Stenographer – A Packet Capture Solution Which Aims To Quickly Spool All Packets To Disk, Then Provide Simple, Fast Access To Subsets Of Those Packets

Stenographer - A Packet Capture Solution Which Aims To Quickly Spool All Packets To Disk, Then Provide Simple, Fast Access To Subsets Of Those Packets

Stenographer is a full-packet-seize utility for buffering packets to disk for intrusion detection and incident response functions. It supplies a excessive-efficiency implementation of NIC-to-disk packet writing, handles deleting these information as disk fills up, and supplies strategies for studying again particular units of packets rapidly and simply.

It is designed to:

  • Write packets to disk, in a short time (~10Gbps on multi-core, multi-disk machines)
  • Store as a lot historical past as it will probably (managing disk utilization, storing longer durations when site visitors slows, then deleting the oldest packets when it hits disk limits)
  • Read a really small proportion (<1%) of packets from disk primarily based on analyst wants

It is NOT designed for:

  • Complex packet processing (TCP stream reassembly, and so forth)
  • It’s quick as a result of it doesn’t do that.  Even with the very minimal, single-cross processing of packets we do, processing ~1Gbps for indexing alone can take >75% of a single core.
  • Processing the info by studying it again from disk additionally doesn’t work:  see subsequent bullet level.
  • Reading again giant quantities of packets (> 1% of packets written)
  • The key idea right here is that disk reads compete with disk writes… you possibly can write at 90% of disk velocity, however that solely offers you 10% of your disk’s time for studying.  Also, we’re writing extremely sequential knowledge, which disks are excellent at doing rapidly, and customarily studying again sparse knowledge with a number of seeks, which disks do slowly.

For additional studying, take a look at for a dialogue of stenographer’s design, or learn for tips on how to set up stenographer on a machine.


Query Language
A consumer requests packets from stenographer by specifying them with a quite simple question language. This language is a straightforward subset of BPF, and consists of the primitives:

host          # Single IP tackle (hostnames not allowed)
internet         # Network with CIDR
internet 1.0.0.Zero masks  # Network with masks
port 80               # Port quantity (UDP or TCP)
ip proto 6            # IP protocol quantity 6
icmp                  # equal to 'ip proto 1'
tcp                   # equal to 'ip proto 6'
udp                   # equal to 'ip proto 17'

# Stenographer-specific time additions:
earlier than 2012-11-03T11:05:00Z      # Packets earlier than a selected time (UTC)
after 2012-11-03T11:05:00-07:00  # Packets after a selected time (with TZ)
earlier than 45m in the past        # Packets earlier than a relative time
earlier than 3h in the past         # Packets after a relative time

NOTE: Relative occasions have to be measured in integer values of hours or minutes as demonstrated above.
Primitives could be mixed with and/&& and with or/||, which have equal precendence and consider left-to-proper. Parens will also be used to group.

(udp and port 514) or (tcp and port 8080)

Stenoread CLI
The stenoread command line script automates pulling packets from Stenographer and presenting them in a usable format to analysts. It requests uncooked packets from stenographer, then runs them via tcpdump to supply a extra full-featured formatting/filtering expertise. The first argument to stenoread is a stenographer question (see ‘Query Language’ above). All different arguments are handed to tcpdump. For instance:

# Request all packets from IP 1.2.3.four port 6543, then do further filtering by
# TCP flag, which typical stenographer doesn't help.
$ stenoread 'host 1.2.3.four and port 6543' 'tcp[tcpflags] & tcp-push != 0'

# Request packets on port 8765, disabling IP decision (-n) and exhibiting
# hyperlink-stage headers (-e) when printing them out.
$ stenoread 'port 8765' -n -e

# Request packets for any IPs within the vary, writing them
# out to an area PCAP file to allow them to be opened in Wireshark.
$ stenoread 'internet' -w /tmp/output_for_wireshark.pcap

To obtain the supply code, set up Go regionally, then run:

$ go get

Go will deal with downloading and putting in all Go libraries that stenographer is dependent upon. To construct stenotype, go into the stenotype listing and run make. You might have to put in the next Ubuntu packages (or their equivalents on different Linux distros):

  • libaio-dev
  • libleveldb-dev
  • libsnappy-dev
  • g++
  • libcap2-bin
  • libseccomp-dev

Obligatory Fine Print
This just isn’t an official Google product (experimental or in any other case), it’s simply code that occurs to be owned by Google.
This code just isn’t supposed (or used) to observe Google’s customers. Its goal is to extend safety on our networks by augmenting our inner monitoring capabilities.


Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.