Stenographer is a full-packet-seize utility for buffering packets to disk for intrusion detection and incident response functions. It supplies a excessive-efficiency implementation of NIC-to-disk packet writing, handles deleting these information as disk fills up, and supplies strategies for studying again particular units of packets rapidly and simply.
It is designed to:
- Write packets to disk, in a short time (~10Gbps on multi-core, multi-disk machines)
- Store as a lot historical past as it will probably (managing disk utilization, storing longer durations when site visitors slows, then deleting the oldest packets when it hits disk limits)
- Read a really small proportion (<1%) of packets from disk primarily based on analyst wants
It is NOT designed for:
- Complex packet processing (TCP stream reassembly, and so forth)
- It’s quick as a result of it doesn’t do that. Even with the very minimal, single-cross processing of packets we do, processing ~1Gbps for indexing alone can take >75% of a single core.
- Processing the info by studying it again from disk additionally doesn’t work: see subsequent bullet level.
- Reading again giant quantities of packets (> 1% of packets written)
- The key idea right here is that disk reads compete with disk writes… you possibly can write at 90% of disk velocity, however that solely offers you 10% of your disk’s time for studying. Also, we’re writing extremely sequential knowledge, which disks are excellent at doing rapidly, and customarily studying again sparse knowledge with a number of seeks, which disks do slowly.
A consumer requests packets from stenographer by specifying them with a quite simple question language. This language is a straightforward subset of BPF, and consists of the primitives:
host 22.214.171.124 # Single IP tackle (hostnames not allowed) internet 126.96.36.199/8 # Network with CIDR internet 1.0.0.Zero masks 255.255.255.0 # Network with masks port 80 # Port quantity (UDP or TCP) ip proto 6 # IP protocol quantity 6 icmp # equal to 'ip proto 1' tcp # equal to 'ip proto 6' udp # equal to 'ip proto 17' # Stenographer-specific time additions: earlier than 2012-11-03T11:05:00Z # Packets earlier than a selected time (UTC) after 2012-11-03T11:05:00-07:00 # Packets after a selected time (with TZ) earlier than 45m in the past # Packets earlier than a relative time earlier than 3h in the past # Packets after a relative time
NOTE: Relative occasions have to be measured in integer values of hours or minutes as demonstrated above.
Primitives could be mixed with and/&& and with or/||, which have equal precendence and consider left-to-proper. Parens will also be used to group.
(udp and port 514) or (tcp and port 8080)
The stenoread command line script automates pulling packets from Stenographer and presenting them in a usable format to analysts. It requests uncooked packets from stenographer, then runs them via tcpdump to supply a extra full-featured formatting/filtering expertise. The first argument to stenoread is a stenographer question (see ‘Query Language’ above). All different arguments are handed to tcpdump. For instance:
# Request all packets from IP 1.2.3.four port 6543, then do further filtering by # TCP flag, which typical stenographer doesn't help. $ stenoread 'host 1.2.3.four and port 6543' 'tcp[tcpflags] & tcp-push != 0' # Request packets on port 8765, disabling IP decision (-n) and exhibiting # hyperlink-stage headers (-e) when printing them out. $ stenoread 'port 8765' -n -e # Request packets for any IPs within the vary 188.8.131.52-184.108.40.206, writing them # out to an area PCAP file to allow them to be opened in Wireshark. $ stenoread 'internet 220.127.116.11/24' -w /tmp/output_for_wireshark.pcap
To obtain the supply code, set up Go regionally, then run:
$ go get github.com/google/stenographer
Go will deal with downloading and putting in all Go libraries that
stenographer is dependent upon. To construct
stenotype, go into the
stenotype listing and run
make. You might have to put in the next Ubuntu packages (or their equivalents on different Linux distros):
Obligatory Fine Print
This just isn’t an official Google product (experimental or in any other case), it’s simply code that occurs to be owned by Google.
This code just isn’t supposed (or used) to observe Google’s customers. Its goal is to extend safety on our networks by augmenting our inner monitoring capabilities.