NAME

SIFTR — Statistical Information For TCP Research

SYNOPSIS

To load the driver as a module at run-time, run the following command as root:

kldload siftr

Alternatively, to load the driver as a module at boot time, add the following line into the loader.conf(5) file:

siftr_load="YES"

DESCRIPTION

The SIFTR (Statistical Information For TCP Research) kernel module logs a range of statistics on active TCP connections to a log file. It provides the ability to make highly granular measurements of TCP connection state, aimed at system administrators, developers and researchers.

CFLAGS+=-DSIFTR_IPV6

enable_time_secs=1238556193    enable_time_usecs=462104 \
siftrver=1.2.2    hz=1000    tcp_rtt_scale=32 \
sysname=FreeBSD    sysver=604000    ipmode=4

o,0xbec491a5,1238556193.463551,172.16.7.28,22,172.16.2.5,55931, \
1073725440,172312,6144,66560,66608,8,1,4,1448,936,1,996,255, \
33304,208,66608,0,208,0

disable_time_secs=1238556197    disable_time_usecs=933607 \
num_inbound_tcp_pkts=356    num_outbound_tcp_pkts=627 \
total_tcp_pkts=983    num_inbound_skipped_pkts_malloc=0 \
num_outbound_skipped_pkts_malloc=0    num_inbound_skipped_pkts_mtx=0 \
num_outbound_skipped_pkts_mtx=0    num_inbound_skipped_pkts_tcb=0 \
num_outbound_skipped_pkts_tcb=0    num_inbound_skipped_pkts_icb=0 \
num_outbound_skipped_pkts_icb=0    total_skipped_tcp_pkts=0 \
flow_list=172.16.7.28;22-172.16.2.5;55931,

IMPLEMENTATION NOTES

SIFTR hooks into the network stack using the pfil(9) interface. In its current incarnation, it hooks into the AF_INET/AF_INET6 (IPv4/IPv6) pfil(9) filtering points, which means it sees packets at the IP layer of the network stack. This means that TCP packets inbound to the stack are intercepted before they have been processed by the TCP layer. Packets outbound from the stack are intercepted after they have been processed by the TCP layer.

The diagram below illustrates how SIFTR inserts itself into the stack.

----------------------------------
           Upper Layers
----------------------------------
    ^                       |
    |                       |
    |                       |
    |                       v
 TCP in                  TCP out
----------------------------------
    ^                      |
    |________     _________|
            |     |
            |     v
           ---------
           | SIFTR |
           ---------
            ^     |
    ________|     |__________
    |                       |
    |                       v
IPv{4/6} in            IPv{4/6} out
----------------------------------
    ^                       |
    |                       |
    |                       v
Layer 2 in             Layer 2 out
----------------------------------
          Physical Layer
----------------------------------

SIFTR uses the alq(9) interface to manage writing data to disk.

At first glance, you might mistakenly think that SIFTR extracts information from individual TCP packets. This is not the case. SIFTR uses TCP packet events (inbound and outbound) for each TCP flow originating from the system to trigger a dump of the state of the TCP control block for that flow. With the PPL set to 1, we are in effect sampling each TCP flow's control block state as frequently as flow packets enter/leave the system. For example, setting PPL to 2 halves the sampling rate i.e., every second flow packet (inbound OR outbound) causes a dump of the control block state.

The distinction between interrogating individual packets versus interrogating the control block is important, because SIFTR does not remove the need for packet capturing tools like tcpdump(1). SIFTR allows you to correlate and observe the cause-and-affect relationship between what you see on the wire (captured using a tool like tcpdump(1)) and changes in the TCP control block corresponding to the flow of interest. It is therefore useful to use SIFTR and a tool like tcpdump(1) to gather the necessary data to piece together the complete picture. Use of either tool on its own will not be able to provide all of the necessary data.

As a result of needing to interrogate the TCP control block, certain packets during the lifecycle of a connection are unable to trigger a SIFTR log message. The initial handshake takes place without the existence of a control block and the final ACK is exchanged when the connection is in the TIMEWAIT state.

SIFTR was designed to minimise the delay introduced to packets traversing the network stack. This design called for a highly optimised and minimal hook function that extracted the minimal details necessary whilst holding the packet up, and passing these details to another thread for actual processing and logging.

This multithreaded design does introduce some contention issues when accessing the data structure shared between the threads of operation. When the hook function tries to place details in the structure, it must first acquire an exclusive lock. Likewise, when the processing thread tries to read details from the structure, it must also acquire an exclusive lock to do so. If one thread holds the lock, the other must wait before it can obtain it. This does introduce some additional bounded delay into the kernel's packet processing code path.

In some cases (e.g., low memory, connection termination), TCP packets that enter the SIFTR pfil(9) hook function will not trigger a log message to be generated. SIFTR refers to this outcome as a "skipped packet". Note that SIFTR always ensures that packets are allowed to continue through the stack, even if they could not successfully trigger a data log message. SIFTR will therefore not introduce any packet loss for TCP/IP packets traversing the network stack.

EXAMPLES

To enable the module's operations, run the following command as root: sysctl net.inet.siftr.enabled=1

To change the granularity of log messages such that 1 log message is generated for every 10 TCP packets per connection, run the following command as root: sysctl net.inet.siftr.ppl=10

To change the log file location to /tmp/siftr.log, run the following command as root: sysctl net.inet.siftr.logfile=/tmp/siftr.log

SEE ALSO

tcpdump(1), tcp(4), sysctl(8), alq(9), pfil(9)

ACKNOWLEDGEMENTS

Development of this software was made possible in part by grants from the Cisco University Research Program Fund at Community Foundation Silicon Valley, and the FreeBSD Foundation.

HISTORY

SIFTR first appeared in FreeBSD 7.4 and FreeBSD 8.2.

SIFTR was first released in 2007 by Lawrence Stewart and James Healy whilst working on the NewTCP research project at Swinburne University of Technology's Centre for Advanced Internet Architectures, Melbourne, Australia, which was made possible in part by a grant from the Cisco University Research Program Fund at Community Foundation Silicon Valley. More details are available at:

http://caia.swin.edu.au/urp/newtcp/

Work on SIFTR v1.2.x was sponsored by the FreeBSD Foundation as part of the "Enhancing the FreeBSD TCP Implementation" project 2008-2009. More details are available at:

http://www.freebsdfoundation.org/

http://caia.swin.edu.au/freebsd/etcp09/

AUTHORS

SIFTR was written by Lawrence Stewart <lstewart@FreeBSD.org> and James Healy <jimmy@deefa.com>.

This manual page was written by Lawrence Stewart <lstewart@FreeBSD.org>.

BUGS

Current known limitations and any relevant workarounds are outlined below:

• The internal queue used to pass information between the threads of operation is currently unbounded. This allows SIFTR to cope with bursty network traffic, but sustained high packet-per-second traffic can cause exhaustion of kernel memory if the processing thread cannot keep up with the packet rate.
• If using SIFTR on a machine that is also running other modules utilising the pfil(9) framework e.g. dummynet(4), ipfw(8), pf(4), the order in which you load the modules is important. You should kldload the other modules first, as this will ensure TCP packets undergo any necessary manipulations before SIFTR "sees" and processes them.
• There is a known, harmless lock order reversal warning between the pfil(9) mutex and tcbinfo TCP lock reported by witness(4) when SIFTR is enabled in a kernel compiled with witness(4) support.
• There is no way to filter which TCP flows you wish to capture data for. Post processing is required to separate out data belonging to particular flows of interest.
• The module does not detect deletion of the log file path. New log messages will simply be lost if the log file being used by SIFTR is deleted whilst the module is set to use the file. Switching to a new log file using the net.inet.siftr.logfile variable will create the new file and allow log messages to begin being written to disk again. The new log file path must differ from the path to the deleted file.
• The hash table used within the code is sized to hold 65536 flows. This is not a hard limit, because chaining is used to handle collisions within the hash table structure. However, we suspect (based on analogies with other hash table performance data) that the hash table look up performance (and therefore the module's packet processing performance) will degrade in an exponential manner as the number of unique flows handled in a module enable/disable cycle approaches and surpasses 65536.
• There is no garbage collection performed on the flow hash table. The only way currently to flush it is to disable SIFTR.
• The PPL variable applies to packets that make it into the processing thread, not total packets received in the hook function. Packets are skipped before the PPL variable is applied, which means there may be a slight discrepancy in the triggering of log messages. For example, if PPL was set to 10, and the 8th packet since the last log message is skipped, the 11th packet will actually trigger the log message to be generated. This is discussed in greater depth in CAIA technical report 070824A.
• At the time of writing, there was no simple way to hook into the TCP layer to intercept packets. SIFTR's use of IP layer hook points means all IP traffic will be processed by the SIFTR pfil(9) hook function, which introduces minor, but nonetheless unnecessary packet delay and processing overhead on the system for non-TCP packets as well. Hooking in at the IP layer is also not ideal from the data gathering point of view. Packets traversing up the stack will be intercepted and cause a log message generation BEFORE they have been processed by the TCP layer, which means we cannot observe the cause-and-affect relationship between inbound events and the corresponding TCP control block as precisely as could be. Ideally, SIFTR should intercept packets after they have been processed by the TCP layer i.e. intercept packets coming up the stack after they have been processed by tcp_input(), and intercept packets coming down the stack after they have been processed by tcp_output(). The current code still gives satisfactory granularity though, as inbound events tend to trigger outbound events, allowing the cause-and-effect to be observed indirectly by capturing the state on outbound events as well.
• The "inflight bytes" value logged by SIFTR does not take into account bytes that have been SACK'ed by the receiving host.
• Packet hash generation does not currently work for IPv6 based TCP packets.
• Compressed notation is not used for IPv6 address representation. This consumes more bytes than is necessary in log output.