natlog - source-nat logging tool
Firewalls like iptables(1) may offer POSTROUTING
(source network address translation, snat) facilities changing the source
address of a host behind the firewall to the address of the host connected
to the outer world. With snat the following combinations of IP
addresses and port numbers are encountered:
- o
- the IP address and port number used by the host protected by (i.e.,
behind) the firewall initiates a connection to the outer world (the source
host, in this manual page referred to as IPsrc, sport);
- o
- the IP address and port number of the host outside (i.e., before) the
firewall that IPsrc connects to (the destination host, in this
manual page referred to as IPdst, dport);
- o
- the IP address and port number of the host where the firewall has been
installed. This host performs the source natting, and its IP-address and
the port it uses when forwarding IPsrc, sport’s requests to
IPdst, dport are in this manual page referred to as IPfw,
fwport. )
Source natting usually uses sport for fwport, but
fwport may already be in use, in which case the firewalling host must
use another, available port to forward communication from IPsrc,
sport to IPdst, dport.
The general scheme that applies to source natting, therefore,
looks like this:
IPsrc:sport is translated by the firewall to IPfw:fwport;
IPfw:fwport is used when communicating with IPdst:dport.
From the perspective of the destination host the communication originates at
IPfw::fwport and consequently all communication (e.g., incident
reports) sent by the systems administrator maintaining IPdst to
IPfw’s systems administrator will refer to IPfw:fwport,
rather than to IPsrc::sport.
Relating IPfw:fwport to IPsrc:sport is difficult
when merely using the standard log facilities provided by iptables
and natlog was developed to fill in that particular niche.
Natlog provides data about source natting in various forms.
The standard logging mode consists of messages sent to the syslog daemon
(cf., rsyslogd(8)) and/or to the standard output stream showing the
essential characteristics of connections using source natting. Here is an
example of a logged message (log-entries occupy single lines; the
line-breaks below are to enhance readability):
NATLOG: from 1338990672:55588 thru 1338990747:807100 (UTC): tcp
192.168.19.72:4467 (via: 129.125.90.132:4467) to
to 200.49.219.180:443; sent: 802, received: 7669
The values 1338990672:55588 and 1338990747:807100 are time stamps
showing the begin- and end-times in seconds:microseconds of a tcp connection
since the beginning of the epoch (Jan 1, 1970, 0:00 UTC). Natlog offers
the --time option for requesting human-readable time specifications
like Nov 2 13:29:11 rather than time representations using seconds and
micro seconds.
The next value (192.168.19.72:4467) represents
IPsrc::sport. This is followed by 129.125.90.132:4467,
representing IPfw:fwport. The third pair of values
(200.49.219.180:443) represents IPdst:dport.
In this example, host 192.168.19.72, using port
4467, connected to host 200.49.219.180, port 443. To
this latter host the connection appears to have originated from
129.125.90.132 port 4467. The log message allows us to
associate this with the `real’ host and port from which the
connection originated: 192.168.19.72:4467.
The final entries show the number of bytes that were sent by the
source-host (IPsrc) and received from the destination-host
(IPdst).
When natlog is terminated it can no longer track
connections that are still open. If natlog was terminated (by a
SIGINT or SIGTERM signal), then it logs a `terminating’
line, followed by an overview of all (potentially) still open connections.
Those connections are flagged with a trailing ’(EOP)’ (end of
program) log-element, and their end-times show natlog’s
termination time. Incomplete connections show (EXPIRED).
In addition to the standard logs the option --log-data is
available. This option requires the path to a file where information is
logged in tabular form, which can easily be processed by statistical
software like R(1). When specifying this option information will be
appended to an existing file. When the log file does not yet exist it is
created. The first line of the thus written log files names the columns of
the table. The column names are (all on one line):
type, srcNr, srcIP, srcPort, dstNr, dstIP, dstPort,
sent, recvd, begin, end, beginTime, endTime, status
Most column labels will be self-explanatory. Type indicates the
connection type, logged as icmp, tcp, or udp; srcNr and
dstNr are the 32 bit numeric values of, respectively, the source
host’s IP address and the destination host’s IP address (decimal
representations); begin and end are the times in seconds since
the beginning of the epoch, corresponding to the times displayed at,
respectively, beginTime and endTime; status indicates the
status of the logged connection information: ok indicates a connection
that was normally completed; expired indicates that the connection was
recognized, but was not normally completed; eop is used for connections
that were still active by the time natlog terminates. When the status
equals expired, the time entries show the times of receiving the first
and last packets of that connection; when eop, then the end and
endTime entries show natlog’s termination time.
Log entries look like this (each entry occupies one line, header
line and logged data lines are right-aligned):
tcp, 101820608, 192.168.17.6, 48886,
4012145084, 188.121.36.239, 80,
430, 2266, 1517387644, 1517387644,
Jan 31 08:34:04:318340, Jan 31 08:34:04:383170, ok
- o
- conntrack: the `conntrack’-mode. This command can only be
used on platforms using iptables(1) where conntrack(1) has
also been installed. Information about snat connections is obtained from
conntrack(1)’s output. In this mode all, or one of the tcp
(the protocol used by default), udp, and icmp layer four protocols can be
monitored.
- When using the conntrack mode the conntrack program will
report sent and received number of bytes unless the option no-bytes
has been specified.
- Conntrack includes the sizes of the IP headers (usually 20 bytes)
in reported byte counts. Thus, icmp packets are usually reported as
having size 84, even though ping(1) reports a payload of 64 bytes.
Since the actual sizes of IP headers cannot be determined from
conntrack’s output, the sizes reported when using
natlog’s conntrack mode are as reported by conntrack,
and are therefore not corrected for IP header lengths. The option
--conntrack-ip-header-size can be used to correct for the (assumed)
IP header sizes.
- Conntrack can also be used to track all connections, not just the
snat connections. If that’s required omit conntrack’s
option -n, and optionally specify option no-via.
- See also the conntrack-command option.
- o
- indevice outdevice: the `devices’-mode. Here,
indevice is the name of the device behind the firewall: addresses
living behind the indevice are source-natted to the firewall
host’s IP address when passed on to the outdevice.
- Outdevice is the name of the device where source-natted packets are
forwarded to, and from where replies for source-natted hosts living behind
the indevice are received. With this command all, or any
combination of the tcp (the protocol monitored by default), udp, and icmp
layer four protocols can be monitored.
- For example, when specifying the arguments
eth1 eth0
thene eth1 is the device behind the firewall, and eth0 is the
device to where source-natted packets are forwared.
- This command can also be used to track all connections using a single
device, instead of merely tracking snat connections. In that case specify
the same devices for indevice and outdevice, and optionally
specify option no-via. E.g.,
eth0 eth0
- o
- infile in-address in-mask outfile out-address out-mask: the
`tcpdump’-mode. This command can be used to process
tcpdump(1) generated binary files, generated on the source-natting
host. If a source natting host uses interface eth1 behind the
firewall and eth0 to connect to the outside world, then the follow
tcpdump commands produce the required binary files (these commands
will normally be run in the background, hence the trailing &):
tcpdump -wi eth0 /tmp/eth0 &
tcpdump -wi eth1 /tmp/eth1 &
To have natlog process these files, end the tcpdump processes,
and transfer the files /tmp/eth0 and /tmp/eth1 to the host
where natlog has been installed. The required addresses and masks
are shown by the ifconfig(1) command. E.g.,
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 129.125.1.123 netmask 255.255.0.0
broadcast 129.125.255.255
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.1.1 netmask 255.255.255.0
broadcast 192.168.1255
The relevant info is shown in the lines following the interface’s
name: the value following inet is the interface’s IP
address, and the value following netmask is the network’s
mask.
- Combining files and addresses, natlog is run as follows (all on one
line):
natlog /tmp/eth0 129.125.1.123 255.255.0.0
/tmp/eth1 192.168.1.1 255.255.255.0
Instead of fully specifying the netmask, netmaks specifications like /24 are
also accepted. In that case the number following the slash indicates the
number of non-zero bits of the netmask. In practice, each value of the
netmask is either 255 (8 bits are set) or 0 (0 bits are set), and so
255.255.0.0 can also be specified as /16, while 255.255.255.0 can be
specified like /24.
See also section SYSTEMD.
- o
- --config=config-path (-c)
The argument config-path defines the path to natlog’s
configuration file. By default it is /etc/natlog.conf. All
configuration options have defaults, which are used when no configuration
file and no command-line options were provided.
- All options, except for config, help, S, terminate, verbose and
version can also be specified in the configuration file. The
configuration file ignores empty lines and all information on lines
beginning with a hash-mark (#). In the configuration file initial
hyphens should be omitted, and option names may immediately be followed by
a colon. Do not surround option values with quotes. Examples:
stdout
syslog-facility: LOCAL0
Command-line options override configuration file options.
- o
- --conntrack-command=path [options]
The path and options to the conntrack(1) program. By default this is
/usr/sbin/conntrack -p tcp -E -n -o timestamp -e NEW,DESTROY
resulting in:
- - Monitoring the tcp layer four protocol;
- Displaying real-time event logs (-E);
- Only use snat connections (-n);
- Displaying time stamps (-o timestamp);
- Logging all new and destroyed (ended) events (-e
NEW,DESTROY);
- Reporting the number of bytes sent- and received by connections;
- By default tcp is monitored. Other protocols can be configured
using the --protocol option.
- The conntrack program must be available when requesting
natlog’s conntrack command. Layer four protocols
other than tcp, udp and icmp are currently not supported. A subset of the
supported protocols may be requested using conntrack’s -p tcp,
-p udp or -p icmp options.
- When all connections should be logged (not just snat connections) then
omit conntrack’s -n option. See also option --no-via
below.
- Unless option --no-bytes is specified the conntrack program reports
the number of sent and received bytes of connections. Conntrack does so
when the value 1 has been written to
/proc/sys/net/netfilter/nf_conntrack_acct. When natlog
starts, and no-bytes has not been specified then natlog
writes 1 to nf_conntrack_acct.
- Note: when specifying the conntrack-command option in the
configuration file do not sourround the command with quotes.
- o
- --conntrack-device=dev
By default conntrack monitors the information made available at the
/proc/net/nf_conntrack device. When another device should be used,
specify it using this option.
- o
- --conntrack-ip-header-size=size
This option is used to correct for the IP header sizes. By default,
conntrack includes these sizes in reported byte counts. By
specifying this option packet sizes reported by conntrack are
reduced by size. Commonly IP headers consist of 20 bytes (so, to
correct for this specify --conntrack-ip-header-size 20).
- o
- --conntrack-restart=max
If the conntrack process prematurely ends it is restarted at most max
times (these are pure restarts: conntrack’s initial startup
is not counted for this option). By default 10 restarts are allowed.
- o
- --debug
Write additional info to the log file. Currently, --debug writes
information about memory consumption to the log file.
- o
- --help (-h)
Write basic usage information to the standard output stream and
terminate.
- o
- --log=argument
By default natlog forwards log messages about natlog and
connection information to the syslog daemon using the DAEMON
facility with priority NOTICE (see below at the syslog*
options). This is identical to specifying the argument syslog.
- Alternatively, specify the argument off to suppress writing log
messages. Any other argument is interpreted as a path-specification to a
file to receive the log messages: log-messages are appended to existing
files. If the log file does not yet exist it is first created.
- The stdout option is handled independently from the log
option: log messages will appear to the standard output stream if
stdout and log: off are both specified.
- o
- --log-data=path
Path specifies the pathname of the file where information about
observed connections is written in tabular form. If path does not
yet exist it is first created. Refer to the DESCRIPTION section for
information about the format of the generated table. Specify
"" as command-line option if the configuration file
specifies a log data file, but no tabular data should be logged for that
natlog run.
- Like the standard log file (option --log) the log-data file
is not rotated if rotation is requested (cf. option log-rotate).
For statistical analyses rotated log-data files can be concatenated
(usually omitting the first (header) line of rotated log-data files).
- o
- --log-rotate=spec
This option specifies the frequency and the number of log-files that are
rotated. By default log-files are not rotated.
To rotate log-files use time[mhd] or time[mhd]nFiles. The
’time’ specification is a number, which must be followed by
m for minutes, h for hours, and d for days.
nFiles specifies the max. number of rotated files. If only
time[mhd] is specified, then nFiles is set to 1. By default
(or if time or nfiles are specified as zero (0)) log files
are not rotated.
Note: when using rsyslogd(1) for logging (i.e., when
specifying --log syslog, see also option syslog-facility
below), then it is assumed that the syslog daemon or a log-file rotation
program like logrotate(8) handles the log file rotations. Rotating
the log-data file is not affected by specifying --log
syslog.
Natlog uses a built-in minimum rotation interval of 30
seconds.
- o
- --no-bytes
By default log-entries show numbers of sent and received bytes. Specify this
option to omit these statistics from log-entries.
- o
- --no-daemon
By default, natlog runs in the background (a daemon). Natlog
runs as an ordinary program (i.e., in the foreground when the option
no-daemon is specified). When running as a daemon, --stdout
(see below) is suppressed, and --verbose messages (see below) are
sent to the syslog daemon, unless --no-syslog was specified. When
using the tcpdump-mode natlog does not run in the background. In
this case, if no-daemon is omitted a warning message is logged, and
natlog continues as an ordinary program.
- o
- --no-dst
Normally, when snat connections are logged the destination IP addresses and
port numbers are logged as ’dst’ entries in log-data files
and as ’to’ entries in log-files. If these destination items
should be omitted specify no-via as configuration parameter or as
option.
- o
- --no-via
Normally, when snat connections are logged the host handling the address
translations are logged as ’via’ entries in log-files. If
the ’via’ entries should be omitted activate no-via
as configuration parameter or as option.
- o
- --pid-file=path (-p)
When natlog runs in the background, then path is the name of
the path of the file holding the daemon’s process-id. By default
this file is /run/natlog.pid. To end the daemon, simply call
natlog --terminate (or send a SIGINT or
SIGTERM signal to the process id mentioned in the pid-file).
Natlog uses SIGHUP and SIGALRM signals for explicit
rotations of log-files (see options --rotate and
--rotate-data below.
- o
- --protocol=specification (-P)
The protocol(s) to monitor. By default the tcp layer four protocol is
monitored. Currently natlog’s conntrack command can monitor
the tcp, udp, and icmp layer four protocols. Using the protocol
option (note: only one protocol option should be specified) any
subset of these protocols can be selected by specifying a colon-separated
subset of tcp, udp, and icmp (e.g., --protocol udp:tcp). The
specification all can be used to monitor all three protocols (tcp,
udp, and icmp).
- o
- --rotate
When --log has been used then this option forces rotating the log
file independently from the interval specified by --log-rotate.
Natlog uses a built-in minimum rotation interval of 30
seconds.
- o
- --rotate-data
When --log-data has been used then this option forces rotating the
log-data file independently from the interval specified by
--log-rotate. Natlog uses a built-in minimum rotation
interval of 30 seconds.
- o
- -S
Use this option as first option, immediately following the program name,
when starting natlog from a systemd(1) natlog.service
file. See also section SYSTEMD below.
- o
- --stdout (-s)
Syslog-equivalent messages are sent to the standard output. This option is
suppressed when natlog runs as a daemon.
- o
- --syslog-facility=facility
The facility that is used to write the syslog messages to. By default this
is DAEMON. For an overview of facilities and their meanings, see,
e.g., syslog(3). With natlog the facilities DAEMON,
LOCAL0, LOCAL1, LOCAL2, LOCAL3, LOCAL4, LOCAL5, LOCAL6, LOCAL7, and
USER can be used.
When rsyslog filtering is used (see that section below)
then rsyslogd(8) uses that instead of the specified facility.
- o
- --syslog-priority=priority
The priority that is used to write the syslog messages to. By default this
is NOTICE. For an overview of priorities and their meanings, see,
e.g., syslog(3). With natlog all defined priorities can be
used. E.g., EMERG, ALERT, CRIT, ERR, WARNING, NOTICE, INFO and
DEBUG.
- o
- --syslog-tag=tag
When syslog messages are generated they can be provided with a tag,
which can be used to filter natlog’s syslog messages from
the log-files. By default the tag NATLOG is used. See also section
RSYSLOG FILTERING below.
- o
- --terminate
When natlog runs as a daemon, the command natlog --terminate
can be issued to terminate the daemon. By default it reads the
daemon’s process ID from natlog’s pid-file (cf.
option pid-file) /run/natlog.pid). If another pid-file holds
the process ID of the natlog program to terminate then specify the
location of the pid-file to use using a command like
natlog --terminate --pid-file=/path/to/the/pid-file
When the daemon could be terminated 0 is returned. Otherwise, an error
message is displayed and 1 is returned.
- o
- --time=spec (-t)
By default time stamps written by natlog are in raw, numeric form.
E.g.,
NATLOG: From 1338990672:55588 thru 1338990747:807100
These time stamps indicate times in seconds:microseconds since the beginning
of the epoch, January 1, 1970, 0:00 UTC. This option can be used to change
the seconds part of the time stamps to more conventional representations.
Specify raw (the default) for the default representation in seconds
since the epoch;
specify utc for a representation like Jun 6 13:29:11, using
Universal Time Coordinated;
specify local for a representation like Jun 6 13:29:11, using
the local time zone defined by the computer running natlog.
- o
- --ttl=secs[ui] (-T)
time-to-live for received connections. At most two time-to-live
specifications can be provided: for udp/icmp connections a letter u
must be appended to the specified seconds. By default 60u is used.
For tcp connections a letter t must be appended to the specified
seconds. By default 3000t is used. Both time-to-live specifications
may be combined: --ttl 120u1800t specifies a time-to-live of two
minutes for udp/icmp connections and a time-to-live of half an hour for
tcp connections. Time-to-live is not used in conntrack-mode.
- o
- --verbose (-V)
Additional messages about natlog’s mode of operation are sent
to the standard output stream. When natlog runs as a daemon these
messages are sent to the syslog daemon, unless --no-syslog was
specified.
When --verbose is specified twice then all actual
configuration parameters are shown just before natlog starts.
When --verbose is specified more often then natlog
ends after reporting the configuration parameters.
- o
- --version (-v)
Write natlog’s version number to the standard output stream
and terminate.
)
An annoying characteristic of systemd(1) is that
environment variables containing blanks are passed as single arguments to
the program being called by their .service files. As a consequence,
it is very hard to provide an environment variable in, e.g.,
/etc/default/natlog specifying natlog’s arguments: in
practice the number of arguments varies, and so even constructions like
ARG1=value1, ARG2=value2, etc. are awkward at best.
As a stopgap for this unwelcome characteristic of systemd
the option -S is provided. When used it must be specified as
natlog’s first argument. Natlog will then inspect all
remaining arguments, splitting arguments containing blanks into separate
arguments, which are then processed by natlog as intended. Be aware
that, to limit the complexity of the splitting-procedure, it is not
full-proof: double- or single-quote delimited string-arguments will also be
split into separate arguments. Unless filenames themselves containing blanks
are passed as arguments to natlog this limitation is probably not
very serious.
As an example, here is an example of systemd’s
ExecStart specification:
ExecStart=/usr/bin/natlog -S -p ${PIDFILE} ${DAEMON_ARGS}
where DAEMON_ARGS might have been specified in /etc/default/natlog
as
DAEMON_ARGS=--log /tmp/natlog.log --log-data /dev/null conntrack
When using rsyslogd(8) property based filters may be used
to filter syslog messages and write them to a file of your choice. E.g., to
filter messages starting with the syslog message tag (e.g., NATLOG)
use
:syslogtag, isequal, "NATLOG:" /var/log/natlog.log
:syslogtag, isequal, "NATLOG:" stop
Note that the colon is part of the tag, but is not specified with
the syslog-tag option.
This causes all messages having the NATLOG: tag to be
written on /var/log/natlog.log after which they are discarded. More
extensive filtering is also supported, see, e.g.,
http://www.rsyslog.com/doc/rsyslog_conf_filter.html and
http://www.rsyslog.com/doc/property_replacer.html
Examples of natlog activations:
- o
- natlog --no-daemon --no-syslog -s br0 eth0
Natlog remains active as a foreground process, no syslog messages are
written, syslog-equivalent message are written to the standard output.
Natlog uses the pcap library to capture packets from the br0
device, which is active behind the firewall, and to capture packets from
the eth0 device, which is the device to where source-natted
packages are sent.
- o
- natlog conntrack
Depending on the options specified in /etc/natlog.conf (or, if not
available, natlog’s default options) source-natted
connections are obtained from conntrack(1). By default
natlog continues as a daemon process, generating syslog messages
using syslog tags NATLOG:, and containing information about
source-natted connections.
Here is natlog’s default configuration file. Empty
lines and lines starting with hash-marks (#) are ignored. Options adhere to
the following syntax:
option value
Option and value are separated by white space, a colon may be appended to option
names:
# This configuration file shows the default option values.
# Options that are *not* active by default have an extra comment-line
# showing ’not by default:’
# all options and values are case sensitive
# see `man natlog’ for further details
# the path and options of the conntrack program:
# when no filtering options are specified, the tcp
# protocol is monitored
# the default command is shown.
# Note: do not surround the conntrack command specification with quotes
#conntrack-command: /usr/sbin/conntrack -E -n -o timestamp -e NEW,DESTROY
# the device used by conntrack
#conntrack-device: /proc/net/nf_conntrack
# correction for the IP header size
# (standard IP header size is 20 bytes)
#conntrack-ip-header-size: 0
# max. number of conntrack restarts
#conntrack-restart: 10
# write additional info to the log file
# not by default:
#debug
# log messages are written to ’pathname’; use ’log: off’ to suppress log
# messages
# not by default:
#log: pathname
# data file containing tabular logs
# not by default:
#log-data: pathname
# tmespec: time[mhd]nFiles - specification for rotating log-files
# not by default:
#log-rotate: timespec
# do not log the sent/received byte counts (default: counts are logged)
# not by default:
#no-bytes
# do not run as a daemon
# not by default:
#no-daemon
# do not log the destination entries
# not by default:
#no-dst
# do not log the via: entries
# not by default:
#no-via
# the path to the pid-file of natlog’s daemon process
#pid-file: /run/natlog.pid
# the protocols that are scanned with the ’conntrack’ command:
# protocol: all - monitors tcp, udp, icmp
# protocol: udp:tcp - monitors upd and tcp (any non-empty subset,
# possibly including icmp is OK)
#protocol: tcp
# write messages to stdout (ignored by daemons)
# not by default:
#stdout
# the default syslog facility:
#syslog-facility: DAEMON
# the default syslog priority:
#syslog-priority: NOTICE
# the default syslog tag:
#syslog-tag: NATLOG
# the default time specification (alternatives: utc, local):
#time: raw
# ttl: time to live (seconds) for udp/icmp connections
#ttl: 60
# end of the configuration file
- o
- /etc/natlog.conf: default configuration file location;
- o
- /etc/default/natlog: arguments for startup scripts;
- o
- /etc/init.d/natlog: SysV startup script;
- o
- /etc/systemd/system/natlog.service: systemd startup script (calling
/etc/init.d/natlog).
conntrack(1), ifconfig(1), iptables(1),
logrotate(8), pcap-filter(7), ping(1), R(1),
rsyslogd(8), syslog(3), systemd(1),
tcpdump(1)
Natlog currently can process tcp, udp and icmp layer four
protocols.
Frank B. Brokken (f.b.brokken@rug.nl).