NAME

python-ipfix - python-ipfix Documentation

IPFIX implementation for Python 3.3.

This module provides a Python interface to IPFIX message streams, and provides tools for building IPFIX Exporting and Collecting Processes. It handles message framing and deframing, encoding and decoding IPFIX data records using templates, and a bridge between IPFIX ADTs and appropriate Python data types.

Before using any of the functions of this module, it is necessary to populate the information model with Information Elements. ipfix.ie.use_iana_default() populates the default IANA IPFIX Information Element Registry shipped with the module; this is the current registry as of release time. ipfix.ie.use_5103_default() populates the reverse counterpart IEs as in RFC 5103. The module also supports the definition of enterprise-specific Information Elements via ipfix.ie.for_spec() and ipfix.ie.use_specfile(); see ipfix.ie for more.

For reading and writing of records to IPFIX message streams with automatic message boundary management, see the ipfix.reader and ipfix.writer modules, respectively. For manual reading and writing of messages, see ipfix.message. In any case, exporters will need to define templates; see ipfix.template.

This module is copyright 2013 Brian Trammell. It is made available under the terms of the GNU Lesser General Public License, version 3 or, at your option, any later version.

Reference documentation for each module is found in the subsections below.

MODULE IPFIX.TYPES

Implementation of IPFIX abstract data types (ADT) and mappings to Python types.

Maps each IPFIX ADT to the corresponding Python type, as below:

Though client code generally will not use this module directly, it defines how each IPFIX abstract data type will be represented in Python, and the concrete IPFIX representation of each type. Type methods operate on buffers, as used internally by the ipfix.message.MessageBuffer class, so we'll create one to illustrate encoding and decoding:

>>> import ipfix.types
>>> buf = memoryview(bytearray(16))

Each of the encoding methods returns the offset into the buffer of the first byte after the encoded value; since we're always encoding to the beginning of the buffer in this example, this is equivalent to the length. We use this to bound the encoded value on subsequent decode.

Integers are represented by the python int type:

>>> unsigned32 = ipfix.types.for_name("unsigned32")
>>> length = unsigned32.encode_single_value_to(42, buf, 0)
>>> buf[0:length].tolist()
[0, 0, 0, 42]
>>> unsigned32.decode_single_value_from(buf, 0, length)
42

...floats by the float type, with the usual caveats about precision:

>>> float32 = ipfix.types.for_name("float32")
>>> length = float32.encode_single_value_to(42.03579, buf, 0)
>>> buf[0:length].tolist()
[66, 40, 36, 166]
>>> float32.decode_single_value_from(buf, 0, length)
42.035789489746094

...strings by the str type, encoded as UTF-8:

>>> string = ipfix.types.for_name("string")
>>> length = string.encode_single_value_to("Grüezi", buf, 0)
>>> buf[0:length].tolist()
[71, 114, 195, 188, 101, 122, 105]
>>> string.decode_single_value_from(buf, 0, length)
'Grüezi'

...addresses as the IPv4Address and IPv6Address types in the ipaddress module:

>>> from ipaddress import ip_address
>>> ipv4Address = ipfix.types.for_name("ipv4Address")
>>> length = ipv4Address.encode_single_value_to(ip_address("198.51.100.27"), buf, 0)
>>> buf[0:length].tolist()
[198, 51, 100, 27]
>>> ipv4Address.decode_single_value_from(buf, 0, length)
IPv4Address('198.51.100.27')
>>> ipv6Address = ipfix.types.for_name("ipv6Address")
>>> length = ipv6Address.encode_single_value_to(ip_address("2001:db8::c0:ffee"), buf, 0)
>>> buf[0:length].tolist()
[32, 1, 13, 184, 0, 0, 0, 0, 0, 0, 0, 0, 0, 192, 255, 238]
>>> ipv6Address.decode_single_value_from(buf, 0, length)
IPv6Address('2001:db8::c0:ffee')

...and the timestamps of various precision as a python datetime, encoded as per RFC5101bis:

>>> from datetime import datetime
>>> from datetime import timezone
>>> dtfmt = "%Y-%m-%d %H:%M:%S.%f"
>>> dt = datetime.strptime("2013-06-21 14:00:03.456789", dtfmt)

dateTimeSeconds truncates microseconds:

>>> dateTimeSeconds = ipfix.types.for_name("dateTimeSeconds")
>>> length = dateTimeSeconds.encode_single_value_to(dt, buf, 0)
>>> buf[0:length].tolist()
[81, 196, 92, 99]
>>> dateTimeSeconds.decode_single_value_from(buf, 0, length).strftime(dtfmt)
'2013-06-21 14:00:03.000000'

dateTimeMilliseconds truncates microseconds to the nearest millisecond:

>>> dateTimeMilliseconds = ipfix.types.for_name("dateTimeMilliseconds")
>>> length = dateTimeMilliseconds.encode_single_value_to(dt, buf, 0)
>>> buf[0:length].tolist()
[0, 0, 1, 63, 103, 8, 228, 128]
>>> dateTimeMilliseconds.decode_single_value_from(buf, 0, length).strftime(dtfmt)
'2013-06-21 14:00:03.456000'

dateTimeMicroseconds exports microseconds fully in NTP format:

>>> dateTimeMicroseconds = ipfix.types.for_name("dateTimeMicroseconds")
>>> length = dateTimeMicroseconds.encode_single_value_to(dt, buf, 0)
>>> buf[0:length].tolist()
[81, 196, 92, 99, 116, 240, 32, 0]
>>> dateTimeMicroseconds.decode_single_value_from(buf, 0, length).strftime(dtfmt)
'2013-06-21 14:00:03.456789'

dateTimeNanoseconds is also supported, but is identical to dateTimeMicroseconds, as the datetime class in Python only supports microsecond-level timing.

MODULE IPFIX.IE

IESpec-based interface to IPFIX information elements, and interface to use the default IPFIX IANA Information Model

An IESpec is a string representation of an IPFIX information element, including all the information required to define it, as documented in Section 9 of http://tools.ietf.org/html/draft-ietf-ipfix-ie-doctors. It has the format:

To specify a new Information Element, a complete IESpec must be passed to for_spec():

>>> import ipfix.ie
>>> e = ipfix.ie.for_spec("myNewInformationElement(35566/1)<string>")
>>> e
InformationElement('myNewInformationElement', 35566, 1, ipfix.types.for_name('string'), 65535)

The string representation of an InformationElement is its IESpec:

>>> str(e)
'myNewInformationElement(35566/1)<string>[65535]'

To get an Information Element already specified, an incomplete specification can be passed; a name or number is enough:

>>> ipfix.ie.use_iana_default()
>>> ipfix.ie.use_5103_default()
>>> str(ipfix.ie.for_spec("octetDeltaCount"))
'octetDeltaCount(0/1)<unsigned64>[8]'
>>> str(ipfix.ie.for_spec("(2)"))
'packetDeltaCount(0/2)<unsigned64>[8]'

Reduced-length encoding and fixed-length sequence types are supported by the for_length method; this is used internally by templates.

>>> str(e.for_length(32))
'myNewInformationElement(35566/1)<string>[32]'

An Information Element object can also be used to translate between native Python and string representations of an Information Element value:

>>> ipfix.ie.for_spec("sourceIPv4Address").parse("192.0.2.19")
IPv4Address('192.0.2.19')
>>> from datetime import datetime
>>> ipfix.ie.for_spec("flowEndMilliseconds").unparse(datetime(2013,6,21,14))
'2013-06-21 14:00:00.000'

Most client code will only need the use_iana_default(), use_5103_default(), and use_specfile() functions; client code using tuple interfaces will need spec_list() as well.

MODULE IPFIX.TEMPLATE

Representation of IPFIX templates. Provides template-based packing and unpacking of data in IPFIX messages.

For reading, templates are handled internally. For writing, use from_ielist() to create a template.

See ipfix.message for examples.

MODULE IPFIX.MESSAGE

Provides the MessageBuffer class for encoding and decoding IPFIX Messages.

This interface allows direct control over Messages; for reading or writing records automatically from/to streams, see ipfix.reader and ipfix.writer, respectively.

To create a message buffer:

>>> import ipfix.message
>>> msg = ipfix.message.MessageBuffer()
>>> msg
<MessageBuffer domain 0 length 0>

To prepare the buffer to write records:

>>> msg.begin_export(8304)
>>> msg
<MessageBuffer domain 8304 length 16 (writing)>

Note that the buffer grows to contain the message header.

To write records to the buffer, first you'll need a template:

>>> import ipfix.ie
>>> ipfix.ie.use_iana_default()
>>> import ipfix.template
>>> tmpl = ipfix.template.from_ielist(256, 
...        ipfix.ie.spec_list(("flowStartMilliseconds",
...                            "sourceIPv4Address",
...                            "destinationIPv4Address",
...                            "packetDeltaCount")))
>>> tmpl
<Template ID 256 count 4 scope 0>

To add the template to the message:

>>> msg.add_template(tmpl)
>>> msg
<MessageBuffer domain 8304 length 40 (writing set 2)>

Note that MessageBuffer.add_template() exports the template when it is written by default, and that the current set ID is 2 (template set).

Now, a set must be created to add records to the message; the set ID must match the ID of the template. MessageBuffer automatically uses the template matching the set ID for record encoding.

>>> msg.export_ensure_set(256)
>>> msg
<MessageBuffer domain 8304 length 44 (writing set 256)>

Records can be added to the set either as dictionaries keyed by IE name:

>>> from datetime import datetime
>>> from ipaddress import ip_address
>>> rec = { "flowStartMilliseconds" : datetime.strptime("2013-06-21 14:00:00", 
...                                       "%Y-%m-%d %H:%M:%S"),
...         "sourceIPv4Address" : ip_address("10.1.2.3"),
...         "destinationIPv4Address" : ip_address("10.5.6.7"),
...         "packetDeltaCount" : 27 }
>>> msg.export_namedict(rec)
>>> msg
<MessageBuffer domain 8304 length 68 (writing set 256)>

or as tuples in template order:

>>> rec = (datetime.strptime("2013-06-21 14:00:02", "%Y-%m-%d %H:%M:%S"),
...        ip_address("10.8.9.11"), ip_address("10.12.13.14"), 33)
>>> msg.export_tuple(rec)
>>> msg
<MessageBuffer domain 8304 length 92 (writing set 256)>

Variable-length information elements will be encoded using the native length of the passed value:

>>> ipfix.ie.for_spec("myNewInformationElement(35566/1)<string>")
InformationElement('myNewInformationElement', 35566, 1, ipfix.types.for_name('string'), 65535)
>>> tmpl = ipfix.template.from_ielist(257, 
...        ipfix.ie.spec_list(("flowStartMilliseconds",
...                            "myNewInformationElement")))
>>> msg.add_template(tmpl)
>>> msg.export_ensure_set(257)
>>> msg
<MessageBuffer domain 8304 length 116 (writing set 257)>
>>> rec = { "flowStartMilliseconds" : datetime.strptime("2013-06-21 14:00:04", 
...                                   "%Y-%m-%d %H:%M:%S"),
...         "myNewInformationElement" : "Grüezi, Y'all" }
>>> msg.export_namedict(rec)
>>> msg
<MessageBuffer domain 8304 length 139 (writing set 257)>

Attempts to write past the end of the message (set via the mtu parameter, default 65535) result in EndOfMessage being raised.

Messages can be written to a stream using MessageBuffer.write_message(), or dumped to a byte array for transmission using MessageBuffer.to_bytes(). The message must be reset before starting to write again.

>>> b = msg.to_bytes()
>>> msg.begin_export()
>>> msg 
<MessageBuffer domain 8304 length 16 (writing)>

Reading happens more or less in reverse. To begin, a message is read from a byte array using MessageBuffer.from_bytes(), or from a stream using MessageBuffer.read_message().

>>> msg.from_bytes(b)
>>> msg
<MessageBuffer domain 8304 length 139 (deframed 4 sets)>

Both of these methods scan the message in advance to find the sets within the message. The records within these sets can then be accessed by iterating over the message. As with export, the records can be accessed as a dictionary mapping IE names to values or as tuples. The dictionary interface is designed for general IPFIX processing applications, such as collectors accepting many types of data, or diagnostic tools for debugging IPFIX export:

>>> for rec in msg.namedict_iterator():
...    print(sorted(rec.items()))
...
[('destinationIPv4Address', IPv4Address('10.5.6.7')), ('flowStartMilliseconds', datetime.datetime(2013, 6, 21, 14, 0)), ('packetDeltaCount', 27), ('sourceIPv4Address', IPv4Address('10.1.2.3'))]
[('destinationIPv4Address', IPv4Address('10.12.13.14')), ('flowStartMilliseconds', datetime.datetime(2013, 6, 21, 14, 0, 2)), ('packetDeltaCount', 33), ('sourceIPv4Address', IPv4Address('10.8.9.11'))]
[('flowStartMilliseconds', datetime.datetime(2013, 6, 21, 14, 0, 4)), ('myNewInformationElement', "Grüezi, Y'all")]

The tuple interface for reading messages is designed for applications with a specific internal data model. It can be much faster than the dictionary interface, as it skips decoding of IEs not requested by the caller, and can skip entire sets not containing all the requested IEs. Requested IEs are specified as an ipfix.ie.InformationElementList instance, from ie.spec_list():

>>> ielist = ipfix.ie.spec_list(["flowStartMilliseconds", "packetDeltaCount"])
>>> for rec in msg.tuple_iterator(ielist):
...     print(rec)
...
(datetime.datetime(2013, 6, 21, 14, 0), 27)
(datetime.datetime(2013, 6, 21, 14, 0, 2), 33)

Notice that the variable-length record written to the message are not returned by this iterator, since that record doesn't include a packetDeltaCount IE. The record is, however, still there:

>>> ielist = ipfix.ie.spec_list(["myNewInformationElement"])
>>> for rec in msg.tuple_iterator(ielist):
...     print(rec)
...
("Grüezi, Y'all",)

MODULE IPFIX.READER

Interface to read IPFIX Messages from a stream.

MODULE IPFIX.WRITER

AUTHOR

Brian Trammell

COPYRIGHT

2013-2022, Brian Trammell