Kernel interface

ipsec is controlled by a key management and policy engine, that reside in the operating system kernel. Key management is the process of associating keys with security associations, also know as SAs. Policy management dictates when new security associations created or destroyed.

The key management engine can be accessed from userland by using PF_KEY sockets. The PF_KEY socket API is defined in RFC2367.

The policy engine is controlled by an extension to the PF_KEY API, setsockopt(2) operations, and sysctl(3) interface. The kernel implements an extended version of the PF_KEY interface and allows the programmer to define IPsec policies which are similar to the per-packet filters. The setsockopt(2) interface is used to define per-socket behavior, and sysctl(3) interface is used to define host-wide default behavior.

The kernel code does not implement a dynamic encryption key exchange protocol such as IKE (Internet Key Exchange). Key exchange protocols are beyond what is necessary in the kernel and should be implemented as daemon processes which call the APIs.

Policy management

IPsec policies can be managed in one of two ways, either by configuring per-socket policies using the setsockopt(2) system calls, or by configuring kernel level packet filter-based policies using the PF_KEY interface, via the setkey(8) you can define IPsec policies against packets using rules similar to packet filtering rules. Refer to setkey(8) on how to use it.

Depending on the socket's address family, IPPROTO_IP or IPPROTO_IPV6 transport level and IP_IPSEC_POLICY or IPV6_IPSEC_POLICY socket options may be used to configure per-socket security policies. A properly-formed IPsec policy specification structure can be created using ipsec_set_policy(3) function and used as socket option value for the setsockopt(2) call.

When setting policies using the setkey(8) command, the “default” option instructs the system to use its default policy, as explained below, for processing packets. The following sysctl variables are available for configuring the system's IPsec behavior. The variables can have one of two values. A 1 means “use”, which means that if there is a security association then use it but if there is not then the packets are not processed by IPsec. The value 2 is synonymous with “require”, which requires that a security association must exist for the packets to move, and not be dropped. These terms are defined in ipsec_set_policy(8).

If the kernel does not find a matching, system wide, policy then the default value is applied. The system wide default policy is specified by the following sysctl(8) variables. 0 means “discard” which asks the kernel to drop the packet. 1 means “none”.

Miscellaneous sysctl variables

When the ipsec protocols are configured for use, all protocols are included in the system. To selectively enable/disable protocols, use sysctl(8).

In addition the following variables are accessible via sysctl(8), for tweaking the kernel's IPsec behavior:

The variables are interpreted as follows:

ipsec.ah_cleartos

If set to non-zero, the kernel clears the type-of-service field in the IPv4 header during AH authentication data computation. This variable is used to get current systems to inter-operate with devices that implement RFC1826 AH. It should be set to non-zero (clear the type-of-service field) for RFC2402 conformance.

ipsec.ah_offsetmask

During AH authentication data computation, the kernel will include a 16bit fragment offset field (including flag bits) in the IPv4 header, after computing logical AND with the variable. The variable is used for inter-operating with devices that implement RFC1826 AH. It should be set to zero (clear the fragment offset field during computation) for RFC2402 conformance.

ipsec.dfbit

This variable configures the kernel behavior on IPv4 IPsec tunnel encapsulation. If set to 0, the DF bit on the outer IPv4 header will be cleared while 1 means that the outer DF bit is set regardless from the inner DF bit and 2 indicates that the DF bit is copied from the inner header to the outer one. The variable is supplied to conform to RFC2401 chapter 6.1.

ipsec.ecn

If set to non-zero, IPv4 IPsec tunnel encapsulation/decapsulation behavior will be friendly to ECN (explicit congestion notification), as documented in draft-ietf-ipsec-ecn-02.txt. gif(4) talks more about the behavior.

ipsec.debug

If set to non-zero, debug messages will be generated via syslog(3).

ipsec.natt_cksum_policy

Controls how the kernel handles TCP and UDP checksums when ESP in UDP encapsulation is used for IPsec transport mode. If set to a non-zero value, the kernel fully recomputes checksums for inbound TCP segments and UDP datagrams after they are decapsulated and decrypted. If set to 0 and original addresses were configured for corresponding SA by the IKE daemon, the kernel incrementally recomputes checksums for inbound TCP segments and UDP datagrams. If addresses were not configured, the checksums are ignored.

ipsec.check_policy_history

Enables strict policy checking for inbound packets. By default, inbound security policies check that packets handled by IPsec have been decrypted and authenticated. If this variable is set to a non-zero value, each packet handled by IPsec is checked against the history of IPsec security associations. The IPsec security protocol, mode, and SA addresses must match.

Variables under the net.inet6.ipsec6 tree have similar meanings to those described above.