IPsec provides two mechanisms for protecting data. The Authentication Header (AH) provides strong integrity, replay protection, and data authentication. AH protects as much of the IP datagram as it can. AH cannot protect fields that change nondeterministically between sender and receiver.

The Encapsulating Security Payload (ESP) provides confidentiality over what it encapsulates, as well as the services that AH provides, but only over that which it encapsulates. ESP's authentication services are optional, which allow ESP and AH to be used together on the same datagram without redundancy.

Authentication and encryption algorithms are used for IPsec. Authentication algorithms produce an integrity checksum value or "digest" based on the data and a key. The size of both the digest and the key are described in authentication algorithm pages. See authmd5h(7M) and authsha1(7M). Encryption algorithms encrypt data with a key. Encryption algorithms operate on data in units of a "block size." The size of both the block size and the key size are described in the encryption algorithm pages. See encr3des(7M) for an example of block size and key size descriptions.

AH and ESP use Security Associations (SA). SA's are entities that specify security properties from one host to another. Two communicating machines require two SAs (at a minimum) to communicate securely. However, communicating machines that use multicast can share the same multicast SA. SAs are managed through the pf_key(7P) interface. For IPv4, automatic SA management is available through the Internet Key Exchange (IKE), as implemented by in.iked(1M). A command-line front-end is available by means of ipseckey(1M). An IPsec SA is identified by a tuple of <AH or ESP, destination IP address, and SPI>. The Security Parameters Index (SPI) is an arbitrary 32-bit value that is transmitted on the wire with an AH or ESP packet. See ipsecah(7P) or ipsecesp(7P) for an explanation about where the SPI falls in a protected packet.

Mechanism and policy are separate. The policy for applying IPsec is enforced on a system-wide or per-socket level. Configuring systemwide policy is done via the ipsecconf(1M) command. Configuring per-socket policy is discussed later in this section.

Systemwide IPsec policy is applied to incoming and outgoing datagrams. Some additional rules can be applied to outgoing datagrams because of the additional data known by the system. Inbound datagrams can be accepted or dropped. The decision to drop or accept an inbound datagram is based on several criteria which sometimes overlap or conflict. Conflict resolution is resolved by which rule is parsed first, with one exception: if a policy entry states that traffic should bypass all other policy, it is automaticaly be accepted. Outbound datagrams are sent with or without protection. Protection may (or may not) indicate specific algorithms. If policy normally would protect a datagram, it can be bypassed either by an exception in systemwide policy or by requesting a bypass in per-socket policy.

Intra-machine traffic policies are enforced, but actual security mechanisms are not applied; rather, the outbound policy on an intra-machine packet translates into an inbound packet with those mechanisms applied.

IPsec policy is enforced in the ip(7P) driver; several ndd tunables for /dev/ip affect policy enforcement. These include:

icmp_accept_clear_messages

If equal to 1 (the default), allow certain cleartext icmp messages to bypass policy. For ICMP echo requests ("ping" messages), protect the response like the request. If zero, treat icmp messages like other IP traffic.

igmp_accept_clear_messages

If 1, allow inbound cleartext IGMP messages to bypass IPsec policy.

pim_accept_clear_messages

If 1, allow inbound cleartext PIM messages to bypass IPsec policy.

The IP_SEC_OPT or IPV6_SEC_OPT socket option is used to set per-socket IPsec policy. The structure used for an IP_SEC_OPT request is:

typedef struct ipsec_req {
    uint_t      ipsr_ah_req;           /* AH request */
    uint_t      ipsr_esp_req;          /* ESP request */
    uint_t      ipsr_self_encap_req;   /* Self-Encap request */
    uint8_t     ipsr_auth_alg;         /* Auth algs for AH */
    uint8_t     ipsr_esp_alg;          /* Encr algs for ESP */
    uint8_t     ipsr_esp_auth_alg;     /* Auth algs for ESP */
} ipsec_req_t;

The IPsec request has fields for both AH and ESP. Algorithms may or may not be specified. The actual request for AH or ESP services can take one of the following values:

IPSEC_PREF_NEVER

Bypass all policy. Only the superuser may request this service.

IPSEC_PREF_REQUIRED

Regardless of other policy, require the use of the IPsec service.

The following value can be logically ORed to an IPSEC_PREF_REQUIRED value:

IPSEC_PREF_UNIQUE

Regardless of other policy, enforce a unique SA for traffic originating from this socket.

In the event IP options not normally encapsulated by ESP need to be, the ipsec_self_encap_req is used to add an additional IP header outside the original one. Algorithm values from <net/pfkeyv2.h> are as follows:

SADB_AALG_MD5HMAC

Uses the MD5-HMAC (RFC 2403) algorithm for authentication. See authmd5h(7M).

SADB_AALG_SHA1HMAC

Uses the SHA1-HMAC (RFC 2404) algorithm for authentication. See authsha1(7M).

SADB_EALG_DESCBC

Uses the DES (RFC 2405) algorithm for encryption. See encrdes(7M).

SADB_EALG_3DESCBC

Uses the Triple DES (RFC 2451) algorithm for encryption. See encr3des(7M).

SADB_EALG_BLOWFISH

Uses the Blowfish (RFC 2451) algorithm for encryption. See encrbfsh(7M).

SADB_EALG_AES

Uses the Advanced Encryption Standard algorithm for encryption. See encraes(7M).

An application should use either the getsockopt(3SOCKET) or the setsockopt(3SOCKET) call to manipulate IPsec requests. For example:

#include <sys/socket.h>
#include <netinet/in.h>
#include <net/pfkeyv2.h>   /* For SADB_*ALG_* */      
/* .... socket setup skipped */     
rc = setsockopt(s, IPPROTO_IP, IP_SEC_OPT,        
   (const char *)&ipsec_req, sizeof (ipsec_req_t));