What is an IPsec VPN? ESP, AH, IKEv2 and tunnel mode explained

02How does IKEv2 set up an IPsec tunnel?

IKEv2 normally needs four messages to bring a tunnel up. RFC 7296 defines them as two exchanges of a request and a response each, IKE_SA_INIT and IKE_AUTH, which establish the IKE SA and the first Child SA. Later exchanges add SAs, rekey them and report errors:

• IKE_SA_INIT: the peers offer and choose algorithms, exchange Diffie-Hellman values and nonces, and detect whether NAT sits between them; everything after this exchange is encrypted
• IKE_AUTH: each side proves its identity with a certificate, a pre-shared key or EAP, and the first Child SA, which carries the ESP traffic, is created with its traffic selectors
• CREATE_CHILD_SA: adds further Child SAs and rekeys existing ones before their lifetimes expire
• INFORMATIONAL: deletes SAs, reports errors and, when sent empty, checks that the peer is still alive

03ESP vs AH: what is the difference?

The Encapsulating Security Payload (ESP) encrypts the packet and protects its integrity. The Authentication Header (AH) only protects integrity, but extends it to the unchanging fields of the outer IP header, including both addresses. That extra coverage is AH's undoing: a NAT device must rewrite the source address, which breaks AH's integrity check, so AH cannot cross NAT.

The standards reflect this. RFC 4301 requires implementations to support ESP and makes AH optional, and RFC 8221 notes that ESP with NULL encryption, which gives integrity only, is preferred over AH because of NAT traversal. In practice ESP is IPsec, and AH appears mostly in older configurations.

ESP, tunnel mode
[ new IP ][ SPI 4 | Seq 4 ][ IV ][ original IP | TCP/UDP | data | pad | padlen 1 | next 1 ][ ICV ]
                                  |<------------------ encrypted ------------------------>|
          |<---------------------------- integrity-protected ---------------------------->|

ESP, transport mode
[ IP ][ SPI 4 | Seq 4 ][ IV ][ TCP/UDP | data | pad | padlen 1 | next 1 ][ ICV ]

AH, transport mode (the ICV also covers the IP header's immutable fields)
[ IP ][ next 1 | len 1 | reserved 2 | SPI 4 | Seq 4 | ICV ][ TCP/UDP | data ]

04IPsec tunnel mode vs transport mode

Tunnel mode encrypts the entire original packet and puts it behind a new IP header addressed to the far gateway, so an observer sees only the two gateway addresses. Site-to-site and remote-access VPNs use it, because the gateways protect traffic for whole networks behind them.

Transport mode keeps the original IP header and protects only what follows it. It saves one IP header and suits host-to-host protection where the endpoints are the communicating hosts. Its best-known use is GRE over IPsec: a GRE tunnel supplies the outer addresses and the ability to carry routing protocols, multicast and IPv6, and transport-mode ESP encrypts the GRE packets between the same two endpoints. For GRE itself, see what GRE tunnelling is.

05Which ports and protocols does IPsec need?

Three things must pass between the peers: UDP 500 for IKE, UDP 4500 for IKE and ESP once NAT is detected, and IP protocol 50 for ESP when there is no NAT. AH, if used, is IP protocol 51. Protocols 50 and 51 have no ports, so a firewall that opens only UDP lets the tunnel negotiate and then pass no traffic, a classic half-working symptom.

Behind NAT, RFC 3948 wraps ESP in UDP 4500 and marks IKE messages on that port with four zero bytes so the receiver can tell the two apart. Where UDP is blocked entirely, RFC 9329 defines IKE and ESP over TCP port 4500.

The ruleset below admits IKE and ESP only from known peers and accepts traffic that arrived inside an SA after decryption. It is a complete input chain with a default drop, so adapt the SSH rule and the peer addresses first; the wider principles are in firewall configuration.

table inet ipsec_edge {
    set ike_peers {
        type ipv4_addr
        elements = { 198.51.100.7, 203.0.113.20 }
    }

    chain input {
        type filter hook input priority filter; policy drop;
        ct state established,related accept
        ct state invalid drop
        iifname "lo" accept
        tcp dport 22 accept
        ip protocol icmp accept
        meta l4proto ipv6-icmp accept

        # IKEv2 and NAT traversal (UDP 500 and 4500), ESP (IP protocol 50)
        ip saddr @ike_peers udp dport { 500, 4500 } accept
        ip saddr @ike_peers ip protocol esp accept

        # traffic that arrived inside an IPsec SA, after decryption
        meta ipsec exists accept
    }
}

# watch negotiation and ESP on the uplink
tcpdump -ni eth0 'udp port 500 or udp port 4500 or ip proto 50'

# list the kernel's SAs without printing keys, and its policies (Linux)
ip xfrm state list nokeys
ip xfrm policy list

06IPsec overhead and MTU: a worked example

Every layer of encapsulation takes bytes from each packet, and the inner MTU must shrink to match, or full-size packets are dropped or fragmented. The figures assume a 1,500-byte IPv4 path and AES-GCM: an 8-byte ESP header, an 8-byte IV (RFC 4106), a 2-byte trailer, a 16-byte integrity tag, and padding so the encrypted part ends on a 4-byte boundary (RFC 4303). AES-CBC with HMAC-SHA-256 costs more, because its IV is 16 bytes and it pads to 16-byte blocks.

Cisco's guidance for GRE over IPsec recommends an inner MTU of 1,400 bytes because it covers the common mode and algorithm combinations; the table shows how much headroom that leaves. Plain GRE is included for comparison, and GRE vs IPIP shows its arithmetic; the MTU and MSS calculator works out other combinations.

ESP tunnel, AES-GCM:      1500 - 20 IP - 8 ESP - 8 IV - 16 ICV = 1448 for packet + pad + 2
                          1448 - 2 trailer           = 1446 inner MTU   -> MSS 1406
same, behind NAT:         1448 - 8 UDP = 1440;  - 2  = 1438 inner MTU   -> MSS 1398
ESP tunnel, AES-CBC:      1500 - 20 - 8 - 16 IV - 16 ICV = 1440 (90 blocks of 16)
                          1440 - 2 trailer           = 1438 inner MTU   -> MSS 1398
GRE over IPsec, transport 1448 - 4 GRE - 2           = 1442 inner MTU   -> MSS 1402
GRE over IPsec, tunnel    1448 - 20 IP - 4 GRE - 2   = 1422 inner MTU   -> MSS 1382
WireGuard over IPv4:      1500 - 20 IP - 8 UDP - 16 header - 16 tag = 1440 -> MSS 1400
WireGuard default MTU:    1500 - 80 (sized for an IPv6 outer header) = 1420

07IPsec vs WireGuard vs OpenVPN vs GRE

All four build tunnels, but only three encrypt, and they differ in how they authenticate, what they need from the firewall and what they cost per packet. GRE is included because it is so often paired with IPsec; on its own it provides no confidentiality at all. In short:

• IPsec when the other end is a firewall, router or cloud VPN gateway, because that is the protocol they all speak
• WireGuard when you control both ends and want a small configuration and low overhead
• OpenVPN when only one TCP or UDP port gets through, or existing clients depend on it
• GRE over IPsec when routing protocols, multicast or several address families must cross an encrypted link

08IPsec and DDoS: what a VPN can and cannot protect

An IPsec gateway is an exposed service in its own right. IKE responders listen on public UDP ports and must do expensive cryptography for strangers, and RFC 7296 names state and CPU exhaustion through session requests from forged addresses as the attacks to expect. Its answer is a stateless cookie: under load the responder replies to the first message with a COOKIE notification and commits no state until the initiator repeats it from a real address. RFC 8019 adds client puzzles and further hardening for DDoS conditions.

Encryption does not make a link harder to flood. A volumetric UDP flood aimed at the gateway's address fills the uplink before any IPsec processing happens, and the tunnel goes down with the link. Filtering has to happen upstream, at network level, and it cannot look inside ESP: an upstream filter sees outer addresses, protocol 50 or UDP 4500, and packet rates. A web reverse proxy cannot help either, because IKE and ESP are not HTTP; Layer 3/4 vs Layer 7 explains the difference.

• Accept IKE and ESP only from known peers, as in the ruleset above; it sheds load from strangers but cannot stop a flood that fills the link
• Keep cookies and, where supported, puzzles enabled on the IKE responder
• Use upstream network-level DDoS filtering for gateway addresses, the only place a flood larger than the uplink can be dropped
• Give critical sites a second gateway address or uplink, so one flooded address does not cut them off

09IPsec timeline: from RFC 1825 to the end of IKEv1

• August 1995: RFC 1825 publishes the first IPsec security architecture
• November 1998: RFC 2401 revises the architecture and RFC 2409 defines IKE, now known as IKEv1
• January 2005: RFC 3948 standardises UDP encapsulation of ESP for NAT traversal
• December 2005: RFC 4301 and RFC 4303 define today's architecture and ESP; RFC 4306 introduces IKEv2
• October 2014: IKEv2 becomes an Internet Standard as RFC 7296
• October 2017: RFC 8221 makes AES-GCM mandatory to implement for ESP
• November 2022: RFC 9329 specifies IKE and ESP over TCP
• April 2023: RFC 9395 deprecates IKEv1