IPsec: AH, ESP, transform sets and IKE
IPsec is a framework of open standards for secure VPNs across untrusted networks, giving peer authentication, confidentiality, integrity and replay detection. IKE authenticates the peers and agrees the keys and algorithms; AH or ESP then protect each packet. ESP encrypts and AH does not, and each runs in tunnel mode, wrapping the whole packet, or transport mode.
By Arif, founder of Daveth · Updated
The full 8 min walkthrough is in Daveth Pro (Network). See what the plan opens
What this page answers
What is IPsec?
A framework of open standards for creating secure VPNs across untrusted networks, offering peer authentication, data confidentiality, data integrity and replay detection. Read it in “4.1 What IPsec protects”
What is the difference between IPsec tunnel mode and transport mode?
Tunnel mode encrypts the entire original packet, headers included, and adds a new IP header, so it works as an overlay. Transport mode protects only the payload and keeps the original IP header outside, so it gives no overlay. Read it in “4.1 What IPsec protects”
What is the difference between AH and ESP?
AH, IP protocol 51, gives integrity, authentication and replay protection but no encryption. ESP, IP protocol 50, adds confidentiality, encrypting the payload. Read it in “4.1 What IPsec protects”
4.1 What IPsec protects
IPsec is "a framework of open standards for creating highly secure virtual private networks" across untrusted networks such as the Internet. It offers four services (Table 16-3):
| Service | What it stops | With |
|---|---|---|
| peer authentication | talking to an impostor | pre-shared keys, digital certificates |
| data confidentiality | eavesdropping | DES, 3DES, AES (DES and 3DES not recommended) |
| data integrity | tampering in transit | HMAC with MD5 or SHA-1 (MD5 not recommended) |
| replay detection | captured traffic sent again | a unique sequence number on every packet |
Protecting a packet is like a firm sending valuables to a branch office: you check it really is the branch you're dealing with, lock the box, seal it so tampering shows, and number every box so a copy stands out. GRE (04-2) gave you none of these; that's why the next lesson wraps GRE in IPsec.
GRE is a postcard with a very long address; IPsec is what turns it into a letter.
The pieces fit together in two layers. IKE authenticates the peers and agrees the keys and algorithms, producing security associations (SAs); AH or ESP then protect each packet with them. Think of it as a handshake before business: nobody opens the boxes until both sides have agreed who they are and which locks to use.
IPsec has two packet headers:
- Authentication Header (AH), IP protocol 51: integrity, authentication and replay protection, but no encryption and no NAT traversal, so the guide recommends it only when authentication is all you want.
- Encapsulating Security Payload (ESP), IP protocol 50: confidentiality (encryption), authentication and replay protection, and it supports NAT traversal.
It's a bit like the difference between a tamper-evident seal and a locked box: AH seals the envelope so you know nobody touched it, but anyone can read it; ESP puts the contents in a locked box.
Each can run in two modes (quiz Q4: tunnel and transport; "transparent" isn't one). The guide describes them for ESP, the header that encrypts:
- Tunnel mode encrypts the entire original packet, headers included, and adds a new IP header; it works as an overlay.
- Transport mode encrypts and authenticates only the payload; the original IP header stays on the outside and routes the packet, so it gives no overlay.
AH has the same two modes but encrypts nothing in either: in tunnel mode the inner addresses stay readable, and in both modes AH also covers parts of the outer IP header (next paragraph).
In Figure 16-3, ESP encrypts from the payload to the ESP trailer and authenticates from the ESP header to the trailer; the outer IP header is outside both. AH works differently: it authenticates "as much of the IP header as possible": the fields that don't change in transit or change predictably, and in IPv4 that includes the source address and the destination address (when no source routing is in use). For IPv6 the list also includes the source and destination addresses (without a routing header). ESP uses "payload" in its own sense: the part of the original packet that ESP encapsulates.
Before running D2, predict: a NAT on the path rewrites the outer source address of an AH packet and of an ESP packet. Which one still verifies at the far end?
D2

That's the wait-what. AH fails behind a NAT, in either mode, because its integrity check covers the very address the NAT changed; ESP survives, because its check starts at the ESP header, and NAT traversal (NAT-T) carries it inside UDP port 4500 so the NAT can track it. RFC 3948 puts the AH half plainly: "the protection of the outer IP addresses in IPsec AH is inherently incompatible with NAT", and the guide says AH "does not support ... NAT traversal". (ESP in transport mode has one more wrinkle behind a NAT: the inner TCP or UDP checksum was computed with the old addresses, so the receiver has to correct it.) If you've ever had a parcel refused because someone re-wrote the address on a sealed label, you've seen AH meet a NAT. The memory hook: IKE hands out the keys; ESP locks the box; AH only seals it.
AH is the security guard who checks your ID so carefully that he refuses you entry for having moved house.
The guide's list, with Cisco's current ratings from its Next Generation Cryptography guidance. It reads like a pantry with use-by dates: everything still opens, but some of it is long past its date.
| Algorithm | Does | Guide | Cisco |
|---|---|---|---|
| DES | encryption, 56-bit key | "very weak and should be avoided" | Avoid |
| 3DES | encryption, DES three times | "no longer recommended" | Legacy |
| AES | encryption, 128, 192 or 256 bits | use it | AES-GCM is NGE |
| MD5 | 128-bit hash (HMAC) | "no longer recommended" | Avoid |
| SHA-1 | 160-bit hash (HMAC) | use SHA | SHA-1 Legacy; HMAC-SHA-1 Acceptable; HMAC-SHA-256 NGE |
| DH groups 1, 2, 5 | key exchange, 768, 1024, 1536 bits | avoid; use 14 and higher | DH-768/1024 Avoid; DH-2048 Acceptable |
| RSA signatures, PSK | peer authentication | both supported | - |
Quiz Q5: DES and 3DES are the encryption protocols to avoid; its other options, AES, GCM and GMAC, are the modern ones (Table 16-4 lists esp-gcm and esp-gmac; esp-aes is 128-bit, esp-gcm 128 by default or 256). Diffie-Hellman is the clever one: an asymmetric, CPU-hungry exchange that lets two peers agree a shared symmetric key, such as an AES key, over an insecure network. It's like two people mixing paint colours in public and each ending up with the same secret shade, without ever showing it.
DES's 56-bit key was strong in the 1970s, which is also when people thought eight-track tapes had a future.
A transform set is "a combination of security protocols and algorithms"; the peers agree on one during the IPsec SA negotiation, and it's applied to the SAs on both. Think of it as an order form both sides must fill in identically before anything ships. Table 16-4 lists the choices, one per type:
| Type (only one allowed) | Examples |
|---|---|
| AH authentication | ah-sha-hmac, ah-sha256-hmac, ah-sha384-hmac, ah-sha512-hmac (ah-md5-hmac not recommended) |
| ESP encryption | esp-aes, esp-aes 192, esp-aes 256, esp-gcm, esp-gmac (esp-des, esp-3des no longer recommended; esp-null, esp-seal) |
| ESP authentication | esp-sha-hmac (esp-md5-hmac no longer recommended) |
| IP compression | comp-lzs |
Pick at most one from each type, and one more rule on top: on IOS XE, AH and ESP algorithms can't go in the same transform set. It acts kind of like two separate menus in one restaurant: choose one dish per course from the ESP menu, or from the AH menu, but you can't combine the two menus on one order. (The guide's table calls the ah-sha256/384/512-hmac transforms "AES" algorithms; they're SHA-2 HMACs.)
A transform set is the only menu where choosing esp-null gets you the full service and no food.
IKE authenticates the two ends and builds the SAs, which the guide also calls IKE tunnels; IKEv1 is RFC 2409, IKEv2 RFC 7296. IKEv1 is built on ISAKMP, which uses UDP port 500; IKE implements ISAKMP with the Oakley and Skeme key-exchange techniques, and on Cisco, IKE and ISAKMP mean the same thing. (IKEv2 uses UDP 500 too, and 4500 behind a NAT.) It works in two phases:
- Phase 1 builds one two-way ISAKMP SA, by main mode (six messages) or aggressive mode (three).
- Phase 2 uses it to build the IPsec SAs by quick mode (three messages); either peer may start it. One negotiation makes two one-way IPsec SAs; the guide says "one on each peer", which means one for each direction of traffic, and QM1 can start several at once. That's quiz Q6: quick mode establishes the IKEv1 IPsec SA.
Before reading on, predict: in main mode, which is the first message that travels encrypted, and why can't it be earlier? D1 lets you check.
In main mode, MM1 and MM2 agree a proposal (hash, encryption, authentication method, DH group, lifetime), MM3 and MM4 run Diffie-Hellman, and only then, with keys shared, MM5 and MM6 exchange identities and authenticate, encrypted. The lifetime (24 hours by default) is the only proposal value that needn't match: the peers take the smaller. Aggressive mode packs the same job into three messages, so it's faster, but "the identities of the two peers... are exposed to eavesdropping", and main mode is also more flexible because it can offer more proposals. The guide says AM1 carries "MM1 through MM3 and MM5"; strictly it's the initiator's parts of those, the proposals, its DH value, a nonce and its identity, since MM2 is the responder's message. Whether AM3 is encrypted is optional in RFC 2409 ("it need not be"). Picture yourself phoning a bank: main mode sets up a private line before anyone gives their name; aggressive mode shouts both names down the corridor to save time.
D1

Step 1 of 10 IKEv1 builds a two-way ISAKMP SA in phase 1 (main mode, six messages), then uses it to build two one-way IPsec SAs in phase 2, quick mode.

Step 2 of 10 MM1: SA proposals: hash, encryption, authentication method, DH group, lifetime.

Step 3 of 10 MM2: the proposal it matched.

Step 4 of 10 MM3: DH key exchange starts.

Step 5 of 10 MM4: its DH key: keys are now shared. From here on, main mode is encrypted, so the identities in MM5 and MM6 are hidden.

Step 6 of 10 MM5: identity and authentication.

Step 7 of 10 MM6: identity and authentication: ISAKMP SA up.

Step 8 of 10 QM1: IPsec proposals and the traffic to protect.

Step 9 of 10 QM2: matching IPsec parameters.

Step 10 of 10 QM3: confirmation: two one-way IPsec SAs. Nine messages in all.
The totals to remember: main plus quick is nine messages, aggressive plus quick six. Perfect forward secrecy (PFS) is a recommended but optional extra Diffie-Hellman in phase 2 so that each session key is independent, "a compromised key does not compromise future keys"; it costs CPU. Consider it your house keys: with PFS, a thief who copies this year's key still can't open next year's lock, because each lock is cut fresh.
Continue the lesson
The whole lesson, with its drawings and its questions. The rest of this section and 9 more after it.












