IPsec

The Internet was built for a world where everyone could be trusted. That world never existed, but we built on that assumption anyway.

IP packets travel in the clear. Anyone along the path can read them, modify them, forge them. The original designers knew this but prioritized getting the network working over securing it. Security could come later.

Later arrived. IPsec is that later—a suite of protocols bolted onto IP that provides what the base protocol lacks: proof that packets came from who they claim, assurance they weren't modified in transit, encryption so intermediaries can't read them, and protection against attackers who capture packets and replay them later.

IPsec operates at the network layer, which means it secures everything above it. Applications don't need to know it exists. Two networks connected by IPsec gateways can communicate as if they were directly connected, with all the encryption happening invisibly at the network boundary.

The Components

IPsec isn't a single protocol. It's a framework with several pieces:

Authentication Header (AH) computes a cryptographic hash over the packet, proving it hasn't been tampered with and came from the claimed source. But AH has a fatal flaw: it can't survive NAT. NAT modifies IP headers, which breaks AH's authentication. AH is largely obsolete.

Encapsulating Security Payload (ESP) encrypts the payload and optionally authenticates it. ESP does everything AH does plus confidentiality. Modern deployments use ESP exclusively.

Internet Key Exchange (IKE) handles the hard problem: how do two machines that have never communicated establish shared secrets for encryption? IKE negotiates cryptographic parameters, authenticates peers, and generates the keys that ESP uses.

Security Association (SA) is the agreement between sender and receiver about how to protect traffic—which algorithms, which keys, which parameters. SAs are one-way, so bidirectional communication needs two of them.

How ESP Works

An ESP packet carries:

• A Security Parameters Index identifying which SA to use
• A sequence number for replay protection
• The encrypted payload
• An authentication tag proving the encrypted data wasn't modified

The sequence number matters. Without it, an attacker could capture an encrypted packet and replay it later. The receiver might not be able to read the packet any more than the attacker can, but it would still process it—potentially authorizing a transaction twice or accepting a command again. Sequence numbers let receivers detect and discard replays.

Transport Mode vs. Tunnel Mode

IPsec can protect packets two ways:

Transport mode encrypts only the payload, leaving the original IP header visible. Efficient, but both endpoints must support IPsec. Used for direct host-to-host communication.

Tunnel mode encrypts the entire original packet—header and all—then wraps it in a new IP header. The original packet becomes cargo. This is how VPNs work: your packet addressed to an internal server gets encrypted and wrapped in a packet addressed to the VPN gateway. The gateway unwraps it and forwards the original.

Tunnel mode hides not just content but traffic patterns. Observers see packets flowing between gateways but can't tell which internal hosts are actually communicating.

The Key Problem

Encryption requires keys. Where do the keys come from?

Manual key distribution doesn't scale. Imagine configuring shared secrets on hundreds of devices, rotating them regularly, and keeping track of which key goes with which peer. This is why IKE exists.

IKE works in two phases:

Phase 1 establishes a secure channel between peers. They authenticate each other (using pre-shared keys, certificates, or other methods) and negotiate a shared secret using Diffie-Hellman. This IKE SA protects the Phase 2 negotiation.

Phase 2 uses the protected channel to negotiate the IPsec SAs that will actually encrypt data traffic. Multiple IPsec SAs can run over a single IKE SA, allowing different security policies for different traffic.

IKEv2 simplified this considerably. It's faster, handles mobile clients better, and is more resistant to denial-of-service attacks.

Authentication

Before encryption can happen, peers must prove their identity. IKE supports several methods:

Pre-shared keys are passwords known to both sides. Simple to set up, but they don't scale. Every pair of peers needs its own key, and weak passwords invite brute-force attacks.

Certificates use public key cryptography. Each peer has a certificate signed by a trusted authority. The peer proves identity by demonstrating possession of the corresponding private key. This scales to large deployments.

EAP (Extensible Authentication Protocol) integrates with existing authentication systems—RADIUS, Active Directory, whatever the organization already uses.

Perfect Forward Secrecy

Here's a scenario: an attacker records all your encrypted VPN traffic for years. Later, through theft or legal compulsion, they obtain your private key. Can they decrypt all that stored traffic?

Without Perfect Forward Secrecy (PFS): yes. The session keys were derived from the long-term key, so compromising the long-term key compromises everything.

With PFS: no. Each session generates fresh ephemeral keys through a new Diffie-Hellman exchange. The long-term key is only used for authentication, not for deriving session keys. Compromise of the long-term key lets attackers impersonate you going forward, but past traffic remains protected.

PFS costs CPU cycles—each new SA requires Diffie-Hellman calculations. For sensitive communications, it's worth it.

The NAT Problem

NAT breaks things.

IPsec was designed assuming end-to-end connectivity with real IP addresses. Then NAT became ubiquitous, hiding entire networks behind single addresses. ESP packets have no port numbers for NAT to translate, and modifying packets breaks authentication.

NAT Traversal (NAT-T) is the workaround: wrap ESP packets in UDP. The UDP header provides port numbers that NAT can translate. The ESP packet rides as payload, untouched.

IKE detects NAT automatically by comparing the source address in the packet with what the receiver actually sees. If they differ, NAT exists, and peers switch to UDP encapsulation.

This is patches on patches. We secured the network layer, then broke it with NAT, then wrapped the security layer in transport-layer headers to get through. Somehow it works.

Performance

Every packet encrypted. Every packet authenticated. Every packet decrypted and verified. This costs CPU cycles.

Modern processors include dedicated instructions for cryptographic operations—AES-NI, SHA extensions. Hardware that supports these can handle IPsec at multi-gigabit speeds with minimal overhead.

Dedicated VPN appliances include cryptographic accelerators for even higher throughput. Software-based IPsec on general-purpose servers may become a bottleneck at high packet rates.

Overhead also comes from the additional headers. Tunnel mode ESP might add 50-100 bytes per packet. On small packets, that's significant. Path MTU Discovery or careful fragmentation handling becomes important.

IPsec and IPv6

IPsec was supposed to be mandatory for IPv6. Every IPv6 implementation would support IPsec, and secure communication would be universal.

That requirement was later relaxed. In practice, IPv6 uses IPsec the same way IPv4 does—as an optional layer for VPNs and protected communications, not as universal encryption.

The larger IPv6 header (40 bytes vs. 20 for IPv4) means IPsec overhead is proportionally smaller, slightly improving efficiency.

Where IPsec Fits

Site-to-site VPNs connect networks across the Internet. Gateways at each site encrypt traffic between them. Users see a unified network; the encryption is invisible.

Remote access VPNs let individuals connect securely from anywhere. The client establishes an IPsec tunnel to a concentrator, gaining access to internal resources.

Host-to-host encryption secures communication between specific servers without application changes. Database servers and application servers might use IPsec to encrypt their traffic automatically.

When Things Break

IPsec failures are opaque. Packets simply don't arrive, with no obvious indication why.

Common causes:

Mismatched parameters. Peers must agree on algorithms, key sizes, and other settings. One side offering AES-256 while the other only accepts AES-128 means no agreement, no SA, no traffic.

Authentication failures. Wrong pre-shared key, expired certificate, clock skew breaking certificate validation.

Firewalls. IKE needs UDP port 500. ESP is IP protocol 50 (not a port—a protocol number). NAT-T needs UDP port 4500. Any of these blocked means IPsec fails.

MTU issues. IPsec overhead pushes packets over the path MTU. If fragmentation is blocked, packets disappear into black holes.

Debugging requires access to IKE logs, which reveal where negotiation failed. Without logs, you're guessing.