Port 500: ISAKMP/IKE — The Handshake Before the Secret

Port 500 carries ISAKMP and IKE traffic over UDP. Every time a VPN tunnel forms between two offices, every time a remote worker connects to their corporate network, every time two routers establish an encrypted link across the Internet, that connection begins with a negotiation on port 500.

This is the port of trust establishment. Before any encrypted data can flow, two machines must agree on how to encrypt it. They must prove their identities to each other. They must generate shared secrets without ever transmitting those secrets. Port 500 is where this seemingly impossible conversation happens.

What Port 500 Does

ISAKMP (Internet Security Association and Key Management Protocol) and IKE (Internet Key Exchange) work together to solve a fundamental problem in cryptography: how do two parties who have never met agree on a shared encryption key without anyone else being able to learn it?

The answer involves mathematics that borders on magic. Diffie-Hellman key exchange, published in 1976 by Whitfield Diffie and Martin Hellman, allows two parties to create a shared secret by exchanging information in plain view.¹ An eavesdropper watching every packet can see all the exchanged data and still cannot compute the shared key. The math works because certain operations are easy to perform but computationally infeasible to reverse.

IKE uses this mathematical foundation to establish what the protocol calls a "Security Association," a mutual agreement between two endpoints about exactly how they will encrypt their traffic: which algorithms, which key lengths, which authentication methods. Port 500 carries this negotiation.²

The Key Exchange Mechanism

The IKE protocol operates in two phases:³

Phase 1 establishes a secure, authenticated channel between two endpoints. The endpoints exchange cryptographic parameters, authenticate each other using pre-shared keys or certificates, and use Diffie-Hellman to generate shared keying material. This creates an IKE Security Association.

Phase 2 uses the secure channel from Phase 1 to negotiate the actual IPsec Security Associations that will protect the data traffic. These negotiations happen encrypted, protected by the keys established in Phase 1.

The protocol typically requires 4 to 6 packets and 2 to 3 round trips to complete. Every packet in this dance serves a purpose: establishing parameters, proving identity, exchanging mathematical values, confirming agreement.⁴

The History

In November 1998, the IETF published three RFCs that defined how IKE would work:⁵

• RFC 2407 defined the Internet IP Security Domain of Interpretation for ISAKMP
• RFC 2408 defined ISAKMP itself, the framework for security association management
• RFC 2409 defined IKE, which combined ISAKMP with the Oakley key exchange protocol

The authors of RFC 2408 included Douglas Maughan and Mark Schneider of the National Security Agency, along with Mark Schertler of Securify, Inc. and Jeff Turner of RABA Technologies, Inc.⁶ The ISAKMP/IKE implementation was jointly developed by Cisco and Microsoft.⁷

IKE itself was a hybrid, combining three earlier protocols: ISAKMP provided the framework for negotiation, Oakley (proposed by Hilarie K. Orman) provided the key determination mechanism based on Diffie-Hellman, and SKEME provided the key generation techniques.⁸ These protocols were designed to be modular, each handling its piece of the puzzle.

The NSA Connection

The presence of NSA cryptographers among the RFC authors raises obvious questions. The same agency responsible for signals intelligence helped design the protocol that protects Internet traffic from surveillance.

This is not hidden or controversial. It is documented on page one of RFC 2408. Douglas Maughan listed his address as "National Security Agency, ATTN: R23, 9800 Savage Road, Ft. Meade, MD 20755-6000."⁹

Whether this represents the NSA contributing genuine security expertise to Internet standards, or something more complicated, the protocol has been publicly analyzed for decades. Maughan later worked at the Department of Homeland Security on cybersecurity R&D initiatives.¹⁰ The standards remain open. The math remains sound. The implementations have been scrutinized by security researchers worldwide.

Security History

ISAKMP implementations have faced their share of vulnerabilities:

In 2005, the PROTOS test suite developed at the University of Oulu in Finland generated abnormal ISAKMP traffic and found multiple implementations vulnerable. Cisco, Juniper, Secgo, and OpenSWAN all released patches.¹¹

In 2015, Cisco ASA devices were found vulnerable to a denial of service attack through improper handling of ISAKMP packets (CVE-2015-6327). An unauthenticated remote attacker could cause the device to reload by sending crafted UDP packets to port 500.¹²

Check Point products experienced a buffer overflow in their ISAKMP implementation that could allow remote code execution with root privileges.¹³

The original IKEv1 protocol had design limitations: it was difficult to extend with new cryptographic standards, lacked native NAT traversal support, and was vulnerable to denial of service attacks during the resource-intensive Phase 1 negotiation.¹⁴

IKEv2: The Evolution

In December 2005, RFC 4306 introduced IKEv2, a major revision that addressed IKEv1's weaknesses.¹⁵ The new version consolidated multiple RFCs into one specification, reduced the number of round trips required, improved DoS protection, added native NAT traversal support, and provided better extensibility.

The current specification is RFC 7296, published in 2014.¹⁶

NAT Traversal and Port 4500

IPsec and Network Address Translation are fundamentally incompatible. IPsec's ESP protocol encrypts layer 4 headers, preventing NAT devices from performing port translation. The encrypted payload cannot be modified without breaking the cryptographic integrity.

NAT Traversal (NAT-T) solves this by encapsulating ESP packets inside UDP. When IKE endpoints detect a NAT device between them, they switch from port 500 to port 4500 and wrap their encrypted traffic in UDP headers that NAT devices can translate.¹⁷

RFC 3947 describes how IKE endpoints detect NAT devices using hash payloads (NAT_DETECTION_SOURCE_IP and NAT_DETECTION_DESTINATION_IP) and how they negotiate the switch to UDP encapsulation.¹⁸

Current Usage

IPsec VPNs remain fundamental to enterprise networking. 93% of organizations use VPNs to secure connections for employees and offices.¹⁹ Over 82% of enterprises use VPNs or SD-WAN technologies to secure remote access.²⁰

The VPN and network security market reached $23 billion in 2024.²¹ Cisco holds approximately 54% of the corporate VPN market.²²

Modern deployments increasingly use IPsec in conjunction with SD-WAN. 45% of large enterprises in North America had adopted SD-WAN by 2023, often with IPsec providing the underlying security.²³ The U.S. Department of Homeland Security's CISA recommends IPsec for federal network segmentation.²⁴

Protocol Details

• Port: 500/UDP (4500/UDP with NAT-T)
• IP Protocol: ESP (50) for encrypted data, AH (51) for authentication
• RFCs: 2407, 2408, 2409 (IKEv1); 4306, 7296 (IKEv2); 3947 (NAT-T)
• IANA Assignment: Internet Security Association and Key Management Protocol

Related Ports