BGP Deep Dive

Every packet you send across the Internet passes through multiple networks owned by different organizations. Your ISP hands it to a backbone provider, who hands it to another provider, who eventually delivers it to the destination network. But how do these networks know where to send your traffic?

Border Gateway Protocol. BGP is how the Internet's 70,000+ independent networks tell each other what they can reach. It's the routing protocol that holds everything together—and it runs on trust.

The Honor System at Global Scale

BGP was designed in 1989, when the Internet was a few hundred networks run by people who mostly knew each other. The protocol reflects that era: when a network announces "I can reach these addresses," BGP believes it.

This trust model has consequences. In 2008, Pakistan Telecom tried to block YouTube domestically by announcing YouTube's IP addresses to their own network. But they accidentally leaked that announcement to the global Internet. Suddenly, networks worldwide believed Pakistan Telecom was the path to YouTube. For two hours, YouTube was unreachable globally because the Internet trusted a lie.

BGP is a handshake protocol running on a global honor system. When someone lies—accidentally or maliciously—the Internet believes them.

Autonomous Systems: The Players

The Internet isn't one network. It's tens of thousands of autonomous systems (ASes), each operated by a different organization—ISPs, corporations, universities, content providers.

Every organization running BGP gets a globally unique Autonomous System Number (ASN) from a regional Internet registry. Originally 16-bit (65,536 possible values), ASNs are now 32-bit to accommodate growth. When you see "AS15169," that's Google. "AS7922" is Comcast. These numbers identify who's who in the routing conversation.

How BGP Routers Talk

BGP routers establish sessions over TCP port 179. Unlike protocols that automatically discover neighbors, BGP requires explicit configuration on both sides—you have to agree to peer.

Two types of peering exist:

External BGP (eBGP) connects routers in different autonomous systems. This is where the real Internet routing happens—networks exchanging information about what they can reach.

Internal BGP (iBGP) distributes those external routes within an organization. If your company has BGP connections to two ISPs, iBGP ensures all your routers know about both paths.

After initial setup, BGP only sends updates when something changes. This efficiency matters because Internet BGP routers track over 900,000 routes. Exchanging full tables constantly would melt the infrastructure.

Path Attributes: More Than Just Reachability

When a BGP router advertises a route, it includes attributes that describe the path:

AS_PATH lists every autonomous system the route has traversed. This prevents loops (routers reject routes containing their own ASN) and reveals how far away a destination is in terms of network hops.

NEXT_HOP tells routers where to forward packets. For eBGP, this is typically the advertising router's address.

LOCAL_PREF sets preference within your own network. Higher values win. If you prefer traffic to exit through your connection to ISP-A rather than ISP-B, you set higher LOCAL_PREF on routes from ISP-A.

MED (Multi-Exit Discriminator) suggests to your neighbor which of your entry points to prefer. Unlike LOCAL_PREF, MED propagates to the neighboring AS.

COMMUNITY tags routes for policy purposes. Want to tell your upstream provider "don't advertise this route to their other customers"? There's a community for that.

Route Selection: The Decision Process

When multiple paths exist to the same destination, BGP follows a decision process. Simplified:

1. Highest LOCAL_PREF (your explicit preference)
2. Shortest AS_PATH (fewest networks to traverse)
3. Lowest MED (neighbor's preference)
4. eBGP over iBGP (prefer external knowledge)
5. Nearest exit point (lowest internal cost)

This isn't just shortest-path routing. BGP is fundamentally about policy. Networks choose paths based on business relationships, not just technical metrics.

Policy: Where Business Meets Routing

BGP lets networks encode business logic into routing decisions:

Customer routes are preferred over peer routes, which are preferred over provider routes. Your customers pay you; you want to carry their traffic. Peers exchange traffic for free; acceptable. Providers charge you; use them only when necessary.

Transit policies determine what you'll carry. If two of your peers should be peering directly with each other, you might refuse to carry traffic between them.

Traffic engineering balances load. Multiple links to the same destination? Use BGP attributes to distribute traffic.

This policy flexibility is why BGP won. Interior routing protocols optimize for shortest paths. BGP optimizes for whatever you decide matters.

Filtering: Defense Against Chaos

Not every route should propagate everywhere. Networks implement filters:

Prefix filters reject routes for address space you know shouldn't come from a particular peer. If your peer advertises your own IP addresses, something's wrong.

AS-path filters reject routes based on which networks they've traversed.

Community filters implement complex policies based on tags attached to routes.

Proper filtering prevents two disasters:

Route leaks happen when a network accidentally advertises routes learned from one peer to another. In 2019, a small Pennsylvania ISP leaked routes that briefly made them appear to be the best path to major chunks of the Internet.

Route hijacks happen when someone advertises address space they don't own. Sometimes accidental, sometimes malicious—BGP can't tell the difference.

Making BGP More Trustworthy

The Internet is slowly addressing BGP's trust problem.

RPKI (Resource Public Key Infrastructure) lets address space owners cryptographically sign which ASNs are authorized to announce their prefixes. BGP routers can validate routes against these signatures and reject unauthorized announcements.

RPKI adoption is growing but incomplete. Many networks now sign their routes; fewer validate incoming routes against signatures.

BGPsec extends RPKI to validate the entire AS path, not just the origin. But computational overhead and deployment challenges have slowed adoption.

For now, the Internet runs on a combination of RPKI where deployed, filtering where implemented, and hope everywhere else.

BGP Convergence: The Speed Problem

When something changes—a link fails, a configuration updates, an announcement withdraws—BGP must converge to a new stable state.

BGP convergence is slow compared to interior protocols. When a route disappears, routers explore alternatives sequentially, potentially trying many invalid paths before settling. This "path exploration" can take minutes.

Mechanisms help:

Graceful restart lets routers maintain forwarding during BGP process restarts, preventing route withdrawals during maintenance.

Add-paths allows advertising multiple paths to the same destination, so alternatives exist before the primary fails.

Route flap damping suppresses unstable routes—though this can extend outages for legitimately troubled networks.

BGP in the Data Center

Modern data centers use BGP internally, not just for Internet connectivity. This "BGP in the data center" approach provides:

• Uniform configuration across all devices
• Fast convergence compared to spanning tree
• Multi-path routing (ECMP) to use all available links
• Integration with overlay networks

Data center BGP differs from Internet BGP: thousands of routes instead of hundreds of thousands, convergence speed prioritized over complex policy.

The Protocol That Shouldn't Work

BGP routes traffic between autonomous systems that have no reason to trust each other, using a protocol that accepts claims at face value, across an infrastructure where a single misconfiguration can break connectivity for millions of users.

And yet it works. Not because BGP is perfectly designed—it has obvious flaws. It works because network operators have built decades of operational practice around it: filtering, monitoring, incident response, gradual security improvements.

BGP is the Internet's routing substrate. Understanding it means understanding how traffic actually flows, why outages happen, and what's at stake when networks make routing decisions.