Brough gave a talk on Internet Peering at last year’s Optical conference, OFC 2014 in San Francisco. With all the debate over net neutrality, it’s more relevant than ever. Here’s a draft of a paper he wrote to go along with his presentation.
Impact of Internet Peering on Network Architectures and Economics
netBlazr Inc., 218 Lincoln St, MA 02134 USA
brough at netblazr dot com
Abstract: The Internet backbone consists of ~6000 independent networks. The technology and economics of how these networks exchange data drives the location of data centers and the location and utilization of high capacity fiber links. We explain how Internet peering works, how it has evolved and trends that will influence future network deployments.
The Internet is made up of millions of independently controlled networks that, together, support nearly 3 billion users . Most of these networks, e.g. home networks and medium and smaller enterprise networks, pay an Internet Service Provider (ISP) to provide access to any valid Internet address. Some larger enterprise networks pay for separate connections to two or more ISPs. Tens of thousands of ISPs support access and aggregation. Finally, just over 6000 ISP networks  form the Internet backbone by exchanging traffic over a sparse mesh of interconnections that use common link protocols and link technology, but two different business arrangements: “Internet transit” and “peering.” With transit, the upstream ISP offers to handle traffic for any Internet address. In peering, operators forward only those packet destined for subscribers on the peer’s network. This difference is key for the economics discussed in section 3.
2. Interconnection Protocols & Technology
All links transport Internet Protocol (IP) data packets with public IP addresses. Peering links (and many transit links) use Border Gateway Protocol (BGP) (currently BGPv4 per RFC 4271) to advertise the IP address ranges for which they will accept traffic. Because BGP is deployed on a link-by-link basis, there is room to negotiate BGP options (of which there are many) in support of the exchange policies of the specific operators.
Typically, each ISP arranges their own data transport circuits to a common meeting point where each ISP has their own router. The actual peering link is usually an Ethernet connection between these routers, either directly or via an in-building peering fabric (effectively an Ethernet switch provided by an Internet exchange operator).
3. Peering economics
For local or regional ISPs, peering economics are dominated by where you are and how much traffic you have. For example, a regional ISP in mid-state Illinois will have limited options for any kind of Internet transit and no local options for peering. Fiber connections from the incumbent telephone company are widely available but expensive. Competing local fiber is rare while long haul fiber routes have relatively few physical access points. Outside of major cities, the rates for Internet transit service can be 10x to 100x greater than at a neutral data center in a major hub like Chicago. So one tradeoff is between locally purchased Internet transit and the combined cost of a transport circuit to Chicago plus Internet transit purchased in Chicago. But additional benefits at a hub like Chicago include the ability to peer with Netflix, Google, Amazon and many others. This can offload 1/2 to 2/3 of the traffic, dramatically reducing the bill for Internet transit.
While the preceding example was a small ISP, the same principals apply at every scale. The goal is to deliver as much of your customers’ traffic directly to the networks with the desired destination addresses and minimize traffic that must transit multiple networks. Larger carriers are typically present in multiple cities and multiple data centers where additional considerations come into effect. The first is “hot-potato routing.” Internet traffic is typically handed off at the first available opportunity. Depending on hand-off points, this could result in one party carrying most cross-country traffic while the other party carries mostly local traffic. To counter this, large carriers’ peering policies typically require connectivity at multiple hubs and relatively balanced traffic flows. The second consideration is economic clout coming from the scale and nature of the operator’s traffic. This shifts over time and among operators leading to peering disputes that have briefly made portions of the Internet unreachable for some users.
4. History and futures
When the NSFnet was turned off in 1995, there were six commercial backbone carriers that did settlement-free peering at four major Internet exchange points. These so-called Tier 1 carriers attempted to form a cartel that did not peer beyond the initial group. However, traffic grew more rapidly than original providers could handle and, by the early 2000s, large groups of secondary carriers were exchanging traffic between each other, effectively forming a donut around the original tier 1 carriers, so that by the mid-2000s the original cartel was irrelevant .
The next major evolution began with the emergence of content delivery networks (CDN) like Akamai in the early 2000s. Whether web surfing or buffering video, lower round trip latency improves user experience, so there is big incentive to move content closer to the user. This also reduces the amount of data that must be carried long distances. Larger networks deploy CDN servers within their networks, so most such traffic is entirely locally. Additionally, to reach the broadest number of networks, most CDN providers host CDN servers at major Internet exchanges. By 2010, the majority of inter-domain traffic went to CDNs and Google had emerged as the 2nd largest ISP in the world, by volume .
The recent trend is the emergence of multi-lateral peering sites where dozens or hundreds of networks exchange traffic at one location or across one tightly interconnected set of buildings. These peering points arose first in Europe and Asia, where they reduced the amount of local traffic that was routed to the US and back . For example, there are 144 networks interconnected at the Hong Kong Internet Exchange. The group open-ix.org is trying to bring this model to North America. In any event, the focus is on shorter paths and less long haul data transmission as this cuts costs and improves user experience.
5. Peering in practice
A 2011 survey of 4,331 ISP networks (86% of backbone carriers) analyzed 142,210 inter-carrier interconnection agreements with some interesting results . 99.5% were handshake agreements based on commonly understood peering principles, without any written contract, and 99.7% were symmetric. So settlement-free peering dominates. While the majority of networks have less than ten peers, a number of multi-lateral peering agreements are visible in the data since participating operators have hundreds of peers.
Because of the costs in setting up and managing interconnections, there are normally minimum traffic conditions for peering. For example, Google peers at 70+ Internet exchanges and 60+ other facilities around the world. Under their policy, networks at any of these exchanges can peer if they have adequate traffic destined for Google: 100 Mbps for US or EU peering, 25 Mbps for Asia but no minimums at African or South American Internet exchanges .
6. Peering Politics
As mentioned above, economic clout plays a role in large carrier interconnection agreements, and occasionally, a breakdown in negotiations results in service disruptions for some users. For example, a November 2010 peering dispute between Level 3 and Comcast arose when Netflix changed CDN partners, from Akamai to Level 3, driving an enormous increase in traffic on Comcast-Level 3 interconnection links. How this was resolved is not public, but the two parties came to some agreement and upgraded links so Comcast customers could again get access to Netflix videos, although nearly three years passed before they announced they had a final agreement .
Strong competition in the Internet backbone causes prices to reflect marginal costs, however access networks in the US and many countries are monopolies or duopolies under dramatically less competitive pressure. Recently, major consumer access ISPs like Verizon, Time Warner, Comcast and France Telecom have been using the size of their subscriber bases to demand payments from major content providers like Netflix and Google [8, 9]. How this will play out remains a matter of speculation (and politics).
The Internet is a voluntary agreement among network operators to exchange traffic for their mutual benefit . This exchange is remarkable: essentially unregulated, to a great extent informal, not well understood by outsiders, and yet, successfully responding to explosive growth and dramatic technical change for nearly two decades. And, all indications are the system will continue to work for decades to come.
 “Key ICT indicators for developed and developing countries and the world (totals and penetration rates),” International Telecommunications Unions (ITU), Geneva, 27 February 2013
 Smith, Philip, “Weekly Routing Table Report,” October 10, 2013, Transit ASes http://thyme.rand.apnic.net/current/data-summary
 Labovitz, Craig, et al, “Internet Inter-domain Traffic,” Proceedings IEEE SigComm ’10, http://doi.acm.org/10.1145/1851275.1851194
 Woodcock, Bill and Vijay Adhikari, “Survey of Characteristics of Internet Carrier Interconnection Agreements,” Packet Clearing House, May 2, 2011,https://www.pch.net/resources/papers/peering-survey/PCH-Peering-Survey-2011.pdf
 Google peering policy, https://peering.google.com/about/peering_policy.html, accessed Oct 10, 2013
 Engebretson, Joan, “Behind the Level 3 – Comcast Peering Settlement,” Telecompetitor, July 17, 2013, http://www.telecompetitor.com/behind-the-level-3-comcast-peering-settlement/
 Ramachandran, Shalini and Drew Fitzgerald, “For Web Firms, Faster Access Comes at a Price”, Wall Street Journal Online, June 20, 2013,http://online.wsj.com/news/articles/SB10001424127887323836504578553170167992666
 Higginbotham, Stacey, “Peering Pressure: The secret battle to control the future of the Internet,” Gigaom, June 19, 2013, http://gigaom.com/2013/06/19/peering-pressure-the-secret-battle-to-control-the-future-of-the-internet/
 Goldstein, Fred, “What does it Mean to be Internet?”, TMCnews, June 8, 2009, http://ipcommunications.tmcnet.com/topics/ip-communications/articles/57573-what-does-it-mean-be-internet.htm