WIX: a Distributed Internet Exchange
Border Gateway Protocol (BGP) is fundamental to the operation of the Internet, because it automates the sharing of routes by a process called advertising. Adjacent routers establish a BGP session and advertise their networks to the other routers. It is a bit like one router saying to another, “Here is my IP address. If you have traffic with any of the following destination addresses, send it to me.” BGP doesn't stop there, though. Routers also share information they have gathered from other routers. They say, “I also know how to get traffic to some other networks. Send that to me as well.”
Typical Internet exchanges are different from the rest of the Internet and use Internal Gateway Protocol (IGP) instead. This protocol does not pass on information to other routers—it only advertises routes within the networks to which it is connected. The router says, “You can send me traffic for these addresses, but you can't pass that address information on to any other routers.”
In 1998, Citylink started work on deploying a BGP/4-based Internet exchange on top of the public Ethernet. One of the key design criteria was a low cost of entry. So the Citylink team took a step back and looked at the exchange problem from a logical point of view: what were the core requirements of an exchange, and how could they use the existing fibre infrastructure to create an environment that would encourage as many people as possible to join? Exchanging traffic via an exchange is called peering.
A key component of Citylink's approach involved placing routers at a customer's point of connection to the fibre infrastructure rather than in a central location. Not a lot of options were available for BGP/4-capable routers, though—you pretty much bought a Cisco or a Cisco (Figure 2).
Because the exchange was distributed, Citylink needed a low-cost mechanism to allow people to peer, and so they started using Zebra on Linux as an alternative routing platform to Cisco.
Because of the limited space in clients' cabinets and the need to be able to buy the same hardware over a reasonable period of time, Citylink now has standardized on two systems. Boards from Soekris Engineering (4501 and 4801) are used for routers up to 30Mb/s, while Nexcom P4 GigE and VIA 100Mb boards are used for 100Mb/s connections and upwards to 300–500Mb/s on a good day. The Linux Embedded Appliance Firewall (LEAF) forms the basis of all software used on Citylink routing hardware.
Customers can use the Citylink routers or whatever suits their needs and budgets. A wide variety are in use—from Dlink and Xyzel at the low end to Juniper and Cisco boxes at the high end. A lot of people also use their own Linux and *BSD boxes. Using PC-based routers does limit the options for network interfaces, but this is not an issue for Citylink, which uses only Ethernet.
In addition to devolving hardware to client's premises, the routing tables for the whole exchange are managed by way of route servers, the operation of which I explain later in this article.
The new exchange was dubbed WIX, the Wellington Internet Exchange. There is a fixed monthly charge for connection, and traffic over the fibre network is free. Because the network often runs right past a potential user's door, it is easy for anyone to connect and peer. And this is exactly what has happened—even end users who could never peer under a traditional model now can. For all exchange users, access to the global Internet still necessitates the purchase of “transit”—access to the global Internet routing table—from at least one ISP.
The open peering approach has made a huge impact on traffic flows and latency. When the Internet first started in New Zealand, the country's single exchange point, at Waikato University, was also the international gateway for the whole country. Two businesses in Wellington wanting to exchange data would send it to their respective ISPs, who sent it upstream to Waikato. The path time for this typically ran 50–200ms. When ISPs began to exchange data directly in Wellington, this rate dropped to 20ms. End-user peering reduced this by another factor of ten, to 2ms.
In addition to shorter and faster network paths, no charges are levied for traffic sent over Citylink by way of the exchange. ISPs never see the traffic, because the exchange directs it through the shortest path to the requesting party.
A number of printing companies peer at WIX and exchange huge publishing files at no cost. One local newspaper runs an FTP server on WIX to which pre-press companies can upload files. Some graphics houses also run media stores for their clients, and these can browsed at no cost, as though they were part of the client's own LAN. These are only a few of the ways that the fibre infrastructure and WIX have helped local businesses to lower costs, expand and innovate.
The distributed exchange environment has been likened to a market square—anyone can trade his or her wares with no cost for participation. Not everyone chooses to peer openly though, and the exchange supports all types of participation. A number of ISPs exchange traffic only with established customers and ignore any other traffic. Some simply use WIX as a neutral point to exchange data with one other party.
About 130 entities are peering at WIX, with close to 1,000 using the Citylink fibre for private purposes or public Internet connectivity. Encouraging peering comes with a downside though—a lot of participants means a lot of routing tables need to be managed.
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Speed Up Your Web Site with Varnish||Jun 19, 2013|
|Non-Linux FOSS: libnotify, OS X Style||Jun 18, 2013|
|Containers—Not Virtual Machines—Are the Future Cloud||Jun 17, 2013|
|Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer||Jun 12, 2013|
|Weechat, Irssi's Little Brother||Jun 11, 2013|
|One Tail Just Isn't Enough||Jun 07, 2013|
- Speed Up Your Web Site with Varnish
- Containers—Not Virtual Machines—Are the Future Cloud
- Linux Systems Administrator
- Lock-Free Multi-Producer Multi-Consumer Queue on Ring Buffer
- Senior Perl Developer
- Technical Support Rep
- Non-Linux FOSS: libnotify, OS X Style
- UX Designer
- RSS Feeds
- It is quiet helping
1 hour 48 min ago
2 hours 5 min ago
- Reachli - Amplifying your
3 hours 21 min ago
4 hours 10 min ago
- good point!
4 hours 13 min ago
- Varnish works!
4 hours 22 min ago
- Reply to comment | Linux Journal
4 hours 51 min ago
- Reply to comment | Linux Journal
7 hours 17 min ago
- Reply to comment | Linux Journal
11 hours 17 min ago
- Yeah, user namespaces are
12 hours 33 min ago