Linux Advanced Routing Tutorial

For years, we used to have a plain-old ADSL in the office—fast download speeds, slow upload, high latency—all that at the cost of $1/GB. We have had so many problems with performance and reliability that after a few years of struggling, we decided to get a second upstream link—SHDSL 5M/5M symmetric link—low latency, consistent speed during the day. It's simply awesome.

Figure 1. Network Overview

However, SHDSL is expensive! Although national traffic is free of charge, international traffic is $5/GB. Compare that with ADSL at $1/GB for both national and international (Table 1).

Table 1. ADSL vs. SHDSL
ADSL SHDSL
Speed 7M down/1M up 5M down/5M up
Price national $1/GB Free
Price international $1/GB $5/GB
Public IP addresses 1 1 + /28

There are clear pros and cons to both. That made me think (and I don't do that often), what if we kept ADSL for international traffic that's not that critical for us, and use SHDSL primarily for national traffic—fast and cheap?

Figure 2. Split Traffic between International and National

We also got a /28 subnet with the SHDSL to use for a DMZ, and obviously, we want all the traffic to and from the DMZ to go via the SHDSL link, regardless of whether it's national or international.

Figure 3. Traffic to/from the DMZ

Routing 101

For my plan to work, the company core router needs some way to tell whether a packet from a workstation in the office to an IP.AD.DR.ES in the wild open Internet is international or national and send it accordingly through ADSL or SHDSL. Routers use their routing tables for deciding the fates and paths of the packets they forward. A simple routing table for an office workstation with an address of 192.168.130.100 may look like this:


[workstation] ~ # ip route show
192.168.130.0/24 dev eth0 proto kernel  scope link  
 ↪src 192.168.130.100
default via 192.168.130.1 dev eth0

This says that all traffic to all other addresses in the 192.168.130.0/24 range is to be sent straight to the local network segment attached to interface eth0. All other traffic follows the "default" route and is handed over to the router with IP 192.168.130.1 to handle it. Let's assume we're sending a packet to 8.8.4.4, Google's public DNS server. For starters, we're trying to "ping 8.8.4.4" and are observing the traffic on the workstation's eth0 interface with tcpdump:


[workstation] ~ # tcpdump -i eth0 -n -s0 -e
listening on eth0, link-type EN10MB (Ethernet), 
 ↪capture size 65535 bytes
[1] 17:53:59.615650 00:16:17:ec:5c:6c > ff:ff:ff:ff:ff:ff, 
 ↪ethertype ARP (0x0806), length 42:
        Request who-has 192.168.130.1 tell 192.168.130.100, 
        ↪length 28
[2] 17:53:59.615775 00:14:c2:5b:4f:2c > 00:16:17:ec:5c:6c, 
 ↪ethertype ARP (0x0806), length 60:
        Reply 192.168.130.1 is-at 00:14:c2:5b:4f:2c, length 46
[3] 17:53:59.615796 00:16:17:ec:5c:6c > 00:14:c2:5b:4f:2c, 
 ↪ethertype IPv4 (0x0800), length 98:
        192.168.130.100 > 8.8.4.4: ICMP echo request, 
        ↪id 3082, seq 1, length 64

My workstation consulted the routing table for the destination IP 8.8.4.4 and realized it should send the packet to the default router 192.168.130.1. For that, it needs the router's low-level Ethernet address (also known as a MAC address), and the first packet in the above tcpdump output, marked [1], is doing exactly that—asking for the MAC address of IP 192.168.130.1 by "broadcasting" the request to all nodes on the network segment. The second packet, marked [2], is a reply—IP 192.168.130.1 is at MAC address 00:14:c2:5b:4f:2c. Finally, the PING packet can be dispatched, with the destination IP 8.8.4.4 to the router's MAC address 00:14:c2:5b:4f:2c (see the line marked [3]).

All good, so now we can assume our router got the packet and will forward it further toward the destination. Let's see what happens on the router.

We've got both the ADSL and the SHDSL links configured, but all traffic is, by default, sent through ADSL. The ADSL modem is at 192.168.128.254. For now, the SHDSL link 203.0.113.36/30 sits idle. Here is the routing table:


[router] ~ # ip route show
[1] 203.0.113.36/30 dev vlan-shdsl  proto kernel  
 ↪scope link  src 203.0.113.38
[2] 192.168.128.0/24 dev vlan-adsl  proto kernel  
 ↪scope link  src 192.168.128.1
[3] 192.168.130.0/24 dev vlan-office  proto kernel  
 ↪scope link  src 192.168.130.1
[4] default via 192.168.128.254 dev vlan-adsl

The first line [1] is the SHDSL link—our router's IP on that link is 203.0.113.38. The second line is the link to the ADSL modem; the third line [3] is the network segment with my workstation, and finally, the fourth line [4] is the default route. All packets that don't match any of the local subnets on lines 1, 2 or 3 are sent down to the ADSL modem at 192.168.128.254 that then will forward them to the ISP 2. That's also the fate of our packet to 8.8.4.4. Let's quickly verify what is going to happen to it by calling ip route get:


[router] ~ # ip route get 8.8.4.4 from 192.168.130.100 
 ↪iif vlan-office
8.8.4.4 from 192.168.130.100 via 192.168.128.254 dev vlan-adsl

As you can see, it will be sent "via 192.168.128.254", which is the ADSL modem. A simple way to check the full network path from my workstation to any given destination address is the traceroute command. It shows all the routers ("hops") along the way to the destination:


[workstation] ~ $ /usr/sbin/traceroute 8.8.4.4
traceroute to 8.8.4.4 (8.8.4.4), 30 hops max, 
 ↪40 byte packets using UDP
 1  gw-vlan-office.e-it.co.nz (192.168.130.1)  
    ↪0.156 ms  0.126 ms  0.124 ms
 2  gw-vlan-adsl.e-it.co.nz (192.168.128.254)  
    ↪0.853 ms  0.831 ms  0.830 ms
 3  core-adsl.isp2 (218.101.x.y)  
    ↪11.765 ms  19.173 ms  19.066 ms
 4  core-xyz.isp2 (203.98.x.y)  
    ↪16.052 ms  15.515 ms  17.153 ms
[... some more hops ...]
13  64.233.x.y (64.233.x.y)  193.826 ms  194.230 ms  194.412 ms
14  * * *
15  google-public-dns-b.google.com (8.8.4.4)  
    ↪196.086 ms  195.909 ms  195.816 ms

As you can see, the first hop is our router 192.168.130.1. The second hop is the ADSL modem 192.168.128.254. The third hop is one of the ISP2's core routers, and so on and on, passing 11 more routers before the packet finally reaches the destination 8.8.4.4, aka google-public-dns-b.google.com.

______________________

Michal Ludvig works for Enterprise IT Ltd in New Zealand as a senior Linux engineer. He's got root access to some of the largest New Zealand corporations, but since he tends to forget his passwords, he is, in general, pretty harmless.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

cool

bahce00's picture

Coverage of the 2007 MWRC has been fairly comprehensive.
http://bahcemobilyam.blogspot.com

very good, post more please

florian's picture

very good post... more on routing with bird/quagga and or advanced routing with linux is welcome. thanks.

nice read! bird config file

Moosa's picture

nice read! bird config file killed the purpose of obscuring AS no. in the table.(just saying)

Very nice and informative.

khappieinstein's picture

Very nice and informative. Inspired me to start Experimenting again. Thanks again. Keep writing. It been long i have touch base with LINUX.

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState