Linux Advanced Routing Tutorial
For years, we used to have a plain-old ADSL in the office—fast download speeds, slow upload, high latency—all that at the cost of $1/GB. We have had so many problems with performance and reliability that after a few years of struggling, we decided to get a second upstream link—SHDSL 5M/5M symmetric link—low latency, consistent speed during the day. It's simply awesome.
Figure 1. Network Overview
However, SHDSL is expensive! Although national traffic is free of charge, international traffic is $5/GB. Compare that with ADSL at $1/GB for both national and international (Table 1).
|Speed||7M down/1M up||5M down/5M up|
|Public IP addresses||1||1 + /28|
There are clear pros and cons to both. That made me think (and I don't do that often), what if we kept ADSL for international traffic that's not that critical for us, and use SHDSL primarily for national traffic—fast and cheap?
Figure 2. Split Traffic between International and National
We also got a /28 subnet with the SHDSL to use for a DMZ, and obviously, we want all the traffic to and from the DMZ to go via the SHDSL link, regardless of whether it's national or international.
Figure 3. Traffic to/from the DMZ
For my plan to work, the company core router needs some way to tell whether a packet from a workstation in the office to an IP.AD.DR.ES in the wild open Internet is international or national and send it accordingly through ADSL or SHDSL. Routers use their routing tables for deciding the fates and paths of the packets they forward. A simple routing table for an office workstation with an address of 192.168.130.100 may look like this:
[workstation] ~ # ip route show 192.168.130.0/24 dev eth0 proto kernel scope link ↪src 192.168.130.100 default via 192.168.130.1 dev eth0
This says that all traffic to all other addresses in the 192.168.130.0/24 range is to be sent straight to the local network segment attached to interface eth0. All other traffic follows the "default" route and is handed over to the router with IP 192.168.130.1 to handle it. Let's assume we're sending a packet to 184.108.40.206, Google's public DNS server. For starters, we're trying to "ping 220.127.116.11" and are observing the traffic on the workstation's eth0 interface with tcpdump:
[workstation] ~ # tcpdump -i eth0 -n -s0 -e listening on eth0, link-type EN10MB (Ethernet), ↪capture size 65535 bytes  17:53:59.615650 00:16:17:ec:5c:6c > ff:ff:ff:ff:ff:ff, ↪ethertype ARP (0x0806), length 42: Request who-has 192.168.130.1 tell 192.168.130.100, ↪length 28  17:53:59.615775 00:14:c2:5b:4f:2c > 00:16:17:ec:5c:6c, ↪ethertype ARP (0x0806), length 60: Reply 192.168.130.1 is-at 00:14:c2:5b:4f:2c, length 46  17:53:59.615796 00:16:17:ec:5c:6c > 00:14:c2:5b:4f:2c, ↪ethertype IPv4 (0x0800), length 98: 192.168.130.100 > 18.104.22.168: ICMP echo request, ↪id 3082, seq 1, length 64
My workstation consulted the routing table for the destination IP 22.214.171.124 and realized it should send the packet to the default router 192.168.130.1. For that, it needs the router's low-level Ethernet address (also known as a MAC address), and the first packet in the above tcpdump output, marked , is doing exactly that—asking for the MAC address of IP 192.168.130.1 by "broadcasting" the request to all nodes on the network segment. The second packet, marked , is a reply—IP 192.168.130.1 is at MAC address 00:14:c2:5b:4f:2c. Finally, the PING packet can be dispatched, with the destination IP 126.96.36.199 to the router's MAC address 00:14:c2:5b:4f:2c (see the line marked ).
All good, so now we can assume our router got the packet and will forward it further toward the destination. Let's see what happens on the router.
We've got both the ADSL and the SHDSL links configured, but all traffic is, by default, sent through ADSL. The ADSL modem is at 192.168.128.254. For now, the SHDSL link 203.0.113.36/30 sits idle. Here is the routing table:
[router] ~ # ip route show  203.0.113.36/30 dev vlan-shdsl proto kernel ↪scope link src 203.0.113.38  192.168.128.0/24 dev vlan-adsl proto kernel ↪scope link src 192.168.128.1  192.168.130.0/24 dev vlan-office proto kernel ↪scope link src 192.168.130.1  default via 192.168.128.254 dev vlan-adsl
The first line  is the SHDSL link—our router's IP on that link is
203.0.113.38. The second line is the link to the ADSL modem; the third
line  is the network segment with my workstation, and finally, the fourth
line  is the default route. All packets that don't match any of the
local subnets on lines 1, 2 or 3 are sent down to the ADSL modem at
192.168.128.254 that then will forward them to the ISP 2. That's also the
fate of our packet to 188.8.131.52. Let's quickly verify what is going to
happen to it by calling
ip route get:
[router] ~ # ip route get 184.108.40.206 from 192.168.130.100 ↪iif vlan-office 220.127.116.11 from 192.168.130.100 via 192.168.128.254 dev vlan-adsl
As you can see, it will be sent "via 192.168.128.254", which is the ADSL modem. A simple way to check the full network path from my workstation to any given destination address is the traceroute command. It shows all the routers ("hops") along the way to the destination:
[workstation] ~ $ /usr/sbin/traceroute 18.104.22.168 traceroute to 22.214.171.124 (126.96.36.199), 30 hops max, ↪40 byte packets using UDP 1 gw-vlan-office.e-it.co.nz (192.168.130.1) ↪0.156 ms 0.126 ms 0.124 ms 2 gw-vlan-adsl.e-it.co.nz (192.168.128.254) ↪0.853 ms 0.831 ms 0.830 ms 3 core-adsl.isp2 (218.101.x.y) ↪11.765 ms 19.173 ms 19.066 ms 4 core-xyz.isp2 (203.98.x.y) ↪16.052 ms 15.515 ms 17.153 ms [... some more hops ...] 13 64.233.x.y (64.233.x.y) 193.826 ms 194.230 ms 194.412 ms 14 * * * 15 google-public-dns-b.google.com (188.8.131.52) ↪196.086 ms 195.909 ms 195.816 ms
As you can see, the first hop is our router 192.168.130.1. The second hop is the ADSL modem 192.168.128.254. The third hop is one of the ISP2's core routers, and so on and on, passing 11 more routers before the packet finally reaches the destination 184.108.40.206, aka google-public-dns-b.google.com.
Michal Ludvig works for Enterprise IT Ltd in New Zealand as a senior Linux engineer. He's got root access to some of the largest New Zealand corporations, but since he tends to forget his passwords, he is, in general, pretty harmless.
Practical books for the most technical people on the planet. Newly available books include:
- Agile Product Development by Ted Schmidt
- Improve Business Processes with an Enterprise Job Scheduler by Mike Diehl
- Finding Your Way: Mapping Your Network to Improve Manageability by Bill Childers
- DIY Commerce Site by Reven Lerner
Plus many more.
- Building a Multisourced Infrastructure Using OpenVPN
- Happy GPL Birthday VLC!
- Unikernels, Docker, and Why You Should Care
- diff -u: What's New in Kernel Development
- What's New in 3D Printing, Part III: the Software
- Giving Silos Their Due
- Controversy at the Linux Foundation
- Don't Burn Your Android Yet
- Non-Linux FOSS: Snk
- Firefox OS