Home ›

Overcoming Asymmetric Routing on Multi-Homed Servers

Apr 05, 2004  By Patrick McManus
 in
Did you pay for two network interfaces and find your server is using only one? Balance the traffic with some simple routing hints.
Experimental Results

Let's now see the results of this technique play out during a real Web serving test. The test consists of transferring a 90KB file 20,000 times. The HTTP transactions are load-balanced across the server's two IP addresses, with an average of 40 connections being performed in parallel.

The ifconfig command reports on an interface's packet counters. Listing 2 shows the output of the ifconfig command after running the test on a vanilla Web server that does not employ the source-based routing approach.

Listing 2. Interface Counters with Destination-Based Routing 

eth0 Link encap:Ethernet  HWaddr 00:E1:AA:7C:51:2C
     inet addr:192.168.16.20  Bcast:192.168.16.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:328008 errors:0 dropped:0 overruns:0 frame:0
     TX packets:1341151 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100
     RX bytes:23963417 (22.8 Mb)  TX bytes:1908125938 (1819.7 Mb)
     Interrupt:19 Base address:0xe400 Memory:dff80000-dffa0000

eth1 Link encap:Ethernet  HWaddr 00:E1:AA:7C:51:2D
     inet addr:192.168.16.21  Bcast:192.168.16.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:346430 errors:0 dropped:0 overruns:0 frame:0
     TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100
     RX bytes:25250075 (24.0 Mb)  TX bytes:0 (0.0 b)
     Interrupt:16 Base address:0xec00 Memory:dffa0000-dffc0000

The server's received traffic, which consists of HTTP requests and TCP acknowledgments for the HTTP responses, is well balanced at roughly 330,000 packets received by each interface. However, the transmission traffic has fallen prey to the asynchronous route problem: interface eth0 has transmitted 1.3 million packets where eth1 has not transmitted any.

Listing 3 contains the output of ifconfig after rebooting the server to clear the interface counters and employing the iproute2 strategy discussed in this article. The test then was run again in the same manner as above.

Listing 3. Interface Counters with Policy Based Routing

eth0 Link encap:Ethernet  HWaddr 00:E1:AA:7C:51:2C
     inet addr:192.168.16.20  Bcast:192.168.16.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:332371 errors:0 dropped:0 overruns:0 frame:0
     TX packets:670341 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100
     RX bytes:24270910 (23.1 Mb)  TX bytes:954045844 (909.8 Mb)
     Interrupt:19 Base address:0xe400 Memory:dff80000-dffa0000

eth1 Link encap:Ethernet  HWaddr 00:E1:AA:7C:51:2D
     inet addr:192.168.16.21  Bcast:192.168.16.255  Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:334110 errors:0 dropped:0 overruns:0 frame:0
     TX packets:670152 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100
     RX bytes:24387875 (23.2 Mb)  TX bytes:954032082 (909.8 Mb)
     Interrupt:16 Base address:0xec00 Memory:dffa0000-dffc0000

The server's received traffic remains well balanced, but the transmission traffic now is equalized at 670,000 packets for each interface.

Conclusion

Source-based routing capabilities are common on high end networking gear, but they rarely are seen or utilized in server environments. Linux has excellent but poorly understood source-based routing support. The whole universe of advanced Linux routing and traffic shaping is well described at lartc.org.

Resources

Effects of Network Asymmetry on TCP Performance: www.eecs.berkeley.edu/IPRO/Summary/97abstracts/padmanab.1.html

Linux Advanced Routing and Traffic Control: www.lartc.org

Patrick McManus (mcmanus@ducksong.com) works as a software engineer for Datapower Technology, near his home in Boston, Massachusetts. He currently is obsessed with reading a biography of each American president.

______________________

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

Local Traffic

Carson Gee's picture
Submitted by Carson Gee (not verified) on Mon, 09/27/2010 - 09:32.

I just ran into this today when fixing some routes. If you want those two interfaces to send traffic normally on their local network ( 192.168.16.0/24 ) without going through the gateway and forming an asymmetric route with hosts on that network you'll need to add:

#ip route add 192.168.16.0/24 dev eth0 tab 1
#ip route add 192.168.16.0/24 dev eth1 tab 2

to use link routing on the local subnet.

Ubuntu ip route commands - what file do I put them in?

tc0nn's picture
Submitted by tc0nn on Sun, 04/04/2010 - 11:07.

So, I tried /etc/network/if-up.d/ip and /etc/rc.local, but all routing breaks when the box reboots. Where should I put these? Currently, I let the box boot up, then run the commands manually and everything works great. Any suggestions?

1. vi

Anonymous's picture
Submitted by Anonymous (not verified) on Fri, 07/09/2010 - 09:20.

1.
vi /etc/init.d/iproutes-asym and add the commands you need in there
chmod 755 /etc/init.d/iproutes-asym
2.
cd /etc/rc3.d
ln -s ../init.d/iproutes S99z-iproutes-asym

this is what my iproutes-asym file looks like

ip route add default via 10.53.1.252 dev eth0 tab 1
ip route add default via 10.53.1.252 dev eth1 tab 2
ip rule add from 10.53.1.55/32 tab 1 priority 500
ip rule add from 10.53.1.54/32 tab 2 priority 600
ip route flush cache

Muchas gracias

anomie's picture
Submitted by anomie (not verified) on Wed, 01/28/2009 - 15:18.

Thanks for putting this together. Proper routing on a multi-homed server is poorly documented by my Linux distro vendor. Your article was a great help in understanding iproute2 (in this context) and getting things working properly.

solutions

Bgs's picture
Submitted by Bgs (not verified) on Wed, 07/02/2008 - 17:29.

Network interface level problem can be solved with bonding too and it's easier to manage. iproute2 can be used to have multiple loadbalancers and/or gateways though.

Need some HELP for linux asymmetric routing

Michael Rack's picture
Submitted by Michael Rack (not verified) on Thu, 05/31/2007 - 06:44.

Hello Friends! I have two ISP-Links from the same Service-Provider. I got for each link an IP-Address on Subnet /30. eth0 runs on x.x.24.66, and eth1 on x.x.24.234.

The default-route is set to x.x.24.233 dev eth1. Now, when a ICMP-Ping reached by x.x.24.234 on eth1, ping will be responded. When a ping reached by x.x.24.66 on eth0, nothing happens.

The ICMP-Ping-Request pass the eth0-interface, but will not be responded via eth1 (default-route)... When i listen on eth1 with tcpdump, there no outgoing-packets to handle ICMP-Responses.

Whats the problem?

Thanks, Mike.
http://www.michaelrack.de

Thank you! Also..

Nathan Dornquast's picture
Submitted by Nathan Dornquast (not verified) on Sat, 05/20/2006 - 00:45.

Patrick,

Thank you! I have been struggling with this for weeks. I wish I had found this article first. This is the first time I have found a good explanation of rules and tables and their relationship in the same place.

Regarding SNAT. I listed two source addresses in my iptables firewall.. it mostly works well. However, some outbound connections fail - most noteably SSH, Yahoo IM, IRC all reset after a short time (though web traffic seems ok). I can SNAT to one of my outbound addresses and use an ip rule to designate a single gateway. This works, but I am no longer NAT load balancing over my two WAN links. Anyone know a solution?

-Nathan

Thank you

atrix's picture
Submitted by atrix (not verified) on Thu, 05/11/2006 - 06:51.

Thank you very much for this excellent article
Best wishes

Super

Cristian C.'s picture
Submitted by Cristian C. (not verified) on Fri, 05/20/2005 - 06:07.

Very nice and educative article. Good reading.

Re: Overcoming Asymmetric Routing on Multi-Homed Servers

Anonymous's picture
Submitted by Anonymous on Tue, 04/06/2004 - 02:00.

Minimalist load balancer. From lartc.org section 4.2.2

# ip route default nexthop via gw_1 nexthop via gw_2

Mohammad Bahathir Hashim
Malaysia.

rules vs. nat

Anonymous's picture
Submitted by Anonymous on Tue, 04/06/2004 - 02:00.

What about the SNAT target in iptables? It modifies the source IP address of the packet, but applies only in the POSTROUTING chain. Are the rules (the policy) evaluated *after* that again? The name POSTROUTING makes me think the routing part is already over...

Re: rules vs. nat

SeanW's picture
Submitted by SeanW (not verified) on Tue, 04/06/2004 - 02:00.

If it's anything like a Cisco router, outbound NAT happens after policy routing, and doesn't get another chance at the policy engine.

Sean

Re: rules vs. nat

Anonymous's picture
Submitted by Anonymous on Wed, 04/07/2004 - 02:00.

The SNAT target allows you to specify multiple source ip's and they will be used one after the other. That would probably give you simple outbound load-balancing.

From the iptables man page:

You can add several --to-source options. If you specify more than one source address, either via an address range or multiple --to-source options, a simple round-robin (one after another in cycle) takes place between these addresses.

L2 vs L3...

SeanW's picture
Submitted by SeanW (not verified) on Mon, 04/05/2004 - 02:00.

Great article on policy routing, but isn't this problem what bonding was designed to solve?

http://linux-ip.net/html/ether-bonding.html
/usr/src/linux-2.4/Documentation/networking/bonding.txt

Sean

Re: L2 vs L3...

Anonymous's picture
Submitted by Anonymous on Tue, 04/06/2004 - 02:00.

From the link you posted:

" Bonding for link aggregation must be supported by both endpoints."

"Bonding for link aggregation

p5k's picture
Submitted by p5k (not verified) on Wed, 07/13/2005 - 14:55.

"Bonding for link aggregation must be supported by both endpoints."

Sounds like something our marriage therapist once told my (now ex-) wife and I... ;) Needless to say, it was NOT supported by *both* endpoints!

Webcast
More Resources
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers

Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.

Learn More

Sponsored by AMD

White Paper
More Resources
Red Hat White Paper: Using an Open Source Framework to Catch the Bad Guy

Built-in forensics, incident response, and security with Red Hat Enterprise Linux 6

Every security policy provides guidance and requirements for ensuring adequate protection of information and data, as well as high-level technical and administrative security requirements for a system in a given environment. Traditionally, providing security for a system focuses on the confidentiality of the information on it. However, protecting the data integrity and system and data availability is just as important. For example, when processing United States intelligence information, there are three attributes that require protection: confidentiality, integrity, and availability.

Learn more about catching the bad guy in this free white paper.

Learn More

Sponsored by DLT Solutions