Policy Routing for Fun and Profit
The default route of the ADSL routing table is ppp0, representing a PPP over Ethernet (PPPoE) link. The Ethernet then is encapsulated into ATM (EoA), and it is ATM cells that traverse the ADSL link to the telco DSLAM.
If the ppp0 interface goes down, the ADSL default route is removed automatically by the kernel and replaced with the main routing table. Thus, if the ADSL line fails, all traffic destined for the ADSL routing table is diverted to the presumably more reliable main routing table. We do get the occasional ADSL outages that are endemic to low cost, unmanaged broadband services like ADSL. These outages last from a few seconds to several hours, but there is no loss of user functionality because the traffic switches transparently to the T1 line. The T1 interface is good backup for the ADSL line, but the reverse is not true. Most of the hosts that use the T1 link do so because they need to use fixed IP addresses; they cannot be serviced adequately with the ADSL line that has a non-fixed IP address.
The default route of the main routing table is wan0 (T1). All traffic coming into this routing table is forwarded to the T1 line.
Outgoing Internet traffic over the ADSL connection comes from servers with routable IP addresses. These address types need to be NATed; otherwise, the traffic routed to the real IP addresses comes back over the T1 line:
iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE
Our tagging and policy routing is shown in Figure 4.
Once we have directed the appropriate traffic to the ADSL line, we need to manage residual T1 traffic so that the usage boundaries are never exceeded. The magic 95th percentile point always must be less than or equal to 128kbps. We first measure the traffic using IP accounting, which allows us to gauge average throughput over a specified time interval.
All incoming and outgoing packets on the T1 line pass through IP accounting rules. Each customer's traffic is measured based on the IP address and direction of the traffic.
A custom dæmon has been written to check the T1 bandwidth used for each five-minute period or bin. Each time the T1 throughput is greater than 128kbps averaged over a five-minute period, a counter is incremented. The 128kbps threshold corresponds to about 4.5MB over the five-minute period.
If the counter reaches 432, that represents the free 36 hours per month. In turn, that equals 5% of the time in a month, and the TC (traffic control) script is executed to clamp the T1 line down to 128kbps, until the start of the next month. The IP accounting configuration file is shown in Listing 2, available from the Linux Journal FTP site [ftp.linuxjournal.com/pub/lj/listings/issue121/7134.tgz].
We usually get through the month without having to clamp the T1 line. Sometimes, however, the free 36 hours are consumed. In this case, traffic control (TC) is used to clamp the bandwidth. The documentation covering traffic control and the tc command can be found at lartc.org/manpages.
We use Qdisc class-based queuing (CBQ) discipline for both the T1 line, wan0 and the local Ethernet, eth0. This is done for both connections to implement flow control in both traffic directions:
tc qdisc add dev wan0 root handle 10: \ cbq bandwidth 1500Kbit avpkt 1000 tc qdisc add dev eth0 root handle 20: \ cbq bandwidth 1500Kbit avpkt 1000
Next, add Global Class with maximum bandwidth for wan0 and eth0. The maximum bandwidth for both lines is 1,500kbps (T1):
tc class add dev wan0 parent 10:0 classid 10:1 \ cbq bandwidth 1500Kbit avpkt 1000 rate 1500Kbit \ allot 1514 weight 150Kbit prio 8 maxburst 0 tc class add dev wan0 parent 20:0 classid 20:1 \ cbq bandwidth 1500Kbit avpkt 1000 rate 1500Kbit \ allot 1514 weight 150Kbit prio 8 maxburst 0
Add User Class with limited bandwidth for both wan0 and eth0. The bandwidth limit we use is 100kbps, not 128kbps. Linux TC is not perfectly accurate, and we determined through trial and error that if we set the bandwidth limit higher than 100kbps, sometimes the burst traffic could go over 128kbps:
tc class add dev wan0 parent 10:1 classid 10:100 \ cbq bandwidth 1500Kbit avpkt 1000 rate 100Kbit \ allot 1514 weight 10Kbit prio 8 maxburst 0 bounded tc class add dev eth0 parent 20:1 classid 20:100 \ cbq bandwidth 1500Kbit avpkt 1000 rate 100Kbit \ allot 1514 weight 10Kbit prio 8 maxburst 0 bounded
Add SFQ queuing discipline for the User Class, on both wan0 and eth0. The default queuing discipline Stochastic Fairness Queueing (SFQ) has been selected. A number of other disciplines also could be employed:
tc qdisc add dev wan0 parent 10:100 \ sfq quantum 1514b perturb 15 tc qdisc add dev eth0 parent 20:100 \ sfq quantum 1514b perturb 15
Bind the traffic tagged number 2 to the User Class Queue for both wan0 and eth0. All traffic destined for the T1 line already has been tagged with number 2. The traffic control only limits the T1 traffic, while letting ADSL traffic flow at its full physical rate:
tc filter add dev wan0 parent 10:0 protocol ip \ prio 25 handle 2 fw flowid 10:100 tc filter add dev eth0 parent 20:0 protocol ip \ prio 25 handle 2 fw flowid 20:100
Free Webinar: Hadoop
How to Build an Optimal Hadoop Cluster to Store and Maintain Unlimited Amounts of Data Using Microservers
Realizing the promise of Apache® Hadoop® requires the effective deployment of compute, memory, storage and networking to achieve optimal results. With its flexibility and multitude of options, it is easy to over or under provision the server infrastructure, resulting in poor performance and high TCO. Join us for an in depth, technical discussion with industry experts from leading Hadoop and server companies who will provide insights into the key considerations for designing and deploying an optimal Hadoop cluster.
Some of key questions to be discussed are:
- What is the “typical” Hadoop cluster and what should be installed on the different machine types?
- Why should you consider the typical workload patterns when making your hardware decisions?
- Are all microservers created equal for Hadoop deployments?
- How do I plan for expansion if I require more compute, memory, storage or networking?
|Designing Electronics with Linux||May 22, 2013|
|Dynamic DNS—an Object Lesson in Problem Solving||May 21, 2013|
|Using Salt Stack and Vagrant for Drupal Development||May 20, 2013|
|Making Linux and Android Get Along (It's Not as Hard as It Sounds)||May 16, 2013|
|Drupal Is a Framework: Why Everyone Needs to Understand This||May 15, 2013|
|Home, My Backup Data Center||May 13, 2013|
- RSS Feeds
- Dynamic DNS—an Object Lesson in Problem Solving
- Making Linux and Android Get Along (It's Not as Hard as It Sounds)
- Designing Electronics with Linux
- Using Salt Stack and Vagrant for Drupal Development
- New Products
- A Topic for Discussion - Open Source Feature-Richness?
- Drupal Is a Framework: Why Everyone Needs to Understand This
- Validate an E-Mail Address with PHP, the Right Way
- What's the tweeting protocol?
- Kernel Problem
5 hours 37 min ago
- BASH script to log IPs on public web server
10 hours 4 min ago
13 hours 40 min ago
- Reply to comment | Linux Journal
14 hours 13 min ago
- All the articles you talked
16 hours 36 min ago
- All the articles you talked
16 hours 39 min ago
- All the articles you talked
16 hours 41 min ago
21 hours 5 min ago
- Keeping track of IP address
22 hours 56 min ago
- Roll your own dynamic dns
1 day 4 hours ago