Kernel Korner - Analysis of the HTB Queuing Discipline

Can Linux do Quality of Service in a way that both offers high throughput and does not exceed the defined bandwidth? Here's a thorough test.
Test Cases and Test Results

When defining the HTB configuration, the following options of the tc class commands were used in order to achieve the required results:

  • rate = the maximum bandwidth a class can use without borrowing from other classes.

  • ceiling = the maximum bandwidth that a class can use, which limits how much bandwidth the class can borrow.

  • burst = the amount of data that could be sent at the ceiling speed before moving to serve the next class.

  • cburst = the amount of data that could be sent at the wire speed before moving to serve the next class.

Most of the traffic in the Internet world is generated by TCP, so packet sizes representative of datagram sizes, such as 64 (TCP Ack), 512 (FTP) and 1,500, were included for all test cases.

Testing Model 1

Figure 6. Test 1: One Stream In, One Stream Out

Table 4. Results for Testing Model 1

Burst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)
DefaultDefault1284033.5
DefaultDefault644022 (Linux halt)
DefaultDefault6432 (Max)29.2
15k15k6432 (Max)30
15k15k51232 & 50 & 7025.3
15k15k1,50032 & 50 & 7025.2
18k18k6432 (Max)29.2
18k18k51232 & 50 & 7030.26
18k18k1,50032 & 50 & 7029.29

From the results in Table 4, the following statements can be made:

  • The maximum bandwidth a Linux machine can forward (receive on one interface and transmit on another interface) with continuous streams of 64-byte packets, is approximately 34Mbit/s.

  • The burst/cburst values, which give the most average accuracy results, are 18k/18k.

  • A linear relation exists between the burst value and the requested rate value. This relationship becomes apparent across tests.

  • The amount of bandwidth pushed on the output interface doesn't affect the accuracy of the results.

Figure 7. Graphic Analysis of Packet Size vs. Output Bandwidth

Figure 7 illustrates the relationship between packet size and output bandwidth after testing various packet sizes. From the results in Table 4 and Figure 7, we can conclude two things: throughput accuracy can be controlled by changing the cburst/burst values and the accuracy bandwidth range size is 2Mbit/s when using packets of sizes between 64 and 1,500 bytes. To verify that a linear relationship exists between the burst/cbursts values and the rate bandwidth, multiple burst and cburst values were tested. Table 5 shows the significant portion of the sampled data from the test case.

Table 5. Relationship between Burst/Cburst and Rate Values

Burst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Assigned-Rate (Mbit/s)
9k9k643217.515
9k9k5123215.1215
9k9k1,5003215.2815
4.8k4.8k64328.968
4.8k4.8k512328.1768
4.8k4.8k1,5003288
3k3k643217.515
3k3k5123215.1215
3k3k1,5003215.2815

18k/18k values were used as a starting point. The burst/cburst values were obtained by using the formula cburst/burst (Kbytes) = 18/(30M/Assign rate). From the results in Table 5, cburst/burst values can be defined dynamically for the rate value, such as when assuming a linear relationship.

Figure 8. Test 2: Three Streams In, One Stream Out

Table 6. Test 2 Results

StreamBurst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Class
117k17k641512.73
217k17k5122017.13
31k1k51242.012
Total18k18k3931.8

Table 6 shows an example of one level of inheritance. Class 2 and class 3 inherit the rate limit specification from class 1 (30Mbit/sec). In this test, the rate ceiling of the child classes is equal to the parent's rate limit, so class 2 and class 3 can borrow up to 30Mbit/sec. The linear relation assumption was used to calculate the cburst/burst values of all classes, based on their desired bandwidth.

Table 6 describes how the linear relationship works in the case of one level of inheritance. In this test, the input stream transmits continuous traffic of 39Mbit/s, and the accumulated output bandwidth is 31.8Mbit/s.

Figure 9. Test 3: Four Streams in, One Stream out

Table 7. Test 3 Results

StreamBurst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Class
11k1k51252.042
26k6k61511.3264
33k3k64105.675
47.8k7.8k5122013.026
Total18k18k5032.05

Table 7 shows the case of two levels of inheritance. Class 2 and class 3 inherit the rate limit specification from class 1 (30Mbit/s). Classes 4, 5 and 6 inherit the rate limit of class 3 (28Mbit/s) and share it based on their own rate limit specifications. In this test, the rate ceiling of the child classes is equal to the parent's rate limit, so classes 4, 5 and 6 can borrow up to 28Mbit/s. The linear relation assumption was used for calculating the cburst/burst values of all classes, based on their desired bandwidth.

From the results of Table 7, it can be observed that the linear relationship works in the case of two levels of inheritance. In this test the input port transmits continuous traffic of 50Mbit/s, and the accumulated output bandwidth is 32.05Mbit/s.

______________________

White Paper
Linux Management with Red Hat Satellite: Measuring Business Impact and ROI

Linux has become a key foundation for supporting today's rapidly growing IT environments. Linux is being used to deploy business applications and databases, trading on its reputation as a low-cost operating environment. For many IT organizations, Linux is a mainstay for deploying Web servers and has evolved from handling basic file, print, and utility workloads to running mission-critical applications and databases, physically, virtually, and in the cloud. As Linux grows in importance in terms of value to the business, managing Linux environments to high standards of service quality — availability, security, and performance — becomes an essential requirement for business success.

Learn More

Sponsored by Red Hat

White Paper
Private PaaS for the Agile Enterprise

If you already use virtualized infrastructure, you are well on your way to leveraging the power of the cloud. Virtualization offers the promise of limitless resources, but how do you manage that scalability when your DevOps team doesn’t scale? In today’s hypercompetitive markets, fast results can make a difference between leading the pack vs. obsolescence. Organizations need more benefits from cloud computing than just raw resources. They need agility, flexibility, convenience, ROI, and control.

Stackato private Platform-as-a-Service technology from ActiveState extends your private cloud infrastructure by creating a private PaaS to provide on-demand availability, flexibility, control, and ultimately, faster time-to-market for your enterprise.

Learn More

Sponsored by ActiveState