Kernel Korner - Analysis of the HTB Queuing Discipline

Can Linux do Quality of Service in a way that both offers high throughput and does not exceed the defined bandwidth? Here's a thorough test.
Test Cases and Test Results

When defining the HTB configuration, the following options of the tc class commands were used in order to achieve the required results:

  • rate = the maximum bandwidth a class can use without borrowing from other classes.

  • ceiling = the maximum bandwidth that a class can use, which limits how much bandwidth the class can borrow.

  • burst = the amount of data that could be sent at the ceiling speed before moving to serve the next class.

  • cburst = the amount of data that could be sent at the wire speed before moving to serve the next class.

Most of the traffic in the Internet world is generated by TCP, so packet sizes representative of datagram sizes, such as 64 (TCP Ack), 512 (FTP) and 1,500, were included for all test cases.

Testing Model 1

Figure 6. Test 1: One Stream In, One Stream Out

Table 4. Results for Testing Model 1

Burst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)
DefaultDefault1284033.5
DefaultDefault644022 (Linux halt)
DefaultDefault6432 (Max)29.2
15k15k6432 (Max)30
15k15k51232 & 50 & 7025.3
15k15k1,50032 & 50 & 7025.2
18k18k6432 (Max)29.2
18k18k51232 & 50 & 7030.26
18k18k1,50032 & 50 & 7029.29

From the results in Table 4, the following statements can be made:

  • The maximum bandwidth a Linux machine can forward (receive on one interface and transmit on another interface) with continuous streams of 64-byte packets, is approximately 34Mbit/s.

  • The burst/cburst values, which give the most average accuracy results, are 18k/18k.

  • A linear relation exists between the burst value and the requested rate value. This relationship becomes apparent across tests.

  • The amount of bandwidth pushed on the output interface doesn't affect the accuracy of the results.

Figure 7. Graphic Analysis of Packet Size vs. Output Bandwidth

Figure 7 illustrates the relationship between packet size and output bandwidth after testing various packet sizes. From the results in Table 4 and Figure 7, we can conclude two things: throughput accuracy can be controlled by changing the cburst/burst values and the accuracy bandwidth range size is 2Mbit/s when using packets of sizes between 64 and 1,500 bytes. To verify that a linear relationship exists between the burst/cbursts values and the rate bandwidth, multiple burst and cburst values were tested. Table 5 shows the significant portion of the sampled data from the test case.

Table 5. Relationship between Burst/Cburst and Rate Values

Burst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Assigned-Rate (Mbit/s)
9k9k643217.515
9k9k5123215.1215
9k9k1,5003215.2815
4.8k4.8k64328.968
4.8k4.8k512328.1768
4.8k4.8k1,5003288
3k3k643217.515
3k3k5123215.1215
3k3k1,5003215.2815

18k/18k values were used as a starting point. The burst/cburst values were obtained by using the formula cburst/burst (Kbytes) = 18/(30M/Assign rate). From the results in Table 5, cburst/burst values can be defined dynamically for the rate value, such as when assuming a linear relationship.

Figure 8. Test 2: Three Streams In, One Stream Out

Table 6. Test 2 Results

StreamBurst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Class
117k17k641512.73
217k17k5122017.13
31k1k51242.012
Total18k18k3931.8

Table 6 shows an example of one level of inheritance. Class 2 and class 3 inherit the rate limit specification from class 1 (30Mbit/sec). In this test, the rate ceiling of the child classes is equal to the parent's rate limit, so class 2 and class 3 can borrow up to 30Mbit/sec. The linear relation assumption was used to calculate the cburst/burst values of all classes, based on their desired bandwidth.

Table 6 describes how the linear relationship works in the case of one level of inheritance. In this test, the input stream transmits continuous traffic of 39Mbit/s, and the accumulated output bandwidth is 31.8Mbit/s.

Figure 9. Test 3: Four Streams in, One Stream out

Table 7. Test 3 Results

StreamBurst (Bytes)Cburst (Bytes)Packet-Size (Bytes)In-Bandwidth (Mbit/s)Out-Bandwidth (Mbit/s)Class
11k1k51252.042
26k6k61511.3264
33k3k64105.675
47.8k7.8k5122013.026
Total18k18k5032.05

Table 7 shows the case of two levels of inheritance. Class 2 and class 3 inherit the rate limit specification from class 1 (30Mbit/s). Classes 4, 5 and 6 inherit the rate limit of class 3 (28Mbit/s) and share it based on their own rate limit specifications. In this test, the rate ceiling of the child classes is equal to the parent's rate limit, so classes 4, 5 and 6 can borrow up to 28Mbit/s. The linear relation assumption was used for calculating the cburst/burst values of all classes, based on their desired bandwidth.

From the results of Table 7, it can be observed that the linear relationship works in the case of two levels of inheritance. In this test the input port transmits continuous traffic of 50Mbit/s, and the accumulated output bandwidth is 32.05Mbit/s.

______________________

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix