Linux tc htb + prio = very slow link

htblinuxtctraffic-managementtraffic-shaping

I'm trying to shape the traffic going out of my DSL link (verified 1 mbit upload) using Linux (3.2) tc, HTB, and PRIO. My Linux box is connected via a Gigabit Ethernet link to the ADSL modem. I want to limit the upload rate using HTB so my packets get queued at the Linux box and not at the modem, and then use PRIO to put packets into priority bands.

So, I did this:

tc qdisc add dev dsl root handle 1: htb default 1
tc class add dev dsl parent 1: classid 1:1 htb rate 950kbit
tc qdisc add dev dsl parent 1:1 handle 2: prio bands 6
tc qdisc add dev dsl parent 2:1 bfifo
tc qdisc add dev dsl parent 2:2 bfifo
tc qdisc add dev dsl parent 2:3 bfifo
tc qdisc add dev dsl parent 2:4 bfifo
tc qdisc add dev dsl parent 2:5 bfifo
tc qdisc add dev dsl parent 2:6 bfifo

Problem is, as soon as I do that, my upload link slows down miserably. I'm getting approx. 25 kbit/s, which is 38 times slower than what I should be getting (950 kbit/s), even though the link is unused.

Interestingly, if I remove the PRIO qdisc but keep the HTB qdisc, then throughput goes up to approx. 550 kbit/s, which is better but still not what I should be getting. Again, the link is unused, so prioritization does not account for this behavior.

Any idea what's wrong with what I'm doing? I've been using the exact same commands for years to shape an Ethernet link to 50 mbit/s without any issues, so I really have no idea why it doesn't work in this situation.

Additional information:

# tc -s -d qdisc ls dev dsl
qdisc htb 1: root refcnt 2 r2q 10 default 1 direct_packets_stat 0 ver 3.17
 Sent 447262 bytes 1168 pkt (dropped 90, overlimits 38 requeues 0)
 backlog 0b 0p requeues 0
qdisc prio 2: parent 1:1 bands 6 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 447262 bytes 1168 pkt (dropped 90, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8043: parent 2:1 limit 1514b
 Sent 84138 bytes 928 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8044: parent 2:2 limit 1514b
 Sent 363124 bytes 240 pkt (dropped 90, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8045: parent 2:3 limit 1514b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8046: parent 2:4 limit 1514b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8047: parent 2:5 limit 1514b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc bfifo 8048: parent 2:6 limit 1514b
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

# tc -s -d class show dev dsl
class htb 1:1 root leaf 2: prio 0 quantum 11875 rate 950000bit ceil 950000bit burst 1599b/8 mpu 0b overhead 0b cburst 1599b/8 mpu 0b overhead 0b level 0
 Sent 478804 bytes 1316 pkt (dropped 90, overlimits 0 requeues 0)
 rate 42232bit 17pps backlog 0b 0p requeues 0
 lended: 1316 borrowed: 0 giants: 0
 tokens: 195781 ctokens: 195781
class prio 2:1 parent 2: leaf 8043:
 Sent 97560 bytes 1064 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 2:2 parent 2: leaf 8044:
 Sent 381244 bytes 252 pkt (dropped 90, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 2:3 parent 2: leaf 8045:
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 2:4 parent 2: leaf 8046:
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 2:5 parent 2: leaf 8047:
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
class prio 2:6 parent 2: leaf 8048:
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Best Answer

Andy Furniss found the culprit when I asked on the LARTC mailing list. What I didn't mention was that I was shaping traffic out of a tagged VLAN interface. The issue is, Linux does not set up a queue for a VLAN interface (i.e. txqueuelen is 0) and prefers to use the physical interface queue instead.

Because of that, my bfifo qdiscs defaulted to a 1514 byte queue (as seen in the tc qdisc ls output), which is way too low, thus creating a bottleneck.

The solution is to make sure the interface has its own queue:

ifconfig dsl txqueuelen 20

bfifo then defaults to a 30 KB (txqueuelen * MTU) queue, which solves the problem.