Linux traffic shaping for large bandwidth (more than 1Gig)

bandwidth-controltraffic-shaping

I am doing an experiment between three computers that are connected with 10Gig interfaces and have RTT of 100 microseconds. Let's say these are machines A, B, and C. Machine A is communicating with B and C. Machines B and C do not communicate with one another.

I want to throttle bandwidth such that the machines A and B have 5Gbps bandwidth and 100 millisecond latency. Whereas, the machines A and C does not have any bandwidth and latency throttling. My goal is to emulate a wide area network with large bandwidth available.

I have tried using HTB and prio to throttle the bandwidth and latency. However, the maximum bandwidth reached is about 350 Mbps. I tested through iperf. My understanding is that we cannot have bandwidth more than 1Gig using TC.

Is there any way or any tool to throttle bandwidth above 1Gig?

I am using the following TC commands:

sudo tc qdisc del dev p4p1 root
sudo tc qdisc add dev p4p1 handle 1: root htb r2q 1000
sudo tc class add dev p4p1 parent 1: classid 1:3 htb rate 1000Mbps
sudo tc qdisc add dev p4p1 parent 1:3 handle 23: netem delay 50ms limit 100000000
sudo tc filter add dev p4p1 protocol ip prio 9 u32 match ip dst 10.96.0.1 flowid 1:3

Best Answer

I had the same trouble you found. Eventually I found something that worked at the 2Gbps I needed to throttle at. This is the script. Modify as you require.

#!/bin/sh
#
# Incoming traffic control
#
DEV=eth0
RATE="2000mbit"

tc qdisc del dev $DEV root
tc qdisc add dev $DEV root handle 1: htb default 10
tc class add dev $DEV parent 1: classid 1:1 htb rate ${RATE} burst 15k
tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${RATE} ceil ${RATE} burst 15k
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10

echo;echo "tc configuration for $DEV:"
tc qdisc show dev $DEV
tc class show dev $DEV

#
# Outgoing traffic control
#
DEV=eth2
tc qdisc del dev $DEV root
tc qdisc add dev $DEV root handle 1: htb default 10
tc class add dev $DEV parent 1: classid 1:1 htb rate ${RATE} burst 15k
tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${RATE} ceil ${RATE} burst 15k
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10

echo;echo "tc configuration for $DEV:"
tc qdisc show dev $DEV
tc class show dev $DEV

Related Solutions

Linux – Linux Traffic Shaping Using TC

Assuming eth0 is a 100mbit Ethernet connection to the Cisco Router, it should be something like this (Isn’t it?):

tc qdisc add dev eth0 root handle 1: htb default 2
# 100 mbps
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
# To LAN traffic
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 99000kbit ceil 100mbit
# IN traffic
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 120kbit
# OUT traffic
tc class add dev eth0 parent 1:1 classid 1:4 htb rate 120kbit

# IN “bands” (one for each PC)
tc class add dev eth0 parent 1:3 classid 1:10 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:11 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:12 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:13 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:14 htb rate 24kbit ceil 120kbit

# OUT “bands” (one for each PC)
tc class add dev eth0 parent 1:4 classid 1:15 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:16 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:17 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:18 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:19 htb rate 24kbit ceil 120kbit

Would get me something like this:

+-----------------------------------------------------------+
|                      100 mbits (1:1)                      |
+---------+------------------------+------------------------+
| 99mbits |   120 kbits In (1:3)   |  120 kbits Out(1:4)    |
+  (1:2)  +----+----+----+----+----+----+----+----+----+----+
+---------+ PC1| PC2| PC3| PC4| PC5| PC1| PC2| PC3| PC4| PC5|
          |1:10|1:11|1:12|1:13|1:14|1:15|1:16|1:17|1:18|1:19|
          +----+----+----+----+----+----+----+----+----+----+

And for each band:

# PC1, IN
tc qdisc add dev eth0 parent 1:10 handle 20: prio
tc qdisc add dev eth0 parent 20:1 handle 22: sfq perturb 10
tc qdisc add dev eth0 parent 20:2 handle 23: sfq perturb 10
tc qdisc add dev eth0 parent 20:3 handle 24: sfq perturb 10

# PC1, OUT
tc qdisc add dev eth0 parent 1:15 handle 21: prio
tc qdisc add dev eth0 parent 21:1 handle 25: sfq perturb 10
tc qdisc add dev eth0 parent 21:2 handle 26: sfq perturb 10
tc qdisc add dev eth0 parent 21:3 handle 27: sfq perturb 10

+--------------------++--------------------+
|       PC1 IN       ||      PC1 OUT       |
+--------------------++--------------------+
|     PRIO (20:0)    ||     PRIO (21:0)    |
|      |      |      ||      |      |      |
| Prio | Prio | Prio || Prio | Prio | Prio |
|   1  |   2  |   3  ||   1  |   2  |   3  |
|(20:1)|(20:2)|(20:3)||(21:1)|(21:2)|(21:3)|
+------+------+------++------+------+------+
|  SFQ |  SFQ |  SFQ ||  SFQ |  SFQ |  SFQ |
|(22:0)|(23:0)|(24:0)||(25:0)|(26:0)|(27:0)|
+------+------+------++------+------+------+

And so on.

The rules should be like this

# PC1, OUT
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 21:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 2 fw flowid 21:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 21:3

# PC1, IN
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 16 fw flowid 20:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 17 fw flowid 20:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 18 fw flowid 20:3

and so on.

Any suggestion, comments, etc? (I have no experience in the field)

Linux – Sharing Bandwidth and Prioritizing Realtime Traffic via HTB, Which Scenario Works Better

If I understand htb correctly, then the rate is "guaranteed". This means you have ideas about the rate of the "realtime" traffic. Only if this rate is exceeded, it will borrow. If several classes want to borrow, the prio should kick in. The guaranteed rates should add up the physical limit. Otherwise it's too much of a hassle.

IMHO, Case A will never really work since you need to have priority or rate-limiting at the root level. The priorities/rates in the different segment don't know anything of each other and will be handled equally.

The thing you probably want is: Put the "rate" for low and normal prio to 0 or close to it and add "ceiling" for the rest of the bandwidth, for high prio you guarantee a rate of 100% of the physical.

Best Answer

Related Solutions

Linux – Linux Traffic Shaping Using TC

Linux – Sharing Bandwidth and Prioritizing Realtime Traffic via HTB, Which Scenario Works Better

Related Topic