Linux – Limiting bandwidth on Ubuntu Linux

iproute2linuxnetworkingtctraffic-shaping

I am in the situation where I have to simulate a P2P-environment (for my masters thesis in computer science). To do that I am using Docker with Ubuntu to create a bunch of virtual machines that is gonna be connected in a BitTorrent network. I then need to make sure that the upload and download rate of the peers can be set, and it is not an option for me to do it in the client (as the client uses sleeping to simulate lower bandwidth and that results in spikes in the rate).

Therefore I am trying to do it for each container. To be honest I don't really care how this is accomplished as long as it works, but I have tried different things with no luck. These are the things I have tried so far:

  1. Trickle
    Trickle seems to be doing the trick, but for some reason, when I start more than 5 Docker containers Trickle will make a lot of them exit without telling me why. I have tried different settings, but you don't have that many knobs to turn when it comes to configuration, so I don't think Trickle will be an option in this scenario.
  2. Wondershaper
    Using Wondershaper seems to work, or at least it limits the bandwidth. The only problem here is that there are no seemingly understandable correlation between the value set in the options and the actual bandwidth. When I set up to have download of 2048 (which should be kbits) the actual download ranges between 550KB and 900KB which seems really weird.
  3. tc
    Using tc, like many people have suggested for similar questions, does indeed limit the bandwidth, but no matter what value I set it always gives me the same bandwidth (around 15-20KB/s).

I have tried following tons of guides and examples, but every single one has either not worked or is described above. I am somewhat at a loss here, so if anyone knows of any reason the above examples should work or have another solution it would be awesome.

What I am looking for is a way to limit a single Linux instance, and then I should be able to make that work for multiple Docker containers.

————— EDIT—————-

I have tried a couple of different tc commands, but one of them is like this 

DEV=eth0
tc qdisc del dev $DEV root
tc qdisc add dev $DEV root handle 1: cbq avpkt 1000 bandwidth 100mbit
tc class add dev $DEV parent 1: classid 1:1 cbq rate 256kbit allot 1500 prio  5 bounded isolated
tc filter add dev $DEV parent 1: protocol ip prio 16 u32 match ip src 0.0.0.0/0 flowid 1:1
tc qdisc add dev $DEV parent 1:1 sfq perturb 10

No matter what rate I set it always gives me download of around 12 KB/s (the default download with no limit set is around 4MB/s)

————EDIT 2 (what I ended up doing)————

It turns out that you can't reliable set the bandwidth for the Docker containers inside the containers, but whenever you make a new container a virtual interface is created on the host machine called vethsomething. If you use for instance Wondershaper on those virtual interfaces the limit has the correct behaviour 🙂

Best Answer

Using tc (because it's the most current, and I'm the most familiar with it), you should be able to slow traffic down without problems.

I have a server acting as the firewall (called 'firewall' -- very creative) and then a second server behind that (called 'mil102'). without any tc commands, scp'ing a file from mil102 to firewall moves at full speed:

root@firewall:/data#scp mil102:/root/test.tgz test.tgz
test.tgz       100%  712MB  71.2MB/s   00:10

Adding the following commands to mil102 (it's easier to shape sending traffic):

#!/bin/sh
DEV=eth0
tc qdisc del dev $DEV root
tc qdisc add dev $DEV handle 1: root htb default 11
tc class add dev $DEV parent 1: classid 1:1 htb rate 4Mbps
tc class add dev $DEV parent 1:1 classid 1:11 htb rate 4Mbit
tc qdisc add dev $DEV parent 1:11 handle 11: sfq perturb 10

Now the same command slows down to 4Mb:

root@firewall:/data#scp mil102:/root/test.tgz test.tgz
test.tgz         0% 6064KB 467.0KB/s   25:48 ETA

I stopped the transfer -- would take far too long. The speed listed in the scp is in bytes, specified in tc in bits, so 467KB * 9 = 4203Kb, close to my 4096Kb limit (thought it would be *8, but I guess there's a parity bit?).

I tried changing to 10Mbit and my scp showed I was moving data at 1.1MB per second (1.1 * 9 = 9.9).

The last line with the 'sfq perturb 10' directive was added to even out traffic flow on a loaded connection. It directs the queue to take packets from each conversation based on a round-robin hash.

You can test with and without -- ssh to a loaded machine without will go in bursts, with will be much smoother (very important for VOIP). The 'perturb 10' tells it to recalculate the hash algorithm every 10 seconds to make it random.

Related Topic