Linux – Linux Traffic Shaping Using TC

linuxtraffic-managementtraffic-shaping

My internet connection goes like this:

Internet <-128kbps link-> Cisco Router (Public IP) <-LAN-> Linux router/server (Public IP) <-LAN-> Regular PCs (Public IPs)

The Cisco router:

  • the first Public IP allocated to my
    institution (/29)
  • is programmed to send all packets
    through the Linux router

The Linux router

  • The second Public IP allocated to my
    institution
  • is programmed to forward packets
    between the regular PCs and the Cisco
    router
  • act as a server too (mail, web, etc.)

The regular PCs (4 of them):

  • The rest of the Public IPs
  • use the Linux router as the gateway

I enabled the iptables packet logging on the Linux router and sometimes and I find out that:

  • Some packets are big, bigger than
    20KB. Is that normal? (yes, it is normal. These are not packets, these are IP datagrams as Some Guy kindly explained)
  • Too many times the transmitted data
    (out to the internet) was bigger than
    16KB. For example in a particular
    second 10572 bytes went in (no
    problem), 63521 bytes went out (to
    the Cisco router). It would take 4
    seconds at least to send that 64KB
    through the 128kbps link. Meanwhile
    the Linux router is sending more data
    to the Cisco router, clogging its
    buffers. Not good.

Now, how can I configure the Linux router to shape traffic in a way that:

  1. Keep transmission speed to the max
    when the traffic is between these
    regular PCs and the Linux server.
  2. Slow down traffic to the outside
    world to avoid clogging the “out”
    line, using all (or almost all) the
    bandwidth available (128 kbps). No
    more “>16KB out seconds” on the
    trace.
  3. Guarantee 24kbps out to each regular
    PC, 24 kbps to the Linux server at
    any time. (8bkps left for overhead
    if necessary). IOW, 5 (pseudo)
    “bands”, 24kbps each one.
  4. In case there is any PC not using
    its full band, fairly share the idle
    bandwidth among the rest of the
    transmitting PCs
  5. Give priority to certain packets
    (DNS lookups, control packets), take
    priority away from others
    (torrent!!!), INSIDE each band, and
    not affecting other bands.

I have already marked (using IP tables –set-xmark option) each outgoing packet for each PC:

  1. Linux router to the outside world,
    high prio
  2. Linux router to the outside world,
    normal prio
  3. Linux router to the outside world,
    low prio
  4. First regular PC to the outside
    world, high prio

… And so on.

Each incoming packet is also marked using this scheme starting from 16.

I’m sorry for this long question, but I have given up to set this up using the tc command, there is too little documentation about traffic shaping and I don’t know where to go next.

Best Answer

Assuming eth0 is a 100mbit Ethernet connection to the Cisco Router, it should be something like this (Isn’t it?):

tc qdisc add dev eth0 root handle 1: htb default 2
# 100 mbps
tc class add dev eth0 parent 1: classid 1:1 htb rate 100mbit
# To LAN traffic
tc class add dev eth0 parent 1:1 classid 1:2 htb rate 99000kbit ceil 100mbit
# IN traffic
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 120kbit
# OUT traffic
tc class add dev eth0 parent 1:1 classid 1:4 htb rate 120kbit

# IN “bands” (one for each PC)
tc class add dev eth0 parent 1:3 classid 1:10 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:11 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:12 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:13 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:3 classid 1:14 htb rate 24kbit ceil 120kbit

# OUT “bands” (one for each PC)
tc class add dev eth0 parent 1:4 classid 1:15 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:16 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:17 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:18 htb rate 24kbit ceil 120kbit
tc class add dev eth0 parent 1:4 classid 1:19 htb rate 24kbit ceil 120kbit

Would get me something like this:

+-----------------------------------------------------------+
|                      100 mbits (1:1)                      |
+---------+------------------------+------------------------+
| 99mbits |   120 kbits In (1:3)   |  120 kbits Out(1:4)    |
+  (1:2)  +----+----+----+----+----+----+----+----+----+----+
+---------+ PC1| PC2| PC3| PC4| PC5| PC1| PC2| PC3| PC4| PC5|
          |1:10|1:11|1:12|1:13|1:14|1:15|1:16|1:17|1:18|1:19|
          +----+----+----+----+----+----+----+----+----+----+

And for each band:

# PC1, IN
tc qdisc add dev eth0 parent 1:10 handle 20: prio
tc qdisc add dev eth0 parent 20:1 handle 22: sfq perturb 10
tc qdisc add dev eth0 parent 20:2 handle 23: sfq perturb 10
tc qdisc add dev eth0 parent 20:3 handle 24: sfq perturb 10

# PC1, OUT
tc qdisc add dev eth0 parent 1:15 handle 21: prio
tc qdisc add dev eth0 parent 21:1 handle 25: sfq perturb 10
tc qdisc add dev eth0 parent 21:2 handle 26: sfq perturb 10
tc qdisc add dev eth0 parent 21:3 handle 27: sfq perturb 10

+--------------------++--------------------+
|       PC1 IN       ||      PC1 OUT       |
+--------------------++--------------------+
|     PRIO (20:0)    ||     PRIO (21:0)    |
|      |      |      ||      |      |      |
| Prio | Prio | Prio || Prio | Prio | Prio |
|   1  |   2  |   3  ||   1  |   2  |   3  |
|(20:1)|(20:2)|(20:3)||(21:1)|(21:2)|(21:3)|
+------+------+------++------+------+------+
|  SFQ |  SFQ |  SFQ ||  SFQ |  SFQ |  SFQ |
|(22:0)|(23:0)|(24:0)||(25:0)|(26:0)|(27:0)|
+------+------+------++------+------+------+

And so on.

The rules should be like this

# PC1, OUT
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 1 fw flowid 21:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 2 fw flowid 21:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 3 fw flowid 21:3

# PC1, IN
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 16 fw flowid 20:1
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 17 fw flowid 20:2
tc filter add dev eth0 protocol ip parent 1:0 prio 1 handle 18 fw flowid 20:3

and so on.

Any suggestion, comments, etc? (I have no experience in the field)

Related Topic