Linux – Having an issue trying to get Gigabit speed across the network (Ubuntu Server)

gigabit-ethernetlinuxnetworkingUbuntu

I've just started looking into the network speeds at my office, the entire network is setup to be "Gigabit". This includes Gb switches, Gb Network cards and Cat 5e cabling.

I'm not expecting the full speed, I just want more than ~90 Mb/s.

I've been running some tests with iperf the linux tools and checking the hardware with ethtool. I have 3 servers and when doing my checks/test I discovered that the two backup servers can access each other at around 450 Mb/s but when using either one of them to connect and test the main server, I only get the 90Mb/s even though ethtool shows the networking card running at 1000/Full.

The only difference between all the server/networking cards is the "Port" which ethtool shows. On the two backup servers the "Port" is shown as MII yet on the other it's shown as "Twisted Pair".

When using ethtool -s to manually set the "Port" to MII on the main server, it looses all connectivity and does not show "Speed" or "Duplex".

Anyway, Am i doing something wrong? Is there a specific reason my main server cannot use Gb when there appears to be no difference except the "Port"?

Best Answer

As @Luis pointed out, 'b' = bit, and 'B' = byte. iperf helps us out by reporting times in Mbits/sec:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.01 GBytes   865 Mbits/sec

Does ifconfig eth0 show any errors on either host? From your reports, it seems as if either the CPU is overloaded on one of the two iperf machines, or the switch (please tell me it's not multiple switches) between them is overloaded, or you have a 100Mbit constriction somewhere between your hosts. I've found people cascaded off of a small 100Mbit switch one too many times, so it's good to know where your cable runs start and finish. Are you using a smart switch? Does it show that your ports are active at gigabit speeds?

Once you figure out your problem, you might want to enable jumbo frames on your hosts in order to get even better throughput when you're making bulk data transfers, provided you have a reasonably-modern switch.