Linux – VMWare vSphere packet loss

linuxnetworkingpacketlossvmware-esxi

I have a Dell Blade Enclosure with 14 blades and ESXi 5.5 on it. On blade 7 and 14 I have round about 65 % packet loss when I ping the ESXi Management interface. On all other blades there ist no packet loss.
The strange thing is, I have this packet loss when I ping from:

blade 1 to blade 7 = 65% packet loss

but, when I ping from blade 7 to blade 1 at the same time I ping from blade 1 to blade 7 I have no packet loss at all … not from 1 to 7 norr from 7 to 1

blade 1 to blade 7 and blade 7 to blade 1 = 0% packet loss

I have increased the Rx buffer in the ESX CLI but it doesn´t help.
When I do a esxtop and go to the networktab I don´t see any packetdrop

PORT-ID USED-BY TEAM-PNIC DNAME PKTTX/s MbTX/s PKTRX/s MbRX/s %DRPTX %DRPRX 33554433 Management n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 33554434 vmnic0 - vSwitch0 22.73 0.04 44.50 0.10 0.00 0.00 33554435 Shadow of vmnic0 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 33554436 vmnic1 - vSwitch0 43.39 0.08 1.91 0.00 0.00 0.00 33554437 Shadow of vmnic1 n/a vSwitch0 0.00 0.00 0.00 0.00 0.00 0.00 33554438 vmk0 all(2) vSwitch0 3.66 0.01 1.91 0.00 0.00 0.00

The only thing I see that there are massive interrupts for 0xef:

VECTOR COUNT/s TIME/int COUNT_0 COUNT_1 COUNT_2 COUNT_3 COUNT_4 COUNT... 0xef 4435.5 1.0 309.2 275.6 402.9 30.5 339.0 15.3 ...

The switch logfile shows me, that the blade network card is flapping sometimes, when I say sometimes I mean once or twice a week for about 1 or 2 minutes.

I don´t think that`s the reason for that but I haven´t any idea anymore what can be the problem. Especially the ping and counter ping situation don´t make any sense to me.
Maybe you can help me?

Best Answer

I found the problem and a solution.

The packet loss was caused by identical MAC-addresses on different switch ports.

The virtual interface VMK0 Blade 7 had the same MAC as the hardware interface of Blade 11.

So what I've done is this: I gave the VMK0 interface a new MAC address. For that I go over the iDRAC interface of the Blade Enclosure and loging into the ESXi "GUI". I removed my both networkcards from the Administration interface, restart the Management Network, added them and restart the Management Network again. So I had some downtime but after that the MAC address was changed and there was no packet loss anymore.

Related Solutions

How to diagnose packet loss

I am a network engineer, so I'll describe this from my perspective.

For me, diagnosing packet loss usually starts with "it's not working very well". From there, I usually try to find kit as close to both ends of the communication (typically, a workstation in an office and a server somewhere) and ping as close to the other end as possible (ideally the "remote end-point", but sometimes there are firewalls I can't send pings through, so will have to settle for a LAN interface on a router) and see if I can see any loss.

If I can see loss, it's usually a case of "not enough bandwidth" or "link with issues" somewhere in-between, so find the route through the network and start from the middle, that usually gives you one end or the other.

If I cannot see loss, the next two steps tend to be "send more pings" or "send larger pings". If that doesn't sort give an indication of what the problem is, it's time to start looking at QoS policies and interface statistics through the whole path between the end-points.

If that doesn't find anything, it's time to start question your assumptions, are you actually suffering from packet loss. The only sure way of finding that is to do simultaneous captures on both ends, either by using WireShark (or equivalent) on the hosts or by hooking up sniffer machines (probably using WireShark or similar) via network taps. Then comes the fun of comparing the two packet captures...

Sometimes, what is attributed as "packet loss" is simply something on the server side being noticeably slower (like, say, moving the database from "on the same LAN" to "20 ms away" and using queries that requires an awful lot of back-and-forth between the front-end and the database).

Linux – How passively monitor for tcp packet loss? (Linux)

For a general sense of the scale of your problem netstat -s will track your total number of retransmissions.

# netstat -s | grep retransmitted
     368644 segments retransmitted

You can aso grep for segments to get a more detailed view:

# netstat -s | grep segments
         149840 segments received
         150373 segments sent out
         161 segments retransmitted
         13 bad segments received

For a deeper dive, you'll probably want to fire up Wireshark.

In Wireshark set your filter to tcp.analysis.retransmission to see retransmissions by flow.

That's the best option I can come up with.

Other dead ends explored:

netfilter/conntrack tools don't seem to keep retransmits
stracing netstat -s showed that it is just printing /proc/net/netstat
column 9 in /proc/net/tcp looked promising, but it unfortunately appears to be unused.

Best Answer

Related Solutions

How to diagnose packet loss

Linux – How passively monitor for tcp packet loss? (Linux)

Related Topic