Cisco – Basic multicast network performance problems

This is highly dependent on the architecture of the particular switch in question. The wide price range for a 48-port "managed switch" (e.g. under $300 to well over $10,000) should tell you there's something fundamentally different going on inside. If you didn't pay very much for your switch (and I hope you didn't), it's quite likely that multicast (and other "enterprise" features) are offloaded to software (if they're supported at all).

Basic unicast forwarding has become rather inexpensive to do in hardware, so I would expect any modern Ethernet switch to perform reasonably well at basic unicast forwarding under light loads.

When you start adding more features to hardware, the cost goes up significantly. For example, forwarding unicast frames is very different than doing packet replication based on dynamic multicast state. Those are both very specialized tasks. It takes specific bits of hardware to do either one well. Most users of low end switches don't have huge multicast needs. Paying for multicast-specific hardware is a waste for these users.

But most networks use a little multicast. Consequently, it's common for manufacturers to implement multicast and other less commonly used features in software. For example, the unicast forwarding hardware would be told to forward anything with a multicast MAC address to an internal port where they are received by a CPU subsystem (or at least a microcontroller of some sort). Then a software process is able to look at the frame, consult the multicast forwarding table, replicate the frame and hand multiple copies back to the hardware, one per port to be forwarded. Obviously lots of features can be added at this point in software without affecting the cost of the switch significantly.

In such a system, the performance will never be anywhere near what it is for unicast. The performance of the CPU will obviously have some impact, but if you're sending that much "exception" traffic that can't be forwarded in hardware, you're doing it wrong. You need to buy a different switch.

In the worst case, a really low end switch won't have any resource protections so the same CPU that's being slammed with multicast traffic will forget that it's also responsible for the care and feeding of everything else in the switch. If the CPU is too busy replicating multicast traffic to keep the unicast tables updated in hardware (or whatever else the CPU is supposed to be doing), you'll have all sorts of problems.

Please update your ESXi hosts. This is the one thing you've really missed in the troubleshooting steps.

Your 5.5 installation is almost 1 year old!!

As of this writing, the current version of ESXi 5.5 is 2718055. The current ESXi 6.0 build number is 2809209.

Dell, HP, doesn't matter... you're still supposed to update your ESXi installations. Many people overlook this, and it's the second most-often cause of unintended downtime in the environments I see.