Odd one-way ping issue that I can’t wrap the head around

The MD3200 connects to servers via SAS; so I'm thinking you mean the MD3200i. It's "normal" to separate vLANs for management, iSCSI, and client traffic. You don't mention what kind of cluster software you're running, but for most it's pretty common for it's cluster/heartbeat traffic to have it's own vLAN.

Depending on your server and hardware each of these may have their own physical NICs or may be shared. Also normal to have a minimum of 4 NIC ports for 1GbE; or 2 ports of 10GbE/FC/IB/NCA and 2 ports of 1GbE. It's also normal to have a L4 switch or router with access controls that allows all of these vLANs to be routed (selectively and where applicable).

Breaking that down:

• Your servers should have at least 4 network ports.
  • Depending on your setup you may want 6 or even 8; it really depends.
• You should have vLANs for:
  • Management (ESX or Hyper-V Hypervisor; iLO/DRAC/iLOM/etc; SAN Management)
  • SAN Traffic (whatever it is: iSCSI, FC, IB, SAS; it should be separate)
  • Client Traffic (your normal traffic, sometimes you'll want more than one, as many as you would have without any of the extra complexity)
  • Clustering/Heartbeat (optional, depending on the software you're using). I run Hyper-V clusters and have heartbeat on the management network, it's not terribly sensitive to minor lag; some cluster software is painfully sensitive to lag and needs to be on its own.
• Each vLAN should have it's own subnet. I would highly recommend using subnets larger than /24 or spacing them out so it's easy to expand them later (I wouldn't recommend anything smaller than /20; ie. 10.0.0.0/20, 10.0.16.0/20, 10.0.32.0/20, etc). There should be some form of routing between subnets. High end switches have this feature, your firewall may as well. if you're lacking a box or VM with pfSense works great.

Other thoughts: The NIC chips really matters. Make sure your servers have something good and Google for cluster or SAN problems with the particular chip. I run Broadcom BCM5709 chips for instance (all my servers use the exact same chip). These chips had problems with past firmware that has been resolved. Intel chips tend to be very good. In any case check it out and be sure to run the latest firmware, drivers, and management software.

I've started to see machines getting incorrect ARP table entries for several SQL Server instances in a failover cluster.

Client servers are alternatively populating their ARP tables with MAC addresses from the correct NIC team and the MAC address from one of the physical NICs (not the necessarily the corresponding NIC team MAC on that server) on a different cluster node.

This is causing intermittent connection failures for clients on the same LAN as the SQL Cluster.

This behavior has been noted by both VM clients as well as physical boxes.

This occurs after a failover and lasts for days.

In order to mitigate this, I've had to set static arp entries on the more troublesome clients.

ENVIRONMENT:

• Windows 2008 R2 SP1 Servers in a failover cluster
• SQL Server 2008 R2 Instances
• Teamed Intel Gigabit NICS
• HP 28XX switches
• Virtual Machines hosted on Windows Server 2008 R2 SP1 Hyper-V

The Intel NIC team creates a virtual adapter with the MAC address of one of the physical NICs.

I have a suspicion that the Intel NIC teaming software is the culprit, but any other troubleshooting thoughts or solutions would be appreciated.

I'm likely going to rebuild the cluster hosts with Server 2012 and use the in-box NIC teaming there (as I have not seen that issue with my testing with that platform).