I've started to see machines getting incorrect ARP table entries for several SQL Server instances in a failover cluster.
Client servers are alternatively populating their ARP tables with MAC addresses from the correct NIC team and the MAC address from one of the physical NICs (not the necessarily the corresponding NIC team MAC on that server) on a different cluster node.
This is causing intermittent connection failures for clients on the same LAN as the SQL Cluster.
This behavior has been noted by both VM clients as well as physical boxes.
This occurs after a failover and lasts for days.
In order to mitigate this, I've had to set static arp entries on the more troublesome clients.
ENVIRONMENT:
- Windows 2008 R2 SP1 Servers in a failover cluster
- SQL Server 2008 R2 Instances
- Teamed Intel Gigabit NICS
- HP 28XX switches
- Virtual Machines hosted on Windows Server 2008 R2 SP1 Hyper-V
The Intel NIC team creates a virtual adapter with the MAC address of one of the physical NICs.
I have a suspicion that the Intel NIC teaming software is the culprit, but any other troubleshooting thoughts or solutions would be appreciated.
I'm likely going to rebuild the cluster hosts with Server 2012 and use the in-box NIC teaming there (as I have not seen that issue with my testing with that platform).
I Think your case with 4 Gigs of RAM is pretty good covered here for Neo4j which also uses Memory Mapped IO and thus the same tuning measures should apply:
http://docs.neo4j.org/chunked/stable/linux-performance-guide.html
So from reading this I think your
vm.dirty_ratio=5
vm.dirty_background_ratio=5
are rather too low than to high.
Best Answer
a) They will fail explicitly. It is your responsibility to deal with it. What I tend to do is to retry the operation after a short grace period if the exception/error indicates that the operation failed because of a communication failure.
b) Implicitly. After the command to step down was issued, there is a 10 second grace period in which no write operations are accepted. During that time, the primary basically waits for the secondaries to catch up (be it necessary or not). For read operations: should they finish within the grace period, everything is fine. Otherwise, the connection is closed by the server, causing an error to be raised on th free client side.
c) The drivers are aware that they are connected to a replica set. If the primary steps down, the driver will notice. No primary = no write operations. Any write operation issued while there's no primary will fail with a according message. As per reads it depends on your read preference. Reads with a preference of
primary
will obviously fail while there is no primary, all other read preference will succeed. I usually issue my request with the default read preference ofprimary
and when it falls because of a missing primary, I reissue the query with a read preference ofsecondaryPreferred
(for the rare case that the primary came up and the first secondary failed while dealing with the original error).d) I am not a node expert, but the node driver is maintained by the Inc, so it is safe to assume it is replica set aware and adheres to the standardsas described above.