So the scenario is we have a web server at our colocation. In front of that web server are two SonicWalls and two L2 non-stacking managed switches. We're trying to find an efficient way to set up redundancy on all levels here (router, switch, machine) with the equipment we already own (so please, no "Go buy X" answers). The SonicWalls have an active/passive HA link, the switches have nothing, and the machine has dual NICs teamed with the same IP address (different MACs).
So each router has a link to each switch. The switches have a trunk link between each other and each switch has a connection to the dual NIC on the PC. Right now, the redundancy works such that I can kill the link between the router/switch or the switch/pc and I still have network access. To accomplish this, I set the ARP flush to 2 minutes on the Sonicwalls (lowest it would go) and 60 seconds on the switches). I can set up a constant PING on the machine and then remove one of the links. I have usually 20-30 seconds down time and then the ping starts back up.
My question is, what are the negative effects of having this low arp flush time? The server runs an online testing platform that people use, and in some cases they'll be downloading audio files. I was told by a peer that if they're downloading a file and the arp cache is flushed on the router, they'll lose their download. My understanding with TCP was that if it doesn't get an ACK in response, it will keep resending that same packet until it does? Given this information, can you see any issues that I might run into?
Best Answer
On networks I have worked on in the past servers that had redundant links had the same MAC address for both links, that MAC address was used for the main traffic to & from the server and the native MAC addresses were used for keep-alives between the links. That method removes the need for constant ARP cache clearing.
Anyway, in answer to your question, yes TCP will retry if necessary, but it probably wouldn't be required because the device that had just had its ARP cache emptied would simply send an ARP request as it would do on its first attempt to contact the device with the uncached address. On the switches, as you may be aware, the address cache will be repopulated almost instantly because it will learn addresses as soon as traffic with an unknown MAC flows through it.