Routing – Sonicwall NSA 2600 routing issues with multiple LAN interfaces configured

We have a Sonicwall NSA 2400 that has been in service for some years now. We've never had an issue with any routing until we "upgraded" to a new NSA 2600 device. Below is a rough idea of the configuration:

NSA 2600 interfaces:

x0 - LAN IP x.x.10.223  x1 - WAN ISP #1   x2 - WAN ISP #2    x3 - WAN ISP #3 and x4 - LAN IP y.y.16.10

x0 and x4 are plugged into an HP Procurve 2920-48G-PoE+ switch and the server is on another HP Procurve 2920-24G switch plugged directly into the other.

The main server we are seeing routing issues with is an IBM AS400 with multiple IPs configured through virtual interfaces on its NIC. The IP addresses are on the LAN segments and are as follows:

x.x.10.4   x.x.10.6   y.y.16.22

This configuration was setup by a vendor for the AS400 and the y.y.16.0 subnet is for replication to be routed out of the x2 ISP #2 interface

The behavior we are seeing has stumped Dell/Sonicwall engineers and so I will describe it as best as I can here.

Upon plugging in all the interfaces, the ARP table starts filling up with the devices on the LAN segments. y.y.16.22, x.x.10.4 and x.x.10.6 do not show up. We have tried adding static ARP entries for them and the pings perform abnormally in that they will respond for a period of time then start dropping off one IP at a time.

Now if we leave the x4 (second LAN interface) unplugged then the x.x.10.4 and x.x.10.6 will appear in the ARP entries initially. We have run packet captures and find that ping requests from the Sonicwall NSA 2600 are 'generated' then 'consumed' but sometimes not forwarded.

We have cleared the ARP cache on the AS400, this is the second NSA 2600 device as the first was RMA'ed after being factory reset 3 times and even testing the settings imported from the 2400. We also tried setting the interface MAC addresses on the NSA 2600 to the same as the 2400. Oh, and we started this ordeal with Dell Powerconnect switches and were told they were an issue (by Sonicwall) so we bought HPs 🙂

Any thoughts, advice or even a direction to look into would be appreciated as we have beat our heads on keyboards for a while, and spent hours on the phone with Dell/Sonicwall "support".

Thanks.

EDIT for RON:

1) correct. with static ARP entries it may ping for a few minutes then drop completely from the firewall but from other PCs on the network it still responds.

2) on packet captures for the NSA 2400 (old device) the ping trace from the firewall to AS400 goes 1-packet Generated by source 2-packet Consumed (by firewall) 3-packet Forwarded to AS400 so in the packet capture on the 2600 it is not forwarding the packets on the NSA 2600.

3) The AS400 has a single NIC with virtual interfaces configured so it is technically a single interface. Other machines on the network can ping the AS400 directly on the x.x.10.0 network and I've added the y.y.16.0 network to a machine and the pings never drop to any of those addresses. This issue seems to only be with the NSA 2600 device, which we've replaced…

Thanks Ron.

EDIT # 2:
Packet captures show that ARP packets are going out but responses are not received when the x4 interface is plugged in. If the x4 interface is unplugged, the ARP responses are received. Same goes for unplugging the x0 interface for the x4 subnet.

EDIT # 3:
Found out the AS400 is using what is called a Proxy ARP configuration. The physical NIC is used as a console for the system and also acts as a proxy for the virtual interfaces configured inside the AS400. The proxy responds to all ARP requests with the physical MAC address and then handles routing internally to the virtual interfaces. I've attached a drawing to clarify the setup as follows.

Still no headway on what is causing this behavior.