Ping Issues in the Same Subnet – Troubleshooting Guide

I'd say it looks normal. But I think you're doing this a bit wrong.

Ping has one function only: probing, i.e. "Hello, are you there" - "Hello, yes I am here". While it shows you the latency between two hosts, it is not a network performance tool.

Your "empty" network is never silent. When you connect two network devices together - any devices - they will start exchanging all kinds of information immediately; availability, capabilities, link status etc. This is (like ICMP) network management traffic, and handled by the CPU.

The primary function of any network device is traffic forwarding. When the first packet of a discussion comes, a Layer 2 switch will first check the MAC/ARP tables to see if the port leading to the destination is known. If not, it will ARP the network to learn the destination port. Once it is known, the traffic forwarding is delegated to the forwarding engine. CPU will not touch packets in that discussion, but it will still keep communicating with the other network devices. L3 forwarding (i.e routing) is often done entirely by the CPU. This depends on the device, some routing switches are capable of hardrware packet forwarding similar to L2 switching. This is known as "route once / switch many" or "Multilayer switching".

The network devices are configured to maintain the primary functionality of the network. All mgmt traffic it's given secondary priority, and ICMP ping has the lowest of low priorities. This is - as far as I know - true for all devices. Allowing ICMP a higher priority would open door for DOS attacks such as Ping of Death and Ping Flood.

Hence tests 2 and 3 are irrelevant. Devices are responding, which shows traffic is flowing correctly. The devices will not sacrifice CPU cycles for ping, they will respond only when (or if!) they have time. Once the network gets more traffic, you can expect higher variation on network device ping responses. It's still expected behavior, their latency has no relation whatsoever to network performance. Host-to-host response times should be less affected.

Tests 1 and 4 are more sensible; but they aren't a valid comparison between L2 and L3 forwarding performance. In test 1 you're pinging a "resource", in test 4 you're pinging a server - i.e. not the same device. While the network may be playing part, the latency depends on the recipient. What's above applies also to any computer; a NIC is a network device and ICMP is handled by CPU... maybe your server is taking it's own sweet time responding. When you're pinging to a different subnet you're bringing in yet another potential source of latency - the default gateways.

If you want to test the forwarding capability of your network, you need to do it differently.

1. Download iperf. This is a simple, but very powerful network performance tool. Start with default TCP testing. You can change the parameters, run UDP test etc. afterwards for more specified testing.
2. Use the same two devices in all testing to exclude host-based differences
3. Where to connect depends on what exactly is tested - cabling, patch panels etc. are also potential sources of latency
4. I would use static IPs in all testing; I don't want my test devices do anything I do not specifically instruct them to do including renewing the IP

The Ping Fallacy

I have seen this before on other host types. Putting in default gateway addresses should prevent this behavior. Without a default gateway, the host sends an ARP for the other host's layer-3 address, and it receives a reply because the hosts are on the same layer-2 domain. With a default gateway, the host would ARP for the gateway's layer-3 address, not the other host's layer-3 address, but there is no gateway address for which to ARP.