Ubuntu Server – Two Interfaces Living Together (in perfect harmony)

You should only have one default gateway. If you remove the gateway line from eth1, it'll all just work (after networking is restarted).

Your routes and configuration look fine.

$: route -n
Destination // Gateway // Genmask // Flags // Metric // Ref // Use // Iface
66.*.*.0       0.0.0.0    255.255.255.248   U   1003      0   0     0  em2
0.0.0.0        66.*.*.1   0.0.0.0           UG     0      0   0     0  em2

The first route, 66.*.*.0/29 with gateway 0.0.0.0 tells your computer to use interface em2 and then make an arp request to find the hardware address of the host you're trying to reach. This is a "connected" route.

The second one is the default route, pointing at your default gateway through em2. If you need to send a packet in another network than 66.*.*.0/29, your computer will make an arp request to find 66.*.*.1 and then send the packets to it.

The only thing in your configuration that could be an issue is the NM_CONTROLLED=yes statement in /etc/sysconfig/network-scripts/ifcfg-em2. This tells the system that this interface is controlled by NetworkManager. This could interfere with your static configuration.

However, even without any default gateway you should be able to ping and ssh from the 66.*.*.0/29 subnet to your machine.

Check layer 1 first, and ensure that the cable is plugged on each side. Use leds on nic and switch, and check if the system sees it correctly:

# mii-tool
eth0: negotiated 1000baseT-FD flow-control, link ok

Then verify if any iptables are dropping the packets. Use iptables -L or iptables-save to check for any rules, and iptables -D <rule> to delete them. Pay attention to the default policy.

Also, on some systems, NetworkManager can configure ufw automatically, and I've had issues with static interface configuration that wasn't seen by NM and hence blocked by ufw.