I'm running 3 Debian-Servers, each with its own IPMI. All IPs are within the same subnet with the same gateway. The problem is, that each server can ping its own IPMI but not the IPMI of the other 2 servers.
That said, all 3 IPMI and Servers are pingable and accessable from the outside by their IPs.
Each Server got 2 NICs, eth0 is the network to the outside and eth1 is used for internal traffic between the servers.
My network configuration looks like this:
eth0
1.2.3.84 (Server1)
1.2.3.85 (Server2)
1.2.3.86 (Server3)
1.2.3.71 (IPMI Server1)
1.2.3.76 (IPMI Server2)
1.2.3.66 (IPMI Server3)
1.2.3.65 (Gateway)
255.255.255.224 (Netmask)
eth1
10.10.10.1 (Server1)
10.10.10.2 (Server2)
10.10.10.3 (Server3)
The /etc/network/interfaces (in this example the one of Server1)
auto eth0
iface eth0 inet static
address 1.2.3.84
netmask 255.255.255.224
gateway 1.2.3.65
# default route to access subnet
up route add -net 1.2.3.64 netmask 255.255.255.224 gw 1.2.3.65 eth0
auto eth1
iface eth1 inet static
address 10.10.10.1
netmask 255.255.255.0
route -n (on server1)
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
1.2.3.64 1.2.3.65 255.255.255.224 UG 0 0 0 eth0
1.2.3.64 0.0.0.0 255.255.255.224 U 0 0 0 eth0
10.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
0.0.0.0 1.2.3.65 0.0.0.0 UG 0 0 0 eth0
Any ideas, why the IPMIs can not be accessed from the other servers?
EDIT
The IPMI used is a "Intel Remote Management Module (RMM)" that uses a shared NIC configured for management and shared with the operating system (according to the manual).
The server – and any external server outside of the subnet – has no problem connecting to the IMPI.
The network-configuration of the RMM of Server1 is:
IP Address: 1.2.3.71
Subnet Mask: 255.255.255.224
Default Gateway: 1.2.3.65
A traceroute to from the server to its own IPMI shows the following
traceroute to 1.2.3.71 (1.2.3.71), 30 hops max, 60 byte packets
1 static.25.184.x.y.clients.your-server.de (y.x.184.25) 0.921 ms 0.914 ms 0.941 ms
2 static.71.3.2.1.clients.your-server.de (1.2.3.71) 10.457 ms 10.460 ms 10.446 ms
Best Answer
I think your biggest problem is (was):
This causes a route to be defined for a network that is physically connected (local) instead of routed (requiring a gateway / route). This route takes precedence over the link-local traffic (it appears first in the routing table).
The system configured this way will not be able to communicate directly with other hosts on the routed network because it will direct all of the traffic to the gateway, and rely on the gateway to forward the traffic.
It seems likely that 1.2.3.65 is not forwarding packets out the interface / network they were received on (I.e. the packet comes from the 1.2.3.64/27, destined for the 1.2.3.64/27 network, so clearly it doesn't need to be forwarded). This prevents the server from communicating indirectly with other hosts on the 1.2.3.64/27 network. Absent direct communication ability and indirect (bounce off the gateway), there is no communication.
Note that the servers can communicate with each other on the 10.10.10.0/24 network, which works fine, and which is probably the IP address configured for the hosts in /etc/hosts or DNS. I suspect individual servers cannot reach other severs on the 1.2.3.64/27 network IP address.
If you remove the configured route for 1.2.3.64/27, the default gateways configured for each system (RMM and server) should be all of the routing you need.
The only strange thing is that the RMM is responding to traffic destined for it (layer 3) even though it is addressed (layer 2) for the gateway. My guess is that since it shares an interface, the RMM snarfs the packets as they cross, regardless of silly things like MAC addresses. The switch is respecting the MAC addresses, not broadcasting the packets, and so the RMM doesn't see packets from other physical devices destined for the gateway.