We have a number of Supermicro machines with IPMI/BMC features. Some of these machines use an onboard BMC, while others use an add-on card.
We are looking into using sideband due to it's reduced costs and cabling requirements. However, some sideband details don't quite make sense.
Sideband requires one ethernet cable which is plugged into an ethernet port on the motherboard. This network port is then shared between the IPMI system and the operating system. From what I read in this Supermicro manual, "Use the same MAC address you are using for LAN1 for the SIMSO IPMI card". However, the IPMI must have a different IP address then the operating system.
How is it possible to have two devices (the operating system and the IPMI) which can listen and transmit on this same physical network port? When a packet arrives at the interface, how does the system determine if this packet is intended for the Operating System or for the IPMI system?
Are these packets handled by the CPU at all, using CPU interrupts? Can packets to the IPMI interface be viewed by the operating system?
Best Answer
I manage a lot of SuperMicro servers using the onboard IPMI. I have a love/hate relationship with the shared (aka sideband) ethernet. In general, the way these things work is that LAN1 appears to have 2 (different) MAC addresses - one is for the IPMI interface, the other your standard Broadcom NIC. Traffic to the IPMI interface (layer 2, based on the MAC address) is magically intercepted below the operating system level and never seen by whatever OS is running.
You've already hit on the one good point for them: less cabling. Now let me cover some of the downsides:
While this has nothing to do with sideband-vs.-dedicated, I'll also note that the tools for accessing host systems are very poorly written. Older IPMI cards don't support anything other than local authentication, making password rotation a total pain. If you're using the KVM-over-IP functionality, you're stuck using an improperly-signed, expired Java applet or a weird Java desktop application that only works on Windows and requires UAC elevation to run. I've found the keyboard entry to be spotty at best, sometimes getting "stuck keys" such that it's impossible to type a password to login without trying 10 times.
I've eventually managed to get 40+ systems working with this arrangement. I've got mostly newer systems I could VLAN the IPMI interfaces onto a separate subnet, and I mostly use the serial console via ipmitool which works very well. For the next generation of servers, I'm looking at Intel's AMT technology with KVM support; as this makes it into the server space, I can see replacing IPMI with this.