Use ipmitool to determine which IPMI link is ‘up’

You may be running in to an unfortunate effect of the Supermicro BMC firmware. When power is applied to the power supply, the BMC powers on immediately. During the boot process the BMC (via Uboot which is booting Linux on the BMC) checks to see if the dedicated IPMI NIC port sees a link state. If not, the shared NIC port will be used. The NIC port selected at BMC boot time will be the NIC port used until the BMC is power cycled, either through a direct BMC reboot or when power is removed from the power supply. Rebooting the system itself will do nothing to the BMC.

This creates a cabling time race condition between plugging in the dedicated IPMI NIC and the power cable which is very obnoxious. Or, for example, if you have a power outtage and the BMC comes up before the switch does, the BMC will select the shared NIC in spite of the dedicated NIC being wired and LAN IPMI access will, in the case of VLANed ports, will be on the wrong network. We experience this more often than we like and find it quite frustrating.

If you were able (which, you won't be able to if the BMC comes up on the "wrong" NIC) to connect over the LAN you could SSH to the BMC using the ADMIN account (default password "ADMIN). When logged on to the BMC via SSH you can see the effect of the Uboot time decision in the command line as shown by the usencsi= option at the end of the command line:

# cat /proc/cmdline 
root=/dev/ramdisk ro ip=none ramdisk_blocksize=4096 console=ttyS0,38400 rootfstype=cramfs bigphysarea=1025 usencsi=0

On my system (X8DTi-LN4F) usencsi=0 means "use the dedicated IPMI NIC."

Of course, this requires that you connect to the BMC via the LAN. I've looked pretty hard with the r1.05 firmware and can find no way to discern the selected NIC accessing IPMI from the host. I've just started to look at the r1.32 firmware for this system. In any case, I don't see your motherboard model listed on the SuperMicro IPMI firmware page here:

• http://www.supermicro.com/support/bios/firmware.aspx

What's most frustrating about this is I know what two bytes I would like to hardwire in the BMC firmware letting us set the IPMI interface either to the dedicated NIC or shared NIC but as far as I can discern there is no setting allowing this.

I manage a lot of SuperMicro servers using the onboard IPMI. I have a love/hate relationship with the shared (aka sideband) ethernet. In general, the way these things work is that LAN1 appears to have 2 (different) MAC addresses - one is for the IPMI interface, the other your standard Broadcom NIC. Traffic to the IPMI interface (layer 2, based on the MAC address) is magically intercepted below the operating system level and never seen by whatever OS is running.

You've already hit on the one good point for them: less cabling. Now let me cover some of the downsides:

• It's particularly difficult to partition the IPMI interface onto a separate subnet in a secure manner. Since the traffic all goes over the same cable, you (almost) always have to have the IPMI interface and the LAN1 interface on the same IP subnet. On the latest motherboards, the IPMI cards now support assigning a VLAN to the IPMI NIC, so you can get some semblance of separation - but the underlying OS could always sniff the traffic for that VLAN. Older BMC controllers don't allow changing the VLAN at all, although tools like ipmitool or ipmicfg will ostensibly let you change it, it just doesn't work.
• You're centralizing your failure points on the system. Doing configuration on a switch and manage to cut yourself off somehow? Congratulations, you've now cut off the primary network connection to your server AND the backup via IPMI. NIC hardware fail? Congratulations, same problem.
• Early SuperMicro IPMI BMCs were notorious for doing wonky things with the network interface. Whether to use the onboard vs. dedicated IPMI port was often determined at power-on (not restart), and would not toggle from there. If you had a power outage and your switch didn't provide power quickly enough, you could end up with the IPMI failing to work because it autodetected the wrong setting.
• I've personally had lots of weird, unexplainable connectivity issues getting the sideband IPMI working reliably. Sometimes I simply couldn't ping the interface IP for a few minutes. Sometimes I'd get a storm of packets on the assigned VLAN, but the traffic all appeared to be dropped.

While this has nothing to do with sideband-vs.-dedicated, I'll also note that the tools for accessing host systems are very poorly written. Older IPMI cards don't support anything other than local authentication, making password rotation a total pain. If you're using the KVM-over-IP functionality, you're stuck using an improperly-signed, expired Java applet or a weird Java desktop application that only works on Windows and requires UAC elevation to run. I've found the keyboard entry to be spotty at best, sometimes getting "stuck keys" such that it's impossible to type a password to login without trying 10 times.

I've eventually managed to get 40+ systems working with this arrangement. I've got mostly newer systems I could VLAN the IPMI interfaces onto a separate subnet, and I mostly use the serial console via ipmitool which works very well. For the next generation of servers, I'm looking at Intel's AMT technology with KVM support; as this makes it into the server space, I can see replacing IPMI with this.