Problems with Supermicro X10DRi-T NICs

Using /conf/crontab, as dlawson pointed out, sounds like an excellent idea to me. This allows me to run a script once a minute that ensures everything but http and ssh is shut down:

/etc/init.d/cdserver stop
/etc/init.d/fdserver stop
/etc/init.d/cim_sfcb stop
/etc/init.d/webgo stop

That still leaves me with a web server with password-based access control (I can see no way to have it validate client certificates) and who knows what remote vulnerabilities. Turning it off when I'm not using it (which is most of the time) seems like a reasonable solution; adding a crontab entry to shut it down every five or ten minutes would catch those cases where someone forgets to shut it down when he's done.

The ssh daemon is a version of dropbear that appears to be fairly heavily modified. It reads usernames and plaintext passwords from /conf/PMConfig.dat (which is also used by the web server), logs in any valid name and password as the root user, and ignores the ~/.ssh/authorized_keys file. This last problem is annoying; it forces you to allow password logins and opens the possibility of backdoors depending on from where-all it gets its names and passwords.

So that's the dilemma you face: how much do you really trust this modified ssh daemon installed on a system that was fairly obviously designed by security-naïve developers? Not much at all, given the number of broken bits of cruft I've seen in their shell scripts. There are unusual naming conventions (/etc/rc?.d/sshd is a symlink to /etc/init.d/ssh), huge amount of code that appear to be unused, and features in just the ssh startup script, such as the /conf/portcfg_ssh file and even the restart command are entirely broken. (Don't try using these; sshd won't restart and you'll be screwed unless you have an existing login. We rebooted the BMC and ended up having to reflash it.)

The best option I can think of, if one's going to use the thing at all, is to start ssh on an alternate port using a cron job, so at least it's less likely to appear in a portscan.

The final component is the IPMI network management ports; I can't see how to turn these off.

Jiri's on the right track with the three options (Dedicated, Share, Failover) for the IPMI interface. The short answer is that yes, you can use LAN1 instead of the dedicated IPMI port, and it generally works that way with the default BIOS settings. It's not possible to run the IPMI on the LAN2 interface.

Here's a more detailed description of the three options:

• Dedicated: Always use the dedicated IPMI interface. This is the option you want if you're trying to have the simplest setup, at the expense of additional cabling.

• Shared: Always use the LAN1 interface. This is the option you want if you're trying to reduce your cabling to each server, and understand the tradeoffs. Under the covers, there's a virtual switch in hardware that's splitting out traffic to the IPMI card from traffic to the rest of the system; the IPMI card has a separate MAC address to differentiate the traffic. On modern Supermicro boards, you can also set the IPMI traffic to run on a different VLAN from the rest of the system, so you can tag the IPMI traffic. There are some definite security implication to this design; it's not difficult for the main system to access the IPMI network, if you were trying to keep them separated. A failure of the LAN1 interface often means that you lose primary and out-of-band connectivity at the same time.

• Failover (factory default): On boot, detect if the dedicated IPMI interface is connected. If so, use the dedicated interface, otherwise fall back to the shared LAN1. I've never found a good use for this option. As best I can tell, this setup is fundamentally flawed - I haven't tested it extensively, but I've heard reports it'll fail to detect the dedicated interface in many circumstances because the upstream switch isn't passing traffic - for example, after a power outage if the switch and system come up simultaneously, or if the switch is still blocking during the spanning tree detection. Combine this with the fact that the check only happens at boot, and it's just generally hard to control what interface you end up using.