Securing a SuperMicro IPMI BMC

Question

I've recently acquired a SuperMicro X8DTU-F motherboard, which has a built-in BMC, which basically runs the IPMI system. It turns out to be a small Linux system running on an ARM processor.

Unfortunately, it's running a boatload of software, much of which I don't need, and I don't have the ability to put it behind a firewall. However, I do want the IPMI functionality. Does anybody who has used one of these have some specific suggestions on how to secure the thing? It boots from what is essentially a ROM filesystem, and there don't even seem to be any hooks to turn off any of the various servers it runs....

I'd also be interested in how I might verify the list of names and passwords that can be used to access the system through all of the various services. The default is ADMIN / ADMIN, but none of the files in /conf or /etc have 'ADMIN' in them, which rather worries me. There are /conf/shadow and /conf/webshadow files, with mysterious 'test' IDs in them, which doesn't make me feel particularly comfortable either.

Solution 1

Using /conf/crontab, as dlawson pointed out, sounds like an excellent idea to me. This allows me to run a script once a minute that ensures everything but http and ssh is shut down:

/etc/init.d/cdserver stop
/etc/init.d/fdserver stop
/etc/init.d/cim_sfcb stop
/etc/init.d/webgo stop

That still leaves me with a web server with password-based access control (I can see no way to have it validate client certificates) and who knows what remote vulnerabilities. Turning it off when I'm not using it (which is most of the time) seems like a reasonable solution; adding a crontab entry to shut it down every five or ten minutes would catch those cases where someone forgets to shut it down when he's done.

The ssh daemon is a version of dropbear that appears to be fairly heavily modified. It reads usernames and plaintext passwords from /conf/PMConfig.dat (which is also used by the web server), logs in any valid name and password as the root user, and ignores the ~/.ssh/authorized_keys file. This last problem is annoying; it forces you to allow password logins and opens the possibility of backdoors depending on from where-all it gets its names and passwords.

So that's the dilemma you face: how much do you really trust this modified ssh daemon installed on a system that was fairly obviously designed by security-naïve developers? Not much at all, given the number of broken bits of cruft I've seen in their shell scripts. There are unusual naming conventions (/etc/rc?.d/sshd is a symlink to /etc/init.d/ssh), huge amount of code that appear to be unused, and features in just the ssh startup script, such as the /conf/portcfg_ssh file and even the restart command are entirely broken. (Don't try using these; sshd won't restart and you'll be screwed unless you have an existing login. We rebooted the BMC and ended up having to reflash it.)

The best option I can think of, if one's going to use the thing at all, is to start ssh on an alternate port using a cron job, so at least it's less likely to appear in a portscan.

The final component is the IPMI network management ports; I can't see how to turn these off.

Solution 2

Ideally, your management network would be a different network to your other network, or at least a different vlan with limited routed access.

These systems aren't really running that many services though:

PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
443/tcp  open  https
555/tcp  open  dsf
5120/tcp open  unknown
5900/tcp open  vnc
5988/tcp open  unknown
MAC Address: 00:30:48:D9:3A:71 (Supermicro Computer)

(and UDP/623 for IPMI itself)

Most of these are needed if you want to do any kind of remote management. If you don't want to do remote management, then you should consider not enabling the IPMI controller at all, or buying an X9DTU board instead ( the -F denotes "built-in BMC")

If you want to do full remote management, can't run your IPMI controllers on a different network, and still want to disable some access, then you can always get the IPMI controller to execute iptables commands. You could script an ssh login to execute the commands, or ask Supermicro for the devkit for the BMC and build a new image with a custom iptables script.

UPDATE

I had another look at our systems here, and the /conf filesystem is mounted rw. None of the init scripts called anything directly in /conf (that I could see), but there is a crontab file. So, I guess you could copy in an iptables script, and edit /conf/crontab to call it at some suitable interval. You'd want it to be run ASAP on BMC init, but you don't neccesarily want it running every minute. Or maybe you don't care.

Solution 3

One thing to consider when securing a Supermicro IPMI is the ssh server. Older versions of the X8SIL-F IPMI code accepted ssh connections no matter what password was given. The software would then check the password and reject or accept the connection, but there was a brief window to create ssh port forwards. People were getting spam/abuse complaints for their IPMI IPs because of this. For the X8SIL-F motherboard, The 2.60 IPMI firmware version fixed the problem (it might have been fixed earlier, 2.54's changelog entry looks like it might be it).

A second problem is an anonymous user with a default password. The anonymous user seems to be fixed in firmware version 2.22.

Solution 4

There is a little trick to enable HTTPS for web-interface of IPMI.

If your IPMI firmware supports that (my 2.04 firmware for X8DTH-iF supports), you could, at first, enable HTTPS access by going to Configuration -> SSL, uploading two PEM files (certificate and private key) and, at second, manually reboot your IPMI module.

Finally, you could access IPMI's web-interface by https://bmc-ip-or-hostname/. I can't say that HTTPS works slower than HTTP.

View more solutions

Comments

[Daniel Lawson]

I've yet to find a correlation between the contents of shadow, webshadow, and what are actually valid users. New users added via BMC management don't appear in these files. Further, the "anonymous", "test1", "test2" users have a shell that doesn't exist on the filesystem.

[Stefan Lasiewski]

Also see Dan Farmer's guide at IPMI Security Best Practices. He released a recent paper detailing some signifigant security problems with IPMI titled ipmi: freight train to hell.

[cjs]

Thanks for that great link. The short summary of the security issues is at fish2.com/ipmi/itrain-gz.html and the very short summary is, "you're utterly screwed." Sigh.

[cjs]

I'd love to have a separate network for management, but unfortunately this is going into someone else's data center, and I can't have that. As for the management, all I really want is https and ssh.

[Daniel Lawson]

You'll never, ever want KVM over lan?

[Daniel Lawson]

You could have a separate firewall unit in front of your host at the DC. That would somewhat resolve the problem. Have you asked Supermicro support for assistance on this? I've found them to be fairly responsive

[cjs]

No. I've no need for KVM over LAN since the BIOS, Grub and the Linux kernel all support a serial console.

[Daniel Lawson]

As I said earlier, it's far better to secure your IPMI controller externally: either by having it on a completely separate vlan or physical network, or secured by a border firewall. This particular model of IPMI/BMC controller happens to run linux, which then prompts you with the notion that you could secure it with iptables. The reality is, however, that the vast majority of BMC/IPMI/whateveryoucall them do not have much, or anything, in the way of firewalling, and so you should not rely on it. Save yourself the hassle and treat your IPMI network as private, secured and nonrouted.

[Daniel Lawson]

Most of your concerns regarding modifications are probably not a problem. Dropbear uses pam, which uses libpamipmi for auth - I've not seen any evidence that it actually reads cleartext passwords directly. Libpamipmi will make an ipmi call to the ipmi stack, and that might well be reading cleartext passwords, but my point is that it doesn't look like the dropbear daemon has been screwed up. I'd be interested to hear in any definitive proof you have to the contrary however.

[cjs]

I disagree that it's "better" to secure your IPMI controller externally; firewalls and the like are more prone to security failures than a host that's properly secured in the first place. However, if you have the ability to use a separate network, that's a good thing, and in the case of IPMI devices like this one, it looks to be almost essential.

[cjs]

Well, we know it's been broken in that there's no way to a) use keys, rather than passwords, and b) disable password authentication.

[Admin]

I did ssh -v, to see what was going on, saw that it was attempting Public Key Authentication. I got it to work by disabling that.

[Admin]

I'm now in ATEN SMASH-CLP System Management Shell, version 1.00