APC Smart-UPS: remote power shutoff?

Question

Given:

1. A server room containing about two dozen rackmount servers and a PBX.
2. About a dozen APC Smart-UPS 2200s providing power to this equipment.
3. A fire suppression project requiring immediate shutoff of power to all servers in the event that the fire suppression agent is released to extinguish an electrical fire.
4. We assume that, using separate hardware, we will be able to generate an SNMP trap from the fire suppression system's alarm relay output, and that this will trigger some kind of a UPS power shutdown script on our Nagios server (which fortunately does not share a rack or a UPS with any other servers).

Question:

1. Is it possible to shut down the power to all devices plugged into an APC UPS via SNMP, assuming that we equip each UPS with an ethernet card? The documentation that I've seen refers extensively to the ability to shut down servers gracefully in the event of a power failure via PowerChute; in this case, we would not have time to shut them down gracefully.
2. Does it matter which model of ethernet adapter we use in order to accomplish this? It appears that the old 10Mbps models (AP9606) are available refurbished for quite a lot less money than it would cost to purchase the current model cards; I am unclear as to whether there are significant differences in feature set, other than ethernet bandwidth.
3. Would it be better, for some reason, to use serial cables instead of ethernet cards?

The intent of this question is to tap into your real-world experience with APC UPSs. To date, we have not used any remote management features, and the documentation that I have found to date does not seem to be clear as to whether power can be shut off unceremoniously via SNMP.

Solution 1

Depending on the precise model of UPS, your best bet is to wire the EPO port on the back of those UPSes to the EPO switch on the fire panel.

Smart-UPS 2200 manual, see page 6.

Since these UPSes actually have an actual Emergency Power Off function built in, in order to make your fire-inspectors happy you need to wire those ports to the EPO function of the fire-panel itself. I'm not an electrician, so I don't know the specifics of how you'd wire 10's of these to a single EPO port on the fire panel, but that's a question for your fire panel system vendor.

Half-assing it with suicide software monitors on servers connected to each UPS is the kind of close-enough that gets fire-inspectors to shake their head in that special way that means you'll have to do it all over again, but right this time.

Solution 2

Note that there is a difference between "shut down" and "unpowered" when it comes to computers. When a server shuts itself off, the board remains powered and the server is fully capable of turning itself back on again (think of wake-on-lan). The only way to un-power an ATX-supplied board is to cut the power at (or to) the power supply. In a fire-surpression case, you want all power removed from the server, not just have the server shut itself off.

That said, if you have any advanced notice at all, the command you should send to the server isn't "gracefully shut down" -- that can take minutes to complete. Instead, your goal should be limited to protecting data integrity. An emergency sync and unmount of all filesystems is a much better option in that it usually takes less than a second to complete, which could be all the time you have.

echo s >| /proc/sysrq-trigger
echo u >| /proc/sysrq-trigger

Solution 3

I for one would never sleep soundly knowing that a Nagios / SNMP solution was doing the job of fire supression.

The only choices I would be comfortable with are: something inside the UPS itself triggers a shutdown when it detects overheating (e.g. ambient temperature significantly above normal); or hard wiring to the EPO port (sometimes called REPO port) with some sort of thermal relay or similar. You want this to work, without the slightest shadow of doubt, under all situations.

Comments

[Hecter]

@sysadmin1138 has offered a far simpler solution to this historical question, but I am leaving the original query intact.

[Hecter]

I have edited the question to clarify how we would receive the alert from the fire suppression system.

[Hecter]

I have edited the question to clarify how we would be alerted about the event and initiate the shutdown. I am not asking about the "normal power loss" scenario because that part is easy (and very well documented)!

[joeqwerty]

+1. Well I stand corrected. I didn't think the 2200 had this capability. Thanks for educating me, sysadmin1138.

[Deb]

I had I look it up to be sure, but it's high enough KVA to be at the grade where that's an issue.

[Hecter]

I completely missed that option, probably because I didn't believe that there would be an easy solution. Fantastic answer. Thanks so much.