How to extinguish a small fire in a server rack to minimize damage to surrounding equipment?

Question

Suppose I have a rack with several servers and other stuff. One of servers overheats severely and either starts smoking or catches fire while there's a serviceman nearby.

If anything similar happens in an apartment and there's a fire extinguisher nearby using the latter promptly often lets extinguish the fire very fast but in case of server rack improper extinguishing may lead to unneeded extra damage to surrounding equipment.

To clarify, I'm talking about a really small fire that one can try to extinguish without risking their life - like grab a nearby extinguisher, discharge it and get the fire extinguished in say ten fifteen seconds.

What's the strategy to extinguish a small local fire in a server rack? What type of extinguisher is to be used? How to minimize damage to surrounding equipment?

Solution 1

Definitely extinguish the fire without regard for the rest of the servers on the same rack. Any properly designed data center will have backup servers at another location for the applications to fall back on. It is better to lose a rack of servers than to lose a life or the whole data center.

Solution 2

This is a strange question, but I'm gonna attempt to answer it anyway.

All electrical fires must be extinguished carefully. Especially if they're still live. Fire departments will all recommend the use of a CO2 extinguisher.

In a datacentre environment, however, I'd (But I don't recommend you) do one of two things.

1. Hit the EPO (Emergency Power Off) and the fire alarm button.
2. Find a CO2 or Halon* extinguisher and attempt to extinguish the fire
3. If that doesn't work. Goto 1.
4. Unconditional Jump: Run out of Fire Exit.

I'd be more concerned about the protection of the rest of the suite than the expense or damage caused by following their procedure for fires in the building. You're operating as a business, so you've got insurance, both for your hardware, and Public Liability. It'll all be covered by that if you've got a decent insurer.

It'd be better all around to hit the fire alarm and let the professionals deal with it. Electrical fires produce evil acrid choking smoke, often containing horrible toxins, and I sure as hell wouldn't wanna be trying to extinguish it without breathing apparatus.

*Halon is a bit of an odd one. It's a fantastically efficient extinguishing agent, but has a nasty side-effect of destroying the Ozone layer. I've only seen Halon extinguishers in datacentres, and even then, there's a lot of paperwork if you discharge one. It works by depleting the oxygen in the applied area, and as a result, it'll suffocate the fire, and you at the same time if you linger too long. Again, this kind of lends itself to calling the fire service. Or activating the building's fire supression system. Don't worry about the expense, that's what insurance is for.

Also.. has this just happened? I can't resist this.

(Disclaimer: I'm an ex fire-cadet, so I've got a pretty good knowledge of fire and it's extinguishing. )

Solution 3

Speaking from organizational experience at my last job, a small fire in one rack can turn into a big fire with a whole rack, and very shortly afterwards, the rack next to it as well. Fire spreads fast. Once the insulation jackets of all of those power and Ethernet cables gets involved it travels. It travels really fast. Then the flames start tickling the ceiling tiles and things get really bad.

We know this from experience, though I no longer have access to the photos since I don't work there anymore.

When you get a fire, any fire, you need to treat it like you would a larger fire. Since server-rack fires can burn fast and hot, you need to have enough fire suppression in place to handle something like that:

What's the best fire suppression for a server room?

Our experience with this issue was related to having insufficient suppression in the area the servers were kept. There may have been sprinklers in the room, but considering the age of the building that was unlikely. There were CO2 extinguishers, but the problem hit when no one was in the building and therefore present to use them. By the time the Fire Department got on site, the building was well on its way to gutted. Bother.

The server room that was involved in this conflagration had maybe one point of the items on that 'best fire suppression' answer. Don't be us.

Solution 4

If you don't know what to do in the event of a server room fire then you shouldn't be in the server room...

By which I mean that if you have enough computers and switch gear in one place to call it a "server room" then you have enough equipment in there that could be a potential fire risk, and at that point someone needs to have a sit down and have a think about procedures for dealing with a fire in that room and all the staff who are allowed in the room should then be trained in that procedure.

You should at the very least have a CO2/Halon/similar hand-held extinguisher in that room and training available for all people who might have to use it.

If the room is large enough and/or contains enough valuable equipment then an integrated fire suppression system should be considered. We have them in both of our server rooms, with FM-200 being our gas of choice as its a fairly new install, rather than Halon, which I don't think you're supposed to use in new installs these days, at least not in the UK.

In both cases, while I'm not a health and safety bore nor do I wish to pretend to be one, this is one area where the right steps to take in the event of a fire in the server room should be given to someone before they are allowed to work in the server room, and the first step in the list of procedures should include a warning about risk of death/injury and instructions to stop fire-fighting attempts and evacuate if there are any possible doubts.

Solution 5

We have a CO2 extinguisher in our comms room. I guess I'd use that if I was in the room when the fire started, and noticed it. Couple of discharges at the base of the fire, then get out even if it appeared to be out, then raise the alarm.

The only time we had a fire in our comms room, it was the AC unit that started it, and luckily it didn't spread. No one stayed around to tackle it - the smoke detectors caught it before anyone did, and the building was evacuated. Having the previous nights backup tapes in a pelican case next to his desk, ready for collection by our offsite storage suppler, our network manager carried those out with him.

The fire brigade came, and it was valuable to let them know the fire was in the server room, and give them an access card. They didn't need to take an axe to the door or destroy anything - they went in with breathing apparatus, located the fire and dealt with it. So one thing worth doing is being around to talk to the fire brigade when they arrive, rather than disappearing down to the pub with everyone else...

It would have been a major disaster for our organisation if the fire brigade had run in and sprayed everything with water. Yes, we have an offsite recovery where we believe we can recover our systems within 48 hours. But we would really rather avoid having to use it.

View more solutions

Comments

[Tom O'Connor]

1) Hit the fire alarm button in your datacentre. 2) Run.

[sharptooth]

@Tom O'Connor: That only minimizes risk to one's life.

[user9517]

This really should be addressed to your local fire prevention office.

[sharptooth]

@Tom O'Connor: Well, I know that, still running away when I'm sure I can spend fifteen seconds to at least try seems quite stupid.

[Sirex]

Not so much. A halon fire suppression activation isn't a fun time, and plastic can burn with poisonous gasses. Either way i wouldn't want to hang around. Opening the server up and providing more air to burn isnt going to help a whole lot. So it depends, if its super small fire and you have an extinguisher near by and can yank the server out and spray it with something electricity safe then maybe, but i wouldn't risk it for anything even remotely dangerous. even burnt fingers suck when you're footing the bills.

[Tom O'Connor]

There's a great phrase on most fire policies (ie, company rules) "If you feel capable of tackling it, do so, but do not take any risks"

[RobW]

This question doesn't sound hypothetical to me. If you're actually trying to come up with a way to justify keeping an actual, known fire risk in your data center, you need to stop and really think if it is worth the risk.

[Hecter]

The biggest problem with this "localized fire suppression tactics" myth is that the first step to extinguishing an electrical fire is to TAKE AWAY THE ELECTRICITY. If you don't have an EPO switch hooked up to every one of your UPSs, you can pretty much forget about it...

[MrGigu]

Minimising the risk to your life is worth more than any stupid servers sitting ina data center. If you're comfortable in the knowledge that you have off site backups, get the hell out of there. If you want to quantify it, the cost to your insurance to replace the entire building and all its contents is probably a few million. The cost to your insurance (and public liability) if you die would be subtantially more, not counting the fact that you're dead, which your family will then have to deal with.

[Rob Moir]

That only minimizes risk to one's life - you say it as if there's something wrong with that. There isn't.

[oracle certified professional]

Are you a manager, perchance?

[Khushboo Tahir]

Leave the fire suppression to the professionals and the server operators then. Smoke from a server fire is physical, lethal and not covered by a software algorithm. Your best fire suppression is to cut the power to the rack and run.

[sharptooth]

Shouldn't item 3 read "... goto exit"?

[Matthew Scharley]

+1, totally agreed. Electrical fires, and fires in general, are very dangerous. Hardware can be replaced. Software can be reinstalled. Data can be restored (you have backups, right?). Lives can't be revived.

[pauska]

+1 for CO2 extinguisher. Using foam or chemical will surely destroy your equipment for good.

[Tom O'Connor]

@pauska and possibly you, in the process.

[skisloski]

You might want to test that pseudo-code for potential fatal errors...

[Joe H.]

Agreed ... not extinguishing the fire means it damages the whole room, so if you only hurt one rack, or the racks immediately next to it, you're still better off.

[MrGigu]

You guys do realise that you're not computer programs, right? You're capable of exiting a for/while or a do loop without specific instructions to do so...

[fluffy]

@sharptooth goto considered harmful. And exit() is TERRIBLE for error handling.

[sharptooth]

@fluffy: Who said goto? break would do.

[dunxd]

@mfinn - yes, we test the site annually. Still don't want to use it though.

[dunxd]

@RoundTower - in an ideal world that would be in place. Welcome to my real world...

[Rory Alsop]

Of course Halon is not used any more - instead we have halon replacements which are better for the ozone layer, but can kill you faster (some not only displace oxygen, which is quite difficult for humans, but can react with the moisture in your mouth/nose to produce nasty toxic substances)

[Rory Alsop]

by human breathable, they mean - you can survive for the time it takes to get to the exit. They work by suppressing oxygen, and us humans need a bit of the old O2

[Tom O'Connor]

Halon isn't used anymore in the EU.. Some places still use it.. The US military still loves the stuff (According to Wikipedia)

[Dan Is Fiddling By Firelight]

The "halon" (not sure what's in the tanks) systems in the com-rooms where I work supposedly don't release enough of whatever they use to be a suffocation hazard; making frostbite if you stand under the nozzles the biggest safety hazard of the system. Caveat: I'm just repeating what I was told in the annual training sessions and haven't actually hit the big red button and stayed inside to test the claims.

[adavid]

Oh sure. But at least it is not (too) corrosive to mucosa ;)

[Rob Moir]

@DanNeely - if you're standing underneath one, frostbite around the face is still a serious hazard. Don't forget that. And there have been reports of pressurised fire suppression systems that discharge gas fast enough to break windows and even rupture (thin) walls. The modern gasses might be better than halon but I'm still not sure I'd care to be standing near one of the vents when it discharged.

[Khushboo Tahir]

Cut power to the rack before bothering with a fire extinguisher, the heat source needs to be eliminated before the fire extinguisher will be useful in suppressing the fire. And A/B class extinguishers are as useful as putting water on a magnesium fire.

[Khushboo Tahir]

@RoryAlsop with the exception of Aerospace where it's kept around for suppressing avionics/aircraft electrical fires. Despite the cessation of production in 1994, there's still enough around to keep us supplied. H3R supplies these for aircraft use.

[Rory Alsop]

@Fiasco - interesting. I didn't realise anyone still used it :-)

[Daniel B.]

This is a bit old, but it was part of the question that I don't see answered: Energized electrical equipment fires should be extinguished with a class C fire extinguisher, but the FIRST thing you should do is cut power (at the breaker/UPS. Please don't try to turn off the power supply on a burning server :p )

[Peter Green]

Halon 1301 can suppress fire without displacing all the oxygen and without being immediately deadly to humans (though it's still advised to get out if you can). Last I heard it was still be used by Eurotunnel on their car shuttle trains for this reason.