Advice on moving a machine room to a new location?

Question

Our company is moving to new offices in a couple of months, and I am responsible for looking after the move of the development servers in the company. most of the dev equipment is in 5, 42U cabinets + rack for switching/routing equipment. How do most people do this sort of thing? Move the cabinent whole or extract the indvidual components and move the racks empty.

any advise on prep and shutdown before the move would be welcome

Solution 1

We have done a couple of moves with racks. Here's what I took away from the experience:

Before:

• label everything. Both ends. And the rack slot it's plugged into too.
• diagram everything.
• if you have important data, schedule a full backup to finish 6-12 hours before the move. At a minimum, validate the last full backup you have.
• plan to take everything out of the rack during the move. At a minimum take out half the mass, from the top down -- this means leave the top empty Don't expect movers to move a top-heavy rack. In a rack with a UPS at the base that means you are taking out probably 2/3 of the servers. Anything which is left racked MUST have rails and be secure in the rails (less than 0.5mm movement, and tolerant of any orientation -- including upside down and face down -- was our margin). We've moved full racks, and half-full racks, and the racks always need to be tilted/rotated, and it is downright scary to watch your livelyhood being dangled over a concrete loading dock.
• in our experience, front and rear doors are more decorative than load-bearing and leaving them on can make moving the racks very awkward because all the good hand-holding points are unavailable. Plan to remove the doors during the actual move. This forces the movers to lift, pull, push, and tug the rack by its strongest parts. Usually side panels can stay on, but be prepared to remove them if the movers think having them gone would help.
• hire someone with an insured truck to to the transportation. DO NOT do it yourself. You might think you are insured, but your insurance company will probably think otherwise. The second last thing you want is to be involved in a three-way blamestorm between your employer's insurance company and your insurance company.
• hire someone insured to physically pick up, move, put down, and (during an accident) drop your gear for you. DO NOT do it yourself. The last thing you want to do is hurt yourself or be involved in a co-worker's injury.
• make sure your insured someones specialize in high-tech moves and can supply blankets, anti-static bubble-wrap, and if necessary crates.

During the move:

• backups are good, right?
• do the wire disconnect yourself. Pull ALL wires that are disconnectable. Each wire should be labelled and go into a box that itself is well labelled.
• let the strong, well insured guys do the server extractions from the rack. Supervise the wrapping personally.
• stay out of the strong, well-insured guys' way when they do the loading and unloading.
• supervise the unwrapping and re-insertion into the racks. If there is ANY question about a server's condition, mention it, and note that it was mentioned.
• If the job is taking a while, have coffee and/or doughnuts available for the guys. If appropriate (and if they've done a good job), cold beer for after the job goes well too.

Physical bring-up:

• double your time estimate for getting things back on track.
• have a build-up plan. A good build up plan includes a checklist of all services, servers, and a test plan.
• validate, validate, validate.
• check any servers which may have been damaged in transit as noted above.
• don't let management cheap out when it comes to food if you are working through meal times. Hungry techs make mistakes. If the bring-up is going to be long and complicated, schedule time to get out of the room for 90 minutes to go get a real meal.

After:

• a cold beer for you may be appropriate, too.

Solution 2

I'm with Kevin on unracking and disconnecting. Racks that I've worked with that are built to ship loaded typically have additional bracing and must be attached to a fixed base (like a pallet) during shipping. Usually the servers have some kind of locking mechanism, too. If you're not moving something that isn't built to be moved loaded don't.

I would add this to Kevin's answer:

Part of documenting is knowing why the various connections and cables are there. You can make all the notes, take all the pictures, etc, that you want to, but at the end of the day if you don't understand why something is there you're running a risk of not knowing what to test to establish, after the move, that you "got everything".

This is a golden opportunity to, as you're documenting the "what's plugged into what" of the move, ask yourself "why is this plugged into this"?

I've had too many cases of the "I'll just patch this here and temporarily change a VLAN assignment" that turns into a semi-permanent configuration that no one documented come back to bite me when moving equipment years later.

Before any equiment goes into a personal vehicle, find out how the company's insurance is handling claims in the event of damage during transport.

Back everything up before you begin moving. Assume that all the equipment will be destroyed during the move because (if you're driving on public roads, at least) it could be. It goes w/o saying that the backup media should not move with the infrastructure gear!

Solution 3

Extract the individual components and move the racks empty. This also has the advantage of allowing you to reconnect things in an optimal configuration and get your wiring exactly the way you want it. Just be sure to document everything before, during and after. Take digital pictures to help.

Solution 4

Buy an awesome label maker, and label everything like you're going insane. Then take it apart and move it piece by piece.

That is all.

Solution 5

We relocated two racks of servers about 60% full about 3 years ago.

It was the most frightening thing I have witnessed in a long time.

Everything was OK in the end, but the movers had to tilt the rack to get it out of the door at our old location. They were very big and very strong, and lucky for me the rack did not drop -- but I can tell you, I was stressing just watching it.

If you must to move the racks full, make certain that you can roll the racks through the doors with out having to tilt them at every door that you will encounter during the move. if you have to tilt, the forget it and start unloading servers :-)

I would personally advise on either emptying the racks first, or like Kevin suggested, getting new racks at your new location. This would also perhaps allow you the flexibility to move the systems over in stages, rather than all at once. Of course that might not work given your specific environment.

View more solutions

Comments

[MikeJ]

thanks to everyone who commented on my question. I tried to upvote everyone who added something to my understanding of what I am facing in the comming weeks. Hopefully, this will go without a hitch (wishful thinking I know).

[KPWINC]

+1 I was going to ask the author, "Have you ever SEEN someone try and move a FULL RACK? Its down right SCARY!" There can be A LOT of weight (not to mention breakable parts) in a fully loaded rack.

[David Mackintosh]

Interesting, but you almost got a -1 because the linked site started playing audio when I clicked on it.

[kmarsh]

+1 for the digital pictures.

[Chad Huneycutt]

lol, I totally forgot about that (go flashblock!) But you can't downvote A-Team :)

[MikeJ]

my predecessor was a dymo nut - all the servers are labeled with name, ip it's reservered. and even color coded etherenet cabled depending on which row the server is in the cabinent.

[MikeJ]

more I thought about it, moving it empty seems the way to go, but unloading the top level servers out of the rack seems very very scary.

[MikeJ]

as cool as this would be, I am not getting this luxury. i have to be down, out, shipped (about 15 city blocks) and up between a Friday and Monday.

[Laura Thomas]

Wow. I hope you have a lot of help. Document like crazy. Is your network architecture changing? If not I'd make sure I had good backups of my switch configs and make sure each server got back in the same rack in the same switch ports. That will hopefully avoid weird "We put this on it's own vlan in 2007 but forgot to document it" problems.

[Laura Thomas]

Getting big servers out of the top of a rack is a two or maybe three person job. I mean sure I can hold a 4U server over my head... but can I do it while releasing the rail catches, and or realigning them? NOPE! Even hired outside techincian labor is cheaper than one dropped server.

[Kara Marfia]

This is great stuff. Wish I'd had it a year ago!

[Dennis Williamson]

+100 for not moving the backup media with the gear.

[Dennis Williamson]

"and make sure it's just just 'yes, the hot's hot'" - s/just just/not just/

[Dennis Williamson]

A lift table or load lift can be helpful for lowering equipment removed from the upper levels of a rack if you get one with high enough reach.

[John Gardeniers]

+1 In addition to all the labelling I like to take photos as well.

[Aashraya Singal]

One of my core matras "Assume everything temporary will be permanent". More often than not, it is.

[MikeJ]

with the idiots that moved the empty racks, I am glad we moved the actual hardware independently

[Krista K]

Product idea: engine hoist reconfigured for server installation...