I've inherited a rat's nest of cabling. What now?

Question

You know, you see pictures like below and sort of chuckle until you actually have to deal with it.

I have just inherited something that looks like the picture below. The culture of the organization does not tolerate down time very well, yet I have been tasked to 'clean it up'. The network functions as it is, and there doesn't seem to be rush to get it done, but I will have to tackle the bear at some point. I get the ugly eye when I mention anything about weekends.

So my question goes, is there sort of a structured approach to this problem?

My Ideas thus far:

• Label, Label, Label
• Make up my patch cables of desired length ahead of time
• Do each subnet at a time (appears that each subnet are for different physical locations)
• Replace one cable at a time for each subnet
• It's easier to get forgiveness than permision?

Solution 1

In no particular order here are some suggestions that have been helpful to me over the years-

1. Can any of the equipment in those racks be eliminated, upgraded or consolidated? It's hard to tell what's there, but in my experience these kinds of messes tend to be aggravated by gear that should have been pulled out years ago.

2. Once you've got some idea of the minimum set of equipment then consider how best to lay it out. The criteria here may vary, but grouping by technology type or business function might make sense. Clearly the proximity of high density devices (i.e. switches) and patch panels and such will immediately be apparent.

3. Use cable management!!! There are both horizontal and vertical cable management solutions. Use both - horizontals around patch panels and other significant concentrations, verticals next to switches and to facilitate risers.

4. It's always surprising, but how power cables are routed should be considered. UPS units in the bottom of racks, PDU selection and diversity all need to be considered before pulling a cable.

5. Keep inventory of common cable lengths. It's late at night and you want to go home. A 3' cable is what's necessary but the closest you have handy is 5'. This is how these kinds of messes develop.

6. Documenting is part of the game, but the importance of labeling cannot be overstated. With clear labels and efficient/clean cabling the number of mistakes will be vastly decreased and troubleshooting simplified.

7. Limit who can pull cables!!! Differing styles and degrees of attention to detail can yield chaos pretty quickly.

Solution 2

Please call a cabling contractor in to spend a day or two onsite to "dress" your cables. I used to spend time dealing with this type of work on my own, but realize that cabling contractors are faster and more organized.

A good cabling contractor is better at this than you are!

They will have the right resources to tag, test, get custom lengths, dress cabling and install the right type of management infrastructure.

It's worth the time and expense in order to end up with a cleaner, more manageable solution.

If cost is an issue, then you can reduce the hired man-hours by bringing the contractor to handle any structured cabling needs (re-terminating patch panels, testing existing runs, etc.)... Looking at that photo, there will be a need... You can buy or have them provide a variety of patch cable lengths and perform that work on your own, tagging and labeling as needed.

I went from this before-to-after by using this approach. It took 6 hours of contractor help, and 8 hours of my work to get the result.

Solution 3

A co-worker and I recently cleaned up a mess that was pretty bad (I might post pictures later if I get some time) and I wholeheartedly disagree with the contractor approach. You will learn significantly more about the system itself and what deficiencies it has if you do the work yourself. Also when you make a mistake, as you or any other mortal is likely to do, you will have a much better idea of how to fix it. Having said all of this here are my suggestions.

• Take your time: this is a marathon not a sprint.
• First remove all the items that are not plugged in on one end. This took a surprising amount out in our situation.
• Label, label, and label again.
• Do a switch at a time.
• Take your time and check your work.

Solution 4

My sympathies to you. I was tasked with a similar problem for a number of cabinets, equally as horrendous.

The approach I took was as follows:

1. Use a spread sheet to make a list of which port is connected to which port on which piece of equipment (arduous manual process of cable tracking). Try to use a spread sheet cell to represent a port, and order collections of cells as a visual representation of each piece of equipment.
2. For each port, also note the VLAN/subnet etc (I did this via a web interface/cli)
3. Write a small macro to theoretically re-wire your equipment based on cabling distance, or other factor of your choice. The result should give you which switch port maps to which patch panel port (or other equipment). It should also indicate how many cables you need and how long they need to be. I did this in a spread sheet by using cells as switch/patch-panel ports and it worked well. It produced a visual diagram of the wiring by drawing lines between cells and colouring the cells with conditional formatting to show their VLAN number(s).
4. Once I had the schematic it took less than an hour after working hours to pull out all the cables and patch them up according to the generated diagram. Then another 30 minutes the next day to fix the "why can't I connect?!" problems :-)

There's really no easy way to do it, but at least the above approach saved me from pulling my hair out.

Good luck!

Screen shots:

Before:

After:

Solution 5

All these answers are incorrect because they omit the critical first step: Define the project. Before you can use any of these ideas (many of which are very good), you need to be able to answer the following questions with absolute certainty

• What systems you are touching-- when and for how long can they go down?
• Which departments will be hit and when will they agree to be inconvenienced?
• If there is a critical system, when can I shut it down? is it backed up?
• How much money can I spend to do this?
• How many people can I borrow?
• How long can I take? (meaning both man-hours to do it and time when it must be done)
• Is there any documentation and should I trust it?

Maybe you know all these things and you didn't go into detail. But unless everyone is on board with your plan, you can end up writing one of those "How I blew up my company" Infoworld columns.

I had this situation once. Turned out we had a box that had to be on and connected. Pull a wire for too long, it would hiccup and crash. If it went down, anyone using it was screwed. Plus, two other systems-- who rarely used it but continually monitored its status-- would freak.

It only took a few minutes with the manual to figure out the order the systems needed to be shut off and a few phone calls to schedule it and one "Don't log into this system remotely over the weekend!" mass e-mail. But that's the difference between nobody noticing and people hating you forever.

Management's responses to definition questions will help you plan other projects. Your work requires them to make occasional choices and you need to learn what, how and why they will choose. You might find you can't hire anyone to help, can't buy 1-foot cables to replace 8-footers, can't shut people down, can't stop taking "can you unjam my printer?" calls while you do it, can't get overtime and can't come in early, stay late or work weekends if it means you won't be there M-F. If so, you know you need a new job.

View more solutions

Comments

[Admin]

Things like this were the reason that I insisted on seeing the infrastructure before accepting any offer.

[Admin]

My suggestion would be to carefully document absolutely everything and keep it to yourself. Then unplug one end of every cable one weekend. Call your boss and renegotiate for a much, much better compensation package. :D

[Admin]

make sure that any devices that support it are set up for dual-connections, both switch-to-switch and switch-to-device. that way you can do a lot of this work without needing downtime. i cleaned up a similar mess with only 2 hours of downtime to a couple of servers and one router (and no switches). but the whole repatching project took me 24 man hours, 20 of which were during business hours.

[Admin]

Looks like we got ourselves a new 404 image for the site

[Admin]

@hydroparadise - Do you have a real photo of what you're working with?

[Admin]

@squillman I thought that was the 404 page at one stage... Clearly the current 404 is much worse with only one colour cable. I can see at least 3 here :-) And going to the site linked to from the 404 page, you can see this image has been submitted there "Messy Backbone!".

[Admin]

@ewwhite Well I thought about it before posting the question, I just didn't want to inadvertently post anything that might be considered 'sensitive information' due to high security concerns. Still actually considering it though.

[Admin]

If it is in fact a "rat's" nest, sometimes the most efficient method is known as the 'when in rome' approach, so my suggestion: chew through all the cabling.

[Admin]

@hydroparadise The sensitive information is usually inside the boxes, not outside it. Please do post higher quality pictures.

[Admin]

Where can I get a camera that displays the data stored inside of a computer?!

[Admin]

@gWaldo The NSA souvenir shop at Fort Meade.

[Admin]

I meant that there's server names, IP addresses, etc posted all over the equipment. Yet, they are super security conscious.

[Admin]

One place I worked dealt with a situation like this by moving to a new building.

[Admin]

@hydroparadise: can you clarify if your organization frowns on all downtime, or just business-hours downtime. I was assuming the former, but one of the answers below assumed that only business-hours downtime was a problem. It would be helpful to know beyond doubt which one you meant.

[Admin]

@iconoclast Well there's two parts to the down time; we have end users insdide and then we have services we provide to our customers. Downtime with end users is somewhat tolerated which is during business hours, but downtime on services is not so much tollerated which is all the time. It's funny though because the only time they would be willing to have downtime on serices is during business hours so that we at least have people available for customer support to kindly say "Why yes, we are down. Is there anything else I can help you with?"

[Admin]

This whole post/thread should be a sticky or the ServerFault equivalent. SO many very precious advices and tips and ideas in it. Amazing.

[Admin]

Looks like Production source code to me.

[Admin]

That is awful. You should post the after picture aswell.

[Admin]

did you ever get around fixing it or did you really end up looking for a new job?

[Admin]

We finally did as of last week. Forgot that we actually do have "scheduled down-time" procedures for the rest of employees to operate under. We used an approach similar to Jak's answer, meticulously documenting EVERYTHING. We had a few switches that needed replacing for compliance reasons, so we took advantage of the upgrade to get the switches much, much closer to the patch pannel. Now, 90% of the patch cables are under 2' while consolidating a number of the smaller switches. Took 3 of us 4 hours to get it completed with everything back up and running. Still, not as nice as ewwhite's pic!

[Admin]

As a side note, BEWARE OF LOOPS! Can kill a network in seconds.

[Admin]

Show us your "after" picture!

[Admin]

Might I suggest the XM42.

[Admin]

The picture is a little misleading. People have written some answers assuming this is your actual server room.

[Aaron Copley]

That's pretty...

[petrus]

moonraker. what a lovely name for a cisco :)

[shrawani karpe]

You make a very compelling arguement for using a outside company.

[ewwhite]

The contractor did the patch panels, the 2-post rack installation, vertical and horizontal cable managers and ladder work. I performed the patching and color-coded the subnets.

[Patrick]

Very well done. Cabling like this should be a standard but sadly you get people who go in plug in cables and let them just hang there. Color coding the subnets is a great idea. I've met some people who even label both ends of the cables which state exactly which port/switch/rack they plug into.

[woliveirajr]

+1 for the "remove all items that are not plugged"

[shrawani karpe]

If I can convince the higher ups to bring in an outside company, I'll gladly mark this correct correct answer.

[Freiheit]

"Test that things still work. Reboot a client PC and make sure that you can still get to things." - You cannot complete this step without first documenting what can access what. 'get to things' for an IT person may be completely different from 'do my job' to the actual users.

[gWaldo]

@Freiheit Your comment strikes me as a bit pedantic. Was I supposed to have an earlier step that says "document everything that all of your users could possibly need to do their jobs", and replace "can get to to things" with "attempt to do the entirety of their job"? Perhaps that would be the ideal in thoroughness, but it strikes me as excessive.

[Dave M]

+1 for inventory of cables. Unless you are very good and have excellent test equipment, buy certified cables of the needed lengths. Saves much grief

[Freiheit]

"document everything that all of your users could possibly need to do their jobs" is clearly excessive, but "can get to things" is incredibly vague. Especially after working a full day, off hours, and you just want to verify that the job is done and go home. Having a well defined and clear punch list of what things to check is important. It can be as simple as "Login as userbob on secretary workstation and access foo-app on jiggle-bunz-server". Just know what your requirements for "the job is done" are as best as you can.

[rnxrx]

I completely agree on the premade cables. Especially with the advent of cat6 and beyond the tolerances are much tighter around all terminations. I can crimp a beautiful looking patch cable and can make neat 110 punchdowns on a panel, but neither necessarily implies that I've built a completely functional network until it has been actively certified. While it might seem to cost a bit more, the truth is that buying these elements as premade (and standardized) as possible saves a great deal in terms of headaches and mystery outages in the future.

[akostadinov]

re "You will learn significantly more about the system itself and what deficiencies it has", what if you DO NOT want to learn about the system deficiencies? :) Really you will sooner or later but it's about choice...

[PlTaylor]

The sooner you get neck deep in the problems the sooner you can proactively fix things. I have a huge preference for doing things myself whenever I can. You have fiber to run between buildings, hire a contractor. You need to splice a couple cables or buy pre-made lengths, do it your self. You will be happier in the long run when you have a more intimate knowledge of the system.

[sehe]

How much money was spent on materials?

[ewwhite]

Estimate $2500-$3500 US in rack equipment. $300 US on cables.

[JamesRyan]

Two problems with this because its a one off a) often pro cabling jobs will leave it hard to make changes b) cables get messy because of changes often in a rush, so it requires a change in procedure for tidiness to last

[JamesRyan]

And when you have regular quick changes to do, when do these eventually get corrected? The point is that the problem IS a discipline/organizational issue, 80% of the people here could run cables as neatly as a contractor if they had the convenience to do so. The thing is, that in itself that doesn't actually solve the problem.

[Ben DeMott]

@AaronCopley Should see equinix in Chicago... (where some of Google is) they are like uber-pro cablers.

[Ben DeMott]

@ewwhite Thanks for sharing the photo, brings back good memories :)

[David Schmitt]

Having an acceptance test is good. Having a fallback-plan when that test fails is better.

[Justin ᚅᚔᚈᚄᚒᚔ]

@rnxrx: Where do you get certified, premade pre-punched patch panels? (yay, alliteration!) I would love to be able to drop in some working panels with the cables already run, rather than pulling all that pesky solid core cable through the walls.

[user1603602]

@Justinᚅᚔᚈᚄᚒᚔ Pre-punched? Not sure what that means, but maybe you're looking for this?

[iconoclast]

If you're going to re-cable this whole mess, it seems to me to be the perfect opportunity to work out a color-coding scheme. You know (or will soon know) what you have, and can figure out what categories make sense and assign colors to each, and post the color key on a laminated piece of paper nearby so everyone else who works with it can be consistent with your scheme. And you can leave the current cable colors/tones (black, white, and gray?) as "non-categorized" while you use (e.g.) blue, red, green, yellow for new cables. It'll also be easy to see what's been replaced and what hasn't.

[iconoclast]

+1: another awesome answer by ewwhite.

[iconoclast]

I don't see why it has to be an either/or proposition. Why couldn't you work with the contractor, and learn from him, and benefit from his expertise. (BTW, +1 for the excellent points you make, even though I only partially agree.)

[iconoclast]

How would this prevent downtime? Maybe no downtime during business hours, but it's not clear from the question that that's okay. I assumed they did not tolerate downtime at all, but perhaps he meant only during business hours. (At the company where I work, downtime would affect end customers, local retail stores which in some cases are open 24 hours a day, as well as other businesses that communicate with our systems, so any downtime at all is a problem.)

[acolyte]

@iconoclast that is a very good point. I had assumed this was an office-setting where after hours downtime would be fine.

[PlTaylor]

Nothing is black or white and this isn't an either or proposition. My personal bias is to do everything I can that will help my understanding of the system and how to troubleshoot it later. If the cabinet is a mess chances are there is a lot more wrong with it than just a rats nest.

[Justin ᚅᚔᚈᚄᚒᚔ]

@NReilingh: I was being a bit flippant, the wording of rnxrx's comment makes it seem like he was suggesting using "premade certified punchdowns", which is obviously impossible with current technology.

[rnxrx]

Sorry - my wording may have been unclear. "These elements" referred to the patches themselves. In the case of fiber, however, the use of MPO cables is a big win. Pre-made cables with modular connectors, snap-in breakouts, etc. No help as far at twisted pair, but potentially a godsend for horizontal scaling of fiber and the use of 40GE or 100GE.

[Henk]

+1 for the spreadsheet (idea and shots).

[shrawani karpe]

+1 for recognizing that this is a project. It's easy to overlook.

[Geoff]

Thank you. All the work you perform in network administration or help desk is a project. "Reset password" or "change toner" don't require much definition... but the fourth time you unjam a printer, it's time to add tasks like "inspect rollers" or "call vendor" to the plan.

[ItsGC]

sorry, jiggle-bunz is the name of my next loadbalancer.

[HaydnWVN]

Great, but that doesn't look like the same rack/equipment in the pics? Was lots of it changed?

[ewwhite]

@HaydnWVN Yep. The old rack was removed and the old cabling had to be destroyed to move the rack. Total cleanup.

[HaydnWVN]

I've been doing similar here but all in-house... From a 16U wall mount mess to a 42U floor standing 20" deep MONSTER (with an NVR in the bottom, ignore the labels as i've still lots of tidying to do lol)

[the-wabbit]

@Justinᚅᚔᚈᚄᚒᚔ A whole number of cable manufacturers (including international ones like Panduit or Corning) offer pre-fabricated and certified cables with Cat6(A) module termination at various lengths. Take a look at the Catalogues for details: catalog.corning.com/CableSystems/en-GB/catalog/…. If your premises allow for it, installation is as simple as pull & click, although excess cable lengths have to be dealt with.

[the-wabbit]

@Jak care to share the spreadsheet with the re-wiring / VLAN reconfiguration algorithm?

[the-wabbit]

You absolutely should label the cable with device/port but only the device/port name at the end of the cable which is less likely to be unplugged when re-routing. If you are routing cables from servers to switches, label the cable with the server name and interface number (of the server, not the switch) on both ends. If you are routing cables from switches to patch panels, the choice is more difficult and depends on your recabling habits - just define the end you are not going to plug when re-routing and use its name as the patch cable label. Again, on both ends.

[Joel Coel]

I'd be happy if we could even get as clean as your "before" picture.

[Vitim.us]

Maybe a program like Eagle PCB Design program (used to design circuit boards of schematics) can help to layout wiring using the built-in routing algorithm they are designed to avoid crossing wires, which will help solving the nest. other similar program will probably do the job

[Jak]

I would love to share the code but I did this as part of a contract so I don't officially own it :-( Its done using VB macros in Excel by calculating the distance between cells.

[Arthor]

If you have the time, you can always do a good job. Being organised is a highly transferable skill, PM, cabling, electronics, life.. take a pick.

[Krista K]

+1 : when touching anything 5 or more years old, it should next be placed into the trash bin. With cable end(s) cut off.

[Dan Pritts]

I really like your idea of mid-span cable labels. I always label both ends of each cable with a serial number and the length. Having the length labeled is handy when you're reaching into the cable box. I use a Brady ID Pal cable label maker. Be sure to get the "industrial" label tape -it actually sticks more or less permanently, unlike typical label maker tape.

[Pavel]

Spreadsheets. Our systems professor once said that Excel is the most important piece of enterprise software in the world, there are even banks that run solely on Excel. But this is the first time I see Excel used for cabling itself!