How to set up a centralized backup server with lots of offsite workstations, intermittent internet connectivity, and stubborn users?

Question

This might be an impossible question.

Context:

We have a bunch of computers across around 1000 users. We have a centralized office where 900 of the users work, most of the time. Most of the computers are laptops. They are very frequently coming on and off the network for hours at a time. Users often take their computers home and do lots of work from home. In addition, there are a handful of users who work elsewhere in the country, who are offline (no internet connection whatsoever) for more than half of the time they use their machines. All of the machines are Windows 7/XP.

Problem:

People are always losing data. One day someone accidentally deletes a bunch of files. The next day someone else installs a bad driver or tries to mess with something in system32 and needs a personal data backup/reinstall of Windows. Because of how many of our business operations are done without an internet connection, and how frequently computers come on- and offline, it's unfeasible to make users use network storage for all of their data. We tried giving them Dropboxes, and they stored their files elsewhere. We bought and deployed Altiris, and they uninstalled it and blamed us when they couldn't get files back that they accidentally deleted while they were offline and hadn't taken a backup in months. We tried teaching them backup best-practices, and using scheduled sync tools to upload things to the network drives, and they turned them off because they "looked like viruses". It doesn't help that many of these users are pretty high up in the business and are not amicable to any sort of "you need to do something regularly because we say so" solution.

Question:

Other than finding another job where IT is treated differently and users are willing to follow best practices, how would people recommend I implement a file backup solution that supports the following:

• Backs up to a centralized server over LAN or WAN whenever a network link becomes available, or on a schedule.
• Supports interrupted/resumed backups (and hopefully file-delta only backups), since connections to the network (WAN or LAN) are often slow and only open for half an hour or so.
• Supports relatively rapid, "I accidentally deleted the TPS reports! Oh no!" single-file recovery, ideally administered from the central backup server rather than the client PC.
• Supports local-to-local file delta backup on a schedule, so that users without a network connection for a few days can still retrieve accidental deletions or whatnot. Ideally, the local stored backups would be pushed up to the server whenever network link is available.
• Isn't configurable on the clients without certain credentials. Because the CFOs (who won't give up their admin rights on the domain) will disable it if they can.
• Backs up the entire hard drive. There are people who are self-righteous about storing things in C:\, or in the recycle bin, or in the C:\Windows dir (yes, I know).

I'm fine integrating multiple products/solutions, or scripting different programs together myself (I'm a somewhat competent programmer), but I've been drawing a blank on where to start. Dropbox is folder-specific, Altiris doesn't cope with LAN outages or interrupted/resumed backups, Volume Shadow Copy is awesome for a local-to-local solution, but I don't know how to push days of stored shadow copies up to a server in a 2 hour window of network access. The company is fine with spending decent money on this, thousands (USD) on a server, and hundreds on clients, if necessary.

I want to emphasize that this isn't a shopping list request. While I wish there was a program out there that did what I want, I've looked pretty hard, and not found anything that fits the bill. Instead, I'm hoping for ideas on where to start hacking things together from scratch/from different technologies to make something stable that works. Cheers!

Solution

I personally use CrashPlan Business Solution for the small office solutions that I provide. It's easy to setup, and backups up to their servers. It's easy to monitor, and doesn't matter where the user is located (WAN or LAN connections).

However, with a larger corporations I would suggest looking at CrashPlan Enterprise Solutions as that option appears to have the ability to:

Build your private cloud with managed hardware engineered exclusively for CrashPlan PROe.

The pricing for the business version is great for small businesses, but the customization and ability to expand is low. Since you're dealing with 100's even close to a thousand clients I would look at the CrashPlan Enterprise first.

The great thing about CrashPlan is that you can 'rollback' or restore deleted files, including individual files. They typically don't have a cap on the amount that you can store (as long as you're willing to pay). It supports all platforms (Windows, Mac, Linux). If users disconnect during a backup, the backup will resume from where it last left, meaning that individual files are sent and monitored. This is similar to dropbox.

There's also a great dashboard which allows you to manage users, devices, and monitoring backup status's from either the web or even your phone (yep they have an app for that)

Here's a small sample of the 'devices' section in the online dashboard:

You also do the restoring or rolling back of files from this dashboard:

I think that this is going to be your best bet as it appears to meet most of your criterion, in a single solution.

The only problems that I don't know about are:

• Supports local-to-local file delta backup on a schedule, so that users without a network connection for a few days can still retrieve accidental deletions or whatnot. Ideally, the local stored backups would be pushed up to the server whenever network link is available.

• Isn't configurable on the clients without certain credentials. Because the CFOs (who won't give up their admin rights on the domain) will disable it if they can.

With bullet one, I'm not sure how the enterprise system works, but I believe that the Business solution pushes straight to the server, and not to the local machine.

With bullet two, again, I'm not sure how the enterprise system works, but the business has a small client that runs (and is visible in the "System Icons" area of Windows) that can be controlled by the user. I would suggest talking to CrashPlan for more details in regards to these issues.

Note: I am NOT an affiliated member/employee/whatever of CrashPlan. I'm just a user that really likes it.

Comments

[Admin]

It sounds to me like you are in the unfortunate position of having to try to achieve this task with both hands tied behind your back.

[Admin]

Spot on. I mean, they're fine giving me lots of MONEY for it. They just don't want to have to change any of their bad habits, or, you know COOPERATE at all. :)

[Admin]

Don't give them admin accounts on their machines. Your users clearly aren't responsible enough to handle it and will happily waste the company's money because of this. Have them sign a document stating it's not your problem when (not if) they use files when they want their admin access back and have the influence to make it happen, making it clear that you strongly advice against it. Create a nice presentation showing data loss statistics and the impact on the business to convince as high up in the chain as possible that this is necessary.

[Admin]

I like the title "How to set up... stubborn users" because it's hilarious.

[Zac B]

Does it support file-level recovery? If a user is backing up 200MB of data and they go off the network for an hour halfway, does it resume or restart the transfer? Does it back up changes locally when a user is offline for a day or more? If not (to that last one), does it have an API/is it easily scriptable to take Shadow Copy data and back it up as well?

[afrazier]

CrashPlan does do file-level recovery, resumes interrupted backups, does compressed and deduped binary deltas, though I don't believe it "holds" backups locally while offline though, but it does use VSS for handling open files.

[Prerak Diwan]

@ZacB I've updated the answer to what I know. (so Yes, and Yes to the first two) As far as the local backups, I'm not for sure, and would suggest talking to CrashPlan in regards to this. Their customer service is quite exceptional as well. Although I haven't really dealt with them that much since it just works, and I really don't have to do much or have a lot of problems with it.