A previous IT worker probably left some backdoors. How can I eliminate them?

Question

I started working for a company that fired a previous IT worker for leaking data.

I can only say the following things:

We use a Firebird DB with an application written by another company, Proxmox, for virtualization of Windows Server 2008 R2, SQL Server, cloud core Mikrotik router, and a few other Mikrotik devices.

I am not 100% sure, but is there some quick way to check if there are some backdoors left, without interrupting internal processes and reformatting everything?

This previous guy was really good, having written software in C++ and C#. I also know that he did some assembler and cracked a few programs in ollydbg.

Solution 1

The only way to be absolutely certain is to wipe every system clean and to reinstall from scratch. You will also need to audit all of the locally generated software and configurations to ensure that they do not contain backdoors. This is a non trivial task which comes with a non trivial cost.

Beyond that then there isn't really much you can do.

Obviously while you're deciding what to do

• Audit all firewall rules for validity
• Audit all accounts for validity
• Audit all sudoers files for validity
• Change all passwords and keys

but that is only scratching the surface.

Solution 2

First, the most important goal of a fired sysadmin is to clean up his past, particularly if it was a departure in bad standing. If an attack on his previous system were to happen he wouldn't stand to gain anything (particularly not his old job), but he could lose a lot. I've been faced with similar fears many times, but in my opinion, they are largely unfounded. I think it is way more likely that if you were to pose him some questions, he would be very nice and helpful to you (which you should then in turn mention to your boss).

---

Now consider the very unlikely case, that he really wants to do something harmful.

Make an archive of your whole network into some - for him - unreachable location (behind a firewall out of his responsibility, hard disk in a locked cabinet, etc).

As soon as you've made this backup, he can't cover his tracks any more. In the case of a rogue attack, it will serve as evidence.

---

You can't ever be 100% sure (except in the case of a reinstallation of the whole network). You can pass through a checklist, but you can't ever be sure that you've checked everything.

Even in the case that you find a hole, you can't prove that it was placed there intentionally. Note, the same problem exists for software developers. Bad work is not a criminal offense, and you can't prove that he "forgot", for example, to change the default database admin password, intentionally. Or that users, whose password was set by him, "accidentally" were granted the privileges required to connect to your top-secret databases.

---

In most cases, the most important part of your system isn't your OSes, but the data which is managed on them. It is particularly so, if this data is private data and the property of your customers. He could have stolen them long before his last workday, encrypted them and saved them in places known only to him. So make sure to also check for traces that indicate that he made copies of your data before he left. Note though that if he did it "properly", you won't find anything.

Solution 3

@JonasWielicki compared this question to one of our canonical security questions (How do I deal with a Compromised Server). I stand by my answer to that question but there's an important difference.

In that question the server was known to be compromised. As far as I understand this question that hasn't been established in this case. As such, I'm not sure we need to be breaking out the flamethrower just yet.

People leave an organisation all the time without anything bad happening, even when they've left under poor terms. Simply being good at programming, which is all the "evidence" you've shown us in the question OP, does not mean that someone is a hacker per se, nor that they're likely to attack you now that they've left.

If you're truly concerned then I would suggest hiring an outside security firm to audit your system. Not only will that hopefully uncover any little surprises that may have been left by the former sysadmin, it will also serve as a good baseline for all your security issues and concerns going forward.

Solution 4

The only way to ensure that no backdoor exist for sure is nuking the system like you said.

If that isn't entirely possible,

1. Consider a baseline secure configuration and analyse of the current configuration is deviant from that.
2. Verify all the suid programs.
3. Analyse all the running processes.
4. Perform a portscan on the system to identify open ports and services.
5. Routinely monitor all outbound and inbound connections and look for rogue connections.

Solution 5

You're going to need to decide how sure you want to be. The cost-benefit is never going to pan out for nuking from orbit.

Are managers demanding assurances or are you just trying to do some reasonable examination of the systems you've inherited?

If it's managers, now you get to find out how reasonable they are. Are they willing to settle for "pretty sure"? Maybe you can follow the fired guy to his new job!

If you're looking to examine your own systems, I'd start with setting up a network monitoring system, like snort. Look for unexpected traffic, like " why is the system talking to this one server in Russia every day?" or "why are people doing IRC over my web server?" (that one happened to me).

I think @peterh's suggestion about making a big archive is a really good idea. I also think his suggestion about the fired guy being helpful is totally realistic. Problems like this turn out to be stupid management 90% of the time.

View more solutions

Comments

[Michael Hampton]

Comments are not for extended discussion; this conversation has been moved to chat.

[Boris the Spider]

Indeed. A full code audit is the only way to be sure - understand what each and every line of code does. It is trivial to add some "special" URL that opens a shell, and hide that in obscure application logic. No amount of scanning, rebooting or checking config files will detect, let alone fix, that.

[Nzall]

Note: "reinstall from scratch" REALLY means from scratch. Get a new machine he couldn't possibly have touched and use that to download fresh installation files for all operating systems and software you use, from the official website, preferably over a secured connection and from an entirely different network. If you cannot find the correct installers online, use a checksum on any existing files before you use them.

[user2357112]

Considering all the physical access the guy would have had, you might need to check for unusual or unexpected hardware.

[Samira Aliyeva]

If you have to use software the guy wrote and you have completed a full code audit then re-compile because you still can't be sure the binaries you have are entirely from the source code you audited.

[user]

Note that peterh proposed How do you search for backdoors from the previous IT person? as a possible duplicate, which would IMO be a far better match than How do I deal with a compromised server?

[Rob Moir]

@MichaelKjörling I'd agree it would make a better dup. I think the question is different enough to the 'search from backdoors one' to survive even if the answers are similar.

[user]

I'm actually somewhat undecided on whether the two cover the same ground. Finding the backdoors is an obvious first step before they can be eliminated; once found, the specific steps would be somewhat different depending on the type of backdoor.

[jwg]

@MichaelKjörling I'd say it's very far from obvious as a first step. As the top answer shows, by far the most common strategy is to assume that everything is tainted, and to wipe, re-install, etc. without making any effort to find backdoors.

[TessellatingHeckler]

@BoristheSpider A full code audit is not the only way to be sure, it's not enough to be sure. You might be able to understand each and every line and see they look safe, but they might still combine to produce an unsafe overall system. Consider the valid regex which crashed StackOverflow or the entries of the Underhanded C code contest. If a 'really good' programmer made two parts of the system look right but act maliciously together, a code audit wouldn't find it

[peterh]

Many passwords are deeply hardcoded into sources and configuration resources. Particularly the ones belonging internal services. If the system is complex, often they can't even find all of them. Often it is not even known that it is a essentially password (well... credential token), for example a 128bit hexa code in a config file, zipped deeply into a .jar file.

[peterh]

Nuking from orbit is costly, it would means months of work, with the high chance of a lot of stability problems. Particularly in big companies, the allowed downtime is essentially zero.

[peterh]

@beldaz Nuking from orbit can easily mean months of work, and the system will be unstable in these months...