How do I know if my Linux server has been hacked?

53,394

Solution 1

  1. Keep a pristine copy of critical system files (such as ls, ps, netstat, md5sum) somewhere, with an md5sum of them, and compare them to the live versions regularly. Rootkits will invariably modify these files. Use these copies if you suspect the originals have been compromised.
  2. aide or tripwire will tell you of any files that have been modified - assuming their databases have not been tampered with.
  3. Configure syslog to send your logfiles to a remote log server where they can't be tampered with by an intruder. Watch these remote logfiles for suspicious activity
  4. read your logs regularly - use logwatch or logcheck to synthesize the critical information.
  5. Know your servers. Know what kinds of activities and logs are normal.

Solution 2

Some things that have tipped me off in the past:

  • High load on a system that should be idle
  • Weird segfaults, eg. from standard utilities like ls (this can happen with broken root kits)
  • Hidden directories in / or /var/ (most script kiddies are too stupid or lazy to cover their tracks)
  • netstat shows open ports that shouldn't be there
  • Daemons in the process list that you normally use different flavours of (eg. bind, but you always use djbdns)

Additionally I've found the there's one reliable sign that a box is compromised: if you have a bad feeling about the diligence (with updates, etc.) of the admin from whom you inherited a system, keep a close eye on it!

Solution 3

You don't.

I know, I know - but it's the paranoid, sad truth, really ;) There are plenty of hints of course, but if the system was targeted specifically - it might be impossible to tell. It's good to understand that nothing is ever completely secure. But we need to work for more secure, so I will point at all the other answers instead ;)

If your system was compromised, none of your system tools can be trusted to reveal the truth.

Solution 4

Tripwire is a commonly used tool - it notifies you when system files have changed, although obviously you need to have it installed beforehand. Otherwise items such as new user accounts you don't know about, weird processes and files you don't recognize, or increased bandwidth usage for no apparent reason are the usual signs.

Other monitoring systems such as Zabbix can be configured to alert you when files such as /etc/passwd are changed.

Solution 5

There's a method of checking hacked servers via kill -

Essentially, when you run "kill -0 $PID" you are sending a nop signal to process identifier $PID. If the process is running, the kill command will exit normally. (FWIW, since you're passing a nop kill signal, nothing will happen to the process). If a process isn't running, the kill command will fail (exit status less than zero).

When your server is hacked / a rootkit is installed, one of the first things it does is tell the kernel to hide the affected processes from the process tables etc. However it can do all sorts of cool things in kernel space to muck around with the processes. And so this means that

a) This check isn't an extensive check, since the well coded/intelligent rootkits will ensure that the kernel will reply with a "process doesn't exist" reply making this check redundant. b) Either way, when a hacked server has a "bad" process running, it's PID usually won't show under /proc.

So, if you're here until now, the method is to kill -0 every available process in the system (anything from 1 -> /proc/sys/kernel/pid_max) and see if there are processes that are running but not reported in /proc.

If some processes do come up as running, but not reported in /proc, you probably do have a problem any way you look at it.

Here's a bash script that implements all that - https://gist.github.com/1032229 . Save that in some file and execute it, if you find a process that comes up unreported in proc, you should have some lead to start digging in.

HTH.

Share:
53,394

Related videos on Youtube

cowgod
Author by

cowgod

Updated on September 17, 2022

Comments

  • cowgod
    cowgod over 1 year

    What are the tell-tale signs that a Linux server has been hacked? Are there any tools that can generate and email an audit report on a scheduled basis?

    • Oskar Duveborn
      Oskar Duveborn about 15 years
      If the state is unknown, there is really no way. That's why it's so important to use trusted installation sources and set up tools like Tripwire before exposing it to anything else than itself.
    • Tom Ritter
      Tom Ritter about 15 years
    • Mithin
      Mithin about 15 years
      You mean "cracked". Hacking is how we got linux in the first place.
    • hookenz
      hookenz over 10 years
      A friend of mine who's server was hosted with us once asked me to look at his server as there seemed to be something odd with it. As soon as I saw the logs I knew something was up. They tried to cover their tracks and I think had installed a root kit but made a bit of a mess of it. Anyway, to cut a long story short, we had to rebuild the whole server from scratch. It took all night and then we configured some security auditing tools.
    • Rodrigo
      Rodrigo over 6 years
      @Matt Mind telling us which tools? Still the same today?
    • hookenz
      hookenz over 6 years
      Rodrigo - i think it was snort. This is going back 15 years
    • Jenny D
      Jenny D over 6 years
  • Renkai
    Renkai over 14 years
    md5 has been weakened severely if not thrown out. You may want to move to sha512.
  • Oskar Duveborn
    Oskar Duveborn over 10 years
    The most advanced attack tools require zero skill these days and are easily available and some are extremely stealthy by default and by design. Botnets / zombies can be idle a long time before being used for harm but bugs in the attack tools can cause unwanted crashes and weird behaviour etc.
  • hub
    hub over 10 years
    Thats really usefull for my homeserver where i dont got the time to maintain the system like a productive work system. Anyhow, could I use this in a professional environment and be "relative" sure about the results? And for the answer to be 3 years old: Is this still a valid method to check for common infection nowadays in 2014?
  • hub
    hub over 10 years
    Is this a cloud service logging into your machine with root rights? What happens if the service is compromised?
  • Cheyne
    Cheyne over 10 years
    You don't need to give it root. You can also choose to use an agent instead, which means your machine polls out instead of SSH calling in. Passwords for things like DB are always stored on your machine, not in the cloud too.
  • fpersyn
    fpersyn about 9 years
    I think -ctime instead of -mtime. Cannot tamper with -ctime
  • GuidedHacking
    GuidedHacking over 3 years
    Just a FYI your link is broken, may want to replace it