What audit log files are created in linux to track a users activities?

Question

One of the machines on our networks was consuming a large amount of Internet bandwidth. However no user was using the system.

When I connected to the server using SSH and ran who it showed that a user is logged in from an IP that I didn't recognize. I didn't create this user on the system. And in /etc/sudoers the user had unlimited root access:

test11   ALL = NOPASSWD: ALL

I terminated the session and changed the root password. I also deleted the user.

Now I'm probably safe but need to analyze why this happened and also what activities were done by user.

How do I find out more information about what was done by this user? What system log files could help me get more info about IP, duration, commands run, internet sites accessed, etc?

Note: this is a default install of a CentOS 6 system. I have not installed any security or auditing software. Nor did I change any system configuration. I need to know what I can extract out of a default install.

Solution

As your system has been compromised, no information you get from that system can be trusted. Only logs that are immediately shipped off to an external system can be trusted (such as real time remote syslog). Meaning if you've got some nightly log rotation to an NFS share, you cannot trust it.

However it is possible the user did not bother covering his/her tracks, and you might have the information still available on the system.

Unfortunately on a default Centos/RHEL install, there is very minimal logging. You're basically restricted to poking around in /var/log. Which log you should dig through depends on what services are running on that box. However I would start with the ssh log. After that look at the logs of any services which run as root, or have sudo access.

If you're lucky, that test11 user might have a home directory, with a .bash_history file containing a history of what was done.

---

Also, as the system has been compromised to the point where an unknown user was able to gain root access, the system must be rebuilt from scratch. You cannot reuse anything from the system. Consider every single file as compromised. I would also recommend against using backups as you don't know how long ago the system was compromised.

Once a user gains root access, there are an unlimited number of backdoors that could be installed. If I were the one who had gained access to your system, simply removing that test11 user and changing the root password wouldn't even slow me down.

---

In the future, there are a few things you can do.

Remote logging

As mentioned, only real-time remote logging can be trusted not to be tampered with. Make sure you have this.

Auditing

There are 2 utilities you should install and use to monitor and audit the critical components of the system. These are auditd and ossec.

These 2 utilities operate differently, but are for the same goal, to watch for abnormal activity.

Terminal logging

There is another auditing tool called pam_tty_audit that works in conjunction with the earlier mentioned auditd utility. pam_tty_audit is a utility you add to your pam stack which logs all input & output across the TTY. Meaning if the user is accessing the box via interactive ssh, their activity would be logged.
Note however that it is of the utmost importance that this log be protected at all costs. This is mostly because of passwords. When the you type your password at a prompt, even though you don't see the password being typed, the pam_tty_audit module will see it, and log it. It is also possible you will cat out (or otherwise view) files containing sensitive information, which will also be logged. Thus this log must be either be immediately shipped off the local system so that it cannot be obtained by intruders, or it must be encrypted (and the decryption key must not be on the local system). Preferably both should be performed, ship off remotely, and encrypt it.