How to see all commands executed by the current user, in all sessions/screen/byobu terminals?

Question

If there is debian box, some users have access to the root account, some of those users open an ssh connection and start executing commands, some open screen or byobu or other similar tools to execute commands.

The command "history" doesn't appear to get full list of executed commands.

What is the best way to get all executed commands ?

Solution 1

This is kind of a hack, but at least it works. It requires you to have root on that server:

Looking at the output of ps aux I see a user's fork of sshd, like for example in this case the user mst:

$ ps aux | grep ssh
mst      19325  0.0  0.0  76268  1920 ?        S    21:20   0:00 sshd: mst@pts/6

So I check the file descriptors of this process like this:

$ sudo ls -lha /proc/19325/fd
total 0
dr-x------ 2 root root   0 Aug 30 21:26 .
dr-xr-xr-x 7 mst  users  0 Aug 30 21:25 ..
lrwx------ 1 root root  64 Aug 30 21:26 0 -> /dev/null
lrwx------ 1 root root  64 Aug 30 21:26 1 -> /dev/null
lrwx------ 1 root root  64 Aug 30 21:26 11 -> /dev/ptmx
lrwx------ 1 root root  64 Aug 30 21:26 12 -> /dev/ptmx
lrwx------ 1 root root  64 Aug 30 21:26 2 -> /dev/null
lrwx------ 1 root root  64 Aug 30 21:26 3 -> socket:[138972]
lrwx------ 1 root root  64 Aug 30 21:26 4 -> socket:[138198]
lrwx------ 1 root root  64 Aug 30 21:26 5 -> socket:[138200]
lrwx------ 1 root root  64 Aug 30 21:26 6 -> socket:[138207]
lr-x------ 1 root root  64 Aug 30 21:26 7 -> pipe:[138212]
l-wx------ 1 root root  64 Aug 30 21:26 8 -> pipe:[138212]
lrwx------ 1 root root  64 Aug 30 21:26 9 -> /dev/ptmx

Three of these links are pointing to /dev/ptmx, these are stdin, stdout and stderr. Since the user's shell is printing all the commands that he inputs, and also the output of these commands I watch his stdout by using strace and filtering for read system calls on the fd number 11 (because 11 is the second link to /dev/ptmx).

sudo strace -e read -s 256 -p 19325 2>&1 | grep 'read(11'

And I can see that the user types an ls command:

read(11, "l", 16384)                    = 1
read(11, "s", 16384)                    = 1

Ok, the output isn't pretty... but works

Solution 2

The best way is to use auditd. You can set it to log all commands issued by a certain user, or all commands issued by all users, or any invocation of a specific command, etc.

The man page for auditctl will give you some examples of rules you may want. In addition, if you don't trust the other root users, I'd advise you to log to a separate server that the others do not have access to.

I'd also suggest that it's better to not give out the root password and not allow ssh logins as root. Instead, allow only logins to the user's own accounts and let them use sudo su to become root. That will also make it possible for auditd to track which user did what.

Comments

[Marek Zakrzewski]

ps -efwauxx and then man ps

[slm]

@sharp12345 - val0x00f's comment will only show the actual live list of what's currently running. It won't show you an exhaustive list of everything that's ever run. Do you want a complete history of everything or just what's currently running?

[sharp12345]

@slm I want a complete list of all commands that have been executed, at least in the past few days.

[Drav Sloan]

take a look at the auditd, which logs all commands issued by users and a whole lot more. security.blogoverflow.com/2013/01/… and whmcr.com/2011/10/14/auditd-logging-all-commands

[Tim]

You can also peek at /root/.bash_history but that will depend on if bash is the shell or not and assuming you want to see roots history. If some of the other users with root access are actually using sudo, you will need to look at the users .bash_history.

[user2914606]

IIRC, bash history stays local (i.e. not in .bash_history) until the bash process terminates. I'm not sure, though.

[JonLord]

Can you explain how I can use auditd to capture the commands in real time? I can capture the commands editing /etc/pam.d/ files but the log have a big delay or it saves only when user logout.