Is there an easy way to log all commands executed, including command line arguments?

24,942

Solution 1

Yes, there is a kernel facility: the audit subsystem. The auditd daemon does the logging, and the command auditctl sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve system call:

auditctl -a exit,always -S execve

To specifically trace the invocation of a specific program, add a filter on the program executable:

auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool

The logs show up in /var/log/audit.log, or wherever your distribution puts them. You need to be root to control the audit subsystem.

Once you're done investigating, use the same command line with -d instead of -a to delete a logging rule, or run auditctl -D to delete all audit rules.

For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.

Solution 2

You could use snoopy.

Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload.

Disclosure: I am current snoopy maintainer.

Solution 3

The Linux kernel "audit" subsystem can do what you need.

e.g. if you run these commands:

auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve

Then every execution event is logged, and a lot of information is provided around that

e.g. this is the output of me running tail /var/log/audit/audit.log

exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64):  cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67

There's some interesting values that can be seen; e.g. "auid" is 500, which is my login ID, even though "uid" is zero ('cos I'm running under su). So even though the user may have switched accounts with su or sudo we can still track back to their "audit ID"

Now those auditctl commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/ directory, on CentOS 7). The exact location will depend on your OS version. The auditctl manual page should help here.

Beware, though... this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!

If necessary the rules can be limited to a specific user, or a specific command.

And also beware; if a user puts the password in the command execution (e.g. mysql --user=username --password=passwd) then this will be logged.

Share:
24,942

Related videos on Youtube

ede
Author by

ede

Updated on September 18, 2022

Comments

  • ede
    ede almost 2 years

    I am trying to find how to log a specific instantiation of rrdtool to see whether the path it is receiving is incorrect.

    I know I could wrap the executable in a shell script that would log the parameters, but I was wondering if there was a more kernel-specific way to monitor for that, perhaps a filesystem callback that sees when a particular /proc/pid/exe matches a given binary?

  • Graeme
    Graeme over 10 years
    Why the -F path=/ust/bin/rrdtool? I don't get how rrdtool is even related software.
  • Gilles 'SO- stop being evil'
    Gilles 'SO- stop being evil' over 10 years
    @Graeme The problem described in the question was tracking an invocation of rrdtool. If you want to log invocations of all programs, drop the -F path=… part (you'll get a lot of logs of course).
  • Graeme
    Graeme over 10 years
    Right... first line of the question. Thanks.
  • r.v
    r.v about 9 years
    Is it possible to do logging for commands directly or indirectly spawned from a particular shell only?
  • Bostjan Skufca Jese
    Bostjan Skufca Jese about 9 years
    I am not sure I understand your question - do you mean "shell" as specific program that is used as shell (bash, dash, zsh etc.), or you mean you would like to log just specific PTY? Snoopy provides filtering framework, but currently only a couple of very basic filters are implemented, see here for the list: link. If you have a concrete usecase that might be applicable to others, please do explain in feature request, and, oh, btw, patches are welcome :)
  • r.v
    r.v about 9 years
    I meant just a specific PTY.
  • Bostjan Skufca Jese
    Bostjan Skufca Jese about 9 years
    There is no specific filter for PTY available ATM. However, you could use snoopy to log everything, including which PTY the event occured on, and then do the filtering in your syslog daemon. I do not know which one you are using, but syslog-ng (for example) can do regex matching, positive or negative.
  • Ruslan
    Ruslan about 5 years
    This is all great, but how do I reset the config it to initial state? Otherwise it'll continue filling the log with new and new launched commands... or is this auditctl command only effective until reboot?
  • Gilles 'SO- stop being evil'
    Gilles 'SO- stop being evil' about 5 years
    @Ruslan The effect of auditctl only survives until reboot, but that's a good point anyway, I've added instructions on removing them without rebooting to my answer.
  • BradTheBluefish
    BradTheBluefish almost 3 years
    @r.v: You can set LD_PRELOAD environment variable to use snoopy with certain applications only.