Is there an easy way to log all commands executed, including command line arguments?
Solution 1
Yes, there is a kernel facility: the audit subsystem. The auditd
daemon does the logging, and the command auditctl
sets up the logging rules. You can log all calls to a specific system alls, with some filtering. If you want to log all commands executed and their arguments, log the execve
system call:
auditctl -a exit,always -S execve
To specifically trace the invocation of a specific program, add a filter on the program executable:
auditctl -a exit,always -S execve -F path=/usr/bin/rrdtool
The logs show up in /var/log/audit.log
, or wherever your distribution puts them. You need to be root to control the audit subsystem.
Once you're done investigating, use the same command line with -d
instead of -a
to delete a logging rule, or run auditctl -D
to delete all audit rules.
For debugging purposes, replacing the program by a wrapper script gives you more flexibility to log things like the environment, information about the parent process, etc.
Solution 2
You could use snoopy.
Snoopy is more lightweight solution as it does not need kernel cooperation. All that is needed is dynamic loader (dl) that preloads snoopy library, path to which is specified in /etc/ld.so.preload
.
Disclosure: I am current snoopy maintainer.
Solution 3
The Linux kernel "audit" subsystem can do what you need.
e.g. if you run these commands:
auditctl -a exit,always -F arch=b64 -S execve
auditctl -a exit,always -F arch=b32 -S execve
Then every execution event is logged, and a lot of information is provided around that
e.g. this is the output of me running tail /var/log/audit/audit.log
exit=0 a0=7f0e4a21e987 a1=7f0e4a21e6b0 a2=7f0e4a21e808 a3=8 items=2 ppid=906 pid=928 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="tail" exe="/usr/bin/tail" subj=kernel key=(null)
type=EXECVE msg=audit(1543671660.203:64): argc=2 a0="tail" a1="/var/log/audit/audit.log"
type=CWD msg=audit(1543671660.203:64): cwd="/home/sweh"
type=PATH msg=audit(1543671660.203:64): item=0 name="/usr/bin/tail" inode=266003 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1543671660.203:64): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=273793 dev=fd:03 mode=0100755 ouid=0 ogid=0 rdev=00:00 obj=unlabeled objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PROCTITLE msg=audit(1543671660.203:64): proctitle=7461696C002F7661722F6C6F672F61756469742F61756469742E6C6F67
There's some interesting values that can be seen; e.g. "auid" is 500, which is my login ID, even though "uid" is zero ('cos I'm running under su
). So even though the user may have switched accounts with su
or sudo
we can still track back to their "audit ID"
Now those auditctl
commands will be lost on a reboot. You can put them into a configuration file (eg in the /etc/audit/rules.d/
directory, on CentOS 7). The exact location will depend on your OS version. The auditctl
manual page should help here.
Beware, though... this will cause a lot of log messages to be generated. Make sure you have enough space on the disk!
If necessary the rules can be limited to a specific user, or a specific command.
And also beware; if a user puts the password in the command execution (e.g. mysql --user=username --password=passwd
) then this will be logged.
Related videos on Youtube
ede
Updated on September 18, 2022Comments
-
ede almost 2 years
I am trying to find how to log a specific instantiation of
rrdtool
to see whether the path it is receiving is incorrect.I know I could wrap the executable in a shell script that would log the parameters, but I was wondering if there was a more kernel-specific way to monitor for that, perhaps a filesystem callback that sees when a particular /proc/pid/exe matches a given binary?
-
Neil over 8 yearsIs there a way to get
auditd
to record the command-line arguments as well as the program ran? serverfault.com/questions/765179/…
-
-
Graeme over 10 yearsWhy the
-F path=/ust/bin/rrdtool
? I don't get howrrdtool
is even related software. -
Gilles 'SO- stop being evil' over 10 years@Graeme The problem described in the question was tracking an invocation of
rrdtool
. If you want to log invocations of all programs, drop the-F path=…
part (you'll get a lot of logs of course). -
Graeme over 10 yearsRight... first line of the question. Thanks.
-
r.v about 9 yearsIs it possible to do logging for commands directly or indirectly spawned from a particular shell only?
-
Bostjan Skufca Jese about 9 yearsI am not sure I understand your question - do you mean "shell" as specific program that is used as shell (bash, dash, zsh etc.), or you mean you would like to log just specific PTY? Snoopy provides filtering framework, but currently only a couple of very basic filters are implemented, see here for the list: link. If you have a concrete usecase that might be applicable to others, please do explain in feature request, and, oh, btw, patches are welcome :)
-
r.v about 9 yearsI meant just a specific PTY.
-
Bostjan Skufca Jese about 9 yearsThere is no specific filter for PTY available ATM. However, you could use snoopy to log everything, including which PTY the event occured on, and then do the filtering in your syslog daemon. I do not know which one you are using, but syslog-ng (for example) can do regex matching, positive or negative.
-
Ruslan about 5 yearsThis is all great, but how do I reset the config it to initial state? Otherwise it'll continue filling the log with new and new launched commands... or is this
auditctl
command only effective until reboot? -
Gilles 'SO- stop being evil' about 5 years@Ruslan The effect of
auditctl
only survives until reboot, but that's a good point anyway, I've added instructions on removing them without rebooting to my answer. -
BradTheBluefish almost 3 years@r.v: You can set
LD_PRELOAD
environment variable to use snoopy with certain applications only.