Any way to determine how files were deleted on Linux/Apache server?

Question

PROBLEM:

I have several missing directories on an offsite Linux/Apache server. I've tracked down about when they went missing from the httpd error logs, and I have a list of SSL logins to the machine (complete with network addresses.)

There are several individuals who have access to my account on the machine, so I am trying to confirm whether the files were removed accidentally or intentionally. If I can determine that the session came from an external network address, then I can be sure it was intentional. Otherwise I will need additional information.

QUESTION:

Is there any way to determine what user account (mine vs. root), network session, or process removed the files? I'm not worried about restoring them as backups are recent.

RELATED QUESTION:

I've tried looking at the bash history for my own account (the only one other than root with permission) and it only seems to extend back to the beginning of my SSL session. Does this imply the bash history was tampered with, or is this normal behavior?

Solution 1

By default, this isn't logged (imagine how big the logs would get). You can add logging with auditctl or inotify/iwatch (interesting options, check them out), or your control panel logging or ftp logs (usually all in /var/log somewhere) or bash history might help you track it down a little. Check your HTTP access logs in case some GET request managed to do this (unlikely, but possible)

If your history isn't there, it has been cleared. This is either done manually, or a command in your .bash_logout or .logout files.

Also, it's SSH not SSL :)

Solution 2

Cleanup is always a hard. Resources you can use:

To determine who was logged in, and from where:

• last / lastlog
• /var/log/secure (shows successful and failed logins most of the time)

To determine what was going on at the time:

• ausearch (assuming you are auditing; if you aren't, you might consider starting)
• sar (to figure out if the system was idle or busy at the time)

If there hasn't been much churn since the incident, the mtime on the directory might have useful info (probably not, but I generally like to collect that kind of data either way)

For the future, get every single person with access to the machine an account of their very own. What you have now is very silly for the reason you have just discovered; You can't know who did what.

Solution 3

If you are running process accounting, you may be able to find the call(s) to rm in the processs accounting log.

Shell history may have been removed, or may be configured for session private history. Concurrent sessions can also cause problems with shell history.

Solution 4

The bash history only gets written when you close the shell (you can deactivate that for the current session). In a running bash session, you can't see the commands you typed in that session in the bash history file - close the session, start a new one and you should be able to see it.

View more solutions

Comments

[James L]

It doesn't operate over SSL, it uses The Secure Shell Transport Layer Protocol: tools.ietf.org/html/rfc4253

[James L]

The history command will show .bash_history plus any history from the current session, but you're right in that it won't appear in .bash_history

[Rob]

! ... wow, I was 99% percent positive SSH operated over SSL; I have no idea where such a strong misconception came from. Thanks for the heads up.

[Rob]

Unfortunately, my boss was who decided to give out my account info to the rest of the IT team... until yesterday, I wasn't even aware. I will have to discuss this with him.