Weird SSH, Server security, I might have been hacked

Question

I am not sure if I've been hacked or not.

I tried to log in through SSH and it wouldn't accept my password. Root login is disabled so I went to rescue and turned root login on and was able to log in as root. As root, I tried to change the password of the affected account with the same password with which I had tried to log in before, passwd replied with "password unchanged". I then changed the password to something else and was able to log in, then changed the password back to the original password and I was again able to log in.

I checked auth.log for password changes but didn't find anything useful.

I also scanned for viruses and rootkits and the server returned this:

ClamAV:

"/bin/busybox Unix.Trojan.Mirai-5607459-1 FOUND"

RKHunter:

"/usr/bin/lwp-request Warning: The command '/usr/bin/lwp-request' has been replaced by a script: /usr/bin/lwp-request: a /usr/bin/perl -w script, ASCII text executable

Warning: Suspicious file types found in /dev:"

It should be noted that my server isn't widely known. I have also changed the SSH port and enabled 2-step verification.

I am worred I got hacked and someone is trying to fool me, "everything is fine don't worry about it".

Solution 1

Like J Rock, I think this is a false positive. I had the same experience.

I received an alarm from 6 different, disparate, geographically separated servers in a short time span. 4 of these servers only existed on a private network. The one thing they had in common was a recent daily.cld update.

So, after checking for some of the typical heuristics of this trojan without success, I booted a vagrant box with my known clean baseline and ran freshclam. This grabbed

"daily.cld is up to date (version: 22950, sigs: 1465879, f-level: 63, builder: neo)"

A subsequent clamav /bin/busybox returned the same "/bin/busybox Unix.Trojan.Mirai-5607459-1 FOUND" alert on the original servers.

Finally, for good measure, I also did a vagrant box from Ubuntu's official box and also got the same "/bin/busybox Unix.Trojan.Mirai-5607459-1 FOUND" (Note, I had to up the memory on this vagrant box from its default 512MB or clamscan failed with 'killed')

Full output from fresh Ubuntu 14.04.5 vagrant box.

root@vagrant-ubuntu-trusty-64:~# freshclam
ClamAV update process started at Fri Jan 27 03:28:30 2017
main.cvd is up to date (version: 57, sigs: 4218790, f-level: 60, builder: amishhammer)
daily.cvd is up to date (version: 22950, sigs: 1465879, f-level: 63, builder: neo)
bytecode.cvd is up to date (version: 290, sigs: 55, f-level: 63, builder: neo)
root@vagrant-ubuntu-trusty-64:~# clamscan /bin/busybox
/bin/busybox: Unix.Trojan.Mirai-5607459-1 FOUND

----------- SCAN SUMMARY -----------
Known viruses: 5679215
Engine version: 0.99.2
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 1.84 MB
Data read: 1.83 MB (ratio 1.01:1)
Time: 7.556 sec (0 m 7 s)
root@vagrant-ubuntu-trusty-64:~#

So, I also believe this is likely to be a false positive.

I will say, rkhunter did not give me the: "/usr/bin/lwp-request Warning" reference, so maybe PhysiOS Quantum is having more than one issue.

EDIT: just noticed that I never explicitly said that all of these servers are Ubuntu 14.04. Other versions may vary?

Solution 2

The ClamAV signature for Unix.Trojan.Mirai-5607459-1 is definitely too broad, so it's likely a false positive, as noted by J Rock and cayleaf.

For example, any file that has all of the following properties will match the signature:

• it's an ELF file;
• it contains the string "watchdog" exactly twice;
• it contains the string "/proc/self" at least once;
• it contains the string "busybox" at least once.

(The whole signature is a bit more complicated, but the above conditions are sufficient for a match.)

For example, you can create such a file with:

$ echo 'main() {printf("watchdog watchdog /proc/self busybox");}' > innocent.c
$ gcc -o innocent innocent.c
$ clamscan --no-summary innocent
innocent: Unix.Trojan.Mirai-5607459-1 FOUND

Any busybox build (on Linux) will usually match the four properties I listed above. It's obviously an ELF file and it will definitely contain the string "busybox" many times. It executes "/proc/self/exe" to run certain applets. Finally, "watchdog" occurs twice: once as an applet name and once inside the string "/var/run/watchdog.pid".

Solution 3

This just showed up today for me as well in my ClamAV scan for /bin/busybox. I'm wondering if the updated database has an error.

Solution 4

I tried to log in through SSH and it wouldn't accept my password. Root login is disabled so I went to rescue and turned root login on and was able to log in as root. As root, I tried to change the password of the affected account with the same password with which I had tried to log in before, passwd replied with "password unchanged". I then changed the password to something else and was able to log in, then changed the password back to the original password and I was again able to log in.

This sounds like expired password. Setting the password (successfully) by root resets the password expiration clock. You could check /var/log/secure (or whatever is the Ubuntu equivalent) and find out why your password was rejected.

View more solutions

Comments

[Josh Morel]

Agree with Michael. Seems like Mirai uses brute-force password guessing to compromise linux hosts - incapsula.com/blog/malware-analysis-mirai-ddos-botnet.html. Using public key authentication would be better than changing the SSH port for security purposes IMHO.

[Rui F Ribeiro]

You might have your var log partition full, and you cant change the root password when in rescue because first you have to mount the partitions properly as read-write

[marcelm]

@JoshMorel I would go further and say that changing the SSH port is detrimental to security. It doesn't help protect anything, but people who do it wrongly feel more secure. So, by feeling more secure without actually being more secure they're worse off than before. Also, I'd say pubkey auth isn't simply better, but a must.

[marcelm]

"... it wouldn't accept my password ... it replied "password unchanged" ... after changing the password to something else I was able to login, I changed the password back to what it was and I was still able to login." - All that could be explained by you making typos in your password (or having caps lock on) before you went to the rescue user.

[JDS]

the busybox trojan detection by clamav happened to me, too, this morning for the first time ever, across ~100 systems; I'm voting false positive. I'd guess clamav updated their sig database to have this false positive start showing up overnight last night

[JDS]

Incidentally, the sha256 hashsum of my busybox on these systems is 7fa3a176871de12832ca8a78b646bc6be92f7f528ee81d1c35bf12aa9929‌​2b1c . These are ubuntu 14.04 systems, and the mtime on the busybox bin is 2013-11-14

[Xiong Chiamiov]

Have you read through /usr/bin/lwp-request? Does it look suspicious?

[Mark]

@marcelm, if you're paying attention to your logs, it's helpful: it gets rid of all the automated scanners, so you know that any log entries are from a serious attacker.

[J Rock]

Scan /bin/busybox on any Ubuntu 14.04 LTS with the latest ClamAV database. It returns infected. This is a false positive, IMO.

[J Rock]

I submitted a false positive report to ClamAV. I also found that vmware player binaries show up as infected with the same trojan. It's likely they have included busybox code.

[Délisson Junio]

Where can I read that signature, and others from ClamAV, out of curiosity?

[PhysiOS]

I am going to change my SSH authentication for a pubkey and I will try to monitor the network connections, but honestly it's really wierd because I even copy and pasted the password and it still rejected it. What should I do with the /usr/bin/lwp-request?

[agregoire]

I also got this notification this morning on a Ubuntu 14.04 server. I compared (sha1sum) my server's /bin/busybox file to the same file on a local VM created from an Ubuntu image and they're identical. So I vote false positive too.

[cayleaf]

I knew someone smarter than me would be able to explain why it was a false positive. Thanks!

[Admin]

@PhysiOSQuantum Nothing. That's also a false positive -- lwp-request is a tool related to a Perl module (metacpan.org/pod/LWP), so it's perfectly normal for it to be a script.