DDoS Virus infection (as a unix service) on a Debian 8 VM Webserver

Question

I maintain a (fully updated) Wordpress for a student team on a Virtual Machine on ~okeanos service for a couple of years. Today, the helpdesk informed me that i am conducting DDoS attacks, which - of course - I am not (this service has my academic credentials connected..). After they suspended the machine and I flamed their mailing system I tried to find out what has happend.

First of all, I run a ps -ej to checkout what is running:

root@snf-25181:~# ps -ej
1545 1545 1545 ? 00:00:00 console-kit-dae
1618 1057 1057 ? 00:00:00 gdm-session-wor
1632 1632 1632 ? 00:01:40 rghuoywvrf
1767 1767 1767 ? 00:00:00 sshd
1769 1769 1769 ? 00:00:00 systemd
1770 1769 1769 ? 00:00:00 (sd-pam)
1775 1767 1767 ? 00:00:00 sshd
1776 1776 1776 pts/0 00:00:00 bash
1849 1849 1776 pts/0 00:00:00 su
1870 1870 1776 pts/0 00:00:00 bash
2246 0 0 ? 00:00:00 kworker/0:0
2797 839 839 ? 00:00:00 apache2
3158 3158 3158 ? 00:00:00 bvxktwwnsb
3162 3162 3162 ? 00:00:00 bvxktwwnsb
3163 3163 3163 ? 00:00:00 bvxktwwnsb
3164 3164 3164 ? 00:00:00 bvxktwwnsb
3165 3165 1776 pts/0 00:00:00 ps

Notice the bvxktwwnsb and the rguoywvrf

Then I did a ps aux to get the services (again, a tail):

Debian-+  1629  0.0  0.0 178300  4444 ?        Sl   16:53   0:00 /usr/lib/dconf/dconf-service
root      1667  0.0  0.0  30744  4436 ?        Ss   16:53   0:00 /sbin/wpa_supplicant -u -s -O /run/wpa_supplicant
root      1670  0.0  0.1 299588  9884 ?        Ssl  16:53   0:00 /usr/lib/packagekit/packagekitd
root      1674  0.0  0.1 1055004 6168 ?        Ssl  16:53   0:00 /usr/sbin/console-kit-daemon --no-daemon
www-data  1923  0.0  0.1 240964  8112 ?        S    16:53   0:00 /usr/sbin/apache2 -k start
pankgeo+  5656  0.0  0.0  27416  3424 ?        Ss   17:03   0:00 /lib/systemd/systemd --user
pankgeo+  5657  0.0  0.0 143108  2408 ?        S    17:03   0:00 (sd-pam)   
root      5893  0.0  0.1 102420  6428 ?        Ss   17:04   0:00 sshd: pankgeorg [priv]
pankgeo+  5904  0.1  0.0 102560  4128 ?        S    17:04   0:02 sshd: pankgeorg@pts/0
pankgeo+  5905  0.2  0.1  16816  6388 pts/0    Ss+  17:04   0:04 -bash      
root      7443  0.0  0.1 102420  6496 ?        Ss   17:07   0:00 sshd: pankgeorg [priv]
pankgeo+  7448  0.0  0.0 102552  4160 ?        S    17:07   0:00 sshd: pankgeorg@pts/1
pankgeo+  7449  0.0  0.1  16468  6228 pts/1    Ss+  17:07   0:01 -bash      
root     17351  0.0  0.0      0     0 ?        S    17:15   0:00 [kworker/0:0]
root     18446  0.0  0.0      0     0 ?        S    17:18   0:00 [kworker/0:2]
root     18488  0.1  0.0      0     0 ?        S    17:18   0:01 [kworker/1:1]
root     22680  1.5  0.0      0     0 ?        S    17:28   0:08 [kworker/1:0]
root     24173  0.0  0.1 102420  6416 ?        Ss   17:31   0:00 sshd: pankgeorg [priv]
pankgeo+ 24181  0.3  0.0 102420  3360 ?        S    17:31   0:01 sshd: pankgeorg@pts/2
pankgeo+ 24182  0.0  0.0  16480  6112 pts/2    Ss   17:31   0:00 -bash      
root     25316  2.3  0.0      0     0 ?        S    17:33   0:06 [kworker/1:2]
root     26777  0.0  0.0      0     0 ?        S    17:35   0:00 [kworker/0:1]
root     26778  0.0  0.0      0     0 ?        S    17:35   0:00 [kworker/0:3]
root     27300  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 cat resolv.conf  #note                        
root     27306  0.0  0.0   1424  1036 ?        Ss   17:38   0:00 gnome-terminal   #from                     
root     27307  0.0  0.0   1424  1036 ?        Ss   17:38   0:00 ifconfig eth0    #here                    
root     27308  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 id               #(DDOS?)         
root     27309  0.0  0.0   1424  1040 ?        Ss   17:38   0:00 ifconfig                        
pankgeo+ 27315  0.0  0.0  11136  2044 pts/2    R+   17:38   0:00 ps aux     

Note the items[-4:-1]. Then I found online about chkconfig --list so I run that and this popped out:

root@snf-25181:/home/pankgeorg# chkconfig --list
acdnfhruvx 0:off 1:off 2:off 3:off 4:off 5:off 6:off
flyymwddwn 0:off 1:off 2:off 3:off 4:off 5:off 6:off

1 to 5 where on but I turned them off. Then I restarted and it has changed name. Then I located the acdnfhruvx and this popped out:

root@snf-25181:~# locate acdnfhruvx
/etc/init.d/acdnfhruvx
/etc/rc1.d/S01acdnfhruvx
/etc/rc2.d/S01acdnfhruvx
/etc/rc3.d/S01acdnfhruvx
/etc/rc4.d/S01acdnfhruvx
/etc/rc5.d/S01acdnfhruvx

The contents of one of them (they are all the same): root@snf-25181:~# cat /etc/init.d/acdnfhruvx #!/bin/sh

chkconfig: 12345 90 90
description: acdnfhruvx
BEGIN INIT INFO
Provides: acdnfhruvx
Required-Start:
Required-Stop:
Default-Start: 1 2 3 4 5
Default-Stop:
Short-Description: acdnfhruvx
END INIT INFO
case $1 in
start)
/bin/acdnfhruvx
;;
stop)
;;
*)
/bin/acdnfhruvx   
;;
esac    

This was found after a restart, so /bin/acdnfhruvx was nowhere. Later I found exes (ELF Formatted) at /usr/bin (I think I can share it if there is a brave man among you)

An extensive list of the commands I saw the machine executing without knowing origin (from successive ps -ejs and ps auxes:

root     27755  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 ifconfig                        
root     27759  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 who                        
root     27760  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 echo "find"                        
root     27761  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27762  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 id                        
root     27805  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 gnome-terminal                        
root     27809  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 ifconfig                        
root     27810  0.0  0.0   1424  1044 ?        Ss   17:40   0:00 sh                        
root     27811  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 sleep 1                        
root     27822  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 netstat -an                        
root     27826  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27829  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 bash                        
root     27833  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 cd /etc                        
root     27834  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 whoami                        
root     27822  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 netstat -an                        
root     27826  0.0  0.0   1424  1036 ?        Ss   17:40   0:00 top                        
root     27829  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 bash                        
root     27833  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 cd /etc                        
root     27834  0.0  0.0   1424  1040 ?        Ss   17:40   0:00 whoami                        

pkilling is pointless, since it always forks, removing files from /etc/init.d/ and /{usr/,}bin is also pointless since after restart there is a new (identical) version of the executable. After all this info, I have two questions: Can I find out HOW I was infected? Can I get rid of this? Thank you in advance!

Solution 1

We suffered a similar infection on Suse, probably through ssh brute force login.

The steps to clean are:

1. Check the file /etc/crontab. You probably have an entry to call the virus every 3 minutes

   */3 * * * * root /etc/cron.hourly/cron.sh
   

   Delete this line.

2. Identify the parent process of the virus. The rguoywvrf in your ps -ej. The other proceses are created and killed continously.
3. Stop it, dont kill it, with kill -STOP 1632
4. Check with another ps -ej that only the parent lives, the children should die quickly
5. Now you can delete the files in /usr/bin and /etc/init.d. There are variants of the virus that also uses /boot or /bin. Use ls -lt | head to look for files that have been modified recently.
6. Check the script in /etc/cron.hourly/cron.sh. In our server it was calling another copy of the virus on /lib/libgcc.so. Delete both files.
7. Now you can kill definitely the rguoywvrf process.

Solution 2

To answer your questions:

1. Without the necessary precautions (off site syslog, IDS, log monitoring, etc) you'll probably never find out what happened.
2. I would have to agree with Matt. You'll invest time to get a machine running which you will never really trust. In my opinion, the best solution is to move the data off site & redo the machine.

Of course, for what it is worth, this is only my opinion. Though, when redoing the machine, you can of course take the necessary precautions & protect yourself better in the future.

Solution 3

this a threat that generates allot of problems because launch a DDOS attack and generate thousands of connections to externals servers on port 80, but i do not if intentionally or not, it tends to overload your connection until the routers/firewalls freeze if there are not DDOS attack rules.

now , how can you remove this threat?

1. find your threat , use

Centos/redhat

ps -ely 

Debian

ps -ej

you will see:

3158 3158 3158 ? 00:00:00 bvxktwwnsb
3162 3162 3162 ? 00:00:00 bvxktwwnsb
3163 3163 3163 ? 00:00:00 bvxktwwnsb
3164 3164 3164 ? 00:00:00 bvxktwwnsb

the "bvxktwwnsb" is your target

2. then you need to boot your linux server on single user mode , make any changes in multiuser mode is pointless, usually you can switch with the following command:

   telinit S

3. after that you need to delete files run at startup

in Centos/Redhat the procedure is

Step a)

cd /etc/init.d          
ll -tr 

the last command order your files in reverse date, you are going to see a last 1 or 2 files at the end with named like

acdnfhruvx
kmrkuwbrng
gqpjiestmf
bvxktwwnsb

you need to see the content

cat /etc/init.d/gqpjiestmf

normally you will see the execution of a file located in /bin or /usr/sbin with same name

you need to delete both files.

Step b)

cd /etc/
ll -tr 

check if your crontab file is recently changed, look at its contents, search for a line

*/3 * * * * root /etc/cron.hourly/udev.sh

or

*/3 * * * * root /etc/cron.hourly/crontab.sh 

you need to edit the file and remove that line.

check the content of udev.sh or crontab.sh and you will see something like this

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
cp /lib/libgcc4.so /lib/libgcc4.4.so
/lib/libgcc4.4.so

you need to remove the "libgcc4.4.so" file or any other mentioned there (changing the permissions would also work, for instance chmod a-x libgcc.so )

reboot your server and everything should be ok.

For debian/ubuntu and relatives use:

locate bvxktwwnsb

and delete the files found in /etc and /bin

hope this help many people.

Comments

[Admin]

If your server has been compromised then it will be very difficult to tell how it was infected and what has been done, because it is trivial for the intruder to doctor/remove the log files. The best practice is to have offsite storage of log files at another location, so if your machine is compromised you'll at least have the logs leading up to the break-in. Ultimately I think you are going to need to reinstall - only way to ensure a clean uninfected system.

[bwDraco]

Your answer can be hard to read because it doesn't appear to be correctly formatted. If you need help, the help center has more information on properly formatting posts.

[mazgalici]

there are some bad scripts on /etc/rc6.d/ , they start with K90

[Mohamed Hafez]

do a find / -name "*rguoywvrf*" to find the other files, replacing rguoywvrf with whatever your file was named

[DarckBlezzer]

Check this link : garasiku.web.id/web/joomla/index.php/security/…