Help! My server has been hacked - .IptabLes and .IptabLex in /boot

Question

I'm running a Ubuntu 6.06 dapper server and it has been hacked. I'll admit up front here that I'm a programmer and not a system administrator so even though I've worked with Unix/Linux for years my skills as a sysadmin are extremely weak.

I host a website that is running apache 2 and tomcat to serve up a Java based Struts web application. The performance of the site was getting really bad and after poking around trying to understand what the problem might be I discovered that a temp directory in my home directory was getting filled with files called getsetup.hb.*

This was new so I dug around a little more and found by running ps -ef that there were a couple of processes running that I wasn't aware of and looked a little strange to me, /boot/.IptabLes and /boot/.IptabLex

I searched the web for a clue as to what these files were and tried running netstat to see what foreign hosts were connected. Both .IptabLes and .IptabLex were connected to IP addresses that belonged to China Telecom.

This was a shock to me because I had the iptables firewall software configured so it would only allow connections to the web server on port 80 and ssh on port 22 was limited to a static IP I use at home so I can access the server remotely.

I killed the processes, removed the files from the /boot directory and also found copies in the root and in /usr which I removed. I also changed the root password.

Logging in again today I see the files are all back again and the connection to the China Telecom IP's have been re-established.

I have no idea how to proceed here. Please forgive my ignorance but I'm hoping someone can direct me as to how to proceed from here to get this resolved. Any suggestions are welcome.

Thanks in advance.

Mike

Solution

As Thomas said on comment 6.06 is WAY past end of life. You need to hire a Linux guy ASAP and help you out with the upgrade process.

In the meantime, Don't Panic! relax and read the following popular topics:

How do you know your server has been compromised?

How do I know if my Linux server has been hacked?

Hopefully you will find the best way to deal with your compromised server reading the following Canonical Question:

How do I deal with a compromised server?

Good Luck!

Comments

[Thomas Ward]

6.06 is WAY past end of life. We can't help you with it, because of the policies of what is on topic on this site. I suggest you nuke your boot data, and get any critical data off after checking to make sure it isn't going to infect your systems again, then upgrade to a supported release that still gets security support.

[web.learner]

Power down the PC, remove the hard drive. Get a new hard drive install a newer version of Ubuntu. Mount your old hard drive in another computer and take out whatever you need (it would be better to just ditch it and use backups, but I suppose you don't have those).

[Wilf]

Take the server OFFLINE, then try and sort it out.

[Eliah Kagan]

Since the system has been compromised, "upgrading" should of course, as Seth says, not involve doing anything with the existing system. Take it down as wilf says, use a live CD to get anything off that has to be recovered (this should not include any part of the operating system, this needs to be reinstalled from scratch--and the more you can restore from backups instead, the better), wipe everything, and install a version of some operating system (Ubuntu is a good choice, but this principle applies to anything) that is currently supported with security updates. 6.06 hasn't been for years.

[Rinzwind]

only 1 valid option: R E I N S T A L L. And make -damn- sure when you restore your backup to 1st investigate the contents.

[Braiam]

"I'm running a Ubuntu 6.06...my server has been hacked" of course, you are running a 8 years old system. There are TONS of updates and patches in between.

[Eliah Kagan]

@Braiam Indeed. At best (assuming only approved "server related" packages), it may have gone just ~2.5 years with no security patches, since Dapper Server reached end-of-life in June 1, 2011. But as you say, even in those 2+ years, there have still been plenty of updates. Plus, it's possible that using a system years past its end-of-life date isn't the only problematic security practice in play here. user237315: I'm sorry this has happened. Assuming you want to move to a supported Ubuntu release, feel free to post questions for help rebuilding on it.

[ufk]

thanks for this post. my gentoo server has been hacked too.

[user237315]

Thanks for your suggestions, I'll take the server offline and install the latest version of Ubuntu. The web application can be rebuilt from source so besides the data in the database there is nothing else I'll need to back up.

[Wilf]

Good Luck! ;-) It hopefully be easy, you can download the new version from here.

[Wilf]

There is an image here if you are refferring to the Hitchiker's Guide To The Galaxy's Don't Panic :D

[Achu]

No, I took it from here :D

[user237315]

Thanks Achu - your compassion under the circumstances is appreciated, I'll do my best!