Personal Log: Digital Exorcism

When you leave the commercial/proprietary software ecosphere and jump into open-source operating systems, you will have to learn how to handle daemons. And once you've created a couple of those daemons yourself, taught them what to do and let them work in production, you gain a lot of experience and confidence in dealing with all kinds of daemons.

Yesterday, a couple of friends from the awesome http://co-munity.net/ecobytes project seemed to be in some sort of possible DDoS trouble and asked for my advice and experience to mitigate the issue. Now, to me, it is a very amazing experience to simply get root access to a lot of machines lately, operated by people which I have never physically met but in this case we are connected by elf-pavlik. And in today's world, voluntarily giving root access to someone else, is the ultimate token of trust or/and friendship. So I'd like to thank you guys for that vote of confidence.

Since it was supposed to be a DDoS, I've had my input filters clamped too early and saw that something was going on and a lot of traffic was moving but it somehow seemed wrong compared to other DDoS investigations I had to do in the past. After some failed attempts to block/null-route a couple of offensive networks, our analysis focus shifted to traffic distribution where we saw that one of the VMs seemed to be the top talker. And it also became clear that the traffic wasn't coming in, it was going out. I didn't take care to look at flow direction at all because I already assumed it was incoming traffic (DDoS).

Here's where Dashboards like this come in handy. You have all relevant metrics at a glance and can compare the current to some “normal” state in the past. Matching graphs and colors visually takes much less time than working on the console to aggregate everything manually for a quick situation overview.

It then quickly became apparent, that one of the VMs was the top talker so we moved onto that box and what started out as DDoS mitigation turned into digital exorcism. You know, when there are daemons that are possessed and controlled by some evil spirit to create some form havoc, mostly motivated purely by the ultimate overlord of all evil: Financial Profit. And what do you do when dealing with evil daemons? You go Exorcist on them.

After verifying that the traffic really was outgoing, it was time to find out what is causing this amount of traffic. In the old days, exorcism was a bit more of a good show I think, today, it's just a couple of people sharing the same tmux session, listening to their favorite kind of music and hacking away with a couple of tcpdumps, iftops, netstats and some other shell mumbo-jumbo. rkhunter didn't identify any rootkits. The daemon concealment seemed done like a crude quick-hack. If I'd have to hide something in a system, I'd definitely make it much harder for someone to track.

At the time two daemons of the kit were running: /.sshd and /http. sshd was located at /usr/sbin/.sshd and http at /etc/http. Both put some sort of pidfile into /tmp/$foo.lod, by which we were able to identify the running processes in the first place. It also seems to carry replacements for lsof,netstat,ps and ss, each one seems to be a unique binary. With the exception of bsd-port/jave, the rest of the identified files are basically just copies of the same with a different name and come with this file signature, which already stood out on a more or less up to date 64-bit machine.

ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), statically linked, for GNU/Linux 2.2.5, not stripped. 

After collecting all files we considered compromised, Virustotal removed the last shreds of doubt.

The main binary seems to contain a big list of IP addresses (probably C&C) and some sort of amplification and packet generation scheme. The observed attack pattern seemed more like a targeted HTTP SYN attack against only a few targets but with maximized bandwidth, trying to utilize the 1 GBit uplink capacity and spreading over 4-6 target hosts, each up to 200 MBit. Further analysis is still ongoing but if someone else would like to take a look at this thing pulled out of the wild, go ahead: