When you leave the commercial/proprietary software ecosphere and jump into open-source operating systems, you will have to learn how to handle daemons. And once you've created a couple of those daemons yourself, taught them what to do and let them work in production, you gain a lot of experience and confidence in dealing with all kinds of daemons.

Yesterday, a couple of friends from the awesome http://co-munity.net/ecobytes project seemed to be in some sort of possible DDoS trouble and asked for my advice and experience to mitigate the issue. Now, to me, it is a very amazing experience to simply get root access to a lot of machines lately, operated by people which I have never physically met but in this case we are connected by elf-pavlik. And in today's world, voluntarily giving root access to someone else, is the ultimate token of trust or/and friendship. So I'd like to thank you guys for that vote of confidence.

Since it was supposed to be a DDoS, I've had my input filters clamped too early and saw that something was going on and a lot of traffic was moving but it somehow seemed wrong compared to other DDoS investigations I had to do in the past. After some failed attempts to block/null-route a couple of offensive networks, our analysis focus shifted to traffic distribution where we saw that one of the VMs seemed to be the top talker. And it also became clear that the traffic wasn't coming in, it was going out. I didn't take care to look at flow direction at all because I already assumed it was incoming traffic (DDoS).

Here's where Dashboards like this come in handy. You have all relevant metrics at a glance and can compare the current to some “normal” state in the past. Matching graphs and colors visually takes much less time than working on the console to aggregate everything manually for a quick situation overview.

It then quickly became apparent, that one of the VMs was the top talker so we moved onto that box and what started out as DDoS mitigation turned into digital exorcism. You know, when there are daemons that are possessed and controlled by some evil spirit to create some form havoc, mostly motivated purely by the ultimate overlord of all evil: Financial Profit. And what do you do when dealing with evil daemons? You go Exorcist on them.

→ Read more...