Web Administration Scripts

During the past month or so, I've also been dealing with an aggressive DDOS (that's a "distributed denial of service") attack on my server, one that's been a huge pain, as you might expect. What's odd is that with multiple domains on the same server, it's one of my less-popular sites that seems to have been the target of the attacks.

So, that's the jumping off point for this article's scripts: analyzing log files to understand what's going on and why.

To start, a handy check is to see how many processes are running, because my DDOS was characterized by a ridiculous number of comment and search scripts being triggered—hundreds a minute. How to check?

The ps command offers a list of running processes at any given time, but for many versions, all you see is the Web server "httpd" without any further details. The -C cmd flag narrows down output only to those processes, like this:


: ps -C httpd
  PID TTY          TIME CMD
20225 ?        00:13:21 httpd
28162 ?        00:00:01 httpd
...
5681 ?        00:00:00 httpd
5683 ?        00:00:00 httpd <defunct>
]]>

(Note the "defunct" process that's about to vanish.)

So one easy test is to see how many httpd processes are running:


$ ps -C httpd | wc -l
108

That seems like a lot, but this server is hosting several sites, including the super-busy AskDaveTaylor.com tech-support site, which sees more than 100k hits/day. So how does this vary over time? Hmm...still working on the command line:


$ while /bin/true
> do
>   ps -C httpd | wc -l
>   sleep 5
> done
108
107
103
99
94
91
87
84
91
121
120
116

So there's a max of 121 and a min of 87. But, what if I actually want to analyze this and get min, max and average over a longer period of time? Here's how I solve it:


#!/bin/sh
# Calculates the number of processes running that matches
#   a set pattern over time, producing min, max and average.
min=999; max=0; average=0; tally=0; sumtotal=0
pattern="httpd"  # ps -C pattern
while /bin/true
do
  count=$(ps -C $pattern | wc -l)
  tally=$(( $tally + 1 ))
  if [ $count -gt $max ] ; then
    max=$count
  fi
  if [ $count -lt $min ] ; then
    min=$count
  fi
  sumtotal=$(( $sumtotal + $count ))
  average=$(( $sumtotal / $tally ))
  echo "Current ps count=$count: min=$min, max=$max, tally=$tally 
  ↪and average=$average"
  sleep 5 # seconds
done
exit 0

Notice in the script that I'm not falling into the trap of calculating the average by having a running average and somehow factoring in the latest value as a diminishing additive, but instead I use a sumtotal variable that keeps having the latest processor count added. That divided by tally is always the average, although at some point this probably would be greater than MAXINT (2**32) and would start to produce bad results. On a modern computer, however, that should take a while. (And the quantum, the period of time between iterations, also can be adjusted. Five seconds might be too granular for a process that's going to be run for hours or even days.)

The following are the first few lines of output. Notice how the min and max vary as the different values are calculated:


sh processes.sh
Current ps count=132: min=132, max=132, tally=1 and average=132
Current ps count=128: min=128, max=132, tally=2 and average=130
Current ps count=124: min=124, max=132, tally=3 and average=128
Current ps count=123: min=123, max=132, tally=4 and average=126

If I let the script run for a longer period of time, the values become a bit more varied:


Current ps count=90: min=76, max=150, tally=70 and average=107

During the 15 minutes or so that I ran the script, an average of 107 "httpd" processes were running, with a minimum of 76 and a max of 150.

Armed with that information, another script could keep an eye on things via a cron job, like this:


#!/bin/sh
# DDOS - keep an eye on process count to 
# detect a blossoming DDOS attack
pattern="httpd"
max=200   # avoid false positives
admin="d1taylor@gmail.com"
count="$(ps -C $pattern | wc -l)"
if [ $count -gt $max ] ; then
  echo "Warning: DDOS in process? Current httpd count = 
  ↪$count" | sendmail $admin
fi
exit 0

That's a superficial solution, however, and it has two problems: 1) what I'd really like is to be able to identify the potential DDOS based on processor count and watch to see if it's sustained over the next few invocations of the script, and 2) once it's triggered, if it is a DDOS, in addition to everything else, I'll also start drowning in e-mail from this script saying essentially the same thing each time. Not good.

What the script needs is contextual memory so it can differentiate between a sudden spike in traffic and a persistent DDOS attack. In the former case, the script might trigger positive, then the next time it runs, it's all within acceptable limits again. In the latter case, once the attack starts, it'll probably just accelerate.

That's the opposite of the e-mail non-repeat condition though, because in the latter case, I want to know that the e-mail has been sent and not send it again within, say, a 60-minute window.

I'll dig in to both of those situations another time. For now, I need to get back to my server and keep bringing things back on-line, program by program, to try to avoid any problems. Stay tuned!

______________________

Dave Taylor has been hacking shell scripts for over thirty years. Really. He's the author of the popular "Wicked Cool Shell Scripts" and can be found on Twitter as @DaveTaylor and more generally at www.DaveTaylorOnline.com.

Comments

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.

12 Years of Windows and Now I'm Hooked on Linux........

M Tucker's picture

I've been playing around with scripting for about 3 months... I've been using Linux based router software for almost 10 years...

I was wondering if any of you advanced users could help me out. I trying to write a script for my routers.. (Gateway Asus using Tomato 1.83 and my router is linksys 54GL v 1.1 running DD-WRT)

I would appreciate any input on some security scripts I can put in my firewalls AND I really want to BLOCK ASIAN SUBNETS..... however; allow Japan to pass..
>> If any of you need any advice on Windows, I'm an expert and locking it down... Please let me know.

Any Ideas?

Tuc

In the suburbs and on country

sollen's picture

In the suburbs and on country roads, however, detectors are common. They may detect when a car arrives at an intersection, when too many cars 90W driving Light For Bulldozers are stacked up at an intersection (to control the length of the light), or when cars have entered a turn lane (in order to activate the arrow light).

Nothing is perfect

Anonymous's picture

Use netfilter/iptables to simply block anything that connects more than X in Y.

Use graphite to monitor the system and fire off an event based on those metrics, ie email, firewall rule, black hole route or how about just reverse attack...

Contact your upstream provider and get them the /dev/null your aggressor.

Another tact

Colin J McDermott's picture

I agree with the other posters, this is not the best way to handle these events. Something like fail2ban, etc would be better. Ie block a ddos attack from one source/ip, monitor and respond if it is from multiple sources. Close connections if they are open and inactive, etc.

A timely article I was reading this morning:
http://everythingsysadmin.com/2013/11/stop-monitoring-if-service-is-up.h...

So you would probably want the server to start responding to a DDOS attack with a scripted response plan.... I.e. a great hint for an article I would love to read.

considering zabbix

fred l.'s picture

if you have several sever to take care of, you should consider zabbix using personalized template to monitor apache and everything around...
however, i like your scripts

You need Opsview Core

Seth F.'s picture

Dave, what you need is opsview core and the nrpe agent on your web server. You can get graphs for various information and set up alerts. NRPE allows you to write your own nagios scripts as well.

Reply to comment | Linux Journal

where my water android's picture

Hi, I do believe this is a great website. I stumbledupon it ;) I will return yet
again since i have book marked it. Money and freedom is the
greatest way to change, may you be rich and continue to help other people.

Webinar
One Click, Universal Protection: Implementing Centralized Security Policies on Linux Systems

As Linux continues to play an ever increasing role in corporate data centers and institutions, ensuring the integrity and protection of these systems must be a priority. With 60% of the world's websites and an increasing share of organization's mission-critical workloads running on Linux, failing to stop malware and other advanced threats on Linux can increasingly impact an organization's reputation and bottom line.

Learn More

Sponsored by Bit9

Webinar
Linux Backup and Recovery Webinar

Most companies incorporate backup procedures for critical data, which can be restored quickly if a loss occurs. However, fewer companies are prepared for catastrophic system failures, in which they lose all data, the entire operating system, applications, settings, patches and more, reducing their system(s) to “bare metal.” After all, before data can be restored to a system, there must be a system to restore it to.

In this one hour webinar, learn how to enhance your existing backup strategies for better disaster recovery preparedness using Storix System Backup Administrator (SBAdmin), a highly flexible bare-metal recovery solution for UNIX and Linux systems.

Learn More

Sponsored by Storix