How To Defend slowloris DDoS With mod_qos (Apache2 On Debian [Lenny])

From the Slowloris description:

Slowloris is designed so that a single machine (probably a Linux/UNIX machine since Windows appears to limit how many sockets you can have open at any given time) can easily tie up a typical web server or proxy server by locking up all of it's threads as they patiently wait for more data. Some servers may have a smaller tolerance for timeouts than others, but Slowloris can compensate for that by customizing the timeouts. There is an added function to help you get started with finding the right sized timeouts as well. As a side note, Slowloris does not consume a lot of resources so modern operating systems don't have a need to start shutting down sockets when they come under attack, which actually in turn makes Slowloris better than a typical flooder in certain circumstances. Think of Slowloris as the HTTP equivalent of a SYN flood.

I recently had to defend a live attack with slowloris-dos from a botnet. The load-impact is very low but http quits serving very fast. A quick approach was to mangle with timeout settings, wich is fine to defend a single attacker but leads into new issues (ie. large NAT on client-side).

mod_qos gives some fine-grained opportunities to scale the number of used connections and to defend an attack according to bandwidth limits. Unfortunately it is only available as source-package and there are many possible settings, wich might be hard to setup for this special case. So I provide the way that helped me.

1. Get the source, build & install

mod_qos is available from sourceforge (http://sourceforge.net/projects/mod-qos/). You will find documentation here http://mod-qos.sourceforge.net/.

cd /tmp/
 wget http://downloads.sourceforge.net/sourceforge/mod-qos/mod_qos-8.13-src.tar.gz?use_mirror=freefr
 tar xvfz mod_qos-8.13-src.tar.gz

You might want to copy & paste the direct link from sourceforge. As we want to compile mod_qos by use of apxs, we need to install the appropriate dev package and gcc of course, ie:

apt-get install apache2-threaded-dev gcc

Now build & install

cd mod_qos-8.13/apache2/
 apxs2 -i -c mod_qos.c

If everything worked fine you'll get something like this:

----------------------------------------------------------------------
 Libraries have been installed in:
 /usr/lib/apache2/modules
 If you ever happen to want to link against installed libraries
 in a given directory, LIBDIR, you must either use libtool, and
 specify the full pathname of the library, or use the `-LLIBDIR'
 flag during linking and do at least one of the following:
 - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
 during execution
 - add LIBDIR to the `LD_RUN_PATH' environment variable
 during linking
 - use the `-Wl,--rpath -Wl,LIBDIR' linker flag
 - have your system administrator add LIBDIR to `/etc/ld.so.conf'
 See any operating system documentation about shared libraries for
 more information, such as the ld(1) and ld.so(8) manual pages.
 ----------------------------------------------------------------------
 chmod 644 /usr/lib/apache2/modules/mod_qos.so

2. Now configure and activate

Go to /etc/apache2/mods-available and add a qos.load and qos.conf file

cd /etc/apache2/mods-available/
 vi qos.load

LoadModule qos_module /usr/lib/apache2/modules/mod_qos.so

vi qos.conf

## QoS Settings
<IfModule mod_qos.c>
    # handles connections from up to 100000 different IPs
    QS_ClientEntries 100000
    # will allow only 50 connections per IP
    QS_SrvMaxConnPerIP 50
    # maximum number of active TCP connections is limited to 256
    MaxClients              256 
    # disables keep-alive when 70% of the TCP connections are occupied:
    QS_SrvMaxConnClose      180
    # minimum request/response speed (deny slow clients blocking the server, ie. slowloris keeping connections open without requesting anything):
    QS_SrvMinDataRate       150 1200
    # and limit request header and body (carefull, that limits uploads and post requests too):
    # LimitRequestFields      30
    # QS_LimitRequestBody     102400
</IfModule>

Now you need to enable the module and restart apache:

a2enmod qos
 /etc/init.d/apache2 restart

If you are able to get the server-status by https://__yourserver__/server-status, you'll find mod_qos enabled and working by giving a statistical summary, like this:

mod_qos 8.13

3. before & after

On top of the server-status page you can see a summary of all apache workers. To see what happens before and whilst the attack, I used a local environment. So the results here show the basic principle not the real world example.

Let's assume a normal server-load like this:

1 requests currently being processed, 5 idle workers
_.._W._...._...................................................
................................................................

While attacking this server with one single machine results to all workers being occupied:

102 requests currently being processed, 0 idle workers
RRRRRRRRRRRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR
RRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRR..........................

As you can see the attack took all available threads of the apache server wich itself stays in state "R" (Keepalive (read)). A normal apache will wait for 300 seconds to close such a connection and as the attacker doesn't send any real request and just keeps all open connections, No other requests will be served anymore.

Now with the QoS-Module enabled, things look different. Although apache starts the maximum number auf child-workers, it is still able to serve "real-people-requests":

1 requests currently being processed, 97 idle workers
________W_______________________________________________________
__________________________________..............................

We configured apache with mod_qos to handle a limited number of connections per IP and to prefer serving "fast" requests. The attack just opens a connection without sending a real http-reqeuest (like GET /index.html HTTP 1.1). So apache now classifies that as a "slow" request as it has to wait very long to get a request/response from the attacker.

A real request sends the "GET" much faster together with additional http-headers. So in case that all workers are occupied by an attack whilst receiving a real request, apache just drops a stale connection and handles the new "serious" request at almost normal speed.

On a real world server you will get pictures like these (reload... reload... reload):

166 requests currently being processed, 0 idle workers
RRRRRCCCCRRRCRCRWRRRCRRCRCCRRCRRCCCRRRKCRRRRRRRRCRRRRRCRCCRC.RRC
C.RCR.RRC.CC.RCRR.RRRCR.C.C.RR.R.RRCCR.R.R...RRRRRRCCRRR.RRKC.CR
.RRR.C..RRRC.CCRRRR.RRC..R.CCRR.CR.RR.R.R.C.RR.C...C.C.RCRR.C.R.
..RRCRC.RCRCC..R.CCCW.............................R.............

97 requests currently being processed, 59 idle workers
___R_C_RR.CC_R_R_RRC_RCR._.R_RR.RRW___RR__RC______R_R_R__.K_._R.
RCRC___.CKRCRRRRRCR__CRR_RRRKR_RR.___.._._...RC..RR.CR.R.CRRR.R.
.R__._..RRRC.__R__R.R_R...._R_R.__.RR..._._..R.R...R.R.RR.R.R._.
.._RR_C.R.R_R.._.RRRR...........................................

51 requests currently being processed, 92 idle workers
___KR_R_K.._._.K__R__._..K.__._.R.W__R___R____.____K_____.__._R.
R_CKRK_.__RRRR_KK__RR.___RKR__R.R.__C..R._...__..R_.__._.K__R._.
.___._.._R__.C._RKR.__C....R__C.R_.__..._.C..K._...K._.._._._...
..___RK.R.___.._.____...........................................

126 requests currently being processed, 0 idle workers
RRKRRRRCRCCRRKRKCRRRKCRRRRRRCKCKRKRRRRRRKCRCCCRRKKKRRRRRC.....R.
.C.R.CR.RRC.RRRRRCCRR.RCCCRRRKR.K.R....R.R....R.....C..R.R.RR.R.
.........C.C....K.K.RRK....R..C.K..KC...C.C....R...C.R..W.C.R...
..KRR.R.R.RRR....RCRR...........................................

4. Links, etc.

I provide this howto as one possible way to solve this particular issue. There might be more ways (and hopefully will be), like using a proxy, loadbalancer, level-7-firewall, etc.

In my case, we had to find a quick solution that is up and running in minutes. For further reading you might find these links helpful:

• http://www.golem.de/0906/67854.html
• http://ha.ckers.org/slowloris/
• http://www.zdnet.de/news/wirtschaft_sicherheit_security_perl_skript__slowloris__legt_apache_websites_lahm_story-39001024-41005602-1.htm
• http://isc.sans.org/diary.html?storyid=6613
• http://mod-qos.sourceforge.net/
• http://ha.ckers.org/blog/20090617/slowloris-http-dos/