Apache attacked by a "slow loris"

Posted Jun 24, 2009 16:17 UTC (Wed) by NAR (subscriber, #1313) [Link]

Posted Jun 24, 2009 16:20 UTC (Wed) by mcmanus (guest, #4569) [Link]

Posted Jun 24, 2009 16:31 UTC (Wed) by smurf (subscriber, #17840) [Link]

Posted Jun 25, 2009 5:29 UTC (Thu) by quotemstr (subscriber, #45331) [Link]

(Reading this over, it's a bit of a ramble. Sorry about that.) Virtual memory isn't the problem per se. Apache isn't running out of virtual memory. The attack is against Apache's own limit on the number of simultaneous outstanding requests. Returning to the supermarket analogy, the problem isn't running out of space in the store for cashiers, but simply tying up all the existing ones.

Now, you can increase these limits in order to defeat the attack. There's the problem: Apache's resource consumption can be too high to service the number of simultaneous connections required. For prefork servers, the problem is especially severe: each simultaneous request requires its own process. Now, any serious operating system can handle thousands of processes; that's not the problem. You can even have thousands of Apache processes, since Apache workers are actually pretty light, memory-wise, and all the code and some of the data structures are shared among all workers.

More an issue is the mod_* idiocy that embeds an interpreter in each worker process*. Then, when you have thousands of processes, you run into a problem far more severe than virtual address space exhaustion: swap thrashing, poor performance, machine lockups, and OOM killing sprees.

Using threads, in principle, helps the problem. Each thread shares the same process, meaning the additional memory occupied by each new simultaneous connection is just the memory needed to keep track of that connection; if all the threads share a single copy of the interpreter and other data structures, I bet you can avoid the slowloris problem entirely by simply setting connection limits high enough.

A small variant of this approach is to use multiplexed I/O, as in lighttpd; that's like using the threaded approach above, but you don't need a thread stack for each simultaneous connection. In practice, if you make the thread stack small enough, you can still fit thousands in a 32-bit virtual address space.

The problem with threads is that you need threads, though! The free world's most inexplicably-popular web scripting language, PHP, doesn't have a thread-safe interpreter. That limitation us to another solution: divorce the heavyweight state from the HTTP handler, and use something like FastCGI to communicate to the interpreter. That way, you can have as many (cheap) HTTP connection handlers (threads, processes, or state machines) as you'd like, and still limit the number of heavyweight interpreters you need. (Using a separate processes for the heavy lifting is better than mixing all the threads together anyway; see below.)

To deal with slowloris attacks, just batch all the headers up in a request before sending the whole request on to the heavyweight process that actually does something meaningful with it. (That's what mod_fastcgi does.) That way, a slowloris-using attacker can pound away at the lightweight worker thread (or process), and only when a complete request is read will the system actually commit to using a significant resource, one of the heavyweight FastCGI servers.

lighttpd can't use embedded interpreters; it's state machine precludes that. Instead, people generally set it up to use FastCGI: guess why lighttpd is not vulnerable to slowloris attacks. If you configure Apache appropriately, you can make it work similarly.

(Before you say, "but wait! FastCGI is pretty slow!" -- that's simply not true. The HTTP user-agent's communication to the server is orders of magnitude slower than the local communication between Apache and a FastCGI server. The FastCGI communication doesn't add any meaningful time to each request, so it doesn't increase latency. And since each application is sequestered into its own process instead of running in each worker process in Apache, total memory use can actually be lower than the mod_* model.

As if that weren't a good enough reason to use FastCGI (or something similar, like scgi), it's also a lot easier to keep track of your web application's system resource usage as distinct from the web server's. You can actually measure the CPU consumption of, say, squirrelmail without conflating it with Apache's.)

Oh, and there's an even better reason: nothing forces a FastCGI server to run as the same user as the web server. Finally, you can put an end to anything remotely related to the web having to be owned by apache. Each application can have its own user, and be limited to its own little corner of the machine just like any other kind of network-facing daemon. Really, mod_* is just an ugly hack that's completely unnecessary today.)