LWN: Comments on "Apache attacked by a "slow loris""

LWN: Comments on "Apache attacked by a "slow loris""https://lwn.net/Articles/338407/ This is a special feed containing comments posted to the individual LWN article titled "Apache attacked by a "slow loris"". en-usWed, 08 May 2024 10:18:56 +0000Wed, 08 May 2024 10:18:56 +0000https://www.rssboard.org/rss-specificationlwn@lwn.netnginx vs slowlorishttps://lwn.net/Articles/339989/ https://lwn.net/Articles/339989/gvyVitya, linux.kiev.ua is running nginx + apache-1.3 and it's rather depressed by slowloris in my tests. Could you please elaborate on how to cook things up so as to protect dynamic paths? I could only come up with proxying stuff for at least some fixed minimal timeout (which is not an option for lots of pages) and googling 'nginx slowloris' doesn't yield anything particularly useful to me. PS: <a rel="nofollow" href="http://nginx.org/">nginx</a> is really an excellent static httpd/reverse proxy, anyone running moderately busy site should consider looking into it. Could drop apache instance numbers an order of magnitude, together with RAM occupied by those.Sat, 04 Jul 2009 17:18:30 +0000Teergubbinghttps://lwn.net/Articles/339211/ https://lwn.net/Articles/339211/rickmoenI have no doubt that subscriber "thedevil" merely mistyped "<a href="http://en.wikipedia.org/wiki/Tarpit_(networking)">teergrubing</a>". Rick Moen rick@linuxmafia.comTue, 30 Jun 2009 01:33:48 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339052/ https://lwn.net/Articles/339052/dlang<div class="FormattedComment"> no I am not arguing that they need to only defend us against dialup users. I am saying that their mistakes have made us vulnerable to _even_ dialup users I argue that using the same variable for 1. how long after a connection do I wait for a request 2. how long can the flow of data pause 3. what is the max length of time that a CGI can run, even if it is passing data continuously is just bad design period, even if the attackers aren't taking advantage of it, these are three very different cases, and the appropriate values are very different between them. this affects the activity of a site even when it's not under attack. if someone is out to get you and is willing to burn/use a large botnet to do it, they are going to get you. I don't care who you are, thousands of bots collectivly have more bandwidth than you do, so they can take you down by just doing legitimate things on your site. the current situation is that trivial attacks can take apache down, at almost no cost to the attackers in terms of load. this means that they can do it on a botnet without doing anything that would make the owners of the machine notice that something is wrong. fixing the timeouts so that each of the three cases above are handled seperatly would force the attacker to use significantly more bandwidth for their attack, this would either raise the possibility that the owners of the machines would notice something was wrong (because the letgitmate use is slow due to the load), or it would force the attackers to use much larger botnets. either of these would make it more expensive for the attacker </div> Mon, 29 Jun 2009 09:36:44 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339048/ https://lwn.net/Articles/339048/njs<div class="FormattedComment"> > timeouts don't help in those cases, but allowing for a connection to tie up a slot for 300 seconds before it sends _any_ request is far worse. it doesn't take a broadband line to take down a server, dialup will do the job. Yeah, so... is your argument that the Apache folks should do a bunch of work to patch a million of these holes to protect us against evil modem users? A fix that only protects us against evil modem users doesn't seem worth the effort. > now in reality the issue isn't when you have one attacker IP (that shows up and is easy to block), it's when you have thousands of attackers, and in that case even a small amount of bandwidth and CPU from each of them can overwhelm a server, and at that point it can become very hard to tell the attackers from legitimate users unless the attacker is dumb enough to do something that stands out Right -- this seems like a more realistic threat model. And how will fiddling with Apache's timeout handling, like you advocate, protect anyone from it? </div> Mon, 29 Jun 2009 09:04:11 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339042/ https://lwn.net/Articles/339042/dlang<div class="FormattedComment"> this attackis the most problem when it takes up thousands of sockets., modern server hardware will happily run a few tens of thousands of apache threads. if you have large files to download (or heavy CGI's to run), the attacker can overwelm you by sending small requests that generate huge responses (or in the case of the CGIs, small requests that take significant amounts of CPU time) timeouts don't help in those cases, but allowing for a connection to tie up a slot for 300 seconds before it sends _any_ request is far worse. it doesn't take a broadband line to take down a server, dialup will do the job. you use the example of 500K/sec of bandwidth to eat up 100 connections, with the default 300 second timeout you need to send 2 64 byte packets every 300 seconds to tie it up, so 100 connections is 6400 bytes * 8 bits/byte / 300 seconds or 170 bps, a thousand connections pushes 2k bps now in reality the issue isn't when you have one attacker IP (that shows up and is easy to block), it's when you have thousands of attackers, and in that case even a small amount of bandwidth and CPU from each of them can overwhelm a server, and at that point it can become very hard to tell the attackers from legitimate users unless the attacker is dumb enough to do something that stands out </div> Mon, 29 Jun 2009 07:24:02 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339030/ https://lwn.net/Articles/339030/njs<div class="FormattedComment"> I don't see how fiddling with timeouts can fix anything. It just means the attacker has to apply the exact same attack slightly differently. For instance, suppose that you have, somewhere on your web-site, a 1 megabyte file. Suppose that you set up your timeouts so that a people have to download at at least 5 KB/s or get cut off (locking out some modem users, but oh well). And suppose that you allow a maximum of 100 Apache worker processes. If I download that file right at the minimum speed, then I can tie up one of those 100 worker processes for 204.8 seconds. To tie up all 100 of them indefinitely, I have to issue ~1 request every 2 seconds, and use 500 KB/s of bandwidth. That's trivially achievable from a residential connection. I won't lock other users *quite* as badly as the full slow loris attack -- other connections will get a chance to be serviced once every 2 seconds! -- but it's near as makes no difference in practice. So fiddling around with header timeouts or whatever might feel good, but isn't going to actually solve the problem, and thus is a waste of time. </div> Mon, 29 Jun 2009 01:31:57 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339029/ https://lwn.net/Articles/339029/njs<div class="FormattedComment"> > Oh, and Squid uses a single thread with non-blocking I/O (asynchronous), yet it is affected The original report said that Squid was affected, but the Squid maintainers can't reproduce it (<a href="http://www.squid-cache.org/bugs/show_bug.cgi?id=2694">http://www.squid-cache.org/bugs/show_bug.cgi?id=2694</a>); looks like a mistake in the original report to me. > and IIS uses a thread-per-connection (if I recall correctly) The article here claims that IIS does not use thread-per-connection, but rather some sort of asynchronous state-machine design (like lighttpd or squid) plus a thread pool to parallelize that state-machine. > Maybe lighttpd and IIS do something like give it 300 seconds to get all the headers, and after that, too bad, you're just cut off No -- they just handle the slow connection as normal. The difference is that for them, an idle connection costs a few bytes of memory describing that connection, and one can easily have thousands of these data structures sitting around without anyone noticing. For Apache, an idle connection ties up an entire server process, and for various reasons you can't have thousands of server processes sitting around. </div> Mon, 29 Jun 2009 01:20:45 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339021/ https://lwn.net/Articles/339021/dlang<div class="FormattedComment"> this is exactly the problem. they have one timeout variable that's used for many different things, and while some of the things need long timeouts, others don't, and could be set much shorter. hopefully this will force the apache team to tackle this issue and seperate the timeouts, but from the article it sounds like they are not responding well. they are right that the basic attack approach of having a botnet of servers connect to an apache server and tie it up is an old attack that has been possible forever. fixing the timeout issues will not address that, and even after fixing the timeouts the attackers can kill the apache server by making legitimate requests that take time to process, but fixing the timeouts will go a long way towards leveling the playing field again, right now it's tilted heavily in favor of the attackers. </div> Sun, 28 Jun 2009 22:05:57 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339020/ https://lwn.net/Articles/339020/dlang<div class="FormattedComment"> TCP ports don't get used up on servers when you have lots of connections. the limit is that you cannot duplicate the set source IP source port destination IP destination port in about a 2 min period when connecting to a server the destination port and destination IP are fixed, so a client can make lots of connections and make it so that no other connections could be made from that source IP, but that doesn't hurt anyone else. that also isn't the attack that's happening here. </div> Sun, 28 Jun 2009 21:59:43 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339015/ https://lwn.net/Articles/339015/pphaneuf<div class="FormattedComment"> Oh, and Squid uses a single thread with non-blocking I/O (asynchronous), yet it is affected, and IIS uses a thread-per-connection (if I recall correctly), just like Apache with the threaded MPM, but it is unaffected. I think the key difference is how timeouts are handled. If the 300 seconds timeout in Apache is reset at every header (or worse, at every packet received, which could be one character each!), then you could stretch something for a long time. Maybe lighttpd and IIS do something like give it 300 seconds to get all the headers, and after that, too bad, you're just cut off (freeing the TCP port for another connection). You could still mount some sort of DoS attack, but the attacker would have to keep it up more intensively, so that very few legitimate clients manage to slip by as well as Slowloris does for affected servers (which is eventually a 100% effectiveness, with next to zero difficulty for the attacker). </div> Sun, 28 Jun 2009 20:43:47 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339014/ https://lwn.net/Articles/339014/job<div class="FormattedComment"> Sort of. Not really. Any simple select-loop (or poll)-based web server would do. But as soon as you serve dynamic content in any way, be it via PHP or any other language, the problem is back again. Most (all?) web site languages are served by worker processes, either built into the web server or stand alone pre-forked. Disarming the attack only on static file URLs is not really a solution. The attack would probably choose apparent dynamic URLs such as .php if this was a real world attack anyway. </div> Sun, 28 Jun 2009 20:38:21 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/339005/ https://lwn.net/Articles/339005/pphaneuf<div class="FormattedComment"> The scarce resources here are TCP ports and memory. Maybe you can make things more memory efficient, or add memory, but the number of TCP ports is both fixed by TCP itself and inconveniently small. No matter how you implement it, there's a fixed cap on the number of TCP connections per IP address. You could add IP addresses to a server, but that would be a waste of another precious resource, since during normal usage, most web servers can't handle the maximum number of connections of a single IP. </div> Sun, 28 Jun 2009 20:23:36 +0000There are no such mpm and there will never be such an mpmhttps://lwn.net/Articles/339010/ https://lwn.net/Articles/339010/marcH<div class="FormattedComment"> > There's a word for people who deliberately conflate the conceptional model of a solution with a specific implementation of that solution: salesmen. But there is worse: those who do not! Software patent lawyers. </div> Sun, 28 Jun 2009 19:41:17 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338951/ https://lwn.net/Articles/338951/dmag<div class="FormattedComment"> > So you're talking about having Apache implement TCP in userspace? No, having a generic library for TCP in user space. > That makes no technical sense whatsoever. Please read the linked article. <a href="http://lwn.net/Articles/169961/">http://lwn.net/Articles/169961/</a> > The kernel implementation is thoroughly debugged, mature, patched regularly, Agreed. > and faster to boot. No. Right now, the driver gets the packet, later the kernel gets around to looking at it (maybe doing checksums, etc), and then much later userspace requests it. If each of these happen on a different CPU, you will waste thousands (ten thousands?) of instructions because of CPU caches and data locking issues. To become a better programmer, read this: <a href="http://lwn.net/Articles/250967/">http://lwn.net/Articles/250967/</a> It's really long, but pay attention to the parts where loops get 10x faster just by re-arranging data structures. > The real problem here is what Apache does after it reads data from a socket. Agreed. TCP sockets is icing on the cake once you've solved the current bottleneck. > Here's one uninformed idea: accept connections and read HTTP requests in one master process, asynchronously. But why does that process *have* to be Apache? Just put another web server in front of it (Nginx,Varnish, etc.). Apaches is really a "big and expensive" single-threaded application server (mod_php, mod_passenger, mod_perl, etc). In fact, Apache isn't especially good at serving static files either. I have to admit, Nginx has the best architecture, (but consider Varnish if caching is a big win). It's like when you go to the big warehouse stores, and before you get to the front of the line, some guy with a scanner has already scanned your cart, and all you do is pay at the register without waiting. </div> Sat, 27 Jun 2009 15:06:16 +0000Time them outhttps://lwn.net/Articles/338948/ https://lwn.net/Articles/338948/jengelh<div class="FormattedComment"> So, use <LimitExcept POST> to allow such slowposting for CMS expecting dialup users. </div> Sat, 27 Jun 2009 13:10:20 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338918/ https://lwn.net/Articles/338918/dlang<div class="FormattedComment"> this attack has been seen in the wild (I've been hit by it, or something so close to it that it doesn't matter) in my opinion, the fix here is for apache to change how it does timeouts. it currently has one timeout variable that is used for three things. 1. time from initial connection to when it gets the header 2. idle time while waiting for new data to arrive 3. total time that a CGI is allowed to run. there are _many_ sites where this timeout needs to be set fairly large to accommodate #3, but that the timeouts for #1 and #2 could be _much_ shorter. </div> Fri, 26 Jun 2009 20:42:52 +0000Varnish is the answerhttps://lwn.net/Articles/338858/ https://lwn.net/Articles/338858/dion<div class="FormattedComment"> Varnish only allocates a worker thread once the entire request has been received from the client, so simply slapping Varnish in front of the webserver under attack will defeat the attack. If all you want to do is to mitigate a Slow Loris attack then just move your web server to a different port and start Varnish with the default configuration on port 80. </div> Fri, 26 Jun 2009 13:04:03 +0000All modern OSes offer alternativehttps://lwn.net/Articles/338864/ https://lwn.net/Articles/338864/tcabot<div class="FormattedComment"> I can imagine different case requiring different solutions. In one case (I think the one you're thinking of) the bulk of what gets served is generated by the back-end servers and the "static" assets are smaller by comparison - site icons, css, js, etc. In that case you're right: a proxy is the way to go because you're concentrating control and configuration in one place. On the other hand, let's say that your site serves massive quantities of "interesting" image files (which I understand was the original use case for nginx). In that case the server needs to be extremely efficient because the working set is so large that a cache wouldn't do much good. Horses for courses. </div> Fri, 26 Jun 2009 12:21:09 +0000Teergrube = tarpithttps://lwn.net/Articles/338818/ https://lwn.net/Articles/338818/xoddam<div class="FormattedComment"> Thanks. When spelled correctly it's German for "tarpit", which I should have guessed anyway. </div> Fri, 26 Jun 2009 03:26:25 +0000Teergubbinghttps://lwn.net/Articles/338817/ https://lwn.net/Articles/338817/ofeeley<div class="FormattedComment"> Afrikaans? </div> Fri, 26 Jun 2009 02:32:32 +0000Teergubbinghttps://lwn.net/Articles/338813/ https://lwn.net/Articles/338813/njs<div class="FormattedComment"> I believe they meant "teergrubing" (= deploying a slow-loris-like attack for Good, i.e. against spammers connected to your mailhost). </div> Fri, 26 Jun 2009 02:17:56 +0000Teergubbinghttps://lwn.net/Articles/338805/ https://lwn.net/Articles/338805/xoddam<div class="FormattedComment"> Google replies: Your search - "Teergubbing - did not match any documents." Tomorrow google will find a document or two at lwn.net. But I'll probably be none the wiser :-) </div> Fri, 26 Jun 2009 00:06:22 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338687/ https://lwn.net/Articles/338687/iabervon<div class="FormattedComment"> HTTP is actually quite stateful: each connection is in the state of expecting responses to particular requests; it's really handy not having to invent TCP in order to figure out what response goes to what request, what's coming back when and where, what you're still waiting for, what you should give up on and start over requesting, etc. Once you have any state at all, it's much easier to use TCP than it is to use UDP and deal with the state at a higher level. Simply getting responses routed back to requesters through firewalls and NAT with UDP is a pain and requires a lot of protocol-specific analysis in a lot of devices. HTTP is "stateless" only in that you often return to the default state, not in that you never leave the default state. UDP is only really appropriate for cases where you don't care if your message is received and you won't get a response to it. </div> Thu, 25 Jun 2009 16:25:20 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338680/ https://lwn.net/Articles/338680/elanthis<div class="FormattedComment"> You can't lower the stack sizes. One of the biggest reasons that Apache administrators have to keep with older Apache MPMs is for crap like the mod_php, which is likely going to crash with lower stack sizes. Honestly, if at the end of the day you don't need any of the Apache modules (maybe you're using regular CGI or fast-CGI or something), then just don't use Apache, but instead use one of the newer, more efficient, more scalable Open web servers, like lighttpd, cherokee, etc. (Unfortunately, the lack of .htaccess support in those servers, and hence the lack of mod_rewrite-compatible user-configurable rewrites and redirects, is a real issue for a lot of hosts. If those servers added an optional module for basic .htaccess-like support, they'd probably skyrocket in adoption.) </div> Thu, 25 Jun 2009 15:35:52 +0000Time them outhttps://lwn.net/Articles/338627/ https://lwn.net/Articles/338627/MathFox<div class="FormattedComment"> If you are running a blog or CMS, some of your users need to upload (POST) stories/texts, images and other media. Depending on the website, that can be quite some data, a few megabytes. Timing out too early will make your editors unhappy (and still allows for enough of a window for slow loris.) I have good experiences with server side proxies (freeing Apache resources for all users on slow links), our problem was getting a 4 MB of HTML to users on slower links. Apache kept all its resources till the last byte was sent. </div> Thu, 25 Jun 2009 10:16:46 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338625/ https://lwn.net/Articles/338625/DarkmereThis "bug" is the same as the SMTP <a href="http://en.wikipedia.org/wiki/Tarpit_(networking)">tarpit</a> and others, but instead of working on the server side, it works on the client side, against the server. So, it works on the same basis that so many other attacks do, find a limited resource on the server, find a way to make the server hit that limit, lean back and reap profits. Thu, 25 Jun 2009 09:52:53 +0000This is fundamental problemhttps://lwn.net/Articles/338624/ https://lwn.net/Articles/338624/dgm<div class="FormattedComment"> > The only thing nginx really needs is decent documentation... But not passionate advocates, it seems. </div> Thu, 25 Jun 2009 09:41:42 +0000Time them outhttps://lwn.net/Articles/338614/ https://lwn.net/Articles/338614/man_lsI'm probably stating the obvious, but why not cut each client after a total time of, say, 20 seconds? Genuine clients should not take more than that in making a request. Such a global timeout would only damage extremely slow network links, which might (arguably) be better off cutting the connection short. Quite often I've seen my trusty Firefox waiting minutes for a site which, unsurprisingly, does not come out after all. Combined with something like what you say (20 connections per IP) it would severely limit the damage of this attack. Each individual slow loris would only be able to tie up 20 threads for 20 seconds. So you would need a fairly extensive network to take a site up.Thu, 25 Jun 2009 06:56:14 +0000There are no such mpm and there will never be such an mpmhttps://lwn.net/Articles/338607/ https://lwn.net/Articles/338607/quotemstr<blockquote>...you need to split web-server in two: lighweight frontend (with state machine, fancy kernel interface and everything) and backend (with php, mysql connections and so on).</blockquote>Agreed. <blockquote>The fact that frontend is called nginx and not "apache enhanced engine accelerator" does not change anything.</blockquote>There are many different ways of realizing the frontend-backend split, and nginx is just one of them. You can use FastCGI server processes; you can run a conventional proxy as a reverse proxy; you can run a specialized reverse proxy that's not called nginx; you can use akamai; or you can do a thousand other things. Plenty of people manage to split their front and back-ends without using nginx. There's a word for people who deliberately conflate the conceptional model of a solution with a specific implementation of that solution: salesmen. Sure, nginx is pretty neat, but it's not the only way to implement what we agree is the necessary architecture.Thu, 25 Jun 2009 05:51:29 +0000All modern OSes offer alternativehttps://lwn.net/Articles/338606/ https://lwn.net/Articles/338606/quotemstr<blockquote>Because you need solutions, not a buzzwords?</blockquote>This coming from somebody who's hawking a specific product as the solution to a whole class of problems? I don't think I'm the one who has to worry about buzzwords here. <blockquote>You still need to configure server.</blockquote>Reading the <a href="http://nginx.net/">nginx</a> webpage, it appears you can configure nginx as a caching reverse proxy. That's fine. My issue is that you pretend it's the only game in town when really, any caching reverse proxy will do. (And feature sets may differ; Varnish, for example, appears to have a more sophisticated load balancer.) Also, I can't fathom why you would want your web accelerator serving content on its own. A caching reverse proxy setup is the only one that makes sense to me: that way, you have one place to configure what's served: the back-end servers. Because the back-end servers already mark what's static and what's not (via cache-control HTTP headers), you shouldn't have to do anything special to push static content to the front-end server, and the reverse proxy asking the back-end servers once in a while for some static content won't make a difference in the overall load.Thu, 25 Jun 2009 05:43:19 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338596/ https://lwn.net/Articles/338596/quotemstr(Reading this over, it's a bit of a ramble. Sorry about that.) Virtual memory isn't the problem per se. Apache isn't running out of virtual memory. The attack is against Apache's own limit on the number of simultaneous outstanding requests. Returning to the supermarket analogy, the problem isn't running out of space in the store for cashiers, but simply tying up all the existing ones. Now, you can increase these limits in order to defeat the attack. There's the problem: Apache's resource consumption can be too high to service the number of simultaneous connections required. For <a href="http://httpd.apache.org/docs/2.0/mod/prefork.html">prefork</a> servers, the problem is especially severe: each simultaneous request requires its own process. Now, any serious operating system can handle thousands of processes; that's not the problem. You can even have thousands of Apache processes, since Apache workers are actually pretty light, memory-wise, and all the code and some of the data structures are shared among all workers. More an issue is the <code>mod_*</code> idiocy that embeds an interpreter in each worker process*. Then, when you have thousands of processes, you run into a problem far more severe than virtual address space exhaustion: swap thrashing, poor performance, machine lockups, and OOM killing sprees. Using threads, in principle, helps the problem. Each thread shares the same process, meaning the additional memory occupied by each new simultaneous connection is just the memory needed to keep track of that connection; if all the threads share a single copy of the interpreter and other data structures, I bet you can avoid the slowloris problem entirely by simply setting connection limits high enough. A small variant of this approach is to use multiplexed I/O, as in lighttpd; that's like using the threaded approach above, but you don't need a thread stack for each simultaneous connection. In practice, if you make the thread stack small enough, you can still fit thousands in a 32-bit virtual address space. The problem with threads is that you need threads, though! The free world's most inexplicably-popular web scripting language, PHP, doesn't have a thread-safe interpreter. That limitation us to another solution: divorce the heavyweight state from the HTTP handler, and use something like FastCGI to communicate to the interpreter. That way, you can have as many (cheap) HTTP connection handlers (threads, processes, or state machines) as you'd like, and still limit the number of heavyweight interpreters you need. (Using a separate processes for the heavy lifting is better than mixing all the threads together anyway; see below.) To deal with slowloris attacks, just batch all the headers up in a request before sending the whole request on to the heavyweight process that actually does something meaningful with it. (That's what mod_fastcgi does.) That way, a slowloris-using attacker can pound away at the lightweight worker thread (or process), and only when a complete request is read will the system actually commit to using a significant resource, one of the heavyweight FastCGI servers. lighttpd can't use embedded interpreters; it's state machine precludes that. Instead, people generally set it up to use FastCGI: guess why lighttpd is not vulnerable to slowloris attacks. If you configure Apache appropriately, you can make it work similarly. (Before you say, "but wait! FastCGI is pretty slow!" -- that's simply not true. The HTTP user-agent's communication to the server is orders of magnitude slower than the local communication between Apache and a FastCGI server. The FastCGI communication doesn't add any meaningful time to each request, so it doesn't increase latency. And since each application is sequestered into its own process instead of running in each worker process in Apache, total memory use can actually be lower than the <code>mod_*</code> model. As if that weren't a good enough reason to use FastCGI (or something similar, like scgi), it's also a lot easier to keep track of your web application's system resource usage as distinct from the web server's. You can actually measure the CPU consumption of, say, squirrelmail without conflating it with Apache's.) Oh, and there's an even better reason: nothing forces a FastCGI server to run as the same user as the web server. Finally, you can put an end to anything remotely related to the web having to be owned by <code>apache</code>. Each application can have its own user, and be limited to its own little corner of the machine just like any other kind of network-facing daemon. Really, <code>mod_*</code> is just an ugly hack that's completely unnecessary today.)Thu, 25 Jun 2009 05:29:07 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338603/ https://lwn.net/Articles/338603/thedevil<div class="FormattedComment"> Teergubbing in reverse? </div> Thu, 25 Jun 2009 05:15:41 +0000This is fundamental problemhttps://lwn.net/Articles/338601/ https://lwn.net/Articles/338601/khim<blockquote>The event MPM apparently isn't vulnerable to Slowloris (just like lighthttpd, IIS etc.). However, it's apparently incompatible with mod_ssl and some other input filters, so it's not a solution for us.</blockquote> Single thread can not simultaneously handle many thousands connections and do heavy-duty processing (PHP, SQL database requests, etc). First one requires response in nanoseconds, second one sometimes take seconds. And when you split these two operations in two threads frontend-backend scheme becomes quite natural. You can use nginx to handle enormous loads with a single decent server and then safely use as many backend systems as needed. It IS possible to create faster and less resource-hungry server then nginx+apache combo, but if you need to alter existing installation... there are no contest. The only thing nginx really needs is decent documentation...Thu, 25 Jun 2009 05:09:02 +0000Or somethinghttps://lwn.net/Articles/338598/ https://lwn.net/Articles/338598/khim<blockquote>So the easy solution, I suppose, is just to use Lighttpd or something like that as a reverse proxy for your Apache server.</blockquote>If you look on the latest <a href="http://news.netcraft.com/archives/2009/06/17/june_2009_web_server_sur ve y.html">survey</a> you'll find out that millions are already running "something like that". Nginx was designed from ground up to work in such situation - if you know your apache process usually generates page 100K in size you can specify this as buffer size to nginx and then your "real" server will be freed in milliseconds and when occasional long page will be generated nginx will wait for backend. Lighthttpd uses similar architecture, but it's less configurable when used as http-accelerator. And of course when you send static pages it makes perfect sense to use sendfile(2) and forget about everything (nginx does more or less that - just a few small structures to handle "keep alive" connections). That's why I can not see what's so important happened: this is well-known apache problem but while it can not be solved with apache alone it can be solved with additional software - and was solved for years by real admins on millions of systems.Thu, 25 Jun 2009 04:53:27 +0000All modern OSes offer alternativehttps://lwn.net/Articles/338597/ https://lwn.net/Articles/338597/khim<blockquote>Why do you need an entirely new web server?</blockquote>Because you need solutions, not a buzzwords? I've explained why you need two servers <a href="http://lwn.net/Articles/338594/">below</a>. Without "real" web server you can serve static pages (icons, images, etc) via sendfile(2) - and this is important for real-world servers.<blockquote>Couldn't you do the same thing with a caching reverse proxy like Varnish?</blockquote>You can name your frontend server "web server", "web accelerator" or use any other term, but if your frontend is "heavily threaded, with each client connection being handled by a separate worker thread" then you just added complexity without any benefits. What'll happen with your frontend if you'll have 50'000 clients with opened connections? Nginx can handle such load on medium-sized system.<blockquote>That way, you only need to configure one set of servers.</blockquote>You still need to configure server. Nginx was designed from the ground up to do two things and do them well: 1. Serve static pages. 2. Work as http-accelerator. Thu, 25 Jun 2009 04:45:13 +0000There are no such mpm and there will never be such an mpmhttps://lwn.net/Articles/338594/ https://lwn.net/Articles/338594/khim<blockquote>does someone know of an mpm, which implements IIS's behavior?</blockquote> This is huge design question: do you want extesibility in your web server or not? Suppose someone went and implemeted such an mpm. Then you server got state machine and everything. Now - what'll happen if single thread is hadnling 1000 clients in your server and this thread called php interpreter? 1000 angry clients, that what (think about it). To make such scheme usable you need to split web-server in two: lighweight frontend (with state machine, fancy kernel interface and everything) and backend (with php, mysql connections and so on). And guess what: such scheme is implemented and as last survey <a href="http://news.netcraft.com/archives/2009/06/17/june_2009_web_server_sur vey.html">shows</a> it is used by millions. The fact that frontend is called nginx and not "apache enhanced engine accelerator" does not change anything.Thu, 25 Jun 2009 04:33:09 +0000All modern OSes offer alternativehttps://lwn.net/Articles/338595/ https://lwn.net/Articles/338595/quotemstrWhy do you need an entirely new web server? Couldn't you do the same thing with a caching reverse proxy like <a href="http://en.wikipedia.org/wiki/Varnish_cache">Varnish</a>? That way, you only need to configure one set of servers.Thu, 25 Jun 2009 04:31:03 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338592/ https://lwn.net/Articles/338592/quotemstrSo you're talking about having Apache implement TCP in userspace? That makes no technical sense whatsoever. The kernel implementation is thoroughly debugged, mature, patched regularly, and faster to boot. Apache will have to maintain just as much state as it does today, and moving TCP to userspace solves nothing. A "socket" is just a handle to a tiny bit of state information describing the connection, and of course it's the right abstraction. It's what the protocol is specified to use, and in-order, streamed delivery is the perfect medium for HTTP anyway. The real problem here is what Apache does after it reads data from a socket. Recall that both lighttpd and IIS use sockets (just like every other network daemon on Earth), and they are not vulnerable to this attack. The counter to this attack is simple, really, and it's conceptually the same as a counter to a SYN flood: only commit your resources when the remote party has committed his own. The problem here is how to shoehorn that idea into Apache's model, which commits resources (in this case, processes) very early. Here's one uninformed idea: accept connections and read HTTP requests in one master process, asynchronously. Only when a complete request has been read send the file descriptor of the connection to a worker; the actual handoff can be achieved using a <code>SCM_RIGHTS</code> over a unix domain socket.Thu, 25 Jun 2009 04:29:18 +0000All modern OSes offer alternativehttps://lwn.net/Articles/338593/ https://lwn.net/Articles/338593/khimHave you seen <a href="http://news.netcraft.com/archives/2009/06/17/june_2009_web_server_sur ve y.html">surveys</a>? Do you know WHY nginx is growing so fast? Do you even know WHAT the nginx is? It's caching web-server. It can serve static web- pages and protect "real" server (often Apache server) from slowloris attack. And it DOES NOT use "a heavy-weight polling syscall" - all moder operating systems offer alternative... P.S. The real motivation was not to fight slowloris attack - it was to reduce server load when it talks with thousands of dial-up clients. Think about it: if you have huge number of very slow clients the dynamic is the same! Server processes or threads are tied for minutes when they serve "feature-rich" pages to clients who only consume 1Kb per second. Apache was unable to work with it (nginx author tried to fix it for years) so new web-server was born. And as statistic shows real admins who are in charge of real sites know all about this problem. Tempest in a teapot...Thu, 25 Jun 2009 04:23:38 +0000Apache attacked by a "slow loris"https://lwn.net/Articles/338591/ https://lwn.net/Articles/338591/marineam<div class="FormattedComment"> In my time maintaining a software download mirror I essentially had to deal with this on a regular basis thanks to the oodles of freeware "Download Accelerators" out there which make the assumption that downloading one file in hundreds of pieces from a site surely must be faster than downloading it in one piece. Just get hit by a few people with slow modems using these one of these things to download all the Debian isos and poof, no one is happy. There is an Apache module called mod_limitipconn floating around that I used to limit each individual ip address to about 20 concurrent connections which seemed to to be a reasonable compromise number between killing the damn broken download accelerators and allowing legitimate proxies for my traffic. </div> Thu, 25 Jun 2009 03:53:30 +0000