Apache Power

PHOTOS © GARY WAGNER

While there would still be a World Wide Web without the Apache Web server, pundits have suggested that it would belong to Microsoft. Since drawing up the plan for the Apache project in 1993, Apache Software Foundation President Brian Behlendorf has helped lead the volunteer development team that proved that you can take on Microsoft and win — just so long as you change the rules.

Lately, the 26-year-old Behlendorf has been focusing on his new company, Collab.net, which aims to provide an integrated development environment for open source developers the world over. Behlendorf met with Linux Magazine editors Adam Goodman and Robert McMillan to discuss Collab.net, open source, the Internet, and his early, crazy days at Wired.

Linux Magazine: Is it true you once owned the McDonalds.com domain?

Brian Behlendorf: In the early days of Wired, one of their authors, Josh Quittner, wrote an article about domain names. How easy it is to get a domain name. Because at that point, no one knew about the Internet except for systems administrators. It was nowhere near the kind of consumer thing that it is today.

So Josh and I sat there and did a whole bunch of whois lookups on the Fortune 500 companies. Most of them were registered, except for McDonald’s. So I sent in the form and pointed it at a hobbyist site of mine and set up the e-mail address ronald@mcdonalds.com to go to me and to Josh Quittner.

In the article Josh describes the phone conversation he had with McDonald’s corporate headquarters trying to find out who there would be interested in registering the domain. He made an honest attempt to give them a chance to register the domain and they couldn’t do it. Their PR people didn’t understand, and their marketing people didn’t understand. What was really funny was that their technical people had registered mcd.com. Eventually Josh gave it back to McDonald’s in exchange for McDonald’s funding a T-1 in perpetuity to a high school in the Bronx.

LM: How did your work at Wired Magazine lead to the Web?

BB: I went to Wired as basically their first Webmaster. At the time they had an e-mail robot that would send back text versions of articles if you wanted them. So my first job was to make that work.

I went from being a $7.00-an-hour contractor at Wired to joining them more full time. In early 1994 we decided: “Hey, there might be some legs to this. I wonder if we could do something a little content-related on this. Maybe we could put some extra stories that didn’t make it to the magazine in here, and maybe create some content just for the Web site.”

So we set up what I think was the first password-based mass Web site, where you logged in, created a password, then got an account that it would tailor content to. That was HotWired. I was the chief engineer behind that, and some of the things that we were doing involved modifications to the server, for example to look up a user from a DBM file database. Really simple stuff.

LM: In some ways it seems like Wired was the nexus of all this stuff that followed after that. They didn’t want to be just a magazine.

BB: For a good year-and-a-half, maybe two years, they were the only magazine talking about things that we talk about every day here. Things like cryptography, like online communities, like really wild and crazy high-tech gadgets. Things like Internet technologies and domain names.

LM: Do you read Wired now?

BB: I still read it from time to time. It’s certainly a much different magazine. It’s more of a business-focused magazine than it used to be. It used to be kind of like Omni meets The Economist.

LM: Who thought of the name Apache?

BB: I had some friends at a company called Enterprise Integration Technology, and somebody there asked me, “What would be your ideal Web server?” So I wrote about a bunch of stuff that I thought was missing from NCSA’s server — some stuff that still isn’t in a lot of Web servers like revision control and stuff like that. I put it on a page and said: “I should come up with a name for this.” The name literally came out of the blue. I wish I could say that it was something fantastic, but it was out of the blue. I put it on a page and then a few months later when this project started, I pointed people to this page and said: “Hey, what do you think of that idea?”

Someone said they liked the name and that it was a really good pun. And I was like, “A pun? What do you mean?” He said, “Well, we’re building a server out of a bunch of software patches, right? So it’s a patchy Web server.” I went, “Oh, all right.”

LM: That never occurred to you when you thought of the name?

BB: When I thought of the name, no. It just sort of connoted: “Take no prisoners. Be kind of aggressive and kick some ass.”

LM: One of the interesting things about Apache is that it doesn’t really have a benevolent dictator like Linux. What is the most comparable piece of open source software to Apache in terms of its development model?

BB: FreeBSD is pretty close in terms of the development being distributed at the top. I think FreeBSD should be given more credit than they get. They started before us. In many ways we copied a lot of what they did.

LM: Self-consciously?

BB: In many ways. As we were getting together, there were people from the FreeBSD community such as Paul Richards who said: “This is how we do it here and this is perhaps how we should do it over here.”

LM: Apache is basically the standard for servers on the Internet. How does the Apache project avoid the kind of infighting that affected the HTTP standard when Netscape and Microsoft became involved? If representatives from either one of those companies had had the right to veto the development of those standards — as core Apache members do — nothing would have ever happened.

BB: I think it’s important that we’re not inventing standards. We’re creating reference implementations of standards, but we’ve always attached the development of Apache to externally defined standards such as the IETF [Internet Engineering Task Force] definition of what HTTP is.

It helps that a lot of the people who work on Apache also work on the standards. Roy Fielding is one of the authors of the HTTP standard. There is a need for that close connection between the group defining the standards and the group implementing them. But we’ve never tried to present ourselves as a standards body.

LM: I know that you had some apprehensions when IBM first approached you about doing work with Apache. Now some time has gone by; IBM is on the Apache team. What if Microsoft were to approach you and say: “We want to participate”?

BB: Earlier on we had this discussion on the Apache list. The question was, “What if Microsoft came up and started contributing patches? Would we give someone from Microsoft access to the CVS [Concurrent Versioning System] commit tree?” We all felt that yes, we would because in order to get to the level where you can start committing code, you have to essentially prove the value of getting there.

We all hated high school popularity contests, right? We all hated the kind of arbitrary nature of who’s in, who’s out, who’s cool, who’s not. I don’t think anyone wanted to emulate that in this kind of system. Even IBM, before they came in, they had to prove that they understood how this process worked by contributing and by defending those patches. I think we’d all feel comfortable if someone from Microsoft actually started posting patches. If they did it consistently and if the quality of the patches were high and if they actually played by the rules that we laid out, sure. Why not?

LM: So has Microsoft provided any patches?

BB: No. They’ve have had somebody on the list almost since the beginning. I’ve talked to people at Microsoft and I think they’ve probably not contributed because they haven’t seen the value of doing it.

LM: Do you see Microsoft’s Internet Information Server [IIS] as the main competitor for Apache?

BB: People run IIS because they built their entire back end on ASP [Microsoft's Active Server Pages]. People rarely run IIS because they were previously running Apache and decided it wasn’t fast enough. If there’s competition I’d say it’s been other HTTP servers out there. Things like kHTTPd, which is the kernel module for Linux, which is a really high-performance Web server that’s directly linked into the kernel.

You’ll always get performance benefits by building something that is very specific to a particular environment or application. Speed was never the most important thing in Apache. It was protocol compliance first, correct operations second, then a tie between portability and speed.

LM: We interviewed Tim O’Reilly a while ago, and he basically said that a lot of the focus on Linux is really missing the point. It’s not Linux but Apache that is the most important piece of software because Apache keeps the Internet protocols open. Microsoft owns the client side; they don’t own the server side, and as long as Apache remains dominant, they won’t be able to own the Internet.

BB: What I think Tim might have meant is that Apache is an important template for how open source projects should view themselves. As a project I think he probably likes the model that it’s run under more than the kind of benevolent-dictator model that a lot of other software is written under.

And that’s an important thing. That’s a good thing. But I think the second thing is this adherence to externally defined standards. And most open source projects have that. Samba to SMB [Server Message Block], Linux to POSIX [Portable Operating System Interface for Unix].

But the role for the intersection between open source and standards is in my mind really interesting. The same way that you have groups of people defining standards through things like the IETF and the W3C [World Wide Web Consortium] and stuff like that and you have other groups implementing those standards. They seem like a nice complement to each other. They seem like as natural as yin-yang, left brain-right brain.

LM: That’s a really different perspective from what I’ve heard from Linux people. If you ask some Linux people today, they would say that Linux is the standard for Unix now. POSIX needs to follow Linux and the Linux kernel as defined by Linus, and that’s end of story.

BB: That model works primarily because Linus is an extremely good project manager, extremely good at gaining consensus. Even though his word is law, so to speak, he never makes a decision without carefully considering it. He never makes a decision without incorporating comments from everyone, even those he disagrees with. And that’s a hallmark of every good leader in the community. He’s essentially running both a standards organization and a software organization at the same time.

LM: OK, about Collab.net. SourceXchange, which is a collaborative development site for people who want to pay for open source development, is the first Collab.net project. Where did that idea come from? Was there a Web site that inspired you to try the SourceXchange model?

BB: There’s a site called the Free Software Bazaar, which someone was running off a .edu/~person account. It was basically: “Here are some ideas of things that need funding. If you want to contribute, write a check.” It was real homegrown.

And I thought, with developers going for six-figure salaries — especially the good ones, and especially those who know how to do open source software development these days — no one’s going to do this, except for the most trivial of bug fixes. In the time it takes some of these developers to read the message itself, they’ve already effectively spent that amount of money.

So we had to focus at a higher level: at doing something that was more than a few days’ worth of work. We needed to have checkpoints in that process, and we needed to reassure people that they wouldn’t do all that work only to have somebody else, who took more shortcuts and delivered lower-quality software that still met the spec, get the business.

So it was about making sure that we built a process that actually had quality control, where people are motivated to write good code rather than just the fastest code they could write, and was usable for large projects. And then we said: “If you’re an independent developer and you really want to work in software, there are things you hate dealing with like paperwork, like collections.” If you’ve written software for someone, going after them for the cash just sucks, right? So we said: “We can act as a trusted third party in this relationship. We have a lot more leverage on the average company than a small, independent developer might, so we’ll take care of collections and we’ll also act as the contract proxy.”

LM: How many developers have you attracted?

BB: Forty-three hundred so far. We have about six different companies that have put in RFPs [Requests for Proposals] live on the site. There are another 10 that have actually signed the sponsor agreement to put RFPs in. Our hope is that we build up to the rate of 20 projects a month
through this year.

LM: How do you make money from SourceXchange?

BB: We charge a fee on top to the sponsor.

LM: What is the business model behind the software that drives SourceXchange, Tigris?

BB: The business model is to provide it as a hosted platform. To go to companies and say: “If you want to have an open source project like HP’s e-Speak technology, we will provide that. We will host that for you on these servers and rather than you having to pay an administrator and bandwidth and machinery and all that kind of stuff.” So it’s a subscription kind of basis.

LM: How is that different from SourceForge [A competitive project run by VA Linux Systems]?

BB: SourceForge’s business model is different. It’s “let’s get everyone using us so we can get lots of traffic.” From a technology perspective it’s different from the back end. They rewrote a whole lot of stuff, whereas our model is to try to use as much existing software as is out there and glue it together with a kind of common glue.

LM: You provide open source development tools to companies that may not have them?

BB: Right. Basically we can build engines. SourceXchange is an engine. Tigris is an engine. They both have independent business models and they each potentially could be companies on their own.

LM: Tigris is open source software, though. Why would someone come to Collab.net to host a Tigris-based project?

BB: Why does somebody host with Above.net rather than Level 3? We’ll manage the servers; we’ll manage the uptime, and by hopefully developing a good track record in doing that, we’ll be able to get these sales.

LM: But any ISP could provide that level of reliability, couldn’t they?

BB: Running this collection of software is actually very nontrivial. So it’s an operations sale: The more complex the software gets, the more reason there is to outsource it. Something very simple like Web hosting, for example, is easy to do on a mass scale, but it’s also easy for people to do themselves. Integrating CVS with all these other tools is not as trivial.

LM: HP is a major partner on SourceXchange. Do you think the established technology companies really “get” open source?

BB: There are clueful and unclueful people in every company. I think in every company the people at the very top and the people who report to them get it. And the engineers and even the product managers at the bottom, so to speak — at the line level — get it.

I think it’s the people in the middle of all of these companies — people who have been charged with worrying about: “We’ve got products to launch and we can’t just drop everything, and you’re going to give away what? I get my bonus based upon how many sales I make. You’re telling me I can’t make that?” That’s where the real entrenched war is.

Today when people buy commercial software, the ones and zeros on the CD are meaningless without the company documentation, trust in the company that they bought it from, and the ability to get support when they need it. Most people probably never call a support number for software they buy, but they know they can. They know they can return it. There are guarantees of service there. The physical box that they buy is just an embodiment of all those services behind it. So one could say that the ability for a company to sell software and never support it and just sell ones and zeros as if it were widgets and out the door and never to be worried about again is gone. But I think any company that thought that was the business is out of business already.

LM: So in a sense, Microsoft is kind of like a blip?

BB: Are you leading me to say something dumb? [laughs] “Behlendorf Says: Microsoft Meaningless in the Next World.” No. Certainly not. Could they retool to be a profitable business — as profitable as they are today — in a world where all of their ones and zeros needed to be free? That’s a big question.

LM: I think it’s interesting because they revolutionized the world. They proved the point that you could just sell those ones and zeros and not much else.

BB: I think it’s a fiction that they’ll be able to ride a little bit longer. But there was software before Microsoft came along and it was free. Free soft-ware’s not a new idea. When Bill Gates sent his message to all the people copying DOS saying: “Hey you kids! Get outta my yard!” that set the stage for something different.

I work from the presumption that software gradually becomes open source. It doesn’t mean that next week everyone’s going to give away all their stuff. It also doesn’t mean that there isn’t a future in commercial software development. I think it does mean that the big companies today are going to have to realize that they are services companies rather than goods companies.

They have this factory model, as if they’re pumping out cars. And their accounting is all based on that kind of accounting as well. They see support as a necessary evil. Something that they have to do and make sure that they don’t piss off people so badly that other people don’t buy their software. Or if they do piss them off, they have to make sure that people have to buy their products anyway. I hope that model changes.

Robert McMillan is executive editor for Linux Magazine. He can be reached at bob@linux-mag.com.