ONLamp.com    
 Published on ONLamp.com (http://www.onlamp.com/)
 See this if you're having trouble printing code examples


Confessions of a Recovering NetBSD Zealot

by Federico Biancuzzi
09/14/2006
The NetBSD Project has stagnated to the point of irrelevance. It has gotten to the point that being associated with the project is often more of a liability than an asset. I will attempt to explain how this happened, what the current state of affairs is, and what needs to be done to attempt to fix the situation.

This is the first paragraph of the email that Charles M. Hannum, one of the creators of the NetBSD Project and the NetBSD Foundation, posted on BSD-related mailing lists.

Federico Biancuzzi interviewed Charles (who was affected by the recent termination of some netbsd.org accounts) and discussed the evolution of the project, funds management, problems with licenses and hardware documentation, the link with vendors, past and current mistakes, and what we can learn from the Linux development process.

Could you introduce yourself?

Charles M. Hannum: I'm one of the creators of the NetBSD Project, and served as its de facto technical lead for a long time. I was also involved in creating the NetBSD Foundation, and served as its president and chairman of the board. (Note: I was never the Foundation's secretary or treasurer.)

How did the NetBSD Project start?

Charles M. Hannum: The four people who started NetBSD were Chris Demetriou, Adam Glass, Theo de Raadt, and me. At the time, Chris and Adam were attending Berkeley and close to graduating. Theo was working for a living. I was doing other things. Chris is still sort of around, but doesn't really do anything; Adam went to work at Microsoft and is now lost to us; and we all know about Theo. There is some info in the NetBSD entry on Wikipedia.

I think the most striking difference between late 1992 and today is that there really was no "web." The original software had been released, and some of us had started using it for small (non-graphical!) websites, but it was still very much a hobbyist thing. More people were getting "email accounts" of various types, but penetration of the internet into homes was still quite low. Even so, this was near the end of the NSFNet era, and there were increasing problems with the backbone being overloaded. There were also some as-yet-undiscovered problems with TCP flow control which caused additional performance problems. And lastly, nobody had started taking network security seriously.

On the open source side, you could liken it to the Stone Age. The operating systems (primarily 386BSD 0.1 and Linux 0.12) were nothing to write home about; they were buggy and incomplete. There were no "desktop" packages or "office" suites of any significance. (I actually ported Applixware to NetBSD under contract, because a certain blue company wanted it for their thin clients that ran NetBSD.) Development of X was hampered by the dissolution of the X Consortium and a vacuum of leadership there. We discovered later that GCC development was also being hampered by mismanagement (since fixed).

What I'd like to stress is that there was no Dummies Guide to Starting an Open Source Project. The term "open source" wasn't even being used yet, but that's beside the point. We were the first big collaborative project to use a version management system. (As examples, Linux used none, and most GNU software at best used "backup files" for version management.) There were no previous successes--or really even significant failures--to look at for guidance about how to organize the project. So we made it up.

I want to elaborate on the point about network security a little. Keep in mind that, in this time frame, most people were still using rlogin and unencrypted telnet. Buffer overflows were rampant. I (and other people) had already started doing experiments with subverting web sites; we all knew it was possible then, but nobody cared. It was several years before most people saw the writing on the wall and started clamping down access to central repositories and whatnot. Even today, most software distributions on the net (and this includes NetBSD) are not signed.

The primary reason for starting NetBSD at that time, ironically enough, is that there was a perceived lack of management in 386BSD. The actual 386BSD release only ran on a handful of systems, and was quite buggy. There was a rapidly growing community around it nonetheless, and many people had contributed patches. However, 386BSD's leader simply vanished. Nobody had any idea what he was doing, or whether he was even looking at the patches or working on another release. Eventually we decided that the only answer was to make a go of it ourselves--that's right, it all started with a fork.

Would you like to talk about the fork that originated OpenBSD?

Charles M. Hannum: No.

Since it came up in the /. thread, though, I would like to make one correction. It's widely claimed that I'm "the one" who ejected Theo from the NetBSD community. That is false. At that time in NetBSD's history, Chris G. Demetriou was playing the role of alpha male, and I wasn't even given a choice. I was certain it was going to bite us in the ass. I think the question for historians is not whether it did bite us in the ass, but how many times and how hard.

Why did you focus on portability?

Charles M. Hannum: At the very beginning, this was not actually a focus. It quickly became apparent that there were a large number of people interested in NetBSD (and open source OSes in general) who were currently on non-x86 platforms. Remember that this was still the 80486 era--PC processors weren't very good. Most "workstation"-class computers were based on something else--a myriad of Motorola 68k and 88k, SPARC, POWER, etc. On the HP9000/300 and SPARC platforms, we also had the advantage of access to code already written--though in both cases the integration was complex, and the code only supported a handful of systems at first.

I was also hanging out (and occasionally doing some work) at the Free Software Foundation, where there were a lot of HP9000/300s--running MORE/BSD, which had long since been abandoned. So I set out to get NetBSD running on these machines. Just getting a cross compiler working at the time was quite a bear, but it didn't take too long before these machines were all running NetBSD.

Yes, several of the earliest NetBSD development systems were owned by the Free Software Foundation.

With working 68k support in hand, we quickly spawned Amiga and Mac development groups. I then helped Theo get the SPARC support from LBL integrated. The whole thing snowballed.

Of course, the fact that you've ported the system to tons of machines does not mean you have good portability. We were still in the days of "copy and edit"--and so, for example, although the Amiga and Mac code [was] heavily cribbed from the HP9000/300 code, they were separate and had different features and bugs. Trying to fix bugs in this environment was making me crazy--and wasting a lot of time--and eventually I just started smashing the code together at high speed and sifting out the common parts. Other developers followed ("lead by example" works sometimes), and so this led into our global thinking about portability architecture, shared device drivers, etc. Oddly enough, Microsoft was working on similar ideas at the same time, in the development of Windows NT.

How was the relationship with hardware manufacturers at that time?

Charles M. Hannum: Terrible. We rarely got documentation from anyone. I actually wrote a lot of device driver code by guessing what the device was supposed to do. (Lots of previous experience reverse-engineering code helped there, I'm sure.) We had severe problems trying to deal with things like the "programmable" Adaptec SCSI controllers that became very popular; it was so bad that I was honestly talking about staging a sit-in at Adaptec HQ, and probably should have done it.

There were some notable exceptions, though. It took a while, but NCR, and later LSI, finally came around and dropped a heaping pile of (fairly good) documentation on me for their SCSI controllers. We did eventually get some material out of BusLogic, but only for their older "heavyweight" controllers (what we call "bha"<-- should bha be in code style? -->).

Intel, for its part, has been pretty good about putting CPU and chipset documentation online for the last several years, which I applaud, but their networking documentation (both wired and wireless) has been extremely poor. The strangest part of the Intel story was when the i82559 manual became restricted, even though it was substantially identical to the i82557 manual [that] had been published in their networking databooks earlier. Most other companies producing Ethernet controllers have been decent to us, except for Broadcom and Marvell, which have [each] been a 100% loss. Wireless vendors have generally been a tremendous annoyance, generating excuses but no documentation--I think Atmel and Realtek are the only exceptions.

We got some scattered documentation from other companies--e.g. Ensoniq and Realtek--but it's sometimes been incomplete and very difficult to make sense of.

Fortunately we didn't have to deal with most of the PC video card circus ourselves; XFree86 and now X.org have taken care of that.

What type of relationship did you have with the license? Is this relationship changed during the time?

Charles M. Hannum: Most of us had a very strong distaste for the so-called "virus" clause in the GPL, and this is the primary reason we did not adopt it. There was also some thinking that CSRG (Berkeley) and the X Consortium had been successful with leaner, looser licenses, so why bother. In retrospect I think this was naive; if you look at the history, you'll see that neither CSRG nor the X Consortium were really successful in getting third parties to contribute back most of their changes--and so what we really got in both cases was a long list of derived but very different, and often incompatible, systems.

Linux has not been wholly successful in this either, and today there are myriad distributions which are subtly incompatible. However, they definitely did better.

If I were doing it again, I might very well switch to the LGPL. I'll just note that it didn't exist at the time.

How much did the (in)famous BSD lawsuit hit NetBSD code base and popularity?

Charles M. Hannum: There was a lot of FUD around this issue--some of it from Linus, actually--and it did cause us some problems. The reality is that we had a signed agreement with USL that essentially said we had to upgrade certain files from their Net/2 versions to 4.4-Lite, and not distribute some other files at all (which we never used in the first place). We were in the process of moving to a 4.4-Lite base anyway, so this had virtually no impact on development. It did, however, delay making our CVS history public--far longer than it should have--because we needed to remove some of that early history in order to meet the conditions of the agreement.

I've never seen the similar agreement between USL and FreeBSD, but my understanding from what I've heard is that it is quite different. This caused some more FUD to be generated, because apparently what we did would not have met the terms of FreeBSD's agreement.

Had Novell not bought USL when it did, it's unclear to me how this would have panned out. I've never been able to convince myself that Berkeley was in the "right." However, Novell put a swift end to the suit, the agreement is very clear, and nobody cares about that early code history any more--so this is all water under the bridge.

Have you read the legal settlement after it was recently made public? Any surprise?

Charles M. Hannum: Yes, I read it. The first thing to note is that this agreement was with Berkeley. We executed a separate agreement with USL (which has not been made public), and that is what governs our relationship. There were no surprises when we read the settlement, but it wasn't really relevant to us.

If you had a separate agreement, you were free to work on your software without problems. Do you think that the general and chaotic FUD about the lawsuit hit you even if you had already solved the problem?

Charles M. Hannum: Absolutely it hurt us. A lot of people (and I don't want to be divisive, but honestly they were mostly Linux proponents, including Linus himself) spread FUD for years about BSD systems being "unsafe"--even after the UCB/USL lawsuit was settled. The fact is that there was no danger in using NetBSD in a product, and a number of companies did so.

How was NetBSD funded during these years? Who managed these funds and how?

Charles M. Hannum: In the beginning, we just put the machines on the UCB and MIT AI lab networks, because we had access to do that and nobody minded. The server equipment was purchased by Chris Demetriou. The project per se had no "funds." Later on, colo space and machines were donated by a variety of organizations (NASA, iki.fi and hut.fi, etc.), and again no money changed hands. Later on we started collecting donations to purchase hardware; colo space was (and is) still donated, primarily by ISC now.

As far as funding marketing work, such as conference appearances and merchandise, most of that was paid for by me, d.b.a. The NetBSD Mission. A fraction of the cost of the Comdex booths was paid for by LinuxMall. Most of the other conferences gave us "free" booths--that means the conference itself didn't charge for the booth, but we (I) still had to pay for everything else (carpeting, furniture, shipping or renting equipment, union labor, etc.). Producing CDs and T-shirts to give away (we tried selling them at conferences, but that didn't go over well, especially at Comdex) was also fairly expensive; it adds up quickly.

Do you have any regret about the way NetBSD promoted the project and did advocacy at conferences around the world?

Charles M. Hannum: No. Unfortunately there is no longer a concerted effort to do this, and particularly to give away copies that people can try. Frankly I'm not sure (and wasn't even then) that giving away copies to install on a PC will impress people much anyway, given that NetBSD's installer is still very primitive compared to the Linux distros. Many reviews have focused on this and lambasted NetBSD for it.

What is your opinion about strong links with vendors? For example, NetBSD and Wasabi, or Linux and Red Hat and other commercial distros, where these Linux distros try to push in their code and favorite features, for example, file systems.

Charles M. Hannum: That one is complicated. Part of the point of the "coup" was obviously to get more Wasabi people into the new hierarchy, and there were some very strange and discouraging actions that resulted from that. Wasabi also marketed itself with the tag line "The NetBSD Company" early on, which was clear trademark infringement--but due to the influence of Wasabi people on the organization, there was nothing I could do about it at the time. Meanwhile, the Foundation was sending legal threats to other people producing T-shirts with NetBSD logos on them, alleging trademark infringement.

In addition, Wasabi actually held up a NetBSD release for more than a week, while failing to tell even the developer community why. (The main release engineer happened to be a Wasabi employee, so this was easy.) After I personally pushed the release out without them, it was finally revealed that this was because there was a secret arrangement, which only Wasabi employees and Wasabi board members knew about, to do a combined NetBSD Foundation and Wasabi press release. They were actually upset at me for preempting this.

But it gets even more incestuous than that. The new bylaws were written, also in secret, by people with an interest in Wasabi. They were presented wholesale to the developer community (even I didn't get a say in them before that, and I was one of the two Foundation board members). I repeatedly asked for a definition of what the goals were for a reorganization, and complained about certain problems (e.g. that the dissolution clause actually instructs the Foundation to do something that's illegal), and was completely ignored. In fact, all input on the proposed bylaws was either discarded or completely ignored; no changes were made.

Let me put the question out there. Is this how you think the relationship between a for-profit company and an open source group should work? Even the biggest distros don't have this kind of influence on Linux.

That said, Wasabi mostly does its own thing now, and doesn't really contribute much of its work to NetBSD. I was concerned at the time about the possibility of a "brain drain"--people going to work for Wasabi and no longer hacking on NetBSD on their own--and it turns out that this has in fact happened (and I've seen multiple complaints about it recently).

The Linux structure is set up such that Red Hat can't push things into the Linux kernel on a whim. They do sometimes keep their own changes and put them in their releases, but there is a concerted effort not to do that too much (possibly more because it would annoy the community than because they don't want to). So Red Hat has not really had an undue influence; I think their contributions have largely been positive.

This is rather unlike the current NetBSD situation, where Wasabi people can effectively do whatever they want, and there is no check and balance.

Can you elaborate on the trademark infringement?

Charles M. Hannum: You probably thought NetBSD was free--like Linux, you can download it, burn some CDs, and sell them. Or make T-shirts or bibs and sell them. Well, it used to be, but it's not any more. If you put NetBSD CDs or other merch up for sale, you can expect to get a nasty letter from the NetBSD Foundation accusing you of trademark infringement and demanding royalties. I don't know how many letters they actually sent, but we've gone from at least 6 CD vendors in the U.S. to one in the last few years.

What do you consider the biggest errors made by the NetBSD Project?

Charles M. Hannum: First, the short list:

  1. Not establishing a source of funding early enough.
  2. Not developing a package system from day one.
  3. Not backporting hardware support to the previous "stable" release.
  4. Not restricting access of developers who repeatedly break things.

Now, I'll explain the points in more detail:

  1. FreeBSD, and most of the Linux distros, associated themselves with CD distributors (or were the distributors) early on. This gave them a source of income, which enabled them to do a lot more promotional work. FreeBSD, for example, got a lot of support from Walnut Creek CD-ROM (including revenue from Linux sales!) to ship out free CDs and merch to user groups.
  2. I actually wanted to develop a package system, but I was told that it was "stupid" because people could just build from source. I think the last nail in that idea occurred when we adopted pkgsrc (and started giving out prebuilt packages), but in a way that was too late. We had already adopted a model where the "OS" was really heavyweight and included a lot of things that probably could have been better handled by packages--e.g. GCC, UUCP, sendmail, groff, etc. Maintaining our own separate build infrastructure for these things has been a pain, and it has really slowed adoption of new versions.
  3. What can I say? This has lost us a lot of users, because the "stable" release often just doesn't work on new hardware. The staleness was so bad that I almost singlehandedly put out two releases (1.3.2 and the "Comdex 1999 preview"), just so that I could press CDs that I considered worthy of being demoed.

    Some of this is also due to poor design decisions. Remember when the P4 came out, and it was revealed that one of the CPUID values was chosen so that Windows would boot? We all laughed at Microsoft's shoddy code. I'm sorry to report that NetBSD has had similar problems far more than once. (I always railed against the idea of including CPU-specific code, rather than feature tests, myself.)

  4. I can't even begin to describe how much of a loss this has been. We honestly had (and have) developers who think that changing all the line-ending whitespace was really important. Or that whether there are some "legacy" K&R-style prototypes actually matters. These people go around committing non-functional changes, which makes source management (and specifically patching and merging) really painful, and which, unfortunately, also often breaks things accidentally. What's worse is that sometimes they don't even bother trying to compile it, much less test it. I wish I was making this up; I wish more that I had made these people go away a long time ago.

Recently NetBSD defined a set of rules and time slots to manage the release engineering process, and also a new numbering scheme. Do you think this will solve any of the problems you have just outlined?

Charles M. Hannum: Not at all. What we see now is just a mad push to release something, no matter what it is. The gold standard has moved entirely from technical excellence to the time on the clock. This isn't a good solution either, and actually it risks losing a lot of users by putting out really buggy releases. There needs to be some balance.

NetBSD is the only BSD project that kept using XFree86 after the license change in 4.4.0. What is your opinion about it?

Charles M. Hannum: This relates to something I mentioned before. The fact that X is part of the "OS," and uses a custom build process, makes it much harder to update--it's really a substantial amount of work. If it was a separate package, I think we would have seen X.org in use a long time ago--someone would have just spent the few hours to make the package and people would have started using it. It's this method that is really the core asset of the "bazaar" idea.

Do you think that we are going towards a universal package system, something like pkgsrc or OpenPKG, or that in the end every OS/distro will develop its own?

Charles M. Hannum: Well, look at the current situation. If you look at just the major options in the Linux community, you have apt, rpm+up2date and rpm+yast. They have all converged on pretty much the same feature set, but they have different UIs, and the packages themselves are maintained by completely different communities (with their own versioning, patches, and selected options). There's no good reason that they need to have different underlying formats (.deb versus .rpm), but they do.

I think fundamentally what leads to a lot of forks in the Linux distros is NIH syndrome. There's no reason to think this won't apply to package systems.

Oddly enough, though FreeBSD ports, NetBSD pkgsrc, and OpenBSD packages have all forked their maintenance and build structures long ago, and they are now quite dissimilar, the prebuilt package format is still quite similar. I think it would be a good idea to at least merge the format and the utilities, if for no other reason than it would enable sysadmins to learn one set of tools for all three systems. Some work on package upgrading and patching also needs to be done; all three systems handle this poorly.

What do you like in the management and code development process of other BSD projects? Which things would you define as "best practices" and would you try to repeat in the NetBSD Project?

Charles M. Hannum: As I alluded to in my original statement, I think FreeBSD has similar problems, and this is a large part of why Dragonfly was created. OpenBSD does have strong leadership, but unfortunately it's a little too strong, and that has caused a lot of people to not contribute. I don't have a guide to what works best, but I can definitely tell you a few things that do not work.

There are quite a few open source projects that I classify as successful--though what that means could be another debate. Two examples I'll give are KDE and X.org. These projects are making good progress and doing really good things.

From your point of view what are the advantages and disadvantages of the Linux development process?

Charles M. Hannum: Well, first we have to define what the "Linux development process" actually is. I think ESR got it wrong. Back then, we had different distros all doing their own thing, making their own sets of kernel patches, and integrating totally different versions of other software. Linux compatibility was terrible--it got so bad that at least one company ported NetBSD's libc and statically linked everything so that their software would actually run on multiple distros.

However, some things have changed. Compatibility has gotten better--though there are still problems. The base Linux kernel distribution has gotten far enough, and complete enough, that there are fewer changes maintained by distros. I think this is in large part because there is a more open and effective (though sometimes seemingly arbitrary) method for getting things into the base kernel.

The thing that has not changed, and that carries through within most distros, is that the individual packages are maintained in fiefdoms. Many of those fiefdoms are impenetrable and inscrutable; it is as if the gate is down, the moat is poisoned, and they occasionally toss a cow over the wall to feed the peasants. In short, I liken the "Linux process" not to a bazaar, but to a set of independent fortresses, all doing their own thing. (BTW, I use such a metaphor only to keep in the spirit of ESR's metaphor.)

This has been a subject of some debate for many years. Back when I was hitting the conference circuit, I sat in on discussions about the problems with this model and how to fix it. The biggest complaint seemed to be the lack of transparency and accountability for upstream and package maintainers, partly because most of them used no version control system at all, or kept all the history private. Because of this, it's much more work to pinpoint bugs, and where they came from. It's also hard to decide whether the maintainer is doing a good job, and whether a fork is mandated, if you have no idea what they're doing.

(I know some people argue that "there's no point in assigning blame," but I'm not talking about blame. Sometimes it's important to understand why a change was committed in the first place; e.g., maybe it was supposed to fix a security hole, but they made a mistake. Obviously you don't want to put the security hole back in.)

In short, I don't see the Linux model as a panacea.

How should open source project leaders assign or remove the commit bit?

Charles M. Hannum: I think the first thing to do is have a firm set of commit standards, as I mentioned. Maybe schedule a particular window during the development process for committing non-functional (i.e. whitespace, function prototype, etc.) changes, so that people can merge branches around them only once rather than a zillion times. Switch to subversion (or git, if that's your choice of Kool-Aid) and make people batch similar commits. Be hard-nosed about removing changes that don't meet standards or break things.

The Linux kernel effectively has such standards now because everything is filtered through a small set of people with reasonable taste (and they're not totally overloaded the way Linus used to be). That's not to say I agree with all of their choices, but they seem to do a good job of enforcing their standards and principles, and I have to respect that.

NetBSD today does a very poor job of setting and meeting standards. I created the mythos of NetBSD having "clean" code, and even I don't buy it any more. By "clean," I certainly never meant "no trailing whitespace"--the idea was that changes were publicly reviewed and cooperatively developed, and we didn't do quick minimalist hacks. This isn't how things are done now; in fact, at least one major architectural change recently was developed entirely in secret and committed wholesale, with zero public review. Despite its rather different beginnings, even the Linux community is more open than that now.

I'm not the only one who was affected by the recent termination of netbsd.org accounts. Of particular note are two other names: Lennart Augustsson and Darren Reed. Lennart single-handedly wrote the entire USB stack in NetBSD, and many of the device class drivers; Darren wrote ipfilter, which is the only fully supported firewall and NAT solution in NetBSD (though there has been some ongoing work to integrate OpenBSD's pf). These two have rather obviously made significant contributions, and their termination is a major loss for the project.

The announcement of this action also contained several factual errors.

After your email made it to the mailing lists of every BSD project (including OpenSolaris), what has happened? What do you think will happen in the near future?

Charles M. Hannum: The overwhelming majority of feedback I've gotten privately (and there has been a lot) is positive. I've heard from many former users and developers who abandoned NetBSD for reasons that I stated. Even comments from Linux and FreeBSD developers have generally been positive. It's been interesting to read. There have been quite a few people calling for a fork, and even offering help, but I'm not going to state an opinion on that at this point.

Not surprisingly, there has been quite a bit of flamage on the NetBSD mailing lists--mostly people trying to stand their ground and pretend that there is nothing wrong.

I think the most interesting part is that several users have specifically said they "didn't know anything was wrong [with the organization]." This is at least partly my fault; I stepped out for a while and waited to see how the circus proceeded without me. All I can say is that it's been an unmitigated train wreck--they've repeatedly grabbed more authority and kicked people out of various positions, made people sign Draconian agreements, done almost everything in secret, and failed to actively promote the project. I find it hard to imagine that an open source project could have worse management.

Federico Biancuzzi is a freelance interviewer. His interviews appeared on publications such as ONLamp.com, LinuxDevCenter.com, SecurityFocus.com, NewsForge.com, Linux.com, TheRegister.co.uk, ArsTechnica.com, the Polish print magazine BSD Magazine, and the Italian print magazine Linux&C.


Return to the BSD DevCenter.

Copyright © 2009 O'Reilly Media, Inc.