"Dear Mr Brooks, or: 
		Software engineering in the free software world"

		Talk given by Alan Cox at LinuxTag 2000, Stuttgart.

Note: 	

This transcript was made after repeated listening to the
actalk_*.ogg files provided by the LinuxTag organisers after
LinuxTag 2000, 29/06/2000-2/07/2000. See 
	http://www.linuxtag.org/2000/english/
	http://www.linuxtag.org/2000/deutsch/
for more on LinuxTag. 

I found the ogg files in ftp://ftp.redhat.de/pub/linuxtag/ ; 
they are also in ftp.linux.org.uk/pub/linux/alan/OGG/ and 
probably elsewhere. I have not attempted to mark where the
slides were referred to because I was not present at the talk
and have no idea. I have also concatenated the talk into one
long file because some of the .ogg files stop in the middle
of sentences. 

Thanks to the LinuxTag people for making it available. 
No thanks to Alan for mumbling. I am not a professional 
transcriber (as anyone reading this will soon notice).
Where he mumbled, I have put "[inaudible]" or "[inaud]". 
Double hyphens ("--") tend to represent those moments 
where he paused and restarted a sentence.
 
People at either the OLS (a year earlier) or Vienna (six 
months later) versions of the talk will notice that yes, it's
a bit different each time.

I have not written the questions up. It was extremely hard to
hear the questions, and all bar one of them were actually about
the kernel instead of about this talk itself. (The exception
was from a questioner who asked how different thing was from
the seventies way of creating UNIX, and Alan didn't know because
"I wasn't there then. I was too busy watching Scooby-Doo".)

Enjoy. 

					--Telsa


[Sound of clapping]

[Faintly] Can everybody hear me? Good...

Okay. First thing, just out of interest: some of you have told me that
the book "The Mythical Man-Month" isn't something particularly commonly 
found in Germany. So how many people have read "The Mythical Man-Month" 
here? How many have heard of it? Oh dear. Oh well. You should all go
and read it. 

"The Mythical Man-Month" is a book which was written by someone who 
had the misfortune to manage IBM's OS/360 operating system release. 
At the time, it was probably the largest software project ever 
attempted. Even though nowadays it's pretty much a tiny effort, the 
work it represents then was huge. He never quite got over the 
experience and he became an academic and wrote a book. [Laughter] 
It's from that book that we get some of the ideas that programmers 
work best in small teams; that adding new programmers to a late 
project will make it later; and a lot of the other basic ideas of 
software engineering. 

So I had been re-reading this book, and I started to look at all of
these Linux and other free software things, thinking, "Some of these
rules don't seem to be working". And I started to wonder how the system
itself actually works. So I put this, the original version of this talk,
together in Ottawa last year. So anyone who was at my talk at OLS
can leave, because you've heard this before.

Okay. Right. First of all, for anyone who's forgotten university
computing, or never did it, this is conventional software engineering. 
This is how most large companies build software. 

The first idea is that you have people working in small teams. This 
is not just a computing idea: if you have a large team of people like 
a large meeting, you end up with a lot of talking and no work done. 
Even people like the military understand this kind of idea: they have 
small groups and hierarchical command structures. 

To get a programming system to work, you need the programmers to 
able to spend enough time programming rather than arguing, to move the
project in a forward direction. And various models have been proposed
based on the number of people who may need to talk to each other. There's
a thing called "combinatorial explosion", which means three or four
people is fine, by the time you've got six people there are seven
hundred and twenty ways they may be talking to each other. [Laughter].
By the time you've got to about ten people, you might as well retire.

You have a hierarchical management structure: that is to say there is a
clear idea of "who is in charge of this project", and "who is in charge
of groups of projects above them". So you have this tree-like 
structure, with the management at the top, who thinks it's a very 
good thing, and the programmers at the bottom, who think otherwise. 

You have controlled access to other teams. I'm generalising here. 
Some large companies don't work this way. There's a general rule
that if you wish to talk to programmers working on another part of the
project, you ask your manager to have a meeting with their manager. 
You tell your manager all the questions you have, which he probably
doesn't understand half of, he tells -- he has a meeting with some
of the other managers, and they discuss this, and simplify it
even further, because none of them really understand what's going on.
[Laughter] It's not even the fault of the management. If you're
not directly involved in a project, any information you get is
second-hand. So you never understand the fine details of what's
going on, even if you understand programming. Um. So these managers
have a meeting, and something vaguely resembling your question ends
up at the other programming team. They explain the answer to their
manager, who translates it to something very simple, who two weeks
later at the next meeting, answers the question as an "action point"
from the previous meeting, and then hopefully your manager transmits
it back to you. So it's very hard to get messages a long way through
the project.

Software engineering people understand this. They try and organise
and say "Sections of the project that need to talk to each other are
close together: for example, under the same management, or at
least the same two layers of managers. And they try and write lots
of specifications. 

So initially you start with a formal specification. This is what you've
agreed with the customer the project has to do. The formal specification
is meant to be a fixed item. It's never fixed because what you and the
customer think it meant are always different. English is an ambiguous
language, German probably likewise. 

>From this, you write a design specification. You carefully break your
project up into lots of small projects; you assign teams to each project, 
so it's all very very neatly and hierarchically organised; and then one
of your team leaves and goes somewhere else... And all these other things
that go on in a company. People move around. 

So you have this very very inflexible tree-like structure and the structure 
of the program is in your design. So when the customer comes along and
says, "That's not what I meant. What I was trying to say was this.." you
actually end up changing -- having to rewrite part of the design document.
Which means you're changing the way people are assigned to the project. 

After a design specification, in formal software engineering, you write
a detailed analysis of the interfaces. Object-oriented programmers
draw pretty graphs showing inheritance and message-passing; and non-object-
orientated programmers normally draw tree-like diagrams which actually
look very similar. And you start writing the code. 

And all these little teams go off and then they start discussing 
bits of code. As you start writing the code, your marketing team goes
out and starts explaining to everybody how wonderful this product is,
and yes, they can deliver it to you in twelve weeks. [Laughter]. It's
very common. This is one of the reasons programmers and marketing people
often don't get on: the marketing people are being told, "We will deliver 
this in twelve weeks" by management -- who thought the programmers said 
they could deliver it in twelve weeks, but they weren't quite sure. Some of
them sort of nodded their heads at meetings at nine in the morning, so
obviously they can deliver in twelve weeks. 


Free software is a little bit different.

First of all, if you say, "This will be finished in twelve weeks,"
and you're not contributing code, you get laughed at. 

It's still very small focused teams. If you look at most free software
projects, all the small ones are normally two, maybe three or four,
people. So something like CD player applets which everybody writes, 
you'll have two or three people working on it.

The bigger projects break down naturally into small groups. So if
you look at KDE or GNOME: the original KDE group was a small group. 
As KDE became more than an initial idea and some very simple bits of
application code, KDE became lots of programs. And each of these
programs has got a small group of people involved. And they all 
talk to each other, these groups. But there's a hierarchy still. 

If you really want to get something done in GNOME, you talk to Miguel
de Icaza. Because Miguel de Icaza is the boss of the GNOME project. 
Nobody made him a manager. He didn't get there because he'd been working 
on it longer than everybody else, although I suppose he had in this case. 
He got there because everybody figured that he was the right guy for
the job. So there is still a hierarchy. If you are working on
something like a GNOME or a KDE project, and you make patches to an
application, the person you ask for permission to add patches to 
the application is the person who is sort of in charge of that 
application. So you can normally look at a piece of code and say 
"Who is responsible for it? Who is the head of that particular 
application, for that particular piece of the project?" So that 
shows there is a hierarchy. 

So even though it's free software and people say we don't have core 
teams, we don't have any kind of hierarchy, there is a hierarchy there,
even if it's not a formal one.

Access points between teams: very different. Instead of going through
managers, you simply email the other person and wait for a reply
from the other person. And hopefully your email will get forwarded, or
more usually you'll get an answer back saying, "No. Talk to somebody 
else". That's a very important difference.

One of the Debian project people conducted a little experiment. He
started tracking some of the messages going into the Debian project 
which went through several people to get to the right destination. He
found it's about a 50% chance of somebody who gets the message
forwarding it on. So if you need to forward the message through three
people, you will actually need to send eight copies of the message. 
on average, to get one copy through. So this idea of sending mail
to someone and then he'll answer back saying, "No. Talk to such and
such a person" is important. Effectively, communication between parts
of a free software project isn't fixed; it is continually changing as 
different pieces of a project need to talk to each other. As two groups 
of people suddenly realise they can co-operate, they will start talking 
directly, not through any third party who may have introduced them. 

When you release a free software project, you do things in a different
order. Firstly, you get some code. Hopefully, it just about works. And
you document it as "Needs fixing, needs this, needs that." But most
free software code, to get other people involved in the project, it has
to work. It doesn't matter if it's hard to compile. It doesn't matter
if it only works on one machine in five. And it doesn't matter if it
eats the data file every so often. So long as sometimes, the right
results happen, people will start to pick up the project and use it. 
They start to use it, and then they have to fix it. 

You have marketing. Marketing is important in free software. This
is why all free software projects nowadays have web sites. The goal 
is different. The goal in most conventional marketing is to get users, 
because users are the people who pay you money to improve the product, 
and the more users you get, the more you can charge for it because the 
harder it is to go somewhere else. In the free software world, you do the 
marketing first. Because the website initially is not targeted at the
people who will use the software. What you are trying to do is to
get developers interested in the actual application. So you're
hoping you can find people who see a use for this software who are
also programmers, or documentors, or artists, even website designers. 
It's not uncommon for someone to put up a website for a project, 
someone will start using the code and say "Well, it's nice code, but
I can't program -- but your website is awful" [Laughter] And so you'll
find web designers and all sorts of people this way. 

The next thing you do is you make it work. And when it all works
nicely, you think about documenting it. Maybe. [Laughter] Alternatively,
you ship the product and hope that somebody writes the documentation
later. [Laughter] One good thing that's happened with KDE is that
people start to regard documentation as part of the program. It's
happening now with GNOME and the Gnome Documentation Project. 
But the fact that people are going, "This program isn't finished. The 
help button don't work" is good, because probably until two years ago, 
an application which had a help button -- a help button! --  was a major 
feature. You mentioned this to everyone: "This application does this, 
this and this, and has help!" For programmers it didn't matter: you read 
the source code. But if you wanted to go sort of beyond just computing 
people who actually have the time to read all the source code, you do 
actually need documentation. We're even starting to document the kernel 
now. Although this will be a very very long-term project. [laughter]


Communication [inaudible] some time later. Software engineering: the
communication between parts of the projects is very very hierarchical. 
It's best drawn as a tree. You put related areas of the project together, 
so you squash people who are working on shared interfaces, you put them 
under the same manager. And hopefully that means the amount of 
communication between them which gets through without being destroyed 
by somebody on the way, will be reasonably good.

Talking directly to other departments is bad. We actually got told
this at one large networking company I used to work for. "You are not
supposed to be talking to people from such-and-such division". The fact
that they already had code that did what we needed was apparently
completely unrelated to official corporate politics. And the other code
had been written on a different person's budget, so we weren't allowed
to use it. So what actually happened was that we talked to each other
anyway a little bit, although it's much harder to do that. It's very 
hard in big companies to know who to talk to because you don't have 
obvious figures standing out in your organisation that you can go to. 
But things get passed around quietly so there's a Chinese Whispers system 
where I found-- this is at IBM, I phoned up so and so and such and such a 
department is doing this. And you get this big company effect where you 
actually find out about your own company projects from people who work for
companies who are working with your company, because they say "Oh,
you're working for such and such? Yes, we're working with them on the
television decoder". "We do television decoders? I didn't know that!"
It happens in all sorts of companies: you get this "I found out
what my company is doing on Slashdot" syndrome in some technical companies. 


In free software, you have the reverse problem. You have mailing
lists. You have things like the debian-policy mailing list. Anything
you ever want to know about policy in the Debian project is on the
mailing list. The question is: where? [Laughter] And you have what
someone called "the information hosepipe". And what you actually 
need to do is to evolve ways of extracting information. So people
will read some mailing lists. A lot of other mailing lists, people 
will search the archives for, or one project member may be involved
reading a particular mailing list. So he will tend to say, "I saw
this. This looks interesting.." You can't actually read all of the
mailing lists. Even trying to read all of the linux-kernel mailing
lists is impossible. If you wanted to read every free software mailing 
list, you would probably need a very large network connection. That and 
a very large number of people to do the actual reading. 

You tend to have charismatic project leaders. That's partly I think
because of the information thing. You need people you can talk to who
know what is going on with all parts of the project. So you have
people like Miguel de Icaza or Linus. Everybody knows who they are
so you always know there is somebody you can go to to say "I'm thinking
of working on this for KDE, this for GNOME, this part of gcc." You 
know roughly who to go to, and they will say, "Well, you need to 
talk to this person, you may well find this mailing list interesting.."
and so these people just build connections. And you have to have people
there that everybody can find. You need to know who to talk to to
get involved in their project. 

Not all of them are charismatic. There are certain projects where you
know who to talk to because they are very very loud. Qmail is a
wonderful example. [Laughter] Everybody knows Dan. He's probably
not the most charismatic person in the world, but everyone knows
who to talk to if they want to do something with qmail. It's very
simple. 

Direct communication is common. You don't go and find a group
of people and say "Please tell so and so we're starting this part
of the project." You actually send mail saying "Who do I talk to?"
You'll get an answer, "Talk to these people". So communication becomes
much more direct. So information is passed much more reliably. 

The downside of this in free software is that a lot of things that
are going on don't get seen. So for example, a beautiful example of
this is GNOME and KDE. You see all of these public lists and people 
complaining and bitching at each other about GNOME isn't any good, 
KDE is crap. But if you look in the actual lists which are much harder 
to find, you have all these people going, "Well, how do we make bonobo 
and kparts work together?" or "How can I get KDE applets to live on a 
GNOME panel?" and the programming people doing all these things saying 
intelligent sensible things, but they're not as loud, and they're 
actually in direct communication with people, so they don't get seen. 

That can also mean that projects end up redoing work, because a
piece of work may be getting done, but the people who are doing 
the work are quiet about it. So you end up repeating the work
before you find out that the project is in progress. And often
what happens, if you post the first beta release of your software, 
then you get a mail from somebody saying "Why didn't you talk to
us?" in a very hurt tone. And you just didn't know they were there.

The design of the code is a bit different. Free software is very
very modular. I regularly take two or three megabytes of patches
for various parts of the kernel, apply all of them to my kernel
tree, and there are no clashes. Because patches affecting each
driver never affect the main code. The main code is very modular, so
if you touch the memory-management software, you don't normally
affect filesystems or drivers or networking. So the whole system 
is oriented about isolating all of the internal details, and 
creating very very modular code. It's important to do that. 

Another difference that's very important is, if you remember back
to the first slide, in a traditional software project, what happens
is you created a design specification. You specify the interfaces
right at the start of the project. In free software, you have what
I call group-owned interfaces, which means that every so often, 
people who are using the interface have a discussion, or a flame war,
or all the various shades in between, and between them they will
evolve or change the interface. And so instead of being fixed, it
gets broken in a very controlled manner. Everyone can break it
who's involved in using the interface, so long as the people
using the interface agree - or most of them agree - that the 
actual original design was wrong, the actual interface will get
changed, all of the drivers will break, and the drivers will all 
then get fixed. So we have this process whereby we're actively 
evolving the interfaces, and again, we have direct communications. 
All of these people end up talking or arguing directly with each 
other about the interface changes needed. It's very very different 
to the fixed nature of traditional project design.

Continuous evolution, except in Kansas, where it's prohibited by
law [laughter]: if you design something which it doesn't work very
well, people don't use it. If you design something which is complicated, 
people will write their own. It's easier sometimes to write your own code 
than read the manual. Assuming the manual has been written yet, which is 
another question [laughter]. So you get very simple interfaces because 
people who create complex interfaces write code that nobody else uses. 

The interfaces are also the reusable part of the system, and not normally
designed to be reusable. What actually happens: someone will write a 
piece of software which is vaguely-- it's fairly general. It's certainly
not a true software engineering reusable component. They only implement 
the bits they need. Someone else comes along a bit later, and they'll 
add the bits that they need. And over time this code will gradually
become more and more like a reusable component in traditional software.

So it becomes more and more generalised. Sometimes people will
rewrite it out of disgust on the way, because it gets more and
more unpleasant as people hack bits on it. But you evolve the 
interface, and what all this means is that nobody ends up writing
all of the interesting routines which aren't needed. 

So in a traditional object-oriented class, you have all of these
fascinating abilities, like for example the ability to delete an item
on a list while you are iterating the list. In a lot of free software
projects, someone's said "Well, we could do that, but nobody does it so
why bother?". So you tend to get smaller and cleaner code which does
everything people need rather than everything which is possible. 

We minimise the interfaces between modules. Simple interfaces. Because
if you write a complex interface, not only will people not use it, 
worse than that, they'll start sending you email asking how it works.
[laughter] And the more email you get about your interface, the more
likely you are to write documentation. Hopefully. But also the more it
indicates the interface is perhaps too complicated. Not always. There
are complicated interfaces which nobody has found a simple version of
yet. They probably exist somewhere. 

Another thing is that modules tend to talk to what we call the core of 
the system, to the core modules. In the kernel, drivers tend to talk 
to the filesystem-- sort of the core filesystem and memory management code. 
There are very few drivers that talk to each other. If you look at 
applications, they ten to talk to the libraries, but they don't tend 
to talk to each other, at least not in an informal manner. They will 
use things like CORBA, actually build real component-based interfaces, 
but at the deep code level, they are all calling into shared library 
services. So there is a very clean structure where pieces of code don't 
arbitrarily go and talk to each other.

Part of that is done by design; part of that is that it just happens
that way, because it's the only way to maintain this kind of software.
If you talk to large numbers of modules, you have to be on large 
numbers of mailing lists so you know about the changes to each module
you talk to. So the less non-core modules you talk to, the less 
time you spend arguing on mailing lists, the more time you spend
programming, and the less people you have to know to make your
project a success. Because if you're only working from the core 
libraries, you can just track some main mailing lists, like CORBA,
KDE or GNOME and you see people saying "Well the next major release
we're going to change this and that." "OK" If you start talking to 
the innards of other programs or other drivers, then every time 
somebody breaks the other program or has a Grand Idea and rewrites
bits, your program breaks mysteriously. And it's quite possible 
that the first thing you actually get to hear about it is when eighty 
people email you saying "I upgraded my gnome such and such and now my 
panel won't start" and you think "Oh my god, what have we done?" Whereas, 
because it's talking to the core of the system, you don't have to 
watch for this kind of application-to-application interfaces. You
can actually say, "These applications deal with this library. All 
that I have to care about is that I am aware of changes to the library
itself." It's much more controlled. 

And this is very close to traditional object-oriented programming. 
A lot of people in that environment favour the idea that objects s
should talk to core services, and if you have a lot of objects all 
talking to each other then you actually have a badly-designed system. 
So in this respect in many ways we agree with traditional software 
engineering. 


Reusability, I talked about slightly out of order. Oh well. I'd
forgotten this slide was up. 

Flexibility is another thing. Most free software projects are designed 
on the basis that other bright people will do clever things with the 
software. So we tend to have things like shell script interfaces, or 
there are ways for example in mailers, like the exim mailer has lots 
of nice facilities: like you can embed Perl into it. Because somebody 
along the way did a few things with Perl and realised  "A great way of 
solving the awkward cases in my mailing system is to embed [inaudible] 
Perl in it" So if you need to do something clever in exim, you can write 
it in Perl and put it in your mailer. 

Lots of packages have "calling the shell script" routines or macro
functions. We have languages like Tcl which is designed for this 
purpose. The idea is to get you very much flexibility in basically
misusing the application, the application's purpose, in new more
useful roles. 

This gets back to Brook's Law. 

Okay. I'm going to make the first claim I'm going to make. Free software 
is generally late. I'll explain why I say this. First of all, traditionally 
in software your marketing people tell you, "In nine months' time, lots
of people will want a package which does something." So you schedule
it so that in nine months' time, you release a package that does this.
Free software is very very different. What happens with most free software 
projects is that you see a neat toy in the shop, you plug it into your 
Linux box, and there isn't a driver for it. So at this point, you 
already want a driver. So by definition your software is already late. 
[laughter]. Because somebody else didn't write it for you, and you've 
now got to go and write it. 

This is true of applications: a lot of free software applications
are written because somebody thinks, "What I need right now is.."
A nice example to use which is even clearer is GNOME. The KDE
people started the KDE project, there was an interesting little
licensing argument which has thankfully gone away, and GNOME was a
response initially to KDE. So it would be fair to say that by any
stretch of the definition, GNOME was late. GNOME is very much -- was 
very much playing catch-up with the KDE project. 

Now, Fred Brooks claims that adding programmers to late projects makes
them later. The GNOME project added two hundred people to a late project
and it didn't make it later. So you have to look at the question, 
"What is actually going on here?" And I started to look at it. I 
went through chunks of GNOME CVS, I also looked at one or two other
projects, some of the KDE project, and I started to look at how these
things fit together. 

The first thing surprised me. The team sizes were very bounded. You
never found any project where there were teams of more than six people.
Even though most of these people probably knew nothing about software
engineering. There were two or three cases where I found larger ones. 
And then you read their mailing list archive and in actual fact they're 
working on two unrelated parts of an app-- a given application. So in 
other words, a team working on the graphics interface and a team working 
on the CD player interface. So in fact it's two teams of small groups of 
people. And I found near enough no exceptions to this. The other larger 
teams I found tended to have less than six active people and a few people 
who'd written bits of code and didn't contribute any more. So again, 
there's a small focused group of active people. 

The important thing: when you add more programmers to a free software
project, it splits. It seems almost instinctively the people involved 
split into sub-projects. So with GNOME, as all of these people kept 
coming into the GNOME project, instead of GNOME being simply just one 
GNOME thing that everybody hacked on, it because gnome-libs for the 
libraries and gnome-core for file manager stuff, and also playing with 
the neat little panel toys. As more people were added, all of these 
little panel applications became a separate project. Then some of these 
little panel applications were joined together and became much more 
complicated. People started writing quite clever little panel applets, 
and also borrowing some nice ones from Window Maker, which also works 
rather nicely with that. Getting your code started. And these individual 
[inaudible] started to have their own teams of programmers. And so the 
project was splitting neatly; and it was splitting along clean maintainable 
lines. It was always splitting where there were clear simple interfaces. 
So like with the panel applications, the interface between a GNOME panel 
application and the GNOME panel, much like the KDE application and the KDE 
panel which is a similar idea. It's a very simple, very clean, very fixed 
interface. So it's quite natural for the applications to separate from the 
panel itself, and then for these panel applications which don't talk to 
each  other to each become a separate project. 

Even within things like the libraries, individual people will become
responsible for a particular chunk of the library. So somebody was 
doing the score-handling for games, which is actually in the standard 
libraries because of security issues: your games to have to be running 
setuid so you can't cheat at the scoretable. Which is apparently very 
very important. [Laughter] Early versions of some of the GNOME applications 
had world-writable score files, and people filed large numbers of bugs 
complaining that  they could cheat at solitaire. [Laughter] So it ended 
up as a library. It inherited a particular person who was only interested 
in the score file stuff. Other people were only interested in things like 
dotfile-parsing, handling all the configuration files. People were 
interested just in graphical areas. There were people who were only 
interested in things like menu bars. That's-- all they ever played with 
was menu bars and putting things on menus. And there's a clean interface 
between menus and the application. So again it makes sense: they are 
splitting things along very very clean interfaces. And this appears to 
be happening without anybody planning it. There is either somebody very 
very well-hidden running around all these free software projects dropping 
hints who has no email address; or people are figuring this out purely 
by instinct. 

There are downsides. We tend to copy code. If you look at Linux serial 
drivers, a lot of them are written by taking the existing serial driver 
and saying "Well, this driver is the nearest to our hardware" so you make 
a copy of it, because you don't want drivers talking to each other: 
because otherwise every time Cyclades updates their driver, your serial 
card driver breaks. And if you start fiddling with the innards of the 
Cyclades driver, Cyclades will probably get quite upset. So you make a 
copy of it. What you inadvertently do then is make a copy of all the 
remaining bugs in it. So sometimes we get a situation where somebody 
will write in and say "If I do this with a particular serial driver, 
it breaks". And so you fix this driver. And you start looking. And you 
fix one, two... maybe somewhere in the region of five or seven other 
drivers at the same time. And this is one of the reasons we keep the 
kernel as a single very large tar file. Because in the nature of it, 
this copy of code and this lack of communication between free software 
users means that if Cyclades fix a bug in their serial driver, the chances 
are that people working on other serial drivers are not on the Cyclades 
mailing list. So they may well miss these changes. And you'll get a 
situation where bugs are being fixed in different drivers, and people 
are having to repeat not the easy bit -- fixing a bug is the easy bit. 
People are having to repeat the problem of finding the bug in each 
individual driver, which is something you don't want to do. So you 
actually do have to make an effort. This is something where unless 
people make an effort, it doesn't seem to come out right, which is to 
make sure that when you change one driver, you know which drivers are 
effectively derived from it. And you have a look at those and you can 
send email prodding all the right people saying, "We found a bug. We 
know your driver is based on our code. Please check if you have the same 
problem". 

That's happened with serial drivers. It's happened with sound drivers.
And it's happening in other areas of the kernel. It'll also happen in 
applications: people will copy the basis, for example, of a simple KDE 
or or GNOME application. Because any application you're going to write 
has got a menu bar, a tool bar, an about box, I would hope a help button... 
[Laughter] So you might as well start with another existing application, 
take out all the rest of it but the menu bar, your buttons, and can you 
put your menu editors on it. Then you need to start writing the inside 
of your application. 

Modularity. We are aggressively modular. We spend a lot of effort
in Linux, free software, trying to make very simple interfaces. We 
want people to be able to reuse our code. They can only reuse our 
code or build on our code if they can understand how it works. 
Or at the very least understand how to use it. 

Complexity is generally a bad thing. There are lots of cases where
we use less complex algorithms. We may take a tiny tiny performance
hit from it, but it means that people can actually maintain the code
long-term. If you put a really clever complex idea into a piece of
code, and that person then leaves the project, what generally happens
is: no-one's quite sure how it works. Nobody wants to fiddle with this
piece of code. And so you start to evolve some compatibility layers 
around this code. Instead of changing it to do what you now need, 
you've actually got a layer which converts between what you used to
need and what you need to do now. And so you get this very very 
messy code if you're not careful. 

Subtle surprises are worst. So we don't care about breaking things. 
When we change an interface, we try and change the interface in such 
a way that every single application or driver using it breaks. Ideally
it won't even compile. Because if you make an interface change, and
someone tries to compile a driver or an application and it breaks, it
won't compile, they immediately know they need to go and fix their
application. If you compile your application, it works beautifully and 
every half an hour or so it crashes, it's very unlikely you're going to 
think "Ah. Somebody has changed the library interface". It's unfortunately
all too likely to be a cause, but it's not the kind of-- it's not the
first thing you think of. 

And so it's much better, when we change the interfaces, not to try and 
be very clever, making it extremely backward-compatible, but in fact to 
say, "Okay, here's a clear point. It's between two major revisions of 
our software. We've already told people we will be breaking some things. 
I'm going to deliberately completely and utterly break this interface".
And then you start to fix the software to use the new version of the
interface. And we do that a lot. Especially with the kernel. 

And you see it a lot with libraries, as well. Rather than change the 
behaviour of a library function, you create a completely new library 
function with a new name. And after a while, you delete the old function, 
so the code simply won't link. You do not want people to miss changes. 

We grow by splitting modules into two. By changing the various
specification of the interfaces. Changing the structure of the design,
changing the interfaces, in most traditional software engineering,
is regarded as one of the worst crimes you can commit. You're allowed 
to do some local changes, so splitting an end-module into two because 
"it's harder than we thought" is fine. But actually changing the way 
pieces of programs talk to each other and doing it in a way which breaks 
other code intentionally is very very different to what software 
engineering teaches people. But it works. 


So, the fundamental differences: 

We have all this ability to communicate. We have email, and mailing
lists. But most of what it's about is trying to avoid having to
read all of these mailing lists. Avoiding having to read documentation.
It's actually about minimising the amount of communication you have to
do, and the amount of-- particularly the amount of co-ordination between
parts of the project. 

So contrary to what you might think, what we actually do because we have 
all this communication ability, is to reduce the amount of communication 
we do. 

There is a difference: all the information is available: you can go and 
look through mailing list archives and search things, which is very 
different perhaps to a lot of companies. But the goal is completely 
the reverse of the one you may think. The goal is not complete 
communication. Complete communication is how you get the debian-policy 
list. It's how you get a lot of these sort of big, big lists which are 
very slow to decide anything because there are just too many people 
involved. 

The management structure is a graph. There is no clear hierarchy.
And for GNOME-- for KDE things you might talk to particular people
at the top of the KDE project, but these people, when they're doing
something which is not KDE, will be talking to people who they see
as the higher-ups in a different project. So it really depends what
you're doing on how someone sort of fits into the hierarchy. If
you're doing a GNOME application, Miguel de Icaza is at the top of
the hierarchy. When Miguel was writing graphics code for the SGIs, 
Miguel was the bottom of the hierarchy and some of the X developers
were the top of the hierarchy. And Miguel existed in both of those
places at the same time, so he was the big [inaud] of GNOME and some
random person working on the X server for the SGI. So it's not a clear 
tree. It all all depends on what you're trying to achieve, who is the
important people within any idea. Which project you are currently
working on determines how people fit in the, sort of, hierarchy. 

You continuously change parts of the hierarchy. You change who
you talk to, and what you talk about. So as two pieces of software
realise they have things they can share, they start to talk to 
each other a lot. When they maybe realise afterwards they were
wrong about the sharing, then the link between the two project
teams will disappear. 

And finally, the people who are project leaders spend a lot of
time not going to meetings, talking about things, but actually
telling particular developers, "You need to talk to this developer,
you need to talk to that person". And the kind of skill these
people tend to have are remembering what hundreds of people are
doing. So in many ways it's not about being a good programmer, 
it's about knowing, and being able to think, "Video. Ah. kwintv. 
that's so and so." When someone comes to you and says, "I'm working 
on a card game": "So and so has been doing the artwork for a card game, 
so and so has been writing the card game menu, this person has been 
writing scoring code, just remembering, being able to say to people, 
"Talk to him, talk to him, talk to him" And so they are --  they're -- 
people are building links, They're almost like routing things in a 
network of human beings, rather than actively passing messages 
themselves. 

We've talked about most of this. Um. Okay. Simple interfaces. Very
modular. Again, it's about reducing the amount of things you need
to know. And in free software, all code is reusable. It may not be
modular or clean code, but there is nothing you cannot fix with an
editor. You have all of the source code so you can steal things 
quite legitimately, however small, however large, from any project.

It's good practice to ask people who are running that project, 
because they probably would prefer you don't split code off, but 
it doesn't really matter how narrowly-designed the code originally 
was. You can always come along and say, "I have another use for 
this code". The ability is always there. 

Frankly, you can write ugly code in free software, contrary to every
claim otherwise. What needs to happen though is that the interfaces
must be clean. So long as people are seeing cleanly how to use the
code, or at worst, how to write new code for the same interfaces, 
they don't tend to worry if the code inside is currently very ugly. 
The interface is nice. The code does the job. Someone has already 
written it. You don't really want to write it again. So you just keep 
quiet about the mess [laughter]. Because you know the interface is 
good, you know that someone, someone else of course, can always come 
along and rewrite the messy code and replace the module if it's ever 
necessary. 

And so one thing which very very surprised me: free software design,
I think, is actually about avoiding knowledge. It's very much about
not having to know things. And that's sort of my, my final conclusion
on that part of it. 


As far as Fred Brooks' "The Mythical Man-Month" goes, it's hard to
say whether he's right or wrong that adding people to a project 
makes it later. Because it depends how you -- what you think of as
the project. In free software projects, you don't actually add
people to a project. You add people to the overall thing. So you
could be adding to the project: it's not clear. But you create
more projects. And so the question is whether or not that counts
as following his claims or breaking them. And I don't think there is
an easy answer. 

Certainly however, we can add people to large problems and create
more projects and more software, and we don't suffer from some of
the traditional problems where a project is late, you add lots of
people, and your project becomes far far later. In part because we
don't spend a lot of time training people. There is a sort of 
assumption that if you're going to join in the project, you will
read the mailing list, you will read the code, and you will figure it out. 
It does create an entry barrier sometimes. So you get to the point
where you can't find programmers: and this is why simple interfaces
become so important. If your interfaces are simple, you don't have 
to answer hundreds of emails from people saying, "I'd like to get involved
but I cannot understand what this function does" or "I cannot
understand how this code can possibly work" Because if you ask
people and you have to answer all those emails, you really will
slow the project down. If your interfaces are simple, then people
hopefully can figure it out themselves. Especially if there's even 
the documentation. And that means they add themselves to the project
without interfering with the existing people working on it, and
that's [inaud] help very much. 

And that's sort of the end of the talk part of this. I wanted to
leave a little time for questions. So... any questions? 

[Applause]