Re: A modest proposal -- We need a patch penguin

Rob Landley
2002-03-01T19:03:32Z

I'm on the road, so this reply probably won't go out for a couple days, but
I'll queue it in my outbox anyway...

On Friday 22 February 2002 11:06 am, Hans Reiser wrote:
> We need to move from discussing whether Linus can scale to whether the
> Linux Community can scale.
>
> Every organization needs to have clearly defined algorithms for
> determining what work is done by who. For the linux community, our work
> consists in part of reviewing patches. Incoherent inconsistent
> delegation is the only reason why we are having scaling problems. We
> have a consistent recurring problem (yes, I know, a few lucky folks like
> me don't have this problem, but it is clear to see that WE as a
> community have this problem).
>
> It is important that there be a consistent feeling among patch
> submitters that they know where to send their patches for
> acceptance/rejection. There should be NO patches which go out, and not
> even a rejection comes back.

This is what the various patchbot projects are trying to address.
(Unfortunately, they seem to have gotten the idea that they need a complete
solution to all possible cases before deploying anything. Just a filtered
patches-only mailing list would be a start. I'd put one up if I had a
server, but as I said I'm moving this month. If nobody else has done it by
march, I might.)

The current Linux community structure seems to be four tiers. Developers,
maintainers, lieutenants, and linus. Linus listens to lieutenants,
lieutenants accept from maintainers, and maintainers accept from developers.

The confusion seems to be that until recently, many maintainers didn't know
who the lieutenants were (who the people Linus actually listens to are, to at
the very least explicitly reject patches once these people have reviewed,
approved, and forwarded them). Hence stuff was getting to maintainers and
then being dropped when forwarded straight to Linus. Linus still hasn't
quite enumerated his lieutenants, but now that people know they exist I
expect they'll become apparent eventually...

> Every organization has clearly defined procedures for allocating the
> flow of work. It is called a management structure. That is what we
> need, and we need a formal well defined and externally visible one. An
> informal undefined network of friends is just not suitable for an
> organization where the flow of email is as large as it is in the Linux
> Community.

It's not a binary state. The fact we need a little more structure doesn't
mean anybody has to start filling out paperwork and blindly following
procedures. :)

The "a little more structure" could be a "how to submit patches" FAQ entry
that says:

1) Develop patch, test, get community feedback if necessary to make sure it
works.
2) Submit to maintainer, get them to review and sign off. Resolve any issues
they have before proceeding.
3) Submit to lieutenant (the maintainer will tell you who this is), get them
to review and sign off. Resolve any issues lieutenant has before proceeding.
4) Submit to Linus, with appropriate endorsements.

If Linus ignores everybody except Lieutenants, that's probably workable as
long as he DOESN'T ignore the lieutenants and people know who the lieutenants
are, and the lieutenants don't ignore the maintainers and the maintainers
don't ignore the developers.

If Linus has two levels of sturgeon's law filters (maintainers and
lieutenants) before he has to explicitly reject something, then asking him
(nicely) to at least reply thumbs up/thumbs down on patches forwarded to him
("bad idea", "fix this", "do it this way instead") by the dozen or so people
he trusts shouldn't overload his bandwidth. (Whether or not he'd actually do
it is still up to him, but that strikes me as the minimum workable long-term
solution.)

So a developer would at least know who they have to please next (maintainer,
lieutenant, or linus) to forward their patch. It still might be a lot of
work to go through the long way, and Linus would probably still accept
interesting patches directly. But the failure case of "my patch is getting
ignored" would have a procedure to go through to get explicitly rejected by
the appropriate person. :)

By the way, sometimes the answer honestly is "I'm busy, submit again after
2.5.7". Which is still better than being ignored. (Stuff like the ALSA
drivers: "Good idea, not now." Ok: When?)

> Linus, I would like you to stop saying that you cannot scale to where
> you can read every email, and start determining what it takes to make
> the Linux Community infrastructure underneath you responsive to patches.

Linus (and the lieutenants under him) have not been the ones experiencing the
problem. Linus accepts all the patches he wants to, and the lieutenants tend
not to be ignored. The top of the pyramid is not where the motivation for
change is most strongly felt...

> Bitkeeper is a great start, but you also need to create a management
> structure and interface that is clearly defined to the external
> community. Saying that the maintainers list is ignored by you means
> that you need to create something that is not ignored by you. You also
> need to create a system (bitkeeper can perhaps help, Larry?) for
> tracking who fails to respond to patches, and (after a few warnings)
> remove them as maintainers.

I don't expect Linus needs to do any of the grunt work here. He just needs
to sign off on the design and actually use the finished solution.

We've got some time on this. If Bitkeeper allows Linus to move to a "pull"
model with his lieutenants, that should allow the system to scale enough to
get 2.6 out the door. It's the layers under those guys who need to shuffle
around to feed their patches into those bitkeeper trees that Linus pulls from.

Of course to make this work, Bitkeeper will somehow need to let Linus
cherry-pick patches from the bitkeeper trees under him and reject others.
I've tried to follow the discussion on this front but I'm not convinced it's
resolved yet. But as I said, there's time...

> Our problems are not novel. Let us apply standard business school
> methodologies to them.

If I remember my kernel-traffic summaries correctly, Eric Raymond was saying
something like this about two years ago. Something about the boy genius
effect? :)

> Hans

Rob