[Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

In the spirit of having a more process than technical based kernel
summit, I'd like to put the topic of the kernel Janitors project up for
discussion.

In the early days, the project was conceived as a way of getting fresh
blood into kernel development by giving them fairly simple but generally
useful tasks and hoping they'd move more into the mainstream.  If we
wind forwards to 2008, there's considerable and rising friction being
generated by janitorial patches.  This is only an example:

http://marc.info/?l=linux-kernel&m=121135889328760

but there are many more.  The greatest problem, as I see it is that by
pouring vitriol like this on newbies, we're really damaging our
reputation as a community that welcomes newcomers and strangling our
necessary supply of willing volunteers.  On the other hand, as a
maintainer, when there's people yelling me at about patches not being
included plus a persistent regressions list and about ten bug reports to
track down, the last thing I want to see within a million miles of my
inbox is a white space fixing patch.  The more of these patches we get,
the worse the problem becomes and the shorter and more inflammatory the
responses get. We can't go on like this.

The most obvious solution might be to shut the Janitors project down, or
at least more tightly manage its TODO list (although a lot of what gets
seen as janitorial patches, like whitespace fixes, isn't on the TODO
list in the first place).  However, since the purpose is to get new
people involved with kernel development, perhaps we should repurpose the
project so it actually does this.  My suggestion is that we replace it
with the kernel bugs project.  Kudos for finding bugs, more for finding
better ways of finding bugs, and the most for finding and actually
fixing a bug.  

Perhaps we should simply start the discussion with the premise that we
want to encourage new people to do useful work and draw them into the
development community and see where it leads.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[H. Peter Anvin]

James Bottomley wrote:
> 
> but there are many more.  The greatest problem, as I see it is that by
> pouring vitriol like this on newbies, we're really damaging our
> reputation as a community that welcomes newcomers and strangling our
> necessary supply of willing volunteers.  On the other hand, as a
> maintainer, when there's people yelling me at about patches not being
> included plus a persistent regressions list and about ten bug reports to
> track down, the last thing I want to see within a million miles of my
> inbox is a white space fixing patch.  The more of these patches we get,
> the worse the problem becomes and the shorter and more inflammatory the
> responses get. We can't go on like this.
> 

As far as whitespace fixes, it seems we should just declare a flag day, 
specifically, right before Linus releases 2.6.26 or .27 he'd run 
"cleanfile" on the WHOLE TREE and check it in (either as one patch or 
several.)

After that, one would either have to use "patch -l" to refresh patches 
across the flag day, or I would write a reverse version of "cleanpatch" 
(one which cleans the minus lines and the context instead of the output) 
to fix them up.

Then the whole tree would be whitespace-clean, it would concentrate the 
pain to a single event, and then we can go on with life.

	-hpa

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Rik van Riel]

On Wed, 28 May 2008 12:20:12 -0500
James Bottomley <James.Bottomley@HansenPartnership.com> wrote:

> The most obvious solution might be to shut the Janitors project down, or
> at least more tightly manage its TODO list 

I've had some limited success with:

http://kernelnewbies.org/KernelProjects

One problem is that people end up beginning with a project but
for some reason stop working on it later - with no indication
as to whether the project ended up being too difficult, or they
ran out of time, or something else happened...

-- 
All rights reversed.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Chris Mason]

On Wednesday 28 May 2008, Rik van Riel wrote:
> On Wed, 28 May 2008 12:20:12 -0500
>
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> > The most obvious solution might be to shut the Janitors project down, or
> > at least more tightly manage its TODO list
>
> I've had some limited success with:
>
> http://kernelnewbies.org/KernelProjects
>
> One problem is that people end up beginning with a project but
> for some reason stop working on it later - with no indication
> as to whether the project ended up being too difficult, or they
> ran out of time, or something else happened...

I think our mistake is the assumption that everyone who wants to contribute to 
the kernel wants to code, or that everyone wants to end up at the top of the 
major feature contributors list.  For lots of people, we're going to be that 
experiment from college they never quite want to admit to later on in life.

Looking at the KernelProjects link, bugzilla is at the buttom and janitors is 
at the top.  I've always thought the janitors project was important, but 
maybe we want to change the emphasis around a bit.

The regressions page on kernelnewbies is out of date, and there are many more 
howtos on coding than on bug hunting.  It doesn't mention linux-next 
anywhere.

I'm not picking on kernelnewbies, it is one of our best resources.  But, going 
back to James' ideas, introducing people to bug hunting is a better way to 
involve them in the community.

Maintainers are easier to interact with when you send good bug reports along 
with a clear way to reproduce it and perhaps a bisection

Getting all of the above is often 95% of the work of fixing it, people are 
most likely to jump into the coding when they've found a bad patch and start 
to wonder if they can fix it themselves.

Along those lines, how about a kernel bug hunting challenge.  The top 
contributor(s) to creating bugzillas that get solved get a new pc, laptop, 
high end graphics card, ssd device...whatever.  We could send the best 
testers hardware that breaks most often...everyone wins.

Other prizes could include tickets to a linux conf of choice (not airline 
tickets, just free entry).

One motivation here is to bring bug reporters from active distro communities 
into testing mainline kernels as well.

-chris

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Jiri Kosina]

On Wed, 28 May 2008, Chris Mason wrote:

> One motivation here is to bring bug reporters from active distro 
> communities into testing mainline kernels as well.

My experience from handling both distro-reported and upstream-reported 
bugs shows that these two worlds/communitites are really quite different 
in their nature.

Bug reporters who report bug in the vanilla kernel are usually perfectly 
fine when someone sends them patch to test, they don't have any problem 
recompiling the patched kernel, testing, and reporting back.

This is not the case with distro kernel bug reporters at all. You usually 
can't just tell them "hey, this patch might fix it, report back if it 
does". Vast majority of theese users expect the kernel package built for 
their distro to be provided, so that they can easily install it and test 
it, but requiring any other effort usually doesn't work.

Just my $0.02.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Pavel Machek]

Hi!

> > One motivation here is to bring bug reporters from active distro 
> > communities into testing mainline kernels as well.
> 
> My experience from handling both distro-reported and upstream-reported 
> bugs shows that these two worlds/communitites are really quite different 
> in their nature.
> 
> Bug reporters who report bug in the vanilla kernel are usually perfectly 
> fine when someone sends them patch to test, they don't have any problem 
> recompiling the patched kernel, testing, and reporting back.
> 
> This is not the case with distro kernel bug reporters at all. You usually 
> can't just tell them "hey, this patch might fix it, report back if it 
> does". Vast majority of theese users expect the kernel package built for 
> their distro to be provided, so that they can easily install it and test 
> it, but requiring any other effort usually doesn't work.

I guess this would be helped if we actually made compiling distro
kernel easy.

Default suse installation does not even contain gcc; I'm not sure if
we provide our kernel in easy-to-use git form, and I don't think many
suse people would be able to create kernel .rpm without help of
autobuild system... autobuild only works behind suse firewall and
buildservice is not really right answer either.

Having compile-me-kernel script that does all the neccessary steps of
compiling/installing distro kernel with custom patches would be great
way forward...

Plus, we do very little q&a on distro kernel w.r.t to non-standard
.config files. Improving that would be some work, but there would be
chance for 'disable CONFIG_NO_HZ and see if it helps' kind of
debugging.

Oh and having make prepare-kernel-for-this-distro in vanilla would
help a lot, too. Usually, vanilla boots and works on distro with
suitable config, but finding that config is not too easy.
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Wed, 2008-05-28 at 15:08 -0400, Rik van Riel wrote:
> On Wed, 28 May 2008 12:20:12 -0500
> James Bottomley <James.Bottomley@HansenPartnership.com> wrote:
> 
> > The most obvious solution might be to shut the Janitors project down, or
> > at least more tightly manage its TODO list 
> 
> I've had some limited success with:
> 
> http://kernelnewbies.org/KernelProjects
> 
> One problem is that people end up beginning with a project but
> for some reason stop working on it later - with no indication
> as to whether the project ended up being too difficult, or they
> ran out of time, or something else happened...

Isn't the kernel mentors project supposed to help us at least get
feedback on that problem (if not prevent it altogether)?

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Woodhouse]

On Wed, 2008-05-28 at 12:20 -0500, James Bottomley wrote:
> In the early days, the project was conceived as a way of getting fresh
> blood into kernel development by giving them fairly simple but generally
> useful tasks and hoping they'd move more into the mainstream.  If we
> wind forwards to 2008, there's considerable and rising friction being
> generated by janitorial patches.  This is only an example:
> 
> http://marc.info/?l=linux-kernel&m=121135889328760
> 
> but there are many more.

The example that sums it up for me is this one:

	http://lkml.org/lkml/2008/5/18/20

We have people making minor cosmetic changes, and not paying even the
_slightest_ attention to what they're doing. This one's a particularly
scary example because it's something even the most non-technical person
should have spotted; there's _no_ excuse. It's the cosmetic equivalent
of a naïve warning fix that leaves the actual bug in place.

I think you're right that the status quo is damaging, and I don't see it
getting any better with the current quality of 'janitoring'. I think the
only way we can salvage anything useful from the janitors project is to
keep a close rein on what tasks are actually undertaken.

But we've pushed back on people doing this kind of thing before, and
pointed them both at the obvious things they've missed in the context of
their patches, and other more useful things they could be doing -- but
we've often received responses along the lines of "but I don't want to
have to _think_!".

It's hard to know where to go from there, and it's not exactly
surprising that we end up frustrated.

-- 
dwmw2

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Wed, 2008-05-28 at 23:49 +0300, David Woodhouse wrote:
> On Wed, 2008-05-28 at 12:20 -0500, James Bottomley wrote:
> > In the early days, the project was conceived as a way of getting fresh
> > blood into kernel development by giving them fairly simple but generally
> > useful tasks and hoping they'd move more into the mainstream.  If we
> > wind forwards to 2008, there's considerable and rising friction being
> > generated by janitorial patches.  This is only an example:
> > 
> > http://marc.info/?l=linux-kernel&m=121135889328760
> > 
> > but there are many more.
> 
> The example that sums it up for me is this one:
> 
> 	http://lkml.org/lkml/2008/5/18/20
> 
> We have people making minor cosmetic changes, and not paying even the
> _slightest_ attention to what they're doing. This one's a particularly
> scary example because it's something even the most non-technical person
> should have spotted; there's _no_ excuse. It's the cosmetic equivalent
> of a naïve warning fix that leaves the actual bug in place.
> 
> I think you're right that the status quo is damaging, and I don't see it
> getting any better with the current quality of 'janitoring'. I think the
> only way we can salvage anything useful from the janitors project is to
> keep a close rein on what tasks are actually undertaken.
> 
> But we've pushed back on people doing this kind of thing before, and
> pointed them both at the obvious things they've missed in the context of
> their patches, and other more useful things they could be doing -- but
> we've often received responses along the lines of "but I don't want to
> have to _think_!".
> 
> It's hard to know where to go from there, and it's not exactly
> surprising that we end up frustrated.

Right, but that's why I think we have to change the process.  If we keep
the Janitors project, then the bar has to be raised so that it becomes
more participatory and thought oriented (i.e. eliminate from the outset
anyone who is not going to graduate from mechanical changes to more
useful ones).  That's why I think bug finding and reporting is a better
thing to do.  There are mechanical aspects to finding bugs but it is a
useful service.  Bug fixing is participatory because we usually do quite
a lot of back and forth between the reporter and the fixer and at the
end of the day quite a few people get curious about how the bug was
triggered in the first place and actually go off and read the code.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Pekka Enberg]

Hi James,

On Thu, May 29, 2008 at 12:01 AM, James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
> Right, but that's why I think we have to change the process.  If we keep
> the Janitors project, then the bar has to be raised so that it becomes
> more participatory and thought oriented (i.e. eliminate from the outset
> anyone who is not going to graduate from mechanical changes to more
> useful ones).

I'm not sure what you expect to happen if we "shut down" the Janitors
project. The important janitorial work doesn't just magically
disappear. For example, we still need people for:

  - Fixing API misuse
  - Converting code from old APIs to new ones
  - Consolidating duplicate code
  - Fixing error handling code
  - Removing unused code
  - De-obfuscating code (e.g. removing bad macro magic, etc.)

And, quite frankly, I don't see what the big fuss is about. I know
we've had way too many "whitespace cleanup" patches in the past six
months or so, but can't we just NAK them politely and be done with it?

                        Pekka

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Thu, 2008-05-29 at 00:31 +0300, Pekka Enberg wrote:
> On Thu, May 29, 2008 at 12:01 AM, James Bottomley
> <James.Bottomley@hansenpartnership.com> wrote:
> > Right, but that's why I think we have to change the process.  If we keep
> > the Janitors project, then the bar has to be raised so that it becomes
> > more participatory and thought oriented (i.e. eliminate from the outset
> > anyone who is not going to graduate from mechanical changes to more
> > useful ones).
> 
> I'm not sure what you expect to happen if we "shut down" the Janitors
> project. The important janitorial work doesn't just magically
> disappear. For example, we still need people for:

I'm just outlining the possible solutions; shutting it down wasn't the
one I advocated. One can argue that janitorial changes with enough
intrinsic value tend to get done anyway regardless of whether we have
the project or not.

>   - Fixing API misuse
>   - Converting code from old APIs to new ones
>   - Consolidating duplicate code
>   - Fixing error handling code
>   - Removing unused code
>   - De-obfuscating code (e.g. removing bad macro magic, etc.)
> 
> And, quite frankly, I don't see what the big fuss is about. I know
> we've had way too many "whitespace cleanup" patches in the past six
> months or so, but can't we just NAK them politely and be done with it?

The problem is that there's something in the way all of this is working
that's causing politeness to get short shrift. In turn that's giving
lkml a larger than normal reputation for being a free for all dog fight
and discouraging potential contributors from coming forwards.

Appeals to be politer tend to only work in the short term (having given
quite a few of them).  I think we're developing a root cause problem in
the way we recruit people to work in the kernel and we have to think
about fixing it there.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Woodhouse]

On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> Appeals to be politer tend to only work in the short term (having given
> quite a few of them).  I think we're developing a root cause problem in
> the way we recruit people to work in the kernel and we have to think
> about fixing it there.

Do we really have a problem recruiting people to work in the kernel? On
what do you base that observation?

On a similar note, do we have any real data on the question of whether
those who are volunteering the patches which raise so much ire would
_ever_ become productive members of the team, even if we were to nurture
them properly?

-- 
dwmw2

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Thu, 2008-05-29 at 01:18 +0300, David Woodhouse wrote:
> On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> > Appeals to be politer tend to only work in the short term (having given
> > quite a few of them).  I think we're developing a root cause problem in
> > the way we recruit people to work in the kernel and we have to think
> > about fixing it there.
> 
> Do we really have a problem recruiting people to work in the kernel? On
> what do you base that observation?

Yes, the median age of the MAINTAINERS is rising.  Not quite at the rate
of a year per year which would show we have practically no turn over,
but it is rising.

However, even if there were no recruitment problem at all, getting more
people involved is always better because it means more contributions.
And contributions (useful ones) are the lifeblood that moves the kernel
forwards.

> On a similar note, do we have any real data on the question of whether
> those who are volunteering the patches which raise so much ire would
> _ever_ become productive members of the team, even if we were to nurture
> them properly?

I don't think so.  But that's not really what I'm saying.  I'm saying we
need to make the process of encouraging useful contributors more
streamlined (as in less aggro on the mailing list).  If that involves
cutting out the less useful ones earlier, so be it.  If we can come up
with a better conversion process, that's great too.  I want to start the
discussion, not necessarily prescribe the solution.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Greg KH]

On Wed, May 28, 2008 at 05:35:27PM -0500, James Bottomley wrote:
> On Thu, 2008-05-29 at 01:18 +0300, David Woodhouse wrote:
> > On Wed, 2008-05-28 at 16:42 -0500, James Bottomley wrote:
> > > Appeals to be politer tend to only work in the short term (having given
> > > quite a few of them).  I think we're developing a root cause problem in
> > > the way we recruit people to work in the kernel and we have to think
> > > about fixing it there.
> > 
> > Do we really have a problem recruiting people to work in the kernel? On
> > what do you base that observation?
> 
> Yes, the median age of the MAINTAINERS is rising.  Not quite at the rate
> of a year per year which would show we have practically no turn over,
> but it is rising.

My raw numbers show that the number of individual kernel contributors
continues to increase with every release, so this might not be as much
of a problem as it's made out to be.

> However, even if there were no recruitment problem at all, getting more
> people involved is always better because it means more contributions.
> And contributions (useful ones) are the lifeblood that moves the kernel
> forwards.

I agree.

thanks,

greg k-h

RE: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Luck, Tony]

> My raw numbers show that the number of individual kernel contributors
> continues to increase with every release, so this might not be as much
> of a problem as it's made out to be.

That depends on whether we are gradually adding to the pool
of developers, or seeing an increasing stream of newcomers
who supply a patch or two before disappearing again.

If you look at the list of contributors some old release for
which we have good data (say 2.6.16).  How many of those people
contributed to each of the following releases? Does the
decay curve look steeper or more gentle if you start from
a more recent release?

-Tony

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Greg KH]

On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
> > My raw numbers show that the number of individual kernel contributors
> > continues to increase with every release, so this might not be as much
> > of a problem as it's made out to be.
> 
> That depends on whether we are gradually adding to the pool
> of developers, or seeing an increasing stream of newcomers
> who supply a patch or two before disappearing again.
> 
> If you look at the list of contributors some old release for
> which we have good data (say 2.6.16).  How many of those people
> contributed to each of the following releases? Does the
> decay curve look steeper or more gentle if you start from
> a more recent release?

I don't know, I haven't tracked the people individually that way, only
looked at the basic numbers of developers per release.

thanks,

greg k-h

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Dave Jones]

On Wed, May 28, 2008 at 05:36:57PM -0700, Greg Kroah-Hartman wrote:
 > On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
 > > > My raw numbers show that the number of individual kernel contributors
 > > > continues to increase with every release, so this might not be as much
 > > > of a problem as it's made out to be.
 > > 
 > > That depends on whether we are gradually adding to the pool
 > > of developers, or seeing an increasing stream of newcomers
 > > who supply a patch or two before disappearing again.
 > > 
 > > If you look at the list of contributors some old release for
 > > which we have good data (say 2.6.16).  How many of those people
 > > contributed to each of the following releases? Does the
 > > decay curve look steeper or more gentle if you start from
 > > a more recent release?
 > 
 > I don't know, I haven't tracked the people individually that way, only
 > looked at the basic numbers of developers per release.

Are you just doing something like
 git log v2.6.16..v2.6.17 | grep ^Author: | sort -u| wc -l  

or do you have some script that maps addresses to people ?
One person may appear once in 2.6.20, and then a half dozen times
in 2.6.21 if they use multiple email addresses for example.
(Also, typos, and people using full hostnames in their sign-off's
 instead of email addresses skew this somewhat).

I'm guessing the latter, due to the graph thing you did. Pointers?

	Dave

-- 
http://www.codemonkey.org.uk

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Greg KH]

On Wed, May 28, 2008 at 09:00:55PM -0400, Dave Jones wrote:
> On Wed, May 28, 2008 at 05:36:57PM -0700, Greg Kroah-Hartman wrote:
>  > On Wed, May 28, 2008 at 04:23:52PM -0700, Luck, Tony wrote:
>  > > > My raw numbers show that the number of individual kernel contributors
>  > > > continues to increase with every release, so this might not be as much
>  > > > of a problem as it's made out to be.
>  > > 
>  > > That depends on whether we are gradually adding to the pool
>  > > of developers, or seeing an increasing stream of newcomers
>  > > who supply a patch or two before disappearing again.
>  > > 
>  > > If you look at the list of contributors some old release for
>  > > which we have good data (say 2.6.16).  How many of those people
>  > > contributed to each of the following releases? Does the
>  > > decay curve look steeper or more gentle if you start from
>  > > a more recent release?
>  > 
>  > I don't know, I haven't tracked the people individually that way, only
>  > looked at the basic numbers of developers per release.
> 
> Are you just doing something like
>  git log v2.6.16..v2.6.17 | grep ^Author: | sort -u| wc -l  

Ah, if it were only so easy :)

> or do you have some script that maps addresses to people ?
> One person may appear once in 2.6.20, and then a half dozen times
> in 2.6.21 if they use multiple email addresses for example.
> (Also, typos, and people using full hostnames in their sign-off's
>  instead of email addresses skew this somewhat).
> 
> I'm guessing the latter, due to the graph thing you did. Pointers?

Yes, it's the later.  Jon Corbet has a great little python tool that we
have used to create the "who is writing the kernel" series of articles.
I've been using it to keep track of who maps to what email address and
for what company for a while now.

An older version can be found at:
	http://www.kernel.org/pub/linux/kernel/people/gregkh/kernel_history/
it's the 'gitdm' tool there.  I don't know if he has an updated version
around anywhere, I suppose as it looks like I'm doing the releases for
it, I can put up a new snapshot if people are really interested.

I also have "cleaned up" versions of the kernel log files for just the
reason you say above.  You would not believe the number of times some
people mispell their own name in a single kernel release...  That makes
it easier to do this kind of mapping.  The cleaned up logs are in that
directory as well.

thanks,

greg k-h

How many contributors are we losing

[Luck, Tony]

Greg wrote:
> I also have "cleaned up" versions of the kernel log files for just the
> reason you say above.  You would not believe the number of times some
> people mispell their own name in a single kernel release...  That makes
> it easier to do this kind of mapping.  The cleaned up logs are in that
> directory as well.

I took those cleaned up log files (which run from 2.6.11 to 2.6.22) and
created some new ones (raw, I didn't try to clean them) for 2.6.23, 2.6.24,
2.6.25 and 2.6.26-sofar.  Then I skimmed through looking for drive-by
contributors (defined as someone who contributes to just one release and
is then never heard from again).

The summary looks like this:
63 in version 2.6.11 never seen again
148 in version 2.6.12 never seen again
128 in version 2.6.13 never seen again
92 in version 2.6.14 never seen again
96 in version 2.6.15 never seen again
122 in version 2.6.16 never seen again
137 in version 2.6.17 never seen again
140 in version 2.6.18 never seen again
135 in version 2.6.19 never seen again
95 in version 2.6.20 never seen again
136 in version 2.6.21 never seen again
153 in version 2.6.22 never seen again
179 in version 2.6.23 never seen again
179 in version 2.6.24 never seen again
304 in version 2.6.25 never seen again

These numbers are somewhat exaggerated by typos (the "cleaned up" files
still have some problems in the "Author:" entry (which is the only one
I looked at).  People add or drop middle initials, or sometimes switch
between "Firstname Lastname" and "Lastname, Firstname", and there are
plenty of obviously garbled entries.

The numbers for the more recent releases may also include
people who are still in the community, but just don't contribute to
every release.

My script didn't look for people that contributed for two or more
releases and then disappeared.

You can skim through the full list at the bottom of this message
and make your own guesses at how much of this data is garbage.
Even if 3/4 of the names here can be discounted, that still leaves
over 500 people who came to us at one point with a patch that was
good enough to be applied and then they left.

-Tony

==== Scanning 2.6.11
Margit Schubert-While
Peter Maydell
Matthias-Christian Ott
Daniel E. Markle
Arkadiusz Miskiewicz
Jarno Paananen
Jonas Munsin
Tvrtko A. Ursulin
Ron Murray
Ulrich Weigand
Martins Krikis
Edwin Olson
Rafael Avila de Espindola
Per Hedblom
Kamezawa Hiroyuki
Tom L. Nguyen
Kazunori Miyazawa
Jim Hague
J. A. Magallon
Prakash Cheemplavam
Stone Wang
Georgi Guninski
Nathan Bryant
Suresh B Siddha
Mat Loikkanen
Bill Rugolsky
Haroldo Gamal
Josh Green
Ian Pratt
Nicolas Bouliane
Esben Nielsen
Thayne Harbaugh
Herbert Potzl
Jan Kasprzak
Hanna V. Linder
Josh Aas
Martin Kogler
Theodore Y. Ts'o
Werner Almesberger
Gary N. Spiess
Olsimar
Maksim Krasnyanskiy
Juergen Quade
Pawel Sikora
Philipp Gortan
Oskar Senft
Samuel Jean
Alex Yustasov
Milton D. Miller II
Fanny Wakizaka
Andris Pavenis
Rogier Wolff
Artem Bityuckiy
Tom Coughlan
Vasia Pupkin
Thomas Spatzier
Franz Pletz
Justin Thiessen
Darrick Wong
Stefan Knoblich
Andrew Theurer
Gabor Egry
David Marlin
63 in version 11 never seen again

==== Scanning 2.6.12
Dave Neuer
johnpol@2ka.mipt.ru
Stefan Ott
dgoeddel@trustedcs.com
ecashin@coraid.com
Daniel Andersen
maartendeprez@scarlet.be
wendyx@us.ibm.com
pingc@wacom.com
hfvogt@gmx.net
Seth Rohit
jkacur@rogers.com
Tim Bird
Cosmin Nicolaescu
Kai Mäkisara
takata@org.rmk.(none)
Kenn Humborg
blaisorblade@yahoo.it
lucasvr@gobolinux.org
Jeff Lackey
Asim Shankar
Daniel Dickman
Rafael Ávila de Espíndola
cbrake@com.rmk.(none)
jason@rightthere.net
Pravin B. Shelar
Eric Brower
François Romieu
hong.liu@intel.com
hwelte@hmw-consulting.de
Chris Elston
Frederic CAND
rene.scharfe@lsrfire.ath.cx
hal@realmsys.com
Emanuele Giaquinta
Wensong Zhang
lucasvr@org.rmk.(none)
vandrove@cz.rmk.(none)
Bharath Ramesh
J. Simonetti
NAKAMURA Kenta
bram.verweij@wanadoo.nl
Mark F. Haigh
mbrancaleoni@tiscali.it
perchrh@pvv.org
carlos.pardo@siliconimage.com
Michael Werner
Ben Greear
kianusch@sk-tech.net
Giovambattista Pulcini
pavel@cz.rmk.(none)
Niu YaWei
richtera@us.ibm.com
Viktor A. Danilov
Bob Montgomery
jan.kiszka@web.de
rene@exactcode.de
Vinay K Nallamothu
McMullan, Jason
svrmgrl@gmx.net
Oliver Korpilla
Jakub Jelínek
n1gp@hotmail.com
kay.sievers@vrfy.org
Mickey Stein
Larry Battraw
Thomas Hood
Brian Waite
gam3@gam3.net
jbj1@ultraemail.net
ken@mvista.com
Jack F Vogel
Sven Anderson
kimball.murray@stratus.com
incent Sanders
Christopher Li
James Harper
relf@os2.ru
Richard Drummond
andi@cosy.sbg.ac.at
aquynh@gmail.com
pj@engr.sgi.com
shaharf@voltaire.com
aaw@rincewind.tv
tero_niemela@yahoo.com
Peter Lundkvist
Albrecht Dreß
jix@bugmachine.ca
dick@com.rmk.(none)
Herbert Pötzl
steven@brudenell.name
Gabor Fekete
Gordon Jin
domen@coderock.org
Kianusch Sayah Karadji
Dely Sy
gijoe@poczta.onet.pl
cl81@gmx.net
ariel@blueslice.com
a.llano@usyscom.com
Hideaki YOSHIFUJI
akropel@rochester.rr.com
David T. Hollis
mlafon@arkoon.net
Jason Davis
Daniel Moore
elueck@de.ibm.com
Liu Tao
liml@rtr.ca
Arthur Huillet
Lars Marowsky-Bree
Rémi Denis-Courmont
afong@org.rmk.(none)
venza@brownhat.org
Marcello Maggioni
Alex Zarochentsev
Goffredo Baroncelli
Vivian Bregier
Jerome Forissier
Stephen Tweedie
matthias.kunze@gmx-topmail.de
mstjohns@mindspring.com
bstroesser@fijitsu-siemens.com
Andree Leidenfrost
minyard@acm.org
ssant@in.ibm.com
Patrick vd Lageweg
Andre Tomt
Frank Beesley
simon@thekelleys.org.uk
duraid@octopus.com.au
Dan Malek
Arjan Bunk
Shawn Starr
Pete Jewell
Francisco Javier
felix@derklecks.de
lkml@einar-lueck.de
tzachar@cs.bgu.ac.il
christian@leber.de
Rich Walker
craig@microtron.org.uk
050035w@acadiau.ca
Darren Williams
Georges Toth
grant_nospam@dodo.com.au
gmenguez@usuarios.retecal.es
Jeroen Vreeken
148 in version 12 never seen again

==== Scanning 2.6.13
Dietmar Eggemann
Lorenzo Hernández García-Hierro
Allan Stirling
Daniele Lacamera
Stefan Sorensen
Masahito Omote
Natalie.Protasevich@unisys.com
Herbert
Simon Derr
Hans-Juergen Tappe
Hien Nguyen
Stephane VOLTZ
Martin Schitter
suzuki
Luca Falavigna
David A. Marlin
David Ranson
Zaur Kambarov
rajesh.shah@intel.com
Jonas Holmberg
Janet Morgan
Loic Le Loarer
Lorenzo Hernandez García-Hierro
david-b@pacbell.net
Michael Paxton
Andrew Hodgson
Michael Iatrou
Philip Pokorny
Vincent Vanackere
Jesse Millan
Gavin Hamill
brian@murphy.dk
Matt
Keith Moore
mochel@digitalimplant.org
Simone Zinanni
Darren Hart
Thomas Charbonnel
Felipe
Hariprasad Nellitheertha
Peter Skipworth
Greg Felix
Jurriaan
Wolfgang Wander
bobl
Dmitry Yusupov
Kurt Wall
Ivan Casado Ruiz
Keiichiro Tokunaga
Sébastien Dugu
Conger, Chris A
long
mike.miller@hp.com
Alexander Schulz
Ian Dall
Michael Gernoth
Jeremy White
Mikael Starvik
Richard Dawe
Nico Pitre
Stanislaw W. Gruszka
Sebastian Witt
Brad Campbell
David Meybohm
Hideki Yamane
Kiril Jovchev
C. Adam Oldham
Olivier Blin
Mike Strosaker
Sean Lee
Steffen Motzer
Sergey Ulanov
Utz Bacher
Gleb Natapov
Wen-chien Jesse Sung
Jurriaan on adsl-gate
Michael Prokop
Nico Golde
Rob Punkunus
BGardner@Wabtec.com
Hifumi Hisashi
Roger Mach
Matt Gillette
Yoav Zach
Chen Shang
Matthew Chapman
JT
John Levon
jayalk@intworks.biz
Bhavesh P. Davda
V. ANANDA KRISHNAN
Dinakar Guniguntala
Mathieu
pavel@ucw.cz
Nicolas S. Dade
Guillaume Autran
Abhijit Karmarkar
Reuben Farrelly
Marian-Nicolae V. Ion
Daniele Gaffuri
Dag Arne Osvik
TINNES Julien
Olivier Croquette
Jakub Bogusz
stephane.eranian@hp.com
Martin Loschwitz
Estelle Hammache
bgardner@wabtec.com
Christoph Schulz
David Chau
Blaisorblade
Sylvain Meyer
Gregory B Frost
Downing, Thomas
Manuel Capinha
Ben Colline
Emmanuel Colbus
Anton Wöllert
Dan Brown
Luca T
Peter Zubaj
Nickolai Zeldovich
Rusty Lynch
Yani Ioannou
Bernhard Fischer
bob.picco
Sven Luther
Joachim Nilsson
128 in version 13 never seen again

==== Scanning 2.6.14
Mark J Cox
Kevin Vigor
Karl Herz
Guy German
Jon Wetzel
Karl Magnus Kolstoe
Dariusz M
Bryan Sutula
Jim Paradis
Erik Waling
Rumen Ivanov Zarev
Brandon Enochs
Renaud Lienhart
Stephane Kardas
Pavol Kurina
Martin Whitaker
Randall Nortman
Iacopo Spalletti
Andrew McDonald
Adam Szalkowski
Mitsuru KANDA
Jean-Denis Boyer
Yann Droneaud
Stephane Doyon
Daniel Horchner
Jason R. Martin
Timothy Thelin
Mauricio Lin
Eric Lemoine
Chris Sykes
Vincent Pelletier
Max Kellermann
Sven Hartge
Pieter Dejaeghere
Avery, Brian
Kenneth Chen
Christian Krause
Denis Lukianov
Philipp Matthias Hahn
Michael Haboustak
Gary Wayne Smith
Paul Schulz
P Draig Brady
Daniel Burcaw
Borislav Deianov
Thomas Maguin
David Johnson
Nakul Saraiya
Tsuneo Yoshioka
Peter Wainwright
Philip Prindeville
J. Suter
Alok N Kataria
Brian Schau
Naveen Gupta
John Kingman
Svante Olofsson
Karl Hegbloom
Scott Talbert
Vincent Sanders
Bagalkote, Sreenivas
Tziporet Koren
Barry Scott
Stephan Brodkorb
Yoshifumi Nishida
Chaskiel Grundman
Dominique Dumont
Christopher Zimmermann
Ion Badulescu
Oliver Dawid
Bernd Petrovitsch
Kostik Belousov
Stephen Evanchik
Mihnea-Costin Grigore
Jose Miguel Goncalves
Wim Coekaerts
Stephen Hemmigner
Zhigang Huo
Stuart McLaren
Krzysztof Benedyczak
KUROSAWA Takahiro
Michael Veeck
Joel Schopp
Ye Jianjun (Joey)
Otto Meier
Martin Wilck
Kolli, Neela Syam
Timothy Baldwin
thomas schorpp
Alexander Krizhanovsky
Baris Cicek
Brett M Russ
92 in version 14 never seen again

==== Scanning 2.6.15
John Linville
David Eriksson
Martin Zwickel
Reimar Doeffinger
Pavel Mihaylov
Elad Lahav
Hiroki Kaminaga
Wei Ni
Chris Elmquist
Toni Mueller
Ray Cole
Richard Hitt
Tom Hughes
Peter Keilty
Vinod G
Hideki Iwamoto
James R. Webb
Alex Aizman
Peter 'p2' De Schrijver
Krzysztof Piotr Oledzki
Krishnakumar R
Jon Ringle
Carlo Comin
Glen Gray
Bernhard Rosenkraenzer
Thomas Riewe
Liu Hong
Konstantin Baidarov
Suzanne Wood
Charles R. Anderson
Ed Kear
Constantine Gavrilov
Antti Andreimann
Hareesh Nagarajan
Kurt Huwig
Scott MacKenzie
Bryan Ford
Mateusz Berezecki
David Engel
Jeff Raubitschek
Luis Miguel Garcia
Ole Reinhardt
Ralph Metzler
Kamble, Nitin A
Igor Popik
Bill Pechter
Simeon Simeonov
Kenth Andersson
Ken-ichirou MATSUZAWA
Guillaume GOURAT
Andrew Isaacson
Tim
Andrey Volkov
Kirk Lapray
Will Dyson
Hiroyuki YAMAMORI
Catalin Climov
Dag-Erling Smorgrav
Damian Wrobel
Paul T Zalac
Mark Weaver
Mark Adams
Laurent riffard
David Shirley
Honza Maly
NooneImportant
Robert W. Boone
Thorsten Maerz
Carlos Silva
Jan Pieter
Paul Vriens
Josef Balatka
Roger While
Denny Priebe
Lubomir Bulej
Martin Hagelin
Edson Seabra
Khalid Aziz
Yuri Vasilevski
Jeff Hansen
Mike Stroyan
Rik Van Riel
Zach Smith
Lars Kotthoff
Hubert WS Lin
Mao, Bibo
Peter Jones
Torsten Seeboth
Stig Telfer
Alexandra Kossovsky
Karoly Lorentey
Alexander Wold
Nikola Valerjev
Mirco Macrelli
James Cleverdon
Mathias Kretschmer
96 in version 15 never seen again

==== Scanning 2.6.16
Juergen E. Fischer
Kevin VanMaren
Tong Li
Joerg Schummer
Takis
Shaun Tancheff
d binderman
Deti Fliegl
Julian Calaby
Keck, David
Fengwei Yin
Rocky Craig
Kaj-Michael Lang
Zinx Verituse
Adam D. Moss
Janosch Machowinski
Damien Douxchamps
Wolfram Joost
Joe Kappus
Oren Laadan
Hugo Santos
Franck
Joshua Giles
Denis MONTERRAT
Thomas Schaefer
Steve Langasek
Christopher Pascoe
Noone Important
Steve Toth
Kurt J. Bosch
Graham Gower
Luis F. Ortiz
Regis Prevot
Andre McCurdy
Andrew Fuller
Eric Van Buggenhaut
Albrecht Dress
John Kacur
Yusuf Iskenderoglu
Suresh Bhogavilli
SAN People
Daniele Gozzi
Avishay Traeger
Jun'ichi "Nick" Nomura
David Kubicek
Marc Koschewski
Christian Lindner
Gregor Maier
Taneli Vahakangas
Alex Woods
A.YOSHIYAMA
Thomas Rosner
Joe
Evgeniy
Michael Richardson
David Binderman
Gergely Tamas
Vitaly Fertman
Joshua Kwan
Samir Bellabes
fabien COSSE
Ryan Hankins
Simon Vogl
Loren M. Lang
Martin Murray
Trent Jaeger
Dirk Mueller
Chris McDermott
Michal Janusz Miroslaw
Matej Kupljen
Jesse Allen
Felix Oxley
Wouter Paesen
Alex Shepard
Ville Skytta
Francois Wellenrieter
Jordan, William P
Carlo Perassi
Tore Anderson
Rene Scharfe
Hendrik Schweppe
Mike Lavender
Richard Mortimer
Ulrich Mueller
Alexandre Duret-Lutz
George Gazurkoff
Martin Gingras
Panagiotis Christeas
Javier Achirica
V. Ananda Krishnan
Ravikiran Thirumalai
Jens-Michael Hoffmann
Richard Lucassen
Albert D. Cahalan
David Elliott
Jenx Axboe
Timothy Charles McGrath
JANAK DESAI
Oliver Weihe
Christoph Dworzak
Michael Matz
Andreas Deresch
Marten Wikstrom
Janak Desai
Martin Drab
Xu, Anthony
Jorn Dreyer
Marko Kohtala
ODonnell, Michael
Lukasz Stemach
Manenti Marco
Gareth Howlett
Tetsuo Takata
Brian Strand
Paul Janzen
Stephen Williams
Robb, Sam
Mishala Pytasz
Nicolas DICHTEL
David Shaw
Ustyugov Roman
Daniel Mueller
122 in version 16 never seen again

==== Scanning 2.6.17
Micon, David
Yotam Medini
Charl Coetzee
Seiji Munetoh
Stefan Schweizer
Doug McLain
Bauke Jan Douma
Ananiev, Leonid I
Rytchkov Alexey
Florian Schlichting
Steve Yang
Marc-Andre Hebert
Rodolfo Quesada
Matheus Izvekov
Weidong
A. Maitland Bottoms
Olivier Blondeau
Magnus Kessler
Lin Feng Shen
Craig Brind
Martin Andersson
Malcom Parsons
Satoshi Oshima
Ron Yorston
Laurent Meyer
Giampiero Giancipoli
Curt Meyers
John Reed Riley
Kevin Winchester
Masc
Klaus Wacker
Jeffrey Vandenbroucke
Bastien Roucaries
Catalin(ux aka Dino) BOIE
Markus Gutschke
David chosrova
Janos Farkas
Rickard Osser
Jamie Lokier
Paul Smith
Alan Modra
Glen Overby
S P
Ken Brush
Tolentino, Matthew E
Mikhail Gusarov
Soyoung Park
Joel H Schopp
Brian Rogan
Mark Salazar
Lorenzo Hernaindez Garci-Hierro
David Basden
Bryan Holty
J.O. Aho
Nathan Bronson
Peter Gruber
Victor V. Vengerov
Tamuki Shoichi
Anders Larsen
Hakon Lovdal
Steven Finney
Julian Bradfield
Arnaud MAZIN
Aras Vaichas
Kirk True
Monty
Thilo Berger
Goldwyn Rodrigues
Jamie Wellnitz
KAI.HSU
Jochen Hein
Brent Cook
Andrew Burri
Razvan Gavril
Petri T. Koistinen
Kalin KOZHUHAROV
Jordi Caubet
Carlos Aguiar
BoyZonder
Ingo Schneider
Peter Teichmann
Bernhard R Link
Philip Gladstone
Jordan Hargrave
Harry Fearnhamm
Dale Sedivec
Chad Reese
Alberto Mardegan
Navin Boppuri
Peter Hartshorn
Alexander Zarochentzev
Michael Owen
Alpt
Ilia Sotnikov
Ashley Clark
Jon Anders Haugum
Ryan Wilson
Charis Kouzinopoulos
Richard Thrippleton
Thibault LE MEUR
Maximilian Rehkopf
Pallipadi, Venkatesh
Hideo AOKI
Olivier Hochreutiner
Dirk Herrendoerfer
Ken Arromdee
Aaron Brooks
Nico Schottelius
James Ring
Ami Perlmutter
Rune Torgersen
Aki M Nyrhinen
shin, jacob
Wang Jun
Horst Kronstorfer
Thayumanavar Sachithanantham
Brian Uhrain says
Erling A. Jacobsen
Artur Skawina
C.Y.M
Ben Woodard
Bart Samwel
John Z. Bohach
Ryan S. Arnold
Bunk
Rudo Thomas
Malte Doersam
Robert Hentosh
Levent Serinol
Tobias Powalowski
Grzegorz Janoszka
Michael Ryan
Matthias Gehre
Kenrik Kretzschmar
Robin H. Johnson
Tomasz Koprowski
Andreas Happe
137 in version 17 never seen again

==== Scanning 2.6.18
Navaho Gunleg
Cameron Hutchinson
Henk Vergonet
Daikichi Osuga
Dustin McIntire
Zoran Marceta
Rich Townsend
Kai Lindhom
olecom
Starikovskiy, Alexey Y
Bartlomiej Swiercz
Muthu Kumar
Olaf Weber
Daniel Kobras
Malcolm Valentine
Hartmut Rick
ravikiran thirumalai
Val Henson
Andy Walker
Shankar Anand
Vladislav Bolkhovitin
Kenneth Lee
Michael LeMay
Christian Lupien
Thomas Horsley
Axel Dyks
Takashi YOSHI
Edwin Huffstutler
Jorge Matias
Thago Galesi
Yingchao Zhou
Kim Oldfield
Piotr Kaczuba
Renzo Davoli
Jonathan Davies
Pavel Mironchik
Peter Milne
Justin Piszcz
Ganapathi CH
Daniel Alomar
Angelo Marconi
Nickolay
Irwan Djajadi
Lothar Englisch
Bart Massey
Michael Rash
Tyler
Jae-hyeon Park
Michael De Backer
Jan "Yenya" Kasprzak
bjdouma
Ernis
Kenneth Crudup
Ralph Siemsen
Frank de Lange
Paul Drynoff
bibo, mao
Lars Jacob
Leubner, Achim
Joerg Ahrens
Przemek Iskra
Peter Ujfalusi
Mark.Zhan
Aukasz Stelmach
D. Peter Siddons
Frode Isaksen
Lv Liangying
Zac Bowling
William Morrrow
George C. Wilson
Raimonds Cicans
Dan Bastone
Saqeb Akhter
Juergen Mell
Ira Weiny
David Wang
Thomas Andrews
Nick Fedchik
Ville Herva
Havasi Ferenc
Remi Denis-Courmont
Christophe Mariac
Aubrey Lee
Linda Knippers
Dmitry Bazhenov
Porpoise
Timothy Sipples
Zang Roy
Rachita Kothiyal
Raymond Burns
Ramachandra K
Daniel R Thompson
Unicorn Chang
Tim Kaiser
Paul Serice
Shailabh Nagar
Orjan Friberg
Shuya MAEDA
Bryan Scott
Nick Martin
Eduard Warkentin
James E Wilson
Joseph Jezak
Constantine Sapuntzakis
Ralf Schlatterbeck
Roberto Castagnola
Marko Macek
Sylvain Pasche
Michal Feix
David Kuehling
Peter Moulder
Matthew Meno
Jani Alinikula
Luca De Cicco
Davide Perini
bert hubert
Philipe De Muyter
Markus Schoder
Yuri Gushin
Fredrik Roubert
Per Dalaon
Chris Lund
Bill Huey
lamikr
Yoav Steinberg
Hiro Yoshioka
Marc Sowen
Michal Ruzicka
Philippe Retornaz
Bin Zhou
Handle X
Pedro Alejandro Lalpez-Valencia
Ralf Hildebrandt
Russ Ross
Willson Callan
Christian Praehauser
Christopher Neufeld
Tushar Gohad
Fredrik Tolf
Thomas Glanzmann
140 in version 18 never seen again

==== Scanning 2.6.19
Manasi Deval
Soos Peter
Jan Luebbe
Dave McCracken
Al Stone
Doug Ledford
George Hansper
Tilman Sauerbeck
Shaun Jackman
Werner Lemberg
Alistair Buxton
Padraig Brady
Joachim Fritschi
Prasanna S.P
dave wysochanski
Troy Heber
Om Narasimhan
Dan Cyr
keios
Mark Assad
Guillaume Munch
Auke-Jan H Kok
Bas Bloemsaat
Simon Tatham
Luke Zhang
Jules Villard
Kaustav Majumdar
Vladimir Avdonin
Thiago Galesi
Sergei Haller
Dave Liu
Bryn Reeves
jens m. noedler
Eran Tromer
Bjorn Schneider
Dmitriy Zavin
Bradley Kite
Mathieu Avila
Lars Gjesse Kjellberg
Noguchi, Masato
Dwayne Grant Mcconnell
Allan Third
Denis M. Sadykov
Raghavendra Biligiri
Brian Walsh
Kenzo Iwami
Yvan Seth
Christian Merkle
Reiner Herrmann
Sam Vilain
Eric Thomas
Jim Lewis
Francisco Larramendi
Diego Beltrami
Erich Chen
Eric Eric Sesterhenn
Dennis Stosberg
Metathronius Galabant
Alexandre Ratchov
Jan Mate
HyeonSeung Jang
Steven Haigh
Nicholas Nunley
Lijun Chen
Ian S. Nelson
David Weinehall
Patrick Jefferson
Eugeny S. Mints
Francesco Fondelli
Marek W
Peter Naulls
Jan-Frode Myklebust
Ira W. Snyder
Marcus Junker
Toshinobu Sugioka
Unai Uribarri
Andrey Liakhovets
keith mannthey
David Bussenschutt
Eric Biederman
Dan Fandrich
Fernando Vazquez
David M. Grimes
Michael Grundy
Alex Sanks
Magnus Sandin
Luke Ross
Sam Hocevar
Fernando J. Pereda
Johannes Steingraeber
Frank Sievertsen
Igor M. Liplianin
Dominic Cerquetti
Adam Henley
Sharyathi Nagesh
Amy Fong
Craig Hughes
Richard Sandiford
Thomas Chou
Joel & Rebecca VanderZee
Richard Fish
Robert S Peterson
Roger Gammans
Rick Koch
Karl-Johan Karlsson
jamal
Justin Carlson
Cory Olmo
Andy Gay
Lebedev, Vladimir P
Jochen Issing
Richard Curnow
Alexander Tuschen
Christian Steineck
Jian Gui
Klaus Frahm
Stphane Witzmann
Shem Multinymous
Mark Howell
Adam Radford
Manuel Francisco Naranjo
Cjacker Huang
Jamie Painter
Adam Tlalka
Kjell Myksvoll
Michal Majchrowicz
Sujoy Gupta
Paul Bonser
Eiichiro Oiwa
Wesley PA4WDH
Ph. Marek
Dave Arlie
Bryce Harrington
Ray Lehtiniemi
Skip Hansen
135 in version 19 never seen again

==== Scanning 2.6.20
Andy Ryan
Manuel Osdoba
Chris Caputo
David Erb
Miguel Angel Alvarez
George Sapountzis
Eagle Jones
Jeet Chaudhuri
Stefan Traby
Davy Chan
Matthijs van Otterdijk
Daniel
stanley cai
Ang Way Chuang
Takamasa Ohtake
Eric Smith
Mikhail Fedotov
James Bursa
Michael Riepe
Martin Williges
Thomas Genty
Nagendra Singh Tomar
Hiroshi Miura
Michael Broughton
Craig Schlenter
Li Yewang
Jose Carlos Garcia Sogo
Derek Fults
Luca Pedrielli
Wojtek Kaniewski
Jeff Chua
Ethan Hsiao
Raz Ben-Jehuda(caro)
Phillip Lougher
Luke Deller
Ricard Wanderlof
Holden Karau
Ryan Underwood
Lars Ellenberg
Andrea A Odetti
Justin Clacherty
Adrian Friedli
Takada
Jesse Huang
Hynek Petrak
Lew Glendenning
Florian Festi
Christian Hesse
Fabrice Knevez
Torsten Ertbjerg Rasmussen
Dave Olsen
Adam Megacz
Frederic Riss
BP, Praveen
Erik Jacobson
Tomi Koivulahti
Rik Bobbaers
Chen, Justin
Artiom Myaskouvskey
Aaron Salter
Rutger Nijlunsing
Guillem Jover
Vijay Kumar
Marton Nemeth
Timo Lindhorst
Yoshimi Ichiyanagi
Matthew McClintock
Nicolas Bellido
Thomas Tuttle
David Clare
Ashwin Chaugule
Henning Schroeer
Henry Nestler
Chris Frey
Dwayne Grant McConnell
Shantanu Goel
Ard van Breemen
Jan Capek
Thomas Hamm
Jason Parekh
Ryan Jackson
Garrett Damore
Kristian Kielhofner
Jun Chen
Filipe Lautert
Lars Immisch
Andrew Beekhof
Christoph Haubrich
Greg Chandler
Simon Bennett
Sven Anders & Marcus Junker
Josef "Jeff" Sipek
Mark Glaisher
Timofei V. Bondarenko
Martin Willi
95 in version 20 never seen again

==== Scanning 2.6.21
Robert Marquardt
Tomasz Kvarsin
Jean-Paul Saman
Taku Izumi
Nigel Williams
Clement Guedez
Rainer Weikusat
Chris Rankin
Mikhail Kouzmich
Valery Podrezov
Jakub Schmidtke
s situert
Joerg Dorchain
Ole Andre Vadla Ravnas
Ozzy
Marcel Siegert
Thomas Bachler
Jeff Morrow
Sanjoy Mahajan
Micke Prag
Carl Love
Matthew C Campbell
Fiodor Suietov
Vassili Karpov
Shinta Sugimoto
Thomas Schleusener
Manfred Gruber
Benjamin Li
Pantelis Koukousoulas
TAKADA Yoshihito
eric wollesen
Brian Pomerantz
Ken Witherow
Jakub W. Jozwicki J
Bruce Fields
Zheng XiaoJun
Kevin Jamieson
Mariusz Domanski
Giuliano Procida
Sergei Organov
Joerg Sommer
Shane Shrybman
Andrea Guzzo
Heiko Baums
Aji Srinivas
Karsten Weiss
Bill Helfinstine
Ruben Vandeginste
Kai Engert
Carl Lundqvist
Matthew Percival
Benjamin Romer
Max Dmitrichenko
Joseph S. Myers
Roland Kletzing
Adhiraj Joshi
Vincent Penne
Zilvinas Valinskas
Corentin Labbe
Rolf Stefan Wilke
Toshimune Konno
Thomas Hoehn
Tommi Kyntola
Antti Seppala
peter fuerst
Luciano Rocha
Hubert Kahlert
Hans-Peter Nilsson
Rui Zhang
Andre Spahlinger
Marcel van Nies
Mike Chan
Frithiof Jensen
Timo Savola
Shlomi Fish
Simon Vallet
Jack Lee
Andrew Johnson
Chris Lesiak
Ken L Johnson
Eric D. Mudama
Martin Stoilov
Andreas Block
Chuck Meade
Martin Schiller
Julius Volz
Evgeny Kravtsunov
Andrew Nayenko
Mikael Nilsson
John Daiker
Michael Leun
Andrew L. Neporada
Simon Depiets
Gard Spreemann
Andrew Donofrio
Vijay Sampath
Jon Dowland
Thomas De Schampheleire
Jan Yenya Kasprzak
Johannes Schlumberger
Antti Palosaari
Richard Fearn
Bjoern Fay
Richard Woodruff
Jin-Bong lee
Thomas Bittermann
Dan Wolstenholme
Lalit Chandivade
Akiyama, Nobuyuki
Peter Eriksen
Alan Tyson
Michael Olberg
Teru KAMOGASHIRA
Raol Sainchez Siles
Michal Wrobel
Jerome Demange
Maciej Zenczykowski
Anthony Godshall
Randy Cushman
Wu, Gilbert
Emil Larsson
Thomas Viehweger
Jamie Clark
Ishimatsu Yasuaki
Juan Pablo Sormani
Paul Rolland
Dylan Taft
Kristian Hogsberg
Valery A. Podrezov
Jan Nijs
Philipp Reisner
Hennerich, Michael
Alexandre Bounine
Phil Blundell
Gerhard Dirschl
Jon K Hellan
136 in version 21 never seen again

==== Scanning 2.6.22
matze
Pat Erley
Danny Budik
Sylvain FORET
Sergey Kiselev
Kevin Welton
Morten Banzon
John Utz
Len Sorensen
Egmont Koblinger
Marko Vrh
Stephen Cameron
Dragos Carp
Sandeep Sanjay Patil
Peter Stokes
vignesh
Marton Nameth
Domenico Andreoli
Leon Leong
Tom Alsberg
Matthew Davidson
Stephen M. Cameron
Witold Filipczyk
Ronny Peine
Uwe Kleine-Kanig
Ratnadeep Joshi
Sami Farin
Jon Paul Maloy
Luis Carlos
Ed Vipas
Thierry Merle
Andre Renaud
Jrgen Schindele
Guido Scholz
Peter P. Waskiewicz Jr.
Patrick McHarrdy
Avuton Olrich
Srinivas Aji
Alex Villacis Lasso
Ludwig Nussel
Alexandra N. Kossovsky
Julian Stecklina
James Puthukattukaran
Christian Rothlaender
Matej Kenda
Servaas Vandenberghe
Alexander Gattin
William Cohen
Aeschbacher, Fabrice
Dave Gilbert
Yehuda Sadeh Weinraub
Christophe Cattelain
Stefan Lucke
Nicu Ioan Petru
Karl Pickett
Alberto Bertogli
Rask Ingemann Lambertsen
Davide Brini
Ronni Nielsen
Kouta Ooizumi
Fabrice Aeschbacher
Stephen Mollett
Eric Rannaud
Scott Wiersdorf
Torsten Kaiser
Hans Engelen
Tian Kevin
Vignesh Babu BM
Simon Richter
Ruslan V. Sushko
Bernhard Kauer
Orczykowski, Juergen
Loic Prylli
Florian Attenberger
Pierre Willenbrock
Charles Pillar
James Carter
Takao Shinohara
Lasse Collin
Abhijit Bhopatkar
Joshua Wise
Dwaine P. Garden
Damian Minkov
Daniel P. Engel
Kenichi Nagai
Holger Magnussen
johan henriksson
Dennis Ranke
James T Klaas
Michael Milner
Corey Mutter
Syed Khasim
John Johansen
Sam Liddicott
Brian Braunstein
Janusz Krzysztofik
Jeffrey Layton
Markus Dahms
Robin H\. Johnson
Hendrik Borghorst
Edward Goggin
Mark Huth
Knobloch, Thomas
Julian Cable
takada
Daniel Wolstenholme
John Feeney
Joshua N. Pritikin
kalash nainwal
Marco Costalba
Jean-Christian de Rivaz
Ragner Magalhaes
Luis Carlos Cobo Rus
Yosef Etigin
James Yang
Mike Accetta
Peter Kovar
Thomas Reitmayr
Paul Zaremba
Eberhard Fahle
Lee Trager
Chris Clayton
Brandon Craig Rhodes
Yaozu Dong
Hans-Juergen Koch
Christian Volkmann
Surya Prabhakar
Marc Butler
Neil \"Superna\" ARMSTRONG
Morrison, Tom
Douglas Landgraf
Joey Goncalves
Holger Smolinski
Shan Lu
Michael Reiss
Jay Lubomirski
wendy xiong
Zach Carter
Utako Kusaka
Roman Moravcik
Myron Stowe
Klaus Kudielka
Richard Lary
Marvin Raaijmakers
MOKUNO Masakazu
Emil Georgiev
Michael Loehr
Noel Kothe
Shashi Rao
Thomas Knobloch
Marco Roeland
Shahrom Sharif
Carlos E. Ugarte
153 in version 22 never seen again

==== Scanning 2.6.23
Oliver McFadden
Tomas Janousek
Marisuz Kozlowski
Nicola Fagnani
Lucas Nussbaum
Pádraig Brady
Songmao Tian
Wyatt Banks
Neil Muller
Ranganathan Desikan
Adit Ranadive
Joan Eslinger
Eric
Wade Berrier
Joshua Hoblitt
Rafael Bilski
Rainer Birkenmaier
Timo Jantunen
Maxim Uvarov
James Le Cuirot
Lucy McCoy
Steve G
Benjamin Gilbert
Mijo Safradin
Søren Hauberg
Kapil Juneja
Dustin Marquess
Patrice Vilchez
tao.ma@oracle.com
young dave
Tim Harvey
henry su
Leandro Dorileo
Alexander Shmelev
Daniel Exner
Clifford Wolf
James Jarvis
Pierre Castella
david m. richter
Andrew Burgess
Kirill Kuvaldin
Niels de Vos
Anderson Briglia
Thomas Dahlmann
Usha Ketineni
Karl Olsen
Stepan Moskovchenko
Edward Hsu
Maarten Bressers
Roger So
Eric Wollesen
George Shapovalov
Dajie Tan
Ville Tervo
Kent Yoder
Jonathan Phenix
Changli Gao
Ranko Zivojnovic
Kazunori Asayama
Herbert van den Bergh
izumi
Michal Marek
Balazs Scheidler
Eduard-Gabriel Munteanu
Erik Johansson
Bruce Ashfield
Elvis Pranskevichus
Brijesh Singh
Toshiyuki Okajima
Milinevsky Dmitry
Vladimir Shebordaev
Thibault Le Meur
jing xiang
samson yeung
C. Scott Ananian
Mike Crash
Yasuaki Ishimatsu
Charlie Shepherd
frederic RODO
Luis Lloret
Kawai, Hidehiro
Nick Bowler
Nathael Pajani
Alessio Igor Bogani
Vovan888@gmail
Romain Goyet
Giuseppe Sacco
IKEDA, Munehiro
Quinn Jensen
TripleX
Ritesh Raj Sarraf
Mark Grondona
Tear
Ting Yang
Douglas Thompson
Klaus Weidner
Rui Sousa
Samuel
Raghava Kondapalli
Reiner Sailer
Thomas Viehmann
geoffrey.levand@am.sony.com
Ryo Dairiki
Jonathan Lim
Kalpak Shah
Suresh Jayaraman
Attila Kinali
Stuart_Hayes@Dell.com
Christian Schmidt
Mike Miller (OS Dev
Imre Kaloz
t.sefzick
Jesper Bengtsson
aherrman@arcor.de
Massimiliano Ghilardi
Milko Krachounov
Thomas Hommel
Fuxin Zhang
David Warman
Ethan Solomita
Serge Belyshev
Jeff Norden
gw.kernel@tnode.com
corentin.labbe
Ricardo Barberis
Meelap Shah
Jaroslav Kysela perex@suse.cz
Nitin Gupta
Ryan Power
Mike Cruse
Rene van Paassen
Nelson, Shannon
Martin Szulecki
Christian Kandeler
Mark Vytlacil
Diogo Kastrup
Dmitry Butskoy
Vinit Agnihotri
nickcheng(é"­å®^è¬(tm)
Joakim Koskela
Danny ter Haar
Minchan Kim
Wolfgang Walter
Stephan Wolf
M4rkusXXL
Andreas Arens
Tony Wan
Jorge Juan Chico
Nicolas George
Juan Lang
Terry Loftin
Christian Heim
John Donoghue
Veena Parat
su henry
Andrey Arapov
Murillo Fernandes Bernardes
Edgar Pisani
Kees Lemmens
Jan Frey
Matt Colyer
Ilpo JÃf¤rvinen
Julien Eyries
sebastian@breakpoint.cc
jacmet@sunsite.dk
Jean-Christophe DUBOIS
Ivan N. Zlatev
Matthew Gregan
Jan Lübbe
Carlos Olalla Martinez
Carlo Beccaria
Vikram Pandita
Eddy L O Jansson
YOSHIFUJI Hideaki / å?è-¤è<±æ~Z
Nate
Eric W. Biderman
Sven-Thorsten Dietrich
Priyanka Gupta
Micah Cowan
179 in version 23 never seen again

==== Scanning 2.6.24
Martti Huttunen
Charles Hardin
Juha Laiho
ben.nizette@iinet.net.au
Dave Wysochanski
Mike Crowe
Pedro
Chris David
Björn Steinbrink
Yann Dirson
Steve Cameron
Thomas Backlund
Andrew McNabb
Jonas Danielsson
Sam Jansen
Alfred E. Heggestad
Veljkovic Srdjan
tonyj@suse.de
Michael J. Evans
Pekka Seppanen
Peter Lund
Dr. David Alan Gilbert
Matt Doran
de Dinechin, Christophe (Integrity VM)
Khelben Blackstaff
Aaron Carroll
Elias Oltmanns
Gregory CLEMENT
Mark Hills
Alon Ziv
Hans J Koch
Mark Nelson
Ilya Frolov
Christian Hohnstaedt
Ed Swarthout
Richard Sharpe
Pekka Seppänen
Mark Ryden
Alex Villacís Lasso
vbarshak@ru.mvista.com
Bryn M. Reeves
Martin Kusserow
Kevin Pedretti
Joe Carnuccio
jidong xiao
saeed bishara
Joshua J Bowman
Daniel Roesen
Philipp Marek
Micah Parrish
sebdeg@ngi.it
chaohong guo
Dirk Hohndel
Danny Baumann
Tobias Poschwatta
Andrew Dyer
Sheela
Ryousei Takano
Edouard Lafargue
Nico Erfurth
John Muir
Yann Chachkoff
Jörn Engel
Dave Dillow
John Traill
Alex Unleashed
Ryan Reading
Adam Jackson
Adel Gadllah
Wolfgang Denk
Nathanael Nerode
Hye-Shik Chang
sonic zhang
minchan kim
Ilya Yanok
Anton Ekblad
Yang, Sheng
Akos Maroy
Hinko Kocevar
Saleem Abdulrasool
Matej Laitl
Chris Paulson-Ellis
William Pettersson
Mirko Lindner
Bryan Kadzban
Tony Li
Philippe Elie
Barry Kasindorf
Matthias Mueller
Francis Moreau
Ludovico Cavedon
ashish kalra
Jonathan Bastien-Filiatrault
Mike Westerhof
Denys
Alex
Keita Maehara
warmcat
Wagner Ferenc
David Smith
Tomoya Adachi
Milan Plzik
Benoit Istin
Keiichi Kii
Aoi Shinkai
Adrian Knoth
Radu Rendec
Brett Warden
Li, Xin B
Damian Jurd
Anton Salikhmetov
Ivo Manca
Andrew Gallatin
agilmore@wirelessbeehive.com
Andrew M. Bishop
barrios
FD Cami
Jerrold Jones
Iustin Pop
Gilles Gigan
Jan Rinze
Signed-off-by@vergenet.net":Simon
James Pearson
George Kibardin
Anton Arapov
trem
\"Talpey, Thomas\
zhejiang
bugme-daemon@bugzilla.kernel.org
Jesper Dangaard Brouer
Rini van Zetten
Matias Zabaljauregui
Lepton Wu
Subbaiah Venkata
Matteo Vit
Kevin R Page
Tom "spot" Callaway
Jeff Bailey
Sellout Bessie
Joachim Steiger
Kazuhiko Kawakami
Jeff Long
Edgar Simo
Surya Prabhakar N
Murali Iyer
Massimo Cirillo
Steven A. Falco
David Daney
Xianghua Xiao
Kuan Luo
Stefan Lippers-Hollmann
Andreas Loibl
Jonas Stare
Philippe Rétornaz
Michael Brunner
Ilpo Järvinen
Andy Lowe
Cyril Gorcunov
Matthias M. Dellweg
Heiko Schocher
Dawid Wrobel
Benedikt Spranger
Michael Mauch
Stuart Swales
Marek Vašut
Mike Miller (OS Dev)
Matthias Goebl
Eugene Konev
Olaf Hartmann
Scott James Remnant
Aleksandar Radovanovic
Chris Poon
Tao Mao
Frank A Kingswood
James Bowes
Vladimir Davydov
Udo A. Steinberg
Carmelo Amoroso
Chaogui Zhang
179 in version 24 never seen again

==== Scanning 2.6.25
Syed Mohammed, Khasim
Ross Burton
Ed Beroset
Brett T. Warden
Assaf Hoffman
Patrick Marchand Latifi
Chidambar 'ilLogict' Zinnoury
K.Tanaka
Hirokazu Takahashi
nickcheng
Min Zhang
Jonathan Lynch
mboton@gmail.com
Prasad P
Rizzo Davide
travis@sgi.com
Jim Paris
Carlo Marcelo Arenas Belon
Joel Soete
Troy Kisky
Veli-Matti Valtonen
Robie Basak
Paul Jimenez
Signed-off by Yi Yang
goda.yusuke
Kim Sandberg
Shan Wei
Ã?ric Piel
clameter@sgi.com
Milan plzik
Franco Lanza
Kevin Lo
Carlos R. Mafra
Jiang Zhe
Maciej Sosnowski
Marin Mitov
sergeh@us.ibm.com
Thomas Horsten
Robert Bragg
Thomas Mingarelli
Bryan Boatright
Nicholas Beck
Tan Swee Heng
Clark Rawlins
Christian Pellegrin
Paul Knowles
Riki Oktarianto
Nikanth Karthikesan
Jack Stone
yakui.zhao@intel.com
Michel Lespinasse
Sebastian Ott
Miguel Boton
Sergio Luis
Kieran Bingham
Ned Forrester
Michael E Brown
Bastien Nocera
Jessica L. Blank
Jan Evert van Grootheest
Drew Fisher
Remy Bohmer
David P. Quigley
Dave Miller
Iñaky Pérez-González
Dan Muntz
tang kai
Steve Welch
Patrick Caulfeld
Johnson Leung
Andrew Burton
Bruno Redondi
Adrian Pardini
Daniel Hokka Zakrisson
Tom Talpey
Kim B. Heino
Silvester Erdeg
Thadeu Lima de Souza Cascardo
Timofei Bondarenko
Mike Snitzer
Gadiyar, Anand
Alexey Zaytsev
Dmitri Monakhov
Christian Glindkamp
Keld Simonsen
Trevor Highland
Andreas Herrmann3
Roy Hashimoto
Markus Metzger
serue@us.ibm.com
Christine Caulfield
Ming Lin
Liam R. Howlett
Lei Ming
Eduardo Habkost
Dmitry Shapin
Rainer Jochem
Keiichi KII
Gaston, Jason D
Anand Gadiyar
Yuri Funduryan
Martin Strubel
Thomas Sujith
Itaru Kitayama
Arjan van dev Ven
Bryan Rosenburg
Marcin Åslusarz
Zhang Xiantao
frederic Rodo
Jean Noel Cordenner
janboe
Bradley Smith
joe@perches.com
Michel Ludwig
len.brown@intel.com
Martin Stava
Marc Boucher
Alex Deucher
Laszlo Attila Toth
Nate Carlson
Greg KH
Daniel Kozák
Klaus Heinrich Kiwi
Vladimir Berezniker
T. H. Huth
Sven Andersen
Dmitry Antipov
Hans Rosenfeld
Krauth.Julien
Pravin M. Bathija
Alex Bounine
Nathaniel Filardo
Pavel Troller
Adrian Bassett
Paul Chavent
Andrew Patterson
Warren Turkal
Stephan Diestelhorst
Alejandro Riveira Fernández
Nicolas Mailhot
Valdis.Kletnieks@vt.edu
SDiZ
Zhang Yanmin
Luís P Mendes
Jan Slupski
Sven Wegener
Jean-Samuel Chenard
Aegis Lin
Michal Sojka
John Reiser
Ulisses Furquim
Laszlo Kajan
Vasily Khoruzhick
Peter Stark
Yang Shi
Jay Schulist
Zhang Le
Jaroslav Barton
Corentin CHARY
TripleX Chung
Michael Abd-El-Malek
Dan Kenigsberg
Tomohiro Kusumi
Stefan Bauer
Bob Wilson
Anatolij Gustschin
fangxiaozhi
Tim Ansell
Nick Warne
Andrew Smith
root
Thomas Betker
Tim Taubert
Mikko Herranen
Gregory Greenman
Aidan Williams
Piotr Roszatycki
Andrey J. Melnikoff (TEMHOTA)
John Lacombe
Surjit Reang
Rabin Vincent
Akira Tsukamoto
Hermann Lauer
Maarten Lankhorst
Lai Jiangshan
bjorn.helgaas@hp.com
Andrew Murray
Brian Wood
Sten Wang
Eamon Walsh
Maciej Cencora
Kevin Vance
Justin Treon
Soeren Moch
Sheng Yongjie (Sam
Yutaro Ebihara
devzero@web.de
Wolfgang Ocker
Priit Laes
Laim Girdwood
Maciej Soltysiak
Bill Hayes
Eduardo Pereira Habkost
Oliver Schuster
Marcus Barrow
Misha Zhilin
Daniel Wagner
EGRY Gabor
Mart Raudsepp
Hideo Aoki
Andre Noll
Mirko
Bruce Duncan
René Bürgel
Neil Turton
David Acker
Bron Gondwana
Stephan Boettcher
Abhishek Sagar
Anton Salnikov
Manish Katiyar
Emil Tantilov
Damien Stuart
Jody Belka
Steve Hardy
Pradeep Satyanarayana
Miguel Botón
Fred L. Templin
Arve Hjønnevåg
Brad Sawatzky
Uwe Kleine-Koenig
Walter T Gruczka
Girish Shilamkar
Yuri Tikhonov
Constantin Baranov
dominik
Bjorn Mork
Carol Hebert
Girish
Giel de Nijs
Shin-ya Okada
Karol Swietlicki
Markus Gaugusch
Thomas Pfaff
Alain Degreffe
Jesse Ahrens
Konstantin Kletschke
Ivo Couckuyt
Marc Gauthier
Dave Anderson
Richard MUSIL
Ronen Shitrit
Ke Wei
Alexey Demin
Rick Farrington
Yousef Lamlum
Petr Cvek
Coleman Kane
arvidjaar@mail.ru
Albert Graham
Dmitry Krivoschekov
Enrik Berkhan
Robert Spitzenpfeil
Josef Jeff Sipek
Alexandre Rusev
npiggin@suse.de
Michael Loeffler
Vitaliy Gusev
Gerald Stralko
Alex Riesen
Marc Dionne
Zhang, Xiantao
Anantha Subramanyam
David Scidmore
Mirko Bordignon
David Ludlow
Mike Day
Stuart Bennett
Peter Teoh
Ian Schram
Fabio Checconi
Lon Hohberger
Libor Pechacek
Dominique Quatravaux
Chaoyu Chen
Eric Dujardin
Leonid Evdokimov
Quentin Barnes
Max Arnold
Serge A. Suchkov
Dirk DeSchepper
Roland Stoll
Minoru Usui
Douglas Kosovic
Andreas Degert
Crane Cai
Heinz-Ado Arnolds
Ihar Hrachyshka
Ctirad Fertr
Stephen Ware
Rémi Denis-Courmont
David Newall
David Cohen
Rafael Ignacio Zurita
304 in version 25 never seen again

Re: How many contributors are we losing

[Willy Tarreau]

On Fri, May 30, 2008 at 01:23:44PM -0700, Luck, Tony wrote:
> Greg wrote:
> > I also have "cleaned up" versions of the kernel log files for just the
> > reason you say above.  You would not believe the number of times some
> > people mispell their own name in a single kernel release...  That makes
> > it easier to do this kind of mapping.  The cleaned up logs are in that
> > directory as well.
> 
> I took those cleaned up log files (which run from 2.6.11 to 2.6.22) and
> created some new ones (raw, I didn't try to clean them) for 2.6.23, 2.6.24,
> 2.6.25 and 2.6.26-sofar.  Then I skimmed through looking for drive-by
> contributors (defined as someone who contributes to just one release and
> is then never heard from again).
> 
> The summary looks like this:
> 63 in version 2.6.11 never seen again
> 148 in version 2.6.12 never seen again
> 128 in version 2.6.13 never seen again
> 92 in version 2.6.14 never seen again
> 96 in version 2.6.15 never seen again
> 122 in version 2.6.16 never seen again
> 137 in version 2.6.17 never seen again
> 140 in version 2.6.18 never seen again
> 135 in version 2.6.19 never seen again
> 95 in version 2.6.20 never seen again
> 136 in version 2.6.21 never seen again
> 153 in version 2.6.22 never seen again
> 179 in version 2.6.23 never seen again
> 179 in version 2.6.24 never seen again
> 304 in version 2.6.25 never seen again
> 
> These numbers are somewhat exaggerated by typos (the "cleaned up" files
> still have some problems in the "Author:" entry (which is the only one
> I looked at).  People add or drop middle initials, or sometimes switch
> between "Firstname Lastname" and "Lastname, Firstname", and there are
> plenty of obviously garbled entries.
> 
> The numbers for the more recent releases may also include
> people who are still in the community, but just don't contribute to
> every release.
> 
> My script didn't look for people that contributed for two or more
> releases and then disappeared.
> 
> You can skim through the full list at the bottom of this message
> and make your own guesses at how much of this data is garbage.
> Even if 3/4 of the names here can be discounted, that still leaves
> over 500 people who came to us at one point with a patch that was
> good enough to be applied and then they left.

It depends how we see this. Having a lot of external contributors is
very good, as it implies that more and more people are getting used
to read and modify the code. What would be interesting would be to
check for people who where there on a regular basis then went away,
even though I admit is harder to find out.

Willy

Re: How many contributors are we losing

[Greg KH]

On Fri, May 30, 2008 at 01:23:44PM -0700, Luck, Tony wrote:
> Greg wrote:
> > I also have "cleaned up" versions of the kernel log files for just the
> > reason you say above.  You would not believe the number of times some
> > people mispell their own name in a single kernel release...  That makes
> > it easier to do this kind of mapping.  The cleaned up logs are in that
> > directory as well.
> 
> I took those cleaned up log files (which run from 2.6.11 to 2.6.22) and
> created some new ones (raw, I didn't try to clean them) for 2.6.23, 2.6.24,
> 2.6.25 and 2.6.26-sofar.  Then I skimmed through looking for drive-by
> contributors (defined as someone who contributes to just one release and
> is then never heard from again).

Well, you do know that the distribution of all of our users are:
	50% only contributed 1 patch
	25% contributed 2
	12% contributed 3
	6% contributed 4
and so on?

Our curve is leveling out much better now though.  For the whole 2.5
release, the top 30 people did over 80% of the work.  Now, the top 30
people are doing 30% of the work.

So it is getting much better, as long as we still continue to keep our
massive rate of change[1] that we have going, and huge number of
developers[2], we should be fine.

So this list doesn't necessarily mean anything is wrong, only that 50%
are one-time contributors.  And I think that shows we are easy to get a
change into our tree from just about anyone, not that we are driving
people away.

thanks,

greg k-h

[1] 7,000 lines added, 2,500 lines removed, 2,400 lines modified, per
day for all of the 2.6.25 release cycle.  That's insane.
[2] 2,598 unique developers from 2.6.20 to 2.6.25

Re: [Ksummit-2008-discuss] How many contributors are we losing

[Grant Grundler]

On Fri, May 30, 2008 at 1:47 PM, Greg KH <greg@kroah.com> wrote:
...
> Well, you do know that the distribution of all of our users are:
>        50% only contributed 1 patch
>        25% contributed 2
>        12% contributed 3
>        6% contributed 4
> and so on?

"contributed" here means a patch was accepted.
This is measuring "attribution", not contribution.

Posting a patch is not trivial and (hopefully) takes a fair
amount of work to prepare for anyone not doing this full
time. I'm not talking about white space changes but
even trivial patches that require some testing.


It would be interesting to scrape the archives of linux-*
and netdev mailing lists to see who is submitting patches
(and how many) and compare that with how many the
same person gets "attribution" for. The fallout rate would
be a better indicator.

> Our curve is leveling out much better now though.  For the whole 2.5
> release, the top 30 people did over 80% of the work.  Now, the top 30
> people are doing 30% of the work.

My guess is top 30 people are spending more time reviewing patches
than writing code. They get "attribution" by adding their SOB lines.

> So it is getting much better, as long as we still continue to keep our
> massive rate of change[1] that we have going, and huge number of
> developers[2], we should be fine.
>
> So this list doesn't necessarily mean anything is wrong, only that 50%
> are one-time contributors.

In general I agree - I don't think the problem is as bad
as some people are claiming. But I want to acknowledge
it is a problem and I think jejb is right in how he is
raising the issue.

> And I think that shows we are easy to get a
> change into our tree from just about anyone, not that we are driving
> people away.

I still don't buy this. I've been contributing to linux kernel since about
1999 and it's definitely not getting easier.  The size of SubmittingPatches
is one indicator of how much work it is to submit a patch.
SubmittingPatches is now 600 lines (3400+ words).

The large number of contributors says nothing about how easy or hard
it is to get a patch into the tree. I think it says more about how many
people are getting paid to work on linux or are exposed to linux.

My own experience with tulip driver certainly isn't one that encourages
people to submit more patches and stay involved. USB patches I've
submitted were trivial (hard to debug and required specific HW to test)
but did get accepted. The first IDE patch I submitted also got rejected
with an answer that didn't help:
   http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg22756.html

That last patch "Worked For Me" and Alan Cox argued for it but it
didn't get further attention. I mention these only because except
for tulip, I wasn't paid to submit or work on those patches.

The problem is with specific maintainers not having BW or interest
in the users' problems. I'm thinking each maintainer should have
some "minions" to assist people submitting bugs/patches like
my issue with IDE until a patch gets accepted or the issue otherwise
resolved.

Another reason I suspect we are seeing more "one-time" contributions
is because of product development sticking with one kernel version
they've cooked themselves for several years.  The project will submit
fewer patches upstream as their kernel "ages" and each patch requires
substantial more work to "forward port". I don't expect there is anything
we can even if we could find volunteers to do that forward porting.

hth,
grant

Re: How many contributors are we losing

[Stefan Richter]

Grant Grundler wrote:
> On Fri, May 30, 2008 at 1:47 PM, Greg KH <greg@kroah.com> wrote:
>> And I think that shows we are easy to get a
>> change into our tree from just about anyone, not that we are driving
>> people away.
> 
> I still don't buy this. I've been contributing to linux kernel since about
> 1999 and it's definitely not getting easier.  The size of SubmittingPatches
> is one indicator of how much work it is to submit a patch.
> SubmittingPatches is now 600 lines (3400+ words).

This growth of SubmittingPatches and related documents also partly means 
that we now have
   - more tools available for QA before posting,
   - more documentation on these tools,
   - more comprehensive documentation on the workflows.
To some degree, this should make contributions easier rather than harder.

It may of course also mean that some expectations are higher now, as you 
are saying AFAIU.  However, with scarce reviewer base (and scarce early 
testers base, compared to the whole userbase), it's not a bad thing for 
the project health if patches, when first posted, already have a good 
level of quality.  We want many good contributions, not just many 
contributions.  Of course if there were many reviewers and many mentors, 
than we could do with lower initial quality of submissions.
-- 
Stefan Richter
-=====-==--- -=-= =====
http://arcgraph.de/sr/

Re: [Ksummit-2008-discuss] How many contributors are we losing

[Daniel Walker]

On Fri, 2008-05-30 at 13:23 -0700, Luck, Tony wrote:

> You can skim through the full list at the bottom of this message
> and make your own guesses at how much of this data is garbage.
> Even if 3/4 of the names here can be discounted, that still leaves
> over 500 people who came to us at one point with a patch that was
> good enough to be applied and then they left.

I know some of the names on the list which seem valid, but they are
still in the community at large they just aren't sending patches .. I'd
imagine not many of these people are gone from Linux, they're still
involved in some way.

(btw, Greg KH is listed under 2.6.25 .. )

Daniel

Re: [Ksummit-2008-discuss] How many contributors are we losing

[Hugh Dickins]

On Fri, 30 May 2008, Luck, Tony wrote:
> 
> I took those cleaned up log files (which run from 2.6.11 to 2.6.22) and
> created some new ones (raw, I didn't try to clean them) for 2.6.23, 2.6.24,
> 2.6.25 and 2.6.26-sofar.  Then I skimmed through looking for drive-by
> contributors (defined as someone who contributes to just one release and
> is then never heard from again).

We've had some great contributions from drive-bys down the years, 
and I see that as a gain rather than a loss.  I suppose it's a
half-full versus half-empty perception.

Interesting list, but as you admit, yes, there are a lot of false
positives in there.  Just to pick on one ...

> ==== Scanning 2.6.11
> Theodore Y. Ts'o

... anyone know what happened to that guy?

Hugh

RE: [Ksummit-2008-discuss] How many contributors are we losing

[Luck, Tony]

> We've had some great contributions from drive-bys down the years, 
> and I see that as a gain rather than a loss.  I suppose it's a
> half-full versus half-empty perception.

Totally agree.  If someone finds one bug, sends us a fix and leaves
as a happy customer that is a wonderful thing.

Here's the (much shorter) list of people that contributed to two
consecutive releases and then disappeared.  These people made at
least two contributions across a 3+ month period. In most cases
this moves them out of the "drive-by" category.

Probably some of them moved on to do different things (or were
promoted to manage people who still work on Linux).

-Tony

Scanning 2.6.11-2.6.12
Jeff Muizelaar
Mikkel Krautz
Eric Lammerts
Marcel Sebek
Brian Waite
Werner Almesberger
Giorgio Padrin
Samuel Jean
Hideaki YOSHIFUJI
Markus Bollinger
Catalin Boie
Jerome Forissier
Stephen Tweedie
John Lenz
Shannon Holland
15 never seen again

Scanning 2.6.12-2.6.13
Narendra Sankar
Steven Cole
Thomas Charbonnel
Steven Scholz
Jun Komuro
Jarkko Raja
Dely Sy
Qu Fuping
Lars Marowsky-Bree
Peter Zubaj
10 never seen again

Scanning 2.6.13-2.6.14
Nick Sillik
Giancarlo Formicuccia
M.Baris Demiray
Nick Wilson
Linda Xie
Victor Fusco
Jan Veldeman
Andreas Steinmetz
8 never seen again

Scanning 2.6.14-2.6.15
Kenneth Tan
Stuart Auchterlonie
Henk
Felix Blyakher
KOVACS Krisztian
Mike Kershaw
Adam Brooks
Hironobu Ishii
Abhay Salunke
9 never seen again

Scanning 2.6.15-2.6.16
Kylene Jo Hall
Alex Aizman
Rui Santos
Chris Humbert
Thomas Young
Daniel Marjamaki
Samir Bellabes
Gabriel A. Devenyi
Marcus Sundberg
9 never seen again

Scanning 2.6.16-2.6.17
Seiji Munetoh
Mattias Nordstrom
Per Liden
Horst Schirmeier
Marco Manenti
Marcin Rudowski
Holger Eitzenberger
Nippun Goel
Roberto Nibali
Karsten Suehring
10 never seen again

Scanning 2.6.17-2.6.18
Thomas Kleffel
Frank Gevaerts
Laurent Meyer
Catherine Zhang
Jean-Luc Leger
Peter Horton
Tomasz Kazmierczak
Dustin Kirkland
Mandy Kirkconnell
Bastiaan Jacques
David Hollister
Wu Fengguang
12 never seen again

Scanning 2.6.18-2.6.19
Manoj Naik
Toralf Foerster
Thiago Galesi
Chris Boot
matthieu castet
Eric Hustvedt
Ben Williamson
Christoph Pfister
David Wang
Forrest Zhao
Chuck Short
Paul Collins
Suparna Bhattacharya
Dennis Munsie
Wong Hoi Sing Edison
15 never seen again

Scanning 2.6.19-2.6.20
Ernie Petrides
Ville Nuorvala
Tony Olech
Chris Lalancette
Ira W. Snyder
Kim Nordlund
David Anders
Sven Anders
nkalmala
Paul B Schroeder
Chad Sellers
11 never seen again

Scanning 2.6.20-2.6.21
Amit S. Kale
Sarah Bailey
Dimitri Gorokhovik
Cyrill V. Gorcunov
Ahmed Darwish
Miika Komu
Thomas Hellstrom
Mario Rossi
Justin Chen
Leonard Norrgard
Dan Carpenter
Masayuki Nakagawa
Thomas Hisch
Judith Lebzelter
James C Georgas
Conke Hu
16 never seen again

Scanning 2.6.21-2.6.22
Alexandr Andreev
Ville-Pekka Vainio
SUGIOKA Toshinobu
Benjamin Li
Aubrey Li
Dwaine P. Garden
Seth Forshee
Robert Peterson
Stuart Yoder
Frank Mandarino
Mark Glines
Monakhov Dmitriy
Mithlesh Thukral
Andrea Paterniani
14 never seen again

Scanning 2.6.22-2.6.23
Jennifer Hunt
Marc St-Jean
Christian Engelmayer
henry su
Bill Gatliff
Shani Moideen
David Lamparter
Tim Hockin
Jan Kratochvil
Marco Gittler
Junio C Hamano
Maik Hampel
Milind Arun Choudhary
Egor Martovetsky
Konstantin Sharlaimov
Ingo Korb
Santosh Rastapur
MOKUNO Masakazu
Denver Gingerich
19 never seen again

Re: [Ksummit-2008-discuss] How many contributors are we losing

[Theodore Tso]

On Fri, May 30, 2008 at 03:05:12PM -0700, Luck, Tony wrote:
> 
> Probably some of them moved on to do different things (or were
> promoted to manage people who still work on Linux).

There's an interesting unspoken assumption here that people who stop
contributing patches which end up in the Linux kernel mailing "no
longer working on the kernel", or "no longer working in Linux", or
"left the community", or (even more of a stretch) people that we have
somehow driven away or that we have failed because they didnt come
back.

Looking at the list.....

> Werner Almesberger

	Original author of LILO, does networking research using the
	Linux kernel.  (Has shown up and presented papers at various
	conferences such as FISL and Linux.conf.au.)

> Stephen Tweedie

	Still working at Red Hat, my last knowledge was that he was in
	Xen hell.....

> Kylene Jo Hall

	Member of IBM Linux Technology Center, Security Team.
	The security team does a lot more than kernel work....

> Dustin Kirkland

	Was a member of the IBM Linux Technology Center, now working
	for Canonical on Ubuntu's Enterprise Server Team

> Suparna Bhattacharya

	Member of the IBM Linux Technology Center, on leave to get an
	advanced degree.

> Junio C Hamano

	Git maintainer.  :-)

So lots of stories, and there are plenty of people who are still
working at a company doing Linux work; they're just not happening to
contribute to a kernel.  They might be fixing bugs for customers, or
forward porting Xen for an enterprise distro, or many other things
that are intimately related to the Linux kernel --- just not in ways
that result in patches into mainline.

So if we really want some hard numbers on how many kernel developers
we are "losing", we would probably have to try to contact some of
these people and see if they are willing to answer a survey.  But
given your numbers, I really don't think it's as big a problem as some
people make it out to be....

						- Ted

Re: [Ksummit-2008-discuss] How many contributors are we losing

[H. Peter Anvin]

Hugh Dickins wrote:
> 
> We've had some great contributions from drive-bys down the years, 
> and I see that as a gain rather than a loss.  I suppose it's a
> half-full versus half-empty perception.
> 

Well, it can be a blessing and it can be a curse.  It mostly depends on 
if anyone else is willing to take over the maintenance afterwards.

	-hpa

Re: How many contributors are we losing

[Josh Boyer]

On Fri, 30 May 2008 13:23:44 -0700
"Luck, Tony" <tony.luck@intel.com> wrote:

I was bored and went through the list.  When it comes to knowing many
kernel developers, I don't consider myself to have very wide-spread
contact with too many people.  But I was able to point out a few that
might not be day-to-day active, but are still around in some capacity.

> Ulrich Weigand

Still around, mostly working on toolchain stuff.

> Theodore Y. Ts'o

I have no idea who this guy is.  What did he even do? ;)

> Artem Bityuckiy

I'm 99% sure this is Artem Bityutskiy, who's listed in MAINTAINERS and
is actively working on UBIFS.

> Tim Bird

Tim is still active on various lists.

> Jakub Jelínek

This is probably Jakub Jelinek, who's a glibc maintainer.

> kay.sievers@vrfy.org

Kay left?  Is this why sysfs keeps breaking?

> David T. Hollis

David is still active on some of the embedded lists.

> Stephen Tweedie

Stephen is at Red Hat working on Xen stuff.

> Dan Malek

Dan pops his head up from time to time.

> Nico Pitre

Nico is still active in the MTD community.  As active as the MTD
community is anyway.

> Utz Bacher

Utz is working on firmware stuff now I think.

> Joel Schopp

Joel has been transitioning to new roles lately.

> John Linville

Crap.  Just when wireless was starting to work... ;)

> NooneImportant

Seriously?  We had someone actually get a patch in with that name?
That seems wrong...

> Peter Jones

Peter is still around.

> Jenx Axboe

Typo.

> Ryan S. Arnold

Ryan sits down the hall from me.  He's working on glibc for PowerPC.

> Dave Arlie

Typo?

> Jörn Engel

This is one of the many convolutions of Joern Engel's name.  He's
working on Logfs and reviewing MTD patches.

> Adam Jackson

Adam is still around.  He's the X maintainer for Fedora.

> Wolfgang Denk

Wolfgang is quite active.

> bugme-daemon@bugzilla.kernel.org

If bugzilla gave up, what kind of sign should we take that as?

> Tom "spot" Callaway

Tom pops up with an occasional sparc patch from time to time.  He's at
Red Hat.

> Steven A. Falco

Steven is still active on the powerpc lists.

> James Bowes

At Red Hat.  Not exactly sure what his full job is.

> Dave Miller

I knew Dave was getting fed up with some things but I didn't think he
quit... ;)

> Alex Bounine

Alex continues to post patches for some of the Tundra devices.

josh

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

From: Greg KH <greg@kroah.com>
Date: Wed, 28 May 2008 15:51:36 -0700

> My raw numbers show that the number of individual kernel contributors
> continues to increase with every release, so this might not be as much
> of a problem as it's made out to be.

And even based upon pure observation, I sometimes cannot keep up
with the patch submissions even if I lock myself up in my office
for a full straight 12 hours of a day.

That tells me that we are almost nearing a situation where we have
more than we need.

I think this "we need more developers" problem really does not exist
currently.  I see new people appearing all the time, and these are
the kind of new people we need, those who do useful work and aren't
in the "whitespace loop".

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

From: James Bottomley <James.Bottomley@HansenPartnership.com>
Date: Wed, 28 May 2008 17:35:27 -0500

> However, even if there were no recruitment problem at all, getting more
> people involved is always better because it means more contributions.

This is not true at all.

If people are getting involved, just for the sake of being involved,
which there is strong evidence of, then it's not a positive thing.

We want people who are passionate about doing things with the
kernel, are self-motivated, and frankly don't need a ton of hand
holding and do not work on things that require absolutely no
thinking.

Look at anyone who is extremely nimble with the kernel, and ask them
what they worked on to get going with development.  Did Andrew Morton
fixup whitespace errors when he was starting to become familiar with
the tree?  Did I?  No, none of us did this stuff.  We read over the
code and learned how it worked, did a port, optimized a lookup
algorithm somewhere.

Consistently we see people turding with whitespace, and not breaking
out of that cycle.  That is a problem.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Peter Zijlstra]

On Wed, 2008-05-28 at 23:09 -0700, David Miller wrote:
> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Date: Wed, 28 May 2008 17:35:27 -0500
> 
> > However, even if there were no recruitment problem at all, getting more
> > people involved is always better because it means more contributions.
> 
> This is not true at all.
> 
> If people are getting involved, just for the sake of being involved,
> which there is strong evidence of, then it's not a positive thing.
> 
> We want people who are passionate about doing things with the
> kernel, are self-motivated, and frankly don't need a ton of hand
> holding and do not work on things that require absolutely no
> thinking.

I fully agree with this, you need passion and persistence - a will to
make a difference.

Also, the kernel is a lot more complex these days than it was like 5
years ago (not that I would know, since I'm not around that long :-),
and that just means you just get a better challenge!

Now getting such people is hard, I've been forever trying to get my
brother to take up a FOSS project, but the drive seems to be missing. If
we find a way to challenge people, to inspire them, that would be good.

> Look at anyone who is extremely nimble with the kernel, and ask them
> what they worked on to get going with development.  Did Andrew Morton
> fixup whitespace errors when he was starting to become familiar with
> the tree?  Did I?  No, none of us did this stuff.  We read over the
> code and learned how it worked, did a port, optimized a lookup
> algorithm somewhere.

I started by rewriting the page reclaim code ;-) you need to start
somewhere.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Wed, 2008-05-28 at 23:09 -0700, David Miller wrote:
> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Date: Wed, 28 May 2008 17:35:27 -0500
> 
> > However, even if there were no recruitment problem at all, getting more
> > people involved is always better because it means more contributions.
> 
> This is not true at all.
> 
> If people are getting involved, just for the sake of being involved,
> which there is strong evidence of, then it's not a positive thing.

OK, I agree with this.  The problem with our current recruitment process
is that it seems to encourage non-useful contributions, which is what I
want this discussion to try to change.

> We want people who are passionate about doing things with the
> kernel, are self-motivated, and frankly don't need a ton of hand
> holding and do not work on things that require absolutely no
> thinking.

Right, so we need to start the newbies process at the point where
thought is required.   That's why I think bug hunting and reporting
might be a better way to go about it.

> Look at anyone who is extremely nimble with the kernel, and ask them
> what they worked on to get going with development.  Did Andrew Morton
> fixup whitespace errors when he was starting to become familiar with
> the tree?  Did I?  No, none of us did this stuff.  We read over the
> code and learned how it worked, did a port, optimized a lookup
> algorithm somewhere.
> 
> Consistently we see people turding with whitespace, and not breaking
> out of that cycle.  That is a problem.

Agreed.  And it's the problem I'd like to address in this discussion
item if it gets on the agenda.  Along with ways of encouraging more
useful contributions.

I'm not convinced we need to have a graduated programme where we try to
draw absolutely everybody in and up; as you say, people with interest
and passion will always draw themselves in naturally.  However, I do
think we need to provide contribution avenues for people who aren't
necessarily aiming to become full time kernel developers.

Like you, I find little value in whitespace patches (and much
annoyance); especially from people who never graduate from them.
However, if we get a user who's willing to test the kernel of the day on
their strange laptop every few days, report any problems or bugs they
see and work with people to fix them, that's a fantastically useful
service; it's relatively easy for them to do and if it's all they ever
do, it's enough to be very useful.

Really, I think that's what our mistake is in the recruitment program
is: we need to start people out on useful tasks that they can do rather
than on tasks we find annoying and useless in the hope they move on to
something better.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Jiri Kosina]

On Thu, 29 May 2008, James Bottomley wrote:

> Really, I think that's what our mistake is in the recruitment program 
> is: we need to start people out on useful tasks that they can do rather 
> than on tasks we find annoying and useless in the hope they move on to 
> something better.

I fully agree, but my impression is that this really is not going to be 
easy. Fixing bugs definitely is a good way to start kernel coding -- it 
forces you to understand the internals of the source, get used to the 
coding style by reading the code, etc. Unfortunately, it's simply not very 
attractive for newcomers.

For example -- I am leading a seminar at university, oriented at linux 
kernel internals. I provide the possibility to students either to

- write some stand-alone interesting kernel project
- fix a few non-trivial bugs in kernel bugzilla
- chose any part of a kernel, learn how it works, and present this to 
  other seminar attendees

The feedback I often get from students (and these guys are studying 
computer science) is

- writing some wholy new interesting kernel project is usually too 
  complicated (both coming with an interesting idea for a project, and 
  doing the coding itself)
- fixing random bugs from kernel bugzilla is boring (spending 10 hours 
  looking for missing '=' doesn't really attract them)

So in overwhelming majority of cases, they just chose the presentation.


All I want to say is that I could very well imagine that a lot of 
newcomers will find "hey, feel free to crawl through bugzilla and fix 
whatever you are able to fix" very non-attractive.
Not that I have any better idea right now, though.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Wilcox]

On Thu, May 29, 2008 at 05:06:15PM +0200, Jiri Kosina wrote:
> I fully agree, but my impression is that this really is not going to be 
> easy. Fixing bugs definitely is a good way to start kernel coding -- it 
> forces you to understand the internals of the source, get used to the 
> coding style by reading the code, etc. Unfortunately, it's simply not very 
> attractive for newcomers.

"Programming is hard, let's go shopping"?

> For example -- I am leading a seminar at university, oriented at linux 
> kernel internals. I provide the possibility to students either to
> 
> - write some stand-alone interesting kernel project
> - fix a few non-trivial bugs in kernel bugzilla
> - chose any part of a kernel, learn how it works, and present this to 
>   other seminar attendees
> 
> The feedback I often get from students (and these guys are studying 
> computer science) is
> 
> - writing some wholy new interesting kernel project is usually too 
>   complicated (both coming with an interesting idea for a project, and 
>   doing the coding itself)
> - fixing random bugs from kernel bugzilla is boring (spending 10 hours 
>   looking for missing '=' doesn't really attract them)
> 
> So in overwhelming majority of cases, they just chose the presentation.

In general, students are going to choose the easiest option ;-)

Perhaps you could try making the process of finding a kernel bug seem
more exciting?  I doubt that most kernel bugs are a simple as a missing =.
But even if they were, it's about the /process/ you take to get to that
boring patch.

> All I want to say is that I could very well imagine that a lot of 
> newcomers will find "hey, feel free to crawl through bugzilla and fix 
> whatever you are able to fix" very non-attractive.

Of course.  We have to make it sexier than that.

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Stefan Richter]

Matthew Wilcox wrote:
> On Thu, May 29, 2008 at 05:06:15PM +0200, Jiri Kosina wrote:
>> All I want to say is that I could very well imagine that a lot of 
>> newcomers will find "hey, feel free to crawl through bugzilla and fix 
>> whatever you are able to fix" very non-attractive.
> 
> Of course.  We have to make it sexier than that.

A newcomer is motivated to fix a non-trivial bug if his own setup is 
seriously affected, no convenient workaround is possible, and it is 
clear that nobody else is gong to fix the bug (e.g. because the code is 
orphaned).

If you have funds, you may have ways to provide better motivators than this.
-- 
Stefan Richter
-=====-==--- -=-= ====-
http://arcgraph.de/sr/

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Jiri Kosina]

On Thu, 29 May 2008, Matthew Wilcox wrote:

> > - fixing random bugs from kernel bugzilla is boring (spending 10 hours 
> >   looking for missing '=' doesn't really attract them)
> Perhaps you could try making the process of finding a kernel bug seem
> more exciting?  I doubt that most kernel bugs are a simple as a missing =.

Indeed. Large amount of kernel bugs is 'kernel doesn't work on this 
hardware setup X, other hardware setups are not affected". That doesn't 
sound too sexy either, unless you are unfortunate enough to have the piece 
of hardware in question, of course.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Rafael J. Wysocki]

On Monday, 2 of June 2008, Jiri Kosina wrote:
> On Thu, 29 May 2008, Matthew Wilcox wrote:
> 
> > > - fixing random bugs from kernel bugzilla is boring (spending 10 hours 
> > >   looking for missing '=' doesn't really attract them)
> > Perhaps you could try making the process of finding a kernel bug seem
> > more exciting?  I doubt that most kernel bugs are a simple as a missing =.
> 
> Indeed. Large amount of kernel bugs is 'kernel doesn't work on this 
> hardware setup X, other hardware setups are not affected". That doesn't 
> sound too sexy either, unless you are unfortunate enough to have the piece 
> of hardware in question, of course.

Well, if you test linux-next or -mm on a regular basis, you are almost
guaranteed to encounter a bug that manifests itself on your hardware, sooner
or later. :-)

Thanks,
Rafael

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Thu, 2008-05-29 at 17:06 +0200, Jiri Kosina wrote:
> On Thu, 29 May 2008, James Bottomley wrote:
> 
> > Really, I think that's what our mistake is in the recruitment program 
> > is: we need to start people out on useful tasks that they can do rather 
> > than on tasks we find annoying and useless in the hope they move on to 
> > something better.
> 
> I fully agree, but my impression is that this really is not going to be 
> easy. Fixing bugs definitely is a good way to start kernel coding -- it 
> forces you to understand the internals of the source, get used to the 
> coding style by reading the code, etc. Unfortunately, it's simply not very 
> attractive for newcomers.

Yes, that's why I think they don't start with bug fixing, they start
with testing (which, of course naturally leads to bug reporting and
possibly even fixing).

> For example -- I am leading a seminar at university, oriented at linux 
> kernel internals. I provide the possibility to students either to
> 
> - write some stand-alone interesting kernel project
> - fix a few non-trivial bugs in kernel bugzilla
> - chose any part of a kernel, learn how it works, and present this to 
>   other seminar attendees
> 
> The feedback I often get from students (and these guys are studying 
> computer science) is
> 
> - writing some wholy new interesting kernel project is usually too 
>   complicated (both coming with an interesting idea for a project, and 
>   doing the coding itself)
> - fixing random bugs from kernel bugzilla is boring (spending 10 hours 
>   looking for missing '=' doesn't really attract them)
> 
> So in overwhelming majority of cases, they just chose the presentation.
> 
> 
> All I want to say is that I could very well imagine that a lot of 
> newcomers will find "hey, feel free to crawl through bugzilla and fix 
> whatever you are able to fix" very non-attractive.

Right ... for me to fix a bug, it really has to be relevant, which
finding some random one in the bugzilla isn't, so the most motivated
person on a bug should be the reporter.  That's why I think we start
newbies off with testing the kernel of the day and seeing what goes
wrong.  If something does, they have the bug right there in front of
them.  Just reporting it and helping others fix it is a useful service.
If they actually move on to investigating and fixing it themselves, so
much the better.

> Not that I have any better idea right now, though.

Well, apart from bug fixing and testing, I don't either ... that's why
I'm proposing a discussion not a solution.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Rafael J. Wysocki]

On Thursday, 29 of May 2008, James Bottomley wrote:
> On Thu, 2008-05-29 at 17:06 +0200, Jiri Kosina wrote:
> > On Thu, 29 May 2008, James Bottomley wrote:
> > 
> > > Really, I think that's what our mistake is in the recruitment program 
> > > is: we need to start people out on useful tasks that they can do rather 
> > > than on tasks we find annoying and useless in the hope they move on to 
> > > something better.
> > 
> > I fully agree, but my impression is that this really is not going to be 
> > easy. Fixing bugs definitely is a good way to start kernel coding -- it 
> > forces you to understand the internals of the source, get used to the 
> > coding style by reading the code, etc. Unfortunately, it's simply not very 
> > attractive for newcomers.
> 
> Yes, that's why I think they don't start with bug fixing, they start
> with testing (which, of course naturally leads to bug reporting and
> possibly even fixing).
> 
> > For example -- I am leading a seminar at university, oriented at linux 
> > kernel internals. I provide the possibility to students either to
> > 
> > - write some stand-alone interesting kernel project
> > - fix a few non-trivial bugs in kernel bugzilla
> > - chose any part of a kernel, learn how it works, and present this to 
> >   other seminar attendees
> > 
> > The feedback I often get from students (and these guys are studying 
> > computer science) is
> > 
> > - writing some wholy new interesting kernel project is usually too 
> >   complicated (both coming with an interesting idea for a project, and 
> >   doing the coding itself)
> > - fixing random bugs from kernel bugzilla is boring (spending 10 hours 
> >   looking for missing '=' doesn't really attract them)
> > 
> > So in overwhelming majority of cases, they just chose the presentation.
> > 
> > 
> > All I want to say is that I could very well imagine that a lot of 
> > newcomers will find "hey, feel free to crawl through bugzilla and fix 
> > whatever you are able to fix" very non-attractive.
> 
> Right ... for me to fix a bug, it really has to be relevant, which
> finding some random one in the bugzilla isn't, so the most motivated
> person on a bug should be the reporter.

Agreed.

> That's why I think we start newbies off with testing the kernel of the day
> and seeing what goes wrong.  If something does, they have the bug right there
> in front of them.  Just reporting it and helping others fix it is a useful service.
> If they actually move on to investigating and fixing it themselves, so
> much the better.

Agreed again.

Generally, I think a good way to start is to ask yourself if there are any
problems with the kernel that you see on your own box and you'd like to fix,
then try to understand why these problems occur and try to provide a solution.

> > Not that I have any better idea right now, though.
> 
> Well, apart from bug fixing and testing, I don't either ... that's why
> I'm proposing a discussion not a solution.

Well, people usually have some problems with the kernel, not necessarily bugs,
but things that are somehow annoying or apparently possible to improve (eg. to
speed up).  Sometimes the kernel doesn't work as expected, etc.  IMO, it would
be nice if people started from looking at things that don't work for them
ideally.

Thanks,
Rafael

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Lars Noschinski]

* Jiri Kosina <jkosina@suse.cz> [08-05-31 11:48]:
>I fully agree, but my impression is that this really is not going to be 
>easy. Fixing bugs definitely is a good way to start kernel coding -- it 
>forces you to understand the internals of the source, get used to the 
>coding style by reading the code, etc. Unfortunately, it's simply not very 
>attractive for newcomers.
>
>For example -- I am leading a seminar at university, oriented at linux 
>kernel internals. I provide the possibility to students either to
>
>- write some stand-alone interesting kernel project
>- fix a few non-trivial bugs in kernel bugzilla
>- chose any part of a kernel, learn how it works, and present this to 
>  other seminar attendees
>
>The feedback I often get from students (and these guys are studying 
>computer science) is
>
>- writing some wholy new interesting kernel project is usually too 
>  complicated (both coming with an interesting idea for a project, and 
>  doing the coding itself)
>- fixing random bugs from kernel bugzilla is boring (spending 10 hours 
>  looking for missing '=' doesn't really attract them)
>
>So in overwhelming majority of cases, they just chose the presentation.

 From my own experience: Last semester, I participated in a "Linux Kernel
Hacking Lab" which involved some smaller tasks (write a module which
presents a circular buffer in /proc; the same as char device; write
a very minimalistic file system, extend it to do something resembling
journalling, ...) and a larger project.

The first two tasks were easily done by reading Linux Device Drivers;
but after the file system task (2 weeks), more than half of the group
dropped out; probably because to get this task done you needed to dig at
least in the minix code.

The project I was later involved on was writing a file system, which
stores blocks with the same content only once. In a two months period
(with other lectures to attend to), we managed to produce such a thing;
but it was very unreliable (which reminds me, I am supposed to publish
the results of this project here)

Most problems resulted of a combination of not enough documentation
(filesystem/page cache/block device interaction) and "not digging deep
enough, so the kernel does not really do what we expected".

So, the code produced in this learning phase is not usable; but I
learned a lot from this. Restarting from scratch now would probably
yield some working code.

At least, know I understand why Linux has such a high rate of changes ;)

>All I want to say is that I could very well imagine that a lot of 
>newcomers will find "hey, feel free to crawl through bugzilla and fix 
>whatever you are able to fix" very non-attractive.
>Not that I have any better idea right now, though.

I probably would not have attended a lab titled like this ;)

store-same-blocksonce (was Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project)

[Pavel Machek]

Hi!

> The project I was later involved on was writing a file 
> system, which
> stores blocks with the same content only once. In a two 
> months period
> (with other lectures to attend to), we managed to 
> produce such a thing;
> but it was very unreliable (which reminds me, I am 
> supposed to publish
> the results of this project here)

Yep, sounds like a cool project, actually. I guess it could be very
useful for people like me that store many kernel trees around, and
sometimes run out of  disk space.
							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Paul Jackson]

David wrote:
> Did Andrew Morton fixup whitespace errors when he was starting to become
> familiar with the tree?  Did I?  No, none of us did this stuff.

I came in knowing I wanted to do cpusets, and intentionally
picked off an "easier" target first -- overhauling the bitmap,
cpumask, nodemask apparatus -- in order to get up to speed.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.940.382.4214

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Wilcox]

On Wed, May 28, 2008 at 12:20:12PM -0500, James Bottomley wrote:
> In the spirit of having a more process than technical based kernel
> summit, I'd like to put the topic of the kernel Janitors project up for
> discussion.

I suspect many of the people contributing to this thread aren't on the
kernel-janitors list.  They're only seeing janitorial-style patches hit
them in the face, and they know they don't like it.

> On the other hand, as a
> maintainer, when there's people yelling me at about patches not being
> included plus a persistent regressions list and about ten bug reports to
> track down, the last thing I want to see within a million miles of my
> inbox is a white space fixing patch.

Part of the problem is that whitespace fixing patches are treated like
they're real patches.  We get conflicts with real patches, they're pinged
like they're real patches, and the contributors name gets put in the
shortlog like it was a real patch.

I liked the trivial service that Rusty had setup.  Patch started getting
conflicts?  Dropped.  Pinged?  By an automated device, not by akpm.  We
didn't have contributor names back then, but maybe we should anonymise
whitespace patches -- just accept that something that could have been
done by a machine isn't worth attaching a name to.

> The most obvious solution might be to shut the Janitors project down, or
> at least more tightly manage its TODO list (although a lot of what gets
> seen as janitorial patches, like whitespace fixes, isn't on the TODO
> list in the first place).

Also, a lot of the trivial patches don't come across the janitors list
first, so shutting the project down will not, IMO, reduce the problem.
So, most obvious, but wrong ;-)

> However, since the purpose is to get new
> people involved with kernel development, perhaps we should repurpose the
> project so it actually does this.  My suggestion is that we replace it
> with the kernel bugs project.  Kudos for finding bugs, more for finding
> better ways of finding bugs, and the most for finding and actually
> fixing a bug.  

That's a good idea.  We'd all be happier if more people spent time on
bug triage, investigation and fixing.

> Perhaps we should simply start the discussion with the premise that we
> want to encourage new people to do useful work and draw them into the
> development community and see where it leads.

There are some of those things going on in the current janitors project,
it's just they're drowned out by the noise.  I'd like to mention a few.

Mark Asselstine has been working to remove the last remnants of cli/sti
from the kernel.  I believe patches to get rid of them all are now
either merged or sitting in trees waiting to be merged.  I found him
receptive to feedback and willing to sit down and really analyse a
driver to figure out what was going on.  If he chooses to continue his
kernel hacking career, he'll be a great asset.

Julia Lawall has a fun tool that spots patterns which are buggy.  I've
worked on a number of patches with her recently where we compare
unsigned integers < 0.  Again, we need people to look at the piece of
code in context to figure out what the original author meant (not too
dissimilar from the coverity reports).

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

From: James Bottomley <James.Bottomley@HansenPartnership.com>
Date: Wed, 28 May 2008 12:20:12 -0500

> In the spirit of having a more process than technical based kernel
> summit, I'd like to put the topic of the kernel Janitors project up for
> discussion.

Not to distract from your points, but I think it's been a huge
mistake to not allow folks to work hard on technical issues
during the summit.

I feel like I'm at the edge of my seat dieing to talk about
some tech topics, but it's all of this process stuff, it makes
me want to disappear into the local cafe during the main
summit with some folks and talk tech.

I think it would even fix the attitude and process issues,
to be able to talk face-to-face about technical issues instead
of always hashing stuff out in email which is the source of
all of the bad behavior.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Benjamin Herrenschmidt]

On Wed, 2008-05-28 at 22:58 -0700, David Miller wrote:
> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Date: Wed, 28 May 2008 12:20:12 -0500
> 
> > In the spirit of having a more process than technical based kernel
> > summit, I'd like to put the topic of the kernel Janitors project up for
> > discussion.
> 
> Not to distract from your points, but I think it's been a huge
> mistake to not allow folks to work hard on technical issues
> during the summit.

Having been stopped a couple of times last year when trying to bring
some technical subjects for the reason that they were "off topic, KS is
for process" I tend to agree :-)

There are some things that are hard to get through the list, or even
IRC, such as wanting to reorganize some part of the core and wanting to
get "live" input and discussion with most arch maintainers.

In my case it was some rework on some of the page table batching, for
which I finally didn't finish the work as I'm not too happy with what I
came up with, but I'm still somewhat convinced there's something to do
there and I could use a bit of a group beat-up in a room.

There are other areas like that which would benefit from that kind of
hard core tech discussion face to face.

However, on the other hand, KS is only 2 days, which doesn't leave room
for that much stuff. And I don't think splitting into sub-groups or
mini-summits is necessarily the answers. There are some areas where it's
useful to do the tech beatup -with- everybody in the room.

Just my 2 cents... Maybe we need 2 summits, the process one and the tech
one ? :-)

Cheers,
Ben.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Theodore Tso]

On Thu, May 29, 2008 at 04:17:19PM +1000, Benjamin Herrenschmidt wrote:
> Having been stopped a couple of times last year when trying to bring
> some technical subjects for the reason that they were "off topic, KS is
> for process" I tend to agree :-)

We did have some technical topics last year.  So I don't think the KS
will ever be purely for process.

> There are some things that are hard to get through the list, or even
> IRC, such as wanting to reorganize some part of the core and wanting to
> get "live" input and discussion with most arch maintainers.
> 
> There are other areas like that which would benefit from that kind of
> hard core tech discussion face to face.

I agree that sometimes face-to-face discussions are crucial to
resolving technical issues.  The problem is that we try to nail down
the agenda a 3-4 weeks ahead of time, and realistically if a
particular topic requires certain stakeholders to be invited, we need
to decide that we're going to do that topic at least 8-9 weeks in
advance, maybe more, so those people can make travel plans and/or get
travel approvals.  And there is always the chance the topic will
resolve itself via e-mail.  So the trick is being able to identifying
the topics where a face-to-face discussion really will be useful.

> However, on the other hand, KS is only 2 days, which doesn't leave room
> for that much stuff. And I don't think splitting into sub-groups or
> mini-summits is necessarily the answers. There are some areas where it's
> useful to do the tech beatup -with- everybody in the room.

First of all, if there is interest in holding some topic-area specific
mini-summits on Sunday before the Kernel Summit, we may be able to
scare up some space.  So if there is interest, please let me know.

Secondly, one of the things which has been suggested in the past is
that we move the BOF's into an afternoon session slot, and maybe push
the last session to run until 7pm, perhaps.  That might make it easier
for people to attend BOF's on the spur of the moment --- which might
be useful in some cases where there is only 10 or so people you need
to hash our some issue.

And, of course, we try to schedule plenty of break time so that some
of these discussions can happy in the hallway.

Do any of these possibilities sound particularly attractive or likely
to address your concerns?  If so, which ones do you think we should
try this year, as an experiment?

						- Ted

RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Thu, 2008-05-29 at 08:45 -0400, Theodore Tso wrote:
>  there is always the chance the topic will resolve itself via e-mail. 

I'm kind of hoping that this is going to be one of those topics that
resolves itself -- but just in case it isn't, I'd like to discuss it at
the kernel summit:

I'd like to remove all firmware blobs from the kernel source tree.

With the intention of making that less contentious, I've done the
following: 

I've added support to the firmware loader for finding firmware in a
special section of the static kernel, so that using request_firmware()
no longer forces you to use an initrd or udev. You can build _arbitrary_
firmware files into your kernel, if you want to. Whether you're allowed
to distribute the resulting vmlinux or not is still a question you ought
to ask your lawyer.

I'm working on converting drivers over to use request_firmware(), and
shifting their firmware blobs into .ihex files in a firmware/ directory
of the source tree. For each one, I'm giving the option of continuing to
build it in statically, using the above mechanism.

I've implemented a 'make firmware_install' Kbuild target, which takes
all those firmware blobs and installs them somewhere where udev can load
them on request.

By the time the kernel summit comes around, we should have made decent
progress on moving _all_ the firmware blobs to the firmware/ directory.
And at that point I'd like to remove them completely, to a separate git
tree and tarball. Those who really want to build them in to their static
kernel would still be able to, but it wouldn't be the default behaviour.

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Greg KH]

On Thu, May 29, 2008 at 07:15:00PM +0300, David Woodhouse wrote:
> On Thu, 2008-05-29 at 08:45 -0400, Theodore Tso wrote:
> >  there is always the chance the topic will resolve itself via e-mail. 
> 
> I'm kind of hoping that this is going to be one of those topics that
> resolves itself -- but just in case it isn't, I'd like to discuss it at
> the kernel summit:
> 
> I'd like to remove all firmware blobs from the kernel source tree.
> 
> With the intention of making that less contentious, I've done the
> following: 
> 
> I've added support to the firmware loader for finding firmware in a
> special section of the static kernel, so that using request_firmware()
> no longer forces you to use an initrd or udev. You can build _arbitrary_
> firmware files into your kernel, if you want to. Whether you're allowed
> to distribute the resulting vmlinux or not is still a question you ought
> to ask your lawyer.
> 
> I'm working on converting drivers over to use request_firmware(), and
> shifting their firmware blobs into .ihex files in a firmware/ directory
> of the source tree. For each one, I'm giving the option of continuing to
> build it in statically, using the above mechanism.
> 
> I've implemented a 'make firmware_install' Kbuild target, which takes
> all those firmware blobs and installs them somewhere where udev can load
> them on request.

I like and agree with all of this.

> By the time the kernel summit comes around, we should have made decent
> progress on moving _all_ the firmware blobs to the firmware/ directory.

That's good too.

> And at that point I'd like to remove them completely, to a separate git
> tree and tarball. Those who really want to build them in to their static
> kernel would still be able to, but it wouldn't be the default behaviour.

This I disagree with.  Leaving it in a single directory makes it easier
for some distros to patch it away to apease their feelings about the
issue.  But for some of use, we like the firmware in our kernels, it
makes it easier to boot for one, and we don't have to deal with
different firmware packages.  So I say leave them in, with the option to
get them elsewhere, like you are doing.

thanks,

greg k-h

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Arjan van de Ven]

Greg KH wrote:
>> And at that point I'd like to remove them completely, to a separate git
>> tree and tarball. Those who really want to build them in to their static
>> kernel would still be able to, but it wouldn't be the default behaviour.
> 
> This I disagree with.  Leaving it in a single directory makes it easier
> for some distros to patch it away to apease their feelings about the
> issue.  But for some of use, we like the firmware in our kernels, it
> makes it easier to boot for one, and we don't have to deal with
> different firmware packages.  So I say leave them in, with the option to
> get them elsewhere, like you are doing.
> 

I very much would like to see a kernel-firmware or something tarbal that contains
a copy of all relevant "freely distributable" firmware, that users can just install
independent of the actual kernel version (and that kbuild would just pick up somehow).
That way we can deal with a lot more firmware without having to pollute the kernel / kernel release process
(after all, the timing is different in terms of releasing) while making it easy
to get the lot of it.

Now.. of course what "freely distributable" means is a huge debate. I don't care ;)

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Matthew Wilcox]

On Thu, May 29, 2008 at 01:29:38PM -0700, Arjan van de Ven wrote:
> I very much would like to see a kernel-firmware or something tarbal that contains
> a copy of all relevant "freely distributable" firmware, that users can just install
> independent of the actual kernel version (and that kbuild would just pick up somehow).
> That way we can deal with a lot more firmware without having to pollute the kernel / kernel release process
> (after all, the timing is different in terms of releasing) while making it easy
> to get the lot of it.

There's definitely two schools of thought on this.  Sometimes firmware
changes (or adds) an interface.  If the kernel driver has to accommodate
new and old firmware, that adds complexity, and we all know that added
complexity means more bugs.  So I can definitely see some vendors
wanting to distribute their firmware with the kernel.

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Yinghai Lu]

On Thu, May 29, 2008 at 1:47 PM, Matthew Wilcox <matthew@wil.cx> wrote:
> On Thu, May 29, 2008 at 01:29:38PM -0700, Arjan van de Ven wrote:
>> I very much would like to see a kernel-firmware or something tarbal that contains
>> a copy of all relevant "freely distributable" firmware, that users can just install
>> independent of the actual kernel version (and that kbuild would just pick up somehow).
>> That way we can deal with a lot more firmware without having to pollute the kernel / kernel release process
>> (after all, the timing is different in terms of releasing) while making it easy
>> to get the lot of it.
>
> There's definitely two schools of thought on this.  Sometimes firmware
> changes (or adds) an interface.  If the kernel driver has to accommodate
> new and old firmware, that adds complexity, and we all know that added
> complexity means more bugs.  So I can definitely see some vendors
> wanting to distribute their firmware with the kernel.
>
driver should check fw version...

YH

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[James Bottomley]

On Thu, 2008-05-29 at 13:55 -0700, Yinghai Lu wrote:
> On Thu, May 29, 2008 at 1:47 PM, Matthew Wilcox <matthew@wil.cx> wrote:
> > On Thu, May 29, 2008 at 01:29:38PM -0700, Arjan van de Ven wrote:
> >> I very much would like to see a kernel-firmware or something tarbal that contains
> >> a copy of all relevant "freely distributable" firmware, that users can just install
> >> independent of the actual kernel version (and that kbuild would just pick up somehow).
> >> That way we can deal with a lot more firmware without having to pollute the kernel / kernel release process
> >> (after all, the timing is different in terms of releasing) while making it easy
> >> to get the lot of it.
> >
> > There's definitely two schools of thought on this.  Sometimes firmware
> > changes (or adds) an interface.  If the kernel driver has to accommodate
> > new and old firmware, that adds complexity, and we all know that added
> > complexity means more bugs.  So I can definitely see some vendors
> > wanting to distribute their firmware with the kernel.
> >
> driver should check fw version...

Drivers with built in firmware usually don't.  Most don't actually have
even a version string they could check.  Why would they: the firmware
with the driver is the right one.   When I converted the aic94xx driver
to go from built in firmware to externally loaded, the first thing I had
to do was to give it a firmware version string in the binary.  It's this
type of problem that makes the conversion such a pain.

James

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Greg KH]

On Thu, May 29, 2008 at 01:55:16PM -0700, Yinghai Lu wrote:
> On Thu, May 29, 2008 at 1:47 PM, Matthew Wilcox <matthew@wil.cx> wrote:
> > On Thu, May 29, 2008 at 01:29:38PM -0700, Arjan van de Ven wrote:
> >> I very much would like to see a kernel-firmware or something tarbal that contains
> >> a copy of all relevant "freely distributable" firmware, that users can just install
> >> independent of the actual kernel version (and that kbuild would just pick up somehow).
> >> That way we can deal with a lot more firmware without having to pollute the kernel / kernel release process
> >> (after all, the timing is different in terms of releasing) while making it easy
> >> to get the lot of it.
> >
> > There's definitely two schools of thought on this.  Sometimes firmware
> > changes (or adds) an interface.  If the kernel driver has to accommodate
> > new and old firmware, that adds complexity, and we all know that added
> > complexity means more bugs.  So I can definitely see some vendors
> > wanting to distribute their firmware with the kernel.
> >
> driver should check fw version...

Not all firmware has the ability to check the version :(

So while Arjan's goal would be nice, Matthew is right, this can't happen
for all types of firmware.

thanks,

greg k-h

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Alan Cox]

> Not all firmware has the ability to check the version :(
> 
> So while Arjan's goal would be nice, Matthew is right, this can't happen
> for all types of firmware.

Well for loaded or packaged firmware it can happen. Just stick a revision
number on the start of the firmware file and check it rather than load it
into the hardware in that specific driver.

Alan

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Fri, 2008-05-30 at 10:20 +0100, Alan Cox wrote:
> > Not all firmware has the ability to check the version :(
> > 
> > So while Arjan's goal would be nice, Matthew is right, this can't happen
> > for all types of firmware.
> 
> Well for loaded or packaged firmware it can happen. Just stick a revision
> number on the start of the firmware file and check it rather than load it
> into the hardware in that specific driver.

Or append it to the name of the firmware file, in some cases.

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Miller]

From: "Yinghai Lu" <yhlu.kernel@gmail.com>
Date: Thu, 29 May 2008 13:55:16 -0700

> driver should check fw version...

This alone is not a reason to rip the firmware out into a seperate
tree.  And I am absolutely not convinced that the cases where this
matters all universally even use firmware versions.

I've installed the wrong ipw2200 on several occaisions.

Furthermore, it's about distributing what works with what it's meant
to work with.  With this seperate scheme, I can still link in the
wrong firmware file (the driver doesn't check the firmware version
until it executes) and the driver won't work.  So this moves the
validation to run time, which users typically don't appreciate.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Yinghai Lu]

On Thu, May 29, 2008 at 2:31 PM, David Miller <davem@davemloft.net> wrote:
> From: "Yinghai Lu" <yhlu.kernel@gmail.com>
> Date: Thu, 29 May 2008 13:55:16 -0700
>
>> driver should check fw version...
>
> This alone is not a reason to rip the firmware out into a seperate
> tree.  And I am absolutely not convinced that the cases where this
> matters all universally even use firmware versions.
>
> I've installed the wrong ipw2200 on several occaisions.
>
> Furthermore, it's about distributing what works with what it's meant
> to work with.  With this seperate scheme, I can still link in the
> wrong firmware file (the driver doesn't check the firmware version
> until it executes) and the driver won't work.  So this moves the
> validation to run time, which users typically don't appreciate.

I agree. I hate to have two klibc (one for 32 bit, and one for 64bit)
to for initramfs to load FW for qlogic...

dwmw2 's new patches to put all fw in /firmware, and built them into
kernel could avoid that...

and could make the build process to check fw version to match with
driver in different kernel version later...

YH

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Takashi Iwai]

At Thu, 29 May 2008 14:57:38 -0700,
Yinghai Lu wrote:
> 
> On Thu, May 29, 2008 at 2:31 PM, David Miller <davem@davemloft.net> wrote:
> > From: "Yinghai Lu" <yhlu.kernel@gmail.com>
> > Date: Thu, 29 May 2008 13:55:16 -0700
> >
> >> driver should check fw version...
> >
> > This alone is not a reason to rip the firmware out into a seperate
> > tree.  And I am absolutely not convinced that the cases where this
> > matters all universally even use firmware versions.
> >
> > I've installed the wrong ipw2200 on several occaisions.
> >
> > Furthermore, it's about distributing what works with what it's meant
> > to work with.  With this seperate scheme, I can still link in the
> > wrong firmware file (the driver doesn't check the firmware version
> > until it executes) and the driver won't work.  So this moves the
> > validation to run time, which users typically don't appreciate.
> 
> I agree. I hate to have two klibc (one for 32 bit, and one for 64bit)
> to for initramfs to load FW for qlogic...
> 
> dwmw2 's new patches to put all fw in /firmware, and built them into
> kernel could avoid that...
> 
> and could make the build process to check fw version to match with
> driver in different kernel version later...

OTOH, it doesn't give you any error at build time even if you forget
to put a firmware image beforehand.  The kernel continues to look for
a non-existing external firmware file.  In the old code, this can't
happen.

It's just a small drawback, and I still like the idea very well,
though.


Takashi

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Fri, 2008-05-30 at 11:52 +0200, Takashi Iwai wrote:
> OTOH, it doesn't give you any error at build time even if you forget
> to put a firmware image beforehand.  The kernel continues to look for
> a non-existing external firmware file.  In the old code, this can't
> happen.

Perhaps we should make depmod also check all MODULE_FIRMWARE() and
complain about missing firmware which might be used? We could make it
(or something else) check for firmwares required by drivers built in to
the kernel, too.

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Miller]

Arjan, by definition the firmware has to be tied to the kernel somehow.
THe datastructure and other aspects are often tied directly to
the firmware version loaded.

At first, people said "hey we're going to make the firmware loadable
from userspace, so sorry now you can't load your driver without a
ramdisk, sorry!"  And that royally pissed me off, and directly impacted
my work flow negatively.

Now, for the cases where we do have in-kernel firmware still, people
are suggesting to seperate that out to a seperate tree as well.
Sorry, that's taking things too far.  I've fought, like, forever, to
keep the tg3 driver with it's firmware in-tree.  I refuse to let the
driver get broken like that, it's staying working, and that means
in-tree and linked into the driver.

If debian or whoever else have these concernes and want to rip the
firmware out, it is one hundred percent their problem to patch things
out of the kernel tree they use.

At least from my perspective this looks like a transfer of burdon from
the folks who want to rip the firmware out, to those of us who find
high value in the firmware staying in the tree.

This is really irritating me, because this is one huge case of frickin
Animal Farm.  First a little was taken away, then a little bit more,
and by the end you have something absolutely nobody would have agreed
to from the beginning.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Arjan van de Ven]

David Miller wrote:
> Arjan, by definition the firmware has to be tied to the kernel somehow.
> THe datastructure and other aspects are often tied directly to
> the firmware version loaded.

agreed that there are cases like this, and I have no personal objection to having
those in the kernel.

> If debian or whoever else have these concernes and want to rip the
> firmware out, it is one hundred percent their problem to patch things
> out of the kernel tree they use.

I don't care at all about the argument from that camp.


My aim was more the opposite: be able to get MORE firmware easily used/loaded,
not less. Right now it's a royal pain for users to get all the right pieces of
firmware.... having ONE place to put all that would go a long way of making that
side of things easier.

If you want to argue that that should be in the kernel tarbal itself, you won't
hear me complain. But others will... and for that a 2nd tarbal might well be the answer.
Just we shouldn't need 100 tarbals.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Thu, 2008-05-29 at 14:11 -0700, Arjan van de Ven wrote: 
> David Miller wrote:
> > Arjan, by definition the firmware has to be tied to the kernel somehow.
> > THe datastructure and other aspects are often tied directly to
> > the firmware version loaded.
> 
> agreed that there are cases like this, and I have no personal objection to having
> those in the kernel.
> 
> > If debian or whoever else have these concernes and want to rip the
> > firmware out, it is one hundred percent their problem to patch things
> > out of the kernel tree they use.
> 
> I don't care at all about the argument from that camp.

I do -- partly because it's not just from the fundamentalist camp,
partly because (after the work I've been doing) it doesn't actually
_hurt_ us to assume that they're right, but mostly because I have a
horrible suspicion that they _are_ right.

Expanding on those three points not in that order...:

The firmware is an independent and separate work in itself. Section 2 of
the GPL talks about such sections of the work, explicitly. The only way
to excuse what we're doing at the moment is to call it 'mere
aggregation' -- an exception which was intended to handle stuff like the
'freeware' CDs on the covers of magazines, distributing a bunch of
unrelated software. Not a coherent work combining software from
different sources into a single entity which works closely together as
one, and where one part is useless without the other.

The more that people claim it would be such a burden to split the
firmware from their driver because they're so closely interrelated, the
more they are arguing against the 'mere aggregation' defence, which was
tenuous enough in the first place.

And it isn't just the nutters. Fedora also wants to ship the firmware in
a separate package from the kernel -- since the alleged GPL violation is
such a _gratuitous_ risk given that we always use an initrd anyway, and
because people want to be able to do 'Free' spins which don't feature
the firmware at all, even in the source packages.

I strongly suspect we'd end up building the Fedora kernel from a special
'linux-2.6.2x-nofirmware.tar.bz2' tarball, rather than using the stock
tarballs if they continue to include the firmware. We do similar things
with 'openssh-5.0p1-noacss.tar.bz2' already, for example.

> My aim was more the opposite: be able to get MORE firmware easily used/loaded,
> not less. Right now it's a royal pain for users to get all the right pieces of
> firmware.... having ONE place to put all that would go a long way of making that
> side of things easier.

There are people who own copyright on firmware who refuse to put it into
the Linux source tree, because their lawyers don't believe the 'mere
aggregation' line, and believe that including it in the kernel source in
any form would require them to license it under the GPL.

But those same companies _would_ consider putting their firmware into a
non-GPL'd 'linux-firmware' repository instead. So by moving our firmware
out into a separate tree, we should actually make it _easier_ for people
to find all the necessary firmware in one place.

It's not as if it's hard to set CONFIG_BUILTIN_FIRMWARE_DIR to point to
your checkout of the linux-firmware repository, and build your kernel
with _whatever_ firmware you choose. You end up with _more_ choice, not
less.

And on the rare occasion that we really do have an incompatible change
of firmware/kernel interaction from one kernel to the next, it really
isn't difficult to add a version number to the name of the firmware
file, and ship both old and new firmwares in the firmware tree for a
while. That's a bogus argument, even if people _do_ manage to come up
with a better example for it, where their firmware has actually changed
in the last couple of years.

> If you want to argue that that should be in the kernel tarbal itself, you won't
> hear me complain. But others will... and for that a 2nd tarbal might well be the answer.

As soon as we have firmware converted to ihex files in the kernel source
tree, I'll set up a 'shadow' git tree like my kernel-headers tree, which
automatically follows the linux-2.6.git tree commit by commit, and
contains just the results you'd get from 'make firmware_install'.

At that point, when the distributions want to do it separately and
individuals who want to build firmware into their kernel really could
continue to do so _very_ easily, I just don't see any good reason why
we'd continue to keep them together.

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Arnaldo Carvalho de Melo]

Em Fri, May 30, 2008 at 02:04:18AM +0300, David Woodhouse escreveu:
> It's not as if it's hard to set CONFIG_BUILTIN_FIRMWARE_DIR to point to
> your checkout of the linux-firmware repository, and build your kernel
> with _whatever_ firmware you choose. You end up with _more_ choice, not
> less.
> 
> And on the rare occasion that we really do have an incompatible change
> of firmware/kernel interaction from one kernel to the next, it really
> isn't difficult to add a version number to the name of the firmware
> file, and ship both old and new firmwares in the firmware tree for a
> while. That's a bogus argument, even if people _do_ manage to come up
> with a better example for it, where their firmware has actually changed
> in the last couple of years.

While updating the bnx2 driver on a 2.6.24.7 based rpm and after reading
some of your messages...

commit df7f1ed6b85b936a4dd341c48e30aa207697997c
Author: Michael Chan <mchan@broadcom.com>
Date:   Tue Jan 29 21:38:06 2008 -0800

    [BNX2]: Update firmware.
    
    Update firmware to support programmable flow control.
    
    Signed-off-by: Michael Chan <mchan@broadcom.com>
    Signed-off-by: David S. Miller <davem@davemloft.net>

After this bnx2.c would be updated to have a:

MODULE_FIRMWARE_DEFAULT("df7f1ed6b85b936a4dd341c48e30aa207697997c");

to be obtained from firmware.git, and if some chip revisions needed
something different we could have a MODULE_FIRMWARE_PCI(vendor, product,
older-git-object) or something along these lines.

Some drivers could well prefer a more loose keying scheme, like
"foo-firmware-v2" and get the latest version of this from firmware.git.

- Arnaldo

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Pavel Machek]

Hi!

> The firmware is an independent and separate work in itself. Section 2 of
> the GPL talks about such sections of the work, explicitly. The only way
> to excuse what we're doing at the moment is to call it 'mere
> aggregation' -- an exception which was intended to handle stuff like the
> 'freeware' CDs on the covers of magazines, distributing a bunch of
> unrelated software. Not a coherent work combining software from
> different sources into a single entity which works closely together as
> one, and where one part is useless without the other.

Wait a moment, haven't you just described linux distribution?

I mean, if aggregation clause does not work for firmware in kernel,
why would it work for packages in distro?

(Actually, seeing some distro EULAs, I wished GPL infected whole
distro so that I'd not have to read the stupid EULA.)

> There are people who own copyright on firmware who refuse to put it into
> the Linux source tree, because their lawyers don't believe the 'mere
> aggregation' line, and believe that including it in the kernel source in
> any form would require them to license it under the GPL.

They can release the firmware under BSD 3-clause, and we can include
it in kernel, then.... right? (Or into linux-firmware or into whatever
package that comes handy).

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Sun, 2008-06-01 at 18:17 +0200, Pavel Machek wrote:
> Hi!
> 
> > The firmware is an independent and separate work in itself. Section 2 of
> > the GPL talks about such sections of the work, explicitly. The only way
> > to excuse what we're doing at the moment is to call it 'mere
> > aggregation' -- an exception which was intended to handle stuff like the
> > 'freeware' CDs on the covers of magazines, distributing a bunch of
> > unrelated software. Not a coherent work combining software from
> > different sources into a single entity which works closely together as
> > one, and where one part is useless without the other.
> 
> Wait a moment, haven't you just described linux distribution?
> 
> I mean, if aggregation clause does not work for firmware in kernel,
> why would it work for packages in distro?
>
> (Actually, seeing some distro EULAs, I wished GPL infected whole
> distro so that I'd not have to read the stupid EULA.)

That's an interesting question, which has concerned me in the past. 
There are certainly those who would claim that the distribution _is_ a
collective work, and not just 'mere aggregation on a volume of a storage
or distribution medium'. Which poses interesting questions.

I think you might _just_ about get away with arguing to the contrary, as
long as you never refer to the distribution as a collective work which
is copyrightable in its own right, or try to put a EULA on it or
anything like that. At least it's _slightly_ more excusable in that case
than what we were talking about before.

Again, there's no definitive answer until/unless it comes to court. But
there's no _guarantee_ that a court would rule that what we're doing is
OK, even though it's a long-standing and universally accepted practice.
You can't apply reductio ad absurdum and declare that the whole
'collective work' part of §2 doesn't exist just because it might lead to
a _really_ surprising conclusion.

> > There are people who own copyright on firmware who refuse to put it into
> > the Linux source tree, because their lawyers don't believe the 'mere
> > aggregation' line, and believe that including it in the kernel source in
> > any form would require them to license it under the GPL.
> 
> They can release the firmware under BSD 3-clause, and we can include
> it in kernel, then.... right? (Or into linux-firmware or into whatever
> package that comes handy).

You may only include it in a GPL'd project if you can distribute the
source code in the preferred form for editing -- unless you claim that
the tightly intertwined combination of driver and firmware is "mere
aggregation on a volume of a storage medium", which many would say it
blatantly is not.

If, on the other hand, we have a separate repository where stuff only
has to be 'redistributable', then that would be OK even without source.

I'm working on reducing the technical barriers to having a separate
repository, to the point where it becomes silly not to do it like that.

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Pavel Machek]

Hi!

> > > The firmware is an independent and separate work in itself. Section 2 of
> > > the GPL talks about such sections of the work, explicitly. The only way
> > > to excuse what we're doing at the moment is to call it 'mere
> > > aggregation' -- an exception which was intended to handle stuff like the
> > > 'freeware' CDs on the covers of magazines, distributing a bunch of
> > > unrelated software. Not a coherent work combining software from
> > > different sources into a single entity which works closely together as
> > > one, and where one part is useless without the other.
> > 
> > Wait a moment, haven't you just described linux distribution?
> > 
> > I mean, if aggregation clause does not work for firmware in kernel,
> > why would it work for packages in distro?
> >
> > (Actually, seeing some distro EULAs, I wished GPL infected whole
> > distro so that I'd not have to read the stupid EULA.)
> 
> That's an interesting question, which has concerned me in the past. 
> There are certainly those who would claim that the distribution _is_ a
> collective work, and not just 'mere aggregation on a volume of a storage
> or distribution medium'. Which poses interesting questions.
> 
> I think you might _just_ about get away with arguing to the contrary, as
> long as you never refer to the distribution as a collective work which
> is copyrightable in its own right, or try to put a EULA on it or
> anything like that. At least it's _slightly_ more excusable in that case
> than what we were talking about before.

Well, distos I seen, did try to claim whole distro is collective work,
and did have eula in the front :-(.

> > > There are people who own copyright on firmware who refuse to put it into
> > > the Linux source tree, because their lawyers don't believe the 'mere
> > > aggregation' line, and believe that including it in the kernel source in
> > > any form would require them to license it under the GPL.
> > 
> > They can release the firmware under BSD 3-clause, and we can include
> > it in kernel, then.... right? (Or into linux-firmware or into whatever
> > package that comes handy).
> 
> You may only include it in a GPL'd project if you can distribute the
> source code in the preferred form for editing -- unless you claim that

Oops, you are right; I overlooked that.

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Mauro Carvalho Chehab]

On Fri, 30 May 2008 02:04:18 +0300
David Woodhouse <dwmw2@infradead.org> wrote:

> The firmware is an independent and separate work in itself. Section 2 of
> the GPL talks about such sections of the work, explicitly.

For me, both inside kernel or on a separate tree would produce similar results.
Yet, a separate tree will be a little more painful.

One alternative to keep this at kernel -git tree would be to write a COPYING or
LICENSE file at firmware dir, explicitly stating that most of those firmwares
have proprietary licenses and are there just to make easier for kernel develoment.

Being in-tree or out kernel tree, we should explicitly state what license each
firmware uses, and having the license terms of proprietary firmwares there.

I would add a metatag at the .ihex file to reference to what license applies to
each firmware. Something like FIRMWARE_LICENSE("some vendor license #1"). For
those legacy firmwares where we don't know, we may simply add "unknown".

Cheers,
Mauro

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Thu, 2008-05-29 at 14:09 -0700, David Miller wrote:
> I've fought, like, forever, to
> keep the tg3 driver with it's firmware in-tree.  I refuse to let the
> driver get broken like that, it's staying working, and that means
> in-tree and linked into the driver.

Am I missing something here, or is the tg3 firmware contained in
'tg3FwText' and similar non-swappable u32 arrays which haven't changed
_ever_ in our current git history?

-- 
dwmw2

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Grant Grundler]

On Thu, May 29, 2008 at 3:11 PM, David Woodhouse <dwmw2@infradead.org> wrote:
> On Thu, 2008-05-29 at 14:09 -0700, David Miller wrote:
>> I've fought, like, forever, to
>> keep the tg3 driver with it's firmware in-tree.  I refuse to let the
>> driver get broken like that, it's staying working, and that means
>> in-tree and linked into the driver.
>
> Am I missing something here, or is the tg3 firmware contained in
> 'tg3FwText' and similar non-swappable u32 arrays which haven't changed
> _ever_ in our current git history?

It certainly has changed. Maybe that was before the move to git.
ISTR, Last time I looked at the tg3 firmware, it was on version 1.4 or 1.5.

thanks,
grant

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Alexandre Oliva]

On May 29, 2008, David Woodhouse <dwmw2@infradead.org> wrote:

> it's not just from the fundamentalist camp, [...]

> And it isn't just the nutters.

FTR, I take offense at that!  Olives are *not* nuts!  I'm a different
kind of appetizer :-)


On May 29, 2008, David Miller <davem@davemloft.net> wrote:

> If debian or whoever else have these concernes and want to rip the
> firmware out, it is one hundred percent their problem to patch things
> out of the kernel tree they use.

I saw this and some other claims upthread that appear to indicate that
this patch hit a nerve that triggers the well-known Free Software
Movement rejection reflex, so common in LKML.

Moving non-Free firmware within, or even out of, the Linux tree,
wouldn't advance our mission at all.  It wouldn't even make
maintaining Linux-libre easier.  I'd rather it weren't merged.
http://www.fsfla.org/~lxoliva/fsfla/linux-libre/
http://fsfla.org/mailman/listinfo/linux-libre

This may be hard for some of you to believe.  I have no doubt some
might even think this is some unethical tactics to get the patch in.
It's not.  I'm serious when I say I'd rather it weren't merged.
Please let me explain.  This will take some understanding of what I
care about, and what I'm trying to accomplish in life; and some
tolerance to arguments involving ideology, freedom and ethics, because
these are values that move me.  I don't ask you to agree with or share
these values, but if you want to make sense of the paragraph above,
you'll have to at least try to understand what matters to me.


The tolerance for non-Free Software in Linux's sources (and anywhere
else), be it non-Free firmware blobs, be it drivers developed under
NDA (whose code is obscured and harder or impossible to understand and
adapt to one's needs as a consequence of the NDA), all revolve around
acceptance, endorsement and even promotion of unethical practices that
I don't want to condone or participate in.

Working towards retaining the ability for people to distribute and use
blobs along with Linux, rather than merely removing the blobs like we
do in Linux-libre, amounts to condoning this practice.  It does not
advance our cause.  In fact, as others pointed out, such changes make
it easier for unethical vendors to add even more of their blobs to
Linux (or co-maintained packages), which is actually detrimental to
our cause:

On May 29, 2008, Arjan van de Ven <arjan@linux.intel.com> wrote:

> My aim was more the opposite: be able to get MORE firmware easily
> used/loaded, not less.

On May 29, 2008, David Woodhouse <dwmw2@infradead.org> wrote:

> those same companies _would_ consider putting their firmware into a
> non-GPL'd 'linux-firmware' repository instead.


And then, this very patchset started out of a distribution's demand to
preserve the ability to include the non-Free blobs as a condition to
ship a Free kernel (*).  (No, it wasn't Debian).  Now, why would I
want Free Software activists to use, endorse, promote and recommend
such a distro, or a usable strict subset thereof, if such a distro
will go to such lengths to endorse and distribute non-Free Software,
and to reject even *adding* a Free alternative?  Besides, tending to
that demand would be condoning and working towards a practice that is
fundamentally incompatible with what matters most to the Free Software
movement.  No, thanks.

(*) No, saying Linux is Free Software, or even Open Source, is
misleading at best.  All recent, and many not-so-recent, Linux
"source" tarballs published there contain software that fails both the
Free Software and the Open Source definition.  I guess repeating a lie
a sufficient number of times eventually does make it true.


Besides, unless Linux made a commitment to remove and keep out any
non-Free Software (blobs, NDA-obscured drivers), we'd still have to
keep an eye on Linux sources and check each release for non-Free
Software, and remove any remaining and newly-added such software.

But this is precisely what we do and the reason we do maintain
Linux-libre.  I'd even say that, the more non-Free Software moves
about in the the kernel, the more work this makes for us.  (Removal
also makes for work for us, but it happens only once, and it's worth
it because it makes things strictly better in the end.)

And no, 'rm -rf firmware' is not the answer; we have no reason to
remove the (few) Free firmwares in there and, unless all of the
non-Free Software is removed and remains out, distros that don't
switch to Linux-libre or do something equivalent on their own would
still end up shipping a non-Free, non-Open Source kernel, and most
would be misleading their users about that.  "Freedom² is a Feature",
and "main/l/linux-2.6" (rather than non-free) come to mind.


Now, I have no expectations whatsoever that head Linux developers
would make such a commitment to keep non-Free Software outside the
kernel source tree.  I wouldn't even bother trying.  That's just not
where their heart is.  I can live with that, and, nevertheless, I
thank them for their indirect contributions to the Free Software
movement.

After all, it's their hard work on Linux that makes it possible for us
to derive quite easily a Free Software kernel from it (Linux-libre),
to put that together with the GNU operating system (minus the kernel)
and then have a completely Free operating system.  That was the goal
of the first big undertaking of the Free Software movement, the GNU
Project, and this goal is finally achieved, available for all in
several GNU/Linux-libre distributions.


To sum it up: the patchset makes it easier for more firmware to go in;
it won't solve the problem (and render Linux-libre obsolete) unless a
commitment nobody would hope for; it makes for more work in
Linux-libre; and it might get more people to condone and even
recommend distros that tolerate, condone and distribute non-Free
Software.

I hope this makes it clear why I dislike the patchset, the approach
and the demand that led to it.  I hope you now see that the suspicions
that this would advance the software freedom cause were unfounded.
That's why I won't take part in developing it: in spite of its
technical soundness, it would make even more work for me in both
Linux-libre and Linux upstream, in exchange for at best a no-op as far
as my goals of freedom are concerned.

Having cleared this up, I'll leave the technical, legal and strategic
aspects for you to discuss among yourselves.

Have fun.  And please don't bother disputing the values that led to my
conclusion, they're firmly set and the flame war would probably just
annoy everyone who doesn't enjoy this kind of discussion.  Now, if you
find any flaws in the reasoning that took me from the premises to the
conclusions, I'd be happy to read about them and discuss them.

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: RFC: Moving firmware blobs out of the kernel.

[Jeff Garzik]

David Woodhouse wrote:
> I'm kind of hoping that this is going to be one of those topics that
> resolves itself -- but just in case it isn't, I'd like to discuss it at
> the kernel summit:
> 
> I'd like to remove all firmware blobs from the kernel source tree.
> 
> With the intention of making that less contentious, I've done the
> following: 
> 
> I've added support to the firmware loader for finding firmware in a
> special section of the static kernel, so that using request_firmware()
> no longer forces you to use an initrd or udev. You can build _arbitrary_
> firmware files into your kernel, if you want to. Whether you're allowed
> to distribute the resulting vmlinux or not is still a question you ought
> to ask your lawyer.
> 
> I'm working on converting drivers over to use request_firmware(), and
> shifting their firmware blobs into .ihex files in a firmware/ directory
> of the source tree. For each one, I'm giving the option of continuing to
> build it in statically, using the above mechanism.
> 
> I've implemented a 'make firmware_install' Kbuild target, which takes
> all those firmware blobs and installs them somewhere where udev can load
> them on request.
> 
> By the time the kernel summit comes around, we should have made decent
> progress on moving _all_ the firmware blobs to the firmware/ directory.
> And at that point I'd like to remove them completely, to a separate git
> tree and tarball. Those who really want to build them in to their static
> kernel would still be able to, but it wouldn't be the default behaviour.


I like your idea, all the way to the point where you actually remove the 
firmware file from the kernel tree :)

I think it's a good move to make the firmware files standalone and not 
#include'd as a C header file, but it's just plain easier to ship them 
with the kernel tree in a lot of cases.  $FOO-firmware packages in 
Fedora prove other methods of firmware distribution are quite usable, 
but that does not therefore imply that in-kernel built-in firmwares have 
no utility.

Personally, living in an imaginary all-open-source world, I wouldn't 
mind seeing firmware source for firmware built into drivers, a la 
drivers/scsi/aic7xxx/aicasm/, living in the kernel tree.

If the firmware has a compatible license and is required for critical 
operations like booting the machine, built-in firmware should remain an 
option.  For certain embedded cases, I could certainly see that 
in-kernel firmware being the best method for firmware distribution, for 
both $Platform's users and $Platform's developers.

	Jeff

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Peter Zijlstra]

On Thu, 2008-05-29 at 15:12 -0400, Jeff Garzik wrote:
> David Woodhouse wrote:
> > I'm kind of hoping that this is going to be one of those topics that
> > resolves itself -- but just in case it isn't, I'd like to discuss it at
> > the kernel summit:
> > 
> > I'd like to remove all firmware blobs from the kernel source tree.
> > 
> > With the intention of making that less contentious, I've done the
> > following: 
> > 
> > I've added support to the firmware loader for finding firmware in a
> > special section of the static kernel, so that using request_firmware()
> > no longer forces you to use an initrd or udev. You can build _arbitrary_
> > firmware files into your kernel, if you want to. Whether you're allowed
> > to distribute the resulting vmlinux or not is still a question you ought
> > to ask your lawyer.
> > 
> > I'm working on converting drivers over to use request_firmware(), and
> > shifting their firmware blobs into .ihex files in a firmware/ directory
> > of the source tree. For each one, I'm giving the option of continuing to
> > build it in statically, using the above mechanism.
> > 
> > I've implemented a 'make firmware_install' Kbuild target, which takes
> > all those firmware blobs and installs them somewhere where udev can load
> > them on request.
> > 
> > By the time the kernel summit comes around, we should have made decent
> > progress on moving _all_ the firmware blobs to the firmware/ directory.
> > And at that point I'd like to remove them completely, to a separate git
> > tree and tarball. Those who really want to build them in to their static
> > kernel would still be able to, but it wouldn't be the default behaviour.
> 
> 
> I like your idea, all the way to the point where you actually remove the 
> firmware file from the kernel tree :)
> 
> I think it's a good move to make the firmware files standalone and not 
> #include'd as a C header file, but it's just plain easier to ship them 
> with the kernel tree in a lot of cases.  $FOO-firmware packages in 
> Fedora prove other methods of firmware distribution are quite usable, 
> but that does not therefore imply that in-kernel built-in firmwares have 
> no utility.
> 
> Personally, living in an imaginary all-open-source world, I wouldn't 
> mind seeing firmware source for firmware built into drivers, a la 
> drivers/scsi/aic7xxx/aicasm/, living in the kernel tree.
> 
> If the firmware has a compatible license and is required for critical 
> operations like booting the machine, built-in firmware should remain an 
> option.  For certain embedded cases, I could certainly see that 
> in-kernel firmware being the best method for firmware distribution, for 
> both $Platform's users and $Platform's developers.

I certainly agree, pushing more and more into initrd just annoys the
hell out of me.

I'd argue to include everything needed to build (and esp cross build) an
initrd into the kernel - up until that point initrds are useless.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[H. Peter Anvin]

Peter Zijlstra wrote:
> 
> I certainly agree, pushing more and more into initrd just annoys the
> hell out of me.
> 
> I'd argue to include everything needed to build (and esp cross build) an
> initrd into the kernel - up until that point initrds are useless.
> 

I tried to push for that two years ago.  I'd be more than happy to pull 
that out of the freezer.

	-hpa

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Alan Cox]

On Thu, 29 May 2008 16:39:48 -0700
"H. Peter Anvin" <hpa@zytor.com> wrote:

> Peter Zijlstra wrote:
> > 
> > I certainly agree, pushing more and more into initrd just annoys the
> > hell out of me.
> > 
> > I'd argue to include everything needed to build (and esp cross build) an
> > initrd into the kernel - up until that point initrds are useless.
> > 
> 
> I tried to push for that two years ago.  I'd be more than happy to pull 
> that out of the freezer.

Its kind of irrelevant if you need an initrd or not. The only question of
relevance is "does it get built when I type make all". Almost all Linux
users are using initrd without problem - because their distro ensures
"make install" and the packaged kernels do the right thing.

Alan

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Peter Zijlstra]

On Fri, 2008-05-30 at 10:31 +0100, Alan Cox wrote:
> On Thu, 29 May 2008 16:39:48 -0700
> "H. Peter Anvin" <hpa@zytor.com> wrote:
> 
> > Peter Zijlstra wrote:
> > > 
> > > I certainly agree, pushing more and more into initrd just annoys the
> > > hell out of me.
> > > 
> > > I'd argue to include everything needed to build (and esp cross build) an
> > > initrd into the kernel - up until that point initrds are useless.
> > > 
> > 
> > I tried to push for that two years ago.  I'd be more than happy to pull 
> > that out of the freezer.
> 
> Its kind of irrelevant if you need an initrd or not. The only question of
> relevance is "does it get built when I type make all". Almost all Linux
> users are using initrd without problem - because their distro ensures
> "make install" and the packaged kernels do the right thing.

Its my own convenience I'm serving here. I hardly ever do a local
install. And for the old machines a local build just isn't even an
option.

It's what benh said; I just want a single netbootable image. And cross
buildling initrds is just impossible - which makes the whole solution
useless.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Jeff Garzik]

Peter Zijlstra wrote:
> It's what benh said; I just want a single netbootable image. And cross
> buildling initrds is just impossible - which makes the whole solution
> useless.


hpa's klibc solution was cross-buildable IIRC.

	Jeff

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Alexandre Oliva]

On May 30, 2008, Peter Zijlstra <a.p.zijlstra@chello.nl> wrote:

> It's what benh said; I just want a single netbootable image. And cross
> buildling initrds is just impossible - which makes the whole solution
> useless.

Unless you actually use the feature added in the first patch of the
series, that enables you to build firmware images into your single
netbootable image, that is ;-)

-- 
Alexandre Oliva         http://www.lsd.ic.unicamp.br/~oliva/
Free Software Evangelist  oliva@{lsd.ic.unicamp.br, gnu.org}
FSFLA Board Member       ¡Sé Libre! => http://www.fsfla.org/
Red Hat Compiler Engineer   aoliva@{redhat.com, gcc.gnu.org}

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[H. Peter Anvin]

Alan Cox wrote:
>>>
>> I tried to push for that two years ago.  I'd be more than happy to pull 
>> that out of the freezer.
> 
> Its kind of irrelevant if you need an initrd or not. The only question of
> relevance is "does it get built when I type make all". Almost all Linux
> users are using initrd without problem - because their distro ensures
> "make install" and the packaged kernels do the right thing.
> 

Obviously.  I'm equally obviously referring to klibc, which was designed 
so that the resulting vmlinux/bzImage/... file contains the necessary 
initramfs, regardless of issues like cross-compilation, draconian size 
restrictions(*), and so forth.

	-hpa

(*) Not saying that a klibc-based initramfs is necessarily smaller than 
the in-kernel code it replaces, but the total size is << than the size 
of the kernel proper, which isn't true when using a full-featured libc.

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Alan Cox]

> (*) Not saying that a klibc-based initramfs is necessarily smaller than 
> the in-kernel code it replaces, but the total size is << than the size 
> of the kernel proper, which isn't true when using a full-featured libc.

True but with a vendor hat on thats not usually a problem on modern
systems and using glibc means less code to maintain, less special cases,
and more flexibility.

For embedded klibc may well be interesting.

Alan

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[H. Peter Anvin]

Alan Cox wrote:
>> (*) Not saying that a klibc-based initramfs is necessarily smaller than 
>> the in-kernel code it replaces, but the total size is << than the size 
>> of the kernel proper, which isn't true when using a full-featured libc.
> 
> True but with a vendor hat on thats not usually a problem on modern
> systems and using glibc means less code to maintain, less special cases,
> and more flexibility.
> 
> For embedded klibc may well be interesting.

For vendor hat meaning full-blown systems, yes that is true, but for 
embedded, or even on some "real" platforms, that definitely matters.

Fundamentally, however, we're never going to have a full-blown libc 
environment built out of the kernel tree, as its size would be 
comparable or larger than the kernel itself.

	-hpa

Re: [Ksummit-2008-discuss] RFC: Moving firmware blobs out of the kernel.

[Benjamin Herrenschmidt]

On Thu, 2008-05-29 at 23:17 +0200, Peter Zijlstra wrote:
> > If the firmware has a compatible license and is required for critical 
> > operations like booting the machine, built-in firmware should remain an 
> > option.  For certain embedded cases, I could certainly see that 
> > in-kernel firmware being the best method for firmware distribution, for 
> > both $Platform's users and $Platform's developers.
> 
> I certainly agree, pushing more and more into initrd just annoys the
> hell out of me.
> 
> I'd argue to include everything needed to build (and esp cross build) an
> initrd into the kernel - up until that point initrds are useless.

Us kernel hackers tend to like self-contained netbootable kernels
that have everything needed to boot a machine all the way to a shell
prompt :-)

David's proposal however do provides for storing the firmwares in
the kernel image, so I have no objection there.

Ben.

Re: RFC: Moving firmware blobs out of the kernel.

[David Woodhouse]

On Thu, 2008-05-29 at 15:12 -0400, Jeff Garzik wrote:
> I like your idea, all the way to the point where you actually remove the 
> firmware file from the kernel tree :)

I've been careful to ensure that what I'm doing is a benefit even
without that final step. But I still think that final step is useful
too.

> If the firmware has a compatible license and is required for critical 
> operations like booting the machine, built-in firmware should remain an 
> option. 

Well, 'compatible licence' is a loaded question. It takes a fairly
wilful misinterpretation of 'mere aggregation', IMHO, to see what we're
currently doing as legal -- but I was trying to avoid that discussion
since it tends to devolve into name-calling and nobody's _actually_
right until/unless it's decided by a court.

But sticking to the technical side -- with what I have now, it's
_easier_ to build external firmware into the kernel than it was before.
My original testing was done with a kernel with the libertas
'usb8388.bin' firmware built in to it, for example. That was never
possible before.

Having firmware files in a separate tarball doesn't prevent you from
building them into your vmlinux if you want to. And there are some
companies who wouldn't allow their firmware to be distributed in the
kernel source tree because it's under GPL, but _would_ consider putting
it in a separate 'kernel-firmware' repository instead. So this should
_also_ mean we can ship more firmware, more easily.

Hell, with git submodules you almost don't need to notice the
difference, do you? Just check out kernel-firmware as a subdirectory of
the source tree (or point CONFIG_BUILTIN_FIRMWARE_DIR at wherever you
_did_ check it out), and you're done.

>  For certain embedded cases, I could certainly see that 
> in-kernel firmware being the best method for firmware distribution, for 
> both $Platform's users and $Platform's developers.

It is not my intention to remove that possibility. I believe I have made
it _more_ feasible; not less.

-- 
dwmw2

Re: RFC: Moving firmware blobs out of the kernel.

[Benjamin Herrenschmidt]

 .../...

> By the time the kernel summit comes around, we should have made decent
> progress on moving _all_ the firmware blobs to the firmware/ directory.
> And at that point I'd like to remove them completely, to a separate git
> tree and tarball. Those who really want to build them in to their static
> kernel would still be able to, but it wouldn't be the default behaviour.

Pretty much on line with others: I like most of what you propose
-except- having the firmwares distributed in a separate git tree. I much
prefer having them still in the main kernel, though distro can choose
not to do so more easily if they are in a separate directory.

Ben.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

Ted, we are adults and professional kernel hackers.

If you put us in a room together, we know what the heck to talk
about and what is relevant.  So saying we need to plan the topics and
invite the right people, that's pure hogwash, just invite the same
kind of people we've always been inviting and it will be fine.

It's absolutely absurd to put the best kernel hackers in the world all
together in the same place and talk about process for 3 days.

Yes, process is important, but I'd say it deserved 1/3 of the summit
time, but not one minute more.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Wilcox]

On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> Ted, we are adults and professional kernel hackers.
> 
> If you put us in a room together, we know what the heck to talk
> about and what is relevant.  So saying we need to plan the topics and
> invite the right people, that's pure hogwash, just invite the same
> kind of people we've always been inviting and it will be fine.

Ooh, do I hear a supporter for my idea from a few years ago of having
half the time in sessions and half the time in the hallway track?

(I always felt the presentations were more or less useless.  Maybe that
was different last year.)

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Greg KH]

On Thu, May 29, 2008 at 02:59:18PM -0600, Matthew Wilcox wrote:
> On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> > Ted, we are adults and professional kernel hackers.
> > 
> > If you put us in a room together, we know what the heck to talk
> > about and what is relevant.  So saying we need to plan the topics and
> > invite the right people, that's pure hogwash, just invite the same
> > kind of people we've always been inviting and it will be fine.
> 
> Ooh, do I hear a supporter for my idea from a few years ago of having
> half the time in sessions and half the time in the hallway track?

I second this idea, a mini "hack-fest" to hash out some of the ideas and
actually try to start to implement them would be very good.

thanks,

greg k-h

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Benjamin Herrenschmidt]

On Thu, 2008-05-29 at 14:59 -0600, Matthew Wilcox wrote:
> On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> > Ted, we are adults and professional kernel hackers.
> > 
> > If you put us in a room together, we know what the heck to talk
> > about and what is relevant.  So saying we need to plan the topics and
> > invite the right people, that's pure hogwash, just invite the same
> > kind of people we've always been inviting and it will be fine.
> 
> Ooh, do I hear a supporter for my idea from a few years ago of having
> half the time in sessions and half the time in the hallway track?
> 
> (I always felt the presentations were more or less useless.  Maybe that
> was different last year.)

Turn presentations into BOFs ? :-)

Ben

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Wilcox]

On Fri, May 30, 2008 at 11:25:41AM +1000, Benjamin Herrenschmidt wrote:
> Turn presentations into BOFs ? :-)

That would certainly reduce the slideware.

jejb complained that this was going to turn into an unconference.
I didn't really know what that meant, so I looked around on
Wikipedia.  It doesn't seem like such a bad idea to me, particularly
some of the more unconventional organising techniques, eg:
http://en.wikipedia.org/wiki/Open_Space_Technology

-- 
Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Theodore Tso]

On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> Ted, we are adults and professional kernel hackers.
> 
> If you put us in a room together, we know what the heck to talk
> about and what is relevant.  So saying we need to plan the topics and
> invite the right people, that's pure hogwash, just invite the same
> kind of people we've always been inviting and it will be fine.

I do assume people we invite are adults.  That being said, *some*
amout of structure is still a good thing.  And as others have pointed
out, sometimes the sharper disagreements are with people who aren't
invited.  If we know we want to try talk about LSM/Apparmor YET AGAIN
(no I'm not advocating this; besides, I thought Linus has already made
it clear that people who depend on path-oriented security are on crack :-), 
we might end up inviting some folks that we might not otherwise
include, just to take one example.

So if people want to talk about things that aren't on the agenda, it's
certainly not carried down by Moses from the mountain and cast in
stone; besides, Charles Heston has already passed away.  :-)

> Yes, process is important, but I'd say it deserved 1/3 of the summit
> time, but not one minute more.

We were about 50-50 last time, roughly.  So let's see if there is
consensus to dial it back this year, based on topics that people
suggest.

						- Ted

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

From: Theodore Tso <tytso@mit.edu>
Date: Thu, 29 May 2008 17:14:04 -0400

> On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> > Yes, process is important, but I'd say it deserved 1/3 of the summit
> > time, but not one minute more.
> 
> We were about 50-50 last time, roughly.  So let's see if there is
> consensus to dial it back this year, based on topics that people
> suggest.

Today at LinuxTAG, Thomas Gleixner and I tossed around the idea
that (assuming a 3 day event) we do tech stuff on day 1, process
stuff on day 2, and back to tech stuff on day 3.

The idea is that you hash out the initial impressions of your
ideas on day 1, your mind works on the problem in the background
on day 2, and they everyone has the answers on day 3 :-)

Another idea we discussed briefly was some kind of organized run
over the bugzilla entries.  And just to make it interesting we could
award some silly prize to whoever fixed the most interesting or
amusing bug during the bug-run, as voted by the attendees.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Thomas Gleixner]

On Thu, 29 May 2008, David Miller wrote:
> From: Theodore Tso <tytso@mit.edu>
> Date: Thu, 29 May 2008 17:14:04 -0400
> 
> > On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> > > Yes, process is important, but I'd say it deserved 1/3 of the summit
> > > time, but not one minute more.
> > 
> > We were about 50-50 last time, roughly.  So let's see if there is
> > consensus to dial it back this year, based on topics that people
> > suggest.
> 
> Today at LinuxTAG, Thomas Gleixner and I tossed around the idea
> that (assuming a 3 day event) we do tech stuff on day 1, process
> stuff on day 2, and back to tech stuff on day 3.
> 
> The idea is that you hash out the initial impressions of your
> ideas on day 1, your mind works on the problem in the background
> on day 2, and they everyone has the answers on day 3 :-)
> 
> Another idea we discussed briefly was some kind of organized run
> over the bugzilla entries.  And just to make it interesting we could
> award some silly prize to whoever fixed the most interesting or
> amusing bug during the bug-run, as voted by the attendees.

I thought more about it and came up with an interesting challange for
the top hackers crowd:

Fix NOHZ/CPUIDLE along with suspend/resume on all participants
laptops, which are probably 50+ different models. That'd be an odd
enough mix of wreckaged hardware / BIOS / ACPI.

Should be fun and solve a bunch of hard to grok bugs in the bugzillas
along the way.

Thanks,
	tglx

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Sun, 2008-06-01 at 16:11 +0200, Thomas Gleixner wrote:
> On Thu, 29 May 2008, David Miller wrote:
> > From: Theodore Tso <tytso@mit.edu>
> > Date: Thu, 29 May 2008 17:14:04 -0400
> > 
> > > On Thu, May 29, 2008 at 01:54:31PM -0700, David Miller wrote:
> > > > Yes, process is important, but I'd say it deserved 1/3 of the summit
> > > > time, but not one minute more.
> > > 
> > > We were about 50-50 last time, roughly.  So let's see if there is
> > > consensus to dial it back this year, based on topics that people
> > > suggest.
> > 
> > Today at LinuxTAG, Thomas Gleixner and I tossed around the idea
> > that (assuming a 3 day event) we do tech stuff on day 1, process
> > stuff on day 2, and back to tech stuff on day 3.
> > 
> > The idea is that you hash out the initial impressions of your
> > ideas on day 1, your mind works on the problem in the background
> > on day 2, and they everyone has the answers on day 3 :-)
> > 
> > Another idea we discussed briefly was some kind of organized run
> > over the bugzilla entries.  And just to make it interesting we could
> > award some silly prize to whoever fixed the most interesting or
> > amusing bug during the bug-run, as voted by the attendees.
> 
> I thought more about it and came up with an interesting challange for
> the top hackers crowd:
> 
> Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> laptops, which are probably 50+ different models. That'd be an odd
> enough mix of wreckaged hardware / BIOS / ACPI.
> 
> Should be fun and solve a bunch of hard to grok bugs in the bugzillas
> along the way.

Well ... in theory.

In practice, for instance, my laptop had suspend/resume working shortly
after I got it, and the widescreen video too.  Most of the problems were
video related, so I did interact with the upstream intel video driver
people, but by and large it was a set of black magic rules to restore
the video to its prior state (in my case, even the vbe tools didn't work
and I had to manually save and restore the pci config space).

The point, though, is I'd be incredibly surprised if a kernel hacker had
an unfixed suspend resume problem (except possibly one that just
appeared in the latest upgrade).  It's a fairly important feature and
hackers tend to get annoyed by problems like this and hack on them until
they're fixed.

However, persuading us all to go to the fix my suspend/resume session at
the plumbers conf (which follows directly) would probably achieve a
fairly sizeable crossection of unfixed laptops and possibly quite a lot
of bug fixing ...

James

s2ram video problems Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Pavel Machek]

Hi!

> > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > laptops, which are probably 50+ different models. That'd be an odd
> > enough mix of wreckaged hardware / BIOS / ACPI.
> > 
> > Should be fun and solve a bunch of hard to grok bugs in the bugzillas
> > along the way.
> 
> Well ... in theory.
> 
> In practice, for instance, my laptop had suspend/resume working shortly
> after I got it, and the widescreen video too.  Most of the problems were
> video related, so I did interact with the upstream intel video driver
> people, but by and large it was a set of black magic rules to restore
> the video to its prior state (in my case, even the vbe tools didn't work
> and I had to manually save and restore the pci config space).

that's s2ram -v, right? Can you submit a whitelist entry so it starts
working for other people, too?

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: s2ram video problems Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Rafael J. Wysocki]

On Sunday, 1 of June 2008, Pavel Machek wrote:
> Hi!
> 
> > > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > > laptops, which are probably 50+ different models. That'd be an odd
> > > enough mix of wreckaged hardware / BIOS / ACPI.
> > > 
> > > Should be fun and solve a bunch of hard to grok bugs in the bugzillas
> > > along the way.
> > 
> > Well ... in theory.
> > 
> > In practice, for instance, my laptop had suspend/resume working shortly
> > after I got it, and the widescreen video too.  Most of the problems were
> > video related, so I did interact with the upstream intel video driver
> > people, but by and large it was a set of black magic rules to restore
> > the video to its prior state (in my case, even the vbe tools didn't work
> > and I had to manually save and restore the pci config space).
> 
> that's s2ram -v, right? Can you submit a whitelist entry so it starts
> working for other people, too?

No, that won't work.  James uses Fedora.

Thanks,
Rafael

Re: s2ram video problems Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Sun, 2008-06-01 at 18:21 +0200, Pavel Machek wrote:
> Hi!
> 
> > > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > > laptops, which are probably 50+ different models. That'd be an odd
> > > enough mix of wreckaged hardware / BIOS / ACPI.
> > > 
> > > Should be fun and solve a bunch of hard to grok bugs in the bugzillas
> > > along the way.
> > 
> > Well ... in theory.
> > 
> > In practice, for instance, my laptop had suspend/resume working shortly
> > after I got it, and the widescreen video too.  Most of the problems were
> > video related, so I did interact with the upstream intel video driver
> > people, but by and large it was a set of black magic rules to restore
> > the video to its prior state (in my case, even the vbe tools didn't work
> > and I had to manually save and restore the pci config space).
> 
> that's s2ram -v, right? Can you submit a whitelist entry so it starts
> working for other people, too?

Actually, it's pm-utils, because I'm using fedora.

This, by the way was years ago, beginning with FC6.  In FC7 we got the
driver to the state where vbestate save/restore worked for it and added
it to the hal database.  Today, at FC9, I've just been busy filing a bug
with fedora because the i915 drm now seems to do everything and actively
screws up if vbestate save/restore is used (so all the work I did with
FC7/8 now needs to be undone).

James

Re: s2ram video problems Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Rafael J. Wysocki]

On Sunday, 1 of June 2008, James Bottomley wrote:
> On Sun, 2008-06-01 at 18:21 +0200, Pavel Machek wrote:
> > Hi!
> > 
> > > > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > > > laptops, which are probably 50+ different models. That'd be an odd
> > > > enough mix of wreckaged hardware / BIOS / ACPI.
> > > > 
> > > > Should be fun and solve a bunch of hard to grok bugs in the bugzillas
> > > > along the way.
> > > 
> > > Well ... in theory.
> > > 
> > > In practice, for instance, my laptop had suspend/resume working shortly
> > > after I got it, and the widescreen video too.  Most of the problems were
> > > video related, so I did interact with the upstream intel video driver
> > > people, but by and large it was a set of black magic rules to restore
> > > the video to its prior state (in my case, even the vbe tools didn't work
> > > and I had to manually save and restore the pci config space).
> > 
> > that's s2ram -v, right? Can you submit a whitelist entry so it starts
> > working for other people, too?
> 
> Actually, it's pm-utils, because I'm using fedora.
> 
> This, by the way was years ago, beginning with FC6.  In FC7 we got the
> driver to the state where vbestate save/restore worked for it and added
> it to the hal database.  Today, at FC9, I've just been busy filing a bug
> with fedora because the i915 drm now seems to do everything and actively
> screws up if vbestate save/restore is used (so all the work I did with
> FC7/8 now needs to be undone).

Well, we have the same problem in s2ram.

Thanks,
Rafael

Re: s2ram video problems Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Garrett]

On Sun, Jun 01, 2008 at 01:04:25PM -0500, James Bottomley wrote:

> This, by the way was years ago, beginning with FC6.  In FC7 we got the
> driver to the state where vbestate save/restore worked for it and added
> it to the hal database.  Today, at FC9, I've just been busy filing a bug
> with fedora because the i915 drm now seems to do everything and actively
> screws up if vbestate save/restore is used (so all the work I did with
> FC7/8 now needs to be undone).

Fixed in pm-utils upstream, which knows not to do anything with recent 
Intel drm. The trick now is to get nvidia and amd to the same level of 
functionality, though in principle we know enough to manage at least the 
nvidia case.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Garrett]

On Sun, Jun 01, 2008 at 04:11:01PM +0200, Thomas Gleixner wrote:

> Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> laptops, which are probably 50+ different models. That'd be an odd
> enough mix of wreckaged hardware / BIOS / ACPI.

If we have in-kernel video reinit for the major vendors by then, sure. 
If not, I suspect the majority of issues we'd hit would just be "My 
screen doesn't come back".

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Thomas Gleixner]

On Sun, 1 Jun 2008, Matthew Garrett wrote:

> On Sun, Jun 01, 2008 at 04:11:01PM +0200, Thomas Gleixner wrote:
> 
> > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > laptops, which are probably 50+ different models. That'd be an odd
> > enough mix of wreckaged hardware / BIOS / ACPI.
> 
> If we have in-kernel video reinit for the major vendors by then, sure. 
> If not, I suspect the majority of issues we'd hit would just be "My 
> screen doesn't come back".

And why wouldn't it be a good idea to spend time on adding the video
reinit right there at the hacking session ?

Thanks,

	tglx

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Matthew Garrett]

On Sun, Jun 01, 2008 at 10:22:48PM +0200, Thomas Gleixner wrote:

> And why wouldn't it be a good idea to spend time on adding the video
> reinit right there at the hacking session ?

Because there's only one guy outside nvidia who understands the nvidia 
BIOS tables right now, and I suspect he's not on the invite list :) I'm 
actually planning on giving the nvidia stuff a go this week.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Benjamin Herrenschmidt]

On Sun, 2008-06-01 at 22:22 +0200, Thomas Gleixner wrote:
> On Sun, 1 Jun 2008, Matthew Garrett wrote:
> 
> > On Sun, Jun 01, 2008 at 04:11:01PM +0200, Thomas Gleixner wrote:
> > 
> > > Fix NOHZ/CPUIDLE along with suspend/resume on all participants
> > > laptops, which are probably 50+ different models. That'd be an odd
> > > enough mix of wreckaged hardware / BIOS / ACPI.
> > 
> > If we have in-kernel video reinit for the major vendors by then, sure. 
> > If not, I suspect the majority of issues we'd hit would just be "My 
> > screen doesn't come back".
> 
> And why wouldn't it be a good idea to spend time on adding the video
> reinit right there at the hacking session ?

You mean adding the right sequence of whacking 200+ non documented chip
registers in the right order with the right values that have to be
subtly different on each revision and each OEM implementation ? :-)

At least for ATI, we do have the necessary infos to do it actually. We
can parse tables in the BIOS, we know how to, we have the format and the
necessary interpreter, it's mostly a matter of doing it (and
re-implementing the interpreter in code that is acceptable to the
kernel, last I looked, the one in X isn't).

For nVidia, it's harder though the "nouveau" folks do have figured how
to run some of the BIOS foo too (same thing, BIOS contains magic
"scripts" that can be interpreted to bootstrap the card).

For others (VIA, XGI, ...) I don't know.

Cheers,
Ben.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Benjamin Herrenschmidt]

On Thu, 2008-05-29 at 08:45 -0400, Theodore Tso wrote:
> On Thu, May 29, 2008 at 04:17:19PM +1000, Benjamin Herrenschmidt wrote:
> > Having been stopped a couple of times last year when trying to bring
> > some technical subjects for the reason that they were "off topic, KS is
> > for process" I tend to agree :-)
> 
> We did have some technical topics last year.  So I don't think the KS
> will ever be purely for process.

Amen :-)

> I agree that sometimes face-to-face discussions are crucial to
> resolving technical issues.  The problem is that we try to nail down
> the agenda a 3-4 weeks ahead of time, and realistically if a
> particular topic requires certain stakeholders to be invited, we need
> to decide that we're going to do that topic at least 8-9 weeks in
> advance, maybe more, so those people can make travel plans and/or get
> travel approvals.  And there is always the chance the topic will
> resolve itself via e-mail.  So the trick is being able to identifying
> the topics where a face-to-face discussion really will be useful.

It's a bit hard, as sometimes, the topics will just show up before hand,
or don't necessarily need special invite list when it's purely something
that needs to be discussed with a good enough panel of arch maintainers.

> First of all, if there is interest in holding some topic-area specific
> mini-summits on Sunday before the Kernel Summit, we may be able to
> scare up some space.  So if there is interest, please let me know.

Ok, if something comes to mind I will :-) The only thing at this stage
would possibly be page table walking (ie, some old work I did on
that to generalize the batch interface to all more-than-one-page
operations such as fork, and to get rid of the per-cpu structures, and
that I never fully finished, along with the mmu notifier stuff and
the various needs around that area such as sleeping during some parts
of page table updates etc...).

We might finally be able to put the whole thing on a table and decide
on a nice / clean solution that I'd be happy to help implement then :-)

Among others, I have had a hard time getting enough input from the
various archs as to get to a design that would fit everybody fine.

We don't -have- to spend time on that, it's just an example, but typically
I think a good one of something that would benefit from a face to face
brainwashing with the arch maintainers & MM people.

> Secondly, one of the things which has been suggested in the past is
> that we move the BOF's into an afternoon session slot, and maybe push
> the last session to run until 7pm, perhaps.  That might make it easier
> for people to attend BOF's on the spur of the moment --- which might
> be useful in some cases where there is only 10 or so people you need
> to hash our some issue.
> 
> And, of course, we try to schedule plenty of break time so that some
> of these discussions can happy in the hallway.
> 
> Do any of these possibilities sound particularly attractive or likely
> to address your concerns?  If so, which ones do you think we should
> try this year, as an experiment?

To be honest, I don't know for sure. A bit like David, I'm not even
certain planning for those things is -that- useful and I must admit
I'm not too fan of long days mostly because I end up always getting
there totally jet lagged :-)

But we can definitely try.

Cheers,
Ben.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[David Miller]

From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: Fri, 30 May 2008 11:20:06 +1000

> To be honest, I don't know for sure. A bit like David, I'm not even
> certain planning for those things is -that- useful and I must admit
> I'm not too fan of long days mostly because I end up always getting
> there totally jet lagged :-)

I personally don't even like having to choose presentation topics when
I go to conferences.  What is interesting for me to talk about usually
is changing righ up to the day or two before I travel to the event.

And I think the same thing applies here.

You can pick some topic now, and by the time we actually get to the
summit those folks might have hashed everything out already.

We should just all show up and discuss in appropriate groups whatever
is pressing and interesting at the moment.

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Jonathan Corbet]

Ben H writes:

> On Wed, 2008-05-28 at 22:58 -0700, David Miller wrote:
> > 
> > Not to distract from your points, but I think it's been a huge
> > mistake to not allow folks to work hard on technical issues
> > during the summit.
> 
> Having been stopped a couple of times last year when trying to bring
> some technical subjects for the reason that they were "off topic, KS is
> for process" I tend to agree :-)

Here's an idea: the discussion list for the 2008 summit has opened up.
This seems like an ideal time to make suggestions for technical topics
which you think would be appropriate for this gathering.  Failing that,
you leave it up to the program committee to try to guess what's on
everybody's minds, and that seems certain to disappoint a lot of people.

jon

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Neil Brown]

On Thursday May 29, corbet@lwn.net wrote:
> Ben H writes:
> 
> > On Wed, 2008-05-28 at 22:58 -0700, David Miller wrote:
> > > 
> > > Not to distract from your points, but I think it's been a huge
> > > mistake to not allow folks to work hard on technical issues
> > > during the summit.
> > 
> > Having been stopped a couple of times last year when trying to bring
> > some technical subjects for the reason that they were "off topic, KS is
> > for process" I tend to agree :-)
> 
> Here's an idea: the discussion list for the 2008 summit has opened up.
> This seems like an ideal time to make suggestions for technical topics
> which you think would be appropriate for this gathering.  Failing that,
> you leave it up to the program committee to try to guess what's on
> everybody's minds, and that seems certain to disappoint a lot of people.
> 

http://lwn.net/Articles/283161/

Barriers

What minimal semantics to Filesystem developers actually want?
What can hardware really provide?
What how can layered mutli-disk devices (md, dm, loop) mediate these?

What interface semantics can we come up with that are sufficiently
general and powerful that everyone will actually implement them
correctly?

However I won't be coming this year, so I won't be able to participate
directly in such a discussion.

NeilBrown

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[James Bottomley]

On Wed, 2008-05-28 at 22:58 -0700, David Miller wrote:
> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Date: Wed, 28 May 2008 12:20:12 -0500
> 
> > In the spirit of having a more process than technical based kernel
> > summit, I'd like to put the topic of the kernel Janitors project up for
> > discussion.
> 
> Not to distract from your points, but I think it's been a huge
> mistake to not allow folks to work hard on technical issues
> during the summit.

Actually, it was wrong of me to give that impression.  The summit is
also all about technical discussions.  However, one of the observations
made over the last few years was that it's hard to have a technical
discussion that actively involves a majority of people in the room, so
most of the technical stuff goes on either in the mini-summits (for
discussions within maintainer areas) or in the hallway (for discussions
across maintainer areas) or in BoF sessions.

However, if there's a technical topic that would occupy the attention of
more people than can conveniently be accommodated in the hallway track,
by all means raise it ... it can be added to the agenda.

> I feel like I'm at the edge of my seat dieing to talk about
> some tech topics, but it's all of this process stuff, it makes
> me want to disappear into the local cafe during the main
> summit with some folks and talk tech.

That's sort of an extension of the hallway track (especially in
Cambridge where there was a convenient pub across the road).

> I think it would even fix the attitude and process issues,
> to be able to talk face-to-face about technical issues instead
> of always hashing stuff out in email which is the source of
> all of the bad behavior.

I agree that having people face to face helps draw heat out of email
discussions, certainly.  My observation though (and it may not be
universally held) has been that the majority of the frictions on the
mailing list isn't necessarily coming from the core people who get
summit invites.

James

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Helge Hafting]

James Bottomley wrote:
[...]
> track down, the last thing I want to see within a million miles of my
> inbox is a white space fixing patch.  The more of these patches we get,
> the worse the problem becomes and the shorter and more inflammatory the
> responses get. We can't go on like this.
> 
This particular problem seems fixable. Put something like this in the 
janitor faq:

"Don't create patches that are whitespace-fixes, or similiar attempts to
make the kernel adhere to the CodingStyle without actually changing the 
working code. The reasons are:
* Any patch disturbs others working on the same file. So change must be
   justified by usefulness above mere pretty formatting.
* The maintainer can run the code through pretty-printing software
   _when it won't disrupt other activity_. Such software will do
   a better cleanup job than you can, and with no human effort at.
Please find something more useful to do than whitespace/style
improvements on existing code."

And if they do this anyway, just reject the patch with a pointer to the 
FAQ entry...

Helge Hafting

Re: [Ksummit-2008-discuss] Fixing the Kernel Janitors project

[Adrian Bunk]

On Wed, May 28, 2008 at 12:20:12PM -0500, James Bottomley wrote:
>...
> The most obvious solution might be to shut the Janitors project down, or
> at least more tightly manage its TODO list (although a lot of what gets
> seen as janitorial patches, like whitespace fixes, isn't on the TODO
> list in the first place).  However, since the purpose is to get new
> people involved with kernel development, perhaps we should repurpose the
> project so it actually does this.  My suggestion is that we replace it
> with the kernel bugs project.  Kudos for finding bugs, more for finding
> better ways of finding bugs, and the most for finding and actually
> fixing a bug.  
> 
> Perhaps we should simply start the discussion with the premise that we
> want to encourage new people to do useful work and draw them into the
> development community and see where it leads.

My attempt in this direction is described at 
http://lkml.org/lkml/2008/5/7/258

Although I do not yet know how it will develop it already resulted in 
some bug reports that might otherwise not have been sent.

> James

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed