Re: A modest proposal -- We need a patch penguin

Re: A modest proposal -- We need a patch penguin

[John Weber]

I would be happy to serve as patch penguin, as I plan on collecting all
patches anyway in my new duties as maintainer of www.linuxhq.com.

I am currently writing code to scan the usual places for linux patches
and automatically add them to our databases.  This would be really
simplified by having patches sent to us.  And, since we already have a
functioning site, we have the hardware/network capacity to serve as
a limitless queue of waiting patches for Linus.  I would love nothing
more than to update the site with information as to the status of these
patches.

( john.weber@linux.org )

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 08:53 pm, John Weber wrote:
> I would be happy to serve as patch penguin, as I plan on collecting all
> patches anyway in my new duties as maintainer of www.linuxhq.com.
>
> I am currently writing code to scan the usual places for linux patches
> and automatically add them to our databases.  This would be really
> simplified by having patches sent to us.  And, since we already have a
> functioning site, we have the hardware/network capacity to serve as
> a limitless queue of waiting patches for Linus.  I would love nothing
> more than to update the site with information as to the status of these
> patches.
>
> ( john.weber@linux.org )

Philosophical question: Would you have a major philosophical objection to 
acting as Dave Jones's secretary and webmaster?  (He is the de facto current 
patch penguin.  I'm just asking for the position to be recognized.  We need 
that before we can really move forward with anything else.  If you were to 
queue patches for Linus and then be ignored by Linus, nothing would have been 
accomplished, and if somebody ELSE then takes your work and integrates it, it 
would be yet more pressure to fork the tree, pressure which I'm trying to 
REDUCE here...)

Remember minix?  Way way way back?  Andrew Tanenbaum had a little kernel, ran 
on intel hardware, came with complete source code.  And he did not accept 
patches, due to his minix book contract and the resulting licensing issues.  
Collaborative development on Linux STARTED in the minix newsgroup, largely by 
recruiting people who were frustrated at trying to get their patches into 
minix.

Remember GNU?  Stalled in the late 80's?  For legal reasons, Richard Stallman 
wanted people to physically sign over their copyrights (on paper he could put 
in his file cabinet) to any code they submitted to the GNU project.  This 
caused way too much friction (and Richard wasn't exactly a coalition building 
statesman either), and eventually people got fed up with the project and took 
their code elsewhere.

These are the kind of pressures that, if they build up high enough, cause 
projects to fork.  It's all different trees with different patches in them, 
and if the patch pressure builds up too high forking is inevitable.  
(Re-integration of forks is also quite possible, they can be short lived.  
But that's the same integration issue, just deferred a bit.)

I'm not saying Linux is in immediate danger of forking, I'm just saying that 
code integration can be a serious limiting factor, and is a potentially 
seperable problem from being a code architect.  I think an explicit full-time 
integration maintainer could reduce/buffer the patch pressure, and that this 
could be good for the project.

Rob

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Mon, 28 Jan 2002, John Weber wrote:

> I would be happy to serve as patch penguin, as I plan on collecting all
> patches anyway in my new duties as maintainer of www.linuxhq.com.

> we have the hardware/network capacity to serve as a limitless queue of
> waiting patches for Linus.

Please don't just accumulate stuff.

It would be useful to know which of the patches still
applies against the most recent 2.2, 2.4 or 2.5 kernel,
so each patch gets some status fields:

1) applies against 2.2
2) applies against 2.4
3) applies against 2.5

4) was applied to 2.2
5) was applied to 2.4
6) was applied to 2.5

7) bitrotted patch, no longer applies and wasn't
   applied ... moved to 'old' queue

kind regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Denis Vlasenko]

On 29 January 2002 09:04, Rik van Riel wrote:
> On Mon, 28 Jan 2002, John Weber wrote:
> > I would be happy to serve as patch penguin, as I plan on collecting all
> > patches anyway in my new duties as maintainer of www.linuxhq.com.
> >
> > we have the hardware/network capacity to serve as a limitless queue of
> > waiting patches for Linus.
>
> Please don't just accumulate stuff.

Right. Accepting any patch is wrong policy. You'll be swamped.
Patch must be marked "applies to 2.N.M", patch tracking system must check 
that automagically.

Also each patch(set) can be commented by general public and by maintainers.
If there is _no_ comment from any of _maintainers_ (i.e. it is not reviewed 
or found too ugly to worth commenting) it is automatically dropped from the 
system after some time.  This will force patch authors to care about code 
quality.

If patch is too old (several releases behind) system can mail author(s):
"Warning. Your patchset #3476346 needs rediffing. It will be dropped 
otherwise"

These "small" details determine whether system is useful or just turns into 
huge pile of patches of questionable value.
--
vda

Re: A modest proposal -- We need a patch tracking system.

[Kervin Pierre]

Public patch tracking system/queue, maybe something derived from bugzilla.

(i) patches are sent to the maintainer and entered into the system.

(ii) reviewed patches are update appropriately, eg. ( "reject - untidy, 
please fix", "accept - expected version 2.4.18pre19" etc. )

(iii) patch versions, updates can be kept, as in mozilla's bugzilla 
site.  And comments on that patch can also be kept right along side the 
code.

Regardless of wether the current system is changed or not, the linux 
kernel would benefit from a central, searchable, public repository of 
patches.

The code is available, bugzilla has all this functionality today.

So here's hoping for a patchzilla.kernel.org :)

--Kervin



Denis Vlasenko wrote:
> On 29 January 2002 09:04, Rik van Riel wrote:
> 
>>On Mon, 28 Jan 2002, John Weber wrote:
>>
>>>I would be happy to serve as patch penguin, as I plan on collecting all
>>>patches anyway in my new duties as maintainer of www.linuxhq.com.
>>>
>>>we have the hardware/network capacity to serve as a limitless queue of
>>>waiting patches for Linus.
>>>
>>Please don't just accumulate stuff.
>>
> 
> Right. Accepting any patch is wrong policy. You'll be swamped.
> Patch must be marked "applies to 2.N.M", patch tracking system must check 
> that automagically.
> 
> Also each patch(set) can be commented by general public and by maintainers.
> If there is _no_ comment from any of _maintainers_ (i.e. it is not reviewed 
> or found too ugly to worth commenting) it is automatically dropped from the 
> system after some time.  This will force patch authors to care about code 
> quality.
> 
> If patch is too old (several releases behind) system can mail author(s):
> "Warning. Your patchset #3476346 needs rediffing. It will be dropped 
> otherwise"
> 
> These "small" details determine whether system is useful or just turns into 
> huge pile of patches of questionable value.
> --
> vda
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 



-- 
http://linuxquestions.org/ - Ask linux questions, give linux help.

Re: A modest proposal -- We need a patch tracking system.

[H. Peter Anvin]

Followup to:  <3C573428.3000404@fit.edu>
By author:    Kervin Pierre <kpierre@fit.edu>
In newsgroup: linux.dev.kernel
>
> Public patch tracking system/queue, maybe something derived from bugzilla.
> 
> (i) patches are sent to the maintainer and entered into the system.
> 
> (ii) reviewed patches are update appropriately, eg. ( "reject - untidy, 
> please fix", "accept - expected version 2.4.18pre19" etc. )
> 
> (iii) patch versions, updates can be kept, as in mozilla's bugzilla 
> site.  And comments on that patch can also be kept right along side the 
> code.
> 
> Regardless of wether the current system is changed or not, the linux 
> kernel would benefit from a central, searchable, public repository of 
> patches.
> 
> The code is available, bugzilla has all this functionality today.
> 
> So here's hoping for a patchzilla.kernel.org :)
> 

If Linus et al signs on to the idea, I'm sure we can build it...

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

John Weber <weber@nyc.rr.com> said:
> I would be happy to serve as patch penguin, as I plan on collecting all
> patches anyway in my new duties as maintainer of www.linuxhq.com.

Complete with "Patches only against 2.4.17 through 2.4.19", "Doesn't
compile con ARM with CONFIG_FOO", "Works fine on AXP"?

Plus search capability: Which files does it touch? What functions/variables
change/appear/dissapear? Etc?

Looks like a _HUGE_ ammount of work...

> I am currently writing code to scan the usual places for linux patches
> and automatically add them to our databases.  This would be really
> simplified by having patches sent to us.  And, since we already have a
> functioning site, we have the hardware/network capacity to serve as
> a limitless queue of waiting patches for Linus.  I would love nothing
> more than to update the site with information as to the status of these
> patches.

Again, as was discussed here: PLEASE do save the complete message. Oh, BTW
the following thread might have caveats, fixes, and important comments.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Olaf Dietsche]

Hi John,

John Weber <weber@nyc.rr.com> writes:

> I am currently writing code to scan the usual places for linux patches
> and automatically add them to our databases.  This would be really
> simplified by having patches sent to us.  And, since we already have a

How about extracting patches from lkml with procmail?

---cut here-->8---
:0 :
* ^sender: linux-kernel-owner@vger.kernel.org
* ^subject:.*patch
{
	:0 Bc:
	* ^--- .*/
	* ^+++ .*/
	linux-kernel-patches
}
---8<--cut here---

This recipe has its limits, but it's a start.

Regards, Olaf.

Re: A modest proposal -- We need a patch penguin

[James Stevenson]

> ---cut here-->8---
> :0 :
> * ^sender: linux-kernel-owner@vger.kernel.org
> * ^subject:.*patch
> {
> :0 Bc:
> * ^--- .*/
> * ^+++ .*/
> linux-kernel-patches
> }
> ---8<--cut here---
>
> This recipe has its limits, but it's a start.

well since most patches have a subject line
starting with [PATCH] its not hard to pull them out
with procmail.

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 07:33 PM 29/01/02 +0100, Olaf Dietsche wrote:

>How about extracting patches from lkml with procmail?
>
>---cut here-->8---
>:0 :
>* ^sender: linux-kernel-owner@vger.kernel.org
>* ^subject:.*patch
>{
>         :0 Bc:
>         * ^--- .*/
>         * ^+++ .*/
>         linux-kernel-patches
>}
>---8<--cut here---
>
>This recipe has its limits, but it's a start.

Actually I was sort of thinking that maybe part of the problem with our 
current system is the noise-to-signal ratio of lkml itself.

Perhaps it's time we set up a specific lkml-patch mailing list, and leave 
lkml for discussions about the problems. Have a script that posts general 
details about patches on lkml when there is a post to lkml-patch if you 
like, so people know and can go and take a look if they want. If you get 
complex, it can vet the patches to see if they apply, before pushing them 
to the list. It also goes well with some sort of patch tracking system (who 
says we can't use a mailing list as a distribution mechanism), if that gets 
the go ahead, while not requiring it.

Another possibility (or could even be combined) is that perhaps we need to 
start separating the mailing list at the code tree level.

eg: The "development" tree (lkml-dev which would currently contain 2.5.x) 
from the "stable" tree (lkml-stable which would currently contain 2.4.x) 
from the "older" trees (lkml-old which would currently contain 
2.2.x/2.0.x), at the mailing list level.

That way, people can concentrate on a specific tree (eg: Linus could 
concentrate on 2.5.x), without getting inundated with all the other stuff. 
This progresses easily when the next "stable" branch hits, so that the 
"dev" list can keep talking about what they plan to do while waiting for 
the stable to fork into the new development tree, and the previous stable 
joins the ranks of the "old" kernels, where it might possibly still get the 
occasional fix.

By reducing the noise (and hey, there is a reason people black-list certain 
subjects on lkml apart from personal/flame war issues), people can 
concentrate on the facts. The less noise (the less traffic?) the more 
likely every message will be read, patches will be checked, etc. Especially 
when you have other "duties" apart from maintaining kernel code, it's not 
always easy keeping up with lkml.


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 12:00:11PM +1100, Stuart Young wrote:
> Perhaps it's time we set up a specific lkml-patch mailing list, and leave 

I like the suggestion (most recently, of Daniel?  pardon if I
miscredit) of having patches-2.[45]@vger.kernel.org type addresses,
which would archive patches, and have a high noise-to-signal ratio.
Maybe even filter out all non-patches.

The big issue I cannot decide upon is whether standard e-mails should be
	To: torvalds@
	CC: patches-2.4@
or just
	To: patches-2.4@

(I'm guessing Linus would prefer the first, but who knows)

Also, something noone has mentioned is out-of-band patches.  Security fixes and other
patches which for various reasons go straight to Linus.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 02:18 am, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 12:00:11PM +1100, Stuart Young wrote:
> > Perhaps it's time we set up a specific lkml-patch mailing list, and leave 
> 
> I like the suggestion (most recently, of Daniel?  pardon if I
> miscredit) of having patches-2.[45]@vger.kernel.org type addresses,
> which would archive patches, and have a high noise-to-signal ratio.
> Maybe even filter out all non-patches.
> 
> The big issue I cannot decide upon is whether standard e-mails should be
> 	To: torvalds@
> 	CC: patches-2.4@
> or just
> 	To: patches-2.4@
> 
> (I'm guessing Linus would prefer the first, but who knows)

I'd say: cc Linus specifically if you think it's something he'd find 
personally interesting.  Leave out the cc if it's a minor bugfix or 
maintainance.

Oh, as somebody suggested in this thread, there is a difference in priority 
between bugfixes and other kinds of patches.  Should buxfixes go to 
patches-xxx@kernel.org with [BUGFIX] in the subject, or would 
bugs-xxx@kernel.org be a better idea?

> Also, something noone has mentioned is out-of-band patches.  Security fixes
> and other patches which for various reasons go straight to Linus.

Out-of-band patches are not going to stop.  The difference is, they will be 
duly noticed after the fact because they should be relatively few in 
comparison to in-band patches.

Another kind of out-of-band patch is where Linus takes the basic idea from 
somebody's patch and completely rewrites it, or does some hacking on his own, 
which he's been known to do.  Somehow I wouldn't expect he'd bother emailing 
the results to himself.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 08:18 PM 29/01/02 -0500, Jeff Garzik wrote:
>On Wed, Jan 30, 2002 at 12:00:11PM +1100, Stuart Young wrote:
> > Perhaps it's time we set up a specific lkml-patch mailing list, and leave
>
>I like the suggestion (most recently, of Daniel?  pardon if I
>miscredit) of having patches-2.[45]@vger.kernel.org type addresses,
>which would archive patches, and have a high noise-to-signal ratio.
>Maybe even filter out all non-patches.
>
>The big issue I cannot decide upon is whether standard e-mails should be
>         To: torvalds@
>         CC: patches-2.4@
>or just
>         To: patches-2.4@
>
>(I'm guessing Linus would prefer the first, but who knows)

Perhaps it'd be easier for patches-2.4 to actually send a copy to whoever 
is the relevant maintainer of a "section" (which could be worked out from 
the path in the patch, as long as it's made relevant to linux/) as well as 
the 2.4 maintainer? There is a lot of things that can be done here.

>Also, something noone has mentioned is out-of-band patches.  Security 
>fixes and other patches which for various reasons go straight to Linus.

Perhaps that is a good use for my lkml-patches idea, which gives those who 
have no avenue a place to post patches so they get picked up.

Something that does need to be done is that various directories under the 
kernel tree need to have someone "who receives patches" for that part, and 
who forwards them onto the kernel maintainer (eg: Linus, Marcello, etc) for 
further review/inclusion/rejection. This way, anything that doesn't fall 
under a particular maintainer gets sectioned off to someone, so it does get 
review, and hopefully a reply.


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 03:58:10 -0500, 
Jeff Garzik <garzik@havoc.gtf.org> wrote:
>On Thu, Jan 31, 2002 at 06:52:55PM +1100, Keith Owens wrote:
>You are missing a huge point.
>	You:	"Look Ma, nothing breaks!"
>	Ma sez: "It's supposed to, silly"

Hypocrisy, Jeff.  In your previous mail you complained that kbuild 2.5
was not ready to go in.  When I point out that not only is it ready but
it can go in without breaking the existing code, then you complain that
I am not breaking anything.  Make up your mind.

>Cleanup does not occur if cruft lives on as "backwards compatibility."
>You simply promote further bitrot and discontinuity.

The old code does not live on indefinitely, it gets removed as soon as
kbuild 2.5 is deemed stable.  Four weeks, tops.

>Let's see if I have this right:
>* You want completely duplicate build and config systems in the kernel,
>  with all the accompanying headaches for maintainers.

Only for as long as it takes to prove that kbuild 2.5 is ready.

>* You want to introduce a new system but don't give a shit about 2.5.

I have given up trying to get patches into 2.5.  The Linus black hole
swallows them all.

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Thu, Jan 31, 2002 at 11:03:07PM +1100, Keith Owens wrote:
> On Thu, 31 Jan 2002 03:58:10 -0500, 
> Jeff Garzik <garzik@havoc.gtf.org> wrote:
> >On Thu, Jan 31, 2002 at 06:52:55PM +1100, Keith Owens wrote:
> >You are missing a huge point.
> >	You:	"Look Ma, nothing breaks!"
> >	Ma sez: "It's supposed to, silly"
> 
> Hypocrisy, Jeff.  In your previous mail you complained that kbuild 2.5
> was not ready to go in.  When I point out that not only is it ready but
> it can go in without breaking the existing code, then you complain that
> I am not breaking anything.  Make up your mind.

You still do not appear to understand.  Please re-read my last message.

If kbuild was ready to go in, the old build system should go away.

ready != not breaking things, if you are breaking things on purpose.


> >Cleanup does not occur if cruft lives on as "backwards compatibility."
> >You simply promote further bitrot and discontinuity.
> 
> The old code does not live on indefinitely, it gets removed as soon as
> kbuild 2.5 is deemed stable.  Four weeks, tops.

You don't prove something stable with most people not using it.


> >Let's see if I have this right:
> >* You want completely duplicate build and config systems in the kernel,
> >  with all the accompanying headaches for maintainers.
> 
> Only for as long as it takes to prove that kbuild 2.5 is ready.

Not the Linux kernel way.

	Jeff

RE: A modest proposal -- We need a patch penguin

[Dana Lacoste]

> The whole point of the pristine tree is to give yourself a tree into 
> which you can import Linus patches.  If you start putting extra stuff
> in there you will get patch rejects.

or in the opposite direction : your changesets sent to linus have to be
patches against the pristine tree, not against your-working-tree-with-
several-patches-that-linus-doesn't-have.

(tying your response to Ingo into this one :)

it makes sense : any submitted patches should be against a known-clean
state, which means that the 'linear' element that people complain about
is actually bk enforcing some rather logical development practices.

but if linus isn't going to accept changesets (only patches) anyways,
then i guess it really doesn't matter :)

dana lacoste
ottawa, canada

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <B51F07F0080AD511AC4A0002A52CAB445B2B2F@ottonexc1.ottawa.loran.com>,
Dana Lacoste  <dana.lacoste@peregrine.com> wrote:
>
>but if linus isn't going to accept changesets (only patches) anyways,
>then i guess it really doesn't matter :)

I'm not going to accept changesets if they require ordering, but one of
the promises of bk is that I _could_ accept them eventually if the
infrastructure is good enough.

		Linus

Re: A modest proposal -- We need a patch penguin

[John L. Males]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

***** I am not on the Kernel Mailing list.  I would appreciate any
replies or references made to this eMail copy me in as well *****

***** My hope is this eMail will become part of the current thread
that was started with Rob Landley's initial eMail of 28 January 2002
*****

Ok, now to the meat of the matter.  I what has been proposed and the
discussion thereafter has had many opinions and thoughts.  I have not
had a chance to read all of the responses and comments, but have read
a number of them to develop a sense of what has been proposed, why it
was proposed and the pros and cons related to same.

First seeing as I am not a familar "face" in this mailing list or to
the community I want give a simple introduction to myself that I
think is very important.  I am a Career QA/Testing person.  For any
that may have any notion about my understanding of operating systems,
compilers, etc, suffice to say my early days involved heavy
modification to an IBM operating system by disassembling it back from
source binary cards and then making the modifications to the
"operating system", compilers, assembles, linker, librarian.  Many of
these very major changes to the OS.  This was before software had
copyright.  Ok I am dating myself.  I also wrote a replacement system
generation/load from scratch/bootstrap of a virgin system as well as
the supporting programs to produce the realted OS files, compilers,
support utilities.  This is not to upstage any developer or kernel
"hacker".  I lay this background information so all know I have an
appreciation of what is involved in maintaing an OS kernel agmonst
other things.

Without any intent to offend any member of the community I want to
note my comments are with a number of years experience related
QA/Testing and Change Control/Management.  I mentioned the above as
what I did not elude to was how this actually was the foundation of
my strong QA/Testing skills.

Ok, lets get to the meat of the matter.  Please bear with me as I
want to simplify what the proposal is about, the objections,
challenges and some of the enhanced suggestions.

Firstly with all due respect to Linus' view, there is a problem here.
 The proposal is a concerned attempt to address some of the problems
and/or formalize the unofficial practices that have been ongoing.

The basic essence of the issue at hand is the number of patches for
any number of reasons not making it into the main kernel tree as
maintained by Linus.  I appreciate some of the pros and cons why this
is as supported in the proposal and in Linus' comments.  I am also
aware of Linus perspective of allowing small forks of the kernel to
allow time and milage to determine how good a feature and/or
implementation is before accepting it into his kernel tree.

What concerns me is there are just too many kernel varients not just
via the Kernel Developer community, but also factoring in the various
distribution kernel varients.  In the kernel community we have not
just different trees, but a whole bunch of patches.  Many seem to
have very good merit.

The core issues with the pile of patches/varients is two fold.  One
not having a central source point to store the patches.  The second
the ability to have a single; choose (i.e. as would be case for VM
work); tree of a "bleeding edge" kernel.  It appears to me much time
is being wasted in trying to get patches accepted on a repeated basis
for a given patch, if it is accepted at all.

What we have despite the suggestion otherwise is a very forked
kernel.  In my humble opinion the matter is already out of control. 
I talk from experience and first hand coping with forked code.  A
development team I know of did fork the code initially due to
problems and time issues to get fixes or enhancements out to clients
in a shorter time.  The "collective" would not work or required much
more effort to work and time seemed not to permit development to do
so.  Does this sound familar, but different driving reasons why this
condition happens for the Kernel?  Suffice to say the matter was on
the same scale as the Kernel, just instead of many parties being
involved such as hackers and distributions in the Linux Kernel case,
it was done by the same company development teams for "business"
reasons to acheive a shorter deliverly of working code to customers.

The essence was the code continued to fork many more times with
several subset varients for each release and from release to release
some fixes/features were not available in each release.  Matters
finially became a support nightmare, and then problems where
customers needed certain fixes or features only for support to find
out some or all of the needs of the customers were mutually exclusive
- - In other words the "collective" had all of them, but implemented at
different patch or version levels.  In my humble opinion the Linux
kernel is close to that point right now.  The company development
teams finailly had to wake up to fact this temporary forking approach
could not longer meet customer needs and more importanly was becoming
a developmet nightmare in managing the code base.  The result was the
company had to put fixes on hold for several months as they tried to
merge the code base back to a single version.  The finial result was
a working finial single version, but at a extremely high cost, not
just in dollars, but in manpower and development effort to bring
things all back into line.

I will not go into the second example, but I have seen this happen
again.  With the same problems and issues, except it was a release
catch up issue and a "merge Master" person dedicated as the primary
developer to bring things back in line between the two product
versions.

It is my considered opinion the Linux kernel is a much larger code
base and level of complexity then either of the obove noted two
examples.  I see the writing on the wall.  Everyone agrees of the key
role Linus has as being the Architect and one who brings a wholeness
to the code standards, interfaces, etc of the Linux Kernel.  As I see
things now, that is slowly starting to slip and erode.  All one has
to do is look at how many Kernel tree varients exist.  Linus himself
may not be conerned, but I am and I think many people are.

I would ask that the community pause for moment and really think
about what is at state here, and if the real risk is worth taking?  I
for one hold the opinion it is not.

The Kernel has grown and it is time to adopt some formal processes
and open discussion to how best to implement those processes.  It is
simply necessary given how popular and important the Linux Kernel has
become and it is also part of the growth process of the Linux Kernel.
 Failing to accept the growth and further maturing of the Linux
Kernel will need to change some of the practices that were managable
when the Linux Kernel was a software "kid" or "early teenager" I am
afraid may end up resulting in its demise or the important guidance
of the "core" Linux team to the code and direction.

I have other thoughts and appreciate there have been arguments
presented for and against the proposal both in public and privately
between various people.  The first and most important step is to
first accept this need to accept it is time to take the approach of a
"staging" tree for all patches.  Once that is accepted, then the next
phase can be to determine how best to approach the challenge.  It
will be a challenge, but the community is so large and diverse I
believe it can accomplish the challenge.  What is important is to
focus on the issue at hand and try to keep a focus on that rather
than the side or down the road issues.  Like development of code, it
is an incremental and progressive itteration process.  This issue is
not different and can be acheived using the same process of thought,
design, debugging and implementation as all good code evolves from.

Thank you for your time and patience in reading my comments and
thoughts.  I hope in reflection and consideration of my background as
a prior assemble programmer, OS/Compiler/System person and my
QA/Testing experience that the community can understand why Rob
Landley's proposal was made and important to the Linux community at
large.


Regards,

John L. Males
Software I.Q. Consulting
Toronto, Ontario
Canada
30 January 2002 02:29
mailto:software_iq@TheOffice.net

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 6.5.8 for non-commercial use <http://www.pgp.com>
Comment: .

iQA/AwUBPFeg1PLzhJbmoDZ+EQJ+0wCghl7QBBKthhAb7aOPDMW9gbb/C+cAoLzE
9SodkZaQXNWNIg3aNBUyAiSE
=sluA
-----END PGP SIGNATURE-----



"What is your goal? Is it to reduce testing, or to reduce risk? Where is the greatest return?"
Linda Hayes is CEO of WorkSoft Inc. She was one of the founders of AutoTester.
>From article "Does Test Automation Save Time?", May 9, 2001 - IT Management

Re: A modest proposal -- We need a patch penguin

[Ed Tomlinson]

Linus Torvalds wrote:
 
> On Tue, 29 Jan 2002, Rob Landley wrote:
>> > >
>> > > Then why not give the subsystem maintainers patch permissions on your
>> > > tree. Sort of like committers.  The problem people have is that
>> > > you're dropping patches from those ten-twenty people you trust.
>> >
>> > No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
>> > _not_ dropping their patches (although I'm sometimes critical of them,
>> > and will tell them that they do not get applied).
>>
>> Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain,
>> Axel Boldt...
> 
> NONE of those are in the ten-twenty people group.
> 
> How many people do you think fits in a small group? Hint. It sure isn't
> all 300 on the maintainers list.
> 
>> Ah.  So being listed in the maintainers list doesn't mean someone is
>> actually a maintainer it makes sense to forward patches to?
> 
> Sure it does.
> 
> It just doesn't mean that they should send stuff to _me_.

This is the salient point.  I have been reading lkml for about two years 
and it was not an obivous one...  

> Did you not understand my point about scalability?  I can work with a
> limited number of people, and those people can work with _their_ limited
> number of people etc etc.

Why not arange the MAINTAINERS file so everyone knows the path you would
like patches to follow?  If everyone understands they should first try
lkml or the MAINTAINER and, once the MAINTAINER and/or lkml agree, the patch 
should be sent (by the MAINTAINER if he/she was involved) to a trustee who 
vets it again and sends it on to you.

Why not formalize the list of 'trustees' in the MAINTAINER files?  

IMO people will happily work with your procedures, but they _do_ have to 
understand them - not always an easy task.
 
<grin>

Ed Tomlinson

Re: A modest proposal -- We need a patch penguin

[Jesse Pollard]

Martin Dalecki <dalecki@evision-ventures.com>:
> 
> Linus Torvalds wrote:
> 
> >On Mon, 28 Jan 2002, Larry McVoy wrote:
> >
> >>What you didn't do, Linus, is paint a picture which allows development
> >>to scale up.
> >>
> >
> >Actually, I thought I did.
> >
> >Basic premise: development is done by humans.
> >
> >Now, look at how humans work. I don't know _anybody_ who works with
> >hundreds of people. You work with 5-10 people, out of a pool of maybe
> >30-50 people. Agreed?
> >
> Not at all. Please have a look at the ARMY. (A tightly hierarchical 
> system...)

And at each level (outside of training) there are usually one supervisor
for 8-15 people. At the lowest - a corpral. next sargent, ...
Though I can accept the lowest defined as a sargent, with an assistant.

And at the top - president, assisted by vice pres over the cabinet. next
level down, Secretary of Defense over Joint Chiefs...


-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: A modest proposal -- We need a patch penguin

[Andi Kleen]

torvalds@transmeta.com (Linus Torvalds) writes:

> In article <200201292208.g0TM8ql17622@ns.caldera.de>,
> Christoph Hellwig  <hch@ns.caldera.de> wrote:
> >In article <p73aduwddni.fsf@oldwotan.suse.de> you wrote:
> >> "Most times". For example the EA patches have badly failed so far, just because
> >> Linus ignored all patches to add sys call numbers for a repeatedly discussed 
> >> and stable API and nobody else can add syscall numbers on i386. 
> >
> >There still seems to be a lot of discussion vs EAs and ACLs.
> >Setting the suboptimal XFS APIs in stone doesn't make the discussion
> >easier.
> 
> In fact, every time I thought that the extended attributes had reached
> some kind of consensus, somebody piped up with some apparently major
> complaint. 
> 
> I think last time it was Al Viro.  Admittedly (_very_ much admittedly),
> making Al happy is really really hard.  His perfectionism makes his
> patches very easy to accept, but they make it hard for others to try to
> make _him_ accept patches.  But since he effectively is the VFS
> maintainer whether he wants it to be written down in MAINTAINERS or not,
> a comment from him on VFS interfaces makes me jump. 
> 
> The last discussion over EA's in my mailbox was early-mid December, and
> there were worries from Al and Stephen Tweedie.  I never heard from the
> worried people whether their worries were calmed.

Stephen's objection was that the EA specification didn't define a standard
format for ACL EAs too. That's certainly true, but outside the EA spec
actually. First there are good reasons to have ACLs mapped to generic EAs.
There are also already other applications of EAs independent
of ACLs, for example there is an DMAPI implementation for XFS which needs
EAs, or HFS and NTFS need to map arbitary EAs defined by the foreign fs
to some common backupable format, and various extended security projects
need generic "blob" EAs too to store their per file security information.
Mapping the ACLs to EAs has the advantage that you can backup/archive/
network sync all these things with a single API, instead of teaching all
your backup tools about first ACLs and then generic EAs. 

About the common ACL spec: there are unfortunately different flavours of
ACLs. All the unix like implementations so far (ext2/3-acl and XFS acl)
use something very near the withdrawn POSIX draft, while NTFS/CIFS/NFSv4
have completely different ACLs, and AFS has another flavour. Trying to map 
these into a single ACL syscall seemed to hard and likely doesn't make 
sense. 

ext2/XFS currently rely on a single user space library that generates 
the POSIX ACLs. NTFS/NFSv4/etc. are out of scope because it's not clear
how to merge them. With generic EAs you could have a different user library
that manages NTFS ACLs (and another one for AFS etc.), it seems there will 
be no clean way around this.

For the ACL format used by the bestbits libacl there is afaik currently
no draft, just some source code, but the semantics are defined by the
POSIX draft and relatively fixed and also quite common in other Unixes. 

Al's objection seemed to be that the patch adding the new VFS functions
passed a struct inode * instead of a struct dentry * in one case. 
Fixing that is a trivial search'n'replace. I think it wasn't done at first
because it required changing the VFS to pass a dentry to i_op->permission() 
and they didn't want to bloat the patch with search-n-replace in every
filesystem yet. If you would integrate it that could be done of course.
[in short it is a non brainer, has nothing to do with the API but is 
just a small implementation detail that can be easily changed] 

Does that answer your questions? 
Would you look at a patch again? 

-Andi

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On 30 Jan 2002, Andi Kleen wrote:
> Does that answer your questions?
> Would you look at a patch again?

That answers the specific questions about Al and Stephen.

It does NOT address whether consensus has been reached in general, and
whether people are happy. Is that the case?

Also, obviously nobody actually took over maintainership of the patch,
because equally obviously nobody has been pinging me about it. For some
reason you seem to want _me_ to go out of my way to search for patches
that are over a month old that I don't know whether they are valid or not,
used or not, or even agreed upon or not.

But yes, it's so much easier to blame me.

			Linus

Re: A modest proposal -- We need a patch penguin

[Nathan Scott]

hi Linus,

On Tue, Jan 29, 2002 at 03:13:14PM -0800, Linus Torvalds wrote:
> 
> On 30 Jan 2002, Andi Kleen wrote:
> > Does that answer your questions?
> > Would you look at a patch again?
> 
> That answers the specific questions about Al and Stephen.

Al had several (additional) issues with the original patch, but I
think we progressively worked through them - Al stopped suggesting
changes at one point anyway, and the level of abuse died away ;),
so I guess he became more satisfied with them.

> It does NOT address whether consensus has been reached in general, and
> whether people are happy. Is that the case?

I believe so - I know that the ext2/ext3 EA/ACL maintainer 
(AndreasG, CCd) is satisfied with the current patches, from
an XFS point of view they satisfy all our needs, and Anton
has chimed in saying this interface will work well for NTFS
EA support (Anton sent me patches and several suggestions as
well, which were included in the patch at the time).

> Also, obviously nobody actually took over maintainership of the patch,
> because equally obviously nobody has been pinging me about it. For some
> reason you seem to want _me_ to go out of my way to search for patches
> that are over a month old that I don't know whether they are valid or not,
> used or not, or even agreed upon or not.
> 
> But yes, it's so much easier to blame me.
> 

Not much point apportioning blame - its as much my fault - I
hadn't heard back from you at all since day 1, so figured you
were just not interested in this stuff, so I stopped sending.

The two patches which seemed to satisfy the most people are
below - they are unchanged from 2.5.0 but should apply to any
2.5.x tree (they are fairly non-intrusive and the system call
table hasn't changed since last time).  A complete userspace
implementation for both EAs and POSIX ACLs exists above these
interfaces.

cheers.

-- 
Nathan


[Patch #1: reserve syscall numbers]

diff -Naur 2.5.0-pristine/arch/i386/kernel/entry.S 2.5.0-reserved/arch/i386/kernel/entry.S
--- 2.5.0-pristine/arch/i386/kernel/entry.S	Sat Nov  3 12:18:49 2001
+++ 2.5.0-reserved/arch/i386/kernel/entry.S	Tue Dec  4 11:57:32 2001
@@ -622,6 +622,18 @@
 	.long SYMBOL_NAME(sys_ni_syscall)	/* Reserved for Security */
 	.long SYMBOL_NAME(sys_gettid)
 	.long SYMBOL_NAME(sys_readahead)	/* 225 */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for setxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for lsetxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for fsetxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for getxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* 230 reserved for lgetxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for fgetxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for listxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for llistxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for flistxattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* 235 reserved for removexattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for lremovexattr */
+	.long SYMBOL_NAME(sys_ni_syscall)	/* reserved for fremovexattr */
 
 	.rept NR_syscalls-(.-sys_call_table)/4
 		.long SYMBOL_NAME(sys_ni_syscall)
diff -Naur 2.5.0-pristine/include/asm-i386/unistd.h 2.5.0-reserved/include/asm-i386/unistd.h
--- 2.5.0-pristine/include/asm-i386/unistd.h	Thu Oct 18 03:03:03 2001
+++ 2.5.0-reserved/include/asm-i386/unistd.h	Tue Dec  4 11:58:21 2001
@@ -230,6 +230,18 @@
 #define __NR_security		223	/* syscall for security modules */
 #define __NR_gettid		224
 #define __NR_readahead		225
+#define __NR_setxattr		226
+#define __NR_lsetxattr		227
+#define __NR_fsetxattr		228
+#define __NR_getxattr		229
+#define __NR_lgetxattr		230
+#define __NR_fgetxattr		231
+#define __NR_listxattr		232
+#define __NR_llistxattr		233
+#define __NR_flistxattr		234
+#define __NR_removexattr	235
+#define __NR_lremovexattr	236
+#define __NR_fremovexattr	237
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */
 


[Patch #2: VFS implementation]

diff -Naur 2.5.0-pristine/arch/i386/kernel/entry.S 2.5.0-xattr/arch/i386/kernel/entry.S
--- 2.5.0-pristine/arch/i386/kernel/entry.S	Sat Nov  3 12:18:49 2001
+++ 2.5.0-xattr/arch/i386/kernel/entry.S	Thu Dec  6 12:59:47 2001
@@ -622,6 +622,18 @@
 	.long SYMBOL_NAME(sys_ni_syscall)	/* Reserved for Security */
 	.long SYMBOL_NAME(sys_gettid)
 	.long SYMBOL_NAME(sys_readahead)	/* 225 */
+	.long SYMBOL_NAME(sys_setxattr)
+	.long SYMBOL_NAME(sys_lsetxattr)
+	.long SYMBOL_NAME(sys_fsetxattr)
+	.long SYMBOL_NAME(sys_getxattr)
+	.long SYMBOL_NAME(sys_lgetxattr)	/* 230 */
+	.long SYMBOL_NAME(sys_fgetxattr)
+	.long SYMBOL_NAME(sys_listxattr)
+	.long SYMBOL_NAME(sys_llistxattr)
+	.long SYMBOL_NAME(sys_flistxattr)
+	.long SYMBOL_NAME(sys_removexattr)	/* 235 */
+	.long SYMBOL_NAME(sys_lremovexattr)
+	.long SYMBOL_NAME(sys_fremovexattr)
 
 	.rept NR_syscalls-(.-sys_call_table)/4
 		.long SYMBOL_NAME(sys_ni_syscall)
diff -Naur 2.5.0-pristine/fs/Makefile 2.5.0-xattr/fs/Makefile
--- 2.5.0-pristine/fs/Makefile	Tue Nov 13 04:34:16 2001
+++ 2.5.0-xattr/fs/Makefile	Thu Dec  6 12:59:47 2001
@@ -14,7 +14,7 @@
 		super.o block_dev.o char_dev.o stat.o exec.o pipe.o namei.o \
 		fcntl.o ioctl.o readdir.o select.o fifo.o locks.o \
 		dcache.o inode.o attr.o bad_inode.o file.o iobuf.o dnotify.o \
-		filesystems.o namespace.o seq_file.o
+		filesystems.o namespace.o seq_file.o xattr.o
 
 ifeq ($(CONFIG_QUOTA),y)
 obj-y += dquot.o
diff -Naur 2.5.0-pristine/fs/xattr.c 2.5.0-xattr/fs/xattr.c
--- 2.5.0-pristine/fs/xattr.c	Thu Jan  1 10:00:00 1970
+++ 2.5.0-xattr/fs/xattr.c	Thu Dec  6 12:59:58 2001
@@ -0,0 +1,341 @@
+/*
+  File: fs/xattr.c
+
+  Extended attribute handling.
+
+  Copyright (C) 2001 by Andreas Gruenbacher <a.gruenbacher@computer.org>
+  Copyright (C) 2001 SGI - Silicon Graphics, Inc <linux-xfs@oss.sgi.com>
+ */
+#include <linux/fs.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/smp_lock.h>
+#include <linux/file.h>
+#include <linux/xattr.h>
+#include <asm/uaccess.h>
+
+/*
+ * Extended attribute memory allocation wrappers, originally
+ * based on the Intermezzo PRESTO_ALLOC/PRESTO_FREE macros.
+ * The vmalloc use here is very uncommon - extended attributes
+ * are supposed to be small chunks of metadata, and it is quite
+ * unusual to have very many extended attributes, so lists tend
+ * to be quite short as well.  The 64K upper limit is derived
+ * from the extended attribute size limit used by XFS.
+ * Intentionally allow zero @size for value/list size requests.
+ */
+static void *
+xattr_alloc(size_t size, size_t limit)
+{
+	void *ptr;
+
+	if (size > limit)
+		return ERR_PTR(-E2BIG);
+
+	if (!size)	/* size request, no buffer is needed */
+		return NULL;
+	else if (size <= PAGE_SIZE)
+		ptr = kmalloc((unsigned long) size, GFP_KERNEL);
+	else
+		ptr = vmalloc((unsigned long) size);
+	if (!ptr)
+		return ERR_PTR(-ENOMEM);
+	return ptr;
+}
+
+static void
+xattr_free(void *ptr, size_t size)
+{
+	if (!size)	/* size request, no buffer was needed */
+		return;
+	else if (size <= PAGE_SIZE)
+		kfree(ptr);
+	else
+		vfree(ptr);
+}
+
+/*
+ * Extended attribute SET operations
+ */
+static long
+setxattr(struct dentry *d, char *name, void *value, size_t size, int flags)
+{
+	int error;
+	void *kvalue;
+	char kname[XATTR_NAME_MAX + 1];
+
+	if (flags & ~(XATTR_CREATE|XATTR_REPLACE))
+		return -EINVAL;
+
+	if (copy_from_user(kname, name, XATTR_NAME_MAX))
+		return -EFAULT;
+	kname[XATTR_NAME_MAX] = '\0';
+
+	kvalue = xattr_alloc(size, XATTR_SIZE_MAX);
+	if (IS_ERR(kvalue))
+		return PTR_ERR(kvalue);
+
+	if (size > 0 && copy_from_user(kvalue, value, size)) {
+		xattr_free(kvalue, size);
+		return -EFAULT;
+	}
+
+	error = -EOPNOTSUPP;
+	if (d->d_inode->i_op && d->d_inode->i_op->setxattr) {
+		lock_kernel();
+		error = d->d_inode->i_op->setxattr(d, kname, kvalue, size, flags);
+		unlock_kernel();
+	}
+
+	xattr_free(kvalue, size);
+	return error;
+}
+
+asmlinkage long
+sys_setxattr(char *path, char *name, void *value, size_t size, int flags)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk(path, &nd);
+	if (error)
+		return error;
+	error = setxattr(nd.dentry, name, value, size, flags);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_lsetxattr(char *path, char *name, void *value, size_t size, int flags)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk_link(path, &nd);
+	if (error)
+		return error;
+	error = setxattr(nd.dentry, name, value, size, flags);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_fsetxattr(int fd, char *name, void *value, size_t size, int flags)
+{
+	struct file *f;
+	int error = -EBADF;
+
+	f = fget(fd);
+	if (!f)
+		return error;
+	error = setxattr(f->f_dentry, name, value, size, flags);
+	fput(f);
+	return error;
+}
+
+/*
+ * Extended attribute GET operations
+ */
+static long
+getxattr(struct dentry *d, char *name, void *value, size_t size)
+{
+	int error;
+	void *kvalue;
+	char kname[XATTR_NAME_MAX + 1];
+
+	if (copy_from_user(kname, name, XATTR_NAME_MAX))
+		return -EFAULT;
+	kname[XATTR_NAME_MAX] = '\0';
+
+	kvalue = xattr_alloc(size, XATTR_SIZE_MAX);
+	if (IS_ERR(kvalue))
+		return PTR_ERR(kvalue);
+
+	error = -EOPNOTSUPP;
+	if (d->d_inode->i_op && d->d_inode->i_op->getxattr) {
+		lock_kernel();
+		error = d->d_inode->i_op->getxattr(d, kname, kvalue, size);
+		unlock_kernel();
+	}
+
+	if (kvalue && error > 0)
+		if (copy_to_user(value, kvalue, size))
+			error = -EFAULT;
+	xattr_free(kvalue, size);
+	return error;
+}
+
+asmlinkage long
+sys_getxattr(char *path, char *name, void *value, size_t size)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk(path, &nd);
+	if (error)
+		return error;
+	error = getxattr(nd.dentry, name, value, size);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_lgetxattr(char *path, char *name, void *value, size_t size)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk_link(path, &nd);
+	if (error)
+		return error;
+	error = getxattr(nd.dentry, name, value, size);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_fgetxattr(int fd, char *name, void *value, size_t size)
+{
+	struct file *f;
+	int error = -EBADF;
+
+	f = fget(fd);
+	if (!f)
+		return error;
+	error = getxattr(f->f_dentry, name, value, size);
+	fput(f);
+	return error;
+}
+
+/*
+ * Extended attribute LIST operations
+ */
+static long
+listxattr(struct dentry *d, char *list, size_t size)
+{
+	int error;
+	char *klist;
+
+	klist = (char *)xattr_alloc(size, XATTR_LIST_MAX);
+	if (IS_ERR(klist))
+		return PTR_ERR(klist);
+
+	error = -EOPNOTSUPP;
+	if (d->d_inode->i_op && d->d_inode->i_op->listxattr) {
+		lock_kernel();
+		error = d->d_inode->i_op->listxattr(d, klist, size);
+		unlock_kernel();
+	}
+
+	if (klist && error > 0)
+		if (copy_to_user(list, klist, size))
+			error = -EFAULT;
+	xattr_free(klist, size);
+	return error;
+}
+
+asmlinkage long
+sys_listxattr(char *path, char *list, size_t size)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk(path, &nd);
+	if (error)
+		return error;
+	error = listxattr(nd.dentry, list, size);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_llistxattr(char *path, char *list, size_t size)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk_link(path, &nd);
+	if (error)
+		return error;
+	error = listxattr(nd.dentry, list, size);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_flistxattr(int fd, char *list, size_t size)
+{
+	struct file *f;
+	int error = -EBADF;
+
+	f = fget(fd);
+	if (!f)
+		return error;
+	error = listxattr(f->f_dentry, list, size);
+	fput(f);
+	return error;
+}
+
+/*
+ * Extended attribute REMOVE operations
+ */
+static long
+removexattr(struct dentry *d, char *name)
+{
+	int error;
+	char kname[XATTR_NAME_MAX + 1];
+
+	if (copy_from_user(kname, name, XATTR_NAME_MAX))
+		return -EFAULT;
+	kname[XATTR_NAME_MAX] = '\0';
+
+	error = -EOPNOTSUPP;
+	if (d->d_inode->i_op && d->d_inode->i_op->removexattr) {
+		lock_kernel();
+		error = d->d_inode->i_op->removexattr(d, kname);
+		unlock_kernel();
+	}
+	return error;
+}
+
+asmlinkage long
+sys_removexattr(char *path, char *name)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk(path, &nd);
+	if (error)
+		return error;
+	error = removexattr(nd.dentry, name);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_lremovexattr(char *path, char *name)
+{
+	struct nameidata nd;
+	int error;
+
+	error = user_path_walk_link(path, &nd);
+	if (error)
+		return error;
+	error = removexattr(nd.dentry, name);
+	path_release(&nd);
+	return error;
+}
+
+asmlinkage long
+sys_fremovexattr(int fd, char *name)
+{
+	struct file *f;
+	int error = -EBADF;
+
+	f = fget(fd);
+	if (!f)
+		return error;
+	error = removexattr(f->f_dentry, name);
+	fput(f);
+	return error;
+}
diff -Naur 2.5.0-pristine/include/asm-i386/unistd.h 2.5.0-xattr/include/asm-i386/unistd.h
--- 2.5.0-pristine/include/asm-i386/unistd.h	Thu Oct 18 03:03:03 2001
+++ 2.5.0-xattr/include/asm-i386/unistd.h	Thu Dec  6 12:59:47 2001
@@ -230,6 +230,18 @@
 #define __NR_security		223	/* syscall for security modules */
 #define __NR_gettid		224
 #define __NR_readahead		225
+#define __NR_setxattr		226
+#define __NR_lsetxattr		227
+#define __NR_fsetxattr		228
+#define __NR_getxattr		229
+#define __NR_lgetxattr		230
+#define __NR_fgetxattr		231
+#define __NR_listxattr		232
+#define __NR_llistxattr		233
+#define __NR_flistxattr		234
+#define __NR_removexattr	235
+#define __NR_lremovexattr	236
+#define __NR_fremovexattr	237
 
 /* user-visible error numbers are in the range -1 - -124: see <asm-i386/errno.h> */
 
diff -Naur 2.5.0-pristine/include/linux/fs.h 2.5.0-xattr/include/linux/fs.h
--- 2.5.0-pristine/include/linux/fs.h	Fri Nov 23 06:46:19 2001
+++ 2.5.0-xattr/include/linux/fs.h	Thu Dec  6 12:59:47 2001
@@ -851,6 +851,10 @@
 	int (*revalidate) (struct dentry *);
 	int (*setattr) (struct dentry *, struct iattr *);
 	int (*getattr) (struct dentry *, struct iattr *);
+	int (*setxattr) (struct dentry *, char *, void *, size_t, int);
+	int (*getxattr) (struct dentry *, char *, void *, size_t);
+	int (*listxattr) (struct dentry *, char *, size_t);
+	int (*removexattr) (struct dentry *, char *);
 };
 
 /*
diff -Naur 2.5.0-pristine/include/linux/limits.h 2.5.0-xattr/include/linux/limits.h
--- 2.5.0-pristine/include/linux/limits.h	Thu Jul 29 03:30:10 1999
+++ 2.5.0-xattr/include/linux/limits.h	Thu Dec  6 12:59:47 2001
@@ -13,6 +13,9 @@
 #define NAME_MAX         255	/* # chars in a file name */
 #define PATH_MAX        4095	/* # chars in a path name */
 #define PIPE_BUF        4096	/* # bytes in atomic write to a pipe */
+#define XATTR_NAME_MAX   255	/* # chars in an extended attribute name */
+#define XATTR_SIZE_MAX 65536	/* size of an extended attribute value (64k) */
+#define XATTR_LIST_MAX 65536	/* size of extended attribute namelist (64k) */
 
 #define RTSIG_MAX	  32
 
diff -Naur 2.5.0-pristine/include/linux/xattr.h 2.5.0-xattr/include/linux/xattr.h
--- 2.5.0-pristine/include/linux/xattr.h	Thu Jan  1 10:00:00 1970
+++ 2.5.0-xattr/include/linux/xattr.h	Thu Dec  6 12:59:47 2001
@@ -0,0 +1,15 @@
+/*
+  File: linux/xattr.h
+
+  Extended attributes handling.
+
+  Copyright (C) 2001 by Andreas Gruenbacher <a.gruenbacher@computer.org>
+  Copyright (C) 2001 SGI - Silicon Graphics, Inc <linux-xfs@oss.sgi.com>
+*/
+#ifndef _LINUX_XATTR_H
+#define _LINUX_XATTR_H
+
+#define XATTR_CREATE	0x1	/* set value, fail if attr already exists */
+#define XATTR_REPLACE	0x2	/* set value, fail if attr does not exist */
+
+#endif	/* _LINUX_XATTR_H */

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Nathan Scott wrote:
>
> Al had several (additional) issues with the original patch, but I
> think we progressively worked through them - Al stopped suggesting
> changes at one point anyway, and the level of abuse died away ;),
> so I guess he became more satisfied with them.

I think you can safely assume that if Al doesn't curse you to hell, he can
be considered happy.

> Not much point apportioning blame - its as much my fault - I
> hadn't heard back from you at all since day 1, so figured you
> were just not interested in this stuff, so I stopped sending.

Basically, you should always consider email to me to be a unreliable
medium, with no explicit congestion control. So think of an email like a
TCP packet, with exponential backoff - except the times are different (in
TCP, the initial timeout is three seconds, and the max timeout is 2
minutes. In "Linus-lossy-network" it makes sense to use different
default and maximum values ;)

		Linus

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 03:59 PM 29/01/02 -0800, Linus Torvalds wrote:
>Basically, you should always consider email to me to be a unreliable
>medium, with no explicit congestion control. So think of an email like a
>TCP packet, with exponential backoff - except the times are different (in
>TCP, the initial timeout is three seconds, and the max timeout is 2
>minutes. In "Linus-lossy-network" it makes sense to use different
>default and maximum values ;)

Actually it's more like UDP. *grin* Least with TCP we get an ACK that the 
connection is accepted, and some sort of status is kept. Not so sure we 
have that with you all the time.

But hey, lots of things run over UDP, just a matter of making sure everyone 
realizes it's not a guaranteed medium really, isn't it?


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 06:13 pm, Linus Torvalds wrote:
> On 30 Jan 2002, Andi Kleen wrote:
> > Does that answer your questions?
> > Would you look at a patch again?
>
> That answers the specific questions about Al and Stephen.
>
> It does NOT address whether consensus has been reached in general, and
> whether people are happy. Is that the case?

Wouldn't it be nice if somebody could collect this sort of information for 
you?

> Also, obviously nobody actually took over maintainership of the patch,
> because equally obviously nobody has been pinging me about it.

I.E. patches get dropped, even when there's intense interest about them, due 
to sheer frustration burning out the patch's maintainer, and scaring away 
other potential maintainers.  (Is this NOT the case?  I could be wrong.  
Please point out the flaw in my logic...)

> For some
> reason you seem to want _me_ to go out of my way to search for patches
> that are over a month old that I don't know whether they are valid or not,
> used or not, or even agreed upon or not.

I thought I was actually proposing somebody else be formally tasked with the 
job of doing that...

> But yes, it's so much easier to blame me.

Maybe just to suggest that you could use a secretary?

> 			Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 08:00:19PM -0500, Rob Landley wrote:
> On Tuesday 29 January 2002 06:13 pm, Linus Torvalds wrote:
> > On 30 Jan 2002, Andi Kleen wrote:
> > > Does that answer your questions?
> > > Would you look at a patch again?
> >
> > That answers the specific questions about Al and Stephen.
> >
> > It does NOT address whether consensus has been reached in general, and
> > whether people are happy. Is that the case?
> 
> Wouldn't it be nice if somebody could collect this sort of information for 
> you?

So, you think "somebody" is willing to track all kernel issues and patches?

I don't think that's possible for any person or automated system.  [it's possible with a
combination of -people- and -automation-, IMHO, though]


> > Also, obviously nobody actually took over maintainership of the patch,
> > because equally obviously nobody has been pinging me about it.
> 
> I.E. patches get dropped, even when there's intense interest about them, due 
> to sheer frustration burning out the patch's maintainer, and scaring away 
> other potential maintainers.  (Is this NOT the case?  I could be wrong.  
> Please point out the flaw in my logic...)

I would point out a flaw if I actually saw any logic...

Are you talking about the EA patch?  See Nathan's response and other responses in this
thread.


> > For some
> > reason you seem to want _me_ to go out of my way to search for patches
> > that are over a month old that I don't know whether they are valid or not,
> > used or not, or even agreed upon or not.
> 
> I thought I was actually proposing somebody else be formally tasked with the 
> job of doing that...

You clearly do not realize the enormity of the task.


> > But yes, it's so much easier to blame me.
> 
> Maybe just to suggest that you could use a secretary?

I suggest that you need a Valium.

	Jeff

RE: A modest proposal -- We need a patch penguin

[Nickolaos Fotopoulos]

I work at a incoming call center and when the phone get to be too much for
us to handle all of overflow automatically goes to them.  This help keep our
client happy and us in business when a client unexpectidly run an TV ad.
Maybe something could be setup to handle the overflow that Linus recieves on
a regular basis.  BTW this is my first post to the list and firstday on the
list so please excuse any ignorance, as i might have just restated what
other have already said.

Nick Fotopoulos

Re: A modest proposal -- We need a patch penguin

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > If a patch gets ignored 33 times in a row then perhaps the person doing
> > the patch should first think really hard about the following 4 issues:
> 
> Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4 liners.
> The big stuff is not the problem most times. That stuff does get ripped to

"Most times". For example the EA patches have badly failed so far, just because
Linus ignored all patches to add sys call numbers for a repeatedly discussed 
and stable API and nobody else can add syscall numbers on i386. 

-Andi

Re: A modest proposal -- We need a patch penguin

[Christoph Hellwig]

In article <p73aduwddni.fsf@oldwotan.suse.de> you wrote:
> "Most times". For example the EA patches have badly failed so far, just because
> Linus ignored all patches to add sys call numbers for a repeatedly discussed 
> and stable API and nobody else can add syscall numbers on i386. 

There still seems to be a lot of discussion vs EAs and ACLs.
Setting the suboptimal XFS APIs in stone doesn't make the discussion
easier.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: A modest proposal -- We need a patch penguin

[Andi Kleen]

On Tue, Jan 29, 2002 at 11:08:52PM +0100, Christoph Hellwig wrote:
> In article <p73aduwddni.fsf@oldwotan.suse.de> you wrote:
> > "Most times". For example the EA patches have badly failed so far, just because
> > Linus ignored all patches to add sys call numbers for a repeatedly discussed 
> > and stable API and nobody else can add syscall numbers on i386. 
> 
> There still seems to be a lot of discussion vs EAs and ACLs.

At least the last l-k discussion ended in relative conclusion as far as 
I remember (only disagreement was from someone wanting to implement them
in sys_reiser4) 

> Setting the suboptimal XFS APIs in stone doesn't make the discussion
> easier.

The presented APIs were not the XFS APIs, but a significantly revised 
version, based on a mix of ext2-acl and XFS and some new changes. 

See http://acl.bestbits.at/man/extattr.2.html and 
http://acl.bestbits.at/man/extattr.5.html

If you think anything is badly "suboptimal" proposal you should have stated
your criticism. 


-Andi

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <200201292208.g0TM8ql17622@ns.caldera.de>,
Christoph Hellwig  <hch@ns.caldera.de> wrote:
>In article <p73aduwddni.fsf@oldwotan.suse.de> you wrote:
>> "Most times". For example the EA patches have badly failed so far, just because
>> Linus ignored all patches to add sys call numbers for a repeatedly discussed 
>> and stable API and nobody else can add syscall numbers on i386. 
>
>There still seems to be a lot of discussion vs EAs and ACLs.
>Setting the suboptimal XFS APIs in stone doesn't make the discussion
>easier.

In fact, every time I thought that the extended attributes had reached
some kind of consensus, somebody piped up with some apparently major
complaint. 

I think last time it was Al Viro.  Admittedly (_very_ much admittedly),
making Al happy is really really hard.  His perfectionism makes his
patches very easy to accept, but they make it hard for others to try to
make _him_ accept patches.  But since he effectively is the VFS
maintainer whether he wants it to be written down in MAINTAINERS or not,
a comment from him on VFS interfaces makes me jump. 

The last discussion over EA's in my mailbox was early-mid December, and
there were worries from Al and Stephen Tweedie.  I never heard from the
worried people whether their worries were calmed.

Maybe they did, and maybe they didn't.  If somebody doesn't tell me that
they are resolved, and that the people who would actually _use_ and
maintain this interface agrees on it, how can you expect me to ever
apply a patch?

			Linus

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> Maybe they did, and maybe they didn't.  If somebody doesn't tell me that
> they are resolved, and that the people who would actually _use_ and
> maintain this interface agrees on it, how can you expect me to ever
> apply a patch?

Ok now I'm confused. What is the criteria that decides "Im waiting silently
for everyone to guess I need answers from two people whom you must deduce
from telepathy" versus "bugger this lets rip this out and try xyz's approach"
 - which you also do at times without everyone being remotely happy on the
subject.

Alan

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Jan 29, 2002  22:56 +0100, Andi Kleen wrote:
> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> > Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4
> > liners. The big stuff is not the problem most times.
>
> "Most times". For example the EA patches have badly failed so far, just
> because Linus ignored all patches to add sys call numbers for a repeatedly
> discussed  and stable API and nobody else can add syscall numbers on i386. 

But at times, keeping things external to the kernel for a good while isn't
a bad thing either.  Basically, once code is part of the kernel, the API
is much harder to change than if it was always an optional patch.

For example, the EA patches have matured a lot in the time that they have
been external to the kernel (i.e. the move towards a common API with ext2
and XFS).  Even so, the ext2 shared EA on-disk representation leaves a
bit to be desired, because it is only useful in the case of shared ACLs,
and fails if you add any other kind of EA type.  See discussions on
ext2-devel for more information.

Similarly, my ext2 online resizing code was (and is) just fine, but when
I implemented the ext3 online resizing code (not yet available) I realized
I needed to change the on-disk format of some structures and this would
be much harder to do if it was part of the official kernel.

Like Ingo was saying, having to look over your code for a while also helps
it mature, and gives you some leeway to change it.  That said, there is
also a benefit to adding code to the kernel, as it increases your user
base a LOT, and no code that is external to the kernel can be considered
as mature as that which is included into the kernel, I think.  Drivers
may be an exception, because either you need the driver or you don't,
and people have little way of testing a driver if they don't have the hw.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Andi Kleen]

On Tue, Jan 29, 2002 at 03:24:26PM -0700, Andreas Dilger wrote:
> On Jan 29, 2002  22:56 +0100, Andi Kleen wrote:
> > Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> > > Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4
> > > liners. The big stuff is not the problem most times.
> >
> > "Most times". For example the EA patches have badly failed so far, just
> > because Linus ignored all patches to add sys call numbers for a repeatedly
> > discussed  and stable API and nobody else can add syscall numbers on i386. 
> 
> But at times, keeping things external to the kernel for a good while isn't
> a bad thing either.  Basically, once code is part of the kernel, the API
> is much harder to change than if it was always an optional patch.
> 
> For example, the EA patches have matured a lot in the time that they have
> been external to the kernel (i.e. the move towards a common API with ext2
> and XFS).  Even so, the ext2 shared EA on-disk representation leaves a

The problem is that there are already 5-6 different incompatible system calls 
with either different numbers or different semantics in significant 
deployment in the wild. EA/ACLs is an important feature for
samba servers so they are rather popular. While it's IMHO ok to do limited
testing in private the critical threshold where incompatible system call
ABIs are just a big problem for linux has long been surpassed. 
One of the success linux/i386 had in the past was to maintain a very stable
kernel ABI, but at least in the EA space this archivement is currently
getting undermined badly. 

-Andi

Re: A modest proposal -- We need a patch penguin

[Greg Boyce]

I have a little bit of input from the masses.  I'm not much of a developer
at this point, but I have been reading lkml for several months and there
have been a few things I've noticed on this topic.

As people on both sides of this argument have pointed out, one single
person can only do so much.  No matter how good you are, you're not going
to catch everything.  Due to this, I'd like to suggest a dual
maintainership.  A primary maintainer for the bug changes, and a secondary
for any small bits that fall through the crack.

The thing about this method is that it's already been proven to work.
Before Marcelo took over 2.4, Linus was the primary maintainer, and Alan
was making sure that the small bits weren't forgotten (As well as
providing some testing for some major changes before they were quite
ready).

Dave Jones appears to be taking the same roll in the 2.5 series, and Alan
is coming back a bit for 2.4 again.

Why not make it official?  The dual tree system seems to work.  It
would be quite similiar to Debian's release system.  A stable, and a
testing branch.  As long as the patches from the secondary maintainer gets
handled in a timely manner, less small changes will fall through the
crack.

Just my 2 cents.

Greg Boyce

Re: A modest proposal -- We need a patch penguin

[Eric S. Raymond]

Ingo Molnar:
>If a patch gets ignored 33 times in a row then perhaps the person doing
>the patch should first think really hard about the following 4 issues:
>
>  - cleanliness
>  - concept
>  - timing
>  - testing
>
>a violation of any of these items can cause patch to be dropped *without
>notice*. Face it, it's not Linus' task to teach people how to code or how
>to write correct patches. Sure, he still does teach people most of the
>time, but you cannot *expect* him to be able to do it 100% of the time.

Since the "33 times in a row" seems to refer to my bad experience with the
Configure.help patches, I think I need to correct a misconception.

The patches in question were *documentation*.  No concept issue, no
timing issue, no testing issue (I don't know what a "cleanliness"
issue would be in this context).  I know that Michael Elizabeth
Chastain, the listed CML1 maintainer, has had similar frustrating
exoeriences trying to get documentation patches folded in.

We're not talking about obscure internals changes that could break the
kernel, we're talking zero-risk patches submitted by official maintainers.  
This is not a situation that ought to require a lot of judgment or
attention on Linus's part.  

The fact that Linus *does* have to pass on all such patches, and is
dropping a lot of them them on the floor, is the clearest possible
example of the weaknesses in the present system.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

RE: A modest proposal -- We need a patch penguin

[Dana Lacoste]

> A correct patch for this one giving a new maintainer was 
> posted to Linux
> kernel already

And here it is again, just in case you missed it :

ARPD SUPPORT
P:	Jonathan Layes
-M:	layes@loran.com
+M:	dlacoste@loran.com
L:	linux-net@vger.kernel.org
S:	Maintained
+W:	http://home.loran.com/~dlacoste/

Re: A modest proposal -- We need a patch penguin

[Alexei Podtelezhnikov]

Hi!

May I add just a pure speculation that some significant part of the problem
lays in human nature of the kernel developers? The perfectly logical path of a 
patch progression and acceptance, which goes through the subsystem  
maintainers, is greatly obscured by the fact that the author's name will 
likely never be mentioned in the his-highness-Linus' ChangeLog. So they send 
them directly or post to lkml, just hoping. So Linus is flooded with patches. 
So the maintainers are fishing for the related bits in the lkml.

So I suggest dropping the names from the holy ChangeLog at all, rather than 
mostly just mentioning well established authorities. Yes, this is likely 
to kill probably the only reward existing in this community. There are 
other ways. It would be cool if maintainers had released their trees before 
the holy resync. With rewarding ChangeLogs, this will attract the mighty 
community towards them. That's exactly how Alan won the love of the community.

www.kernel.org should have WHO-TO with the maintainers contacts. Gee, I 
wonder who decides on the personality of the particular maintainer. The 
unmaintained parts - well, Linus, unless you find who maintains it, it's 
your responsibility, it's not nice to ignore patches for several months. 
If you are not interested, tough but even a person who you dislike will 
suffice the community. People just want to know who to bother.

A.

PS. you guessed, I have no chance to be in the ChangeLog since I only 
watch debates here. 

Re: A modest proposal -- we need a patch penguin

[Brad Chapman]

Mr. Landley,

	WOW!

	I think you have described the problem EXACTLY. Just from what I have
read on lkml, it would seem that all the stuff that you describe is accurate.
I think that what you've written is very well written, clearly describes
what you believe must be done, and shows how having this new job would be
beneficial.

	BTW, I posted an article on KernelTrap (www.kerneltrap.com).

Thanks,

Brad

=====
Brad Chapman

Permanent e-mails: kakadu_croc@yahoo.com
		   jabiru_croc@yahoo.com
		   tanami_croc@devel.lbsd.net

__________________________________________________
Do You Yahoo!?
Great stuff seeking new owners in Yahoo! Auctions! 
http://auctions.yahoo.com

A modest proposal -- We need a patch penguin

[Rob Landley]

Patch Penguin Proposal.

 1) Executive summary.
 2) The problem.
 3) The solution.
 4) Ramifications.

 -- Executive summary.

Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are 
getting dropped on the floor on a regular basis. This is burning out 
maintainers and is increasing the number of different kernel trees (not yet a 
major fork, but a lot of cracks and fragmentation are showing under the 
stress). Linus needs an integration lieutenant, and he needs one NOW.

We need to create the office of "patch penguin", whose job would be to make 
Linus's life easier by doing what Alan Cox used to do, and what Dave Jones is 
unofficially doing right now. (In fact, I'd like to nominate Dave Jones for 
the office, although it's Linus's decision and it would be nice if Dave got 
Alan Cox's blessing as well.)

And if we understand this position, and formalize it, we can make better use 
of it. It can solve a lot of problems in linux development.

 -- The problem.

Linus doesn't scale, and his current way of coping is to silently drop the 
vast majority of patches submitted to him onto the floor. Most of the time 
there is no judgement involved when this code gets dropped. Patches that fix 
compile errors get dropped. Code from subsystem maintainers that Linus 
himself designated gets dropped. A build of the tree now spits out numerous 
easily fixable warnings, when at one time it was warning-free. Finished code 
regularly goes unintegrated for months at a time, being repeatedly resynced 
and re-diffed against new trees until the code's maintainer gets sick of it. 
This is extremely frustrating to developers, users, and vendors, and is 
burning out the maintainers. It is a huge source of unnecessary work. The 
situation needs to be resolved. Fast.

If you think I'm preaching to the choir, skip to the next bit called "the 
solution". If not, read on...

The Linux tree came very close to forking during 2.4. We went eleven months 
without a development tree. The founding of the Functionally Overloaded Linux 
Kernel project was a symptom of an enormous unintegrated patch backlog 
building up pressure until at least a small fork was necessary. Even with 2.5 
out, the current high number of seperate development trees accepting 
third-party is still alarmingly high. Linus and Marcelo have been joined by 
Dave Jones, Alan Cox, Andrea Arcangeli, Michael Cohen, and others, with 
distributors maintaining their own significantly forked kernel trees as a 
matter of course. Developers like Andrea, Robert Love and Rik van Riel either 
distribute others' relatively unrelated patches with their patch sets, or 
base their patches on other, as yet unintegrated patches for extended periods 
of time.

Discussion of this problem was covered by kernel traffic and Linux Weekly 
News:

http://kt.zork.net/kernel-traffic/kt20020114_150.html#5
http://lwn.net/2002/0103/kernel.php3 (search for "patch management").

During 2.4, the version skew between Alan Cox's tree and Linus's tree got as 
bad as it's ever been. Several of the bug fixes in Alan's tree (which he 
stopped maintaining months ago) still are not present in 2.4.17 or 2.5. Rik 
van Riel has publicly complained that Linus dropped his VM patches on the 
floor for several months, a contributing factor to the 2.4 VM's failure to 
stabilize for almost a -YEAR- after its release. (This is a bad sign. Whether 
Rik's or Andrea's VM is superior is a side issue. Alan Cox, and through him 
Red Hat, got Rik's VM to work acceptably by integrating patches from that 
VM's maintainer. The fact Linus didn't do as well is a symptom of this larger 
problem. The kind of subsystem swapping so major it requires a new maintainer 
should not be necessary during a stable series.)

Speaking of Andrea Arcangeli, he's now integrating third-party patches into 
his own released development trees, because 2.5 isn't suitable to do 
development against and 2.4 doesn't have existing work (like low latency) 
he's trying to extend.

Andre Hedrick just had to throw a temper tantrum to get any attention paid to 
his IDE patches, and he's the official IDE maintainer. Eric Raymond tells me 
his help file updates have now been ignored for 33 consecutive releases 
(again, he's the maintainer), and this combined with recent major changes 
Linus unilaterally made to 2.5's help files (still without syncing with the 
maintainer's version before doing so) has created a lot of extra and totally 
unnecessary work for Eric, and he tells me it's made the version skew between 
2.4 and 2.5 almost unmanageable.

Andrew Morton's lock splitting low latency work has no forseeable schedule 
for inclusion, despite the fact it's needed whether or not Robert Love's 
preemption patch goes in. Ingo Molnar's O(1) scheduler did go in, but that 
was largely a case of good timing: it came right on the heels of a public 
argument about why Linus had not accepted patches to the scheduler for 
several years. The inclusion of code like JFS, XFS, and Daniel Phillips' ext2 
indexing patch are left up to distributions to integrate and ship long before 
they make it into Linus's tree. (Remember the software raid code in 2.2?) 
These are just the patches that have persisted, how much other good work 
doesn't last because its author loses interest after six months of the silent 
treatment? The mere existence of the "Functionally Overloaded Linux Kernel" 
(FOLK) project, to collect together as many unintegrated patches as possible, 
was a warning sign that things were not going smoothly on the patch 
integration front.

The release of 2.5 has helped a bit, but by no means solved the problem. Dave 
Jones started his tree because 2.4 fixes Marcello had accepted were not 
finding their way into 2.5. Even code like Keith Owens' new build system and 
CML2, both of which Linus approved for inclusion at the Kernel summit almost 
a year ago and even tentatively scheduled for before 2.5.2, are still not 
integrated with no clear idea if they ever will be. (Yes Linus can change his 
mind about including them, but total silence isn't the best way to indicate 
this. Why leave Keith and Eric hanging, and wasting months or even years of 
their time still working on code Linus may not want?)

The fact that 2.5 has "pre" releases seems suggestive of a change in mindset. 
A patch now has to be widely used, tested, and recommended even to get into 
2.5, yet how does the patch get such testing before it's integrated into a 
buildable kernel? Chicken and egg problem there, you want more testing but 
without integration each testers can test fewer patches at once.

There has even been public talk of CRON jobs to resubmit patches to Linus 
periodically for as long as it takes him to get around to dealing with them. 
Linus has actually endorsed this approach (provided that re-testing of the 
patches against the newest release before they are remailed is also 
automated). This effort has spawned a mailing list. That's just nuts. The 
fact that Linus doesn't scale isn't going to be solved by increasing the 
amount of mail he gets. When desperate measures are being seriously 
considered, we must be in desperate times.

-- The solution.

The community needs to offload some work from Linus, so he can focus on what 
he does that nobody else can. This offloading and delegation has been done 
before, with the introduction of subsystem maintainers. We just need to 
extend the maintainer concept to include an official and recognized 
integration maintainer.

During 2.1, when Linus burned out, responsibility for various subsystems were 
delegated to lieutenants to make Linus' job more manageable. Lieutenants are 
maintainers of various parts of the tree who collect and clean up patches, 
and make the first wave of obvious decisions ("this patch doesn't apply", 
"this bit doesn't compile", "my machine panicked", "look, it's not SMP safe") 
before sending tested code off to Linus. Linus still spends a lot of his time 
reading and auditing code, but by increasing the average quality of the code 
Linus looks at, the maintainers make his job easier. The more work 
maintainers can do the less Linus has to.

So what tasks does Linus still personally do? He's an architect. He steers 
the project, vetoing patches he doesn't like and suggesting changes in 
direction to the developers. And he's the final integrator, pulling together 
dispirate patches into one big system.

The job of architect is something only Linus can do. The job of integrator is 
something many people can do. Alan Cox did it for years. Dave Jones is doing 
it now, and Michael Cohen has yet another tree. Every Linux distributor has 
its own tree. Integrating patches so they don't conflict and porting them to 
new versions is hard work, but not brain surgery.

Linus is acting as a central collection point for patches, and patches are 
getting lost on their way to that collection point. Patches as big as UML and 
EXT3 were going into Alan Cox's tree during 2.4, and now new patches are 
going into Dave Jones's tree to be tested out and queued for Linus.

Integration is a task that can be delegated, and it has been. In Perl's 
model, Larry Wall is the benevolent dictator and architect, but integration 
work is done by the current holder of the "patch pumpkin". In Linux, Alan Cox 
used to be the de facto holder of the patch penguin, and now Dave Jones is 
maintaining a tree which can accept and integrate patches, and then feed them 
on to Linus when Linus is ready.

This system should be formalized, we need the patch penguin to become 
official. The patch penguin seems to have passed from Alan Cox to Dave Jones. 
If we recognize this, we can make much better use of it.

 --- Ramifications.

The holder of the patch penguin's job would be to accept patches from people 
other than linus, make them work together in a publicly compilable and 
testable tree, and feed good patches on to Linus. This may sound simple and 
obvious, but it's currenlty not happening and its noticeable by its absence.

The purpose of the patch penguin tree is to make life easier, both for Linus 
and the developer community. With a designated patch collector and 
integrator, Linus's job becomes easier. Linus would still maintain and 
release his own kernel trees (the way he did when Alan Cox regularly fed him 
patches), and Linus could still veto any patch he doesn't like (whether it 
came from the patch penguin, directly from a subsystem maintainer, or 
elsewhere). But Linus wouldn't be asked to act as the kernel's public CVS 
tree anymore. He could focus on being the architect.

The bulk of the patch penguin's work would be to accept, integrate, and 
update patches from designated subsystem maintainers, maintaining his own 
tree (seperate from linus's) containing the integrated collection of pending 
patches awaiting inclusion in Linus's tree. Patches submitted directly to the 
patch penguin could be redirected to subsystem maintainers where appropriate, 
or bounced with a message to that effect (at the patch penguin's option). 
This integration and patch tracking work is a fairly boring, thankless task, 
but it's work somebody other than Linus can do, which Linus has to do 
otherwise. (And which Linus is NOT doing a good job at right now.)

The rest of the patch-penguin holder's job is to feed Linus patches. The 
patch penguin holder's tree would fundamentally be a delta against the most 
recent release from Linus (like the "-ac patches" used to be). If Linus takes 
several releases to get around to looking at a new patch, the patch penguin 
would resynchronize their tree with each new Linus release, doing the fixups 
on the pending patch set (or bullying the source of each patch into doing 
so). It would be the patch penguin's job to keep up with Linus, not the other 
way around. When a change happens in Linus's tree that didn't come from the 
patch penguin, the patch penguin integrates the change into their tree 
automatically.

The holder of the patch penguin would feed Linus good patches, by Linus's 
standards. Not just tested ones, but small bite-sized patches, one per email 
as plain text includes, with an explanation of what each patch does at the 
top of the mail. (Just the way Linus likes them. :) Current pending patches 
from the patch penguin tree could even be kept at a public place (like 
kernel.org) so Linus could pull rather than push, and grab them when he has 
time. The patch penguin tree would make sure that when Linus is ready for a 
patch, the patch is ready for Linus.

The patch penguin tree can act as a buffer between Linus and the flood of 
patches from the field. When Linus is not ready for a patch yet, he can hold 
off on taking it into his tree, and doesn't have to worry about the patch 
being lost or out of date by the time he's ready to accept it. When Linus is 
focusing on something like the new block I/O code, the backlog of other 
patches naturally feeds into the patch penguin tree until Linus is ready to 
look at them. People won't have to complain about dropped patches, and Linus 
doesn't have to worry that patches haven't been tested enough before being 
submitted to him. Users who want to live on the really bleeding edge have a 
place to go for a kernel that's likely to break. Testers can find bugs en 
masse without having to do integration work (which is in and of itself a 
source of potential bugs).

Linus would still have veto power. He gets to reject any patch he doesn't 
like, and can ask for the integration lieutenant to back that patch out of 
the patch penguin tree. That's one big difference between the patch penguin 
tree and Linus's tree: the patch penguin tree is provisional. Stuff that goes 
in it can still get backed out in a version or two. Of course Linus would 
have to explicitly reject a patch to get it out of the patch penguin tree, 
meaning its developer stops fruitlessly re-submitting it to Linus, and maybe 
even gets a quick comment from Linus as to WHY it was unacceptable so they 
can fix it. (From the developer's point of view, this is a good thing. They 
either get feedback of closure.)

Linus sometimes needs time off. Not just for vacations, but to focus on 
specific subsections, like integrating Jens Axobe's BIO patches at the start 
of 2.5. Currently, these periods hopelessly stall or tangling development. 
But in Linus's absence, the patch penguin could continue to maintain a delta 
against the last Linus tree, and generate a sequence of small individual 
patches like a trail of bread crumbs for Linus to follow when he gets back. 
Linus could take a month off, and catch back up in a fraction of that time 
when he returned. (And if Linus were to get hit by a bus, the same 
infrastructure would allow the community to select and support a new 
architect, which might help companies like IBM sleep better at night.) And if 
Linus rejected patches halfway through the bread crumb trail requiring a lot 
of shuffling in later patches, well, that's more work for the patch penguin, 
not more work for Linus.

One reason Linus doesn't like CVS is he won't let other people check code 
into his tree. (This is not a capricious decision on Linus's part: no 
architect can function if he doesn't know what's in the system. Code review 
of EVERYTHING is a vital part of Linus's job.) With a patch penguin tree, 
there's no more pressure on Linus to use CVS. The patch penguin can use CVS 
if he wants to, and if he wants to give the subsystem maintainers commit 
access to the patch penguin tree, that's his business. The patch penguin's 
own internal housekeeping toolset shouldn't affect the patches he feeds on to 
Linus one way or the other.

Again, Linus likes stuff tested by a wide audience before it's sent to him. 
With a growing list of multiple trees maintained by Dave Jones, Alan Cox, 
Michael Cohen, Andrea Arcangeli, development and testing become fragmented. 
With a single patch penguin tree, the patches drain into a common pool and 
the backlog of unintegrated patches can't build up dangerous amounts of 
pressure to interfere with development. A single shared "pending" tree means 
the largest possible group of potential testers.

Re: A modest proposal -- We need a patch penguin

[Matthew D. Pitts]

Rob and company...

I agree 100% with this idea. I have several projects in the planning stages
that i want to integrate into 2.5, but I do NOT want them dropped with no
explanation. If i had a computer that I could devote to it, I would be
willing to volenteer to be patch penguin.

Matthew D. Pitts

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Mon, 28 Jan 2002 09:10:56 -0500
Rob Landley <landley@trommello.org> wrote:

> Patch Penguin Proposal.
> 
> [...]

You mean some sort of proxy/two-tier development? A "commit/rollback"
transaction model on the kernel itself?

I deeply agree with you, especially in keeping "many eyes" to look at the
same kernel tree, and not chosing one of the many subtrees; as added bonus,
this stuff is buzzword compliant! What we can ask more? :)

Now, Linus' call to accept _your_ patch. Fingers crossed already.

-- FM

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <200201290137.g0T1bwB24120@karis.localdomain>,
Francesco Munda  <syylk@libero.it> wrote:
>
>I deeply agree with you, especially in keeping "many eyes" to look at the
>same kernel tree, and not chosing one of the many subtrees; as added bonus,
>this stuff is buzzword compliant! What we can ask more? :)

Some thinking, for one thing.

One "patch penguin" scales no better than I do. In fact, I will claim
that most of them scale a whole lot worse. 

The fact is, we've had "patch penguins" pretty much forever, and they
are called subsystem maintainers.  They maintain their own subsystem, ie
people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
(USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
Molnar (scheduler), Jeff Garzik (network drivers) etc etc. 

If there are problems with certain patches, it tends to be due to one or
more of:

 - the subsystem is badly modularized (quite common, originally. I don't
   think many people realize how _far_ Linux has come in the last five
   years wrt issues like architectures, driver independence, filesystem
   infrastructure etc). And it still happens.

 - lack of maintainer interest. Many "maintainers" are less interested
   in true merging than in trying to just push whatever code they have,
   and only ever grow their patches instead of keeping them in pieces.

   This is usually a matter of getting used to it, and the best people
   get used to it really quickly (Andrea, for example, used to not do
   this well at all, but look at how he does it now! From a merge
   standpoint, his patches have gone from "horrible" to "very good")

 - personality/communication issues. Yes, they happen. I've tried to
   have other people be "filters" for the people I cannot work with, but
   I have to say that most of the time when I can't work with somebody,
   others have real problems with those people too. 

   (An example of a very successful situation: David Miller and Alexey
   Kuznetsov: Alexey used to have these rather uncontrolled patches that
   I couldn't judge or work with but that obviously had a lot of
   potential, and David acting as a filter made them a very successful
   team.)

In short, if you have areas or patches that you feel have had problems,
ask yourself _why_ those areas have problems. 

A word of warning: good maintainers are hard to find.  Getting more of
them helps, but at some point it can actually be more useful to help the
_existing_ ones.  I've got about ten-twenty people I really trust, and
quite frankly, the way people work is hardcoded in our DNA.  Nobody
"really trusts" hundreds of people.  The way to make these things scale
out more is to increase the network of trust not by trying to push it on
me, but by making it more of a _network_, not a star-topology around me. 

In short: don't try to come up with a "patch penguin".  Instead try to
help existing maintainers, or maybe help grow new ones. THAT is the way
to scalability.

		Linus

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 10:23 pm, Linus Torvalds wrote:
> In article <200201290137.g0T1bwB24120@karis.localdomain>,
>
> Francesco Munda  <syylk@libero.it> wrote:
> >I deeply agree with you, especially in keeping "many eyes" to look at the
> >same kernel tree, and not chosing one of the many subtrees; as added
> > bonus, this stuff is buzzword compliant! What we can ask more? :)
>
> Some thinking, for one thing.
>
> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse.

Sure.  But Alan doesn't, and Dave Jones (with enough experience) shouldn't.

You have architecture duties.  You're worried about the future of the code.  
You have to understand just about everybody's subsystem, so you can veto a 
patch from somebody like Jens Axboe or Andre Hedrick if you have an objection 
to it.

An integration maintainer would NOT be making any major architectural 
decisions, they would be integrating the code from the maintainers, 
collecting the patches for the unmaintained areas of code, and resolving 
issues between maintainers that are purely implementation details.

Then you code review what they do anyway as the architect, saying whether or 
not it's a good idea.  But you don't have to deal with the obvious grunt work 
that's largely a matter of figuring out which bits don't compile because 
person A was not talking to person B.

> The fact is, we've had "patch penguins" pretty much forever, and they
> are called subsystem maintainers.  They maintain their own subsystem, ie
> people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
> (USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
> Molnar (scheduler), Jeff Garzik (network drivers) etc etc.

I'm suggesting an integration maintainer, whose explicit job is to put 
together patches from the various subsystem maintainers, and only directly 
accept patches for areas of code that do not HAVE any other subsystem 
maintainer.

Alan Cox used to do this (and is starting to do it again for Marcelo in 2.4). 
 Dave Jones is currently the guy doing this for you, taking patches, sorting 
through them, and then feeding them on to you.

> If there are problems with certain patches, it tends to be due to one or
> more of:
>
>  - the subsystem is badly modularized (quite common, originally. I don't
>    think many people realize how _far_ Linux has come in the last five
>    years wrt issues like architectures, driver independence, filesystem
>    infrastructure etc). And it still happens.

Yup.  Architecture issue.  Still your problem, I'm fraid.

>  - lack of maintainer interest. Many "maintainers" are less interested
>    in true merging than in trying to just push whatever code they have,
>    and only ever grow their patches instead of keeping them in pieces.

Patch penguin's job.  Foist this grunt work off on him.

>    This is usually a matter of getting used to it, and the best people
>    get used to it really quickly (Andrea, for example, used to not do
>    this well at all, but look at how he does it now! From a merge
>    standpoint, his patches have gone from "horrible" to "very good")

And needed patches from people who aren't very good have to wait years for 
the developer to learn how to feed you the right kind of patches?

If the patch is for a specific subsystem, then obviously your first line of 
defense fighting off sturgeon's law is the subsystem maintainer.  But you 
don't seem to have been taking patches even from subsystem maintainers in a 
timely manner, and how can people tell the difference between you dropping a 
patch because you have an objection to it and you dropping a patch because 
your mailbox overfloweth?  (You keep complaining people never send you 
patches.  People are suggesting automated patch remailers to spam your 
mailbox even harder.  There has GOT to be a better way...)

>  - personality/communication issues. Yes, they happen. I've tried to
>    have other people be "filters" for the people I cannot work with, but
>    I have to say that most of the time when I can't work with somebody,
>    others have real problems with those people too.
>
>    (An example of a very successful situation: David Miller and Alexey
>    Kuznetsov: Alexey used to have these rather uncontrolled patches that
>    I couldn't judge or work with but that obviously had a lot of
>    potential, and David acting as a filter made them a very successful
>    team.)
>
> In short, if you have areas or patches that you feel have had problems,
> ask yourself _why_ those areas have problems.

Query: Do you not believe you have been dropping a significant number of good 
patches on the floor?

> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and
> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me.

You don't see an integration maintainer as a step in the right direction?  
(It's not a star topology, it's a tree.)

Having lots of dispirate overlapping trees fragments development, fragments 
the testers, and makes an awful lot more WORK for everybody involved.

> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

Are you saying that Alan Cox's didn't serve a purpose during the 2.2 kernel 
time frame, and that Dave Jones is currently wasting his time? 

I'm confused here: "don't try to come up with a patch penguin", "help 
existing maintainers" (that's the patch penguin's job) "help grow new 
[maintainers]"...  The patch penguin IS an integration maintainer.  That's 
what I'm talking about.  (Patch penguin, patch pumpkin.  Patch pumkin, patch 
penguin.  I can say "integration maintainer" if it would help...)

I missed a curve somewhere.  Maybe the original message wasn't clear?  I am 
suggesting making Dave Jones the integration maintainer (a position he 
currently unofficially holds, and which Alan Cox did before him), and simply 
telling everybody who's complaining that their patches are getting silently 
dropped or ignored to try getting them into HIS tree first before bothering 
you about it.

I'm not asking for a major change here, I'm talking about clarifying the 
current ad-hoc development process.  Formalizing an existing de facto 
position so people farther out in the development process know what to do and 
where to go.

> 		Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Mon, 28 Jan 2002, Rob Landley wrote:
>
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
> > quite frankly, the way people work is hardcoded in our DNA.  Nobody
> > "really trusts" hundreds of people.  The way to make these things scale
> > out more is to increase the network of trust not by trying to push it on
> > me, but by making it more of a _network_, not a star-topology around me.
>
> You don't see an integration maintainer as a step in the right direction?
> (It's not a star topology, it's a tree.)

No, I don't really think an "integration manager" works well.

I think it helps a lot to have people pick up patches that nobody else
wants to maintain, and to gather them up. Andrea does that to some degree.
But it is _much_ better if you have somebody who is a point-man for
specific areas.

The problem with an overall guy is that there can't be too many of them.
The very thing you are _complaining_ about is in fact that there are a
number of over-all guys without clear focus, which only leads to confusion
about who handles what.

Clarity is good.

> Are you saying that Alan Cox's didn't serve a purpose during the 2.2 kernel
> time frame, and that Dave Jones is currently wasting his time?

No, I'm saying that there are not very many peopel who can do it, and who
can get the kind of trust that they are _everywhere_. Let's face it, Alan
grew to be respected because he did lots of different stuff over many
years, and he proved himself more than capable. And I suspect he's _quite_
happy not being in the middle of it right now.. It's a tough job.

It's a lot more likely to find people who can maintain _parts_. And if
there are patches that fall out of those parts, that tends to indicate a
lack of modularity, and perhaps a lack of maintainer for those parts.

And more likely, even if you _do_ find ten people who can do everything,
you don't want them to.

		Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Mon, Jan 28, 2002 at 10:00:19PM -0800, Linus Torvalds wrote:
> And more likely, even if you _do_ find ten people who can do everything,
> you don't want them to.

[and other things, in general saying that he wasn't happy with the approach]

What you didn't do, Linus, is paint a picture which allows development
to scale up.  Perhaps you don't want it to do so; looking back on Sun's
development process has taught me that a lot of it was just a way to
slow things down enough that the architects could keep up with the many
hacks going on here there and everywhere.  If this was Sun, I'd say that
slowing things down is good.

But cast your mind back to your "Linux is evolution" line of reasoning
and ask yourself if that does not mean that allowing the fastest
forward progress is what you want.  It would seem so, but at this point,
Linus, it is hard to predict what you think, so I'll pass on guessing.

What would be nice is if you came out with a clear statement, similar
to Rob's written summary, that said what it is that you would like to
see happen to address the issues that have come up over and over again.

The alternative is that sooner or later someone or some group will come
up with a better answer, there will be an ugly fight, and a lot of time
will get wasted when we could have learned how to grow up nicely.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Mon, 28 Jan 2002, Larry McVoy wrote:
>
> What you didn't do, Linus, is paint a picture which allows development
> to scale up.

Actually, I thought I did.

Basic premise: development is done by humans.

Now, look at how humans work. I don't know _anybody_ who works with
hundreds of people. You work with 5-10 people, out of a pool of maybe
30-50 people. Agreed?

Now, look at the suggested "patch penguin", and realize that the
suggestion doesn't add any scaling at all: it only adds a simple layer
that has all the same scaling problems.

What I'm saying is

 - I'm never going to work with hundreds of people directly, because it is
   fundamentally against my nature, by virtue of being human.

 - adding a "patch penguin" doesn't help, because _he_ (or she, although I
   didn't see any women nominated) is also going to be human. So either
   the patch penguin is going to do a bad job (and I won't start trusting
   him/her), or the patch penguin is going to have all the same issues
   people complain about.

Those are obvious truths. If you don't see them as being obvious truths,
you just haven't been thinking things through.

Did Alan do a good job? Yes. He did a _great_ job. But let's face it: (a)
he got really tired of doing it and (b) it really works only with one or
two Alan's, not with more - because with more you get people complaining
about the -aa tree vs the -dj tree vs the -marcelo tree vs the -linus
tree.

So Alan doesn't scale up either - I doubt you'll find a "better Alan", and
I _seriously_ doubt you'll be able to have multiple Alan's.

Does anybody really doubt this?

Now, if you've read this far, and you agree with these issues, I suspect
you know the solution as well as I do. It's the setup I already mentioned:
a network of maintainers, each of whom knows other maintainers.

And there's overlap. I'm not talking about a hierarchy here: it's not the
"general at the top" kind of tree-like setup. The network driver people
are in the same general vicinity as the people doing network protocols,
and there is obviously a lot of overlap.

Are there problems with maintainership? Yes. The main one being that it's
too damn easy to step on any toes. Which is why you want the modularity,
ie you do NOT want to have files issues that slash across different
maintenance boundaries (unless they have clearly defined interfaces:
that's where the importance of a clear VFS interface design comes in etc)

For an example of this, look at init/main.c in 2.2.x, in 2.4.x, and in
2.5.x. BIG changes. And most of the changes have very little to do with
what that file actually does (relatively little), and _everything_ to do
with the fact that it is the file that "instantiates" almost everything
and thus crosses all boundaries.

And notice how the "initcall" etc setup has changed, and cut a lot of
those dependencies. That's very much by design: look at what a device
driver had to do (and know) to be either a compiled-in driver or a modular
driver a few years ago. And look at a driver today.

In short, I'm saying that the true path to scalability is:

 - lack of dependencies on a source level

   This implies good interfaces that do NOT have common source files for
   different projects. In particular, it implies dynamic add/remove
   without the rest of the system having to know at all.

 - lack of people (whether patch-penguins or me) who have to follow
   everything.

   This, in turn, implies two things:

   (a) it is not the maintainers who pull things into their trees (because
       they aren't always there, and they don't know everything). It is
       the developers who _push_ onto maintainers.

   (b) if you as a developer cannot find a maintainer who you know, it is
       YOUR problem, and you cannot blame some super-penguin in the blue
       yonder for not caring about you. You may have to maintain your
       patch-set yourself. Or you should find a maintainer who cares about
       the work you do, and who helps feed it forward.

I know, I know. It's easier to whine about other people than it is to take
responsibility for your own actions. It's so easy to complain about "Linus
doesn't apply my patches", and so hard to just face the fact that Linus
never _will_ care about all patches, and that if you cannot find anybody
else to care about them either, then maybe they should die or you should
take care and feed them yourself.

I'm a bastard. I'm spending a lot of time applying patches, but it's not
my whole life. Never has been, never will be. I'll always be better at
applying patches from those ten people that I know and I trust than I'll
be at applying patches from people I don't trust.

If you're a developer, and you can't get through to me, ask yourself if
you can get through to somebody else.

And if you can't get through to anybody else either, ask yourself whether
maybe the problem is at _your_ end.

			Linus

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Linus Torvalds wrote:

>On Mon, 28 Jan 2002, Larry McVoy wrote:
>
>>What you didn't do, Linus, is paint a picture which allows development
>>to scale up.
>>
>
>Actually, I thought I did.
>
>Basic premise: development is done by humans.
>
>Now, look at how humans work. I don't know _anybody_ who works with
>hundreds of people. You work with 5-10 people, out of a pool of maybe
>30-50 people. Agreed?
>
Not at all. Please have a look at the ARMY. (A tightly hierarchical 
system...)

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >Now, look at how humans work. I don't know _anybody_ who works with
> >hundreds of people. You work with 5-10 people, out of a pool of maybe
> >30-50 people. Agreed?
> >
> Not at all. Please have a look at the ARMY. (A tightly hierarchical
> system...)

a general wont work with hundreds of people *directly*.

and i doubt 'giving orders' qualifies as 'works with'. 'Works with' is the
close circle around the general whom he talks to about current issues and
whose advice he listens to.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Stephan von Krawczynski]

On Tue, 29 Jan 2002 12:45:46 +0100
Martin Dalecki <dalecki@evision-ventures.com> wrote:

> Linus Torvalds wrote:
> 
> >On Mon, 28 Jan 2002, Larry McVoy wrote:
> >
> >>What you didn't do, Linus, is paint a picture which allows development
> >>to scale up.
> >>
> >
> >Actually, I thought I did.
> >
> >Basic premise: development is done by humans.
> >
> >Now, look at how humans work. I don't know _anybody_ who works with
> >hundreds of people. You work with 5-10 people, out of a pool of maybe
> >30-50 people. Agreed?
> >
> Not at all. Please have a look at the ARMY. (A tightly hierarchical 
> system...)

Shoot me: where the heck is the creative/innovative element inside the ARMY?
It just died somewhere down the hierarchy tree...
Ants are a very successful species, too, but they will hardly ever write
software (personal guess).

Regards,
Stephan

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 12:37 pm, Stephan von Krawczynski wrote:
> On Tue, 29 Jan 2002 12:45:46 +0100
>
> Martin Dalecki <dalecki@evision-ventures.com> wrote:
> > Linus Torvalds wrote:
> > >On Mon, 28 Jan 2002, Larry McVoy wrote:
> > >>What you didn't do, Linus, is paint a picture which allows development
> > >>to scale up.
> > >
> > >Actually, I thought I did.
> > >
> > >Basic premise: development is done by humans.
> > >
> > >Now, look at how humans work. I don't know _anybody_ who works with
> > >hundreds of people. You work with 5-10 people, out of a pool of maybe
> > >30-50 people. Agreed?
> >
> > Not at all. Please have a look at the ARMY. (A tightly hierarchical
> > system...)
>
> Shoot me: where the heck is the creative/innovative element inside the
> ARMY? It just died somewhere down the hierarchy tree...
> Ants are a very successful species, too, but they will hardly ever write
> software (personal guess).
>
> Regards,
> Stephan

This is only marginally on-topic (at best), but I agree the army is a very 
bad model for distributed collaborative development organization to try to 
emulation.  I have in fact written a series about this back when I wrote 
stock market investment columns (strange hobby for a programmer, I know...)

http://www.fool.com/news/foth/2000/foth000731.htm
http://www.fool.com/news/foth/2000/foth000913.htm
http://www.fool.com/news/foth/2000/foth000905.htm
http://www.fool.com/news/foth/2000/foth000918.htm
http://www.fool.com/news/foth/2000/foth000925.htm
http://www.fool.com/portfolios/rulemaker/2000/rulemaker000928.htm
http://www.fool.com/news/foth/2000/foth001002.htm

Linux development is a fan club.  And if you don't think fan clubs can 
mobilize and coordinate truly huge amounts of effort, you've obviously never 
been to worldcon.

But "the prototype and the fan club" is a seperate thing.  Maybe if 
somebody's really really bored I'll give them that talk during a BOF at LWE...

Rob

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Tue, 29 Jan 2002, Rob Landley wrote:

> Linux development is a fan club.

*plonk*

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 06:37 pm, Stephan von Krawczynski wrote:
> On Tue, 29 Jan 2002 12:45:46 +0100
> Martin Dalecki <dalecki@evision-ventures.com> wrote:
> > Linus Torvalds wrote:
> > >On Mon, 28 Jan 2002, Larry McVoy wrote:
> > >>What you didn't do, Linus, is paint a picture which allows development
> > >>to scale up.
> > >
> > >Actually, I thought I did.
> > >
> > >Basic premise: development is done by humans.
> > >
> > >Now, look at how humans work. I don't know _anybody_ who works with
> > >hundreds of people. You work with 5-10 people, out of a pool of maybe
> > >30-50 people. Agreed?
> > >
> > Not at all. Please have a look at the ARMY. (A tightly hierarchical 
> > system...)
> 
> Shoot me: where the heck is the creative/innovative element inside the ARMY?
> It just died somewhere down the hierarchy tree...
> Ants are a very successful species, too, but they will hardly ever write
> software (personal guess).

Correct, they don't write it, they evolve it.

/me ducks and runs for cover

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:
 
> Now, if you've read this far, and you agree with these issues, I suspect
> you know the solution as well as I do. It's the setup I already mentioned:
> a network of maintainers, each of whom knows other maintainers.
> 
> And there's overlap. I'm not talking about a hierarchy here: it's not the
> "general at the top" kind of tree-like setup. The network driver people
> are in the same general vicinity as the people doing network protocols,
> and there is obviously a lot of overlap.

So the kernel maintainership becomes a network of maintainers.  Then
we only have to understand the routing protocols.  Currently the
routing tables appear to have Linus as the default route.  As there
are currently kernel subsystems that do not have a real maintainer, it
may reasonable to have a misc maintainer.  Who looks after the
orphaned code, rejects/ignores patches for code that does have
active maintainers, and looks for people to be maintainers of the
orphaned code.  

The key is having enough human to human protocol that there is someone
besides Linus you can send your code to.  Or at least when there isn't
people are looking for someone.

Free Software obtains a lot of it's value by many people scratching an
itch and fixing a little bug, or adding a little feature, sending the
code off and then they go off to something else.  We need to have the
maintainer routing protocol clear enough, and the maintainer coverage
good enough so we can accumulate most of the bug fixes from the fly by
night hackers.  

So does anyone have any good ideas about how to build up routing
tables?  And almost more importantly how to make certain we have good
maintainer coverage over the entire kernel?

Eric

Re: A modest proposal -- We need a patch penguin

[Momchil Velikov]

>>>>> "Eric" == Eric W Biederman <ebiederm@xmission.com> writes:

Eric> So does anyone have any good ideas about how to build up routing
Eric> tables?

Umm, broadcasting to lkml ?:)

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 02:19 pm, Eric W. Biederman wrote:
> So the kernel maintainership becomes a network of maintainers.  Then
> we only have to understand the routing protocols.  Currently the
> routing tables appear to have Linus as the default route.  As there
> are currently kernel subsystems that do not have a real maintainer, it
> may reasonable to have a misc maintainer.  Who looks after the
> orphaned code, rejects/ignores patches for code that does have
> active maintainers, and looks for people to be maintainers of the
> orphaned code.  
> 
> The key is having enough human to human protocol that there is someone
> besides Linus you can send your code to.  Or at least when there isn't
> people are looking for someone.
> 
> Free Software obtains a lot of it's value by many people scratching an
> itch and fixing a little bug, or adding a little feature, sending the
> code off and then they go off to something else.  We need to have the
> maintainer routing protocol clear enough, and the maintainer coverage
> good enough so we can accumulate most of the bug fixes from the fly by
> night hackers.  
> 
> So does anyone have any good ideas about how to build up routing
> tables?  And almost more importantly how to make certain we have good
> maintainer coverage over the entire kernel?

Yes, we should cc our patches to a patchbot:

  patches-2.5@kernel.org -> goes to linus
  patches-2.4@kernel.org -> goes to marcello
  patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
  etc.

The vast sea of eyeballs will do the rest.  A web interface would be a nice 
bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is the 
essential ingredient.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 06:51 pm, Daniel Phillips wrote:
> On January 29, 2002 02:19 pm, Eric W. Biederman wrote:
> > So the kernel maintainership becomes a network of maintainers.  Then
> > we only have to understand the routing protocols.  Currently the
> > routing tables appear to have Linus as the default route.  As there
> > are currently kernel subsystems that do not have a real maintainer, it
> > may reasonable to have a misc maintainer.  Who looks after the
> > orphaned code, rejects/ignores patches for code that does have
> > active maintainers, and looks for people to be maintainers of the
> > orphaned code.
> >
> > The key is having enough human to human protocol that there is someone
> > besides Linus you can send your code to.  Or at least when there isn't
> > people are looking for someone.
> >
> > Free Software obtains a lot of it's value by many people scratching an
> > itch and fixing a little bug, or adding a little feature, sending the
> > code off and then they go off to something else.  We need to have the
> > maintainer routing protocol clear enough, and the maintainer coverage
> > good enough so we can accumulate most of the bug fixes from the fly by
> > night hackers.
> >
> > So does anyone have any good ideas about how to build up routing
> > tables?  And almost more importantly how to make certain we have good
> > maintainer coverage over the entire kernel?
>
> Yes, we should cc our patches to a patchbot:
>
>   patches-2.5@kernel.org -> goes to linus
>   patches-2.4@kernel.org -> goes to marcello
>   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
>   etc.
>
> The vast sea of eyeballs will do the rest.  A web interface would be a nice
> bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is
> the essential ingredient.

And of course there's not much point in having patches go to that list that 
AREN'T public (um, they're for inclusion into a public tree, right)?  So the 
patchbot might as well distribute to a mailing list, as long as there's some 
variety of moderation (possibly just a procmail recipe) to delete everything 
that didn't actually have a patch in it.  (Yet another discussion list is 
unlikely to help matters too much.)

Of course this still doesn't address the problem of patches going stale if 
they're not applied for a version or two and something else that goes in 
breaks them...

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 08:33:03PM -0500, Rob Landley wrote:
> > Yes, we should cc our patches to a patchbot:
> >
> >   patches-2.5@kernel.org -> goes to linus
> >   patches-2.4@kernel.org -> goes to marcello
> >   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
> >   etc.
> >
> > The vast sea of eyeballs will do the rest.  A web interface would be a nice
> > bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is
> > the essential ingredient.
> 
> And of course there's not much point in having patches go to that list that 
> AREN'T public

If mail sent to the above addresses is not public somehow, the idea
is a non-starter.


> Of course this still doesn't address the problem of patches going stale if 
> they're not applied for a version or two and something else that goes in 
> breaks them...

If you really want to be a patch penguin then.... just do it.

You don't need specific permission to pick up, update, and maintain
patches that don't make it into the Linus tree on the first try.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:46 pm, Jeff Garzik wrote:
> On Tue, Jan 29, 2002 at 08:33:03PM -0500, Rob Landley wrote:

> > Of course this still doesn't address the problem of patches going stale
> > if they're not applied for a version or two and something else that goes
> > in breaks them...
>
> If you really want to be a patch penguin then.... just do it.
>
> You don't need specific permission to pick up, update, and maintain
> patches that don't make it into the Linus tree on the first try.

I'm not asking to become a patch penguin, and the various other people who 
volunteered early on, though well intentioned, slightly missed my point as 
well.

We used to HAVE a patch penguin.  "Miscelaneous maintainer", "integration 
lieutenant", call the position what you will.  His name was Alan Cox.  He 
recently abdicated the position, which has since gradually been assumed by 
Dave Jones.  There is serious pressure on Dave Jones's tree to accept the 
kind of patches Alan used to (and which Alan is still accepting for 2.4 and 
queuing for Marcelo).  If Dave continues to put out a tree, he would have to 
work fairly hard to avoid becoming Alan Cox's successor.

I was hoping for some sort of indication that if patches DID get into Dave's 
tree, it would be a step towards their eventual consideration by Linus.  Not 
a guarantee of inclusion, of course, but it would be nice to know if 
inclusion in Dave's tree would move the patch one step towards Linus, or 
would just head down a cul-de-sac and additional fragmentation of the 
development process.

I was also trying to point out that there seems to be a recurring role here, 
which used to be identified with "just Alan" but has now passed to another 
person, while still maintaining a noticeable portion of its character.  It 
might be nice to recognize it as such.

Also, I was trying to encourage ONE beta tree in order to DISCOURAGE the 
fragmented proliferation of version-skewed trees accepting third party 
patches that seem to have been cropping up recently.  (See the linux weekly 
news and kernel traffic links in the original posting that started this whole 
thread.)  In the absence of an -ac tree to accept patches that show 
significant promise but are not ready for Linus and vice versa, patches are 
accumulating in multiple trees.  This fragments the tester base, and seemed 
to me to be a less efficient way than the way things worked before Alan quit.

There seems to have been widespread misinterpretation of these objectives...

> 	Jeff

Rob

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> Yes, we should cc our patches to a patchbot:
>
>   patches-2.5@kernel.org -> goes to linus
>   patches-2.4@kernel.org -> goes to marcello
>   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
>   etc.

I'd rather make the patchbot more intelligent, that means it analyzes the
patch and produces a list of touched files. People can now register to get
notified about patches, which changes areas they are interested in.
In the simplest configuration nothing would change for Linus, but patches
wouldn't get lost and people could be notified if their patch was applied
or if it doesn't apply anymore. Other people have a place to search for
patches and they can check whether something was already fixed.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 11:39 am, Roman Zippel wrote:
> Hi,
> 
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> 
> > Yes, we should cc our patches to a patchbot:
> >
> >   patches-2.5@kernel.org -> goes to linus
> >   patches-2.4@kernel.org -> goes to marcello
> >   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
> >   etc.
> 
> I'd rather make the patchbot more intelligent, that means it analyzes the
> patch and produces a list of touched files. People can now register to get
> notified about patches, which changes areas they are interested in.

But they can already do that, by subscribing to the respective mailing list 
(obviously, the bot posts to the list as well as forwarding to the 
maintainer) and running the mails through a filter of their choice.

> In the simplest configuration nothing would change for Linus, but patches
> wouldn't get lost and people could be notified if their patch was applied
> or if it doesn't apply anymore.

OK, it would be nice, but you wouldn't want to pile on so many features that 
this never gets implemented would you?  The minimal thing that forwards and 
posts patches is what we need now.  Like any other software it can be 
improved over time.

> Other people have a place to search for patches and they can check whether 
> something was already fixed.

Automating the applied/dropped status is clearly the next problem to tackle, 
but that's harder, it involves behavioral changes on the maintainers side.  
(Pragmatically, providing a web interface so somebody whose job it is to do 
that, can efficiently post 'applied' messages to the list would get the job 
done without making anyone learn new tools or change the way they work.)

By the way, who is going to code this?  Or are we determined to make 
ourselves look like wankers once again, by putting considerably more time 
into the lkml flamewar than goes into producing working code?

(Hint: I am not going to code it, nor should I since I should be working in 
the kernel.)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> > I'd rather make the patchbot more intelligent, that means it analyzes the
> > patch and produces a list of touched files. People can now register to get
> > notified about patches, which changes areas they are interested in.
>
> But they can already do that, by subscribing to the respective mailing list
> (obviously, the bot posts to the list as well as forwarding to the
> maintainer) and running the mails through a filter of their choice.

What about unmaintained parts?

> > In the simplest configuration nothing would change for Linus, but patches
> > wouldn't get lost and people could be notified if their patch was applied
> > or if it doesn't apply anymore.
>
> OK, it would be nice, but you wouldn't want to pile on so many features that
> this never gets implemented would you?  The minimal thing that forwards and
> posts patches is what we need now.  Like any other software it can be
> improved over time.

That's what I have in mind. What we can already do now is to store
incoming patches into some directory. That would give us already some
basic data to work with and we can start to implement new features as they
are needed.

> Automating the applied/dropped status is clearly the next problem to tackle,
> but that's harder, it involves behavioral changes on the maintainers side.

What "behavioral changes"? Maintainers should react in some way or another
react to patches no matter where come from.

> (Pragmatically, providing a web interface so somebody whose job it is to do
> that, can efficiently post 'applied' messages to the list would get the job
> done without making anyone learn new tools or change the way they work.)

Web interfaces can be nice, but the bulk work should be doable by mail.
For changes in areas which have a maintainer, the mail to Linus could
include a note "forwarded to maintainer x" and Linus can still decide,
whether he applies the patch or waits for the OK from the maintainer.

> By the way, who is going to code this?  Or are we determined to make
> ourselves look like wankers once again, by putting considerably more time
> into the lkml flamewar than goes into producing working code?
>
> (Hint: I am not going to code it, nor should I since I should be working in
> the kernel.)

That's a known problem, I have no time either, but we should give anyone
interested in this some example data. This data has to come from the
kernel hackers, but patch management system is better implemented by
non-kernel hackers.

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 01:00 am, Linus Torvalds wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:
> > > A word of warning: good maintainers are hard to find.  Getting more of
> > > them helps, but at some point it can actually be more useful to help
> > > the _existing_ ones.  I've got about ten-twenty people I really trust,
> > > and quite frankly, the way people work is hardcoded in our DNA.  Nobody
> > > "really trusts" hundreds of people.  The way to make these things scale
> > > out more is to increase the network of trust not by trying to push it
> > > on me, but by making it more of a _network_, not a star-topology around
> > > me.
> >
> > You don't see an integration maintainer as a step in the right direction?
> > (It's not a star topology, it's a tree.)
>
> No, I don't really think an "integration manager" works well.

So what was Alan Cox doing all those years?  What is Dave Jones currently 
doing?

> I think it helps a lot to have people pick up patches that nobody else
> wants to maintain, and to gather them up. Andrea does that to some degree.

It's not a question of patches people don't want to maintain, it's a question 
of patches getting much wider testing and better feedback when they're in a 
larger tree, and the maintainers of the various patches getting better 
warning about other patches that break them or that they break other patches.

When two developers share a common tree, they notice when they break each 
other's stuff, and they resolve it.  When two developers go off in isolation, 
they break each other's stuff as a matter of course.  And testers who have to 
hunt down a patch are are willing to apply it generally aren't the ones who 
raise an objection once it gets applied to the next tree they download.

> But it is _much_ better if you have somebody who is a point-man for
> specific areas.

I'm not proposing replacing the current subsystem maintainers.  But are the 
current subsystem maintainers happy?

I thought they weren't, but I guess that by their silence, they must be 
thrilled, so...  (Sorry, I seem to be getting a lot more support in private 
than anybody is willing to cc: to the list.  I'm new at this politics 
business...)

> The problem with an overall guy is that there can't be too many of them.
> The very thing you are _complaining_ about is in fact that there are a
> number of over-all guys without clear focus, which only leads to confusion
> about who handles what.
>
> Clarity is good.

The fact Jens Axboe handles one system, Stephen C. Tweedie another, Andre 
Hedrick a third, Rik van Riel a fourth, and Eric Raymond a fifth, is not 
particularly confusing.  It's when the integration and debugging of Jens' 
patches in 2.5 blocks the inclusion of basically anything else for a month or 
two, and then Andre Hedrick has to mount a publicity campaign on linux-kernel 
to get any attention paid to his patches, and Eric's help patches get ignored 
for 33 consecutive releases.

Rik was replaced by Andrea as the VM maintainer, and Rik has publicly stated 
that he thinks you were dropping his VM patches for months at a time, while 
he was the maintainer and the VM was a subsystem definitely in need of 
patches.  Are you saying that the system was working well?  Are you saying 
that it was a one-time thing that is now resolved and won't recur?

Okay, maybe a lot of this is all miscommunication.  But that just identifies 
the TYPE of the problem, doesn't it?

> > Are you saying that Alan Cox's didn't serve a purpose during the 2.2
> > kernel time frame, and that Dave Jones is currently wasting his time?
>
> No, I'm saying that there are not very many peopel who can do it, and who
> can get the kind of trust that they are _everywhere_. Let's face it, Alan
> grew to be respected because he did lots of different stuff over many
> years, and he proved himself more than capable. And I suspect he's _quite_
> happy not being in the middle of it right now.. It's a tough job.

It is a tough job, and I understand that not everybody can be a good 
maintainer.  But currently at least Alan, Dave Jones, and Andrea are all 
maintaining their own public trees, from which they break out patches to send 
on to an "official" Linux tree.  (As for Alan not being "in the middle of 
it", he IS doing his tree again.  He's just doing it for 2.4.  He's basically 
being Marcelo's integration lieutenant.  Whatever he's burned out on, it's 
apparently not the job of maintaining a tree.  And he's doing it for Marcelo, 
whose architect role is largely rejecting as much as he possibly can since 
2.4 is not a development branch...)

You currently HAVE a de facto integration lieutenant, or else I totally 
misunderstand what Dave Jones is doing.  This is not a position for which 
applicants currently need to be interviewed, is it?  (Do you have a complaint 
with the job Dave is doing?)

> It's a lot more likely to find people who can maintain _parts_. And if
> there are patches that fall out of those parts, that tends to indicate a
> lack of modularity, and perhaps a lack of maintainer for those parts.

Sure.  But how do the maintainers piece together their code, resolve the 
obvious conflicts, and get the new stuff tested by live users in the field 
who want to live dangerously?  They USED to feed stuff into the -ac tree, 
months if not YEARS before you accepted (or rejected) it.  That's not my 
opinion or my recommendation, that's history.  I'm simply proposing that 
people consider the fact it might be an important and natural part of the 
process.  (When Alan stopped doing it, somebody else basically got shanghaied 
into doing it.)

> And more likely, even if you _do_ find ten people who can do everything,
> you don't want them to.

No, you want one guy with final responsibility for maintaining any tree.  
Committees produce mostly compromises and deadlocks.  That's why I proposed 
one guy for this job.  As I said, the CVS thing was a confusing side issue.  
(An easier way for the maintainers to do lower-friction merges with the 
integration maintainer, who would by the CVS administrator and would still 
have final say over what goes into his tree.)

But the -ac tree did not serve the same purpose as your tree did, and I was 
under the strong impression that the -ac tree DID serve a purpose.  (And, for 
Marcelo, is starting to do so again.)

There is currently no tree for provisionally integrating code.  Or for taking 
the flood of new driver patches that Alan Cox always fielded.  Not code from 
left field, but code like keith owens' new kbuild, CML2, or rik van riel's 
reverse mapping patches.  Things which have a strong possiblity of being 
integrated (two of the above you okayed at the kernel summit, one you've 
expressed interest in), and are ready for wider testing.

> 		Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Tue, Jan 29, 2002 at 02:33:24AM -0500, Rob Landley wrote:
> 
> I'm not proposing replacing the current subsystem maintainers.  But are the 
> current subsystem maintainers happy?

I'll speak up here as a subsystem maintainer and say that I'm happy with
the current situation.  I integrate a wide variety of USB driver patches
from lots of different people (and usually in lots of different formats
against different kernel trees) and feed them to Linus/Marcelo/Alan in
small chunks that can be easily applied against their latest kernel
version. 

Sure, sometimes my patches get dropped, but you forgot to mention the
most important thing a kernel programmer needs to have, persistence :)

thanks,

greg k-h

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 02:33:24AM -0500, Rob Landley wrote:
> I'm not proposing replacing the current subsystem maintainers.  But are the 
> current subsystem maintainers happy?

I think the system works, if you understand the system ;-)

Finding a maintainer for a piece of code is sometimes a jumble, but if
people want their patches in the kernel they need to be proactive about
it.

I'm -glad- Linus does not usually apply drop-n-run patches.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 10:00 PM 28/01/02 -0800, Linus Torvalds wrote:
>I think it helps a lot to have people pick up patches that nobody else
>wants to maintain, and to gather them up. Andrea does that to some degree.
>But it is _much_ better if you have somebody who is a point-man for
>specific areas.
>
>The problem with an overall guy is that there can't be too many of them.
>The very thing you are _complaining_ about is in fact that there are a
>number of over-all guys without clear focus, which only leads to confusion
>about who handles what.
>
>Clarity is good.

Perhaps what we need is a patch maintenance system? Now I'm not talking 
CVS, I'm talking about something that is, in reality, pretty simple. 
Something that does the following:

  1. People can submit patches, which are given a unique patch ID.
  2. Notifications of patches are passed on to (from a selection or 
automatic detection):
    a. A module maintainer
    b. A section maintainer
    c. A tree maintainer
    d. Linus Torvalds
  3. The patches can be reviewed, and immediately:
    a. Dropped with a canned reject message
    b. Dropped with a custom reject message
    c. Dropped but archived for later review
    d. Suspended/Skipped for later review
    e. Redelegated up/down to a maintainer
    f. Accepted, pending testing
  4. If someone wants to know why their patch is not being accepted:
    a. They can easily look up the current status
    b. There is a common reference place
    c. If their patch is rejected, they can ask for more detail

Yes it's a tiny patch tracking system. The idea is that if we can make the 
thing simple to use, simple to understand, and simple to maintain, people 
will actually use it. Submitting patches by mail in freeform, while 
reasonably simple, leads to overall complication, and may take longer to 
sort through.

It doesn't have to be that involved. A shell script (or perl) could happily 
do the submission job (which could all still be done by mail, however in a 
more formatted approach). It'd also be simple to do some sort of web 
interface if anyone was actually inclined.

Later, maybe, we can have it track possible conflicts (between waiting 
patches) and flag them as such so the maintainer is aware of the fact.

If we can automate some simple, key parts of the kernel maintenance, then 
this could help immensely, and let everyone get on with the job than the 
hassle. Maybe we want to re-invent this wheel, maybe we don't.

PS: Remember this is only a suggestion. I'm not infallible (is anyone?), 
all I can do is give the big guy ideas (even bad ones), so he can make 
better decisions. *grin*


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[Nix N. Nix]

On Tue, 2002-01-29 at 02:10, Stuart Young wrote:
> At 10:00 PM 28/01/02 -0800, Linus Torvalds wrote:
> >I think it helps a lot to have people pick up patches that nobody else
> >wants to maintain, and to gather them up. Andrea does that to some degree.
> >But it is _much_ better if you have somebody who is a point-man for
> >specific areas.
> >
> >The problem with an overall guy is that there can't be too many of them.
> >The very thing you are _complaining_ about is in fact that there are a
> >number of over-all guys without clear focus, which only leads to confusion
> >about who handles what.
> >
> >Clarity is good.
> 
> Perhaps what we need is a patch maintenance system? Now I'm not talking 
> CVS, I'm talking about something that is, in reality, pretty simple. 
> Something that does the following:
> 
>   1. People can submit patches, which are given a unique patch ID.
>   2. Notifications of patches are passed on to (from a selection or 
> automatic detection):
>     a. A module maintainer
>     b. A section maintainer
>     c. A tree maintainer
>     d. Linus Torvalds
>   3. The patches can be reviewed, and immediately:
>     a. Dropped with a canned reject message
>     b. Dropped with a custom reject message
>     c. Dropped but archived for later review
>     d. Suspended/Skipped for later review
>     e. Redelegated up/down to a maintainer
>     f. Accepted, pending testing
>   4. If someone wants to know why their patch is not being accepted:
>     a. They can easily look up the current status
>     b. There is a common reference place
>     c. If their patch is rejected, they can ask for more detail

Kinda like bugzilla, but for patches ? Am I off the deep end here ?


Just a thought.
> 
> Yes it's a tiny patch tracking system. The idea is that if we can make the 
> thing simple to use, simple to understand, and simple to maintain, people 
> will actually use it. Submitting patches by mail in freeform, while 
> reasonably simple, leads to overall complication, and may take longer to 
> sort through.
> 
> It doesn't have to be that involved. A shell script (or perl) could happily 
> do the submission job (which could all still be done by mail, however in a 
> more formatted approach). It'd also be simple to do some sort of web 
> interface if anyone was actually inclined.
> 
> Later, maybe, we can have it track possible conflicts (between waiting 
> patches) and flag them as such so the maintainer is aware of the fact.
> 
> If we can automate some simple, key parts of the kernel maintenance, then 
> this could help immensely, and let everyone get on with the job than the 
> hassle. Maybe we want to re-invent this wheel, maybe we don't.
> 
> PS: Remember this is only a suggestion. I'm not infallible (is anyone?), 
> all I can do is give the big guy ideas (even bad ones), so he can make 
> better decisions. *grin*
> 
> 
> Stuart Young - sgy@amc.com.au
> (aka Cefiar) - cefiar1@optushome.com.au
> 
> [All opinions expressed in the above message are my]
> [own and not necessarily the views of my employer..]
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: A modest proposal -- We need a patch penguin

[Patrick Mochel]

On Tue, 29 Jan 2002, Stuart Young wrote:
> Perhaps what we need is a patch maintenance system? Now I'm not talking 
> CVS, I'm talking about something that is, in reality, pretty simple. 
> Something that does the following:

Ah, the old Patch Management Problem. It's like an old friend (or a bad 
rash). 

AFAIK, something like this was first proposed here:

http://lists.insecure.org/linux-kernel/2000/Sep/2468.html

At that time, we were in the midst of the 2.4.0-test? series. Many things 
were unstable and/or volatile. Linus was receiving an ungodly number of 
patches, and releasing a new -pre patch about every day. 

One of the main problems was that many patches simply didn't apply. What 
a patch was diffed against would become obsolete so quickly, that many 
patches were rendered useless by the time they were even read. And, there 
was the same limitation concerning Linus's ability and desire to reply to 
every single email. 

The concept is very simple and described well in the email. So, I will not 
expound on it here. Unfortunately, the project was dropped internally. 

The problem has come up a few times in the last few months. Several people 
have expressed interest in having something like it. Some already do. 
Several people have said they were working on something like it. 
Unfortunately, I think most of those people got distracted with their 
other full-time jobs or more intersting work.

I brought the topic up here at OSDL a few months ago for use both 
internally and externally. Also, with the notion of integrating our STP 
(Scalable Test Platform). We've had several discussions about it, what it 
would look like, and how it would work. We've also had the chance to talk 
with a few of the other kernel maintainers in the area (face-to-face 
meetings really do a long way).

The conclusions were this:

Is it necessary?	No.
Could it be useful?	Yes.
Would people use it?	Probably. 
Would everyone use it?	No.
Would Linus use it?	Probably Not. 


Which is all pretty obvious. You can't please everyone.

We're going to develop a system internally and are willing to host a 
system for the rest of the world to use. We're not looking to design a 
be-all, end-all solution. Basically, just a system that can automate 
things like applying patches and compiling. 

If a patch succeeds, it then goes to the maintainer to which it was sent. 
The maintainer can then accept or reject the patch. Either with 
explanation or not. The submitter can then track what patches were 
accepted. 

Submitted patches can also go to a public mailing list and/or exported via 
a public (read: web) interface. Of course, there should be ways to 
override the publicity for OOB patches and sensitive items. 

Writing the software is really not that difficult. But, we want something 
that people like and can use, as well as modular and extensible. So, we're 
aiming for simplicity and modularity. 

So, the obvious question is 'So, where is it at now?'. Not much further 
than the conceptualizing stage. The two people actually writing the 
software are a bit over-subscribed ATM, though we do have some pretty 
pictures. 

I'm currently in the waiting queue for a Sourceforge project. Once that is 
live, there will be a mailing list to which the discussion can be moved 
and kept alive. Anyone and everyone interested is welcome to submit their 
ideas and suggestions. Via the SF project, we will submit our designs and 
post our progress on the system. 

If you prefer a more private forum, feel free to email me and/or the Man 
with the Plan: Nathan Dabney at smurf@osdl.org.

	-pat

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

Apropos of nothing in particular:

> (It's not a star topology, it's a tree.)

There is no difference between a star and a tree, except how you draw the 
picture.

-- 
Daniel

RE: A modest proposal -- We need a patch penguin

[George Bonser]

I dunno. I tend to agree with Linus from a management standpoint.
You delegate responsibility to those you trust and accept their
judgment. You also must try to ensure that the thing is built so
that decisions made by Bill can't step on Bob's work.  Delegation
is one thing ... but it is the architecture that makes sure that
delegation is possible. I am not a kernel hacker but don't feel
that I have to be in this case to comment. What Linus is saying
has nothing at all to do with Linux but is more along the lines
of how to get work done in any kind of project that is larger than
any single person.

While all kinds of software might get written for project management
and revision control, projects boil down to how the thing is built,
the style of the person in charge, and the people around that person.

Finding people that are willing to take responsibility for portions
of the code base and develop a relationship of trust over time while
being able to work with the individuals submitting patches for their
bug fixes/ideas is a big task.  If you could gather all of
those people together, I will bet you they will be a successful team
in just about ANYTHING they choose to work on. I think what Linus is
striving toward is what any good manager wants and when such a team
comes together, watch out, things are gonna happen fast.

Don't think of it as a star or a tree, think of it as a star of trees.



> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of
> Daniel Phillips
> Sent: Monday, January 28, 2002 11:39 PM
> To: Rob Landley; linux-kernel@vger.kernel.org
> Subject: Re: A modest proposal -- We need a patch penguin
>
>
> Apropos of nothing in particular:
>
> > (It's not a star topology, it's a tree.)
>
> There is no difference between a star and a tree, except
> how you draw the
> picture.
>
> --
> Daniel
> -
> To unsubscribe from this list: send the line "unsubscribe
> Linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: A modest proposal -- We need a patch penguin

[Xavier Bestel]

le mar 29-01-2002 à 08:38, Daniel Phillips a écrit :
> Apropos of nothing in particular:
> 
> > (It's not a star topology, it's a tree.)
> 
> There is no difference between a star and a tree, except how you draw the 
> picture.

I think a star is a tree only 1 unit deep. No subchildren. (In this
case)

	Xav

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Mon, 28 Jan 2002, Rob Landley wrote:

> (You keep complaining people never send you patches.  People are
> suggesting automated patch remailers to spam your mailbox even harder.
> There has GOT to be a better way...)

None of the examples you cited so far are convincing to me, and i'd like
to explain why. I've created and submitted thousands of patches to the
Linux kernel over the past 4 years (my patch archive doesnt go back more
than 4 years):

  # ls patches | wc -l
     2818

a fair percentage of those went to Linus as well, and while having seen
some of them rejected does hurt mentally, i couldnt list one reject from
Linus that i wouldnt reject *today*. But i sure remember being frustrated
about rejects when they happened. In any case, i have some experience in
submitting patches and i'm maintaining a few subsystems, so here's my take
on the 'patch penguin' issue:

If a patch gets ignored 33 times in a row then perhaps the person doing
the patch should first think really hard about the following 4 issues:

  - cleanliness
  - concept
  - timing
  - testing

a violation of any of these items can cause patch to be dropped *without
notice*. Face it, it's not Linus' task to teach people how to code or how
to write correct patches. Sure, he still does teach people most of the
time, but you cannot *expect* him to be able to do it 100% of the time.

1) cleanliness

code cleanliness is a well-know issue, see Documentation/CodingStyle.  If
a patch has such problems then maintainers are very likely to help - Linus
probably wont and shouldnt. I'm truly shocked sometimes, how many active
and experienced kernel developers do not follow these guidelines. While
the Linux coding style might be arbitrary in places, all coding styles are
arbitrary in some areas, and only one thing is important above all:
consistency between kernel subsystems. If i go from one kernel subsystem
to another then i'd like to have the same 'look and feel' of source code -
i think this is a natural desire we all share. If anyone doesnt see the
importance of this issue then i'd say he hasnt seen, hacked and maintained
enough kernel code yet. I'd say the absolute positive example here is Al
Viro. I think most people just do not realize the huge amount of
background cleanup work Al did in the past 2 years. And guess what? I bet
Linus would be willing to apply Al's next patch blindfolded.

impact: a patch penguin might help here - but he probably wont scale as
well as the current set of experienced kernel hackers scale, many of whom
are happy to review patches for code cleanliness (and other) issues.

2) concept

many of the patches which were rejected for a long time are *difficult*
issues. And one thing many patch submitters miss: even if the concept of
the patch is correct, you first have to start by cleaning up *old* code,
see issue 1). Your patch is not worth a dime if you leave in old cruft, or
if the combination of old cruft and your new code is confusing. Also, make
sure the patch is discussed and developed openly, not on some obscure
list. linux-kernel@vger.kernel.org will do most of the time. I do not want
to name specific patches that violate this point (doing that in public
just offends people needlessly - and i could just as well list some of my
older patches), but i could list 5 popular patches immediately.

impact: a patch penguin just wont solve this concept issue, because, by
definition, he doesnt deal with design issues. And most of the big patch
rejections happen due to exactly these concept issues.

3) timing

kernel source code just cannot go through arbitrary transitions. Eg. right
now the scheduler is being cleaned up (so far it was more than 50
sub-patches and they are still coming) - and work is going on to maximize
the quality of the preemption patch, but until the base scheduler has
stabilized there is just no point in applying the preemption patch - no
matter how good the preemption patch is. Robert understands this very
much. Many other people do not.

impact: a patch penguin just wont solve this issue, because a patch
penguin cannot let his tree transition arbitrarily either. Only separately
maintained and tested patches/trees can handle this issue.

4) testing

there are code areas and methods which need more rigorous testing and
third-party feedback - no matter how good the patch. Most notably, if a
patch exports some new user-space visible interface, then this item
applies. An example is the aio patch, which had all 3 items right but was
rejected due to this item. [things are improving very well on the aio
front so i think this will change in the near future.]

impact: a patch penguin just wont solve this issue, because his job, by
definition, is not to keep patches around indefinitely, but to filter them
to Linus. Only separately maintained patches/trees help here. More people
are willing to maintain separate trees is good (-dj, -ac, -aa, etc.), one
tree can do a nontrivial transition at a time, and by having more of them
we can eg. get one of them testing aio, the other one testing some other
big change. A single patch penguin will be able to do only one nontrivial
transition - and it's not his task to do nontrivial transitions to begin
with.


Many people who dont actually maintain any Linux code are quoting Rik's
complains as an example. I'll now have to go on record disagreeing with
Rik humbly, i believe he has done a number of patch-management mistakes
during his earlier VM development, and i strongly believe the reason why
Linus ignored some of his patches were due to these issues. Rik's flames
against Linus are understandable but are just that: flames. Fortunately
Rik has learned meanwhile (we all do) and his rmap patches are IMHO
top-notch. Joining the Andrea improvements and Rik's tree could provide a
truly fantastic VM. [i'm not going to say anything about the IDE patches
situation because while i believe Rik understands public criticism, i
failed to have an impact on Andre before :-) ]

also, many people just start off with a single big patch. That just doesnt
work and you'll likely violate one of the 4 items without even noticing
it. Start small, because for small patches people will have the few
minutes needed to teach you. The bigger a patch, the harder it is to
review it, and the less likely it happens. Also, if a few or your patches
have gone into the Linux tree that does not mean you are senior kernel
hacker and can start off writing the one big, multi-megabyte super-feature
you dreamt about for years. Start small and increase the complexity of
your patches slowly - and perhaps later on you'll notice that that
super-feature isnt all that super anymore. People also underestimate the
kind of complexity explosion that occurs if a large patch is created.
Instead of 1-2 places, you can create 100-200 problems.

face it, most of the patches rejected by Linus are not due to overload. He
doesnt guarantee to say why he rejects patches - *and he must not*. Just
knowing that your patch got rejected and thinking it all over again often
helps finding problems that Linus missed first time around. If you submit
to Linus then you better know exactly what you do.

if you are uncertain about why a patch got rejected, then shake off your
frustration and ask *others*. Many kernel developers, including myself,
are happy to help reviewing patches. But people do have egos, and it
happens very rarely that people ask it on public lists why their patches
got rejected, because people do not like talking about failures. And the
human nature makes it much easier to attack than to talk about failures.
Which fact alone pretty much shows that most of the time the problem is
with the patch submitter, not with Linus.

it's so much easier to blame Linus, or maintainers. It's so much easier to
fire off an email flaming Linus and getting off the steam than to actually
accept the possibility of mistake and *fix* the patch. I'll go on record
saying that good patches are not ignored, even these days when the number
of active kernel hackers has multipled. People might have to go through
several layers first, and finding some kernel hacker who is not as loaded
as Linus to review your patch might be necessery as well (especially if
the patch is complex), but if you go through the right layers then you can
be sure that nothing worthwile gets rejected arbitrarily.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 02:54 pm, Ingo Molnar wrote:
> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:
> 
>   - cleanliness
>   - concept
>   - timing
>   - testing
> 
> a violation of any of these items can cause patch to be dropped *without
> notice*. Face it, it's not Linus' task to teach people how to code or how
> to write correct patches. Sure, he still does teach people most of the
> time, but you cannot *expect* him to be able to do it 100% of the time.

While I agree in general with most of your remarks, I think you're being a 
little too glib here.  Consider my patch to fix group descriptor corruption 
in Ext2, submitted half a dozen times to Linus and other maintainers over the 
course of two years, which was clearly explained, passed scrutiny on 
ext2-devel and lkml, fixed a real problem that really bit people and which 
I'd been running myself over the entire period.  Which one of cleanliness, 
concept, timing or testing did I violate?

If the answer is 'none of the above', then what is wrong with this picture?

--
Daniel

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> [...] Consider my patch to fix group descriptor corruption in Ext2,
> submitted half a dozen times to Linus and other maintainers over the
> course of two years, which was clearly explained, passed scrutiny on
> ext2-devel and lkml, fixed a real problem that really bit people and
> which I'd been running myself over the entire period.  Which one of
> cleanliness, concept, timing or testing did I violate?
>
> If the answer is 'none of the above', then what is wrong with this
> picture?

am i correct that you are referring to this patch?:

   http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html

was this the first iteration of your patch? Your mail is a little more
than 1 year old. You rated the patch as: 'The fix below is kind of
gross.'. Clearly, this does not help getting patches applied.

the ext2 bh-handling code had cleanliness issues before. I had ext2
patches rejected by Linus because they kept the method of passing around
double-pointers, and i have to agree that the code was far from clean. Al
did lots of cleanups in this area, and i think he fixed this issue as
well, didnt he? So where is the problem exactly, does 2.4 still have this
bug?

in terms of 2.2 and 2.0, you should contact the respective maintainers.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Ville Herva]

On Tue, Jan 29, 2002 at 03:52:20PM +0100, you [Ingo Molnar] claimed:
> 
>    http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html

(...)
 
> does 2.4 still have this bug?

My understanding is that Al did fix it.

> in terms of 2.2 and 2.0, you should contact the respective maintainers.

It has been submitted now. David Weinehall merged it in 2.0.40-rc1 and I
understand that it's in Alan's 2.2.21pre queue as well.


-- v --

v@iki.fi

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 03:52 pm, Ingo Molnar wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> 
> > [...] Consider my patch to fix group descriptor corruption in Ext2,
> > submitted half a dozen times to Linus and other maintainers over the
> > course of two years, which was clearly explained, passed scrutiny on
> > ext2-devel and lkml, fixed a real problem that really bit people and
> > which I'd been running myself over the entire period.  Which one of
> > cleanliness, concept, timing or testing did I violate?
> >
> > If the answer is 'none of the above', then what is wrong with this
> > picture?
> 
> am i correct that you are referring to this patch?:
> 
>    http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html
> 
> was this the first iteration of your patch? Your mail is a little more
> than 1 year old.

No, there are versions before that.  The first version, which really was 
inadequate because I didn't know about diff -u at the time (my first patch) 
is about 23 months old.

> You rated the patch as: 'The fix below is kind of
> gross.'. Clearly, this does not help getting patches applied.

Note who the email is addressed to.  I have tried many different techniques 
for communicating with this gentleman, including self-deprecation, and they 
all seem to have the same result: no patch applied, long wait, eventually 
some other patch a long time later will obsolete my patch in some way, and 
the whole thing drifts off into forgotten history.  Err, almost forgotten, 
because the bad taste remains.

And yes, there was a successor to the patch in which I did the job 'properly' 
by cleaning up some other infrastructure instead of just fixing the bug 
locally.  There was also a long lag after I created and submitted that 
version before the bug was actually fixed, and then it was only fixed in 2.4.

All of this only 'since you asked'.  I'd prefer not to dwell on it further, 
but as you could imagine, this story would not have developed this way if we 
have even a minimal form of patch tracking.  At least the bugs would have 
been fixed in all trees, nearly two years earlier.

> the ext2 bh-handling code had cleanliness issues before. I had ext2
> patches rejected by Linus because they kept the method of passing around
> double-pointers, and i have to agree that the code was far from clean.

Exactly.  The successor patch to the 'kind of gross' patch got rid of the 
double-pointers, it was the proper fix, though there is still no excuse for 
leaving the bug hanging around while coming up with the better version.

> Al did lots of cleanups in this area, and i think he fixed this issue as
> well, didnt he? So where is the problem exactly, does 2.4 still have this
> bug?

Oh yes, there are a few problems with what happened:

  - It left the bug circulating out in the wild far longer than
    necessary, and it bit people, pissing them off, especially when
    they figured out there was a patch not applied.

  - While it got fixed in the 2.4 tree, it didn't get fixed in 2.2 or
    for all I know, 2.0.

  - It pissed me off.

> in terms of 2.2 and 2.0, you should contact the respective maintainers.

This was taken care of by a good samaritan:

   http://marc.theaimsgroup.com/?l=linux-kernel&m=100989249313641&w=2

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Andrew Morton]

Daniel Phillips wrote:
> 
> Note who the email is addressed to.  I have tried many different techniques
> for communicating with this gentleman, including self-deprecation, and they
> all seem to have the same result: no patch applied, long wait, eventually
> some other patch a long time later will obsolete my patch in some way, and
> the whole thing drifts off into forgotten history.  Err, almost forgotten,
> because the bad taste remains.
> 
> And yes, there was a successor to the patch in which I did the job 'properly'
> by cleaning up some other infrastructure instead of just fixing the bug
> locally.  There was also a long lag after I created and submitted that
> version before the bug was actually fixed, and then it was only fixed in 2.4.
> 

When all this ext2 fuss was going on, I went into ext3, changed
a few lines, fixed the bug and typed `cvs commit'.

I just thought you might enjoy hearing that.

-

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> Note who the email is addressed to.  I have tried many different techniques 
> for communicating with this gentleman, including self-deprecation, and they 
> all seem to have the same result

Trying a bit of intellectual honesty would help big way.

Realizing that ext2 patches should be sent to ext2 maintainers would help
even more.

You've spent _months_ ignoring the idea above.  You've tried many different
techniques for what, exactly?  To push that stuff to a guy who is not, was not
and had never been maintainer of the code in question?  Wow.

And yes, it had been told to you from the very beginning.  tytso, sct and akpm
are the right guys for such stuff.  It's their code, they do maintain it
and I think in all cases I've sent ext2 patches it was only after ACK from
ext2 folks.

If it took you a fscking year to realize that, despite having it explained to
you in details...  Don't you feel yourself an idiot?

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 05:37 am, Alexander Viro wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> > Note who the email is addressed to.  I have tried many different techniques 
> > for communicating with this gentleman, including self-deprecation, and they 
> > all seem to have the same result
> 
> Trying a bit of intellectual honesty would help big way.

I've been entirely straightforward and honest.  If I were intellectually
dishonest, I would smile and take the crap from you, as others do  But that
is not me as you know, and I suppose that is why you let your venom out.
(And don't say you don't, I have irc logs enough to prove that point.)

By the way, do you think that your constant dissing of me, typically
behind my back, makes people respect you more?

> Realizing that ext2 patches should be sent to ext2 maintainers would help
> even more.
> 
> You've spent _months_ ignoring the idea above.  You've tried many different
> techniques for what, exactly?  To push that stuff to a guy who is not, was not
> and had never been maintainer of the code in question?  Wow.

Linus just called you the ext2 maintainer.  If you do not consider yourself
to be the ext2 maintainer, when was the last time you submitted a patch
through Ted?

In any event, a reasonable patch was submitted to Ted:

   http://marc.theaimsgroup.com/?l=ext2-devel&m=99039802717798&w=2

with zero results one way or the other, probably because Ted, who hadn't
been seen on the ext2-devel list for some time at that point, was up to his
ears in something else.  A version of the patch was forwarded to you at
that time, and you also subscribe to ext2-devel, so you knew the whole
store, including the fact that Andrew had signed of on it:

  http://marc.theaimsgroup.com/?l=ext2-devel&m=99054430703022&w=2

Now, you could say that at this point the ball was in my/Ted's court, and
you'd be right, except for the fact that there was yet another go around on
it, just before 2.5 opened, when Alan for some reason wanted to wait for 2.5
to open, and again just after 2.5 opened, when I offered the patch again and
you refused it because you planned to obsolete it.

I found the whole story fairly distasteful, and even so, I would have
forgotten about it if Villa Herva had not noticed that I was being jerked
around, and brought it to the attention of the community.  By the way, I
had nothing to do with this, I'd never heard of him before he made his
post.

A similar story was played out with the fs.h cleanups.  I'm unhappy with
the way you handled that, as well.

> And yes, it had been told to you from the very beginning.  tytso, sct and akpm
> are the right guys for such stuff.  It's their code, they do maintain it
> and I think in all cases I've sent ext2 patches it was only after ACK from
> ext2 folks.
>
> If it took you a fscking year to realize that, despite having it explained to
> you in details...  Don't you feel yourself an idiot?

No I do not, and it is precisely that kind of remark that makes you hard
or impossible to get along with.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

Calm down guys.  Al, Daniel, peace.

It was fun while people were just ragging on me (it tends to happen about
every 6 months or so, and I have a thick skin - and every time it happens
some problems do get unearthed and that's fine), but let's not make this
degenerate into a real hate-fest..

Shake hands, please.

-- tangential --

One thing intrigued me in this thread - which was not the discussion
itself, but the fact that Rik is using bitkeeper.

How many other people are actually using bitkeeper already for the kernel?
I know the ppc guys have, for a long time, but who else is? bk, unlike
CVS, should at least be _able_ to handle a "network of people" kind of
approach.

		Linus

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> 
> How many other people are actually using bitkeeper already for the kernel?

I am, for the USB and PCI hotplug stuff:
	http://linuxusb.bkbits.net/

It makes tracking what patches got applied, and which didn't, and
forward porting those that didn't to the next release, a breeze.

My trees are world readable, and people are welcome to send me patches
against it, or even bitkeeper changesets, but I have yet to receive one
of those yet :)

thanks,

greg k-h

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 02:48 am, Linus Torvalds wrote:

> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
>
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

One thing that's intrigued ME is the explanation of the hierarchy of 
maintainers.  There ARE specific people that patches should be reviewed by 
before being sent to you, there even seems to be a directed graph of them.

It would be kind of nice if it was documented enough that at least the 
maintainers in the maintainers list knew what it was, and who they should 
forward stuff on to after reviewing it.  That might go a ways towards 
addressing the "hitting resend isn't working" problem...

You've said that the tier under you (who you DO semi-reliably accept patches 
from) is a group of ten to twenty people.  If we knew who those people were, 
we could bug them to name THEIR secretary lists (or figure it out from the 
maintainers list)...

In your original response to the patch penguin proposal, you mentioned:

>The fact is, we've had "patch penguins" pretty much forever, and they
>are called subsystem maintainers.  They maintain their own subsystem, ie
>people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
>(USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
>Molnar (scheduler), Jeff Garzik (network drivers) etc etc. 

You also said:

> The VM stuff right now seems to be Andrea, Dave or you yourself.

That was responding to Rik van Riel, I'm guessing "Dave" is Dave Jones(?), 
and of course Andrea would be Andrea Arcangeli.  It seems that Andrea 
Arcangeli is the default VM maintainer, and that Dave Jones is gradually 
getting sucked into Alan Cox's old position as "miscelaneous maintainer" 
putting out a "this needs wider testing" tree.

The above seems to be about the full list I can assemble from recent emails.  
(You've also used David Miller again as an example in a later email, you put  
Paul Mackerras as a subordinate maintainer under him, and "Greg" 
(Kroah-Hartmann) had Johannes Erdfelt under him handling UHCI.  This isn't 
really new information about the top ten, more like some examples to help in 
tree building under them.)

This is eleven "top level" maintainers, one of whom is handling ext3 which 
sounds kind of odd...  (If David Miller is networking and Jeff Garzik is 
network drivers, would there be a "filesystem drivers" guy paired off with Al 
Viro?  Does EXT2 go through Andrew Morton as well?  Would Hans Reiser submit 
directly to you for ReiserFS patches, or should he get a signoff from...  
Um...  Andrew?  Al?  Try to get it into the -dj tree first?  Could I have a 
hint?)

To clarify what I'm aiming at: Are these eleven people a significant portion 
of the group of people who, if code makes it as far as them and they sign off 
on it, you'd then be willing to at least review it and if necessary 
explicitly reject? [1]  Should some of them be forwarding their patches to 
somebody other than you?  Are there more people on the list that lower level 
maintainers should be funneling patches to in order to eventually get them 
into your tree?

A two tier maintainer system definitely sounds like an improvement if that 
will help the process scale.  It's just that today is the first I've heard 
about it, and I had TRIED to study the situation before opening my big 
mouth...

> 		Linus

Rob

[Footnote 1] Implicit rejections can be REALLY stressful when combined with 
delaying the of inclusion of code that isn't actually rejected, but just not 
convenient to include right now.  It means that code that isn't merged 
immediately soon starts to smell of failure.  The throught process seems to 
go "If Linus hasn't accepted it, and Linus ignores patches he's rejecting, 
maybe he's rejecting this.  If so, the reason is something we need to figure 
out on our own, so let's all pile on the code and start badmouthing it until 
we figure out why Linus doesn't like it."  This can easily go beyond useful 
code review into pointless flame wars.  Arranging a system where it's 
possible to have some kind of progress indicator (even a "distance from 
Linus" index as patches progress through the maintainer tree) seems like a 
good thing to me...

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Rob Landley wrote:

>
>
>This is eleven "top level" maintainers, one of whom is handling ext3 which 
>sounds kind of odd...  (If David Miller is networking and Jeff Garzik is 
>network drivers, would there be a "filesystem drivers" guy paired off with Al 
>Viro?  Does EXT2 go through Andrew Morton as well?  Would Hans Reiser submit 
>directly to you for ReiserFS patches, or should he get a signoff from...  
>Um...  Andrew?  Al?  Try to get it into the -dj tree first?  Could I have a 
>hint?)
>

There is a maintainers list somewhere in the kernel tree.  I am listed 
there as  the ReiserFS maintainer, and I send our patches directly to 
Linus and Marcelo.  I don't think that a filesystems maintainer would be 
easily achieved, since if we agreed about architecture we would have 
written the same filesystems.   We can't even agree about whether 
streams and extended attributes should be implemented as files, and as 
for whether keyword search and database functionality should go into the 
filesystem namespace.....

So, there is a maintainers list, and for many subsystems it works fairly 
well.  For ReiserFS, while I review and approve all patches we accept, 
Oleg Drokin is my patch whirlwind who does the work of testing and 
inspecting line by line for bugs (I inspect more for desirable 
functionality).

ReiserFS has been well-tended by Marcelo, so things are working well for 
us.  Dave Jones tends to 2.5 ReiserFS patches quite nicely also.  None 
of our 2.5 patches are earth-shattering, so I think it is very 
reasonable for Linus to pay attention to, say,  bio stuff for now rather 
than our patches (I am sure he will eventually fold them in from Dave 
Jones's tree.)  I worry more that I haven't had a few hours to brief 
Linus on the strategic direction of Reiser4, and what I think is needed 
longterm to compete with Longhorn, but this is probably my fault for not 
asking him for it.

I know that others have had real problems in this regard, and I don't 
discount them, but ReiserFS patches are going well at this time.

Hans

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

I gave up on CVS for internal stuff with the -ac patches. I ended up keeping
a running patch and a complete archive of the submissions. The archive so
I can ensure info gets back and forth neatly.

(tangientially further)

Larry - can bitkeeper easily be persuaded to take "messages" back all the way 
to the true originator of a change. Ie if a diff gets to Linus he can reject
a given piece of change and without passing messages back down the chain
ensure they get the reply as to why it was rejected, and even if
nobody filled anything in that it was looked at and rejected by xyz at
time/date ?

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 10:14:49AM +0000, Alan Cox wrote:
> Larry - can bitkeeper easily be persuaded to take "messages" back all the way 
> to the true originator of a change. Ie if a diff gets to Linus he can reject
> a given piece of change and without passing messages back down the chain
> ensure they get the reply as to why it was rejected, and even if
> nobody filled anything in that it was looked at and rejected by xyz at
> time/date ?

It's certainly possible and there are changes we could make to make it more
useful.  Right now, there is no record of a change if it goes and then gets
rejected right back out; it's as if you patched and then you did a reverse
patch.

The good news is that each change (patch) has an identifier, they look like

    awc@bitmover.bitmover.com|ChangeSet|20011230212716|39200

and if we kept a record of those that were rejected, it would be trivial for
a developer to track whether his change was in/not seen/rejected.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
 
> -- tangential --
> 
> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
> 
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

It does in some ways anyhow.  Following things downstream is rather
painless, but one of the things we in the PPC tree hit alot is when we
have a new file in one of the sub trees and want to move it up to the
'stable' tree, or when it shows up in your/marcelo's tree.  bk send only
works for same base tree type things (ie a clone of tree X, some
changes, not a clone of tree Y, which was a clone of tree X but has lots
of changes and has tree X changes pulled in frequently).  Unfortunaly I
don't think this is an easy problem to work on either.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> It does in some ways anyhow.  Following things downstream is rather
> painless, but one of the things we in the PPC tree hit alot is when we
> have a new file in one of the sub trees and want to move it up to the
> 'stable' tree

Summary: only an issue because Linus isn't using BK.

Details:

The source of this problem is that there is no Linux/BK tree.  To 
understand, read on.  The PPC team "tracks" the kernel using BK.
There is a paper with pictures describing what they do at 

	http://www.bitkeeper.com/tracking.ps

To do this, they have a BK repository which is the "pristine" source,
i.e., it has nothing but released code from Linus.  They import new work
into that repository using "bk import -tpatch" which takes traditional
patches.

There is a "child" repository which is a copy of the "pristine".  The
child repository is used to hold not only the pristine work but also
all the work from the PPC team.  Think of it as "Linux+PPC".

To get patches back to Linus, the PPC maintainer (used to be Cort, now
Paul) will export changes as a traditional patch from the "Linux+PPC"
repository and send them to Linus.  These changes, sometimes modified,
make their way back to the PPC team through the "pristine" tree.  That's
the source of the problem.  

So the work flow is

	patches -> pristine
		      v
		      v
		   Linux+PPC -> patches to Linus

The problem is when a file is created in the Linux+PPC tree, sent to Linus,
comes back as a patch, is imported in pristine, and then is sent to 
Linux+PPC.

BitKeeper tracks files just like a file system, by a BK form of an inode.
That file that came in as a patch is a different inode, what is logically
the same file was created twice.  Much like if you created foo and bar
and then said "mv foo bar", BitKeeper will complain at you that the file
exists already.

This problem goes away if the PPC team could send Linus BK patches and 
Linus sent out his changes as BK patches.  It doesn't require a wholesale
switch to BK, Linus can still take traditional patches and send them out,
but if he maintained a BK tree as well, the problem Troy described goes
away.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > It does in some ways anyhow.  Following things downstream is rather
> > painless, but one of the things we in the PPC tree hit alot is when we
> > have a new file in one of the sub trees and want to move it up to the
> > 'stable' tree
> 
> Summary: only an issue because Linus isn't using BK.

Then how do we do this in the bk trees period?  To give a concrete
example, I want to move arch/ppc/platforms/prpmc750_setup.c from
2_4_devel into 2_4, without loosing history.  How?  And just this file
and not all of _devel.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> > 
> > Summary: only an issue because Linus isn't using BK.
> 
> Then how do we do this in the bk trees period?  To give a concrete
> example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> 2_4_devel into 2_4, without loosing history.  How?  And just this file
> and not all of _devel.

That question doesn't parse.  There are multiple ways you can do it but 
once you do patches will no longer import cleanly from Linus.  The whole
point of the pristine tree is to give yourself a tree into which you can
import Linus patches.  If you start putting extra stuff in there you will
get patch rejects.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:11:34AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> > On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> > > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > > It does in some ways anyhow.  Following things downstream is rather
> > > > painless, but one of the things we in the PPC tree hit alot is when we
> > > > have a new file in one of the sub trees and want to move it up to the
> > > > 'stable' tree
> > > 
> > > Summary: only an issue because Linus isn't using BK.
> > 
> > Then how do we do this in the bk trees period?  To give a concrete
> > example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> > 2_4_devel into 2_4, without loosing history.  How?  And just this file
> > and not all of _devel.
> 
> That question doesn't parse.  There are multiple ways you can do it but 
> once you do patches will no longer import cleanly from Linus.  The whole
> point of the pristine tree is to give yourself a tree into which you can
> import Linus patches.  If you start putting extra stuff in there you will
> get patch rejects.

Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
again.  One of the problems we hit frequently is that we have to move
files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
But just creating a normal patch, or cp'ing the files means when we pull
linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
have to move one of the files (the previously existing one) into the
deleted dir.  How do we cleanly move just a few files from a child tree
into the parent?  I think this is a lot like what would happen, if Linus
used BK and we wanted to send him support for some platforms, but not
all of the other changes we have.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

> Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> again.  One of the problems we hit frequently is that we have to move
> files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> But just creating a normal patch, or cp'ing the files means when we pull
> linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> have to move one of the files (the previously existing one) into the
> deleted dir.  How do we cleanly move just a few files from a child tree
> into the parent?  I think this is a lot like what would happen, if Linus
> used BK and we wanted to send him support for some platforms, but not
> all of the other changes we have.

BitKeeper is like a distributed, replicated file system with atomic changes.
That has certain advantages, much like a database.  What you are asking 
violates the database rules, if I understand you properly.  Are you asking
to move part of a changeset?  That's a no no, that's like moving the 
increment to your bank account without the decrement to mine; the banks
frown on that :-)

Or are you asking more about the out of order stuff, i.e., whole changesets
are fine but not all of them.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:37:56AM -0800, Larry McVoy wrote:
> > Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> > again.  One of the problems we hit frequently is that we have to move
> > files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> > But just creating a normal patch, or cp'ing the files means when we pull
> > linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> > have to move one of the files (the previously existing one) into the
> > deleted dir.  How do we cleanly move just a few files from a child tree
> > into the parent?  I think this is a lot like what would happen, if Linus
> > used BK and we wanted to send him support for some platforms, but not
> > all of the other changes we have.
> 
> BitKeeper is like a distributed, replicated file system with atomic changes.
> That has certain advantages, much like a database.  What you are asking 
> violates the database rules, if I understand you properly.  Are you asking
> to move part of a changeset?  That's a no no, that's like moving the 
> increment to your bank account without the decrement to mine; the banks
> frown on that :-)
> 
> Or are you asking more about the out of order stuff, i.e., whole changesets
> are fine but not all of them.

Unfortunatly I think the PPC tree has hit both cases :)  The restriction
that everything gets moved as a changeset is fine tho. One problem is
an out-of-order (or rather a single) changeset which creates a few
files.

The other problem is we create a file (say
arch/ppc/kernel/prpmc750_setup.c) and then 4-5 changesets effect this
file (code, code, bk mv, code, code).  If this is doable in multiple
out-of-order sends to the parent, that'd probably be OK.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> > Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> > again.  One of the problems we hit frequently is that we have to move
> > files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> > But just creating a normal patch, or cp'ing the files means when we pull
> > linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> > have to move one of the files (the previously existing one) into the
> > deleted dir.  How do we cleanly move just a few files from a child tree
> > into the parent?  I think this is a lot like what would happen, if Linus
> > used BK and we wanted to send him support for some platforms, but not
> > all of the other changes we have.
> 
> BitKeeper is like a distributed, replicated file system with atomic changes.
> That has certain advantages, much like a database.  What you are asking 
> violates the database rules, if I understand you properly.  Are you asking
> to move part of a changeset?  That's a no no, that's like moving the 
> increment to your bank account without the decrement to mine; the banks
> frown on that :-)
> 
> Or are you asking more about the out of order stuff, i.e., whole changesets
> are fine but not all of them.

If I understand it correctly, yes, you want to `push' only one changeset (the
creation of the new file) to the parent repository. Either directly (through
push), or through creating a patch in the child repository and importing it in
the parent repository.

[ Disclaimer: I'm not that familiar with the  problem Tom mentions ]

However, why couldn't BK automatically resolve this?

In BK, a file creation (or a rename) is simply a changeset, just like a change
to the contents of a file (i.e. a patch that affects one file only), right?

If I modify a file in the child repository, and that change ends up in the
same file in the parent repository (i.e. Linus applied the patch I sent there),
BK will automatically resolve the issue when I do a pull in my child
repository.  How is this different from a new file I added in the child
repository, and the same file (with the same contents, or contents from a
previous revision in the child repository) that got added in the parent later?
If I do a pull, BK should `merge' the change (a new file)? Or am I missing
something?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Paul Mackerras]

Larry McVoy writes:

> On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> > Then how do we do this in the bk trees period?  To give a concrete
> > example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> > 2_4_devel into 2_4, without loosing history.  How?  And just this file
> > and not all of _devel.
> 
> That question doesn't parse.  There are multiple ways you can do it but 
> once you do patches will no longer import cleanly from Linus.  The whole
> point of the pristine tree is to give yourself a tree into which you can
> import Linus patches.  If you start putting extra stuff in there you will
> get patch rejects.

I think there is a misunderstanding here: we actually have 3 trees:

linux_2_4		"pristine" tree, identical to Marcelo's latest
linuxppc_2_4		"stable" tree, stuff we are pushing to Marcelo
linuxppc_2_4_devel	"devel" tree, bleeding edge stuff

Normally linuxppc_2_4 pulls from linux_2_4 and linuxppc_2_4_devel
pulls from linuxppc_2_4.  That is, linuxppc_2_4_devel has all of the
changesets that are in linuxppc_2_4, and more.  When Marcelo does a
new release the changes go into linux_2_4 and propagate from there
into linuxppc_2_4 and then linuxppc_2_4_devel.

Now when we decide that some stuff in linuxppc_2_4_devel has matured
to the point where we want it in linuxppc_2_4, what we currently do,
conceptually at least, is to generate a patch with the changes we want
and apply that to the linuxppc_2_4 tree.  If we had the ability to
apply changesets out-of-order, presumably what we could do is to push
the particular changesets of interest from linuxppc_2_4_devel back up
into linuxppc_2_4.  Then when we pulled from linuxppc_2_4 into
linuxppc_2_4_devel, bk would presumably say "got that one already"
about those changesets.

At the moment the process of applying a patch to linuxppc_2_4 and
doing the pull into linuxppc_2_4_devel results in conflicts which bk
mostly handles automatically *except* in the cases where the patch
creates new files.

Paul.

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > It does in some ways anyhow.  Following things downstream is rather
> > painless, but one of the things we in the PPC tree hit alot is when we
> > have a new file in one of the sub trees and want to move it up to the
> > 'stable' tree
>
> Summary: only an issue because Linus isn't using BK.

Bitkeeper also seems to have some problems applying out-of-order
changesets or applying them partially.

Changesets sent by 'bk send' are also much harder to read than
unidiffs ;)

I think for bitkeeper to be useful for the kernel we really need:

1) 'bk send' format Linus can read easily

2) the ability to send individual changes (for example, the
   foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

3) the ability for Linus to apply patches that are slightly
   "out of order" - a direct consequence of (2)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> >
> > Summary: only an issue because Linus isn't using BK.
> 
> Bitkeeper also seems to have some problems applying out-of-order
> changesets or applying them partially.
> 
> Changesets sent by 'bk send' are also much harder to read than
> unidiffs ;)
> 
> I think for bitkeeper to be useful for the kernel we really need:
> 
> 1) 'bk send' format Linus can read easily

I think you can do bk send -u which spits out a unified diff in the
comments of the file or so.

> 2) the ability to send individual changes (for example, the
>    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

This is sort of what I was getting at, execpt in this case foo_net.c is
also a new file as well.  Myself and Paul haven't found a good way to do
this :(

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> >
> > Summary: only an issue because Linus isn't using BK.
> 
> Bitkeeper also seems to have some problems applying out-of-order
> changesets or applying them partially.

It does indeed and I think this is a far more serious issue than, for
example, the shouting SCCS subdirectories.  So let's discuss it.

> Changesets sent by 'bk send' are also much harder to read than
> unidiffs ;)

Yeah but if you do a bk send -d it prefixes them with unidiffs or 
you can do 
	
	cat patch | bk receive /home/bk/vmtree
	cd /home/bk/vmtree/RESYNC
	bk csets

and you are looking at the changes in the changeset viewer which most
people think is nicer than unidiffs.

> I think for bitkeeper to be useful for the kernel we really need:
> 
> 1) 'bk send' format Linus can read easily

That's done.

> 2) the ability to send individual changes (for example, the
>    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

That's possible now but not a really good idea.

> 3) the ability for Linus to apply patches that are slightly
>    "out of order" - a direct consequence of (2)

This is really the main point.  There are two issues, one is out of order
and the other is what we call "false dependencies".  I think it is the 
latter that bites you most of the time.  The reason you want out of order
is because of the false dependencies, at least that is my belief, let
me tell you what they are and you tell me.

Suppose that you make 3 changes, a driver change, a vm change, and a 
networking change.  Suppose that you want to send the networking change
to Linus.  With patch, you just diff 'em and send and hope that patch
can put it together on the other end.  With BK as it stands today, there
is a linear dependency between all three changes and if you want to send
the networking change, you also have to send the other 2.

How much of the out order stuff goes away if you could send changes out
of order as long as they did not overlap (touch the same files)?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:32:54AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
[snip]
> > 2) the ability to send individual changes (for example, the
> >    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff
> 
> That's possible now but not a really good idea.

How is it possible and what can go wrong?  This will be a common thing I
think, so it'd be a good thing to know (and have reliable and be a good
idea in some form).

> > 3) the ability for Linus to apply patches that are slightly
> >    "out of order" - a direct consequence of (2)
> 
> This is really the main point.  There are two issues, one is out of order
> and the other is what we call "false dependencies".  I think it is the 
> latter that bites you most of the time.  The reason you want out of order
> is because of the false dependencies, at least that is my belief, let
> me tell you what they are and you tell me.

I think that sounds about right.  If changeset 1.123 and 1.124 are only
related in that one got commited first (and for the sake of argument,
isn't done just yet) and the other is ready to go, the second one is
stuck.

> Suppose that you make 3 changes, a driver change, a vm change, and a 
> networking change.  Suppose that you want to send the networking change
> to Linus.  With patch, you just diff 'em and send and hope that patch
> can put it together on the other end.  With BK as it stands today, there
> is a linear dependency between all three changes and if you want to send
> the networking change, you also have to send the other 2.
> 
> How much of the out order stuff goes away if you could send changes out
> of order as long as they did not overlap (touch the same files)?

I think that'd help a good deal of the cases.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:43:36AM -0700, Tom Rini wrote:
> On Wed, Jan 30, 2002 at 08:32:54AM -0800, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> [snip]
> > > 2) the ability to send individual changes (for example, the
> > >    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff
> > 
> > That's possible now but not a really good idea.
> 
> How is it possible and what can go wrong?  This will be a common thing I
> think, so it'd be a good thing to know (and have reliable and be a good
> idea in some form).

It's possible in that we can generate regular diffs for any range of
any files.  Cort used to do this all the time to send a subset of the
tree to Linus as a regular patch, talk to him about getting his scripts.
Or talk to Paul, I suspect he does it as well.

> > > 3) the ability for Linus to apply patches that are slightly
> > >    "out of order" - a direct consequence of (2)
> > 
> > This is really the main point.  There are two issues, one is out of order
> > and the other is what we call "false dependencies".  I think it is the 
> > latter that bites you most of the time.  The reason you want out of order
> > is because of the false dependencies, at least that is my belief, let
> > me tell you what they are and you tell me.
> 
> I think that sounds about right.  If changeset 1.123 and 1.124 are only
> related in that one got commited first (and for the sake of argument,
> isn't done just yet) and the other is ready to go, the second one is
> stuck.

Indeed.  We're painfully aware of this.

> > Suppose that you make 3 changes, a driver change, a vm change, and a 
> > networking change.  Suppose that you want to send the networking change
> > to Linus.  With patch, you just diff 'em and send and hope that patch
> > can put it together on the other end.  With BK as it stands today, there
> > is a linear dependency between all three changes and if you want to send
> > the networking change, you also have to send the other 2.
> > 
> > How much of the out order stuff goes away if you could send changes out
> > of order as long as they did not overlap (touch the same files)?
> 
> I think that'd help a good deal of the cases.

So here's the deal on this topic.  The out of order thing is a much
bigger deal for a large open source project than it is for our commercial
customers.  It's also hard to do correctly, it would take at least a
month.  Linus has flirted with using BitKeeper multiple times and then
gotten distracted.  If we had dropped everything and fixed the issues he
needs fixed rather than the issues our commercial customers need fixed,
we'd be out of business and you'd have the lovely task of trying to make
this work in the BitKeeper source.  You can believe me or not, but you'd
have very little chance of getting it to work, the BK source base is 
a lot larger and more complex than the generic part of the Linux kernel.

If enough of the kernel people start using BitKeeper and demanding the
out of order stuff, we'll do it.  The lead time is about a month, so
you have to deal with that.  On the other hand, if this turns into yet
another kernel "she loves, she loves me not, she loves me, she loves me
not" about BK, we're not going to fix the out of order stuff until it
is the most important issue for our commercial customers.

Hopefully, you'll take this in the spirit in which it is intended.
We want to help out the kernel team, that was the reason for writing
BitKeeper.  We have to survive, however, and that means paying attention
to the commercial needs as well.  If/when it looks like Linus & Co are
serious about using BK, I'll staff a couple of people to address the 
out of order stuff and commit to a delivery date.  It's clear that it
is the biggest "gotcha" about BK & the kernel work flow.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> How much of the out order stuff goes away if you could send changes
> out of order as long as they did not overlap (touch the same files)?

could this be made: 'as long as they do not touch the same lines of code,
taking 3 lines of context into account'? (ie. unified diff definition of
'collisions' context.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 07:35:03PM +0100, Ingo Molnar wrote:
> 
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> 
> > How much of the out order stuff goes away if you could send changes
> > out of order as long as they did not overlap (touch the same files)?
> 
> could this be made: 'as long as they do not touch the same lines of code,
> taking 3 lines of context into account'? (ie. unified diff definition of
> 'collisions' context.)

No.  What you described is diff/patch.  We have that already and if it
really worked in all the cases there would be no need for BitKeeper to
exist.  I'll be the first to admit that BK is too pedantic about change
ordering and atomicity, but you need to see that there is a spectrum and
if we slid BK over to what you described it would be a meaningless tool,
it would basically be a lot of code implementing what people already do
with diff/patch.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 07:35:03PM +0100, Ingo Molnar wrote:
> > On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > > How much of the out order stuff goes away if you could send changes
> > > out of order as long as they did not overlap (touch the same files)?
> >
> > could this be made: 'as long as they do not touch the same lines of code,
> > taking 3 lines of context into account'? (ie. unified diff definition of
> > 'collisions' context.)
>
> No.  What you described is diff/patch.  We have that already and if it
> really worked in all the cases there would be no need for BitKeeper to
> exist.  I'll be the first to admit that BK is too pedantic about
> change ordering and atomicity, but you need to see that there is a
> spectrum and if we slid BK over to what you described it would be a
> meaningless tool,

OK, so why not put the boundary at the same point as where
bitkeeper still manages to automatically merge branches and
where it gives up ?

(this seems to be somewhat finer grained than the whole-file
level, but more picky and intelligent than patch/diff)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> No.  What you described is diff/patch.  We have that already and if it
> really worked in all the cases there would be no need for BitKeeper to
> exist.  I'll be the first to admit that BK is too pedantic about
> change ordering and atomicity, but you need to see that there is a
> spectrum and if we slid BK over to what you described it would be a
> meaningless tool, it would basically be a lot of code implementing
> what people already do with diff/patch.

eg. i sent 8 different scheduler update patches 5 days ago:

 [patch] [sched] fork-fix 2.5.3-pre5
 [patch] [sched] yield-fixes 2.5.3-pre5
 [patch] [sched] SCHED_RR fix, 2.5.3-pre5
 [patch] [sched] set_cpus_allowed() fix, 2.5.3-pre5
 [patch] [sched] entry.S offset fix, 2.5.3-pre5.
 [patch] [sched] cpu_logical_map fixes, balancing, 2.5.3-pre5
 [patch] [sched] compiler warning fix, 2.5.3-pre3
 [patch] [sched] unlock_task_rq() cleanup, 2.5.3-pre3

these patches, while many of them are touching the same file (sched.c) are
functionally orthogonal, and can be applied in any order. Linus has
applied all of them, but he might have omitted any questionable one and
still apply the rest.

how would such changes be expressed via BK, and would it be possible for
Linus to reject/accept an arbitrary set of these patches?

	Ingo

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 07:48:35PM +0100, Ingo Molnar wrote:
> eg. i sent 8 different scheduler update patches 5 days ago:
> 
>  [patch] [sched] fork-fix 2.5.3-pre5
>  [patch] [sched] yield-fixes 2.5.3-pre5
>  [patch] [sched] SCHED_RR fix, 2.5.3-pre5
>  [patch] [sched] set_cpus_allowed() fix, 2.5.3-pre5
>  [patch] [sched] entry.S offset fix, 2.5.3-pre5.
>  [patch] [sched] cpu_logical_map fixes, balancing, 2.5.3-pre5
>  [patch] [sched] compiler warning fix, 2.5.3-pre3
>  [patch] [sched] unlock_task_rq() cleanup, 2.5.3-pre3
> 
> these patches, while many of them are touching the same file (sched.c) are
> functionally orthogonal, and can be applied in any order. Linus has
> applied all of them, but he might have omitted any questionable one and
> still apply the rest.
> 
> how would such changes be expressed via BK, and would it be possible for
> Linus to reject/accept an arbitrary set of these patches?

There is a way to do this in BK that would work just fine.  It pushes some
work back onto the developer, but if you are willing to do it, we have no
problem doing what you want with BK in its current form and I suspect that
the work is very similar to what you are already doing.

In your list above, all of the patches are against 2.5.3-pre5.  If you did
the work for each patch against the 2.5.3-pre5 baseline, checking it in,
saving the BK patch, removing that changeset from the tree, and then going
onto the next change, what you'd have is exactly the same set of patches
as you have no.  Literally.  You could type the appropriate command to BK
and you should be able to generate a bit for bit identical patch.

In BK's mind, what you have done is to make a very "bushy" set of changes,
they are all "side by side".  If you think of a graph of changes, you started
with

			[older changes]
			      v
			  [2.5.3-pre4]
			      v
			  [2.5.3-pre5]

and then you added one change below that, multiple times.  If you were to
combine all of those changes in a BK tree, it would look like

			[older changes]
			      v
			  [2.5.3-pre4]
			      v
			  [2.5.3-pre5]
  [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]

and BK would be happy to allow you to send any subset of the sched changes
to Linus and it would work *exactly* how you want it to work.  If we could
get people to work like this, there are no problems.  Just to make it really
clear you would do this

	for p in patch_list
	do	bk vi sched.c	# that will lock it if isn't
		hack, debug, test
		bk citool	# check it in and make a changeset
		bk makepatch -r+ > ~/bk-patches/patch.$p
		bk undo -fr+	# get back to the same baseline
	done

Here's what people actually do.  They make the first change, then make
the second change in the same tree that already has the first change,
and so on.  BitKeeper faithfully records the linear sequence of changes
and enforces that the changes propogate as that linear sequence.  You can
skip some at the end but you can't skip any in the middle.

In your particular case, we really need out of order changesets to allow
the second type of work flow and cherry picking.  However, a fairly common
case is that the changes are all in unrelated files and *even then* 
BitKeeper enforces the linearity.  That's the problem I think we need to
fix first, it's not a complete solution, but it is the 80-90% solution.
Until we give you a 100% solution, you have to realize that you are making
side by side changes and actually do it that way.

If any of this is not clear to anyone, please speak up and I'll try and 
draw some pictures and add some explanation.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> There is a way to do this in BK that would work just fine.  It pushes some
> work back onto the developer, but if you are willing to do it, we have no
> problem doing what you want with BK in its current form and I suspect that
> the work is very similar to what you are already doing.

The thing is, bk should be able to do this on its own.

The rule on check-in should be: if the resultant changeset can be
automatically merged with an earlier changeset, it should be _parallel_ to
that changeset, not linear.

And note the "automatically merged" part. That still guarantees your
database consistency at all levels.

Let us assume that you have a tree that looks like

	a -> b -> c

together with modifications "d". Now, "bk ci" (or, because this is more
expensive than a plain "ci", you can call it "bk backmerge" or something,
and all sane people will just stop using "ci") should instead of doing a
plain

	a -> b -> c -> d

it would see how far back it can go with an automatic merge and add "d" at
the _furthest_ point possible. So let's assume that "d" really cannot be
merged with "b" but doesn't clash with "c", so what you'd create with "bk
backmerge" is the "maximally parallel version:

	a -> b -> c
		\
		 > d

Now, you'll say that this is exponentially expensive to do, and I agree.
But CPU-time is cheap, and notice that this "backmerge" would be done not
in one centralized location, but in parallel at different developers.

(Yeah, my example may look "cheap", but if you do exclusively backmerging,
then after a while your trees will have lots of very wide development, and
the m,ore interesting backmerge is when you already have

	a -> b -> c -> f

		-> d

		-> e

and you're adding "g", and it doesn't merge with "c" but not with "d" and
"e", so you have to test every path backwards to see where you can push
it. In this case you'd end up with

	a -> b  -> c -> f

		     -> g

		-> d

		-> e

kind of tree.)

Just how expensive would that be? I'd be willing to make my machine sweat
a bit, if it would just automatically generate the most parallel
changesets possible.. And I bet you could do _this_ fairly easily if you
didn't care too much about trying to be too efficient.

You're saying your merges are perfect. USE that knowledge, and make it do
"bk backmerge".

(Once you do backmerge, the false dependencies no longer exist, AND
suddenly "bk" actually gives people information that they didn't have
before: the revision history actually tells you the _true_ dependencies in
the tree, not some stupid "this is the order in which things were checked
in" stuff that doesn't add any value that can't be seen from the changeset
directly)

Larry, give me that, and I'll promise to use bk exclusively for two months
to shake it out..

		Linus

Re: A modest proposal -- We need a patch penguin

[Georg Nikodym]

On Wed, 2002-01-30 at 13:23, Linus Torvalds wrote:

[really interesting and insightful comments about revision graphs]

The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
the change that is 'g'.  It's actually a label that implies a point in
time as well as all the change that came before it.  Stated differently,
it is a set.

Using your graph:

        a -> b -> c -> f

                -> d

                -> e

and the way that people currently think (and thus speak) of these
things, saying that you're using a version 'e' kernel is ambiguous
because it may or may not have 'c' or 'd'.  This ambiguity also
complicates the task of reproducing a tree at some known state later.

You, as the center of the known universe may not need to concern
yourself with such trifles, but speaking as one of those fabled
commercial customers, the ability to say unambiguously say what's been
changed (read: fixed) is really important.

All that said, I like your idea about revision graph compression.  My
concerns might simply be mitigated by being able to insert "release"
points (simply a tag that implies/includes all the preceding
changesets).

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 02:38:21PM -0500, Georg Nikodym wrote:

[snip]
> and the way that people currently think (and thus speak) of these
> things, saying that you're using a version 'e' kernel is ambiguous
> because it may or may not have 'c' or 'd'.  This ambiguity also
> complicates the task of reproducing a tree at some known state later.

Well, this is what tags are for.  The ambiguity in changesets is OK,
since it's possible anyhow with multiple people (and with some creative
work maybe, 'c' would be before 'd', but at the same level, so 'c'
wouldn't get 'd', but this might break the new behavior so..)  But if
you do:
a b (tag v1) c e (tag v2)
             d
	     f (added after the v2 tag)

it should be possible to have the 'v2' tag say what changsets it had, or
even what rev of each file it was made to.

Larry, how does BK handle this now?  Ive been thinking about this for a
bit and am kind of curious now..

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On 30 Jan 2002, Georg Nikodym wrote:
>
> The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
> the change that is 'g'.  It's actually a label that implies a point in
> time as well as all the change that came before it.  Stated differently,
> it is a set.

I disagree.

"g" is really just one thing: it is "the changes".

However, you are obviously correct that any changes are inherently
dependent on the context those changes are in. And there are multiple
contexts.

One context is simply the "when in time" context, which is interesting, as
when you serialize all changesets you see the time-wise development. And
this is really the _only_ context that BK (and most other source control
tools) really think about.

However, in another sense, the "when in time" context is completely
meaningless. Should you care what the _temporal_ relationship between two
independent patches is? I say "Absolutely not!". The temporal relationship
only hides the _real_ revision information, which is what the patch
actually depends on.

And note that my suggestion to have a "bk backmerge" does not remove the
temporal relationships. All changesets already have a timestamp (they
clearly have to have it, just so that you can see when they happened and
say "what did this tree look like one month ago?"). So we already _have_
the temporal information, and encoding that temporal information into the
"relationship" information actually ends up costing you quite dearly.

I'd say that most (maybe all) of the complaints about not being able to
apply changesets in any random order comes exactly from the fact that
developers _know_ that temporal relationships are often not relevant. From
a development standpoint, temporal relationships are only relevant if they
match the _causal_ relationships, and even then you can often find patches
that are _caused_ by previous patches, but that are valid on their own
(and can indeed be more important, or even completely obviate the need for
the original patch).

So what I'm saying is that from a patch revision standpoint, temporal
information is useless. You still have enough information to recreate the
tree "at time 'g'" by just doing the equivalent of "bk get -c<date-of-g>".

See?

> You, as the center of the known universe may not need to concern
> yourself with such trifles, but speaking as one of those fabled
> commercial customers, the ability to say unambiguously say what's been
> changed (read: fixed) is really important.

But you don't lose that ability. You still have all the information you
used to have, you just have even _more_ information. You have the
information on notjust when the change was checked in (temporal), you have
the information on what files/changes it really _depended_ on.

Now, after those arguments, I'll just finish off with saying that I
actually agree with you to some degree - as I already said in private
email to Larry, I would definitely also want to have a way of stopping
back-merging at some point. In particular, when I'd tag a relase (ie
something like Linux-2.5.3, I would potentially also want to set a
"backmerge marker tag" which basically tells the backmerge logic that it's
not worth it to try to backmerge past that version tag.

That would decrease the chance of confusion considerably, and it would
also avoid the exponential complexity problem. Let's face it, exponential
algorithms can be really useful, but you do want to have some way of
making sure that the bad behaviour never happens in practice. A way of
limiting the set of changelogs to be considered for backmerging not only
means that humans don't get as confused, the computer also won't have to
work insanely to try to go back to Linux version 0.01 if a small patch
happened to apply all the way back.

		Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 01:17:25PM -0800, Linus Torvalds wrote:
> 
> On 30 Jan 2002, Georg Nikodym wrote:
> >
> > The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
> > the change that is 'g'.  It's actually a label that implies a point in
> > time as well as all the change that came before it.  Stated differently,
> > it is a set.
> 
> I disagree.
> 
> "g" is really just one thing: it is "the changes".

In some system that you are imagining, you are correct.  In BK, you are
flat wrong and you've missed an important point.

In BK, "g" means two things: the diffs and the stuff that has no diffs.
In other words, both what changed and exactly what was in the tree at
the time the change was created.

In your mind, "g" is just the diffs applied to any tree which allows
the diffs to apply cleanly by patch, and even more trees where they
don't apply cleanly but you can fix up the patch rejects by hand.

The BK "g" is a more pwoerful statement than the Linus "g".  
The Linus "g" is a more flexible thing than the BK "g".

The BK "g" is more powerful because when I say "I've tested g", I'm
saying one hell of a lot more exact statement then when you say "I've
tested g".  The BK "g" is reproducible, without exception.  Your "g"
is reproducible if and only if I happen to have exactly the same tree
that you had when you applied the "g" patch.

The real power of the BK way is that testing builds up incrementally,
one on top of the other, but in your way, each time you apply the 
patch in some other context, the testing is invalidate.  You can
swear up and down all you want that it doesn't make any difference
and every time you do I will come up with a different example where
a whole team of Linus types said the same thing and were wrong.  If
you apply the patch to a different context, you have to start the
testing process over.  No ifs, ands, or buts.  If I can't run cmp(1)
on the object and get no diffs, it's not the same object.

> One context is simply the "when in time" context, which is interesting, as
> when you serialize all changesets you see the time-wise development. And
> this is really the _only_ context that BK (and most other source control
> tools) really think about.

If that's what you believe about BK, you haven't the faintest idea of
how it works.  BK has, right now, in the bits that you have on your
disk, the ability to recreate any change with all, some, or none of the
previous changes.  bk help sccscat.  You can create a version of the 
file which has only the odd numbered changes in it if that is what you
want.  Try that in other systems or with your patches.

> However, in another sense, the "when in time" context is completely
> meaningless. Should you care what the _temporal_ relationship between two
> independent patches is? I say "Absolutely not!".

And I (and the entire release management and software support world)
say  "you're wrong".  I'll grant you are right a lot, but lots of times
you'll be wrong.  I've seen enough changes like "remove all the trailing
whitespace, it can't hurt anything" and it breaks the product.  

> I'd say that most (maybe all) of the complaints about not being able to
> apply changesets in any random order comes exactly from the fact that
> developers _know_ that temporal relationships are often not relevant.

Phooey.  They _think_ they know.  You, Linus, are better than average
on this topic but you make mistakes too.  Any the point you are missing
is that in the face of a random set of inputs, any time you change
anything, you need to restart the test cycle.  The BK way lets you
isolate exactly what caused the problem.  We can, and do, run binary
search over changes to figure out what caused the problem.  As outlined
in Documentation/bug-hunting.txt.  The difference is that we track
it down to exactly the change that caused the problem and you can't,
you are varying more variables than BK does.  I'm not saying that you
shouldn't be able to vary things, I'm saying that it has a cost, just
like not being able to vary it has a cost.  You are painting a picture
that says you can vary the order and it won't matter.  That's flat out
wrong and I think you know it.

> So what I'm saying is that from a patch revision standpoint, temporal
> information is useless. You still have enough information to recreate the
> tree "at time 'g'" by just doing the equivalent of "bk get -c<date-of-g>".
> 
> See?

No, I don't.  It's a distributed system and there is lots of parallel
development and date-of-g may have many matches.  And it's relatively
meaningless since you are applying g in multiple contexts.  In BK as
it stands, "g" is an invariant.  It means one thing: the state of the
tree immediately after "g" was checked in.  What does your "g" mean?
Some diffs.  Where do they work?  You don't know.  You have to go try
it to find out.

That's all fine, just admit that you have lost something before you
throw it away.  There is no way we'll change BK to allow what you are
asking for until you get it.  I understand what you want, I understand
the implications, I need you to really understand what it is that you
are asking.

I also think we need some face time, I'd like to draw you some pictures
and I find the keyboard just too slow for that.  I think this would be
a lot faster in person.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Linus Torvalds wrote:
[snip]
> Now, after those arguments, I'll just finish off with saying that I
> actually agree with you to some degree - as I already said in private
> email to Larry, I would definitely also want to have a way of stopping
> back-merging at some point. In particular, when I'd tag a relase (ie
> something like Linux-2.5.3, I would potentially also want to set a
> "backmerge marker tag" which basically tells the backmerge logic that it's
> not worth it to try to backmerge past that version tag.
> 
> That would decrease the chance of confusion considerably, and it would
> also avoid the exponential complexity problem. Let's face it, exponential
> algorithms can be really useful, but you do want to have some way of
> making sure that the bad behaviour never happens in practice. A way of
> limiting the set of changelogs to be considered for backmerging not only
> means that humans don't get as confused, the computer also won't have to
> work insanely to try to go back to Linux version 0.01 if a small patch
> happened to apply all the way back.
> 
>                 Linus
[and earlier...]
> However, you are obviously correct that any changes are inherently
> dependent on the context those changes are in. And there are multiple
> contexts.

What about the design context?

I'm a bit concerned about the design-level inter-dependencies of
changesets that don't result in source-level dependencies.

Hypothetical situation:
Changeset adds driver for device Q.  Now, let's further suppose that in
2.5.x we have perfect modularity for drivers and at that time Q is
added... we just add a directory, linux/drivers/Qdrv.  It won't conflict
with 2.5, 2.4.x, 2.2.x, etc..  However, because 2.2.x does not have the
hooks necesary to see it, Q won't work on those kernels.  There is a
design-level dependency in changeset q.

This would be indirectly addressed with the 'backmerg marker tag', but I
have a nagging doubt.

Maybe a better example:
What about changing a global variable (say, from 'events' to
'global_events')?  You change it in the global place, yielding a
changeset.  Later, the individual users change the name.  But if an
individual user has seen very little change in the time before the
global_events change (and the global location has been changing a lot),
that patchset could backmerge beyond the global change.

Say 'f' was the global change, and 'g' was an individual change. 
Backmerge could yield:

        a -> b -> c -> f

                -> d

                -> e
          -> g

even though 'g' really depends on 'f'.

Thoughts?

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Eli Carter wrote:
> > However, you are obviously correct that any changes are inherently
> > dependent on the context those changes are in. And there are multiple
> > contexts.
>
> What about the design context?
>
> I'm a bit concerned about the design-level inter-dependencies of
> changesets that don't result in source-level dependencies.

Well, I'm personally worried about _no_ inter-dependencies that show up as
source-level dependencies that are impossible to break.

If you want to have a "known working version", that's what tagging is for:
basically a list of all patch-sets that makes up the current tree. That
includes all the dependencies.

> Hypothetical situation:
> Changeset adds driver for device Q.  Now, let's further suppose that in
> 2.5.x we have perfect modularity for drivers and at that time Q is
> added... we just add a directory, linux/drivers/Qdrv.  It won't conflict
> with 2.5, 2.4.x, 2.2.x, etc..  However, because 2.2.x does not have the
> hooks necesary to see it, Q won't work on those kernels.  There is a
> design-level dependency in changeset q.

Not so hypothetical, and entirely true. Which is, why I suggest that such
"deep merging" wouldn't actually go past tags.

Let's face it, the source control tool cannot know what works and what
doesn't. The only thing it can ever know about is "what applies". It can
take the approach that everything only applies to the last branch - which
is the traditional approach, but which introduces dependencies that simply
aren't there, and that _cannot_ be cut. This is what bk does now.

But the other approach is to say "whatever applies, applies, and as long
as I don't lose revision information I'll move it backwards". That has
other problems (as you point out), but now those problems are manageable,
and have solutions.

I'd rather take the problem that can be solved over the problem that
cannot.

The fact is, development _often_ is in the situation where somebody does a
quick-and-dirty fix, and then only later goes in and digs deeper and does
the right fix that makes the original fix completely unnecessary.

The way BK works now, if we call the quick-and-dirty fix "A", and the real
fix "B", the developer has a really hard time just sending "B" to me. He'd
have to re-clone an earlier BK tree, re-do B in that tree, and then send
me the second version.

I'm suggesting that he just send me B, and get rid of his tree. There are
no dependencies on A, and I do not want _crap_ in my tree just because A
was a temporary state for him.

			Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> The way BK works now, if we call the quick-and-dirty fix "A", and the real
> fix "B", the developer has a really hard time just sending "B" to me. He'd
> have to re-clone an earlier BK tree, re-do B in that tree, and then send
> me the second version.
> 
> I'm suggesting that he just send me B, and get rid of his tree. There are
> no dependencies on A, and I do not want _crap_ in my tree just because A
> was a temporary state for him.

And you just lost some useful information.  The fact that so-and-so did
fix A and then did B is actually useful.  It tells me that A didn't work
and B does.  You think it's "crap" and by tossing it dooms all future
developers to rethink the A->B transition.

There is a reason that commercial companies guard their revision history
and fight like mad to preserve it.  It contains useful information,
even the bad stuff is useful.

Some stuff may be so bad that it shouldn't ever get in the tree, but you
don't accept anything at all from those people in general.  If Al Viro
takes one pass at a problem and it works well enough that it gets in 
the tree, and then later does a pass two that cleans it up, I can learn
from that.  That's very useful information, his brain frequently shines
a light in a dark corner but I'd miss a lot of that without the history.

Your approach is constantly dropping useful information on the floor.
It may not be useful to you but it's useful to virtually everyone
else.  Saving that information will increase the quality and reduce
the quantity of the patches you get.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> And you just lost some useful information.

No. If the useless crap ends up hiding the real points in the revision
history, getting rid of crud is _good_.

Remember how I asked for a way to "batch up" revision history? Same issue.
Crap is crap, and it more often hides the real issues than enlightens
anything at all.

		Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > And you just lost some useful information.
>
> No. If the useless crap ends up hiding the real points in the revision
> history, getting rid of crud is _good_.

Actually, allowing the deep merges to go past tags could
be useful for dragging bugfixes between the 2.4 and 2.5
kernels ...

... but I think the 'dragging' analogy is something we'll
want to keep here, not back merging across tags by default
but _trying_ to do the backmerge on demand only, when the
user wants to drag a changeset from 2.4 to 2.5.

We could just have a revtool-like interface for that.

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 05:36 pm, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> > The way BK works now, if we call the quick-and-dirty fix "A", and the
> > real fix "B", the developer has a really hard time just sending "B" to
> > me. He'd have to re-clone an earlier BK tree, re-do B in that tree, and
> > then send me the second version.
> >
> > I'm suggesting that he just send me B, and get rid of his tree. There are
> > no dependencies on A, and I do not want _crap_ in my tree just because A
> > was a temporary state for him.
>
> And you just lost some useful information.  The fact that so-and-so did
> fix A and then did B is actually useful.  It tells me that A didn't work
> and B does.  You think it's "crap" and by tossing it dooms all future
> developers to rethink the A->B transition.

<rant>

The noise to signal ratio is too high.  I think Linus has made it clear that 
he actively does not WANT this information.  (The "A" kind of patch is 
generally posted to linux-kernel, where it is buried deep in the flood.)

If developers can't ever make temporary changes to their tree which they do 
NOT intend to send to linus, they can't FUNCTION.  (Except my not doing 
development in said tree.)

They can, of course, explicitly do an end run around your "bondage and 
discipline" design by doing the "patch against the base tree" thing you 
suggested earlier.  Or just having it create plain diffs.  But if they have 
to go to lengths to work around your design to accomplish what THEY want (not 
what you want for them), then the tool is broken.

> There is a reason that commercial companies guard their revision history
> and fight like mad to preserve it.  It contains useful information,
> even the bad stuff is useful.

Do you REALLY think that Linus wants the experimental, quickly-reverted crutf 
of 300 maintainers accumulating in his tree?

Linux development is not a commercial company.  It is FAR more decentralized. 
 There are WAY more developers, doing WAY more experimentation, than most 
commercial companies could EVER afford to fund the man-hours for.  A 
commercial company generaly doesn't have bored college students futzing 
around with random ideas that have a 95% chance of failure, but occasionally 
produce something brilliant.  And a month of experimental baggage tag along 
with a twenty line patch is just insane. 

Trying out way more bad code than good is probably the NORM for the Linux 
development model.  Certainly outside of the core maintainers and 
lieutenants.  What you're basically saying is that people have to be really 
careful about ever putting any code into their tree, or else just extract 
straight patches from bitkeeper and put up with losing the tracking 
information and comments to avoid having your design ideas cram megabytes of 
cruft down their throat.

Good grief, -I- can see this is a bad idea...

> Some stuff may be so bad that it shouldn't ever get in the tree, but you
> don't accept anything at all from those people in general.

Not directly, no.  So basically, you're trying very hard to prevent bitkeeper 
from spreading far down the maintainer tree, due to the exponentially 
increasing number of overridden patches that bitkeeper will suck out of 
everybody's trees no matter how hard they try to avoid passing that garbage 
on to Linus.

Remember Linus's main job?  Code reviewing everythign and making 
architectural decisions?  Why on earth are you trying to force the poor man 
to read code that the submitter does NOT present to him as the solution?  
(There's 8 zillion ways NOT to fix a given problem.  We're trying to REDUCE 
the bandwidth demands on the guy...)

AAAAAAAAAH!

Okay, I'm better now.

(Sorry, this is a hot button issue with me.  Tool makers who insist they know 
how those tools should be used and what for, and thus reject feedback from 
users asking for greater flexibility with a "no, you don't want to DO that".  
Hammer vendors should not tell me what kind of nails to use.)

> If Al Viro
> takes one pass at a problem and it works well enough that it gets in
> the tree, and then later does a pass two that cleans it up, I can learn
> from that.  That's very useful information, his brain frequently shines
> a light in a dark corner but I'd miss a lot of that without the history.

So go read linux-kernel.

Giving people the OPTION of folding this cruft into the tree is one thing.  
FORCING them to do so is just WRONG.

> Your approach is constantly dropping useful information on the floor.

Information which does not belong in Linus's tree.  (You're basically saying 
Linus should add a subset of the rejected patch set to his tree's revision 
history.  Does it sound like a dumber idea to have Linus put EVERY rejected 
patch he deletes into his tree's history in some automated way?)

Monolithic evil.  Proper tool for proper job, don't try to force the job to 
adapt to what you think the tool is good for.

> It may not be useful to you but it's useful to virtually everyone
> else.

I would like to go on record as saying I don't consider this useful.  I don't 
have always enough bandwidth to read through every -pre diff.  This stuff 
gets discussed on linux-kernel.  People are talking about a patch archive 
system which may save rejected patches for posterity.  This is a seperate 
problem, and has a chance of succeeding exactly because it is NOT tangled 
with the issue of source control for the main tree.

> Saving that information will increase the quality and reduce
> the quantity of the patches you get.

Uh, Larry?  By definition, adding unnecessary reverted patches for dependency 
purposes to the set of patches Linus would have to apply to his tree is 
increasing the number of patches Linus actually would have to deal with, if 
he was using bitkeeper-to-bitkeeper.  You are FORCING people to do everything 
as diff -u and drop MORE information, because YOU are not being flexible here.

</rant>

Rob

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 06:18:11PM -0500, Rob Landley wrote:
> > And you just lost some useful information.  The fact that so-and-so did
> > fix A and then did B is actually useful.  It tells me that A didn't work
> > and B does.  You think it's "crap" and by tossing it dooms all future
> > developers to rethink the A->B transition.
> 
> <rant>

I'll see your rant and raise you a nickel.

> If developers can't ever make temporary changes to their tree which they do 
> NOT intend to send to linus, they can't FUNCTION.  (Except my not doing 
> development in said tree.)

Of course they can do that.  They get to decide what they push and
what they don't.  I don't think you understand BK.  What we are talking
about is the problem that the change has been committed to CVS, other
changes are built on top of it, and now Linus realizes that the change
was bad news.  The problem is extracting stuff out of the middle which
has already been built upon for more stuff.  How would you propose solving
that problem because that is the problem statement?

If someone sends Linus a patch, he checks into BK or CVS or whatever,
he then gets 5 other patches and applies them in BK/CVS, and THEN he
wants to take out the first patch, how would you suggest we do that?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 08:57 pm, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 06:18:11PM -0500, Rob Landley wrote:
> > > And you just lost some useful information.  The fact that so-and-so did
> > > fix A and then did B is actually useful.  It tells me that A didn't
> > > work and B does.  You think it's "crap" and by tossing it dooms all
> > > future developers to rethink the A->B transition.
> >
> > <rant>
>
> I'll see your rant and raise you a nickel.
>
> > If developers can't ever make temporary changes to their tree which they
> > do NOT intend to send to linus, they can't FUNCTION.  (Except my not
> > doing development in said tree.)
>
> Of course they can do that.  They get to decide what they push and
> what they don't.  I don't think you understand BK.

Entirely possible.  Quite likely in fact, I'm trying to pick it up as I go 
along.  (I've fired up the docs, but haven't had time to read too far yet.  
Trying to finish some paying work before hitting LWE tomorrow.)

> What we are talking
> about is the problem that the change has been committed to CVS, other
> changes are built on top of it, and now Linus realizes that the change
> was bad news.

The inflexibility of CVS relative to simply applying or reversing patches to 
a source tree on disk is a documented reason Linus doesn't use CVS.  Don't 
compare to CVS, compare to what Linus is currently using.  Beating a straw 
man doesn't HELP.

> The problem is extracting stuff out of the middle which
> has already been built upon for more stuff.  How would you propose solving
> that problem because that is the problem statement?

I'm not quite sure how Linus does this, but how I'd do it is keep around the 
last shipped tree and assemble a patch set against it.  If stuff gets really 
screwed up (a change in the middle needs to be reverted but doesn't unapply 
cleanly), then you can always revert back to the last shipped tree, re-apply 
the patches before the dead one, and then re-apply the patches afterwards 
fixing up rejects as necessary.  (And if I were Linus and the fixup took more 
than 30 seconds, I'd probably throw the dependent ones back down to a 
lieutenant or maintainer with a quick and dirty explanation and have THEM 
fixup the diffs, possibly by making them apply cleanly to the next released 
-pre version and submitting them again then.)  Bounces and even reversions 
due to conflicts are eminently understandable and part of the cost of doing 
business.  (It can be annoying at times, but debugging always is...)

This isn't me trying to make policy, this is me trying to guestimate what's 
going on today.  (I'm not trying to speak for Linus, just explain my 
understanding of how Linus works.)  Now that needs to translate into 
bitkeeper, and if it's HARDER to do in bitkeeper, bitkeeper probably needs to 
be fixed.

> If someone sends Linus a patch, he checks into BK or CVS or whatever,
> he then gets 5 other patches and applies them in BK/CVS, and THEN he
> wants to take out the first patch, how would you suggest we do that?

If the patch no longer unapplies cleanly, then a reversed patch to take it 
out may have to be applied to the tree.  In Linus's tree, this can also 
happen if he's shipped a -pre release with the old patch in it, so reverting 
it would need to be in the next incremental patch Linus releases anyway 
because we're beyond a checkpoint.  (Write barrier, changes committed to 
kernel.org, no longer able to be reordered in cache...)

But if the patch DOES unapply cleanly, and we haven't checkpointed yet, 
there's no good reason Linus can't just revert it.  There should definitely 
be an option to delete all traces of it from the hierarchy (carving moses' 
name off the pillars and all that).

Reverting a change out of order should not ARTIFICIALLY cause conflicts just 
because that's not the way bitkeeper expects people to think.  That would be 
another case of "false dependencies", and gets us back to Linus' backmerge 
concept.  This is sort of reverting a backmerge.

When there is a logistical problem with simply reverting a change because 
other changes really are on top of it, then logically the removal is either 
sort of a patch, or the later changes need to be temporarily reverted and 
fixed up before being reapplied.  (This is not conceptually new, as I said it 
happens with patches all the time.  Whether bitkeeper provides better tools 
to do this than diff, patch, and vi do is an open question.  It might be 
possible to make some sort of "--force --nodeps" way to revert a patch by the 
short and curlies, followed by a manual fixup of the patches that went in on 
top of it with that "bitkeeper fix -c" thing, but probably isn't worth the 
effort and I'm just not going there right now.)

But pointing out that "that history is valuable, leave the old residue in the 
tree even if it hasn't been sent out anywhere yet" is bogus.  Between 
checkpoints, Linus could translate his entire tree into fortran and back if 
he wanted to, and nobody else should ever have to care if he doesn't want 
them to.  The set of changes Linus chooses to ship to turn pre7 into pre8 is 
his choice, not yours.

What Linus seems to want is the shortest and most convenient straight line 
from checkpoint to checkpoint to be stored in his tree.  (Basically, like the 
patch files he puts out NOW, only more granular, with his changelog 
information broken up and stored with each change.)  How can bitkeeper 
provide THAT?  (If it can do more, great.  But if it can't provide this basic 
functionality without some bell or whistle interfering and making it hard to 
use, then there's a problem.  Please be very clear when you're talking about 
an an enhancement over and above this level of functionality when it is NOT a 
requirement...)

Rob

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

Sigh.  OK, I'm taking one more pass at trying to get you to see the light
and then I give up.  You need to understand that you haven't begun to
understand the problem, that you need to think a lot more before speaking,
and that it's really rude to shoot down a system that you haven't even
managed to through the basics of "hello world".  Food for thought.

On Wed, Jan 30, 2002 at 10:12:20PM -0500, Rob Landley wrote:
> The inflexibility of CVS relative to simply applying or reversing patches to 
> a source tree on disk is a documented reason Linus doesn't use CVS.  Don't 
> compare to CVS, compare to what Linus is currently using.  Beating a straw 
> man doesn't HELP.

Go read the archives, patch import/export is not ever, to the best of 
my knowledge, mentioned as a reason for Linus not liking CVS.  It's way
down on the issues list.  File renames, repository hierarchies, work
flow, reproducibility, I've talked with Linus about all of those as issues
but patch import/export never came up.  And CVS isn't remotely as good as
BK at that.  

> > The problem is extracting stuff out of the middle which
> > has already been built upon for more stuff.  How would you propose solving
> > that problem because that is the problem statement?
> 
> I'm not quite sure how Linus does this, but how I'd do it is [a really 
> complicated solution based on patches that won't work]

Think.  What you are describing is basically what Linus does today.  And
noone, including you, is happy.  You're the guy who started this thread.

> > If someone sends Linus a patch, he checks into BK or CVS or whatever,
> > he then gets 5 other patches and applies them in BK/CVS, and THEN he
> > wants to take out the first patch, how would you suggest we do that?
> 
> If the patch no longer unapplies cleanly, then a reversed patch to take it 
> out may have to be applied to the tree.  

Great, now we're getting somewhere.  You can take out a patch in
BK, including old ones way back in time, with a "bk cset -x<rev>".
Works great and no anti-patch is needed, so it's actually better than
what you described.

However, what you described *completely* misses the point.  Linus isn't
asking for an anti-patch, he doesn't want the bad patch in the revision
history at all.  He wants to be able to go backwards, across revisions,
and remove stuff in the middle.  He doesn't want the checkin comments,
he doesn't want the data, he wants no sign the patch was ever in the
revision history.

Do you start to see the problem?  You were yelling and screaming 
"BitKeeper sucks because it can't take a patch out" when in fact
it can do exactly what you said it can't.  On top of that, what you
were complaining about isn't the point.  The thing that you say 
BK can't do, but it can, is not what Linus wants.  Not even close.

And you haven't begun to understand that BK is a distributed, replicated
system.  You can turn all that off and you've got CVS, so turning it
off isn't an option.  Leaving it on means that the revision history is
replicated.  So it isn't an option to collapse a pile of changes into
a smaller pile, the bigger pile will come back the next time he updates
with the other tree.

And once again, you come back with another post that shows you just want
to yell and you haven't thought it through, into my kill file you go,
my blood pressure goes down, you get to yell all you want, everybody
is happy.  I'm willing to try and make BK do what is needed here; I'm
not willing to tolerate people who don't think.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> However, what you described *completely* misses the point.  Linus isn't
> asking for an anti-patch, he doesn't want the bad patch in the revision
> history at all.  He wants to be able to go backwards, across revisions,
> and remove stuff in the middle.  He doesn't want the checkin comments,
> he doesn't want the data, he wants no sign the patch was ever in the
> revision history.

I can't speak for Linus, but my main problem with BK is similar to what
you'd described.  Here's what I'm usually doing and what I'd like to
be able to do with BK:

Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.

I add a branch that consists of B' (B backported to original) and
ABB'^{-1}.  It joins the original at AB.

I backport C to B'.  Now I've got B', C', ABC(B'C')^{-1}.  Again, it
joins the original branch.

Repeat for D and E.  Now I've got the following picture (apologies for BUAG):

* -B'-> * -C'-> * -D'-> * -E'-> *
|                              /
A                            crap
V                            V
* -B-> * -C-> * -D-> * -E-> *

_Now_ I change the direction of last arrow.  Yes, it's more or less reverted
A.  And now I want to consider the top branch as the main history.

IOW, what I want is ability to play revisionist.  And it's not limited to
removing patches - if I've found a bug in A,  I want to be able to add A-fix
and move it all way back to right after A.  And merge them.  B, C, D and E
might have changed from that, but that's what I want.  Moreover, I might
have some junk left in the end (i.e. ABCDEA-fix == (AA-fix)B'C'D'E'noise)
and I'd really like to be able to say that (AA-fix)B'C'D'E' is the main
history now and other path (ABCDE A-fix noise^{-1}) is buried.

If you can give a way to do that - I'm happy.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.

If you just want to make A's changes go away, that's trivial:

	bk get -e -xA foo.c
	bk delta -y"kill A's changes"

all done.

If you want to make A go away out of the graph, that's only possible if
you have the only copy of the graph containing A.  Since BK replicates
the history, you only get to do what you want before you push your changes
to someone else.  No surgery after you let the cat out of the bag.

You have a repository and you haven't propogated all this stuff to
someplace else, then this is easy, we'll just rebuild the history minus A.
You or I could write a shell script using BK commands to do it in about
10 minutes.

It's a distributed, replicated file system which uses the revision history
to propogate changes.  If you don't care about talking to anyone else,
you can do whatever you want.  If you want to give someone your history
and then change it, no way.  That's rewriting what happened.

Now does it make more sense?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> > Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.
> 
> If you just want to make A's changes go away, that's trivial:
> 
> 	bk get -e -xA foo.c
> 	bk delta -y"kill A's changes"
> 
> all done.
> 
> If you want to make A go away out of the graph, that's only possible if
> you have the only copy of the graph containing A.  Since BK replicates
> the history, you only get to do what you want before you push your changes
> to someone else.  No surgery after you let the cat out of the bag.
> 
> You have a repository and you haven't propogated all this stuff to
> someplace else, then this is easy, we'll just rebuild the history minus A.
> You or I could write a shell script using BK commands to do it in about
> 10 minutes.
> 
> It's a distributed, replicated file system which uses the revision history
> to propogate changes.  If you don't care about talking to anyone else,
> you can do whatever you want.  If you want to give someone your history
> and then change it, no way.  That's rewriting what happened.
> 
> Now does it make more sense?

Sigh...  OK, let me try to put it in a different way.

I don't want A (or entire old path) to disappear.  What I want is ability
to have two paths leading to the same point + ability to mark one of
them as "more interesting".

I.e. the result I want is _two_ sets of changesets with the same compositions.

And _that_ is compatible with replication - I simply want the new path in
revision graph to be propagated.  Along with the "this path is more
interesting" being written on the new one.

Can that be done?  I want a way to re-split the set of deltas.  I'm perfectly
happy with old one staying around, as long as we remember that results of
old and new are the same object and that new is a prefered way to look at
the damn thing.

I suspect that it could be doable with with something as simple as "if you
ask to merge two identical nodes, I'll just mark them as such and ask which
is more interesting".  IIRC, BK doesn't require tree topology on the graph -
it can have converging branches.

_If_ that is feasible - the rest can be scripted around it.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 01:02:53AM -0500, Alexander Viro wrote:
> I don't want A (or entire old path) to disappear.  What I want is ability
> to have two paths leading to the same point + ability to mark one of
> them as "more interesting".
> 
> I.e. the result I want is _two_ sets of changesets with the same compositions.

Ahh, you want LODs.  And they neatly solve the problem you described.
And a bunch of others.  Think of a LOD as a revision history graph.
Imagine being able to create a new, empty (or partially populated)
"container".  That container is a LOD.  You can do set operations from
one LOD to the other.  They are a lot like branches except that they
themselves can branch & merge.

The way that we'd do what you wanted is you'd create a new LOD, 
stick B, C, D, E into it, and make it the default LOD in your 
repository.  

LODs have some very nice attributes - each change is a set element, the
LOD is nothing more than a recorded history of what set elements are in
this LOD, and you can cherry pick from one LOD to the other.  Out of
order, sparsely, whatever.

The only restriction is that you have to have all the changes in your
graph.  There is no concept of a sparse graph.  You can trim off stuff
that happens after some point but you can't remove points in the middle,
even if they are in the other LOD.  Is that OK?

Linus first sounded like he'd accept this as an answer and then later it
fell out of favor because even though he could hide a bad changeset in
another LOD, he didn't want it in the graph at all.  I don't know how
to do that.

The other gotcha is that LODs are only partially implemented and are 
going to stay that way until we achieve concensus on how BK should
work for you.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Wed, Jan 30, 2002 at 09:08:35PM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> > Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.
> 
> If you just want to make A's changes go away, that's trivial:
> 
> 	bk get -e -xA foo.c
> 	bk delta -y"kill A's changes"
> 
> all done.

The real problem with this approach is it leads to information overload.

Go look at linuxppc_2_4_devel with sccstool and try to track major 
changes over the last 6 months. 

You can't. You are completely overwhelmed by the day-to-day thrashing of 
'bug fix this, bug fix that', and all the lines on the screen from the 
wacky merges we wind up doing in that tree.

I think what Linus and Viro really want is not to rewrite history
(although there are times when it would be nice), but say "I don't think
this change is worth looking at". Keep the data in the database, because
you have to to maintain consistency, but don't let me see it unless I ask
3 times, and say please.

This is the reason why I have a 'linuxppc_2_4_galileo' tree.. it's got a 
bunch of crap that will *only* ever be usefull to archeologists who want 
to figure out my or someone else's thought process. The code is ugly. The 
only reason I have another tree is because I want to work with other 
people and keep a record for myself in case I break something in the 
process.

Now, because I don't want to pollute the main tree, once this works, I'm 
going to merge this stuff manually with dirdiff over to 2_4_devel, and 
check it in.

And you're going to lose that information anyway (well, except on 
openlogging.org, but that's just comments), because once I have this 
working and checked into _devel, the _galileo tree is going to get 
deleted because nobody cares.

It is usefull to note that the human brain has evolved the capacity to
forget. Yes, sometimes it makes a mistake and forgets the wrong thing,
but let's not be too quick to forget there may be a lesson here, and 
everything is NOT always worth keeping. 

If you don't have some capacity to 'lose' information in your distributed
database, there will come a point in time that it will stop being
scalable. Either moore's law may break, or information will be going in
faster than you can keep up by depending on faster CPU's/more memory/disk
space to handle all that history.

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

Troy said:
> The real problem with this approach is it leads to information overload.
> 
> Go look at linuxppc_2_4_devel with sccstool and try to track major 
> changes over the last 6 months. 
> 
> You can't. You are completely overwhelmed by the day-to-day thrashing of 
> 'bug fix this, bug fix that', and all the lines on the screen from the 
> wacky merges we wind up doing in that tree.

I agree with this and the rest of your message.  Here's what we are doing
to address it:

a) I have a version of BK where revtool (aka sccstool) shows only the 
   tagged releases.  It's cool.  It also has a feature where you can
   select a node and ask it to color all versions which contain this
   node (seems like you'd never need that until you see a heavily used
   BK tree like Troy has).

b) part of the problem is the "merge" deltas in the ChangeSet file.
   They really need to be hidden or removed completely.  As a side
   effect of making the ChangeSet file more flexible a la Linus' 
   requests (doesn't give all that he wants but part of it), I 
   think these will go away.

c) LODs.  One thing a LOD can do for you is to allow you to have your
   private LOD into which you do a ton of changes.  Then you can do
   a "rollup include" into a public LOD, like the PPC LOD.  We then
   give you a LOD aware revtool and the information overload starts
   to go away (but preserves the information if you need it).

> I think what Linus and Viro really want is not to rewrite history
> (although there are times when it would be nice), but say "I don't think
> this change is worth looking at". Keep the data in the database, because
> you have to to maintain consistency, but don't let me see it unless I ask
> 3 times, and say please.

If we could agree that this is true, I'm ecstatic.  BK needs at least part
of what you said to be true.  If you can convince Linus that it doesn't 
matter if the data is there as long as his view is clean, that solves some
of the nasty problems.

That said, I'm sympathetic to the "I make lotso changes and I want to 
collapse them into one big change" problem.  It's certainly technically
possible to make BK do that, but then you have to *know* that nobody
else has a BK repo with your old detailed changes in it, or if they 
do, they won't ever try to push them back to you (or Linus or ...).  
It's not an error if they do, it's just that BK will view them as 
different changes and automerge them right back into the history.
So then you'll have both the collapsed version and the detailed version
which puts you worse off than when you started.

That's the whole issue with the "history rewrite".  I'll give you history
rewrite, but you need to understand what it means.  I think the current
BK users get it.  I think the BK future users don't get that it is all
one big replicated distributed slightly (or not so slightly) out of sync
database that wants to sync up when it can.  So if you rewrite history,
BK has no way of knowing that you did that.  I suppose we could teach
though so that it would reject the uncollapsed changes but that has its
own issues.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Thu, Jan 31, 2002 at 02:50:31AM -0500, Rob Landley wrote:
> On Thursday 31 January 2002 01:37 am, Larry McVoy wrote:
> 
> > That said, I'm sympathetic to the "I make lotso changes and I want to
> > collapse them into one big change" problem.  It's certainly technically
> > possible to make BK do that, but then you have to *know* that nobody
> > else has a BK repo with your old detailed changes in it, or if they
> > do, they won't ever try to push them back to you (or Linus or ...).
> 
> No, bitkeeper simply has to know. :)
> 
> Put in a node that says "this change collapses this range of other changes" 
> with a range or list of change IDs, and then when you do your next merge with 
> another tree, bitkeeper has the info it needs to avoid sucking in dupes.  If 
> the node says you have that change already, you don't need to suck it in from 
> the other tree.
> 
> > It's not an error if they do, it's just that BK will view them as
> > different changes and automerge them right back into the history.
> > So then you'll have both the collapsed version and the detailed version
> > which puts you worse off than when you started.
> 
> Just teach BK that the collapsed version includes everything in the detailed 
> version.  (Even if that's not technically true, teaching one system to lie to 
> another is an important part of programming... :)  Linus wanted checkpoint 
> functionality to limit backmerges, this seems sort of related-ish.  
> (Boundaries on change sets, merging change sets...)
> 
> Is there an implementation reason why this is particularly hard, or some evil 
> nasty side effects to such an approach that we should know about?


Can you detect the 'collapsed vs full version' thing, and force it to be 
a merge conflict? That, and working LOD support would probably get most 
of what I want (until I try the new version and find more stuff I want 
:P)

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> Can you detect the 'collapsed vs full version' thing, and force it to be 
> a merge conflict? That, and working LOD support would probably get most 
> of what I want (until I try the new version and find more stuff I want 
> :P)

Are you sure you want that?  If so, that would work today, it's about a
20 line script.  You clone the tree, collapse all the stuff into a new
changeset, and pull.  It will all automerge.  But now you have the detailed
stuff and the non-detailed stuff in the same tree, which I doubt is what
you want.  I thought the point was to remove information, not double it.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Thu, Jan 31, 2002 at 09:19:14AM -0800, Larry McVoy wrote:
> On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> > Can you detect the 'collapsed vs full version' thing, and force it to be 
> > a merge conflict? That, and working LOD support would probably get most 
> > of what I want (until I try the new version and find more stuff I want 
> > :P)
> 
> Are you sure you want that?  If so, that would work today, it's about a
> 20 line script.  You clone the tree, collapse all the stuff into a new
> changeset, and pull.  It will all automerge.  But now you have the detailed
> stuff and the non-detailed stuff in the same tree, which I doubt is what
> you want.  I thought the point was to remove information, not double it.

Well, what I meant was have some kind of pointer in the collapsed stuff 
that conflicts with the detailed stuff, and requires the user to pick 
which on they want. Ideally, this could default to user picks, but a 
repository policy of 'only take collapsed versions' could be used for 
upstream trees, say like linuxppc_2_4.  (linuxppc_2_4_devel could take 
detailed versions).

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

cc list trimmed.

On Thu, 31 Jan 2002 09:19:14 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
>> Can you detect the 'collapsed vs full version' thing, and force it to be 
>> a merge conflict? That, and working LOD support would probably get most 
>> of what I want (until I try the new version and find more stuff I want 
>> :P)
>
>Are you sure you want that?  If so, that would work today, it's about a
>20 line script.  You clone the tree, collapse all the stuff into a new
>changeset, and pull.  It will all automerge.  But now you have the detailed
>stuff and the non-detailed stuff in the same tree, which I doubt is what
>you want.  I thought the point was to remove information, not double it.

That sounds almost like what I was looking for, with two differences.

(1) Implement the collapsed set so bk records that it is equivalent to
    the individual patchsets.  Only record that information in my tree.
    I need the detailed history of what changes went into the collapsed
    set, nobody else does.

(2) Somebody else creates a change against the collapsed set and I pull
    that change.  bk notices that the change is again a collapsed set
    for which I have local detail.  The external change becomes a
    branch off the last detailed patch in the collapsed set.

Example.

I have individual changes c1-c17 which are not externally visible.

Tell bk to generate collapsed patch A from c1-c17.  A is externally
visible, without the detailed internal change history of c1-c17.  This
is the equivalent of exporting a patch but it is recorded in bk.

I continue development with c18 onwards, based off c17.

Somebody makes change B against A.  B is externally visible.

I pull B.  bk recognises that B is against A for which local data
exists and therefore B is not really against A but is against c17.
bk creates B as a branch against c17, in parallel with c18.

Outside world sees A->B.  I see A[c1-c17], c17->c18 ..., c17->B (two
branches).

That processing model hides all the backtracking and partial checkins
from the outside world, which only sees the final patchset A, no
information overload.  It allows me to continue with internal
development with all the information that I need.  And it allows me to
automatically take back changes, identify that the changes are in
parallel to my internal changes and merge while keeping local details.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
> That sounds almost like what I was looking for, with two differences.
> 
> (1) Implement the collapsed set so bk records that it is equivalent to
>     the individual patchsets.  Only record that information in my tree.
>     I need the detailed history of what changes went into the collapsed
>     set, nobody else does.
> 
> (2) Somebody else creates a change against the collapsed set and I pull
>     that change.  bk notices that the change is again a collapsed set
>     for which I have local detail.  The external change becomes a
>     branch off the last detailed patch in the collapsed set.

This is certainly possible to do.  However, unless you are willing to fund
this development, we aren't going to do it.  We will pick up the costs of
making changes that you want if and only if we have commercial customers
who want (or are likely to want) the same thing.  Nothing personal, it's
a business and we make tradeoffs like that all the time.

Collapsing is relatively easy, it's tracking the same content in two
different sets of deltas which is hard to get exactly correct.  Certainly
possible but I can visualize what it would take and it would be messy and
disruptive to the source base for an obscure feature that is unlikely to
be used.

Why don't you actually use BK for a while and see if you really think
you need this feature.  The fact that our customers aren't clamoring for
it should tell you something.  They do work as hard and on as much code
(in many cases on the same code) as you do.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 17:04:28 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
>> That sounds almost like what I was looking for, with two differences.
>> 
>> (1) Implement the collapsed set so bk records that it is equivalent to
>>     the individual patchsets.  Only record that information in my tree.
>>     I need the detailed history of what changes went into the collapsed
>>     set, nobody else does.
>> 
>> (2) Somebody else creates a change against the collapsed set and I pull
>>     that change.  bk notices that the change is again a collapsed set
>>     for which I have local detail.  The external change becomes a
>>     branch off the last detailed patch in the collapsed set.
>
>This is certainly possible to do.  However, unless you are willing to fund
>this development, we aren't going to do it.  We will pick up the costs of
>making changes that you want if and only if we have commercial customers
>who want (or are likely to want) the same thing.  Nothing personal, it's
>a business and we make tradeoffs like that all the time.

Understood.

>Collapsing is relatively easy, it's tracking the same content in two
>different sets of deltas which is hard to get exactly correct.  Certainly
>possible but I can visualize what it would take and it would be messy and
>disruptive to the source base for an obscure feature that is unlikely to
>be used.
>
>Why don't you actually use BK for a while and see if you really think
>you need this feature.  The fact that our customers aren't clamoring for
>it should tell you something.  They do work as hard and on as much code
>(in many cases on the same code) as you do.

This is the way that I use PRCS now and it fits the diff/patch model
for distributing kernel code that most people are used to, while
reducing the concerns about information overload.

With PRCS I have branches galore with lots of little changes.  The
outside world sees complete patch sets, not the individual changes.
When they send a patch back I work out which internal change it is
against and start a new branch against it.  The downside with PRCS is
that the creation of the patch set and storing on an ftp site is a
manual process, as is identifying which internal change a patch
response is against and starting a new branch against the last internal
change.

If bk could automate the creation and tracking of meta patchsets I
would convert tomorrow, the ability to automatically distribute changes
is what I miss in PRCS.  But if using bk means that I cannot
automatically separate and track the internal and external patches then
there is no benefit to me in converting.  If I have to clone a
repository to roll up internal patches into an external set and I
cannot automatically pull changes against the external set back into my
working repository then bk gives me no advantages.

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:
> On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
> > That sounds almost like what I was looking for, with two differences.
> > 
> > (1) Implement the collapsed set so bk records that it is equivalent to
> >     the individual patchsets.  Only record that information in my tree.
> >     I need the detailed history of what changes went into the collapsed
> >     set, nobody else does.
> > 
> > (2) Somebody else creates a change against the collapsed set and I pull
> >     that change.  bk notices that the change is again a collapsed set
> >     for which I have local detail.  The external change becomes a
> >     branch off the last detailed patch in the collapsed set.
> 
> This is certainly possible to do.  However, unless you are willing to fund
> this development, we aren't going to do it.  We will pick up the costs of
> making changes that you want if and only if we have commercial customers
> who want (or are likely to want) the same thing.  Nothing personal, it's
> a business and we make tradeoffs like that all the time.

I wonder how your commercial customers develop code then. Either each
programmer futzes around in his/her own tree, and then creates a patch (or
some such) for everybody to see (then I don't see the point of source
control as a help to the individual developer), or everybody sees all the
backtracking going on everywhere (in which case the repository is a mostly
useless mess AFAICS).
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Fri, 1 Feb 2002, Horst von Brand wrote:

> I wonder how your commercial customers develop code then. Either each
> programmer futzes around in his/her own tree, and then creates a patch
> (or some such) for everybody to see (then I don't see the point of
> source control as a help to the individual developer), or everybody
> sees all the backtracking going on everywhere (in which case the
> repository is a mostly useless mess AFAICS).

If the object is to minimise confusion by not showing
back-tracked changes, why not simply allow the user
to mark changesets with a "visibility":

1) hidden, for stuff which shouldn't be seen by default,
   like backed out changes, etc..
2) small, individual development steps to achieve a new
   feature
3) normal, the normal commits
4) major (tagged versions ?)

This way the user can select how detailed the overview
of the versions should be.

Also, when viewing a changeset/version of a certain
priority, bitkeeper could optionally "fold in" the
hidden changesets between the last changeset and the
one the user wants to view.

Would this be a workable scheme ?

(keeps the bitkeeper repository intact, can reduce
the confusion)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 09:30:56AM -0200, Rik van Riel wrote:
> On Fri, 1 Feb 2002, Horst von Brand wrote:
> If the object is to minimise confusion by not showing
> back-tracked changes, why not simply allow the user
> to mark changesets with a "visibility":

That's what LODs do.  You can do all your work in your "branch", when you
are ready, you do a branch-to-branch pull which collapses the view of all
your changesets down to one in the other view.

I'd love it if you could get Linus to buy into this as an acceptable answer.

I do agree that there are times when you really want to collapse a pile
of changes into one and I'm willing to write that code if it becomes
agreed that it is widely useful.  It's maintaining both versions of 
the changes, the collapsed and uncollapsed, that I don't want to do.
That would be a nightmare in the source base and I don't believe there
is substantial real benefit.  Either the changes are valuable or they
aren't.  If they are valuable enough that you want to save them then
you should let the rest of the world see them.  If they aren't, then
they aren't.  I'm sure you can find cases that don't match that view
but I'm equally sure they are a very small percentage.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Fri, 1 Feb 2002 08:43:27 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>I do agree that there are times when you really want to collapse a pile
>of changes into one and I'm willing to write that code if it becomes
>agreed that it is widely useful.  It's maintaining both versions of 
>the changes, the collapsed and uncollapsed, that I don't want to do.

Hand waving solutions without seeing the bk code ...

Don't maintain both versions of the changes.

Everybody except the person with the uncollapsed set only sees the
collapsed set, no problem there.

The person with the uncollapsed set only has one version of the
changes, in the uncollapsed set.

Identify each externally visible patchset with a unique id, you
probably already do this.

Define a meta patchset entry which maps the uncollapsed line of patches
to the collapsed set.  That is, don't duplicate the changes, add a
mapping instead.

The meta entry maps the externally visible patchset to the internal set
by listing just the start and end point of the LOD.  These are the
entries that would be given to export patch.

Add a global repository option, show/hide detail.  The default is show
detail, the current behaviour.

For show detail, you cannot use meta patchset entries.  Everything is
visible to the rest of the world.

For hide detail, you must define meta entries for what you want to be
visible, the rest of the world can only see what you expose.  That is,
you cannot publish some detail entries and some meta entries, you must
choose one or the other at the repository level.

Meta sets are single level, no collapse within collapse.  A meta set is
externally visible, to collapse a meta set into a meta-meta set is
equivalent to rewriting the distributed bk history, don't allow that.

No duplication of changes.  No rewriting of bk history.  Users who want
to hide their detail changes and only expose the result can do so.
Users who only check in working (as opposed to hacking) code are not
affected.  Users who want the extra flexibility incur the cost of
defining meta sets before they can publish anything.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 12:11:30PM +0100, Horst von Brand wrote:
> > This is certainly possible to do.  However, unless you are willing to fund
> > this development, we aren't going to do it.  We will pick up the costs of
> > making changes that you want if and only if we have commercial customers
> > who want (or are likely to want) the same thing.  Nothing personal, it's
> > a business and we make tradeoffs like that all the time.
> 
> I wonder how your commercial customers develop code then. Either each
> programmer futzes around in his/her own tree, and then creates a patch (or
> some such) for everybody to see (then I don't see the point of source
> control as a help to the individual developer), or everybody sees all the
> backtracking going on everywhere (in which case the repository is a mostly
> useless mess AFAICS).

You are presupposing that all the developers are checking in many bad changes
and only one good change.  And that all the bad changes are obscuring the
good ones.  That a correct statement of your beliefs?

If so, what you are describing is called "hacking" in the negative
sense of the word, and what my customers do is called "programming".
It's quite rare to see the sort of mess that you described, it happens,
but it is rare.  I don'tknow how else to explain it, but it is not the
norm in the professional world to try a zillion different approaches
and revision control each and every one.

The norm is:
	clone a repository
	edit the files
	modify/compile/debug until it works
	check in
	push the patch up the shared repository
I'm really at a loss as to why that shouldn't be the norm here as well.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Wayne Scott]

From: Larry McVoy <lm@bitmover.com>
> If so, what you are describing is called "hacking" in the negative
> sense of the word, and what my customers do is called "programming".
> It's quite rare to see the sort of mess that you described, it happens,
> but it is rare.  I don'tknow how else to explain it, but it is not the
> norm in the professional world to try a zillion different approaches
> and revision control each and every one.
> 
> The norm is:
> 	clone a repository
> 	edit the files
> 	modify/compile/debug until it works
> 	check in
> 	push the patch up the shared repository

Or they create do a line of development in a repository with commits
and then determine that it wasn't working.  No problem throw it away
and start over from a clean copy.  Since the repositories are
distributed, private branches just disappear if you don't like them.

-Wayne

Re: A modest proposal -- We need a patch penguin

[Nix N. Nix]

On Thu, 2002-01-31 at 12:19, Larry McVoy wrote: 
> On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> > Can you detect the 'collapsed vs full version' thing, and force it to be 
> > a merge conflict? That, and working LOD support would probably get most 
> > of what I want (until I try the new version and find more stuff I want 
> > :P)
> 
> Are you sure you want that?  If so, that would work today, it's about a
> 20 line script.  You clone the tree, collapse all the stuff into a new
> changeset, and pull.  It will all automerge.  But now you have the detailed
> stuff and the non-detailed stuff in the same tree, which I doubt is what
> you want.  I thought the point was to remove information, not double it.

Sounds to me like you should have the /option/ to double your info,
which does not mean that the whole world should start seeing your stuff
double. You must "fool" the other trees into believing that you are the
second Mozart (you get everything right the first time around) and only
to yourself will you admit that it took 15 different tries and you
dead-ended yourself 15 different ways.  Under these conditions you would
have all your blunders documented, but only for yourself.



Regards.

> -- 
> ---
> Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 10:51 pm, Larry McVoy wrote:

>Sigh.  OK, I'm taking one more pass at trying to get you to see the light
>and then I give up.

And I'm trimming the cc list back a bit...

> > I'm not quite sure how Linus does this, but how I'd do it is [a really
> > complicated solution based on patches that won't work]

A) It's not really all that complicated.  The minimal commands are "untar" 
and "patch".  If you don't feel like doing fixups at all (with a third 
command, an editor), the rejected patches can be bounced.  It does mean 
you're keeping track of patches manually, but I'm just talking about what 
happens between -pre releases, which is not an infinitely large set of 
patches to wrangle...

B) It's a solution that is working, and has been for years.  It certainly has 
its disadvantages, but it does in fact function.  What the existing process 
is used to do on a regular basis is the definition of the minimum level of 
functionality I would expect from bitkeeper.

I'm not defending the patch method.  I'm certainly not advocating it as the 
ultimate solution.  But it has worked for years, and a replacement that can't 
easily do things it can easily do seems flawed to me.

If you'd like to say that I'm wrong about bitkeeper and that it CAN easily do 
the above.  (That's kind of the reply I expected.)  But attacking the 
desirability of doing things that the existing patch managment process 
handles on a fairly regular basis...  Seems counter-productive to me.

> Think.  What you are describing is basically what Linus does today.  And
> noone, including you, is happy.  You're the guy who started this thread.

I wasn't trying to get Linus to use CVS.  That's just a fringe benefit.  I 
was trying to figure out how to deal with patches getting dropped to the 
point people were suggesting automatic remailers.  I didn't directly suggest 
changing what Linus did with patches on his own hard drive.  I went out of my 
way to craft a proposal where he DIDN'T have to change any of that.  I was 
mostly talking about the method by which patches got to him, which is where I 
(right or wrong) percieved the existence of a problem.

You can say I suggested the wrong solution, and you can say I identified the 
wrong problem.  But please don't project your desires onto me and try to use 
that to explain why you think I should be agreeing with you.

To clarify: I'm HAPPY to see Linus giving bitkeeper another try.  I think 
this is an EXTREMELY positive development.  I do NOT want to discourage it.  
I also don't want to see it fail because you expect Linus to adapt to 
bitkeeper rather than trying to understand what Linus does and adapt 
bitkeeper to be a good tool for him.

(Some of these are simple documentation questions.  How does Linus do THIS.  
And I intend to go RTFM when I have time to tackle a new project (I.E. not 
tonight), but I'm mostly following up on your replies to questions other 
people originally asked, which I didn't consider to be a real answer to the 
question.)

> However, what you described *completely* misses the point.

The message you're replying to was answering your question, which to me 
seemed to be you changing the subject form the earlier "people have to send 
linus patch A in order to send him a bitkeeper change set with patch B, even 
if they don't want to".  I wasn't talking about  what goes on within Linus's 
tree, it's about what people send to him.  (Bitkeeper change sets seem to 
have descriptions and potentially automatic notification back to the original 
patch submitter that they got looked at and/or applied, which is a potential 
improvement over raw text patches.  The ordering requirements do not seem to 
me to be a pure improvement, but instead something that takes extra effort to 
work around.)


> Do you start to see the problem?  You were yelling and screaming
> "BitKeeper sucks because it can't take a patch out" when in fact
> it can do exactly what you said it can't.

Yelling and screaming?  Was I?  (I think you read too much into the rant tag. 
 The point of one of those things is "not everybody is expected to agree with 
me on this"...)

There's a difference between "if you can't take the patch out, then maybe 
bitkeeper sucks" and "bitkeeper definitely sucks because I just KNOW you 
can't..."  You seem to have missed the conditional and homed right in on 
defending your baby from hypothetical criticism.  Maybe I'm not expressing 
myself clearly.  It's happened before...

You just answered my question: bitkeeper can take the patch out, the problem 
isn't a real problem.  Thanks, that's what I wanted to know.

> On top of that, what you
> were complaining about isn't the point.  The thing that you say
> BK can't do, but it can, is not what Linus wants.  Not even close.
>
> And you haven't begun to understand that BK is a distributed, replicated
> system.  You can turn all that off and you've got CVS, so turning it
> off isn't an option.

If you turn off the distributed/replicated nature of bitkeeper, it acquires 
problems with file renames and reproducibility?  Without being distributed, 
you have problems with file renames and reproducibility?  Several other 
developers use a CVS or BK repositories which nobody else ever directly 
checks any code into.  They import patches, and then do their own development 
in that tree.  The source management system is purely for their own 
convenience.  Code goes to other developers as unified diffs.

I'm not talking about fun bells and whistles with which Linus could IMPROVE 
his process.  I'm trying to make sure there are no holes that stop him from 
doing what he could do before.  (There SHOULDN'T be.  But if there aren't, 
why did it take so long to adopt?  Seems worth a check.  Several people who 
use bitkeeper have said things along the lines of "I want to do this and 
haven't figured out how", and you actually seem to have replied "no you don't 
really want to do that" on more than one occasion.  I'm hoping this is a 
miscommunication, which is why I'm trying to follow up.  I mostly just wanted 
clarification and reassurance...)

> Leaving it on means that the revision history is replicated.

You seem to have an unquestioning assumption that infinitely replicating the 
revision history is a good thing.

Linus himself seems to have denied this assumption.  Just because a 
downstream developer took two months to come up with a subsystem and did 
eight hundred seperate checkins of which only maybe 20% of the code made it 
into the final version, does NOT mean that Linus cares.  That level of detail 
IS more information, but there's a limit to how much information you want 
before it becomes cruft in Linus's tree.

> So it isn't an option to collapse a pile of changes into a smaller pile,
> the bigger pile will come back the next time he updates
> with the other tree.

And I'm saying I'm not convinced this is necessarily a good thing.

> And once again, you come back with another post that shows you just want
> to yell

I occasionally use caps to emphasize a word because you can't put italics or 
underlines in non-html email.  I do not however PUT MULTIPLE CAPITALIZED 
WORDS TOGETHER.  That is shouting, yes...)

> and you haven't thought it through, into my kill file you go,

By all means.  I've made it back into Al Viro's now.  (I'm a bit confused how 
I managed to get OUT of it, but I probably changed email addresses...)

Generally, I don't consider putting my fingers in my ears and going "la la la 
I can't hear you" to be the logical equivalent to proving your point in an 
argument.  When I don't feel progress can be made, I generally just stop 
replying.

> my blood pressure goes down, you get to yell all you want, everybody
> is happy.  I'm willing to try and make BK do what is needed here; I'm
> not willing to tolerate people who don't think.

And of course the other person not seeing your point is always entirely 
because they're all not thinking?  There's no possibility of legitimate 
miscommunication, or legitimate difference of opinion about anything?

That must save time.

Rob

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 00:16:13 -0500, 
Rob Landley <landley@trommello.org> wrote:
>Linus himself seems to have denied this assumption.  Just because a 
>downstream developer took two months to come up with a subsystem and did 
>eight hundred seperate checkins of which only maybe 20% of the code made it 
>into the final version, does NOT mean that Linus cares.  That level of detail 
>IS more information, but there's a limit to how much information you want 
>before it becomes cruft in Linus's tree.

I agree with Rob on this.  My PRCS tree for 2.4 has 950+ patch sets in
it but there is no way I would inflict that level of detail on the rest
of the world.  I follow the model of check in often, even if it does
not work, so I can backtrack and take another branch if the code hits a
dead end.

Lots of small checkins and branching means a lot of history which is
useful to me but to nobody else.  When I release a patch I pick a start
point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  That single
patch against 2.4.17 is all the outside world needs to see, the only
history is "kdb v2.1 2.4.17 common-2".  It does not matter if I
backtracked and discarded some twigs to get to that final end point,
the rest of the world only cares about the end point.

For that model to work (which is effectively what diff/patch does now),
developers need a repository system that can consolidate lots of little
changes into a single patchset.  Distribute and replicate the single
patchset, only the original developer retains the individual steps that
made up the combined patchset.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
> When I release a patch I pick a start
> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  

bk export -tpatch -r17.1,17.37

Does exactly the same thing.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 21:55:23 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
>> When I release a patch I pick a start
>> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
>> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  
>
>bk export -tpatch -r17.1,17.37
>
>Does exactly the same thing.

Now you've confused me :).  Does that replicate the history or not?

I know that bk can generate a patch which is fine for people not using
bk, but one of the selling points of bk is the ability to replicate
history entries.  My point is that full replication of history may be
too much detail for anybody except the original developer.  If bk can
consolidate a series of patchsets into one big patchset (not patch)
which becomes the unit of distribution then the problem of too much
history can be solved.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 05:03:11PM +1100, Keith Owens wrote:
> On Wed, 30 Jan 2002 21:55:23 -0800, 
> Larry McVoy <lm@bitmover.com> wrote:
> >On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
> >> When I release a patch I pick a start
> >> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
> >> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  
> >
> >bk export -tpatch -r17.1,17.37
> >
> >Does exactly the same thing.
> 
> Now you've confused me :).  Does that replicate the history or not?

Nope.

> I know that bk can generate a patch which is fine for people not using
> bk, but one of the selling points of bk is the ability to replicate
> history entries.  

Yup.

> My point is that full replication of history may be
> too much detail for anybody except the original developer.  If bk can
> consolidate a series of patchsets into one big patchset (not patch)
> which becomes the unit of distribution then the problem of too much
> history can be solved.

If all you mean is that you don't want to have to tell it what to send,
yes, it does that automatically.  If you start with 100 changes, 
I clone your tree, you add 200 more, all I do to get them is say

	bk pull

it will send them all, quickly (works very nicely over a modem or 
a long latency link like a satellite).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 22:07:20 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 05:03:11PM +1100, Keith Owens wrote:
>> My point is that full replication of history may be
>> too much detail for anybody except the original developer.  If bk can
>> consolidate a series of patchsets into one big patchset (not patch)
>> which becomes the unit of distribution then the problem of too much
>> history can be solved.
>
>If all you mean is that you don't want to have to tell it what to send,
>yes, it does that automatically.  If you start with 100 changes, 
>I clone your tree, you add 200 more, all I do to get them is say
>
>	bk pull
>
>it will send them all, quickly (works very nicely over a modem or 
>a long latency link like a satellite).

AFAICT this is the heart of Rob's problem.  He (and I) do not want you
to see all 200 changes.  Some changes are dead ends, some are temporary
bug fixes that I know will be replaced later, IOW they are my jottings,
not for public release.  Replicating the lot to everybody is just
polluting the other trees.

OTOH if I can tell bk :-

* Take all the changes in the direct line from 17.1 to 17.37.
* Ignore any extraneous branches off that line.
* Ignore other changes that were applied in the same time period but
  are not on the direct line, I am also making changes to 2.4.18-pre6
  at the same time.
* Generate a consolidated patchset which is visible to the outside
  world.
* Hide anything not explicitly marked as visible.
* When the consolidated patchset comes back from the master tree,
  recognise that it is equivalent to 17.1 through 17.37 on my tree,
  even though nobody else has the individual changes.

Then I can choose to make kdb v2.1 2.4.17 common-2 visible as an
entity (17.1 to 17.37), without telling everybody else what other
changes are going on in my tree.  bk pull only sees the consolidated
changes I want to make visible.

Re: A modest proposal -- We need a patch penguin

[Kenneth Johansson]

Larry McVoy wrote:

> On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> > The way BK works now, if we call the quick-and-dirty fix "A", and the real
> > fix "B", the developer has a really hard time just sending "B" to me. He'd
> > have to re-clone an earlier BK tree, re-do B in that tree, and then send
> > me the second version.
> >
> > I'm suggesting that he just send me B, and get rid of his tree. There are
> > no dependencies on A, and I do not want _crap_ in my tree just because A
> > was a temporary state for him.
>
> And you just lost some useful information.  The fact that so-and-so did
> fix A and then did B is actually useful.  It tells me that A didn't work
> and B does.  You think it's "crap" and by tossing it dooms all future
> developers to rethink the A->B transition.
>

I think Linus meant that A never got sent out before the developer did the B
version. Now A could be a even bigger bug than what it was intended to fix so the
developer really dont wan't the world to se that sucker but can't just send the B
changeset as it depends on A. So I guess he needs a easy way to make A just go
away. Basically just collaps A and B into the same changeset. This should probably
ony work on changeset that has not been pushed to other trees.

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Thu, Jan 31, 2002 at 09:09:06AM +0100, Christoph Rohland wrote:
> Hi Linus,
> 
> On Wed, 30 Jan 2002, Linus Torvalds wrote:
> > it would see how far back it can go with an automatic merge and add
> > "d" at the _furthest_ point possible. 
> 
> No, I would prefer a way where the developer gives the merge point and
> bk checks if it merges cleanly. Else it is too easy to have merge
> points which are semantically wrong.

Well, provided the 'backmerge' respects tag, or certain kinds of tags
(ie the tree is 'soft tagged' as v2.5.4-pre3, v2.5.4-pre2, v2.5.4-pre1
and 'hard tagged' as v2.5.3.  'backmerge' will attempt to move a change
back only as far as v2.5.3, since v2.5.3 had an API change here.

Or the other option, since this isn't the _default_ behavior, but an
optional one is to give backmerge a 'don't go past tag' since the
developer should be aware that the API changed at v2.5.3 or v2.5.4-pre2
and even tho the change might apply cleanly further back, since it's
updating the driver to the new API, don't try anyways.)

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> and then you added one change below that, multiple times.  If you were to
> combine all of those changes in a BK tree, it would look like
>
> 			[older changes]
> 			      v
> 			  [2.5.3-pre4]
> 			      v
> 			  [2.5.3-pre5]
>   [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]

I'm porting rmap to 2.5 now, doing just this.

One thing I noticed was that the space usage of all the
bk trees I'm using in order to keep the different changes
individually pullable is about 1.5 GB now.

Not too big an issue for me, but it might be an issue if
every bkbits.net user starts doing this ... ;)

cheers,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Tue, Feb 12, 2002 at 08:59:34PM -0200, Rik van Riel wrote:
> One thing I noticed was that the space usage of all the
> bk trees I'm using in order to keep the different changes
> individually pullable is about 1.5 GB now.

We just need the "bk relink" command and that should be ~100M or so.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Feb 12, 2002  20:59 -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > and then you added one change below that, multiple times.  If you were to
> > combine all of those changes in a BK tree, it would look like
> >
> > 			[older changes]
> > 			      v
> > 			  [2.5.3-pre4]
> > 			      v
> > 			  [2.5.3-pre5]
> >   [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]
> 
> I'm porting rmap to 2.5 now, doing just this.
> 
> One thing I noticed was that the space usage of all the
> bk trees I'm using in order to keep the different changes
> individually pullable is about 1.5 GB now.

Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
that since the rmap changes are fairly localized that you would only
get multiple copies of a limited number of files, and it wouldn't
increase the size of each repository very much.

As Larry mentioned, you could re-merge these trees.  The following script
will probably be enough, since we don't want/need to compare all files
in each tree, only SCCS and BitKeeper files that are in the same place
in the heirarchy.  Very lightly tested - Larry will have to tell me if
it is OK to hard-link everything in SCCS and BitKeeper repositories,
or if there are some files that should be ignored.

On my e2fsprogs tree, it takes 12s to relink a clone to its parent,
saving 12/19MB = 63% reduction in space per clone.

Cheers, Andreas
=========================== bkrelink =====================================
#!/bin/sh
# A script to relink files in BitKeeper repositories if they were not
# created with "bk clone -l" or if the same changes were made to both
# repositories.
#
# Andreas Dilger <adilger@turbolabs.com>  02/12/2002

PROG=bkrelink

usage() {
	echo "usage: $PROG <parent BK tree> <clone BK tree>" 1>&2 && exit 1
}

[ $# -ne 2 ] && usage
[ ! -d "$1/BitKeeper" ] && usage
[ ! -d "$2/BitKeeper" ] && usage

PTREE=$1
CTREE=$2

#DEBUG=1
say() {
	[ "$DEBUG" ] && echo "$*"
	return 0
}

do_link() {
	echo "$PROG: hard-linking $2 to $1"
	ln -f $1 $2
}

# We need to do some ugly things with the find processes to keep the relative
# paths correct in each tree.  Likewise, | read will run in a separate process
# so we need to do the checks in a subshell so all the stat fields are set.
(cd $CTREE
say "$PROG: finding in $CTREE" 1>&2
find . -type d \( -name BitKeeper -o -name SCCS \) ) | while read DIR; do
(cd $CTREE/$DIR
say "$PROG: looking in $CTREE/$DIR" 1>&2
find . -type f ) | while read FILE; do
 	PFILE=$PTREE/$DIR/$FILE
 	CFILE=$CTREE/$DIR/$FILE

	say "$PROG: checking $CFILE, $PFILE"

	[ ! -f "$PFILE" ] && say "$PROG: $PFILE not found" && continue
	[ ! -f "$CFILE" ] && say "$PROG: $CFILE not found" && continue

	[ "$DEBUG" ] && stat -t $PFILE && stat -t $CFILE
	stat -t $PFILE | { read JNK PSZ JNK JNK PUSR PGRP PDEV PINO PLINK JNK
	stat -t $CFILE | { read JNK CSZ JNK JNK CUSR CGRP CDEV CINO CLINK JNK

	# do the easy test (size compare) first
	[ $CSZ != $PSZ ] && say "size mismatch: $CSZ != $PSZ" && continue
	# can't hard link across devices
	[ $CDEV != $PDEV ] && say "dev mismatch: $CDEV != $PDEV" && continue
	# already hard linked (same device number, same inode numer)
	[ $CINO == $PINO ] && say "ino match: $CINO == $PINO" && continue
	[ $CUSR != $PUSR ] && say "user mismatch: $CUSR != $PUSR" && continue
	[ $CGRP != $PGRP ] && say "group mismatch: $CGRP != $PGRP" && continue

	#echo "$PROG: comparing $CFILE, $PFILE"

	cmp --quiet $CFILE $PFILE || continue

	# We try to have only a single target against which we link.
	# If in doubt, move links towards the specified parent.
	if [ $CLINK -eq 1 ]; then
		do_link $PFILE $CFILE
	elif [ $PLINK -eq 1 ]; then
		do_link $CFILE $PFILE
	else
		do_link $PFILE $CFILE
	fi
	}
	}
done
done
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 12 Feb 2002, Andreas Dilger wrote:

> Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
> that since the rmap changes are fairly localized that you would only
> get multiple copies of a limited number of files, and it wouldn't
> increase the size of each repository very much.

the problem is, i'd like to see all these changes in a single tree, and
i'd like to be able to specify whether two changesets have semantic
dependencies on each other or not. BK would still enforce 'hard
orthogonality' - ie. two changesets that change the same line of code
cannot be defined as nondependent on each other, BK should refuse the
checking in of such a changeset. The default dependency should be
something like 'this changeset is dependent on all previous changesets
committed to this repository' - but if the developer wants it then it
should be possible to un-depend two changesets.

it's also true in the other direction: two changesets that have no hard
conflicts could still have semantic dependencies, it's the responsibility
of the developer.

(detail: it might even be possible to define two changesets as orthogonal
even if there are hard conflicts between them. For this to work the
developer has to provide the correct merge in both directions. If that is
done then BK should allow to make the two changesets independent.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Feb 13, 2002  13:07 +0100, Ingo Molnar wrote:
> On Tue, 12 Feb 2002, Andreas Dilger wrote:
> > Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
> > that since the rmap changes are fairly localized that you would only
> > get multiple copies of a limited number of files, and it wouldn't
> > increase the size of each repository very much.
> 
> the problem is, i'd like to see all these changes in a single tree, and
> i'd like to be able to specify whether two changesets have semantic
> dependencies on each other or not.

Oh, I agree.  My response was only to Rik's mention that his multiple trees
take up too much space.  I would personally also want to be able to separate
independent changes out of my tree rather than having many repositories.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

We need to move from discussing whether Linus can scale to whether the 
Linux Community can scale.

Every organization needs to have clearly defined algorithms for 
determining what work is done by who.  For the linux community, our work 
consists in part of reviewing patches.  Incoherent inconsistent 
delegation is the only reason why we are having scaling problems.  We 
have a consistent recurring problem (yes, I know, a few lucky folks like 
me don't have this problem, but it is clear to see that WE as a 
community have this problem).

It is important that there be  a consistent feeling among patch 
submitters that they know where to send their patches for 
acceptance/rejection.  There should be NO patches which go out, and not 
even a rejection comes back.  

Every organization has clearly defined procedures for allocating the 
flow of work.  It is called a management structure.  That is what we 
need, and we need a formal well defined and externally visible one.  An 
informal undefined network of friends is just not suitable for an 
organization where the flow of email is as large as it is in the Linux 
Community.

Linus, I would like you to stop saying that you cannot scale to where 
you can read every email, and start determining what it takes to make 
the Linux Community infrastructure underneath you responsive to patches. 
 Bitkeeper is a great start, but you also need to create  a management 
structure and interface that is clearly defined to the external 
community.  Saying that the maintainers list is ignored by you means 
that you need to create something that is not ignored by you.  You also 
need to create a system (bitkeeper can perhaps help, Larry?) for 
tracking who fails to respond to patches, and (after a few warnings) 
remove them as maintainers.  

Our problems are not novel.  Let us apply standard business school 
methodologies to them.

Hans

Re: A modest proposal -- We need a patch penguin

[Mark Hahn]

On Fri, 22 Feb 2002, Hans Reiser wrote:
...
> Our problems are not novel.  Let us apply standard business school 
> methodologies to them.

eh?  "creative destruction"?  "disruptive techologies"?  
hmm, maybe it should have been *factory* vs bazaar.

Re: A modest proposal -- We need a patch penguin

[Randy.Dunlap]

On Fri, 22 Feb 2002, Hans Reiser wrote:

| We need to move from discussing whether Linus can scale to whether the
| Linux Community can scale.

I have to agree with much of what Hans has written here.

and one of the biggest things that would help in this regard, IMHO,
is to (dare I say "require") provide documentation for kernel
API changes or semantics.  "Read the source" or "Use the source"
doesn't scale well either, when 10K kernel developers are
trying to use a new widget in 2.5.4, but they all ask questions
on lkml about how it's done.

Let's keep Documentation/* stuff up to date.  Whether it's in
text or DocBook format doesn't matter much.
Or have web pages for it if that's preferred by the project or
individual(s).

  ~Randy

| Every organization needs to have clearly defined algorithms for
| determining what work is done by who.  For the linux community, our work
| consists in part of reviewing patches.  Incoherent inconsistent
| delegation is the only reason why we are having scaling problems.  We
| have a consistent recurring problem (yes, I know, a few lucky folks like
| me don't have this problem, but it is clear to see that WE as a
| community have this problem).
|
| It is important that there be  a consistent feeling among patch
| submitters that they know where to send their patches for
| acceptance/rejection.  There should be NO patches which go out, and not
| even a rejection comes back.
|
| Every organization has clearly defined procedures for allocating the
| flow of work.  It is called a management structure.  That is what we
| need, and we need a formal well defined and externally visible one.  An
| informal undefined network of friends is just not suitable for an
| organization where the flow of email is as large as it is in the Linux
| Community.
|
| Linus, I would like you to stop saying that you cannot scale to where
| you can read every email, and start determining what it takes to make
| the Linux Community infrastructure underneath you responsive to patches.
|  Bitkeeper is a great start, but you also need to create  a management
| structure and interface that is clearly defined to the external
| community.  Saying that the maintainers list is ignored by you means
| that you need to create something that is not ignored by you.  You also
| need to create a system (bitkeeper can perhaps help, Larry?) for
| tracking who fails to respond to patches, and (after a few warnings)
| remove them as maintainers.
|
| Our problems are not novel.  Let us apply standard business school
| methodologies to them.

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 25 February 2002 12:13 pm, Randy.Dunlap wrote:
> On Fri, 22 Feb 2002, Hans Reiser wrote:
> | We need to move from discussing whether Linus can scale to whether the
> | Linux Community can scale.
>
> I have to agree with much of what Hans has written here.
>
> and one of the biggest things that would help in this regard, IMHO,
> is to (dare I say "require") provide documentation for kernel
> API changes or semantics.  "Read the source" or "Use the source"
> doesn't scale well either, when 10K kernel developers are
> trying to use a new widget in 2.5.4, but they all ask questions
> on lkml about how it's done.
>
> Let's keep Documentation/* stuff up to date.  Whether it's in
> text or DocBook format doesn't matter much.
> Or have web pages for it if that's preferred by the project or
> individual(s).

Random comment:

It's design documentation that's needed.  Looking at the code you can see 
what it's doing, but you sometimes have to read an amazing amount of it to 
even guess at WHY...  The code doesn't always tell you about the author's 
intentions, just the implementation.  And sometimes the code is wrong 
anyway...

I believe the kernelnewbies project is working on this, but haven't been able 
to follow it.  (I just moved back to Austin, am recovering from food 
poisioning, picking an old job back up...  Not following much of anything at 
the moment...)

>   ~Randy

Rob

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Rob Landley wrote:
>
>>Let's keep Documentation/* stuff up to date.  Whether it's in
>>text or DocBook format doesn't matter much.
>>Or have web pages for it if that's preferred by the project or
>>individual(s).
>>
> 
> Random comment:
> 
> It's design documentation that's needed. 

Another random comment:

1. docs.sun.com, ssh sunsite.prv ... man system-xxx
2. AT&T The UNIX operating system.
3. Watch out at O'REILY.

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

I'm on the road, so this reply probably won't go out for a couple days, but 
I'll queue it in my outbox anyway...

On Friday 22 February 2002 11:06 am, Hans Reiser wrote:
> We need to move from discussing whether Linus can scale to whether the
> Linux Community can scale.
>
> Every organization needs to have clearly defined algorithms for
> determining what work is done by who.  For the linux community, our work
> consists in part of reviewing patches.  Incoherent inconsistent
> delegation is the only reason why we are having scaling problems.  We
> have a consistent recurring problem (yes, I know, a few lucky folks like
> me don't have this problem, but it is clear to see that WE as a
> community have this problem).
>
> It is important that there be  a consistent feeling among patch
> submitters that they know where to send their patches for
> acceptance/rejection.  There should be NO patches which go out, and not
> even a rejection comes back.

This is what the various patchbot projects are trying to address.  
(Unfortunately, they seem to have gotten the idea that they need a complete 
solution to all possible cases before deploying anything.  Just a filtered 
patches-only mailing list would be a start.  I'd put one up if I had a 
server, but as I said I'm moving this month.  If nobody else has done it by 
march, I might.)

The current Linux community structure seems to be four tiers.  Developers, 
maintainers, lieutenants, and linus.  Linus listens to lieutenants, 
lieutenants accept from maintainers, and maintainers accept from developers.

The confusion seems to be that until recently, many maintainers didn't know 
who the lieutenants were (who the people Linus actually listens to are, to at 
the very least explicitly reject patches once these people have reviewed, 
approved, and forwarded them).  Hence stuff was getting to maintainers and 
then being dropped when forwarded straight to Linus.  Linus still hasn't 
quite enumerated his lieutenants, but now that people know they exist I 
expect they'll become apparent eventually...

> Every organization has clearly defined procedures for allocating the
> flow of work.  It is called a management structure.  That is what we
> need, and we need a formal well defined and externally visible one.  An
> informal undefined network of friends is just not suitable for an
> organization where the flow of email is as large as it is in the Linux
> Community.

It's not a binary state.  The fact we need a little more structure doesn't 
mean anybody has to start filling out paperwork and blindly following 
procedures. :)

The "a little more structure" could be a "how to submit patches" FAQ entry 
that says:

1) Develop patch, test, get community feedback if necessary to make sure it 
works.
2) Submit to maintainer, get them to review and sign off.  Resolve any issues 
they have before proceeding.
3) Submit to lieutenant (the maintainer will tell you who this is), get them 
to review and sign off.  Resolve any issues lieutenant has before proceeding.
4) Submit to Linus, with appropriate endorsements.

If Linus ignores everybody except Lieutenants, that's probably workable as 
long as he DOESN'T ignore the lieutenants and people know who the lieutenants 
are, and the lieutenants don't ignore the maintainers and the maintainers 
don't ignore the developers.

If Linus has two levels of sturgeon's law filters (maintainers and 
lieutenants) before he has to explicitly reject something, then asking him 
(nicely) to at least reply thumbs up/thumbs down on patches forwarded to him 
("bad idea", "fix this", "do it this way instead") by the dozen or so people 
he trusts shouldn't overload his bandwidth.  (Whether or not he'd actually do 
it is still up to him, but that strikes me as the minimum workable long-term 
solution.)

So a developer would at least know who they have to please next (maintainer, 
lieutenant, or linus) to forward their patch.  It still might be a lot of 
work to go through the long way, and Linus would probably still accept 
interesting patches directly.  But the failure case of "my patch is getting 
ignored" would have a procedure to go through to get explicitly rejected by 
the appropriate person. :)

By the way, sometimes the answer honestly is "I'm busy, submit again after 
2.5.7".  Which is still better than being ignored.  (Stuff like the ALSA 
drivers: "Good idea, not now."  Ok: When?)

> Linus, I would like you to stop saying that you cannot scale to where
> you can read every email, and start determining what it takes to make
> the Linux Community infrastructure underneath you responsive to patches.

Linus (and the lieutenants under him) have not been the ones experiencing the 
problem.  Linus accepts all the patches he wants to, and the lieutenants tend 
not to be ignored.  The top of the pyramid is not where the motivation for 
change is most strongly felt...

>  Bitkeeper is a great start, but you also need to create  a management
> structure and interface that is clearly defined to the external
> community.  Saying that the maintainers list is ignored by you means
> that you need to create something that is not ignored by you.  You also
> need to create a system (bitkeeper can perhaps help, Larry?) for
> tracking who fails to respond to patches, and (after a few warnings)
> remove them as maintainers.

I don't expect Linus needs to do any of the grunt work here.  He just needs 
to sign off on the design and actually use the finished solution.

We've got some time on this.  If Bitkeeper allows Linus to move to a "pull" 
model with his lieutenants, that should allow the system to scale enough to 
get 2.6 out the door.  It's the layers under those guys who need to shuffle 
around to feed their patches into those bitkeeper trees that Linus pulls from.

Of course to make this work, Bitkeeper will somehow need to let Linus 
cherry-pick patches from the bitkeeper trees under him and reject others.  
I've tried to follow the discussion on this front but I'm not convinced it's 
resolved yet.  But as I said, there's time...

> Our problems are not novel.  Let us apply standard business school
> methodologies to them.

If I remember my kernel-traffic summaries correctly, Eric Raymond was saying 
something like this about two years ago.  Something about the boy genius 
effect? :)

> Hans

Rob

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Rob Landley wrote:

>>
>
>Linus (and the lieutenants under him) have not been the ones experiencing the 
>problem.  Linus accepts all the patches he wants to, and the lieutenants tend 
>not to be ignored.  The top of the pyramid is not where the motivation for 
>change is most strongly felt...
>

Partially true in most social systems.  However, the top of the pyramid 
has some substantial motivation to see a better kernel come into 
existence.  I personally feel motivated to ensure that ReiserFS patches 
don't get lost, because those 5-10% improvements really do add up over 
time.  We may also hope that they feel some pride of craftsmanship in 
constructing their social system --- I suspect that pride in what they 
do is more likely to move them.

Hans

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Ingo Molnar wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> > How much of the out order stuff goes away if you could send changes
> > out of order as long as they did not overlap (touch the same files)?
>
> could this be made: 'as long as they do not touch the same lines of
> code, taking 3 lines of context into account'? (ie. unified diff
> definition of 'collisions' context.)

That would be _wonderful_ and fix the last bitkeeper
problem I'm having once in a while.

cheers,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Josh MacDonald]

Quoting Rik van Riel (riel@conectiva.com.br):
> On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > > How much of the out order stuff goes away if you could send changes
> > > out of order as long as they did not overlap (touch the same files)?
> >
> > could this be made: 'as long as they do not touch the same lines of
> > code, taking 3 lines of context into account'? (ie. unified diff
> > definition of 'collisions' context.)
> 
> That would be _wonderful_ and fix the last bitkeeper
> problem I'm having once in a while.

This would seem to require a completely new tool for you to specify which 
hunks within a certain file belong to which changeset.  I can see why
Larry objects.  What's your solution?

-josh

-- 
PRCS version control system    http://sourceforge.net/projects/prcs
Xdelta storage & transport     http://sourceforge.net/projects/xdelta
Need a concurrent skip list?   http://sourceforge.net/projects/skiplist

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 08:59:03AM -0800, Josh MacDonald wrote:
> Quoting Rik van Riel (riel@conectiva.com.br):
> > On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > > On Wed, 30 Jan 2002, Larry McVoy wrote:
> > >
> > > > How much of the out order stuff goes away if you could send changes
> > > > out of order as long as they did not overlap (touch the same files)?
> > >
> > > could this be made: 'as long as they do not touch the same lines of
> > > code, taking 3 lines of context into account'? (ie. unified diff
> > > definition of 'collisions' context.)
> > 
> > That would be _wonderful_ and fix the last bitkeeper
> > problem I'm having once in a while.
> 
> This would seem to require a completely new tool for you to specify which 
> hunks within a certain file belong to which changeset.  I can see why
> Larry objects.  What's your solution?

We actually can go from any line to the changeset which created that line
relatively quickly (milliseconds in hot cache, second or so in cold cache).
And we have a design which has been proven to work in the past at HP which
would allow a fully general out of order, at the changeset level application
of changes.  It's a bit complex to describe here and it has been 6 months 
away from being done for 3 years, so don't hold your breath.  I think the
better short term answer is to fix the false dependency problem, use regular
diff/patch for the places where there really are dependencies, and then do
the completely general one.

We are doing the general solution, which we call lines of development, our
commercial customers want it as well.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Jan 30, 2002  08:59 -0800, Josh MacDonald wrote:
> Quoting Rik van Riel (riel@conectiva.com.br):
> > On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > > On Wed, 30 Jan 2002, Larry McVoy wrote:
> > >
> > > > How much of the out order stuff goes away if you could send changes
> > > > out of order as long as they did not overlap (touch the same files)?
> > >
> > > could this be made: 'as long as they do not touch the same lines of
> > > code, taking 3 lines of context into account'? (ie. unified diff
> > > definition of 'collisions' context.)
> > 
> > That would be _wonderful_ and fix the last bitkeeper
> > problem I'm having once in a while.
> 
> This would seem to require a completely new tool for you to specify which 
> hunks within a certain file belong to which changeset.  I can see why
> Larry objects.  What's your solution?

Please see my other email on this subject (out of order BK CSETs).
One relatively easy solution is "proxy CSETs", which describe on a
per-line basis (or xdelta checksum boundaries, even better) the changes
made in "false parent" CSETs, but do not actually contain the changes.
This allows the reciever to resolve the false dependencies, and also
allows the sender to validate (at proxy CSET creation time) that the
out-of-order CSET they are about to send, in fact, does not depend on
any of the "false parent" CSETs.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Rik van Riel wrote:

> On Wed, 30 Jan 2002, Ingo Molnar wrote:
>
> > could this be made: 'as long as they do not touch the same lines of
> > code, taking 3 lines of context into account'? (ie. unified diff
> > definition of 'collisions' context.)
>
> That would be _wonderful_ and fix the last bitkeeper problem I'm
> having once in a while.

perhaps there should also be some sort of authority needed to allow such
'violation' of current BK rules: while the patches to conflict in terms of
source code file, we can override it and tell it that they really dont
conflict.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Val Henson]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> 
> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
> 
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

I'm one of the ppc people so I don't really count, but...

I've used bitkeeper for the kernel for a year now.

One of the issues in the "network of people" approach is how much time
and effort it takes to maintain a separate tree while waiting for
changes to be merged into the main tree.  Bitkeeper really helps here.
I've been maintaining a tree with significant differences from the
mainline linuxppc tree, and I can pull and merge new changes without
hand-editing a file 99% of the time.  Maintaining my own tree is now a
minor annoyance, instead of a major pain.

-VAL

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> Linus just called you the ext2 maintainer.

Message-ID, please?

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alexander Viro wrote:
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > Linus just called you the ext2 maintainer.
>
> Message-ID, please?

I called you the VFS maintainer ("whether you like it or not" I think I
said. Although I can't find the message right now).

Now, that obviously does imply a certain control over low-level
filesystems, but it really mainly implies a control over the _interfaces_
used to talk the the filesystem, not the filesystem itself.

I personally really wouldn't mind seeing most filesystem patches coming
through Al (and, in fact, in the inode trimming patches that is partly
what as been happening), but I have this nagging suspicion that some
filesystem maintainers would rather eat barbed wire (*).

		Linus

(*) The discussions between Gooch and Al are always "interesting", to name
some names.

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Linus Torvalds wrote:

> Now, that obviously does imply a certain control over low-level
> filesystems, but it really mainly implies a control over the _interfaces_
> used to talk the the filesystem, not the filesystem itself.
> 
> I personally really wouldn't mind seeing most filesystem patches coming
> through Al (and, in fact, in the inode trimming patches that is partly

I would, though.  Inode-trimming is a separate story - it's a massive series of
interface-changing patches (55 chunks already merged, more to follow) and
any help is certainly welcome - it's a friggin' lot of work (and there was
quite a help - from Jeff, Urban, Christoph...)

	However, I really don't want to be in position when patches to fs
internals are fed through me.  I can give comments.  I can do code review.
I can look through the code and discuss VFS/VM/etc. changes that might be
useful.  I can actually decide to do these changes myself.  That's all nice
and dandy, but let's face it - most of filesystem internals patches are pretty
local and fs maintainers _MUST_ be the first recepients of such patches.

	I don't have Alan's patience.  And I don't believe that I can run
a clearinghouse tree for *all* fs patches - it's pretty much guaranteed to end
up with burnout in a month or so.  BTW, IIRC Jeff Garzik had been doing
something similar unofficially, but I've no idea how he feels about giving
it official status.

	Frankly, the only real issue in that thread was that we _do_ need
a tree specifically for small fixes.  Preferably - quickly getting merged
into the main tree.  And that's a hard work - davej seems to be doing that
and I admire the efforts he's able and willing to put into that stuff.
I know that I couldn't pull that off.

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alexander Viro wrote:
>
> 	Frankly, the only real issue in that thread was that we _do_ need
> a tree specifically for small fixes.  Preferably - quickly getting merged
> into the main tree.

A "small stuff" maintainer may indeed be a good idea. The maintainer could
be the same as somebody who does bigger stuff too, but they should be
clearly different things - trivial one-liners that do not add anything
new, only fix obvious stuff (to the point where nobody even needs to think
about it - if I'd start getting any even halfway questionable patches from
the "small stuff" maintainer, it wouldn't work).

		Linus

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:21 am, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Alexander Viro wrote:
> >
> > 	Frankly, the only real issue in that thread was that we _do_ need
> > a tree specifically for small fixes.  Preferably - quickly getting merged
> > into the main tree.
> 
> A "small stuff" maintainer may indeed be a good idea. The maintainer could
> be the same as somebody who does bigger stuff too, but they should be
> clearly different things - trivial one-liners that do not add anything
> new, only fix obvious stuff (to the point where nobody even needs to think
> about it - if I'd start getting any even halfway questionable patches from
> the "small stuff" maintainer, it wouldn't work).

But that's exactly what Arnaldo Carvalho de Melo[1] does, has been doing for 
more than a year, surely you've noticed?  On top of being the nicest guy in 
the world, as far as I can tell.

[1] Most of us call him <acme>.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> A "small stuff" maintainer may indeed be a good idea. The maintainer could
> be the same as somebody who does bigger stuff too, but they should be
> clearly different things - trivial one-liners that do not add anything
> new, only fix obvious stuff (to the point where nobody even needs to think
> about it - if I'd start getting any even halfway questionable patches from
> the "small stuff" maintainer, it wouldn't work).

So if someone you trusted actually started batching up small fixes and 
sending you things like

"37 random documentation updates - no code changed", "11 patches to fix
kmalloc checks", "maintainers updates to 6 network drivers"

that would work sanely ? I think that would actually fix a lot of the stuff 
getting lost right now. Its mostly small stuff, often from new people, or from
folks who met a bug, fixed it and have a totally seperate and rather more 
important (to them) project and deadline to meet that is going walkies.

It also increases bandwidth for sorting out the big stuff.

The other related question is device driver implementation stuff (not interfaces
and abstractions). You don't seem to check that much anyway, or have any taste
in device drivers 8) so should that be part of the small fixing job ?

Alan

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 10:06:35AM +0000, Alan Cox wrote:
> The other related question is device driver implementation stuff (not interfaces
> and abstractions). You don't seem to check that much anyway, or have any taste
> in device drivers 8) so should that be part of the small fixing job ?

I've often dreamt of an overall "drivers maintainer" or perhaps just an
unmaintained-drivers maintainer:  a person with taste who could give
driver patches a glance, when noone else does.
(and no I'm not volunteering :))

	Jeff, dreams on

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Wed, Jan 30, 2002 at 05:18:55AM -0500, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 10:06:35AM +0000, Alan Cox wrote:
> > The other related question is device driver implementation stuff (not interfaces
> > and abstractions). You don't seem to check that much anyway, or have any taste
> > in device drivers 8) so should that be part of the small fixing job ?
> 
> I've often dreamt of an overall "drivers maintainer" or perhaps just an
> unmaintained-drivers maintainer:  a person with taste who could give
> driver patches a glance, when noone else does.
> (and no I'm not volunteering :))

I have had that same dream too, Jeff :)
Especially after spelunking through the SCSI drivers, and being amazed
that only one of them uses the, now two year old, pci_register_driver()
interface (which means that only that driver works properly in PCI
hotplug systems.)

Having someone with "taste" to run driver patches by first would have
been a great help when I started out writing them.  I've been trying to
provide that resource for the new USB drivers.

greg k-h

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> Especially after spelunking through the SCSI drivers, and being amazed
> that only one of them uses the, now two year old, pci_register_driver()
> interface (which means that only that driver works properly in PCI
> hotplug systems.)

I doubt it does actually. The problem with pci register driver and scsi is
that the two subsystems are designed with violently conflicting goals. Once
DaveJ or someone does the proposed scsi cleanups it'll become the natural
not the obscenely complicated way to do a scsi driver, as well as sorting out
the pcmcia and cardbus scsi mess, and the card failed/recovered stuff once and
for all.

Alan

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 06:35:15PM +0000, Alan Cox wrote:
> > Especially after spelunking through the SCSI drivers, and being amazed
> > that only one of them uses the, now two year old, pci_register_driver()
> > interface (which means that only that driver works properly in PCI
> > hotplug systems.)
> 
> I doubt it does actually. The problem with pci register driver and scsi is
> that the two subsystems are designed with violently conflicting goals. Once
> DaveJ or someone does the proposed scsi cleanups it'll become the natural
> not the obscenely complicated way to do a scsi driver, as well as sorting out
> the pcmcia and cardbus scsi mess, and the card failed/recovered stuff once and
> for all.

I disagree... I outlined a workable scheme for hotplugging SCSI
controllers to Justin Gibbs a long time ago, when the new aic7xxx was
first being merged.  Using the new PCI API was fairly easy, handling the
disk-disappearing-from-under-you problem was a bit more annoying :)

	Jeff

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 01:29:39PM -0500, Jeff Garzik wrote:
> I disagree... I outlined a workable scheme for hotplugging SCSI
> controllers to Justin Gibbs a long time ago, when the new aic7xxx was
> first being merged.  Using the new PCI API was fairly easy, handling the
> disk-disappearing-from-under-you problem was a bit more annoying :)

And indeed the aic7xxx driver does handle hotplugging.
It just doesn't handle hot-un-plugging.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 09:11:26AM -0800, Greg KH wrote:
> I have had that same dream too, Jeff :)
> Especially after spelunking through the SCSI drivers, and being amazed
> that only one of them uses the, now two year old, pci_register_driver()
> interface (which means that only that driver works properly in PCI
> hotplug systems.)

Spelunking through the SCSI drivers involves great deal of
bravery.  That pile of dung does not need a "small-stuff"
maintainer.  It needs to be forcefully ejected and replaced with
extreme prejudice.  It is amazing that ancient stuff works as
well as it does...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> bravery.  That pile of dung does not need a "small-stuff"
> maintainer.  It needs to be forcefully ejected and replaced with
> extreme prejudice.  It is amazing that ancient stuff works as
> well as it does...

A lot of the apparently really ugly drivers turned out to be very good code
hiding under 10 years of history and core code changes and
assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
or hang - Linus apparently merged untested stuff to the old broken driver.

Alan

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 11:06:04PM +0000, Alan Cox wrote:
> > bravery.  That pile of dung does not need a "small-stuff"
> > maintainer.  It needs to be forcefully ejected and replaced with
> > extreme prejudice.  It is amazing that ancient stuff works as
> > well as it does...
> 
> A lot of the apparently really ugly drivers turned out to be very good code
> hiding under 10 years of history and core code changes and
> assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
> use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
> or hang - Linus apparently merged untested stuff to the old broken driver.

This is in the latest -ac kernels?  Cool, I'll go take a close
look.  I'm very anxious to see a SCSI layer that doesn't suck
get put in place,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Andre Hedrick]

On Wed, 30 Jan 2002, Erik Andersen wrote:

> On Wed Jan 30, 2002 at 11:06:04PM +0000, Alan Cox wrote:
> > > bravery.  That pile of dung does not need a "small-stuff"
> > > maintainer.  It needs to be forcefully ejected and replaced with
> > > extreme prejudice.  It is amazing that ancient stuff works as
> > > well as it does...
> > 
> > A lot of the apparently really ugly drivers turned out to be very good code
> > hiding under 10 years of history and core code changes and
> > assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
> > use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
> > or hang - Linus apparently merged untested stuff to the old broken driver.
> 
> This is in the latest -ac kernels?  Cool, I'll go take a close
> look.  I'm very anxious to see a SCSI layer that doesn't suck
> get put in place,

Given me another development tree of time to create one and it will be a
done deal, but I have enough to clean up in what is started now.

Cheers,

Andre Hedrick
Linux Disk Certification Project                Linux ATA Development

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 04:48:48PM -0700, Erik Andersen wrote:
 > > assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
 > > use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
 > > or hang - Linus apparently merged untested stuff to the old broken driver.
 > 
 > This is in the latest -ac kernels?

 Even better, it's in 2.4 mainline.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> This is in the latest -ac kernels?  Cool, I'll go take a close
> look.  I'm very anxious to see a SCSI layer that doesn't suck
> get put in place,

The scsi mid layer is a seperate problem, and its getting there already in
2.5. Chunks of nasty scsi special cases keep dissappearing with the bio stuff

The NCR5380 stuff fixes what was an amazingly crufty unmaintained driver

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alan Cox wrote:
>
> So if someone you trusted actually started batching up small fixes and
> sending you things like
>
> "37 random documentation updates - no code changed", "11 patches to fix
> kmalloc checks", "maintainers updates to 6 network drivers"
>
> that would work sanely ?

Yes. That would take a whole lot of load off me - load I currently handle
by just not sweating the small stuff, and concentrating on the things I
think are important.

> The other related question is device driver implementation stuff (not interfaces
> and abstractions). You don't seem to check that much anyway, or have any taste
> in device drivers 8) so should that be part of the small fixing job ?

I think it has some of the same issues, but I really would prefer to have
it in a separate batch.

Quite frankly, this is a large part of what you did..

		Linus

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Wed, 30 Jan 2002, Linus Torvalds wrote:

> 
> On Wed, 30 Jan 2002, Alan Cox wrote:
> >
> > So if someone you trusted actually started batching up small fixes and
> > sending you things like
> >
> > "37 random documentation updates - no code changed", "11 patches to fix
> > kmalloc checks", "maintainers updates to 6 network drivers"
> >
> > that would work sanely ?
> 
> Yes. That would take a whole lot of load off me - load I currently handle
> by just not sweating the small stuff, and concentrating on the things I
> think are important.

Once more beating a dead horse, you don't improve scalability by finding a
better way to push everything through one person. "Random documentation
updates" and "corrections to MAINTAINERS mailing addresses" could and
should be approved by someone else.

So should the 1-2-3 liners which are clearly and obviously tiny bug fixes
for obvious problems, off by one, mistyped lock names, adding casts to
make the kernel compile w/o hundreds of "you don't understand C type
rules" warnings.

The way to get crap out of your life is to trust some people to identify
changes of this type and leave you to code review significant changes. The
most efficient way to do something is to avoid having to do it all all,
not by doing the wrong thing better. Pushing hard to you is like hand
coding a bubble sort in assembler, the problem is not in the
implementation but the algorithm. 

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Alan Cox wrote:

>>A "small stuff" maintainer may indeed be a good idea. The maintainer could
>>be the same as somebody who does bigger stuff too, but they should be
>>clearly different things - trivial one-liners that do not add anything
>>new, only fix obvious stuff (to the point where nobody even needs to think
>>about it - if I'd start getting any even halfway questionable patches from
>>the "small stuff" maintainer, it wouldn't work).
>>
And then we are still just discussing here how to get things IN. But 
there apparently currently is
nearly no way to get things OUT of the kernel tree. Old obsolete drivers 
used by some
computer since archeologists should be killed (Atari, Amiga, support, 
obsolete drivers and so on).
Just let *them* maintains theyr separate kernel tree...

Re: A modest proposal -- We need a patch penguin

[Ian Molton]

On a sunny Thu, 31 Jan 2002 13:14:55 +0100 Martin Dalecki gathered a sheaf
of electrons and etched in their motions the following immortal words:

> Old obsolete drivers used by some
> computer since archeologists should be killed (Atari, Amiga, support, 
> obsolete drivers and so on).
> Just let *them* maintains theyr separate kernel tree...

Yeah, keep linux for those X86 purists.

Great philosophy.

Just because hardware is OLD doesnt mean it cant integrate fairly cleanly
with the current linux kernel.

Why shouldnt Linux run on my 15 year old Acorn A400 (my current project) ?

Linux is about FUN.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, Martin Dalecki wrote:

> And then we are still just discussing here how to get things IN. But
> there apparently currently is nearly no way to get things OUT of the
> kernel tree. Old obsolete drivers used by some computer since
> archeologists should be killed (Atari, Amiga, support, obsolete
> drivers and so on). Just let *them* maintains theyr separate kernel
> tree...

'old' architectures do not hinder development - they are separate, and
they have to update their stuff. (and i think the m68k port is used by
many other people and not CS archeologists.) Old drivers are not a true
problem either - if they dont compile that's the problem of the
maintainer. Occasionally old drivers get zapped (mainly when there is a
new replacement driver).

	Ingo

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Thu, 31 Jan 2002, Ingo Molnar wrote:

> 'old' architectures do not hinder development - they are separate, and
> they have to update their stuff. (and i think the m68k port is used by

... unless they play silly buggers with the internals of VM.

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Thu, 31 Jan 2002, Alexander Viro wrote:

> > 'old' architectures do not hinder development - they are separate, and
> > they have to update their stuff. (and i think the m68k port is used by
>
> ... unless they play silly buggers with the internals of VM.

As long as there is still someone who can respond to problems in the arch
part, I don't really think it's a problem, or was it?

bye, Roman

Re: A modest proposal -- We need a patch penguin

[David Weinehall]

On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
> 
> On Thu, 31 Jan 2002, Martin Dalecki wrote:
> 
> > And then we are still just discussing here how to get things IN. But
> > there apparently currently is nearly no way to get things OUT of the
> > kernel tree. Old obsolete drivers used by some computer since
> > archeologists should be killed (Atari, Amiga, support, obsolete
> > drivers and so on). Just let *them* maintains theyr separate kernel
> > tree...
> 
> 'old' architectures do not hinder development - they are separate, and
> they have to update their stuff. (and i think the m68k port is used by
> many other people and not CS archeologists.) Old drivers are not a true
> problem either - if they dont compile that's the problem of the
> maintainer. Occasionally old drivers get zapped (mainly when there is a
> new replacement driver).

To testify that even really old hardware is used, I recently received a
patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
xd.c-driver. Not particularly recent hardware, but the person who sent
the patch uses it. Why deny him usage of his hardware when it doesn't
intrude upon the rest of the codebase?


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

David Weinehall wrote:

>On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>
>>On Thu, 31 Jan 2002, Martin Dalecki wrote:
>>
>>>And then we are still just discussing here how to get things IN. But
>>>there apparently currently is nearly no way to get things OUT of the
>>>kernel tree. Old obsolete drivers used by some computer since
>>>archeologists should be killed (Atari, Amiga, support, obsolete
>>>drivers and so on). Just let *them* maintains theyr separate kernel
>>>tree...
>>>
>>'old' architectures do not hinder development - they are separate, and
>>they have to update their stuff. (and i think the m68k port is used by
>>many other people and not CS archeologists.) Old drivers are not a true
>>problem either - if they dont compile that's the problem of the
>>maintainer. Occasionally old drivers get zapped (mainly when there is a
>>new replacement driver).
>>
>
>To testify that even really old hardware is used, I recently received a
>patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
>xd.c-driver. Not particularly recent hardware, but the person who sent
>the patch uses it. Why deny him usage of his hardware when it doesn't
>intrude upon the rest of the codebase?
>

He should feel free to use the 2.0.xx kernel up no end. Nobody denys it 
to him. But from the mainline
it all should get out of the sight for the developement.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, David Weinehall wrote:

> On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>
> > 'old' architectures do not hinder development - they are separate, and
> > they have to update their stuff. (and i think the m68k port is used by
> > many other people and not CS archeologists.) Old drivers are not a true
> > problem either - if they dont compile that's the problem of the
> > maintainer. Occasionally old drivers get zapped (mainly when there is a
> > new replacement driver).
>
> To testify that even really old hardware is used, I recently received
> a patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
> xd.c-driver. Not particularly recent hardware, but the person who sent
> the patch uses it. Why deny him usage of his hardware when it doesn't
> intrude upon the rest of the codebase?

exactly. Cruft hanging around does hurt in the 'generic' kernel. There is
'leaf' code where it hurts much less. Sure, we'd like to have clean code
everywhere, and a driver with a clean and recent codebase will get more
attention from the architecture point of view, but to the user, an
outdated but working driver is better than no driver at all.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Thu, 31 Jan 2002, David Weinehall wrote:
>
>>On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>>
>>>'old' architectures do not hinder development - they are separate, and
>>>they have to update their stuff. (and i think the m68k port is used by
>>>many other people and not CS archeologists.) Old drivers are not a true
>>>problem either - if they dont compile that's the problem of the
>>>maintainer. Occasionally old drivers get zapped (mainly when there is a
>>>new replacement driver).
>>>
>>To testify that even really old hardware is used, I recently received
>>a patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
>>xd.c-driver. Not particularly recent hardware, but the person who sent
>>the patch uses it. Why deny him usage of his hardware when it doesn't
>>intrude upon the rest of the codebase?
>>
>
>exactly. Cruft hanging around does hurt in the 'generic' kernel. There is
>'leaf' code where it hurts much less. Sure, we'd like to have clean code
>everywhere, and a driver with a clean and recent codebase will get more
>attention from the architecture point of view, but to the user, an
>outdated but working driver is better than no driver at all.
>
It's an incredibble bandwidth waste for 99.99% of people downolading 
2.5.xx and it *is* making architectural
changes in the kernel harder, becouse the modularisatoin of the kernel 
isn't nearly as perfect as you try
to disguise it here. Please just have a look at the consequences of the 
kdev_t changes, which where necessary
since already about 8 years. And then my these is somehow tautological 
if it doesn't apply now, it will
apply in about 4 years. At some point in time there is the need to let 
some things go - the problem
is more fundamental.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, Martin Dalecki wrote:

> It's an incredibble bandwidth waste for 99.99% of people downolading
> 2.5.xx and it *is* making architectural changes in the kernel harder,
> becouse the modularisatoin of the kernel isn't nearly as perfect as
> you try to disguise it here. Please just have a look at the
> consequences of the kdev_t changes, which where necessary since
> already about 8 years. And then my these is somehow tautological if it
> doesn't apply now, it will apply in about 4 years. At some point in
> time there is the need to let some things go - the problem is more
> fundamental.

it's not mandatory for the developer to push every interface change into
every driver or every architecture. Sure, if some code has not been kept
in sync for a long time then it should be zapped, but the pure fact that
something is less often used should not make it a candidate for zapping.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Thu, Jan 31, 2002 at 04:07:52PM +0100, Ingo Molnar wrote:
> it's not mandatory for the developer to push every interface change into
> every driver or every architecture. Sure, if some code has not been kept
> in sync for a long time then it should be zapped,

add "by the maintainer, if they are still around" here please.

> but the pure fact that
> something is less often used should not make it a candidate for zapping.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Thu, 31 Jan 2002, Martin Dalecki wrote:
> Alan Cox wrote:
> >>A "small stuff" maintainer may indeed be a good idea. The maintainer could
> >>be the same as somebody who does bigger stuff too, but they should be
> >>clearly different things - trivial one-liners that do not add anything
> >>new, only fix obvious stuff (to the point where nobody even needs to think
> >>about it - if I'd start getting any even halfway questionable patches from
> >>the "small stuff" maintainer, it wouldn't work).
> >>
> And then we are still just discussing here how to get things IN. But 
> there apparently currently is
> nearly no way to get things OUT of the kernel tree. Old obsolete drivers 
> used by some
> computer since archeologists should be killed (Atari, Amiga, support, 
> obsolete drivers and so on).
> Just let *them* maintains theyr separate kernel tree...

Come'on, m68k is not dead yet!

We do our best to keep the m68k tree in sync. In fact that's much less work
than feeding back our changes to Linus, since hacking code needs less retries
than sending patches :-)

(but we all know that since we're discussing it in this thread... ;-)

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 01:21:09AM -0800, Linus Torvalds wrote:
 > 
 > A "small stuff" maintainer may indeed be a good idea. The maintainer could
 > be the same as somebody who does bigger stuff too, but they should be
 > clearly different things - trivial one-liners that do not add anything
 > new, only fix obvious stuff (to the point where nobody even needs to think
 > about it - if I'd start getting any even halfway questionable patches from
 > the "small stuff" maintainer, it wouldn't work).

 The difficult part comes when slightly larger changesets are needed,
 just to make things compile again for some people.
 See yesterdays subsection changes from Keith I forwarded you for
 example. To me, it had looked fine, it had good discussion on
 l-k, and it solved a known problem. I was surprised you threw it
 back for more changes (but glad, I want the best solution too, and
 taking a quick glance to the mail I've not read yet, it looks like
 Keith has bettered his original solution).

 Most of the bits I've sent you so far have been "small stuff".
 And things will likely continue to be so. There are large chunks
 in my tree, but I've absolutely no intention of feeding you those.
 Things like the input layer/console layer reworking are the
 responsibility of $maintainer to push your way.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 09:09 am, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Alexander Viro wrote:
> > On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > > Linus just called you the ext2 maintainer.
> >
> > Message-ID, please?
> 
> I called you the VFS maintainer ("whether you like it or not" I think I
> said. Although I can't find the message right now).

Correct, I was wrong to say that, in fact, I should have posted the whole 
email to 'drafts' as I normally do with such non-constructive tripe.  I don't 
know what got into me.  Al, please accept my apologies.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> On January 30, 2002 09:09 am, Linus Torvalds wrote:
> > On Wed, 30 Jan 2002, Alexander Viro wrote:
> > > On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > > > Linus just called you the ext2 maintainer.
> > >
> > > Message-ID, please?
> > 
> > I called you the VFS maintainer ("whether you like it or not" I think I
> > said. Although I can't find the message right now).
> 
> Correct, I was wrong to say that, in fact, I should have posted the whole 
> email to 'drafts' as I normally do with such non-constructive tripe.  I don't 
> know what got into me.  Al, please accept my apologies.

Accepted.

Re: A modest proposal -- We need a patch penguin

[Kees Bakker, Kees Bakker]

>>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:

Alexander> On Wed, 30 Jan 2002, Daniel Phillips wrote:

>> Linus just called you the ext2 maintainer.

Alexander> Message-ID, please?

From: torvalds@transmeta.com (Linus Torvalds)
Subject: Re: A modest proposal -- We need a patch penguin
Date: 	Tue, 29 Jan 2002 22:22:46 +0000 (UTC)
Message-ID: <a377bn$1go$1@penguin.transmeta.com>

Re: A modest proposal -- We need a patch penguin

[Charles Cazabon]

Daniel Phillips <phillips@bonn-fries.net> wrote:
> 
> Linus just called you the ext2 maintainer.

Did he?  I only saw him call Al Viro the de-facto VFS maintainer.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                         <charlesc@discworld.dyndns.org>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Oliver Xymoron]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> Exactly.  The successor patch to the 'kind of gross' patch got rid of the
> double-pointers, it was the proper fix, though there is still no excuse for
> leaving the bug hanging around while coming up with the better version.

The gross fixes tend to get dropped because if they're in, the proper fix
loses priority. FIXMEs can take many years to fix. The problem seems not
to be the dropping of the patch so much as the dropping of the bug report
and bug tracking is an altogether different problem.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 08:41 am, Oliver Xymoron wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> 
> > Exactly.  The successor patch to the 'kind of gross' patch got rid of the
> > double-pointers, it was the proper fix, though there is still no excuse for
> > leaving the bug hanging around while coming up with the better version.
> 
> The gross fixes tend to get dropped because if they're in, the proper fix
> loses priority. FIXMEs can take many years to fix. The problem seems not
> to be the dropping of the patch so much as the dropping of the bug report
> and bug tracking is an altogether different problem.

The problem was the dropping of the patch.  A bunch of things contributed
to it, and at this point I believe the main one was having no patch
submission system.  I should know, I was on the dirty end of this stick.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:

Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4 liners.
The big stuff is not the problem most times. That stuff does get ripped to
shreds and picked over as is needed. (Except device drivers, Linus alas has
absolutely no taste in device drivers 8))

People collecting up patches _does_ help big time for all the small fixes.
Especially ones disciplined enough to keep the originals they applied so
they can feed stuff on with that tag. If I sent Linus on a patch that said
"You've missed this fix by Andrew Morton" then Linus knew it was probably
right for example.

> it. Start small, because for small patches people will have the few

Start small and your obvious one line diff, or 3 line typo fix will be
ignored for a decade. There were critical fixes that Linus dropped
repeatedly between 2.4.2 and 2.4.16 or so which ended up being holes in every
non-ac based distro.

Alan

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> The big stuff is not the problem most times. [...]

oh, i agree, but still the big stuff is that gets quoted in most emails
that try to invent the next big patch submission method ...

> People collecting up patches _does_ help big time for all the small
> fixes.

yes. This started with you and multiple people do it currently.

> Especially ones disciplined enough to keep the originals they applied
> so they can feed stuff on with that tag. If I sent Linus on a patch
> that said "You've missed this fix by Andrew Morton" then Linus knew it
> was probably right for example.

yes. This is what maintainers do. You, when collecting patches for the -ac
tree, are in essence a trusted jolly joker maintainer, very disciplined to
filter the trivial stuff from the nontrivial stuff.

> Start small and your obvious one line diff, or 3 line typo fix will be
> ignored for a decade. There were critical fixes that Linus dropped
> repeatedly between 2.4.2 and 2.4.16 or so which ended up being holes
> in every non-ac based distro.

(while i still do not claim that things are perfect, i'd like to see
specific examples nevertheless.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Dave McCracken]

--On Tuesday, January 29, 2002 13:22:05 +0000 Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

> Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4
> liners. The big stuff is not the problem most times. That stuff does get
> ripped to shreds and picked over as is needed. (Except device drivers,
> Linus alas has absolutely no taste in device drivers 8))
> 
> People collecting up patches _does_ help big time for all the small fixes.
> Especially ones disciplined enough to keep the originals they applied so
> they can feed stuff on with that tag. If I sent Linus on a patch that said
> "You've missed this fix by Andrew Morton" then Linus knew it was probably
> right for example.

I think this is a big part of the problem.  What's needed, and what Alan
used to provide, is a maintainer for the miscellaneous parts of the kernel
that aren't covered by the various subsystem maintainers.  We need someone
who can accept that one or two line fix, get it tested, and make sure Linus
sees it.

I gather Dave Jones is taking on that role to some extent.  If so, perhaps
he should be officially listed in the MAINTAINERS file.  Whether it's Dave
or someone else, I think this is a critical need.

Dave McCracken

======================================================================
Dave McCracken          IBM Linux Base Kernel Team      1-512-838-3059
dmccr@us.ibm.com                                        T/L   678-3059

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:54 am, Ingo Molnar wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:

>   - cleanliness
>   - concept
>   - timing
>   - testing
>
> a violation of any of these items can cause patch to be dropped *without
> notice*. Face it, it's not Linus' task to teach people how to code or how
> to write correct patches. Sure, he still does teach people most of the
> time, but you cannot *expect* him to be able to do it 100% of the time.

I'm trying to identify stuff that isn't necessarily Linus's job, and doesn't 
seem to be being done, and seeing if somebody ELSE can do it.  My proposal 
was my take on improving things.  If now is not the time for it, maybe 
smaller steps can be taken first.

Possibly Linus needs a "bounce this message to kernelnewbies.org's 'please 
teach this guy how to program' list?" key, as an alternative to the delete 
key?  For patches that try to do something useful and simply don't manage to?

This is one of the things I thought a patch penguin (and troupe of volunteer 
patch secretaries operating under a patch penguin) might be able to do.  
Right now, there's no troupe of patch secretaries because the patch 
submission queue is linus's mailbox.  (The real irreplaceable job of the 
patch penguin is queueing stuff for linus split out of the tree, so the patch 
penguin doesn't have to scale that much better than Linus does.  Other people 
working with the patch penguin could theoretically help 
massage/test/integrate/update patches.  And of course linus could still take 
patches directly from maintainers if he had the bandwidth to do so.  He can 
obviously even write his own code when he has time...)

Linux-kernel used to serve all these functions (it was the patch queue and 
the patch cleanup and sorting discussion list, and some people on it were a 
bit like patch secretaries), but the volume has gotten too high for it to 
function as such anymore.  Posting a patch here less than a dozen times 
doesn't really count as submitting it to Linus anymore.

One demarcation that COULD be made is 2.4 vs 2.5.  There could probably be 
seperate "stable" vs "development" lists.  Probably been suggested before and 
shot down, but does work aimed at Marcelo and work aimed at Linus really need 
to be on the same list?

> 2) concept
>
> many of the patches which were rejected for a long time are *difficult*
> issues. And one thing many patch submitters miss: even if the concept of
> the patch is correct, you first have to start by cleaning up *old* code,
> see issue 1). Your patch is not worth a dime if you leave in old cruft, or
> if the combination of old cruft and your new code is confusing. Also, make
> sure the patch is discussed and developed openly, not on some obscure
> list. linux-kernel@vger.kernel.org will do most of the time. I do not want
> to name specific patches that violate this point (doing that in public
> just offends people needlessly - and i could just as well list some of my
> older patches), but i could list 5 popular patches immediately.
>
> impact: a patch penguin just wont solve this concept issue, because, by
> definition, he doesnt deal with design issues. And most of the big patch
> rejections happen due to exactly these concept issues.

Definitely Linus's job.  But of those top 5 patches, how many of the patch 
pushers have had their ideas actually critiqued so they know why they're not 
going in?  (If it's just a question of "this work needs to be done 
first/also", that's a manageable problem.)

> 3) timing
>
> kernel source code just cannot go through arbitrary transitions. Eg. right
> now the scheduler is being cleaned up (so far it was more than 50
> sub-patches and they are still coming) - and work is going on to maximize
> the quality of the preemption patch, but until the base scheduler has
> stabilized there is just no point in applying the preemption patch - no
> matter how good the preemption patch is. Robert understands this very
> much. Many other people do not.
>
> impact: a patch penguin just wont solve this issue, because a patch
> penguin cannot let his tree transition arbitrarily either. Only separately
> maintained and tested patches/trees can handle this issue.

A patch penguin's tree could apply more "speculative" patches than Linus, 
because the nature of the tree is that some of the patches in it get backed 
out, or at the very least don't go on to Linus yet.

It's also a nice buffer of patches for Linus.  Even a relatively small buffer 
is a good thing to prevent data loss (16550a anyone?  Better than nothing...) 
 As long as the patches in the queue are maintained and kept up to date 
(which is work that is not being coordinated right now: person A's change 
breaks person B's patch, how does person A know if he doesn't apply person 
B's patch to his tree?).

And there's also the possibility that "judgement call" patches the patch 
penguin didn't want to include could be listed as "known to apply cleanly 
against this tree, but not included".  Just a page listing URLs to patches 
that are being tracked.  This has not previously been done by Alan, Dave, or 
Andrea, and maybe there's a reason why...

> 4) testing
>
> there are code areas and methods which need more rigorous testing and
> third-party feedback - no matter how good the patch. Most notably, if a
> patch exports some new user-space visible interface, then this item
> applies. An example is the aio patch, which had all 3 items right but was
> rejected due to this item. [things are improving very well on the aio
> front so i think this will change in the near future.]
>
> impact: a patch penguin just wont solve this issue, because his job, by
> definition, is not to keep patches around indefinitely, but to filter them
> to Linus.

Yes and no. Alan did more than that.  His tree contained stuff (like User 
Mode Linux) that he didn't immediately mean to forward to Linus.  Stuff HAS 
historically gone into patch penguin trees because the patch penguin likes 
the idea but believes it needs wider testing.

This stuff is usually a compile option.  XFS and JFS come to mind here, 
although that just points out we need a unified journaling layer, which is an 
ongoing discussion.  (Will a unified journaling layer come about until a tree 
exists that contains XFS, JFS, EXT3, and ReiserFS, and then people start 
scrutinizing the mess and combining common code?  I dunno...)

> Only separately maintained patches/trees help here. More people
> are willing to maintain separate trees is good (-dj, -ac, -aa, etc.), one
> tree can do a nontrivial transition at a time,

That's a seperately maintained patch that has not been integrated into a 
tree.  All patches apply to a Linux tree in order to get compiled into a 
system, and all patches could be downloaded as a tree (as the XFS guys do).

When I say tree I'm talking about a tree that's integrating patches from more 
than one source.  This is a step that comes after the patch has gone about as 
far as it's likely to as a seperate patch, seems to work pretty well, and 
needs testing by a wider audience in order to squeeze out more bugs or get 
more code review and comments on its design.

> and by having more of them
> we can eg. get one of them testing aio, the other one testing some other
> big change.

Sure.  This happens now.  The question is, what happens after the JFS people 
say "okay, we've reached version 1.0 now, please try this" and the patch 
still doesn't get integrated into a "beat me, whip me, make me break" tree 
for wider testing for months and months and months?

> A single patch penguin will be able to do only one nontrivial
> transition - and it's not his task to do nontrivial transitions to begin
> with.

I don't know what you mean here.

> Many people who dont actually maintain any Linux code are quoting Rik's
> complains as an example. I'll now have to go on record disagreeing with
> Rik humbly, i believe he has done a number of patch-management mistakes
> during his earlier VM development, and i strongly believe the reason why
> Linus ignored some of his patches were due to these issues. Rik's flames
> against Linus are understandable but are just that: flames. Fortunately
> Rik has learned meanwhile (we all do) and his rmap patches are IMHO
> top-notch. Joining the Andrea improvements and Rik's tree could provide a
> truly fantastic VM. [i'm not going to say anything about the IDE patches
> situation because while i believe Rik understands public criticism, i
> failed to have an impact on Andre before :-) ]

I understand that a lot of the problems aren't purely on Linus's end.  You 
didn't add Richard Gooch to that list with Rik and Andre (although Al seems 
to have decided it's one of his missions in life to keep Richard adhering to 
the coding style.... :)

But right now the individual maintainers need to be really good at patch 
management or the system breaks down.  This isn't exactly a scalability 
question, this is a reliability question.  There's no failure recovery 
mechanism here: if one part of the distributed system breaks the results are 
visible at the end.  You can't scale to more components if all of them most 
work perfectly every time and expect to have a more reliable system.

> also, many people just start off with a single big patch. That just doesnt
> work and you'll likely violate one of the 4 items without even noticing
> it. Start small, because for small patches people will have the few
> minutes needed to teach you.

The kernel is currently FULL of warnings when it used to have none, and 
outright compile errors that go unfixed for several versions if they occur in 
less-often used subsystems.

Small patches seem more likely to get dropped than big ones.  This could be 
an artifact of perception, I dunno...

> The bigger a patch, the harder it is to
> review it, and the less likely it happens. Also, if a few or your patches
> have gone into the Linux tree that does not mean you are senior kernel
> hacker and can start off writing the one big, multi-megabyte super-feature
> you dreamt about for years. Start small and increase the complexity of
> your patches slowly - and perhaps later on you'll notice that that
> super-feature isnt all that super anymore. People also underestimate the
> kind of complexity explosion that occurs if a large patch is created.
> Instead of 1-2 places, you can create 100-200 problems.
>
> face it, most of the patches rejected by Linus are not due to overload. He
> doesnt guarantee to say why he rejects patches - *and he must not*. Just
> knowing that your patch got rejected and thinking it all over again often
> helps finding problems that Linus missed first time around. If you submit
> to Linus then you better know exactly what you do.

There seems to be a perception that on at least some of the occasions Linus 
said "it didn't get applied because nobody ever sent me that patch", people 
were under the impression that the patch HAD been sent to him.

Sending a patch and hearing no reply, resending the same patch and then 
having it applied...  That sends mixed signals.  Resending the same patch a 
dozen times before it gets applied, how do you know if it's being rejected 
for a reason or if it's just a bad time?

> it's so much easier to blame Linus, or maintainers.

It's not a question of "blame".  Why does everybody keep thinking it's a 
question of blame?

It's "There seems to be a problem.  How do we fix it?"

The replies I've gotten range from denying there is any problem, assurances 
that the situation is functioning as maximum possible efficiency and cannot 
be improved, assurances that the problem is purely perceptual on my part, and 
a couple variations on "just live with it".

If this is the consensus...

> It's so much easier to
> fire off an email flaming Linus and getting off the steam than to actually
> accept the possibility of mistake and *fix* the patch. I'll go on record
> saying that good patches are not ignored, even these days when the number
> of active kernel hackers has multipled.

So patches that fix simple build breakage in things like the radeon or ymfpci 
drivers do not need to be resubmitted for multiple point releases?

> People might have to go through
> several layers first, and finding some kernel hacker who is not as loaded
> as Linus to review your patch might be necessery as well (especially if
> the patch is complex), but if you go through the right layers then you can
> be sure that nothing worthwile gets rejected arbitrarily.

Uh-huh.  Find people to review your patch, go through the right layers...

And a paralell tree to Linus's, dedicated to patch processing and tracking 
patches, with a patch submission system dedicated to routing patches to the 
proper maintainers, reviewing and cross-checking patches from maintainers, 
resolving conflicts, subjecting the lot to public scrutiny and being a 
one-stop-shopping place for people who want to find bugs in something...  
Like Alan Cox did for years, and like Dave Jones is doing now...  This is a 
totally different subject then?

Are people saying this is a bad thing?  Saying that this is useless?

Oh well, it was just a suggestion.  Seemed kind of a safe one since we were 
already mostly DOING it...

> 	Ingo

Rob

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Rob Landley wrote:

> And a paralell tree to Linus's, dedicated to patch processing and
> tracking patches, with a patch submission system dedicated to routing
> patches to the proper maintainers, reviewing and cross-checking
> patches from maintainers, resolving conflicts, subjecting the lot to
> public scrutiny and being a one-stop-shopping place for people who
> want to find bugs in something...  Like Alan Cox did for years, and
> like Dave Jones is doing now...  This is a totally different subject
> then?

you are banging on open doors. We have and need multiple trees. And no, a
*single* 'integration' or 'patch penguin' tree will not be able to solve
this problem. The 'small stuff' tree is a tree that does *not* apply
nontrivial patches. It's a tree for *pure* small stuff.

also, the -ac, -dj and -aa trees all act as a 'small stuff' tree currently
but the problem Alan pointed out is that Linus often rejects small stuff
from these sources as well, which creates high latency for small stuff and
decreases the 'end user' quality of the Linus tree. (and also we lose some
of the newbie developers who by definition start with small stuff.) So by
making a *separate* and *small stuff only* tree Linus could start trusting
those patches as small-stuff-only.

A small stuff tree will not and cannot replace the multiple experimental
trees that explore riskier patches (but only a few a time) like the -dj
tree or the -ac tree. (although i'd say the -ac tree isnt purely that,
it's more like a productization tree. The -dj and the devel-based -aa tree
is a good example.)

this way all the people who have the experience and stamina to integrate
patches can act as an experimental ground to 'cook' bigger patches before
they are sent to Linus. Linus' tree is 'cooking' a few patches as well,
but only in orthogonal areas. Eg. right now we have the bio changes, the
vfs cleanups, the device handing cleanups, and the scheduler cleanups
going on in parallel. The -dj tree might (and does) 'cook' patches that
shouldnt be applied to the Linus tree right now even if they were
'perfect' as a starting point. [i still have to see a complex patch that
is truly perfect and needs no iterations. Much of the true integration
steps are still done in the Linus tree these days.]

but integration (of nontrivial patches) on such level *can* be
parallelized to a certain degree. If patches are 'pre-cooked' well (in the
-ac, -dj and -aa, etc. trees and actual users see and test them) then the
load on the Linus tree and the latency of transition of the Linus tree can
be decreased somewhat. But i think we are still very far from the point
when Linus gets only 'perfect' (nontrivial-) patches. I doubt we'll ever
reach that point, and in that case Linus wont have much fun himself so i
doubt we want to reach that point :)

the small stuff tree on the other hand does not need to be parallelized,
small stuff is atomic and such patches scale almost infinitely. So a
single small stuff tree could indeed not only serve as a trusted source
for Linus, but could also take off the load from the other trees so they
can concentrate on the not-so-small-stuff like driver updates and other
subsystem updates, or even bigger patches. Formalizing and automating the
small-stuff tree might work as well, due to its inherent simplicity.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

mingo@elte.hu (Ingo Molnar)  wrote on 29.01.02 in <Pine.LNX.4.33.0201291324560.3610-100000@localhost.localdomain>:

> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:
>
>   - cleanliness
>   - concept
>   - timing
>   - testing

IIRC, the number 33 referred to esr's Configure.help patch. Which of these  
did he violate?

MfG Kai

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:

 > >   - cleanliness
 > >   - concept
 > >   - timing
 > >   - testing
 > 
 > IIRC, the number 33 referred to esr's Configure.help patch. Which of these  
 > did he violate?

 Timing.  Linus was busy focusing on the block layer.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Denis Vlasenko]

On 30 January 2002 00:46, Dave Jones wrote:
> On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
>  > >   - cleanliness
>  > >   - concept
>  > >   - timing
>  > >   - testing
>  >
>  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
>  > these did he violate?
>
>  Timing.  Linus was busy focusing on the block layer.

Sounds alarming. Linus didn't take documentation updates from designated 
maintainer? For many months? I can't believe in argument that updates were 
able to break _anything_, it's only documentation, right? I could understand 
this if these updates were sent by little known person, but Eric?!

Clearly a scalability problem here :-)

I won't post to this thread more.
Why? It's a flamewar, more _words_ will do nothing. _Action_ is needed.
--
vda

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:57:02AM -0200, Denis Vlasenko wrote:
> On 30 January 2002 00:46, Dave Jones wrote:
> > On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
> >  > >   - cleanliness
> >  > >   - concept
> >  > >   - timing
> >  > >   - testing
> >  >
> >  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
> >  > these did he violate?
> >
> >  Timing.  Linus was busy focusing on the block layer.
> 
> Sounds alarming. Linus didn't take documentation updates from designated 
> maintainer? For many months? I can't believe in argument that updates were 
> able to break _anything_, it's only documentation, right? I could understand 
> this if these updates were sent by little known person, but Eric?!
> 
> Clearly a scalability problem here :-)

Oh-my-lord.  Please re-read this thread, and especially Linus's
2.5.3-pre5 changelog announcement.

Configure.help needed to be split up.  Eric?! was told this repeatedly,
but he did not listen.  Hopefully he will listen to feedback regarding
CML2...  He has even been told repeatedly that he does not
listen to feedback ;-)

	Jeff, chuckling

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 03:29 am, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 09:57:02AM -0200, Denis Vlasenko wrote:
> > On 30 January 2002 00:46, Dave Jones wrote:
> > > On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
> > >  > >   - cleanliness
> > >  > >   - concept
> > >  > >   - timing
> > >  > >   - testing
> > >  >
> > >  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
> > >  > these did he violate?
> > >
> > >  Timing.  Linus was busy focusing on the block layer.
> >
> > Sounds alarming. Linus didn't take documentation updates from designated
> > maintainer? For many months? I can't believe in argument that updates
> > were able to break _anything_, it's only documentation, right? I could
> > understand this if these updates were sent by little known person, but
> > Eric?!
> >
> > Clearly a scalability problem here :-)
>
> Oh-my-lord.  Please re-read this thread, and especially Linus's
> 2.5.3-pre5 changelog announcement.
>
> Configure.help needed to be split up.  Eric?! was told this repeatedly,
> but he did not listen.  Hopefully he will listen to feedback regarding
> CML2...  He has even been told repeatedly that he does not
> listen to feedback ;-)
>
> 	Jeff, chuckling

I spoke to Eric earlier today.  (We're co-doing a presenatation at LinuxWorld 
Expo on thursday.)

His take on it was that he understood that Configure.help needed to be split 
up, but since the file was used by CML1 and he was NOT the CML1 maintainer, 
he didn't believe he had the authority to unilaterally change the file format 
in a way that would seriously break CML1.  (And, as it happens, now that the 
change has gone in, it seems the existing configurators are currently broken 
and have no help.)

Considering how much he's been warned so far about the need for CML2 to 
maintain as much compatability as possible with CML1, change as little 
behavior as possible in its first version, and not to make intrustive changes 
into the rest of the codebase...  I think he expected to be flamed alive if 
he broke up the help file before CML2 went in.

I.E. There was a miscommunication.  (The drop from Linus was an actual 
reject, but without an explanation of why it was rejected the reject didn't 
get resolved.  For 33 consecutive versions...)

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> Considering how much he's been warned so far about the need for CML2 to 
> maintain as much compatability as possible with CML1,

Pardon me while I laugh my ass off.


> behavior as possible in its first version, and not to make intrustive changes 
> into the rest of the codebase...  I think he expected to be flamed alive if 
> he broke up the help file before CML2 went in.

> I.E. There was a miscommunication.  (The drop from Linus was an actual 
> reject, but without an explanation of why it was rejected the reject didn't 
> get resolved.  For 33 consecutive versions...)

Getting told something point blank, multiple times, is definitely
-something-.  I suppose you could call that miscommunication.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 04:43 am, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> > Considering how much he's been warned so far about the need for CML2 to
> > maintain as much compatability as possible with CML1,
>
> Pardon me while I laugh my ass off.

[waits...]

> > behavior as possible in its first version, and not to make intrustive
> > changes into the rest of the codebase...  I think he expected to be
> > flamed alive if he broke up the help file before CML2 went in.
> >
> > I.E. There was a miscommunication.  (The drop from Linus was an actual
> > reject, but without an explanation of why it was rejected the reject
> > didn't get resolved.  For 33 consecutive versions...)
>
> Getting told something point blank, multiple times, is definitely
> -something-.  I suppose you could call that miscommunication.

I'm under the impression CML2 already supports the split-up per-directory 
help files, and did long before Linus actually split it up.  Therefore, Eric 
hasn't entirely been ignoring the issue, has he?

Yes, I would call it a miscommunication.

(By the way, if you really want to fix the current cml1 stuff in the 
cheesiest manner possible, what would be wrong with some variant of "find . 
-name "*.hlp" | xargs cat > oldhelpfile.hlp"?  Then the old help file becomes 
a generated file of the new help files.  Why mess with tcl/tk?  Put it in the 
make file as a dependency.  Pardon me if somebody fixed it last night, I seem 
have 91 emails to wade through since then on the patch penguin fallout 
alone...)

> 	Jeff

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 02:40:25PM -0500, Rob Landley wrote:
> On Wednesday 30 January 2002 04:43 am, Jeff Garzik wrote:
> > On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> > > Considering how much he's been warned so far about the need for CML2 to
> > > maintain as much compatability as possible with CML1,
> >
> > Pardon me while I laugh my ass off.
> 
> [waits...]

Keep waiting, I'm still laughing.


> I'm under the impression CML2 already supports the split-up per-directory 
> help files, and did long before Linus actually split it up.  Therefore, Eric 
> hasn't entirely been ignoring the issue, has he?

>From the kernel's point of view, yes, he has.


> (By the way, if you really want to fix the current cml1 stuff in the 
> cheesiest manner possible, what would be wrong with some variant of "find . 
> -name "*.hlp" | xargs cat > oldhelpfile.hlp"?  Then the old help file becomes 
> a generated file of the new help files.  Why mess with tcl/tk?  Put it in the 
> make file as a dependency.  Pardon me if somebody fixed it last night, I seem 
> have 91 emails to wade through since then on the patch penguin fallout 
> alone...)

That's a hack.  Fix it the right way.

<broken record> this is a devel series, we can afford to wait for the
better fix </broken record>

	Jeff

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> able to break _anything_, it's only documentation, right? I could understand 
> this if these updates were sent by little known person, but Eric?!

Oh come on. In the kernel world Eric is a little known person.

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Ingo Molnar wrote:

> None of the examples you cited so far are convincing to me, and i'd like
> to explain why. I've created and submitted thousands of patches to the
> Linux kernel over the past 4 years (my patch archive doesnt go back more
> than 4 years):
> 
>   # ls patches | wc -l
>      2818
> 
> a fair percentage of those went to Linus as well, and while having seen
> some of them rejected does hurt mentally, i couldnt list one reject from
> Linus that i wouldnt reject *today*. But i sure remember being frustrated
> about rejects when they happened. In any case, i have some experience in
> submitting patches and i'm maintaining a few subsystems, so here's my take
> on the 'patch penguin' issue:
> 
> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:

1 - why doesn't someone at least ack the damn patch?
2 - why doesn't someone at least ack the damn patch?
3 - why doesn't someone at least ack the damn patch?
4 - why doesn't someone at least ack the damn patch?

There is no better way to avoid getting something good from someone than
to ignore them completely. The fact that things like reisser f/s patches
from the creator don't get in is an example, or don't you think they are
good enough?

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Tomasz Kłoczko]

On Tue, 29 Jan 2002, Ingo Molnar wrote:
[..]
> 1) cleanliness
> 
> code cleanliness is a well-know issue, see Documentation/CodingStyle.  If
> a patch has such problems then maintainers are very likely to help - Linus
> probably wont and shouldnt.

I think place in each directory .indent.pro file with proper coding style
configuration and reduce Documentation/CodingStyle to how to use indent
tool can will solve many currunt problems with proper patches form and
will probaly take smaller amout disk space (or aprox the same) than
current Documentation/CodingStyle. Even if current indent can't handle
correctly current kernel coding style IMHO it will be better inves few
minutes on some changes to current indent behavior for bring this tool
abilities for reindent source code in way described in
Documentation/CodingStyle .. (?)

kloczek
-- 
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: kloczek@rudy.mif.pg.gda.pl*

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 10:23 pm, Linus Torvalds wrote:

> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse.

Oh, one other thing.  I didn't emphasize the possibility that the patch 
penguin might eventually run a public CVS tree that the various subsystem 
maintainers might be granted commit access to, because I didn't want to 
confuse the issue.

You don't use CVS, this proposal is not asking you to use CVS, and you seem 
to dislike other people using CVS.  I'm under the impression this is because 
CVS blurs together the patches you then need to receive and code review to do 
your job as architect of the Linux kernel.

It takes a skilled human being to extract clean patches from a CVS tree and 
feed them on to you in the format you prefer: one per email, plain text, with 
a description at the top.  Clean, atomic patches that do exactly one thing, 
and don't have any cross-reference dependencies on any other pending patches 
in the patch set.  No automated CVS-like tool will ever be able to do that 
AND resolve conflicts between patches.  You need a human.

Extracting patches out of a CVS tree and hand-massaging them into a format 
you would accept would be a big part of the integration maintainer's job.  
If they chose to run a CVS tree.  (Backing the patches out if you rejected 
the whole idea of that particular patch would be a lot of work too, but it 
would also be part of the integration maintainer's job.)

Rob

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Linus Torvalds wrote:

>Some thinking, for one thing.
>
>One "patch penguin" scales no better than I do. In fact, I will claim
>that most of them scale a whole lot worse. 
>
Bla bla bla... Just tell how frequenty do I have to tell the world, that 
the read_ahead array is a write
only variable inside the kernel and therefore not used at 
all?????!!!!!!!!!!

Re: A modest proposal -- We need a patch penguin

[Christoph Hellwig]

Hi Martin,

In article <3C568C52.2060707@evision-ventures.com> you wrote:
>>One "patch penguin" scales no better than I do. In fact, I will claim
>>that most of them scale a whole lot worse. 
>>
> Bla bla bla... Just tell how frequenty do I have to tell the world, that 
> the read_ahead array is a write
> only variable inside the kernel and therefore not used at 
> all?????!!!!!!!!!!

It IS used. (hint: take a look at fs/hfs/file.c).

I still don't think maintainig this array is worth just for hfs
readahead, so the below patch disables it and gets rid of read_ahead.

Jens, could you check the patch and include it in your next batch of
block-layer changes for Linus?

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.


diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/acorn/block/mfmhd.c linux/drivers/acorn/block/mfmhd.c
--- ../master/linux-2.5.3-pre6/drivers/acorn/block/mfmhd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/acorn/block/mfmhd.c	Tue Jan 29 14:07:41 2002
@@ -1442,7 +1442,6 @@
 	hdc63463_irqpollmask	= irqmask;
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB?) read ahread */
 
 	add_gendisk(&mfm_gendisk);
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/DAC960.c linux/drivers/block/DAC960.c
--- ../master/linux-2.5.3-pre6/drivers/block/DAC960.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/DAC960.c	Tue Jan 29 13:57:06 2002
@@ -1964,10 +1964,6 @@
   Controller->GenericDiskInfo.sizes = Controller->PartitionSizes;
   blksize_size[MajorNumber] = Controller->BlockSizes;
   /*
-    Initialize Read Ahead to 128 sectors.
-  */
-  read_ahead[MajorNumber] = 128;
-  /*
     Complete initialization of the Generic Disk Information structure.
   */
   Controller->GenericDiskInfo.major = MajorNumber;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/acsi.c linux/drivers/block/acsi.c
--- ../master/linux-2.5.3-pre6/drivers/block/acsi.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/acsi.c	Tue Jan 29 13:57:19 2002
@@ -1785,7 +1785,6 @@
 	STramMask = ATARIHW_PRESENT(EXTD_DMA) ? 0x00000000 : 0xff000000;
 	
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &acsi_lock);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&acsi_gendisk);
 
 #ifdef CONFIG_ATARI_SLM
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/blkpg.c linux/drivers/block/blkpg.c
--- ../master/linux-2.5.3-pre6/drivers/block/blkpg.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/blkpg.c	Tue Jan 29 13:58:51 2002
@@ -227,18 +227,6 @@
 			intval = (is_read_only(dev) != 0);
 			return put_user(intval, (int *)(arg));
 
-		case BLKRASET:
-			if(!capable(CAP_SYS_ADMIN))
-				return -EACCES;
-			if(arg > 0xff)
-				return -EINVAL;
-			read_ahead[major(dev)] = arg;
-			return 0;
-		case BLKRAGET:
-			if (!arg)
-				return -EINVAL;
-			return put_user(read_ahead[major(dev)], (long *) arg);
-
 		case BLKFRASET:
 			if (!capable(CAP_SYS_ADMIN))
 				return -EACCES;
@@ -319,6 +307,9 @@
 			set_blocksize(dev, intval);
 			return 0;
 
+		case BLKRASET:
+		case BLKRAGET:
+			/* this information is no more used by the kernel */
 		default:
 			return -EINVAL;
 	}
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/cciss.c linux/drivers/block/cciss.c
--- ../master/linux-2.5.3-pre6/drivers/block/cciss.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/cciss.c	Tue Jan 29 13:59:03 2002
@@ -2542,7 +2542,6 @@
 
 	/* fill in the other Kernel structs */
 	blksize_size[MAJOR_NR+i] = hba[i]->blocksizes;
-        read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 	/* Fill in the gendisk data */ 	
 	hba[i]->gendisk.major = MAJOR_NR + i;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/cpqarray.c linux/drivers/block/cpqarray.c
--- ../master/linux-2.5.3-pre6/drivers/block/cpqarray.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/cpqarray.c	Tue Jan 29 13:59:14 2002
@@ -481,7 +481,6 @@
 		blk_queue_max_phys_segments(q, SG_MAX);
 
 		blksize_size[MAJOR_NR+i] = ida_blocksizes + (i*256);
-		read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 		ida_gendisk[i].major = MAJOR_NR + i;
 		ida_gendisk[i].major_name = "ida";
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/ll_rw_blk.c linux/drivers/block/ll_rw_blk.c
--- ../master/linux-2.5.3-pre6/drivers/block/ll_rw_blk.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/ll_rw_blk.c	Tue Jan 29 13:59:28 2002
@@ -54,10 +54,6 @@
  */
 DECLARE_TASK_QUEUE(tq_disk);
 
-/* This specifies how many sectors to read ahead on the disk. */
-
-int read_ahead[MAX_BLKDEV];
-
 /* blk_dev_struct is:
  *	request_queue
  *	*queue
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pcd.c linux/drivers/block/paride/pcd.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pcd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/paride/pcd.c	Tue Jan 29 14:00:33 2002
@@ -358,7 +358,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &pcd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 
 	for (i=0;i<PCD_UNITS;i++) pcd_blocksizes[i] = 1024;
         blksize_size[MAJOR_NR] = pcd_blocksizes;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pd.c linux/drivers/block/paride/pd.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/paride/pd.c	Tue Jan 29 14:00:41 2002
@@ -397,7 +397,6 @@
 	q = BLK_DEFAULT_QUEUE(MAJOR_NR);
 	blk_init_queue(q, DEVICE_REQUEST, &pd_lock);
 	blk_queue_max_sectors(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	pd_gendisk.major = major;
 	pd_gendisk.major_name = name;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pf.c linux/drivers/block/paride/pf.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pf.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/paride/pf.c	Tue Jan 29 14:00:23 2002
@@ -363,7 +363,6 @@
 	blk_init_queue(q, DEVICE_REQUEST, &pf_spin_lock);
 	blk_queue_max_phys_segments(q, cluster);
 	blk_queue_max_hw_segments(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	for (i=0;i<PF_UNITS;i++) pf_blocksizes[i] = 1024;
 	blksize_size[MAJOR_NR] = pf_blocksizes;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/ps2esdi.c linux/drivers/block/ps2esdi.c
--- ../master/linux-2.5.3-pre6/drivers/block/ps2esdi.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/ps2esdi.c	Tue Jan 29 14:00:11 2002
@@ -177,8 +177,6 @@
 	}
 	/* set up some global information - indicating device specific info */
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &ps2esdi_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
-
 	/* some minor housekeeping - setup the global gendisk structure */
 	add_gendisk(&ps2esdi_gendisk);
 	ps2esdi_geninit();
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/xd.c linux/drivers/block/xd.c
--- ../master/linux-2.5.3-pre6/drivers/block/xd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/xd.c	Tue Jan 29 13:59:39 2002
@@ -171,7 +171,6 @@
 	}
 	devfs_handle = devfs_mk_dir (NULL, xd_gendisk.major_name, NULL);
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &xd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 	add_gendisk(&xd_gendisk);
 	xd_geninit();
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/aztcd.c linux/drivers/cdrom/aztcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/aztcd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/cdrom/aztcd.c	Tue Jan 29 14:05:15 2002
@@ -1927,7 +1927,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &aztSpin);
 	blksize_size[MAJOR_NR] = aztcd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &azt_fops, 0);
 
 	if ((azt_port == 0x1f0) || (azt_port == 0x170))
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/cdu31a.c linux/drivers/cdrom/cdu31a.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/cdu31a.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/cdrom/cdu31a.c	Tue Jan 29 14:05:21 2002
@@ -3442,7 +3442,6 @@
 		blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR),
 			       DEVICE_REQUEST,
 			       &cdu31a_lock);
-		read_ahead[MAJOR_NR] = CDU31A_READAHEAD;
 		cdu31a_block_size = 1024;	/* 1kB default block size */
 		/* use 'mount -o block=2048' */
 		blksize_size[MAJOR_NR] = &cdu31a_block_size;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/cm206.c linux/drivers/cdrom/cm206.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/cm206.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/cm206.c	Tue Jan 29 14:05:43 2002
@@ -1503,7 +1503,6 @@
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &cm206_lock);
 	blksize_size[MAJOR_NR] = cm206_blocksizes;
-	read_ahead[MAJOR_NR] = 16;	/* reads ahead what? */
 	init_bh(CM206_BH, cm206_bh);
 
 	memset(cd, 0, sizeof(*cd));	/* give'm some reasonable value */
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/gscd.c linux/drivers/cdrom/gscd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/gscd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/gscd.c	Tue Jan 29 14:05:50 2002
@@ -1022,7 +1022,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &gscd_lock);
 	blksize_size[MAJOR_NR] = gscd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 
 	disk_state = 0;
 	gscdPresent = 1;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/mcd.c linux/drivers/cdrom/mcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/mcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/mcd.c	Tue Jan 29 14:05:56 2002
@@ -1075,7 +1075,6 @@
 	blksize_size[MAJOR_NR] = mcd_blocksizes;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcd_spinlock);
-	read_ahead[MAJOR_NR] = 4;
 
 	/* check for card */
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/mcdx.c linux/drivers/cdrom/mcdx.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/mcdx.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/mcdx.c	Tue Jan 29 14:06:01 2002
@@ -1184,7 +1184,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcdx_lock);
-	read_ahead[MAJOR_NR] = READ_AHEAD;
 	blksize_size[MAJOR_NR] = mcdx_blocksizes;
 
 	xtrace(INIT, "init() subscribe irq and i/o\n");
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/optcd.c linux/drivers/cdrom/optcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/optcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/optcd.c	Tue Jan 29 14:06:06 2002
@@ -2062,7 +2062,6 @@
 	blksize_size[MAJOR_NR] = &blksize;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &optcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	request_region(optcd_port, 4, "optcd");
 	register_disk(NULL, mk_kdev(MAJOR_NR,0), 1, &opt_fops, 0);
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sbpcd.c linux/drivers/cdrom/sbpcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sbpcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sbpcd.c	Tue Jan 29 14:07:22 2002
@@ -4532,11 +4532,7 @@
 	} /* end of CDROMREADAUDIO */
 		
 	case BLKRASET:
-		if(!capable(CAP_SYS_ADMIN)) RETURN_UP(-EACCES);
-		if(kdev_none(cdi->dev)) RETURN_UP(-EINVAL);
-		if(arg > 0xff) RETURN_UP(-EINVAL);
-		read_ahead[major(cdi->dev)] = arg;
-		RETURN_UP(0);
+		return -EINVAL;
 	default:
 		msg(DBG_IOC,"ioctl: unknown function request %04X\n", cmd);
 		RETURN_UP(-EINVAL);
@@ -5870,7 +5866,6 @@
 	(BLK_DEFAULT_QUEUE(MAJOR_NR))->front_merge_fn = dont_bh_merge_fn;
 	(BLK_DEFAULT_QUEUE(MAJOR_NR))->merge_requests_fn = dont_merge_requests_fn;
 #endif
-	read_ahead[MAJOR_NR] = buffers * (CD_FRAMESIZE / 512);
 	
 	request_region(CDo_command,4,major_name);
 	
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sjcd.c linux/drivers/cdrom/sjcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sjcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sjcd.c	Tue Jan 29 14:06:10 2002
@@ -1695,7 +1695,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,&sjcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &sjcd_fops, 0);
 
 	if (check_region(sjcd_base, 4)) {
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sonycd535.c linux/drivers/cdrom/sonycd535.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sonycd535.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sonycd535.c	Tue Jan 29 14:06:16 2002
@@ -1598,7 +1598,6 @@
 				}
 				blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &sonycd535_lock);
 				blksize_size[MAJOR_NR] = &sonycd535_block_size;
-				read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read-ahead */
 
 				sony_toc = (struct s535_sony_toc *)
 					kmalloc(sizeof *sony_toc, GFP_KERNEL);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/hd.c linux/drivers/ide/hd.c
--- ../master/linux-2.5.3-pre6/drivers/ide/hd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/ide/hd.c	Tue Jan 29 14:04:07 2002
@@ -837,7 +837,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &hd_lock);
 	blk_queue_max_sectors(BLK_DEFAULT_QUEUE(MAJOR_NR), 255);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&hd_gendisk);
 	init_timer(&device_timer);
 	device_timer.function = hd_times_out;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-cd.c linux/drivers/ide/ide-cd.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-cd.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-cd.c	Tue Jan 29 14:04:51 2002
@@ -2662,7 +2662,6 @@
 	int major = HWIF(drive)->major;
 	int minor = drive->select.b.unit << PARTN_BITS;
 
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW, BLKRAGET, BLKRASET, TYPE_INT, 0, 255, 1, 2, &read_ahead[major], NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW, BLKFRAGET, BLKFRASET, TYPE_INTA, 0, INT_MAX, 1, 1024, &max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"dsc_overlap",		SETTING_RW, -1, -1, TYPE_BYTE, 0, 1, 1,	1, &drive->dsc_overlap, NULL);
 }
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-disk.c linux/drivers/ide/ide-disk.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-disk.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-disk.c	Tue Jan 29 14:04:45 2002
@@ -931,7 +931,6 @@
 	ide_add_setting(drive,	"bswap",		SETTING_READ,					-1,			-1,			TYPE_BYTE,	0,	1,				1,	1,	&drive->bswap,			NULL);
 	ide_add_setting(drive,	"multcount",		id ? SETTING_RW : SETTING_READ,			HDIO_GET_MULTCOUNT,	HDIO_SET_MULTCOUNT,	TYPE_BYTE,	0,	id ? id->max_multsect : 0,	1,	1,	&drive->mult_count,		set_multcount);
 	ide_add_setting(drive,	"nowerr",		SETTING_RW,					HDIO_GET_NOWERR,	HDIO_SET_NOWERR,	TYPE_BYTE,	0,	1,				1,	1,	&drive->nowerr,			set_nowerr);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	1,	&read_ahead[major],		NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	4096,			PAGE_SIZE,	1024,	&max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"lun",			SETTING_RW,					-1,			-1,			TYPE_INT,	0,	7,				1,	1,	&drive->lun,			NULL);
 	ide_add_setting(drive,	"wcache",		SETTING_RW,					HDIO_GET_WCACHE,	HDIO_SET_WCACHE,	TYPE_BYTE,	0,	1,				1,	1,	&drive->wcache,			write_cache);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-floppy.c linux/drivers/ide/ide-floppy.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-floppy.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-floppy.c	Tue Jan 29 14:04:40 2002
@@ -1968,7 +1968,6 @@
 	ide_add_setting(drive,	"bios_cyl",		SETTING_RW,					-1,			-1,			TYPE_INT,	0,	1023,				1,	1,	&drive->bios_cyl,		NULL);
 	ide_add_setting(drive,	"bios_head",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	255,				1,	1,	&drive->bios_head,		NULL);
 	ide_add_setting(drive,	"bios_sect",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	63,				1,	1,	&drive->bios_sect,		NULL);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	2,	&read_ahead[major],		NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	INT_MAX,			1,	1024,	&max_readahead[major][minor],	NULL);
 
 }
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-probe.c linux/drivers/ide/ide-probe.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-probe.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-probe.c	Tue Jan 29 14:04:22 2002
@@ -937,7 +937,6 @@
 	init_gendisk(hwif);
 	blk_dev[hwif->major].data = hwif;
 	blk_dev[hwif->major].queue = ide_get_queue;
-	read_ahead[hwif->major] = 8;	/* (4kB) */
 	hwif->present = 1;	/* success */
 
 	return hwif->present;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/md/md.c linux/drivers/md/md.c
--- ../master/linux-2.5.3-pre6/drivers/md/md.c	Tue Jan 29 11:24:21 2002
+++ linux/drivers/md/md.c	Tue Jan 29 13:56:35 2002
@@ -1737,7 +1737,6 @@
 	register_disk(&md_gendisk, mk_kdev(MAJOR_NR,mdidx(mddev)),
 			1, &md_fops, md_size[mdidx(mddev)]<<1);
 
-	read_ahead[MD_MAJOR] = 1024;
 	return (0);
 }
 
@@ -3177,11 +3176,7 @@
 	sz += sprintf(page+sz, "\n");
 
 
-	sz += sprintf(page+sz, "read_ahead ");
-	if (read_ahead[MD_MAJOR] == INT_MAX)
-		sz += sprintf(page+sz, "not set\n");
-	else
-		sz += sprintf(page+sz, "%d sectors\n", read_ahead[MD_MAJOR]);
+	sz += sprintf(page+sz, "read_ahead not set\n");
 
 	ITERATE_MDDEV(mddev,tmp) {
 		sz += sprintf(page + sz, "md%d : %sactive", mdidx(mddev),
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/s390/block/xpram.c linux/drivers/s390/block/xpram.c
--- ../master/linux-2.5.3-pre6/drivers/s390/block/xpram.c	Tue Jan 15 10:59:08 2002
+++ linux/drivers/s390/block/xpram.c	Tue Jan 29 14:02:25 2002
@@ -163,12 +163,11 @@
 
 static int major    = XPRAM_MAJOR;
 static int devs     = XPRAM_DEVS;
-static int rahead   = XPRAM_RAHEAD;
 static int sizes[XPRAM_MAX_DEVS] = { 0, };
 static int blksize  = XPRAM_BLKSIZE;
 static int hardsect = XPRAM_HARDSECT;
 
-int xpram_devs, xpram_rahead;
+int xpram_devs;
 int xpram_blksize, xpram_hardsect;
 int xpram_mem_avail = 0;
 unsigned long xpram_sizes[XPRAM_MAX_DEVS];
@@ -660,21 +659,7 @@
 		return 0;
 
 	case BLKRAGET: /* return the readahead value, 0x1263 */
-		if (!arg)  return -EINVAL;
-		err = 0; /* verify_area_20(VERIFY_WRITE, (long *) arg, sizeof(long));
-		          * if (err) return err;
-                          */
-		put_user(read_ahead[MAJOR(inode->i_rdev)], (long *)arg);
-
-		return 0;
-
 	case BLKRASET: /* set the readahead value, 0x1262 */
-		if (!capable(CAP_SYS_ADMIN)) return -EACCES;
-		if (arg > 0xff) return -EINVAL; /* limit it */
-		read_ahead[MAJOR(inode->i_rdev)] = arg;
-                atomic_eieio();
-		return 0;
-
 	case BLKRRPART: /* re-read partition table: can't do it, 0x1259 */
 		return -EINVAL;
 
@@ -685,7 +670,6 @@
 						* BLKROGET
                                                 */
 #endif /* V22 */
-
 	case HDIO_GETGEO:
 		/*
 		 * get geometry: we have to fake one...  trim the size to a
@@ -940,7 +924,6 @@
 				 * snoozing with a debugger.
 				 */
 
-	xpram_rahead   = rahead;
 	xpram_blksize  = blksize;
 	xpram_hardsect = hardsect;
 
@@ -1029,7 +1012,7 @@
 	PRINT_INFO("  %d kB expanded memory found.\n",xpram_mem_avail );
 
 	/*
-	 * Assign the other needed values: request, rahead, size, blksize,
+	 * Assign the other needed values: request, size, blksize,
 	 * hardsect. All the minor devices feature the same value.
 	 * Note that `xpram' defines all of them to allow testing non-default
 	 * values. A real device could well avoid setting values in global
@@ -1042,7 +1025,6 @@
 	q = BLK_DEFAULT_QUEUE (major);
 	blk_init_queue (q, xpram_request);
 #endif /* V22/V24 */
-	read_ahead[major] = xpram_rahead;
 
 	/* we want to have XPRAM_UNUSED blocks security buffer between devices */
 	mem_usable=xpram_mem_avail-(XPRAM_UNUSED*(xpram_devs-1));
@@ -1181,7 +1163,6 @@
 	kfree(xpram_hardsects);
 	hardsect_size[major] = NULL;
  fail_malloc:
-	read_ahead[major] = 0;
 #if (XPRAM_VERSION == 22)
 	blk_dev[major].request_fn = NULL;
 #endif /* V22 */
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/s390/char/tapeblock.c linux/drivers/s390/char/tapeblock.c
--- ../master/linux-2.5.3-pre6/drivers/s390/char/tapeblock.c	Sat Dec 22 20:06:57 2001
+++ linux/drivers/s390/char/tapeblock.c	Tue Jan 29 14:02:34 2002
@@ -101,7 +101,6 @@
     }
     if (tapeblock_major == 0) tapeblock_major = result;   /* accept dynamic major number*/
     INIT_BLK_DEV(tapeblock_major,tape_request_fn,tapeblock_getqueue,NULL);
-    read_ahead[tapeblock_major]=TAPEBLOCK_READAHEAD;
     PRINT_WARN(KERN_ERR " tape gets major %d for block device\n", result);
     blk_size[tapeblock_major] = (int*) kmalloc (256*sizeof(int),GFP_ATOMIC);
     memset(blk_size[tapeblock_major],0,256*sizeof(int));
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/scsi/sd.c linux/drivers/scsi/sd.c
--- ../master/linux-2.5.3-pre6/drivers/scsi/sd.c	Tue Jan 15 10:59:09 2002
+++ linux/drivers/scsi/sd.c	Tue Jan 29 14:03:01 2002
@@ -1179,7 +1179,7 @@
 		add_gendisk(&(sd_gendisks[i]));
 	}
 
-	for (i = 0; i < sd_template.dev_max; ++i)
+	for (i = 0; i < sd_template.dev_max; ++i) {
 		if (!rscsi_disks[i].capacity && rscsi_disks[i].device) {
 			sd_init_onedisk(i);
 			if (!rscsi_disks[i].has_part_table) {
@@ -1190,17 +1190,6 @@
 				rscsi_disks[i].has_part_table = 1;
 			}
 		}
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 60 blocks (120 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. We can only respect this with the
-	 * granularity of every 16 disks (one device major).
-	 */
-	for (i = 0; i < N_USED_SD_MAJORS; i++) {
-		read_ahead[SD_MAJOR(i)] =
-		    (rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device
-		     && rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device->host->sg_tablesize)
-		    ? 120	/* 120 sector read-ahead */
-		    : 4;	/* 4 sector read-ahead */
 	}
 
 	return;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/scsi/sr.c linux/drivers/scsi/sr.c
--- ../master/linux-2.5.3-pre6/drivers/scsi/sr.c	Tue Jan 29 11:24:22 2002
+++ linux/drivers/scsi/sr.c	Tue Jan 29 14:03:53 2002
@@ -785,16 +785,6 @@
                                     &sr_bdops, NULL);
 		register_cdrom(&scsi_CDs[i].cdi);
 	}
-
-
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 16 blocks (32 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. */
-	if (scsi_CDs[0].device && scsi_CDs[0].device->host->sg_tablesize)
-		read_ahead[MAJOR_NR] = 32;	/* 32 sector read-ahead.  Always removable. */
-	else
-		read_ahead[MAJOR_NR] = 4;	/* 4 sector read-ahead */
-
 }
 
 static void sr_detach(Scsi_Device * SDp)
@@ -846,7 +836,6 @@
 		kfree(sr_blocksizes);
 		sr_blocksizes = NULL;
 	}
-	read_ahead[MAJOR_NR] = 0;
 	blk_clear(MAJOR_NR);
 
 	sr_template.dev_max = 0;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/fs/hfs/file.c linux/fs/hfs/file.c
--- ../master/linux-2.5.3-pre6/fs/hfs/file.c	Tue Jan 15 10:59:13 2002
+++ linux/fs/hfs/file.c	Tue Jan 29 14:09:18 2002
@@ -309,6 +309,7 @@
 	blocks = (count+offset+HFS_SECTOR_SIZE-1) >> HFS_SECTOR_SIZE_BITS;
 
 	bhb = bhe = buflist;
+#if 0 /* for readahead we have the pagecache..  --hch */
 	if (reada) {
 		if (blocks < read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9)) {
 			blocks = read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9);
@@ -317,6 +318,7 @@
 			blocks = size - block;
 		}
 	}
+#endif
 
 	/* We do this in a two stage process.  We first try and
 	   request as many blocks as we can, then we wait for the
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/include/linux/blkdev.h linux/include/linux/blkdev.h
--- ../master/linux-2.5.3-pre6/include/linux/blkdev.h	Tue Jan 29 11:24:24 2002
+++ linux/include/linux/blkdev.h	Tue Jan 29 13:54:49 2002
@@ -341,7 +341,6 @@
 #endif
 	blksize_size[major] = NULL;
 	max_readahead[major] = NULL;
-	read_ahead[major] = 0;
 }
 
 extern inline int get_hardsect_size(kdev_t dev)
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/include/linux/fs.h linux/include/linux/fs.h
--- ../master/linux-2.5.3-pre6/include/linux/fs.h	Tue Jan 29 11:24:24 2002
+++ linux/include/linux/fs.h	Tue Jan 29 13:54:36 2002
@@ -1484,7 +1484,6 @@
 
 extern ssize_t char_read(struct file *, char *, size_t, loff_t *);
 extern ssize_t block_read(struct file *, char *, size_t, loff_t *);
-extern int read_ahead[];
 
 extern ssize_t char_write(struct file *, const char *, size_t, loff_t *);
 extern ssize_t block_write(struct file *, const char *, size_t, loff_t *);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/kernel/ksyms.c linux/kernel/ksyms.c
--- ../master/linux-2.5.3-pre6/kernel/ksyms.c	Tue Jan 29 11:24:24 2002
+++ linux/kernel/ksyms.c	Tue Jan 29 13:54:15 2002
@@ -509,7 +509,6 @@
 EXPORT_SYMBOL(clear_inode);
 EXPORT_SYMBOL(___strtok);
 EXPORT_SYMBOL(init_special_inode);
-EXPORT_SYMBOL(read_ahead);
 EXPORT_SYMBOL(__get_hash_table);
 EXPORT_SYMBOL(new_inode);
 EXPORT_SYMBOL(insert_inode_hash);

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> I still don't think maintainig this array is worth just for hfs
> readahead, so the below patch disables it and gets rid of read_ahead.
> 
> Jens, could you check the patch and include it in your next batch of
> block-layer changes for Linus?

What would be significantly more useful would be to make it actually work.
Lots of drivers benefit from control over readahead sizes - both the
stunningly slow low end stuff and the high end raid cards that often want
to get hit by very large I/O requests (eg 128K for the ami megaraid)

Alan

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Alan Cox wrote:

>>I still don't think maintainig this array is worth just for hfs
>>readahead, so the below patch disables it and gets rid of read_ahead.
>>
>>Jens, could you check the patch and include it in your next batch of
>>block-layer changes for Linus?
>>
>
>What would be significantly more useful would be to make it actually work.
>Lots of drivers benefit from control over readahead sizes - both the
>stunningly slow low end stuff and the high end raid cards that often want
>to get hit by very large I/O requests (eg 128K for the ami megaraid)
>
No you are wrong. This array is supposed to provide a readahead setting 
on the driver level, which is bogous, since
it's something that *should* not be exposed to the upper layers at all. 
Please note as well that
 we have already max_readahead in struut block_device as well. Please 
note that this array only has
a granularity of major block device numbers which is compleatly bogous 
for example for a disk and
cd-rom hanging on a IDE interface. And so on and so on... It's really 
better to let it go.

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> No you are wrong. This array is supposed to provide a readahead setting 
> on the driver level, which is bogous, since
> it's something that *should* not be exposed to the upper layers at all. 

Right.

>  we have already max_readahead in struut block_device as well. Please 
> note that this array only has

Ok. Now I look at it again yes - the array is completely surplus to current
requirements. 2.5 nicely sorts out the queues 

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Christoph Hellwig wrote:

>Hi Martin,
>
>In article <3C568C52.2060707@evision-ventures.com> you wrote:
>
>>>One "patch penguin" scales no better than I do. In fact, I will claim
>>>that most of them scale a whole lot worse. 
>>>
>>Bla bla bla... Just tell how frequenty do I have to tell the world, that 
>>the read_ahead array is a write
>>only variable inside the kernel and therefore not used at 
>>all?????!!!!!!!!!!
>>
>
>It IS used. (hint: take a look at fs/hfs/file.c).
>

Right, but the usage there is semantically *invalid*.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >One "patch penguin" scales no better than I do. In fact, I will claim
> >that most of them scale a whole lot worse.

> Bla bla bla... Just tell how frequenty do I have to tell the world,
> that the read_ahead array is a write only variable inside the kernel
> and therefore not used at all?????!!!!!!!!!!

tell Jens. He goes about fixing it all, not just the most visible pieces
that showed how much the Linux block IO code sucked. And guess what? His
patches are being accepted, and the Linux 2.5 block IO code is evolving
rapidly. Sometimes keeping broken code around as an incentive to fix it
*for real* is better than trying to massage the broken code somewhat.

a patch penguin doesnt solve this particular problem, by definition he
just wont fix the block IO code.

any other 'examples'?

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Tue, 29 Jan 2002, Martin Dalecki wrote:
>
>>Bla bla bla... Just tell how frequenty do I have to tell the world,
>>that the read_ahead array is a write only variable inside the kernel
>>and therefore not used at all?????!!!!!!!!!!
>>
>tell Jens. He goes about fixing it all, not just the most visible pieces
>that showed how much the Linux block IO code sucked. And guess what? His
>patches are being accepted, and the Linux 2.5 block IO code is evolving
>rapidly. Sometimes keeping broken code around as an incentive to fix it
>*for real* is better than trying to massage the broken code somewhat.
>
There is nothing easier to fix then this. You just have to grep for it, 
or just remove the declaration and wait
to be hit by this during the compilation. And most interrestingly this 
is *easier* then sending a patch!
A patch for this particular problem tend't to
1. Touch many things (however in a trivial way!)
2. Have spurious conflicts in terms of synchronization with the overall 
developement tree of the maintainer
in question.

Now dear linus tell me a better way to deal with *this* kind of problem 
then using CVS for example
where not a single man has the overall controll.

Yes my opinnin is indeed that in reality our problem is that Linus just 
doesn't want to give up
some kind of controll - no more no less.

>
>
>a patch penguin doesnt solve this particular problem, by definition he
>just wont fix the block IO code.
>
>any other 'examples'
>

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >tell Jens. He goes about fixing it all, not just the most visible pieces
> >that showed how much the Linux block IO code sucked. And guess what? His
> >patches are being accepted, and the Linux 2.5 block IO code is evolving
> >rapidly. Sometimes keeping broken code around as an incentive to fix it
> >*for real* is better than trying to massage the broken code somewhat.
> >

> There is nothing easier to fix then this. You just have to grep for
> it, or just remove the declaration and wait to be hit by this during
> the compilation. [...]

you have completely and totally ignored my argument.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Tue, 29 Jan 2002, Martin Dalecki wrote:
>
>>>tell Jens. He goes about fixing it all, not just the most visible pieces
>>>that showed how much the Linux block IO code sucked. And guess what? His
>>>patches are being accepted, and the Linux 2.5 block IO code is evolving
>>>rapidly. Sometimes keeping broken code around as an incentive to fix it
>>>*for real* is better than trying to massage the broken code somewhat.
>>>
>>There is nothing easier to fix then this. You just have to grep for
>>it, or just remove the declaration and wait to be hit by this during
>>the compilation. [...]
>>
>>you have completely and totally ignored my argument.	
>>
And you didn't look at the issue. Abstract arguments from you sound only 
like a dialogue with
a AI programm.

1. Telling Jest - he is supposed to read lkml. I'm continuously raising 
this issue since several moths already.
2. *for real* - removing it is the REAL fix.

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> rapidly. Sometimes keeping broken code around as an incentive to fix it
> *for real* is better than trying to massage the broken code somewhat.
> 
> a patch penguin doesnt solve this particular problem, by definition he
> just wont fix the block IO code.

Ingo, you should have a look at my mailbox and the people sick of trying to
get Linus to take 3 liners to fix NODEV type stuff and being ignored so that
2.5.* still doesn't even compile or boot for many people.

Dave in doing the patch hoovering at least ensures these are picked up. You
think if this carries on anyone will be running Linus tree in 9 months ?

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> Ingo, you should have a look at my mailbox and the people sick of
> trying to get Linus to take 3 liners to fix NODEV type stuff and being
> ignored so that 2.5.* still doesn't even compile or boot for many
> people.

for code areas where there is not active maintainer or the maintainer has
ignored patches? Eg. the majority of the kdev transition patches went in
smoothly.

but i'm not claiming that everything is rosy (i would post patches if
everything was rosy in Linux-land), but i disagree with the majority of
serious examples i've seen cited.

> Dave in doing the patch hoovering at least ensures these are picked
> up. You think if this carries on anyone will be running Linus tree in
> 9 months ?

Dave and you doing patch hoovering is indeed very good. One reason is that
it multithreads the introduction of more risky stuff (by splitting up the
testing effort), and builds up confidence in more complex patches. This is
especially important in the stable cycle - eg. it happened not once that
had eg. some APIC cleanup that was too risky to be added to the stable
branch, and which went into your branch and lived a few iterations (got a
few bugreports) and then went to Linus.

Obviously you wont apply all the complex patches at once - i remember that
occasionally you delayed certain patches of mine because something else
was happening in your tree at that monent. You are simply doing
*different* transitions, but you are constrained by the same basic limits
as Linus' tree is.

Another reason is that you do much more housekeeping in areas that are not
actively maintained. But wouldnt it be better if there were active
maintainers in those areas as well so you could spend more time on eg.
doing the kernel-stack coloring changes?

but i truly believe that for the hard issues there is no solution, and
that most of the patch rejects are due to hard issues.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> for code areas where there is not active maintainer or the maintainer has
> ignored patches? Eg. the majority of the kdev transition patches went in
> smoothly.

No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
if you do some asking. And a measurable number of the listed maintainer
addresses just bounce.

> Another reason is that you do much more housekeeping in areas that are not
> actively maintained. But wouldnt it be better if there were active
> maintainers in those areas as well so you could spend more time on eg.
> doing the kernel-stack coloring changes?

There never will be maintainers proper for large amounts of stuff, and the
longer Linus deletes and ignores everything from someone new the less people
will bother sending to him. Just look at the size of the diff set between all
the vendor kernels and Linus 2.4.x trees before the giant -ac merge.

Think gcc, think egcs. History is merely beginning to repeat itself

Alan

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 01:40:50PM +0000, Alan Cox wrote:
 > 
 > No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
 > if you do some asking. And a measurable number of the listed maintainer
 > addresses just bounce.
 
 That's something that should really be fixed.
 I believe a while back someone was going to send a ping to all
 the listed addresses in MAINTAINERS. Doing this again may not
 be a bad idea.

 > There never will be maintainers proper for large amounts of stuff, and the
 > longer Linus deletes and ignores everything from someone new the less people
 > will bother sending to him. Just look at the size of the diff set between all
 > the vendor kernels and Linus 2.4.x trees before the giant -ac merge.

 Now that we have an open development branch again, perhaps its
 time for a lot of the things that have been proven stable in vendor
 kernels for a long time to get a looksee in mainline.
 Some things I feel will likely still be vendor-kernel only for some time.
 And some of them, rightly so.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Henning P. Schmiedehausen]

Dave Jones <davej@suse.de> writes:

> Now that we have an open development branch again, perhaps its
> time for a lot of the things that have been proven stable in vendor
> kernels for a long time to get a looksee in mainline.
> Some things I feel will likely still be vendor-kernel only for some time.
> And some of them, rightly so.

Bah. RedHat puts what? 120+? Patches into 2.2 (!) to ship their vendor
kernel. And it is still much more stable than any 2.4 I've encountered
till today.

I personally run a heavily patched 2.2 (+aa, +ide +reiser +ext3 +raid
and so on) and still get uptimes on busy and heavily loaded servers
(think newsserver with 50+ MBit/sec continous traffic. Think web
accelerator for one of the busiest web sites in Germany. Think mail
system for 1M users) far beyond 200 days uptime.

Will these patches ever be integrated? No. And same will go to some
sore spots of 2.4. Think RAID for 2.2. Think NFS for 2.2 where we
almost had to strangle Alan just to put in the most obvious bug fixes
from Trond :-) . In what? 2.2.16?

Do we have NFSv3 over TCP (which is in the real world available for
how many years?)? A really reliable IDE driver (Hi Andre :-) )? A raid
code that won't stuble over recoverable SCSI errors? That does not
interact badly with some FS types (and of course with those where RAID
would be really interesting)?

We have a kernel based Webserver. But not reliable media detection for
run-of-the-mill network cards. Something which "that other OS" has
since 1995.

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> [...] And a measurable number of the listed maintainer addresses just
> bounce.

out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
bounced physically. (whether they bounce logically is another question.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 05:15:15PM +0100, Ingo Molnar wrote:
 > > [...] And a measurable number of the listed maintainer addresses just
 > > bounce.
 > 
 > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
 > bounced physically. (whether they bounce logically is another question.)

 Care to remove the bogus ones from MAINTAINERS and send it to Linus/Me?
 (Keep the entries, but remove the address. (Or better perhaps to mark
 it 'out of order' in case of mailserver failure on $maintainers side.)

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Tue, Jan 29, 2002 at 03:27:32PM +0100, Dave Jones wrote:
> On Tue, Jan 29, 2002 at 05:15:15PM +0100, Ingo Molnar wrote:
>  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
>  > bounced physically. (whether they bounce logically is another question.)
> 
>  Care to remove the bogus ones from MAINTAINERS and send it to Linus/Me?
>  (Keep the entries, but remove the address. (Or better perhaps to mark
>  it 'out of order' in case of mailserver failure on $maintainers side.)

If we're going to be doing this periodically, it might be an idea to
put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
at the top of the file.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Russell King <rmk@arm.linux.org.uk> said:

[...]

> If we're going to be doing this periodically, it might be an idea to
> put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
> at the top of the file.

Perhaps add a "Last checked: field to each (too)?

But then again, patches to MAINTAINERS are silently dropped. Perhaps this
should be posted on kernel.org?
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Wed, Jan 30, 2002 at 10:44:24AM +0100, Horst von Brand wrote:
> Russell King <rmk@arm.linux.org.uk> said:
> > If we're going to be doing this periodically, it might be an idea to
> > put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
> > at the top of the file.
> 
> Perhaps add a "Last checked: field to each (too)?

Old ground - see Message-ID: <20020129144307.B6542@flint.arm.linux.org.uk>

> But then again, patches to MAINTAINERS are silently dropped. Perhaps this
> should be posted on kernel.org?

I suspect that's because Linus gets patches to it from people he's never
heard of before (ie, the new people taking over), but without seeing
anything that says that the old maintainer has gone.

I think this is something that someone could pick up this, and be the
contact point for both maintainers leaving, and new maintainers
taking over - if you like the MAINTAINERS maintainer.  Obviously this
should be someone Linus trusts.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Dave Jones wrote:

>  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
>  > bounced physically. (whether they bounce logically is another question.)
>
>  Care to remove the bogus ones from MAINTAINERS

yeah, was in the process of doing that. Patch against 2.5.3-pre6 attached.
Altogether 13 addresses are affected. I have only removed the
hard-bouncing email addresses, names and list names remain (of course).

	Ingo

--- linux/MAINTAINERS.orig	Tue Jan 29 15:11:04 2002
+++ linux/MAINTAINERS	Tue Jan 29 15:16:31 2002
@@ -154,8 +154,6 @@

 AD1816 SOUND DRIVER
 P:	Thorsten Knabe
-M:	Thorsten Knabe <tek@rbg.informatik.tu-darmstadt.de>
-M:	Thorsten Knabe <tek01@hrzpub.tu-darmstadt.de>
 W:	http://www.student.informatik.tu-darmstadt.de/~tek/projects/linux.html
 W:	http://www.tu-darmstadt.de/~tek01/projects/linux.html
 S:	Maintained
@@ -216,7 +214,6 @@

 ARPD SUPPORT
 P:	Jonathan Layes
-M:	layes@loran.com
 L:	linux-net@vger.kernel.org
 S:	Maintained

@@ -235,7 +232,6 @@

 BERKSHIRE PRODUCTS PC WATCHDOG DRIVER
 P:	Kenji Hollis
-M:	kenji@bitgate.com
 W:	http://ftp.bitgate.com/pcwd/
 S:	Maintained

@@ -433,13 +429,11 @@
 DIGI INTL. EPCA DRIVER
 P:	Chad Schwartz
 M:      support@dgii.com
-M:      chads@dgii.com
 L:      digilnux@dgii.com
 S:      Maintained

 DIGI RIGHTSWITCH NETWORK DRIVER
 P:	Rick Richardson
-M:	rick@remotepoint.com
 L:	linux-net@vger.kernel.org
 W:	http://www.dgii.com/linux/
 S:	Maintained
@@ -485,7 +479,6 @@

 DRM DRIVERS
 P:	Rik Faith
-M:	faith@valinux.com
 L:	dri-devel@lists.sourceforge.net
 S:	Supported

@@ -497,7 +490,6 @@

 EATA-DMA SCSI DRIVER
 P:	Michael Neuffer
-M:	mike@i-Connect.Net
 L:	linux-eata@i-connect.net, linux-scsi@vger.kernel.org
 S:	Maintained

@@ -927,7 +919,6 @@

 LOGICAL VOLUME MANAGER
 P:     Heinz Mauelshagen
-M:     mge@sistina.de
 L:     linux-LVM@sistina.com
 W:     http://www.sistina.com/lvm
 S:     Maintained
@@ -1134,7 +1125,6 @@

 OLYMPIC NETWORK DRIVER
 P:	Peter De Shrijver
-M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
 P:	Mike Phillips
 M:	mikep@linuxtr.net
 L:	linux-net@vger.kernel.org
@@ -1293,7 +1283,6 @@

 RISCOM8 DRIVER
 P:	Dmitry Gorodchanin
-M:	pgmdsg@ibi.com
 L:	linux-kernel@vger.kernel.org
 S:	Maintained

@@ -1660,13 +1649,11 @@
 USB SERIAL BELKIN F5U103 DRIVER
 P:	William Greathouse
 M:	wgreathouse@smva.com
-M:	wgreathouse@myfavoritei.com
 L:	linux-usb-users@lists.sourceforge.net
 L:	linux-usb-devel@lists.sourceforge.net
 S:	Maintained

 USB SERIAL CYBERJACK PINPAD/E-COM DRIVER
-M:	linux-usb@sii.li
 L:	linux-usb-users@lists.sourceforge.net
 L:	linux-usb-devel@lists.sourceforge.net
 S:	Supported
@@ -1792,7 +1779,6 @@

 ZF MACHZ WATCHDOG
 P:	Fernando Fuganti
-M:	fuganti@conectiva.com.br
 M:	fuganti@netbank.com.br
 W:	http://cvs.conectiva.com.br/drivers/ZFL-watchdog/
 S:	Maintained

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

>  ARPD SUPPORT
>  P:	Jonathan Layes
> -M:	layes@loran.com
>  L:	linux-net@vger.kernel.org
>  S:	Maintained

A correct patch for this one giving a new maintainer was posted to Linux
kernel already

>  DRM DRIVERS
>  P:	Rik Faith
> -M:	faith@valinux.com
>  L:	dri-devel@lists.sourceforge.net
>  S:	Supported

Rik moved from VA so I suspect you want to look in the RH internal directory
for the correct one there 8)

Alan

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> A correct patch for this one giving a new maintainer was posted to
> Linux kernel already

ok.

> >  DRM DRIVERS
> >  P:	Rik Faith
> > -M:	faith@valinux.com
> >  L:	dri-devel@lists.sourceforge.net
> >  S:	Supported
>
> Rik moved from VA so I suspect you want to look in the RH internal directory
> for the correct one there 8)

*blush* :-) The correct patch is:

-M:	faith@valinux.com
+M:	faith@redhat.com

	Ingo

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Ingo Molnar wrote:
> 
> On Tue, 29 Jan 2002, Dave Jones wrote:
> 
> >  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
> >  > bounced physically. (whether they bounce logically is another question.)
> >
> >  Care to remove the bogus ones from MAINTAINERS
> 
> yeah, was in the process of doing that. Patch against 2.5.3-pre6 attached.
> Altogether 13 addresses are affected. I have only removed the
> hard-bouncing email addresses, names and list names remain (of course).
> 
>         Ingo

Humble suggestion:  Add a date field for "took over maintainence
on/before: yyyy-mm-dd" and a field for "last verified: yyyy-mm-dd" so we
know when we last checked on the existance/etc. of a maintainer.  (And
maybe an "AWAL on/before: yyyy-mm-dd" for those without known working
addresses.)

Thoughts?

Ah, I see Russell King made a similar suggestion...

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

Patrick Mauritz noticed that this one is actually a typo in the
MAINTAINERS file:

> -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be

so the correct (and existing) email address is:

-M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
+M:	p2@ace.ulyssis.student.ac.be

	Ingo

Re: A modest proposal -- We need a patch penguin

[Patrick Mauritz]

On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
> +M:	p2@ace.ulyssis.student.ac.be
fixing the fix:
+M:	p2@ace.ulyssis.student.kuleuven.ac.be

sorry,
patrick mauritz
-- 
,------------------------------------------------------------------------.
>   In the Beginning there was nothing, which exploded - Yeah right...   <
|------------------------------------------------------------------------|
>           Debian GNU/Linux, apt-get into it | www.debian.org           <
`------------------------------------------------------------------------'
 If you gave an infinite number of monkeys an infinite number of typewr..
                         Hey! that's the internet!

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in <20020129145344.GC2611@hydra>:

> On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
> > +M:	p2@ace.ulyssis.student.ac.be
> fixing the fix:
> +M:	p2@ace.ulyssis.student.kuleuven.ac.be

I thought that was the bouncing address?!

MfG Kai

Re: A modest proposal -- We need a patch penguin

[Arnaldo Carvalho de Melo]

Em Tue, Jan 29, 2002 at 10:03:00PM +0200, Kai Henningsen escreveu:
> oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in <20020129145344.GC2611@hydra>:
> 
> > On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
                                ^^
> > > +M:	p2@ace.ulyssis.student.ac.be
> > fixing the fix:
> > +M:	p2@ace.ulyssis.student.kuleuven.ac.be
                        ^^
> I thought that was the bouncing address?!

got it? :)

- Arnaldo

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

acme@conectiva.com.br (Arnaldo Carvalho de Melo)  wrote on 30.01.02 in <20020130031528.GF973@conectiva.com.br>:

> Em Tue, Jan 29, 2002 at 10:03:00PM +0200, Kai Henningsen escreveu:
> > oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in
> > <20020129145344.GC2611@hydra>:
> >
> > > On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > > > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
>                                 ^^
> > > > +M:	p2@ace.ulyssis.student.ac.be
> > > fixing the fix:
> > > +M:	p2@ace.ulyssis.student.kuleuven.ac.be
>                         ^^
> > I thought that was the bouncing address?!
>
> got it? :)

Now. I certainly looked at those lines for several minutes without ...

Sometimes a wdiff would be better, it seems :-)


MfG Kai

Re: A modest proposal -- We need a patch penguin

[toon]

On Tue, Jan 29, 2002 at 05:36:07PM +0100, Ingo Molnar wrote:
> 
> 
> @@ -927,7 +919,6 @@
> 
>  LOGICAL VOLUME MANAGER
>  P:     Heinz Mauelshagen
> -M:     mge@sistina.de
>  L:     linux-LVM@sistina.com
>  W:     http://www.sistina.com/lvm
>  S:     Maintained

I just checked the linux-lvm mailing list.
Heinz seems to be mailing from: mauelshagen@sistina.com

Regards,
Toon.
-- 
 /"\                             |   Windows XP:
 \ /     ASCII RIBBON CAMPAIGN   |        "Sorry Dave...
  X        AGAINST HTML MAIL     |         I'm afraid I can't do that."
 / \

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Ingo Molnar <mingo@elte.hu> said:

[...]

>  ARPD SUPPORT
>  P:	Jonathan Layes
> -M:	layes@loran.com
>  L:	linux-net@vger.kernel.org
>  S:	Maintained

I seem to remember a message on lkml from a gal claiming (a) that Loran had
been bought by somebody else, and (b) that she had taken over
maintainership of this, after somewhat of elbowing around with the new
ownership. She asked the entry to be corrected...
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:40 am, Alan Cox wrote:
> > for code areas where there is not active maintainer or the maintainer has
> > ignored patches? Eg. the majority of the kdev transition patches went in
> > smoothly.
>
> No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
> if you do some asking. And a measurable number of the listed maintainer
> addresses just bounce.

I'm under the impression Michael Elizabeth Chastain is one such burned out 
maintainer, but hasn't been able to hand over maintainership because Linus 
keeps dropping his patch to change the maintainers file to say "Peter 
Samuelson", and he eventually just gave up trying.

I could be wrong about this.  Ask him.  Or maybe his maintainer hand-over 
patch needs more code review?

> > Another reason is that you do much more housekeeping in areas that are
> > not actively maintained. But wouldnt it be better if there were active
> > maintainers in those areas as well so you could spend more time on eg.
> > doing the kernel-stack coloring changes?
>
> There never will be maintainers proper for large amounts of stuff, and the
> longer Linus deletes and ignores everything from someone new the less
> people will bother sending to him.

Case in point:

--- linux/arch/i386/boot/bootsect.S.old Tue Jan  1 19:41:22 2002
+++ linux/arch/i386/boot/bootsect.S     Tue Jan  1 19:44:02 2002
@@ -158,9 +158,7 @@
        movw    $sread, %si             # the boot sector has already been 
read
        movw    %ax, (%si)

-       xorw    %ax, %ax                # reset FDC
-       xorb    %dl, %dl
-       int     $0x13
+       call    kill_motor              # reset FDC
        movw    $0x0200, %bx            # address = 512, in INITSEG
 next_step:
        movb    setup_sects, %al

Dumb little nit I noticed a few weeks ago, but never bothered to follow up 
on, because it's just not worth it.  Not that potentially saving 3 bytes out 
of the boot sector is a BAD thing, but it's not good enough to be worth the 
effort anymore.  Warning fixing patches are largely the same way: easy to do, 
but why?

This didn't strike me as a healthy development, really...

> Just look at the size of the diff set
> between all the vendor kernels and Linus 2.4.x trees before the giant -ac
> merge.
>
> Think gcc, think egcs. History is merely beginning to repeat itself

I was actually hoping to AVOID that.  (There IS still time.  We're not that 
badly off.  Yet.  I'm just a bit nervous about direction.  The kind of 
stresses I've seen seem (to me) unlikely to improve with time...)

And we ARE using a patch penguin.  You were around, and Dave is around.  I'm 
kind of confused at the level of resistence to formally recognizing what 
basically IS current practice, and has been for YEARS.  (The reason for 
naming the position is we can't just say "alan's tree" anymore.  The position 
went from one person to another person, and as such the position seemed to 
need to be recognized as being separate from the individual.  I didn't expect 
to hit a brick wall on that.  It  didn't seem like a revolutionary idea to 
me...)

> Alan

Re: A modest proposal -- We need a patch penguin

[Eric S. Raymond]

Rob Landley <landley@trommello.org>:
> And we ARE using a patch penguin.  You were around, and Dave is
> around.  I'm kind of confused at the level of resistence to formally
> recognizing what basically IS current practice, and has been for
> YEARS.  (The reason for naming the position is we can't just say
> "alan's tree" anymore.  The position went from one person to another
> person, and as such the position seemed to need to be recognized as
> being separate from the individual.  I didn't expect to hit a brick
> wall on that.  It didn't seem like a revolutionary idea to me...)

Alas.  That's because, like most Americans these days, you're
historically illiterate.  What we are facing here is a *very* familiar
problem to social and institutional historians. 

All movements founded by charismatic leaders like Linus eventually hit
this same wall -- the point at which the charisma of the founder and
the individual ability of the disciples he personally attracts are no
longer adequate to meet the challenges of success, and some way to
institutionalize and distribute the leader's role has to be found.
Movements that fail to make this transition die, generally by
implosion or fragmenting into feuding sub-sects.

If you were familiar with the historical precedents, Rob, you would 
understand that your modest proposal re-enacts a common pattern.
A relatively junior member of the movement, one with few political
ties, sees the developing stress fractures in the organization of
the movement and proposes a modest, incremental change to relieve
some of them.  Conservatives interpret the attempt to separate 
and institutionalize part of the founder's role as an attack on
the authority of the founder.  Huge flamewars ensue, with the
original pragmatic sense of the proposal often being lost as it
becomes a political football in the movement's internal status games.

Sometimes the first such attempt at institutionization succeeds.  More
often, the movement has to go through a series of escalating crises
(burning up would-be reformers each time) before anyone finally
succeeds in changing the movement's internal culture.

Religions go through this.  Secular social movements go through this.
Companies founded by brilliant entrepreneurs go through this (the
B-schools have a whole literature on "entrepreneurial overcontrol" and
its consequences).  It's one of the dramas that gets perpetually
re-enacted; it's built in to our wiring.  The unhappy truth is that
even *successful* transitions of this kind are invariably painful, and
often leave deep scars on the survivors and on the institution that
arises from the transition.

*Never* expect this sort of transition to be easy, especially when the
positions people are taking are as much about personal identity and
values as they are about "success" in whatever terms the movement
defines it.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Re: A modest proposal -- We need a patch penguin

[Mark Hahn]

On Tue, 29 Jan 2002, Eric S. Raymond wrote:
...
> Alas.  That's because, like most Americans these days, you're
> historically illiterate.  What we are facing here is a *very* familiar

can majordomo be configured to rewrite Erik's name as "Pappy Raymond"?

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Alan Cox wrote:

> Ingo, you should have a look at my mailbox and the people sick of trying to
> get Linus to take 3 liners to fix NODEV type stuff and being ignored so that
> 2.5.* still doesn't even compile or boot for many people.
> 
> Dave in doing the patch hoovering at least ensures these are picked up. You
> think if this carries on anyone will be running Linus tree in 9 months ?

Linus, I think I hear people you said you trust telling you this... take
the little bits out of band and TRUST people to give you patches which go
in 2.5.x now, not when or if you get to it. Having you approve trivial
stuff is a waste of what you do best, and I think all the versions show
you're not even being effective at that. Don't you HATE looking at
spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
it out and let other people do it, and work on the fun stuff.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Craig Christophel]

> Linus, I think I hear people you said you trust telling you this... take
> the little bits out of band and TRUST people to give you patches which go
> in 2.5.x now, not when or if you get to it. Having you approve trivial
> stuff is a waste of what you do best, and I think all the versions show
> you're not even being effective at that. Don't you HATE looking at
> spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
> it out and let other people do it, and work on the fun stuff.

	aasn, (as a side note)  It's really hard to allow people to just change the 
little peices.  The little peices soon become larger and more complex.  This 
is a really difficult transition in moving from (my personal perspective) an 
exciting young code base to a maturing and well functioning/planned setup.  
The only thing I have to say is that Linus had better pick his talent and 
friends well, because even if the codebase does not split today, what is to 
keep it from doing so in the future when even more complexities arise.  

	Linus,
		It doesn't have to be difficult, just as you have maintainers for the 
"stable" series yet still have a say on what happens, define the level of 
modularity you would like to happen for different maintainers to be able to 
take care of their little peices and it will happen.  Please do not allow the 
BSD split to happen here.  Be proactive and take a hard look at what you 
really want.  

	Please,

Craig.

Re: A modest proposal -- We need a patch penguin

[Shawn]

On 01/29, Craig Christophel said something like:
> > Linus, I think I hear people you said you trust telling you this... take
> > the little bits out of band and TRUST people to give you patches which go
> > in 2.5.x now, not when or if you get to it. Having you approve trivial
> > stuff is a waste of what you do best, and I think all the versions show
> > you're not even being effective at that. Don't you HATE looking at
> > spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
> > it out and let other people do it, and work on the fun stuff.
> 
> 	aasn, (as a side note)  It's really hard to allow people to just change the 
> little peices.  The little peices soon become larger and more complex.  This 

I believe he was talking about some of the example one-liners cited
earlier. Not a significant chance of those getting "more complex" such
that they become cumbersome...

> is a really difficult transition in moving from (my personal perspective) an 
> exciting young code base to a maturing and well functioning/planned setup.  
> The only thing I have to say is that Linus had better pick his talent and 
> friends well, because even if the codebase does not split today, what is to 
> keep it from doing so in the future when even more complexities arise.  

The codebase has been, is now, and forever will be split, due to
differing goals, and the fact that one size does not fit all... Not to
mention personal taste. The splits will only be as large a practicality
allows, given the number of trees trying to sync from, or with -linus.
The -ac tree got pretty splorked off for a while, though.

By the way, Linus, Alan, Al Viro, Marcello, Ingo, Stephen, Rik, Dave,
Dave, any other Daves, Geert, Jens, Andre, Richard, hpa, Hans, and
everyone I can't think of, THANKS for making my machine run fast, die
hard, and just for putting out better code than a huge corporation can!

--
Shawn Leas
core@enodev.com

For my birthday I got a humidifier and a de-humidifier... I put them
in the same room and let them fight it out...
						-- Stephen Wright

Re: A modest proposal -- We need a patch penguin

[Skip Ford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
[snip]
> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and

Then why not give the subsystem maintainers patch permissions on your tree.
Sort of like committers.  The problem people have is that you're dropping
patches from those ten-twenty people you trust.

Each subsystem maintainer should handle patches to that subsystem, and
you should remove your own patch permissions for only those subsystems.
You could get involved with only changes in direction that affect more
than one subsystem.

> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me. 
> 
> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

- -- 
Skip  ID: 0x7EDDDB0A
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAjxWsfkACgkQBMKxVH7d2wpAfQCfRMoMirEH2/KRYMowNkZyMCBi
CEYAoM4akKt8Ifl20cvkA5UG8Kb4p9Tb
=q5bM
-----END PGP SIGNATURE-----

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Skip Ford wrote:
>
> Linus Torvalds wrote:
> [snip]
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
>
> Then why not give the subsystem maintainers patch permissions on your tree.
> Sort of like committers.  The problem people have is that you're dropping
> patches from those ten-twenty people you trust.

No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
_not_ dropping their patches (although I'm sometimes critical of them,
and will tell them that they do not get applied).

Sure, it happens occasionally that they really do get dropped, just
because I get too much email, but these people know how to re-send every
once in a while, and keep their patches separate.

I think there is some confusion about who I trust. Being listed as
MAINTAINER doesn't mean you are automatically trusted. The MAINTAINERS
list is not meant for me _at_all_ in fact, it's meant more as one of the
places for _others_ to search for a contact with.

Examples of people who I trust: Ingo Molnar, Jeff Garzik, Alan Cox, Al
Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
"good taste" for a long time. But it's not always a long process - some
of you may remember Bill Hawes, for example, who came out of nowhere
rather quickly.

There are other categories: Andrea, for example, is in a category all of
his own under "absolutely stunning, but sometimes somewhat erratic", which
just means that I have to think a lot more about his patches. I love his
experimentation, especially now that he maintains separate patches (and
I'd also love for him to be more active in pushing the non-experimental
parts towards me, hint hint, Andrea)

And there are categories of people who just own a big enough chunk that is
separate enough that I trust them for that area: architecture maintainers
etc tend to be here, as long as they only affect their own architecture.

But you have to realize that there are a _lot_ of people on the
maintainers list that I don't implicitly trust. And being loud and
wellknown on the mailing lists or IRC channels doesn't make them any more
trusted.

			Linus

Re: A modest proposal -- We need a patch penguin

[Michael Sterrett -Mr. Bones.-]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> "But it's not always a long process - some of you may remember Bill
> Hawes, for example, who came out of nowhere rather quickly.

I remember being very impressed by Bill.  What ever happened to him?

Michael Sterrett
  -Mr. Bones.-
michael.sterrett@coat.com

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 12:36 pm, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Skip Ford wrote:
> > Linus Torvalds wrote:
> > [snip]
> >
> > > A word of warning: good maintainers are hard to find.  Getting more of
> > > them helps, but at some point it can actually be more useful to help
> > > the _existing_ ones.  I've got about ten-twenty people I really trust,
> > > and
> >
> > Then why not give the subsystem maintainers patch permissions on your
> > tree. Sort of like committers.  The problem people have is that you're
> > dropping patches from those ten-twenty people you trust.
>
> No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> _not_ dropping their patches (although I'm sometimes critical of them,
> and will tell them that they do not get applied).

Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel 
Boldt...

> I think there is some confusion about who I trust. Being listed as
> MAINTAINER doesn't mean you are automatically trusted. The MAINTAINERS
> list is not meant for me _at_all_ in fact, it's meant more as one of the
> places for _others_ to search for a contact with.

Ah.  So being listed in the maintainers list doesn't mean someone is actually 
a maintainer it makes sense to forward patches to?

Are you backing away from the maintainer system, or were the rest of us 
misinterpreting what it meant all along?  Does being a maintainer mean you 
feel you have delegated any actual authority to them at all, or is it merely 
a third party tech support contact point?

You seem to be saying "send patches to maintainers, support the maintainers 
better, but don't expect me to necessarily take patches from them".  Who 
should patches be sent TO?

> Examples of people who I trust: Ingo Molnar, Jeff Garzik, Alan Cox, Al
> Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> "good taste" for a long time. But it's not always a long process - some
> of you may remember Bill Hawes, for example, who came out of nowhere
> rather quickly.

So listed "maintainers" may need to forward patches to these people, and get 
them to sign off on them, in order to get their patches at least reviewed for 
inclusion into your tree?

If that's the process, fine.  I'm just trying to clarify what the process IS, 
other than spamming your mailbox with a cron job (as has been suggested and 
actually taken seriously as long as re-testing is also automated)...

> And there are categories of people who just own a big enough chunk that is
> separate enough that I trust them for that area: architecture maintainers
> etc tend to be here, as long as they only affect their own architecture.
>
> But you have to realize that there are a _lot_ of people on the
> maintainers list that I don't implicitly trust. And being loud and
> wellknown on the mailing lists or IRC channels doesn't make them any more
> trusted.

I've noticed that rather a lot of development seems to be moving to IRC.  How 
is this NOT to be interpreted as a lack of endorsement of the functionality 
of the other channels?  Is IRC "just nice", or does IRC address a problem not 
otherwise being addressed?

> 			Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Rob Landley wrote:
> > >
> > > Then why not give the subsystem maintainers patch permissions on your
> > > tree. Sort of like committers.  The problem people have is that you're
> > > dropping patches from those ten-twenty people you trust.
> >
> > No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> > _not_ dropping their patches (although I'm sometimes critical of them,
> > and will tell them that they do not get applied).
>
> Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
> Boldt...

NONE of those are in the ten-twenty people group.

How many people do you think fits in a small group? Hint. It sure isn't
all 300 on the maintainers list.

> Ah.  So being listed in the maintainers list doesn't mean someone is actually
> a maintainer it makes sense to forward patches to?

Sure it does.

It just doesn't mean that they should send stuff to _me_.

Did you not understand my point about scalability?  I can work with a
limited number of people, and those people can work with _their_ limited
number of people etc etc.

The MAINTAINERS file is _not_ a list of people I work with on a daily
basis. In fact, I don't necessarily even recognize the names of all those
people.

Let's take an example. Let's say that you had a patch for ppp. You'd send
the patch to Paul Mackerras. He, in turn, would send his patches to David
Miller (who knows a hell of a lot better what it's all about than I do).
And he in turn sends them to me.

They are both maintainers. That doesn't mean that I necessarily work with
every maintainer directly.

Or look at USB: I get the USB patches from Greg, and he gets them from
various different people. Johannes Erdfelt is the maintainer for uhci.c,
and he sends them to Greg, not to me.

Why? Because having hundreds of people emailing me _obviously_ doesn't
scale. Never has, never will. It may work over short timeperiods wih lots
of energy, but it obviously isn't a stable setup.

			Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> > Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth
> > Chastain, Axel Boldt...
>
> NONE of those are in the ten-twenty people group.
>
> How many people do you think fits in a small group? Hint. It sure
> isn't all 300 on the maintainers list.

That's fine with me, but _who_ do I send VM patches to if
I can't send them to you ?

There is no maintainer for mm/* or kernel/*, it's just you.

I agree it would be nice to have somebody from within the
ten-twenty people group take care of that, but we'll need
to have _somebody_ ...

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Rik van Riel wrote:
>
> That's fine with me, but _who_ do I send VM patches to if
> I can't send them to you ?

The VM stuff right now seems to be Andrea, Dave or you yourself (right now
I just wish you would split up your patches like Andrea does, that way I
can cherry-pick).

> There is no maintainer for mm/* or kernel/*, it's just you.

As to kernel/ there are actually maintainers for some sub-areas, the most
noticeable being Ingo on the scheduler. The rest of kernel/ hasn't ever
been much of a problem, really.

The VM is a big issue, of course. And that one isn't likely to go away
anytime soon as a point of contention. And it's not easy to modularize,
apart from the obvious pieces (ie "filemap.c" vs the rest).

You may not believe me when I say so, but I personally _really_ hope your
rmap patches will work out. I may not have believed in your patches in a
2.4.x kind of timeframe, but for 2.6.x I'm more optimistic. As to how to
actually modularize it better to make points of contention smaller, I
don't know how.

At the same time, while I can understand your personal pain, I don't think
most of the problems have been with the VM (maintenance-wise, that is.
Most of the _technical_ problems really have been with the VM, it's just
the most complex piece).

		Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Tue, 29 Jan 2002, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Rik van Riel wrote:
> >
> > That's fine with me, but _who_ do I send VM patches to if
> > I can't send them to you ?
>
> The VM stuff right now seems to be Andrea, Dave or you yourself (right
> now I just wish you would split up your patches like Andrea does, that
> way I can cherry-pick).

I will.  It's not split up at the moment because I'd like to
work a bit more on -rmap before submitting it for inclusion
and bitkeeper is a really nice tool to help me carry the patch
from version to version.

If -rmap makes it into the kernel I'll work with small patches
in the same style as Andrea, that's just the easiest way to
work.

I'll also take some of rusty's scripts to automatically check
if a patch (1) has been applied to the latest kernel or
(2) if it still applies cleanly.

Basically I'm looking for a way to minimise the work of
carrying the -rmap VM across kernel versions, so I can spend
my time doing development and cleaning up the code further.

> The VM is a big issue, of course. And that one isn't likely to go away
> anytime soon as a point of contention. And it's not easy to modularize,
> apart from the obvious pieces (ie "filemap.c" vs the rest).

Actually some stuff can be modularised somewhat. Christoph
Hellwig for example has a nice patch which replaces all
knowledge of page->wait with wake_up_page().

This makes the fact of whether we're using per-page waitqueues
or hashed waitqueues completely invisible to the rest of the
kernel.

Similar things are possible for other areas of the code.

> You may not believe me when I say so, but I personally _really_ hope your
> rmap patches will work out. I may not have believed in your patches in a
> 2.4.x kind of timeframe, but for 2.6.x I'm more optimistic. As to how to
> actually modularize it better to make points of contention smaller, I
> don't know how.

One thing William Irwin (and others, myself too) have been
looking at is making the pagemap_lru_lock per-zone.

This would allow us to "split up" memory in zones and have
each CPU start the allocation chain at its own zone.

This works out in practice because the reverse mapping code
allows us to scan and free memory by physical address,
meaning that reclaim_page() becomes a per-CPU local thing
under light memory loads.

Of course the current problem with that code is truncate
and the lock ordering between the pagemap_lru_lock and the
page_cache_lock ... something to look at later.

kind regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Daniel Jacobowitz]

On Tue, Jan 29, 2002 at 03:50:43PM -0800, Linus Torvalds wrote:
> > Ah.  So being listed in the maintainers list doesn't mean someone is actually
> > a maintainer it makes sense to forward patches to?
> 
> Sure it does.
> 
> It just doesn't mean that they should send stuff to _me_.
> 
> Did you not understand my point about scalability?  I can work with a
> limited number of people, and those people can work with _their_ limited
> number of people etc etc.
> 
> The MAINTAINERS file is _not_ a list of people I work with on a daily
> basis. In fact, I don't necessarily even recognize the names of all those
> people.

I understand that the sort of careful hierarchy that drives this
process is, by nature, an informal thing.  But it would still be nice
if _suggestions_ on how it worked were written down somewhere.  When
you've got patches that don't have a clear relevant maintainer, it
would be nice to have something more specific than "post to
linux-kernel and pray someone picks it up" to run with!

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: A modest proposal -- We need a patch penguin

[Chris Ricker]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> They are both maintainers. That doesn't mean that I necessarily work with
> every maintainer directly.
> 
> Or look at USB: I get the USB patches from Greg, and he gets them from
> various different people. Johannes Erdfelt is the maintainer for uhci.c,
> and he sends them to Greg, not to me.
> 
> Why? Because having hundreds of people emailing me _obviously_ doesn't
> scale. Never has, never will. It may work over short timeperiods wih lots
> of energy, but it obviously isn't a stable setup.

Linus,

That's fine, but there's a major problem with your scheme.  What happens
with all the stuff for which no one is listed in MAINTAINERS?  For example,
no one owns linux/Documentation.  As the person nominally in charge of
linux/Documentation/Changes, there's no one between me and you, period, let
alone anyone between me and you that you trust....  And I realize that you
don't consider documentation very important, but there are other segments of
the Linux source tree for which this breakdown in hierarchy is also true....

later,
chris

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Chris Ricker wrote:
>
> That's fine, but there's a major problem with your scheme.  What happens
> with all the stuff for which no one is listed in MAINTAINERS?

I have to admit that personally I've always found the MAINTAINERS file
more of an irritation than anything else. The first place _I_ tend to look
personally is actually in the source files themselves (although that may
be a false statistic - the kind of people I tend to have to look up aren't
the main maintainers at all, but more single driver people etc).

It might not be a bad idea to just make that "mention maintainer at the
top of the file" the common case.

		Linus

Re: A modest proposal -- We need a patch penguin

[Miles Lane]

On Tue, 2002-01-29 at 16:44, Linus Torvalds wrote:
> 
> On Tue, 29 Jan 2002, Chris Ricker wrote:
> >
> > That's fine, but there's a major problem with your scheme.  What happens
> > with all the stuff for which no one is listed in MAINTAINERS?
> 
> I have to admit that personally I've always found the MAINTAINERS file
> more of an irritation than anything else. The first place _I_ tend to look
> personally is actually in the source files themselves (although that may
> be a false statistic - the kind of people I tend to have to look up aren't
> the main maintainers at all, but more single driver people etc).
> 
> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

I do similarly when I am testing Gnome software, but there
I have the CVS sources to look at, including carefully updated
ChangeLog files.  I find the ChangeLogs and the output of
"cvs log ChangeLog" to be highly informative and helpful when
attempting to track down the appropriate person to contact.
Is it feasible to set up a read-only anonymous cvs server for
the kernel tree?  It seems to me that it would be nice to 
good to have ChangeLogs for the kernel directories as well.

	Miles

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:38 pm, Miles Lane wrote:
> On Tue, 2002-01-29 at 16:44, Linus Torvalds wrote:
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
>
> I do similarly when I am testing Gnome software, but there
> I have the CVS sources to look at, including carefully updated
> ChangeLog files.  I find the ChangeLogs and the output of
> "cvs log ChangeLog" to be highly informative and helpful when
> attempting to track down the appropriate person to contact.
> Is it feasible to set up a read-only anonymous cvs server for
> the kernel tree?  It seems to me that it would be nice to
> good to have ChangeLogs for the kernel directories as well.

This isn't necessarily a problem for Linus to handle.

Right now, it's pretty easy to find/generate diffs between each "pre 
release".  Each of those could be incrementally fed into a CVS server, and 
bang, you have a revision history.  The granualrity might not be the 
greatest, but it's a start, and it can be done retroactively.  (I vaguely 
remember hearing some work along these lines...)

Now to get the kind of patch level granuarity that Linus likes to have made 
available to the rest of the world, you need the actual patches, as applied, 
made available.  Getting a patch penguin (I.E. Alan Cox or Dave Jones) to do 
this might not be too hard (as long as it's not too much work), but not 
enough patches go through them at the moment to necessarily make it 
worthwhile.

Long ago I suggested that since the way Linus works is "append various emails 
to a big file, then feed that to patch(1) at the end of a mail run", it 
should be possible to send Linus a perl script that copies the individual 
emails from the big file to a mailing list when he patches his tree.  Not 
just the actual patch, but the whole email with the description of the fix 
and everything.  (Again, no guarantee he wouldn't back them out again, but 
it's something that really requires no extra work on his part, gives 
immediate acknowledgement that he's looked at something, and gives the rest 
of the world access to the level of granularity he expects to receive from 
them.)

Of course until such a script is actually written, with a mailing list set up 
for it to post to (read-only except for Linus), it's just an idle thought I 
haven't had time to pursue.  (The diffs between pre-versions have generally 
been good enough for me personally, so...)

If the "patches-to-linus" list does get implemented, it would probably also 
be fairly easy to automatically match new pre-X->pre-Y diffs against the 
recent patches from the list, and extract most of the information that way.  
(Assuming Linus doesn't modify them too much, or end up taking a lot of 
patches from other sources.  A human would probably still have to do at least 
part of it, but it might be an improvement on just putting the whole big 
version diff in the cvs tree as one lump...)

> 	Miles

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 03:06:15AM -0500, Rob Landley wrote:
> If the "patches-to-linus" list does get implemented, it would probably also 
> be fairly easy to automatically match new pre-X->pre-Y diffs against the 
> recent patches from the list, and extract most of the information that way.  
> (Assuming Linus doesn't modify them too much, or end up taking a lot of 
> patches from other sources.  A human would probably still have to do at least 
> part of it, but it might be an improvement on just putting the whole big 
> version diff in the cvs tree as one lump...)

Instead of doing this stuff half-assed, just convince Linus to use BK :)

	Jeff

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 03:47:46AM -0500, Jeff Garzik wrote:
> Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> 	Jeff

It might be good if I (and others who are interested) worked on a document
describing the work flow that has been discussed here, in particular
the sort of network of people that Linus mentioned and Dave showed in
his ascii art.  That document would be a useful thing to hand to newbies
regardless of whether BK or any other CVS is in the picture because it
would describe how things work today, more or less.

It is also useful as a stake in the ground which says that any CVS
which wants to be considered in this problem space has to solve that
work flow, or come pretty darn close.  If BK can do it, that would be
good, that's why we built it.  If it can't, you still get a useful doc
describing how things work and we can go back and try and fix BK.

If you think this is a good idea and you are one of the 20-30 maintainers
close to the center of the network, let me know.  If anyone wants to work
on this or just watch the progress, I can have a mailing list set up 
tomorrow.  If there is interest in the doc, I think I could have something
with nice pictures and text in a week.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <20020130034746.K32317@havoc.gtf.org>,
Jeff Garzik  <garzik@havoc.gtf.org> wrote:
>
>Instead of doing this stuff half-assed, just convince Linus to use BK :)

The thing is, I actually _want_ to use BK (as opposed to CVS, which I
really don't think cuts it). 

I still dislike some things (those SHOUTING SCCS files) in bk, and let's
be honest: I've used CVS, but I've never really used BK. Larry has given
me the demos, and I actually decided to re-do the examples, but it takes
time and effort to get used to new tools, and I'm a bit worried that
I'll find other things to hate than just those loud filenames.

This is partly why I asked how many people use it..

		Linus

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:33:19AM +0000, Linus Torvalds wrote:
> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
> be honest: I've used CVS, but I've never really used BK. Larry has given
> me the demos, and I actually decided to re-do the examples, but it takes
> time and effort to get used to new tools, and I'm a bit worried that
> I'll find other things to hate than just those loud filenames.

One issue I'm interested in, and Larry and I have chatted about this a
couple times, is making sure that the "standard" patch flow isn't
affected... and what I mean by that is out-of-order and/or modified
patches.

Say you apply patches A, B, and E from an Al Viro patch series,
reject D, and apply patch C but tweak it yourself [sb->s_id is case
in point IIRC].  Say further that Al sent you a BK patch.  (ha! but
bear with me :))  I want to be confident that BK does not cause
downstream patches to impose constraints on you which prevent or make
difficult weird cases like this, just to ensure that BK's idea of a
global tree remains intact.

Experience and additional BK knowledge on my part will likely clear this
up, but IIRC this was one of the larger issues with not only you but
many others concurrently developing on what I would call the "global
Linux tree."

Obviously this wouldn't apply if you fed BK patches into GNU patch, and
then issued the commit from there...  but that way is a bit lossy, since
you would need to recreate rename information among other things.

In any case, I think BK is pretty nifty so far, but want to practice
by importing all Linux patches into a tree before converting my own
"gkernel" cvs to BK.  (tytso disagrees and thinks that there should be a
separate BK tree for 2.4, 2.5,... IMHO: ug.)

	Jeff,
	who should really get sleep before tomorrow's LW-NY

Re: A modest proposal -- We need a patch penguin

[Momchil Velikov]

>>>>> "Linus" == Linus Torvalds <torvalds@transmeta.com> writes:

Linus> In article <20020130034746.K32317@havoc.gtf.org>,
Linus> Jeff Garzik  <garzik@havoc.gtf.org> wrote:
>> 
>> Instead of doing this stuff half-assed, just convince Linus to use BK :)

Linus> The thing is, I actually _want_ to use BK (as opposed to CVS, which I
Linus> really don't think cuts it). 

Well, I know now the whole linux community will hate me, but this has
never stopped me before, it won't now.

How about using something FREE ?

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:33 am, Linus Torvalds wrote:
> In article <20020130034746.K32317@havoc.gtf.org>,
> Jeff Garzik  <garzik@havoc.gtf.org> wrote:
> >
> >Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> The thing is, I actually _want_ to use BK (as opposed to CVS, which I
> really don't think cuts it). 
> 
> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
> be honest: I've used CVS, but I've never really used BK. Larry has given
> me the demos, and I actually decided to re-do the examples, but it takes
> time and effort to get used to new tools, and I'm a bit worried that
> I'll find other things to hate than just those loud filenames.

Oh gosh, I hate those too.  (Yes, this is a "me too".)  Larry, could we 
*please* have that metadata in a .file?

/me also detests those shouting CVS's in cvs trees and is easily put off
by such trivial things

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Albert D. Cahalan]

Daniel Phillips writes:
> On January 30, 2002 10:33 am, Linus Torvalds wrote:

>> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
>> be honest: I've used CVS, but I've never really used BK. Larry has given
>> me the demos, and I actually decided to re-do the examples, but it takes
>> time and effort to get used to new tools, and I'm a bit worried that
>> I'll find other things to hate than just those loud filenames.
>
> Oh gosh, I hate those too.  (Yes, this is a "me too".)  Larry, could we 
> *please* have that metadata in a .file?

Try "man ls":

       -I, --ignore=PATTERN
              do not list implied entries matching shell PATTERN


So then something like this...

alias ls='/bin/ls --ignore=SCCS'

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> 
> So then something like this...
> 
> alias ls='/bin/ls --ignore=SCCS'

Oh, that's very useful. Considering that everything else still finds them, 
like find, shell autocompletion etc.

The only thing "--ignore=xxx" is useful for is hackers that want to break 
into your system and hide their files.

		Linus

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On April 5, 2002 03:21 am, Linus Torvalds wrote:
> On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> > 
> > So then something like this...
> > 
> > alias ls='/bin/ls --ignore=SCCS'
> 
> Oh, that's very useful. Considering that everything else still finds them, 
> like find, shell autocompletion etc.
> 
> The only thing "--ignore=xxx" is useful for is hackers that want to break 
> into your system and hide their files.

And anyway, Larry sorta/kinda agreed to let us hide his bk metadata in one or
more hidden files, and when I grab him for clubbing^W dinner in a few days
I'll have a good chance to beat on him further to actually get that little
feature, which means more to me than it really should, personally.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Thu, 4 Apr 2002, Linus Torvalds wrote:
> On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> > 
> > So then something like this...
> > 
> > alias ls='/bin/ls --ignore=SCCS'
> 
> Oh, that's very useful. Considering that everything else still finds them, 
> like find, shell autocompletion etc.
> 
> The only thing "--ignore=xxx" is useful for is hackers that want to break 
                                                 ^^^^^^^
> into your system and hide their files.

Ugh, this is linux-kernel! (cfr. signature)

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Jeff Garzik wrote:

> Instead of doing this stuff half-assed, just convince Linus to use BK :)

I don't care what Linus uses, but Linus decision should not lock other
developers into using the same tools, e.g. it should not become
inconvenient to send simple patches. The basic communication tools should
still be mail and patches. What we IMO need is a patch management system
not a source management system.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> I don't care what Linus uses, but Linus decision should not lock other
> developers into using the same tools, e.g. it should not become
> inconvenient to send simple patches. The basic communication tools should
> still be mail and patches. What we IMO need is a patch management system
> not a source management system.

Thats been promised long back. And Linus said many times both in mail and
in person that if he started using bitkeeper he wouldnt force others to do
so.

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Alan Cox wrote:

> Thats been promised long back. And Linus said many times both in mail and
> in person that if he started using bitkeeper he wouldnt force others to do
> so.

I know and that wasn't really my problem. It is the "Linus should just
use bk and all problems are solved" attitude, which makes me nervous.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 06:29:04PM +0100, Roman Zippel wrote:
> I know and that wasn't really my problem. It is the "Linus should just
> use bk and all problems are solved" attitude, which makes me nervous.

If that's referring to my message, I was just cheerleading :)

We are all fully aware of the multitude of issues that need to be
examined before BK or any other patch management system can be
adequately implemented.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 01:59:56PM +0100, Roman Zippel wrote:
> Hi,
> 
> On Wed, 30 Jan 2002, Jeff Garzik wrote:
> 
> > Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> I don't care what Linus uses, but Linus decision should not lock other
> developers into using the same tools, e.g. it should not become
> inconvenient to send simple patches. The basic communication tools should
> still be mail and patches. What we IMO need is a patch management system
> not a source management system.

BK can happily be used as a patch management system and it can, and has
for years, been able to accept and generate traditional patches.  Linus
could maintain the source in a BK tree and make it available as both
a BK tree and traditional patches.  It's a one line command to generate
a release patch and another one line command to generate the release
tarball.

By the way, you can send BK patches exactly the way that you send regular
patches, with the difference being that BK has an optional way of wrapping
them up in uuencode (or whatever) so that mailers don't stomp on them.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Jochen Friedrich]

Hi Larry,

> with the difference being that BK has an optional way of wrapping
> them up in uuencode (or whatever) so that mailers don't stomp on them.

isn't that just the same as sending them as attchment? And isn't that
discouraged?

Cheers,
--jochen

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 05:34:12PM +0100, Jochen Friedrich wrote:
> Hi Larry,
> 
> > with the difference being that BK has an optional way of wrapping
> > them up in uuencode (or whatever) so that mailers don't stomp on them.
> 
> isn't that just the same as sending them as attchment? And isn't that
> discouraged?

We have a generic wrapping/unwrapping mechanism.  The wrapping can be as
draconian as a uuencode/mimencoded attachment or as light as a crc envelope.
We don't care, we're mechanism providers in this area, not policy setters.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 05:34:12PM +0100, Jochen Friedrich wrote:
> > with the difference being that BK has an optional way of wrapping
> > them up in uuencode (or whatever) so that mailers don't stomp on them.
> 
> isn't that just the same as sending them as attchment? And isn't that
> discouraged?

gcc developers love that uuencoded stuff for large files it seems, but
it's not big in l-k land...  plaintext patches are preferred.  The main
requirement is that the patch is NOT obscured by base64 or uuencoding.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Larry McVoy wrote:

> > What we IMO need is a patch management system
> > not a source management system.
>
> BK can happily be used as a patch management system and it can, and has
> for years, been able to accept and generate traditional patches.  Linus
> could maintain the source in a BK tree and make it available as both
> a BK tree and traditional patches.

It's not really the same or that's not what I mean with patch management
system or can bk also manage the patches, which Linus drops?
What I have in mind is a patch management system which tracks the status
of unapplied patches. The status could be:
- patch applies cleanly to tree x.
- patch is approved/disappoved by y.
- patch is in tree z since version...
This system should not only support Linus, but also other tree
maintainers, so they can pick patches they want to integrate into their
trees, which could also feed back information which patch conflicts with
another patch (this could be done by the patchbot as well, but humans are
usually better at judging which patch is more important).
Linus again could use this information to decide which patches he
integrates into his in own tree, so he can easier sync up with other
trees.
My suggestion would be to setup an alias for Linus and/or Marcelo, which
just collects patches send to them. Anyone interested in implementing a
patchbot can use this data to test and demonstrate his/her implementation.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:06:17PM +0100, Roman Zippel wrote:
> > > What we IMO need is a patch management system
> > > not a source management system.
> >
> > BK can happily be used as a patch management system and it can, and has
> > for years, been able to accept and generate traditional patches.  Linus
> > could maintain the source in a BK tree and make it available as both
> > a BK tree and traditional patches.
> 
> It's not really the same or that's not what I mean with patch management
> system or can bk also manage the patches, which Linus drops?
> What I have in mind is a patch management system which tracks the status
> of unapplied patches. The status could be:
> - patch applies cleanly to tree x.
> - patch is approved/disappoved by y.
> - patch is in tree z since version...

Yeah, I think BK can do this, most of what you are describing is covered
by already existing BK commands and practices.  There are literally dozens
of different sites using BK to track the kernel and both internal and
external sources of patches.  I'm sure there are issues that need to be
resolved and we'll try to do so.  

I might be mistaken, I also get the feeling that your real issue might
be that you don't like/understand/something BK and you are pushing for a
different answer.  That's cool, there are now two patchbot projects you
can go join and start coding.  Some of our biggest fans are people who
have tried that and discovered exactly how complex the problem space
really is, so it's actually in my best interest to encourage you to
go help out with one of those projects.  If you solve the problem, the
kernel benefits and I get to learn something: that's good.  Or you don't
and you'll come to respect BK: that's good too, at least I like it.

So have some fun, this is actually a more challenging area than any
kernel work I've ever done or seen, including SMP threading, so the more
you get into it, the more fun you can have (assuming that banging your
brain against some hard problems meets your fun definition).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Larry McVoy wrote:

> I might be mistaken, I also get the feeling that your real issue might
> be that you don't like/understand/something BK and you are pushing for a
> different answer.  That's cool, there are now two patchbot projects you
> can go join and start coding.

Um, I really have no time for this, I have no problem with sending
patches to whoever/whatever.
I'm not arguing for/against bk at all, if you can demonstrate how bk
solves all of our problems and people want to use it, I had absolutely
no problem with that.
But you are correct my experiences with bk are limited and not very
encouraging. Now I only use it to extract specific versions from the ppc
tree and to import that into the apus tree. Maybe a "bk howto for cvs
users" might help, which shows typical cvs use cases and how to do them
with bk, the documentation I found only stresses the bad sides of cvs.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 10:02:05PM +0100, Roman Zippel wrote:
> Larry McVoy wrote:
> 
> > I might be mistaken, I also get the feeling that your real issue might
> > be that you don't like/understand/something BK and you are pushing for a
> > different answer.  That's cool, there are now two patchbot projects you
> > can go join and start coding.
> 
> Um, I really have no time for this
> [BK for cvs users request deleted]

Apparently not.  It might help if you read the docs, this has been there 
for months if not years:

http://www.bitkeeper.com/Documentation.HOWTO.CVS.html
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

Larry McVoy wrote:

> > [BK for cvs users request deleted]
> 
> Apparently not.  It might help if you read the docs, this has been there
> for months if not years:
> 
> http://www.bitkeeper.com/Documentation.HOWTO.CVS.html

I meant the equivalent of cvs uses cases not the equivalent of three cvs
commands, e.g. how would I handle cvs branches and join branches, how do
I check out a specific version/date, how do I track external sources,
which are not using bk, how do I get grep/cscope/... working without
doubling the needed disk space with bk -r co.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 11:13:29PM +0100, Roman Zippel wrote:
> I meant the equivalent of cvs uses cases not the equivalent of three cvs
> commands, e.g. how would I handle cvs branches and join branches, how do
> I check out a specific version/date, how do I track external sources,
> which are not using bk, how do I get grep/cscope/... working without
> doubling the needed disk space with bk -r co.

s/branch/repository/ - every repository is essentially a vendor branch that
works.

bk clone -r<rev> reproduces the tree as of that rev.

bk -r grep is very useful.

Tracking external sources is fairly obvious, and BK excels at it, and
virtually 100% of our users have figured out how to do it without asking
us questions.

All of your issues are actually in the docs and well documented.  And if
you had asked us how to do it, we would have pointed you at the docs or
fixed them if they were incomplete.  I'm at a loss as to why you want to
prove to the entire lkml that you can't/won't read the docs, but hey,
if that's your bag go for it.  I'm willing to answer your questions as
they come up, so keep 'em coming.  It just helps educate the general 
list, though I suspect they may get a bit tired of it because most people
do read the docs and are more interested in the harder problems.  So
maybe we should take this offline?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

Larry McVoy wrote:

> All of your issues are actually in the docs and well documented.  And if
> you had asked us how to do it, we would have pointed you at the docs or
> fixed them if they were incomplete.  I'm at a loss as to why you want to
> prove to the entire lkml that you can't/won't read the docs, but hey,
> if that's your bag go for it.  I'm willing to answer your questions as
> they come up, so keep 'em coming.

This is going nowhere, so that will be my last statement.
I knew most of the answers, I was just trying to give an idea of the
problems a cvs user has to deal with if he is confronted with bk. I
evaluated bk for myself and decided that it's not the right tool for
_me_, so I have no questions.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Chris Ricker]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

You snipped the part I was actually interested in.  Let me try again.

We're agreed that the files themselves are the best indicator of where to
route patches, and that MAINTAINERS isn't useful for much besides deciding
who should get IPO offers ;-).  What I'm wondering is where I, as someone
who is listed in some of the Documentation/* stuff as its maintainer, should
be sending patches.  You want a hierarchy, and I think that's perfectly
reasonable, but I have no idea who the layer of the hierarchy between me and
you is....

later,
chris

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Chris Ricker wrote:
>
> We're agreed that the files themselves are the best indicator of where to
> route patches, and that MAINTAINERS isn't useful for much besides deciding
> who should get IPO offers ;-).  What I'm wondering is where I, as someone
> who is listed in some of the Documentation/* stuff as its maintainer, should
> be sending patches.  You want a hierarchy, and I think that's perfectly
> reasonable, but I have no idea who the layer of the hierarchy between me and
> you is....

Ahh..

I had the same problem with Documentation/Configure.help, as you saw.

My solution in that case (when the issue came to a flame-fest) was to just
split up the documentation - which makes it a whole lot more maintainable
for everybody, and also makes it fairly explicit who maintains it for most
cases.

Basically, I'd really like documentation to go with the thing it
documents. This is something where the docbook stuff helped noticeably.

			Linus

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 06:54:04PM -0800, Linus Torvalds wrote:
> Basically, I'd really like documentation to go with the thing it
> documents. This is something where the docbook stuff helped noticeably.

Well...  kinda sorta.

We -used- to really have documentation with the thing it documents, like
drivers/foo/README.  In-lined source comments are one piece of the
puzzle yes, but the -bulk- of the docs are not anywhere near the thing
it documents.

I actually don't like stuffing documents in Documentation/DocBook
proper...  I've put docs for two drivers in there, but if the trend
continues a new dir structure will need to evolve.

Either we'll want Documentation/DocBook/<category>, or move the docbook
docs into the standard Documentation/* hierarchy......... or we'll start
moving docs back outside Documentation/*

	Jeff, on a semi-tangent

Re: A modest proposal -- We need a patch penguin

[Matthew D. Pitts]

Chris,

Thank you for saying this... I have things I would like do/add to the kernel
and I am not sure who to send them to.

Also, is there presently a maintainer for Supermount? If not, I would be
willing to pick it up for 2.5.x, as it is one of the things I want to work
on.

Matthew D. Pitts

----- Original Message -----
From: "Chris Ricker" <kaboom@gatech.edu>
To: "Linus Torvalds" <torvalds@transmeta.com>
Cc: "World Domination Now!" <linux-kernel@vger.kernel.org>
Sent: Tuesday, January 29, 2002 9:45 PM
Subject: Re: A modest proposal -- We need a patch penguin


> On Tue, 29 Jan 2002, Linus Torvalds wrote:
>
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
>
> You snipped the part I was actually interested in.  Let me try again.
>
> We're agreed that the files themselves are the best indicator of where to
> route patches, and that MAINTAINERS isn't useful for much besides deciding
> who should get IPO offers ;-).  What I'm wondering is where I, as someone
> who is listed in some of the Documentation/* stuff as its maintainer,
should
> be sending patches.  You want a hierarchy, and I think that's perfectly
> reasonable, but I have no idea who the layer of the hierarchy between me
and
> you is....
>
> later,
> chris
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 07:49:41AM -0500, Matthew D. Pitts wrote:
 > 
 > Also, is there presently a maintainer for Supermount? If not, I would be
 > willing to pick it up for 2.5.x, as it is one of the things I want to work
 > on.

 I believe Juan Quintela <quintela@mandrakesoft.com>  did some work on 
 it in 2.4 for Mandrake's kernel.
 
-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Juan Quintela]

>>>>> "matthew" == Matthew D Pitts <mpitts@suite224.net> writes:

matthew> Chris,
matthew> Thank you for saying this... I have things I would like do/add to the kernel
matthew> and I am not sure who to send them to.

matthew> Also, is there presently a maintainer for Supermount? If not, I would be
matthew> willing to pick it up for 2.5.x, as it is one of the things I want to work
matthew> on.


I am working on it.  Actual version worked quite well in 2.4.x series,
but last kernels are showing same problems, worknig on that.

Later, Juan.


-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 07:49 am, Matthew D. Pitts wrote:
> Chris,
>
> Thank you for saying this... I have things I would like do/add to the
> kernel and I am not sure who to send them to.

No, if you're not a maintainer then you still send them to the maintainer in 
the MAINTAINERS file.

The interesting question is, who does THAT maintainer send them to.  (We seem 
to be heading for a four-tiered system, with Linus at the top, a dozen or so 
lieutenants under him, and then the specific maintainers under them.  With 
individual developers submitting patches being the fourth tier.  Patches go 
from developer, to maintainer, to lieutenant, to linus.)

This doesn't sound like a bad thing for scalability reasons, and should also 
help address the "I sent my patch directly to linus a dozen times and I 
didn't hear back" problem.

The problem right now is a lot of the maintainers don't seem to know who 
their corresponding lieutenant is.  We're still waiting for clarification 
from Linus...


> Also, is there presently a maintainer for Supermount? If not, I would be
> willing to pick it up for 2.5.x, as it is one of the things I want to work
> on.

I didn't spot one in MAINTAINERS.  The email at the top of "mount.h" says:

> * Author:  Marco van Wieringen <mvw@planets.elm.net>

So that might be a good person to ask.  Of course who knows how old that 
email address is... :)

> Matthew D. Pitts

Rob

Re: A modest proposal -- We need a patch penguin

[Francois Romieu]

Rob Landley <landley@trommello.org> :
[...]
> The problem right now is a lot of the maintainers don't seem to know who 
> their corresponding lieutenant is.  We're still waiting for clarification 
> from Linus...

Feel free to send your patches here if you're lost.

-- 
Ueimor

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 05:03 pm, Francois Romieu wrote:
> Rob Landley <landley@trommello.org> :
> [...]
>
> > The problem right now is a lot of the maintainers don't seem to know who
> > their corresponding lieutenant is.  We're still waiting for clarification
> > from Linus...
>
> Feel free to send your patches here if you're lost.

I'm not a maintainer, just the friend of one who came close to burnout.

The "we" who are waiting for clarification is larger than the maintainer set, 
because if the maintainers can't function the developers who depend on the 
maintainers can't function either.

Rob

Re: A modest proposal -- We need a patch penguin

[Jesse Pollard]

---------  Received message begins Here  ---------
Rob Landley <landley@trommello.org>:
> 
> On Wednesday 30 January 2002 07:49 am, Matthew D. Pitts wrote:
> > Chris,
> >
> > Thank you for saying this... I have things I would like do/add to the
> > kernel and I am not sure who to send them to.
> 
> No, if you're not a maintainer then you still send them to the maintainer in 
> the MAINTAINERS file.
> 
> The interesting question is, who does THAT maintainer send them to.  (We seem 
> to be heading for a four-tiered system, with Linus at the top, a dozen or so 
> lieutenants under him, and then the specific maintainers under them.  With 
> individual developers submitting patches being the fourth tier.  Patches go 
> from developer, to maintainer, to lieutenant, to linus.)
> 
> This doesn't sound like a bad thing for scalability reasons, and should also 
> help address the "I sent my patch directly to linus a dozen times and I 
> didn't hear back" problem.
> 
> The problem right now is a lot of the maintainers don't seem to know who 
> their corresponding lieutenant is.  We're still waiting for clarification 
> from Linus...

Ummm. this might be silly, but shouldn't those announcements come from
the lieutenants?

Linus has announced who he accepts patches frin, and who will be doing the 2.0,
2.2, and 2.4 maintenance. It would seem logical to have those lieutenants
announce their maintainers.

How would Linus actually know who, (after his lieutenants) SHOULD send mail
to the lieutenants?

That is the problem in the first place...

It would help to have the information in the MAINTAINERS file though. As well
as the auxilary mailing lists supporting that activity. That way, users
who find a bug/create a patch/whatever would have an easier time locating
where to send the patch. Especially when it doesn't directly affect the
core kernel.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> Linus has announced who he accepts patches frin, and who will be doing the
> 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those 
> lieutenants announce their maintainers.

Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of 
2.5, does it make sense for Marcelo to announce the maintainer of usb for
2.4?

It's not as simple as you'd think.  Reason: it's not a tree, it's an
acyclic graph.  Hopefully.  ;-)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 09:39 pm, Daniel Phillips wrote:
> On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > Linus has announced who he accepts patches frin, and who will be doing
> > the 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those
> > lieutenants announce their maintainers.
>
> Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of
> 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> 2.4?
>
> It's not as simple as you'd think.  Reason: it's not a tree, it's an
> acyclic graph.  Hopefully.  ;-)

I'm still trying to figure out who all the lieutenants are.  (It seems Andre 
Hedrick reports to Jens Axboe, Rik van Riel might actually report to.. Andrea 
Arcangeli?  (Or Dave Jones.)  But who does Eric send his help patches to?  Is 
Kieth Owens at the top level or what?  It seems like both Kieth and Eric are 
also under Dave Jones.  I guess "patch penguin" is just "Miscelaneous 
Lieutenant".  Makes sense i the new context, I suppose...)

I expect it will all get worked out eventually.  Now that the secret of the 
difference between maintainers and lieutenants is out.  The thread seems to 
be dying down a bit... :)

Rob

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 04:29 am, Rob Landley wrote:
> On Wednesday 30 January 2002 09:39 pm, Daniel Phillips wrote:
> > On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > > Linus has announced who he accepts patches frin, and who will be doing
> > > the 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those
> > > lieutenants announce their maintainers.
> >
> > Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of
> > 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> > 2.4?
> >
> > It's not as simple as you'd think.  Reason: it's not a tree, it's an
> > acyclic graph.  Hopefully.  ;-)
> 
> I'm still trying to figure out who all the lieutenants are.

You will never figure that out, it isn't predefined.  It reshapes itself on the
fly, and is really defined by what is going on at any given time.  That said,
it's usually possible to figure out how the main maintainers are, and what to
send where, just don't hope to ever nail that down in a rigid structure.  It's
not rigid.

> (It seems Andre 
> Hedrick reports to Jens Axboe, Rik van Riel might actually report to.. Andrea 
> Arcangeli?  (Or Dave Jones.)  But who does Eric send his help patches to?  Is 
> Kieth Owens at the top level or what?  It seems like both Kieth and Eric are 
> also under Dave Jones.  I guess "patch penguin" is just "Miscelaneous 
> Lieutenant".  Makes sense i the new context, I suppose...)
> 
> I expect it will all get worked out eventually.  Now that the secret of the 
> difference between maintainers and lieutenants is out.

By the way, that never was a secret to anybody in active development.

> The thread seems to be dying down a bit... :)

Right, people are working on solutions.  As usual, though much dung did fly,
Linus comes out smelling like a rose.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:

> You will never figure that out, it isn't predefined.  It reshapes itself on
> the fly, and is really defined by what is going on at any given time.  That
> said, it's usually possible to figure out how the main maintainers are, and
> what to send where, just don't hope to ever nail that down in a rigid
> structure.  It's not rigid.

As long as the maintainers know who the lieutenants are, nobody under them 
should really have to care that much...

> > I expect it will all get worked out eventually.  Now that the secret of
> > the difference between maintainers and lieutenants is out.
>
> By the way, that never was a secret to anybody in active development.

I.E. the people who knew it knew it, and hence never noticed the problem...

There are, however, some people writing largeish bits of code that did not in 
fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
files and CML2, Kieth Owens' new build process...  Maybe it was even a factor 
in Alan Cox burning out (you'd have to ask him about that)...

> > The thread seems to be dying down a bit... :)
>
> Right, people are working on solutions.  As usual, though much dung did
> fly, Linus comes out smelling like a rose.

Gee, what a suprise. :)

Rob

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 00:32:40 -0500, 
Rob Landley <landley@trommello.org> wrote:
>On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
>> > I expect it will all get worked out eventually.  Now that the secret of
>> > the difference between maintainers and lieutenants is out.
>>
>> By the way, that never was a secret to anybody in active development.
>
>I.E. the people who knew it knew it, and hence never noticed the problem...
>
>There are, however, some people writing largeish bits of code that did not in 
>fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
>files and CML2, Kieth Owens' new build process... 

Both ESR and I definitely know about this process but kbuild is one of
the awkward systems that affects the entire kernel.  The final kbuild
system goes straight to Linus, there is nobody else to send it to.

kbuild 2.5 must match the current makefiles and config settings before
it can go in so it is impractible to target anything expect the
standard kernel, there is far too much work involved in tracking
makefile and config changes in -ac, -dj, -whoever.  Even if kbuild was
done against another tree, it would have to be redone and reverified
before sending to Linus, there is no way to extract kbuild 2.5 from a
divergent tree and expect it work on Linus's tree.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 06:32 am, Rob Landley wrote:
> On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
> > Rob Landley apparently wrote:
> > > I expect it will all get worked out eventually.  Now that the secret of
> > > the difference between maintainers and lieutenants is out.
> >
> > By the way, that never was a secret to anybody in active development.
> 
> I.E. the people who knew it knew it, and hence never noticed the problem...
> 
> There are, however, some people writing largeish bits of code that did not in 
> fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
> files and CML2, Kieth Owens' new build process...

They all know who the lieutenants are, I can assure you.

> Maybe it was even a factor in Alan Cox burning out (you'd have to ask him
> about that)...

Whatever gave you the idea that Alan is burnt out?

A observation: before proposing how we should fix the Linux development
process, perhaps you should have studied it enough to know how it works,
first.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Thu, Jan 31, 2002 at 12:32:40AM -0500, Rob Landley wrote:
> On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
> > > I expect it will all get worked out eventually.  Now that the secret of
> > > the difference between maintainers and lieutenants is out.

> > By the way, that never was a secret to anybody in active development.
> 
> I.E. the people who knew it knew it, and hence never noticed the problem...
> 
> There are, however, some people writing largeish bits of code that did not in 
> fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
> files and CML2, Kieth Owens' new build process...

ESR was told things repeatedly and they didn't sink in.
(And I note you have been told this repeatedly, too.)
Did you miss Alan's comment as well?  Apparently so...
http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/1567.html

Andre knows his shit ten ways to Sunday, but one must speak ATA
not English with him.  Definite communications problem, despite the
fact that he writes solid low level driver code and tests it pretty
thoroughly.  It was clear at the beginning of 2.5.x that Jens' bio
stuff was going in and Andre's stuff would conflict with it.  Didn't
make Andre happy, but it was a good decision.  And now Andre's basic
taskfile stuff has been merged, so life is good.  I'm looking forward
to all the doors that taskfile has opened to the Linux kernel.

I dunno how Keith became one of your examples.  Maybe I missed it,
but I have not seen an announcement and review proving that kbuild
was ready for merging into 2.5.x.  With all due respect to the kbuild
list, I have seen a couple times "...but this was discussed and decided
upon on the kbuild list" and it turns to be an issue that definitely
requires further discussion and thought.

There is no secret.  Only willful ignorance.

If people writing largish pieces of code in isolation and expect them to
be applied without being cognizent of other development and feedback, I
-expect- their work to be dropped.  That is an example of a WORKING not
broken system.

The Linux kernel way is really evolution not revolution.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 07:27 am, Jeff Garzik wrote:
> ESR was told things repeatedly and they didn't sink in.
> (And I note you have been told this repeatedly, too.)
> Did you miss Alan's comment as well?  Apparently so...
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/1567.html

With due regard to Alan, repeating the BS does not make it true.  Eric is in 
fact very wellknown in the kernel world, just not as a kernel hacker.  I'm 
perfectly content with his level of contribution, I think he's more than 
earned the right to spout.

And by the way, the wait for CML2+kbuild 2.5 is painful, I detest the old 
creaking cruft so much, I feel like I got something on me every time I change 
an option.  It hates me too, and retaliates by breaking itself in as many 
creative ways as it can.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 10:29:39PM -0500, Rob Landley wrote:
> I expect it will all get worked out eventually.  Now that the secret of the 
> difference between maintainers and lieutenants is out.  The thread seems to 
> be dying down a bit... :)

There Is No Cabal

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 22:41:12 -0500, 
Jeff Garzik <garzik@havoc.gtf.org> wrote:
>On Wed, Jan 30, 2002 at 10:29:39PM -0500, Rob Landley wrote:
>> I expect it will all get worked out eventually.  Now that the secret of the 
>> difference between maintainers and lieutenants is out.  The thread seems to 
>> be dying down a bit... :)
>
>There Is No Cabal

Actually the correct secret phrase is "There is no Eric conspiracy".
Hi Bruce.

http://tuxedo.org/~esr/ecsl/

Re: A modest proposal -- We need a patch penguin

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > Linus has announced who he accepts patches frin, and who will be doing the
> > 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those 
> > lieutenants announce their maintainers.
> 
> Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of 
> 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> 2.4?
> 
> It's not as simple as you'd think.  Reason: it's not a tree, it's an
> acyclic graph.  Hopefully.  ;-)

Actually, it does make sense - It still doesn't prevent someone from
announcing a higher level person for a subsystem, or even a person at
the same level.

Announcements shouldn't be rigid, it's up to who the lieutenants will
accept patches from. They can still accept them from outside the announced
lists, though that may increase the amount of effort.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Tue, Jan 29, 2002 at 04:44:12PM -0800, Linus Torvalds wrote:
> I have to admit that personally I've always found the MAINTAINERS file
> more of an irritation than anything else. The first place _I_ tend to look
> personally is actually in the source files themselves (although that may
> be a false statistic - the kind of people I tend to have to look up aren't
> the main maintainers at all, but more single driver people etc).
> 
> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

There's one problem with that though - if someone maintains many files,
and his email address changes, you end up with a large patch changing all
those email addresses in every file.

IMHO its far better to have someone's name at the top of each file, and
put the email addresses in the MAINTAINERS file.

People don't change their names often, but email addresses do change.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:19:12AM +0000, Russell King wrote:
> On Tue, Jan 29, 2002 at 04:44:12PM -0800, Linus Torvalds wrote:
> > I have to admit that personally I've always found the MAINTAINERS file
> > more of an irritation than anything else. The first place _I_ tend to look
> > personally is actually in the source files themselves (although that may
> > be a false statistic - the kind of people I tend to have to look up aren't
> > the main maintainers at all, but more single driver people etc).
> > 
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
> 
> There's one problem with that though - if someone maintains many files,
> and his email address changes, you end up with a large patch changing all
> those email addresses in every file.
> 
> IMHO its far better to have someone's name at the top of each file, and
> put the email addresses in the MAINTAINERS file.

Also FWIW I go to MAINTAINERS file first, when I construct the CC line
for patches sent to Linus.  Poking around the source is annoying and not
terribly scalable in my experience.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Jacob Luna Lundberg]

On Wed, 30 Jan 2002, Russell King wrote:
> There's one problem with that though - if someone maintains many files,
> and his email address changes, you end up with a large patch changing all
> those email addresses in every file.

Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
to people who make it into MAINTAINERS then?  Of course, someone would
have to maintain the accounts...  ;)

-Jacob

--

Reechani, Sentrosi, Vasi

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Wed, Jan 30, 2002 at 11:55:38AM -0800, Jacob Luna Lundberg wrote:
> 
> On Wed, 30 Jan 2002, Russell King wrote:
> > There's one problem with that though - if someone maintains many files,
> > and his email address changes, you end up with a large patch changing all
> > those email addresses in every file.
> 
> Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> to people who make it into MAINTAINERS then?  Of course, someone would
> have to maintain the accounts...  ;)

Already covered that ground.  Excellent way of having an endless supply
of spam fired at you.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Wed, 30 Jan 2002, Jacob Luna Lundberg wrote:

> 
> On Wed, 30 Jan 2002, Russell King wrote:
> > There's one problem with that though - if someone maintains many files,
> > and his email address changes, you end up with a large patch changing all
> > those email addresses in every file.
> 
> Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> to people who make it into MAINTAINERS then?  Of course, someone would
> have to maintain the accounts...  ;)

Just as a talking point, it should be possible to have a daemon scan mail
the lkml for [PATCH] and read the filenames from the patch itself, and do
a file to maintainer lookup followed by a mail. Obviously it would have to
have a human for some cases, but that's not all that bad, at least the
patch program could assign a number and post a list of patches to lkml on
a regular basis.

The hard part is the file to maintainer map, so the program can pick the
best maintainer, and possibly on a regular (daily) basis a single list of
patches to other maintainers: "this patch was sent to XXX bacause most of
the files are hers,but some are yours so you might want to check." And of
course XXX would be told that the patch changed other's files as well.

All patches would be given a number for discussion, after eyeball of the
first 20 patches I saw, I guess that 60-80% could unambiguously go to the
correct maintainer.

I realize this is less complex and wonderful than the schemes proposed,
therefore it might easily actually happen... and it takes no effort except
reading the mail, if the maintainer doesn't care to use the notification
s/he can ignore it, at least the submitter can be sure it was remailed and
to whom.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:56 pm, Bill Davidsen wrote:
> On Wed, 30 Jan 2002, Jacob Luna Lundberg wrote:
> > On Wed, 30 Jan 2002, Russell King wrote:
> > > There's one problem with that though - if someone maintains many files,
> > > and his email address changes, you end up with a large patch changing all
> > > those email addresses in every file.
> > 
> > Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> > to people who make it into MAINTAINERS then?  Of course, someone would
> > have to maintain the accounts...  ;)
> 
> Just as a talking point, it should be possible to have a daemon scan mail
> the lkml for [PATCH] and read the filenames from the patch itself, and do
> a file to maintainer lookup followed by a mail. Obviously it would have to
> have a human for some cases, but that's not all that bad, at least the
> patch program could assign a number and post a list of patches to lkml on
> a regular basis.
> 
> The hard part is the file to maintainer map, so the program can pick the
> best maintainer, and possibly on a regular (daily) basis a single list of
> patches to other maintainers: "this patch was sent to XXX bacause most of
> the files are hers,but some are yours so you might want to check." And of
> course XXX would be told that the patch changed other's files as well.
> 
> All patches would be given a number for discussion, after eyeball of the
> first 20 patches I saw, I guess that 60-80% could unambiguously go to the
> correct maintainer.
> 
> I realize this is less complex and wonderful than the schemes proposed,

Is that bad?

> therefore it might easily actually happen... and it takes no effort except
> reading the mail, if the maintainer doesn't care to use the notification
> s/he can ignore it, at least the submitter can be sure it was remailed and
> to whom.

One (and only one) step ahead of you.  Please have a look at what we're
doing here:

   http://killeri.net/cgi-bin/alias/ezmlm-cgi

And yes, we're already thinking about Russell's concerns with spam.  I
think that issue is under control.

-- 
Daniel

RE: A modest proposal -- We need a patch penguin

[Karl]

>Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org)
>to people who make it into MAINTAINERS then?  Of course, someone would
>hve to maintain the accounts...  ;)


  As someone who normally just watches I will meekly add, that makes a LOT
of sense. Because the address could be IDE_Subsystem@vger.kernel.org and
then if that person passed on the maintainership, the maintainer patch would
have the correct e mail address and only the name would be out of date. Thus
keeping ease of submission at least.

Karl Tatgenhorst

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 06:50 pm, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Rob Landley wrote:
>
> > Ah.  So being listed in the maintainers list doesn't mean someone is
> > actually a maintainer it makes sense to forward patches to?
>
> Sure it does.
>
> It just doesn't mean that they should send stuff to _me_.
>
> Did you not understand my point about scalability?

I was asking for clarification.

> I can work with a
> limited number of people, and those people can work with _their_ limited
> number of people etc etc.

I.E. a tree structure.

> The MAINTAINERS file is _not_ a list of people I work with on a daily
> basis. In fact, I don't necessarily even recognize the names of all those
> people.
>
> Let's take an example. Let's say that you had a patch for ppp. You'd send
> the patch to Paul Mackerras. He, in turn, would send his patches to David
> Miller (who knows a hell of a lot better what it's all about than I do).
> And he in turn sends them to me.
>
> They are both maintainers. That doesn't mean that I necessarily work with
> every maintainer directly.

Okay, so there's a tree of maintainers, and some maintainers seem unaware 
that they should be sending their patches to other maintainers rather than 
directly to you?

Does this seem like a valid assessment of at least part of the problem?

> Why? Because having hundreds of people emailing me _obviously_ doesn't
> scale. Never has, never will. It may work over short timeperiods wih lots
> of energy, but it obviously isn't a stable setup.

Well at least we agree on something. :)

> 			Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Henning P. Schmiedehausen]

Linus Torvalds <torvalds@transmeta.com> writes:

>> Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
>> Boldt...

>NONE of those are in the ten-twenty people group.

>How many people do you think fits in a small group? Hint. It sure isn't
>all 300 on the maintainers list.

Can you tell us, who the people in your peer group are? That would
make sending patches much easier.

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: A modest proposal -- We need a patch penguin

[Daniel Egger]

Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:

> Or look at USB: I get the USB patches from Greg, and he gets them from
> various different people. Johannes Erdfelt is the maintainer for uhci.c,
> and he sends them to Greg, not to me.

What about creating a small document that states who's the correct
recipient for a subsystem? This would prevent dotzends of questions
like "Where do I send my patches?" and turn them into a RTFF.
 
-- 
Servus,
       Daniel

Re: A modest proposal -- We need a patch penguin

[Andre Hedrick]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> 
> On Tue, 29 Jan 2002, Rob Landley wrote:
> > > >
> > > > Then why not give the subsystem maintainers patch permissions on your
> > > > tree. Sort of like committers.  The problem people have is that you're
> > > > dropping patches from those ten-twenty people you trust.
> > >
> > > No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> > > _not_ dropping their patches (although I'm sometimes critical of them,
> > > and will tell them that they do not get applied).
> >
> > Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
> > Boldt...
> 
> NONE of those are in the ten-twenty people group.

Well it is nice to know the facts now.  How about having just a little
more courage and publish your offical tree of who is "IN" and "OUT" so
folks can decide for themselves.  It is an honor to be in the lesser class
of folks who care more but less is accepted.

As for clean coding, you have to make a mess to in one part of a room by
pushing the contents to a corner and weeding out the useful.  Since you
have always stated public/private/in-person the sensitve nature of the
changes to the low-level storage drivers, those who have tried to promote
regression testing of layers and isolation points, have been ignored,
laughed, scorned, etc ...

Regardless if many people see the need and other are tired of being the
garbage collector of blame, when valid and proper solutions have been
presented in the past, specifically "FILE SYSTEM CORRUPTION".

Only after a few cases of pointing out flaws and failures in darwinisms
development, few became ignored.  Noting the total freedom you take to
blanket blame folks for issues which they are not responsible for
creating, an easy target allows one to ignore ultimate responsiblity.

> How many people do you think fits in a small group? Hint. It sure isn't
> all 300 on the maintainers list.

Not at all but again why not draft the offical "Linus IN/OUT Tree".

> > Ah.  So being listed in the maintainers list doesn't mean someone is actually
> > a maintainer it makes sense to forward patches to?
> 
> Sure it does.
> 
> It just doesn't mean that they should send stuff to _me_.

Some have gotten a strong grasp of the obvious nature of this point first
hand, regardless ...

Maybe you should consider taken an agreed code base migration change when
it is suggested and agreed upon, instead of ignoring comments and
suggestions for changes.  Just buy design when I get done w/ ATA and maybe
ATAPI so that it is clean and obvious to the reader, I would consider
tearing into the ABANDONED SCSI CORE.  However, I expect to find the same
uphill battle and may do it for the joy of exactness, but rediscover the
same problem of a global design change.

Something you need to understand, and I honestly expect you to ignore,
is a responsible an proper OS protects the hardware from accepting bad
command operations.  Given "LINUX" is a UNIX environment, that does not
give it the right to ignore comman sense.  However, to get along in your
world, all have accepted users are allowed and expect to have no
safeguards.  So when a problem comes up and it is ugly, it gets batted
down because the solution is wrong, and then quietly adopted without
acknowledgement the very solution proposed.

regards,

--andre

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 02:13 PM 30/01/02 +0100, Daniel Egger wrote:
>Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:
>
> > Or look at USB: I get the USB patches from Greg, and he gets them from
> > various different people. Johannes Erdfelt is the maintainer for uhci.c,
> > and he sends them to Greg, not to me.
>
>What about creating a small document that states who's the correct
>recipient for a subsystem? This would prevent dotzends of questions
>like "Where do I send my patches?" and turn them into a RTFF.

A reworking of MAINTAINERS could be beneficial and help achieve this.

Linus mentioned that he prefers to look at the code to see who to talk to. 
Others have mentioned this may be nice, but makes it hard to get some sort 
of overall view, plus since programmers can be inconsistent in stuff like 
this, it may not always happen. How about we turn the problem upside down, 
and figure out how to get the code easily referenced in MAINTAINERS?

What I'm thinking is that we could add (multiple?) lines into MAINTAINERS 
that specify the actual FILES in the kernel (in reference to linux/) that 
someone works on or maintains. We don't have to list every file (wildcards, 
regex's, etc, can work too), plus you can list the maintainers of various 
areas of the code (such as generic maintainers of all the files under a 
part of the kernel file tree) by just listing what directories they 
control. Something so that it's dead simple to extract who maintains this file.

Here's a possible example:

Say I'm looking at the SiS/Trident Audio Driver, and I have a patch I want 
to send to a maintainer. The file I'm working on is:

linux/drivers/sound/trident.*

If I could easily search MAINTAINERS for who maintains this file, I'm made. 
If I can't find that, I start trimming the search (to say 
linux/drivers/sound/, which would be the sound maintainer).

If we say add an F: field to maintainers (at the end of the maintainers 
record), you can easily do things like...

grep -B 10 "F: linux/drivers/sound/trident" /usr/src/linux/MAINTAINERS

...and get some sort of results (-B is "before context, which displays 
lines before the found line, and is quite useful in this sort of situation) 
that help. This is just a quick and dirty example, and I'm sure someone 
could easily write a small script that could parse the output better, do 
things like automatically cut the search back till it finds a match, etc.

This could also be used to figure out a tree of who does what, which is 
probably not a bad idea.

Just an idea of course. *grin*


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[David Lang]

Eric posted a proposal to split the maintainers info across all
directories/files sometime last year and got shouted down to the point of
not mentioning it anymore (which as we know takes quite a bit for him :-)

does someone want to dig up that proposal and recirculate it rather then
starting from scratch again?

David Lang


 On Thu, 31 Jan 2002, Stuart Young wrote:

> Date: Thu, 31 Jan 2002 12:16:29 +1100
> From: Stuart Young <sgy@amc.com.au>
> To: linux-kernel@vger.kernel.org
> Cc: Daniel Egger <degger@fhm.edu>, Linus Torvalds <torvalds@transmeta.com>
> Subject: Re: A modest proposal -- We need a patch penguin
>
> At 02:13 PM 30/01/02 +0100, Daniel Egger wrote:
> >Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:
> >
> > > Or look at USB: I get the USB patches from Greg, and he gets them from
> > > various different people. Johannes Erdfelt is the maintainer for uhci.c,
> > > and he sends them to Greg, not to me.
> >
> >What about creating a small document that states who's the correct
> >recipient for a subsystem? This would prevent dotzends of questions
> >like "Where do I send my patches?" and turn them into a RTFF.
>
> A reworking of MAINTAINERS could be beneficial and help achieve this.
>
> Linus mentioned that he prefers to look at the code to see who to talk to.
> Others have mentioned this may be nice, but makes it hard to get some sort
> of overall view, plus since programmers can be inconsistent in stuff like
> this, it may not always happen. How about we turn the problem upside down,
> and figure out how to get the code easily referenced in MAINTAINERS?
>
> What I'm thinking is that we could add (multiple?) lines into MAINTAINERS
> that specify the actual FILES in the kernel (in reference to linux/) that
> someone works on or maintains. We don't have to list every file (wildcards,
> regex's, etc, can work too), plus you can list the maintainers of various
> areas of the code (such as generic maintainers of all the files under a
> part of the kernel file tree) by just listing what directories they
> control. Something so that it's dead simple to extract who maintains this file.
>
> Here's a possible example:
>
> Say I'm looking at the SiS/Trident Audio Driver, and I have a patch I want
> to send to a maintainer. The file I'm working on is:
>
> linux/drivers/sound/trident.*
>
> If I could easily search MAINTAINERS for who maintains this file, I'm made.
> If I can't find that, I start trimming the search (to say
> linux/drivers/sound/, which would be the sound maintainer).
>
> If we say add an F: field to maintainers (at the end of the maintainers
> record), you can easily do things like...
>
> grep -B 10 "F: linux/drivers/sound/trident" /usr/src/linux/MAINTAINERS
>
> ...and get some sort of results (-B is "before context, which displays
> lines before the found line, and is quite useful in this sort of situation)
> that help. This is just a quick and dirty example, and I'm sure someone
> could easily write a small script that could parse the output better, do
> things like automatically cut the search back till it finds a match, etc.
>
> This could also be used to figure out a tree of who does what, which is
> probably not a bad idea.
>
> Just an idea of course. *grin*
>
>
> Stuart Young - sgy@amc.com.au
> (aka Cefiar) - cefiar1@optushome.com.au
>
> [All opinions expressed in the above message are my]
> [own and not necessarily the views of my employer..]
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> > Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> > "good taste" for a long time. But it's not always a long process - some
> > of you may remember Bill Hawes, for example, who came out of nowhere
> > rather quickly.
> 
> So listed "maintainers" may need to forward patches to these people, and get 
> them to sign off on them, in order to get their patches at least reviewed for 
> inclusion into your tree?

Count me out of that job. If you want something in 2.5 don't bug me. I
simply don't care

Alan

Re: A modest proposal -- We need a patch penguin

[Shawn]

On 01/29, Alan Cox said something like:
> > > Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> > > "good taste" for a long time. But it's not always a long process - some
> > > of you may remember Bill Hawes, for example, who came out of nowhere
> > > rather quickly.
> > 
> > So listed "maintainers" may need to forward patches to these people, and get 
> > them to sign off on them, in order to get their patches at least reviewed for 
> > inclusion into your tree?
> 
> Count me out of that job. If you want something in 2.5 don't bug me. I
> simply don't care

Now this scares the piss out of me...

--
Shawn Leas
core@enodev.com

I bought a self learning record to learn spanish, I turned it on and
went to sleep, the record got stuck, the next day I could only stutter in
spanish.
						-- Stephen Wright

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 09:30 am, Skip Ford wrote:
> Linus Torvalds wrote:
> [snip]
>
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
>
> Then why not give the subsystem maintainers patch permissions on your tree.
> Sort of like committers.  The problem people have is that you're dropping
> patches from those ten-twenty people you trust.

I understand why he doesn't do that: he can't function if the code is 
changing under him in ways that suprise him.  (Especially he can't function 
as architect without doing code inspection.)

Linus DOES apply larger patches from maintainers with less scrutiny, but 
there still IS scrutiny of each patch.  (At the very least, checking which 
files it touches.)

> Each subsystem maintainer should handle patches to that subsystem, and
> you should remove your own patch permissions for only those subsystems.
> You could get involved with only changes in direction that affect more
> than one subsystem.

Linus also reserves the right to mess with a maintainer's code and force a 
patch back down the tree for them to resync with.  He just did it with the 
help files (after a "private flamewar").  In this case, the maintainer was 
caught by suprise, claiming to be unaware that Linus expected him to make 
that change, which just seems to be one more example of a lack of 
communication between Linus and a maintainer.  This time, instead of Linus 
not getting the maintainer's message (patch), the maintainer doesn't get 
linus's message ("Go do this, in this order".)  So we've got examples of 
messages getting dropped in both directions, making the maintainer look 
inattentive when he claims otherwise, and making Linus look inattentive when 
HE claims otherwise...

Rob

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> In short, if you have areas or patches that you feel have had problems,
> ask yourself _why_ those areas have problems. 

Easy, because the patch process has bogged down due to bad design and has
failed to scale. And you simply refuse to believe that there is a problem.

> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and
> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me. 

The problem is that you don't trust ANYONE. There is no reason why you
should be looking at small obvious patches to bugs (note bugs, not
enhancements). In the last batch of messages I see agreement from Alan Cox
and Eric Raymond that things are backed up. I see reiser filesystem
patches, from the original developer, labeled "third try." Quite bluntly
he is a hell of a lot better qualified to do bug fixes in that area than
you are.
 
> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

I'm sure this will be ignored, but here's what should happen, and will,
just maybe not in the Linus kernel series.

BUGS:

There should be a place to send bugs. Every bug should be acknowleged so
mailbots to keep resending will not be needed. People are setting this up
now, the question is not if it will be done this way, but if it will be
done through or around you.

Each bug should be eyeballed by someone with a clue just to see that the
report states a problem, a way to verify the problem, and doesn't grossly
misstate the intended behaviour. Then someone at the maintainer level
would look at the patch, check it carefully, and reject formally or apply.

Here's the heresy: every bug patch should be promoted to the current
kernel unless it is rejected for reason. This will get fixes in, but more
importantly will force people to look at the damn things...

Reasons to drop a patch:
1 - no bug, this process is for bugs, no news features or improvements.
Offhand I see bugs when you have a crash (oops), a failure to compile, or
filesystem corruption. Someone may want to add drivers totally not
working, but I didn't say it.

2 - no bug, you misunderstand the behaviour, failed to provide a test case
or persuasive argument (you are scanning from 0 to N instead of 0 to N-1
type stuff, you are locking one lock and unlocking another, etc).

3 - no fix, the solution doesn't work.

4 - not readable or understandable.

5 - changes some global part of the kernel and would or could impact other
things.

Note that while "there's a better way" can cause an add to the to-do list,
I do not intend to have the users suffer an actual bug if there is a
viable solution.

To all the people who say that "if you don't like the way Linus does
things write your own kernel," I say that's what is happening with the
-aa, -ac, etc lines. That's why I'm typing this on something called
2.4.17-jl15-ll, because I'm in need of a kernel which runs on a small
machine and doesn't piss me off with lousy response to complex stuff like
echoing what I type.

Someone mentioned the army. Note that hte general doesn't decide where to
dig the foxhole. It's about delegation, and while I doubt Linus read this
far, that's how things scale beyond what one person can do.

Fork is not just a system call, I'd hate to see FreeLinux, NetLinux,
OpenLinux, Linux386, along BSD lines, but while people will wait for
enhancements, I don't find that they are nearly so willing to wait for
fixes. Before we have vendor versions which actually start to differ in
syscall behaviour, let's get a handle on this. It's time for the
maintenence to evolve just as the kernel has, and become something bigger
and better. We need "patch-threads" soon, lest we thrash ourself silly.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Bill Davidsen wrote:

>
>The problem is that you don't trust ANYONE. There is no reason why you
>should be looking at small obvious patches to bugs (note bugs, not
>enhancements). In the last batch of messages I see agreement from Alan Cox
>and Eric Raymond that things are backed up. I see reiser filesystem
>patches, from the original developer, labeled "third try." Quite bluntly
>he is a hell of a lot better qualified to do bug fixes in that area than
>you are.
>
Whoa, in this case it is Marcelo, and his responsiveness is on the whole 
simply superb.  Right complaint, wrong example.  Marcelo also does a 
great job of having good reasons for rejecting patches on the rare 
occasions when he does reject them.  Is the real problem that you are 
comparing guys who have neither a day job nor a family to someone who 
has both?  I think maybe it is.  Linus, do you have other work 
responsibilities besides Linux?  With funding provided , could you work 
solely on Linux?

I remember when I used to hold a day job while working on ReiserFS and 
trying to manage a team of full-time programmers by email.  It sucked. 
 There is crap in ReiserFS that went in only because I didn't have time 
to fight it, and Reiser4 is a very much higher level of quality mainly 
because I can invest a lot more time into it.

Hans

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Tue, 29 Jan 2002 03:23:11 +0000 (UTC)
torvalds@transmeta.com (Linus Torvalds) wrote:

> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse. 

I could hardly disagree.

> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

Ok, I won't come up with anything. :)

I just heard a bell ringing, with Rob's message. It's still a faint ring, but
the debate spurred on l-k has shown that the bell is indeed ringing
somewhere. Many opinions were quite harsh on you, some even ungrateful; I'm
sorry if this caused any trouble.

What I see (from the viewpoint of some random user giving a try to test
kernels just for the fun of doing it) is not a problem with subsys
maintainers. I don't even _see_ them, from my pov. I trust them because you
do, and that's enough for me.

But I start to feel the need for someone to throw me an "unified development
tree". Like -ac was in 2.4: some source you trust from which pulling a kernel
to be tested. I need it, or better I prefer it over a forest of cross-patched
subtrees where I can find kernels bleedingly optimized in some area and
lacking trivial fixes in others.

A matter of convenience, actually. To which you can just answer "go away,
lazy scum". And probably will! :)

But also convenience for you.

You says, righteously, that you want tested patches. OTOH, to have them
tested, peer review would want to have them available somewhere to be tested,
for us crunching drones. Eventually in a tree where I can easily find all the
patches which are subject to scrutiny.

So we both could take advantage of what you call a "layer": someone (better:
more than someone - one man doesn't scale) who gathers stuff and readies ONE
unified, test-time-ready kernel, who has all the reviewing eyeballs, who goes
at you with the results of the tests, re-chunkifying(tm) the patches, and
letting you discard what you don't like and keep the (tested) things you see
fitting.

In the upper spheres, there could be a split of workload between who packages
the kernel to be thrown to the test-dogs (AC in the past, DJ now, ore than
one man in the future?), and who actually pioneers code and steers the kernel
after the packager gathers enough feedback from testers (that's you, wow! :)
).

Of course there has to be trust at that level, and I agree with you that
without trust with few, very selected people, you can't go ahead blindly
gathering debris from everyone.

In short: I think there are too many concurrent, overlapping development
trees, with a web of crosspatches that are honestly difficult to follow from
my "download, make, lilo, reboot, report" viewpoint. A fragmentation in the
to-be-tested code. A single "reference development" tree would be most
welcome.

I didn't intend to ask it to *you*. Probably natural evolution of kernel
development will pop out a good solution anyway.

The "patch penguin" as solution is just an idea. Bad as you want, but it
comes from a sensation Rob had. And I feel it too. And maybe others, less
silly than me.

The idea is rejectable, of course. The sensation, a bit less.

Have fun,

-- Francesco Munda

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:03:55AM +0100, Francesco Munda wrote:
> In short: I think there are too many concurrent, overlapping development
> trees, with a web of crosspatches that are honestly difficult to follow from
> my "download, make, lilo, reboot, report" viewpoint. A fragmentation in the
> to-be-tested code. A single "reference development" tree would be most
> welcome.

2.5.x is the reference development tree.

People building outside-the-kernel patchkits is indeed useful for
end users to conveniently test a bunch of patches... but attempting
to merge various concurrent trees would be murder on code quality.
Do we want XFS ACLs in the reference development tree, just to yank
or modify syscalls before the final revision?  No.  Therefore, XFS
needs to be in its own tree until its ready.

Notice the gcc team will create a branch for development, even
during a development cycle (ie. no freeze at all), just to ensure
that large or complex changes do not destabilize the tree until they
are really ready.

Finally, even in a devel cycle kernel hackers need some semblance of a
sane tree in order to do their own development.  If tons of hackers
are blazing away committing all sorts of code, you have nothing but
a tree of mass confusion, in a nice package for users to test.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Wed, 30 Jan 2002 03:39:06 -0500
Jeff Garzik <garzik@havoc.gtf.org> wrote:

> 2.5.x is the reference development tree.
> 
> [...]

Ok, I buy it.

The amount of messages, and thoughts on the subject, fairly satisfied me. I
wanted to see a reaction, and I indeed saw one. And I like what I've seen:
the point Rob brought up was not dismissed lightheartedly, but instead
spurred a discussion on many aspects of kernel development. Adoption of BK,
maintainers hierarchies, trust relationships between demigods, patch handling
clarifications, you name it...

Who can disagree that talking about these topics, finding solutions (to
existing and future problems), and in general doing reality checks is a good
thing?

And the "small patches maintainer" is a figure that should be there. World
will keep spinning if nobody takes the buck, but having one such maintainer
has been generally greeted as a good idea.

We can have many trees to play with to develop different changes, but that
trivial one-line fixup should be in every tree, not just in "my favorite
maintainer's" tree. Risking a wipeout while testing a new scheduler/VM/dev
handler is one thing. Risking a wipeout for a bug that's patched in
everyone's else tree but not in my own, is another. :)

Trivial fixes and largish changes should be handled in different ways, this
thread seems to have summed up. Maybe by different people, to split some
workload more evenly.

Many interesting points were touched. Let's wait and see.

-- Francesco

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 08:37 pm, Francesco Munda wrote:
> On Mon, 28 Jan 2002 09:10:56 -0500
>
> Rob Landley <landley@trommello.org> wrote:
> > Patch Penguin Proposal.
> >
> > [...]
>
> You mean some sort of proxy/two-tier development? A "commit/rollback"
> transaction model on the kernel itself?

Think how Alan Cox's tree used to work.  Just because Alan accepted a patch 
didn't guarantee Linus wasn't going to come up with a reason to shoot it 
down.  It just meant the patch wasn't going to be ignored, and if it WAS 
dropped there would probably going to be some kind of explanation.

Whether the patch penguin wants to use some kind of tool to maintain their 
tree (like CVS) with a "commit/rollback" model is a seperate issue.  Linus 
isn't going to use it, and linus isn't going to have to see it.  Linus gets 
the kid of patches he likes, which have already had merge clashes and the 
really obvious thinkos resolved before he sees them, and have probably even 
been tested by the foolhardy individuals currently downloading the -ac, -dj, 
and -aa trees.

Right now, Alan's tree is in the process of going back into circulation.  He 
tells me that his tree is basically a delta against marcello (2.4), and DJ is 
doing a delta against linus (2.5).  Over time, the need for a 2.4 delta will 
probably diminish as new development shifts over to 2.5.  Right now, the 
patch constipation we've been seeing is, in my opinion, directing development 
to occur against 2.4 that should at the very least be eyeing 2.5.  (Alan is 
probably NOT interested in integrating patches that Marcelo has no intention 
of eventually integrating into 2.5.  So he's not taking the new development 
integration pressure off, that's DJ's job.)

I think DJ could definitely use a clearer mandate.

> I deeply agree with you, especially in keeping "many eyes" to look at the
> same kernel tree, and not chosing one of the many subtrees; as added bonus,
> this stuff is buzzword compliant! What we can ask more? :)
>
> Now, Linus' call to accept _your_ patch. Fingers crossed already.

I'm getting a lot more support off the list than on the list.  People seem to 
be afraid to cc: linux-kernel.  I underestimated how deeply steeped in 
politics this issue seems to have become.  It seems a fairly straightforward 
optmiziation, mainly a clarification of of the way things have been done in 
the past and a formalization of a position that got a bit confused in the 
transition from one officeholder to another.

Before posting here, I bounced an earlier draft off of both Alan Cox and Dave 
Jones.  Alan's response was, and I quote:

> I'm certainly fine with DaveJ being the victim 8)

Dave didn't seem to have any major objections but raised a lot technical 
points to the effect of "I'm already doing this bit".  Both of them gave me 
permission to post most of our conversation to the list, but seem unwilling 
to do it themselves. :)

I've gotten several other agreements, some from people trying to find an 
off-list place we could discuss it (okay, so what's THIS list for again)?  
And one person, who shall remain nameless (at least as long as he refuses to 
speak for himself. :) brought up the subject of Linus co-designing bitkeeper 
way back when to cope with exactly some of these problems.

Bitkeeper is a technical tool attempting to deal with a social problem.  
Merging patches, resolving conflicts between them, testing them, and keeping 
them current as the tree changes under them requires programmer work.  A 
human needs to do it.  Whether that human uses bitkeeper, CVS, a directory 
full of patch files, or manually keeps all the patches in printouts in a shoe 
box is a side issue.  A human can feed Linus better patches than any software 
tool possibly could.

Now if the patch penguin wants to use bitkeeper for his own internal 
patch-wrangling, that's a seperate issue.  One you should take up with the 
patch penguin, once we have one.  (Of course the developer community and the 
maintainers might exert some pressure on the patch penguin to use CVS, but 
how is this a bad thing from Linus's perspective: it means they'e NOT bugging 
HIM about using CVS anymore.  And again, this is an enhancement/detail that 
can be resolved later.)

As for attracting Linus's attention, there's a penguin and egg problem here: 
without an integration lieutenant Linus is largely too swamped to reliably be 
aware of this kind of thread on the list, so how can he get the suggestion to 
anoint someone with holy penguin pee to basically act as his secretary and 
clean up this mess of patches so he can properly sort through them once 
they've been organized and laid out in front of him in nice neat rows.  Hence 
the drive to get people to agree to it so the thread grows large enough to 
attract Linus's attention, and also passes his "it's been discussed enough to 
find any particularly obvious holes with it" filter...

So everybody who thinks this is a good idea, please say so.  Those who don't 
like it, please say so too so the objection can be aired and maybe resolved.  
The core idea here really is to save Linus time and effort.  Everything else 
is either a direct consequence of that, or a fringe benefit.

> -- FM

Rob

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Mon, Jan 28, 2002 at 10:42:19PM -0500, Rob Landley wrote:
 
 > probably diminish as new development shifts over to 2.5.  Right now, the 
 > patch constipation we've been seeing is, in my opinion, directing development 
 > to occur against 2.4 that should at the very least be eyeing 2.5.  (Alan is 
 > probably NOT interested in integrating patches that Marcelo has no intention 
 > of eventually integrating into 2.5.  So he's not taking the new development 
 > integration pressure off, that's DJ's job.)
 > 
 > I think DJ could definitely use a clearer mandate.

 * Initially, -dj was "pick up fixes from 2.4".

 * Then when Linus broke various other parts of 2.5, I took fixes
   for various bits. (Some of those went back his way, others didn't,
   others are still in the process)
   (I'm a believer in the 'eat your own dogfood' thing, and run my
   tree on several testboxes -- being able to compile/boot/test
   this tree became more important at the cost of the tree growing
   a little further away from -linus)

 * Some developers also wanting to develop against 2.5 found the
   quickest way to get a compilable, workable 2.5 tree was to
   grab my snapshot, and work against my tree until Linus gets his
   together.  And hence, the input layer & fb layer changes.
   This was one I had to think about a bit before deciding if
   I was going to start accepting such patches.
   In theory, as we're now in 2.5, there should be no need for this,
   but whilst Linus is busy focusing on the block layer, scheduler
   or other flavour of the week, James, Vojtech etc can at least
   get some extra testers before their code hits -linus.
   By the time that the new input/fb stuff is ready for Linus' tree
   hopefully a lot of the more obvious problems will be shaken out,
   and Linus can have a set of patches for a "new xxx layer" that
   works for at least everyone who's been testing it in -dj.

 Where to go from here? More of the same. It's a fulltime job
 keeping up with Marcelo & Linus, and reviewing, merging, and
 chasing down the right people.  One thing I'm not entirely 
 enthusiastic about doing, is making policy decisions.
 I've had questions from people asking me if I'll merge xxx's
 implementation of ACLs for example. Without knowing which way
 Linus is going to turn on such an issue, I'm naturally hesitant.

 Another thing of note is that the merge process with Linus
 isn't as straightforward as running splitdiff, and pushing the
 chunks to Linus. Some bits require a timing (although this is
 sometimes hard to get right) so I can push him filesystem
 changes when Al isn't turning the VFS upside down for eg.
 Other bits I won't push because maintainers have mailed me
 asking me not to.  And other bits, because the maintainers
 can do a better job of splitting,pushing and describing than
 I can (typical example: the fbdev/input stuff)
 
 > Dave didn't seem to have any major objections but raised a lot technical 
 > points to the effect of "I'm already doing this bit".  Both of them gave me 
 > permission to post most of our conversation to the list, but seem unwilling 
 > to do it themselves. :)

 Time, Headcold, time, blah, excuses 8)
 But to reiterate, yes. Most of what you described is exactly whats
 taking place, although a lot of it happens behind the scenes, not
 on Linux-kernel, not on irc, but me being a pita chasing maintainers
 "Hey xxx sent me a patch, aren't you working on this? You two should
  talk..". It's like being a switchboard operator at times, plugging
 in the right cables, connecting the right people.
 
 > As for attracting Linus's attention, there's a penguin and egg problem here: 
 > without an integration lieutenant Linus is largely too swamped to reliably be 
 > aware of this kind of thread on the list

 Linus' concern that people don't scale is perhaps not unfounded.
 Since I started doing this, the number of hours involved has increased
 on a day by day basis. If there comes a time where >I'm< not scaling
 and start dropping patches, then maybe an extra tier is needed. *shrug*
 For now at least, things seem to be working out quite well on the whole.
 I'm not aware of any particularly important fix/cleanup that has been 
 dropped on the floor since I started scooping them up.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Padraig Brady]

Francesco Munda wrote:

> On Mon, 28 Jan 2002 09:10:56 -0500
> Rob Landley <landley@trommello.org> wrote:
> 
> 
>>Patch Penguin Proposal.
>>
>>[...]
>>
> 
> You mean some sort of proxy/two-tier development? A "commit/rollback"
> transaction model on the kernel itself?


Dave Jones described the current model very succinctly in:

http://marc.theaimsgroup.com/?l=linux-kernel&m=100966905916285&w=2

He also mentioned a big problem. People not honouring/realising
there position in the tree, (trying to get in the ChangeLog?).
True, the only way to scale it is add another level at the current
bottleneck, but this must be more than 1 person or it won't help,
as it'll just move the bottelneck back a little.

Personally I think automated tools (like bitkeeper) would help
more than another level in the hierarchy.

Currently the way I see it [should be] currently is:

random hackers
| | | | | | |
| maintainers
| | | |
combiners
| |
Linus

I.E. Linus just gets input from the combiners which
test logic from the maintainers in combination. Also
random hackers should input to the combiners and not Linus
if there isn't an appropriate maintainer for their code.

Padraig.

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Tue, 29 Jan 2002 12:23:26 +0000
Padraig Brady <padraig@antefacto.com> wrote:

> Currently the way I see it [should be] currently is:
> 
> [cut-n-pasted graph]
> 
> I.E. Linus just gets input from the combiners which
> test logic from the maintainers in combination. Also
> random hackers should input to the combiners and not Linus
> if there isn't an appropriate maintainer for their code.

Quite descriptive and useful, thanks.

Let me raise a point. And extend your graph:

random hackers
| | | | | | |
| maintainers -< subsys testers
| | | |
combiners -< tree testers
| |
Linus

Who you call combiners... How many of them should release independent trees
to be thrown at us test-dogs? My point of view is neither the hacker, nor the
maintainer nor the combiner one. Nor Linus, thank god! :) It's the guy who
risks his filesystem integrity with some 2.X.Y-preZ-testW-QQ-KK kernel.

How many crosspatched sources I should look at, to try my luck with?

Have fun,

-- Francesco

Re: A modest proposal -- We need a patch penguin

[Andrew Pimlott]

Rob, you make a nice case, but consider a few points.

One,

> This integration and patch tracking work is a fairly boring, thankless task, 
> but it's work somebody other than Linus can do, which Linus has to do 
> otherwise. (And which Linus is NOT doing a good job at right now.)

... are you _sure_ that Linus does this?  My sense is that he mostly
eschews integration grunt-work.  If that is so, it's possible that
Linus is already operating near top efficiency, and that his
throughput is as high as he wants it to be!  Linus has pointed out
more than once that a big part of his job is to limit change.  Maybe
he's happy with the current rate of change in 2.5.  (That doesn't
mean everything is optimal--he might wish for higher quality changes
or a different mix of changes, just not more.)

Two, Linus has argued that maintainers are his patch penguins;
whereas you favor a single integration point between the maintainers
and Linus.  This has advantages and disadvantages, but on the whole,
I think it is better if Linus works directly with subsystem
maintainers.  To the extent that Linux is modular, there is little
need for the extra layer (so it is just overhead).  And when there
is a real conflict between subsystems--that's probably just the time
when Linus and the maintainers need to be collaborating directly!
The only "but" is that many people find it hard to work with Linus.
However, Linus made clear in his message that he considers this a
solvable problem (and maybe one you should work on!).

> Finished code 
> regularly goes unintegrated for months at a time, being repeatedly resynced 
> and re-diffed against new trees until the code's maintainer gets sick of it. 

Assuming that your system doesn't dramatically increase Linus's
throughput, code will still have to be re-diffed.  I don't agree
that thrusting all the merging onto one person is the right
solution.  That person is a _much_ bigger scalability bottleneck
than Linus, because (by your definition of the role) he can't drop
patches!  So he will inevitably become overwhelmed, and then we have
a bigger mess.

Frankly, if I were a maintainer, I would want the patch that finally
gets integrated to be one that I produce, not one re-diffed by
someone less familiar with the subsystem.  So, I side with Linus's
"tough, that's part of the maintainer's job" stance.  (Now, tools to
help resync, and to eliminate the tedium of re-submitting to Linus
periodically, would be welcome.)

> Several of the bug fixes in Alan's tree (which he 
> stopped maintaining months ago) still are not present in 2.4.17 or 2.5.

This is plain evidence that a single integration point (and there is
no better than Alan) isn't a panacea.

Three, regarding your complaint about "clean-up" patches being
dropped: maybe this just means there is a maintainer missing from
the pantheon: the clean-up maintainer.

Andrew

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 06:51 am, Andrew Pimlott wrote:
> Three, regarding your complaint about "clean-up" patches being
> dropped: maybe this just means there is a maintainer missing from
> the pantheon: the clean-up maintainer.

Oh, you mean acme :-)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> throughput is as high as he wants it to be!  Linus has pointed out
> more than once that a big part of his job is to limit change.  Maybe
> he's happy with the current rate of change in 2.5.  (That doesn't
> mean everything is optimal--he might wish for higher quality changes
> or a different mix of changes, just not more.)

Progress happens at its own rate. Linus can no more control rate of change
than you can put a waterfall into low gear. There is a difference between
refusing stuff where the quality is low and losing stuff which is clear
fixes

> Two, Linus has argued that maintainers are his patch penguins;
> whereas you favor a single integration point between the maintainers
> and Linus.  This has advantages and disadvantages, but on the whole,
> I think it is better if Linus works directly with subsystem

Perl I think very much shows otherwise. Right now we have a maze of partially 
integrated trees which overlap, clash when the people send stuff to Linus and
worse.

When you have one or two integrators you have a single tree pretty much everyone
builds new stuff from and which people maintain small diffs relative to. At
the end of the day that ends up like the older -ac tree, and with the same
conditions - notably that anything in it might be going to see /dev/null not
Linus if its shown to be flawed or not done well.

Alan

Re: A modest proposal -- We need a patch penguin

[Andrew Pimlott]

On Tue, Jan 29, 2002 at 01:06:09PM +0000, Alan Cox wrote:
> Andrew wrote:
> > Two, Linus has argued that maintainers are his patch penguins;
> > whereas you favor a single integration point between the maintainers
> > and Linus.  This has advantages and disadvantages, but on the whole,
> > I think it is better if Linus works directly with subsystem
> 
> Perl I think very much shows otherwise.

I'm really not sure about this example.  I assume you mean Perl 5.
Last I checked, Perl didn't really operate the way Rob suggests.
There is a "patch pumpking", but he is more analogous to Linus than
to Alan.  In particular, Larry Wall does not review the pumpking's
work at all (he instead sets general direction and makes key design
decisions).  If Perl doesn't have the problems observed in Linux, I
think it is because 1) Perl is smaller, 2) Perl 5 is largely in
bug-fix mode, 3) Perl has a culture of accepting patches with less
scrutiny (without meaning this as a slam, I think you can see
evidence of this in the Perl 5 source base).

> When you have one or two integrators you have a single tree pretty
> much everyone builds new stuff from and which people maintain
> small diffs relative to. At the end of the day that ends up like
> the older -ac tree, and with the same conditions - notably that
> anything in it might be going to see /dev/null not Linus if its
> shown to be flawed or not done well.

There is an upper bound to the size of the delta one person can
maintain (well, assuming his goal is to sync those changes with
Linus).  Unless Linus's throughput increases dramatically, the
integrator's delta will grow until it reaches that bound.  At that
point, the integrator has to drop patches (or give up!).  How do you
get around this?

Andrew

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> Linus).  Unless Linus's throughput increases dramatically, the
> integrator's delta will grow until it reaches that bound.  At that
> point, the integrator has to drop patches (or give up!).  How do you
> get around this?

Historically at that point the old maintainer is overthrown 8) Look at
Minix versus Linux, gcc versus egcs etc

Re: A modest proposal -- We need a patch penguin

[John Alvord]

On Tue, 29 Jan 2002 13:06:09 +0000 (GMT), Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

>> throughput is as high as he wants it to be!  Linus has pointed out
>> more than once that a big part of his job is to limit change.  Maybe
>> he's happy with the current rate of change in 2.5.  (That doesn't
>> mean everything is optimal--he might wish for higher quality changes
>> or a different mix of changes, just not more.)
>
>Progress happens at its own rate. Linus can no more control rate of change
>than you can put a waterfall into low gear. There is a difference between
>refusing stuff where the quality is low and losing stuff which is clear
>fixes
>
>> Two, Linus has argued that maintainers are his patch penguins;
>> whereas you favor a single integration point between the maintainers
>> and Linus.  This has advantages and disadvantages, but on the whole,
>> I think it is better if Linus works directly with subsystem
>
>Perl I think very much shows otherwise. Right now we have a maze of partially 
>integrated trees which overlap, clash when the people send stuff to Linus and
>worse.
>
>When you have one or two integrators you have a single tree pretty much everyone
>builds new stuff from and which people maintain small diffs relative to. At
>the end of the day that ends up like the older -ac tree, and with the same
>conditions - notably that anything in it might be going to see /dev/null not
>Linus if its shown to be flawed or not done well.
>
Multiple integrator-trees dilute the tester pool, which is a major
limitation on progress.

john alvord

Re: A modest proposal -- We need a patch penguin

[Matthias Andree]

On Mon, 28 Jan 2002, Rob Landley wrote:

> The holder of the patch penguin would feed Linus good patches, by Linus's 
> standards. Not just tested ones, but small bite-sized patches, one per email 
> as plain text includes, with an explanation of what each patch does at the 
> top of the mail. (Just the way Linus likes them. :) Current pending patches 
> from the patch penguin tree could even be kept at a public place (like 
> kernel.org) so Linus could pull rather than push, and grab them when he has 
> time. The patch penguin tree would make sure that when Linus is ready for a 
> patch, the patch is ready for Linus.

Looks like a manual re-implementation of a bug/request/patch tracker
like sourceforge's, bugzilla or whatever, with some additions. A patch
is added to the system, it gets a version tag, and you just pull it, and
mark it closed if applied to Linus' tree. If Linus releases a new tree,
the patch is marked stale until the maintainer uploads an updated patch
or just reopens it to mark "still applies unchanged to new version". (No
CVS involved, BTW.)

> One reason Linus doesn't like CVS is he won't let other people check code 
> into his tree. (This is not a capricious decision on Linus's part: no 
> architect can function if he doesn't know what's in the system. Code review 
> of EVERYTHING is a vital part of Linus's job.) With a patch penguin tree, 

That's one of the arguments Peter Miller uses to endorse his "Aegis"
system: separate the integration part. FreeBSD also seem to follow a
similar idea: there are maintainers and committers. Maintainers usually
do not commit, but file PRs tagged as "maintainer-update".

(Note: I've never used aegis myself, it seems as though it had some
implications with development spread to various locations, someone
comment on that.)

Also, I'm not sure how good Bitkeeper fits here, or whether subversion
will help in this way (one might consider feeding suggestions to the
subversion team, http://subversion.tigris.org/, if they do atomic
commits, one might consider holding them off until blessed by an
integrator).

-- 
Matthias Andree

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."         Benjamin Franklin

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 10:55 am, Matthias Andree wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:
> 
> > The holder of the patch penguin would feed Linus good patches, by Linus's 
> > standards. Not just tested ones, but small bite-sized patches, one per email 
> > as plain text includes, with an explanation of what each patch does at the 
> > top of the mail. (Just the way Linus likes them. :) Current pending patches 
> > from the patch penguin tree could even be kept at a public place (like 
> > kernel.org) so Linus could pull rather than push, and grab them when he has 
> > time. The patch penguin tree would make sure that when Linus is ready for a 
> > patch, the patch is ready for Linus.
> 
> Looks like a manual re-implementation of a bug/request/patch tracker
> like sourceforge's, bugzilla or whatever, with some additions.

And you load a patch into it by emailing to the bot, not via the web
interface.  The web interface is just for a) reporting b) maintainance, i.e., 
closing out a patch that got applied in some altered form, or applied with no 
notification to the bot, or obsoleted.

> A patch
> is added to the system, it gets a version tag, and you just pull it, and
> mark it closed if applied to Linus' tree. If Linus releases a new tree,
> the patch is marked stale until the maintainer uploads an updated patch
> or just reopens it to mark "still applies unchanged to new version". (No
> CVS involved, BTW.)

Yes, very much yes.  This way it just looks like regular email to Linus - 
except for some hopefully useful bookkeeping gack prepended to the top of the 
mail by the bot - and doesn't change the way he works at all.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jim McDonald]

On Tue, 2002-01-29 at 09:55, Matthias Andree wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:

[...]

> Also, I'm not sure how good Bitkeeper fits here, or whether subversion
> will help in this way (one might consider feeding suggestions to the
> subversion team, http://subversion.tigris.org/, if they do atomic
> commits, one might consider holding them off until blessed by an
> integrator).

I'm not sure that a CVS-type solution is going to fix the problem here. 
>From what I can see, the problems that people are bringing up are as
follows:

   - some patches sent to the list get dropped without comment
   - people are worried about Linus' scalability in handling patches
   - patches time-out quite quickly with the speed of development of
     the kernel, which results in patches not getting applied because
     by the time they get looked they have gone stale
   - people seem to often be unsure about where to send patches
     for unmaintained code, and with no direct maintainer it seems
     that these patches automatically fall outside of Linus'
     trusted kernel people and stand a very small chance of getting
     implemented

I must admit that I agree with Linus' position in most places, but the
result of that is two-fold: a lot of people are left in the dark as to
the state of their patches (ditched, pending, bad style, etc), and a lot
of patch work is duplicated as different people fix the same problem
every couple of months because the fixes are never applied.

We have two solutions - fix the way that all of this works, or try to
patch around the resultant problems.  It looks like number 1 is not
going to happen, so why not do something with 2?  There has been a lot
of talk about putting together a system that re-sends patches every
month or so to lkml, let's write something that does this.  We could get
a number of advantages from this:

   - identification of patches (cleanup, performance improvement, bug 
     fix, new functionality, ...)
   - automatic identification of responsible maintainer (direct from
     patch) to email on submission of patch
   - ability to automatically re-diff patch against latest kernel
     versions and get submitter to re-apply if required
   - simple rejection of patches with minimal effort from maintainers
   - easy way for wannabe-patchers to see what patches are already 
     pending so they can concentrate on other areas and not duplicate
     effort

Note that this isn't that far from SourceForge, but probably too far to
make it worthwhile trying to set it up as an SF project.  If we do this
I figure that at worst it allows for auto-resending of patches that have
slipped through the cracks, and at best it will give people a far more
suitable mechanism for patch submission and tracking than just email.

Comments?  I'm willing to write it if someone is willing to host it.

> Matthias Andree

Cheers,
Jim.
-- 
Jim McDonald - Jim@mcdee.net

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Rob Landley wrote:
> Patch Penguin Proposal.

Ok, I mailed Rob privately in support of this; now I'm saying it
publicly: I support this. (And I thought it well written.)

I've submitted small patches for things as I have gotten the chance.  I
submitted those to Alan because they were small fixes.  He gave feedback
(Thanks Alan!) and I tried to fix things better.  He took care of
feeding Linus--something I'm not in a possition to do.
I believe we need a patch-penguin or something similar.  Linus wants
subsystem maintainers... maybe make an official bugfix-maintainer? 
Whoever it is needs to be officially recognized by Linus and probably
featured on /. or something so people who create those 3-4 line patches
that fix a bug that bit them will know not to mail Linus.

(Returning to the shadows where I belong... ;) )

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 04:51 pm, Eli Carter wrote:
> I believe we need a patch-penguin or something similar.  Linus wants
> subsystem maintainers... maybe make an official bugfix-maintainer? 
> Whoever it is needs to be officially recognized by Linus and probably
> featured on /. or something so people who create those 3-4 line patches
> that fix a bug that bit them will know not to mail Linus.

That would be acme, wouldn't it?

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jordan Mendelson]

On Monday, January 28, 2002, at 06:10  AM, Rob Landley wrote:

> Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are
> getting dropped on the floor on a regular basis. This is burning out
> maintainers and is increasing the number of different kernel trees (not 
> yet a
> major fork, but a lot of cracks and fragmentation are showing under the
> stress). Linus needs an integration lieutenant, and he needs one NOW.

While I'm not going to debate the fact that maintainence of the kernel 
is suboptimal, I would like to point out that the system that has been 
put in place is the correct way of doing it if you have someone with the 
vision necessary to oversee development. There are many ways to go about 
developing software, but many of them were created because of the lack 
of a strong leader and fortunately the Linux community has one (many in 
fact).

> We need to create the office of "patch penguin", whose job would be to 
> make
> Linus's life easier by doing what Alan Cox used to do, and what Dave 
> Jones is
> unofficially doing right now. (In fact, I'd like to nominate Dave Jones 
> for
> the office, although it's Linus's decision and it would be nice if Dave 
> got
> Alan Cox's blessing as well.)

Having one person who looks after patches is just going to create 
another bottleneck. The real problem is not Linus not accepting patches, 
but the process by which the patches are sent to him and the general 
impatience of the developer community.

We all know the kernel is broken up into a set of subsystems and those 
subsystems have maintainers. What is lacking however appears to be the 
leadership by these maintainers to:

* Delegate ownership of components in their subsystems to others.

* Prioritize small easily code reviewable patches for submission to 
Linus with adequate information about the contents of the changes 
including who has reviewed it, who wrote it, why it should be included 
and it's testing status is.

* Provide feedback to patch creators about the status of their patch 
including reasons why it was denied, suggestions for improvement, etc.

Of course, this isn't universally true. There are many instances where, 
especially in the case of device drivers, delegation has taken place, 
but a formal process of accepting patches that is actually adhered to by 
independent developers still appears to be lacking.

> Linus doesn't scale, and his current way of coping is to silently drop 
> the
> vast majority of patches submitted to him onto the floor. Most of the 
> time
> there is no judgement involved when this code gets dropped. Patches 
> that fix
> compile errors get dropped. Code from subsystem maintainers that Linus
> himself designated gets dropped. A build of the tree now spits out 
> numerous
> easily fixable warnings, when at one time it was warning-free. Finished 
> code
> regularly goes unintegrated for months at a time, being repeatedly 
> resynced
> and re-diffed against new trees until the code's maintainer gets sick 
> of it.
> This is extremely frustrating to developers, users, and vendors, and is
> burning out the maintainers. It is a huge source of unnecessary work. 
> The
> situation needs to be resolved. Fast.

I ague that the vast majority of patches submitted to Linus should be 
dropped on the floor. There is no reason why a compilation bug fix for 
an IDE chipset driver, PCI subsystem layer or even top-level Makefile 
should be sent directly to Linus by an independent developer.

The amount of time necessary to walk through code submitted by those who 
you don't know, don't trust and simply don't have an understanding of 
proper kernel development makes submitting patches directly to Linus 
like calling up the President to contest the suspension of your child 
from school. You may have a valid fix for a bug, but the system simply 
doesn't work if everyone decides to bypass local authority and go right 
to the top.

> The fact that 2.5 has "pre" releases seems suggestive of a change in 
> mindset.
> A patch now has to be widely used, tested, and recommended even to get 
> into
> 2.5, yet how does the patch get such testing before it's integrated 
> into a
> buildable kernel? Chicken and egg problem there, you want more testing 
> but
> without integration each testers can test fewer patches at once.

There is no chicken and egg problem. A patch gets wide testing by 
submitting it for peer review exactly like what goes on day in and day 
out on linux-kernel and various other subsystem mailing lists. The patch 
gets integrated into various forks of the tree for one reason or another 
and obtains even wider testing and sooner or later, it makes it into the 
kernel.

The patch tracking problem does exist. Maintainers loose track of 
changes. People integrate changes into their forks and the patch creator 
assumes that because someone is using their patch, their work is done. 
Worst of all, the people maintaining these forked trees forget where the 
patches they applied came from and bug fixes and public comments get 
missed.

There has been a lot of discussion about augmenting the kernel 
development process with automated tools of one sort of another, but 
tools are only as good as the people using them.

* Linus's email is obviously overloaded so the first thing that should 
be done is to get him a mailing list where only people he trusts can 
post patches they have reviewed. As a matter of policy, every other 
patch should simply be ignored by Linus.

* A hierarchical structure should be drawn out that shows who owns what 
and the flow of patches. This exists somewhat in the MAINTAINERS file, 
but I have a feeling that in some cases, the chain of command or is 
mostly unknown as not all maintainers should have direct contact with 
Linus. A real benefit would be to place the maintainers email address or 
associated mailing list in the top of every file they own (which exists 
mostly, but I don't believe it is part of a formal process.)

* A system of documentation for patches should be put in place. Each 
patch should have associated information about why it exists, who has 
approved it, what priority it is, change lists and of course, who wrote 
it. When a comment is made, it needs to be sent back to the author of 
the patch otherwise the author is just going to become frustrated and 
attempt to escalate the issue himself.

* A system by which maintainers can prioritize the patches they submit 
to Linus. Some patches are simply more important than others and 
developers are just going to have to deal with the fact that submitting 
a patch doesn't mean getting it integrated the next day.

Now this may sound like a lot of bureaucracy, but if the number of 
people Linus trusts can be expanded and the hierarchy kept shallow, the 
faster patches can be accepted or denied.

A final word. One thing that I have noticed over the years is because of 
the frantic pace of development, people tend to get a bit impatient when 
submitting patches. This mentality has also infected the average user 
who seem to upgrade to the bleeding edge for no apparent reason. What 
people need to remember is the 'stable' label given to the 2.4 tree 
refers to the type and frequency of changes made to the code base, not 
the stability of a running kernel. As developers, we need to be more 
patient when submitting patches. The world doesn't end because a patch 
doesn't make it into the bleeding edge kernels. If distributions need 
the patch, they'll integrate it, but that fact alone doesn't justify the 
patch making it into the kernel.

Of course that's just my opinion, I could be wrong.

Jordan

Re: A modest proposal -- We need a patch penguin

[Ragnar Hojland Espinosa]

On Tue, Jan 29, 2002 at 11:46:55AM -0800, Jordan Mendelson wrote:
> Having one person who looks after patches is just going to create 
> another bottleneck. The real problem is not Linus not accepting patches, 
> but the process by which the patches are sent to him and the general 
> impatience of the developer community.

Which in turn is quite natural when the developer community sees valuable
obvious patches not applied for no reason at all.  Linus, would you accept
some modifications to your MUA so that keys besides deleting the patch(es)
mail canned messages to the submitters?
-- 
____/|  Ragnar Højland      Freedom - Linux - OpenGL |    Brainbench MVP
\ o.O|  PGP94C4B2F0D27DE025BE2302C104B78C56 B72F0822 | for Unix Programming
 =(_)=  "Thou shalt not follow the NULL pointer for  | (www.brainbench.com)
   U     chaos and madness await thee at its end."