A modest proposal -- We need a patch penguin

A modest proposal -- We need a patch penguin

[Rob Landley]

Patch Penguin Proposal.

 1) Executive summary.
 2) The problem.
 3) The solution.
 4) Ramifications.

 -- Executive summary.

Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are 
getting dropped on the floor on a regular basis. This is burning out 
maintainers and is increasing the number of different kernel trees (not yet a 
major fork, but a lot of cracks and fragmentation are showing under the 
stress). Linus needs an integration lieutenant, and he needs one NOW.

We need to create the office of "patch penguin", whose job would be to make 
Linus's life easier by doing what Alan Cox used to do, and what Dave Jones is 
unofficially doing right now. (In fact, I'd like to nominate Dave Jones for 
the office, although it's Linus's decision and it would be nice if Dave got 
Alan Cox's blessing as well.)

And if we understand this position, and formalize it, we can make better use 
of it. It can solve a lot of problems in linux development.

 -- The problem.

Linus doesn't scale, and his current way of coping is to silently drop the 
vast majority of patches submitted to him onto the floor. Most of the time 
there is no judgement involved when this code gets dropped. Patches that fix 
compile errors get dropped. Code from subsystem maintainers that Linus 
himself designated gets dropped. A build of the tree now spits out numerous 
easily fixable warnings, when at one time it was warning-free. Finished code 
regularly goes unintegrated for months at a time, being repeatedly resynced 
and re-diffed against new trees until the code's maintainer gets sick of it. 
This is extremely frustrating to developers, users, and vendors, and is 
burning out the maintainers. It is a huge source of unnecessary work. The 
situation needs to be resolved. Fast.

If you think I'm preaching to the choir, skip to the next bit called "the 
solution". If not, read on...

The Linux tree came very close to forking during 2.4. We went eleven months 
without a development tree. The founding of the Functionally Overloaded Linux 
Kernel project was a symptom of an enormous unintegrated patch backlog 
building up pressure until at least a small fork was necessary. Even with 2.5 
out, the current high number of seperate development trees accepting 
third-party is still alarmingly high. Linus and Marcelo have been joined by 
Dave Jones, Alan Cox, Andrea Arcangeli, Michael Cohen, and others, with 
distributors maintaining their own significantly forked kernel trees as a 
matter of course. Developers like Andrea, Robert Love and Rik van Riel either 
distribute others' relatively unrelated patches with their patch sets, or 
base their patches on other, as yet unintegrated patches for extended periods 
of time.

Discussion of this problem was covered by kernel traffic and Linux Weekly 
News:

http://kt.zork.net/kernel-traffic/kt20020114_150.html#5
http://lwn.net/2002/0103/kernel.php3 (search for "patch management").

During 2.4, the version skew between Alan Cox's tree and Linus's tree got as 
bad as it's ever been. Several of the bug fixes in Alan's tree (which he 
stopped maintaining months ago) still are not present in 2.4.17 or 2.5. Rik 
van Riel has publicly complained that Linus dropped his VM patches on the 
floor for several months, a contributing factor to the 2.4 VM's failure to 
stabilize for almost a -YEAR- after its release. (This is a bad sign. Whether 
Rik's or Andrea's VM is superior is a side issue. Alan Cox, and through him 
Red Hat, got Rik's VM to work acceptably by integrating patches from that 
VM's maintainer. The fact Linus didn't do as well is a symptom of this larger 
problem. The kind of subsystem swapping so major it requires a new maintainer 
should not be necessary during a stable series.)

Speaking of Andrea Arcangeli, he's now integrating third-party patches into 
his own released development trees, because 2.5 isn't suitable to do 
development against and 2.4 doesn't have existing work (like low latency) 
he's trying to extend.

Andre Hedrick just had to throw a temper tantrum to get any attention paid to 
his IDE patches, and he's the official IDE maintainer. Eric Raymond tells me 
his help file updates have now been ignored for 33 consecutive releases 
(again, he's the maintainer), and this combined with recent major changes 
Linus unilaterally made to 2.5's help files (still without syncing with the 
maintainer's version before doing so) has created a lot of extra and totally 
unnecessary work for Eric, and he tells me it's made the version skew between 
2.4 and 2.5 almost unmanageable.

Andrew Morton's lock splitting low latency work has no forseeable schedule 
for inclusion, despite the fact it's needed whether or not Robert Love's 
preemption patch goes in. Ingo Molnar's O(1) scheduler did go in, but that 
was largely a case of good timing: it came right on the heels of a public 
argument about why Linus had not accepted patches to the scheduler for 
several years. The inclusion of code like JFS, XFS, and Daniel Phillips' ext2 
indexing patch are left up to distributions to integrate and ship long before 
they make it into Linus's tree. (Remember the software raid code in 2.2?) 
These are just the patches that have persisted, how much other good work 
doesn't last because its author loses interest after six months of the silent 
treatment? The mere existence of the "Functionally Overloaded Linux Kernel" 
(FOLK) project, to collect together as many unintegrated patches as possible, 
was a warning sign that things were not going smoothly on the patch 
integration front.

The release of 2.5 has helped a bit, but by no means solved the problem. Dave 
Jones started his tree because 2.4 fixes Marcello had accepted were not 
finding their way into 2.5. Even code like Keith Owens' new build system and 
CML2, both of which Linus approved for inclusion at the Kernel summit almost 
a year ago and even tentatively scheduled for before 2.5.2, are still not 
integrated with no clear idea if they ever will be. (Yes Linus can change his 
mind about including them, but total silence isn't the best way to indicate 
this. Why leave Keith and Eric hanging, and wasting months or even years of 
their time still working on code Linus may not want?)

The fact that 2.5 has "pre" releases seems suggestive of a change in mindset. 
A patch now has to be widely used, tested, and recommended even to get into 
2.5, yet how does the patch get such testing before it's integrated into a 
buildable kernel? Chicken and egg problem there, you want more testing but 
without integration each testers can test fewer patches at once.

There has even been public talk of CRON jobs to resubmit patches to Linus 
periodically for as long as it takes him to get around to dealing with them. 
Linus has actually endorsed this approach (provided that re-testing of the 
patches against the newest release before they are remailed is also 
automated). This effort has spawned a mailing list. That's just nuts. The 
fact that Linus doesn't scale isn't going to be solved by increasing the 
amount of mail he gets. When desperate measures are being seriously 
considered, we must be in desperate times.

-- The solution.

The community needs to offload some work from Linus, so he can focus on what 
he does that nobody else can. This offloading and delegation has been done 
before, with the introduction of subsystem maintainers. We just need to 
extend the maintainer concept to include an official and recognized 
integration maintainer.

During 2.1, when Linus burned out, responsibility for various subsystems were 
delegated to lieutenants to make Linus' job more manageable. Lieutenants are 
maintainers of various parts of the tree who collect and clean up patches, 
and make the first wave of obvious decisions ("this patch doesn't apply", 
"this bit doesn't compile", "my machine panicked", "look, it's not SMP safe") 
before sending tested code off to Linus. Linus still spends a lot of his time 
reading and auditing code, but by increasing the average quality of the code 
Linus looks at, the maintainers make his job easier. The more work 
maintainers can do the less Linus has to.

So what tasks does Linus still personally do? He's an architect. He steers 
the project, vetoing patches he doesn't like and suggesting changes in 
direction to the developers. And he's the final integrator, pulling together 
dispirate patches into one big system.

The job of architect is something only Linus can do. The job of integrator is 
something many people can do. Alan Cox did it for years. Dave Jones is doing 
it now, and Michael Cohen has yet another tree. Every Linux distributor has 
its own tree. Integrating patches so they don't conflict and porting them to 
new versions is hard work, but not brain surgery.

Linus is acting as a central collection point for patches, and patches are 
getting lost on their way to that collection point. Patches as big as UML and 
EXT3 were going into Alan Cox's tree during 2.4, and now new patches are 
going into Dave Jones's tree to be tested out and queued for Linus.

Integration is a task that can be delegated, and it has been. In Perl's 
model, Larry Wall is the benevolent dictator and architect, but integration 
work is done by the current holder of the "patch pumpkin". In Linux, Alan Cox 
used to be the de facto holder of the patch penguin, and now Dave Jones is 
maintaining a tree which can accept and integrate patches, and then feed them 
on to Linus when Linus is ready.

This system should be formalized, we need the patch penguin to become 
official. The patch penguin seems to have passed from Alan Cox to Dave Jones. 
If we recognize this, we can make much better use of it.

 --- Ramifications.

The holder of the patch penguin's job would be to accept patches from people 
other than linus, make them work together in a publicly compilable and 
testable tree, and feed good patches on to Linus. This may sound simple and 
obvious, but it's currenlty not happening and its noticeable by its absence.

The purpose of the patch penguin tree is to make life easier, both for Linus 
and the developer community. With a designated patch collector and 
integrator, Linus's job becomes easier. Linus would still maintain and 
release his own kernel trees (the way he did when Alan Cox regularly fed him 
patches), and Linus could still veto any patch he doesn't like (whether it 
came from the patch penguin, directly from a subsystem maintainer, or 
elsewhere). But Linus wouldn't be asked to act as the kernel's public CVS 
tree anymore. He could focus on being the architect.

The bulk of the patch penguin's work would be to accept, integrate, and 
update patches from designated subsystem maintainers, maintaining his own 
tree (seperate from linus's) containing the integrated collection of pending 
patches awaiting inclusion in Linus's tree. Patches submitted directly to the 
patch penguin could be redirected to subsystem maintainers where appropriate, 
or bounced with a message to that effect (at the patch penguin's option). 
This integration and patch tracking work is a fairly boring, thankless task, 
but it's work somebody other than Linus can do, which Linus has to do 
otherwise. (And which Linus is NOT doing a good job at right now.)

The rest of the patch-penguin holder's job is to feed Linus patches. The 
patch penguin holder's tree would fundamentally be a delta against the most 
recent release from Linus (like the "-ac patches" used to be). If Linus takes 
several releases to get around to looking at a new patch, the patch penguin 
would resynchronize their tree with each new Linus release, doing the fixups 
on the pending patch set (or bullying the source of each patch into doing 
so). It would be the patch penguin's job to keep up with Linus, not the other 
way around. When a change happens in Linus's tree that didn't come from the 
patch penguin, the patch penguin integrates the change into their tree 
automatically.

The holder of the patch penguin would feed Linus good patches, by Linus's 
standards. Not just tested ones, but small bite-sized patches, one per email 
as plain text includes, with an explanation of what each patch does at the 
top of the mail. (Just the way Linus likes them. :) Current pending patches 
from the patch penguin tree could even be kept at a public place (like 
kernel.org) so Linus could pull rather than push, and grab them when he has 
time. The patch penguin tree would make sure that when Linus is ready for a 
patch, the patch is ready for Linus.

The patch penguin tree can act as a buffer between Linus and the flood of 
patches from the field. When Linus is not ready for a patch yet, he can hold 
off on taking it into his tree, and doesn't have to worry about the patch 
being lost or out of date by the time he's ready to accept it. When Linus is 
focusing on something like the new block I/O code, the backlog of other 
patches naturally feeds into the patch penguin tree until Linus is ready to 
look at them. People won't have to complain about dropped patches, and Linus 
doesn't have to worry that patches haven't been tested enough before being 
submitted to him. Users who want to live on the really bleeding edge have a 
place to go for a kernel that's likely to break. Testers can find bugs en 
masse without having to do integration work (which is in and of itself a 
source of potential bugs).

Linus would still have veto power. He gets to reject any patch he doesn't 
like, and can ask for the integration lieutenant to back that patch out of 
the patch penguin tree. That's one big difference between the patch penguin 
tree and Linus's tree: the patch penguin tree is provisional. Stuff that goes 
in it can still get backed out in a version or two. Of course Linus would 
have to explicitly reject a patch to get it out of the patch penguin tree, 
meaning its developer stops fruitlessly re-submitting it to Linus, and maybe 
even gets a quick comment from Linus as to WHY it was unacceptable so they 
can fix it. (From the developer's point of view, this is a good thing. They 
either get feedback of closure.)

Linus sometimes needs time off. Not just for vacations, but to focus on 
specific subsections, like integrating Jens Axobe's BIO patches at the start 
of 2.5. Currently, these periods hopelessly stall or tangling development. 
But in Linus's absence, the patch penguin could continue to maintain a delta 
against the last Linus tree, and generate a sequence of small individual 
patches like a trail of bread crumbs for Linus to follow when he gets back. 
Linus could take a month off, and catch back up in a fraction of that time 
when he returned. (And if Linus were to get hit by a bus, the same 
infrastructure would allow the community to select and support a new 
architect, which might help companies like IBM sleep better at night.) And if 
Linus rejected patches halfway through the bread crumb trail requiring a lot 
of shuffling in later patches, well, that's more work for the patch penguin, 
not more work for Linus.

One reason Linus doesn't like CVS is he won't let other people check code 
into his tree. (This is not a capricious decision on Linus's part: no 
architect can function if he doesn't know what's in the system. Code review 
of EVERYTHING is a vital part of Linus's job.) With a patch penguin tree, 
there's no more pressure on Linus to use CVS. The patch penguin can use CVS 
if he wants to, and if he wants to give the subsystem maintainers commit 
access to the patch penguin tree, that's his business. The patch penguin's 
own internal housekeeping toolset shouldn't affect the patches he feeds on to 
Linus one way or the other.

Again, Linus likes stuff tested by a wide audience before it's sent to him. 
With a growing list of multiple trees maintained by Dave Jones, Alan Cox, 
Michael Cohen, Andrea Arcangeli, development and testing become fragmented. 
With a single patch penguin tree, the patches drain into a common pool and 
the backlog of unintegrated patches can't build up dangerous amounts of 
pressure to interfere with development. A single shared "pending" tree means 
the largest possible group of potential testers.

Re: A modest proposal -- We need a patch penguin

[Matthew D. Pitts]

Rob and company...

I agree 100% with this idea. I have several projects in the planning stages
that i want to integrate into 2.5, but I do NOT want them dropped with no
explanation. If i had a computer that I could devote to it, I would be
willing to volenteer to be patch penguin.

Matthew D. Pitts

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Mon, 28 Jan 2002 09:10:56 -0500
Rob Landley <landley@trommello.org> wrote:

> Patch Penguin Proposal.
> 
> [...]

You mean some sort of proxy/two-tier development? A "commit/rollback"
transaction model on the kernel itself?

I deeply agree with you, especially in keeping "many eyes" to look at the
same kernel tree, and not chosing one of the many subtrees; as added bonus,
this stuff is buzzword compliant! What we can ask more? :)

Now, Linus' call to accept _your_ patch. Fingers crossed already.

-- FM

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <200201290137.g0T1bwB24120@karis.localdomain>,
Francesco Munda  <syylk@libero.it> wrote:
>
>I deeply agree with you, especially in keeping "many eyes" to look at the
>same kernel tree, and not chosing one of the many subtrees; as added bonus,
>this stuff is buzzword compliant! What we can ask more? :)

Some thinking, for one thing.

One "patch penguin" scales no better than I do. In fact, I will claim
that most of them scale a whole lot worse. 

The fact is, we've had "patch penguins" pretty much forever, and they
are called subsystem maintainers.  They maintain their own subsystem, ie
people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
(USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
Molnar (scheduler), Jeff Garzik (network drivers) etc etc. 

If there are problems with certain patches, it tends to be due to one or
more of:

 - the subsystem is badly modularized (quite common, originally. I don't
   think many people realize how _far_ Linux has come in the last five
   years wrt issues like architectures, driver independence, filesystem
   infrastructure etc). And it still happens.

 - lack of maintainer interest. Many "maintainers" are less interested
   in true merging than in trying to just push whatever code they have,
   and only ever grow their patches instead of keeping them in pieces.

   This is usually a matter of getting used to it, and the best people
   get used to it really quickly (Andrea, for example, used to not do
   this well at all, but look at how he does it now! From a merge
   standpoint, his patches have gone from "horrible" to "very good")

 - personality/communication issues. Yes, they happen. I've tried to
   have other people be "filters" for the people I cannot work with, but
   I have to say that most of the time when I can't work with somebody,
   others have real problems with those people too. 

   (An example of a very successful situation: David Miller and Alexey
   Kuznetsov: Alexey used to have these rather uncontrolled patches that
   I couldn't judge or work with but that obviously had a lot of
   potential, and David acting as a filter made them a very successful
   team.)

In short, if you have areas or patches that you feel have had problems,
ask yourself _why_ those areas have problems. 

A word of warning: good maintainers are hard to find.  Getting more of
them helps, but at some point it can actually be more useful to help the
_existing_ ones.  I've got about ten-twenty people I really trust, and
quite frankly, the way people work is hardcoded in our DNA.  Nobody
"really trusts" hundreds of people.  The way to make these things scale
out more is to increase the network of trust not by trying to push it on
me, but by making it more of a _network_, not a star-topology around me. 

In short: don't try to come up with a "patch penguin".  Instead try to
help existing maintainers, or maybe help grow new ones. THAT is the way
to scalability.

		Linus

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 10:23 pm, Linus Torvalds wrote:
> In article <200201290137.g0T1bwB24120@karis.localdomain>,
>
> Francesco Munda  <syylk@libero.it> wrote:
> >I deeply agree with you, especially in keeping "many eyes" to look at the
> >same kernel tree, and not chosing one of the many subtrees; as added
> > bonus, this stuff is buzzword compliant! What we can ask more? :)
>
> Some thinking, for one thing.
>
> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse.

Sure.  But Alan doesn't, and Dave Jones (with enough experience) shouldn't.

You have architecture duties.  You're worried about the future of the code.  
You have to understand just about everybody's subsystem, so you can veto a 
patch from somebody like Jens Axboe or Andre Hedrick if you have an objection 
to it.

An integration maintainer would NOT be making any major architectural 
decisions, they would be integrating the code from the maintainers, 
collecting the patches for the unmaintained areas of code, and resolving 
issues between maintainers that are purely implementation details.

Then you code review what they do anyway as the architect, saying whether or 
not it's a good idea.  But you don't have to deal with the obvious grunt work 
that's largely a matter of figuring out which bits don't compile because 
person A was not talking to person B.

> The fact is, we've had "patch penguins" pretty much forever, and they
> are called subsystem maintainers.  They maintain their own subsystem, ie
> people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
> (USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
> Molnar (scheduler), Jeff Garzik (network drivers) etc etc.

I'm suggesting an integration maintainer, whose explicit job is to put 
together patches from the various subsystem maintainers, and only directly 
accept patches for areas of code that do not HAVE any other subsystem 
maintainer.

Alan Cox used to do this (and is starting to do it again for Marcelo in 2.4). 
 Dave Jones is currently the guy doing this for you, taking patches, sorting 
through them, and then feeding them on to you.

> If there are problems with certain patches, it tends to be due to one or
> more of:
>
>  - the subsystem is badly modularized (quite common, originally. I don't
>    think many people realize how _far_ Linux has come in the last five
>    years wrt issues like architectures, driver independence, filesystem
>    infrastructure etc). And it still happens.

Yup.  Architecture issue.  Still your problem, I'm fraid.

>  - lack of maintainer interest. Many "maintainers" are less interested
>    in true merging than in trying to just push whatever code they have,
>    and only ever grow their patches instead of keeping them in pieces.

Patch penguin's job.  Foist this grunt work off on him.

>    This is usually a matter of getting used to it, and the best people
>    get used to it really quickly (Andrea, for example, used to not do
>    this well at all, but look at how he does it now! From a merge
>    standpoint, his patches have gone from "horrible" to "very good")

And needed patches from people who aren't very good have to wait years for 
the developer to learn how to feed you the right kind of patches?

If the patch is for a specific subsystem, then obviously your first line of 
defense fighting off sturgeon's law is the subsystem maintainer.  But you 
don't seem to have been taking patches even from subsystem maintainers in a 
timely manner, and how can people tell the difference between you dropping a 
patch because you have an objection to it and you dropping a patch because 
your mailbox overfloweth?  (You keep complaining people never send you 
patches.  People are suggesting automated patch remailers to spam your 
mailbox even harder.  There has GOT to be a better way...)

>  - personality/communication issues. Yes, they happen. I've tried to
>    have other people be "filters" for the people I cannot work with, but
>    I have to say that most of the time when I can't work with somebody,
>    others have real problems with those people too.
>
>    (An example of a very successful situation: David Miller and Alexey
>    Kuznetsov: Alexey used to have these rather uncontrolled patches that
>    I couldn't judge or work with but that obviously had a lot of
>    potential, and David acting as a filter made them a very successful
>    team.)
>
> In short, if you have areas or patches that you feel have had problems,
> ask yourself _why_ those areas have problems.

Query: Do you not believe you have been dropping a significant number of good 
patches on the floor?

> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and
> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me.

You don't see an integration maintainer as a step in the right direction?  
(It's not a star topology, it's a tree.)

Having lots of dispirate overlapping trees fragments development, fragments 
the testers, and makes an awful lot more WORK for everybody involved.

> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

Are you saying that Alan Cox's didn't serve a purpose during the 2.2 kernel 
time frame, and that Dave Jones is currently wasting his time? 

I'm confused here: "don't try to come up with a patch penguin", "help 
existing maintainers" (that's the patch penguin's job) "help grow new 
[maintainers]"...  The patch penguin IS an integration maintainer.  That's 
what I'm talking about.  (Patch penguin, patch pumpkin.  Patch pumkin, patch 
penguin.  I can say "integration maintainer" if it would help...)

I missed a curve somewhere.  Maybe the original message wasn't clear?  I am 
suggesting making Dave Jones the integration maintainer (a position he 
currently unofficially holds, and which Alan Cox did before him), and simply 
telling everybody who's complaining that their patches are getting silently 
dropped or ignored to try getting them into HIS tree first before bothering 
you about it.

I'm not asking for a major change here, I'm talking about clarifying the 
current ad-hoc development process.  Formalizing an existing de facto 
position so people farther out in the development process know what to do and 
where to go.

> 		Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Mon, 28 Jan 2002, Rob Landley wrote:
>
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
> > quite frankly, the way people work is hardcoded in our DNA.  Nobody
> > "really trusts" hundreds of people.  The way to make these things scale
> > out more is to increase the network of trust not by trying to push it on
> > me, but by making it more of a _network_, not a star-topology around me.
>
> You don't see an integration maintainer as a step in the right direction?
> (It's not a star topology, it's a tree.)

No, I don't really think an "integration manager" works well.

I think it helps a lot to have people pick up patches that nobody else
wants to maintain, and to gather them up. Andrea does that to some degree.
But it is _much_ better if you have somebody who is a point-man for
specific areas.

The problem with an overall guy is that there can't be too many of them.
The very thing you are _complaining_ about is in fact that there are a
number of over-all guys without clear focus, which only leads to confusion
about who handles what.

Clarity is good.

> Are you saying that Alan Cox's didn't serve a purpose during the 2.2 kernel
> time frame, and that Dave Jones is currently wasting his time?

No, I'm saying that there are not very many peopel who can do it, and who
can get the kind of trust that they are _everywhere_. Let's face it, Alan
grew to be respected because he did lots of different stuff over many
years, and he proved himself more than capable. And I suspect he's _quite_
happy not being in the middle of it right now.. It's a tough job.

It's a lot more likely to find people who can maintain _parts_. And if
there are patches that fall out of those parts, that tends to indicate a
lack of modularity, and perhaps a lack of maintainer for those parts.

And more likely, even if you _do_ find ten people who can do everything,
you don't want them to.

		Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Mon, Jan 28, 2002 at 10:00:19PM -0800, Linus Torvalds wrote:
> And more likely, even if you _do_ find ten people who can do everything,
> you don't want them to.

[and other things, in general saying that he wasn't happy with the approach]

What you didn't do, Linus, is paint a picture which allows development
to scale up.  Perhaps you don't want it to do so; looking back on Sun's
development process has taught me that a lot of it was just a way to
slow things down enough that the architects could keep up with the many
hacks going on here there and everywhere.  If this was Sun, I'd say that
slowing things down is good.

But cast your mind back to your "Linux is evolution" line of reasoning
and ask yourself if that does not mean that allowing the fastest
forward progress is what you want.  It would seem so, but at this point,
Linus, it is hard to predict what you think, so I'll pass on guessing.

What would be nice is if you came out with a clear statement, similar
to Rob's written summary, that said what it is that you would like to
see happen to address the issues that have come up over and over again.

The alternative is that sooner or later someone or some group will come
up with a better answer, there will be an ugly fight, and a lot of time
will get wasted when we could have learned how to grow up nicely.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Mon, 28 Jan 2002, Larry McVoy wrote:
>
> What you didn't do, Linus, is paint a picture which allows development
> to scale up.

Actually, I thought I did.

Basic premise: development is done by humans.

Now, look at how humans work. I don't know _anybody_ who works with
hundreds of people. You work with 5-10 people, out of a pool of maybe
30-50 people. Agreed?

Now, look at the suggested "patch penguin", and realize that the
suggestion doesn't add any scaling at all: it only adds a simple layer
that has all the same scaling problems.

What I'm saying is

 - I'm never going to work with hundreds of people directly, because it is
   fundamentally against my nature, by virtue of being human.

 - adding a "patch penguin" doesn't help, because _he_ (or she, although I
   didn't see any women nominated) is also going to be human. So either
   the patch penguin is going to do a bad job (and I won't start trusting
   him/her), or the patch penguin is going to have all the same issues
   people complain about.

Those are obvious truths. If you don't see them as being obvious truths,
you just haven't been thinking things through.

Did Alan do a good job? Yes. He did a _great_ job. But let's face it: (a)
he got really tired of doing it and (b) it really works only with one or
two Alan's, not with more - because with more you get people complaining
about the -aa tree vs the -dj tree vs the -marcelo tree vs the -linus
tree.

So Alan doesn't scale up either - I doubt you'll find a "better Alan", and
I _seriously_ doubt you'll be able to have multiple Alan's.

Does anybody really doubt this?

Now, if you've read this far, and you agree with these issues, I suspect
you know the solution as well as I do. It's the setup I already mentioned:
a network of maintainers, each of whom knows other maintainers.

And there's overlap. I'm not talking about a hierarchy here: it's not the
"general at the top" kind of tree-like setup. The network driver people
are in the same general vicinity as the people doing network protocols,
and there is obviously a lot of overlap.

Are there problems with maintainership? Yes. The main one being that it's
too damn easy to step on any toes. Which is why you want the modularity,
ie you do NOT want to have files issues that slash across different
maintenance boundaries (unless they have clearly defined interfaces:
that's where the importance of a clear VFS interface design comes in etc)

For an example of this, look at init/main.c in 2.2.x, in 2.4.x, and in
2.5.x. BIG changes. And most of the changes have very little to do with
what that file actually does (relatively little), and _everything_ to do
with the fact that it is the file that "instantiates" almost everything
and thus crosses all boundaries.

And notice how the "initcall" etc setup has changed, and cut a lot of
those dependencies. That's very much by design: look at what a device
driver had to do (and know) to be either a compiled-in driver or a modular
driver a few years ago. And look at a driver today.

In short, I'm saying that the true path to scalability is:

 - lack of dependencies on a source level

   This implies good interfaces that do NOT have common source files for
   different projects. In particular, it implies dynamic add/remove
   without the rest of the system having to know at all.

 - lack of people (whether patch-penguins or me) who have to follow
   everything.

   This, in turn, implies two things:

   (a) it is not the maintainers who pull things into their trees (because
       they aren't always there, and they don't know everything). It is
       the developers who _push_ onto maintainers.

   (b) if you as a developer cannot find a maintainer who you know, it is
       YOUR problem, and you cannot blame some super-penguin in the blue
       yonder for not caring about you. You may have to maintain your
       patch-set yourself. Or you should find a maintainer who cares about
       the work you do, and who helps feed it forward.

I know, I know. It's easier to whine about other people than it is to take
responsibility for your own actions. It's so easy to complain about "Linus
doesn't apply my patches", and so hard to just face the fact that Linus
never _will_ care about all patches, and that if you cannot find anybody
else to care about them either, then maybe they should die or you should
take care and feed them yourself.

I'm a bastard. I'm spending a lot of time applying patches, but it's not
my whole life. Never has been, never will be. I'll always be better at
applying patches from those ten people that I know and I trust than I'll
be at applying patches from people I don't trust.

If you're a developer, and you can't get through to me, ask yourself if
you can get through to somebody else.

And if you can't get through to anybody else either, ask yourself whether
maybe the problem is at _your_ end.

			Linus

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Linus Torvalds wrote:

>On Mon, 28 Jan 2002, Larry McVoy wrote:
>
>>What you didn't do, Linus, is paint a picture which allows development
>>to scale up.
>>
>
>Actually, I thought I did.
>
>Basic premise: development is done by humans.
>
>Now, look at how humans work. I don't know _anybody_ who works with
>hundreds of people. You work with 5-10 people, out of a pool of maybe
>30-50 people. Agreed?
>
Not at all. Please have a look at the ARMY. (A tightly hierarchical 
system...)

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >Now, look at how humans work. I don't know _anybody_ who works with
> >hundreds of people. You work with 5-10 people, out of a pool of maybe
> >30-50 people. Agreed?
> >
> Not at all. Please have a look at the ARMY. (A tightly hierarchical
> system...)

a general wont work with hundreds of people *directly*.

and i doubt 'giving orders' qualifies as 'works with'. 'Works with' is the
close circle around the general whom he talks to about current issues and
whose advice he listens to.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Stephan von Krawczynski]

On Tue, 29 Jan 2002 12:45:46 +0100
Martin Dalecki <dalecki@evision-ventures.com> wrote:

> Linus Torvalds wrote:
> 
> >On Mon, 28 Jan 2002, Larry McVoy wrote:
> >
> >>What you didn't do, Linus, is paint a picture which allows development
> >>to scale up.
> >>
> >
> >Actually, I thought I did.
> >
> >Basic premise: development is done by humans.
> >
> >Now, look at how humans work. I don't know _anybody_ who works with
> >hundreds of people. You work with 5-10 people, out of a pool of maybe
> >30-50 people. Agreed?
> >
> Not at all. Please have a look at the ARMY. (A tightly hierarchical 
> system...)

Shoot me: where the heck is the creative/innovative element inside the ARMY?
It just died somewhere down the hierarchy tree...
Ants are a very successful species, too, but they will hardly ever write
software (personal guess).

Regards,
Stephan

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 12:37 pm, Stephan von Krawczynski wrote:
> On Tue, 29 Jan 2002 12:45:46 +0100
>
> Martin Dalecki <dalecki@evision-ventures.com> wrote:
> > Linus Torvalds wrote:
> > >On Mon, 28 Jan 2002, Larry McVoy wrote:
> > >>What you didn't do, Linus, is paint a picture which allows development
> > >>to scale up.
> > >
> > >Actually, I thought I did.
> > >
> > >Basic premise: development is done by humans.
> > >
> > >Now, look at how humans work. I don't know _anybody_ who works with
> > >hundreds of people. You work with 5-10 people, out of a pool of maybe
> > >30-50 people. Agreed?
> >
> > Not at all. Please have a look at the ARMY. (A tightly hierarchical
> > system...)
>
> Shoot me: where the heck is the creative/innovative element inside the
> ARMY? It just died somewhere down the hierarchy tree...
> Ants are a very successful species, too, but they will hardly ever write
> software (personal guess).
>
> Regards,
> Stephan

This is only marginally on-topic (at best), but I agree the army is a very 
bad model for distributed collaborative development organization to try to 
emulation.  I have in fact written a series about this back when I wrote 
stock market investment columns (strange hobby for a programmer, I know...)

http://www.fool.com/news/foth/2000/foth000731.htm
http://www.fool.com/news/foth/2000/foth000913.htm
http://www.fool.com/news/foth/2000/foth000905.htm
http://www.fool.com/news/foth/2000/foth000918.htm
http://www.fool.com/news/foth/2000/foth000925.htm
http://www.fool.com/portfolios/rulemaker/2000/rulemaker000928.htm
http://www.fool.com/news/foth/2000/foth001002.htm

Linux development is a fan club.  And if you don't think fan clubs can 
mobilize and coordinate truly huge amounts of effort, you've obviously never 
been to worldcon.

But "the prototype and the fan club" is a seperate thing.  Maybe if 
somebody's really really bored I'll give them that talk during a BOF at LWE...

Rob

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Tue, 29 Jan 2002, Rob Landley wrote:

> Linux development is a fan club.

*plonk*

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 06:37 pm, Stephan von Krawczynski wrote:
> On Tue, 29 Jan 2002 12:45:46 +0100
> Martin Dalecki <dalecki@evision-ventures.com> wrote:
> > Linus Torvalds wrote:
> > >On Mon, 28 Jan 2002, Larry McVoy wrote:
> > >>What you didn't do, Linus, is paint a picture which allows development
> > >>to scale up.
> > >
> > >Actually, I thought I did.
> > >
> > >Basic premise: development is done by humans.
> > >
> > >Now, look at how humans work. I don't know _anybody_ who works with
> > >hundreds of people. You work with 5-10 people, out of a pool of maybe
> > >30-50 people. Agreed?
> > >
> > Not at all. Please have a look at the ARMY. (A tightly hierarchical 
> > system...)
> 
> Shoot me: where the heck is the creative/innovative element inside the ARMY?
> It just died somewhere down the hierarchy tree...
> Ants are a very successful species, too, but they will hardly ever write
> software (personal guess).

Correct, they don't write it, they evolve it.

/me ducks and runs for cover

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Eric W. Biederman]

Linus Torvalds <torvalds@transmeta.com> writes:
 
> Now, if you've read this far, and you agree with these issues, I suspect
> you know the solution as well as I do. It's the setup I already mentioned:
> a network of maintainers, each of whom knows other maintainers.
> 
> And there's overlap. I'm not talking about a hierarchy here: it's not the
> "general at the top" kind of tree-like setup. The network driver people
> are in the same general vicinity as the people doing network protocols,
> and there is obviously a lot of overlap.

So the kernel maintainership becomes a network of maintainers.  Then
we only have to understand the routing protocols.  Currently the
routing tables appear to have Linus as the default route.  As there
are currently kernel subsystems that do not have a real maintainer, it
may reasonable to have a misc maintainer.  Who looks after the
orphaned code, rejects/ignores patches for code that does have
active maintainers, and looks for people to be maintainers of the
orphaned code.  

The key is having enough human to human protocol that there is someone
besides Linus you can send your code to.  Or at least when there isn't
people are looking for someone.

Free Software obtains a lot of it's value by many people scratching an
itch and fixing a little bug, or adding a little feature, sending the
code off and then they go off to something else.  We need to have the
maintainer routing protocol clear enough, and the maintainer coverage
good enough so we can accumulate most of the bug fixes from the fly by
night hackers.  

So does anyone have any good ideas about how to build up routing
tables?  And almost more importantly how to make certain we have good
maintainer coverage over the entire kernel?

Eric

Re: A modest proposal -- We need a patch penguin

[Momchil Velikov]

>>>>> "Eric" == Eric W Biederman <ebiederm@xmission.com> writes:

Eric> So does anyone have any good ideas about how to build up routing
Eric> tables?

Umm, broadcasting to lkml ?:)

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 02:19 pm, Eric W. Biederman wrote:
> So the kernel maintainership becomes a network of maintainers.  Then
> we only have to understand the routing protocols.  Currently the
> routing tables appear to have Linus as the default route.  As there
> are currently kernel subsystems that do not have a real maintainer, it
> may reasonable to have a misc maintainer.  Who looks after the
> orphaned code, rejects/ignores patches for code that does have
> active maintainers, and looks for people to be maintainers of the
> orphaned code.  
> 
> The key is having enough human to human protocol that there is someone
> besides Linus you can send your code to.  Or at least when there isn't
> people are looking for someone.
> 
> Free Software obtains a lot of it's value by many people scratching an
> itch and fixing a little bug, or adding a little feature, sending the
> code off and then they go off to something else.  We need to have the
> maintainer routing protocol clear enough, and the maintainer coverage
> good enough so we can accumulate most of the bug fixes from the fly by
> night hackers.  
> 
> So does anyone have any good ideas about how to build up routing
> tables?  And almost more importantly how to make certain we have good
> maintainer coverage over the entire kernel?

Yes, we should cc our patches to a patchbot:

  patches-2.5@kernel.org -> goes to linus
  patches-2.4@kernel.org -> goes to marcello
  patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
  etc.

The vast sea of eyeballs will do the rest.  A web interface would be a nice 
bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is the 
essential ingredient.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 06:51 pm, Daniel Phillips wrote:
> On January 29, 2002 02:19 pm, Eric W. Biederman wrote:
> > So the kernel maintainership becomes a network of maintainers.  Then
> > we only have to understand the routing protocols.  Currently the
> > routing tables appear to have Linus as the default route.  As there
> > are currently kernel subsystems that do not have a real maintainer, it
> > may reasonable to have a misc maintainer.  Who looks after the
> > orphaned code, rejects/ignores patches for code that does have
> > active maintainers, and looks for people to be maintainers of the
> > orphaned code.
> >
> > The key is having enough human to human protocol that there is someone
> > besides Linus you can send your code to.  Or at least when there isn't
> > people are looking for someone.
> >
> > Free Software obtains a lot of it's value by many people scratching an
> > itch and fixing a little bug, or adding a little feature, sending the
> > code off and then they go off to something else.  We need to have the
> > maintainer routing protocol clear enough, and the maintainer coverage
> > good enough so we can accumulate most of the bug fixes from the fly by
> > night hackers.
> >
> > So does anyone have any good ideas about how to build up routing
> > tables?  And almost more importantly how to make certain we have good
> > maintainer coverage over the entire kernel?
>
> Yes, we should cc our patches to a patchbot:
>
>   patches-2.5@kernel.org -> goes to linus
>   patches-2.4@kernel.org -> goes to marcello
>   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
>   etc.
>
> The vast sea of eyeballs will do the rest.  A web interface would be a nice
> bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is
> the essential ingredient.

And of course there's not much point in having patches go to that list that 
AREN'T public (um, they're for inclusion into a public tree, right)?  So the 
patchbot might as well distribute to a mailing list, as long as there's some 
variety of moderation (possibly just a procmail recipe) to delete everything 
that didn't actually have a patch in it.  (Yet another discussion list is 
unlikely to help matters too much.)

Of course this still doesn't address the problem of patches going stale if 
they're not applied for a version or two and something else that goes in 
breaks them...

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 08:33:03PM -0500, Rob Landley wrote:
> > Yes, we should cc our patches to a patchbot:
> >
> >   patches-2.5@kernel.org -> goes to linus
> >   patches-2.4@kernel.org -> goes to marcello
> >   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
> >   etc.
> >
> > The vast sea of eyeballs will do the rest.  A web interface would be a nice
> > bonus, but 'patch sent and seen to be sent, to whom, when, what, why' is
> > the essential ingredient.
> 
> And of course there's not much point in having patches go to that list that 
> AREN'T public

If mail sent to the above addresses is not public somehow, the idea
is a non-starter.


> Of course this still doesn't address the problem of patches going stale if 
> they're not applied for a version or two and something else that goes in 
> breaks them...

If you really want to be a patch penguin then.... just do it.

You don't need specific permission to pick up, update, and maintain
patches that don't make it into the Linus tree on the first try.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:46 pm, Jeff Garzik wrote:
> On Tue, Jan 29, 2002 at 08:33:03PM -0500, Rob Landley wrote:

> > Of course this still doesn't address the problem of patches going stale
> > if they're not applied for a version or two and something else that goes
> > in breaks them...
>
> If you really want to be a patch penguin then.... just do it.
>
> You don't need specific permission to pick up, update, and maintain
> patches that don't make it into the Linus tree on the first try.

I'm not asking to become a patch penguin, and the various other people who 
volunteered early on, though well intentioned, slightly missed my point as 
well.

We used to HAVE a patch penguin.  "Miscelaneous maintainer", "integration 
lieutenant", call the position what you will.  His name was Alan Cox.  He 
recently abdicated the position, which has since gradually been assumed by 
Dave Jones.  There is serious pressure on Dave Jones's tree to accept the 
kind of patches Alan used to (and which Alan is still accepting for 2.4 and 
queuing for Marcelo).  If Dave continues to put out a tree, he would have to 
work fairly hard to avoid becoming Alan Cox's successor.

I was hoping for some sort of indication that if patches DID get into Dave's 
tree, it would be a step towards their eventual consideration by Linus.  Not 
a guarantee of inclusion, of course, but it would be nice to know if 
inclusion in Dave's tree would move the patch one step towards Linus, or 
would just head down a cul-de-sac and additional fragmentation of the 
development process.

I was also trying to point out that there seems to be a recurring role here, 
which used to be identified with "just Alan" but has now passed to another 
person, while still maintaining a noticeable portion of its character.  It 
might be nice to recognize it as such.

Also, I was trying to encourage ONE beta tree in order to DISCOURAGE the 
fragmented proliferation of version-skewed trees accepting third party 
patches that seem to have been cropping up recently.  (See the linux weekly 
news and kernel traffic links in the original posting that started this whole 
thread.)  In the absence of an -ac tree to accept patches that show 
significant promise but are not ready for Linus and vice versa, patches are 
accumulating in multiple trees.  This fragments the tester base, and seemed 
to me to be a less efficient way than the way things worked before Alan quit.

There seems to have been widespread misinterpretation of these objectives...

> 	Jeff

Rob

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> Yes, we should cc our patches to a patchbot:
>
>   patches-2.5@kernel.org -> goes to linus
>   patches-2.4@kernel.org -> goes to marcello
>   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
>   etc.

I'd rather make the patchbot more intelligent, that means it analyzes the
patch and produces a list of touched files. People can now register to get
notified about patches, which changes areas they are interested in.
In the simplest configuration nothing would change for Linus, but patches
wouldn't get lost and people could be notified if their patch was applied
or if it doesn't apply anymore. Other people have a place to search for
patches and they can check whether something was already fixed.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 11:39 am, Roman Zippel wrote:
> Hi,
> 
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> 
> > Yes, we should cc our patches to a patchbot:
> >
> >   patches-2.5@kernel.org -> goes to linus
> >   patches-2.4@kernel.org -> goes to marcello
> >   patches-usb@kernel.org -> goes to gregkh, regardless of 2.4/2.5
> >   etc.
> 
> I'd rather make the patchbot more intelligent, that means it analyzes the
> patch and produces a list of touched files. People can now register to get
> notified about patches, which changes areas they are interested in.

But they can already do that, by subscribing to the respective mailing list 
(obviously, the bot posts to the list as well as forwarding to the 
maintainer) and running the mails through a filter of their choice.

> In the simplest configuration nothing would change for Linus, but patches
> wouldn't get lost and people could be notified if their patch was applied
> or if it doesn't apply anymore.

OK, it would be nice, but you wouldn't want to pile on so many features that 
this never gets implemented would you?  The minimal thing that forwards and 
posts patches is what we need now.  Like any other software it can be 
improved over time.

> Other people have a place to search for patches and they can check whether 
> something was already fixed.

Automating the applied/dropped status is clearly the next problem to tackle, 
but that's harder, it involves behavioral changes on the maintainers side.  
(Pragmatically, providing a web interface so somebody whose job it is to do 
that, can efficiently post 'applied' messages to the list would get the job 
done without making anyone learn new tools or change the way they work.)

By the way, who is going to code this?  Or are we determined to make 
ourselves look like wankers once again, by putting considerably more time 
into the lkml flamewar than goes into producing working code?

(Hint: I am not going to code it, nor should I since I should be working in 
the kernel.)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> > I'd rather make the patchbot more intelligent, that means it analyzes the
> > patch and produces a list of touched files. People can now register to get
> > notified about patches, which changes areas they are interested in.
>
> But they can already do that, by subscribing to the respective mailing list
> (obviously, the bot posts to the list as well as forwarding to the
> maintainer) and running the mails through a filter of their choice.

What about unmaintained parts?

> > In the simplest configuration nothing would change for Linus, but patches
> > wouldn't get lost and people could be notified if their patch was applied
> > or if it doesn't apply anymore.
>
> OK, it would be nice, but you wouldn't want to pile on so many features that
> this never gets implemented would you?  The minimal thing that forwards and
> posts patches is what we need now.  Like any other software it can be
> improved over time.

That's what I have in mind. What we can already do now is to store
incoming patches into some directory. That would give us already some
basic data to work with and we can start to implement new features as they
are needed.

> Automating the applied/dropped status is clearly the next problem to tackle,
> but that's harder, it involves behavioral changes on the maintainers side.

What "behavioral changes"? Maintainers should react in some way or another
react to patches no matter where come from.

> (Pragmatically, providing a web interface so somebody whose job it is to do
> that, can efficiently post 'applied' messages to the list would get the job
> done without making anyone learn new tools or change the way they work.)

Web interfaces can be nice, but the bulk work should be doable by mail.
For changes in areas which have a maintainer, the mail to Linus could
include a note "forwarded to maintainer x" and Linus can still decide,
whether he applies the patch or waits for the OK from the maintainer.

> By the way, who is going to code this?  Or are we determined to make
> ourselves look like wankers once again, by putting considerably more time
> into the lkml flamewar than goes into producing working code?
>
> (Hint: I am not going to code it, nor should I since I should be working in
> the kernel.)

That's a known problem, I have no time either, but we should give anyone
interested in this some example data. This data has to come from the
kernel hackers, but patch management system is better implemented by
non-kernel hackers.

Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On January 30, 2002 01:39 pm, Roman Zippel wrote:
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> 
> > > I'd rather make the patchbot more intelligent, that means it analyzes 
the
> > > patch and produces a list of touched files. People can now register to 
get
> > > notified about patches, which changes areas they are interested in.
> >
> > But they can already do that, by subscribing to the respective mailing 
list
> > (obviously, the bot posts to the list as well as forwarding to the
> > maintainer) and running the mails through a filter of their choice.
> 
> What about unmaintained parts?

One or both of patches-2.4@kernel.org or patches-2.5@kernel.org

> > > In the simplest configuration nothing would change for Linus, but 
patches
> > > wouldn't get lost and people could be notified if their patch was 
applied
> > > or if it doesn't apply anymore.
> >
> > OK, it would be nice, but you wouldn't want to pile on so many features 
that
> > this never gets implemented would you?  The minimal thing that forwards 
and
> > posts patches is what we need now.  Like any other software it can be
> > improved over time.
> 
> That's what I have in mind. What we can already do now is to store
> incoming patches into some directory. That would give us already some
> basic data to work with and we can start to implement new features as they
> are needed.

Yes, mine the data.

> > Automating the applied/dropped status is clearly the next problem to 
tackle,
> > but that's harder, it involves behavioral changes on the maintainers side.
> 
> What "behavioral changes"? Maintainers should react in some way or another
> react to patches no matter where come from.

They already have their own ways of reacting.  I don't hear a lot of pleading
from maintainers for tools to help them respond to patches; most seem to be
satisfied with Mutt or whatever.  I feel sure that any attempt to force
changes to such personal practices will be met with a solid wall of
disinterest.

> > (Pragmatically, providing a web interface so somebody whose job it is to 
do
> > that, can efficiently post 'applied' messages to the list would get the 
job
> > done without making anyone learn new tools or change the way they work.)
> 
> Web interfaces can be nice, but the bulk work should be doable by mail.

Oh yes, certainly.

> For changes in areas which have a maintainer, the mail to Linus could
> include a note "forwarded to maintainer x" and Linus can still decide,
> whether he applies the patch or waits for the OK from the maintainer.

Or just cc it to lkml and don't bother Linus.  If Linus wants to know about
traffic to maintainers he can look in the mail archives (he won't).

> > By the way, who is going to code this?  Or are we determined to make
> > ourselves look like wankers once again, by putting considerably more time
> > into the lkml flamewar than goes into producing working code?
> >
> > (Hint: I am not going to code it, nor should I since I should be working 
in
> > the kernel.)
> 
> That's a known problem, I have no time either, but we should give anyone
> interested in this some example data. This data has to come from the
> kernel hackers, but patch management system is better implemented by
> non-kernel hackers.

<deleted>
That's the problem.  I have seen at least two attempted starts on a patchbot
and know of a third (Dan Quinlan was going to do something 1.5 years ago).
So at this point I want to step out of this discussion.  There is going to
be no patchbot without a coder to write it, so why spend more time talking
about it?
</deleted>

OK, on a less pessimistic note, I'll make a small effort to find a volunteer 
to code this, under advisement that it will be a thankless task (everybody 
will complain about everything) and may not even get accepted by Linus or 
anyone else (yes, and?).  So here it is:

   Wanted: a scripting person who has a clue about MTAs and wants to 
   contribute to the kernel.  Please step up to the table over here
   and sign in blood, then we will tell you what your mission is.
   Nobody will thank you for any of the work you do or reward you in
   any way, except for the right to bask in the glory and fame of
   being the one who ended the patchbot wars.  And maybe, just maybe
   get that coveted Slashdot interview.

OK. that's it, if somebody bites I'll gladly participate in a design thread, 
otherwise I think this is just going to sleep until the next bi-monthly 
patchbot flameup.

/me goes back to work

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Rasmus Andersen]

On Wed, Jan 30, 2002 at 02:28:04PM +0100, Daniel Phillips wrote:
>    Wanted: a scripting person who has a clue about MTAs and wants to 
>    contribute to the kernel.  Please step up to the table over here
>    and sign in blood, then we will tell you what your mission is.
>    Nobody will thank you for any of the work you do or reward you in
>    any way, except for the right to bask in the glory and fame of
>    being the one who ended the patchbot wars.  And maybe, just maybe
>    get that coveted Slashdot interview.
> 
> OK. that's it, if somebody bites I'll gladly participate in a design thread, 
> otherwise I think this is just going to sleep until the next bi-monthly 
> patchbot flameup.

I'll bite. I was noodling with this in the background already, so
I have some thoughts at home which I'll be happy to write up and
send to the list. Other people, notably John Weber (linuxhq)
and Patrick Mochel(? odsl) stated that they were working on
something too. As I dont do this for a living (dont mean to
imply that they are), I wouldn't want to want to be in the
way for the big boys :)

If I understand correctly, the bot would, in its basic incarnation,
accept patches (at patchbot@somewhere), stamp them with an uid,
and forward them to various places, e.g., lists, maintainers etc
and let the sumbitter know the patch uid. A mailing list archive
would then be the patch store. Basic filtering could be done by
the bot to reject non-patches etc.

The bot could also:
* Annotate patches with files touched.
* Try to apply patches and take action based on succces failure.
* Try to compile the resulting tree (based on a submitted .config)
  and take action based on the results.
* Store submissions locally and do the steps above for new
  kernel revisions, resubmtting them if appropriate.

Yes, the compile step kinda made the HW requirements go through
the roof.

I have some code already to handle some of this but typically,
I started at the wrong end and did the patch/compile stuff
first :) Ah, BTW, that is in python. I dont see a problem
with that.

Please comment but I may be offline till later this evening.

Regards,
  Rasmus

PS: Daniel have made me aware of another volunteer, Kalle
Kivimaa. I have added him to the list and could obviously
work with him.

Re: Wanted: Volunteer to code a Patchbot

[Rasmus Andersen]

On Wed, Jan 30, 2002 at 04:11:05PM +0100, Rasmus Andersen wrote:
> If I understand correctly, the bot would, in its basic incarnation,
> accept patches (at patchbot@somewhere), stamp them with an uid,
> and forward them to various places, e.g., lists, maintainers etc
> and let the sumbitter know the patch uid. A mailing list archive
> would then be the patch store. Basic filtering could be done by
> the bot to reject non-patches etc.

Somehow, I totally forgot the security disclaimer for some of
the points. Obviously, mindlessly patching a makefile and
executing it would be a Bad Idea. If no satisfying solution
to this can be found, this (execute/compile) step could be 
foregone.

Thanks to Tommy Faasen for raising this point.

Regards,
  Rasmus

Re: Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On January 30, 2002 04:28 pm, Rasmus Andersen wrote:
> On Wed, Jan 30, 2002 at 04:11:05PM +0100, Rasmus Andersen wrote:
> > If I understand correctly, the bot would, in its basic incarnation,
> > accept patches (at patchbot@somewhere), stamp them with an uid,
> > and forward them to various places, e.g., lists, maintainers etc
> > and let the sumbitter know the patch uid. A mailing list archive
> > would then be the patch store. Basic filtering could be done by
> > the bot to reject non-patches etc.
> 
> Somehow, I totally forgot the security disclaimer for some of
> the points. Obviously, mindlessly patching a makefile and
> executing it would be a Bad Idea. If no satisfying solution
> to this can be found, this (execute/compile) step could be 
> foregone.
> 
> Thanks to Tommy Faasen for raising this point.

I'd say, don't try to run it, just see if it applies cleanly.

Speaking of security, we can't expect Matti to take care of blocking spam
on the patch lists the way he does on lkml, so that is going to have to
be handled, or the system will fall apart.  Well, spammers are not going
to be bright enough to send correctly formed patches that apply without
rejects I hope, so maybe that is a non-problem.

The patchbot will have to understand the concept of a patch set, a
series of patches that apply in a particular order.  If it can handle
that it probably doesn't need a general way of handling inter-patch
relationships, at least to start.

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Stuart Young]

At 04:46 PM 30/01/02 +0100, Daniel Phillips wrote:
>On January 30, 2002 04:28 pm, Rasmus Andersen wrote:
> > Somehow, I totally forgot the security disclaimer for some of
> > the points. Obviously, mindlessly patching a makefile and
> > executing it would be a Bad Idea. If no satisfying solution
> > to this can be found, this (execute/compile) step could be
> > foregone.
> >
> > Thanks to Tommy Faasen for raising this point.
>
>I'd say, don't try to run it, just see if it applies cleanly.
>
>Speaking of security, we can't expect Matti to take care of blocking spam
>on the patch lists the way he does on lkml, so that is going to have to
>be handled, or the system will fall apart.  Well, spammers are not going
>to be bright enough to send correctly formed patches that apply without
>rejects I hope, so maybe that is a non-problem.

Possibly, but then it'll reply to the spammer and you'll get bounces left 
and right. Perhaps it's a simple case that the patcher submitting will have 
to have registered the email address before submitting their patch. Only 
needs to be done once (not every time a patch is submitted, that's mad!), 
and weeds out the noise.


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On January 31, 2002 01:49 am, Stuart Young wrote:
> At 04:46 PM 30/01/02 +0100, Daniel Phillips wrote:
> >On January 30, 2002 04:28 pm, Rasmus Andersen wrote:
> > > Somehow, I totally forgot the security disclaimer for some of
> > > the points. Obviously, mindlessly patching a makefile and
> > > executing it would be a Bad Idea. If no satisfying solution
> > > to this can be found, this (execute/compile) step could be
> > > foregone.
> > >
> > > Thanks to Tommy Faasen for raising this point.
> >
> >I'd say, don't try to run it, just see if it applies cleanly.
> >
> >Speaking of security, we can't expect Matti to take care of blocking spam
> >on the patch lists the way he does on lkml, so that is going to have to
> >be handled, or the system will fall apart.  Well, spammers are not going
> >to be bright enough to send correctly formed patches that apply without
> >rejects I hope, so maybe that is a non-problem.
> 
> Possibly, but then it'll reply to the spammer and you'll get bounces left 
> and right. Perhaps it's a simple case that the patcher submitting will have 
> to have registered the email address before submitting their patch. Only 
> needs to be done once (not every time a patch is submitted, that's mad!), 
> and weeds out the noise.

Yes, that's a point for discussion.  Certainly, a patchbot list like 
patches-2.5-maintainer should require registration, and in fact, registration 
for this list will be by invitation.  It's not so clear what the policy 
should be on the patches-2.5 list.  Openness is a nice thing to be able to 
boast about.  Maybe the thing to do is try it open, and see how it works out.

We also have to worry about malicious spamming of the patch list.  I've heard 
this happened to kuro5hin's story submission queue - there is no accounting 
for all the forms of insect life out there.

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Stuart Young]

At 02:26 AM 31/01/02 +0100, Daniel Phillips wrote:
>On January 31, 2002 01:49 am, Stuart Young wrote:
> > Possibly, but then it'll reply to the spammer and you'll get bounces left
> > and right. Perhaps it's a simple case that the patcher submitting will 
> have
> > to have registered the email address before submitting their patch. Only
> > needs to be done once (not every time a patch is submitted, that's mad!),
> > and weeds out the noise.
>
>Yes, that's a point for discussion.  Certainly, a patchbot list like
>patches-2.5-maintainer should require registration, and in fact, registration
>for this list will be by invitation.  It's not so clear what the policy
>should be on the patches-2.5 list.  Openness is a nice thing to be able to
>boast about.  Maybe the thing to do is try it open, and see how it works out.

True. But if it comes to it, a once off authentication to 'allow' an e-mail 
address weeds out a hell of a lot of spambots.

>We also have to worry about malicious spamming of the patch list.  I've heard
>this happened to kuro5hin's story submission queue - there is no accounting
>for all the forms of insect life out there.

Malicious spamming will happen no matter what you do, unless you vet the 
subscriptions manually.


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: Wanted: Volunteer to code a Patchbot

[Rik van Riel]

On Thu, 31 Jan 2002, Stuart Young wrote:

> Possibly, but then it'll reply to the spammer and you'll get bounces left
> and right. Perhaps it's a simple case that the patcher submitting will have
> to have registered the email address before submitting their patch. Only
> needs to be done once (not every time a patch is submitted, that's mad!),
> and weeds out the noise.

--------------------------------------------------------------
This is the patchbot auto-reply.

You tried to send me a patch (attached below) but I don't
know you.  To confirm that you exist (and aren't a spammer)
please reply to this message.

After receiving your reply your queued patches will be
published.
--------------------------------------------------------------

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: Wanted: Volunteer to code a Patchbot

[Patrick Mauritz]

On Thu, Jan 31, 2002 at 11:51:53AM -0200, Rik van Riel wrote:
> --------------------------------------------------------------
> This is the patchbot auto-reply.
> 
> You tried to send me a patch (attached below) but I don't
> know you.  To confirm that you exist (and aren't a spammer)
> please reply to this message.
> 
> After receiving your reply your queued patches will be
> published.
> --------------------------------------------------------------

ok, so then I look for some open relay and the email address of my
neighbour I dislike and send some hundred mails with his address in the 
From: field - lotsa fun...

patrick mauritz
-- 
,------------------------------------------------------------------------.
>   In the Beginning there was nothing, which exploded - Yeah right...   <
|------------------------------------------------------------------------|
>                      plex86 NOW! | www.plex86.org                      <
`------------------------------------------------------------------------'
         Your right to swing your fist ends where my nose begins

Re: Wanted: Volunteer to code a Patchbot

[Jan Harkes]

On Thu, Jan 31, 2002 at 04:29:20PM +0100, Patrick Mauritz wrote:
> On Thu, Jan 31, 2002 at 11:51:53AM -0200, Rik van Riel wrote:
> > --------------------------------------------------------------
> > This is the patchbot auto-reply.
> > 
> > You tried to send me a patch (attached below) but I don't
> > know you.  To confirm that you exist (and aren't a spammer)
> > please reply to this message.
> > 
> > After receiving your reply your queued patches will be
> > published.
> > --------------------------------------------------------------
> 
> ok, so then I look for some open relay and the email address of my
> neighbour I dislike and send some hundred mails with his address in the 
> From: field - lotsa fun...

Drop anything that is not text/plain and doesn't contain

diff -urN [--exclude-from=dontdiff]
--- yyy
+++ zzz

Maybe bounce when the diff includes any files that shouldn't be part of
the diff (http://www.moses.uklinux.net/patches/dontdiff) with a nice
message to get that file and add --exclude-from=dontdiff or explain that
patches can get dropped silently when they are not in unidiff format,
include generated files, or have any type of non text/plain attachment.

And whenever spam starts 'adhering' to the suggested format of linux
kernel patches we'll get an interesting kernel indeed. Where are the
many (human) eyes in this picture anyways. Do people get to vote
for/veto patches in the patchbot queue?

Evil thought, patchbot would turn into some form of a 'slashdot' with
karma whores and petrified goats and such. Hmm, maybe we should keep
things the way they are right now.

Jan

Re: Wanted: Volunteer to code a Patchbot

[Horst von Brand]

Stuart Young <sgy@amc.com.au> said:

[...]

> Possibly, but then it'll reply to the spammer and you'll get bounces left 
> and right. Perhaps it's a simple case that the patcher submitting will have 
> to have registered the email address before submitting their patch. Only 
> needs to be done once (not every time a patch is submitted, that's mad!), 
> and weeds out the noise.

And then lkml will be swamped with questions as to why the automated patch
system doesn't work, or it will just not be used at all because it is more
work than just firing off a patch at lkml.
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On January 31, 2002 11:05 pm, Horst von Brand wrote:
> Stuart Young <sgy@amc.com.au> said:
> 
> [...]
> 
> > Possibly, but then it'll reply to the spammer and you'll get bounces left 
> > and right. Perhaps it's a simple case that the patcher submitting will
> > have to have registered the email address before submitting their patch. 
> > Only needs to be done once (not every time a patch is submitted, that's 
> > mad!), and weeds out the noise.
> 
> And then lkml will be swamped with questions as to why the automated patch
> system doesn't work, or it will just not be used at all because it is more
> work than just firing off a patch at lkml.

The plan is to have both open and registered-users-only patchbots.  The 
second kind is the kind to which maintainers themselves submit to, so the 
forwarded stream of patches is guaranteed to come from trustworthy sources.  
Maintainers themselves can configure their own patchbots to be open or closed 
as they see fit.  In essense, neither submitters not maintainers will see any 
change at all in their procedures, except for the address to which they send 
the patch.[1]

There will be a very significant change in the results of this process from 
the submitter's point of view, since everybody will know where to look to see 
what patches have been submitted, to whom, when, why etc.

There are a lot of things we can do with the patches once they're all sitting 
in the patchbot's database, including tracking the state - applied, rejected, 
being revised, etc.  That's for later, the task at hand is simply to clarify 
and streamline the lines of communication between submitters and maintainers.

[1] Submitters *may* chose to fill in a few lines of metadata in their patch 
to specify, for example, a one-line description which is different from the 
email subject, or that they are not interested in confirmation.  Such 
metadata is not required - the patchbots will accept patches in exactly the 
format we are used to.

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Stuart Young]

At 11:51 AM 31/01/02 -0200, Rik van Riel wrote:
>On Thu, 31 Jan 2002, Stuart Young wrote:
>
>--------------------------------------------------------------
>This is the patchbot auto-reply.
>
>You tried to send me a patch (attached below) but I don't
>know you.  To confirm that you exist (and aren't a spammer)
>please reply to this message.
>
>After receiving your reply your queued patches will be
>published.
>--------------------------------------------------------------

Beautiful. Just beautiful. *grin*

Just hope the spammers don't catch on and start sending replies.


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

(reposted to fix the word wrap)

On January 30, 2002 01:39 pm, Roman Zippel wrote:
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> 
> > > I'd rather make the patchbot more intelligent, that means it analyzes the
> > > patch and produces a list of touched files. People can now register to get
> > > notified about patches, which changes areas they are interested in.
> >
> > But they can already do that, by subscribing to the respective mailing list
> > (obviously, the bot posts to the list as well as forwarding to the
> > maintainer) and running the mails through a filter of their choice.
> 
> What about unmaintained parts?

One or both of patches-2.4@kernel.org or patches-2.5@kernel.org

> > > In the simplest configuration nothing would change for Linus, but patches
> > > wouldn't get lost and people could be notified if their patch was applied
> > > or if it doesn't apply anymore.
> >
> > OK, it would be nice, but you wouldn't want to pile on so many features that
> > this never gets implemented would you?  The minimal thing that forwards and
> > posts patches is what we need now.  Like any other software it can be
> > improved over time.
> 
> That's what I have in mind. What we can already do now is to store
> incoming patches into some directory. That would give us already some
> basic data to work with and we can start to implement new features as they
> are needed.

Yes, mine the data.

> > Automating the applied/dropped status is clearly the next problem to tackle,
> > but that's harder, it involves behavioral changes on the maintainers side.
> 
> What "behavioral changes"? Maintainers should react in some way or another
> react to patches no matter where come from.

They already have their own ways of reacting.  I don't hear a lot of pleading
from maintainers for tools to help them respond to patches; most seem to be
satisfied with Mutt or whatever.  I feel sure that any attempt to force
changes to such personal practices will be met with a solid wall of
disinterest.

> > (Pragmatically, providing a web interface so somebody whose job it is to do
> > that, can efficiently post 'applied' messages to the list would get the job
> > done without making anyone learn new tools or change the way they work.)
> 
> Web interfaces can be nice, but the bulk work should be doable by mail.

Oh yes, certainly.

> For changes in areas which have a maintainer, the mail to Linus could
> include a note "forwarded to maintainer x" and Linus can still decide,
> whether he applies the patch or waits for the OK from the maintainer.

Or just cc it to lkml and don't bother Linus.  If Linus wants to know about
traffic to maintainers he can look in the mail archives (he won't).

> > By the way, who is going to code this?  Or are we determined to make
> > ourselves look like wankers once again, by putting considerably more time
> > into the lkml flamewar than goes into producing working code?
> >
> > (Hint: I am not going to code it, nor should I since I should be working in
> > the kernel.)
> 
> That's a known problem, I have no time either, but we should give anyone
> interested in this some example data. This data has to come from the
> kernel hackers, but patch management system is better implemented by
> non-kernel hackers.

<deleted>
That's the problem.  I have seen at least two attempted starts on a patchbot
and know of a third (Dan Quinlan was going to do something 1.5 years ago).
So at this point I want to step out of this discussion.  There is going to
be no patchbot without a coder to write it, so why spend more time talking
about it?
</deleted>

OK, on a less pessimistic note, I'll make a small effort to find a volunteer 
to code this, under advisement that it will be a thankless task (everybody 
will complain about everything) and may not even get accepted by Linus or 
anyone else (yes, and?).  So here it is:

   Wanted: a scripting person who has a clue about MTAs and wants to 
   contribute to the kernel.  Please step up to the table over here
   and sign in blood, then we will tell you what your mission is.
   Nobody will thank you for any of the work you do or reward you in
   any way, except for the right to bask in the glory and fame of
   being the one who ended the patchbot wars.  And maybe, just maybe
   get that coveted Slashdot interview.

OK. that's it, if somebody bites I'll gladly participate in a design thread, 
otherwise I think this is just going to sleep until the next bi-monthly 
patchbot flameup.

/me goes back to work

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Tim Waugh]

[-- Attachment #1: Type: text/plain, Size: 882 bytes --]

On Wed, Jan 30, 2002 at 02:45:59PM +0100, Daniel Phillips wrote:

> (reposted to fix the word wrap)
> 
> On January 30, 2002 01:39 pm, Roman Zippel wrote:
> > On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > 
> > > > I'd rather make the patchbot more intelligent, that means it
> > > > analyzes the patch and produces a list of touched
> > > > files. People can now register to get notified about patches,
> > > > which changes areas they are interested in.
> > >
> > > But they can already do that, by subscribing to the respective
> > > mailing list (obviously, the bot posts to the list as well as
> > > forwarding to the maintainer) and running the mails through a
> > > filter of their choice.

I realise I'm jumping in half-way through a thread, but people
interested in doing this might want to look at how lsdiff and
filterdiff can help.

Tim.
</plug>

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Wanted: Volunteer to code a Patchbot

[Patrick Mochel]

Hi everyone. 

> <deleted>
> That's the problem.  I have seen at least two attempted starts on a patchbot
> and know of a third (Dan Quinlan was going to do something 1.5 years ago).
> So at this point I want to step out of this discussion.  There is going to
> be no patchbot without a coder to write it, so why spend more time talking
> about it?
> </deleted>
> 
> OK, on a less pessimistic note, I'll make a small effort to find a volunteer 
> to code this, under advisement that it will be a thankless task (everybody 
> will complain about everything) and may not even get accepted by Linus or 
> anyone else (yes, and?).  So here it is:
> 
>    Wanted: a scripting person who has a clue about MTAs and wants to 
>    contribute to the kernel.  Please step up to the table over here
>    and sign in blood, then we will tell you what your mission is.
>    Nobody will thank you for any of the work you do or reward you in
>    any way, except for the right to bask in the glory and fame of
>    being the one who ended the patchbot wars.  And maybe, just maybe
>    get that coveted Slashdot interview.
> 
> OK. that's it, if somebody bites I'll gladly participate in a design thread, 
> otherwise I think this is just going to sleep until the next bi-monthly 
> patchbot flameup.

As promised yesterday, a Sourceforge project has been started for a Linux 
Kernel Patch Management System (lk-pms). The page is at:

http://sourceforge.net/projects/lk-pms/

A mailing has been set up for discussion of the development of the 
project. We would like to move the discussion off of linux-kernel onto 
this new mailing list and begin to formalize the concept and the design.

We (OSDL) are volunteering resources for the development of the project: 
developer time, hardware, and bandwidth. We are willing to host it on our 
servers, and would like to eventually integrate such a system with our 
automated test system (STP). Any and all feedback is welcome, preferably 
on the list, though private mail is also welcome. 


Thanks,

	-pat

Re: Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On January 30, 2002 06:46 pm, Patrick Mochel wrote:
> As promised yesterday, a Sourceforge project has been started for a Linux 
> Kernel Patch Management System (lk-pms). The page is at:
> 
> http://sourceforge.net/projects/lk-pms/

OK, we have competing patchbot projects:

   http://killeri.net/cgi-bin/alias/ezmlm-cgi

I don't see a single thing wrong with that, it gives us twice as many chances 
for one to succeed.

> A mailing has been set up for discussion of the development of the 
> project. We would like to move the discussion off of linux-kernel onto 
> this new mailing list and begin to formalize the concept and the design.
> 
> We (OSDL) are volunteering resources for the development of the project: 
> developer time, hardware, and bandwidth. We are willing to host it on our 
> servers, and would like to eventually integrate such a system with our 
> automated test system (STP). Any and all feedback is welcome, preferably 
> on the list, though private mail is also welcome. 

Ah, yum, we are jealous.

But not very ;-)

May the best bot win...

-- 
Daniel

Re: Wanted: Volunteer to code a Patchbot

[Peter C. Norton]

BTW, in the interest of not re-inventing the wheel, and I'm sorry if this
has been mentioned already in this thread, but for confirmations,
whitelist/blacklist stuff for the patchbots take a look at tmda, at
http://tmda.sourceforge.net.

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.

Re: Wanted: Volunteer to code a Patchbot

[Daniel Phillips]

On February 3, 2002 07:54 pm, Peter C. Norton wrote:
> BTW, in the interest of not re-inventing the wheel, and I'm sorry if this
> has been mentioned already in this thread, but for confirmations,
> whitelist/blacklist stuff for the patchbots take a look at tmda, at
> http://tmda.sourceforge.net.

Thanks for this, but whitelist/blacklist is just one aspect of the patchbot, 
and far from the most important aspect.  I think tmda might have a little 
trouble validating a patch against a kernel tree, for example

In Python the white/blacklist handling is trivial anyway.  A workable 
confirmation feature was coded in a couple of hours by the guys (Rasmus and 
Kalle) working on the bot, and it does exactly what we want it to.  Progress 
marches on, Python rocks.  (So do Rasmus and Kalle.)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 01:00 am, Linus Torvalds wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:
> > > A word of warning: good maintainers are hard to find.  Getting more of
> > > them helps, but at some point it can actually be more useful to help
> > > the _existing_ ones.  I've got about ten-twenty people I really trust,
> > > and quite frankly, the way people work is hardcoded in our DNA.  Nobody
> > > "really trusts" hundreds of people.  The way to make these things scale
> > > out more is to increase the network of trust not by trying to push it
> > > on me, but by making it more of a _network_, not a star-topology around
> > > me.
> >
> > You don't see an integration maintainer as a step in the right direction?
> > (It's not a star topology, it's a tree.)
>
> No, I don't really think an "integration manager" works well.

So what was Alan Cox doing all those years?  What is Dave Jones currently 
doing?

> I think it helps a lot to have people pick up patches that nobody else
> wants to maintain, and to gather them up. Andrea does that to some degree.

It's not a question of patches people don't want to maintain, it's a question 
of patches getting much wider testing and better feedback when they're in a 
larger tree, and the maintainers of the various patches getting better 
warning about other patches that break them or that they break other patches.

When two developers share a common tree, they notice when they break each 
other's stuff, and they resolve it.  When two developers go off in isolation, 
they break each other's stuff as a matter of course.  And testers who have to 
hunt down a patch are are willing to apply it generally aren't the ones who 
raise an objection once it gets applied to the next tree they download.

> But it is _much_ better if you have somebody who is a point-man for
> specific areas.

I'm not proposing replacing the current subsystem maintainers.  But are the 
current subsystem maintainers happy?

I thought they weren't, but I guess that by their silence, they must be 
thrilled, so...  (Sorry, I seem to be getting a lot more support in private 
than anybody is willing to cc: to the list.  I'm new at this politics 
business...)

> The problem with an overall guy is that there can't be too many of them.
> The very thing you are _complaining_ about is in fact that there are a
> number of over-all guys without clear focus, which only leads to confusion
> about who handles what.
>
> Clarity is good.

The fact Jens Axboe handles one system, Stephen C. Tweedie another, Andre 
Hedrick a third, Rik van Riel a fourth, and Eric Raymond a fifth, is not 
particularly confusing.  It's when the integration and debugging of Jens' 
patches in 2.5 blocks the inclusion of basically anything else for a month or 
two, and then Andre Hedrick has to mount a publicity campaign on linux-kernel 
to get any attention paid to his patches, and Eric's help patches get ignored 
for 33 consecutive releases.

Rik was replaced by Andrea as the VM maintainer, and Rik has publicly stated 
that he thinks you were dropping his VM patches for months at a time, while 
he was the maintainer and the VM was a subsystem definitely in need of 
patches.  Are you saying that the system was working well?  Are you saying 
that it was a one-time thing that is now resolved and won't recur?

Okay, maybe a lot of this is all miscommunication.  But that just identifies 
the TYPE of the problem, doesn't it?

> > Are you saying that Alan Cox's didn't serve a purpose during the 2.2
> > kernel time frame, and that Dave Jones is currently wasting his time?
>
> No, I'm saying that there are not very many peopel who can do it, and who
> can get the kind of trust that they are _everywhere_. Let's face it, Alan
> grew to be respected because he did lots of different stuff over many
> years, and he proved himself more than capable. And I suspect he's _quite_
> happy not being in the middle of it right now.. It's a tough job.

It is a tough job, and I understand that not everybody can be a good 
maintainer.  But currently at least Alan, Dave Jones, and Andrea are all 
maintaining their own public trees, from which they break out patches to send 
on to an "official" Linux tree.  (As for Alan not being "in the middle of 
it", he IS doing his tree again.  He's just doing it for 2.4.  He's basically 
being Marcelo's integration lieutenant.  Whatever he's burned out on, it's 
apparently not the job of maintaining a tree.  And he's doing it for Marcelo, 
whose architect role is largely rejecting as much as he possibly can since 
2.4 is not a development branch...)

You currently HAVE a de facto integration lieutenant, or else I totally 
misunderstand what Dave Jones is doing.  This is not a position for which 
applicants currently need to be interviewed, is it?  (Do you have a complaint 
with the job Dave is doing?)

> It's a lot more likely to find people who can maintain _parts_. And if
> there are patches that fall out of those parts, that tends to indicate a
> lack of modularity, and perhaps a lack of maintainer for those parts.

Sure.  But how do the maintainers piece together their code, resolve the 
obvious conflicts, and get the new stuff tested by live users in the field 
who want to live dangerously?  They USED to feed stuff into the -ac tree, 
months if not YEARS before you accepted (or rejected) it.  That's not my 
opinion or my recommendation, that's history.  I'm simply proposing that 
people consider the fact it might be an important and natural part of the 
process.  (When Alan stopped doing it, somebody else basically got shanghaied 
into doing it.)

> And more likely, even if you _do_ find ten people who can do everything,
> you don't want them to.

No, you want one guy with final responsibility for maintaining any tree.  
Committees produce mostly compromises and deadlocks.  That's why I proposed 
one guy for this job.  As I said, the CVS thing was a confusing side issue.  
(An easier way for the maintainers to do lower-friction merges with the 
integration maintainer, who would by the CVS administrator and would still 
have final say over what goes into his tree.)

But the -ac tree did not serve the same purpose as your tree did, and I was 
under the strong impression that the -ac tree DID serve a purpose.  (And, for 
Marcelo, is starting to do so again.)

There is currently no tree for provisionally integrating code.  Or for taking 
the flood of new driver patches that Alan Cox always fielded.  Not code from 
left field, but code like keith owens' new kbuild, CML2, or rik van riel's 
reverse mapping patches.  Things which have a strong possiblity of being 
integrated (two of the above you okayed at the kernel summit, one you've 
expressed interest in), and are ready for wider testing.

> 		Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Tue, Jan 29, 2002 at 02:33:24AM -0500, Rob Landley wrote:
> 
> I'm not proposing replacing the current subsystem maintainers.  But are the 
> current subsystem maintainers happy?

I'll speak up here as a subsystem maintainer and say that I'm happy with
the current situation.  I integrate a wide variety of USB driver patches
from lots of different people (and usually in lots of different formats
against different kernel trees) and feed them to Linus/Marcelo/Alan in
small chunks that can be easily applied against their latest kernel
version. 

Sure, sometimes my patches get dropped, but you forgot to mention the
most important thing a kernel programmer needs to have, persistence :)

thanks,

greg k-h

MAINTANIANCE [was Re: A modest proposal -- We need a patch penguin]

[James Simmons]

> I'm not proposing replacing the current subsystem maintainers.  But are the 
> current subsystem maintainers happy?

   Usually I don't get involved in these discussions but I like to share
my experiences here and I also like for people see it from the point of
view of Linus.  
   A few years ago I got involved with developement of the framebuffer
layer. Now I work along with Geert on the improvement of this subsystem. 
Do I consider myself a maintainer? Its just a title. All I know is I work
with Geert on this stuff. He has been doing that subsystem alot longer
than I have been and I have the up most high respect for him. So I would
never do anything without his okay. 
   One thing I can tell you is both me and Geert have jobs that don't pay
us to work directly on this (it would be nice HINT!!!!) . We do it out of
our free time. Unfortunely our free time is limited. I know I as well as
Geert receive email from various people on how to write drivers or even
recieve a bunch of patches. I have tried to look at every patch and read
every email. I can tell you it is so hard. Often I don't reply for days or
even weeks at a time. I feel bad sometimes about this but I can't help it.
BTW I don't ever ignore any emails. They might sit there for some time but
eventually I do answer them. Plus I do keep every patch I have been sent.
I even have been sent free hardware to test people's work on. I have
hardware I have been sitting around for several months ago but because I'm
so swamp I just don't get the time to test things on it. Another thing I
have experienced is having my head bite off by driver writers when I
reworked their code or told them this is the way the code should be
written. 
   As for Linus trusting or not trusting me. I hope he doesn't trust me
because I don't even trust myself sometimes. That is why I like to send
out my stuff on mailing list for people to see it. So people can voice
their options and I do see improvements. I'm thankful for trees like DJ
and alan's because I get to see a large audience test it. I have sent in
patches for the dj that worked for me and for a bunch of other people it 
blew up. I see it as a much needed test bed. Only when nearly everyone
that is affected by my work is happy I would consider it worthy to send to
Linus. 
   I can't even imagine being Linus. First we do see him getting flamed
for wanting things a certain way or reworking someone else code. This can
be good if done right. Second he recieves alot of patches every day. I
don't blame him for not looking at a lot of them. In fact I wouldn't doubt
he has a filter for the people he trust to go int one box and the others
go into another box. 
             
   What is the solution for this problem? With my expeirences I have come
up with I found the following to work best. Unfortunely not all have been
applied to fbdev developement but I'm pushing it this way. These are
the goals I'm aiming for.


----------------------- For new device drivers ---------------------------

I.   First we have setup a mailing list for this. He driver writers can
     post their drivers for code review and testing. 

II.  Second step is to place it in a area where people know here to go for
     the latest thing. For fbdev we have setup a CVS where people can ask
     for CVS access and can place their driver their. I like to do it this
     way because it is way to hard for me to track every patch. Plus we
     have had several people write the same driver independent of each
     other.

III. Place it in a beta tree. Announce it to the world.

IV.  The maintainer submits it to Linus. 

----------------------- API changes --------------------------------------

I.   Post a proposal on a mailing list for that subsystem.

II.  Everyone comments. Go to 1 until most people are happy.

III. Next a document maintiner steps forward to write real docs. (We
     haven't done this). I really think every subsystem needs this.

IV.  Start making patches and do above.
   
   . ---
   |o_o |
   |:_/ |   Give Micro$oft the Bird!!!!
  //   \ \  Use Linux!!!!
 (|     | )
 /'_   _/`\
 ___)=(___/

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 02:33:24AM -0500, Rob Landley wrote:
> I'm not proposing replacing the current subsystem maintainers.  But are the 
> current subsystem maintainers happy?

I think the system works, if you understand the system ;-)

Finding a maintainer for a piece of code is sometimes a jumble, but if
people want their patches in the kernel they need to be proactive about
it.

I'm -glad- Linus does not usually apply drop-n-run patches.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 10:00 PM 28/01/02 -0800, Linus Torvalds wrote:
>I think it helps a lot to have people pick up patches that nobody else
>wants to maintain, and to gather them up. Andrea does that to some degree.
>But it is _much_ better if you have somebody who is a point-man for
>specific areas.
>
>The problem with an overall guy is that there can't be too many of them.
>The very thing you are _complaining_ about is in fact that there are a
>number of over-all guys without clear focus, which only leads to confusion
>about who handles what.
>
>Clarity is good.

Perhaps what we need is a patch maintenance system? Now I'm not talking 
CVS, I'm talking about something that is, in reality, pretty simple. 
Something that does the following:

  1. People can submit patches, which are given a unique patch ID.
  2. Notifications of patches are passed on to (from a selection or 
automatic detection):
    a. A module maintainer
    b. A section maintainer
    c. A tree maintainer
    d. Linus Torvalds
  3. The patches can be reviewed, and immediately:
    a. Dropped with a canned reject message
    b. Dropped with a custom reject message
    c. Dropped but archived for later review
    d. Suspended/Skipped for later review
    e. Redelegated up/down to a maintainer
    f. Accepted, pending testing
  4. If someone wants to know why their patch is not being accepted:
    a. They can easily look up the current status
    b. There is a common reference place
    c. If their patch is rejected, they can ask for more detail

Yes it's a tiny patch tracking system. The idea is that if we can make the 
thing simple to use, simple to understand, and simple to maintain, people 
will actually use it. Submitting patches by mail in freeform, while 
reasonably simple, leads to overall complication, and may take longer to 
sort through.

It doesn't have to be that involved. A shell script (or perl) could happily 
do the submission job (which could all still be done by mail, however in a 
more formatted approach). It'd also be simple to do some sort of web 
interface if anyone was actually inclined.

Later, maybe, we can have it track possible conflicts (between waiting 
patches) and flag them as such so the maintainer is aware of the fact.

If we can automate some simple, key parts of the kernel maintenance, then 
this could help immensely, and let everyone get on with the job than the 
hassle. Maybe we want to re-invent this wheel, maybe we don't.

PS: Remember this is only a suggestion. I'm not infallible (is anyone?), 
all I can do is give the big guy ideas (even bad ones), so he can make 
better decisions. *grin*


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[Nix N. Nix]

On Tue, 2002-01-29 at 02:10, Stuart Young wrote:
> At 10:00 PM 28/01/02 -0800, Linus Torvalds wrote:
> >I think it helps a lot to have people pick up patches that nobody else
> >wants to maintain, and to gather them up. Andrea does that to some degree.
> >But it is _much_ better if you have somebody who is a point-man for
> >specific areas.
> >
> >The problem with an overall guy is that there can't be too many of them.
> >The very thing you are _complaining_ about is in fact that there are a
> >number of over-all guys without clear focus, which only leads to confusion
> >about who handles what.
> >
> >Clarity is good.
> 
> Perhaps what we need is a patch maintenance system? Now I'm not talking 
> CVS, I'm talking about something that is, in reality, pretty simple. 
> Something that does the following:
> 
>   1. People can submit patches, which are given a unique patch ID.
>   2. Notifications of patches are passed on to (from a selection or 
> automatic detection):
>     a. A module maintainer
>     b. A section maintainer
>     c. A tree maintainer
>     d. Linus Torvalds
>   3. The patches can be reviewed, and immediately:
>     a. Dropped with a canned reject message
>     b. Dropped with a custom reject message
>     c. Dropped but archived for later review
>     d. Suspended/Skipped for later review
>     e. Redelegated up/down to a maintainer
>     f. Accepted, pending testing
>   4. If someone wants to know why their patch is not being accepted:
>     a. They can easily look up the current status
>     b. There is a common reference place
>     c. If their patch is rejected, they can ask for more detail

Kinda like bugzilla, but for patches ? Am I off the deep end here ?


Just a thought.
> 
> Yes it's a tiny patch tracking system. The idea is that if we can make the 
> thing simple to use, simple to understand, and simple to maintain, people 
> will actually use it. Submitting patches by mail in freeform, while 
> reasonably simple, leads to overall complication, and may take longer to 
> sort through.
> 
> It doesn't have to be that involved. A shell script (or perl) could happily 
> do the submission job (which could all still be done by mail, however in a 
> more formatted approach). It'd also be simple to do some sort of web 
> interface if anyone was actually inclined.
> 
> Later, maybe, we can have it track possible conflicts (between waiting 
> patches) and flag them as such so the maintainer is aware of the fact.
> 
> If we can automate some simple, key parts of the kernel maintenance, then 
> this could help immensely, and let everyone get on with the job than the 
> hassle. Maybe we want to re-invent this wheel, maybe we don't.
> 
> PS: Remember this is only a suggestion. I'm not infallible (is anyone?), 
> all I can do is give the big guy ideas (even bad ones), so he can make 
> better decisions. *grin*
> 
> 
> Stuart Young - sgy@amc.com.au
> (aka Cefiar) - cefiar1@optushome.com.au
> 
> [All opinions expressed in the above message are my]
> [own and not necessarily the views of my employer..]
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: A modest proposal -- We need a patch penguin

[Patrick Mochel]

On Tue, 29 Jan 2002, Stuart Young wrote:
> Perhaps what we need is a patch maintenance system? Now I'm not talking 
> CVS, I'm talking about something that is, in reality, pretty simple. 
> Something that does the following:

Ah, the old Patch Management Problem. It's like an old friend (or a bad 
rash). 

AFAIK, something like this was first proposed here:

http://lists.insecure.org/linux-kernel/2000/Sep/2468.html

At that time, we were in the midst of the 2.4.0-test? series. Many things 
were unstable and/or volatile. Linus was receiving an ungodly number of 
patches, and releasing a new -pre patch about every day. 

One of the main problems was that many patches simply didn't apply. What 
a patch was diffed against would become obsolete so quickly, that many 
patches were rendered useless by the time they were even read. And, there 
was the same limitation concerning Linus's ability and desire to reply to 
every single email. 

The concept is very simple and described well in the email. So, I will not 
expound on it here. Unfortunately, the project was dropped internally. 

The problem has come up a few times in the last few months. Several people 
have expressed interest in having something like it. Some already do. 
Several people have said they were working on something like it. 
Unfortunately, I think most of those people got distracted with their 
other full-time jobs or more intersting work.

I brought the topic up here at OSDL a few months ago for use both 
internally and externally. Also, with the notion of integrating our STP 
(Scalable Test Platform). We've had several discussions about it, what it 
would look like, and how it would work. We've also had the chance to talk 
with a few of the other kernel maintainers in the area (face-to-face 
meetings really do a long way).

The conclusions were this:

Is it necessary?	No.
Could it be useful?	Yes.
Would people use it?	Probably. 
Would everyone use it?	No.
Would Linus use it?	Probably Not. 


Which is all pretty obvious. You can't please everyone.

We're going to develop a system internally and are willing to host a 
system for the rest of the world to use. We're not looking to design a 
be-all, end-all solution. Basically, just a system that can automate 
things like applying patches and compiling. 

If a patch succeeds, it then goes to the maintainer to which it was sent. 
The maintainer can then accept or reject the patch. Either with 
explanation or not. The submitter can then track what patches were 
accepted. 

Submitted patches can also go to a public mailing list and/or exported via 
a public (read: web) interface. Of course, there should be ways to 
override the publicity for OOB patches and sensitive items. 

Writing the software is really not that difficult. But, we want something 
that people like and can use, as well as modular and extensible. So, we're 
aiming for simplicity and modularity. 

So, the obvious question is 'So, where is it at now?'. Not much further 
than the conceptualizing stage. The two people actually writing the 
software are a bit over-subscribed ATM, though we do have some pretty 
pictures. 

I'm currently in the waiting queue for a Sourceforge project. Once that is 
live, there will be a mailing list to which the discussion can be moved 
and kept alive. Anyone and everyone interested is welcome to submit their 
ideas and suggestions. Via the SF project, we will submit our designs and 
post our progress on the system. 

If you prefer a more private forum, feel free to email me and/or the Man 
with the Plan: Nathan Dabney at smurf@osdl.org.

	-pat

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

Apropos of nothing in particular:

> (It's not a star topology, it's a tree.)

There is no difference between a star and a tree, except how you draw the 
picture.

-- 
Daniel

RE: A modest proposal -- We need a patch penguin

[George Bonser]

I dunno. I tend to agree with Linus from a management standpoint.
You delegate responsibility to those you trust and accept their
judgment. You also must try to ensure that the thing is built so
that decisions made by Bill can't step on Bob's work.  Delegation
is one thing ... but it is the architecture that makes sure that
delegation is possible. I am not a kernel hacker but don't feel
that I have to be in this case to comment. What Linus is saying
has nothing at all to do with Linux but is more along the lines
of how to get work done in any kind of project that is larger than
any single person.

While all kinds of software might get written for project management
and revision control, projects boil down to how the thing is built,
the style of the person in charge, and the people around that person.

Finding people that are willing to take responsibility for portions
of the code base and develop a relationship of trust over time while
being able to work with the individuals submitting patches for their
bug fixes/ideas is a big task.  If you could gather all of
those people together, I will bet you they will be a successful team
in just about ANYTHING they choose to work on. I think what Linus is
striving toward is what any good manager wants and when such a team
comes together, watch out, things are gonna happen fast.

Don't think of it as a star or a tree, think of it as a star of trees.



> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of
> Daniel Phillips
> Sent: Monday, January 28, 2002 11:39 PM
> To: Rob Landley; linux-kernel@vger.kernel.org
> Subject: Re: A modest proposal -- We need a patch penguin
>
>
> Apropos of nothing in particular:
>
> > (It's not a star topology, it's a tree.)
>
> There is no difference between a star and a tree, except
> how you draw the
> picture.
>
> --
> Daniel
> -
> To unsubscribe from this list: send the line "unsubscribe
> Linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: A modest proposal -- We need a patch penguin

[Xavier Bestel]

le mar 29-01-2002 à 08:38, Daniel Phillips a écrit :
> Apropos of nothing in particular:
> 
> > (It's not a star topology, it's a tree.)
> 
> There is no difference between a star and a tree, except how you draw the 
> picture.

I think a star is a tree only 1 unit deep. No subchildren. (In this
case)

	Xav

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Mon, 28 Jan 2002, Rob Landley wrote:

> (You keep complaining people never send you patches.  People are
> suggesting automated patch remailers to spam your mailbox even harder.
> There has GOT to be a better way...)

None of the examples you cited so far are convincing to me, and i'd like
to explain why. I've created and submitted thousands of patches to the
Linux kernel over the past 4 years (my patch archive doesnt go back more
than 4 years):

  # ls patches | wc -l
     2818

a fair percentage of those went to Linus as well, and while having seen
some of them rejected does hurt mentally, i couldnt list one reject from
Linus that i wouldnt reject *today*. But i sure remember being frustrated
about rejects when they happened. In any case, i have some experience in
submitting patches and i'm maintaining a few subsystems, so here's my take
on the 'patch penguin' issue:

If a patch gets ignored 33 times in a row then perhaps the person doing
the patch should first think really hard about the following 4 issues:

  - cleanliness
  - concept
  - timing
  - testing

a violation of any of these items can cause patch to be dropped *without
notice*. Face it, it's not Linus' task to teach people how to code or how
to write correct patches. Sure, he still does teach people most of the
time, but you cannot *expect* him to be able to do it 100% of the time.

1) cleanliness

code cleanliness is a well-know issue, see Documentation/CodingStyle.  If
a patch has such problems then maintainers are very likely to help - Linus
probably wont and shouldnt. I'm truly shocked sometimes, how many active
and experienced kernel developers do not follow these guidelines. While
the Linux coding style might be arbitrary in places, all coding styles are
arbitrary in some areas, and only one thing is important above all:
consistency between kernel subsystems. If i go from one kernel subsystem
to another then i'd like to have the same 'look and feel' of source code -
i think this is a natural desire we all share. If anyone doesnt see the
importance of this issue then i'd say he hasnt seen, hacked and maintained
enough kernel code yet. I'd say the absolute positive example here is Al
Viro. I think most people just do not realize the huge amount of
background cleanup work Al did in the past 2 years. And guess what? I bet
Linus would be willing to apply Al's next patch blindfolded.

impact: a patch penguin might help here - but he probably wont scale as
well as the current set of experienced kernel hackers scale, many of whom
are happy to review patches for code cleanliness (and other) issues.

2) concept

many of the patches which were rejected for a long time are *difficult*
issues. And one thing many patch submitters miss: even if the concept of
the patch is correct, you first have to start by cleaning up *old* code,
see issue 1). Your patch is not worth a dime if you leave in old cruft, or
if the combination of old cruft and your new code is confusing. Also, make
sure the patch is discussed and developed openly, not on some obscure
list. linux-kernel@vger.kernel.org will do most of the time. I do not want
to name specific patches that violate this point (doing that in public
just offends people needlessly - and i could just as well list some of my
older patches), but i could list 5 popular patches immediately.

impact: a patch penguin just wont solve this concept issue, because, by
definition, he doesnt deal with design issues. And most of the big patch
rejections happen due to exactly these concept issues.

3) timing

kernel source code just cannot go through arbitrary transitions. Eg. right
now the scheduler is being cleaned up (so far it was more than 50
sub-patches and they are still coming) - and work is going on to maximize
the quality of the preemption patch, but until the base scheduler has
stabilized there is just no point in applying the preemption patch - no
matter how good the preemption patch is. Robert understands this very
much. Many other people do not.

impact: a patch penguin just wont solve this issue, because a patch
penguin cannot let his tree transition arbitrarily either. Only separately
maintained and tested patches/trees can handle this issue.

4) testing

there are code areas and methods which need more rigorous testing and
third-party feedback - no matter how good the patch. Most notably, if a
patch exports some new user-space visible interface, then this item
applies. An example is the aio patch, which had all 3 items right but was
rejected due to this item. [things are improving very well on the aio
front so i think this will change in the near future.]

impact: a patch penguin just wont solve this issue, because his job, by
definition, is not to keep patches around indefinitely, but to filter them
to Linus. Only separately maintained patches/trees help here. More people
are willing to maintain separate trees is good (-dj, -ac, -aa, etc.), one
tree can do a nontrivial transition at a time, and by having more of them
we can eg. get one of them testing aio, the other one testing some other
big change. A single patch penguin will be able to do only one nontrivial
transition - and it's not his task to do nontrivial transitions to begin
with.


Many people who dont actually maintain any Linux code are quoting Rik's
complains as an example. I'll now have to go on record disagreeing with
Rik humbly, i believe he has done a number of patch-management mistakes
during his earlier VM development, and i strongly believe the reason why
Linus ignored some of his patches were due to these issues. Rik's flames
against Linus are understandable but are just that: flames. Fortunately
Rik has learned meanwhile (we all do) and his rmap patches are IMHO
top-notch. Joining the Andrea improvements and Rik's tree could provide a
truly fantastic VM. [i'm not going to say anything about the IDE patches
situation because while i believe Rik understands public criticism, i
failed to have an impact on Andre before :-) ]

also, many people just start off with a single big patch. That just doesnt
work and you'll likely violate one of the 4 items without even noticing
it. Start small, because for small patches people will have the few
minutes needed to teach you. The bigger a patch, the harder it is to
review it, and the less likely it happens. Also, if a few or your patches
have gone into the Linux tree that does not mean you are senior kernel
hacker and can start off writing the one big, multi-megabyte super-feature
you dreamt about for years. Start small and increase the complexity of
your patches slowly - and perhaps later on you'll notice that that
super-feature isnt all that super anymore. People also underestimate the
kind of complexity explosion that occurs if a large patch is created.
Instead of 1-2 places, you can create 100-200 problems.

face it, most of the patches rejected by Linus are not due to overload. He
doesnt guarantee to say why he rejects patches - *and he must not*. Just
knowing that your patch got rejected and thinking it all over again often
helps finding problems that Linus missed first time around. If you submit
to Linus then you better know exactly what you do.

if you are uncertain about why a patch got rejected, then shake off your
frustration and ask *others*. Many kernel developers, including myself,
are happy to help reviewing patches. But people do have egos, and it
happens very rarely that people ask it on public lists why their patches
got rejected, because people do not like talking about failures. And the
human nature makes it much easier to attack than to talk about failures.
Which fact alone pretty much shows that most of the time the problem is
with the patch submitter, not with Linus.

it's so much easier to blame Linus, or maintainers. It's so much easier to
fire off an email flaming Linus and getting off the steam than to actually
accept the possibility of mistake and *fix* the patch. I'll go on record
saying that good patches are not ignored, even these days when the number
of active kernel hackers has multipled. People might have to go through
several layers first, and finding some kernel hacker who is not as loaded
as Linus to review your patch might be necessery as well (especially if
the patch is complex), but if you go through the right layers then you can
be sure that nothing worthwile gets rejected arbitrarily.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 02:54 pm, Ingo Molnar wrote:
> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:
> 
>   - cleanliness
>   - concept
>   - timing
>   - testing
> 
> a violation of any of these items can cause patch to be dropped *without
> notice*. Face it, it's not Linus' task to teach people how to code or how
> to write correct patches. Sure, he still does teach people most of the
> time, but you cannot *expect* him to be able to do it 100% of the time.

While I agree in general with most of your remarks, I think you're being a 
little too glib here.  Consider my patch to fix group descriptor corruption 
in Ext2, submitted half a dozen times to Linus and other maintainers over the 
course of two years, which was clearly explained, passed scrutiny on 
ext2-devel and lkml, fixed a real problem that really bit people and which 
I'd been running myself over the entire period.  Which one of cleanliness, 
concept, timing or testing did I violate?

If the answer is 'none of the above', then what is wrong with this picture?

--
Daniel

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> [...] Consider my patch to fix group descriptor corruption in Ext2,
> submitted half a dozen times to Linus and other maintainers over the
> course of two years, which was clearly explained, passed scrutiny on
> ext2-devel and lkml, fixed a real problem that really bit people and
> which I'd been running myself over the entire period.  Which one of
> cleanliness, concept, timing or testing did I violate?
>
> If the answer is 'none of the above', then what is wrong with this
> picture?

am i correct that you are referring to this patch?:

   http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html

was this the first iteration of your patch? Your mail is a little more
than 1 year old. You rated the patch as: 'The fix below is kind of
gross.'. Clearly, this does not help getting patches applied.

the ext2 bh-handling code had cleanliness issues before. I had ext2
patches rejected by Linus because they kept the method of passing around
double-pointers, and i have to agree that the code was far from clean. Al
did lots of cleanups in this area, and i think he fixed this issue as
well, didnt he? So where is the problem exactly, does 2.4 still have this
bug?

in terms of 2.2 and 2.0, you should contact the respective maintainers.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Ville Herva]

On Tue, Jan 29, 2002 at 03:52:20PM +0100, you [Ingo Molnar] claimed:
> 
>    http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html

(...)
 
> does 2.4 still have this bug?

My understanding is that Al did fix it.

> in terms of 2.2 and 2.0, you should contact the respective maintainers.

It has been submitted now. David Weinehall merged it in 2.0.40-rc1 and I
understand that it's in Alan's 2.2.21pre queue as well.


-- v --

v@iki.fi

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 03:52 pm, Ingo Molnar wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> 
> > [...] Consider my patch to fix group descriptor corruption in Ext2,
> > submitted half a dozen times to Linus and other maintainers over the
> > course of two years, which was clearly explained, passed scrutiny on
> > ext2-devel and lkml, fixed a real problem that really bit people and
> > which I'd been running myself over the entire period.  Which one of
> > cleanliness, concept, timing or testing did I violate?
> >
> > If the answer is 'none of the above', then what is wrong with this
> > picture?
> 
> am i correct that you are referring to this patch?:
> 
>    http://www.uwsg.iu.edu/hypermail/linux/kernel/0011.3/0861.html
> 
> was this the first iteration of your patch? Your mail is a little more
> than 1 year old.

No, there are versions before that.  The first version, which really was 
inadequate because I didn't know about diff -u at the time (my first patch) 
is about 23 months old.

> You rated the patch as: 'The fix below is kind of
> gross.'. Clearly, this does not help getting patches applied.

Note who the email is addressed to.  I have tried many different techniques 
for communicating with this gentleman, including self-deprecation, and they 
all seem to have the same result: no patch applied, long wait, eventually 
some other patch a long time later will obsolete my patch in some way, and 
the whole thing drifts off into forgotten history.  Err, almost forgotten, 
because the bad taste remains.

And yes, there was a successor to the patch in which I did the job 'properly' 
by cleaning up some other infrastructure instead of just fixing the bug 
locally.  There was also a long lag after I created and submitted that 
version before the bug was actually fixed, and then it was only fixed in 2.4.

All of this only 'since you asked'.  I'd prefer not to dwell on it further, 
but as you could imagine, this story would not have developed this way if we 
have even a minimal form of patch tracking.  At least the bugs would have 
been fixed in all trees, nearly two years earlier.

> the ext2 bh-handling code had cleanliness issues before. I had ext2
> patches rejected by Linus because they kept the method of passing around
> double-pointers, and i have to agree that the code was far from clean.

Exactly.  The successor patch to the 'kind of gross' patch got rid of the 
double-pointers, it was the proper fix, though there is still no excuse for 
leaving the bug hanging around while coming up with the better version.

> Al did lots of cleanups in this area, and i think he fixed this issue as
> well, didnt he? So where is the problem exactly, does 2.4 still have this
> bug?

Oh yes, there are a few problems with what happened:

  - It left the bug circulating out in the wild far longer than
    necessary, and it bit people, pissing them off, especially when
    they figured out there was a patch not applied.

  - While it got fixed in the 2.4 tree, it didn't get fixed in 2.2 or
    for all I know, 2.0.

  - It pissed me off.

> in terms of 2.2 and 2.0, you should contact the respective maintainers.

This was taken care of by a good samaritan:

   http://marc.theaimsgroup.com/?l=linux-kernel&m=100989249313641&w=2

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Andrew Morton]

Daniel Phillips wrote:
> 
> Note who the email is addressed to.  I have tried many different techniques
> for communicating with this gentleman, including self-deprecation, and they
> all seem to have the same result: no patch applied, long wait, eventually
> some other patch a long time later will obsolete my patch in some way, and
> the whole thing drifts off into forgotten history.  Err, almost forgotten,
> because the bad taste remains.
> 
> And yes, there was a successor to the patch in which I did the job 'properly'
> by cleaning up some other infrastructure instead of just fixing the bug
> locally.  There was also a long lag after I created and submitted that
> version before the bug was actually fixed, and then it was only fixed in 2.4.
> 

When all this ext2 fuss was going on, I went into ext3, changed
a few lines, fixed the bug and typed `cvs commit'.

I just thought you might enjoy hearing that.

-

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> Note who the email is addressed to.  I have tried many different techniques 
> for communicating with this gentleman, including self-deprecation, and they 
> all seem to have the same result

Trying a bit of intellectual honesty would help big way.

Realizing that ext2 patches should be sent to ext2 maintainers would help
even more.

You've spent _months_ ignoring the idea above.  You've tried many different
techniques for what, exactly?  To push that stuff to a guy who is not, was not
and had never been maintainer of the code in question?  Wow.

And yes, it had been told to you from the very beginning.  tytso, sct and akpm
are the right guys for such stuff.  It's their code, they do maintain it
and I think in all cases I've sent ext2 patches it was only after ACK from
ext2 folks.

If it took you a fscking year to realize that, despite having it explained to
you in details...  Don't you feel yourself an idiot?

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 05:37 am, Alexander Viro wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> > Note who the email is addressed to.  I have tried many different techniques 
> > for communicating with this gentleman, including self-deprecation, and they 
> > all seem to have the same result
> 
> Trying a bit of intellectual honesty would help big way.

I've been entirely straightforward and honest.  If I were intellectually
dishonest, I would smile and take the crap from you, as others do  But that
is not me as you know, and I suppose that is why you let your venom out.
(And don't say you don't, I have irc logs enough to prove that point.)

By the way, do you think that your constant dissing of me, typically
behind my back, makes people respect you more?

> Realizing that ext2 patches should be sent to ext2 maintainers would help
> even more.
> 
> You've spent _months_ ignoring the idea above.  You've tried many different
> techniques for what, exactly?  To push that stuff to a guy who is not, was not
> and had never been maintainer of the code in question?  Wow.

Linus just called you the ext2 maintainer.  If you do not consider yourself
to be the ext2 maintainer, when was the last time you submitted a patch
through Ted?

In any event, a reasonable patch was submitted to Ted:

   http://marc.theaimsgroup.com/?l=ext2-devel&m=99039802717798&w=2

with zero results one way or the other, probably because Ted, who hadn't
been seen on the ext2-devel list for some time at that point, was up to his
ears in something else.  A version of the patch was forwarded to you at
that time, and you also subscribe to ext2-devel, so you knew the whole
store, including the fact that Andrew had signed of on it:

  http://marc.theaimsgroup.com/?l=ext2-devel&m=99054430703022&w=2

Now, you could say that at this point the ball was in my/Ted's court, and
you'd be right, except for the fact that there was yet another go around on
it, just before 2.5 opened, when Alan for some reason wanted to wait for 2.5
to open, and again just after 2.5 opened, when I offered the patch again and
you refused it because you planned to obsolete it.

I found the whole story fairly distasteful, and even so, I would have
forgotten about it if Villa Herva had not noticed that I was being jerked
around, and brought it to the attention of the community.  By the way, I
had nothing to do with this, I'd never heard of him before he made his
post.

A similar story was played out with the fs.h cleanups.  I'm unhappy with
the way you handled that, as well.

> And yes, it had been told to you from the very beginning.  tytso, sct and akpm
> are the right guys for such stuff.  It's their code, they do maintain it
> and I think in all cases I've sent ext2 patches it was only after ACK from
> ext2 folks.
>
> If it took you a fscking year to realize that, despite having it explained to
> you in details...  Don't you feel yourself an idiot?

No I do not, and it is precisely that kind of remark that makes you hard
or impossible to get along with.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

Calm down guys.  Al, Daniel, peace.

It was fun while people were just ragging on me (it tends to happen about
every 6 months or so, and I have a thick skin - and every time it happens
some problems do get unearthed and that's fine), but let's not make this
degenerate into a real hate-fest..

Shake hands, please.

-- tangential --

One thing intrigued me in this thread - which was not the discussion
itself, but the fact that Rik is using bitkeeper.

How many other people are actually using bitkeeper already for the kernel?
I know the ppc guys have, for a long time, but who else is? bk, unlike
CVS, should at least be _able_ to handle a "network of people" kind of
approach.

		Linus

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> 
> How many other people are actually using bitkeeper already for the kernel?

I am, for the USB and PCI hotplug stuff:
	http://linuxusb.bkbits.net/

It makes tracking what patches got applied, and which didn't, and
forward porting those that didn't to the next release, a breeze.

My trees are world readable, and people are welcome to send me patches
against it, or even bitkeeper changesets, but I have yet to receive one
of those yet :)

thanks,

greg k-h

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 02:48 am, Linus Torvalds wrote:

> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
>
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

One thing that's intrigued ME is the explanation of the hierarchy of 
maintainers.  There ARE specific people that patches should be reviewed by 
before being sent to you, there even seems to be a directed graph of them.

It would be kind of nice if it was documented enough that at least the 
maintainers in the maintainers list knew what it was, and who they should 
forward stuff on to after reviewing it.  That might go a ways towards 
addressing the "hitting resend isn't working" problem...

You've said that the tier under you (who you DO semi-reliably accept patches 
from) is a group of ten to twenty people.  If we knew who those people were, 
we could bug them to name THEIR secretary lists (or figure it out from the 
maintainers list)...

In your original response to the patch penguin proposal, you mentioned:

>The fact is, we've had "patch penguins" pretty much forever, and they
>are called subsystem maintainers.  They maintain their own subsystem, ie
>people like David Miller (networking), Kai Germaschewski (ISDN), Greg KH
>(USB), Ben Collins (firewire), Al Viro (VFS), Andrew Morton (ext3), Ingo
>Molnar (scheduler), Jeff Garzik (network drivers) etc etc. 

You also said:

> The VM stuff right now seems to be Andrea, Dave or you yourself.

That was responding to Rik van Riel, I'm guessing "Dave" is Dave Jones(?), 
and of course Andrea would be Andrea Arcangeli.  It seems that Andrea 
Arcangeli is the default VM maintainer, and that Dave Jones is gradually 
getting sucked into Alan Cox's old position as "miscelaneous maintainer" 
putting out a "this needs wider testing" tree.

The above seems to be about the full list I can assemble from recent emails.  
(You've also used David Miller again as an example in a later email, you put  
Paul Mackerras as a subordinate maintainer under him, and "Greg" 
(Kroah-Hartmann) had Johannes Erdfelt under him handling UHCI.  This isn't 
really new information about the top ten, more like some examples to help in 
tree building under them.)

This is eleven "top level" maintainers, one of whom is handling ext3 which 
sounds kind of odd...  (If David Miller is networking and Jeff Garzik is 
network drivers, would there be a "filesystem drivers" guy paired off with Al 
Viro?  Does EXT2 go through Andrew Morton as well?  Would Hans Reiser submit 
directly to you for ReiserFS patches, or should he get a signoff from...  
Um...  Andrew?  Al?  Try to get it into the -dj tree first?  Could I have a 
hint?)

To clarify what I'm aiming at: Are these eleven people a significant portion 
of the group of people who, if code makes it as far as them and they sign off 
on it, you'd then be willing to at least review it and if necessary 
explicitly reject? [1]  Should some of them be forwarding their patches to 
somebody other than you?  Are there more people on the list that lower level 
maintainers should be funneling patches to in order to eventually get them 
into your tree?

A two tier maintainer system definitely sounds like an improvement if that 
will help the process scale.  It's just that today is the first I've heard 
about it, and I had TRIED to study the situation before opening my big 
mouth...

> 		Linus

Rob

[Footnote 1] Implicit rejections can be REALLY stressful when combined with 
delaying the of inclusion of code that isn't actually rejected, but just not 
convenient to include right now.  It means that code that isn't merged 
immediately soon starts to smell of failure.  The throught process seems to 
go "If Linus hasn't accepted it, and Linus ignores patches he's rejecting, 
maybe he's rejecting this.  If so, the reason is something we need to figure 
out on our own, so let's all pile on the code and start badmouthing it until 
we figure out why Linus doesn't like it."  This can easily go beyond useful 
code review into pointless flame wars.  Arranging a system where it's 
possible to have some kind of progress indicator (even a "distance from 
Linus" index as patches progress through the maintainer tree) seems like a 
good thing to me...

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Rob Landley wrote:

>
>
>This is eleven "top level" maintainers, one of whom is handling ext3 which 
>sounds kind of odd...  (If David Miller is networking and Jeff Garzik is 
>network drivers, would there be a "filesystem drivers" guy paired off with Al 
>Viro?  Does EXT2 go through Andrew Morton as well?  Would Hans Reiser submit 
>directly to you for ReiserFS patches, or should he get a signoff from...  
>Um...  Andrew?  Al?  Try to get it into the -dj tree first?  Could I have a 
>hint?)
>

There is a maintainers list somewhere in the kernel tree.  I am listed 
there as  the ReiserFS maintainer, and I send our patches directly to 
Linus and Marcelo.  I don't think that a filesystems maintainer would be 
easily achieved, since if we agreed about architecture we would have 
written the same filesystems.   We can't even agree about whether 
streams and extended attributes should be implemented as files, and as 
for whether keyword search and database functionality should go into the 
filesystem namespace.....

So, there is a maintainers list, and for many subsystems it works fairly 
well.  For ReiserFS, while I review and approve all patches we accept, 
Oleg Drokin is my patch whirlwind who does the work of testing and 
inspecting line by line for bugs (I inspect more for desirable 
functionality).

ReiserFS has been well-tended by Marcelo, so things are working well for 
us.  Dave Jones tends to 2.5 ReiserFS patches quite nicely also.  None 
of our 2.5 patches are earth-shattering, so I think it is very 
reasonable for Linus to pay attention to, say,  bio stuff for now rather 
than our patches (I am sure he will eventually fold them in from Dave 
Jones's tree.)  I worry more that I haven't had a few hours to brief 
Linus on the strategic direction of Reiser4, and what I think is needed 
longterm to compete with Longhorn, but this is probably my fault for not 
asking him for it.

I know that others have had real problems in this regard, and I don't 
discount them, but ReiserFS patches are going well at this time.

Hans

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

I gave up on CVS for internal stuff with the -ac patches. I ended up keeping
a running patch and a complete archive of the submissions. The archive so
I can ensure info gets back and forth neatly.

(tangientially further)

Larry - can bitkeeper easily be persuaded to take "messages" back all the way 
to the true originator of a change. Ie if a diff gets to Linus he can reject
a given piece of change and without passing messages back down the chain
ensure they get the reply as to why it was rejected, and even if
nobody filled anything in that it was looked at and rejected by xyz at
time/date ?

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 10:14:49AM +0000, Alan Cox wrote:
> Larry - can bitkeeper easily be persuaded to take "messages" back all the way 
> to the true originator of a change. Ie if a diff gets to Linus he can reject
> a given piece of change and without passing messages back down the chain
> ensure they get the reply as to why it was rejected, and even if
> nobody filled anything in that it was looked at and rejected by xyz at
> time/date ?

It's certainly possible and there are changes we could make to make it more
useful.  Right now, there is no record of a change if it goes and then gets
rejected right back out; it's as if you patched and then you did a reverse
patch.

The good news is that each change (patch) has an identifier, they look like

    awc@bitmover.bitmover.com|ChangeSet|20011230212716|39200

and if we kept a record of those that were rejected, it would be trivial for
a developer to track whether his change was in/not seen/rejected.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
 
> -- tangential --
> 
> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
> 
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

It does in some ways anyhow.  Following things downstream is rather
painless, but one of the things we in the PPC tree hit alot is when we
have a new file in one of the sub trees and want to move it up to the
'stable' tree, or when it shows up in your/marcelo's tree.  bk send only
works for same base tree type things (ie a clone of tree X, some
changes, not a clone of tree Y, which was a clone of tree X but has lots
of changes and has tree X changes pulled in frequently).  Unfortunaly I
don't think this is an easy problem to work on either.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> It does in some ways anyhow.  Following things downstream is rather
> painless, but one of the things we in the PPC tree hit alot is when we
> have a new file in one of the sub trees and want to move it up to the
> 'stable' tree

Summary: only an issue because Linus isn't using BK.

Details:

The source of this problem is that there is no Linux/BK tree.  To 
understand, read on.  The PPC team "tracks" the kernel using BK.
There is a paper with pictures describing what they do at 

	http://www.bitkeeper.com/tracking.ps

To do this, they have a BK repository which is the "pristine" source,
i.e., it has nothing but released code from Linus.  They import new work
into that repository using "bk import -tpatch" which takes traditional
patches.

There is a "child" repository which is a copy of the "pristine".  The
child repository is used to hold not only the pristine work but also
all the work from the PPC team.  Think of it as "Linux+PPC".

To get patches back to Linus, the PPC maintainer (used to be Cort, now
Paul) will export changes as a traditional patch from the "Linux+PPC"
repository and send them to Linus.  These changes, sometimes modified,
make their way back to the PPC team through the "pristine" tree.  That's
the source of the problem.  

So the work flow is

	patches -> pristine
		      v
		      v
		   Linux+PPC -> patches to Linus

The problem is when a file is created in the Linux+PPC tree, sent to Linus,
comes back as a patch, is imported in pristine, and then is sent to 
Linux+PPC.

BitKeeper tracks files just like a file system, by a BK form of an inode.
That file that came in as a patch is a different inode, what is logically
the same file was created twice.  Much like if you created foo and bar
and then said "mv foo bar", BitKeeper will complain at you that the file
exists already.

This problem goes away if the PPC team could send Linus BK patches and 
Linus sent out his changes as BK patches.  It doesn't require a wholesale
switch to BK, Linus can still take traditional patches and send them out,
but if he maintained a BK tree as well, the problem Troy described goes
away.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > It does in some ways anyhow.  Following things downstream is rather
> > painless, but one of the things we in the PPC tree hit alot is when we
> > have a new file in one of the sub trees and want to move it up to the
> > 'stable' tree
> 
> Summary: only an issue because Linus isn't using BK.

Then how do we do this in the bk trees period?  To give a concrete
example, I want to move arch/ppc/platforms/prpmc750_setup.c from
2_4_devel into 2_4, without loosing history.  How?  And just this file
and not all of _devel.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> > 
> > Summary: only an issue because Linus isn't using BK.
> 
> Then how do we do this in the bk trees period?  To give a concrete
> example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> 2_4_devel into 2_4, without loosing history.  How?  And just this file
> and not all of _devel.

That question doesn't parse.  There are multiple ways you can do it but 
once you do patches will no longer import cleanly from Linus.  The whole
point of the pristine tree is to give yourself a tree into which you can
import Linus patches.  If you start putting extra stuff in there you will
get patch rejects.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:11:34AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> > On Wed, Jan 30, 2002 at 08:03:08AM -0800, Larry McVoy wrote:
> > > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > > It does in some ways anyhow.  Following things downstream is rather
> > > > painless, but one of the things we in the PPC tree hit alot is when we
> > > > have a new file in one of the sub trees and want to move it up to the
> > > > 'stable' tree
> > > 
> > > Summary: only an issue because Linus isn't using BK.
> > 
> > Then how do we do this in the bk trees period?  To give a concrete
> > example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> > 2_4_devel into 2_4, without loosing history.  How?  And just this file
> > and not all of _devel.
> 
> That question doesn't parse.  There are multiple ways you can do it but 
> once you do patches will no longer import cleanly from Linus.  The whole
> point of the pristine tree is to give yourself a tree into which you can
> import Linus patches.  If you start putting extra stuff in there you will
> get patch rejects.

Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
again.  One of the problems we hit frequently is that we have to move
files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
But just creating a normal patch, or cp'ing the files means when we pull
linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
have to move one of the files (the previously existing one) into the
deleted dir.  How do we cleanly move just a few files from a child tree
into the parent?  I think this is a lot like what would happen, if Linus
used BK and we wanted to send him support for some platforms, but not
all of the other changes we have.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

> Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> again.  One of the problems we hit frequently is that we have to move
> files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> But just creating a normal patch, or cp'ing the files means when we pull
> linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> have to move one of the files (the previously existing one) into the
> deleted dir.  How do we cleanly move just a few files from a child tree
> into the parent?  I think this is a lot like what would happen, if Linus
> used BK and we wanted to send him support for some platforms, but not
> all of the other changes we have.

BitKeeper is like a distributed, replicated file system with atomic changes.
That has certain advantages, much like a database.  What you are asking 
violates the database rules, if I understand you properly.  Are you asking
to move part of a changeset?  That's a no no, that's like moving the 
increment to your bank account without the decrement to mine; the banks
frown on that :-)

Or are you asking more about the out of order stuff, i.e., whole changesets
are fine but not all of them.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:37:56AM -0800, Larry McVoy wrote:
> > Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> > again.  One of the problems we hit frequently is that we have to move
> > files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> > But just creating a normal patch, or cp'ing the files means when we pull
> > linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> > have to move one of the files (the previously existing one) into the
> > deleted dir.  How do we cleanly move just a few files from a child tree
> > into the parent?  I think this is a lot like what would happen, if Linus
> > used BK and we wanted to send him support for some platforms, but not
> > all of the other changes we have.
> 
> BitKeeper is like a distributed, replicated file system with atomic changes.
> That has certain advantages, much like a database.  What you are asking 
> violates the database rules, if I understand you properly.  Are you asking
> to move part of a changeset?  That's a no no, that's like moving the 
> increment to your bank account without the decrement to mine; the banks
> frown on that :-)
> 
> Or are you asking more about the out of order stuff, i.e., whole changesets
> are fine but not all of them.

Unfortunatly I think the PPC tree has hit both cases :)  The restriction
that everything gets moved as a changeset is fine tho. One problem is
an out-of-order (or rather a single) changeset which creates a few
files.

The other problem is we create a file (say
arch/ppc/kernel/prpmc750_setup.c) and then 4-5 changesets effect this
file (code, code, bk mv, code, code).  If this is doable in multiple
out-of-order sends to the parent, that'd probably be OK.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> > Er, not the pristine tree, the linuxppc_2_4 tree, sorry.  I'll try
> > again.  One of the problems we hit frequently is that we have to move
> > files from linuxppc_2_4_devel into linuxppc_2_4, once they prove stable.
> > But just creating a normal patch, or cp'ing the files means when we pull
> > linuxppc_2_4 back into linuxppc_2_4_devel we get a file conflict, and
> > have to move one of the files (the previously existing one) into the
> > deleted dir.  How do we cleanly move just a few files from a child tree
> > into the parent?  I think this is a lot like what would happen, if Linus
> > used BK and we wanted to send him support for some platforms, but not
> > all of the other changes we have.
> 
> BitKeeper is like a distributed, replicated file system with atomic changes.
> That has certain advantages, much like a database.  What you are asking 
> violates the database rules, if I understand you properly.  Are you asking
> to move part of a changeset?  That's a no no, that's like moving the 
> increment to your bank account without the decrement to mine; the banks
> frown on that :-)
> 
> Or are you asking more about the out of order stuff, i.e., whole changesets
> are fine but not all of them.

If I understand it correctly, yes, you want to `push' only one changeset (the
creation of the new file) to the parent repository. Either directly (through
push), or through creating a patch in the child repository and importing it in
the parent repository.

[ Disclaimer: I'm not that familiar with the  problem Tom mentions ]

However, why couldn't BK automatically resolve this?

In BK, a file creation (or a rename) is simply a changeset, just like a change
to the contents of a file (i.e. a patch that affects one file only), right?

If I modify a file in the child repository, and that change ends up in the
same file in the parent repository (i.e. Linus applied the patch I sent there),
BK will automatically resolve the issue when I do a pull in my child
repository.  How is this different from a new file I added in the child
repository, and the same file (with the same contents, or contents from a
previous revision in the child repository) that got added in the parent later?
If I do a pull, BK should `merge' the change (a new file)? Or am I missing
something?

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Paul Mackerras]

Larry McVoy writes:

> On Wed, Jan 30, 2002 at 09:07:07AM -0700, Tom Rini wrote:
> > Then how do we do this in the bk trees period?  To give a concrete
> > example, I want to move arch/ppc/platforms/prpmc750_setup.c from
> > 2_4_devel into 2_4, without loosing history.  How?  And just this file
> > and not all of _devel.
> 
> That question doesn't parse.  There are multiple ways you can do it but 
> once you do patches will no longer import cleanly from Linus.  The whole
> point of the pristine tree is to give yourself a tree into which you can
> import Linus patches.  If you start putting extra stuff in there you will
> get patch rejects.

I think there is a misunderstanding here: we actually have 3 trees:

linux_2_4		"pristine" tree, identical to Marcelo's latest
linuxppc_2_4		"stable" tree, stuff we are pushing to Marcelo
linuxppc_2_4_devel	"devel" tree, bleeding edge stuff

Normally linuxppc_2_4 pulls from linux_2_4 and linuxppc_2_4_devel
pulls from linuxppc_2_4.  That is, linuxppc_2_4_devel has all of the
changesets that are in linuxppc_2_4, and more.  When Marcelo does a
new release the changes go into linux_2_4 and propagate from there
into linuxppc_2_4 and then linuxppc_2_4_devel.

Now when we decide that some stuff in linuxppc_2_4_devel has matured
to the point where we want it in linuxppc_2_4, what we currently do,
conceptually at least, is to generate a patch with the changes we want
and apply that to the linuxppc_2_4 tree.  If we had the ability to
apply changesets out-of-order, presumably what we could do is to push
the particular changesets of interest from linuxppc_2_4_devel back up
into linuxppc_2_4.  Then when we pulled from linuxppc_2_4 into
linuxppc_2_4_devel, bk would presumably say "got that one already"
about those changesets.

At the moment the process of applying a patch to linuxppc_2_4 and
doing the pull into linuxppc_2_4_devel results in conflicts which bk
mostly handles automatically *except* in the cases where the patch
creates new files.

Paul.

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > It does in some ways anyhow.  Following things downstream is rather
> > painless, but one of the things we in the PPC tree hit alot is when we
> > have a new file in one of the sub trees and want to move it up to the
> > 'stable' tree
>
> Summary: only an issue because Linus isn't using BK.

Bitkeeper also seems to have some problems applying out-of-order
changesets or applying them partially.

Changesets sent by 'bk send' are also much harder to read than
unidiffs ;)

I think for bitkeeper to be useful for the kernel we really need:

1) 'bk send' format Linus can read easily

2) the ability to send individual changes (for example, the
   foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

3) the ability for Linus to apply patches that are slightly
   "out of order" - a direct consequence of (2)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> >
> > Summary: only an issue because Linus isn't using BK.
> 
> Bitkeeper also seems to have some problems applying out-of-order
> changesets or applying them partially.
> 
> Changesets sent by 'bk send' are also much harder to read than
> unidiffs ;)
> 
> I think for bitkeeper to be useful for the kernel we really need:
> 
> 1) 'bk send' format Linus can read easily

I think you can do bk send -u which spits out a unified diff in the
comments of the file or so.

> 2) the ability to send individual changes (for example, the
>    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

This is sort of what I was getting at, execpt in this case foo_net.c is
also a new file as well.  Myself and Paul haven't found a good way to do
this :(

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 08:42:33AM -0700, Tom Rini wrote:
> > > On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> > > It does in some ways anyhow.  Following things downstream is rather
> > > painless, but one of the things we in the PPC tree hit alot is when we
> > > have a new file in one of the sub trees and want to move it up to the
> > > 'stable' tree
> >
> > Summary: only an issue because Linus isn't using BK.
> 
> Bitkeeper also seems to have some problems applying out-of-order
> changesets or applying them partially.

It does indeed and I think this is a far more serious issue than, for
example, the shouting SCCS subdirectories.  So let's discuss it.

> Changesets sent by 'bk send' are also much harder to read than
> unidiffs ;)

Yeah but if you do a bk send -d it prefixes them with unidiffs or 
you can do 
	
	cat patch | bk receive /home/bk/vmtree
	cd /home/bk/vmtree/RESYNC
	bk csets

and you are looking at the changes in the changeset viewer which most
people think is nicer than unidiffs.

> I think for bitkeeper to be useful for the kernel we really need:
> 
> 1) 'bk send' format Linus can read easily

That's done.

> 2) the ability to send individual changes (for example, the
>    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff

That's possible now but not a really good idea.

> 3) the ability for Linus to apply patches that are slightly
>    "out of order" - a direct consequence of (2)

This is really the main point.  There are two issues, one is out of order
and the other is what we call "false dependencies".  I think it is the 
latter that bites you most of the time.  The reason you want out of order
is because of the false dependencies, at least that is my belief, let
me tell you what they are and you tell me.

Suppose that you make 3 changes, a driver change, a vm change, and a 
networking change.  Suppose that you want to send the networking change
to Linus.  With patch, you just diff 'em and send and hope that patch
can put it together on the other end.  With BK as it stands today, there
is a linear dependency between all three changes and if you want to send
the networking change, you also have to send the other 2.

How much of the out order stuff goes away if you could send changes out
of order as long as they did not overlap (touch the same files)?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 08:32:54AM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
[snip]
> > 2) the ability to send individual changes (for example, the
> >    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff
> 
> That's possible now but not a really good idea.

How is it possible and what can go wrong?  This will be a common thing I
think, so it'd be a good thing to know (and have reliable and be a good
idea in some form).

> > 3) the ability for Linus to apply patches that are slightly
> >    "out of order" - a direct consequence of (2)
> 
> This is really the main point.  There are two issues, one is out of order
> and the other is what we call "false dependencies".  I think it is the 
> latter that bites you most of the time.  The reason you want out of order
> is because of the false dependencies, at least that is my belief, let
> me tell you what they are and you tell me.

I think that sounds about right.  If changeset 1.123 and 1.124 are only
related in that one got commited first (and for the sake of argument,
isn't done just yet) and the other is ready to go, the second one is
stuck.

> Suppose that you make 3 changes, a driver change, a vm change, and a 
> networking change.  Suppose that you want to send the networking change
> to Linus.  With patch, you just diff 'em and send and hope that patch
> can put it together on the other end.  With BK as it stands today, there
> is a linear dependency between all three changes and if you want to send
> the networking change, you also have to send the other 2.
> 
> How much of the out order stuff goes away if you could send changes out
> of order as long as they did not overlap (touch the same files)?

I think that'd help a good deal of the cases.

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:43:36AM -0700, Tom Rini wrote:
> On Wed, Jan 30, 2002 at 08:32:54AM -0800, Larry McVoy wrote:
> > On Wed, Jan 30, 2002 at 02:14:52PM -0200, Rik van Riel wrote:
> [snip]
> > > 2) the ability to send individual changes (for example, the
> > >    foo_net.c fixes from 1.324 and 1.350) in one nice unidiff
> > 
> > That's possible now but not a really good idea.
> 
> How is it possible and what can go wrong?  This will be a common thing I
> think, so it'd be a good thing to know (and have reliable and be a good
> idea in some form).

It's possible in that we can generate regular diffs for any range of
any files.  Cort used to do this all the time to send a subset of the
tree to Linus as a regular patch, talk to him about getting his scripts.
Or talk to Paul, I suspect he does it as well.

> > > 3) the ability for Linus to apply patches that are slightly
> > >    "out of order" - a direct consequence of (2)
> > 
> > This is really the main point.  There are two issues, one is out of order
> > and the other is what we call "false dependencies".  I think it is the 
> > latter that bites you most of the time.  The reason you want out of order
> > is because of the false dependencies, at least that is my belief, let
> > me tell you what they are and you tell me.
> 
> I think that sounds about right.  If changeset 1.123 and 1.124 are only
> related in that one got commited first (and for the sake of argument,
> isn't done just yet) and the other is ready to go, the second one is
> stuck.

Indeed.  We're painfully aware of this.

> > Suppose that you make 3 changes, a driver change, a vm change, and a 
> > networking change.  Suppose that you want to send the networking change
> > to Linus.  With patch, you just diff 'em and send and hope that patch
> > can put it together on the other end.  With BK as it stands today, there
> > is a linear dependency between all three changes and if you want to send
> > the networking change, you also have to send the other 2.
> > 
> > How much of the out order stuff goes away if you could send changes out
> > of order as long as they did not overlap (touch the same files)?
> 
> I think that'd help a good deal of the cases.

So here's the deal on this topic.  The out of order thing is a much
bigger deal for a large open source project than it is for our commercial
customers.  It's also hard to do correctly, it would take at least a
month.  Linus has flirted with using BitKeeper multiple times and then
gotten distracted.  If we had dropped everything and fixed the issues he
needs fixed rather than the issues our commercial customers need fixed,
we'd be out of business and you'd have the lovely task of trying to make
this work in the BitKeeper source.  You can believe me or not, but you'd
have very little chance of getting it to work, the BK source base is 
a lot larger and more complex than the generic part of the Linux kernel.

If enough of the kernel people start using BitKeeper and demanding the
out of order stuff, we'll do it.  The lead time is about a month, so
you have to deal with that.  On the other hand, if this turns into yet
another kernel "she loves, she loves me not, she loves me, she loves me
not" about BK, we're not going to fix the out of order stuff until it
is the most important issue for our commercial customers.

Hopefully, you'll take this in the spirit in which it is intended.
We want to help out the kernel team, that was the reason for writing
BitKeeper.  We have to survive, however, and that means paying attention
to the commercial needs as well.  If/when it looks like Linus & Co are
serious about using BK, I'll staff a couple of people to address the 
out of order stuff and commit to a delivery date.  It's clear that it
is the biggest "gotcha" about BK & the kernel work flow.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> How much of the out order stuff goes away if you could send changes
> out of order as long as they did not overlap (touch the same files)?

could this be made: 'as long as they do not touch the same lines of code,
taking 3 lines of context into account'? (ie. unified diff definition of
'collisions' context.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 07:35:03PM +0100, Ingo Molnar wrote:
> 
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> 
> > How much of the out order stuff goes away if you could send changes
> > out of order as long as they did not overlap (touch the same files)?
> 
> could this be made: 'as long as they do not touch the same lines of code,
> taking 3 lines of context into account'? (ie. unified diff definition of
> 'collisions' context.)

No.  What you described is diff/patch.  We have that already and if it
really worked in all the cases there would be no need for BitKeeper to
exist.  I'll be the first to admit that BK is too pedantic about change
ordering and atomicity, but you need to see that there is a spectrum and
if we slid BK over to what you described it would be a meaningless tool,
it would basically be a lot of code implementing what people already do
with diff/patch.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 07:35:03PM +0100, Ingo Molnar wrote:
> > On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > > How much of the out order stuff goes away if you could send changes
> > > out of order as long as they did not overlap (touch the same files)?
> >
> > could this be made: 'as long as they do not touch the same lines of code,
> > taking 3 lines of context into account'? (ie. unified diff definition of
> > 'collisions' context.)
>
> No.  What you described is diff/patch.  We have that already and if it
> really worked in all the cases there would be no need for BitKeeper to
> exist.  I'll be the first to admit that BK is too pedantic about
> change ordering and atomicity, but you need to see that there is a
> spectrum and if we slid BK over to what you described it would be a
> meaningless tool,

OK, so why not put the boundary at the same point as where
bitkeeper still manages to automatically merge branches and
where it gives up ?

(this seems to be somewhat finer grained than the whole-file
level, but more picky and intelligent than patch/diff)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> No.  What you described is diff/patch.  We have that already and if it
> really worked in all the cases there would be no need for BitKeeper to
> exist.  I'll be the first to admit that BK is too pedantic about
> change ordering and atomicity, but you need to see that there is a
> spectrum and if we slid BK over to what you described it would be a
> meaningless tool, it would basically be a lot of code implementing
> what people already do with diff/patch.

eg. i sent 8 different scheduler update patches 5 days ago:

 [patch] [sched] fork-fix 2.5.3-pre5
 [patch] [sched] yield-fixes 2.5.3-pre5
 [patch] [sched] SCHED_RR fix, 2.5.3-pre5
 [patch] [sched] set_cpus_allowed() fix, 2.5.3-pre5
 [patch] [sched] entry.S offset fix, 2.5.3-pre5.
 [patch] [sched] cpu_logical_map fixes, balancing, 2.5.3-pre5
 [patch] [sched] compiler warning fix, 2.5.3-pre3
 [patch] [sched] unlock_task_rq() cleanup, 2.5.3-pre3

these patches, while many of them are touching the same file (sched.c) are
functionally orthogonal, and can be applied in any order. Linus has
applied all of them, but he might have omitted any questionable one and
still apply the rest.

how would such changes be expressed via BK, and would it be possible for
Linus to reject/accept an arbitrary set of these patches?

	Ingo

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 07:48:35PM +0100, Ingo Molnar wrote:
> eg. i sent 8 different scheduler update patches 5 days ago:
> 
>  [patch] [sched] fork-fix 2.5.3-pre5
>  [patch] [sched] yield-fixes 2.5.3-pre5
>  [patch] [sched] SCHED_RR fix, 2.5.3-pre5
>  [patch] [sched] set_cpus_allowed() fix, 2.5.3-pre5
>  [patch] [sched] entry.S offset fix, 2.5.3-pre5.
>  [patch] [sched] cpu_logical_map fixes, balancing, 2.5.3-pre5
>  [patch] [sched] compiler warning fix, 2.5.3-pre3
>  [patch] [sched] unlock_task_rq() cleanup, 2.5.3-pre3
> 
> these patches, while many of them are touching the same file (sched.c) are
> functionally orthogonal, and can be applied in any order. Linus has
> applied all of them, but he might have omitted any questionable one and
> still apply the rest.
> 
> how would such changes be expressed via BK, and would it be possible for
> Linus to reject/accept an arbitrary set of these patches?

There is a way to do this in BK that would work just fine.  It pushes some
work back onto the developer, but if you are willing to do it, we have no
problem doing what you want with BK in its current form and I suspect that
the work is very similar to what you are already doing.

In your list above, all of the patches are against 2.5.3-pre5.  If you did
the work for each patch against the 2.5.3-pre5 baseline, checking it in,
saving the BK patch, removing that changeset from the tree, and then going
onto the next change, what you'd have is exactly the same set of patches
as you have no.  Literally.  You could type the appropriate command to BK
and you should be able to generate a bit for bit identical patch.

In BK's mind, what you have done is to make a very "bushy" set of changes,
they are all "side by side".  If you think of a graph of changes, you started
with

			[older changes]
			      v
			  [2.5.3-pre4]
			      v
			  [2.5.3-pre5]

and then you added one change below that, multiple times.  If you were to
combine all of those changes in a BK tree, it would look like

			[older changes]
			      v
			  [2.5.3-pre4]
			      v
			  [2.5.3-pre5]
  [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]

and BK would be happy to allow you to send any subset of the sched changes
to Linus and it would work *exactly* how you want it to work.  If we could
get people to work like this, there are no problems.  Just to make it really
clear you would do this

	for p in patch_list
	do	bk vi sched.c	# that will lock it if isn't
		hack, debug, test
		bk citool	# check it in and make a changeset
		bk makepatch -r+ > ~/bk-patches/patch.$p
		bk undo -fr+	# get back to the same baseline
	done

Here's what people actually do.  They make the first change, then make
the second change in the same tree that already has the first change,
and so on.  BitKeeper faithfully records the linear sequence of changes
and enforces that the changes propogate as that linear sequence.  You can
skip some at the end but you can't skip any in the middle.

In your particular case, we really need out of order changesets to allow
the second type of work flow and cherry picking.  However, a fairly common
case is that the changes are all in unrelated files and *even then* 
BitKeeper enforces the linearity.  That's the problem I think we need to
fix first, it's not a complete solution, but it is the 80-90% solution.
Until we give you a 100% solution, you have to realize that you are making
side by side changes and actually do it that way.

If any of this is not clear to anyone, please speak up and I'll try and 
draw some pictures and add some explanation.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> There is a way to do this in BK that would work just fine.  It pushes some
> work back onto the developer, but if you are willing to do it, we have no
> problem doing what you want with BK in its current form and I suspect that
> the work is very similar to what you are already doing.

The thing is, bk should be able to do this on its own.

The rule on check-in should be: if the resultant changeset can be
automatically merged with an earlier changeset, it should be _parallel_ to
that changeset, not linear.

And note the "automatically merged" part. That still guarantees your
database consistency at all levels.

Let us assume that you have a tree that looks like

	a -> b -> c

together with modifications "d". Now, "bk ci" (or, because this is more
expensive than a plain "ci", you can call it "bk backmerge" or something,
and all sane people will just stop using "ci") should instead of doing a
plain

	a -> b -> c -> d

it would see how far back it can go with an automatic merge and add "d" at
the _furthest_ point possible. So let's assume that "d" really cannot be
merged with "b" but doesn't clash with "c", so what you'd create with "bk
backmerge" is the "maximally parallel version:

	a -> b -> c
		\
		 > d

Now, you'll say that this is exponentially expensive to do, and I agree.
But CPU-time is cheap, and notice that this "backmerge" would be done not
in one centralized location, but in parallel at different developers.

(Yeah, my example may look "cheap", but if you do exclusively backmerging,
then after a while your trees will have lots of very wide development, and
the m,ore interesting backmerge is when you already have

	a -> b -> c -> f

		-> d

		-> e

and you're adding "g", and it doesn't merge with "c" but not with "d" and
"e", so you have to test every path backwards to see where you can push
it. In this case you'd end up with

	a -> b  -> c -> f

		     -> g

		-> d

		-> e

kind of tree.)

Just how expensive would that be? I'd be willing to make my machine sweat
a bit, if it would just automatically generate the most parallel
changesets possible.. And I bet you could do _this_ fairly easily if you
didn't care too much about trying to be too efficient.

You're saying your merges are perfect. USE that knowledge, and make it do
"bk backmerge".

(Once you do backmerge, the false dependencies no longer exist, AND
suddenly "bk" actually gives people information that they didn't have
before: the revision history actually tells you the _true_ dependencies in
the tree, not some stupid "this is the order in which things were checked
in" stuff that doesn't add any value that can't be seen from the changeset
directly)

Larry, give me that, and I'll promise to use bk exclusively for two months
to shake it out..

		Linus

Re: A modest proposal -- We need a patch penguin

[Georg Nikodym]

On Wed, 2002-01-30 at 13:23, Linus Torvalds wrote:

[really interesting and insightful comments about revision graphs]

The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
the change that is 'g'.  It's actually a label that implies a point in
time as well as all the change that came before it.  Stated differently,
it is a set.

Using your graph:

        a -> b -> c -> f

                -> d

                -> e

and the way that people currently think (and thus speak) of these
things, saying that you're using a version 'e' kernel is ambiguous
because it may or may not have 'c' or 'd'.  This ambiguity also
complicates the task of reproducing a tree at some known state later.

You, as the center of the known universe may not need to concern
yourself with such trifles, but speaking as one of those fabled
commercial customers, the ability to say unambiguously say what's been
changed (read: fixed) is really important.

All that said, I like your idea about revision graph compression.  My
concerns might simply be mitigated by being able to insert "release"
points (simply a tag that implies/includes all the preceding
changesets).

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Wed, Jan 30, 2002 at 02:38:21PM -0500, Georg Nikodym wrote:

[snip]
> and the way that people currently think (and thus speak) of these
> things, saying that you're using a version 'e' kernel is ambiguous
> because it may or may not have 'c' or 'd'.  This ambiguity also
> complicates the task of reproducing a tree at some known state later.

Well, this is what tags are for.  The ambiguity in changesets is OK,
since it's possible anyhow with multiple people (and with some creative
work maybe, 'c' would be before 'd', but at the same level, so 'c'
wouldn't get 'd', but this might break the new behavior so..)  But if
you do:
a b (tag v1) c e (tag v2)
             d
	     f (added after the v2 tag)

it should be possible to have the 'v2' tag say what changsets it had, or
even what rev of each file it was made to.

Larry, how does BK handle this now?  Ive been thinking about this for a
bit and am kind of curious now..

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On 30 Jan 2002, Georg Nikodym wrote:
>
> The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
> the change that is 'g'.  It's actually a label that implies a point in
> time as well as all the change that came before it.  Stated differently,
> it is a set.

I disagree.

"g" is really just one thing: it is "the changes".

However, you are obviously correct that any changes are inherently
dependent on the context those changes are in. And there are multiple
contexts.

One context is simply the "when in time" context, which is interesting, as
when you serialize all changesets you see the time-wise development. And
this is really the _only_ context that BK (and most other source control
tools) really think about.

However, in another sense, the "when in time" context is completely
meaningless. Should you care what the _temporal_ relationship between two
independent patches is? I say "Absolutely not!". The temporal relationship
only hides the _real_ revision information, which is what the patch
actually depends on.

And note that my suggestion to have a "bk backmerge" does not remove the
temporal relationships. All changesets already have a timestamp (they
clearly have to have it, just so that you can see when they happened and
say "what did this tree look like one month ago?"). So we already _have_
the temporal information, and encoding that temporal information into the
"relationship" information actually ends up costing you quite dearly.

I'd say that most (maybe all) of the complaints about not being able to
apply changesets in any random order comes exactly from the fact that
developers _know_ that temporal relationships are often not relevant. From
a development standpoint, temporal relationships are only relevant if they
match the _causal_ relationships, and even then you can often find patches
that are _caused_ by previous patches, but that are valid on their own
(and can indeed be more important, or even completely obviate the need for
the original patch).

So what I'm saying is that from a patch revision standpoint, temporal
information is useless. You still have enough information to recreate the
tree "at time 'g'" by just doing the equivalent of "bk get -c<date-of-g>".

See?

> You, as the center of the known universe may not need to concern
> yourself with such trifles, but speaking as one of those fabled
> commercial customers, the ability to say unambiguously say what's been
> changed (read: fixed) is really important.

But you don't lose that ability. You still have all the information you
used to have, you just have even _more_ information. You have the
information on notjust when the change was checked in (temporal), you have
the information on what files/changes it really _depended_ on.

Now, after those arguments, I'll just finish off with saying that I
actually agree with you to some degree - as I already said in private
email to Larry, I would definitely also want to have a way of stopping
back-merging at some point. In particular, when I'd tag a relase (ie
something like Linux-2.5.3, I would potentially also want to set a
"backmerge marker tag" which basically tells the backmerge logic that it's
not worth it to try to backmerge past that version tag.

That would decrease the chance of confusion considerably, and it would
also avoid the exponential complexity problem. Let's face it, exponential
algorithms can be really useful, but you do want to have some way of
making sure that the bad behaviour never happens in practice. A way of
limiting the set of changelogs to be considered for backmerging not only
means that humans don't get as confused, the computer also won't have to
work insanely to try to go back to Linux version 0.01 if a small patch
happened to apply all the way back.

		Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 01:17:25PM -0800, Linus Torvalds wrote:
> 
> On 30 Jan 2002, Georg Nikodym wrote:
> >
> > The thing that's missing here is that 'g' (or 1.7) doesn't just refer to
> > the change that is 'g'.  It's actually a label that implies a point in
> > time as well as all the change that came before it.  Stated differently,
> > it is a set.
> 
> I disagree.
> 
> "g" is really just one thing: it is "the changes".

In some system that you are imagining, you are correct.  In BK, you are
flat wrong and you've missed an important point.

In BK, "g" means two things: the diffs and the stuff that has no diffs.
In other words, both what changed and exactly what was in the tree at
the time the change was created.

In your mind, "g" is just the diffs applied to any tree which allows
the diffs to apply cleanly by patch, and even more trees where they
don't apply cleanly but you can fix up the patch rejects by hand.

The BK "g" is a more pwoerful statement than the Linus "g".  
The Linus "g" is a more flexible thing than the BK "g".

The BK "g" is more powerful because when I say "I've tested g", I'm
saying one hell of a lot more exact statement then when you say "I've
tested g".  The BK "g" is reproducible, without exception.  Your "g"
is reproducible if and only if I happen to have exactly the same tree
that you had when you applied the "g" patch.

The real power of the BK way is that testing builds up incrementally,
one on top of the other, but in your way, each time you apply the 
patch in some other context, the testing is invalidate.  You can
swear up and down all you want that it doesn't make any difference
and every time you do I will come up with a different example where
a whole team of Linus types said the same thing and were wrong.  If
you apply the patch to a different context, you have to start the
testing process over.  No ifs, ands, or buts.  If I can't run cmp(1)
on the object and get no diffs, it's not the same object.

> One context is simply the "when in time" context, which is interesting, as
> when you serialize all changesets you see the time-wise development. And
> this is really the _only_ context that BK (and most other source control
> tools) really think about.

If that's what you believe about BK, you haven't the faintest idea of
how it works.  BK has, right now, in the bits that you have on your
disk, the ability to recreate any change with all, some, or none of the
previous changes.  bk help sccscat.  You can create a version of the 
file which has only the odd numbered changes in it if that is what you
want.  Try that in other systems or with your patches.

> However, in another sense, the "when in time" context is completely
> meaningless. Should you care what the _temporal_ relationship between two
> independent patches is? I say "Absolutely not!".

And I (and the entire release management and software support world)
say  "you're wrong".  I'll grant you are right a lot, but lots of times
you'll be wrong.  I've seen enough changes like "remove all the trailing
whitespace, it can't hurt anything" and it breaks the product.  

> I'd say that most (maybe all) of the complaints about not being able to
> apply changesets in any random order comes exactly from the fact that
> developers _know_ that temporal relationships are often not relevant.

Phooey.  They _think_ they know.  You, Linus, are better than average
on this topic but you make mistakes too.  Any the point you are missing
is that in the face of a random set of inputs, any time you change
anything, you need to restart the test cycle.  The BK way lets you
isolate exactly what caused the problem.  We can, and do, run binary
search over changes to figure out what caused the problem.  As outlined
in Documentation/bug-hunting.txt.  The difference is that we track
it down to exactly the change that caused the problem and you can't,
you are varying more variables than BK does.  I'm not saying that you
shouldn't be able to vary things, I'm saying that it has a cost, just
like not being able to vary it has a cost.  You are painting a picture
that says you can vary the order and it won't matter.  That's flat out
wrong and I think you know it.

> So what I'm saying is that from a patch revision standpoint, temporal
> information is useless. You still have enough information to recreate the
> tree "at time 'g'" by just doing the equivalent of "bk get -c<date-of-g>".
> 
> See?

No, I don't.  It's a distributed system and there is lots of parallel
development and date-of-g may have many matches.  And it's relatively
meaningless since you are applying g in multiple contexts.  In BK as
it stands, "g" is an invariant.  It means one thing: the state of the
tree immediately after "g" was checked in.  What does your "g" mean?
Some diffs.  Where do they work?  You don't know.  You have to go try
it to find out.

That's all fine, just admit that you have lost something before you
throw it away.  There is no way we'll change BK to allow what you are
asking for until you get it.  I understand what you want, I understand
the implications, I need you to really understand what it is that you
are asking.

I also think we need some face time, I'd like to draw you some pictures
and I find the keyboard just too slow for that.  I think this would be
a lot faster in person.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Linus Torvalds wrote:
[snip]
> Now, after those arguments, I'll just finish off with saying that I
> actually agree with you to some degree - as I already said in private
> email to Larry, I would definitely also want to have a way of stopping
> back-merging at some point. In particular, when I'd tag a relase (ie
> something like Linux-2.5.3, I would potentially also want to set a
> "backmerge marker tag" which basically tells the backmerge logic that it's
> not worth it to try to backmerge past that version tag.
> 
> That would decrease the chance of confusion considerably, and it would
> also avoid the exponential complexity problem. Let's face it, exponential
> algorithms can be really useful, but you do want to have some way of
> making sure that the bad behaviour never happens in practice. A way of
> limiting the set of changelogs to be considered for backmerging not only
> means that humans don't get as confused, the computer also won't have to
> work insanely to try to go back to Linux version 0.01 if a small patch
> happened to apply all the way back.
> 
>                 Linus
[and earlier...]
> However, you are obviously correct that any changes are inherently
> dependent on the context those changes are in. And there are multiple
> contexts.

What about the design context?

I'm a bit concerned about the design-level inter-dependencies of
changesets that don't result in source-level dependencies.

Hypothetical situation:
Changeset adds driver for device Q.  Now, let's further suppose that in
2.5.x we have perfect modularity for drivers and at that time Q is
added... we just add a directory, linux/drivers/Qdrv.  It won't conflict
with 2.5, 2.4.x, 2.2.x, etc..  However, because 2.2.x does not have the
hooks necesary to see it, Q won't work on those kernels.  There is a
design-level dependency in changeset q.

This would be indirectly addressed with the 'backmerg marker tag', but I
have a nagging doubt.

Maybe a better example:
What about changing a global variable (say, from 'events' to
'global_events')?  You change it in the global place, yielding a
changeset.  Later, the individual users change the name.  But if an
individual user has seen very little change in the time before the
global_events change (and the global location has been changing a lot),
that patchset could backmerge beyond the global change.

Say 'f' was the global change, and 'g' was an individual change. 
Backmerge could yield:

        a -> b -> c -> f

                -> d

                -> e
          -> g

even though 'g' really depends on 'f'.

Thoughts?

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Eli Carter wrote:
> > However, you are obviously correct that any changes are inherently
> > dependent on the context those changes are in. And there are multiple
> > contexts.
>
> What about the design context?
>
> I'm a bit concerned about the design-level inter-dependencies of
> changesets that don't result in source-level dependencies.

Well, I'm personally worried about _no_ inter-dependencies that show up as
source-level dependencies that are impossible to break.

If you want to have a "known working version", that's what tagging is for:
basically a list of all patch-sets that makes up the current tree. That
includes all the dependencies.

> Hypothetical situation:
> Changeset adds driver for device Q.  Now, let's further suppose that in
> 2.5.x we have perfect modularity for drivers and at that time Q is
> added... we just add a directory, linux/drivers/Qdrv.  It won't conflict
> with 2.5, 2.4.x, 2.2.x, etc..  However, because 2.2.x does not have the
> hooks necesary to see it, Q won't work on those kernels.  There is a
> design-level dependency in changeset q.

Not so hypothetical, and entirely true. Which is, why I suggest that such
"deep merging" wouldn't actually go past tags.

Let's face it, the source control tool cannot know what works and what
doesn't. The only thing it can ever know about is "what applies". It can
take the approach that everything only applies to the last branch - which
is the traditional approach, but which introduces dependencies that simply
aren't there, and that _cannot_ be cut. This is what bk does now.

But the other approach is to say "whatever applies, applies, and as long
as I don't lose revision information I'll move it backwards". That has
other problems (as you point out), but now those problems are manageable,
and have solutions.

I'd rather take the problem that can be solved over the problem that
cannot.

The fact is, development _often_ is in the situation where somebody does a
quick-and-dirty fix, and then only later goes in and digs deeper and does
the right fix that makes the original fix completely unnecessary.

The way BK works now, if we call the quick-and-dirty fix "A", and the real
fix "B", the developer has a really hard time just sending "B" to me. He'd
have to re-clone an earlier BK tree, re-do B in that tree, and then send
me the second version.

I'm suggesting that he just send me B, and get rid of his tree. There are
no dependencies on A, and I do not want _crap_ in my tree just because A
was a temporary state for him.

			Linus

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> The way BK works now, if we call the quick-and-dirty fix "A", and the real
> fix "B", the developer has a really hard time just sending "B" to me. He'd
> have to re-clone an earlier BK tree, re-do B in that tree, and then send
> me the second version.
> 
> I'm suggesting that he just send me B, and get rid of his tree. There are
> no dependencies on A, and I do not want _crap_ in my tree just because A
> was a temporary state for him.

And you just lost some useful information.  The fact that so-and-so did
fix A and then did B is actually useful.  It tells me that A didn't work
and B does.  You think it's "crap" and by tossing it dooms all future
developers to rethink the A->B transition.

There is a reason that commercial companies guard their revision history
and fight like mad to preserve it.  It contains useful information,
even the bad stuff is useful.

Some stuff may be so bad that it shouldn't ever get in the tree, but you
don't accept anything at all from those people in general.  If Al Viro
takes one pass at a problem and it works well enough that it gets in 
the tree, and then later does a pass two that cleans it up, I can learn
from that.  That's very useful information, his brain frequently shines
a light in a dark corner but I'd miss a lot of that without the history.

Your approach is constantly dropping useful information on the floor.
It may not be useful to you but it's useful to virtually everyone
else.  Saving that information will increase the quality and reduce
the quantity of the patches you get.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> And you just lost some useful information.

No. If the useless crap ends up hiding the real points in the revision
history, getting rid of crud is _good_.

Remember how I asked for a way to "batch up" revision history? Same issue.
Crap is crap, and it more often hides the real issues than enlightens
anything at all.

		Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > And you just lost some useful information.
>
> No. If the useless crap ends up hiding the real points in the revision
> history, getting rid of crud is _good_.

Actually, allowing the deep merges to go past tags could
be useful for dragging bugfixes between the 2.4 and 2.5
kernels ...

... but I think the 'dragging' analogy is something we'll
want to keep here, not back merging across tags by default
but _trying_ to do the backmerge on demand only, when the
user wants to drag a changeset from 2.4 to 2.5.

We could just have a revtool-like interface for that.

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 05:36 pm, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> > The way BK works now, if we call the quick-and-dirty fix "A", and the
> > real fix "B", the developer has a really hard time just sending "B" to
> > me. He'd have to re-clone an earlier BK tree, re-do B in that tree, and
> > then send me the second version.
> >
> > I'm suggesting that he just send me B, and get rid of his tree. There are
> > no dependencies on A, and I do not want _crap_ in my tree just because A
> > was a temporary state for him.
>
> And you just lost some useful information.  The fact that so-and-so did
> fix A and then did B is actually useful.  It tells me that A didn't work
> and B does.  You think it's "crap" and by tossing it dooms all future
> developers to rethink the A->B transition.

<rant>

The noise to signal ratio is too high.  I think Linus has made it clear that 
he actively does not WANT this information.  (The "A" kind of patch is 
generally posted to linux-kernel, where it is buried deep in the flood.)

If developers can't ever make temporary changes to their tree which they do 
NOT intend to send to linus, they can't FUNCTION.  (Except my not doing 
development in said tree.)

They can, of course, explicitly do an end run around your "bondage and 
discipline" design by doing the "patch against the base tree" thing you 
suggested earlier.  Or just having it create plain diffs.  But if they have 
to go to lengths to work around your design to accomplish what THEY want (not 
what you want for them), then the tool is broken.

> There is a reason that commercial companies guard their revision history
> and fight like mad to preserve it.  It contains useful information,
> even the bad stuff is useful.

Do you REALLY think that Linus wants the experimental, quickly-reverted crutf 
of 300 maintainers accumulating in his tree?

Linux development is not a commercial company.  It is FAR more decentralized. 
 There are WAY more developers, doing WAY more experimentation, than most 
commercial companies could EVER afford to fund the man-hours for.  A 
commercial company generaly doesn't have bored college students futzing 
around with random ideas that have a 95% chance of failure, but occasionally 
produce something brilliant.  And a month of experimental baggage tag along 
with a twenty line patch is just insane. 

Trying out way more bad code than good is probably the NORM for the Linux 
development model.  Certainly outside of the core maintainers and 
lieutenants.  What you're basically saying is that people have to be really 
careful about ever putting any code into their tree, or else just extract 
straight patches from bitkeeper and put up with losing the tracking 
information and comments to avoid having your design ideas cram megabytes of 
cruft down their throat.

Good grief, -I- can see this is a bad idea...

> Some stuff may be so bad that it shouldn't ever get in the tree, but you
> don't accept anything at all from those people in general.

Not directly, no.  So basically, you're trying very hard to prevent bitkeeper 
from spreading far down the maintainer tree, due to the exponentially 
increasing number of overridden patches that bitkeeper will suck out of 
everybody's trees no matter how hard they try to avoid passing that garbage 
on to Linus.

Remember Linus's main job?  Code reviewing everythign and making 
architectural decisions?  Why on earth are you trying to force the poor man 
to read code that the submitter does NOT present to him as the solution?  
(There's 8 zillion ways NOT to fix a given problem.  We're trying to REDUCE 
the bandwidth demands on the guy...)

AAAAAAAAAH!

Okay, I'm better now.

(Sorry, this is a hot button issue with me.  Tool makers who insist they know 
how those tools should be used and what for, and thus reject feedback from 
users asking for greater flexibility with a "no, you don't want to DO that".  
Hammer vendors should not tell me what kind of nails to use.)

> If Al Viro
> takes one pass at a problem and it works well enough that it gets in
> the tree, and then later does a pass two that cleans it up, I can learn
> from that.  That's very useful information, his brain frequently shines
> a light in a dark corner but I'd miss a lot of that without the history.

So go read linux-kernel.

Giving people the OPTION of folding this cruft into the tree is one thing.  
FORCING them to do so is just WRONG.

> Your approach is constantly dropping useful information on the floor.

Information which does not belong in Linus's tree.  (You're basically saying 
Linus should add a subset of the rejected patch set to his tree's revision 
history.  Does it sound like a dumber idea to have Linus put EVERY rejected 
patch he deletes into his tree's history in some automated way?)

Monolithic evil.  Proper tool for proper job, don't try to force the job to 
adapt to what you think the tool is good for.

> It may not be useful to you but it's useful to virtually everyone
> else.

I would like to go on record as saying I don't consider this useful.  I don't 
have always enough bandwidth to read through every -pre diff.  This stuff 
gets discussed on linux-kernel.  People are talking about a patch archive 
system which may save rejected patches for posterity.  This is a seperate 
problem, and has a chance of succeeding exactly because it is NOT tangled 
with the issue of source control for the main tree.

> Saving that information will increase the quality and reduce
> the quantity of the patches you get.

Uh, Larry?  By definition, adding unnecessary reverted patches for dependency 
purposes to the set of patches Linus would have to apply to his tree is 
increasing the number of patches Linus actually would have to deal with, if 
he was using bitkeeper-to-bitkeeper.  You are FORCING people to do everything 
as diff -u and drop MORE information, because YOU are not being flexible here.

</rant>

Rob

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 06:18:11PM -0500, Rob Landley wrote:
> > And you just lost some useful information.  The fact that so-and-so did
> > fix A and then did B is actually useful.  It tells me that A didn't work
> > and B does.  You think it's "crap" and by tossing it dooms all future
> > developers to rethink the A->B transition.
> 
> <rant>

I'll see your rant and raise you a nickel.

> If developers can't ever make temporary changes to their tree which they do 
> NOT intend to send to linus, they can't FUNCTION.  (Except my not doing 
> development in said tree.)

Of course they can do that.  They get to decide what they push and
what they don't.  I don't think you understand BK.  What we are talking
about is the problem that the change has been committed to CVS, other
changes are built on top of it, and now Linus realizes that the change
was bad news.  The problem is extracting stuff out of the middle which
has already been built upon for more stuff.  How would you propose solving
that problem because that is the problem statement?

If someone sends Linus a patch, he checks into BK or CVS or whatever,
he then gets 5 other patches and applies them in BK/CVS, and THEN he
wants to take out the first patch, how would you suggest we do that?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 08:57 pm, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 06:18:11PM -0500, Rob Landley wrote:
> > > And you just lost some useful information.  The fact that so-and-so did
> > > fix A and then did B is actually useful.  It tells me that A didn't
> > > work and B does.  You think it's "crap" and by tossing it dooms all
> > > future developers to rethink the A->B transition.
> >
> > <rant>
>
> I'll see your rant and raise you a nickel.
>
> > If developers can't ever make temporary changes to their tree which they
> > do NOT intend to send to linus, they can't FUNCTION.  (Except my not
> > doing development in said tree.)
>
> Of course they can do that.  They get to decide what they push and
> what they don't.  I don't think you understand BK.

Entirely possible.  Quite likely in fact, I'm trying to pick it up as I go 
along.  (I've fired up the docs, but haven't had time to read too far yet.  
Trying to finish some paying work before hitting LWE tomorrow.)

> What we are talking
> about is the problem that the change has been committed to CVS, other
> changes are built on top of it, and now Linus realizes that the change
> was bad news.

The inflexibility of CVS relative to simply applying or reversing patches to 
a source tree on disk is a documented reason Linus doesn't use CVS.  Don't 
compare to CVS, compare to what Linus is currently using.  Beating a straw 
man doesn't HELP.

> The problem is extracting stuff out of the middle which
> has already been built upon for more stuff.  How would you propose solving
> that problem because that is the problem statement?

I'm not quite sure how Linus does this, but how I'd do it is keep around the 
last shipped tree and assemble a patch set against it.  If stuff gets really 
screwed up (a change in the middle needs to be reverted but doesn't unapply 
cleanly), then you can always revert back to the last shipped tree, re-apply 
the patches before the dead one, and then re-apply the patches afterwards 
fixing up rejects as necessary.  (And if I were Linus and the fixup took more 
than 30 seconds, I'd probably throw the dependent ones back down to a 
lieutenant or maintainer with a quick and dirty explanation and have THEM 
fixup the diffs, possibly by making them apply cleanly to the next released 
-pre version and submitting them again then.)  Bounces and even reversions 
due to conflicts are eminently understandable and part of the cost of doing 
business.  (It can be annoying at times, but debugging always is...)

This isn't me trying to make policy, this is me trying to guestimate what's 
going on today.  (I'm not trying to speak for Linus, just explain my 
understanding of how Linus works.)  Now that needs to translate into 
bitkeeper, and if it's HARDER to do in bitkeeper, bitkeeper probably needs to 
be fixed.

> If someone sends Linus a patch, he checks into BK or CVS or whatever,
> he then gets 5 other patches and applies them in BK/CVS, and THEN he
> wants to take out the first patch, how would you suggest we do that?

If the patch no longer unapplies cleanly, then a reversed patch to take it 
out may have to be applied to the tree.  In Linus's tree, this can also 
happen if he's shipped a -pre release with the old patch in it, so reverting 
it would need to be in the next incremental patch Linus releases anyway 
because we're beyond a checkpoint.  (Write barrier, changes committed to 
kernel.org, no longer able to be reordered in cache...)

But if the patch DOES unapply cleanly, and we haven't checkpointed yet, 
there's no good reason Linus can't just revert it.  There should definitely 
be an option to delete all traces of it from the hierarchy (carving moses' 
name off the pillars and all that).

Reverting a change out of order should not ARTIFICIALLY cause conflicts just 
because that's not the way bitkeeper expects people to think.  That would be 
another case of "false dependencies", and gets us back to Linus' backmerge 
concept.  This is sort of reverting a backmerge.

When there is a logistical problem with simply reverting a change because 
other changes really are on top of it, then logically the removal is either 
sort of a patch, or the later changes need to be temporarily reverted and 
fixed up before being reapplied.  (This is not conceptually new, as I said it 
happens with patches all the time.  Whether bitkeeper provides better tools 
to do this than diff, patch, and vi do is an open question.  It might be 
possible to make some sort of "--force --nodeps" way to revert a patch by the 
short and curlies, followed by a manual fixup of the patches that went in on 
top of it with that "bitkeeper fix -c" thing, but probably isn't worth the 
effort and I'm just not going there right now.)

But pointing out that "that history is valuable, leave the old residue in the 
tree even if it hasn't been sent out anywhere yet" is bogus.  Between 
checkpoints, Linus could translate his entire tree into fortran and back if 
he wanted to, and nobody else should ever have to care if he doesn't want 
them to.  The set of changes Linus chooses to ship to turn pre7 into pre8 is 
his choice, not yours.

What Linus seems to want is the shortest and most convenient straight line 
from checkpoint to checkpoint to be stored in his tree.  (Basically, like the 
patch files he puts out NOW, only more granular, with his changelog 
information broken up and stored with each change.)  How can bitkeeper 
provide THAT?  (If it can do more, great.  But if it can't provide this basic 
functionality without some bell or whistle interfering and making it hard to 
use, then there's a problem.  Please be very clear when you're talking about 
an an enhancement over and above this level of functionality when it is NOT a 
requirement...)

Rob

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

Sigh.  OK, I'm taking one more pass at trying to get you to see the light
and then I give up.  You need to understand that you haven't begun to
understand the problem, that you need to think a lot more before speaking,
and that it's really rude to shoot down a system that you haven't even
managed to through the basics of "hello world".  Food for thought.

On Wed, Jan 30, 2002 at 10:12:20PM -0500, Rob Landley wrote:
> The inflexibility of CVS relative to simply applying or reversing patches to 
> a source tree on disk is a documented reason Linus doesn't use CVS.  Don't 
> compare to CVS, compare to what Linus is currently using.  Beating a straw 
> man doesn't HELP.

Go read the archives, patch import/export is not ever, to the best of 
my knowledge, mentioned as a reason for Linus not liking CVS.  It's way
down on the issues list.  File renames, repository hierarchies, work
flow, reproducibility, I've talked with Linus about all of those as issues
but patch import/export never came up.  And CVS isn't remotely as good as
BK at that.  

> > The problem is extracting stuff out of the middle which
> > has already been built upon for more stuff.  How would you propose solving
> > that problem because that is the problem statement?
> 
> I'm not quite sure how Linus does this, but how I'd do it is [a really 
> complicated solution based on patches that won't work]

Think.  What you are describing is basically what Linus does today.  And
noone, including you, is happy.  You're the guy who started this thread.

> > If someone sends Linus a patch, he checks into BK or CVS or whatever,
> > he then gets 5 other patches and applies them in BK/CVS, and THEN he
> > wants to take out the first patch, how would you suggest we do that?
> 
> If the patch no longer unapplies cleanly, then a reversed patch to take it 
> out may have to be applied to the tree.  

Great, now we're getting somewhere.  You can take out a patch in
BK, including old ones way back in time, with a "bk cset -x<rev>".
Works great and no anti-patch is needed, so it's actually better than
what you described.

However, what you described *completely* misses the point.  Linus isn't
asking for an anti-patch, he doesn't want the bad patch in the revision
history at all.  He wants to be able to go backwards, across revisions,
and remove stuff in the middle.  He doesn't want the checkin comments,
he doesn't want the data, he wants no sign the patch was ever in the
revision history.

Do you start to see the problem?  You were yelling and screaming 
"BitKeeper sucks because it can't take a patch out" when in fact
it can do exactly what you said it can't.  On top of that, what you
were complaining about isn't the point.  The thing that you say 
BK can't do, but it can, is not what Linus wants.  Not even close.

And you haven't begun to understand that BK is a distributed, replicated
system.  You can turn all that off and you've got CVS, so turning it
off isn't an option.  Leaving it on means that the revision history is
replicated.  So it isn't an option to collapse a pile of changes into
a smaller pile, the bigger pile will come back the next time he updates
with the other tree.

And once again, you come back with another post that shows you just want
to yell and you haven't thought it through, into my kill file you go,
my blood pressure goes down, you get to yell all you want, everybody
is happy.  I'm willing to try and make BK do what is needed here; I'm
not willing to tolerate people who don't think.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> However, what you described *completely* misses the point.  Linus isn't
> asking for an anti-patch, he doesn't want the bad patch in the revision
> history at all.  He wants to be able to go backwards, across revisions,
> and remove stuff in the middle.  He doesn't want the checkin comments,
> he doesn't want the data, he wants no sign the patch was ever in the
> revision history.

I can't speak for Linus, but my main problem with BK is similar to what
you'd described.  Here's what I'm usually doing and what I'd like to
be able to do with BK:

Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.

I add a branch that consists of B' (B backported to original) and
ABB'^{-1}.  It joins the original at AB.

I backport C to B'.  Now I've got B', C', ABC(B'C')^{-1}.  Again, it
joins the original branch.

Repeat for D and E.  Now I've got the following picture (apologies for BUAG):

* -B'-> * -C'-> * -D'-> * -E'-> *
|                              /
A                            crap
V                            V
* -B-> * -C-> * -D-> * -E-> *

_Now_ I change the direction of last arrow.  Yes, it's more or less reverted
A.  And now I want to consider the top branch as the main history.

IOW, what I want is ability to play revisionist.  And it's not limited to
removing patches - if I've found a bug in A,  I want to be able to add A-fix
and move it all way back to right after A.  And merge them.  B, C, D and E
might have changed from that, but that's what I want.  Moreover, I might
have some junk left in the end (i.e. ABCDEA-fix == (AA-fix)B'C'D'E'noise)
and I'd really like to be able to say that (AA-fix)B'C'D'E' is the main
history now and other path (ABCDE A-fix noise^{-1}) is buried.

If you can give a way to do that - I'm happy.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.

If you just want to make A's changes go away, that's trivial:

	bk get -e -xA foo.c
	bk delta -y"kill A's changes"

all done.

If you want to make A go away out of the graph, that's only possible if
you have the only copy of the graph containing A.  Since BK replicates
the history, you only get to do what you want before you push your changes
to someone else.  No surgery after you let the cat out of the bag.

You have a repository and you haven't propogated all this stuff to
someplace else, then this is easy, we'll just rebuild the history minus A.
You or I could write a shell script using BK commands to do it in about
10 minutes.

It's a distributed, replicated file system which uses the revision history
to propogate changes.  If you don't care about talking to anyone else,
you can do whatever you want.  If you want to give someone your history
and then change it, no way.  That's rewriting what happened.

Now does it make more sense?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> > Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.
> 
> If you just want to make A's changes go away, that's trivial:
> 
> 	bk get -e -xA foo.c
> 	bk delta -y"kill A's changes"
> 
> all done.
> 
> If you want to make A go away out of the graph, that's only possible if
> you have the only copy of the graph containing A.  Since BK replicates
> the history, you only get to do what you want before you push your changes
> to someone else.  No surgery after you let the cat out of the bag.
> 
> You have a repository and you haven't propogated all this stuff to
> someplace else, then this is easy, we'll just rebuild the history minus A.
> You or I could write a shell script using BK commands to do it in about
> 10 minutes.
> 
> It's a distributed, replicated file system which uses the revision history
> to propogate changes.  If you don't care about talking to anyone else,
> you can do whatever you want.  If you want to give someone your history
> and then change it, no way.  That's rewriting what happened.
> 
> Now does it make more sense?

Sigh...  OK, let me try to put it in a different way.

I don't want A (or entire old path) to disappear.  What I want is ability
to have two paths leading to the same point + ability to mark one of
them as "more interesting".

I.e. the result I want is _two_ sets of changesets with the same compositions.

And _that_ is compatible with replication - I simply want the new path in
revision graph to be propagated.  Along with the "this path is more
interesting" being written on the new one.

Can that be done?  I want a way to re-split the set of deltas.  I'm perfectly
happy with old one staying around, as long as we remember that results of
old and new are the same object and that new is a prefered way to look at
the damn thing.

I suspect that it could be doable with with something as simple as "if you
ask to merge two identical nodes, I'll just mark them as such and ask which
is more interesting".  IIRC, BK doesn't require tree topology on the graph -
it can have converging branches.

_If_ that is feasible - the rest can be scripted around it.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 01:02:53AM -0500, Alexander Viro wrote:
> I don't want A (or entire old path) to disappear.  What I want is ability
> to have two paths leading to the same point + ability to mark one of
> them as "more interesting".
> 
> I.e. the result I want is _two_ sets of changesets with the same compositions.

Ahh, you want LODs.  And they neatly solve the problem you described.
And a bunch of others.  Think of a LOD as a revision history graph.
Imagine being able to create a new, empty (or partially populated)
"container".  That container is a LOD.  You can do set operations from
one LOD to the other.  They are a lot like branches except that they
themselves can branch & merge.

The way that we'd do what you wanted is you'd create a new LOD, 
stick B, C, D, E into it, and make it the default LOD in your 
repository.  

LODs have some very nice attributes - each change is a set element, the
LOD is nothing more than a recorded history of what set elements are in
this LOD, and you can cherry pick from one LOD to the other.  Out of
order, sparsely, whatever.

The only restriction is that you have to have all the changes in your
graph.  There is no concept of a sparse graph.  You can trim off stuff
that happens after some point but you can't remove points in the middle,
even if they are in the other LOD.  Is that OK?

Linus first sounded like he'd accept this as an answer and then later it
fell out of favor because even though he could hide a bad changeset in
another LOD, he didn't want it in the graph at all.  I don't know how
to do that.

The other gotcha is that LODs are only partially implemented and are 
going to stay that way until we achieve concensus on how BK should
work for you.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Wed, Jan 30, 2002 at 09:08:35PM -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 11:58:11PM -0500, Alexander Viro wrote:
> > Suppose I have 5 deltas - A, B, C, D, E.  I want to kill A.
> 
> If you just want to make A's changes go away, that's trivial:
> 
> 	bk get -e -xA foo.c
> 	bk delta -y"kill A's changes"
> 
> all done.

The real problem with this approach is it leads to information overload.

Go look at linuxppc_2_4_devel with sccstool and try to track major 
changes over the last 6 months. 

You can't. You are completely overwhelmed by the day-to-day thrashing of 
'bug fix this, bug fix that', and all the lines on the screen from the 
wacky merges we wind up doing in that tree.

I think what Linus and Viro really want is not to rewrite history
(although there are times when it would be nice), but say "I don't think
this change is worth looking at". Keep the data in the database, because
you have to to maintain consistency, but don't let me see it unless I ask
3 times, and say please.

This is the reason why I have a 'linuxppc_2_4_galileo' tree.. it's got a 
bunch of crap that will *only* ever be usefull to archeologists who want 
to figure out my or someone else's thought process. The code is ugly. The 
only reason I have another tree is because I want to work with other 
people and keep a record for myself in case I break something in the 
process.

Now, because I don't want to pollute the main tree, once this works, I'm 
going to merge this stuff manually with dirdiff over to 2_4_devel, and 
check it in.

And you're going to lose that information anyway (well, except on 
openlogging.org, but that's just comments), because once I have this 
working and checked into _devel, the _galileo tree is going to get 
deleted because nobody cares.

It is usefull to note that the human brain has evolved the capacity to
forget. Yes, sometimes it makes a mistake and forgets the wrong thing,
but let's not be too quick to forget there may be a lesson here, and 
everything is NOT always worth keeping. 

If you don't have some capacity to 'lose' information in your distributed
database, there will come a point in time that it will stop being
scalable. Either moore's law may break, or information will be going in
faster than you can keep up by depending on faster CPU's/more memory/disk
space to handle all that history.

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

Troy said:
> The real problem with this approach is it leads to information overload.
> 
> Go look at linuxppc_2_4_devel with sccstool and try to track major 
> changes over the last 6 months. 
> 
> You can't. You are completely overwhelmed by the day-to-day thrashing of 
> 'bug fix this, bug fix that', and all the lines on the screen from the 
> wacky merges we wind up doing in that tree.

I agree with this and the rest of your message.  Here's what we are doing
to address it:

a) I have a version of BK where revtool (aka sccstool) shows only the 
   tagged releases.  It's cool.  It also has a feature where you can
   select a node and ask it to color all versions which contain this
   node (seems like you'd never need that until you see a heavily used
   BK tree like Troy has).

b) part of the problem is the "merge" deltas in the ChangeSet file.
   They really need to be hidden or removed completely.  As a side
   effect of making the ChangeSet file more flexible a la Linus' 
   requests (doesn't give all that he wants but part of it), I 
   think these will go away.

c) LODs.  One thing a LOD can do for you is to allow you to have your
   private LOD into which you do a ton of changes.  Then you can do
   a "rollup include" into a public LOD, like the PPC LOD.  We then
   give you a LOD aware revtool and the information overload starts
   to go away (but preserves the information if you need it).

> I think what Linus and Viro really want is not to rewrite history
> (although there are times when it would be nice), but say "I don't think
> this change is worth looking at". Keep the data in the database, because
> you have to to maintain consistency, but don't let me see it unless I ask
> 3 times, and say please.

If we could agree that this is true, I'm ecstatic.  BK needs at least part
of what you said to be true.  If you can convince Linus that it doesn't 
matter if the data is there as long as his view is clean, that solves some
of the nasty problems.

That said, I'm sympathetic to the "I make lotso changes and I want to 
collapse them into one big change" problem.  It's certainly technically
possible to make BK do that, but then you have to *know* that nobody
else has a BK repo with your old detailed changes in it, or if they 
do, they won't ever try to push them back to you (or Linus or ...).  
It's not an error if they do, it's just that BK will view them as 
different changes and automerge them right back into the history.
So then you'll have both the collapsed version and the detailed version
which puts you worse off than when you started.

That's the whole issue with the "history rewrite".  I'll give you history
rewrite, but you need to understand what it means.  I think the current
BK users get it.  I think the BK future users don't get that it is all
one big replicated distributed slightly (or not so slightly) out of sync
database that wants to sync up when it can.  So if you rewrite history,
BK has no way of knowing that you did that.  I suppose we could teach
though so that it would reject the uncollapsed changes but that has its
own issues.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Thu, Jan 31, 2002 at 02:50:31AM -0500, Rob Landley wrote:
> On Thursday 31 January 2002 01:37 am, Larry McVoy wrote:
> 
> > That said, I'm sympathetic to the "I make lotso changes and I want to
> > collapse them into one big change" problem.  It's certainly technically
> > possible to make BK do that, but then you have to *know* that nobody
> > else has a BK repo with your old detailed changes in it, or if they
> > do, they won't ever try to push them back to you (or Linus or ...).
> 
> No, bitkeeper simply has to know. :)
> 
> Put in a node that says "this change collapses this range of other changes" 
> with a range or list of change IDs, and then when you do your next merge with 
> another tree, bitkeeper has the info it needs to avoid sucking in dupes.  If 
> the node says you have that change already, you don't need to suck it in from 
> the other tree.
> 
> > It's not an error if they do, it's just that BK will view them as
> > different changes and automerge them right back into the history.
> > So then you'll have both the collapsed version and the detailed version
> > which puts you worse off than when you started.
> 
> Just teach BK that the collapsed version includes everything in the detailed 
> version.  (Even if that's not technically true, teaching one system to lie to 
> another is an important part of programming... :)  Linus wanted checkpoint 
> functionality to limit backmerges, this seems sort of related-ish.  
> (Boundaries on change sets, merging change sets...)
> 
> Is there an implementation reason why this is particularly hard, or some evil 
> nasty side effects to such an approach that we should know about?


Can you detect the 'collapsed vs full version' thing, and force it to be 
a merge conflict? That, and working LOD support would probably get most 
of what I want (until I try the new version and find more stuff I want 
:P)

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> Can you detect the 'collapsed vs full version' thing, and force it to be 
> a merge conflict? That, and working LOD support would probably get most 
> of what I want (until I try the new version and find more stuff I want 
> :P)

Are you sure you want that?  If so, that would work today, it's about a
20 line script.  You clone the tree, collapse all the stuff into a new
changeset, and pull.  It will all automerge.  But now you have the detailed
stuff and the non-detailed stuff in the same tree, which I doubt is what
you want.  I thought the point was to remove information, not double it.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Troy Benjegerdes]

On Thu, Jan 31, 2002 at 09:19:14AM -0800, Larry McVoy wrote:
> On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> > Can you detect the 'collapsed vs full version' thing, and force it to be 
> > a merge conflict? That, and working LOD support would probably get most 
> > of what I want (until I try the new version and find more stuff I want 
> > :P)
> 
> Are you sure you want that?  If so, that would work today, it's about a
> 20 line script.  You clone the tree, collapse all the stuff into a new
> changeset, and pull.  It will all automerge.  But now you have the detailed
> stuff and the non-detailed stuff in the same tree, which I doubt is what
> you want.  I thought the point was to remove information, not double it.

Well, what I meant was have some kind of pointer in the collapsed stuff 
that conflicts with the detailed stuff, and requires the user to pick 
which on they want. Ideally, this could default to user picks, but a 
repository policy of 'only take collapsed versions' could be used for 
upstream trees, say like linuxppc_2_4.  (linuxppc_2_4_devel could take 
detailed versions).

-- 
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's 
why I draw cartoons. It's my life." -- Charles Schulz

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

cc list trimmed.

On Thu, 31 Jan 2002 09:19:14 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
>> Can you detect the 'collapsed vs full version' thing, and force it to be 
>> a merge conflict? That, and working LOD support would probably get most 
>> of what I want (until I try the new version and find more stuff I want 
>> :P)
>
>Are you sure you want that?  If so, that would work today, it's about a
>20 line script.  You clone the tree, collapse all the stuff into a new
>changeset, and pull.  It will all automerge.  But now you have the detailed
>stuff and the non-detailed stuff in the same tree, which I doubt is what
>you want.  I thought the point was to remove information, not double it.

That sounds almost like what I was looking for, with two differences.

(1) Implement the collapsed set so bk records that it is equivalent to
    the individual patchsets.  Only record that information in my tree.
    I need the detailed history of what changes went into the collapsed
    set, nobody else does.

(2) Somebody else creates a change against the collapsed set and I pull
    that change.  bk notices that the change is again a collapsed set
    for which I have local detail.  The external change becomes a
    branch off the last detailed patch in the collapsed set.

Example.

I have individual changes c1-c17 which are not externally visible.

Tell bk to generate collapsed patch A from c1-c17.  A is externally
visible, without the detailed internal change history of c1-c17.  This
is the equivalent of exporting a patch but it is recorded in bk.

I continue development with c18 onwards, based off c17.

Somebody makes change B against A.  B is externally visible.

I pull B.  bk recognises that B is against A for which local data
exists and therefore B is not really against A but is against c17.
bk creates B as a branch against c17, in parallel with c18.

Outside world sees A->B.  I see A[c1-c17], c17->c18 ..., c17->B (two
branches).

That processing model hides all the backtracking and partial checkins
from the outside world, which only sees the final patchset A, no
information overload.  It allows me to continue with internal
development with all the information that I need.  And it allows me to
automatically take back changes, identify that the changes are in
parallel to my internal changes and merge while keeping local details.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
> That sounds almost like what I was looking for, with two differences.
> 
> (1) Implement the collapsed set so bk records that it is equivalent to
>     the individual patchsets.  Only record that information in my tree.
>     I need the detailed history of what changes went into the collapsed
>     set, nobody else does.
> 
> (2) Somebody else creates a change against the collapsed set and I pull
>     that change.  bk notices that the change is again a collapsed set
>     for which I have local detail.  The external change becomes a
>     branch off the last detailed patch in the collapsed set.

This is certainly possible to do.  However, unless you are willing to fund
this development, we aren't going to do it.  We will pick up the costs of
making changes that you want if and only if we have commercial customers
who want (or are likely to want) the same thing.  Nothing personal, it's
a business and we make tradeoffs like that all the time.

Collapsing is relatively easy, it's tracking the same content in two
different sets of deltas which is hard to get exactly correct.  Certainly
possible but I can visualize what it would take and it would be messy and
disruptive to the source base for an obscure feature that is unlikely to
be used.

Why don't you actually use BK for a while and see if you really think
you need this feature.  The fact that our customers aren't clamoring for
it should tell you something.  They do work as hard and on as much code
(in many cases on the same code) as you do.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 17:04:28 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
>> That sounds almost like what I was looking for, with two differences.
>> 
>> (1) Implement the collapsed set so bk records that it is equivalent to
>>     the individual patchsets.  Only record that information in my tree.
>>     I need the detailed history of what changes went into the collapsed
>>     set, nobody else does.
>> 
>> (2) Somebody else creates a change against the collapsed set and I pull
>>     that change.  bk notices that the change is again a collapsed set
>>     for which I have local detail.  The external change becomes a
>>     branch off the last detailed patch in the collapsed set.
>
>This is certainly possible to do.  However, unless you are willing to fund
>this development, we aren't going to do it.  We will pick up the costs of
>making changes that you want if and only if we have commercial customers
>who want (or are likely to want) the same thing.  Nothing personal, it's
>a business and we make tradeoffs like that all the time.

Understood.

>Collapsing is relatively easy, it's tracking the same content in two
>different sets of deltas which is hard to get exactly correct.  Certainly
>possible but I can visualize what it would take and it would be messy and
>disruptive to the source base for an obscure feature that is unlikely to
>be used.
>
>Why don't you actually use BK for a while and see if you really think
>you need this feature.  The fact that our customers aren't clamoring for
>it should tell you something.  They do work as hard and on as much code
>(in many cases on the same code) as you do.

This is the way that I use PRCS now and it fits the diff/patch model
for distributing kernel code that most people are used to, while
reducing the concerns about information overload.

With PRCS I have branches galore with lots of little changes.  The
outside world sees complete patch sets, not the individual changes.
When they send a patch back I work out which internal change it is
against and start a new branch against it.  The downside with PRCS is
that the creation of the patch set and storing on an ftp site is a
manual process, as is identifying which internal change a patch
response is against and starting a new branch against the last internal
change.

If bk could automate the creation and tracking of meta patchsets I
would convert tomorrow, the ability to automatically distribute changes
is what I miss in PRCS.  But if using bk means that I cannot
automatically separate and track the internal and external patches then
there is no benefit to me in converting.  If I have to clone a
repository to roll up internal patches into an external set and I
cannot automatically pull changes against the external set back into my
working repository then bk gives me no advantages.

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:
> On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
> > That sounds almost like what I was looking for, with two differences.
> > 
> > (1) Implement the collapsed set so bk records that it is equivalent to
> >     the individual patchsets.  Only record that information in my tree.
> >     I need the detailed history of what changes went into the collapsed
> >     set, nobody else does.
> > 
> > (2) Somebody else creates a change against the collapsed set and I pull
> >     that change.  bk notices that the change is again a collapsed set
> >     for which I have local detail.  The external change becomes a
> >     branch off the last detailed patch in the collapsed set.
> 
> This is certainly possible to do.  However, unless you are willing to fund
> this development, we aren't going to do it.  We will pick up the costs of
> making changes that you want if and only if we have commercial customers
> who want (or are likely to want) the same thing.  Nothing personal, it's
> a business and we make tradeoffs like that all the time.

I wonder how your commercial customers develop code then. Either each
programmer futzes around in his/her own tree, and then creates a patch (or
some such) for everybody to see (then I don't see the point of source
control as a help to the individual developer), or everybody sees all the
backtracking going on everywhere (in which case the repository is a mostly
useless mess AFAICS).
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Fri, 1 Feb 2002, Horst von Brand wrote:

> I wonder how your commercial customers develop code then. Either each
> programmer futzes around in his/her own tree, and then creates a patch
> (or some such) for everybody to see (then I don't see the point of
> source control as a help to the individual developer), or everybody
> sees all the backtracking going on everywhere (in which case the
> repository is a mostly useless mess AFAICS).

If the object is to minimise confusion by not showing
back-tracked changes, why not simply allow the user
to mark changesets with a "visibility":

1) hidden, for stuff which shouldn't be seen by default,
   like backed out changes, etc..
2) small, individual development steps to achieve a new
   feature
3) normal, the normal commits
4) major (tagged versions ?)

This way the user can select how detailed the overview
of the versions should be.

Also, when viewing a changeset/version of a certain
priority, bitkeeper could optionally "fold in" the
hidden changesets between the last changeset and the
one the user wants to view.

Would this be a workable scheme ?

(keeps the bitkeeper repository intact, can reduce
the confusion)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

2.4.16 cannot connect to www.sun.com

[Joe Wong]

Hello all,

  For some reason after I upgraded to 2.4.16, I cannot connect to
www.sun.com anymore. It also happens on some other sites. Anyone know what
might be the problem? I have no problem using 2.4.7.

TIA.

- Joe

Re: 2.4.16 cannot connect to www.sun.com

[Chris Chabot]

Try echo 0 > /proc/sys/net/ipv4/tcp_ecn

I dont know, but ecn can prevent you from reaching some locations on the 
net.. could be that 2.4.17 turns it on by default.

    -- Chris


Joe Wong wrote:

>Hello all,
>
>  For some reason after I upgraded to 2.4.16, I cannot connect to
>www.sun.com anymore. It also happens on some other sites. Anyone know what
>might be the problem? I have no problem using 2.4.7.
>
>TIA.
>
>- Joe
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>

Re: 2.4.16 cannot connect to www.sun.com

[David Woodhouse]

joewong@tkodog.no-ip.com said:
> 
>   For some reason after I upgraded to 2.4.16, I cannot connect to
> www.sun.com anymore. It also happens on some other sites. Anyone know
> what might be the problem? I have no problem using 2.4.7. 

http://www.tux.org/lkml/#s14-2

--
dwmw2

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 09:30:56AM -0200, Rik van Riel wrote:
> On Fri, 1 Feb 2002, Horst von Brand wrote:
> If the object is to minimise confusion by not showing
> back-tracked changes, why not simply allow the user
> to mark changesets with a "visibility":

That's what LODs do.  You can do all your work in your "branch", when you
are ready, you do a branch-to-branch pull which collapses the view of all
your changesets down to one in the other view.

I'd love it if you could get Linus to buy into this as an acceptable answer.

I do agree that there are times when you really want to collapse a pile
of changes into one and I'm willing to write that code if it becomes
agreed that it is widely useful.  It's maintaining both versions of 
the changes, the collapsed and uncollapsed, that I don't want to do.
That would be a nightmare in the source base and I don't believe there
is substantial real benefit.  Either the changes are valuable or they
aren't.  If they are valuable enough that you want to save them then
you should let the rest of the world see them.  If they aren't, then
they aren't.  I'm sure you can find cases that don't match that view
but I'm equally sure they are a very small percentage.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Fri, 1 Feb 2002 08:43:27 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>I do agree that there are times when you really want to collapse a pile
>of changes into one and I'm willing to write that code if it becomes
>agreed that it is widely useful.  It's maintaining both versions of 
>the changes, the collapsed and uncollapsed, that I don't want to do.

Hand waving solutions without seeing the bk code ...

Don't maintain both versions of the changes.

Everybody except the person with the uncollapsed set only sees the
collapsed set, no problem there.

The person with the uncollapsed set only has one version of the
changes, in the uncollapsed set.

Identify each externally visible patchset with a unique id, you
probably already do this.

Define a meta patchset entry which maps the uncollapsed line of patches
to the collapsed set.  That is, don't duplicate the changes, add a
mapping instead.

The meta entry maps the externally visible patchset to the internal set
by listing just the start and end point of the LOD.  These are the
entries that would be given to export patch.

Add a global repository option, show/hide detail.  The default is show
detail, the current behaviour.

For show detail, you cannot use meta patchset entries.  Everything is
visible to the rest of the world.

For hide detail, you must define meta entries for what you want to be
visible, the rest of the world can only see what you expose.  That is,
you cannot publish some detail entries and some meta entries, you must
choose one or the other at the repository level.

Meta sets are single level, no collapse within collapse.  A meta set is
externally visible, to collapse a meta set into a meta-meta set is
equivalent to rewriting the distributed bk history, don't allow that.

No duplication of changes.  No rewriting of bk history.  Users who want
to hide their detail changes and only expose the result can do so.
Users who only check in working (as opposed to hacking) code are not
affected.  Users who want the extra flexibility incur the cost of
defining meta sets before they can publish anything.

Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rob Landley]

On Friday 01 February 2002 06:30 am, Rik van Riel wrote:
> On Fri, 1 Feb 2002, Horst von Brand wrote:
> > I wonder how your commercial customers develop code then. Either each
> > programmer futzes around in his/her own tree, and then creates a patch
> > (or some such) for everybody to see (then I don't see the point of
> > source control as a help to the individual developer), or everybody
> > sees all the backtracking going on everywhere (in which case the
> > repository is a mostly useless mess AFAICS).
>
> If the object is to minimise confusion by not showing
> back-tracked changes, why not simply allow the user
> to mark changesets with a "visibility":
>
> 1) hidden, for stuff which shouldn't be seen by default,
>    like backed out changes, etc..
> 2) small, individual development steps to achieve a new
>    feature
> 3) normal, the normal commits
> 4) major (tagged versions ?)
>
> This way the user can select how detailed the overview
> of the versions should be.

A workaround, not a fix.

First of all, not everybody's got a 100 gigabyte drive, cable modem, and 
Athlon 900 yet.  You're talking about potentially taking accidental cruft 
from everybody who uses bitkeeper in 5 years.  Some countries still pay for 
data by the byte, and big servers like kernel.org still care about bandwidth 
issues a lot.

Yeah data storage and transfer, and the memory and CPU power to churn through 
it, is getting cheaper all the time.  But we're talking about expending 
resources shoveling information that even the original developer considers 
completely pointless to maintain and propogate.  Your bitkeeper repositories 
could become enormous.  The amount of proagated state would at least by 
multiplied by the number of developers working on the tree who use bitkeeper, 
meaning spreading the use of bitkeeper would have distinct downsides.  The 
result is that maintainers/lieutenants/linus would almost certainly want to 
take a clean patch rather than a bitkeeper cruftball, on the size and 
cleanliness issue alone.

Secondly, it makes Linus's code review job a LOT harder to have unnecessary 
data in his change sets.  And of course you could say "Linus would never have 
to look at that info", but you'd be wrong.  Stupid example: Somebody patches 
a file to include a copy of decss (or encryption code, or the copyrighted 
ramblings of the lawsuit-happy cult of scientology) and then adds another 
patch to revert it before making a small fix to the file.  The bitkeeper 
change now includes legally questionable code in its back-history, a hot 
potato we probably REALLY don't want to be involved with.

You don't have to worry about malicious use to see a problem.  For-profit 
intellectual property companies generally want to be really clear about what 
code they export.  Think about proprietary drivers that get "cleaned up" 
before being released under the GPL, with the removal code licensed from 
third parties, or patented, or still-proprietary code.  If they can't 
collapse changesets, they simply can't use bitkeeper change sets when 
communicating source code with anyone outside their organization.  Ever.  End 
of story.  Who knows what code might slip through otherwise?  They have to 
audit the entire revision history rather than just the patch they mean to 
send.  That's a nightmare.  The lawyers would never approve ANYTHING for 
release, except as a patch file.

So Linus wouldn't want it, for-profit companies wouldn't want to give it to 
Linus, developers on the end of a 56k modem wouldn't want it, and it could 
potentially cost sites kernel.org a lot of extra bandwidth and data storage 
charges.  This is in exchange for a the possibility that some future 
archeologist or grad student might want to mine the history for anecdotes to 
put in the biography of some developer.

(I'm writing a history book, by the way.  I know all about having 10x as much 
detail as you are ever going to use, because I -AM- the researcher.  Go visit 
the charles babbage institute at the university of minnesota sometime.  It's 
great, but it's not actually all that useful unless you really are looking 
for something specific...)

Another random anecdote on the subject...

http://groups.google.com/groups?selm=789312483snz%40unseen.demon.co.uk

> Also, when viewing a changeset/version of a certain
> priority, bitkeeper could optionally "fold in" the
> hidden changesets between the last changeset and the
> one the user wants to view.
>
> Would this be a workable scheme ?

Depends on your definition of "workable".  It's certainly an improvement, but 
it's still not a fix for the fundamental design assumption in bitkeeper.  
When I do NOT want to propagate my internal state to the rest of the world, I 
seem to have no choice in the matter except by NOT using bitkeeper to 
transfer the data.

I agree that the ability to at least hide stuff would be an improvement.  And 
it should be easily implementable in combination with a checkpoint/tagging 
mechanism (it's just a view issue), but it's still working around a fault in 
bitkeeper that it has no real infrastructure for collapsing together change 
sets in a way that intentionally loses information when that really is what 
you want to do.  The most it could do is hide the problem.

> (keeps the bitkeeper repository intact, can reduce
> the confusion)

Yeah.  But the point is that people often honestly do not want to keep the 
bitkeeper repository intact.

Think about partitioning into different repositories, to do development in 
different branches.  If people don't want to be able to partition their data, 
bitkeeper wouldn't have the concept of multiple repositories on the same 
machine, would it?  You can partition data when you store it, but not 
partition it when you propagate it.  (The concept here is "this info does not 
leave my tree".  This is a concept people really do want.)

It's possible this could be handled by doing ALL new development work in a 
seperate repository, and then ONLY exporting patches from that repository and 
never doing bitkeeper repository merges.  But it's still working around a 
design flaw in bitkeeper.  The way bitkeeper is designed forces you to retain 
cruft you don't want, and actively tries to prevent you from being able to 
get rid of any of it.  I.E. there are things you can do with "diff" and 
"patch", fairly easily, which are not possible to do without bypassing 
bitkeeper.

I can see the mindset that went into that, but I disagree with it.

> regards,
>
> Rik

Rob

(P.S.  You know, If larry gave us the code to bitkeeper, I'm sure we could 
add the features we wanted.... :)

(P.P.S.  Yeah, I know the story on that one, just tweaking his nose...)

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rik van Riel]

On Fri, 1 Feb 2002, Rob Landley wrote:

> (P.S.  You know, If larry gave us the code to bitkeeper, I'm sure we
> could add the features we wanted.... :)

Why would he need to "give you the source" when you can just
download it from bitkeeper.com ?

Are you really this much of a lazy bum that you prefer
badmouthing Larry over searching bitkeeper.com for five
minutes and grabbing the bitkeeper source ? ;)

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rob Landley]

On Saturday 02 February 2002 10:03 am, Rik van Riel wrote:

> Are you really this much of a lazy bum that you prefer
> badmouthing Larry over searching bitkeeper.com for five
> minutes and grabbing the bitkeeper source ? ;)

Actually, I got as far as the questionaire and mailback to get the 
registration required to download anything from bitmoover before putting it 
on my "to do" list, and went and looked at the online docs instead.  (I move 
back to Austin soon, I'm kind of swamped...)

I honestly didn't know the source was available.  I vaguely remembered that 
if bitkeeper were to go out of business the source code would become 
available (under the GPL) after a few months, which kind of implied that 
wasn't the case now.

As for bitkeeper's website, it doesn't mention source code on the main page, 
it doesn't mention any so on the downloads page or the questionaire following 
it, the "free software" link from the downloads page mentions lmbench, 
bitcluster, and webroff but -not- bitkeeper, and the reference manual doesn't 
have a mention of source code (if it does, I missed it) until you get to the 
licensing page (documention->reference manual->licensing) which has "the 
bitkeeper license" that says portions of it (two libraries and the installer) 
are available as GPL code, but the rest would only be released as GPL 180 
days after bitkeeper goes out of businesss).

That actually took a little more than five minutes.  They I went and did 
something else... :)

Looking at the license again, section 2B does seem to imply it's possible to 
get non-GPL source code from bitmover (although it doesn't say it's not 
purchased source code).

But looking looking at the license again, this section under "licensee 
obligations" is certainly interesting:

>     (a)   Maintaining Open Logging Feature: You hereby warrant
>             that you will not take any action to disable or oth-
>             erwise interfere with the Open  Logging  feature  of
>             the BitKeeper Software.  You hereby warrant that you
>             will take any necessary actions to ensure  that  the
>             BitKeeper  Software successfully transmits the Meta-
>             data to an Open Logging server within  72  hours  of
>             the  creation of said Metadata.  By transmitting the
>             Metadata to an Open Logging server, You hereby grant
>             BitMover,  or  any other operator of an Open Logging
>             server, permission to republish the Metadata sent by
>             the BitKeeper Software to the Open Logging server.

Meaning if I use my laptop offline for more than three days, I could be in 
violation of the bitkeeper license.  And under section 4.2, you can pay to 
have that requirement removed.

Section 3C also mentions there's ANOTHER license included in the download (a 
sort of terms of use) which, if you don't comply with it, seems to imply your 
access to bitkeeper's public servers could be revoked...  (How does this 
interact with the "bitkeeper must be able to log your activities" 
requirement?  I don't think I want to know...)

I think I'll stick with diff/patch/rcs for now, thanks.

> regards,
>
> Rik

Rob

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 12:11:30PM +0100, Horst von Brand wrote:
> > This is certainly possible to do.  However, unless you are willing to fund
> > this development, we aren't going to do it.  We will pick up the costs of
> > making changes that you want if and only if we have commercial customers
> > who want (or are likely to want) the same thing.  Nothing personal, it's
> > a business and we make tradeoffs like that all the time.
> 
> I wonder how your commercial customers develop code then. Either each
> programmer futzes around in his/her own tree, and then creates a patch (or
> some such) for everybody to see (then I don't see the point of source
> control as a help to the individual developer), or everybody sees all the
> backtracking going on everywhere (in which case the repository is a mostly
> useless mess AFAICS).

You are presupposing that all the developers are checking in many bad changes
and only one good change.  And that all the bad changes are obscuring the
good ones.  That a correct statement of your beliefs?

If so, what you are describing is called "hacking" in the negative
sense of the word, and what my customers do is called "programming".
It's quite rare to see the sort of mess that you described, it happens,
but it is rare.  I don'tknow how else to explain it, but it is not the
norm in the professional world to try a zillion different approaches
and revision control each and every one.

The norm is:
	clone a repository
	edit the files
	modify/compile/debug until it works
	check in
	push the patch up the shared repository
I'm really at a loss as to why that shouldn't be the norm here as well.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Wayne Scott]

From: Larry McVoy <lm@bitmover.com>
> If so, what you are describing is called "hacking" in the negative
> sense of the word, and what my customers do is called "programming".
> It's quite rare to see the sort of mess that you described, it happens,
> but it is rare.  I don'tknow how else to explain it, but it is not the
> norm in the professional world to try a zillion different approaches
> and revision control each and every one.
> 
> The norm is:
> 	clone a repository
> 	edit the files
> 	modify/compile/debug until it works
> 	check in
> 	push the patch up the shared repository

Or they create do a line of development in a repository with commits
and then determine that it wasn't working.  No problem throw it away
and start over from a clean copy.  Since the repositories are
distributed, private branches just disappear if you don't like them.

-Wayne

Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rob Landley]

On Friday 01 February 2002 11:38 am, Larry McVoy wrote:

> You are presupposing that all the developers are checking in many bad
> changes and only one good change.  And that all the bad changes are
> obscuring the good ones.  That a correct statement of your beliefs?
>
> If so, what you are describing is called "hacking" in the negative
> sense of the word, and what my customers do is called "programming".

Designing while coding is not a bad thing.  It's often considerably more 
efficient than spending a bunch of time up front coming up with an ivory 
tower design that doesn't work in practice.  Very few battle plans survive 
the first engagement with the enemy.  (It's nice to HAVE one.  But if you 
can't adapt when you're in the thick of things, you're in trouble.)

> It's quite rare to see the sort of mess that you described, it happens,
> but it is rare.

In your user base, sure.  Because your tool can't handle it.

Please don't confuse effect with cause...

> I don'tknow how else to explain it, but it is not the
> norm in the professional world to try a zillion different approaches
> and revision control each and every one.

The distinction here is "should bitkeeper be of use to the company", or 
"should bitkeeper be of use to the individual developer".  Bitkeeper is aimed 
at only serving ONE of those two niches right now (as I'll explain in a 
moment), and one thing to realise is that the development teams at companies 
are made up of individual developers who might find bitkeeper significantly 
more useful if it met more of their needs.

> The norm is:
> 	clone a repository
> 	edit the files
> 	modify/compile/debug until it works
> 	check in
> 	push the patch up the shared repository
> I'm really at a loss as to why that shouldn't be the norm here as well.

You'll notice that bitkeeper is totally useless between the clone and the 
check in, in your model.  It can't really be used DURING development, only 
before and after.

You may not notice this so much in the Fortune 500.  I've worked for some 
really big companies.  Your commercial developers are often implementing to a 
spec that was argued over in meetings, with donuts, for weeks before an 
"implementor" was allowed near a keyboard.  This IS a different development 
model from the way open source generally works. :)  (And the bigger the 
company, the more likely they are to ensure their design is perfected before 
they ever actually try to implement a prototype and find out the whole theory 
doesn't really work in practice.  And you wonder how anybody could ever 
manage to spend a billion dollars a year writing software and still ship 
crap.  But that's a seperate argument.  Neither method is inherently 
superior, and most real world development is a blance between up-front design 
and implementation evolution.  Go too far in either direction and you get 
chaos or stagnation.)

As for a simple example of when your model above breaks down, a lot of 
developers who use things like emacs have their source control system as part 
of their integrated development environment.  When they "save and compile", 
it's checked into the RCS (often with a terse three or four word comment 
that's intended as a label to jog the developer's memory).  For one thing, 
this is a nice backup against stupid things like fat-fingering "gcc blah.c -o 
blah.c", which I personally did first thing in the morning (before the 
caffiene could kick in) a few days ago and set myself back just about a whole 
day because my laptop does NOT have any variant of source control on it.  
(Screaming is certainly a quick and aerobic way to wake up, I will admit 
that.  I wouldn't recommend it though.)

The real reason people go with full-blown source control for their backups 
is so that when they ARE doing modify/compile/debug, they can easily back up 
to what they had an hour ago if they find out they went down a culdesac (as 
happens a lot: not always "I couldn't make this work if I kept going" but 
"wait, there's an easier way to do this").  Of course not all changes in 
direction are clean reverts.  Sometimes you want to save part of your work 
and change the other parts yourself, and it's easier to fix up manually and 
check the fixup in than reimplement just to look like you knew what you were 
doing all along.  And sometimes if it's a bug fix inserting a bunch of 
printfs and test code while actually fixing a multi-stage bug, they don't 
bother rolling back to remove the printfs.  Once they've fixed the bug, it's 
easier to delete the instrumentation and false tries rather than roll back to 
a clean version and recreate the entire (possibly long and complicated) fix 
against the fresh tree.)

The problem is, if they use bitkeeper (with a temporary respository), all 
these temporary commits (debugging tries saved in case they want to roll back 
during development) get propagated into the main repository when they do a 
merge.  They can't tell it "done, okay, squash this into one atomic change to 
check in somewhere else, with the whole change history as maybe one comment".

What you're saying is that people who use the source control tools in a way 
you do not expect them to are bad programmers.  A stylistic difference does 
not equal stupidity.

People sometimes really do want to be able to squash between checkpoints, 
which might just be a case of squashing a scratch repository used to create 
the checkpoint.  (If the scratch repository is deleted in the process, then 
the "merge granular and squashed versions" thing doesn't really arise, so 
possibly people just want to be able to do their development in a temporary 
repository which they squash and delete when they issue a release.  Not a 
100% solution, it means the original developer can't keep the original highly 
granular version around without potentially confusing bitkeeper, but it's 
probably an acceptable solution in 80-90% of cases, and is probably doable 
with a script rather than any real change to bitkeeper.)

You do not seem to be used to implementors doing any real designing, and you 
do not seem to be used to your source control system being used during the 
throes of minute-by-minute development.  (Once again, you seem to be arguing 
against  feedback from the community, by saying "you don't really want to do 
that,  your process is bad and should be adapted to my tools".  I hope this 
is just another miscommunication.  We're trying to say "your tools aren't any 
good at what we're trying to use them for, but they could be."  If you don't 
want your tool to serve a certain group of users, fine.  Your tools may serve 
an existing niche fairly well, but if you'd like to expand to a broader 
audience listening to feedback would be helpful.)

Perhaps some of this is just a documentation issue.  If we can explain what 
we want but can't easily figure out how to get bitkeeper to do it, some kind 
of tutorial or prepackaged scripts might be a good idea...

Rob

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Charles Cazabon]

Rob Landley <landley@trommello.org> wrote:
> 
> The problem is, if they use bitkeeper (with a temporary respository), all 
> these temporary commits (debugging tries saved in case they want to roll back 
> during development) get propagated into the main repository when they do a 
> merge.  They can't tell it "done, okay, squash this into one atomic change to 
> check in somewhere else, with the whole change history as maybe one comment".

Something like:

  bk start-temporary-fork
  [hack hack hack]
  bk commit
  [hack hack hack]
  bk revert 
  [hack hack hack]
  bk commit 
  [hack hack hack]
  bk commit
  bk fold-back-into-main-tree-as-one-atomic-update

?

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                            <linux@discworld.dyndns.org>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Larry McVoy]

On Fri, Feb 01, 2002 at 07:19:28PM -0600, Charles Cazabon wrote:
> Rob Landley <landley@trommello.org> wrote:
> > 
> > The problem is, if they use bitkeeper (with a temporary respository), all 
> > these temporary commits (debugging tries saved in case they want to roll back 
> > during development) get propagated into the main repository when they do a 
> > merge.  They can't tell it "done, okay, squash this into one atomic change to 
> > check in somewhere else, with the whole change history as maybe one comment".
> 
> Something like:
> 
>   bk start-temporary-fork

    bk clone main temporary-fork

>   [hack hack hack]
>   bk commit
>   [hack hack hack]
>   bk revert 

    bk fix -c

>   [hack hack hack]
>   bk commit 
>   [hack hack hack]
>   bk commit
>   bk fold-back-into-main-tree-as-one-atomic-update

    bk push

All exists, works as described, no changes necessary.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Charles Cazabon]

Larry McVoy <lm@bitmover.com> wrote:
> 
>     bk clone main temporary-fork
> 
> >   [hack hack hack]
> >   bk commit
> >   [hack hack hack]
> 
>     bk fix -c
> 
> >   [hack hack hack]
> >   bk commit 
> >   [hack hack hack]
> >   bk commit
> 
>     bk push
> 
> All exists, works as described, no changes necessary.

But will the changes which were committed and then reverted from the
temporary tree show up in the main tree after the "push"?  There should be no
evidence that they ever took place, so as not to clutter Linus' tree with
changes a developer made and had no intention of sending to Linus.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                            <linux@discworld.dyndns.org>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Larry McVoy]

On Fri, Feb 01, 2002 at 06:45:59PM -0500, Rob Landley wrote:
> > The norm is:
> > 	clone a repository
> > 	edit the files
> > 	modify/compile/debug until it works
> > 	check in
> > 	push the patch up the shared repository
> > I'm really at a loss as to why that shouldn't be the norm here as well.
> 
> You'll notice that bitkeeper is totally useless between the clone and the 
> check in, in your model.  It can't really be used DURING development, only 
> before and after.

What nonsense.  Go read the docs.

> As for a simple example of when your model above breaks down, a lot of 
> developers who use things like emacs have their source control system as part 
> of their integrated development environment.  When they "save and compile", 
> it's checked into the RCS (often with a terse three or four word comment 
> that's intended as a label to jog the developer's memory).  

Hey genuis.  Download BK, install it, get into emacs, and type the key
strokes you just described.  What happens?  I'll let you in on a little
secret, smart boy, it does exactly what it should do.  Works just like
it were RCS or SCCS.  It was carefully designed behave exactly like SCCS
so that emacs/make/patch/etc all just know how to use it.  Try patching
a BK repository w/ patch(1) and you'll see

    retrieve file from revision system with lock?

Just works.  Fits neatly with the tools that we all know and love.

So explain to me again how it is that the tool is "totally useless"
when it works *exactly* the way you say it should?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rik van Riel]

On Fri, 1 Feb 2002, Rob Landley wrote:
> On Friday 01 February 2002 11:38 am, Larry McVoy wrote:
>
> > You are presupposing that all the developers are checking in many bad
> > changes and only one good change.  And that all the bad changes are
> > obscuring the good ones.  That a correct statement of your beliefs?
> >
> > If so, what you are describing is called "hacking" in the negative
> > sense of the word, and what my customers do is called "programming".
>
> Designing while coding is not a bad thing.  It's often considerably more
> efficient than spending a bunch of time up front coming up with an ivory
> tower design that doesn't work in practice.  Very few battle plans survive
> the first engagement with the enemy.  (It's nice to HAVE one.  But if you
> can't adapt when you're in the thick of things, you're in trouble.)

You're confusing designing with "coming up with the most
inflexible idea possible".

I believe this says more about your experience (or skills)
with designing than about the practice of coming up with
a design before you start implementing.

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Rob Landley]

On Friday 01 February 2002 06:11 am, Horst von Brand wrote:
> Larry McVoy <lm@bitmover.com> said:
> > On Fri, Feb 01, 2002 at 11:29:58AM +1100, Keith Owens wrote:
> > > That sounds almost like what I was looking for, with two differences.
> > >
> > > (1) Implement the collapsed set so bk records that it is equivalent to
> > >     the individual patchsets.  Only record that information in my tree.
> > >     I need the detailed history of what changes went into the collapsed
> > >     set, nobody else does.
> > >
> > > (2) Somebody else creates a change against the collapsed set and I pull
> > >     that change.  bk notices that the change is again a collapsed set
> > >     for which I have local detail.  The external change becomes a
> > >     branch off the last detailed patch in the collapsed set.
> >
> > This is certainly possible to do.  However, unless you are willing to
> > fund this development, we aren't going to do it.  We will pick up the
> > costs of making changes that you want if and only if we have commercial
> > customers who want (or are likely to want) the same thing.  Nothing
> > personal, it's a business and we make tradeoffs like that all the time.
>
> I wonder how your commercial customers develop code then. Either each
> programmer futzes around in his/her own tree, and then creates a patch (or
> some such) for everybody to see (then I don't see the point of source
> control as a help to the individual developer), or everybody sees all the
> backtracking going on everywhere (in which case the repository is a mostly
> useless mess AFAICS).

Speaking from my experience on OS/2 back at IBM (I.E. about as big as it 
gets), they simply don't mind having amazing amounts of cruft in the database 
dating back to at least the last time they switched source control systems.

Under those circumstances, looking at the code history becomes a major 
archaeological expedition, and you either still have the original 
implementors around who have it all in their heads and don't NEED the 
database, or you have people who just look at the code and try to guess what 
it means, possibly asking colleagues about particularly confusing bits.

IBM's source control system always struck me as a graveyard of old code more 
than anything else.  Lots of the really old stuff was stored offline on tapes 
and CDs filed lockers nobody ever opened, with backup copies at some big "we 
have a vault beneath NORAD" data warehousing company.

That's my impression, anyway.  (A few years out of date now.)  My experience 
with the source control system was that it DID have the complete revision 
history in it going back to the 1980's, but it was far more work than it was 
worth to try to mine through it to find something unless you were 
specifically looking to place blame and prove something wasn't YOUR fault.  
Nobody really ever had time to actually go through it for any other reason, 
we had far too much new stuff backlogged.  (And yeah a lot of changes were 
made where some old timer would pipe up afterwards "we tried that five years 
ago, and it didn't work for the same reason you just noticed".  But this was 
the kind of state that was usefully kept in people's heads, not in the source 
control system.)

Now in an open source, the source control system with history might be a 
useful educational resource.  (Non-commercial developers don't have to hit 
the ground running the way commercial ones do.)  But too much granularity 
would definitely diminish that usefulness.  Flood people with low signal and 
high noise, and only archaeologists will ever care about it.

A system that maintained obscene amounts of granularity but HID it (showing 
you only diffs between release versions unless you specifically asked so see 
more detail) would be a distinct improvement over a system that forces every 
query be an archaeological dig.  But since 99% of the people do NOT want to 
go on an archaeological dig, and since for most things only the most active 
10% of the developers even care about the -pre releases and only have time to 
track/sync with the main releases...

Maintaining the "reverted before it left the original implementors tree" 
state at ALL is clearly more trouble than it's worth.  If the original 
IMPLEMENTOR is likely to delete that after they ship, nobody else should EVER 
care unless they're writing some sort of biography of that developer or 
simply engaged in hero-worship.

I.E. yes, I think we honestly do want an easy way to limit the granularity of 
propogated diffs.  We can do this right now by exporting to patch and 
re-importing, but it seems that if we do, then bitkeeper's sync mechanism 
becomes a problem to be worked around.  I'd say this instance of all-out-war 
between what developers are trying to do and what bitkeeper is trying to do 
highlights a design gap in bitkeeper.

Just my opinion, your mileage may vary. :)

Rob

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Larry McVoy]

On Fri, Feb 01, 2002 at 03:47:16PM -0500, Rob Landley wrote:
> That's my impression, anyway.  (A few years out of date now.)  My experience 
> with the source control system was that it DID have the complete revision 
> history in it going back to the 1980's, but it was far more work than it was 
> worth to try to mine through it to find something ...

bk revtool
double click on the last rev, brings up an annotated listing,
click on a line, brings up the checkin comments for that line,
double click on a line, brings up a changeset viewer showing the
entire patch which contained that line.

Now tell me, Rob, wouldn't you find that useful?  I find it useful,
I find it extremely useful, I can fix bugs a factor of 10 (at least)
faster with it than without.  Not having it is like not having ctags,
how the hell do you find anything?

> specifically looking to place blame and prove something wasn't YOUR fault.  

Only losers need to go look to place blame.  Programmers fix bugs.  This
information dramatically improves your ability to fix bugs.  If you
programmed more and talked less, you might understand this concept.

> I.E. yes, I think we honestly do want an easy way to limit the granularity of 
> propogated diffs.  

So suppose I'm joe developer and I make 10 changesets to add some feature.
Somewhere in there I caused a problem.  The changesets have been collapsed
so I now have to wade through all of them to figure out what the problem is.
If I had all of them, I could 

	bk clone
	while true
	do
		make
		test_for_bug || break
		bk undo -fr+
	done

which quickly walks through all of them and finds me exactly the one that
caused the problem.  

The more I collapse, the harder this is.  But wait, you say, if BK lety
me do it both ways, the original developer has all this detail.  Great,
that means we took a system that could let the whole world figure out the
problem and we made it so only one guy can do it fast.  Not real smart.

I'm sympathetic to the "wait, I made this changeset and then I found
another bug" problem and we have a command that lets you reedit a
changeset.  And we're talking with Linus about making that even more
flexible.  But the rules are you can only change history on things you
haven't propogated.

If you want to help make BitKeeper better, it would be a lot more
productive if you used it and figured out how it really works instead of
critizing how you think it works.  It's a lot closer to what you want than
you think it is, and it is much closer than any other system in the world.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Bitkeeper change granularity (was Re: A modest proposal -- We need a patch penguin)

[Henning P. Schmiedehausen]

Larry McVoy <lm@bitmover.com> writes:

>Only losers need to go look to place blame.  Programmers fix bugs.  This

And this from the guy who loves to blame others for their bugs in public
(... because we did so at SUN) and thought that it improves "team morale".

Isn't this a losers' behaviour?

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: A modest proposal -- We need a patch penguin

[Nix N. Nix]

On Thu, 2002-01-31 at 12:19, Larry McVoy wrote: 
> On Thu, Jan 31, 2002 at 11:13:37AM -0600, Troy Benjegerdes wrote:
> > Can you detect the 'collapsed vs full version' thing, and force it to be 
> > a merge conflict? That, and working LOD support would probably get most 
> > of what I want (until I try the new version and find more stuff I want 
> > :P)
> 
> Are you sure you want that?  If so, that would work today, it's about a
> 20 line script.  You clone the tree, collapse all the stuff into a new
> changeset, and pull.  It will all automerge.  But now you have the detailed
> stuff and the non-detailed stuff in the same tree, which I doubt is what
> you want.  I thought the point was to remove information, not double it.

Sounds to me like you should have the /option/ to double your info,
which does not mean that the whole world should start seeing your stuff
double. You must "fool" the other trees into believing that you are the
second Mozart (you get everything right the first time around) and only
to yourself will you admit that it took 15 different tries and you
dead-ended yourself 15 different ways.  Under these conditions you would
have all your blunders documented, but only for yourself.



Regards.

> -- 
> ---
> Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 10:51 pm, Larry McVoy wrote:

>Sigh.  OK, I'm taking one more pass at trying to get you to see the light
>and then I give up.

And I'm trimming the cc list back a bit...

> > I'm not quite sure how Linus does this, but how I'd do it is [a really
> > complicated solution based on patches that won't work]

A) It's not really all that complicated.  The minimal commands are "untar" 
and "patch".  If you don't feel like doing fixups at all (with a third 
command, an editor), the rejected patches can be bounced.  It does mean 
you're keeping track of patches manually, but I'm just talking about what 
happens between -pre releases, which is not an infinitely large set of 
patches to wrangle...

B) It's a solution that is working, and has been for years.  It certainly has 
its disadvantages, but it does in fact function.  What the existing process 
is used to do on a regular basis is the definition of the minimum level of 
functionality I would expect from bitkeeper.

I'm not defending the patch method.  I'm certainly not advocating it as the 
ultimate solution.  But it has worked for years, and a replacement that can't 
easily do things it can easily do seems flawed to me.

If you'd like to say that I'm wrong about bitkeeper and that it CAN easily do 
the above.  (That's kind of the reply I expected.)  But attacking the 
desirability of doing things that the existing patch managment process 
handles on a fairly regular basis...  Seems counter-productive to me.

> Think.  What you are describing is basically what Linus does today.  And
> noone, including you, is happy.  You're the guy who started this thread.

I wasn't trying to get Linus to use CVS.  That's just a fringe benefit.  I 
was trying to figure out how to deal with patches getting dropped to the 
point people were suggesting automatic remailers.  I didn't directly suggest 
changing what Linus did with patches on his own hard drive.  I went out of my 
way to craft a proposal where he DIDN'T have to change any of that.  I was 
mostly talking about the method by which patches got to him, which is where I 
(right or wrong) percieved the existence of a problem.

You can say I suggested the wrong solution, and you can say I identified the 
wrong problem.  But please don't project your desires onto me and try to use 
that to explain why you think I should be agreeing with you.

To clarify: I'm HAPPY to see Linus giving bitkeeper another try.  I think 
this is an EXTREMELY positive development.  I do NOT want to discourage it.  
I also don't want to see it fail because you expect Linus to adapt to 
bitkeeper rather than trying to understand what Linus does and adapt 
bitkeeper to be a good tool for him.

(Some of these are simple documentation questions.  How does Linus do THIS.  
And I intend to go RTFM when I have time to tackle a new project (I.E. not 
tonight), but I'm mostly following up on your replies to questions other 
people originally asked, which I didn't consider to be a real answer to the 
question.)

> However, what you described *completely* misses the point.

The message you're replying to was answering your question, which to me 
seemed to be you changing the subject form the earlier "people have to send 
linus patch A in order to send him a bitkeeper change set with patch B, even 
if they don't want to".  I wasn't talking about  what goes on within Linus's 
tree, it's about what people send to him.  (Bitkeeper change sets seem to 
have descriptions and potentially automatic notification back to the original 
patch submitter that they got looked at and/or applied, which is a potential 
improvement over raw text patches.  The ordering requirements do not seem to 
me to be a pure improvement, but instead something that takes extra effort to 
work around.)


> Do you start to see the problem?  You were yelling and screaming
> "BitKeeper sucks because it can't take a patch out" when in fact
> it can do exactly what you said it can't.

Yelling and screaming?  Was I?  (I think you read too much into the rant tag. 
 The point of one of those things is "not everybody is expected to agree with 
me on this"...)

There's a difference between "if you can't take the patch out, then maybe 
bitkeeper sucks" and "bitkeeper definitely sucks because I just KNOW you 
can't..."  You seem to have missed the conditional and homed right in on 
defending your baby from hypothetical criticism.  Maybe I'm not expressing 
myself clearly.  It's happened before...

You just answered my question: bitkeeper can take the patch out, the problem 
isn't a real problem.  Thanks, that's what I wanted to know.

> On top of that, what you
> were complaining about isn't the point.  The thing that you say
> BK can't do, but it can, is not what Linus wants.  Not even close.
>
> And you haven't begun to understand that BK is a distributed, replicated
> system.  You can turn all that off and you've got CVS, so turning it
> off isn't an option.

If you turn off the distributed/replicated nature of bitkeeper, it acquires 
problems with file renames and reproducibility?  Without being distributed, 
you have problems with file renames and reproducibility?  Several other 
developers use a CVS or BK repositories which nobody else ever directly 
checks any code into.  They import patches, and then do their own development 
in that tree.  The source management system is purely for their own 
convenience.  Code goes to other developers as unified diffs.

I'm not talking about fun bells and whistles with which Linus could IMPROVE 
his process.  I'm trying to make sure there are no holes that stop him from 
doing what he could do before.  (There SHOULDN'T be.  But if there aren't, 
why did it take so long to adopt?  Seems worth a check.  Several people who 
use bitkeeper have said things along the lines of "I want to do this and 
haven't figured out how", and you actually seem to have replied "no you don't 
really want to do that" on more than one occasion.  I'm hoping this is a 
miscommunication, which is why I'm trying to follow up.  I mostly just wanted 
clarification and reassurance...)

> Leaving it on means that the revision history is replicated.

You seem to have an unquestioning assumption that infinitely replicating the 
revision history is a good thing.

Linus himself seems to have denied this assumption.  Just because a 
downstream developer took two months to come up with a subsystem and did 
eight hundred seperate checkins of which only maybe 20% of the code made it 
into the final version, does NOT mean that Linus cares.  That level of detail 
IS more information, but there's a limit to how much information you want 
before it becomes cruft in Linus's tree.

> So it isn't an option to collapse a pile of changes into a smaller pile,
> the bigger pile will come back the next time he updates
> with the other tree.

And I'm saying I'm not convinced this is necessarily a good thing.

> And once again, you come back with another post that shows you just want
> to yell

I occasionally use caps to emphasize a word because you can't put italics or 
underlines in non-html email.  I do not however PUT MULTIPLE CAPITALIZED 
WORDS TOGETHER.  That is shouting, yes...)

> and you haven't thought it through, into my kill file you go,

By all means.  I've made it back into Al Viro's now.  (I'm a bit confused how 
I managed to get OUT of it, but I probably changed email addresses...)

Generally, I don't consider putting my fingers in my ears and going "la la la 
I can't hear you" to be the logical equivalent to proving your point in an 
argument.  When I don't feel progress can be made, I generally just stop 
replying.

> my blood pressure goes down, you get to yell all you want, everybody
> is happy.  I'm willing to try and make BK do what is needed here; I'm
> not willing to tolerate people who don't think.

And of course the other person not seeing your point is always entirely 
because they're all not thinking?  There's no possibility of legitimate 
miscommunication, or legitimate difference of opinion about anything?

That must save time.

Rob

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 00:16:13 -0500, 
Rob Landley <landley@trommello.org> wrote:
>Linus himself seems to have denied this assumption.  Just because a 
>downstream developer took two months to come up with a subsystem and did 
>eight hundred seperate checkins of which only maybe 20% of the code made it 
>into the final version, does NOT mean that Linus cares.  That level of detail 
>IS more information, but there's a limit to how much information you want 
>before it becomes cruft in Linus's tree.

I agree with Rob on this.  My PRCS tree for 2.4 has 950+ patch sets in
it but there is no way I would inflict that level of detail on the rest
of the world.  I follow the model of check in often, even if it does
not work, so I can backtrack and take another branch if the code hits a
dead end.

Lots of small checkins and branching means a lot of history which is
useful to me but to nobody else.  When I release a patch I pick a start
point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  That single
patch against 2.4.17 is all the outside world needs to see, the only
history is "kdb v2.1 2.4.17 common-2".  It does not matter if I
backtracked and discarded some twigs to get to that final end point,
the rest of the world only cares about the end point.

For that model to work (which is effectively what diff/patch does now),
developers need a repository system that can consolidate lots of little
changes into a single patchset.  Distribute and replicate the single
patchset, only the original developer retains the individual steps that
made up the combined patchset.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
> When I release a patch I pick a start
> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  

bk export -tpatch -r17.1,17.37

Does exactly the same thing.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 21:55:23 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
>> When I release a patch I pick a start
>> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
>> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  
>
>bk export -tpatch -r17.1,17.37
>
>Does exactly the same thing.

Now you've confused me :).  Does that replicate the history or not?

I know that bk can generate a patch which is fine for people not using
bk, but one of the selling points of bk is the ability to replicate
history entries.  My point is that full replication of history may be
too much detail for anybody except the original developer.  If bk can
consolidate a series of patchsets into one big patchset (not patch)
which becomes the unit of distribution then the problem of too much
history can be solved.

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Thu, Jan 31, 2002 at 05:03:11PM +1100, Keith Owens wrote:
> On Wed, 30 Jan 2002 21:55:23 -0800, 
> Larry McVoy <lm@bitmover.com> wrote:
> >On Thu, Jan 31, 2002 at 04:46:43PM +1100, Keith Owens wrote:
> >> When I release a patch I pick a start
> >> point (base 2.4.17, patch set 17.1) and an end point (kdb v2.1 2.4.17
> >> common-2, patchset 17.37) and prcs diff -r 17.1 -r 17.37.  
> >
> >bk export -tpatch -r17.1,17.37
> >
> >Does exactly the same thing.
> 
> Now you've confused me :).  Does that replicate the history or not?

Nope.

> I know that bk can generate a patch which is fine for people not using
> bk, but one of the selling points of bk is the ability to replicate
> history entries.  

Yup.

> My point is that full replication of history may be
> too much detail for anybody except the original developer.  If bk can
> consolidate a series of patchsets into one big patchset (not patch)
> which becomes the unit of distribution then the problem of too much
> history can be solved.

If all you mean is that you don't want to have to tell it what to send,
yes, it does that automatically.  If you start with 100 changes, 
I clone your tree, you add 200 more, all I do to get them is say

	bk pull

it will send them all, quickly (works very nicely over a modem or 
a long latency link like a satellite).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 22:07:20 -0800, 
Larry McVoy <lm@bitmover.com> wrote:
>On Thu, Jan 31, 2002 at 05:03:11PM +1100, Keith Owens wrote:
>> My point is that full replication of history may be
>> too much detail for anybody except the original developer.  If bk can
>> consolidate a series of patchsets into one big patchset (not patch)
>> which becomes the unit of distribution then the problem of too much
>> history can be solved.
>
>If all you mean is that you don't want to have to tell it what to send,
>yes, it does that automatically.  If you start with 100 changes, 
>I clone your tree, you add 200 more, all I do to get them is say
>
>	bk pull
>
>it will send them all, quickly (works very nicely over a modem or 
>a long latency link like a satellite).

AFAICT this is the heart of Rob's problem.  He (and I) do not want you
to see all 200 changes.  Some changes are dead ends, some are temporary
bug fixes that I know will be replaced later, IOW they are my jottings,
not for public release.  Replicating the lot to everybody is just
polluting the other trees.

OTOH if I can tell bk :-

* Take all the changes in the direct line from 17.1 to 17.37.
* Ignore any extraneous branches off that line.
* Ignore other changes that were applied in the same time period but
  are not on the direct line, I am also making changes to 2.4.18-pre6
  at the same time.
* Generate a consolidated patchset which is visible to the outside
  world.
* Hide anything not explicitly marked as visible.
* When the consolidated patchset comes back from the master tree,
  recognise that it is equivalent to 17.1 through 17.37 on my tree,
  even though nobody else has the individual changes.

Then I can choose to make kdb v2.1 2.4.17 common-2 visible as an
entity (17.1 to 17.37), without telling everybody else what other
changes are going on in my tree.  bk pull only sees the consolidated
changes I want to make visible.

Re: A modest proposal -- We need a patch penguin

[Kenneth Johansson]

Larry McVoy wrote:

> On Wed, Jan 30, 2002 at 02:17:05PM -0800, Linus Torvalds wrote:
> > The way BK works now, if we call the quick-and-dirty fix "A", and the real
> > fix "B", the developer has a really hard time just sending "B" to me. He'd
> > have to re-clone an earlier BK tree, re-do B in that tree, and then send
> > me the second version.
> >
> > I'm suggesting that he just send me B, and get rid of his tree. There are
> > no dependencies on A, and I do not want _crap_ in my tree just because A
> > was a temporary state for him.
>
> And you just lost some useful information.  The fact that so-and-so did
> fix A and then did B is actually useful.  It tells me that A didn't work
> and B does.  You think it's "crap" and by tossing it dooms all future
> developers to rethink the A->B transition.
>

I think Linus meant that A never got sent out before the developer did the B
version. Now A could be a even bigger bug than what it was intended to fix so the
developer really dont wan't the world to se that sucker but can't just send the B
changeset as it depends on A. So I guess he needs a easy way to make A just go
away. Basically just collaps A and B into the same changeset. This should probably
ony work on changeset that has not been pushed to other trees.

Re: A modest proposal -- We need a patch penguin

[Tom Rini]

On Thu, Jan 31, 2002 at 09:09:06AM +0100, Christoph Rohland wrote:
> Hi Linus,
> 
> On Wed, 30 Jan 2002, Linus Torvalds wrote:
> > it would see how far back it can go with an automatic merge and add
> > "d" at the _furthest_ point possible. 
> 
> No, I would prefer a way where the developer gives the merge point and
> bk checks if it merges cleanly. Else it is too easy to have merge
> points which are semantically wrong.

Well, provided the 'backmerge' respects tag, or certain kinds of tags
(ie the tree is 'soft tagged' as v2.5.4-pre3, v2.5.4-pre2, v2.5.4-pre1
and 'hard tagged' as v2.5.3.  'backmerge' will attempt to move a change
back only as far as v2.5.3, since v2.5.3 had an API change here.

Or the other option, since this isn't the _default_ behavior, but an
optional one is to give backmerge a 'don't go past tag' since the
developer should be aware that the API changed at v2.5.3 or v2.5.4-pre2
and even tho the change might apply cleanly further back, since it's
updating the driver to the new API, don't try anyways.)

-- 
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Larry McVoy wrote:

> and then you added one change below that, multiple times.  If you were to
> combine all of those changes in a BK tree, it would look like
>
> 			[older changes]
> 			      v
> 			  [2.5.3-pre4]
> 			      v
> 			  [2.5.3-pre5]
>   [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]

I'm porting rmap to 2.5 now, doing just this.

One thing I noticed was that the space usage of all the
bk trees I'm using in order to keep the different changes
individually pullable is about 1.5 GB now.

Not too big an issue for me, but it might be an issue if
every bkbits.net user starts doing this ... ;)

cheers,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Tue, Feb 12, 2002 at 08:59:34PM -0200, Rik van Riel wrote:
> One thing I noticed was that the space usage of all the
> bk trees I'm using in order to keep the different changes
> individually pullable is about 1.5 GB now.

We just need the "bk relink" command and that should be ~100M or so.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Feb 12, 2002  20:59 -0200, Rik van Riel wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
> > and then you added one change below that, multiple times.  If you were to
> > combine all of those changes in a BK tree, it would look like
> >
> > 			[older changes]
> > 			      v
> > 			  [2.5.3-pre4]
> > 			      v
> > 			  [2.5.3-pre5]
> >   [sched1] [sched2] [sched3] [sched4] [sched5] [sched6] [sched7]
> 
> I'm porting rmap to 2.5 now, doing just this.
> 
> One thing I noticed was that the space usage of all the
> bk trees I'm using in order to keep the different changes
> individually pullable is about 1.5 GB now.

Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
that since the rmap changes are fairly localized that you would only
get multiple copies of a limited number of files, and it wouldn't
increase the size of each repository very much.

As Larry mentioned, you could re-merge these trees.  The following script
will probably be enough, since we don't want/need to compare all files
in each tree, only SCCS and BitKeeper files that are in the same place
in the heirarchy.  Very lightly tested - Larry will have to tell me if
it is OK to hard-link everything in SCCS and BitKeeper repositories,
or if there are some files that should be ignored.

On my e2fsprogs tree, it takes 12s to relink a clone to its parent,
saving 12/19MB = 63% reduction in space per clone.

Cheers, Andreas
=========================== bkrelink =====================================
#!/bin/sh
# A script to relink files in BitKeeper repositories if they were not
# created with "bk clone -l" or if the same changes were made to both
# repositories.
#
# Andreas Dilger <adilger@turbolabs.com>  02/12/2002

PROG=bkrelink

usage() {
	echo "usage: $PROG <parent BK tree> <clone BK tree>" 1>&2 && exit 1
}

[ $# -ne 2 ] && usage
[ ! -d "$1/BitKeeper" ] && usage
[ ! -d "$2/BitKeeper" ] && usage

PTREE=$1
CTREE=$2

#DEBUG=1
say() {
	[ "$DEBUG" ] && echo "$*"
	return 0
}

do_link() {
	echo "$PROG: hard-linking $2 to $1"
	ln -f $1 $2
}

# We need to do some ugly things with the find processes to keep the relative
# paths correct in each tree.  Likewise, | read will run in a separate process
# so we need to do the checks in a subshell so all the stat fields are set.
(cd $CTREE
say "$PROG: finding in $CTREE" 1>&2
find . -type d \( -name BitKeeper -o -name SCCS \) ) | while read DIR; do
(cd $CTREE/$DIR
say "$PROG: looking in $CTREE/$DIR" 1>&2
find . -type f ) | while read FILE; do
 	PFILE=$PTREE/$DIR/$FILE
 	CFILE=$CTREE/$DIR/$FILE

	say "$PROG: checking $CFILE, $PFILE"

	[ ! -f "$PFILE" ] && say "$PROG: $PFILE not found" && continue
	[ ! -f "$CFILE" ] && say "$PROG: $CFILE not found" && continue

	[ "$DEBUG" ] && stat -t $PFILE && stat -t $CFILE
	stat -t $PFILE | { read JNK PSZ JNK JNK PUSR PGRP PDEV PINO PLINK JNK
	stat -t $CFILE | { read JNK CSZ JNK JNK CUSR CGRP CDEV CINO CLINK JNK

	# do the easy test (size compare) first
	[ $CSZ != $PSZ ] && say "size mismatch: $CSZ != $PSZ" && continue
	# can't hard link across devices
	[ $CDEV != $PDEV ] && say "dev mismatch: $CDEV != $PDEV" && continue
	# already hard linked (same device number, same inode numer)
	[ $CINO == $PINO ] && say "ino match: $CINO == $PINO" && continue
	[ $CUSR != $PUSR ] && say "user mismatch: $CUSR != $PUSR" && continue
	[ $CGRP != $PGRP ] && say "group mismatch: $CGRP != $PGRP" && continue

	#echo "$PROG: comparing $CFILE, $PFILE"

	cmp --quiet $CFILE $PFILE || continue

	# We try to have only a single target against which we link.
	# If in doubt, move links towards the specified parent.
	if [ $CLINK -eq 1 ]; then
		do_link $PFILE $CFILE
	elif [ $PLINK -eq 1 ]; then
		do_link $CFILE $PFILE
	else
		do_link $PFILE $CFILE
	fi
	}
	}
done
done
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 12 Feb 2002, Andreas Dilger wrote:

> Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
> that since the rmap changes are fairly localized that you would only
> get multiple copies of a limited number of files, and it wouldn't
> increase the size of each repository very much.

the problem is, i'd like to see all these changes in a single tree, and
i'd like to be able to specify whether two changesets have semantic
dependencies on each other or not. BK would still enforce 'hard
orthogonality' - ie. two changesets that change the same line of code
cannot be defined as nondependent on each other, BK should refuse the
checking in of such a changeset. The default dependency should be
something like 'this changeset is dependent on all previous changesets
committed to this repository' - but if the developer wants it then it
should be possible to un-depend two changesets.

it's also true in the other direction: two changesets that have no hard
conflicts could still have semantic dependencies, it's the responsibility
of the developer.

(detail: it might even be possible to define two changesets as orthogonal
even if there are hard conflicts between them. For this to work the
developer has to provide the correct merge in both directions. If that is
done then BK should allow to make the two changesets independent.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Feb 13, 2002  13:07 +0100, Ingo Molnar wrote:
> On Tue, 12 Feb 2002, Andreas Dilger wrote:
> > Is this using "bk clone -l" or just "bk clone"?  I would _imagine_
> > that since the rmap changes are fairly localized that you would only
> > get multiple copies of a limited number of files, and it wouldn't
> > increase the size of each repository very much.
> 
> the problem is, i'd like to see all these changes in a single tree, and
> i'd like to be able to specify whether two changesets have semantic
> dependencies on each other or not.

Oh, I agree.  My response was only to Rik's mention that his multiple trees
take up too much space.  I would personally also want to be able to separate
independent changes out of my tree rather than having many repositories.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

We need to move from discussing whether Linus can scale to whether the 
Linux Community can scale.

Every organization needs to have clearly defined algorithms for 
determining what work is done by who.  For the linux community, our work 
consists in part of reviewing patches.  Incoherent inconsistent 
delegation is the only reason why we are having scaling problems.  We 
have a consistent recurring problem (yes, I know, a few lucky folks like 
me don't have this problem, but it is clear to see that WE as a 
community have this problem).

It is important that there be  a consistent feeling among patch 
submitters that they know where to send their patches for 
acceptance/rejection.  There should be NO patches which go out, and not 
even a rejection comes back.  

Every organization has clearly defined procedures for allocating the 
flow of work.  It is called a management structure.  That is what we 
need, and we need a formal well defined and externally visible one.  An 
informal undefined network of friends is just not suitable for an 
organization where the flow of email is as large as it is in the Linux 
Community.

Linus, I would like you to stop saying that you cannot scale to where 
you can read every email, and start determining what it takes to make 
the Linux Community infrastructure underneath you responsive to patches. 
 Bitkeeper is a great start, but you also need to create  a management 
structure and interface that is clearly defined to the external 
community.  Saying that the maintainers list is ignored by you means 
that you need to create something that is not ignored by you.  You also 
need to create a system (bitkeeper can perhaps help, Larry?) for 
tracking who fails to respond to patches, and (after a few warnings) 
remove them as maintainers.  

Our problems are not novel.  Let us apply standard business school 
methodologies to them.

Hans

Re: A modest proposal -- We need a patch penguin

[Mark Hahn]

On Fri, 22 Feb 2002, Hans Reiser wrote:
...
> Our problems are not novel.  Let us apply standard business school 
> methodologies to them.

eh?  "creative destruction"?  "disruptive techologies"?  
hmm, maybe it should have been *factory* vs bazaar.

Re: A modest proposal -- We need a patch penguin

[Randy.Dunlap]

On Fri, 22 Feb 2002, Hans Reiser wrote:

| We need to move from discussing whether Linus can scale to whether the
| Linux Community can scale.

I have to agree with much of what Hans has written here.

and one of the biggest things that would help in this regard, IMHO,
is to (dare I say "require") provide documentation for kernel
API changes or semantics.  "Read the source" or "Use the source"
doesn't scale well either, when 10K kernel developers are
trying to use a new widget in 2.5.4, but they all ask questions
on lkml about how it's done.

Let's keep Documentation/* stuff up to date.  Whether it's in
text or DocBook format doesn't matter much.
Or have web pages for it if that's preferred by the project or
individual(s).

  ~Randy

| Every organization needs to have clearly defined algorithms for
| determining what work is done by who.  For the linux community, our work
| consists in part of reviewing patches.  Incoherent inconsistent
| delegation is the only reason why we are having scaling problems.  We
| have a consistent recurring problem (yes, I know, a few lucky folks like
| me don't have this problem, but it is clear to see that WE as a
| community have this problem).
|
| It is important that there be  a consistent feeling among patch
| submitters that they know where to send their patches for
| acceptance/rejection.  There should be NO patches which go out, and not
| even a rejection comes back.
|
| Every organization has clearly defined procedures for allocating the
| flow of work.  It is called a management structure.  That is what we
| need, and we need a formal well defined and externally visible one.  An
| informal undefined network of friends is just not suitable for an
| organization where the flow of email is as large as it is in the Linux
| Community.
|
| Linus, I would like you to stop saying that you cannot scale to where
| you can read every email, and start determining what it takes to make
| the Linux Community infrastructure underneath you responsive to patches.
|  Bitkeeper is a great start, but you also need to create  a management
| structure and interface that is clearly defined to the external
| community.  Saying that the maintainers list is ignored by you means
| that you need to create something that is not ignored by you.  You also
| need to create a system (bitkeeper can perhaps help, Larry?) for
| tracking who fails to respond to patches, and (after a few warnings)
| remove them as maintainers.
|
| Our problems are not novel.  Let us apply standard business school
| methodologies to them.

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 25 February 2002 12:13 pm, Randy.Dunlap wrote:
> On Fri, 22 Feb 2002, Hans Reiser wrote:
> | We need to move from discussing whether Linus can scale to whether the
> | Linux Community can scale.
>
> I have to agree with much of what Hans has written here.
>
> and one of the biggest things that would help in this regard, IMHO,
> is to (dare I say "require") provide documentation for kernel
> API changes or semantics.  "Read the source" or "Use the source"
> doesn't scale well either, when 10K kernel developers are
> trying to use a new widget in 2.5.4, but they all ask questions
> on lkml about how it's done.
>
> Let's keep Documentation/* stuff up to date.  Whether it's in
> text or DocBook format doesn't matter much.
> Or have web pages for it if that's preferred by the project or
> individual(s).

Random comment:

It's design documentation that's needed.  Looking at the code you can see 
what it's doing, but you sometimes have to read an amazing amount of it to 
even guess at WHY...  The code doesn't always tell you about the author's 
intentions, just the implementation.  And sometimes the code is wrong 
anyway...

I believe the kernelnewbies project is working on this, but haven't been able 
to follow it.  (I just moved back to Austin, am recovering from food 
poisioning, picking an old job back up...  Not following much of anything at 
the moment...)

>   ~Randy

Rob

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Rob Landley wrote:
>
>>Let's keep Documentation/* stuff up to date.  Whether it's in
>>text or DocBook format doesn't matter much.
>>Or have web pages for it if that's preferred by the project or
>>individual(s).
>>
> 
> Random comment:
> 
> It's design documentation that's needed. 

Another random comment:

1. docs.sun.com, ssh sunsite.prv ... man system-xxx
2. AT&T The UNIX operating system.
3. Watch out at O'REILY.

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

I'm on the road, so this reply probably won't go out for a couple days, but 
I'll queue it in my outbox anyway...

On Friday 22 February 2002 11:06 am, Hans Reiser wrote:
> We need to move from discussing whether Linus can scale to whether the
> Linux Community can scale.
>
> Every organization needs to have clearly defined algorithms for
> determining what work is done by who.  For the linux community, our work
> consists in part of reviewing patches.  Incoherent inconsistent
> delegation is the only reason why we are having scaling problems.  We
> have a consistent recurring problem (yes, I know, a few lucky folks like
> me don't have this problem, but it is clear to see that WE as a
> community have this problem).
>
> It is important that there be  a consistent feeling among patch
> submitters that they know where to send their patches for
> acceptance/rejection.  There should be NO patches which go out, and not
> even a rejection comes back.

This is what the various patchbot projects are trying to address.  
(Unfortunately, they seem to have gotten the idea that they need a complete 
solution to all possible cases before deploying anything.  Just a filtered 
patches-only mailing list would be a start.  I'd put one up if I had a 
server, but as I said I'm moving this month.  If nobody else has done it by 
march, I might.)

The current Linux community structure seems to be four tiers.  Developers, 
maintainers, lieutenants, and linus.  Linus listens to lieutenants, 
lieutenants accept from maintainers, and maintainers accept from developers.

The confusion seems to be that until recently, many maintainers didn't know 
who the lieutenants were (who the people Linus actually listens to are, to at 
the very least explicitly reject patches once these people have reviewed, 
approved, and forwarded them).  Hence stuff was getting to maintainers and 
then being dropped when forwarded straight to Linus.  Linus still hasn't 
quite enumerated his lieutenants, but now that people know they exist I 
expect they'll become apparent eventually...

> Every organization has clearly defined procedures for allocating the
> flow of work.  It is called a management structure.  That is what we
> need, and we need a formal well defined and externally visible one.  An
> informal undefined network of friends is just not suitable for an
> organization where the flow of email is as large as it is in the Linux
> Community.

It's not a binary state.  The fact we need a little more structure doesn't 
mean anybody has to start filling out paperwork and blindly following 
procedures. :)

The "a little more structure" could be a "how to submit patches" FAQ entry 
that says:

1) Develop patch, test, get community feedback if necessary to make sure it 
works.
2) Submit to maintainer, get them to review and sign off.  Resolve any issues 
they have before proceeding.
3) Submit to lieutenant (the maintainer will tell you who this is), get them 
to review and sign off.  Resolve any issues lieutenant has before proceeding.
4) Submit to Linus, with appropriate endorsements.

If Linus ignores everybody except Lieutenants, that's probably workable as 
long as he DOESN'T ignore the lieutenants and people know who the lieutenants 
are, and the lieutenants don't ignore the maintainers and the maintainers 
don't ignore the developers.

If Linus has two levels of sturgeon's law filters (maintainers and 
lieutenants) before he has to explicitly reject something, then asking him 
(nicely) to at least reply thumbs up/thumbs down on patches forwarded to him 
("bad idea", "fix this", "do it this way instead") by the dozen or so people 
he trusts shouldn't overload his bandwidth.  (Whether or not he'd actually do 
it is still up to him, but that strikes me as the minimum workable long-term 
solution.)

So a developer would at least know who they have to please next (maintainer, 
lieutenant, or linus) to forward their patch.  It still might be a lot of 
work to go through the long way, and Linus would probably still accept 
interesting patches directly.  But the failure case of "my patch is getting 
ignored" would have a procedure to go through to get explicitly rejected by 
the appropriate person. :)

By the way, sometimes the answer honestly is "I'm busy, submit again after 
2.5.7".  Which is still better than being ignored.  (Stuff like the ALSA 
drivers: "Good idea, not now."  Ok: When?)

> Linus, I would like you to stop saying that you cannot scale to where
> you can read every email, and start determining what it takes to make
> the Linux Community infrastructure underneath you responsive to patches.

Linus (and the lieutenants under him) have not been the ones experiencing the 
problem.  Linus accepts all the patches he wants to, and the lieutenants tend 
not to be ignored.  The top of the pyramid is not where the motivation for 
change is most strongly felt...

>  Bitkeeper is a great start, but you also need to create  a management
> structure and interface that is clearly defined to the external
> community.  Saying that the maintainers list is ignored by you means
> that you need to create something that is not ignored by you.  You also
> need to create a system (bitkeeper can perhaps help, Larry?) for
> tracking who fails to respond to patches, and (after a few warnings)
> remove them as maintainers.

I don't expect Linus needs to do any of the grunt work here.  He just needs 
to sign off on the design and actually use the finished solution.

We've got some time on this.  If Bitkeeper allows Linus to move to a "pull" 
model with his lieutenants, that should allow the system to scale enough to 
get 2.6 out the door.  It's the layers under those guys who need to shuffle 
around to feed their patches into those bitkeeper trees that Linus pulls from.

Of course to make this work, Bitkeeper will somehow need to let Linus 
cherry-pick patches from the bitkeeper trees under him and reject others.  
I've tried to follow the discussion on this front but I'm not convinced it's 
resolved yet.  But as I said, there's time...

> Our problems are not novel.  Let us apply standard business school
> methodologies to them.

If I remember my kernel-traffic summaries correctly, Eric Raymond was saying 
something like this about two years ago.  Something about the boy genius 
effect? :)

> Hans

Rob

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Rob Landley wrote:

>>
>
>Linus (and the lieutenants under him) have not been the ones experiencing the 
>problem.  Linus accepts all the patches he wants to, and the lieutenants tend 
>not to be ignored.  The top of the pyramid is not where the motivation for 
>change is most strongly felt...
>

Partially true in most social systems.  However, the top of the pyramid 
has some substantial motivation to see a better kernel come into 
existence.  I personally feel motivated to ensure that ReiserFS patches 
don't get lost, because those 5-10% improvements really do add up over 
time.  We may also hope that they feel some pride of craftsmanship in 
constructing their social system --- I suspect that pride in what they 
do is more likely to move them.

Hans

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Wed, 30 Jan 2002, Ingo Molnar wrote:
> On Wed, 30 Jan 2002, Larry McVoy wrote:
>
> > How much of the out order stuff goes away if you could send changes
> > out of order as long as they did not overlap (touch the same files)?
>
> could this be made: 'as long as they do not touch the same lines of
> code, taking 3 lines of context into account'? (ie. unified diff
> definition of 'collisions' context.)

That would be _wonderful_ and fix the last bitkeeper
problem I'm having once in a while.

cheers,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Josh MacDonald]

Quoting Rik van Riel (riel@conectiva.com.br):
> On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > On Wed, 30 Jan 2002, Larry McVoy wrote:
> >
> > > How much of the out order stuff goes away if you could send changes
> > > out of order as long as they did not overlap (touch the same files)?
> >
> > could this be made: 'as long as they do not touch the same lines of
> > code, taking 3 lines of context into account'? (ie. unified diff
> > definition of 'collisions' context.)
> 
> That would be _wonderful_ and fix the last bitkeeper
> problem I'm having once in a while.

This would seem to require a completely new tool for you to specify which 
hunks within a certain file belong to which changeset.  I can see why
Larry objects.  What's your solution?

-josh

-- 
PRCS version control system    http://sourceforge.net/projects/prcs
Xdelta storage & transport     http://sourceforge.net/projects/xdelta
Need a concurrent skip list?   http://sourceforge.net/projects/skiplist

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 08:59:03AM -0800, Josh MacDonald wrote:
> Quoting Rik van Riel (riel@conectiva.com.br):
> > On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > > On Wed, 30 Jan 2002, Larry McVoy wrote:
> > >
> > > > How much of the out order stuff goes away if you could send changes
> > > > out of order as long as they did not overlap (touch the same files)?
> > >
> > > could this be made: 'as long as they do not touch the same lines of
> > > code, taking 3 lines of context into account'? (ie. unified diff
> > > definition of 'collisions' context.)
> > 
> > That would be _wonderful_ and fix the last bitkeeper
> > problem I'm having once in a while.
> 
> This would seem to require a completely new tool for you to specify which 
> hunks within a certain file belong to which changeset.  I can see why
> Larry objects.  What's your solution?

We actually can go from any line to the changeset which created that line
relatively quickly (milliseconds in hot cache, second or so in cold cache).
And we have a design which has been proven to work in the past at HP which
would allow a fully general out of order, at the changeset level application
of changes.  It's a bit complex to describe here and it has been 6 months 
away from being done for 3 years, so don't hold your breath.  I think the
better short term answer is to fix the false dependency problem, use regular
diff/patch for the places where there really are dependencies, and then do
the completely general one.

We are doing the general solution, which we call lines of development, our
commercial customers want it as well.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Andreas Dilger]

On Jan 30, 2002  08:59 -0800, Josh MacDonald wrote:
> Quoting Rik van Riel (riel@conectiva.com.br):
> > On Wed, 30 Jan 2002, Ingo Molnar wrote:
> > > On Wed, 30 Jan 2002, Larry McVoy wrote:
> > >
> > > > How much of the out order stuff goes away if you could send changes
> > > > out of order as long as they did not overlap (touch the same files)?
> > >
> > > could this be made: 'as long as they do not touch the same lines of
> > > code, taking 3 lines of context into account'? (ie. unified diff
> > > definition of 'collisions' context.)
> > 
> > That would be _wonderful_ and fix the last bitkeeper
> > problem I'm having once in a while.
> 
> This would seem to require a completely new tool for you to specify which 
> hunks within a certain file belong to which changeset.  I can see why
> Larry objects.  What's your solution?

Please see my other email on this subject (out of order BK CSETs).
One relatively easy solution is "proxy CSETs", which describe on a
per-line basis (or xdelta checksum boundaries, even better) the changes
made in "false parent" CSETs, but do not actually contain the changes.
This allows the reciever to resolve the false dependencies, and also
allows the sender to validate (at proxy CSET creation time) that the
out-of-order CSET they are about to send, in fact, does not depend on
any of the "false parent" CSETs.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Wed, 30 Jan 2002, Rik van Riel wrote:

> On Wed, 30 Jan 2002, Ingo Molnar wrote:
>
> > could this be made: 'as long as they do not touch the same lines of
> > code, taking 3 lines of context into account'? (ie. unified diff
> > definition of 'collisions' context.)
>
> That would be _wonderful_ and fix the last bitkeeper problem I'm
> having once in a while.

perhaps there should also be some sort of authority needed to allow such
'violation' of current BK rules: while the patches to conflict in terms of
source code file, we can override it and tell it that they really dont
conflict.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Val Henson]

On Tue, Jan 29, 2002 at 11:48:05PM -0800, Linus Torvalds wrote:
> 
> One thing intrigued me in this thread - which was not the discussion
> itself, but the fact that Rik is using bitkeeper.
> 
> How many other people are actually using bitkeeper already for the kernel?
> I know the ppc guys have, for a long time, but who else is? bk, unlike
> CVS, should at least be _able_ to handle a "network of people" kind of
> approach.

I'm one of the ppc people so I don't really count, but...

I've used bitkeeper for the kernel for a year now.

One of the issues in the "network of people" approach is how much time
and effort it takes to maintain a separate tree while waiting for
changes to be merged into the main tree.  Bitkeeper really helps here.
I've been maintaining a tree with significant differences from the
mainline linuxppc tree, and I can pull and merge new changes without
hand-editing a file 99% of the time.  Maintaining my own tree is now a
minor annoyance, instead of a major pain.

-VAL

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> Linus just called you the ext2 maintainer.

Message-ID, please?

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alexander Viro wrote:
> On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > Linus just called you the ext2 maintainer.
>
> Message-ID, please?

I called you the VFS maintainer ("whether you like it or not" I think I
said. Although I can't find the message right now).

Now, that obviously does imply a certain control over low-level
filesystems, but it really mainly implies a control over the _interfaces_
used to talk the the filesystem, not the filesystem itself.

I personally really wouldn't mind seeing most filesystem patches coming
through Al (and, in fact, in the inode trimming patches that is partly
what as been happening), but I have this nagging suspicion that some
filesystem maintainers would rather eat barbed wire (*).

		Linus

(*) The discussions between Gooch and Al are always "interesting", to name
some names.

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Linus Torvalds wrote:

> Now, that obviously does imply a certain control over low-level
> filesystems, but it really mainly implies a control over the _interfaces_
> used to talk the the filesystem, not the filesystem itself.
> 
> I personally really wouldn't mind seeing most filesystem patches coming
> through Al (and, in fact, in the inode trimming patches that is partly

I would, though.  Inode-trimming is a separate story - it's a massive series of
interface-changing patches (55 chunks already merged, more to follow) and
any help is certainly welcome - it's a friggin' lot of work (and there was
quite a help - from Jeff, Urban, Christoph...)

	However, I really don't want to be in position when patches to fs
internals are fed through me.  I can give comments.  I can do code review.
I can look through the code and discuss VFS/VM/etc. changes that might be
useful.  I can actually decide to do these changes myself.  That's all nice
and dandy, but let's face it - most of filesystem internals patches are pretty
local and fs maintainers _MUST_ be the first recepients of such patches.

	I don't have Alan's patience.  And I don't believe that I can run
a clearinghouse tree for *all* fs patches - it's pretty much guaranteed to end
up with burnout in a month or so.  BTW, IIRC Jeff Garzik had been doing
something similar unofficially, but I've no idea how he feels about giving
it official status.

	Frankly, the only real issue in that thread was that we _do_ need
a tree specifically for small fixes.  Preferably - quickly getting merged
into the main tree.  And that's a hard work - davej seems to be doing that
and I admire the efforts he's able and willing to put into that stuff.
I know that I couldn't pull that off.

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alexander Viro wrote:
>
> 	Frankly, the only real issue in that thread was that we _do_ need
> a tree specifically for small fixes.  Preferably - quickly getting merged
> into the main tree.

A "small stuff" maintainer may indeed be a good idea. The maintainer could
be the same as somebody who does bigger stuff too, but they should be
clearly different things - trivial one-liners that do not add anything
new, only fix obvious stuff (to the point where nobody even needs to think
about it - if I'd start getting any even halfway questionable patches from
the "small stuff" maintainer, it wouldn't work).

		Linus

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:21 am, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Alexander Viro wrote:
> >
> > 	Frankly, the only real issue in that thread was that we _do_ need
> > a tree specifically for small fixes.  Preferably - quickly getting merged
> > into the main tree.
> 
> A "small stuff" maintainer may indeed be a good idea. The maintainer could
> be the same as somebody who does bigger stuff too, but they should be
> clearly different things - trivial one-liners that do not add anything
> new, only fix obvious stuff (to the point where nobody even needs to think
> about it - if I'd start getting any even halfway questionable patches from
> the "small stuff" maintainer, it wouldn't work).

But that's exactly what Arnaldo Carvalho de Melo[1] does, has been doing for 
more than a year, surely you've noticed?  On top of being the nicest guy in 
the world, as far as I can tell.

[1] Most of us call him <acme>.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> A "small stuff" maintainer may indeed be a good idea. The maintainer could
> be the same as somebody who does bigger stuff too, but they should be
> clearly different things - trivial one-liners that do not add anything
> new, only fix obvious stuff (to the point where nobody even needs to think
> about it - if I'd start getting any even halfway questionable patches from
> the "small stuff" maintainer, it wouldn't work).

So if someone you trusted actually started batching up small fixes and 
sending you things like

"37 random documentation updates - no code changed", "11 patches to fix
kmalloc checks", "maintainers updates to 6 network drivers"

that would work sanely ? I think that would actually fix a lot of the stuff 
getting lost right now. Its mostly small stuff, often from new people, or from
folks who met a bug, fixed it and have a totally seperate and rather more 
important (to them) project and deadline to meet that is going walkies.

It also increases bandwidth for sorting out the big stuff.

The other related question is device driver implementation stuff (not interfaces
and abstractions). You don't seem to check that much anyway, or have any taste
in device drivers 8) so should that be part of the small fixing job ?

Alan

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 10:06:35AM +0000, Alan Cox wrote:
> The other related question is device driver implementation stuff (not interfaces
> and abstractions). You don't seem to check that much anyway, or have any taste
> in device drivers 8) so should that be part of the small fixing job ?

I've often dreamt of an overall "drivers maintainer" or perhaps just an
unmaintained-drivers maintainer:  a person with taste who could give
driver patches a glance, when noone else does.
(and no I'm not volunteering :))

	Jeff, dreams on

Re: A modest proposal -- We need a patch penguin

[Greg KH]

On Wed, Jan 30, 2002 at 05:18:55AM -0500, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 10:06:35AM +0000, Alan Cox wrote:
> > The other related question is device driver implementation stuff (not interfaces
> > and abstractions). You don't seem to check that much anyway, or have any taste
> > in device drivers 8) so should that be part of the small fixing job ?
> 
> I've often dreamt of an overall "drivers maintainer" or perhaps just an
> unmaintained-drivers maintainer:  a person with taste who could give
> driver patches a glance, when noone else does.
> (and no I'm not volunteering :))

I have had that same dream too, Jeff :)
Especially after spelunking through the SCSI drivers, and being amazed
that only one of them uses the, now two year old, pci_register_driver()
interface (which means that only that driver works properly in PCI
hotplug systems.)

Having someone with "taste" to run driver patches by first would have
been a great help when I started out writing them.  I've been trying to
provide that resource for the new USB drivers.

greg k-h

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> Especially after spelunking through the SCSI drivers, and being amazed
> that only one of them uses the, now two year old, pci_register_driver()
> interface (which means that only that driver works properly in PCI
> hotplug systems.)

I doubt it does actually. The problem with pci register driver and scsi is
that the two subsystems are designed with violently conflicting goals. Once
DaveJ or someone does the proposed scsi cleanups it'll become the natural
not the obscenely complicated way to do a scsi driver, as well as sorting out
the pcmcia and cardbus scsi mess, and the card failed/recovered stuff once and
for all.

Alan

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 06:35:15PM +0000, Alan Cox wrote:
> > Especially after spelunking through the SCSI drivers, and being amazed
> > that only one of them uses the, now two year old, pci_register_driver()
> > interface (which means that only that driver works properly in PCI
> > hotplug systems.)
> 
> I doubt it does actually. The problem with pci register driver and scsi is
> that the two subsystems are designed with violently conflicting goals. Once
> DaveJ or someone does the proposed scsi cleanups it'll become the natural
> not the obscenely complicated way to do a scsi driver, as well as sorting out
> the pcmcia and cardbus scsi mess, and the card failed/recovered stuff once and
> for all.

I disagree... I outlined a workable scheme for hotplugging SCSI
controllers to Justin Gibbs a long time ago, when the new aic7xxx was
first being merged.  Using the new PCI API was fairly easy, handling the
disk-disappearing-from-under-you problem was a bit more annoying :)

	Jeff

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 01:29:39PM -0500, Jeff Garzik wrote:
> I disagree... I outlined a workable scheme for hotplugging SCSI
> controllers to Justin Gibbs a long time ago, when the new aic7xxx was
> first being merged.  Using the new PCI API was fairly easy, handling the
> disk-disappearing-from-under-you problem was a bit more annoying :)

And indeed the aic7xxx driver does handle hotplugging.
It just doesn't handle hot-un-plugging.

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 09:11:26AM -0800, Greg KH wrote:
> I have had that same dream too, Jeff :)
> Especially after spelunking through the SCSI drivers, and being amazed
> that only one of them uses the, now two year old, pci_register_driver()
> interface (which means that only that driver works properly in PCI
> hotplug systems.)

Spelunking through the SCSI drivers involves great deal of
bravery.  That pile of dung does not need a "small-stuff"
maintainer.  It needs to be forcefully ejected and replaced with
extreme prejudice.  It is amazing that ancient stuff works as
well as it does...

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> bravery.  That pile of dung does not need a "small-stuff"
> maintainer.  It needs to be forcefully ejected and replaced with
> extreme prejudice.  It is amazing that ancient stuff works as
> well as it does...

A lot of the apparently really ugly drivers turned out to be very good code
hiding under 10 years of history and core code changes and
assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
or hang - Linus apparently merged untested stuff to the old broken driver.

Alan

Re: A modest proposal -- We need a patch penguin

[Erik Andersen]

On Wed Jan 30, 2002 at 11:06:04PM +0000, Alan Cox wrote:
> > bravery.  That pile of dung does not need a "small-stuff"
> > maintainer.  It needs to be forcefully ejected and replaced with
> > extreme prejudice.  It is amazing that ancient stuff works as
> > well as it does...
> 
> A lot of the apparently really ugly drivers turned out to be very good code
> hiding under 10 years of history and core code changes and
> assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
> use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
> or hang - Linus apparently merged untested stuff to the old broken driver.

This is in the latest -ac kernels?  Cool, I'll go take a close
look.  I'm very anxious to see a SCSI layer that doesn't suck
get put in place,

 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Andre Hedrick]

On Wed, 30 Jan 2002, Erik Andersen wrote:

> On Wed Jan 30, 2002 at 11:06:04PM +0000, Alan Cox wrote:
> > > bravery.  That pile of dung does not need a "small-stuff"
> > > maintainer.  It needs to be forcefully ejected and replaced with
> > > extreme prejudice.  It is amazing that ancient stuff works as
> > > well as it does...
> > 
> > A lot of the apparently really ugly drivers turned out to be very good code
> > hiding under 10 years of history and core code changes and
> > assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
> > use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
> > or hang - Linus apparently merged untested stuff to the old broken driver.
> 
> This is in the latest -ac kernels?  Cool, I'll go take a close
> look.  I'm very anxious to see a SCSI layer that doesn't suck
> get put in place,

Given me another development tree of time to create one and it will be a
done deal, but I have enough to clean up in what is started now.

Cheers,

Andre Hedrick
Linux Disk Certification Project                Linux ATA Development

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 04:48:48PM -0700, Erik Andersen wrote:
 > > assumptions. See the NCR5380 stuff I've now all done (in 2.4.18pre) - dont 
 > > use 2.5.* NCR5380 it'll probably corrupt your system if it doesn't just die
 > > or hang - Linus apparently merged untested stuff to the old broken driver.
 > 
 > This is in the latest -ac kernels?

 Even better, it's in 2.4 mainline.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> This is in the latest -ac kernels?  Cool, I'll go take a close
> look.  I'm very anxious to see a SCSI layer that doesn't suck
> get put in place,

The scsi mid layer is a seperate problem, and its getting there already in
2.5. Chunks of nasty scsi special cases keep dissappearing with the bio stuff

The NCR5380 stuff fixes what was an amazingly crufty unmaintained driver

[PATCH] fix for 2.4.18-pre7 SCSI namespace conflict

[Erik Andersen]

On Thu Jan 31, 2002 at 12:33:16AM +0000, Alan Cox wrote:
> > This is in the latest -ac kernels?  Cool, I'll go take a close
> > look.  I'm very anxious to see a SCSI layer that doesn't suck
> > get put in place,
> 
> The scsi mid layer is a seperate problem, and its getting there already in
> 2.5. Chunks of nasty scsi special cases keep dissappearing with the bio stuff
> 
> The NCR5380 stuff fixes what was an amazingly crufty unmaintained driver

Nice work.  Just for giggles I decided to compile in the whole
pile of SCSI drivers.  Some namespace issues vs NCR5380 popped up
pretty quickly...  It turns out that pas16.c, t128.c, dmx3191d.c,
and dtc.c have taken the misguided approach of doing a 
    #include "NCR5380.c" 
Ick!  Clearly there is an important uniform SCSI driver layer
that is entirely missing since driver authors are tempted to save  
time by doing wholesale #includes of other drivers....

Anyways, here is the bug:

    pas16.o: In function `NCR5380_timer_fn':
    pas16.o(.text+0x35c): multiple definition of `NCR5380_timer_fn'
    g_NCR5380.o(.text+0x2ec): first defined here
    t128.o: In function `NCR5380_timer_fn':
    t128.o(.text+0x2cc): multiple definition of `NCR5380_timer_fn'
    g_NCR5380.o(.text+0x2ec): first defined here
    dmx3191d.o: In function `NCR5380_timer_fn':
    dmx3191d.o(.text+0x2bc): multiple definition of `NCR5380_timer_fn'
    g_NCR5380.o(.text+0x2ec): first defined here
    dtc.o: In function `NCR5380_timer_fn':
    dtc.o(.text+0x30c): multiple definition of `NCR5380_timer_fn'
    g_NCR5380.o(.text+0x2ec): first defined here
    make[3]: *** [scsidrv.o] Error 1
    make[3]: Leaving directory `/home/andersen/imager/linux/drivers/scsi'
    make[2]: *** [first_rule] Error 2
    make[2]: Leaving directory `/home/andersen/imager/linux/drivers/scsi'
    make[1]: *** [_subdir_scsi] Error 2
    make[1]: Leaving directory `/home/andersen/imager/linux/drivers'
    make: *** [_dir_drivers] Error 2


Here is a trivial and "obviously correct" fix.

--- linux/drivers/scsi.orig/NCR5380.c	Fri Dec 21 10:41:55 2001
+++ linux/drivers/scsi/NCR5380.c	Wed Jan 30 17:56:42 2002
@@ -612,7 +612,7 @@
  *	Locks: disables irqs, takes and frees io_request_lock
  */
  
-void NCR5380_timer_fn(unsigned long unused)
+static void NCR5380_timer_fn(unsigned long unused)
 {
 	struct Scsi_Host *instance;
 
 -Erik

--
Erik B. Andersen             http://codepoet-consulting.com/
--This message was written using 73% post-consumer electrons--

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Wed, 30 Jan 2002, Alan Cox wrote:
>
> So if someone you trusted actually started batching up small fixes and
> sending you things like
>
> "37 random documentation updates - no code changed", "11 patches to fix
> kmalloc checks", "maintainers updates to 6 network drivers"
>
> that would work sanely ?

Yes. That would take a whole lot of load off me - load I currently handle
by just not sweating the small stuff, and concentrating on the things I
think are important.

> The other related question is device driver implementation stuff (not interfaces
> and abstractions). You don't seem to check that much anyway, or have any taste
> in device drivers 8) so should that be part of the small fixing job ?

I think it has some of the same issues, but I really would prefer to have
it in a separate batch.

Quite frankly, this is a large part of what you did..

		Linus

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Wed, 30 Jan 2002, Linus Torvalds wrote:

> 
> On Wed, 30 Jan 2002, Alan Cox wrote:
> >
> > So if someone you trusted actually started batching up small fixes and
> > sending you things like
> >
> > "37 random documentation updates - no code changed", "11 patches to fix
> > kmalloc checks", "maintainers updates to 6 network drivers"
> >
> > that would work sanely ?
> 
> Yes. That would take a whole lot of load off me - load I currently handle
> by just not sweating the small stuff, and concentrating on the things I
> think are important.

Once more beating a dead horse, you don't improve scalability by finding a
better way to push everything through one person. "Random documentation
updates" and "corrections to MAINTAINERS mailing addresses" could and
should be approved by someone else.

So should the 1-2-3 liners which are clearly and obviously tiny bug fixes
for obvious problems, off by one, mistyped lock names, adding casts to
make the kernel compile w/o hundreds of "you don't understand C type
rules" warnings.

The way to get crap out of your life is to trust some people to identify
changes of this type and leave you to code review significant changes. The
most efficient way to do something is to avoid having to do it all all,
not by doing the wrong thing better. Pushing hard to you is like hand
coding a bubble sort in assembler, the problem is not in the
implementation but the algorithm. 

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Alan Cox wrote:

>>A "small stuff" maintainer may indeed be a good idea. The maintainer could
>>be the same as somebody who does bigger stuff too, but they should be
>>clearly different things - trivial one-liners that do not add anything
>>new, only fix obvious stuff (to the point where nobody even needs to think
>>about it - if I'd start getting any even halfway questionable patches from
>>the "small stuff" maintainer, it wouldn't work).
>>
And then we are still just discussing here how to get things IN. But 
there apparently currently is
nearly no way to get things OUT of the kernel tree. Old obsolete drivers 
used by some
computer since archeologists should be killed (Atari, Amiga, support, 
obsolete drivers and so on).
Just let *them* maintains theyr separate kernel tree...

Re: A modest proposal -- We need a patch penguin

[Ian Molton]

On a sunny Thu, 31 Jan 2002 13:14:55 +0100 Martin Dalecki gathered a sheaf
of electrons and etched in their motions the following immortal words:

> Old obsolete drivers used by some
> computer since archeologists should be killed (Atari, Amiga, support, 
> obsolete drivers and so on).
> Just let *them* maintains theyr separate kernel tree...

Yeah, keep linux for those X86 purists.

Great philosophy.

Just because hardware is OLD doesnt mean it cant integrate fairly cleanly
with the current linux kernel.

Why shouldnt Linux run on my 15 year old Acorn A400 (my current project) ?

Linux is about FUN.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, Martin Dalecki wrote:

> And then we are still just discussing here how to get things IN. But
> there apparently currently is nearly no way to get things OUT of the
> kernel tree. Old obsolete drivers used by some computer since
> archeologists should be killed (Atari, Amiga, support, obsolete
> drivers and so on). Just let *them* maintains theyr separate kernel
> tree...

'old' architectures do not hinder development - they are separate, and
they have to update their stuff. (and i think the m68k port is used by
many other people and not CS archeologists.) Old drivers are not a true
problem either - if they dont compile that's the problem of the
maintainer. Occasionally old drivers get zapped (mainly when there is a
new replacement driver).

	Ingo

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Thu, 31 Jan 2002, Ingo Molnar wrote:

> 'old' architectures do not hinder development - they are separate, and
> they have to update their stuff. (and i think the m68k port is used by

... unless they play silly buggers with the internals of VM.

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Thu, 31 Jan 2002, Alexander Viro wrote:

> > 'old' architectures do not hinder development - they are separate, and
> > they have to update their stuff. (and i think the m68k port is used by
>
> ... unless they play silly buggers with the internals of VM.

As long as there is still someone who can respond to problems in the arch
part, I don't really think it's a problem, or was it?

bye, Roman

Re: A modest proposal -- We need a patch penguin

[David Weinehall]

On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
> 
> On Thu, 31 Jan 2002, Martin Dalecki wrote:
> 
> > And then we are still just discussing here how to get things IN. But
> > there apparently currently is nearly no way to get things OUT of the
> > kernel tree. Old obsolete drivers used by some computer since
> > archeologists should be killed (Atari, Amiga, support, obsolete
> > drivers and so on). Just let *them* maintains theyr separate kernel
> > tree...
> 
> 'old' architectures do not hinder development - they are separate, and
> they have to update their stuff. (and i think the m68k port is used by
> many other people and not CS archeologists.) Old drivers are not a true
> problem either - if they dont compile that's the problem of the
> maintainer. Occasionally old drivers get zapped (mainly when there is a
> new replacement driver).

To testify that even really old hardware is used, I recently received a
patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
xd.c-driver. Not particularly recent hardware, but the person who sent
the patch uses it. Why deny him usage of his hardware when it doesn't
intrude upon the rest of the codebase?


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

David Weinehall wrote:

>On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>
>>On Thu, 31 Jan 2002, Martin Dalecki wrote:
>>
>>>And then we are still just discussing here how to get things IN. But
>>>there apparently currently is nearly no way to get things OUT of the
>>>kernel tree. Old obsolete drivers used by some computer since
>>>archeologists should be killed (Atari, Amiga, support, obsolete
>>>drivers and so on). Just let *them* maintains theyr separate kernel
>>>tree...
>>>
>>'old' architectures do not hinder development - they are separate, and
>>they have to update their stuff. (and i think the m68k port is used by
>>many other people and not CS archeologists.) Old drivers are not a true
>>problem either - if they dont compile that's the problem of the
>>maintainer. Occasionally old drivers get zapped (mainly when there is a
>>new replacement driver).
>>
>
>To testify that even really old hardware is used, I recently received a
>patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
>xd.c-driver. Not particularly recent hardware, but the person who sent
>the patch uses it. Why deny him usage of his hardware when it doesn't
>intrude upon the rest of the codebase?
>

He should feel free to use the 2.0.xx kernel up no end. Nobody denys it 
to him. But from the mainline
it all should get out of the sight for the developement.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, David Weinehall wrote:

> On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>
> > 'old' architectures do not hinder development - they are separate, and
> > they have to update their stuff. (and i think the m68k port is used by
> > many other people and not CS archeologists.) Old drivers are not a true
> > problem either - if they dont compile that's the problem of the
> > maintainer. Occasionally old drivers get zapped (mainly when there is a
> > new replacement driver).
>
> To testify that even really old hardware is used, I recently received
> a patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
> xd.c-driver. Not particularly recent hardware, but the person who sent
> the patch uses it. Why deny him usage of his hardware when it doesn't
> intrude upon the rest of the codebase?

exactly. Cruft hanging around does hurt in the 'generic' kernel. There is
'leaf' code where it hurts much less. Sure, we'd like to have clean code
everywhere, and a driver with a clean and recent codebase will get more
attention from the architecture point of view, but to the user, an
outdated but working driver is better than no driver at all.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Thu, 31 Jan 2002, David Weinehall wrote:
>
>>On Thu, Jan 31, 2002 at 03:17:52PM +0100, Ingo Molnar wrote:
>>
>>>'old' architectures do not hinder development - they are separate, and
>>>they have to update their stuff. (and i think the m68k port is used by
>>>many other people and not CS archeologists.) Old drivers are not a true
>>>problem either - if they dont compile that's the problem of the
>>>maintainer. Occasionally old drivers get zapped (mainly when there is a
>>>new replacement driver).
>>>
>>To testify that even really old hardware is used, I recently received
>>a patch for 2.0.xx to add autodetection for wd1002s-wx2 in the
>>xd.c-driver. Not particularly recent hardware, but the person who sent
>>the patch uses it. Why deny him usage of his hardware when it doesn't
>>intrude upon the rest of the codebase?
>>
>
>exactly. Cruft hanging around does hurt in the 'generic' kernel. There is
>'leaf' code where it hurts much less. Sure, we'd like to have clean code
>everywhere, and a driver with a clean and recent codebase will get more
>attention from the architecture point of view, but to the user, an
>outdated but working driver is better than no driver at all.
>
It's an incredibble bandwidth waste for 99.99% of people downolading 
2.5.xx and it *is* making architectural
changes in the kernel harder, becouse the modularisatoin of the kernel 
isn't nearly as perfect as you try
to disguise it here. Please just have a look at the consequences of the 
kdev_t changes, which where necessary
since already about 8 years. And then my these is somehow tautological 
if it doesn't apply now, it will
apply in about 4 years. At some point in time there is the need to let 
some things go - the problem
is more fundamental.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Thu, 31 Jan 2002, Martin Dalecki wrote:

> It's an incredibble bandwidth waste for 99.99% of people downolading
> 2.5.xx and it *is* making architectural changes in the kernel harder,
> becouse the modularisatoin of the kernel isn't nearly as perfect as
> you try to disguise it here. Please just have a look at the
> consequences of the kdev_t changes, which where necessary since
> already about 8 years. And then my these is somehow tautological if it
> doesn't apply now, it will apply in about 4 years. At some point in
> time there is the need to let some things go - the problem is more
> fundamental.

it's not mandatory for the developer to push every interface change into
every driver or every architecture. Sure, if some code has not been kept
in sync for a long time then it should be zapped, but the pure fact that
something is less often used should not make it a candidate for zapping.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Thu, Jan 31, 2002 at 04:07:52PM +0100, Ingo Molnar wrote:
> it's not mandatory for the developer to push every interface change into
> every driver or every architecture. Sure, if some code has not been kept
> in sync for a long time then it should be zapped,

add "by the maintainer, if they are still around" here please.

> but the pure fact that
> something is less often used should not make it a candidate for zapping.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Thu, 31 Jan 2002, Martin Dalecki wrote:
> Alan Cox wrote:
> >>A "small stuff" maintainer may indeed be a good idea. The maintainer could
> >>be the same as somebody who does bigger stuff too, but they should be
> >>clearly different things - trivial one-liners that do not add anything
> >>new, only fix obvious stuff (to the point where nobody even needs to think
> >>about it - if I'd start getting any even halfway questionable patches from
> >>the "small stuff" maintainer, it wouldn't work).
> >>
> And then we are still just discussing here how to get things IN. But 
> there apparently currently is
> nearly no way to get things OUT of the kernel tree. Old obsolete drivers 
> used by some
> computer since archeologists should be killed (Atari, Amiga, support, 
> obsolete drivers and so on).
> Just let *them* maintains theyr separate kernel tree...

Come'on, m68k is not dead yet!

We do our best to keep the m68k tree in sync. In fact that's much less work
than feeding back our changes to Linus, since hacking code needs less retries
than sending patches :-)

(but we all know that since we're discussing it in this thread... ;-)

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 01:21:09AM -0800, Linus Torvalds wrote:
 > 
 > A "small stuff" maintainer may indeed be a good idea. The maintainer could
 > be the same as somebody who does bigger stuff too, but they should be
 > clearly different things - trivial one-liners that do not add anything
 > new, only fix obvious stuff (to the point where nobody even needs to think
 > about it - if I'd start getting any even halfway questionable patches from
 > the "small stuff" maintainer, it wouldn't work).

 The difficult part comes when slightly larger changesets are needed,
 just to make things compile again for some people.
 See yesterdays subsection changes from Keith I forwarded you for
 example. To me, it had looked fine, it had good discussion on
 l-k, and it solved a known problem. I was surprised you threw it
 back for more changes (but glad, I want the best solution too, and
 taking a quick glance to the mail I've not read yet, it looks like
 Keith has bettered his original solution).

 Most of the bits I've sent you so far have been "small stuff".
 And things will likely continue to be so. There are large chunks
 in my tree, but I've absolutely no intention of feeding you those.
 Things like the input layer/console layer reworking are the
 responsibility of $maintainer to push your way.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 09:09 am, Linus Torvalds wrote:
> On Wed, 30 Jan 2002, Alexander Viro wrote:
> > On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > > Linus just called you the ext2 maintainer.
> >
> > Message-ID, please?
> 
> I called you the VFS maintainer ("whether you like it or not" I think I
> said. Although I can't find the message right now).

Correct, I was wrong to say that, in fact, I should have posted the whole 
email to 'drafts' as I normally do with such non-constructive tripe.  I don't 
know what got into me.  Al, please accept my apologies.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alexander Viro]

On Wed, 30 Jan 2002, Daniel Phillips wrote:

> On January 30, 2002 09:09 am, Linus Torvalds wrote:
> > On Wed, 30 Jan 2002, Alexander Viro wrote:
> > > On Wed, 30 Jan 2002, Daniel Phillips wrote:
> > > > Linus just called you the ext2 maintainer.
> > >
> > > Message-ID, please?
> > 
> > I called you the VFS maintainer ("whether you like it or not" I think I
> > said. Although I can't find the message right now).
> 
> Correct, I was wrong to say that, in fact, I should have posted the whole 
> email to 'drafts' as I normally do with such non-constructive tripe.  I don't 
> know what got into me.  Al, please accept my apologies.

Accepted.

Re: A modest proposal -- We need a patch penguin

[Kees Bakker, Kees Bakker]

>>>>> "Alexander" == Alexander Viro <viro@math.psu.edu> writes:

Alexander> On Wed, 30 Jan 2002, Daniel Phillips wrote:

>> Linus just called you the ext2 maintainer.

Alexander> Message-ID, please?

From: torvalds@transmeta.com (Linus Torvalds)
Subject: Re: A modest proposal -- We need a patch penguin
Date: 	Tue, 29 Jan 2002 22:22:46 +0000 (UTC)
Message-ID: <a377bn$1go$1@penguin.transmeta.com>

Re: A modest proposal -- We need a patch penguin

[Charles Cazabon]

Daniel Phillips <phillips@bonn-fries.net> wrote:
> 
> Linus just called you the ext2 maintainer.

Did he?  I only saw him call Al Viro the de-facto VFS maintainer.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon                         <charlesc@discworld.dyndns.org>
GPL'ed software available at:  http://www.qcc.sk.ca/~charlesc/software/
-----------------------------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Oliver Xymoron]

On Tue, 29 Jan 2002, Daniel Phillips wrote:

> Exactly.  The successor patch to the 'kind of gross' patch got rid of the
> double-pointers, it was the proper fix, though there is still no excuse for
> leaving the bug hanging around while coming up with the better version.

The gross fixes tend to get dropped because if they're in, the proper fix
loses priority. FIXMEs can take many years to fix. The problem seems not
to be the dropping of the patch so much as the dropping of the bug report
and bug tracking is an altogether different problem.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 08:41 am, Oliver Xymoron wrote:
> On Tue, 29 Jan 2002, Daniel Phillips wrote:
> 
> > Exactly.  The successor patch to the 'kind of gross' patch got rid of the
> > double-pointers, it was the proper fix, though there is still no excuse for
> > leaving the bug hanging around while coming up with the better version.
> 
> The gross fixes tend to get dropped because if they're in, the proper fix
> loses priority. FIXMEs can take many years to fix. The problem seems not
> to be the dropping of the patch so much as the dropping of the bug report
> and bug tracking is an altogether different problem.

The problem was the dropping of the patch.  A bunch of things contributed
to it, and at this point I believe the main one was having no patch
submission system.  I should know, I was on the dirty end of this stick.

-- 
Daniel

bug tracking (was Re: A modest proposal -- We need a patch penguin)

[Jeff Garzik]

On Wed, Jan 30, 2002 at 01:41:22AM -0600, Oliver Xymoron wrote:
> The gross fixes tend to get dropped because if they're in, the proper fix
> loses priority. FIXMEs can take many years to fix. The problem seems not
> to be the dropping of the patch so much as the dropping of the bug report
> and bug tracking is an altogether different problem.

Indeed.  The issue of kernel bug tracking gets pondered and discussed
every few months it seems (not without need, mind you).

To tie this back into the original whine from RobL, what we do NOT need
is a patch secretary.  What we do need, desperately, is
(a) a bug-tracking system, and
(b) at least one sharp person, with bunches of help from kernel
developers and users alike, to close fixed bugs, ping users, clear out
garbage so that the bug database has a very high signal-to-noise ratio.

Good kernel bug tracking can be done, but it requires human maintenance,
by someone or someones with a brain.  It cannot be done without plenty
of automation, though, as tytso (god bless him for trying!) showed...

Such would be a significant boon to -all- Linux users.

	Jeff

RE: bug tracking (was Re: A modest proposal -- We need a patch penguin)

[Chris Funderburg]

I'm confused.

Wouldn't Bugzilla be perfect for this?  I run a slightly modified
version
for the company I work for.  You could have as many administrators as
you
need, and use categories for different kernel subsystems.  The
maintainers
could be set-up as QA contacts, and it's really easy to maintain.

How about http://bugzilla.kernel.org (assuming the servers get fixed
someday)

Just a thought.

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org
[mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jeff Garzik
Sent: 30 January 2002 08:10
To: Oliver Xymoron
Cc: Daniel Phillips; linux-kernel
Subject: bug tracking (was Re: A modest proposal -- We need a patch
penguin)


On Wed, Jan 30, 2002 at 01:41:22AM -0600, Oliver Xymoron wrote:
> The gross fixes tend to get dropped because if they're in, the proper
fix
> loses priority. FIXMEs can take many years to fix. The problem seems
not
> to be the dropping of the patch so much as the dropping of the bug
report
> and bug tracking is an altogether different problem.

Indeed.  The issue of kernel bug tracking gets pondered and discussed
every few months it seems (not without need, mind you).

To tie this back into the original whine from RobL, what we do NOT need
is a patch secretary.  What we do need, desperately, is
(a) a bug-tracking system, and
(b) at least one sharp person, with bunches of help from kernel
developers and users alike, to close fixed bugs, ping users, clear out
garbage so that the bug database has a very high signal-to-noise ratio.

Good kernel bug tracking can be done, but it requires human maintenance,
by someone or someones with a brain.  It cannot be done without plenty
of automation, though, as tytso (god bless him for trying!) showed...

Such would be a significant boon to -all- Linux users.

	Jeff


-
To unsubscribe from this list: send the line "unsubscribe linux-kernel"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: bug tracking (was Re: A modest proposal -- We need a patch penguin)

[Oliver Xymoron]

On Wed, 30 Jan 2002, Chris Funderburg wrote:

> Wouldn't Bugzilla be perfect for this?  I run a slightly modified
> version for the company I work for.  You could have as many
> administrators as you need, and use categories for different kernel
> subsystems.  The maintainers could be set-up as QA contacts, and it's
> really easy to maintain.

Bugzilla would be fine. But the tools alone are only a small part of the
problem. Good bug tracking requires people to enter good bug descriptions,
and sift through the database and remove duplicates, close old bugs,
adjust priorities, etc. I personally loathe such work even though I
recognize the value of it, and I suspect most of the major maintainers do
too. The problem is, they're the ones getting the bug reports. We have to
move from the current situation to one where there are trusted and
effective bug database trackers receiving the reports and that's a pretty
big transition.

Note that MJC already has a bugzilla up for his tree. Don't know if he's
found someone to manage it yet though. If that Bugzilla were to track,
say, the dj tree and the rmap patch too, it might get closer to critical
mass. Redhat's already got a Bugzilla for their mostly AC kernel, and
they might be willing to copy mainline kernel bugs to another Bugzilla.

-- 
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:

Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4 liners.
The big stuff is not the problem most times. That stuff does get ripped to
shreds and picked over as is needed. (Except device drivers, Linus alas has
absolutely no taste in device drivers 8))

People collecting up patches _does_ help big time for all the small fixes.
Especially ones disciplined enough to keep the originals they applied so
they can feed stuff on with that tag. If I sent Linus on a patch that said
"You've missed this fix by Andrew Morton" then Linus knew it was probably
right for example.

> it. Start small, because for small patches people will have the few

Start small and your obvious one line diff, or 3 line typo fix will be
ignored for a decade. There were critical fixes that Linus dropped
repeatedly between 2.4.2 and 2.4.16 or so which ended up being holes in every
non-ac based distro.

Alan

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> The big stuff is not the problem most times. [...]

oh, i agree, but still the big stuff is that gets quoted in most emails
that try to invent the next big patch submission method ...

> People collecting up patches _does_ help big time for all the small
> fixes.

yes. This started with you and multiple people do it currently.

> Especially ones disciplined enough to keep the originals they applied
> so they can feed stuff on with that tag. If I sent Linus on a patch
> that said "You've missed this fix by Andrew Morton" then Linus knew it
> was probably right for example.

yes. This is what maintainers do. You, when collecting patches for the -ac
tree, are in essence a trusted jolly joker maintainer, very disciplined to
filter the trivial stuff from the nontrivial stuff.

> Start small and your obvious one line diff, or 3 line typo fix will be
> ignored for a decade. There were critical fixes that Linus dropped
> repeatedly between 2.4.2 and 2.4.16 or so which ended up being holes
> in every non-ac based distro.

(while i still do not claim that things are perfect, i'd like to see
specific examples nevertheless.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Dave McCracken]

--On Tuesday, January 29, 2002 13:22:05 +0000 Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

> Lots of the stuff getting missed is tiny little fixes, obvious 3 or 4
> liners. The big stuff is not the problem most times. That stuff does get
> ripped to shreds and picked over as is needed. (Except device drivers,
> Linus alas has absolutely no taste in device drivers 8))
> 
> People collecting up patches _does_ help big time for all the small fixes.
> Especially ones disciplined enough to keep the originals they applied so
> they can feed stuff on with that tag. If I sent Linus on a patch that said
> "You've missed this fix by Andrew Morton" then Linus knew it was probably
> right for example.

I think this is a big part of the problem.  What's needed, and what Alan
used to provide, is a maintainer for the miscellaneous parts of the kernel
that aren't covered by the various subsystem maintainers.  We need someone
who can accept that one or two line fix, get it tested, and make sure Linus
sees it.

I gather Dave Jones is taking on that role to some extent.  If so, perhaps
he should be officially listed in the MAINTAINERS file.  Whether it's Dave
or someone else, I think this is a critical need.

Dave McCracken

======================================================================
Dave McCracken          IBM Linux Base Kernel Team      1-512-838-3059
dmccr@us.ibm.com                                        T/L   678-3059

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:54 am, Ingo Molnar wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:

>   - cleanliness
>   - concept
>   - timing
>   - testing
>
> a violation of any of these items can cause patch to be dropped *without
> notice*. Face it, it's not Linus' task to teach people how to code or how
> to write correct patches. Sure, he still does teach people most of the
> time, but you cannot *expect* him to be able to do it 100% of the time.

I'm trying to identify stuff that isn't necessarily Linus's job, and doesn't 
seem to be being done, and seeing if somebody ELSE can do it.  My proposal 
was my take on improving things.  If now is not the time for it, maybe 
smaller steps can be taken first.

Possibly Linus needs a "bounce this message to kernelnewbies.org's 'please 
teach this guy how to program' list?" key, as an alternative to the delete 
key?  For patches that try to do something useful and simply don't manage to?

This is one of the things I thought a patch penguin (and troupe of volunteer 
patch secretaries operating under a patch penguin) might be able to do.  
Right now, there's no troupe of patch secretaries because the patch 
submission queue is linus's mailbox.  (The real irreplaceable job of the 
patch penguin is queueing stuff for linus split out of the tree, so the patch 
penguin doesn't have to scale that much better than Linus does.  Other people 
working with the patch penguin could theoretically help 
massage/test/integrate/update patches.  And of course linus could still take 
patches directly from maintainers if he had the bandwidth to do so.  He can 
obviously even write his own code when he has time...)

Linux-kernel used to serve all these functions (it was the patch queue and 
the patch cleanup and sorting discussion list, and some people on it were a 
bit like patch secretaries), but the volume has gotten too high for it to 
function as such anymore.  Posting a patch here less than a dozen times 
doesn't really count as submitting it to Linus anymore.

One demarcation that COULD be made is 2.4 vs 2.5.  There could probably be 
seperate "stable" vs "development" lists.  Probably been suggested before and 
shot down, but does work aimed at Marcelo and work aimed at Linus really need 
to be on the same list?

> 2) concept
>
> many of the patches which were rejected for a long time are *difficult*
> issues. And one thing many patch submitters miss: even if the concept of
> the patch is correct, you first have to start by cleaning up *old* code,
> see issue 1). Your patch is not worth a dime if you leave in old cruft, or
> if the combination of old cruft and your new code is confusing. Also, make
> sure the patch is discussed and developed openly, not on some obscure
> list. linux-kernel@vger.kernel.org will do most of the time. I do not want
> to name specific patches that violate this point (doing that in public
> just offends people needlessly - and i could just as well list some of my
> older patches), but i could list 5 popular patches immediately.
>
> impact: a patch penguin just wont solve this concept issue, because, by
> definition, he doesnt deal with design issues. And most of the big patch
> rejections happen due to exactly these concept issues.

Definitely Linus's job.  But of those top 5 patches, how many of the patch 
pushers have had their ideas actually critiqued so they know why they're not 
going in?  (If it's just a question of "this work needs to be done 
first/also", that's a manageable problem.)

> 3) timing
>
> kernel source code just cannot go through arbitrary transitions. Eg. right
> now the scheduler is being cleaned up (so far it was more than 50
> sub-patches and they are still coming) - and work is going on to maximize
> the quality of the preemption patch, but until the base scheduler has
> stabilized there is just no point in applying the preemption patch - no
> matter how good the preemption patch is. Robert understands this very
> much. Many other people do not.
>
> impact: a patch penguin just wont solve this issue, because a patch
> penguin cannot let his tree transition arbitrarily either. Only separately
> maintained and tested patches/trees can handle this issue.

A patch penguin's tree could apply more "speculative" patches than Linus, 
because the nature of the tree is that some of the patches in it get backed 
out, or at the very least don't go on to Linus yet.

It's also a nice buffer of patches for Linus.  Even a relatively small buffer 
is a good thing to prevent data loss (16550a anyone?  Better than nothing...) 
 As long as the patches in the queue are maintained and kept up to date 
(which is work that is not being coordinated right now: person A's change 
breaks person B's patch, how does person A know if he doesn't apply person 
B's patch to his tree?).

And there's also the possibility that "judgement call" patches the patch 
penguin didn't want to include could be listed as "known to apply cleanly 
against this tree, but not included".  Just a page listing URLs to patches 
that are being tracked.  This has not previously been done by Alan, Dave, or 
Andrea, and maybe there's a reason why...

> 4) testing
>
> there are code areas and methods which need more rigorous testing and
> third-party feedback - no matter how good the patch. Most notably, if a
> patch exports some new user-space visible interface, then this item
> applies. An example is the aio patch, which had all 3 items right but was
> rejected due to this item. [things are improving very well on the aio
> front so i think this will change in the near future.]
>
> impact: a patch penguin just wont solve this issue, because his job, by
> definition, is not to keep patches around indefinitely, but to filter them
> to Linus.

Yes and no. Alan did more than that.  His tree contained stuff (like User 
Mode Linux) that he didn't immediately mean to forward to Linus.  Stuff HAS 
historically gone into patch penguin trees because the patch penguin likes 
the idea but believes it needs wider testing.

This stuff is usually a compile option.  XFS and JFS come to mind here, 
although that just points out we need a unified journaling layer, which is an 
ongoing discussion.  (Will a unified journaling layer come about until a tree 
exists that contains XFS, JFS, EXT3, and ReiserFS, and then people start 
scrutinizing the mess and combining common code?  I dunno...)

> Only separately maintained patches/trees help here. More people
> are willing to maintain separate trees is good (-dj, -ac, -aa, etc.), one
> tree can do a nontrivial transition at a time,

That's a seperately maintained patch that has not been integrated into a 
tree.  All patches apply to a Linux tree in order to get compiled into a 
system, and all patches could be downloaded as a tree (as the XFS guys do).

When I say tree I'm talking about a tree that's integrating patches from more 
than one source.  This is a step that comes after the patch has gone about as 
far as it's likely to as a seperate patch, seems to work pretty well, and 
needs testing by a wider audience in order to squeeze out more bugs or get 
more code review and comments on its design.

> and by having more of them
> we can eg. get one of them testing aio, the other one testing some other
> big change.

Sure.  This happens now.  The question is, what happens after the JFS people 
say "okay, we've reached version 1.0 now, please try this" and the patch 
still doesn't get integrated into a "beat me, whip me, make me break" tree 
for wider testing for months and months and months?

> A single patch penguin will be able to do only one nontrivial
> transition - and it's not his task to do nontrivial transitions to begin
> with.

I don't know what you mean here.

> Many people who dont actually maintain any Linux code are quoting Rik's
> complains as an example. I'll now have to go on record disagreeing with
> Rik humbly, i believe he has done a number of patch-management mistakes
> during his earlier VM development, and i strongly believe the reason why
> Linus ignored some of his patches were due to these issues. Rik's flames
> against Linus are understandable but are just that: flames. Fortunately
> Rik has learned meanwhile (we all do) and his rmap patches are IMHO
> top-notch. Joining the Andrea improvements and Rik's tree could provide a
> truly fantastic VM. [i'm not going to say anything about the IDE patches
> situation because while i believe Rik understands public criticism, i
> failed to have an impact on Andre before :-) ]

I understand that a lot of the problems aren't purely on Linus's end.  You 
didn't add Richard Gooch to that list with Rik and Andre (although Al seems 
to have decided it's one of his missions in life to keep Richard adhering to 
the coding style.... :)

But right now the individual maintainers need to be really good at patch 
management or the system breaks down.  This isn't exactly a scalability 
question, this is a reliability question.  There's no failure recovery 
mechanism here: if one part of the distributed system breaks the results are 
visible at the end.  You can't scale to more components if all of them most 
work perfectly every time and expect to have a more reliable system.

> also, many people just start off with a single big patch. That just doesnt
> work and you'll likely violate one of the 4 items without even noticing
> it. Start small, because for small patches people will have the few
> minutes needed to teach you.

The kernel is currently FULL of warnings when it used to have none, and 
outright compile errors that go unfixed for several versions if they occur in 
less-often used subsystems.

Small patches seem more likely to get dropped than big ones.  This could be 
an artifact of perception, I dunno...

> The bigger a patch, the harder it is to
> review it, and the less likely it happens. Also, if a few or your patches
> have gone into the Linux tree that does not mean you are senior kernel
> hacker and can start off writing the one big, multi-megabyte super-feature
> you dreamt about for years. Start small and increase the complexity of
> your patches slowly - and perhaps later on you'll notice that that
> super-feature isnt all that super anymore. People also underestimate the
> kind of complexity explosion that occurs if a large patch is created.
> Instead of 1-2 places, you can create 100-200 problems.
>
> face it, most of the patches rejected by Linus are not due to overload. He
> doesnt guarantee to say why he rejects patches - *and he must not*. Just
> knowing that your patch got rejected and thinking it all over again often
> helps finding problems that Linus missed first time around. If you submit
> to Linus then you better know exactly what you do.

There seems to be a perception that on at least some of the occasions Linus 
said "it didn't get applied because nobody ever sent me that patch", people 
were under the impression that the patch HAD been sent to him.

Sending a patch and hearing no reply, resending the same patch and then 
having it applied...  That sends mixed signals.  Resending the same patch a 
dozen times before it gets applied, how do you know if it's being rejected 
for a reason or if it's just a bad time?

> it's so much easier to blame Linus, or maintainers.

It's not a question of "blame".  Why does everybody keep thinking it's a 
question of blame?

It's "There seems to be a problem.  How do we fix it?"

The replies I've gotten range from denying there is any problem, assurances 
that the situation is functioning as maximum possible efficiency and cannot 
be improved, assurances that the problem is purely perceptual on my part, and 
a couple variations on "just live with it".

If this is the consensus...

> It's so much easier to
> fire off an email flaming Linus and getting off the steam than to actually
> accept the possibility of mistake and *fix* the patch. I'll go on record
> saying that good patches are not ignored, even these days when the number
> of active kernel hackers has multipled.

So patches that fix simple build breakage in things like the radeon or ymfpci 
drivers do not need to be resubmitted for multiple point releases?

> People might have to go through
> several layers first, and finding some kernel hacker who is not as loaded
> as Linus to review your patch might be necessery as well (especially if
> the patch is complex), but if you go through the right layers then you can
> be sure that nothing worthwile gets rejected arbitrarily.

Uh-huh.  Find people to review your patch, go through the right layers...

And a paralell tree to Linus's, dedicated to patch processing and tracking 
patches, with a patch submission system dedicated to routing patches to the 
proper maintainers, reviewing and cross-checking patches from maintainers, 
resolving conflicts, subjecting the lot to public scrutiny and being a 
one-stop-shopping place for people who want to find bugs in something...  
Like Alan Cox did for years, and like Dave Jones is doing now...  This is a 
totally different subject then?

Are people saying this is a bad thing?  Saying that this is useless?

Oh well, it was just a suggestion.  Seemed kind of a safe one since we were 
already mostly DOING it...

> 	Ingo

Rob

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Rob Landley wrote:

> And a paralell tree to Linus's, dedicated to patch processing and
> tracking patches, with a patch submission system dedicated to routing
> patches to the proper maintainers, reviewing and cross-checking
> patches from maintainers, resolving conflicts, subjecting the lot to
> public scrutiny and being a one-stop-shopping place for people who
> want to find bugs in something...  Like Alan Cox did for years, and
> like Dave Jones is doing now...  This is a totally different subject
> then?

you are banging on open doors. We have and need multiple trees. And no, a
*single* 'integration' or 'patch penguin' tree will not be able to solve
this problem. The 'small stuff' tree is a tree that does *not* apply
nontrivial patches. It's a tree for *pure* small stuff.

also, the -ac, -dj and -aa trees all act as a 'small stuff' tree currently
but the problem Alan pointed out is that Linus often rejects small stuff
from these sources as well, which creates high latency for small stuff and
decreases the 'end user' quality of the Linus tree. (and also we lose some
of the newbie developers who by definition start with small stuff.) So by
making a *separate* and *small stuff only* tree Linus could start trusting
those patches as small-stuff-only.

A small stuff tree will not and cannot replace the multiple experimental
trees that explore riskier patches (but only a few a time) like the -dj
tree or the -ac tree. (although i'd say the -ac tree isnt purely that,
it's more like a productization tree. The -dj and the devel-based -aa tree
is a good example.)

this way all the people who have the experience and stamina to integrate
patches can act as an experimental ground to 'cook' bigger patches before
they are sent to Linus. Linus' tree is 'cooking' a few patches as well,
but only in orthogonal areas. Eg. right now we have the bio changes, the
vfs cleanups, the device handing cleanups, and the scheduler cleanups
going on in parallel. The -dj tree might (and does) 'cook' patches that
shouldnt be applied to the Linus tree right now even if they were
'perfect' as a starting point. [i still have to see a complex patch that
is truly perfect and needs no iterations. Much of the true integration
steps are still done in the Linus tree these days.]

but integration (of nontrivial patches) on such level *can* be
parallelized to a certain degree. If patches are 'pre-cooked' well (in the
-ac, -dj and -aa, etc. trees and actual users see and test them) then the
load on the Linus tree and the latency of transition of the Linus tree can
be decreased somewhat. But i think we are still very far from the point
when Linus gets only 'perfect' (nontrivial-) patches. I doubt we'll ever
reach that point, and in that case Linus wont have much fun himself so i
doubt we want to reach that point :)

the small stuff tree on the other hand does not need to be parallelized,
small stuff is atomic and such patches scale almost infinitely. So a
single small stuff tree could indeed not only serve as a trusted source
for Linus, but could also take off the load from the other trees so they
can concentrate on the not-so-small-stuff like driver updates and other
subsystem updates, or even bigger patches. Formalizing and automating the
small-stuff tree might work as well, due to its inherent simplicity.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

mingo@elte.hu (Ingo Molnar)  wrote on 29.01.02 in <Pine.LNX.4.33.0201291324560.3610-100000@localhost.localdomain>:

> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:
>
>   - cleanliness
>   - concept
>   - timing
>   - testing

IIRC, the number 33 referred to esr's Configure.help patch. Which of these  
did he violate?

MfG Kai

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:

 > >   - cleanliness
 > >   - concept
 > >   - timing
 > >   - testing
 > 
 > IIRC, the number 33 referred to esr's Configure.help patch. Which of these  
 > did he violate?

 Timing.  Linus was busy focusing on the block layer.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Denis Vlasenko]

On 30 January 2002 00:46, Dave Jones wrote:
> On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
>  > >   - cleanliness
>  > >   - concept
>  > >   - timing
>  > >   - testing
>  >
>  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
>  > these did he violate?
>
>  Timing.  Linus was busy focusing on the block layer.

Sounds alarming. Linus didn't take documentation updates from designated 
maintainer? For many months? I can't believe in argument that updates were 
able to break _anything_, it's only documentation, right? I could understand 
this if these updates were sent by little known person, but Eric?!

Clearly a scalability problem here :-)

I won't post to this thread more.
Why? It's a flamewar, more _words_ will do nothing. _Action_ is needed.
--
vda

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:57:02AM -0200, Denis Vlasenko wrote:
> On 30 January 2002 00:46, Dave Jones wrote:
> > On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
> >  > >   - cleanliness
> >  > >   - concept
> >  > >   - timing
> >  > >   - testing
> >  >
> >  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
> >  > these did he violate?
> >
> >  Timing.  Linus was busy focusing on the block layer.
> 
> Sounds alarming. Linus didn't take documentation updates from designated 
> maintainer? For many months? I can't believe in argument that updates were 
> able to break _anything_, it's only documentation, right? I could understand 
> this if these updates were sent by little known person, but Eric?!
> 
> Clearly a scalability problem here :-)

Oh-my-lord.  Please re-read this thread, and especially Linus's
2.5.3-pre5 changelog announcement.

Configure.help needed to be split up.  Eric?! was told this repeatedly,
but he did not listen.  Hopefully he will listen to feedback regarding
CML2...  He has even been told repeatedly that he does not
listen to feedback ;-)

	Jeff, chuckling

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 03:29 am, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 09:57:02AM -0200, Denis Vlasenko wrote:
> > On 30 January 2002 00:46, Dave Jones wrote:
> > > On Tue, Jan 29, 2002 at 09:51:00PM +0200, Kai Henningsen wrote:
> > >  > >   - cleanliness
> > >  > >   - concept
> > >  > >   - timing
> > >  > >   - testing
> > >  >
> > >  > IIRC, the number 33 referred to esr's Configure.help patch. Which of
> > >  > these did he violate?
> > >
> > >  Timing.  Linus was busy focusing on the block layer.
> >
> > Sounds alarming. Linus didn't take documentation updates from designated
> > maintainer? For many months? I can't believe in argument that updates
> > were able to break _anything_, it's only documentation, right? I could
> > understand this if these updates were sent by little known person, but
> > Eric?!
> >
> > Clearly a scalability problem here :-)
>
> Oh-my-lord.  Please re-read this thread, and especially Linus's
> 2.5.3-pre5 changelog announcement.
>
> Configure.help needed to be split up.  Eric?! was told this repeatedly,
> but he did not listen.  Hopefully he will listen to feedback regarding
> CML2...  He has even been told repeatedly that he does not
> listen to feedback ;-)
>
> 	Jeff, chuckling

I spoke to Eric earlier today.  (We're co-doing a presenatation at LinuxWorld 
Expo on thursday.)

His take on it was that he understood that Configure.help needed to be split 
up, but since the file was used by CML1 and he was NOT the CML1 maintainer, 
he didn't believe he had the authority to unilaterally change the file format 
in a way that would seriously break CML1.  (And, as it happens, now that the 
change has gone in, it seems the existing configurators are currently broken 
and have no help.)

Considering how much he's been warned so far about the need for CML2 to 
maintain as much compatability as possible with CML1, change as little 
behavior as possible in its first version, and not to make intrustive changes 
into the rest of the codebase...  I think he expected to be flamed alive if 
he broke up the help file before CML2 went in.

I.E. There was a miscommunication.  (The drop from Linus was an actual 
reject, but without an explanation of why it was rejected the reject didn't 
get resolved.  For 33 consecutive versions...)

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> Considering how much he's been warned so far about the need for CML2 to 
> maintain as much compatability as possible with CML1,

Pardon me while I laugh my ass off.


> behavior as possible in its first version, and not to make intrustive changes 
> into the rest of the codebase...  I think he expected to be flamed alive if 
> he broke up the help file before CML2 went in.

> I.E. There was a miscommunication.  (The drop from Linus was an actual 
> reject, but without an explanation of why it was rejected the reject didn't 
> get resolved.  For 33 consecutive versions...)

Getting told something point blank, multiple times, is definitely
-something-.  I suppose you could call that miscommunication.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 04:43 am, Jeff Garzik wrote:
> On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> > Considering how much he's been warned so far about the need for CML2 to
> > maintain as much compatability as possible with CML1,
>
> Pardon me while I laugh my ass off.

[waits...]

> > behavior as possible in its first version, and not to make intrustive
> > changes into the rest of the codebase...  I think he expected to be
> > flamed alive if he broke up the help file before CML2 went in.
> >
> > I.E. There was a miscommunication.  (The drop from Linus was an actual
> > reject, but without an explanation of why it was rejected the reject
> > didn't get resolved.  For 33 consecutive versions...)
>
> Getting told something point blank, multiple times, is definitely
> -something-.  I suppose you could call that miscommunication.

I'm under the impression CML2 already supports the split-up per-directory 
help files, and did long before Linus actually split it up.  Therefore, Eric 
hasn't entirely been ignoring the issue, has he?

Yes, I would call it a miscommunication.

(By the way, if you really want to fix the current cml1 stuff in the 
cheesiest manner possible, what would be wrong with some variant of "find . 
-name "*.hlp" | xargs cat > oldhelpfile.hlp"?  Then the old help file becomes 
a generated file of the new help files.  Why mess with tcl/tk?  Put it in the 
make file as a dependency.  Pardon me if somebody fixed it last night, I seem 
have 91 emails to wade through since then on the patch penguin fallout 
alone...)

> 	Jeff

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 02:40:25PM -0500, Rob Landley wrote:
> On Wednesday 30 January 2002 04:43 am, Jeff Garzik wrote:
> > On Wed, Jan 30, 2002 at 04:38:51AM -0500, Rob Landley wrote:
> > > Considering how much he's been warned so far about the need for CML2 to
> > > maintain as much compatability as possible with CML1,
> >
> > Pardon me while I laugh my ass off.
> 
> [waits...]

Keep waiting, I'm still laughing.


> I'm under the impression CML2 already supports the split-up per-directory 
> help files, and did long before Linus actually split it up.  Therefore, Eric 
> hasn't entirely been ignoring the issue, has he?

>From the kernel's point of view, yes, he has.


> (By the way, if you really want to fix the current cml1 stuff in the 
> cheesiest manner possible, what would be wrong with some variant of "find . 
> -name "*.hlp" | xargs cat > oldhelpfile.hlp"?  Then the old help file becomes 
> a generated file of the new help files.  Why mess with tcl/tk?  Put it in the 
> make file as a dependency.  Pardon me if somebody fixed it last night, I seem 
> have 91 emails to wade through since then on the patch penguin fallout 
> alone...)

That's a hack.  Fix it the right way.

<broken record> this is a devel series, we can afford to wait for the
better fix </broken record>

	Jeff

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> able to break _anything_, it's only documentation, right? I could understand 
> this if these updates were sent by little known person, but Eric?!

Oh come on. In the kernel world Eric is a little known person.

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Ingo Molnar wrote:

> None of the examples you cited so far are convincing to me, and i'd like
> to explain why. I've created and submitted thousands of patches to the
> Linux kernel over the past 4 years (my patch archive doesnt go back more
> than 4 years):
> 
>   # ls patches | wc -l
>      2818
> 
> a fair percentage of those went to Linus as well, and while having seen
> some of them rejected does hurt mentally, i couldnt list one reject from
> Linus that i wouldnt reject *today*. But i sure remember being frustrated
> about rejects when they happened. In any case, i have some experience in
> submitting patches and i'm maintaining a few subsystems, so here's my take
> on the 'patch penguin' issue:
> 
> If a patch gets ignored 33 times in a row then perhaps the person doing
> the patch should first think really hard about the following 4 issues:

1 - why doesn't someone at least ack the damn patch?
2 - why doesn't someone at least ack the damn patch?
3 - why doesn't someone at least ack the damn patch?
4 - why doesn't someone at least ack the damn patch?

There is no better way to avoid getting something good from someone than
to ignore them completely. The fact that things like reisser f/s patches
from the creator don't get in is an example, or don't you think they are
good enough?

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Tomasz Kłoczko]

On Tue, 29 Jan 2002, Ingo Molnar wrote:
[..]
> 1) cleanliness
> 
> code cleanliness is a well-know issue, see Documentation/CodingStyle.  If
> a patch has such problems then maintainers are very likely to help - Linus
> probably wont and shouldnt.

I think place in each directory .indent.pro file with proper coding style
configuration and reduce Documentation/CodingStyle to how to use indent
tool can will solve many currunt problems with proper patches form and
will probaly take smaller amout disk space (or aprox the same) than
current Documentation/CodingStyle. Even if current indent can't handle
correctly current kernel coding style IMHO it will be better inves few
minutes on some changes to current indent behavior for bring this tool
abilities for reindent source code in way described in
Documentation/CodingStyle .. (?)

kloczek
-- 
-----------------------------------------------------------
*Ludzie nie mają problemów, tylko sobie sami je stwarzają*
-----------------------------------------------------------
Tomasz Kłoczko, sys adm @zie.pg.gda.pl|*e-mail: kloczek@rudy.mif.pg.gda.pl*

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 10:23 pm, Linus Torvalds wrote:

> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse.

Oh, one other thing.  I didn't emphasize the possibility that the patch 
penguin might eventually run a public CVS tree that the various subsystem 
maintainers might be granted commit access to, because I didn't want to 
confuse the issue.

You don't use CVS, this proposal is not asking you to use CVS, and you seem 
to dislike other people using CVS.  I'm under the impression this is because 
CVS blurs together the patches you then need to receive and code review to do 
your job as architect of the Linux kernel.

It takes a skilled human being to extract clean patches from a CVS tree and 
feed them on to you in the format you prefer: one per email, plain text, with 
a description at the top.  Clean, atomic patches that do exactly one thing, 
and don't have any cross-reference dependencies on any other pending patches 
in the patch set.  No automated CVS-like tool will ever be able to do that 
AND resolve conflicts between patches.  You need a human.

Extracting patches out of a CVS tree and hand-massaging them into a format 
you would accept would be a big part of the integration maintainer's job.  
If they chose to run a CVS tree.  (Backing the patches out if you rejected 
the whole idea of that particular patch would be a lot of work too, but it 
would also be part of the integration maintainer's job.)

Rob

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Linus Torvalds wrote:

>Some thinking, for one thing.
>
>One "patch penguin" scales no better than I do. In fact, I will claim
>that most of them scale a whole lot worse. 
>
Bla bla bla... Just tell how frequenty do I have to tell the world, that 
the read_ahead array is a write
only variable inside the kernel and therefore not used at 
all?????!!!!!!!!!!

Re: A modest proposal -- We need a patch penguin

[Christoph Hellwig]

Hi Martin,

In article <3C568C52.2060707@evision-ventures.com> you wrote:
>>One "patch penguin" scales no better than I do. In fact, I will claim
>>that most of them scale a whole lot worse. 
>>
> Bla bla bla... Just tell how frequenty do I have to tell the world, that 
> the read_ahead array is a write
> only variable inside the kernel and therefore not used at 
> all?????!!!!!!!!!!

It IS used. (hint: take a look at fs/hfs/file.c).

I still don't think maintainig this array is worth just for hfs
readahead, so the below patch disables it and gets rid of read_ahead.

Jens, could you check the patch and include it in your next batch of
block-layer changes for Linus?

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.


diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/acorn/block/mfmhd.c linux/drivers/acorn/block/mfmhd.c
--- ../master/linux-2.5.3-pre6/drivers/acorn/block/mfmhd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/acorn/block/mfmhd.c	Tue Jan 29 14:07:41 2002
@@ -1442,7 +1442,6 @@
 	hdc63463_irqpollmask	= irqmask;
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB?) read ahread */
 
 	add_gendisk(&mfm_gendisk);
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/DAC960.c linux/drivers/block/DAC960.c
--- ../master/linux-2.5.3-pre6/drivers/block/DAC960.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/DAC960.c	Tue Jan 29 13:57:06 2002
@@ -1964,10 +1964,6 @@
   Controller->GenericDiskInfo.sizes = Controller->PartitionSizes;
   blksize_size[MajorNumber] = Controller->BlockSizes;
   /*
-    Initialize Read Ahead to 128 sectors.
-  */
-  read_ahead[MajorNumber] = 128;
-  /*
     Complete initialization of the Generic Disk Information structure.
   */
   Controller->GenericDiskInfo.major = MajorNumber;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/acsi.c linux/drivers/block/acsi.c
--- ../master/linux-2.5.3-pre6/drivers/block/acsi.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/acsi.c	Tue Jan 29 13:57:19 2002
@@ -1785,7 +1785,6 @@
 	STramMask = ATARIHW_PRESENT(EXTD_DMA) ? 0x00000000 : 0xff000000;
 	
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &acsi_lock);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&acsi_gendisk);
 
 #ifdef CONFIG_ATARI_SLM
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/blkpg.c linux/drivers/block/blkpg.c
--- ../master/linux-2.5.3-pre6/drivers/block/blkpg.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/blkpg.c	Tue Jan 29 13:58:51 2002
@@ -227,18 +227,6 @@
 			intval = (is_read_only(dev) != 0);
 			return put_user(intval, (int *)(arg));
 
-		case BLKRASET:
-			if(!capable(CAP_SYS_ADMIN))
-				return -EACCES;
-			if(arg > 0xff)
-				return -EINVAL;
-			read_ahead[major(dev)] = arg;
-			return 0;
-		case BLKRAGET:
-			if (!arg)
-				return -EINVAL;
-			return put_user(read_ahead[major(dev)], (long *) arg);
-
 		case BLKFRASET:
 			if (!capable(CAP_SYS_ADMIN))
 				return -EACCES;
@@ -319,6 +307,9 @@
 			set_blocksize(dev, intval);
 			return 0;
 
+		case BLKRASET:
+		case BLKRAGET:
+			/* this information is no more used by the kernel */
 		default:
 			return -EINVAL;
 	}
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/cciss.c linux/drivers/block/cciss.c
--- ../master/linux-2.5.3-pre6/drivers/block/cciss.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/cciss.c	Tue Jan 29 13:59:03 2002
@@ -2542,7 +2542,6 @@
 
 	/* fill in the other Kernel structs */
 	blksize_size[MAJOR_NR+i] = hba[i]->blocksizes;
-        read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 	/* Fill in the gendisk data */ 	
 	hba[i]->gendisk.major = MAJOR_NR + i;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/cpqarray.c linux/drivers/block/cpqarray.c
--- ../master/linux-2.5.3-pre6/drivers/block/cpqarray.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/cpqarray.c	Tue Jan 29 13:59:14 2002
@@ -481,7 +481,6 @@
 		blk_queue_max_phys_segments(q, SG_MAX);
 
 		blksize_size[MAJOR_NR+i] = ida_blocksizes + (i*256);
-		read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 		ida_gendisk[i].major = MAJOR_NR + i;
 		ida_gendisk[i].major_name = "ida";
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/ll_rw_blk.c linux/drivers/block/ll_rw_blk.c
--- ../master/linux-2.5.3-pre6/drivers/block/ll_rw_blk.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/ll_rw_blk.c	Tue Jan 29 13:59:28 2002
@@ -54,10 +54,6 @@
  */
 DECLARE_TASK_QUEUE(tq_disk);
 
-/* This specifies how many sectors to read ahead on the disk. */
-
-int read_ahead[MAX_BLKDEV];
-
 /* blk_dev_struct is:
  *	request_queue
  *	*queue
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pcd.c linux/drivers/block/paride/pcd.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pcd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/paride/pcd.c	Tue Jan 29 14:00:33 2002
@@ -358,7 +358,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &pcd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 
 	for (i=0;i<PCD_UNITS;i++) pcd_blocksizes[i] = 1024;
         blksize_size[MAJOR_NR] = pcd_blocksizes;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pd.c linux/drivers/block/paride/pd.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/paride/pd.c	Tue Jan 29 14:00:41 2002
@@ -397,7 +397,6 @@
 	q = BLK_DEFAULT_QUEUE(MAJOR_NR);
 	blk_init_queue(q, DEVICE_REQUEST, &pd_lock);
 	blk_queue_max_sectors(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	pd_gendisk.major = major;
 	pd_gendisk.major_name = name;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/paride/pf.c linux/drivers/block/paride/pf.c
--- ../master/linux-2.5.3-pre6/drivers/block/paride/pf.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/paride/pf.c	Tue Jan 29 14:00:23 2002
@@ -363,7 +363,6 @@
 	blk_init_queue(q, DEVICE_REQUEST, &pf_spin_lock);
 	blk_queue_max_phys_segments(q, cluster);
 	blk_queue_max_hw_segments(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	for (i=0;i<PF_UNITS;i++) pf_blocksizes[i] = 1024;
 	blksize_size[MAJOR_NR] = pf_blocksizes;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/ps2esdi.c linux/drivers/block/ps2esdi.c
--- ../master/linux-2.5.3-pre6/drivers/block/ps2esdi.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/block/ps2esdi.c	Tue Jan 29 14:00:11 2002
@@ -177,8 +177,6 @@
 	}
 	/* set up some global information - indicating device specific info */
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &ps2esdi_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
-
 	/* some minor housekeeping - setup the global gendisk structure */
 	add_gendisk(&ps2esdi_gendisk);
 	ps2esdi_geninit();
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/block/xd.c linux/drivers/block/xd.c
--- ../master/linux-2.5.3-pre6/drivers/block/xd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/block/xd.c	Tue Jan 29 13:59:39 2002
@@ -171,7 +171,6 @@
 	}
 	devfs_handle = devfs_mk_dir (NULL, xd_gendisk.major_name, NULL);
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &xd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 	add_gendisk(&xd_gendisk);
 	xd_geninit();
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/aztcd.c linux/drivers/cdrom/aztcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/aztcd.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/cdrom/aztcd.c	Tue Jan 29 14:05:15 2002
@@ -1927,7 +1927,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &aztSpin);
 	blksize_size[MAJOR_NR] = aztcd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &azt_fops, 0);
 
 	if ((azt_port == 0x1f0) || (azt_port == 0x170))
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/cdu31a.c linux/drivers/cdrom/cdu31a.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/cdu31a.c	Tue Jan 15 10:59:05 2002
+++ linux/drivers/cdrom/cdu31a.c	Tue Jan 29 14:05:21 2002
@@ -3442,7 +3442,6 @@
 		blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR),
 			       DEVICE_REQUEST,
 			       &cdu31a_lock);
-		read_ahead[MAJOR_NR] = CDU31A_READAHEAD;
 		cdu31a_block_size = 1024;	/* 1kB default block size */
 		/* use 'mount -o block=2048' */
 		blksize_size[MAJOR_NR] = &cdu31a_block_size;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/cm206.c linux/drivers/cdrom/cm206.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/cm206.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/cm206.c	Tue Jan 29 14:05:43 2002
@@ -1503,7 +1503,6 @@
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &cm206_lock);
 	blksize_size[MAJOR_NR] = cm206_blocksizes;
-	read_ahead[MAJOR_NR] = 16;	/* reads ahead what? */
 	init_bh(CM206_BH, cm206_bh);
 
 	memset(cd, 0, sizeof(*cd));	/* give'm some reasonable value */
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/gscd.c linux/drivers/cdrom/gscd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/gscd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/gscd.c	Tue Jan 29 14:05:50 2002
@@ -1022,7 +1022,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &gscd_lock);
 	blksize_size[MAJOR_NR] = gscd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 
 	disk_state = 0;
 	gscdPresent = 1;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/mcd.c linux/drivers/cdrom/mcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/mcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/mcd.c	Tue Jan 29 14:05:56 2002
@@ -1075,7 +1075,6 @@
 	blksize_size[MAJOR_NR] = mcd_blocksizes;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcd_spinlock);
-	read_ahead[MAJOR_NR] = 4;
 
 	/* check for card */
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/mcdx.c linux/drivers/cdrom/mcdx.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/mcdx.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/mcdx.c	Tue Jan 29 14:06:01 2002
@@ -1184,7 +1184,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcdx_lock);
-	read_ahead[MAJOR_NR] = READ_AHEAD;
 	blksize_size[MAJOR_NR] = mcdx_blocksizes;
 
 	xtrace(INIT, "init() subscribe irq and i/o\n");
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/optcd.c linux/drivers/cdrom/optcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/optcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/optcd.c	Tue Jan 29 14:06:06 2002
@@ -2062,7 +2062,6 @@
 	blksize_size[MAJOR_NR] = &blksize;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &optcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	request_region(optcd_port, 4, "optcd");
 	register_disk(NULL, mk_kdev(MAJOR_NR,0), 1, &opt_fops, 0);
 
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sbpcd.c linux/drivers/cdrom/sbpcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sbpcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sbpcd.c	Tue Jan 29 14:07:22 2002
@@ -4532,11 +4532,7 @@
 	} /* end of CDROMREADAUDIO */
 		
 	case BLKRASET:
-		if(!capable(CAP_SYS_ADMIN)) RETURN_UP(-EACCES);
-		if(kdev_none(cdi->dev)) RETURN_UP(-EINVAL);
-		if(arg > 0xff) RETURN_UP(-EINVAL);
-		read_ahead[major(cdi->dev)] = arg;
-		RETURN_UP(0);
+		return -EINVAL;
 	default:
 		msg(DBG_IOC,"ioctl: unknown function request %04X\n", cmd);
 		RETURN_UP(-EINVAL);
@@ -5870,7 +5866,6 @@
 	(BLK_DEFAULT_QUEUE(MAJOR_NR))->front_merge_fn = dont_bh_merge_fn;
 	(BLK_DEFAULT_QUEUE(MAJOR_NR))->merge_requests_fn = dont_merge_requests_fn;
 #endif
-	read_ahead[MAJOR_NR] = buffers * (CD_FRAMESIZE / 512);
 	
 	request_region(CDo_command,4,major_name);
 	
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sjcd.c linux/drivers/cdrom/sjcd.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sjcd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sjcd.c	Tue Jan 29 14:06:10 2002
@@ -1695,7 +1695,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,&sjcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &sjcd_fops, 0);
 
 	if (check_region(sjcd_base, 4)) {
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/cdrom/sonycd535.c linux/drivers/cdrom/sonycd535.c
--- ../master/linux-2.5.3-pre6/drivers/cdrom/sonycd535.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/cdrom/sonycd535.c	Tue Jan 29 14:06:16 2002
@@ -1598,7 +1598,6 @@
 				}
 				blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &sonycd535_lock);
 				blksize_size[MAJOR_NR] = &sonycd535_block_size;
-				read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read-ahead */
 
 				sony_toc = (struct s535_sony_toc *)
 					kmalloc(sizeof *sony_toc, GFP_KERNEL);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/hd.c linux/drivers/ide/hd.c
--- ../master/linux-2.5.3-pre6/drivers/ide/hd.c	Tue Jan 15 10:59:06 2002
+++ linux/drivers/ide/hd.c	Tue Jan 29 14:04:07 2002
@@ -837,7 +837,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &hd_lock);
 	blk_queue_max_sectors(BLK_DEFAULT_QUEUE(MAJOR_NR), 255);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&hd_gendisk);
 	init_timer(&device_timer);
 	device_timer.function = hd_times_out;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-cd.c linux/drivers/ide/ide-cd.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-cd.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-cd.c	Tue Jan 29 14:04:51 2002
@@ -2662,7 +2662,6 @@
 	int major = HWIF(drive)->major;
 	int minor = drive->select.b.unit << PARTN_BITS;
 
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW, BLKRAGET, BLKRASET, TYPE_INT, 0, 255, 1, 2, &read_ahead[major], NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW, BLKFRAGET, BLKFRASET, TYPE_INTA, 0, INT_MAX, 1, 1024, &max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"dsc_overlap",		SETTING_RW, -1, -1, TYPE_BYTE, 0, 1, 1,	1, &drive->dsc_overlap, NULL);
 }
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-disk.c linux/drivers/ide/ide-disk.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-disk.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-disk.c	Tue Jan 29 14:04:45 2002
@@ -931,7 +931,6 @@
 	ide_add_setting(drive,	"bswap",		SETTING_READ,					-1,			-1,			TYPE_BYTE,	0,	1,				1,	1,	&drive->bswap,			NULL);
 	ide_add_setting(drive,	"multcount",		id ? SETTING_RW : SETTING_READ,			HDIO_GET_MULTCOUNT,	HDIO_SET_MULTCOUNT,	TYPE_BYTE,	0,	id ? id->max_multsect : 0,	1,	1,	&drive->mult_count,		set_multcount);
 	ide_add_setting(drive,	"nowerr",		SETTING_RW,					HDIO_GET_NOWERR,	HDIO_SET_NOWERR,	TYPE_BYTE,	0,	1,				1,	1,	&drive->nowerr,			set_nowerr);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	1,	&read_ahead[major],		NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	4096,			PAGE_SIZE,	1024,	&max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"lun",			SETTING_RW,					-1,			-1,			TYPE_INT,	0,	7,				1,	1,	&drive->lun,			NULL);
 	ide_add_setting(drive,	"wcache",		SETTING_RW,					HDIO_GET_WCACHE,	HDIO_SET_WCACHE,	TYPE_BYTE,	0,	1,				1,	1,	&drive->wcache,			write_cache);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-floppy.c linux/drivers/ide/ide-floppy.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-floppy.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-floppy.c	Tue Jan 29 14:04:40 2002
@@ -1968,7 +1968,6 @@
 	ide_add_setting(drive,	"bios_cyl",		SETTING_RW,					-1,			-1,			TYPE_INT,	0,	1023,				1,	1,	&drive->bios_cyl,		NULL);
 	ide_add_setting(drive,	"bios_head",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	255,				1,	1,	&drive->bios_head,		NULL);
 	ide_add_setting(drive,	"bios_sect",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	63,				1,	1,	&drive->bios_sect,		NULL);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	2,	&read_ahead[major],		NULL);
 	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	INT_MAX,			1,	1024,	&max_readahead[major][minor],	NULL);
 
 }
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/ide/ide-probe.c linux/drivers/ide/ide-probe.c
--- ../master/linux-2.5.3-pre6/drivers/ide/ide-probe.c	Tue Jan 29 11:24:20 2002
+++ linux/drivers/ide/ide-probe.c	Tue Jan 29 14:04:22 2002
@@ -937,7 +937,6 @@
 	init_gendisk(hwif);
 	blk_dev[hwif->major].data = hwif;
 	blk_dev[hwif->major].queue = ide_get_queue;
-	read_ahead[hwif->major] = 8;	/* (4kB) */
 	hwif->present = 1;	/* success */
 
 	return hwif->present;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/md/md.c linux/drivers/md/md.c
--- ../master/linux-2.5.3-pre6/drivers/md/md.c	Tue Jan 29 11:24:21 2002
+++ linux/drivers/md/md.c	Tue Jan 29 13:56:35 2002
@@ -1737,7 +1737,6 @@
 	register_disk(&md_gendisk, mk_kdev(MAJOR_NR,mdidx(mddev)),
 			1, &md_fops, md_size[mdidx(mddev)]<<1);
 
-	read_ahead[MD_MAJOR] = 1024;
 	return (0);
 }
 
@@ -3177,11 +3176,7 @@
 	sz += sprintf(page+sz, "\n");
 
 
-	sz += sprintf(page+sz, "read_ahead ");
-	if (read_ahead[MD_MAJOR] == INT_MAX)
-		sz += sprintf(page+sz, "not set\n");
-	else
-		sz += sprintf(page+sz, "%d sectors\n", read_ahead[MD_MAJOR]);
+	sz += sprintf(page+sz, "read_ahead not set\n");
 
 	ITERATE_MDDEV(mddev,tmp) {
 		sz += sprintf(page + sz, "md%d : %sactive", mdidx(mddev),
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/s390/block/xpram.c linux/drivers/s390/block/xpram.c
--- ../master/linux-2.5.3-pre6/drivers/s390/block/xpram.c	Tue Jan 15 10:59:08 2002
+++ linux/drivers/s390/block/xpram.c	Tue Jan 29 14:02:25 2002
@@ -163,12 +163,11 @@
 
 static int major    = XPRAM_MAJOR;
 static int devs     = XPRAM_DEVS;
-static int rahead   = XPRAM_RAHEAD;
 static int sizes[XPRAM_MAX_DEVS] = { 0, };
 static int blksize  = XPRAM_BLKSIZE;
 static int hardsect = XPRAM_HARDSECT;
 
-int xpram_devs, xpram_rahead;
+int xpram_devs;
 int xpram_blksize, xpram_hardsect;
 int xpram_mem_avail = 0;
 unsigned long xpram_sizes[XPRAM_MAX_DEVS];
@@ -660,21 +659,7 @@
 		return 0;
 
 	case BLKRAGET: /* return the readahead value, 0x1263 */
-		if (!arg)  return -EINVAL;
-		err = 0; /* verify_area_20(VERIFY_WRITE, (long *) arg, sizeof(long));
-		          * if (err) return err;
-                          */
-		put_user(read_ahead[MAJOR(inode->i_rdev)], (long *)arg);
-
-		return 0;
-
 	case BLKRASET: /* set the readahead value, 0x1262 */
-		if (!capable(CAP_SYS_ADMIN)) return -EACCES;
-		if (arg > 0xff) return -EINVAL; /* limit it */
-		read_ahead[MAJOR(inode->i_rdev)] = arg;
-                atomic_eieio();
-		return 0;
-
 	case BLKRRPART: /* re-read partition table: can't do it, 0x1259 */
 		return -EINVAL;
 
@@ -685,7 +670,6 @@
 						* BLKROGET
                                                 */
 #endif /* V22 */
-
 	case HDIO_GETGEO:
 		/*
 		 * get geometry: we have to fake one...  trim the size to a
@@ -940,7 +924,6 @@
 				 * snoozing with a debugger.
 				 */
 
-	xpram_rahead   = rahead;
 	xpram_blksize  = blksize;
 	xpram_hardsect = hardsect;
 
@@ -1029,7 +1012,7 @@
 	PRINT_INFO("  %d kB expanded memory found.\n",xpram_mem_avail );
 
 	/*
-	 * Assign the other needed values: request, rahead, size, blksize,
+	 * Assign the other needed values: request, size, blksize,
 	 * hardsect. All the minor devices feature the same value.
 	 * Note that `xpram' defines all of them to allow testing non-default
 	 * values. A real device could well avoid setting values in global
@@ -1042,7 +1025,6 @@
 	q = BLK_DEFAULT_QUEUE (major);
 	blk_init_queue (q, xpram_request);
 #endif /* V22/V24 */
-	read_ahead[major] = xpram_rahead;
 
 	/* we want to have XPRAM_UNUSED blocks security buffer between devices */
 	mem_usable=xpram_mem_avail-(XPRAM_UNUSED*(xpram_devs-1));
@@ -1181,7 +1163,6 @@
 	kfree(xpram_hardsects);
 	hardsect_size[major] = NULL;
  fail_malloc:
-	read_ahead[major] = 0;
 #if (XPRAM_VERSION == 22)
 	blk_dev[major].request_fn = NULL;
 #endif /* V22 */
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/s390/char/tapeblock.c linux/drivers/s390/char/tapeblock.c
--- ../master/linux-2.5.3-pre6/drivers/s390/char/tapeblock.c	Sat Dec 22 20:06:57 2001
+++ linux/drivers/s390/char/tapeblock.c	Tue Jan 29 14:02:34 2002
@@ -101,7 +101,6 @@
     }
     if (tapeblock_major == 0) tapeblock_major = result;   /* accept dynamic major number*/
     INIT_BLK_DEV(tapeblock_major,tape_request_fn,tapeblock_getqueue,NULL);
-    read_ahead[tapeblock_major]=TAPEBLOCK_READAHEAD;
     PRINT_WARN(KERN_ERR " tape gets major %d for block device\n", result);
     blk_size[tapeblock_major] = (int*) kmalloc (256*sizeof(int),GFP_ATOMIC);
     memset(blk_size[tapeblock_major],0,256*sizeof(int));
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/scsi/sd.c linux/drivers/scsi/sd.c
--- ../master/linux-2.5.3-pre6/drivers/scsi/sd.c	Tue Jan 15 10:59:09 2002
+++ linux/drivers/scsi/sd.c	Tue Jan 29 14:03:01 2002
@@ -1179,7 +1179,7 @@
 		add_gendisk(&(sd_gendisks[i]));
 	}
 
-	for (i = 0; i < sd_template.dev_max; ++i)
+	for (i = 0; i < sd_template.dev_max; ++i) {
 		if (!rscsi_disks[i].capacity && rscsi_disks[i].device) {
 			sd_init_onedisk(i);
 			if (!rscsi_disks[i].has_part_table) {
@@ -1190,17 +1190,6 @@
 				rscsi_disks[i].has_part_table = 1;
 			}
 		}
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 60 blocks (120 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. We can only respect this with the
-	 * granularity of every 16 disks (one device major).
-	 */
-	for (i = 0; i < N_USED_SD_MAJORS; i++) {
-		read_ahead[SD_MAJOR(i)] =
-		    (rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device
-		     && rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device->host->sg_tablesize)
-		    ? 120	/* 120 sector read-ahead */
-		    : 4;	/* 4 sector read-ahead */
 	}
 
 	return;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/drivers/scsi/sr.c linux/drivers/scsi/sr.c
--- ../master/linux-2.5.3-pre6/drivers/scsi/sr.c	Tue Jan 29 11:24:22 2002
+++ linux/drivers/scsi/sr.c	Tue Jan 29 14:03:53 2002
@@ -785,16 +785,6 @@
                                     &sr_bdops, NULL);
 		register_cdrom(&scsi_CDs[i].cdi);
 	}
-
-
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 16 blocks (32 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. */
-	if (scsi_CDs[0].device && scsi_CDs[0].device->host->sg_tablesize)
-		read_ahead[MAJOR_NR] = 32;	/* 32 sector read-ahead.  Always removable. */
-	else
-		read_ahead[MAJOR_NR] = 4;	/* 4 sector read-ahead */
-
 }
 
 static void sr_detach(Scsi_Device * SDp)
@@ -846,7 +836,6 @@
 		kfree(sr_blocksizes);
 		sr_blocksizes = NULL;
 	}
-	read_ahead[MAJOR_NR] = 0;
 	blk_clear(MAJOR_NR);
 
 	sr_template.dev_max = 0;
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/fs/hfs/file.c linux/fs/hfs/file.c
--- ../master/linux-2.5.3-pre6/fs/hfs/file.c	Tue Jan 15 10:59:13 2002
+++ linux/fs/hfs/file.c	Tue Jan 29 14:09:18 2002
@@ -309,6 +309,7 @@
 	blocks = (count+offset+HFS_SECTOR_SIZE-1) >> HFS_SECTOR_SIZE_BITS;
 
 	bhb = bhe = buflist;
+#if 0 /* for readahead we have the pagecache..  --hch */
 	if (reada) {
 		if (blocks < read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9)) {
 			blocks = read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9);
@@ -317,6 +318,7 @@
 			blocks = size - block;
 		}
 	}
+#endif
 
 	/* We do this in a two stage process.  We first try and
 	   request as many blocks as we can, then we wait for the
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/include/linux/blkdev.h linux/include/linux/blkdev.h
--- ../master/linux-2.5.3-pre6/include/linux/blkdev.h	Tue Jan 29 11:24:24 2002
+++ linux/include/linux/blkdev.h	Tue Jan 29 13:54:49 2002
@@ -341,7 +341,6 @@
 #endif
 	blksize_size[major] = NULL;
 	max_readahead[major] = NULL;
-	read_ahead[major] = 0;
 }
 
 extern inline int get_hardsect_size(kdev_t dev)
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/include/linux/fs.h linux/include/linux/fs.h
--- ../master/linux-2.5.3-pre6/include/linux/fs.h	Tue Jan 29 11:24:24 2002
+++ linux/include/linux/fs.h	Tue Jan 29 13:54:36 2002
@@ -1484,7 +1484,6 @@
 
 extern ssize_t char_read(struct file *, char *, size_t, loff_t *);
 extern ssize_t block_read(struct file *, char *, size_t, loff_t *);
-extern int read_ahead[];
 
 extern ssize_t char_write(struct file *, const char *, size_t, loff_t *);
 extern ssize_t block_write(struct file *, const char *, size_t, loff_t *);
diff -uNr -Xdontdiff ../master/linux-2.5.3-pre6/kernel/ksyms.c linux/kernel/ksyms.c
--- ../master/linux-2.5.3-pre6/kernel/ksyms.c	Tue Jan 29 11:24:24 2002
+++ linux/kernel/ksyms.c	Tue Jan 29 13:54:15 2002
@@ -509,7 +509,6 @@
 EXPORT_SYMBOL(clear_inode);
 EXPORT_SYMBOL(___strtok);
 EXPORT_SYMBOL(init_special_inode);
-EXPORT_SYMBOL(read_ahead);
 EXPORT_SYMBOL(__get_hash_table);
 EXPORT_SYMBOL(new_inode);
 EXPORT_SYMBOL(insert_inode_hash);

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> I still don't think maintainig this array is worth just for hfs
> readahead, so the below patch disables it and gets rid of read_ahead.
> 
> Jens, could you check the patch and include it in your next batch of
> block-layer changes for Linus?

What would be significantly more useful would be to make it actually work.
Lots of drivers benefit from control over readahead sizes - both the
stunningly slow low end stuff and the high end raid cards that often want
to get hit by very large I/O requests (eg 128K for the ami megaraid)

Alan

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Alan Cox wrote:

>>I still don't think maintainig this array is worth just for hfs
>>readahead, so the below patch disables it and gets rid of read_ahead.
>>
>>Jens, could you check the patch and include it in your next batch of
>>block-layer changes for Linus?
>>
>
>What would be significantly more useful would be to make it actually work.
>Lots of drivers benefit from control over readahead sizes - both the
>stunningly slow low end stuff and the high end raid cards that often want
>to get hit by very large I/O requests (eg 128K for the ami megaraid)
>
No you are wrong. This array is supposed to provide a readahead setting 
on the driver level, which is bogous, since
it's something that *should* not be exposed to the upper layers at all. 
Please note as well that
 we have already max_readahead in struut block_device as well. Please 
note that this array only has
a granularity of major block device numbers which is compleatly bogous 
for example for a disk and
cd-rom hanging on a IDE interface. And so on and so on... It's really 
better to let it go.

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> No you are wrong. This array is supposed to provide a readahead setting 
> on the driver level, which is bogous, since
> it's something that *should* not be exposed to the upper layers at all. 

Right.

>  we have already max_readahead in struut block_device as well. Please 
> note that this array only has

Ok. Now I look at it again yes - the array is completely surplus to current
requirements. 2.5 nicely sorts out the queues 

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Christoph Hellwig wrote:

>Hi Martin,
>
>In article <3C568C52.2060707@evision-ventures.com> you wrote:
>
>>>One "patch penguin" scales no better than I do. In fact, I will claim
>>>that most of them scale a whole lot worse. 
>>>
>>Bla bla bla... Just tell how frequenty do I have to tell the world, that 
>>the read_ahead array is a write
>>only variable inside the kernel and therefore not used at 
>>all?????!!!!!!!!!!
>>
>
>It IS used. (hint: take a look at fs/hfs/file.c).
>

Right, but the usage there is semantically *invalid*.

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >One "patch penguin" scales no better than I do. In fact, I will claim
> >that most of them scale a whole lot worse.

> Bla bla bla... Just tell how frequenty do I have to tell the world,
> that the read_ahead array is a write only variable inside the kernel
> and therefore not used at all?????!!!!!!!!!!

tell Jens. He goes about fixing it all, not just the most visible pieces
that showed how much the Linux block IO code sucked. And guess what? His
patches are being accepted, and the Linux 2.5 block IO code is evolving
rapidly. Sometimes keeping broken code around as an incentive to fix it
*for real* is better than trying to massage the broken code somewhat.

a patch penguin doesnt solve this particular problem, by definition he
just wont fix the block IO code.

any other 'examples'?

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Tue, 29 Jan 2002, Martin Dalecki wrote:
>
>>Bla bla bla... Just tell how frequenty do I have to tell the world,
>>that the read_ahead array is a write only variable inside the kernel
>>and therefore not used at all?????!!!!!!!!!!
>>
>tell Jens. He goes about fixing it all, not just the most visible pieces
>that showed how much the Linux block IO code sucked. And guess what? His
>patches are being accepted, and the Linux 2.5 block IO code is evolving
>rapidly. Sometimes keeping broken code around as an incentive to fix it
>*for real* is better than trying to massage the broken code somewhat.
>
There is nothing easier to fix then this. You just have to grep for it, 
or just remove the declaration and wait
to be hit by this during the compilation. And most interrestingly this 
is *easier* then sending a patch!
A patch for this particular problem tend't to
1. Touch many things (however in a trivial way!)
2. Have spurious conflicts in terms of synchronization with the overall 
developement tree of the maintainer
in question.

Now dear linus tell me a better way to deal with *this* kind of problem 
then using CVS for example
where not a single man has the overall controll.

Yes my opinnin is indeed that in reality our problem is that Linus just 
doesn't want to give up
some kind of controll - no more no less.

>
>
>a patch penguin doesnt solve this particular problem, by definition he
>just wont fix the block IO code.
>
>any other 'examples'
>

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Martin Dalecki wrote:

> >tell Jens. He goes about fixing it all, not just the most visible pieces
> >that showed how much the Linux block IO code sucked. And guess what? His
> >patches are being accepted, and the Linux 2.5 block IO code is evolving
> >rapidly. Sometimes keeping broken code around as an incentive to fix it
> >*for real* is better than trying to massage the broken code somewhat.
> >

> There is nothing easier to fix then this. You just have to grep for
> it, or just remove the declaration and wait to be hit by this during
> the compilation. [...]

you have completely and totally ignored my argument.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Martin Dalecki]

Ingo Molnar wrote:

>On Tue, 29 Jan 2002, Martin Dalecki wrote:
>
>>>tell Jens. He goes about fixing it all, not just the most visible pieces
>>>that showed how much the Linux block IO code sucked. And guess what? His
>>>patches are being accepted, and the Linux 2.5 block IO code is evolving
>>>rapidly. Sometimes keeping broken code around as an incentive to fix it
>>>*for real* is better than trying to massage the broken code somewhat.
>>>
>>There is nothing easier to fix then this. You just have to grep for
>>it, or just remove the declaration and wait to be hit by this during
>>the compilation. [...]
>>
>>you have completely and totally ignored my argument.	
>>
And you didn't look at the issue. Abstract arguments from you sound only 
like a dialogue with
a AI programm.

1. Telling Jest - he is supposed to read lkml. I'm continuously raising 
this issue since several moths already.
2. *for real* - removing it is the REAL fix.

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> rapidly. Sometimes keeping broken code around as an incentive to fix it
> *for real* is better than trying to massage the broken code somewhat.
> 
> a patch penguin doesnt solve this particular problem, by definition he
> just wont fix the block IO code.

Ingo, you should have a look at my mailbox and the people sick of trying to
get Linus to take 3 liners to fix NODEV type stuff and being ignored so that
2.5.* still doesn't even compile or boot for many people.

Dave in doing the patch hoovering at least ensures these are picked up. You
think if this carries on anyone will be running Linus tree in 9 months ?

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> Ingo, you should have a look at my mailbox and the people sick of
> trying to get Linus to take 3 liners to fix NODEV type stuff and being
> ignored so that 2.5.* still doesn't even compile or boot for many
> people.

for code areas where there is not active maintainer or the maintainer has
ignored patches? Eg. the majority of the kdev transition patches went in
smoothly.

but i'm not claiming that everything is rosy (i would post patches if
everything was rosy in Linux-land), but i disagree with the majority of
serious examples i've seen cited.

> Dave in doing the patch hoovering at least ensures these are picked
> up. You think if this carries on anyone will be running Linus tree in
> 9 months ?

Dave and you doing patch hoovering is indeed very good. One reason is that
it multithreads the introduction of more risky stuff (by splitting up the
testing effort), and builds up confidence in more complex patches. This is
especially important in the stable cycle - eg. it happened not once that
had eg. some APIC cleanup that was too risky to be added to the stable
branch, and which went into your branch and lived a few iterations (got a
few bugreports) and then went to Linus.

Obviously you wont apply all the complex patches at once - i remember that
occasionally you delayed certain patches of mine because something else
was happening in your tree at that monent. You are simply doing
*different* transitions, but you are constrained by the same basic limits
as Linus' tree is.

Another reason is that you do much more housekeeping in areas that are not
actively maintained. But wouldnt it be better if there were active
maintainers in those areas as well so you could spend more time on eg.
doing the kernel-stack coloring changes?

but i truly believe that for the hard issues there is no solution, and
that most of the patch rejects are due to hard issues.

	Ingo

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> for code areas where there is not active maintainer or the maintainer has
> ignored patches? Eg. the majority of the kdev transition patches went in
> smoothly.

No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
if you do some asking. And a measurable number of the listed maintainer
addresses just bounce.

> Another reason is that you do much more housekeeping in areas that are not
> actively maintained. But wouldnt it be better if there were active
> maintainers in those areas as well so you could spend more time on eg.
> doing the kernel-stack coloring changes?

There never will be maintainers proper for large amounts of stuff, and the
longer Linus deletes and ignores everything from someone new the less people
will bother sending to him. Just look at the size of the diff set between all
the vendor kernels and Linus 2.4.x trees before the giant -ac merge.

Think gcc, think egcs. History is merely beginning to repeat itself

Alan

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 01:40:50PM +0000, Alan Cox wrote:
 > 
 > No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
 > if you do some asking. And a measurable number of the listed maintainer
 > addresses just bounce.
 
 That's something that should really be fixed.
 I believe a while back someone was going to send a ping to all
 the listed addresses in MAINTAINERS. Doing this again may not
 be a bad idea.

 > There never will be maintainers proper for large amounts of stuff, and the
 > longer Linus deletes and ignores everything from someone new the less people
 > will bother sending to him. Just look at the size of the diff set between all
 > the vendor kernels and Linus 2.4.x trees before the giant -ac merge.

 Now that we have an open development branch again, perhaps its
 time for a lot of the things that have been proven stable in vendor
 kernels for a long time to get a looksee in mainline.
 Some things I feel will likely still be vendor-kernel only for some time.
 And some of them, rightly so.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Henning P. Schmiedehausen]

Dave Jones <davej@suse.de> writes:

> Now that we have an open development branch again, perhaps its
> time for a lot of the things that have been proven stable in vendor
> kernels for a long time to get a looksee in mainline.
> Some things I feel will likely still be vendor-kernel only for some time.
> And some of them, rightly so.

Bah. RedHat puts what? 120+? Patches into 2.2 (!) to ship their vendor
kernel. And it is still much more stable than any 2.4 I've encountered
till today.

I personally run a heavily patched 2.2 (+aa, +ide +reiser +ext3 +raid
and so on) and still get uptimes on busy and heavily loaded servers
(think newsserver with 50+ MBit/sec continous traffic. Think web
accelerator for one of the busiest web sites in Germany. Think mail
system for 1M users) far beyond 200 days uptime.

Will these patches ever be integrated? No. And same will go to some
sore spots of 2.4. Think RAID for 2.2. Think NFS for 2.2 where we
almost had to strangle Alan just to put in the most obvious bug fixes
from Trond :-) . In what? 2.2.16?

Do we have NFSv3 over TCP (which is in the real world available for
how many years?)? A really reliable IDE driver (Hi Andre :-) )? A raid
code that won't stuble over recoverable SCSI errors? That does not
interact badly with some FS types (and of course with those where RAID
would be really interesting)?

We have a kernel based Webserver. But not reliable media detection for
run-of-the-mill network cards. Something which "that other OS" has
since 1995.

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> [...] And a measurable number of the listed maintainer addresses just
> bounce.

out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
bounced physically. (whether they bounce logically is another question.)

	Ingo

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Tue, Jan 29, 2002 at 05:15:15PM +0100, Ingo Molnar wrote:
 > > [...] And a measurable number of the listed maintainer addresses just
 > > bounce.
 > 
 > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
 > bounced physically. (whether they bounce logically is another question.)

 Care to remove the bogus ones from MAINTAINERS and send it to Linus/Me?
 (Keep the entries, but remove the address. (Or better perhaps to mark
 it 'out of order' in case of mailserver failure on $maintainers side.)

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Tue, Jan 29, 2002 at 03:27:32PM +0100, Dave Jones wrote:
> On Tue, Jan 29, 2002 at 05:15:15PM +0100, Ingo Molnar wrote:
>  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
>  > bounced physically. (whether they bounce logically is another question.)
> 
>  Care to remove the bogus ones from MAINTAINERS and send it to Linus/Me?
>  (Keep the entries, but remove the address. (Or better perhaps to mark
>  it 'out of order' in case of mailserver failure on $maintainers side.)

If we're going to be doing this periodically, it might be an idea to
put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
at the top of the file.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Russell King <rmk@arm.linux.org.uk> said:

[...]

> If we're going to be doing this periodically, it might be an idea to
> put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
> at the top of the file.

Perhaps add a "Last checked: field to each (too)?

But then again, patches to MAINTAINERS are silently dropped. Perhaps this
should be posted on kernel.org?
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Wed, Jan 30, 2002 at 10:44:24AM +0100, Horst von Brand wrote:
> Russell King <rmk@arm.linux.org.uk> said:
> > If we're going to be doing this periodically, it might be an idea to
> > put "out of order since dd mmm yyyy" and a "last checked dd mmm yyyy"
> > at the top of the file.
> 
> Perhaps add a "Last checked: field to each (too)?

Old ground - see Message-ID: <20020129144307.B6542@flint.arm.linux.org.uk>

> But then again, patches to MAINTAINERS are silently dropped. Perhaps this
> should be posted on kernel.org?

I suspect that's because Linus gets patches to it from people he's never
heard of before (ie, the new people taking over), but without seeing
anything that says that the old maintainer has gone.

I think this is something that someone could pick up this, and be the
contact point for both maintainers leaving, and new maintainers
taking over - if you like the MAINTAINERS maintainer.  Obviously this
should be someone Linus trusts.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Dave Jones wrote:

>  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
>  > bounced physically. (whether they bounce logically is another question.)
>
>  Care to remove the bogus ones from MAINTAINERS

yeah, was in the process of doing that. Patch against 2.5.3-pre6 attached.
Altogether 13 addresses are affected. I have only removed the
hard-bouncing email addresses, names and list names remain (of course).

	Ingo

--- linux/MAINTAINERS.orig	Tue Jan 29 15:11:04 2002
+++ linux/MAINTAINERS	Tue Jan 29 15:16:31 2002
@@ -154,8 +154,6 @@

 AD1816 SOUND DRIVER
 P:	Thorsten Knabe
-M:	Thorsten Knabe <tek@rbg.informatik.tu-darmstadt.de>
-M:	Thorsten Knabe <tek01@hrzpub.tu-darmstadt.de>
 W:	http://www.student.informatik.tu-darmstadt.de/~tek/projects/linux.html
 W:	http://www.tu-darmstadt.de/~tek01/projects/linux.html
 S:	Maintained
@@ -216,7 +214,6 @@

 ARPD SUPPORT
 P:	Jonathan Layes
-M:	layes@loran.com
 L:	linux-net@vger.kernel.org
 S:	Maintained

@@ -235,7 +232,6 @@

 BERKSHIRE PRODUCTS PC WATCHDOG DRIVER
 P:	Kenji Hollis
-M:	kenji@bitgate.com
 W:	http://ftp.bitgate.com/pcwd/
 S:	Maintained

@@ -433,13 +429,11 @@
 DIGI INTL. EPCA DRIVER
 P:	Chad Schwartz
 M:      support@dgii.com
-M:      chads@dgii.com
 L:      digilnux@dgii.com
 S:      Maintained

 DIGI RIGHTSWITCH NETWORK DRIVER
 P:	Rick Richardson
-M:	rick@remotepoint.com
 L:	linux-net@vger.kernel.org
 W:	http://www.dgii.com/linux/
 S:	Maintained
@@ -485,7 +479,6 @@

 DRM DRIVERS
 P:	Rik Faith
-M:	faith@valinux.com
 L:	dri-devel@lists.sourceforge.net
 S:	Supported

@@ -497,7 +490,6 @@

 EATA-DMA SCSI DRIVER
 P:	Michael Neuffer
-M:	mike@i-Connect.Net
 L:	linux-eata@i-connect.net, linux-scsi@vger.kernel.org
 S:	Maintained

@@ -927,7 +919,6 @@

 LOGICAL VOLUME MANAGER
 P:     Heinz Mauelshagen
-M:     mge@sistina.de
 L:     linux-LVM@sistina.com
 W:     http://www.sistina.com/lvm
 S:     Maintained
@@ -1134,7 +1125,6 @@

 OLYMPIC NETWORK DRIVER
 P:	Peter De Shrijver
-M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
 P:	Mike Phillips
 M:	mikep@linuxtr.net
 L:	linux-net@vger.kernel.org
@@ -1293,7 +1283,6 @@

 RISCOM8 DRIVER
 P:	Dmitry Gorodchanin
-M:	pgmdsg@ibi.com
 L:	linux-kernel@vger.kernel.org
 S:	Maintained

@@ -1660,13 +1649,11 @@
 USB SERIAL BELKIN F5U103 DRIVER
 P:	William Greathouse
 M:	wgreathouse@smva.com
-M:	wgreathouse@myfavoritei.com
 L:	linux-usb-users@lists.sourceforge.net
 L:	linux-usb-devel@lists.sourceforge.net
 S:	Maintained

 USB SERIAL CYBERJACK PINPAD/E-COM DRIVER
-M:	linux-usb@sii.li
 L:	linux-usb-users@lists.sourceforge.net
 L:	linux-usb-devel@lists.sourceforge.net
 S:	Supported
@@ -1792,7 +1779,6 @@

 ZF MACHZ WATCHDOG
 P:	Fernando Fuganti
-M:	fuganti@conectiva.com.br
 M:	fuganti@netbank.com.br
 W:	http://cvs.conectiva.com.br/drivers/ZFL-watchdog/
 S:	Maintained

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

>  ARPD SUPPORT
>  P:	Jonathan Layes
> -M:	layes@loran.com
>  L:	linux-net@vger.kernel.org
>  S:	Maintained

A correct patch for this one giving a new maintainer was posted to Linux
kernel already

>  DRM DRIVERS
>  P:	Rik Faith
> -M:	faith@valinux.com
>  L:	dri-devel@lists.sourceforge.net
>  S:	Supported

Rik moved from VA so I suspect you want to look in the RH internal directory
for the correct one there 8)

Alan

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

On Tue, 29 Jan 2002, Alan Cox wrote:

> A correct patch for this one giving a new maintainer was posted to
> Linux kernel already

ok.

> >  DRM DRIVERS
> >  P:	Rik Faith
> > -M:	faith@valinux.com
> >  L:	dri-devel@lists.sourceforge.net
> >  S:	Supported
>
> Rik moved from VA so I suspect you want to look in the RH internal directory
> for the correct one there 8)

*blush* :-) The correct patch is:

-M:	faith@valinux.com
+M:	faith@redhat.com

	Ingo

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Ingo Molnar wrote:
> 
> On Tue, 29 Jan 2002, Dave Jones wrote:
> 
> >  > out of the 300+ email addresses in the MAINTAINERS file, 15 addresses
> >  > bounced physically. (whether they bounce logically is another question.)
> >
> >  Care to remove the bogus ones from MAINTAINERS
> 
> yeah, was in the process of doing that. Patch against 2.5.3-pre6 attached.
> Altogether 13 addresses are affected. I have only removed the
> hard-bouncing email addresses, names and list names remain (of course).
> 
>         Ingo

Humble suggestion:  Add a date field for "took over maintainence
on/before: yyyy-mm-dd" and a field for "last verified: yyyy-mm-dd" so we
know when we last checked on the existance/etc. of a maintainer.  (And
maybe an "AWAL on/before: yyyy-mm-dd" for those without known working
addresses.)

Thoughts?

Ah, I see Russell King made a similar suggestion...

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Ingo Molnar]

Patrick Mauritz noticed that this one is actually a typo in the
MAINTAINERS file:

> -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be

so the correct (and existing) email address is:

-M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
+M:	p2@ace.ulyssis.student.ac.be

	Ingo

Re: A modest proposal -- We need a patch penguin

[Patrick Mauritz]

On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
> +M:	p2@ace.ulyssis.student.ac.be
fixing the fix:
+M:	p2@ace.ulyssis.student.kuleuven.ac.be

sorry,
patrick mauritz
-- 
,------------------------------------------------------------------------.
>   In the Beginning there was nothing, which exploded - Yeah right...   <
|------------------------------------------------------------------------|
>           Debian GNU/Linux, apt-get into it | www.debian.org           <
`------------------------------------------------------------------------'
 If you gave an infinite number of monkeys an infinite number of typewr..
                         Hey! that's the internet!

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in <20020129145344.GC2611@hydra>:

> On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
> > +M:	p2@ace.ulyssis.student.ac.be
> fixing the fix:
> +M:	p2@ace.ulyssis.student.kuleuven.ac.be

I thought that was the bouncing address?!

MfG Kai

Re: A modest proposal -- We need a patch penguin

[Arnaldo Carvalho de Melo]

Em Tue, Jan 29, 2002 at 10:03:00PM +0200, Kai Henningsen escreveu:
> oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in <20020129145344.GC2611@hydra>:
> 
> > On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
                                ^^
> > > +M:	p2@ace.ulyssis.student.ac.be
> > fixing the fix:
> > +M:	p2@ace.ulyssis.student.kuleuven.ac.be
                        ^^
> I thought that was the bouncing address?!

got it? :)

- Arnaldo

Re: A modest proposal -- We need a patch penguin

[Kai Henningsen]

acme@conectiva.com.br (Arnaldo Carvalho de Melo)  wrote on 30.01.02 in <20020130031528.GF973@conectiva.com.br>:

> Em Tue, Jan 29, 2002 at 10:03:00PM +0200, Kai Henningsen escreveu:
> > oxygene@studentenbude.ath.cx (Patrick Mauritz)  wrote on 29.01.02 in
> > <20020129145344.GC2611@hydra>:
> >
> > > On Tue, Jan 29, 2002 at 05:47:27PM +0100, Ingo Molnar wrote:
> > > > -M:	p2@ace.ulyssis.sutdent.kuleuven.ac.be
>                                 ^^
> > > > +M:	p2@ace.ulyssis.student.ac.be
> > > fixing the fix:
> > > +M:	p2@ace.ulyssis.student.kuleuven.ac.be
>                         ^^
> > I thought that was the bouncing address?!
>
> got it? :)

Now. I certainly looked at those lines for several minutes without ...

Sometimes a wdiff would be better, it seems :-)


MfG Kai

Re: update to MAINTAINERS list

[Andreas Dilger]

On Jan 29, 2002  17:36 +0100, Ingo Molnar wrote:
> @@ -927,7 +919,6 @@
> 
>  LOGICAL VOLUME MANAGER
>  P:     Heinz Mauelshagen
> -M:     mge@sistina.de
>  L:     linux-LVM@sistina.com
>  W:     http://www.sistina.com/lvm
>  S:     Maintained

mge@sistina.com or mauelshagen@sistina.com

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: A modest proposal -- We need a patch penguin

[toon]

On Tue, Jan 29, 2002 at 05:36:07PM +0100, Ingo Molnar wrote:
> 
> 
> @@ -927,7 +919,6 @@
> 
>  LOGICAL VOLUME MANAGER
>  P:     Heinz Mauelshagen
> -M:     mge@sistina.de
>  L:     linux-LVM@sistina.com
>  W:     http://www.sistina.com/lvm
>  S:     Maintained

I just checked the linux-lvm mailing list.
Heinz seems to be mailing from: mauelshagen@sistina.com

Regards,
Toon.
-- 
 /"\                             |   Windows XP:
 \ /     ASCII RIBBON CAMPAIGN   |        "Sorry Dave...
  X        AGAINST HTML MAIL     |         I'm afraid I can't do that."
 / \

Re: A modest proposal -- We need a patch penguin

[Horst von Brand]

Ingo Molnar <mingo@elte.hu> said:

[...]

>  ARPD SUPPORT
>  P:	Jonathan Layes
> -M:	layes@loran.com
>  L:	linux-net@vger.kernel.org
>  S:	Maintained

I seem to remember a message on lkml from a gal claiming (a) that Loran had
been bought by somebody else, and (b) that she had taken over
maintainership of this, after somewhat of elbowing around with the new
ownership. She asked the entry to be corrected...
-- 
Horst von Brand			     http://counter.li.org # 22616

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:40 am, Alan Cox wrote:
> > for code areas where there is not active maintainer or the maintainer has
> > ignored patches? Eg. the majority of the kdev transition patches went in
> > smoothly.
>
> No you merely aren't watching. Most of the maintainers btw are ignoring 2.5
> if you do some asking. And a measurable number of the listed maintainer
> addresses just bounce.

I'm under the impression Michael Elizabeth Chastain is one such burned out 
maintainer, but hasn't been able to hand over maintainership because Linus 
keeps dropping his patch to change the maintainers file to say "Peter 
Samuelson", and he eventually just gave up trying.

I could be wrong about this.  Ask him.  Or maybe his maintainer hand-over 
patch needs more code review?

> > Another reason is that you do much more housekeeping in areas that are
> > not actively maintained. But wouldnt it be better if there were active
> > maintainers in those areas as well so you could spend more time on eg.
> > doing the kernel-stack coloring changes?
>
> There never will be maintainers proper for large amounts of stuff, and the
> longer Linus deletes and ignores everything from someone new the less
> people will bother sending to him.

Case in point:

--- linux/arch/i386/boot/bootsect.S.old Tue Jan  1 19:41:22 2002
+++ linux/arch/i386/boot/bootsect.S     Tue Jan  1 19:44:02 2002
@@ -158,9 +158,7 @@
        movw    $sread, %si             # the boot sector has already been 
read
        movw    %ax, (%si)

-       xorw    %ax, %ax                # reset FDC
-       xorb    %dl, %dl
-       int     $0x13
+       call    kill_motor              # reset FDC
        movw    $0x0200, %bx            # address = 512, in INITSEG
 next_step:
        movb    setup_sects, %al

Dumb little nit I noticed a few weeks ago, but never bothered to follow up 
on, because it's just not worth it.  Not that potentially saving 3 bytes out 
of the boot sector is a BAD thing, but it's not good enough to be worth the 
effort anymore.  Warning fixing patches are largely the same way: easy to do, 
but why?

This didn't strike me as a healthy development, really...

> Just look at the size of the diff set
> between all the vendor kernels and Linus 2.4.x trees before the giant -ac
> merge.
>
> Think gcc, think egcs. History is merely beginning to repeat itself

I was actually hoping to AVOID that.  (There IS still time.  We're not that 
badly off.  Yet.  I'm just a bit nervous about direction.  The kind of 
stresses I've seen seem (to me) unlikely to improve with time...)

And we ARE using a patch penguin.  You were around, and Dave is around.  I'm 
kind of confused at the level of resistence to formally recognizing what 
basically IS current practice, and has been for YEARS.  (The reason for 
naming the position is we can't just say "alan's tree" anymore.  The position 
went from one person to another person, and as such the position seemed to 
need to be recognized as being separate from the individual.  I didn't expect 
to hit a brick wall on that.  It  didn't seem like a revolutionary idea to 
me...)

> Alan

Re: A modest proposal -- We need a patch penguin

[Eric S. Raymond]

Rob Landley <landley@trommello.org>:
> And we ARE using a patch penguin.  You were around, and Dave is
> around.  I'm kind of confused at the level of resistence to formally
> recognizing what basically IS current practice, and has been for
> YEARS.  (The reason for naming the position is we can't just say
> "alan's tree" anymore.  The position went from one person to another
> person, and as such the position seemed to need to be recognized as
> being separate from the individual.  I didn't expect to hit a brick
> wall on that.  It didn't seem like a revolutionary idea to me...)

Alas.  That's because, like most Americans these days, you're
historically illiterate.  What we are facing here is a *very* familiar
problem to social and institutional historians. 

All movements founded by charismatic leaders like Linus eventually hit
this same wall -- the point at which the charisma of the founder and
the individual ability of the disciples he personally attracts are no
longer adequate to meet the challenges of success, and some way to
institutionalize and distribute the leader's role has to be found.
Movements that fail to make this transition die, generally by
implosion or fragmenting into feuding sub-sects.

If you were familiar with the historical precedents, Rob, you would 
understand that your modest proposal re-enacts a common pattern.
A relatively junior member of the movement, one with few political
ties, sees the developing stress fractures in the organization of
the movement and proposes a modest, incremental change to relieve
some of them.  Conservatives interpret the attempt to separate 
and institutionalize part of the founder's role as an attack on
the authority of the founder.  Huge flamewars ensue, with the
original pragmatic sense of the proposal often being lost as it
becomes a political football in the movement's internal status games.

Sometimes the first such attempt at institutionization succeeds.  More
often, the movement has to go through a series of escalating crises
(burning up would-be reformers each time) before anyone finally
succeeds in changing the movement's internal culture.

Religions go through this.  Secular social movements go through this.
Companies founded by brilliant entrepreneurs go through this (the
B-schools have a whole literature on "entrepreneurial overcontrol" and
its consequences).  It's one of the dramas that gets perpetually
re-enacted; it's built in to our wiring.  The unhappy truth is that
even *successful* transitions of this kind are invariably painful, and
often leave deep scars on the survivors and on the institution that
arises from the transition.

*Never* expect this sort of transition to be easy, especially when the
positions people are taking are as much about personal identity and
values as they are about "success" in whatever terms the movement
defines it.
-- 
		<a href="http://www.tuxedo.org/~esr/">Eric S. Raymond</a>

Re: A modest proposal -- We need a patch penguin

[Mark Hahn]

On Tue, 29 Jan 2002, Eric S. Raymond wrote:
...
> Alas.  That's because, like most Americans these days, you're
> historically illiterate.  What we are facing here is a *very* familiar

can majordomo be configured to rewrite Erik's name as "Pappy Raymond"?

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Alan Cox wrote:

> Ingo, you should have a look at my mailbox and the people sick of trying to
> get Linus to take 3 liners to fix NODEV type stuff and being ignored so that
> 2.5.* still doesn't even compile or boot for many people.
> 
> Dave in doing the patch hoovering at least ensures these are picked up. You
> think if this carries on anyone will be running Linus tree in 9 months ?

Linus, I think I hear people you said you trust telling you this... take
the little bits out of band and TRUST people to give you patches which go
in 2.5.x now, not when or if you get to it. Having you approve trivial
stuff is a waste of what you do best, and I think all the versions show
you're not even being effective at that. Don't you HATE looking at
spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
it out and let other people do it, and work on the fun stuff.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Craig Christophel]

> Linus, I think I hear people you said you trust telling you this... take
> the little bits out of band and TRUST people to give you patches which go
> in 2.5.x now, not when or if you get to it. Having you approve trivial
> stuff is a waste of what you do best, and I think all the versions show
> you're not even being effective at that. Don't you HATE looking at
> spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
> it out and let other people do it, and work on the fun stuff.

	aasn, (as a side note)  It's really hard to allow people to just change the 
little peices.  The little peices soon become larger and more complex.  This 
is a really difficult transition in moving from (my personal perspective) an 
exciting young code base to a maturing and well functioning/planned setup.  
The only thing I have to say is that Linus had better pick his talent and 
friends well, because even if the codebase does not split today, what is to 
keep it from doing so in the future when even more complexities arise.  

	Linus,
		It doesn't have to be difficult, just as you have maintainers for the 
"stable" series yet still have a say on what happens, define the level of 
modularity you would like to happen for different maintainers to be able to 
take care of their little peices and it will happen.  Please do not allow the 
BSD split to happen here.  Be proactive and take a hard look at what you 
really want.  

	Please,

Craig.

Re: A modest proposal -- We need a patch penguin

[Shawn]

On 01/29, Craig Christophel said something like:
> > Linus, I think I hear people you said you trust telling you this... take
> > the little bits out of band and TRUST people to give you patches which go
> > in 2.5.x now, not when or if you get to it. Having you approve trivial
> > stuff is a waste of what you do best, and I think all the versions show
> > you're not even being effective at that. Don't you HATE looking at
> > spelling errors, off-by-one logic, corner cases, and stuff like that? Farm
> > it out and let other people do it, and work on the fun stuff.
> 
> 	aasn, (as a side note)  It's really hard to allow people to just change the 
> little peices.  The little peices soon become larger and more complex.  This 

I believe he was talking about some of the example one-liners cited
earlier. Not a significant chance of those getting "more complex" such
that they become cumbersome...

> is a really difficult transition in moving from (my personal perspective) an 
> exciting young code base to a maturing and well functioning/planned setup.  
> The only thing I have to say is that Linus had better pick his talent and 
> friends well, because even if the codebase does not split today, what is to 
> keep it from doing so in the future when even more complexities arise.  

The codebase has been, is now, and forever will be split, due to
differing goals, and the fact that one size does not fit all... Not to
mention personal taste. The splits will only be as large a practicality
allows, given the number of trees trying to sync from, or with -linus.
The -ac tree got pretty splorked off for a while, though.

By the way, Linus, Alan, Al Viro, Marcello, Ingo, Stephen, Rik, Dave,
Dave, any other Daves, Geert, Jens, Andre, Richard, hpa, Hans, and
everyone I can't think of, THANKS for making my machine run fast, die
hard, and just for putting out better code than a huge corporation can!

--
Shawn Leas
core@enodev.com

For my birthday I got a humidifier and a de-humidifier... I put them
in the same room and let them fight it out...
						-- Stephen Wright

Re: A modest proposal -- We need a patch penguin

[Skip Ford]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
[snip]
> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and

Then why not give the subsystem maintainers patch permissions on your tree.
Sort of like committers.  The problem people have is that you're dropping
patches from those ten-twenty people you trust.

Each subsystem maintainer should handle patches to that subsystem, and
you should remove your own patch permissions for only those subsystems.
You could get involved with only changes in direction that affect more
than one subsystem.

> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me. 
> 
> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

- -- 
Skip  ID: 0x7EDDDB0A
-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAjxWsfkACgkQBMKxVH7d2wpAfQCfRMoMirEH2/KRYMowNkZyMCBi
CEYAoM4akKt8Ifl20cvkA5UG8Kb4p9Tb
=q5bM
-----END PGP SIGNATURE-----

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Skip Ford wrote:
>
> Linus Torvalds wrote:
> [snip]
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
>
> Then why not give the subsystem maintainers patch permissions on your tree.
> Sort of like committers.  The problem people have is that you're dropping
> patches from those ten-twenty people you trust.

No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
_not_ dropping their patches (although I'm sometimes critical of them,
and will tell them that they do not get applied).

Sure, it happens occasionally that they really do get dropped, just
because I get too much email, but these people know how to re-send every
once in a while, and keep their patches separate.

I think there is some confusion about who I trust. Being listed as
MAINTAINER doesn't mean you are automatically trusted. The MAINTAINERS
list is not meant for me _at_all_ in fact, it's meant more as one of the
places for _others_ to search for a contact with.

Examples of people who I trust: Ingo Molnar, Jeff Garzik, Alan Cox, Al
Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
"good taste" for a long time. But it's not always a long process - some
of you may remember Bill Hawes, for example, who came out of nowhere
rather quickly.

There are other categories: Andrea, for example, is in a category all of
his own under "absolutely stunning, but sometimes somewhat erratic", which
just means that I have to think a lot more about his patches. I love his
experimentation, especially now that he maintains separate patches (and
I'd also love for him to be more active in pushing the non-experimental
parts towards me, hint hint, Andrea)

And there are categories of people who just own a big enough chunk that is
separate enough that I trust them for that area: architecture maintainers
etc tend to be here, as long as they only affect their own architecture.

But you have to realize that there are a _lot_ of people on the
maintainers list that I don't implicitly trust. And being loud and
wellknown on the mailing lists or IRC channels doesn't make them any more
trusted.

			Linus

Re: A modest proposal -- We need a patch penguin

[Michael Sterrett -Mr. Bones.-]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> "But it's not always a long process - some of you may remember Bill
> Hawes, for example, who came out of nowhere rather quickly.

I remember being very impressed by Bill.  What ever happened to him?

Michael Sterrett
  -Mr. Bones.-
michael.sterrett@coat.com

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 12:36 pm, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Skip Ford wrote:
> > Linus Torvalds wrote:
> > [snip]
> >
> > > A word of warning: good maintainers are hard to find.  Getting more of
> > > them helps, but at some point it can actually be more useful to help
> > > the _existing_ ones.  I've got about ten-twenty people I really trust,
> > > and
> >
> > Then why not give the subsystem maintainers patch permissions on your
> > tree. Sort of like committers.  The problem people have is that you're
> > dropping patches from those ten-twenty people you trust.
>
> No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> _not_ dropping their patches (although I'm sometimes critical of them,
> and will tell them that they do not get applied).

Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel 
Boldt...

> I think there is some confusion about who I trust. Being listed as
> MAINTAINER doesn't mean you are automatically trusted. The MAINTAINERS
> list is not meant for me _at_all_ in fact, it's meant more as one of the
> places for _others_ to search for a contact with.

Ah.  So being listed in the maintainers list doesn't mean someone is actually 
a maintainer it makes sense to forward patches to?

Are you backing away from the maintainer system, or were the rest of us 
misinterpreting what it meant all along?  Does being a maintainer mean you 
feel you have delegated any actual authority to them at all, or is it merely 
a third party tech support contact point?

You seem to be saying "send patches to maintainers, support the maintainers 
better, but don't expect me to necessarily take patches from them".  Who 
should patches be sent TO?

> Examples of people who I trust: Ingo Molnar, Jeff Garzik, Alan Cox, Al
> Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> "good taste" for a long time. But it's not always a long process - some
> of you may remember Bill Hawes, for example, who came out of nowhere
> rather quickly.

So listed "maintainers" may need to forward patches to these people, and get 
them to sign off on them, in order to get their patches at least reviewed for 
inclusion into your tree?

If that's the process, fine.  I'm just trying to clarify what the process IS, 
other than spamming your mailbox with a cron job (as has been suggested and 
actually taken seriously as long as re-testing is also automated)...

> And there are categories of people who just own a big enough chunk that is
> separate enough that I trust them for that area: architecture maintainers
> etc tend to be here, as long as they only affect their own architecture.
>
> But you have to realize that there are a _lot_ of people on the
> maintainers list that I don't implicitly trust. And being loud and
> wellknown on the mailing lists or IRC channels doesn't make them any more
> trusted.

I've noticed that rather a lot of development seems to be moving to IRC.  How 
is this NOT to be interpreted as a lack of endorsement of the functionality 
of the other channels?  Is IRC "just nice", or does IRC address a problem not 
otherwise being addressed?

> 			Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Rob Landley wrote:
> > >
> > > Then why not give the subsystem maintainers patch permissions on your
> > > tree. Sort of like committers.  The problem people have is that you're
> > > dropping patches from those ten-twenty people you trust.
> >
> > No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> > _not_ dropping their patches (although I'm sometimes critical of them,
> > and will tell them that they do not get applied).
>
> Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
> Boldt...

NONE of those are in the ten-twenty people group.

How many people do you think fits in a small group? Hint. It sure isn't
all 300 on the maintainers list.

> Ah.  So being listed in the maintainers list doesn't mean someone is actually
> a maintainer it makes sense to forward patches to?

Sure it does.

It just doesn't mean that they should send stuff to _me_.

Did you not understand my point about scalability?  I can work with a
limited number of people, and those people can work with _their_ limited
number of people etc etc.

The MAINTAINERS file is _not_ a list of people I work with on a daily
basis. In fact, I don't necessarily even recognize the names of all those
people.

Let's take an example. Let's say that you had a patch for ppp. You'd send
the patch to Paul Mackerras. He, in turn, would send his patches to David
Miller (who knows a hell of a lot better what it's all about than I do).
And he in turn sends them to me.

They are both maintainers. That doesn't mean that I necessarily work with
every maintainer directly.

Or look at USB: I get the USB patches from Greg, and he gets them from
various different people. Johannes Erdfelt is the maintainer for uhci.c,
and he sends them to Greg, not to me.

Why? Because having hundreds of people emailing me _obviously_ doesn't
scale. Never has, never will. It may work over short timeperiods wih lots
of energy, but it obviously isn't a stable setup.

			Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> > Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth
> > Chastain, Axel Boldt...
>
> NONE of those are in the ten-twenty people group.
>
> How many people do you think fits in a small group? Hint. It sure
> isn't all 300 on the maintainers list.

That's fine with me, but _who_ do I send VM patches to if
I can't send them to you ?

There is no maintainer for mm/* or kernel/*, it's just you.

I agree it would be nice to have somebody from within the
ten-twenty people group take care of that, but we'll need
to have _somebody_ ...

regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Rik van Riel wrote:
>
> That's fine with me, but _who_ do I send VM patches to if
> I can't send them to you ?

The VM stuff right now seems to be Andrea, Dave or you yourself (right now
I just wish you would split up your patches like Andrea does, that way I
can cherry-pick).

> There is no maintainer for mm/* or kernel/*, it's just you.

As to kernel/ there are actually maintainers for some sub-areas, the most
noticeable being Ingo on the scheduler. The rest of kernel/ hasn't ever
been much of a problem, really.

The VM is a big issue, of course. And that one isn't likely to go away
anytime soon as a point of contention. And it's not easy to modularize,
apart from the obvious pieces (ie "filemap.c" vs the rest).

You may not believe me when I say so, but I personally _really_ hope your
rmap patches will work out. I may not have believed in your patches in a
2.4.x kind of timeframe, but for 2.6.x I'm more optimistic. As to how to
actually modularize it better to make points of contention smaller, I
don't know how.

At the same time, while I can understand your personal pain, I don't think
most of the problems have been with the VM (maintenance-wise, that is.
Most of the _technical_ problems really have been with the VM, it's just
the most complex piece).

		Linus

Re: A modest proposal -- We need a patch penguin

[Rik van Riel]

On Tue, 29 Jan 2002, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Rik van Riel wrote:
> >
> > That's fine with me, but _who_ do I send VM patches to if
> > I can't send them to you ?
>
> The VM stuff right now seems to be Andrea, Dave or you yourself (right
> now I just wish you would split up your patches like Andrea does, that
> way I can cherry-pick).

I will.  It's not split up at the moment because I'd like to
work a bit more on -rmap before submitting it for inclusion
and bitkeeper is a really nice tool to help me carry the patch
from version to version.

If -rmap makes it into the kernel I'll work with small patches
in the same style as Andrea, that's just the easiest way to
work.

I'll also take some of rusty's scripts to automatically check
if a patch (1) has been applied to the latest kernel or
(2) if it still applies cleanly.

Basically I'm looking for a way to minimise the work of
carrying the -rmap VM across kernel versions, so I can spend
my time doing development and cleaning up the code further.

> The VM is a big issue, of course. And that one isn't likely to go away
> anytime soon as a point of contention. And it's not easy to modularize,
> apart from the obvious pieces (ie "filemap.c" vs the rest).

Actually some stuff can be modularised somewhat. Christoph
Hellwig for example has a nice patch which replaces all
knowledge of page->wait with wake_up_page().

This makes the fact of whether we're using per-page waitqueues
or hashed waitqueues completely invisible to the rest of the
kernel.

Similar things are possible for other areas of the code.

> You may not believe me when I say so, but I personally _really_ hope your
> rmap patches will work out. I may not have believed in your patches in a
> 2.4.x kind of timeframe, but for 2.6.x I'm more optimistic. As to how to
> actually modularize it better to make points of contention smaller, I
> don't know how.

One thing William Irwin (and others, myself too) have been
looking at is making the pagemap_lru_lock per-zone.

This would allow us to "split up" memory in zones and have
each CPU start the allocation chain at its own zone.

This works out in practice because the reverse mapping code
allows us to scan and free memory by physical address,
meaning that reclaim_page() becomes a per-CPU local thing
under light memory loads.

Of course the current problem with that code is truncate
and the lock ordering between the pagemap_lru_lock and the
page_cache_lock ... something to look at later.

kind regards,

Rik
-- 
"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Re: A modest proposal -- We need a patch penguin

[Daniel Jacobowitz]

On Tue, Jan 29, 2002 at 03:50:43PM -0800, Linus Torvalds wrote:
> > Ah.  So being listed in the maintainers list doesn't mean someone is actually
> > a maintainer it makes sense to forward patches to?
> 
> Sure it does.
> 
> It just doesn't mean that they should send stuff to _me_.
> 
> Did you not understand my point about scalability?  I can work with a
> limited number of people, and those people can work with _their_ limited
> number of people etc etc.
> 
> The MAINTAINERS file is _not_ a list of people I work with on a daily
> basis. In fact, I don't necessarily even recognize the names of all those
> people.

I understand that the sort of careful hierarchy that drives this
process is, by nature, an informal thing.  But it would still be nice
if _suggestions_ on how it worked were written down somewhere.  When
you've got patches that don't have a clear relevant maintainer, it
would be nice to have something more specific than "post to
linux-kernel and pray someone picks it up" to run with!

-- 
Daniel Jacobowitz                           Carnegie Mellon University
MontaVista Software                         Debian GNU/Linux Developer

Re: A modest proposal -- We need a patch penguin

[Chris Ricker]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> They are both maintainers. That doesn't mean that I necessarily work with
> every maintainer directly.
> 
> Or look at USB: I get the USB patches from Greg, and he gets them from
> various different people. Johannes Erdfelt is the maintainer for uhci.c,
> and he sends them to Greg, not to me.
> 
> Why? Because having hundreds of people emailing me _obviously_ doesn't
> scale. Never has, never will. It may work over short timeperiods wih lots
> of energy, but it obviously isn't a stable setup.

Linus,

That's fine, but there's a major problem with your scheme.  What happens
with all the stuff for which no one is listed in MAINTAINERS?  For example,
no one owns linux/Documentation.  As the person nominally in charge of
linux/Documentation/Changes, there's no one between me and you, period, let
alone anyone between me and you that you trust....  And I realize that you
don't consider documentation very important, but there are other segments of
the Linux source tree for which this breakdown in hierarchy is also true....

later,
chris

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Chris Ricker wrote:
>
> That's fine, but there's a major problem with your scheme.  What happens
> with all the stuff for which no one is listed in MAINTAINERS?

I have to admit that personally I've always found the MAINTAINERS file
more of an irritation than anything else. The first place _I_ tend to look
personally is actually in the source files themselves (although that may
be a false statistic - the kind of people I tend to have to look up aren't
the main maintainers at all, but more single driver people etc).

It might not be a bad idea to just make that "mention maintainer at the
top of the file" the common case.

		Linus

Re: A modest proposal -- We need a patch penguin

[Miles Lane]

On Tue, 2002-01-29 at 16:44, Linus Torvalds wrote:
> 
> On Tue, 29 Jan 2002, Chris Ricker wrote:
> >
> > That's fine, but there's a major problem with your scheme.  What happens
> > with all the stuff for which no one is listed in MAINTAINERS?
> 
> I have to admit that personally I've always found the MAINTAINERS file
> more of an irritation than anything else. The first place _I_ tend to look
> personally is actually in the source files themselves (although that may
> be a false statistic - the kind of people I tend to have to look up aren't
> the main maintainers at all, but more single driver people etc).
> 
> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

I do similarly when I am testing Gnome software, but there
I have the CVS sources to look at, including carefully updated
ChangeLog files.  I find the ChangeLogs and the output of
"cvs log ChangeLog" to be highly informative and helpful when
attempting to track down the appropriate person to contact.
Is it feasible to set up a read-only anonymous cvs server for
the kernel tree?  It seems to me that it would be nice to 
good to have ChangeLogs for the kernel directories as well.

	Miles

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 08:38 pm, Miles Lane wrote:
> On Tue, 2002-01-29 at 16:44, Linus Torvalds wrote:
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
>
> I do similarly when I am testing Gnome software, but there
> I have the CVS sources to look at, including carefully updated
> ChangeLog files.  I find the ChangeLogs and the output of
> "cvs log ChangeLog" to be highly informative and helpful when
> attempting to track down the appropriate person to contact.
> Is it feasible to set up a read-only anonymous cvs server for
> the kernel tree?  It seems to me that it would be nice to
> good to have ChangeLogs for the kernel directories as well.

This isn't necessarily a problem for Linus to handle.

Right now, it's pretty easy to find/generate diffs between each "pre 
release".  Each of those could be incrementally fed into a CVS server, and 
bang, you have a revision history.  The granualrity might not be the 
greatest, but it's a start, and it can be done retroactively.  (I vaguely 
remember hearing some work along these lines...)

Now to get the kind of patch level granuarity that Linus likes to have made 
available to the rest of the world, you need the actual patches, as applied, 
made available.  Getting a patch penguin (I.E. Alan Cox or Dave Jones) to do 
this might not be too hard (as long as it's not too much work), but not 
enough patches go through them at the moment to necessarily make it 
worthwhile.

Long ago I suggested that since the way Linus works is "append various emails 
to a big file, then feed that to patch(1) at the end of a mail run", it 
should be possible to send Linus a perl script that copies the individual 
emails from the big file to a mailing list when he patches his tree.  Not 
just the actual patch, but the whole email with the description of the fix 
and everything.  (Again, no guarantee he wouldn't back them out again, but 
it's something that really requires no extra work on his part, gives 
immediate acknowledgement that he's looked at something, and gives the rest 
of the world access to the level of granularity he expects to receive from 
them.)

Of course until such a script is actually written, with a mailing list set up 
for it to post to (read-only except for Linus), it's just an idle thought I 
haven't had time to pursue.  (The diffs between pre-versions have generally 
been good enough for me personally, so...)

If the "patches-to-linus" list does get implemented, it would probably also 
be fairly easy to automatically match new pre-X->pre-Y diffs against the 
recent patches from the list, and extract most of the information that way.  
(Assuming Linus doesn't modify them too much, or end up taking a lot of 
patches from other sources.  A human would probably still have to do at least 
part of it, but it might be an improvement on just putting the whole big 
version diff in the cvs tree as one lump...)

> 	Miles

Rob

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 03:06:15AM -0500, Rob Landley wrote:
> If the "patches-to-linus" list does get implemented, it would probably also 
> be fairly easy to automatically match new pre-X->pre-Y diffs against the 
> recent patches from the list, and extract most of the information that way.  
> (Assuming Linus doesn't modify them too much, or end up taking a lot of 
> patches from other sources.  A human would probably still have to do at least 
> part of it, but it might be an improvement on just putting the whole big 
> version diff in the cvs tree as one lump...)

Instead of doing this stuff half-assed, just convince Linus to use BK :)

	Jeff

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 03:47:46AM -0500, Jeff Garzik wrote:
> Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> 	Jeff

It might be good if I (and others who are interested) worked on a document
describing the work flow that has been discussed here, in particular
the sort of network of people that Linus mentioned and Dave showed in
his ascii art.  That document would be a useful thing to hand to newbies
regardless of whether BK or any other CVS is in the picture because it
would describe how things work today, more or less.

It is also useful as a stake in the ground which says that any CVS
which wants to be considered in this problem space has to solve that
work flow, or come pretty darn close.  If BK can do it, that would be
good, that's why we built it.  If it can't, you still get a useful doc
describing how things work and we can go back and try and fix BK.

If you think this is a good idea and you are one of the 20-30 maintainers
close to the center of the network, let me know.  If anyone wants to work
on this or just watch the progress, I can have a mailing list set up 
tomorrow.  If there is interest in the doc, I think I could have something
with nice pictures and text in a week.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <20020130034746.K32317@havoc.gtf.org>,
Jeff Garzik  <garzik@havoc.gtf.org> wrote:
>
>Instead of doing this stuff half-assed, just convince Linus to use BK :)

The thing is, I actually _want_ to use BK (as opposed to CVS, which I
really don't think cuts it). 

I still dislike some things (those SHOUTING SCCS files) in bk, and let's
be honest: I've used CVS, but I've never really used BK. Larry has given
me the demos, and I actually decided to re-do the examples, but it takes
time and effort to get used to new tools, and I'm a bit worried that
I'll find other things to hate than just those loud filenames.

This is partly why I asked how many people use it..

		Linus

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:33:19AM +0000, Linus Torvalds wrote:
> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
> be honest: I've used CVS, but I've never really used BK. Larry has given
> me the demos, and I actually decided to re-do the examples, but it takes
> time and effort to get used to new tools, and I'm a bit worried that
> I'll find other things to hate than just those loud filenames.

One issue I'm interested in, and Larry and I have chatted about this a
couple times, is making sure that the "standard" patch flow isn't
affected... and what I mean by that is out-of-order and/or modified
patches.

Say you apply patches A, B, and E from an Al Viro patch series,
reject D, and apply patch C but tweak it yourself [sb->s_id is case
in point IIRC].  Say further that Al sent you a BK patch.  (ha! but
bear with me :))  I want to be confident that BK does not cause
downstream patches to impose constraints on you which prevent or make
difficult weird cases like this, just to ensure that BK's idea of a
global tree remains intact.

Experience and additional BK knowledge on my part will likely clear this
up, but IIRC this was one of the larger issues with not only you but
many others concurrently developing on what I would call the "global
Linux tree."

Obviously this wouldn't apply if you fed BK patches into GNU patch, and
then issued the commit from there...  but that way is a bit lossy, since
you would need to recreate rename information among other things.

In any case, I think BK is pretty nifty so far, but want to practice
by importing all Linux patches into a tree before converting my own
"gkernel" cvs to BK.  (tytso disagrees and thinks that there should be a
separate BK tree for 2.4, 2.5,... IMHO: ug.)

	Jeff,
	who should really get sleep before tomorrow's LW-NY

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Andreas Dilger]

On Wed, Jan 30, 2002 at 09:33:19AM +0000, Linus Torvalds wrote:
> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
> be honest: I've used CVS, but I've never really used BK. Larry has given
> me the demos, and I actually decided to re-do the examples, but it takes
> time and effort to get used to new tools, and I'm a bit worried that
> I'll find other things to hate than just those loud filenames.

Well, the one benefit of using SCCS directories (which are only 1/3
louder than CVS directories) is that tools like patch, make, ctags, emacs
(I believe), etc. already understand what they are and how to extract
the latest version of a file from there.  If these tools were changed
to also recognize .SCCS dirs, then BK could eventually follow suit, but
it would be impractical until they are widely available.*

On Jan 30, 2002  05:07 -0500, Jeff Garzik wrote:
> One issue I'm interested in, and Larry and I have chatted about this a
> couple times, is making sure that the "standard" patch flow isn't
> affected... and what I mean by that is out-of-order and/or modified
> patches.

I would have to agree.  Ted uses BK for e2fsprogs, and there have been
several times when I try to send him a CSET, but he is unable to apply
it because it is missing dependencies, even though I know those prior
CSETs are actually independent changes that just happen to touch the
same files.

It is double-plus bad when those changes are not just ones I've forgotten,
but ones that I know Ted does not want to have in his tree, or are not
in a state where I want to send them to him yet.  This makes me keep
several local repositories - pristine, changes for Ted, changes for me,
etc.  Not fatal, but not as easy as keeping a single tree and pulling
out diffs as needed.

Also, (BK feature request time) there are times when I've done a 'bk citool'
and committed a bunch of changes into a CSET, and later on done some more
testing which revealed a bug or found that I'd forgotten to change the
man page to track the changes I made.  I'd much rather be able to merge
some more changes into the same CSET instead of creating a second CSET and
now have two CSETs to ship around for what is logically a single change.**

I think it would quickly become a nightmare if you had to submit (and
have accepted!) all of your changes to Linus IN ORDER.  As it stands now,
there are lots of discrete changes I have in my ext2 tree, and whenever
I get around to it or when people hit the same bug as me I generate a
patch, edit out the irrelevant parts, and send it out.***

Granted, it is hard to keep distributed BK repositories consistent if you
apply patches out of order, but at the same time, this is how development
works in real life.  Two solutions I can see to this:

1) Allow out-of-order CSET application (maybe with some sort of warning
   that Linus can turn off, because _every_ CSET he would get would be
   missing dependencies from somebody's tree).
2) Allow "proxy" CSETs to be included which say "the changes from adilger
   adilger@lynx.adilger.int|ChangeSet|20011226061040|56205 changed lines
   X-Y, Z of file fs/ext2/super.c" but doesn't actually contain those
   changes, so that the CSET dependency graph is still kept intact.  The
   proxies would clearly be marked as such in the repository.  You would
   (at proxy CSET creation time) validate that these proxies in fact DO NOT
   change any of the same lines that the later CSET changes (saves you from
   sending a patch that won't merge).  Later on, you can really send those
   CSETs to replace the proxies, or if there are conflicting changes in
   the upstream tree it is up to you to resolve them.

> Obviously this wouldn't apply if you fed BK patches into GNU patch, and
> then issued the commit from there...  but that way is a bit lossy, since
> you would need to recreate rename information among other things.

It would also lose the BK CSET identification, which would tell the
original submitter (and his local repository) that the patch was applied,
and when Linus sent out new patches/CSETs, the original submitter would
have to manually resolve these conflicts each time.  Maybe if BK had a
feature like patch, which says 'It appears that the changes in this CSET
have already been applied in <foo>, let's use <foo> instead'.

Cheers, Andreas

(*)  Larry, time to submit patches now so we can use BK for 2.7 ;-)
(**) Maybe I just don't know enough about how to use BK.  There are also
     a lot of distributed database issues involved of which I'm unaware.
(***)Maybe this is just another manifestation of fixes getting lost
     because they need a lot of attention to get into the kernel.
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Larry McVoy]

On Wed, Jan 30, 2002 at 10:24:59AM -0700, Andreas Dilger wrote:
> Well, the one benefit of using SCCS directories (which are only 1/3
> louder than CVS directories) is that tools like patch, make, ctags, emacs
> (I believe), etc. already understand what they are and how to extract
> the latest version of a file from there.  

Yup, I like 'em for that reason as well.  And you can do a

	bk clean # removes all non-modified working files
	ls	 # should see nothing but SCCS/

to clean up all that extra cruft that invariably winds up in your tree.

> I would have to agree.  Ted uses BK for e2fsprogs, and there have been
> several times when I try to send him a CSET, but he is unable to apply
> it because it is missing dependencies, even though I know those prior
> CSETs are actually independent changes that just happen to touch the
> same files.

Please see the response to Ingo and see if you could do what I suggested
there.  I believe that would work fine, if not, let me know.

> Also, (BK feature request time) there are times when I've done a 'bk citool'
> and committed a bunch of changes into a CSET, and later on done some more
> testing which revealed a bug or found that I'd forgotten to change the
> man page to track the changes I made.  I'd much rather be able to merge
> some more changes into the same CSET instead of creating a second CSET and
> now have two CSETs to ship around for what is logically a single change.**

In the next release, there is a "bk fix -c" that does what you want.  It
reedits the entire changeset, and preserves all your checkin comments. 
You don't want to use this if the changeset has propogated out of your
tree, but I use it all the time for exactly the situation you describe
above.  It will be in bk-2.1.4.

> I think it would quickly become a nightmare if you had to submit (and
> have accepted!) all of your changes to Linus IN ORDER.  As it stands now,
> there are lots of discrete changes I have in my ext2 tree, and whenever
> I get around to it or when people hit the same bug as me I generate a
> patch, edit out the irrelevant parts, and send it out.***

Yeah, we know.  Read the other mails and tell me if you think that the 
false dependency remove largely fixes this, somewhat fixes it, or doesn't
help.

> Granted, it is hard to keep distributed BK repositories consistent if you
> apply patches out of order, but at the same time, this is how development
> works in real life.  Two solutions I can see to this:
> 
> 1) Allow out-of-order CSET application (maybe with some sort of warning
>    that Linus can turn off, because _every_ CSET he would get would be
>    missing dependencies from somebody's tree).

This one is unlikely to happen, it is a much harder problem technically than
it appears on the surface.

> 2) Allow "proxy" CSETs to be included which say "the changes from adilger
>    adilger@lynx.adilger.int|ChangeSet|20011226061040|56205 changed lines
>    X-Y, Z of file fs/ext2/super.c" but doesn't actually contain those
>    changes, so that the CSET dependency graph is still kept intact.  

In other words, "proxy" == "place holder".  This is doable but it makes
synchronizing repositories quite a bit more complex.  I.e., if I pull
from you and I have place holders and you have the real thing, should
I get 'em or not?  If the answer is always "or not", then this is a 
doable thing.  We'd need to modify the "bk takepatch" code path to do
the right thing when given the real patch but that's doable as well.
Interesting idea, we could do this and it would be relatively fast to
do, like a week or two.

> It would also lose the BK CSET identification, which would tell the
> original submitter (and his local repository) that the patch was applied,

We wouldn't lose it, we'd have to add some extra state to say it is in there
but only as a placeholder, so keep bugging Linus.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Andreas Dilger]

On Jan 30, 2002  09:34 -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 10:24:59AM -0700, Andreas Dilger wrote:
> > Well, the one benefit of using SCCS directories (which are only 1/3
> > louder than CVS directories) is that tools like patch, make, ctags, emacs
> > (I believe), etc. already understand what they are and how to extract
> > the latest version of a file from there.  
> 
> Yup, I like 'em for that reason as well.  And you can do a
> 
> 	bk clean # removes all non-modified working files
> 	ls	 # should see nothing but SCCS/
> 
> to clean up all that extra cruft that invariably winds up in your tree.

OK, so how about adding (optional) .SCCS and .BitKeeper support to BK
for those that care about the "cleanliness" of the tree?

> > I would have to agree.  Ted uses BK for e2fsprogs, and there have been
> > several times when I try to send him a CSET, but he is unable to apply
> > it because it is missing dependencies, even though I know those prior
> > CSETs are actually independent changes that just happen to touch the
> > same files.
> 
> Please see the response to Ingo and see if you could do what I suggested
> there.  I believe that would work fine, if not, let me know.

Yes, technically it works fine, but practically not.  For example, I want
to test _all_ of the changes I make, and to test them each individually
is a lot of work.  Putting them all in the same tree, and testing them
as a group is a lot less work.  More importantly, this is how people do
their work in real life, so we don't want to change how people work to
fit the tool, but vice versa.

> > Also, (BK feature request time) there are times when I've done a 'bk citool'
> > and committed a bunch of changes into a CSET, and later on done some more
> > testing which revealed a bug or found that I'd forgotten to change the
> > man page to track the changes I made.  I'd much rather be able to merge
> > some more changes into the same CSET instead of creating a second CSET and
> > now have two CSETs to ship around for what is logically a single change.**
> 
> In the next release, there is a "bk fix -c" that does what you want.  It
> reedits the entire changeset, and preserves all your checkin comments. 
> You don't want to use this if the changeset has propogated out of your
> tree, but I use it all the time for exactly the situation you describe
> above.  It will be in bk-2.1.4.

Yes, I had thought about the CSET propogation issue also, but in my cases
it is generally not an issue.  If the CSET got a different version/ID than
the original, it would help avoid such issues also.

I suppose the parallel (if 'bk fix -c' doesn't allow it) would be to allow
splitting a CSET into smaller parts, so in case you committed two unrelated
changes to a single CSET (i.e. overzealousness in the 'bk citool' window),
you could unmerge them for separate inclusion if needed.

> > I think it would quickly become a nightmare if you had to submit (and
> > have accepted!) all of your changes to Linus IN ORDER.  As it stands now,
> > there are lots of discrete changes I have in my ext2 tree, and whenever
> > I get around to it or when people hit the same bug as me I generate a
> > patch, edit out the irrelevant parts, and send it out.***
> 
> Yeah, we know.  Read the other mails and tell me if you think that the 
> false dependency remove largely fixes this, somewhat fixes it, or doesn't
> help.

Yes, it is mostly the false dependency issue at work.  While avoiding this
on a per-file basis is a start, you really need to avoid it on a per-line
basis.

> > 2) Allow "proxy" CSETs to be included which say "the changes from adilger
> >    adilger@lynx.adilger.int|ChangeSet|20011226061040|56205 changed lines
> >    X-Y, Z of file fs/ext2/super.c" but doesn't actually contain those
> >    changes, so that the CSET dependency graph is still kept intact.  
> 
> In other words, "proxy" == "place holder".

Yes.

> This is doable but it makes synchronizing repositories quite a bit more
> complex.  I.e., if I pull from you and I have place holders and you have
> the real thing, should I get 'em or not?

I would say yes, always.  In my (limited) BK experience, when I pull I
generally want to get everything, and I push/send only specific things.
The proxies would only be needed if you didn't want everything.

> If the answer is always "or not", then this is a doable thing.  We'd
> need to modify the "bk takepatch" code path to do the right thing when
> given the real patch but that's doable as well.  Interesting idea, we
> could do this and it would be relatively fast to do, like a week or two.

Yes, the good thing about the proxy CSET idea is that it allows the
reciever to avoid the actual out-of-order patch application problems,
and still keeps the knowledge about per-line changes to a file.

As for what would happen when someone changes the lines that a proxy CSET
refers to, I would imagine that it would be like a merge where all of the
potential changes from the proxy are discarded.

> > It would also lose the BK CSET identification, which would tell the
> > original submitter (and his local repository) that the patch was applied,
> 
> We wouldn't lose it, we'd have to add some extra state to say it is in there
> but only as a placeholder, so keep bugging Linus.

This is out of context.  This was referring to applying CSETs as regular
patches, and is unrelated to the proxy CSET idea.

The mere fact that a proxy CSET might be visible (with submitter
identification) would be useful.  Some people might use the real CSETs
instead of the proxies (ala developing against -dj or -ac kernels instead
of -linus kernels) and it wouldn't introduce extra conflicts/merges later.
If they actually needed what Linus had as a proxy CSET, then it would be
grounds for merging that proxy for real.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Larry McVoy]

On Wed, Jan 30, 2002 at 01:03:19PM -0700, Andreas Dilger wrote:
> > Please see the response to Ingo and see if you could do what I suggested
> > there.  I believe that would work fine, if not, let me know.
> 
> Yes, technically it works fine, but practically not.  For example, I want
> to test _all_ of the changes I make, and to test them each individually
> is a lot of work.  Putting them all in the same tree, and testing them
> as a group is a lot less work.  More importantly, this is how people do
> their work in real life, so we don't want to change how people work to
> fit the tool, but vice versa.

There is a subtle point that this comment misses, it's worth thinking about.

This thread has been about the idea of being able to send any one of those
changes out in isolation, right?  That's the problem we are solving.  But
your statement is that you want to test them all at once, testing them one
at a time is too much work.

Doesn't that mean that you don't even know if these changes compile, let
alone run, when you send them out individually?  You haven't tested them,
you've only tested the set of them as one unit.

In many environments, BK is doing exactly the right thing.  It lets you send
the stuff you've tested together.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Jeff Garzik]

On Thu, Jan 31, 2002 at 09:11:10AM -0800, Larry McVoy wrote:
> This thread has been about the idea of being able to send any one of those
> changes out in isolation, right?  That's the problem we are solving.  But
> your statement is that you want to test them all at once, testing them one
> at a time is too much work.

Maybe, maybe not.  When hacking on filesystems I try to produce
"viro-style" patches, which are a series of patches, each containing
a single transformation.  Each one is tested in isolation in addition to
the final product.  Extremely useful for nipping problems in the bud
sooner rather than later.

	Jeff

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Andreas Dilger]

On Jan 31, 2002  09:11 -0800, Larry McVoy wrote:
> On Wed, Jan 30, 2002 at 01:03:19PM -0700, Andreas Dilger wrote:
> > Yes, technically it works fine, but practically not.  For example, I want
> > to test _all_ of the changes I make, and to test them each individually
> > is a lot of work.  Putting them all in the same tree, and testing them
> > as a group is a lot less work.  More importantly, this is how people do
> > their work in real life, so we don't want to change how people work to
> > fit the tool, but vice versa.
> 
> This thread has been about the idea of being able to send any one of those
> changes out in isolation, right?  That's the problem we are solving.

But what you are proposing is that I keep N trees for each of my N changes
against the baseline, keep all of those N trees up-to-date, compile
and reboot each of the N kernels for each local or upstream change, and
possibly have N! different kernels to test each combination of changes.

> But your statement is that you want to test them all at once, testing
> them one at a time is too much work.

I guess I wasn't very clear then.  I will probably test changes I make
in _order_, but not necessarily in _isolation_.  I may also not test
_every_ change I make individually if it is fairly minor and "obvious".
If the changes are orthogonal, testing kernel+A and testing kernel+A+B
should be enough to tell me that B works without A.  That means I should
be able to send out B without everyone needing A in order to test it.

> Doesn't that mean that you don't even know if these changes compile, let
> alone run, when you send them out individually?  You haven't tested them,
> you've only tested the set of them as one unit.

It boils down to "how much testing is enough" for each of the separate
changes.  Is eyeballing them enough?  Is compiling enough?  Is a single
reboot enough?  I don't have N machines, let alone N!, to test each of
the N changes I have in my tree individually.

There is also value in saying "I've had this patch in my kernel for X
{days,weeks,months} and it works fine", and by your statement above I
could only do this with a single change.

What I'm saying is that I will code a specific change A, test it, and then
usually go on to code the next change B in the tree that has A in it.
Yes, in some cases testing B in isolation is needed (big changes, or
changes which need to be benchmarked in isolation).  In general you
wouldn't make change A if it wasn't worthwhile, and after it's done why
would you not want to continue using it?

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: real BK usage (was: A modest proposal -- We need a patch penguin)

[Linus Torvalds]

On Wed, 30 Jan 2002, Andreas Dilger wrote:
>
> Well, the one benefit of using SCCS directories (which are only 1/3
> louder than CVS directories)

Note that I dislike CVS too. So it's not 1/3 loader than CSV, it's
infinitely louder than nothing at all, and it's quite noticeably louder
than a ".bitkeeper" subdirectory.

>			 is that tools like patch, make, ctags,
> emacs (I believe), etc. already understand what they are and how to
> extract the latest version of a file from there.

So past stupidities would keep you from doing it _right_?

>				  If these tools were
> changed to also recognize .SCCS dirs, then BK could eventually follow
> suit, but it would be impractical until they are widely available.*

Don't be silly. It obviously works the other way. Nobody patches lots of
different tools for a situation that doesn't even exist. But patching
_one_ tool (bk) to be sane makes sense, and then if/when people start
using them, the other tools will certainly follow.

> I would have to agree.  Ted uses BK for e2fsprogs, and there have been
> several times when I try to send him a CSET, but he is unable to apply
> it because it is missing dependencies, even though I know those prior
> CSETs are actually independent changes that just happen to touch the
> same files.

I won't use changesets for this reason, and Larry knows it. I'd still
apply patches, even if I was using bk. It's not as if everybody else would
use bk anyway.

The advantage of bk is that unlike CVS I can use bk in many different
places, and just clone the bk trees. Let's face it, CVS branches suck,
always have, and always will. CVS doesn't allow you to have different CVS
trees, and if one of them starts to look successful, you merge that tree
into your main one.

So I'd personally use changesets just for my _own_ use.

Now, Larry has promised me usable changesets for a long time, but it
obviously hasn't happened yet.

		Linus

Re: A modest proposal -- We need a patch penguin

[Momchil Velikov]

>>>>> "Linus" == Linus Torvalds <torvalds@transmeta.com> writes:

Linus> In article <20020130034746.K32317@havoc.gtf.org>,
Linus> Jeff Garzik  <garzik@havoc.gtf.org> wrote:
>> 
>> Instead of doing this stuff half-assed, just convince Linus to use BK :)

Linus> The thing is, I actually _want_ to use BK (as opposed to CVS, which I
Linus> really don't think cuts it). 

Well, I know now the whole linux community will hate me, but this has
never stopped me before, it won't now.

How about using something FREE ?

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:33 am, Linus Torvalds wrote:
> In article <20020130034746.K32317@havoc.gtf.org>,
> Jeff Garzik  <garzik@havoc.gtf.org> wrote:
> >
> >Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> The thing is, I actually _want_ to use BK (as opposed to CVS, which I
> really don't think cuts it). 
> 
> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
> be honest: I've used CVS, but I've never really used BK. Larry has given
> me the demos, and I actually decided to re-do the examples, but it takes
> time and effort to get used to new tools, and I'm a bit worried that
> I'll find other things to hate than just those loud filenames.

Oh gosh, I hate those too.  (Yes, this is a "me too".)  Larry, could we 
*please* have that metadata in a .file?

/me also detests those shouting CVS's in cvs trees and is easily put off
by such trivial things

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Albert D. Cahalan]

Daniel Phillips writes:
> On January 30, 2002 10:33 am, Linus Torvalds wrote:

>> I still dislike some things (those SHOUTING SCCS files) in bk, and let's
>> be honest: I've used CVS, but I've never really used BK. Larry has given
>> me the demos, and I actually decided to re-do the examples, but it takes
>> time and effort to get used to new tools, and I'm a bit worried that
>> I'll find other things to hate than just those loud filenames.
>
> Oh gosh, I hate those too.  (Yes, this is a "me too".)  Larry, could we 
> *please* have that metadata in a .file?

Try "man ls":

       -I, --ignore=PATTERN
              do not list implied entries matching shell PATTERN


So then something like this...

alias ls='/bin/ls --ignore=SCCS'

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> 
> So then something like this...
> 
> alias ls='/bin/ls --ignore=SCCS'

Oh, that's very useful. Considering that everything else still finds them, 
like find, shell autocompletion etc.

The only thing "--ignore=xxx" is useful for is hackers that want to break 
into your system and hide their files.

		Linus

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On April 5, 2002 03:21 am, Linus Torvalds wrote:
> On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> > 
> > So then something like this...
> > 
> > alias ls='/bin/ls --ignore=SCCS'
> 
> Oh, that's very useful. Considering that everything else still finds them, 
> like find, shell autocompletion etc.
> 
> The only thing "--ignore=xxx" is useful for is hackers that want to break 
> into your system and hide their files.

And anyway, Larry sorta/kinda agreed to let us hide his bk metadata in one or
more hidden files, and when I grab him for clubbing^W dinner in a few days
I'll have a good chance to beat on him further to actually get that little
feature, which means more to me than it really should, personally.

-- 
Daniel

Re: patch-2.4.19-pre5-ac2

[Jonathan A. Davis]

The radeon updates in pre5-ac2 seem to make a minor mess out of my Radeon
7500's console fb.  After X starts up -- switching back to a text console
results in artifacts from the X display contents plus borked scrolling.  
No tendency to crash though and switching back to X results in a normal X
display.  I dropped out the patches to:

drivers/char/drm/radeon_state.c
drivers/video/radeon.h
drivers/video/radeonfb.c

Which returned things to normal.  I'm not up enough on the kernel fb stuff 
to dig into the patch guts very effectively.

The mb chipset is a VIA K266A with very conservative (no overclock, 
etc...) settings.  Processor is an Athlon XP 1700+.

Here is some boot/config info:

--dmesg--
radeonfb: ref_clk=2700, ref_div=12, xclk=23000 defaults
Console: switching to colour frame buffer device 80x30
radeonfb: ATI Radeon 7500 QW  DDR SGRAM 64 MB
radeonfb: DVI port no monitor connected
radeonfb: CRT port CRT monitor connected

--lspci--
01:00.0 VGA compatible controller: ATI Technologies Inc: Unknown device 
5157 (prog-if 00 [VGA])
        Subsystem: ATI Technologies Inc: Unknown device 013a
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr- Stepping+ SERR- FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- 
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 32 (2000ns min), cache line size 08
        Interrupt: pin A routed to IRQ 11
        Region 0: Memory at e0000000 (32-bit, prefetchable) [size=128M]
        Region 1: I/O ports at c000 [size=256]
        Region 2: Memory at ed000000 (32-bit, non-prefetchable) [size=64K]
        Expansion ROM at <unassigned> [disabled] [size=128K]
        Capabilities: [58] AGP version 2.0
                Status: RQ=47 SBA+ 64bit- FW- Rate=x1,x2
                Command: RQ=31 SBA+ AGP+ 64bit- FW- Rate=x2
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA 
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-

--XFree86.0.log--
(--) PCI:*(1:0:0) ATI Radeon 7500 QW rev 0, Mem @ 0xe0000000/27, 
0xed000000/16,I/O @ 0xc000/8
...


-- 

-Jonathan <davis@jdhouse.org>

Re: patch-2.4.19-pre5-ac2

[Peter Horton]

On Thu, Apr 04, 2002 at 08:19:49PM -0600, Jonathan A. Davis wrote:
> 
> The radeon updates in pre5-ac2 seem to make a minor mess out of my Radeon
> 7500's console fb.  After X starts up -- switching back to a text console
> results in artifacts from the X display contents plus borked scrolling.  
> No tendency to crash though and switching back to X results in a normal X
> display.  I dropped out the patches to:
> 

Yep. The accelerator needs resetting on each console switch so that we
can cope when X leaves it in a funky state. The new patch I posted
yesterday should fix it, or for a quick fix add the lines

	if(accel)
		radeon_engine_init_var();

after the call to do_install_cmap() in radeon_fb_setvar().

P.

Re: patch-2.4.19-pre5-ac2

[Geert Uytterhoeven]

On Fri, 5 Apr 2002, Peter Horton wrote:
> On Thu, Apr 04, 2002 at 08:19:49PM -0600, Jonathan A. Davis wrote:
> > The radeon updates in pre5-ac2 seem to make a minor mess out of my Radeon
> > 7500's console fb.  After X starts up -- switching back to a text console
> > results in artifacts from the X display contents plus borked scrolling.  
> > No tendency to crash though and switching back to X results in a normal X
> > display.  I dropped out the patches to:
> > 
> 
> Yep. The accelerator needs resetting on each console switch so that we
> can cope when X leaves it in a funky state. The new patch I posted
> yesterday should fix it, or for a quick fix add the lines
> 
> 	if(accel)
> 		radeon_engine_init_var();
> 
> after the call to do_install_cmap() in radeon_fb_setvar().

You have to reinit the acceleration engine if FB_ACCELF_TEXT is set again. If
this is already the case, you may be running an X server that's not fbdev aware
(or aren't using `option UseFBDev'?)

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Geert Uytterhoeven]

On Thu, 4 Apr 2002, Linus Torvalds wrote:
> On Thu, 4 Apr 2002, Albert D. Cahalan wrote:
> > 
> > So then something like this...
> > 
> > alias ls='/bin/ls --ignore=SCCS'
> 
> Oh, that's very useful. Considering that everything else still finds them, 
> like find, shell autocompletion etc.
> 
> The only thing "--ignore=xxx" is useful for is hackers that want to break 
                                                 ^^^^^^^
> into your system and hide their files.

Ugh, this is linux-kernel! (cfr. signature)

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Jeff Garzik wrote:

> Instead of doing this stuff half-assed, just convince Linus to use BK :)

I don't care what Linus uses, but Linus decision should not lock other
developers into using the same tools, e.g. it should not become
inconvenient to send simple patches. The basic communication tools should
still be mail and patches. What we IMO need is a patch management system
not a source management system.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> I don't care what Linus uses, but Linus decision should not lock other
> developers into using the same tools, e.g. it should not become
> inconvenient to send simple patches. The basic communication tools should
> still be mail and patches. What we IMO need is a patch management system
> not a source management system.

Thats been promised long back. And Linus said many times both in mail and
in person that if he started using bitkeeper he wouldnt force others to do
so.

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Alan Cox wrote:

> Thats been promised long back. And Linus said many times both in mail and
> in person that if he started using bitkeeper he wouldnt force others to do
> so.

I know and that wasn't really my problem. It is the "Linus should just
use bk and all problems are solved" attitude, which makes me nervous.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 06:29:04PM +0100, Roman Zippel wrote:
> I know and that wasn't really my problem. It is the "Linus should just
> use bk and all problems are solved" attitude, which makes me nervous.

If that's referring to my message, I was just cheerleading :)

We are all fully aware of the multitude of issues that need to be
examined before BK or any other patch management system can be
adequately implemented.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 01:59:56PM +0100, Roman Zippel wrote:
> Hi,
> 
> On Wed, 30 Jan 2002, Jeff Garzik wrote:
> 
> > Instead of doing this stuff half-assed, just convince Linus to use BK :)
> 
> I don't care what Linus uses, but Linus decision should not lock other
> developers into using the same tools, e.g. it should not become
> inconvenient to send simple patches. The basic communication tools should
> still be mail and patches. What we IMO need is a patch management system
> not a source management system.

BK can happily be used as a patch management system and it can, and has
for years, been able to accept and generate traditional patches.  Linus
could maintain the source in a BK tree and make it available as both
a BK tree and traditional patches.  It's a one line command to generate
a release patch and another one line command to generate the release
tarball.

By the way, you can send BK patches exactly the way that you send regular
patches, with the difference being that BK has an optional way of wrapping
them up in uuencode (or whatever) so that mailers don't stomp on them.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Jochen Friedrich]

Hi Larry,

> with the difference being that BK has an optional way of wrapping
> them up in uuencode (or whatever) so that mailers don't stomp on them.

isn't that just the same as sending them as attchment? And isn't that
discouraged?

Cheers,
--jochen

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 05:34:12PM +0100, Jochen Friedrich wrote:
> Hi Larry,
> 
> > with the difference being that BK has an optional way of wrapping
> > them up in uuencode (or whatever) so that mailers don't stomp on them.
> 
> isn't that just the same as sending them as attchment? And isn't that
> discouraged?

We have a generic wrapping/unwrapping mechanism.  The wrapping can be as
draconian as a uuencode/mimencoded attachment or as light as a crc envelope.
We don't care, we're mechanism providers in this area, not policy setters.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 05:34:12PM +0100, Jochen Friedrich wrote:
> > with the difference being that BK has an optional way of wrapping
> > them up in uuencode (or whatever) so that mailers don't stomp on them.
> 
> isn't that just the same as sending them as attchment? And isn't that
> discouraged?

gcc developers love that uuencoded stuff for large files it seems, but
it's not big in l-k land...  plaintext patches are preferred.  The main
requirement is that the patch is NOT obscured by base64 or uuencoding.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

On Wed, 30 Jan 2002, Larry McVoy wrote:

> > What we IMO need is a patch management system
> > not a source management system.
>
> BK can happily be used as a patch management system and it can, and has
> for years, been able to accept and generate traditional patches.  Linus
> could maintain the source in a BK tree and make it available as both
> a BK tree and traditional patches.

It's not really the same or that's not what I mean with patch management
system or can bk also manage the patches, which Linus drops?
What I have in mind is a patch management system which tracks the status
of unapplied patches. The status could be:
- patch applies cleanly to tree x.
- patch is approved/disappoved by y.
- patch is in tree z since version...
This system should not only support Linus, but also other tree
maintainers, so they can pick patches they want to integrate into their
trees, which could also feed back information which patch conflicts with
another patch (this could be done by the patchbot as well, but humans are
usually better at judging which patch is more important).
Linus again could use this information to decide which patches he
integrates into his in own tree, so he can easier sync up with other
trees.
My suggestion would be to setup an alias for Linus and/or Marcelo, which
just collects patches send to them. Anyone interested in implementing a
patchbot can use this data to test and demonstrate his/her implementation.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 09:06:17PM +0100, Roman Zippel wrote:
> > > What we IMO need is a patch management system
> > > not a source management system.
> >
> > BK can happily be used as a patch management system and it can, and has
> > for years, been able to accept and generate traditional patches.  Linus
> > could maintain the source in a BK tree and make it available as both
> > a BK tree and traditional patches.
> 
> It's not really the same or that's not what I mean with patch management
> system or can bk also manage the patches, which Linus drops?
> What I have in mind is a patch management system which tracks the status
> of unapplied patches. The status could be:
> - patch applies cleanly to tree x.
> - patch is approved/disappoved by y.
> - patch is in tree z since version...

Yeah, I think BK can do this, most of what you are describing is covered
by already existing BK commands and practices.  There are literally dozens
of different sites using BK to track the kernel and both internal and
external sources of patches.  I'm sure there are issues that need to be
resolved and we'll try to do so.  

I might be mistaken, I also get the feeling that your real issue might
be that you don't like/understand/something BK and you are pushing for a
different answer.  That's cool, there are now two patchbot projects you
can go join and start coding.  Some of our biggest fans are people who
have tried that and discovered exactly how complex the problem space
really is, so it's actually in my best interest to encourage you to
go help out with one of those projects.  If you solve the problem, the
kernel benefits and I get to learn something: that's good.  Or you don't
and you'll come to respect BK: that's good too, at least I like it.

So have some fun, this is actually a more challenging area than any
kernel work I've ever done or seen, including SMP threading, so the more
you get into it, the more fun you can have (assuming that banging your
brain against some hard problems meets your fun definition).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Larry McVoy wrote:

> I might be mistaken, I also get the feeling that your real issue might
> be that you don't like/understand/something BK and you are pushing for a
> different answer.  That's cool, there are now two patchbot projects you
> can go join and start coding.

Um, I really have no time for this, I have no problem with sending
patches to whoever/whatever.
I'm not arguing for/against bk at all, if you can demonstrate how bk
solves all of our problems and people want to use it, I had absolutely
no problem with that.
But you are correct my experiences with bk are limited and not very
encouraging. Now I only use it to extract specific versions from the ppc
tree and to import that into the apus tree. Maybe a "bk howto for cvs
users" might help, which shows typical cvs use cases and how to do them
with bk, the documentation I found only stresses the bad sides of cvs.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 10:02:05PM +0100, Roman Zippel wrote:
> Larry McVoy wrote:
> 
> > I might be mistaken, I also get the feeling that your real issue might
> > be that you don't like/understand/something BK and you are pushing for a
> > different answer.  That's cool, there are now two patchbot projects you
> > can go join and start coding.
> 
> Um, I really have no time for this
> [BK for cvs users request deleted]

Apparently not.  It might help if you read the docs, this has been there 
for months if not years:

http://www.bitkeeper.com/Documentation.HOWTO.CVS.html
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

Larry McVoy wrote:

> > [BK for cvs users request deleted]
> 
> Apparently not.  It might help if you read the docs, this has been there
> for months if not years:
> 
> http://www.bitkeeper.com/Documentation.HOWTO.CVS.html

I meant the equivalent of cvs uses cases not the equivalent of three cvs
commands, e.g. how would I handle cvs branches and join branches, how do
I check out a specific version/date, how do I track external sources,
which are not using bk, how do I get grep/cscope/... working without
doubling the needed disk space with bk -r co.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Wed, Jan 30, 2002 at 11:13:29PM +0100, Roman Zippel wrote:
> I meant the equivalent of cvs uses cases not the equivalent of three cvs
> commands, e.g. how would I handle cvs branches and join branches, how do
> I check out a specific version/date, how do I track external sources,
> which are not using bk, how do I get grep/cscope/... working without
> doubling the needed disk space with bk -r co.

s/branch/repository/ - every repository is essentially a vendor branch that
works.

bk clone -r<rev> reproduces the tree as of that rev.

bk -r grep is very useful.

Tracking external sources is fairly obvious, and BK excels at it, and
virtually 100% of our users have figured out how to do it without asking
us questions.

All of your issues are actually in the docs and well documented.  And if
you had asked us how to do it, we would have pointed you at the docs or
fixed them if they were incomplete.  I'm at a loss as to why you want to
prove to the entire lkml that you can't/won't read the docs, but hey,
if that's your bag go for it.  I'm willing to answer your questions as
they come up, so keep 'em coming.  It just helps educate the general 
list, though I suspect they may get a bit tired of it because most people
do read the docs and are more interested in the harder problems.  So
maybe we should take this offline?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A modest proposal -- We need a patch penguin

[Roman Zippel]

Hi,

Larry McVoy wrote:

> All of your issues are actually in the docs and well documented.  And if
> you had asked us how to do it, we would have pointed you at the docs or
> fixed them if they were incomplete.  I'm at a loss as to why you want to
> prove to the entire lkml that you can't/won't read the docs, but hey,
> if that's your bag go for it.  I'm willing to answer your questions as
> they come up, so keep 'em coming.

This is going nowhere, so that will be my last statement.
I knew most of the answers, I was just trying to give an idea of the
problems a cvs user has to deal with if he is confronted with bk. I
evaluated bk for myself and decided that it's not the right tool for
_me_, so I have no questions.

bye, Roman

Re: A modest proposal -- We need a patch penguin

[Chris Ricker]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

You snipped the part I was actually interested in.  Let me try again.

We're agreed that the files themselves are the best indicator of where to
route patches, and that MAINTAINERS isn't useful for much besides deciding
who should get IPO offers ;-).  What I'm wondering is where I, as someone
who is listed in some of the Documentation/* stuff as its maintainer, should
be sending patches.  You want a hierarchy, and I think that's perfectly
reasonable, but I have no idea who the layer of the hierarchy between me and
you is....

later,
chris

Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

On Tue, 29 Jan 2002, Chris Ricker wrote:
>
> We're agreed that the files themselves are the best indicator of where to
> route patches, and that MAINTAINERS isn't useful for much besides deciding
> who should get IPO offers ;-).  What I'm wondering is where I, as someone
> who is listed in some of the Documentation/* stuff as its maintainer, should
> be sending patches.  You want a hierarchy, and I think that's perfectly
> reasonable, but I have no idea who the layer of the hierarchy between me and
> you is....

Ahh..

I had the same problem with Documentation/Configure.help, as you saw.

My solution in that case (when the issue came to a flame-fest) was to just
split up the documentation - which makes it a whole lot more maintainable
for everybody, and also makes it fairly explicit who maintains it for most
cases.

Basically, I'd really like documentation to go with the thing it
documents. This is something where the docbook stuff helped noticeably.

			Linus

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Tue, Jan 29, 2002 at 06:54:04PM -0800, Linus Torvalds wrote:
> Basically, I'd really like documentation to go with the thing it
> documents. This is something where the docbook stuff helped noticeably.

Well...  kinda sorta.

We -used- to really have documentation with the thing it documents, like
drivers/foo/README.  In-lined source comments are one piece of the
puzzle yes, but the -bulk- of the docs are not anywhere near the thing
it documents.

I actually don't like stuffing documents in Documentation/DocBook
proper...  I've put docs for two drivers in there, but if the trend
continues a new dir structure will need to evolve.

Either we'll want Documentation/DocBook/<category>, or move the docbook
docs into the standard Documentation/* hierarchy......... or we'll start
moving docs back outside Documentation/*

	Jeff, on a semi-tangent

Re: A modest proposal -- We need a patch penguin

[Matthew D. Pitts]

Chris,

Thank you for saying this... I have things I would like do/add to the kernel
and I am not sure who to send them to.

Also, is there presently a maintainer for Supermount? If not, I would be
willing to pick it up for 2.5.x, as it is one of the things I want to work
on.

Matthew D. Pitts

----- Original Message -----
From: "Chris Ricker" <kaboom@gatech.edu>
To: "Linus Torvalds" <torvalds@transmeta.com>
Cc: "World Domination Now!" <linux-kernel@vger.kernel.org>
Sent: Tuesday, January 29, 2002 9:45 PM
Subject: Re: A modest proposal -- We need a patch penguin


> On Tue, 29 Jan 2002, Linus Torvalds wrote:
>
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
>
> You snipped the part I was actually interested in.  Let me try again.
>
> We're agreed that the files themselves are the best indicator of where to
> route patches, and that MAINTAINERS isn't useful for much besides deciding
> who should get IPO offers ;-).  What I'm wondering is where I, as someone
> who is listed in some of the Documentation/* stuff as its maintainer,
should
> be sending patches.  You want a hierarchy, and I think that's perfectly
> reasonable, but I have no idea who the layer of the hierarchy between me
and
> you is....
>
> later,
> chris
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Wed, Jan 30, 2002 at 07:49:41AM -0500, Matthew D. Pitts wrote:
 > 
 > Also, is there presently a maintainer for Supermount? If not, I would be
 > willing to pick it up for 2.5.x, as it is one of the things I want to work
 > on.

 I believe Juan Quintela <quintela@mandrakesoft.com>  did some work on 
 it in 2.4 for Mandrake's kernel.
 
-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Juan Quintela]

>>>>> "matthew" == Matthew D Pitts <mpitts@suite224.net> writes:

matthew> Chris,
matthew> Thank you for saying this... I have things I would like do/add to the kernel
matthew> and I am not sure who to send them to.

matthew> Also, is there presently a maintainer for Supermount? If not, I would be
matthew> willing to pick it up for 2.5.x, as it is one of the things I want to work
matthew> on.


I am working on it.  Actual version worked quite well in 2.4.x series,
but last kernels are showing same problems, worknig on that.

Later, Juan.


-- 
In theory, practice and theory are the same, but in practice they 
are different -- Larry McVoy

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 07:49 am, Matthew D. Pitts wrote:
> Chris,
>
> Thank you for saying this... I have things I would like do/add to the
> kernel and I am not sure who to send them to.

No, if you're not a maintainer then you still send them to the maintainer in 
the MAINTAINERS file.

The interesting question is, who does THAT maintainer send them to.  (We seem 
to be heading for a four-tiered system, with Linus at the top, a dozen or so 
lieutenants under him, and then the specific maintainers under them.  With 
individual developers submitting patches being the fourth tier.  Patches go 
from developer, to maintainer, to lieutenant, to linus.)

This doesn't sound like a bad thing for scalability reasons, and should also 
help address the "I sent my patch directly to linus a dozen times and I 
didn't hear back" problem.

The problem right now is a lot of the maintainers don't seem to know who 
their corresponding lieutenant is.  We're still waiting for clarification 
from Linus...


> Also, is there presently a maintainer for Supermount? If not, I would be
> willing to pick it up for 2.5.x, as it is one of the things I want to work
> on.

I didn't spot one in MAINTAINERS.  The email at the top of "mount.h" says:

> * Author:  Marco van Wieringen <mvw@planets.elm.net>

So that might be a good person to ask.  Of course who knows how old that 
email address is... :)

> Matthew D. Pitts

Rob

Re: A modest proposal -- We need a patch penguin

[Francois Romieu]

Rob Landley <landley@trommello.org> :
[...]
> The problem right now is a lot of the maintainers don't seem to know who 
> their corresponding lieutenant is.  We're still waiting for clarification 
> from Linus...

Feel free to send your patches here if you're lost.

-- 
Ueimor

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 05:03 pm, Francois Romieu wrote:
> Rob Landley <landley@trommello.org> :
> [...]
>
> > The problem right now is a lot of the maintainers don't seem to know who
> > their corresponding lieutenant is.  We're still waiting for clarification
> > from Linus...
>
> Feel free to send your patches here if you're lost.

I'm not a maintainer, just the friend of one who came close to burnout.

The "we" who are waiting for clarification is larger than the maintainer set, 
because if the maintainers can't function the developers who depend on the 
maintainers can't function either.

Rob

Re: A modest proposal -- We need a patch penguin

[Jesse Pollard]

---------  Received message begins Here  ---------
Rob Landley <landley@trommello.org>:
> 
> On Wednesday 30 January 2002 07:49 am, Matthew D. Pitts wrote:
> > Chris,
> >
> > Thank you for saying this... I have things I would like do/add to the
> > kernel and I am not sure who to send them to.
> 
> No, if you're not a maintainer then you still send them to the maintainer in 
> the MAINTAINERS file.
> 
> The interesting question is, who does THAT maintainer send them to.  (We seem 
> to be heading for a four-tiered system, with Linus at the top, a dozen or so 
> lieutenants under him, and then the specific maintainers under them.  With 
> individual developers submitting patches being the fourth tier.  Patches go 
> from developer, to maintainer, to lieutenant, to linus.)
> 
> This doesn't sound like a bad thing for scalability reasons, and should also 
> help address the "I sent my patch directly to linus a dozen times and I 
> didn't hear back" problem.
> 
> The problem right now is a lot of the maintainers don't seem to know who 
> their corresponding lieutenant is.  We're still waiting for clarification 
> from Linus...

Ummm. this might be silly, but shouldn't those announcements come from
the lieutenants?

Linus has announced who he accepts patches frin, and who will be doing the 2.0,
2.2, and 2.4 maintenance. It would seem logical to have those lieutenants
announce their maintainers.

How would Linus actually know who, (after his lieutenants) SHOULD send mail
to the lieutenants?

That is the problem in the first place...

It would help to have the information in the MAINTAINERS file though. As well
as the auxilary mailing lists supporting that activity. That way, users
who find a bug/create a patch/whatever would have an easier time locating
where to send the patch. Especially when it doesn't directly affect the
core kernel.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> Linus has announced who he accepts patches frin, and who will be doing the
> 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those 
> lieutenants announce their maintainers.

Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of 
2.5, does it make sense for Marcelo to announce the maintainer of usb for
2.4?

It's not as simple as you'd think.  Reason: it's not a tree, it's an
acyclic graph.  Hopefully.  ;-)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 09:39 pm, Daniel Phillips wrote:
> On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > Linus has announced who he accepts patches frin, and who will be doing
> > the 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those
> > lieutenants announce their maintainers.
>
> Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of
> 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> 2.4?
>
> It's not as simple as you'd think.  Reason: it's not a tree, it's an
> acyclic graph.  Hopefully.  ;-)

I'm still trying to figure out who all the lieutenants are.  (It seems Andre 
Hedrick reports to Jens Axboe, Rik van Riel might actually report to.. Andrea 
Arcangeli?  (Or Dave Jones.)  But who does Eric send his help patches to?  Is 
Kieth Owens at the top level or what?  It seems like both Kieth and Eric are 
also under Dave Jones.  I guess "patch penguin" is just "Miscelaneous 
Lieutenant".  Makes sense i the new context, I suppose...)

I expect it will all get worked out eventually.  Now that the secret of the 
difference between maintainers and lieutenants is out.  The thread seems to 
be dying down a bit... :)

Rob

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 04:29 am, Rob Landley wrote:
> On Wednesday 30 January 2002 09:39 pm, Daniel Phillips wrote:
> > On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > > Linus has announced who he accepts patches frin, and who will be doing
> > > the 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those
> > > lieutenants announce their maintainers.
> >
> > Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of
> > 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> > 2.4?
> >
> > It's not as simple as you'd think.  Reason: it's not a tree, it's an
> > acyclic graph.  Hopefully.  ;-)
> 
> I'm still trying to figure out who all the lieutenants are.

You will never figure that out, it isn't predefined.  It reshapes itself on the
fly, and is really defined by what is going on at any given time.  That said,
it's usually possible to figure out how the main maintainers are, and what to
send where, just don't hope to ever nail that down in a rigid structure.  It's
not rigid.

> (It seems Andre 
> Hedrick reports to Jens Axboe, Rik van Riel might actually report to.. Andrea 
> Arcangeli?  (Or Dave Jones.)  But who does Eric send his help patches to?  Is 
> Kieth Owens at the top level or what?  It seems like both Kieth and Eric are 
> also under Dave Jones.  I guess "patch penguin" is just "Miscelaneous 
> Lieutenant".  Makes sense i the new context, I suppose...)
> 
> I expect it will all get worked out eventually.  Now that the secret of the 
> difference between maintainers and lieutenants is out.

By the way, that never was a secret to anybody in active development.

> The thread seems to be dying down a bit... :)

Right, people are working on solutions.  As usual, though much dung did fly,
Linus comes out smelling like a rose.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:

> You will never figure that out, it isn't predefined.  It reshapes itself on
> the fly, and is really defined by what is going on at any given time.  That
> said, it's usually possible to figure out how the main maintainers are, and
> what to send where, just don't hope to ever nail that down in a rigid
> structure.  It's not rigid.

As long as the maintainers know who the lieutenants are, nobody under them 
should really have to care that much...

> > I expect it will all get worked out eventually.  Now that the secret of
> > the difference between maintainers and lieutenants is out.
>
> By the way, that never was a secret to anybody in active development.

I.E. the people who knew it knew it, and hence never noticed the problem...

There are, however, some people writing largeish bits of code that did not in 
fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
files and CML2, Kieth Owens' new build process...  Maybe it was even a factor 
in Alan Cox burning out (you'd have to ask him about that)...

> > The thread seems to be dying down a bit... :)
>
> Right, people are working on solutions.  As usual, though much dung did
> fly, Linus comes out smelling like a rose.

Gee, what a suprise. :)

Rob

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Thu, 31 Jan 2002 00:32:40 -0500, 
Rob Landley <landley@trommello.org> wrote:
>On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
>> > I expect it will all get worked out eventually.  Now that the secret of
>> > the difference between maintainers and lieutenants is out.
>>
>> By the way, that never was a secret to anybody in active development.
>
>I.E. the people who knew it knew it, and hence never noticed the problem...
>
>There are, however, some people writing largeish bits of code that did not in 
>fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
>files and CML2, Kieth Owens' new build process... 

Both ESR and I definitely know about this process but kbuild is one of
the awkward systems that affects the entire kernel.  The final kbuild
system goes straight to Linus, there is nobody else to send it to.

kbuild 2.5 must match the current makefiles and config settings before
it can go in so it is impractible to target anything expect the
standard kernel, there is far too much work involved in tracking
makefile and config changes in -ac, -dj, -whoever.  Even if kbuild was
done against another tree, it would have to be redone and reverified
before sending to Linus, there is no way to extract kbuild 2.5 from a
divergent tree and expect it work on Linus's tree.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 06:32 am, Rob Landley wrote:
> On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
> > Rob Landley apparently wrote:
> > > I expect it will all get worked out eventually.  Now that the secret of
> > > the difference between maintainers and lieutenants is out.
> >
> > By the way, that never was a secret to anybody in active development.
> 
> I.E. the people who knew it knew it, and hence never noticed the problem...
> 
> There are, however, some people writing largeish bits of code that did not in 
> fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
> files and CML2, Kieth Owens' new build process...

They all know who the lieutenants are, I can assure you.

> Maybe it was even a factor in Alan Cox burning out (you'd have to ask him
> about that)...

Whatever gave you the idea that Alan is burnt out?

A observation: before proposing how we should fix the Linux development
process, perhaps you should have studied it enough to know how it works,
first.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Thu, Jan 31, 2002 at 12:32:40AM -0500, Rob Landley wrote:
> On Wednesday 30 January 2002 10:40 pm, Daniel Phillips wrote:
> > > I expect it will all get worked out eventually.  Now that the secret of
> > > the difference between maintainers and lieutenants is out.

> > By the way, that never was a secret to anybody in active development.
> 
> I.E. the people who knew it knew it, and hence never noticed the problem...
> 
> There are, however, some people writing largeish bits of code that did not in 
> fact seem to know it.  Andre Hedrick's IDE work, Eric Raymond with the help 
> files and CML2, Kieth Owens' new build process...

ESR was told things repeatedly and they didn't sink in.
(And I note you have been told this repeatedly, too.)
Did you miss Alan's comment as well?  Apparently so...
http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/1567.html

Andre knows his shit ten ways to Sunday, but one must speak ATA
not English with him.  Definite communications problem, despite the
fact that he writes solid low level driver code and tests it pretty
thoroughly.  It was clear at the beginning of 2.5.x that Jens' bio
stuff was going in and Andre's stuff would conflict with it.  Didn't
make Andre happy, but it was a good decision.  And now Andre's basic
taskfile stuff has been merged, so life is good.  I'm looking forward
to all the doors that taskfile has opened to the Linux kernel.

I dunno how Keith became one of your examples.  Maybe I missed it,
but I have not seen an announcement and review proving that kbuild
was ready for merging into 2.5.x.  With all due respect to the kbuild
list, I have seen a couple times "...but this was discussed and decided
upon on the kbuild list" and it turns to be an issue that definitely
requires further discussion and thought.

There is no secret.  Only willful ignorance.

If people writing largish pieces of code in isolation and expect them to
be applied without being cognizent of other development and feedback, I
-expect- their work to be dropped.  That is an example of a WORKING not
broken system.

The Linux kernel way is really evolution not revolution.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 31, 2002 07:27 am, Jeff Garzik wrote:
> ESR was told things repeatedly and they didn't sink in.
> (And I note you have been told this repeatedly, too.)
> Did you miss Alan's comment as well?  Apparently so...
> http://www.uwsg.iu.edu/hypermail/linux/kernel/0201.3/1567.html

With due regard to Alan, repeating the BS does not make it true.  Eric is in 
fact very wellknown in the kernel world, just not as a kernel hacker.  I'm 
perfectly content with his level of contribution, I think he's more than 
earned the right to spout.

And by the way, the wait for CML2+kbuild 2.5 is painful, I detest the old 
creaking cruft so much, I feel like I got something on me every time I change 
an option.  It hates me too, and retaliates by breaking itself in as many 
creative ways as it can.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 10:29:39PM -0500, Rob Landley wrote:
> I expect it will all get worked out eventually.  Now that the secret of the 
> difference between maintainers and lieutenants is out.  The thread seems to 
> be dying down a bit... :)

There Is No Cabal

Re: A modest proposal -- We need a patch penguin

[Keith Owens]

On Wed, 30 Jan 2002 22:41:12 -0500, 
Jeff Garzik <garzik@havoc.gtf.org> wrote:
>On Wed, Jan 30, 2002 at 10:29:39PM -0500, Rob Landley wrote:
>> I expect it will all get worked out eventually.  Now that the secret of the 
>> difference between maintainers and lieutenants is out.  The thread seems to 
>> be dying down a bit... :)
>
>There Is No Cabal

Actually the correct secret phrase is "There is no Eric conspiracy".
Hi Bruce.

http://tuxedo.org/~esr/ecsl/

Re: [lkml] Re: A modest proposal -- We need a patch penguin

[Ian Soboroff]

Jeff Garzik <garzik@havoc.gtf.org> writes:

> There Is No Cabal

or, alternatively, "If you want *BSD, you know where to find it."  The
funny thing is that the BSDs have all this hierarchy and whatnot, and
they still fight about it.

OTOH, I used to avoid Debian because it looked like more of an
ideology than a distribution, but once I ignored the sacred texts, it
was possible to learn that the distribution itself works _really_
well, and I could accept that was a result of the church.  But I still
don't know how to build a .deb.

ian

Re: [lkml] Re: A modest proposal -- We need a patch penguin

[Linus Torvalds]

In article <9cfy9iefvbt.fsf@rogue.ncsl.nist.gov>,
Ian Soboroff  <ian.soboroff@nist.gov> wrote:
>
>or, alternatively, "If you want *BSD, you know where to find it."  The
>funny thing is that the BSDs have all this hierarchy and whatnot, and
>they still fight about it.

I think the fact is that the grass is always greener somewhere else, and
all approaches have their problems. 

And it's always easier to point out problems with existing setups than
it is to come up with constructive changes. People end up wanting to
re-design everything, because that way they can make sure the problems
are gone - without really even knowing what new problems will appear
after a re-designed process.

The same thing happens in coding too, of course - you just _know_ you
can solve some problem by changing how something is done, and when you
actually code it up, you notice that "yes, I solved the problem", but
you also notice that "but now I have this other thing..". 

This is why trial-and-error is such a powerful way of doing things: try
many things, and yes, they all have their problems, but on the whole you
probably end up selecting the approaches where the problems are the
_least_ irritating.

The BIG problem with things like project management is that you simply
_cannot_ do lots of different trial-and-error things.  Sure, you can
try, and you'll get very Dilbertesque results: "The answer to all
problems: re-organize". 

Anyway, I'm actually personally willing to make small trials, and right
now I'm trying to see if it makes any difference if I try to use BK for
a month or two. I seriously doubt it will really "fix" everything, but
neither do I think big re-organizations and patch-lists will. But I'd be
stupid if I wasn't willing to try something.

(So far, trying out BK has only meant that I have spent _none_ of my
time merging patches and reading email, and most of my time writing
helper scripts and emails to Larry to make it possible to use BK in sane
ways that suit me. And I'll doubt you'll see any real productivity
increase from me for a while ;)

			Linus

Re: [lkml] Re: A modest proposal -- We need a patch penguin

[Larry McVoy]

On Fri, Feb 01, 2002 at 05:31:21AM +0000, Linus Torvalds wrote:
> (So far, trying out BK has only meant that I have spent _none_ of my
> time merging patches and reading email, and most of my time writing
> helper scripts and emails to Larry to make it possible to use BK in sane
> ways that suit me. And I'll doubt you'll see any real productivity
> increase from me for a while ;)

Hey, we're hacking away as well, we'll have some fixes for you ASAP.
I'm just adding a few touches to Wayne's diff change you wanted.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [lkml] Re: A modest proposal -- We need a patch penguin

[Craig Schlenter]

On Fri, Feb 01, 2002 at 05:31:21AM +0000, Linus Torvalds wrote:
[snip]
> Anyway, I'm actually personally willing to make small trials, and right
> now I'm trying to see if it makes any difference if I try to use BK [snip]

Are there any other cute pieces of technology that might be able to
help? I'm thinking specifically about a couple of scripts, bits of
procmail and changes to pine perhaps (or whatever mailer you use) so
that email/patch management becomes a bit easier.

The rationale behind most of the ideas below is to get better feedback
to people that email you with the same number of keystrokes as it would
have taken to simply delete the mail/thread and to better organise the
mail you get. I see you're already reading l-k through a newsreader ...

Some ideas:

- a 'patch-tester' that tries to auto-apply patches to a test tree and
if the application fails, flags the message in some way before popping
it in your mailbox eg. [SUCCESSFUL-PATCH] or [FAILED-PATCH] with the
failure as a text attachment to glance at in case you do want to try
to apply it.

- some procmail goo to dump patches that aren't sent as text attachments
or whatever, with a cute autoresponse to the sender giving details
of proper patch submission procedure.

- threads that are deleted or procmailed will trigger an autoresponse to
the sender if your name is in the TO or CC fields ( or the subject
contains [PATCH] ) saying you've nuked the thread and if they really
want a response from you it should be sent under a different name and
provide a pointer to a 'submitting patches to Linus' page. Perhaps
there could be two types nuking emails: 'nuked since I've got 50 million
emails in my inbox' and 'thread is uninteresting, nuked on
[DATE] while reading [MSGID]' ...

- some hotkeys to autoreply to things with canned responses ...  I am
reminded of the 'tick a box' sort of email things but that can be
driven with 1 or 2 keypresses to reply and send the right response:
[ ] You have sent me a patch that does not apply cleanly
[x] Read the codingstyle doc!!! This is awful. Not applied
[ ] This doesn't even compile. Not applied.
[ ] Applied. Thank you.
[ ] Resend diff against latest pre-patch please.
[ ] Resend via the relevant maintainer [link to list] please.
[ ] I can't accept your marriage proposal. I'm already married.
[ ] Send money to [BANK DETAILS]
[ ] Yes
[ ] Never!
etc. ... Some of these might be triggered automatically perhaps eg.
based on the code that a patch touches, the sender etc.

Anyway, just ideas ... there must be something that will make life a
little easier if it was automated and there are probably lots of people
wanting to put a 'I wrote a script that Linus uses' stamp on their CV so
you wouldn't even have to code the stuff yourself :)

Cheers,

--Craig

Re: A modest proposal -- We need a patch penguin

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> On January 30, 2002 11:39 pm, Jesse Pollard wrote:
> > Linus has announced who he accepts patches frin, and who will be doing the
> > 2.0, 2.2, and 2.4 maintenance. It would seem logical to have those 
> > lieutenants announce their maintainers.
> 
> Logical flaw: Marcelo is the maintainer of 2.4, Linus is the maintainer of 
> 2.5, does it make sense for Marcelo to announce the maintainer of usb for
> 2.4?
> 
> It's not as simple as you'd think.  Reason: it's not a tree, it's an
> acyclic graph.  Hopefully.  ;-)

Actually, it does make sense - It still doesn't prevent someone from
announcing a higher level person for a subsystem, or even a person at
the same level.

Announcements shouldn't be rigid, it's up to who the lieutenants will
accept patches from. They can still accept them from outside the announced
lists, though that may increase the amount of effort.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Tue, Jan 29, 2002 at 04:44:12PM -0800, Linus Torvalds wrote:
> I have to admit that personally I've always found the MAINTAINERS file
> more of an irritation than anything else. The first place _I_ tend to look
> personally is actually in the source files themselves (although that may
> be a false statistic - the kind of people I tend to have to look up aren't
> the main maintainers at all, but more single driver people etc).
> 
> It might not be a bad idea to just make that "mention maintainer at the
> top of the file" the common case.

There's one problem with that though - if someone maintains many files,
and his email address changes, you end up with a large patch changing all
those email addresses in every file.

IMHO its far better to have someone's name at the top of each file, and
put the email addresses in the MAINTAINERS file.

People don't change their names often, but email addresses do change.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:19:12AM +0000, Russell King wrote:
> On Tue, Jan 29, 2002 at 04:44:12PM -0800, Linus Torvalds wrote:
> > I have to admit that personally I've always found the MAINTAINERS file
> > more of an irritation than anything else. The first place _I_ tend to look
> > personally is actually in the source files themselves (although that may
> > be a false statistic - the kind of people I tend to have to look up aren't
> > the main maintainers at all, but more single driver people etc).
> > 
> > It might not be a bad idea to just make that "mention maintainer at the
> > top of the file" the common case.
> 
> There's one problem with that though - if someone maintains many files,
> and his email address changes, you end up with a large patch changing all
> those email addresses in every file.
> 
> IMHO its far better to have someone's name at the top of each file, and
> put the email addresses in the MAINTAINERS file.

Also FWIW I go to MAINTAINERS file first, when I construct the CC line
for patches sent to Linus.  Poking around the source is annoying and not
terribly scalable in my experience.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Jacob Luna Lundberg]

On Wed, 30 Jan 2002, Russell King wrote:
> There's one problem with that though - if someone maintains many files,
> and his email address changes, you end up with a large patch changing all
> those email addresses in every file.

Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
to people who make it into MAINTAINERS then?  Of course, someone would
have to maintain the accounts...  ;)

-Jacob

--

Reechani, Sentrosi, Vasi

Re: A modest proposal -- We need a patch penguin

[Russell King]

On Wed, Jan 30, 2002 at 11:55:38AM -0800, Jacob Luna Lundberg wrote:
> 
> On Wed, 30 Jan 2002, Russell King wrote:
> > There's one problem with that though - if someone maintains many files,
> > and his email address changes, you end up with a large patch changing all
> > those email addresses in every file.
> 
> Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> to people who make it into MAINTAINERS then?  Of course, someone would
> have to maintain the accounts...  ;)

Already covered that ground.  Excellent way of having an endless supply
of spam fired at you.

-- 
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Wed, 30 Jan 2002, Jacob Luna Lundberg wrote:

> 
> On Wed, 30 Jan 2002, Russell King wrote:
> > There's one problem with that though - if someone maintains many files,
> > and his email address changes, you end up with a large patch changing all
> > those email addresses in every file.
> 
> Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> to people who make it into MAINTAINERS then?  Of course, someone would
> have to maintain the accounts...  ;)

Just as a talking point, it should be possible to have a daemon scan mail
the lkml for [PATCH] and read the filenames from the patch itself, and do
a file to maintainer lookup followed by a mail. Obviously it would have to
have a human for some cases, but that's not all that bad, at least the
patch program could assign a number and post a list of patches to lkml on
a regular basis.

The hard part is the file to maintainer map, so the program can pick the
best maintainer, and possibly on a regular (daily) basis a single list of
patches to other maintainers: "this patch was sent to XXX bacause most of
the files are hers,but some are yours so you might want to check." And of
course XXX would be told that the patch changed other's files as well.

All patches would be given a number for discussion, after eyeball of the
first 20 patches I saw, I guess that 60-80% could unambiguously go to the
correct maintainer.

I realize this is less complex and wonderful than the schemes proposed,
therefore it might easily actually happen... and it takes no effort except
reading the mail, if the maintainer doesn't care to use the notification
s/he can ignore it, at least the submitter can be sure it was remailed and
to whom.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 30, 2002 10:56 pm, Bill Davidsen wrote:
> On Wed, 30 Jan 2002, Jacob Luna Lundberg wrote:
> > On Wed, 30 Jan 2002, Russell King wrote:
> > > There's one problem with that though - if someone maintains many files,
> > > and his email address changes, you end up with a large patch changing all
> > > those email addresses in every file.
> > 
> > Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org) 
> > to people who make it into MAINTAINERS then?  Of course, someone would
> > have to maintain the accounts...  ;)
> 
> Just as a talking point, it should be possible to have a daemon scan mail
> the lkml for [PATCH] and read the filenames from the patch itself, and do
> a file to maintainer lookup followed by a mail. Obviously it would have to
> have a human for some cases, but that's not all that bad, at least the
> patch program could assign a number and post a list of patches to lkml on
> a regular basis.
> 
> The hard part is the file to maintainer map, so the program can pick the
> best maintainer, and possibly on a regular (daily) basis a single list of
> patches to other maintainers: "this patch was sent to XXX bacause most of
> the files are hers,but some are yours so you might want to check." And of
> course XXX would be told that the patch changed other's files as well.
> 
> All patches would be given a number for discussion, after eyeball of the
> first 20 patches I saw, I guess that 60-80% could unambiguously go to the
> correct maintainer.
> 
> I realize this is less complex and wonderful than the schemes proposed,

Is that bad?

> therefore it might easily actually happen... and it takes no effort except
> reading the mail, if the maintainer doesn't care to use the notification
> s/he can ignore it, at least the submitter can be sure it was remailed and
> to whom.

One (and only one) step ahead of you.  Please have a look at what we're
doing here:

   http://killeri.net/cgi-bin/alias/ezmlm-cgi

And yes, we're already thinking about Russell's concerns with spam.  I
think that issue is under control.

-- 
Daniel

RE: A modest proposal -- We need a patch penguin

[Karl]

>Why not have real fun and give out e-mail@vger.kernel.org (or @kernel.org)
>to people who make it into MAINTAINERS then?  Of course, someone would
>hve to maintain the accounts...  ;)


  As someone who normally just watches I will meekly add, that makes a LOT
of sense. Because the address could be IDE_Subsystem@vger.kernel.org and
then if that person passed on the maintainership, the maintainer patch would
have the correct e mail address and only the name would be out of date. Thus
keeping ease of submission at least.

Karl Tatgenhorst

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 06:50 pm, Linus Torvalds wrote:
> On Tue, 29 Jan 2002, Rob Landley wrote:
>
> > Ah.  So being listed in the maintainers list doesn't mean someone is
> > actually a maintainer it makes sense to forward patches to?
>
> Sure it does.
>
> It just doesn't mean that they should send stuff to _me_.
>
> Did you not understand my point about scalability?

I was asking for clarification.

> I can work with a
> limited number of people, and those people can work with _their_ limited
> number of people etc etc.

I.E. a tree structure.

> The MAINTAINERS file is _not_ a list of people I work with on a daily
> basis. In fact, I don't necessarily even recognize the names of all those
> people.
>
> Let's take an example. Let's say that you had a patch for ppp. You'd send
> the patch to Paul Mackerras. He, in turn, would send his patches to David
> Miller (who knows a hell of a lot better what it's all about than I do).
> And he in turn sends them to me.
>
> They are both maintainers. That doesn't mean that I necessarily work with
> every maintainer directly.

Okay, so there's a tree of maintainers, and some maintainers seem unaware 
that they should be sending their patches to other maintainers rather than 
directly to you?

Does this seem like a valid assessment of at least part of the problem?

> Why? Because having hundreds of people emailing me _obviously_ doesn't
> scale. Never has, never will. It may work over short timeperiods wih lots
> of energy, but it obviously isn't a stable setup.

Well at least we agree on something. :)

> 			Linus

Rob

Re: A modest proposal -- We need a patch penguin

[Henning P. Schmiedehausen]

Linus Torvalds <torvalds@transmeta.com> writes:

>> Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
>> Boldt...

>NONE of those are in the ten-twenty people group.

>How many people do you think fits in a small group? Hint. It sure isn't
>all 300 on the maintainers list.

Can you tell us, who the people in your peer group are? That would
make sending patches much easier.

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: A modest proposal -- We need a patch penguin

[Daniel Egger]

Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:

> Or look at USB: I get the USB patches from Greg, and he gets them from
> various different people. Johannes Erdfelt is the maintainer for uhci.c,
> and he sends them to Greg, not to me.

What about creating a small document that states who's the correct
recipient for a subsystem? This would prevent dotzends of questions
like "Where do I send my patches?" and turn them into a RTFF.
 
-- 
Servus,
       Daniel

Re: A modest proposal -- We need a patch penguin

[Andre Hedrick]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> 
> On Tue, 29 Jan 2002, Rob Landley wrote:
> > > >
> > > > Then why not give the subsystem maintainers patch permissions on your
> > > > tree. Sort of like committers.  The problem people have is that you're
> > > > dropping patches from those ten-twenty people you trust.
> > >
> > > No. Ask them, and they will (I bet) pretty uniformly tell you that I'm
> > > _not_ dropping their patches (although I'm sometimes critical of them,
> > > and will tell them that they do not get applied).
> >
> > Andre Hedrick, Eric Raymond, Rik van Riel, Michael Elizabeth Chastain, Axel
> > Boldt...
> 
> NONE of those are in the ten-twenty people group.

Well it is nice to know the facts now.  How about having just a little
more courage and publish your offical tree of who is "IN" and "OUT" so
folks can decide for themselves.  It is an honor to be in the lesser class
of folks who care more but less is accepted.

As for clean coding, you have to make a mess to in one part of a room by
pushing the contents to a corner and weeding out the useful.  Since you
have always stated public/private/in-person the sensitve nature of the
changes to the low-level storage drivers, those who have tried to promote
regression testing of layers and isolation points, have been ignored,
laughed, scorned, etc ...

Regardless if many people see the need and other are tired of being the
garbage collector of blame, when valid and proper solutions have been
presented in the past, specifically "FILE SYSTEM CORRUPTION".

Only after a few cases of pointing out flaws and failures in darwinisms
development, few became ignored.  Noting the total freedom you take to
blanket blame folks for issues which they are not responsible for
creating, an easy target allows one to ignore ultimate responsiblity.

> How many people do you think fits in a small group? Hint. It sure isn't
> all 300 on the maintainers list.

Not at all but again why not draft the offical "Linus IN/OUT Tree".

> > Ah.  So being listed in the maintainers list doesn't mean someone is actually
> > a maintainer it makes sense to forward patches to?
> 
> Sure it does.
> 
> It just doesn't mean that they should send stuff to _me_.

Some have gotten a strong grasp of the obvious nature of this point first
hand, regardless ...

Maybe you should consider taken an agreed code base migration change when
it is suggested and agreed upon, instead of ignoring comments and
suggestions for changes.  Just buy design when I get done w/ ATA and maybe
ATAPI so that it is clean and obvious to the reader, I would consider
tearing into the ABANDONED SCSI CORE.  However, I expect to find the same
uphill battle and may do it for the joy of exactness, but rediscover the
same problem of a global design change.

Something you need to understand, and I honestly expect you to ignore,
is a responsible an proper OS protects the hardware from accepting bad
command operations.  Given "LINUX" is a UNIX environment, that does not
give it the right to ignore comman sense.  However, to get along in your
world, all have accepted users are allowed and expect to have no
safeguards.  So when a problem comes up and it is ugly, it gets batted
down because the solution is wrong, and then quietly adopted without
acknowledgement the very solution proposed.

regards,

--andre

Re: A modest proposal -- We need a patch penguin

[Stuart Young]

At 02:13 PM 30/01/02 +0100, Daniel Egger wrote:
>Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:
>
> > Or look at USB: I get the USB patches from Greg, and he gets them from
> > various different people. Johannes Erdfelt is the maintainer for uhci.c,
> > and he sends them to Greg, not to me.
>
>What about creating a small document that states who's the correct
>recipient for a subsystem? This would prevent dotzends of questions
>like "Where do I send my patches?" and turn them into a RTFF.

A reworking of MAINTAINERS could be beneficial and help achieve this.

Linus mentioned that he prefers to look at the code to see who to talk to. 
Others have mentioned this may be nice, but makes it hard to get some sort 
of overall view, plus since programmers can be inconsistent in stuff like 
this, it may not always happen. How about we turn the problem upside down, 
and figure out how to get the code easily referenced in MAINTAINERS?

What I'm thinking is that we could add (multiple?) lines into MAINTAINERS 
that specify the actual FILES in the kernel (in reference to linux/) that 
someone works on or maintains. We don't have to list every file (wildcards, 
regex's, etc, can work too), plus you can list the maintainers of various 
areas of the code (such as generic maintainers of all the files under a 
part of the kernel file tree) by just listing what directories they 
control. Something so that it's dead simple to extract who maintains this file.

Here's a possible example:

Say I'm looking at the SiS/Trident Audio Driver, and I have a patch I want 
to send to a maintainer. The file I'm working on is:

linux/drivers/sound/trident.*

If I could easily search MAINTAINERS for who maintains this file, I'm made. 
If I can't find that, I start trimming the search (to say 
linux/drivers/sound/, which would be the sound maintainer).

If we say add an F: field to maintainers (at the end of the maintainers 
record), you can easily do things like...

grep -B 10 "F: linux/drivers/sound/trident" /usr/src/linux/MAINTAINERS

...and get some sort of results (-B is "before context, which displays 
lines before the found line, and is quite useful in this sort of situation) 
that help. This is just a quick and dirty example, and I'm sure someone 
could easily write a small script that could parse the output better, do 
things like automatically cut the search back till it finds a match, etc.

This could also be used to figure out a tree of who does what, which is 
probably not a bad idea.

Just an idea of course. *grin*


Stuart Young - sgy@amc.com.au
(aka Cefiar) - cefiar1@optushome.com.au

[All opinions expressed in the above message are my]
[own and not necessarily the views of my employer..]

Re: A modest proposal -- We need a patch penguin

[David Lang]

Eric posted a proposal to split the maintainers info across all
directories/files sometime last year and got shouted down to the point of
not mentioning it anymore (which as we know takes quite a bit for him :-)

does someone want to dig up that proposal and recirculate it rather then
starting from scratch again?

David Lang


 On Thu, 31 Jan 2002, Stuart Young wrote:

> Date: Thu, 31 Jan 2002 12:16:29 +1100
> From: Stuart Young <sgy@amc.com.au>
> To: linux-kernel@vger.kernel.org
> Cc: Daniel Egger <degger@fhm.edu>, Linus Torvalds <torvalds@transmeta.com>
> Subject: Re: A modest proposal -- We need a patch penguin
>
> At 02:13 PM 30/01/02 +0100, Daniel Egger wrote:
> >Am Mit, 2002-01-30 um 00.50 schrieb Linus Torvalds:
> >
> > > Or look at USB: I get the USB patches from Greg, and he gets them from
> > > various different people. Johannes Erdfelt is the maintainer for uhci.c,
> > > and he sends them to Greg, not to me.
> >
> >What about creating a small document that states who's the correct
> >recipient for a subsystem? This would prevent dotzends of questions
> >like "Where do I send my patches?" and turn them into a RTFF.
>
> A reworking of MAINTAINERS could be beneficial and help achieve this.
>
> Linus mentioned that he prefers to look at the code to see who to talk to.
> Others have mentioned this may be nice, but makes it hard to get some sort
> of overall view, plus since programmers can be inconsistent in stuff like
> this, it may not always happen. How about we turn the problem upside down,
> and figure out how to get the code easily referenced in MAINTAINERS?
>
> What I'm thinking is that we could add (multiple?) lines into MAINTAINERS
> that specify the actual FILES in the kernel (in reference to linux/) that
> someone works on or maintains. We don't have to list every file (wildcards,
> regex's, etc, can work too), plus you can list the maintainers of various
> areas of the code (such as generic maintainers of all the files under a
> part of the kernel file tree) by just listing what directories they
> control. Something so that it's dead simple to extract who maintains this file.
>
> Here's a possible example:
>
> Say I'm looking at the SiS/Trident Audio Driver, and I have a patch I want
> to send to a maintainer. The file I'm working on is:
>
> linux/drivers/sound/trident.*
>
> If I could easily search MAINTAINERS for who maintains this file, I'm made.
> If I can't find that, I start trimming the search (to say
> linux/drivers/sound/, which would be the sound maintainer).
>
> If we say add an F: field to maintainers (at the end of the maintainers
> record), you can easily do things like...
>
> grep -B 10 "F: linux/drivers/sound/trident" /usr/src/linux/MAINTAINERS
>
> ...and get some sort of results (-B is "before context, which displays
> lines before the found line, and is quite useful in this sort of situation)
> that help. This is just a quick and dirty example, and I'm sure someone
> could easily write a small script that could parse the output better, do
> things like automatically cut the search back till it finds a match, etc.
>
> This could also be used to figure out a tree of who does what, which is
> probably not a bad idea.
>
> Just an idea of course. *grin*
>
>
> Stuart Young - sgy@amc.com.au
> (aka Cefiar) - cefiar1@optushome.com.au
>
> [All opinions expressed in the above message are my]
> [own and not necessarily the views of my employer..]
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> > Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> > "good taste" for a long time. But it's not always a long process - some
> > of you may remember Bill Hawes, for example, who came out of nowhere
> > rather quickly.
> 
> So listed "maintainers" may need to forward patches to these people, and get 
> them to sign off on them, in order to get their patches at least reviewed for 
> inclusion into your tree?

Count me out of that job. If you want something in 2.5 don't bug me. I
simply don't care

Alan

Re: A modest proposal -- We need a patch penguin

[Shawn]

On 01/29, Alan Cox said something like:
> > > Viro, David Miller, Greg KH, Andrew Morton etc. They've shown what I call
> > > "good taste" for a long time. But it's not always a long process - some
> > > of you may remember Bill Hawes, for example, who came out of nowhere
> > > rather quickly.
> > 
> > So listed "maintainers" may need to forward patches to these people, and get 
> > them to sign off on them, in order to get their patches at least reviewed for 
> > inclusion into your tree?
> 
> Count me out of that job. If you want something in 2.5 don't bug me. I
> simply don't care

Now this scares the piss out of me...

--
Shawn Leas
core@enodev.com

I bought a self learning record to learn spanish, I turned it on and
went to sleep, the record got stuck, the next day I could only stutter in
spanish.
						-- Stephen Wright

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Tuesday 29 January 2002 09:30 am, Skip Ford wrote:
> Linus Torvalds wrote:
> [snip]
>
> > A word of warning: good maintainers are hard to find.  Getting more of
> > them helps, but at some point it can actually be more useful to help the
> > _existing_ ones.  I've got about ten-twenty people I really trust, and
>
> Then why not give the subsystem maintainers patch permissions on your tree.
> Sort of like committers.  The problem people have is that you're dropping
> patches from those ten-twenty people you trust.

I understand why he doesn't do that: he can't function if the code is 
changing under him in ways that suprise him.  (Especially he can't function 
as architect without doing code inspection.)

Linus DOES apply larger patches from maintainers with less scrutiny, but 
there still IS scrutiny of each patch.  (At the very least, checking which 
files it touches.)

> Each subsystem maintainer should handle patches to that subsystem, and
> you should remove your own patch permissions for only those subsystems.
> You could get involved with only changes in direction that affect more
> than one subsystem.

Linus also reserves the right to mess with a maintainer's code and force a 
patch back down the tree for them to resync with.  He just did it with the 
help files (after a "private flamewar").  In this case, the maintainer was 
caught by suprise, claiming to be unaware that Linus expected him to make 
that change, which just seems to be one more example of a lack of 
communication between Linus and a maintainer.  This time, instead of Linus 
not getting the maintainer's message (patch), the maintainer doesn't get 
linus's message ("Go do this, in this order".)  So we've got examples of 
messages getting dropped in both directions, making the maintainer look 
inattentive when he claims otherwise, and making Linus look inattentive when 
HE claims otherwise...

Rob

Re: A modest proposal -- We need a patch penguin

[Bill Davidsen]

On Tue, 29 Jan 2002, Linus Torvalds wrote:

> In short, if you have areas or patches that you feel have had problems,
> ask yourself _why_ those areas have problems. 

Easy, because the patch process has bogged down due to bad design and has
failed to scale. And you simply refuse to believe that there is a problem.

> A word of warning: good maintainers are hard to find.  Getting more of
> them helps, but at some point it can actually be more useful to help the
> _existing_ ones.  I've got about ten-twenty people I really trust, and
> quite frankly, the way people work is hardcoded in our DNA.  Nobody
> "really trusts" hundreds of people.  The way to make these things scale
> out more is to increase the network of trust not by trying to push it on
> me, but by making it more of a _network_, not a star-topology around me. 

The problem is that you don't trust ANYONE. There is no reason why you
should be looking at small obvious patches to bugs (note bugs, not
enhancements). In the last batch of messages I see agreement from Alan Cox
and Eric Raymond that things are backed up. I see reiser filesystem
patches, from the original developer, labeled "third try." Quite bluntly
he is a hell of a lot better qualified to do bug fixes in that area than
you are.
 
> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

I'm sure this will be ignored, but here's what should happen, and will,
just maybe not in the Linus kernel series.

BUGS:

There should be a place to send bugs. Every bug should be acknowleged so
mailbots to keep resending will not be needed. People are setting this up
now, the question is not if it will be done this way, but if it will be
done through or around you.

Each bug should be eyeballed by someone with a clue just to see that the
report states a problem, a way to verify the problem, and doesn't grossly
misstate the intended behaviour. Then someone at the maintainer level
would look at the patch, check it carefully, and reject formally or apply.

Here's the heresy: every bug patch should be promoted to the current
kernel unless it is rejected for reason. This will get fixes in, but more
importantly will force people to look at the damn things...

Reasons to drop a patch:
1 - no bug, this process is for bugs, no news features or improvements.
Offhand I see bugs when you have a crash (oops), a failure to compile, or
filesystem corruption. Someone may want to add drivers totally not
working, but I didn't say it.

2 - no bug, you misunderstand the behaviour, failed to provide a test case
or persuasive argument (you are scanning from 0 to N instead of 0 to N-1
type stuff, you are locking one lock and unlocking another, etc).

3 - no fix, the solution doesn't work.

4 - not readable or understandable.

5 - changes some global part of the kernel and would or could impact other
things.

Note that while "there's a better way" can cause an add to the to-do list,
I do not intend to have the users suffer an actual bug if there is a
viable solution.

To all the people who say that "if you don't like the way Linus does
things write your own kernel," I say that's what is happening with the
-aa, -ac, etc lines. That's why I'm typing this on something called
2.4.17-jl15-ll, because I'm in need of a kernel which runs on a small
machine and doesn't piss me off with lousy response to complex stuff like
echoing what I type.

Someone mentioned the army. Note that hte general doesn't decide where to
dig the foxhole. It's about delegation, and while I doubt Linus read this
far, that's how things scale beyond what one person can do.

Fork is not just a system call, I'd hate to see FreeLinux, NetLinux,
OpenLinux, Linux386, along BSD lines, but while people will wait for
enhancements, I don't find that they are nearly so willing to wait for
fixes. Before we have vendor versions which actually start to differ in
syscall behaviour, let's get a handle on this. It's time for the
maintenence to evolve just as the kernel has, and become something bigger
and better. We need "patch-threads" soon, lest we thrash ourself silly.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Re: A modest proposal -- We need a patch penguin

[Hans Reiser]

Bill Davidsen wrote:

>
>The problem is that you don't trust ANYONE. There is no reason why you
>should be looking at small obvious patches to bugs (note bugs, not
>enhancements). In the last batch of messages I see agreement from Alan Cox
>and Eric Raymond that things are backed up. I see reiser filesystem
>patches, from the original developer, labeled "third try." Quite bluntly
>he is a hell of a lot better qualified to do bug fixes in that area than
>you are.
>
Whoa, in this case it is Marcelo, and his responsiveness is on the whole 
simply superb.  Right complaint, wrong example.  Marcelo also does a 
great job of having good reasons for rejecting patches on the rare 
occasions when he does reject them.  Is the real problem that you are 
comparing guys who have neither a day job nor a family to someone who 
has both?  I think maybe it is.  Linus, do you have other work 
responsibilities besides Linux?  With funding provided , could you work 
solely on Linux?

I remember when I used to hold a day job while working on ReiserFS and 
trying to manage a team of full-time programmers by email.  It sucked. 
 There is crap in ReiserFS that went in only because I didn't have time 
to fight it, and Reiser4 is a very much higher level of quality mainly 
because I can invest a lot more time into it.

Hans

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Tue, 29 Jan 2002 03:23:11 +0000 (UTC)
torvalds@transmeta.com (Linus Torvalds) wrote:

> One "patch penguin" scales no better than I do. In fact, I will claim
> that most of them scale a whole lot worse. 

I could hardly disagree.

> In short: don't try to come up with a "patch penguin".  Instead try to
> help existing maintainers, or maybe help grow new ones. THAT is the way
> to scalability.

Ok, I won't come up with anything. :)

I just heard a bell ringing, with Rob's message. It's still a faint ring, but
the debate spurred on l-k has shown that the bell is indeed ringing
somewhere. Many opinions were quite harsh on you, some even ungrateful; I'm
sorry if this caused any trouble.

What I see (from the viewpoint of some random user giving a try to test
kernels just for the fun of doing it) is not a problem with subsys
maintainers. I don't even _see_ them, from my pov. I trust them because you
do, and that's enough for me.

But I start to feel the need for someone to throw me an "unified development
tree". Like -ac was in 2.4: some source you trust from which pulling a kernel
to be tested. I need it, or better I prefer it over a forest of cross-patched
subtrees where I can find kernels bleedingly optimized in some area and
lacking trivial fixes in others.

A matter of convenience, actually. To which you can just answer "go away,
lazy scum". And probably will! :)

But also convenience for you.

You says, righteously, that you want tested patches. OTOH, to have them
tested, peer review would want to have them available somewhere to be tested,
for us crunching drones. Eventually in a tree where I can easily find all the
patches which are subject to scrutiny.

So we both could take advantage of what you call a "layer": someone (better:
more than someone - one man doesn't scale) who gathers stuff and readies ONE
unified, test-time-ready kernel, who has all the reviewing eyeballs, who goes
at you with the results of the tests, re-chunkifying(tm) the patches, and
letting you discard what you don't like and keep the (tested) things you see
fitting.

In the upper spheres, there could be a split of workload between who packages
the kernel to be thrown to the test-dogs (AC in the past, DJ now, ore than
one man in the future?), and who actually pioneers code and steers the kernel
after the packager gathers enough feedback from testers (that's you, wow! :)
).

Of course there has to be trust at that level, and I agree with you that
without trust with few, very selected people, you can't go ahead blindly
gathering debris from everyone.

In short: I think there are too many concurrent, overlapping development
trees, with a web of crosspatches that are honestly difficult to follow from
my "download, make, lilo, reboot, report" viewpoint. A fragmentation in the
to-be-tested code. A single "reference development" tree would be most
welcome.

I didn't intend to ask it to *you*. Probably natural evolution of kernel
development will pop out a good solution anyway.

The "patch penguin" as solution is just an idea. Bad as you want, but it
comes from a sensation Rob had. And I feel it too. And maybe others, less
silly than me.

The idea is rejectable, of course. The sensation, a bit less.

Have fun,

-- Francesco Munda

Re: A modest proposal -- We need a patch penguin

[Jeff Garzik]

On Wed, Jan 30, 2002 at 09:03:55AM +0100, Francesco Munda wrote:
> In short: I think there are too many concurrent, overlapping development
> trees, with a web of crosspatches that are honestly difficult to follow from
> my "download, make, lilo, reboot, report" viewpoint. A fragmentation in the
> to-be-tested code. A single "reference development" tree would be most
> welcome.

2.5.x is the reference development tree.

People building outside-the-kernel patchkits is indeed useful for
end users to conveniently test a bunch of patches... but attempting
to merge various concurrent trees would be murder on code quality.
Do we want XFS ACLs in the reference development tree, just to yank
or modify syscalls before the final revision?  No.  Therefore, XFS
needs to be in its own tree until its ready.

Notice the gcc team will create a branch for development, even
during a development cycle (ie. no freeze at all), just to ensure
that large or complex changes do not destabilize the tree until they
are really ready.

Finally, even in a devel cycle kernel hackers need some semblance of a
sane tree in order to do their own development.  If tons of hackers
are blazing away committing all sorts of code, you have nothing but
a tree of mass confusion, in a nice package for users to test.

	Jeff

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Wed, 30 Jan 2002 03:39:06 -0500
Jeff Garzik <garzik@havoc.gtf.org> wrote:

> 2.5.x is the reference development tree.
> 
> [...]

Ok, I buy it.

The amount of messages, and thoughts on the subject, fairly satisfied me. I
wanted to see a reaction, and I indeed saw one. And I like what I've seen:
the point Rob brought up was not dismissed lightheartedly, but instead
spurred a discussion on many aspects of kernel development. Adoption of BK,
maintainers hierarchies, trust relationships between demigods, patch handling
clarifications, you name it...

Who can disagree that talking about these topics, finding solutions (to
existing and future problems), and in general doing reality checks is a good
thing?

And the "small patches maintainer" is a figure that should be there. World
will keep spinning if nobody takes the buck, but having one such maintainer
has been generally greeted as a good idea.

We can have many trees to play with to develop different changes, but that
trivial one-line fixup should be in every tree, not just in "my favorite
maintainer's" tree. Risking a wipeout while testing a new scheduler/VM/dev
handler is one thing. Risking a wipeout for a bug that's patched in
everyone's else tree but not in my own, is another. :)

Trivial fixes and largish changes should be handled in different ways, this
thread seems to have summed up. Maybe by different people, to split some
workload more evenly.

Many interesting points were touched. Let's wait and see.

-- Francesco

PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

The attached 3 patches serve the following purposes:

1. Make the i810_audio.c driver working again. Well it's tested on my 
private hardware.

2. Make the bttv driver work again. I know that there is a GREAT REWRITE 
of this
  driver underway, but still it's a bit annoying to miss the TV until 
that's ready.

3. This is just fixing an obvious mistake from the final release and let's
  the whole compile at all on IA32.



[-- Attachment #2: i810_audio.patch --]
[-- Type: text/plain, Size: 1616 bytes --]

diff -ur linux/drivers/sound/i810_audio.c linux-new/drivers/sound/i810_audio.c
--- linux/drivers/sound/i810_audio.c	Tue Feb 12 18:32:55 2002
+++ linux-new/drivers/sound/i810_audio.c	Mon Feb 11 01:34:36 2002
@@ -939,7 +939,7 @@
 	  
 		for(i=0;i<dmabuf->numfrag;i++)
 		{
-			sg->busaddr=virt_to_bus(dmabuf->rawbuf+dmabuf->fragsize*i);
+			sg->busaddr=virt_to_phys(dmabuf->rawbuf+dmabuf->fragsize*i);
 			// the card will always be doing 16bit stereo
 			sg->control=dmabuf->fragsamples;
 			if(state->card->pci_id == PCI_DEVICE_ID_SI_7012)
@@ -954,7 +954,7 @@
 		}
 		spin_lock_irqsave(&state->card->lock, flags);
 		outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
-		outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
+		outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
 		outb(0, state->card->iobase+c->port+OFF_CIV);
 		outb(0, state->card->iobase+c->port+OFF_LVI);
 
@@ -1669,7 +1669,7 @@
 	if (size > (PAGE_SIZE << dmabuf->buforder))
 		goto out;
 	ret = -EAGAIN;
-	if (remap_page_range(vma->vm_start, virt_to_phys(dmabuf->rawbuf),
+	if (remap_page_range(vma, vma->vm_start, virt_to_phys(dmabuf->rawbuf),
 			     size, vma->vm_page_prot))
 		goto out;
 	dmabuf->mapped = 1;
@@ -1722,7 +1722,7 @@
 		}
 		if (c != NULL) {
 			outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
-			outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
+			outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
 			outb(0, state->card->iobase+c->port+OFF_CIV);
 			outb(0, state->card->iobase+c->port+OFF_LVI);
 		}

[-- Attachment #3: bttv.patch --]
[-- Type: text/plain, Size: 10013 bytes --]

diff -ur linux/drivers/media/video/bttv-driver.c linux-new/drivers/media/video/bttv-driver.c
--- linux/drivers/media/video/bttv-driver.c	Tue Feb 12 18:32:53 2002
+++ linux-new/drivers/media/video/bttv-driver.c	Mon Feb 11 01:42:05 2002
@@ -166,23 +166,23 @@
 	return ret;
 }
 
-static inline unsigned long uvirt_to_bus(unsigned long adr) 
+static inline unsigned long uvirt_to_phys(unsigned long adr)
 {
         unsigned long kva, ret;
 
         kva = uvirt_to_kva(pgd_offset(current->mm, adr), adr);
-	ret = virt_to_bus((void *)kva);
+	ret = virt_to_phys((void *)kva);
         MDEBUG(printk("uv2b(%lx-->%lx)", adr, ret));
         return ret;
 }
 
-static inline unsigned long kvirt_to_bus(unsigned long adr) 
+static inline unsigned long kvirt_to_phys(unsigned long adr)
 {
         unsigned long va, kva, ret;
 
         va = VMALLOC_VMADDR(adr);
         kva = uvirt_to_kva(pgd_offset_k(va), va);
-	ret = virt_to_bus((void *)kva);
+	ret = virt_to_phys((void *)kva);
         MDEBUG(printk("kv2b(%lx-->%lx)", adr, ret));
         return ret;
 }
@@ -530,29 +530,29 @@
   
 	if (bttv_debug > 1)
 		printk("bttv%d: vbi1: po=%08lx pe=%08lx\n",
-		       btv->nr,virt_to_bus(po), virt_to_bus(pe));
+		       btv->nr,virt_to_phys(po), virt_to_phys(pe));
         
 	*(po++)=cpu_to_le32(BT848_RISC_SYNC|BT848_FIFO_STATUS_FM1); *(po++)=0;
 	for (i=0; i<VBI_MAXLINES; i++) 
 	{
 		*(po++)=cpu_to_le32(VBI_RISC);
-		*(po++)=cpu_to_le32(kvirt_to_bus((unsigned long)btv->vbibuf+i*2048));
+		*(po++)=cpu_to_le32(kvirt_to_phys((unsigned long)btv->vbibuf+i*2048));
 	}
 	*(po++)=cpu_to_le32(BT848_RISC_JUMP);
-	*(po++)=cpu_to_le32(virt_to_bus(btv->risc_jmp+4));
+	*(po++)=cpu_to_le32(virt_to_phys(btv->risc_jmp+4));
 
 	*(pe++)=cpu_to_le32(BT848_RISC_SYNC|BT848_FIFO_STATUS_FM1); *(pe++)=0;
 	for (i=VBI_MAXLINES; i<VBI_MAXLINES*2; i++) 
 	{
 		*(pe++)=cpu_to_le32(VBI_RISC);
-		*(pe++)=cpu_to_le32(kvirt_to_bus((unsigned long)btv->vbibuf+i*2048));
+		*(pe++)=cpu_to_le32(kvirt_to_phys((unsigned long)btv->vbibuf+i*2048));
 	}
 	*(pe++)=cpu_to_le32(BT848_RISC_JUMP|BT848_RISC_IRQ|(0x01<<16));
-	*(pe++)=cpu_to_le32(virt_to_bus(btv->risc_jmp+10));
+	*(pe++)=cpu_to_le32(virt_to_phys(btv->risc_jmp+10));
 
 	if (bttv_debug > 1)
 		printk("bttv%d: vbi2: po=%08lx pe=%08lx\n",
-		       btv->nr,virt_to_bus(po), virt_to_bus(pe));
+		       btv->nr,virt_to_phys(po), virt_to_phys(pe));
 }
 
 static int fmtbppx2[16] = {
@@ -599,9 +599,9 @@
 	for (line=0; line < 640; line++)
 	{
                 *(ro++)=cpu_to_le32(BT848_RISC_WRITE|bpl|BT848_RISC_SOL|BT848_RISC_EOL);
-                *(ro++)=cpu_to_le32(kvirt_to_bus(vadr));
+                *(ro++)=cpu_to_le32(kvirt_to_phys(vadr));
                 *(re++)=cpu_to_le32(BT848_RISC_WRITE|bpl|BT848_RISC_SOL|BT848_RISC_EOL);
-                *(re++)=cpu_to_le32(kvirt_to_bus(vadr+gbufsize/2));
+                *(re++)=cpu_to_le32(kvirt_to_phys(vadr+gbufsize/2));
                 vadr+=bpl;
 	}
 	
@@ -629,7 +629,7 @@
 
 	if (bttv_debug > 1)
 		printk("bttv%d: prisc1: ro=%08lx re=%08lx\n",
-		       btv->nr,virt_to_bus(ro), virt_to_bus(re));
+		       btv->nr,virt_to_phys(ro), virt_to_phys(re));
 
 	switch(fmt)
 	{
@@ -705,13 +705,13 @@
 		 
 		 *((*rp)++)=cpu_to_le32(rcmd|bl);
 		 *((*rp)++)=cpu_to_le32(blcb|(blcr<<16));
-		 *((*rp)++)=cpu_to_le32(kvirt_to_bus(vadr));
+		 *((*rp)++)=cpu_to_le32(kvirt_to_phys(vadr));
 		 vadr+=bl;
 		 if((rcmd&(15<<28))==BT848_RISC_WRITE123)
 		 {
-		 	*((*rp)++)=cpu_to_le32(kvirt_to_bus(cbadr));
+		 	*((*rp)++)=cpu_to_le32(kvirt_to_phys(cbadr));
 		 	cbadr+=blcb;
-		 	*((*rp)++)=cpu_to_le32(kvirt_to_bus(cradr));
+		 	*((*rp)++)=cpu_to_le32(kvirt_to_phys(cradr));
 		 	cradr+=blcr;
 		 }
 		 
@@ -726,7 +726,7 @@
 	
 	if (bttv_debug > 1)
 		printk("bttv%d: prisc2: ro=%08lx re=%08lx\n",
-		       btv->nr,virt_to_bus(ro), virt_to_bus(re));
+		       btv->nr,virt_to_phys(ro), virt_to_phys(re));
 
 	return 0;
 }
@@ -751,7 +751,7 @@
 
 	if (bttv_debug > 1)
 		printk("bttv%d: vrisc1: ro=%08lx re=%08lx\n",
-		       btv->nr,virt_to_bus(ro), virt_to_bus(re));
+		       btv->nr,virt_to_phys(ro), virt_to_phys(re));
 	
 	inter = (height>tvnorms[btv->win.norm].sheight/2) ? 1 : 0;
 	bpl=width*fmtbppx2[palette2fmt[palette]&0xf]/2;
@@ -773,25 +773,25 @@
                 {
 		        *((*rp)++)=cpu_to_le32(BT848_RISC_WRITE|BT848_RISC_SOL|
 			        BT848_RISC_EOL|bpl); 
-			*((*rp)++)=cpu_to_le32(kvirt_to_bus(vadr));
+			*((*rp)++)=cpu_to_le32(kvirt_to_phys(vadr));
 			vadr+=bpl;
 		}
 		else
 		{
 		        todo=bpl;
 		        *((*rp)++)=cpu_to_le32(BT848_RISC_WRITE|BT848_RISC_SOL|bl);
-			*((*rp)++)=cpu_to_le32(kvirt_to_bus(vadr));
+			*((*rp)++)=cpu_to_le32(kvirt_to_phys(vadr));
 			vadr+=bl;
 			todo-=bl;
 			while (todo>PAGE_SIZE)
 			{
 			        *((*rp)++)=cpu_to_le32(BT848_RISC_WRITE|PAGE_SIZE);
-				*((*rp)++)=cpu_to_le32(kvirt_to_bus(vadr));
+				*((*rp)++)=cpu_to_le32(kvirt_to_phys(vadr));
 				vadr+=PAGE_SIZE;
 				todo-=PAGE_SIZE;
 			}
 			*((*rp)++)=cpu_to_le32(BT848_RISC_WRITE|BT848_RISC_EOL|todo);
-			*((*rp)++)=cpu_to_le32(kvirt_to_bus(vadr));
+			*((*rp)++)=cpu_to_le32(kvirt_to_phys(vadr));
 			vadr+=todo;
 		}
 	}
@@ -803,7 +803,7 @@
 
 	if (bttv_debug > 1)
 		printk("bttv%d: vrisc2: ro=%08lx re=%08lx\n",
-		       btv->nr,virt_to_bus(ro), virt_to_bus(re));
+		       btv->nr,virt_to_phys(ro), virt_to_phys(re));
 	
 	return 0;
 }
@@ -896,7 +896,7 @@
 
 	if (bttv_debug)
 		printk("bttv%d: clip: ro=%08lx re=%08lx\n",
-		       btv->nr,virt_to_bus(ro), virt_to_bus(re));
+		       btv->nr,virt_to_phys(ro), virt_to_phys(re));
 
 	if ((clipmap=vmalloc(VIDEO_CLIPMAP_SIZE))==NULL) {
 		/* can't clip, don't generate any risc code */
@@ -1213,8 +1213,8 @@
 	btv->gbuf[mp->frame].fmt     = palette2fmt[mp->format];
 	btv->gbuf[mp->frame].width   = mp->width;
 	btv->gbuf[mp->frame].height  = mp->height;
-	btv->gbuf[mp->frame].ro      = virt_to_bus(ro);
-	btv->gbuf[mp->frame].re      = virt_to_bus(re);
+	btv->gbuf[mp->frame].ro      = virt_to_phys(ro);
+	btv->gbuf[mp->frame].re      = virt_to_phys(re);
 
 #if 1
 	if (mp->height <= tvnorms[btv->win.norm].sheight/2 &&
@@ -1341,7 +1341,7 @@
 	btwrite(0xfffffUL, BT848_INT_STAT);
 	btand(~15, BT848_GPIO_DMA_CTL);
 	btwrite(0, BT848_SRESET);
-	btwrite(virt_to_bus(btv->risc_jmp+2),
+	btwrite(virt_to_phys(btv->risc_jmp+2),
 		BT848_RISC_STRT_ADD);
 
 	/* enforce pll reprogramming */
@@ -2371,7 +2371,7 @@
 
 	if (bttv_debug > 1)
 		printk("bttv%d: set_risc_jmp %08lx:",
-		       btv->nr,virt_to_bus(btv->risc_jmp));
+		       btv->nr,virt_to_phys(btv->risc_jmp));
 
 	/* Sync to start of odd field */
 	btv->risc_jmp[0]=cpu_to_le32(BT848_RISC_SYNC|BT848_RISC_RESYNC
@@ -2382,12 +2382,12 @@
 	btv->risc_jmp[2]=cpu_to_le32(BT848_RISC_JUMP|(0xd<<20));
 	if (flags&8) {
 		if (bttv_debug > 1)
-			printk(" ev=%08lx",virt_to_bus(btv->vbi_odd));
-		btv->risc_jmp[3]=cpu_to_le32(virt_to_bus(btv->vbi_odd));
+			printk(" ev=%08lx",virt_to_phys(btv->vbi_odd));
+		btv->risc_jmp[3]=cpu_to_le32(virt_to_phys(btv->vbi_odd));
 	} else {
 		if (bttv_debug > 1)
 			printk(" -----------");
-		btv->risc_jmp[3]=cpu_to_le32(virt_to_bus(btv->risc_jmp+4));
+		btv->risc_jmp[3]=cpu_to_le32(virt_to_phys(btv->risc_jmp+4));
 	}
 
         /* Jump to odd sub */
@@ -2400,12 +2400,12 @@
 	} else if ((flags&2) &&
 		   (!btv->win.interlace || 0 == btv->risc_cap_even)) {
 		if (bttv_debug > 1)
-			printk(" eo=%08lx",virt_to_bus(btv->risc_scr_odd));
-		btv->risc_jmp[5]=cpu_to_le32(virt_to_bus(btv->risc_scr_odd));
+			printk(" eo=%08lx",virt_to_phys(btv->risc_scr_odd));
+		btv->risc_jmp[5]=cpu_to_le32(virt_to_phys(btv->risc_scr_odd));
 	} else {
 		if (bttv_debug > 1)
 			printk(" -----------");
-		btv->risc_jmp[5]=cpu_to_le32(virt_to_bus(btv->risc_jmp+6));
+		btv->risc_jmp[5]=cpu_to_le32(virt_to_phys(btv->risc_jmp+6));
 	}
 
 
@@ -2418,12 +2418,12 @@
 	btv->risc_jmp[8]=cpu_to_le32(BT848_RISC_JUMP);
 	if (flags&4) {
 		if (bttv_debug > 1)
-			printk(" ov=%08lx",virt_to_bus(btv->vbi_even));
-		btv->risc_jmp[9]=cpu_to_le32(virt_to_bus(btv->vbi_even));
+			printk(" ov=%08lx",virt_to_phys(btv->vbi_even));
+		btv->risc_jmp[9]=cpu_to_le32(virt_to_phys(btv->vbi_even));
 	} else {
 		if (bttv_debug > 1)
 			printk(" -----------");
-		btv->risc_jmp[9]=cpu_to_le32(virt_to_bus(btv->risc_jmp+10));
+		btv->risc_jmp[9]=cpu_to_le32(virt_to_phys(btv->risc_jmp+10));
 	}
 
 	/* Jump to even sub */
@@ -2436,12 +2436,12 @@
 	} else if ((flags&1) &&
 		   btv->win.interlace) {
 		if (bttv_debug > 1)
-			printk(" oo=%08lx",virt_to_bus(btv->risc_scr_even));
-		btv->risc_jmp[11]=cpu_to_le32(virt_to_bus(btv->risc_scr_even));
+			printk(" oo=%08lx",virt_to_phys(btv->risc_scr_even));
+		btv->risc_jmp[11]=cpu_to_le32(virt_to_phys(btv->risc_scr_even));
 	} else {
 		if (bttv_debug > 1)
 			printk(" -----------");
-		btv->risc_jmp[11]=cpu_to_le32(virt_to_bus(btv->risc_jmp+12));
+		btv->risc_jmp[11]=cpu_to_le32(virt_to_phys(btv->risc_jmp+12));
 	}
 
 	if (btv->gq_start) {
@@ -2449,7 +2449,7 @@
 	} else {
 		btv->risc_jmp[12]=cpu_to_le32(BT848_RISC_JUMP);
 	}
-	btv->risc_jmp[13]=cpu_to_le32(virt_to_bus(btv->risc_jmp));
+	btv->risc_jmp[13]=cpu_to_le32(virt_to_phys(btv->risc_jmp));
 
 	/* enable cpaturing and DMA */
 	if (bttv_debug > 1)
@@ -2546,10 +2546,10 @@
 		return -1;
 	btv->vbi_odd=btv->risc_jmp+16;
 	btv->vbi_even=btv->vbi_odd+256;
-	btv->bus_vbi_odd=virt_to_bus(btv->risc_jmp+12);
-	btv->bus_vbi_even=virt_to_bus(btv->risc_jmp+6);
+	btv->bus_vbi_odd=virt_to_phys(btv->risc_jmp+12);
+	btv->bus_vbi_even=virt_to_phys(btv->risc_jmp+6);
 
-	btwrite(virt_to_bus(btv->risc_jmp+2), BT848_RISC_STRT_ADD);
+	btwrite(virt_to_phys(btv->risc_jmp+2), BT848_RISC_STRT_ADD);
 	btv->vbibuf=(unsigned char *) vmalloc_32(VBIBUF_SIZE);
 	if (!btv->vbibuf) 
 		return -1;
@@ -2719,7 +2719,7 @@
 			if (btv->errors < BTTV_ERRORS) {
 				spin_lock(&btv->s_lock);
 				btand(~15, BT848_GPIO_DMA_CTL);
-				btwrite(virt_to_bus(btv->risc_jmp+2),
+				btwrite(virt_to_phys(btv->risc_jmp+2),
 					BT848_RISC_STRT_ADD);
 				bt848_set_geo(btv,0);
 				bt848_set_risc_jmps(btv,-1);

[-- Attachment #4: compile-2.5.4.patch --]
[-- Type: text/plain, Size: 1299 bytes --]

diff -ur linux-2.5.4/arch/i386/kernel/process.c linux/arch/i386/kernel/process.c
--- linux-2.5.4/arch/i386/kernel/process.c	Mon Feb 11 02:50:06 2002
+++ linux/arch/i386/kernel/process.c	Tue Feb 12 19:58:33 2002
@@ -468,6 +468,14 @@
 }
 
 /*
+ * Return saved PC of a blocked thread.
+ */
+unsigned long thread_saved_pc(struct task_struct *tsk)
+{
+	return ((unsigned long *)tsk->thread.esp)[3];
+}
+
+/*
  * No need to lock the MM as we are the last user
  */
 void release_segments(struct mm_struct *mm)
diff -ur linux-2.5.4/include/asm-i386/processor.h linux/include/asm-i386/processor.h
--- linux-2.5.4/include/asm-i386/processor.h	Mon Feb 11 02:50:08 2002
+++ linux/include/asm-i386/processor.h	Tue Feb 12 20:03:54 2002
@@ -436,13 +436,8 @@
 extern void copy_segments(struct task_struct *p, struct mm_struct * mm);
 extern void release_segments(struct mm_struct * mm);
 
-/*
- * Return saved PC of a blocked thread.
- */
-static inline unsigned long thread_saved_pc(struct task_struct *tsk)
-{
-	return ((unsigned long *)tsk->thread->esp)[3];
-}
+/* Return saved PC of a blocked thread. */
+extern unsigned long thread_saved_pc(struct task_struct *tsk);
 
 unsigned long get_wchan(struct task_struct *p);
 #define KSTK_EIP(tsk)	(((unsigned long *)(4096+(unsigned long)(tsk)->thread_info))[1019])

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Jeff Garzik]

Martin Dalecki wrote:
sg->busaddr=virt_to_bus(dmabuf->rawbuf+dmabuf->fragsize*i);
> +                       sg->busaddr=virt_to_phys(dmabuf->rawbuf+dmabuf->fragsize*i);
>                         // the card will always be doing 16bit stereo
>                         sg->control=dmabuf->fragsamples;
>                         if(state->card->pci_id == PCI_DEVICE_ID_SI_7012)
> @@ -954,7 +954,7 @@
>                 }
>                 spin_lock_irqsave(&state->card->lock, flags);
>                 outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
> -               outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
> +               outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);


These changes are wrong.  The addresses desired need to be obtained from
the pci_alloc_consistent return values.


>                         outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
> -                       outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
> +                       outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);

likewise

> -       ret = virt_to_bus((void *)kva);
> +       ret = virt_to_phys((void *)kva);

>          va = VMALLOC_VMADDR(adr);
>          kva = uvirt_to_kva(pgd_offset_k(va), va);
> -       ret = virt_to_bus((void *)kva);
> +       ret = virt_to_phys((void *)kva);

> -                      btv->nr,virt_to_bus(po), virt_to_bus(pe));
> +                      btv->nr,virt_to_phys(po), virt_to_phys(pe));

...likewise, etc.

This works on silly x86 but is not portable at all...  definitely not
for application.


-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

Jeff Garzik wrote:

>Martin Dalecki wrote:
>sg->busaddr=virt_to_bus(dmabuf->rawbuf+dmabuf->fragsize*i);
>
>>+                       sg->busaddr=virt_to_phys(dmabuf->rawbuf+dmabuf->fragsize*i);
>>                        // the card will always be doing 16bit stereo
>>                        sg->control=dmabuf->fragsamples;
>>                        if(state->card->pci_id == PCI_DEVICE_ID_SI_7012)
>>@@ -954,7 +954,7 @@
>>                }
>>                spin_lock_irqsave(&state->card->lock, flags);
>>                outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
>>-               outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
>>+               outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
>>
>
>
>These changes are wrong.  The addresses desired need to be obtained from
>the pci_alloc_consistent return values.
>
>>                        outb(2, state->card->iobase+c->port+OFF_CR);   /* reset DMA machine */
>>-                       outl(virt_to_bus(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
>>+                       outl(virt_to_phys(&c->sg[0]), state->card->iobase+c->port+OFF_BDBAR);
>>
>
>likewise
>
>>-       ret = virt_to_bus((void *)kva);
>>+       ret = virt_to_phys((void *)kva);
>>
>>         va = VMALLOC_VMADDR(adr);
>>         kva = uvirt_to_kva(pgd_offset_k(va), va);
>>-       ret = virt_to_bus((void *)kva);
>>+       ret = virt_to_phys((void *)kva);
>>
>>-                      btv->nr,virt_to_bus(po), virt_to_bus(pe));
>>+                      btv->nr,virt_to_phys(po), virt_to_phys(pe));
>>
>
>...likewise, etc.
>
>This works on silly x86 but is not portable at all...  definitely not
>for application.
>

The bttv we can argue about, I was just tagging it as beeing a quick fix...

Of course I admit that I have taken the easy shoot here. But it wasn't 
possible
to me to deduce the proper thing to do by looking at the patches.
This is the usual way I deal with API changes: Have a look at what has 
been done
to the other candidates and do the analogous thing where you need it.

But please just show me a non x86 architecture which is using the 
i810_audio driver!

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[David S. Miller]

   From: Martin Dalecki <dalecki@evision-ventures.com>
   Date: Wed, 13 Feb 2002 13:40:59 +0100

   Of course I admit that I have taken the easy shoot here. But it wasn't 
   possible
   to me to deduce the proper thing to do by looking at the patches.
   This is the usual way I deal with API changes: Have a look at what has 
   been done
   to the other candidates and do the analogous thing where you need it.
   
The API hasn't changed, it is being enforced.  The PCI DMA api
has existed for years.  Please read Documentation/DMA-mapping.txt
so that you may learn how to properly convert drivers.

   But please just show me a non x86 architecture which is using the 
   i810_audio driver!

Because if all drivers are consistently using the portable interfaces,
people writing new drivers will know exactly what to do.

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

David S. Miller wrote:

>   From: Martin Dalecki <dalecki@evision-ventures.com>
>   Date: Wed, 13 Feb 2002 13:40:59 +0100
>
>   Of course I admit that I have taken the easy shoot here. But it wasn't 
>   possible
>   to me to deduce the proper thing to do by looking at the patches.
>   This is the usual way I deal with API changes: Have a look at what has 
>   been done
>   to the other candidates and do the analogous thing where you need it.
>   
>The API hasn't changed, it is being enforced.  The PCI DMA api
>has existed for years.  Please read Documentation/DMA-mapping.txt
>so that you may learn how to properly convert drivers.
>
>   But please just show me a non x86 architecture which is using the 
>   i810_audio driver!
>
>Because if all drivers are consistently using the portable interfaces,
>people writing new drivers will know exactly what to do.
>
Agred. I see now in patch-2.5.2 and the changes to ymfpci.c how to deal 
with this.
I was already suspicious that the continuous back and forth address 
space conversion
in bttv.c could be dealt with by just doing it once at initialization 
time and storing both
the physical and virtuall mapping. Will do it, when I have pyhsical 
access to the corresponding
hardware at hand.

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

Martin Dalecki wrote:

> Jeff Garzik wrote:
>
>> These changes are wrong.  The addresses desired need to be obtained from
>> the pci_alloc_consistent return values. 
>
>
> The bttv we can argue about, I was just tagging it as beeing a quick 
> fix...
>
> Of course I admit that I have taken the easy shoot here. But it wasn't 
> possible
> to me to deduce the proper thing to do by looking at the patches.
> This is the usual way I deal with API changes: Have a look at what has 
> been done
> to the other candidates and do the analogous thing where you need it.
>
> But please just show me a non x86 architecture which is using the 
> i810_audio driver! 


Ah OK now I see that the API changes you where referencing too are 
missing in bttv.c
already for a longer time then the 2.5.3->2.5.4 stage. This makes it 
clear how to deal with that.


>

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Alan Cox]

> But please just show me a non x86 architecture which is using the 
> i810_audio driver! 

To start with the i810 audio code is the same code as is used for the AMD768
southbridge which can be used with an Alpha processor + AMD762

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > But please just show me a non x86 architecture which is using the 
> > i810_audio driver! 
> 
> To start with the i810 audio code is the same code as is used for the AMD768
> southbridge which can be used with an Alpha processor + AMD762

Or equally fun I won't be surprised if the i870 chipset for the next
generation ia64 itanium processor (mckinley) could use this code.

Eric

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Linus Torvalds]

On Wed, 13 Feb 2002, Jeff Garzik wrote:
>
> These changes are wrong.  The addresses desired need to be obtained from
> the pci_alloc_consistent return values.

Let's face it, people won't care about the complex PCI interfaces unless
they give you something useful.

On most PC's, they don't.

In short, don't expect driver writers to care about somethign that simply
doesn't matter to them. Rant and rail all you want, but I think the thing
needs to be simplified to be acceptable to people. Many of these things
basically exists only on PC's, and are done on 1:1 pages (ie GFP_KERNEL),
so "virt_to_phys()" works.

Basic rule: it's up to _other_ architectures to fix drivers that don't
work for them. Always has been. Because there's no way you can get the
people who just want to have something working to care.

		Linus

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[David S. Miller]

   From: Linus Torvalds <torvalds@transmeta.com>
   Date: Wed, 13 Feb 2002 10:30:48 -0800 (PST)
   
   Basic rule: it's up to _other_ architectures to fix drivers that don't
   work for them. Always has been. Because there's no way you can get the
   people who just want to have something working to care.

And if nobody else ends up doing it, you are right it will be people
like Jeff and myself doing it.

So what we are asking is to allow a few weeks for that and not crap up
the tree meanwhile.  This is so that the cases that need to be
converted are harder to find.

Actually, you're only half right in one regard.  Most people I've
pointed to Documentation/DMA-mapping.txt have responded "Oh, never saw
that before, that looks easy to do.  Thanks I'll fix it up properly
for you."

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

David S. Miller wrote:

>   From: Linus Torvalds <torvalds@transmeta.com>
>   Date: Wed, 13 Feb 2002 10:30:48 -0800 (PST)
>   
>   Basic rule: it's up to _other_ architectures to fix drivers that don't
>   work for them. Always has been. Because there's no way you can get the
>   people who just want to have something working to care.
>
>And if nobody else ends up doing it, you are right it will be people
>like Jeff and myself doing it.
>
>So what we are asking is to allow a few weeks for that and not crap up
>the tree meanwhile.  This is so that the cases that need to be
>converted are harder to find.
>
If you try to use them, then they are not hard to find - things just 
break for you and you fix tem.
If you are fixing things for the "store" Linus is right that indeed it's 
just a waiste of time on your behalf.

>Actually, you're only half right in one regard.  Most people I've
>pointed to Documentation/DMA-mapping.txt have responded "Oh, never saw
>that before, that looks easy to do.  Thanks I'll fix it up properly
>for you."
>

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Jeff Garzik]

Martin Dalecki wrote:
> 
> David S. Miller wrote:
> 
> >   From: Linus Torvalds <torvalds@transmeta.com>
> >   Date: Wed, 13 Feb 2002 10:30:48 -0800 (PST)
> >
> >   Basic rule: it's up to _other_ architectures to fix drivers that don't
> >   work for them. Always has been. Because there's no way you can get the
> >   people who just want to have something working to care.
> >
> >And if nobody else ends up doing it, you are right it will be people
> >like Jeff and myself doing it.
> >
> >So what we are asking is to allow a few weeks for that and not crap up
> >the tree meanwhile.  This is so that the cases that need to be
> >converted are harder to find.
> >
> If you try to use them, then they are not hard to find - things just
> break for you and you fix tem.

Applying a patch like s/virt_to_bus/virt_to_phys/ makes it more
difficult to find the right spots to change later.

Patience :)  We will fix i810_audio and the other notables.  As I noted
in the other message just now, i810_audio and bttv fixes are already
floating around, and hopefully should appear on lkml or in a Linus
pre-patch soon...


> If you are fixing things for the "store" Linus is right that indeed it's
> just a waiste of time on your behalf.

I can tell you it is -not- a waste of time.  It's not a waste of time
when a vendor appears out of nowhere, having copied a driver of yours. 
Since you did it right(tm), the vendor driver is portable, even though
its original source was for x86-only hardware.  It's not a waste of time
when people copy your code or learn from your code.  It's not a waste of
time when spiffy new x86 hardware appears that has useful IOMMU stuff,
making a driver's use of the PCI DMA API automatically useful for that
new hardware.

I agree with Linus that there is little motivation for someone to
continue patching a driver, after they have fixed the problem they set
out to fix.  But that is not the same as saying driver portability is
worthless... far from it.

	Jeff


-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Linus Torvalds]

On Wed, 13 Feb 2002, Jeff Garzik wrote:
>
> Applying a patch like s/virt_to_bus/virt_to_phys/ makes it more
> difficult to find the right spots to change later.

Yes and no.

The thing is, for architectures that care, you can just grep for
"virt_to_phys()". It's basically _never_ the right thing to do on
something like sparc.

My personal preference would actually be to keep "virt_to_bus()" for x86
for now, and undo the change to make it complain. Instead, make it
complain on other architectures where it _is_ wrong, so that you don't
have to fix up drivers that simply aren't an issue. What's the point of
breaking some drivers that only exist on x86?

That, together with a warning and educating more driver writers.

		Linus

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

Linus Torvalds wrote:

>
>On Wed, 13 Feb 2002, Jeff Garzik wrote:
>
>>Applying a patch like s/virt_to_bus/virt_to_phys/ makes it more
>>difficult to find the right spots to change later.
>>
>
>Yes and no.
>
>The thing is, for architectures that care, you can just grep for
>"virt_to_phys()". It's basically _never_ the right thing to do on
>something like sparc.
>
>My personal preference would actually be to keep "virt_to_bus()" for x86
>for now, and undo the change to make it complain. Instead, make it
>complain on other architectures where it _is_ wrong, so that you don't
>have to fix up drivers that simply aren't an issue. What's the point of
>breaking some drivers that only exist on x86?
>
I think that the suggestion from Jeff Garzik, that there is currently 
just too much of code
duplication for quite common cases in drivers is the right way to go. 
Most of the
stuff doing the virt_to_phys is doing quite common things from a broader 
point of view.

Well even worser, there is quite a lot of code replication there as well 
...  see for example the
ide and scsi midlayers ;-). The whole hostadapter/iobus/device stuff 
handling could be made common
at least. There is no real need for different driver handler lookup 
mechanisms between them.

HWIF(device)-> and Scsi_Host_Templ come into mind...

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Jeff Garzik]

In the specific case mentioned in $subject, i810_audio has already been
fixed by Pete Zaitcev "the right way", and IIRC Gerd or somebody posted
a better patch for bttv, too.  So $subject is actually taken care of...


"David S. Miller" wrote:
> And if nobody else ends up doing it, you are right it will be people
> like Jeff and myself doing it.

and have been doing it in the past :)


> So what we are asking is to allow a few weeks for that and not crap up
> the tree meanwhile.  This is so that the cases that need to be
> converted are harder to find.

Yes, please..


> Actually, you're only half right in one regard.  Most people I've
> pointed to Documentation/DMA-mapping.txt have responded "Oh, never saw
> that before, that looks easy to do.  Thanks I'll fix it up properly
> for you."

I still get plenty of the "but virt_to_phys works fine for me" crowd too
:)

	Jeff



-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Linus Torvalds]

On Wed, 13 Feb 2002, David S. Miller wrote:
>
> Actually, you're only half right in one regard.  Most people I've
> pointed to Documentation/DMA-mapping.txt have responded "Oh, never saw
> that before, that looks easy to do.  Thanks I'll fix it up properly
> for you."

Fair enough. Educating driver writers is always good, and the good ones
will try to do their best even when it doesn't matter for them. I just
wanted to make sure that we don't make driver writers have to chew off too
big a piece.

(One example of this: look at how the big changes in the ll_rw_block
infrastructure were done - the whole locking thing was basically set up to
avoid having to change legacy drivers in anything but trivial ways, while
still sanely getting rid of io_request_lock. Similarly, while the BIO
stuff was a big change on a mid level layer, there was considerable effort
to make it easy to port drivers that didn't try to be clever to it).

Some drivers are written by people who are really passionate about kernel
internals, and that's good.

But many of them are barely working, written by people who don't care
about the rest of the kernel (or even about the driver itself, they just
wanted to have a working machine and forget about it), and if we make
those kinds of drivers do extra work, it's just not going to work.

		Linus

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Jeff Garzik]

Linus Torvalds wrote:
> But many of them are barely working, written by people who don't care
> about the rest of the kernel (or even about the driver itself, they just
> wanted to have a working machine and forget about it), and if we make
> those kinds of drivers do extra work, it's just not going to work.

Which is why, IMO, we should endeavor to make drivers as cookie-cutter
and dirt simple to create as possible.  It will probably take many
months, but I would like to continually factor out from the net drivers
not only common code but -design patterns-.

As an experiment a couple months ago, I got most of the PCI net drivers
down to ~200-300 lines of C code apiece, by factoring out common code
patterns into M4 macros.  "m4 netdrivers.m4 epic100.tmpl > epic100.c"

I would prefer to make drivers so dirt simple that people don't need to
worry about details like PCI DMA API changes...

	Jeff, dreams on

-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Pavel Machek]

Hi!

> As an experiment a couple months ago, I got most of the PCI net drivers
> down to ~200-300 lines of C code apiece, by factoring out common code
> patterns into M4 macros.  "m4 netdrivers.m4 epic100.tmpl > epic100.c"

This is slightly extreme, right?

But I'd like to see resulting epic100.tmpl ;-).
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Jeff Garzik]

Pavel Machek wrote:
> 
> Hi!
> 
> > As an experiment a couple months ago, I got most of the PCI net drivers
> > down to ~200-300 lines of C code apiece, by factoring out common code
> > patterns into M4 macros.  "m4 netdrivers.m4 epic100.tmpl > epic100.c"
> 
> This is slightly extreme, right?
> 
> But I'd like to see resulting epic100.tmpl ;-).

When you have to maintain more than 10 "cookie cutter" net drivers that
are 80-90% the same, you start to want such extremes :)

-- 
Jeff Garzik      | "I went through my candy like hot oatmeal
Building 1024    |  through an internally-buttered weasel."
MandrakeSoft     |             - goats.com

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Linus Torvalds]

On Thu, 14 Feb 2002, Jeff Garzik wrote:
> > 
> > But I'd like to see resulting epic100.tmpl ;-).
> 
> When you have to maintain more than 10 "cookie cutter" net drivers that
> are 80-90% the same, you start to want such extremes :)

It's not necessarily a bad idea to have a more capable preprocessor than
the C preprocessor. I've cursed preprocessor limitations before, and I
there was some discussion about using m4 several years ago (and I mean
_several_ years ago - if I were to guess I'd say 6-8 years ago..).

		Linus

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Martin Dalecki]

Linus Torvalds wrote:

>On Thu, 14 Feb 2002, Jeff Garzik wrote:
>
>>>But I'd like to see resulting epic100.tmpl ;-).
>>>
>>When you have to maintain more than 10 "cookie cutter" net drivers that
>>are 80-90% the same, you start to want such extremes :)
>>
>
>It's not necessarily a bad idea to have a more capable preprocessor than
>the C preprocessor. I've cursed preprocessor limitations before, and I
>there was some discussion about using m4 several years ago (and I mean
>_several_ years ago - if I were to guess I'd say 6-8 years ago..).
>

The idal solution would be some kind of stripped down C++ for some of 
those problems...
No rtti, no templates, no exceptions, no additional cruft requirng back 
behind you runtime
support for the language, but just plain simple direct struct 
inheritance kind off ;-).

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Olaf Titz]

> The idal solution would be some kind of stripped down C++ for some of
> those problems...
> No rtti, no templates, no exceptions, no additional cruft requirng back
> behind you runtime
> support for the language, but just plain simple direct struct
> inheritance kind off ;-).

I have a preprocessor for that. It's part of my squid-filters package
but could be generally useful as well.

<URL:http://sites.inka.de/~bigred/devel/squid-filter.html>

Olaf

PATCH 2.5.5-pre1 dead arrays.

[Martin Dalecki]

[-- Attachment #1: Type: text/plain, Size: 150 bytes --]

Just the usual removal of the dead global arrays and associated cruft.
(Thistime not affecting lvm, which BTW. doesn't compile currently anyway 
;-).

[-- Attachment #2: dead-arrays.2.5.5.patch --]
[-- Type: text/plain, Size: 41250 bytes --]

diff -ur linux-2.5.4/Documentation/cdrom/sbpcd linux/Documentation/cdrom/sbpcd
--- linux-2.5.4/Documentation/cdrom/sbpcd	Mon Feb 11 02:50:10 2002
+++ linux/Documentation/cdrom/sbpcd	Fri Feb 15 09:27:00 2002
@@ -613,8 +613,8 @@
 	printf("READ          d      READ RAW     w       READ AUDIO   A\n");
 	printf("MS-INFO       M      TOC          T       START        S\n");
 	printf("SET EJECTSW   X      DEVICE       D       DEBUG        Y\n");
-	printf("AUDIO_BUFSIZ  Z      RESET        R       BLKRASET     B\n");
-	printf("SET VOLUME    v      GET VOLUME   V\n");
+	printf("AUDIO_BUFSIZ  Z      RESET        R       SET VOLUME   v\n");
+	printf("GET VOLUME    V\n");
 }
 
 /*
@@ -882,12 +882,6 @@
 			rc=ioctl(drive,CDROMRESET);
 			if (rc<0) printf("CDROMRESET: rc=%d.\n",rc);
 			break;
-		case 'B': /* set the driver's (?) read ahead value */
-			printf("enter read-ahead size: ? ");
-			scanf("%d",&i);
-			rc=ioctl(drive,BLKRASET,i);
-			if (rc<0) printf("BLKRASET: rc=%d.\n",rc);
-			break;
 #ifdef AZT_PRIVATE_IOCTLS /*not supported by every CDROM driver*/
 		case 'd':
 			printf("Address (min:sec:frm)  ");
diff -ur linux-2.5.4/arch/mips64/kernel/ioctl32.c linux/arch/mips64/kernel/ioctl32.c
--- linux-2.5.4/arch/mips64/kernel/ioctl32.c	Mon Feb 11 02:50:17 2002
+++ linux/arch/mips64/kernel/ioctl32.c	Fri Feb 15 09:27:00 2002
@@ -760,10 +760,6 @@
 	IOCTL32_HANDLER(BLKGETSIZE, w_long),
 
 	IOCTL32_DEFAULT(BLKFLSBUF),
-	IOCTL32_DEFAULT(BLKRASET),
-	IOCTL32_HANDLER(BLKRAGET, w_long),
-	IOCTL32_DEFAULT(BLKFRASET),
-	IOCTL32_HANDLER(BLKFRAGET, w_long),
 	IOCTL32_DEFAULT(BLKSECTSET),
 	IOCTL32_HANDLER(BLKSECTGET, w_long),
 	IOCTL32_DEFAULT(BLKSSZGET),
diff -ur linux-2.5.4/arch/sparc64/kernel/ioctl32.c linux/arch/sparc64/kernel/ioctl32.c
--- linux-2.5.4/arch/sparc64/kernel/ioctl32.c	Mon Feb 11 02:50:14 2002
+++ linux/arch/sparc64/kernel/ioctl32.c	Fri Feb 15 09:27:00 2002
@@ -3997,8 +3997,6 @@
 COMPATIBLE_IOCTL(BLKROGET)
 COMPATIBLE_IOCTL(BLKRRPART)
 COMPATIBLE_IOCTL(BLKFLSBUF)
-COMPATIBLE_IOCTL(BLKRASET)
-COMPATIBLE_IOCTL(BLKFRASET)
 COMPATIBLE_IOCTL(BLKSECTSET)
 COMPATIBLE_IOCTL(BLKSSZGET)
 COMPATIBLE_IOCTL(BLKBSZGET)
@@ -4626,10 +4624,8 @@
 HANDLE_IOCTL(SIOCRTMSG, ret_einval)
 HANDLE_IOCTL(SIOCGSTAMP, do_siocgstamp)
 HANDLE_IOCTL(HDIO_GETGEO, hdio_getgeo)
-HANDLE_IOCTL(BLKRAGET, w_long)
 HANDLE_IOCTL(BLKGETSIZE, w_long)
 HANDLE_IOCTL(0x1260, broken_blkgetsize)
-HANDLE_IOCTL(BLKFRAGET, w_long)
 HANDLE_IOCTL(BLKSECTGET, w_long)
 HANDLE_IOCTL(BLKPG, blkpg_ioctl_trans)
 HANDLE_IOCTL(FBIOPUTCMAP32, fbiogetputcmap)
diff -ur linux-2.5.4/drivers/acorn/block/mfmhd.c linux/drivers/acorn/block/mfmhd.c
--- linux-2.5.4/drivers/acorn/block/mfmhd.c	Mon Feb 11 02:50:11 2002
+++ linux/drivers/acorn/block/mfmhd.c	Fri Feb 15 09:27:00 2002
@@ -1208,15 +1208,6 @@
 			return -EFAULT;
 		return 0;
 
-	case BLKFRASET:
-		if (!capable(CAP_SYS_ADMIN))
-			return -EACCES;
-		max_readahead[major][minor] = arg;
-		return 0;
-
-	case BLKFRAGET:
-		return put_user(max_readahead[major][minor], (long *) arg);
-
 	case BLKSECTGET:
 		return put_user(max_sectors[major][minor], (long *) arg);
 
@@ -1230,8 +1221,6 @@
 	case BLKFLSBUF:
 	case BLKROSET:
 	case BLKROGET:
-	case BLKRASET:
-	case BLKRAGET:
 	case BLKPG:
 		return blk_ioctl(dev, cmd, arg);
 
@@ -1442,7 +1431,6 @@
 	hdc63463_irqpollmask	= irqmask;
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB?) read ahread */
 
 	add_gendisk(&mfm_gendisk);
 
diff -ur linux-2.5.4/drivers/block/DAC960.c linux/drivers/block/DAC960.c
--- linux-2.5.4/drivers/block/DAC960.c	Mon Feb 11 02:50:18 2002
+++ linux/drivers/block/DAC960.c	Fri Feb 15 09:27:00 2002
@@ -1964,10 +1964,6 @@
   Controller->GenericDiskInfo.sizes = Controller->PartitionSizes;
   blksize_size[MajorNumber] = Controller->BlockSizes;
   /*
-    Initialize Read Ahead to 128 sectors.
-  */
-  read_ahead[MajorNumber] = 128;
-  /*
     Complete initialization of the Generic Disk Information structure.
   */
   Controller->GenericDiskInfo.major = MajorNumber;
@@ -5399,8 +5395,6 @@
 			   sizeof(DiskGeometry_T)) ? -EFAULT : 0);
     case BLKGETSIZE:
     case BLKGETSIZE64:
-    case BLKRAGET:
-    case BLKRASET:
     case BLKFLSBUF:
     case BLKBSZGET:
     case BLKBSZSET:
diff -ur linux-2.5.4/drivers/block/acsi.c linux/drivers/block/acsi.c
--- linux-2.5.4/drivers/block/acsi.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/block/acsi.c	Fri Feb 15 09:27:00 2002
@@ -1785,7 +1785,6 @@
 	STramMask = ATARIHW_PRESENT(EXTD_DMA) ? 0x00000000 : 0xff000000;
 	
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &acsi_lock);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&acsi_gendisk);
 
 #ifdef CONFIG_ATARI_SLM
diff -ur linux-2.5.4/drivers/block/ataflop.c linux/drivers/block/ataflop.c
--- linux-2.5.4/drivers/block/ataflop.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/block/ataflop.c	Fri Feb 15 09:27:00 2002
@@ -1573,8 +1573,6 @@
 	switch (cmd) {
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKFLSBUF:
 			return blk_ioctl(device, cmd, param);
 	}
diff -ur linux-2.5.4/drivers/block/blkpg.c linux/drivers/block/blkpg.c
--- linux-2.5.4/drivers/block/blkpg.c	Mon Feb 11 02:50:08 2002
+++ linux/drivers/block/blkpg.c	Fri Feb 15 09:27:00 2002
@@ -29,7 +29,7 @@
  */
 
 #include <linux/errno.h>
-#include <linux/fs.h>			/* for BLKRASET, ... */
+#include <linux/fs.h>			/* for BLKROSET, ... */
 #include <linux/sched.h>		/* for capable() */
 #include <linux/blk.h>			/* for set_device_ro() */
 #include <linux/blkpg.h>
@@ -227,31 +227,6 @@
 			intval = (is_read_only(dev) != 0);
 			return put_user(intval, (int *)(arg));
 
-		case BLKRASET:
-			if(!capable(CAP_SYS_ADMIN))
-				return -EACCES;
-			if(arg > 0xff)
-				return -EINVAL;
-			read_ahead[major(dev)] = arg;
-			return 0;
-		case BLKRAGET:
-			if (!arg)
-				return -EINVAL;
-			return put_user(read_ahead[major(dev)], (long *) arg);
-
-		case BLKFRASET:
-			if (!capable(CAP_SYS_ADMIN))
-				return -EACCES;
-			if (!(iptr = max_readahead[major(dev)]))
-				return -EINVAL;
-			iptr[minor(dev)] = arg;
-			return 0;
-
-		case BLKFRAGET:
-			if (!(iptr = max_readahead[major(dev)]))
-				return -EINVAL;
-			return put_user(iptr[minor(dev)], (long *) arg);
-
 		case BLKSECTGET:
 			if ((q = blk_get_queue(dev)) == NULL)
 				return -EINVAL;
diff -ur linux-2.5.4/drivers/block/cciss.c linux/drivers/block/cciss.c
--- linux-2.5.4/drivers/block/cciss.c	Fri Feb 15 09:26:27 2002
+++ linux/drivers/block/cciss.c	Fri Feb 15 09:27:00 2002
@@ -468,8 +468,6 @@
 	case BLKBSZGET:
 	case BLKROSET:
 	case BLKROGET:
-	case BLKRASET:
-	case BLKRAGET:
 	case BLKPG:
 		return blk_ioctl(inode->i_rdev, cmd, arg);
 	case CCISS_GETPCIINFO:
@@ -2543,7 +2541,6 @@
 
 	/* fill in the other Kernel structs */
 	blksize_size[MAJOR_NR+i] = hba[i]->blocksizes;
-        read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 	/* Fill in the gendisk data */ 	
 	hba[i]->gendisk.major = MAJOR_NR + i;
diff -ur linux-2.5.4/drivers/block/cpqarray.c linux/drivers/block/cpqarray.c
--- linux-2.5.4/drivers/block/cpqarray.c	Fri Feb 15 09:26:27 2002
+++ linux/drivers/block/cpqarray.c	Fri Feb 15 09:27:00 2002
@@ -481,7 +481,6 @@
 		blk_queue_max_phys_segments(q, SG_MAX);
 
 		blksize_size[MAJOR_NR+i] = ida_blocksizes + (i*256);
-		read_ahead[MAJOR_NR+i] = READ_AHEAD;
 
 		ida_gendisk[i].major = MAJOR_NR + i;
 		ida_gendisk[i].major_name = "ida";
@@ -1179,8 +1178,6 @@
 	case BLKBSZGET:
 	case BLKROSET:
 	case BLKROGET:
-	case BLKRASET:
-	case BLKRAGET:
 	case BLKPG:
 		return blk_ioctl(inode->i_rdev, cmd, arg);
 
diff -ur linux-2.5.4/drivers/block/floppy.c linux/drivers/block/floppy.c
--- linux-2.5.4/drivers/block/floppy.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/block/floppy.c	Fri Feb 15 09:27:00 2002
@@ -3448,8 +3448,6 @@
 	switch (cmd) {
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKFLSBUF:
 			return blk_ioctl(device, cmd, param);
 	}
diff -ur linux-2.5.4/drivers/block/ll_rw_blk.c linux/drivers/block/ll_rw_blk.c
--- linux-2.5.4/drivers/block/ll_rw_blk.c	Fri Feb 15 09:26:27 2002
+++ linux/drivers/block/ll_rw_blk.c	Fri Feb 15 09:27:00 2002
@@ -54,10 +54,6 @@
  */
 DECLARE_TASK_QUEUE(tq_disk);
 
-/* This specifies how many sectors to read ahead on the disk. */
-
-int read_ahead[MAX_BLKDEV];
-
 /* blk_dev_struct is:
  *	request_queue
  *	*queue
@@ -84,11 +80,6 @@
 int * blksize_size[MAX_BLKDEV];
 
 /*
- * The following tunes the read-ahead algorithm in mm/filemap.c
- */
-int * max_readahead[MAX_BLKDEV];
-
-/*
  * How many reqeusts do we allocate per queue,
  * and how many do we "batch" on freeing them?
  */
@@ -1685,7 +1676,6 @@
 		dev->queue = NULL;
 
 	memset(ro_bits,0,sizeof(ro_bits));
-	memset(max_readahead, 0, sizeof(max_readahead));
 
 	total_ram = nr_free_pages() << (PAGE_SHIFT - 10);
 
diff -ur linux-2.5.4/drivers/block/paride/pcd.c linux/drivers/block/paride/pcd.c
--- linux-2.5.4/drivers/block/paride/pcd.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/block/paride/pcd.c	Fri Feb 15 09:27:00 2002
@@ -358,7 +358,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &pcd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 
 	for (i=0;i<PCD_UNITS;i++) pcd_blocksizes[i] = 1024;
         blksize_size[MAJOR_NR] = pcd_blocksizes;
diff -ur linux-2.5.4/drivers/block/paride/pd.c linux/drivers/block/paride/pd.c
--- linux-2.5.4/drivers/block/paride/pd.c	Mon Feb 11 02:50:12 2002
+++ linux/drivers/block/paride/pd.c	Fri Feb 15 09:27:00 2002
@@ -397,7 +397,6 @@
 	q = BLK_DEFAULT_QUEUE(MAJOR_NR);
 	blk_init_queue(q, DEVICE_REQUEST, &pd_lock);
 	blk_queue_max_sectors(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	pd_gendisk.major = major;
 	pd_gendisk.major_name = name;
@@ -480,8 +479,6 @@
 	    case BLKGETSIZE64:
 	    case BLKROSET:
 	    case BLKROGET:
-	    case BLKRASET:
-	    case BLKRAGET:
 	    case BLKFLSBUF:
 	    case BLKPG:
 		return blk_ioctl(inode->i_rdev, cmd, arg);
diff -ur linux-2.5.4/drivers/block/paride/pf.c linux/drivers/block/paride/pf.c
--- linux-2.5.4/drivers/block/paride/pf.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/block/paride/pf.c	Fri Feb 15 09:27:00 2002
@@ -363,7 +363,6 @@
 	blk_init_queue(q, DEVICE_REQUEST, &pf_spin_lock);
 	blk_queue_max_phys_segments(q, cluster);
 	blk_queue_max_hw_segments(q, cluster);
-        read_ahead[MAJOR_NR] = 8;       /* 8 sector (4kB) read ahead */
         
 	for (i=0;i<PF_UNITS;i++) pf_blocksizes[i] = 1024;
 	blksize_size[MAJOR_NR] = pf_blocksizes;
@@ -433,8 +432,6 @@
                 return put_user((u64)PF.capacity << 9,(u64 *)arg);
 	    case BLKROSET:
 	    case BLKROGET:
-	    case BLKRASET:
-	    case BLKRAGET:
 	    case BLKFLSBUF:
 		return blk_ioctl(inode->i_rdev, cmd, arg);
             default:
diff -ur linux-2.5.4/drivers/block/ps2esdi.c linux/drivers/block/ps2esdi.c
--- linux-2.5.4/drivers/block/ps2esdi.c	Mon Feb 11 02:50:06 2002
+++ linux/drivers/block/ps2esdi.c	Fri Feb 15 09:27:01 2002
@@ -177,7 +177,6 @@
 	}
 	/* set up some global information - indicating device specific info */
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &ps2esdi_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 
 	/* some minor housekeeping - setup the global gendisk structure */
 	add_gendisk(&ps2esdi_gendisk);
@@ -1108,8 +1107,6 @@
 		case BLKGETSIZE64:
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKFLSBUF:
 		case BLKBSZGET:
 		case BLKBSZSET:
diff -ur linux-2.5.4/drivers/block/xd.c linux/drivers/block/xd.c
--- linux-2.5.4/drivers/block/xd.c	Mon Feb 11 02:50:13 2002
+++ linux/drivers/block/xd.c	Fri Feb 15 09:27:01 2002
@@ -171,7 +171,6 @@
 	}
 	devfs_handle = devfs_mk_dir (NULL, xd_gendisk.major_name, NULL);
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &xd_lock);
-	read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read ahead */
 	add_gendisk(&xd_gendisk);
 	xd_geninit();
 
@@ -355,8 +354,6 @@
 		case BLKFLSBUF:
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKPG:
 			return blk_ioctl(inode->i_rdev, cmd, arg);
 
diff -ur linux-2.5.4/drivers/cdrom/aztcd.c linux/drivers/cdrom/aztcd.c
--- linux-2.5.4/drivers/cdrom/aztcd.c	Mon Feb 11 02:50:14 2002
+++ linux/drivers/cdrom/aztcd.c	Fri Feb 15 09:27:01 2002
@@ -1927,7 +1927,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &aztSpin);
 	blksize_size[MAJOR_NR] = aztcd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &azt_fops, 0);
 
 	if ((azt_port == 0x1f0) || (azt_port == 0x170))
diff -ur linux-2.5.4/drivers/cdrom/cdu31a.c linux/drivers/cdrom/cdu31a.c
--- linux-2.5.4/drivers/cdrom/cdu31a.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/cdrom/cdu31a.c	Fri Feb 15 09:27:01 2002
@@ -3442,7 +3442,6 @@
 		blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR),
 			       DEVICE_REQUEST,
 			       &cdu31a_lock);
-		read_ahead[MAJOR_NR] = CDU31A_READAHEAD;
 		cdu31a_block_size = 1024;	/* 1kB default block size */
 		/* use 'mount -o block=2048' */
 		blksize_size[MAJOR_NR] = &cdu31a_block_size;
diff -ur linux-2.5.4/drivers/cdrom/cm206.c linux/drivers/cdrom/cm206.c
--- linux-2.5.4/drivers/cdrom/cm206.c	Mon Feb 11 02:50:14 2002
+++ linux/drivers/cdrom/cm206.c	Fri Feb 15 09:27:01 2002
@@ -1503,7 +1503,6 @@
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &cm206_lock);
 	blksize_size[MAJOR_NR] = cm206_blocksizes;
-	read_ahead[MAJOR_NR] = 16;	/* reads ahead what? */
 	init_bh(CM206_BH, cm206_bh);
 
 	memset(cd, 0, sizeof(*cd));	/* give'm some reasonable value */
diff -ur linux-2.5.4/drivers/cdrom/gscd.c linux/drivers/cdrom/gscd.c
--- linux-2.5.4/drivers/cdrom/gscd.c	Mon Feb 11 02:50:07 2002
+++ linux/drivers/cdrom/gscd.c	Fri Feb 15 09:27:01 2002
@@ -1022,7 +1022,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &gscd_lock);
 	blksize_size[MAJOR_NR] = gscd_blocksizes;
-	read_ahead[MAJOR_NR] = 4;
 
 	disk_state = 0;
 	gscdPresent = 1;
diff -ur linux-2.5.4/drivers/cdrom/mcd.c linux/drivers/cdrom/mcd.c
--- linux-2.5.4/drivers/cdrom/mcd.c	Mon Feb 11 02:50:08 2002
+++ linux/drivers/cdrom/mcd.c	Fri Feb 15 09:27:01 2002
@@ -1075,7 +1075,6 @@
 	blksize_size[MAJOR_NR] = mcd_blocksizes;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcd_spinlock);
-	read_ahead[MAJOR_NR] = 4;
 
 	/* check for card */
 
diff -ur linux-2.5.4/drivers/cdrom/mcdx.c linux/drivers/cdrom/mcdx.c
--- linux-2.5.4/drivers/cdrom/mcdx.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/cdrom/mcdx.c	Fri Feb 15 09:27:01 2002
@@ -1184,7 +1184,6 @@
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &mcdx_lock);
-	read_ahead[MAJOR_NR] = READ_AHEAD;
 	blksize_size[MAJOR_NR] = mcdx_blocksizes;
 
 	xtrace(INIT, "init() subscribe irq and i/o\n");
diff -ur linux-2.5.4/drivers/cdrom/optcd.c linux/drivers/cdrom/optcd.c
--- linux-2.5.4/drivers/cdrom/optcd.c	Mon Feb 11 02:50:12 2002
+++ linux/drivers/cdrom/optcd.c	Fri Feb 15 09:27:01 2002
@@ -2062,7 +2062,6 @@
 	blksize_size[MAJOR_NR] = &blksize;
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,
 		       &optcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	request_region(optcd_port, 4, "optcd");
 	register_disk(NULL, mk_kdev(MAJOR_NR,0), 1, &opt_fops, 0);
 
diff -ur linux-2.5.4/drivers/cdrom/sbpcd.c linux/drivers/cdrom/sbpcd.c
--- linux-2.5.4/drivers/cdrom/sbpcd.c	Mon Feb 11 02:50:11 2002
+++ linux/drivers/cdrom/sbpcd.c	Fri Feb 15 09:27:01 2002
@@ -4531,12 +4531,6 @@
 		RETURN_UP(0);
 	} /* end of CDROMREADAUDIO */
 		
-	case BLKRASET:
-		if(!capable(CAP_SYS_ADMIN)) RETURN_UP(-EACCES);
-		if(kdev_none(cdi->dev)) RETURN_UP(-EINVAL);
-		if(arg > 0xff) RETURN_UP(-EINVAL);
-		read_ahead[major(cdi->dev)] = arg;
-		RETURN_UP(0);
 	default:
 		msg(DBG_IOC,"ioctl: unknown function request %04X\n", cmd);
 		RETURN_UP(-EINVAL);
diff -ur linux-2.5.4/drivers/cdrom/sjcd.c linux/drivers/cdrom/sjcd.c
--- linux-2.5.4/drivers/cdrom/sjcd.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/cdrom/sjcd.c	Fri Feb 15 09:27:01 2002
@@ -1695,7 +1695,6 @@
 	}
 
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST,&sjcd_lock);
-	read_ahead[MAJOR_NR] = 4;
 	register_disk(NULL, mk_kdev(MAJOR_NR, 0), 1, &sjcd_fops, 0);
 
 	if (check_region(sjcd_base, 4)) {
diff -ur linux-2.5.4/drivers/cdrom/sonycd535.c linux/drivers/cdrom/sonycd535.c
--- linux-2.5.4/drivers/cdrom/sonycd535.c	Mon Feb 11 02:50:14 2002
+++ linux/drivers/cdrom/sonycd535.c	Fri Feb 15 09:27:01 2002
@@ -1598,7 +1598,6 @@
 				}
 				blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &sonycd535_lock);
 				blksize_size[MAJOR_NR] = &sonycd535_block_size;
-				read_ahead[MAJOR_NR] = 8;	/* 8 sector (4kB) read-ahead */
 
 				sony_toc = (struct s535_sony_toc *)
 					kmalloc(sizeof *sony_toc, GFP_KERNEL);
diff -ur linux-2.5.4/drivers/ide/ataraid.c linux/drivers/ide/ataraid.c
--- linux-2.5.4/drivers/ide/ataraid.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/ide/ataraid.c	Fri Feb 15 09:27:01 2002
@@ -289,7 +289,6 @@
 	hardsect_size[ATAMAJOR] = NULL;
 	blk_size[ATAMAJOR] = NULL;
 	blksize_size[ATAMAJOR] = NULL;                       
-	max_readahead[ATAMAJOR] = NULL;
 
 	del_gendisk(&ataraid_gendisk);
         
diff -ur linux-2.5.4/drivers/ide/hd.c linux/drivers/ide/hd.c
--- linux-2.5.4/drivers/ide/hd.c	Mon Feb 11 02:50:10 2002
+++ linux/drivers/ide/hd.c	Fri Feb 15 09:27:01 2002
@@ -652,8 +652,6 @@
 		case BLKGETSIZE64:
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKFLSBUF:
 		case BLKPG:
 			return blk_ioctl(inode->i_rdev, cmd, arg);
@@ -837,7 +835,6 @@
 	}
 	blk_init_queue(BLK_DEFAULT_QUEUE(MAJOR_NR), DEVICE_REQUEST, &hd_lock);
 	blk_queue_max_sectors(BLK_DEFAULT_QUEUE(MAJOR_NR), 255);
-	read_ahead[MAJOR_NR] = 8;		/* 8 sector (4kB) read-ahead */
 	add_gendisk(&hd_gendisk);
 	init_timer(&device_timer);
 	device_timer.function = hd_times_out;
diff -ur linux-2.5.4/drivers/ide/ide-cd.c linux/drivers/ide/ide-cd.c
--- linux-2.5.4/drivers/ide/ide-cd.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/ide/ide-cd.c	Fri Feb 15 09:27:01 2002
@@ -2659,11 +2659,6 @@
 
 static void ide_cdrom_add_settings(ide_drive_t *drive)
 {
-	int major = HWIF(drive)->major;
-	int minor = drive->select.b.unit << PARTN_BITS;
-
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW, BLKRAGET, BLKRASET, TYPE_INT, 0, 255, 1, 2, &read_ahead[major], NULL);
-	ide_add_setting(drive,	"file_readahead",	SETTING_RW, BLKFRAGET, BLKFRASET, TYPE_INTA, 0, INT_MAX, 1, 1024, &max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"dsc_overlap",		SETTING_RW, -1, -1, TYPE_BYTE, 0, 1, 1,	1, &drive->dsc_overlap, NULL);
 }
 
diff -ur linux-2.5.4/drivers/ide/ide-disk.c linux/drivers/ide/ide-disk.c
--- linux-2.5.4/drivers/ide/ide-disk.c	Fri Feb 15 09:26:27 2002
+++ linux/drivers/ide/ide-disk.c	Fri Feb 15 09:27:01 2002
@@ -901,8 +901,6 @@
 static void idedisk_add_settings(ide_drive_t *drive)
 {
 	struct hd_driveid *id = drive->id;
-	int major = HWIF(drive)->major;
-	int minor = drive->select.b.unit << PARTN_BITS;
 
 	ide_add_setting(drive,	"bios_cyl",		SETTING_RW,					-1,			-1,			TYPE_INT,	0,	65535,				1,	1,	&drive->bios_cyl,		NULL);
 	ide_add_setting(drive,	"bios_head",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	255,				1,	1,	&drive->bios_head,		NULL);
@@ -911,8 +909,6 @@
 	ide_add_setting(drive,	"bswap",		SETTING_READ,					-1,			-1,			TYPE_BYTE,	0,	1,				1,	1,	&drive->bswap,			NULL);
 	ide_add_setting(drive,	"multcount",		id ? SETTING_RW : SETTING_READ,			HDIO_GET_MULTCOUNT,	HDIO_SET_MULTCOUNT,	TYPE_BYTE,	0,	id ? id->max_multsect : 0,	1,	1,	&drive->mult_count,		set_multcount);
 	ide_add_setting(drive,	"nowerr",		SETTING_RW,					HDIO_GET_NOWERR,	HDIO_SET_NOWERR,	TYPE_BYTE,	0,	1,				1,	1,	&drive->nowerr,			set_nowerr);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	1,	&read_ahead[major],		NULL);
-	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	4096,			PAGE_SIZE,	1024,	&max_readahead[major][minor],	NULL);
 	ide_add_setting(drive,	"lun",			SETTING_RW,					-1,			-1,			TYPE_INT,	0,	7,				1,	1,	&drive->lun,			NULL);
 	ide_add_setting(drive,	"wcache",		SETTING_RW,					HDIO_GET_WCACHE,	HDIO_SET_WCACHE,	TYPE_BYTE,	0,	1,				1,	1,	&drive->wcache,			write_cache);
 	ide_add_setting(drive,	"acoustic",		SETTING_RW,					HDIO_GET_ACOUSTIC,	HDIO_SET_ACOUSTIC,	TYPE_BYTE,	0,	254,				1,	1,	&drive->acoustic,		set_acoustic);
diff -ur linux-2.5.4/drivers/ide/ide-floppy.c linux/drivers/ide/ide-floppy.c
--- linux-2.5.4/drivers/ide/ide-floppy.c	Mon Feb 11 02:50:12 2002
+++ linux/drivers/ide/ide-floppy.c	Fri Feb 15 09:27:01 2002
@@ -1962,14 +1962,9 @@
 
 static void idefloppy_add_settings(ide_drive_t *drive)
 {
-	int major = HWIF(drive)->major;
-	int minor = drive->select.b.unit << PARTN_BITS;
-
 	ide_add_setting(drive,	"bios_cyl",		SETTING_RW,					-1,			-1,			TYPE_INT,	0,	1023,				1,	1,	&drive->bios_cyl,		NULL);
 	ide_add_setting(drive,	"bios_head",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	255,				1,	1,	&drive->bios_head,		NULL);
 	ide_add_setting(drive,	"bios_sect",		SETTING_RW,					-1,			-1,			TYPE_BYTE,	0,	63,				1,	1,	&drive->bios_sect,		NULL);
-	ide_add_setting(drive,	"breada_readahead",	SETTING_RW,					BLKRAGET,		BLKRASET,		TYPE_INT,	0,	255,				1,	2,	&read_ahead[major],		NULL);
-	ide_add_setting(drive,	"file_readahead",	SETTING_RW,					BLKFRAGET,		BLKFRASET,		TYPE_INTA,	0,	INT_MAX,			1,	1024,	&max_readahead[major][minor],	NULL);
 
 }
 
diff -ur linux-2.5.4/drivers/ide/ide-probe.c linux/drivers/ide/ide-probe.c
--- linux-2.5.4/drivers/ide/ide-probe.c	Mon Feb 11 02:50:06 2002
+++ linux/drivers/ide/ide-probe.c	Fri Feb 15 09:27:01 2002
@@ -788,8 +788,7 @@
 static void init_gendisk (ide_hwif_t *hwif)
 {
 	struct gendisk *gd;
-	unsigned int unit, units, minors;
-	int *bs, *max_ra;
+	unsigned int unit, units, minors, i;
 	extern devfs_handle_t ide_devfs_handle;
 
 #if 1
@@ -802,33 +801,25 @@
 	}
 #endif
 
-	minors    = units * (1<<PARTN_BITS);
-	gd        = kmalloc (sizeof(struct gendisk), GFP_KERNEL);
+	minors = units * (1<<PARTN_BITS);
+
+	gd = kmalloc (sizeof(struct gendisk), GFP_KERNEL);
 	if (!gd)
 		goto err_kmalloc_gd;
 	gd->sizes = kmalloc (minors * sizeof(int), GFP_KERNEL);
 	if (!gd->sizes)
 		goto err_kmalloc_gd_sizes;
-	gd->part  = kmalloc (minors * sizeof(struct hd_struct), GFP_KERNEL);
+	gd->part = kmalloc (minors * sizeof(struct hd_struct), GFP_KERNEL);
 	if (!gd->part)
 		goto err_kmalloc_gd_part;
-	bs        = kmalloc (minors*sizeof(int), GFP_KERNEL);
-	if (!bs)
+	blksize_size[hwif->major] = kmalloc (minors*sizeof(int), GFP_KERNEL);
+	if (!blksize_size[hwif->major])
 		goto err_kmalloc_bs;
-	max_ra    = kmalloc (minors*sizeof(int), GFP_KERNEL);
-	if (!max_ra)
-		goto err_kmalloc_max_ra;
 
 	memset(gd->part, 0, minors * sizeof(struct hd_struct));
 
-	/* cdroms and msdos f/s are examples of non-1024 blocksizes */
-	blksize_size[hwif->major] = bs;
-	max_readahead[hwif->major] = max_ra;
-	for (unit = 0; unit < minors; ++unit) {
-		*bs++ = BLOCK_SIZE;
-		*max_ra++ = MAX_READAHEAD;
-	}
-
+	for (i = 0; i < minors; ++i)
+	    blksize_size[hwif->major][i] = BLOCK_SIZE;
 	for (unit = 0; unit < units; ++unit)
 		hwif->drives[unit].part = &gd->part[unit << PARTN_BITS];
 
@@ -875,8 +866,6 @@
 	}
 	return;
 
-err_kmalloc_max_ra:
-	kfree(bs);
 err_kmalloc_bs:
 	kfree(gd->part);
 err_kmalloc_gd_part:
@@ -937,7 +926,6 @@
 	init_gendisk(hwif);
 	blk_dev[hwif->major].data = hwif;
 	blk_dev[hwif->major].queue = ide_get_queue;
-	read_ahead[hwif->major] = 8;	/* (4kB) */
 	hwif->present = 1;	/* success */
 
 	return hwif->present;
diff -ur linux-2.5.4/drivers/ide/ide.c linux/drivers/ide/ide.c
--- linux-2.5.4/drivers/ide/ide.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/ide/ide.c	Fri Feb 15 09:27:01 2002
@@ -2126,7 +2126,6 @@
 	 */
 	unregister_blkdev(hwif->major, hwif->name);
 	kfree(blksize_size[hwif->major]);
-	kfree(max_readahead[hwif->major]);
 	blk_dev[hwif->major].data = NULL;
 	blk_dev[hwif->major].queue = NULL;
 	blk_clear(hwif->major);
diff -ur linux-2.5.4/drivers/md/lvm.c linux/drivers/md/lvm.c
--- linux-2.5.4/drivers/md/lvm.c	Mon Feb 11 02:50:08 2002
+++ linux/drivers/md/lvm.c	Fri Feb 15 09:27:01 2002
@@ -226,10 +226,6 @@
 
 #include "lvm-internal.h"
 
-#define	LVM_CORRECT_READ_AHEAD( a) \
-   if      ( a < LVM_MIN_READ_AHEAD || \
-             a > LVM_MAX_READ_AHEAD) a = LVM_MAX_READ_AHEAD;
-
 #ifndef WRITEA
 #  define WRITEA WRITE
 #endif
@@ -883,29 +879,6 @@
 		invalidate_buffers(inode->i_rdev);
 		break;
 
-
-	case BLKRASET:
-		/* set read ahead for block device */
-		if (!capable(CAP_SYS_ADMIN)) return -EACCES;
-
-		P_IOCTL("BLKRASET: %ld sectors for %s\n",
-			(long) arg, kdevname(inode->i_rdev));
-
-		if ((long) arg < LVM_MIN_READ_AHEAD ||
-		    (long) arg > LVM_MAX_READ_AHEAD)
-			return -EINVAL;
-		lv_ptr->lv_read_ahead = (long) arg;
-		break;
-
-
-	case BLKRAGET:
-		/* get current read ahead setting */
-		P_IOCTL("BLKRAGET %d\n", lv_ptr->lv_read_ahead);
-		if (put_user(lv_ptr->lv_read_ahead, (long *)arg))
-			return -EFAULT;
-		break;
-
-
 	case HDIO_GETGEO:
 		/* get disk geometry */
 		P_IOCTL("%s -- lvm_blk_ioctl -- HDIO_GETGEO\n", lvm_name);
@@ -2035,7 +2008,6 @@
 	lvm_size[minor(lv_ptr->lv_dev)] = lv_ptr->lv_size >> 1;
 	vg_lv_map[minor(lv_ptr->lv_dev)].vg_number = vg_ptr->vg_number;
 	vg_lv_map[minor(lv_ptr->lv_dev)].lv_number = lv_ptr->lv_number;
-	LVM_CORRECT_READ_AHEAD(lv_ptr->lv_read_ahead);
 	vg_ptr->lv_cur++;
 	lv_ptr->lv_status = lv_status_save;
 
diff -ur linux-2.5.4/drivers/md/md.c linux/drivers/md/md.c
--- linux-2.5.4/drivers/md/md.c	Mon Feb 11 02:50:13 2002
+++ linux/drivers/md/md.c	Fri Feb 15 09:27:01 2002
@@ -1737,7 +1737,6 @@
 	register_disk(&md_gendisk, mk_kdev(MAJOR_NR,mdidx(mddev)),
 			1, &md_fops, md_size[mdidx(mddev)]<<1);
 
-	read_ahead[MD_MAJOR] = 1024;
 	return (0);
 }
 
@@ -2622,8 +2621,6 @@
 						(u64 *) arg);
 			goto done;
 
-		case BLKRAGET:
-		case BLKRASET:
 		case BLKFLSBUF:
 		case BLKBSZGET:
 		case BLKBSZSET:
@@ -3176,13 +3173,6 @@
 
 	sz += sprintf(page+sz, "\n");
 
-
-	sz += sprintf(page+sz, "read_ahead ");
-	if (read_ahead[MD_MAJOR] == INT_MAX)
-		sz += sprintf(page+sz, "not set\n");
-	else
-		sz += sprintf(page+sz, "%d sectors\n", read_ahead[MD_MAJOR]);
-
 	ITERATE_MDDEV(mddev,tmp) {
 		sz += sprintf(page + sz, "md%d : %sactive", mdidx(mddev),
 						mddev->pers ? "" : "in");
@@ -3622,7 +3612,6 @@
 	}
 	blksize_size[MAJOR_NR] = md_blocksizes;
 	blk_size[MAJOR_NR] = md_size;
-	max_readahead[MAJOR_NR] = md_maxreadahead;
 
 	dprintk("md: sizeof(mdp_super_t) = %d\n", (int)sizeof(mdp_super_t));
 
@@ -3658,9 +3647,6 @@
 	/* forward all md request to md_make_request */
 	blk_queue_make_request(BLK_DEFAULT_QUEUE(MAJOR_NR), md_make_request);
 
-
-	read_ahead[MAJOR_NR] = INT_MAX;
-
 	add_gendisk(&md_gendisk);
 
 	md_recovery_thread = md_register_thread(md_do_recovery, NULL, name);
diff -ur linux-2.5.4/drivers/message/i2o/i2o_block.c linux/drivers/message/i2o/i2o_block.c
--- linux-2.5.4/drivers/message/i2o/i2o_block.c	Mon Feb 11 02:50:11 2002
+++ linux/drivers/message/i2o/i2o_block.c	Fri Feb 15 09:27:01 2002
@@ -1104,8 +1104,6 @@
 		case BLKFLSBUF:
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKPG:
 			return blk_ioctl(inode->i_rdev, cmd, arg);
 			
diff -ur linux-2.5.4/drivers/s390/block/dasd.c linux/drivers/s390/block/dasd.c
--- linux-2.5.4/drivers/s390/block/dasd.c	Mon Feb 11 02:50:16 2002
+++ linux/drivers/s390/block/dasd.c	Fri Feb 15 09:27:01 2002
@@ -2489,8 +2489,6 @@
 	case BLKSSZGET:
 	case BLKROSET:
 	case BLKROGET:
-	case BLKRASET:
-	case BLKRAGET:
 	case BLKFLSBUF:
 	case BLKPG:
 	case BLKELVGET:
diff -ur linux-2.5.4/drivers/s390/block/xpram.c linux/drivers/s390/block/xpram.c
--- linux-2.5.4/drivers/s390/block/xpram.c	Mon Feb 11 02:50:11 2002
+++ linux/drivers/s390/block/xpram.c	Fri Feb 15 09:27:01 2002
@@ -163,12 +163,11 @@
 
 static int major    = XPRAM_MAJOR;
 static int devs     = XPRAM_DEVS;
-static int rahead   = XPRAM_RAHEAD;
 static int sizes[XPRAM_MAX_DEVS] = { 0, };
 static int blksize  = XPRAM_BLKSIZE;
 static int hardsect = XPRAM_HARDSECT;
 
-int xpram_devs, xpram_rahead;
+int xpram_devs;
 int xpram_blksize, xpram_hardsect;
 int xpram_mem_avail = 0;
 unsigned long xpram_sizes[XPRAM_MAX_DEVS];
@@ -659,26 +658,9 @@
 		if ( capable(CAP_SYS_ADMIN) )invalidate_buffers(inode->i_rdev);
 		return 0;
 
-	case BLKRAGET: /* return the readahead value, 0x1263 */
-		if (!arg)  return -EINVAL;
-		err = 0; /* verify_area_20(VERIFY_WRITE, (long *) arg, sizeof(long));
-		          * if (err) return err;
-                          */
-		put_user(read_ahead[MAJOR(inode->i_rdev)], (long *)arg);
-
-		return 0;
-
-	case BLKRASET: /* set the readahead value, 0x1262 */
-		if (!capable(CAP_SYS_ADMIN)) return -EACCES;
-		if (arg > 0xff) return -EINVAL; /* limit it */
-		read_ahead[MAJOR(inode->i_rdev)] = arg;
-                atomic_eieio();
-		return 0;
-
 	case BLKRRPART: /* re-read partition table: can't do it, 0x1259 */
 		return -EINVAL;
 
-
 #if (XPRAM_VERSION == 22)
 		RO_IOCTLS(inode->i_rdev, arg); /* the default RO operations 
                                                 * BLKROSET
@@ -940,7 +922,6 @@
 				 * snoozing with a debugger.
 				 */
 
-	xpram_rahead   = rahead;
 	xpram_blksize  = blksize;
 	xpram_hardsect = hardsect;
 
@@ -1029,7 +1010,7 @@
 	PRINT_INFO("  %d kB expanded memory found.\n",xpram_mem_avail );
 
 	/*
-	 * Assign the other needed values: request, rahead, size, blksize,
+	 * Assign the other needed values: request, size, blksize,
 	 * hardsect. All the minor devices feature the same value.
 	 * Note that `xpram' defines all of them to allow testing non-default
 	 * values. A real device could well avoid setting values in global
@@ -1042,7 +1023,6 @@
 	q = BLK_DEFAULT_QUEUE (major);
 	blk_init_queue (q, xpram_request);
 #endif /* V22/V24 */
-	read_ahead[major] = xpram_rahead;
 
 	/* we want to have XPRAM_UNUSED blocks security buffer between devices */
 	mem_usable=xpram_mem_avail-(XPRAM_UNUSED*(xpram_devs-1));
@@ -1181,7 +1161,6 @@
 	kfree(xpram_hardsects);
 	hardsect_size[major] = NULL;
  fail_malloc:
-	read_ahead[major] = 0;
 #if (XPRAM_VERSION == 22)
 	blk_dev[major].request_fn = NULL;
 #endif /* V22 */
diff -ur linux-2.5.4/drivers/s390/char/tapeblock.c linux/drivers/s390/char/tapeblock.c
--- linux-2.5.4/drivers/s390/char/tapeblock.c	Mon Feb 11 02:50:15 2002
+++ linux/drivers/s390/char/tapeblock.c	Fri Feb 15 09:27:01 2002
@@ -101,7 +101,6 @@
     }
     if (tapeblock_major == 0) tapeblock_major = result;   /* accept dynamic major number*/
     INIT_BLK_DEV(tapeblock_major,tape_request_fn,tapeblock_getqueue,NULL);
-    read_ahead[tapeblock_major]=TAPEBLOCK_READAHEAD;
     PRINT_WARN(KERN_ERR " tape gets major %d for block device\n", result);
     blk_size[tapeblock_major] = (int*) kmalloc (256*sizeof(int),GFP_ATOMIC);
     memset(blk_size[tapeblock_major],0,256*sizeof(int));
diff -ur linux-2.5.4/drivers/scsi/ide-scsi.c linux/drivers/scsi/ide-scsi.c
--- linux-2.5.4/drivers/scsi/ide-scsi.c	Mon Feb 11 02:50:08 2002
+++ linux/drivers/scsi/ide-scsi.c	Fri Feb 15 09:27:01 2002
@@ -259,11 +259,10 @@
 	printk("]\n");
 }
 
-static void idescsi_end_request (byte uptodate, ide_hwgroup_t *hwgroup)
+static int idescsi_end_request(ide_drive_t *drive, int uptodate)
 {
-	ide_drive_t *drive = hwgroup->drive;
 	idescsi_scsi_t *scsi = drive->driver_data;
-	struct request *rq = hwgroup->rq;
+	struct request *rq = HWGROUP(drive)->rq;
 	idescsi_pc_t *pc = (idescsi_pc_t *) rq->special;
 	int log = test_bit(IDESCSI_LOG_CMD, &scsi->log);
 	struct Scsi_Host *host;
@@ -271,8 +270,8 @@
 	unsigned long flags;
 
 	if (!(rq->flags & REQ_SPECIAL)) {
-		ide_end_request (uptodate, hwgroup);
-		return;
+		ide_end_request(drive, uptodate);
+		return 0;
 	}
 	ide_end_drive_cmd (drive, 0, 0);
 	if (rq->errors >= ERROR_MAX) {
@@ -302,6 +301,8 @@
 	idescsi_free_bio (rq->bio);
 	kfree(pc); kfree(rq);
 	scsi->pc = NULL;
+
+	return 0;
 }
 
 static inline unsigned long get_timeout(idescsi_pc_t *pc)
@@ -341,7 +342,7 @@
 		ide__sti();
 		if (status & ERR_STAT)
 			rq->errors++;
-		idescsi_end_request (1, HWGROUP(drive));
+		idescsi_end_request(drive, 1);
 		return ide_stopped;
 	}
 	bcount = IN_BYTE (IDE_BCOUNTH_REG) << 8 | IN_BYTE (IDE_BCOUNTL_REG);
@@ -470,7 +471,7 @@
 		return idescsi_issue_pc (drive, (idescsi_pc_t *) rq->special);
 	}
 	blk_dump_rq_flags(rq, "ide-scsi: unsup command");
-	idescsi_end_request (0,HWGROUP (drive));
+	idescsi_end_request(drive, 0);
 	return ide_stopped;
 }
 
@@ -541,7 +542,6 @@
  */
 static ide_driver_t idescsi_driver = {
 	name:			"ide-scsi",
-	version:		IDESCSI_VERSION,
 	media:			ide_scsi,
 	busy:			0,
 	supports_dma:		1,
diff -ur linux-2.5.4/drivers/scsi/sd.c linux/drivers/scsi/sd.c
--- linux-2.5.4/drivers/scsi/sd.c	Fri Feb 15 09:26:27 2002
+++ linux/drivers/scsi/sd.c	Fri Feb 15 09:27:01 2002
@@ -228,8 +228,6 @@
 		case BLKGETSIZE64:
 		case BLKROSET:
 		case BLKROGET:
-		case BLKRASET:
-		case BLKRAGET:
 		case BLKFLSBUF:
 		case BLKSSZGET:
 		case BLKPG:
@@ -1190,18 +1188,6 @@
 				rscsi_disks[i].has_part_table = 1;
 			}
 		}
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 60 blocks (120 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. We can only respect this with the
-	 * granularity of every 16 disks (one device major).
-	 */
-	for (i = 0; i < N_USED_SD_MAJORS; i++) {
-		read_ahead[SD_MAJOR(i)] =
-		    (rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device
-		     && rscsi_disks[i * SCSI_DISKS_PER_MAJOR].device->host->sg_tablesize)
-		    ? 120	/* 120 sector read-ahead */
-		    : 4;	/* 4 sector read-ahead */
-	}
 
 	return;
 }
diff -ur linux-2.5.4/drivers/scsi/sr.c linux/drivers/scsi/sr.c
--- linux-2.5.4/drivers/scsi/sr.c	Mon Feb 11 02:50:12 2002
+++ linux/drivers/scsi/sr.c	Fri Feb 15 09:27:01 2002
@@ -785,16 +785,6 @@
                                     &sr_bdops, NULL);
 		register_cdrom(&scsi_CDs[i].cdi);
 	}
-
-
-	/* If our host adapter is capable of scatter-gather, then we increase
-	 * the read-ahead to 16 blocks (32 sectors).  If not, we use
-	 * a two block (4 sector) read ahead. */
-	if (scsi_CDs[0].device && scsi_CDs[0].device->host->sg_tablesize)
-		read_ahead[MAJOR_NR] = 32;	/* 32 sector read-ahead.  Always removable. */
-	else
-		read_ahead[MAJOR_NR] = 4;	/* 4 sector read-ahead */
-
 }
 
 static void sr_detach(Scsi_Device * SDp)
@@ -846,7 +836,6 @@
 		kfree(sr_blocksizes);
 		sr_blocksizes = NULL;
 	}
-	read_ahead[MAJOR_NR] = 0;
 	blk_clear(MAJOR_NR);
 
 	sr_template.dev_max = 0;
diff -ur linux-2.5.4/drivers/scsi/sr_ioctl.c linux/drivers/scsi/sr_ioctl.c
--- linux-2.5.4/drivers/scsi/sr_ioctl.c	Mon Feb 11 02:50:09 2002
+++ linux/drivers/scsi/sr_ioctl.c	Fri Feb 15 09:27:01 2002
@@ -550,8 +550,6 @@
 		return put_user((u64)scsi_CDs[target].capacity << 9, (u64 *)arg);
 	case BLKROSET:
 	case BLKROGET:
-	case BLKRASET:
-	case BLKRAGET:
 	case BLKFLSBUF:
 	case BLKSSZGET:
 		return blk_ioctl(cdi->dev, cmd, arg);
diff -ur linux-2.5.4/fs/hfs/file.c linux/fs/hfs/file.c
--- linux-2.5.4/fs/hfs/file.c	Fri Feb 15 09:26:31 2002
+++ linux/fs/hfs/file.c	Fri Feb 15 09:27:01 2002
@@ -164,8 +164,7 @@
 	if (left <= 0) {
 		return 0;
 	}
-	if ((read = hfs_do_read(inode, HFS_I(inode)->fork, pos,
-				buf, left, filp->f_reada != 0)) > 0) {
+	if ((read = hfs_do_read(inode, HFS_I(inode)->fork, pos, buf, left)) > 0) {
 	        *ppos += read;
 		filp->f_reada = 1;
 	}
@@ -292,7 +291,7 @@
  * It has been changed to take into account that HFS files have no holes.
  */
 hfs_s32 hfs_do_read(struct inode *inode, struct hfs_fork * fork, hfs_u32 pos,
-		    char * buf, hfs_u32 count, int reada)
+		    char * buf, hfs_u32 count)
 {
 	kdev_t dev = inode->i_dev;
 	hfs_s32 size, chars, offset, block, blocks, read = 0;
@@ -313,14 +312,6 @@
 	blocks = (count+offset+HFS_SECTOR_SIZE-1) >> HFS_SECTOR_SIZE_BITS;
 
 	bhb = bhe = buflist;
-	if (reada) {
-		if (blocks < read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9)) {
-			blocks = read_ahead[major(dev)] / (HFS_SECTOR_SIZE>>9);
-		}
-		if (block + blocks > size) {
-			blocks = size - block;
-		}
-	}
 
 	/* We do this in a two stage process.  We first try and
 	   request as many blocks as we can, then we wait for the
diff -ur linux-2.5.4/include/linux/blkdev.h linux/include/linux/blkdev.h
--- linux-2.5.4/include/linux/blkdev.h	Mon Feb 11 02:50:12 2002
+++ linux/include/linux/blkdev.h	Fri Feb 15 09:27:01 2002
@@ -314,11 +314,8 @@
 extern void generic_unplug_device(void *);
 
 extern int * blk_size[MAX_BLKDEV];
-
 extern int * blksize_size[MAX_BLKDEV];
 
-extern int * max_readahead[MAX_BLKDEV];
-
 #define MAX_PHYS_SEGMENTS 128
 #define MAX_HW_SEGMENTS 128
 #define MAX_SECTORS 255
@@ -340,8 +337,6 @@
 	blk_size_in_bytes[major] = NULL;
 #endif
 	blksize_size[major] = NULL;
-	max_readahead[major] = NULL;
-	read_ahead[major] = 0;
 }
 
 extern inline int get_hardsect_size(kdev_t dev)
diff -ur linux-2.5.4/include/linux/fs.h linux/include/linux/fs.h
--- linux-2.5.4/include/linux/fs.h	Fri Feb 15 09:26:32 2002
+++ linux/include/linux/fs.h	Fri Feb 15 09:27:01 2002
@@ -173,10 +173,12 @@
 #define BLKRRPART  _IO(0x12,95)	/* re-read partition table */
 #define BLKGETSIZE _IO(0x12,96)	/* return device size /512 (long *arg) */
 #define BLKFLSBUF  _IO(0x12,97)	/* flush buffer cache */
-#define BLKRASET   _IO(0x12,98)	/* Set read ahead for block device */
+#if 0				/* Obsolete, these don't do anything. */
+#define BLKRASET   _IO(0x12,98)	/* set read ahead for block device */
 #define BLKRAGET   _IO(0x12,99)	/* get current read ahead setting */
 #define BLKFRASET  _IO(0x12,100)/* set filesystem (mm/filemap.c) read-ahead */
 #define BLKFRAGET  _IO(0x12,101)/* get filesystem (mm/filemap.c) read-ahead */
+#endif
 #define BLKSECTSET _IO(0x12,102)/* set max sectors per request (ll_rw_blk.c) */
 #define BLKSECTGET _IO(0x12,103)/* get max sectors per request (ll_rw_blk.c) */
 #define BLKSSZGET  _IO(0x12,104)/* get block device sector size */
@@ -1492,8 +1494,6 @@
 
 extern ssize_t char_read(struct file *, char *, size_t, loff_t *);
 extern ssize_t block_read(struct file *, char *, size_t, loff_t *);
-extern int read_ahead[];
-
 extern ssize_t char_write(struct file *, const char *, size_t, loff_t *);
 extern ssize_t block_write(struct file *, const char *, size_t, loff_t *);
 
diff -ur linux-2.5.4/kernel/ksyms.c linux/kernel/ksyms.c
--- linux-2.5.4/kernel/ksyms.c	Fri Feb 15 09:26:32 2002
+++ linux/kernel/ksyms.c	Fri Feb 15 09:27:01 2002
@@ -323,7 +323,6 @@
 EXPORT_SYMBOL(tq_disk);
 EXPORT_SYMBOL(init_buffer);
 EXPORT_SYMBOL(refile_buffer);
-EXPORT_SYMBOL(max_readahead);
 EXPORT_SYMBOL(wipe_partitions);
 
 /* tty routines */
@@ -525,7 +524,6 @@
 EXPORT_SYMBOL(clear_inode);
 EXPORT_SYMBOL(___strtok);
 EXPORT_SYMBOL(init_special_inode);
-EXPORT_SYMBOL(read_ahead);
 EXPORT_SYMBOL(__get_hash_table);
 EXPORT_SYMBOL(new_inode);
 EXPORT_SYMBOL(insert_inode_hash);
diff -ur linux-2.5.4/mm/filemap.c linux/mm/filemap.c
--- linux-2.5.4/mm/filemap.c	Mon Feb 11 02:50:14 2002
+++ linux/mm/filemap.c	Fri Feb 15 09:27:01 2002
@@ -1120,7 +1120,7 @@
  *
  * Asynchronous read-ahead risks:
  * ------------------------------
- * In order to maximize overlapping, we must start some asynchronous read 
+ * In order to maximize overlapping, we must start some asynchronous read
  * request from the device, as soon as possible.
  * We must be very careful about:
  * - The number of effective pending IO read requests.
@@ -1131,13 +1131,6 @@
  *   64k if defined (4K page size assumed).
  */
 
-static inline int get_max_readahead(struct inode * inode)
-{
-	if (kdev_none(inode->i_dev) || !max_readahead[major(inode->i_dev)])
-		return MAX_READAHEAD;
-	return max_readahead[major(inode->i_dev)][minor(inode->i_dev)];
-}
-
 static void generic_file_readahead(int reada_ok,
 	struct file * filp, struct inode * inode,
 	struct page * page)
@@ -1146,7 +1139,6 @@
 	unsigned long index = page->index;
 	unsigned long max_ahead, ahead;
 	unsigned long raend;
-	int max_readahead = get_max_readahead(inode);
 
 	end_index = inode->i_size >> PAGE_CACHE_SHIFT;
 
@@ -1231,8 +1223,8 @@
 
 		filp->f_ramax += filp->f_ramax;
 
-		if (filp->f_ramax > max_readahead)
-			filp->f_ramax = max_readahead;
+		if (filp->f_ramax > MAX_READAHEAD)
+			filp->f_ramax = MAX_READAHEAD;
 
 #ifdef PROFILE_READAHEAD
 		profile_readahead((reada_ok == 2), filp);
@@ -1278,7 +1270,6 @@
 	struct page *cached_page;
 	int reada_ok;
 	int error;
-	int max_readahead = get_max_readahead(inode);
 
 	cached_page = NULL;
 	index = *ppos >> PAGE_CACHE_SHIFT;
@@ -1318,9 +1309,9 @@
 			filp->f_ramax = needed;
 
 		if (reada_ok && filp->f_ramax < MIN_READAHEAD)
-				filp->f_ramax = MIN_READAHEAD;
-		if (filp->f_ramax > max_readahead)
-			filp->f_ramax = max_readahead;
+			filp->f_ramax = MIN_READAHEAD;
+		if (filp->f_ramax > MAX_READAHEAD)
+			filp->f_ramax = MAX_READAHEAD;
 	}
 
 	for (;;) {
@@ -1808,8 +1799,7 @@
 {
 	unsigned long ra_window;
 
-	ra_window = get_max_readahead(vma->vm_file->f_dentry->d_inode);
-	ra_window = CLUSTER_OFFSET(ra_window + CLUSTER_PAGES - 1);
+	ra_window = CLUSTER_OFFSET(MAX_READAHEAD + CLUSTER_PAGES - 1);
 
 	/* vm_raend is zero if we haven't read ahead in this area yet.  */
 	if (vma->vm_raend == 0)

Re: PATCH 2.5.4 i810_audio, bttv, working at all.

[Alan Cox]

>  		{
> -			sg->busaddr=virt_to_bus(dmabuf->rawbuf+dmabuf->fragsize*i);
> +			sg->busaddr=virt_to_phys(dmabuf->rawbuf+dmabuf->fragsize*i);

No don't do this. The code needs changing to use the new PCI API

Re: A modest proposal -- We need a patch penguin

[Rob Landley]

On Monday 28 January 2002 08:37 pm, Francesco Munda wrote:
> On Mon, 28 Jan 2002 09:10:56 -0500
>
> Rob Landley <landley@trommello.org> wrote:
> > Patch Penguin Proposal.
> >
> > [...]
>
> You mean some sort of proxy/two-tier development? A "commit/rollback"
> transaction model on the kernel itself?

Think how Alan Cox's tree used to work.  Just because Alan accepted a patch 
didn't guarantee Linus wasn't going to come up with a reason to shoot it 
down.  It just meant the patch wasn't going to be ignored, and if it WAS 
dropped there would probably going to be some kind of explanation.

Whether the patch penguin wants to use some kind of tool to maintain their 
tree (like CVS) with a "commit/rollback" model is a seperate issue.  Linus 
isn't going to use it, and linus isn't going to have to see it.  Linus gets 
the kid of patches he likes, which have already had merge clashes and the 
really obvious thinkos resolved before he sees them, and have probably even 
been tested by the foolhardy individuals currently downloading the -ac, -dj, 
and -aa trees.

Right now, Alan's tree is in the process of going back into circulation.  He 
tells me that his tree is basically a delta against marcello (2.4), and DJ is 
doing a delta against linus (2.5).  Over time, the need for a 2.4 delta will 
probably diminish as new development shifts over to 2.5.  Right now, the 
patch constipation we've been seeing is, in my opinion, directing development 
to occur against 2.4 that should at the very least be eyeing 2.5.  (Alan is 
probably NOT interested in integrating patches that Marcelo has no intention 
of eventually integrating into 2.5.  So he's not taking the new development 
integration pressure off, that's DJ's job.)

I think DJ could definitely use a clearer mandate.

> I deeply agree with you, especially in keeping "many eyes" to look at the
> same kernel tree, and not chosing one of the many subtrees; as added bonus,
> this stuff is buzzword compliant! What we can ask more? :)
>
> Now, Linus' call to accept _your_ patch. Fingers crossed already.

I'm getting a lot more support off the list than on the list.  People seem to 
be afraid to cc: linux-kernel.  I underestimated how deeply steeped in 
politics this issue seems to have become.  It seems a fairly straightforward 
optmiziation, mainly a clarification of of the way things have been done in 
the past and a formalization of a position that got a bit confused in the 
transition from one officeholder to another.

Before posting here, I bounced an earlier draft off of both Alan Cox and Dave 
Jones.  Alan's response was, and I quote:

> I'm certainly fine with DaveJ being the victim 8)

Dave didn't seem to have any major objections but raised a lot technical 
points to the effect of "I'm already doing this bit".  Both of them gave me 
permission to post most of our conversation to the list, but seem unwilling 
to do it themselves. :)

I've gotten several other agreements, some from people trying to find an 
off-list place we could discuss it (okay, so what's THIS list for again)?  
And one person, who shall remain nameless (at least as long as he refuses to 
speak for himself. :) brought up the subject of Linus co-designing bitkeeper 
way back when to cope with exactly some of these problems.

Bitkeeper is a technical tool attempting to deal with a social problem.  
Merging patches, resolving conflicts between them, testing them, and keeping 
them current as the tree changes under them requires programmer work.  A 
human needs to do it.  Whether that human uses bitkeeper, CVS, a directory 
full of patch files, or manually keeps all the patches in printouts in a shoe 
box is a side issue.  A human can feed Linus better patches than any software 
tool possibly could.

Now if the patch penguin wants to use bitkeeper for his own internal 
patch-wrangling, that's a seperate issue.  One you should take up with the 
patch penguin, once we have one.  (Of course the developer community and the 
maintainers might exert some pressure on the patch penguin to use CVS, but 
how is this a bad thing from Linus's perspective: it means they'e NOT bugging 
HIM about using CVS anymore.  And again, this is an enhancement/detail that 
can be resolved later.)

As for attracting Linus's attention, there's a penguin and egg problem here: 
without an integration lieutenant Linus is largely too swamped to reliably be 
aware of this kind of thread on the list, so how can he get the suggestion to 
anoint someone with holy penguin pee to basically act as his secretary and 
clean up this mess of patches so he can properly sort through them once 
they've been organized and laid out in front of him in nice neat rows.  Hence 
the drive to get people to agree to it so the thread grows large enough to 
attract Linus's attention, and also passes his "it's been discussed enough to 
find any particularly obvious holes with it" filter...

So everybody who thinks this is a good idea, please say so.  Those who don't 
like it, please say so too so the objection can be aired and maybe resolved.  
The core idea here really is to save Linus time and effort.  Everything else 
is either a direct consequence of that, or a fringe benefit.

> -- FM

Rob

Re: A modest proposal -- We need a patch penguin

[Dave Jones]

On Mon, Jan 28, 2002 at 10:42:19PM -0500, Rob Landley wrote:
 
 > probably diminish as new development shifts over to 2.5.  Right now, the 
 > patch constipation we've been seeing is, in my opinion, directing development 
 > to occur against 2.4 that should at the very least be eyeing 2.5.  (Alan is 
 > probably NOT interested in integrating patches that Marcelo has no intention 
 > of eventually integrating into 2.5.  So he's not taking the new development 
 > integration pressure off, that's DJ's job.)
 > 
 > I think DJ could definitely use a clearer mandate.

 * Initially, -dj was "pick up fixes from 2.4".

 * Then when Linus broke various other parts of 2.5, I took fixes
   for various bits. (Some of those went back his way, others didn't,
   others are still in the process)
   (I'm a believer in the 'eat your own dogfood' thing, and run my
   tree on several testboxes -- being able to compile/boot/test
   this tree became more important at the cost of the tree growing
   a little further away from -linus)

 * Some developers also wanting to develop against 2.5 found the
   quickest way to get a compilable, workable 2.5 tree was to
   grab my snapshot, and work against my tree until Linus gets his
   together.  And hence, the input layer & fb layer changes.
   This was one I had to think about a bit before deciding if
   I was going to start accepting such patches.
   In theory, as we're now in 2.5, there should be no need for this,
   but whilst Linus is busy focusing on the block layer, scheduler
   or other flavour of the week, James, Vojtech etc can at least
   get some extra testers before their code hits -linus.
   By the time that the new input/fb stuff is ready for Linus' tree
   hopefully a lot of the more obvious problems will be shaken out,
   and Linus can have a set of patches for a "new xxx layer" that
   works for at least everyone who's been testing it in -dj.

 Where to go from here? More of the same. It's a fulltime job
 keeping up with Marcelo & Linus, and reviewing, merging, and
 chasing down the right people.  One thing I'm not entirely 
 enthusiastic about doing, is making policy decisions.
 I've had questions from people asking me if I'll merge xxx's
 implementation of ACLs for example. Without knowing which way
 Linus is going to turn on such an issue, I'm naturally hesitant.

 Another thing of note is that the merge process with Linus
 isn't as straightforward as running splitdiff, and pushing the
 chunks to Linus. Some bits require a timing (although this is
 sometimes hard to get right) so I can push him filesystem
 changes when Al isn't turning the VFS upside down for eg.
 Other bits I won't push because maintainers have mailed me
 asking me not to.  And other bits, because the maintainers
 can do a better job of splitting,pushing and describing than
 I can (typical example: the fbdev/input stuff)
 
 > Dave didn't seem to have any major objections but raised a lot technical 
 > points to the effect of "I'm already doing this bit".  Both of them gave me 
 > permission to post most of our conversation to the list, but seem unwilling 
 > to do it themselves. :)

 Time, Headcold, time, blah, excuses 8)
 But to reiterate, yes. Most of what you described is exactly whats
 taking place, although a lot of it happens behind the scenes, not
 on Linux-kernel, not on irc, but me being a pita chasing maintainers
 "Hey xxx sent me a patch, aren't you working on this? You two should
  talk..". It's like being a switchboard operator at times, plugging
 in the right cables, connecting the right people.
 
 > As for attracting Linus's attention, there's a penguin and egg problem here: 
 > without an integration lieutenant Linus is largely too swamped to reliably be 
 > aware of this kind of thread on the list

 Linus' concern that people don't scale is perhaps not unfounded.
 Since I started doing this, the number of hours involved has increased
 on a day by day basis. If there comes a time where >I'm< not scaling
 and start dropping patches, then maybe an extra tier is needed. *shrug*
 For now at least, things seem to be working out quite well on the whole.
 I'm not aware of any particularly important fix/cleanup that has been 
 dropped on the floor since I started scooping them up.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: A modest proposal -- We need a patch penguin

[Padraig Brady]

Francesco Munda wrote:

> On Mon, 28 Jan 2002 09:10:56 -0500
> Rob Landley <landley@trommello.org> wrote:
> 
> 
>>Patch Penguin Proposal.
>>
>>[...]
>>
> 
> You mean some sort of proxy/two-tier development? A "commit/rollback"
> transaction model on the kernel itself?


Dave Jones described the current model very succinctly in:

http://marc.theaimsgroup.com/?l=linux-kernel&m=100966905916285&w=2

He also mentioned a big problem. People not honouring/realising
there position in the tree, (trying to get in the ChangeLog?).
True, the only way to scale it is add another level at the current
bottleneck, but this must be more than 1 person or it won't help,
as it'll just move the bottelneck back a little.

Personally I think automated tools (like bitkeeper) would help
more than another level in the hierarchy.

Currently the way I see it [should be] currently is:

random hackers
| | | | | | |
| maintainers
| | | |
combiners
| |
Linus

I.E. Linus just gets input from the combiners which
test logic from the maintainers in combination. Also
random hackers should input to the combiners and not Linus
if there isn't an appropriate maintainer for their code.

Padraig.

Re: A modest proposal -- We need a patch penguin

[Francesco Munda]

On Tue, 29 Jan 2002 12:23:26 +0000
Padraig Brady <padraig@antefacto.com> wrote:

> Currently the way I see it [should be] currently is:
> 
> [cut-n-pasted graph]
> 
> I.E. Linus just gets input from the combiners which
> test logic from the maintainers in combination. Also
> random hackers should input to the combiners and not Linus
> if there isn't an appropriate maintainer for their code.

Quite descriptive and useful, thanks.

Let me raise a point. And extend your graph:

random hackers
| | | | | | |
| maintainers -< subsys testers
| | | |
combiners -< tree testers
| |
Linus

Who you call combiners... How many of them should release independent trees
to be thrown at us test-dogs? My point of view is neither the hacker, nor the
maintainer nor the combiner one. Nor Linus, thank god! :) It's the guy who
risks his filesystem integrity with some 2.X.Y-preZ-testW-QQ-KK kernel.

How many crosspatched sources I should look at, to try my luck with?

Have fun,

-- Francesco

Re: A modest proposal -- We need a patch penguin

[Andrew Pimlott]

Rob, you make a nice case, but consider a few points.

One,

> This integration and patch tracking work is a fairly boring, thankless task, 
> but it's work somebody other than Linus can do, which Linus has to do 
> otherwise. (And which Linus is NOT doing a good job at right now.)

... are you _sure_ that Linus does this?  My sense is that he mostly
eschews integration grunt-work.  If that is so, it's possible that
Linus is already operating near top efficiency, and that his
throughput is as high as he wants it to be!  Linus has pointed out
more than once that a big part of his job is to limit change.  Maybe
he's happy with the current rate of change in 2.5.  (That doesn't
mean everything is optimal--he might wish for higher quality changes
or a different mix of changes, just not more.)

Two, Linus has argued that maintainers are his patch penguins;
whereas you favor a single integration point between the maintainers
and Linus.  This has advantages and disadvantages, but on the whole,
I think it is better if Linus works directly with subsystem
maintainers.  To the extent that Linux is modular, there is little
need for the extra layer (so it is just overhead).  And when there
is a real conflict between subsystems--that's probably just the time
when Linus and the maintainers need to be collaborating directly!
The only "but" is that many people find it hard to work with Linus.
However, Linus made clear in his message that he considers this a
solvable problem (and maybe one you should work on!).

> Finished code 
> regularly goes unintegrated for months at a time, being repeatedly resynced 
> and re-diffed against new trees until the code's maintainer gets sick of it. 

Assuming that your system doesn't dramatically increase Linus's
throughput, code will still have to be re-diffed.  I don't agree
that thrusting all the merging onto one person is the right
solution.  That person is a _much_ bigger scalability bottleneck
than Linus, because (by your definition of the role) he can't drop
patches!  So he will inevitably become overwhelmed, and then we have
a bigger mess.

Frankly, if I were a maintainer, I would want the patch that finally
gets integrated to be one that I produce, not one re-diffed by
someone less familiar with the subsystem.  So, I side with Linus's
"tough, that's part of the maintainer's job" stance.  (Now, tools to
help resync, and to eliminate the tedium of re-submitting to Linus
periodically, would be welcome.)

> Several of the bug fixes in Alan's tree (which he 
> stopped maintaining months ago) still are not present in 2.4.17 or 2.5.

This is plain evidence that a single integration point (and there is
no better than Alan) isn't a panacea.

Three, regarding your complaint about "clean-up" patches being
dropped: maybe this just means there is a maintainer missing from
the pantheon: the clean-up maintainer.

Andrew

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 06:51 am, Andrew Pimlott wrote:
> Three, regarding your complaint about "clean-up" patches being
> dropped: maybe this just means there is a maintainer missing from
> the pantheon: the clean-up maintainer.

Oh, you mean acme :-)

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> throughput is as high as he wants it to be!  Linus has pointed out
> more than once that a big part of his job is to limit change.  Maybe
> he's happy with the current rate of change in 2.5.  (That doesn't
> mean everything is optimal--he might wish for higher quality changes
> or a different mix of changes, just not more.)

Progress happens at its own rate. Linus can no more control rate of change
than you can put a waterfall into low gear. There is a difference between
refusing stuff where the quality is low and losing stuff which is clear
fixes

> Two, Linus has argued that maintainers are his patch penguins;
> whereas you favor a single integration point between the maintainers
> and Linus.  This has advantages and disadvantages, but on the whole,
> I think it is better if Linus works directly with subsystem

Perl I think very much shows otherwise. Right now we have a maze of partially 
integrated trees which overlap, clash when the people send stuff to Linus and
worse.

When you have one or two integrators you have a single tree pretty much everyone
builds new stuff from and which people maintain small diffs relative to. At
the end of the day that ends up like the older -ac tree, and with the same
conditions - notably that anything in it might be going to see /dev/null not
Linus if its shown to be flawed or not done well.

Alan

Re: A modest proposal -- We need a patch penguin

[Andrew Pimlott]

On Tue, Jan 29, 2002 at 01:06:09PM +0000, Alan Cox wrote:
> Andrew wrote:
> > Two, Linus has argued that maintainers are his patch penguins;
> > whereas you favor a single integration point between the maintainers
> > and Linus.  This has advantages and disadvantages, but on the whole,
> > I think it is better if Linus works directly with subsystem
> 
> Perl I think very much shows otherwise.

I'm really not sure about this example.  I assume you mean Perl 5.
Last I checked, Perl didn't really operate the way Rob suggests.
There is a "patch pumpking", but he is more analogous to Linus than
to Alan.  In particular, Larry Wall does not review the pumpking's
work at all (he instead sets general direction and makes key design
decisions).  If Perl doesn't have the problems observed in Linux, I
think it is because 1) Perl is smaller, 2) Perl 5 is largely in
bug-fix mode, 3) Perl has a culture of accepting patches with less
scrutiny (without meaning this as a slam, I think you can see
evidence of this in the Perl 5 source base).

> When you have one or two integrators you have a single tree pretty
> much everyone builds new stuff from and which people maintain
> small diffs relative to. At the end of the day that ends up like
> the older -ac tree, and with the same conditions - notably that
> anything in it might be going to see /dev/null not Linus if its
> shown to be flawed or not done well.

There is an upper bound to the size of the delta one person can
maintain (well, assuming his goal is to sync those changes with
Linus).  Unless Linus's throughput increases dramatically, the
integrator's delta will grow until it reaches that bound.  At that
point, the integrator has to drop patches (or give up!).  How do you
get around this?

Andrew

Re: A modest proposal -- We need a patch penguin

[Alan Cox]

> Linus).  Unless Linus's throughput increases dramatically, the
> integrator's delta will grow until it reaches that bound.  At that
> point, the integrator has to drop patches (or give up!).  How do you
> get around this?

Historically at that point the old maintainer is overthrown 8) Look at
Minix versus Linux, gcc versus egcs etc

Re: A modest proposal -- We need a patch penguin

[John Alvord]

On Tue, 29 Jan 2002 13:06:09 +0000 (GMT), Alan Cox
<alan@lxorguk.ukuu.org.uk> wrote:

>> throughput is as high as he wants it to be!  Linus has pointed out
>> more than once that a big part of his job is to limit change.  Maybe
>> he's happy with the current rate of change in 2.5.  (That doesn't
>> mean everything is optimal--he might wish for higher quality changes
>> or a different mix of changes, just not more.)
>
>Progress happens at its own rate. Linus can no more control rate of change
>than you can put a waterfall into low gear. There is a difference between
>refusing stuff where the quality is low and losing stuff which is clear
>fixes
>
>> Two, Linus has argued that maintainers are his patch penguins;
>> whereas you favor a single integration point between the maintainers
>> and Linus.  This has advantages and disadvantages, but on the whole,
>> I think it is better if Linus works directly with subsystem
>
>Perl I think very much shows otherwise. Right now we have a maze of partially 
>integrated trees which overlap, clash when the people send stuff to Linus and
>worse.
>
>When you have one or two integrators you have a single tree pretty much everyone
>builds new stuff from and which people maintain small diffs relative to. At
>the end of the day that ends up like the older -ac tree, and with the same
>conditions - notably that anything in it might be going to see /dev/null not
>Linus if its shown to be flawed or not done well.
>
Multiple integrator-trees dilute the tester pool, which is a major
limitation on progress.

john alvord

Re: A modest proposal -- We need a patch penguin

[Matthias Andree]

On Mon, 28 Jan 2002, Rob Landley wrote:

> The holder of the patch penguin would feed Linus good patches, by Linus's 
> standards. Not just tested ones, but small bite-sized patches, one per email 
> as plain text includes, with an explanation of what each patch does at the 
> top of the mail. (Just the way Linus likes them. :) Current pending patches 
> from the patch penguin tree could even be kept at a public place (like 
> kernel.org) so Linus could pull rather than push, and grab them when he has 
> time. The patch penguin tree would make sure that when Linus is ready for a 
> patch, the patch is ready for Linus.

Looks like a manual re-implementation of a bug/request/patch tracker
like sourceforge's, bugzilla or whatever, with some additions. A patch
is added to the system, it gets a version tag, and you just pull it, and
mark it closed if applied to Linus' tree. If Linus releases a new tree,
the patch is marked stale until the maintainer uploads an updated patch
or just reopens it to mark "still applies unchanged to new version". (No
CVS involved, BTW.)

> One reason Linus doesn't like CVS is he won't let other people check code 
> into his tree. (This is not a capricious decision on Linus's part: no 
> architect can function if he doesn't know what's in the system. Code review 
> of EVERYTHING is a vital part of Linus's job.) With a patch penguin tree, 

That's one of the arguments Peter Miller uses to endorse his "Aegis"
system: separate the integration part. FreeBSD also seem to follow a
similar idea: there are maintainers and committers. Maintainers usually
do not commit, but file PRs tagged as "maintainer-update".

(Note: I've never used aegis myself, it seems as though it had some
implications with development spread to various locations, someone
comment on that.)

Also, I'm not sure how good Bitkeeper fits here, or whether subversion
will help in this way (one might consider feeding suggestions to the
subversion team, http://subversion.tigris.org/, if they do atomic
commits, one might consider holding them off until blessed by an
integrator).

-- 
Matthias Andree

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."         Benjamin Franklin

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 10:55 am, Matthias Andree wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:
> 
> > The holder of the patch penguin would feed Linus good patches, by Linus's 
> > standards. Not just tested ones, but small bite-sized patches, one per email 
> > as plain text includes, with an explanation of what each patch does at the 
> > top of the mail. (Just the way Linus likes them. :) Current pending patches 
> > from the patch penguin tree could even be kept at a public place (like 
> > kernel.org) so Linus could pull rather than push, and grab them when he has 
> > time. The patch penguin tree would make sure that when Linus is ready for a 
> > patch, the patch is ready for Linus.
> 
> Looks like a manual re-implementation of a bug/request/patch tracker
> like sourceforge's, bugzilla or whatever, with some additions.

And you load a patch into it by emailing to the bot, not via the web
interface.  The web interface is just for a) reporting b) maintainance, i.e., 
closing out a patch that got applied in some altered form, or applied with no 
notification to the bot, or obsoleted.

> A patch
> is added to the system, it gets a version tag, and you just pull it, and
> mark it closed if applied to Linus' tree. If Linus releases a new tree,
> the patch is marked stale until the maintainer uploads an updated patch
> or just reopens it to mark "still applies unchanged to new version". (No
> CVS involved, BTW.)

Yes, very much yes.  This way it just looks like regular email to Linus - 
except for some hopefully useful bookkeeping gack prepended to the top of the 
mail by the bot - and doesn't change the way he works at all.

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jim McDonald]

On Tue, 2002-01-29 at 09:55, Matthias Andree wrote:
> On Mon, 28 Jan 2002, Rob Landley wrote:

[...]

> Also, I'm not sure how good Bitkeeper fits here, or whether subversion
> will help in this way (one might consider feeding suggestions to the
> subversion team, http://subversion.tigris.org/, if they do atomic
> commits, one might consider holding them off until blessed by an
> integrator).

I'm not sure that a CVS-type solution is going to fix the problem here. 
>From what I can see, the problems that people are bringing up are as
follows:

   - some patches sent to the list get dropped without comment
   - people are worried about Linus' scalability in handling patches
   - patches time-out quite quickly with the speed of development of
     the kernel, which results in patches not getting applied because
     by the time they get looked they have gone stale
   - people seem to often be unsure about where to send patches
     for unmaintained code, and with no direct maintainer it seems
     that these patches automatically fall outside of Linus'
     trusted kernel people and stand a very small chance of getting
     implemented

I must admit that I agree with Linus' position in most places, but the
result of that is two-fold: a lot of people are left in the dark as to
the state of their patches (ditched, pending, bad style, etc), and a lot
of patch work is duplicated as different people fix the same problem
every couple of months because the fixes are never applied.

We have two solutions - fix the way that all of this works, or try to
patch around the resultant problems.  It looks like number 1 is not
going to happen, so why not do something with 2?  There has been a lot
of talk about putting together a system that re-sends patches every
month or so to lkml, let's write something that does this.  We could get
a number of advantages from this:

   - identification of patches (cleanup, performance improvement, bug 
     fix, new functionality, ...)
   - automatic identification of responsible maintainer (direct from
     patch) to email on submission of patch
   - ability to automatically re-diff patch against latest kernel
     versions and get submitter to re-apply if required
   - simple rejection of patches with minimal effort from maintainers
   - easy way for wannabe-patchers to see what patches are already 
     pending so they can concentrate on other areas and not duplicate
     effort

Note that this isn't that far from SourceForge, but probably too far to
make it worthwhile trying to set it up as an SF project.  If we do this
I figure that at worst it allows for auto-resending of patches that have
slipped through the cracks, and at best it will give people a far more
suitable mechanism for patch submission and tracking than just email.

Comments?  I'm willing to write it if someone is willing to host it.

> Matthias Andree

Cheers,
Jim.
-- 
Jim McDonald - Jim@mcdee.net

Re: A modest proposal -- We need a patch penguin

[Eli Carter]

Rob Landley wrote:
> Patch Penguin Proposal.

Ok, I mailed Rob privately in support of this; now I'm saying it
publicly: I support this. (And I thought it well written.)

I've submitted small patches for things as I have gotten the chance.  I
submitted those to Alan because they were small fixes.  He gave feedback
(Thanks Alan!) and I tried to fix things better.  He took care of
feeding Linus--something I'm not in a possition to do.
I believe we need a patch-penguin or something similar.  Linus wants
subsystem maintainers... maybe make an official bugfix-maintainer? 
Whoever it is needs to be officially recognized by Linus and probably
featured on /. or something so people who create those 3-4 line patches
that fix a bug that bit them will know not to mail Linus.

(Returning to the shadows where I belong... ;) )

Eli
--------------------.     Real Users find the one combination of bizarre
Eli Carter           \ input values that shuts down the system for days.
eli.carter(a)inet.com `-------------------------------------------------

Re: A modest proposal -- We need a patch penguin

[Daniel Phillips]

On January 29, 2002 04:51 pm, Eli Carter wrote:
> I believe we need a patch-penguin or something similar.  Linus wants
> subsystem maintainers... maybe make an official bugfix-maintainer? 
> Whoever it is needs to be officially recognized by Linus and probably
> featured on /. or something so people who create those 3-4 line patches
> that fix a bug that bit them will know not to mail Linus.

That would be acme, wouldn't it?

-- 
Daniel

Re: A modest proposal -- We need a patch penguin

[Jordan Mendelson]

On Monday, January 28, 2002, at 06:10  AM, Rob Landley wrote:

> Okay everybody, this is getting rediculous. Patches FROM MAINTAINERS are
> getting dropped on the floor on a regular basis. This is burning out
> maintainers and is increasing the number of different kernel trees (not 
> yet a
> major fork, but a lot of cracks and fragmentation are showing under the
> stress). Linus needs an integration lieutenant, and he needs one NOW.

While I'm not going to debate the fact that maintainence of the kernel 
is suboptimal, I would like to point out that the system that has been 
put in place is the correct way of doing it if you have someone with the 
vision necessary to oversee development. There are many ways to go about 
developing software, but many of them were created because of the lack 
of a strong leader and fortunately the Linux community has one (many in 
fact).

> We need to create the office of "patch penguin", whose job would be to 
> make
> Linus's life easier by doing what Alan Cox used to do, and what Dave 
> Jones is
> unofficially doing right now. (In fact, I'd like to nominate Dave Jones 
> for
> the office, although it's Linus's decision and it would be nice if Dave 
> got
> Alan Cox's blessing as well.)

Having one person who looks after patches is just going to create 
another bottleneck. The real problem is not Linus not accepting patches, 
but the process by which the patches are sent to him and the general 
impatience of the developer community.

We all know the kernel is broken up into a set of subsystems and those 
subsystems have maintainers. What is lacking however appears to be the 
leadership by these maintainers to:

* Delegate ownership of components in their subsystems to others.

* Prioritize small easily code reviewable patches for submission to 
Linus with adequate information about the contents of the changes 
including who has reviewed it, who wrote it, why it should be included 
and it's testing status is.

* Provide feedback to patch creators about the status of their patch 
including reasons why it was denied, suggestions for improvement, etc.

Of course, this isn't universally true. There are many instances where, 
especially in the case of device drivers, delegation has taken place, 
but a formal process of accepting patches that is actually adhered to by 
independent developers still appears to be lacking.

> Linus doesn't scale, and his current way of coping is to silently drop 
> the
> vast majority of patches submitted to him onto the floor. Most of the 
> time
> there is no judgement involved when this code gets dropped. Patches 
> that fix
> compile errors get dropped. Code from subsystem maintainers that Linus
> himself designated gets dropped. A build of the tree now spits out 
> numerous
> easily fixable warnings, when at one time it was warning-free. Finished 
> code
> regularly goes unintegrated for months at a time, being repeatedly 
> resynced
> and re-diffed against new trees until the code's maintainer gets sick 
> of it.
> This is extremely frustrating to developers, users, and vendors, and is
> burning out the maintainers. It is a huge source of unnecessary work. 
> The
> situation needs to be resolved. Fast.

I ague that the vast majority of patches submitted to Linus should be 
dropped on the floor. There is no reason why a compilation bug fix for 
an IDE chipset driver, PCI subsystem layer or even top-level Makefile 
should be sent directly to Linus by an independent developer.

The amount of time necessary to walk through code submitted by those who 
you don't know, don't trust and simply don't have an understanding of 
proper kernel development makes submitting patches directly to Linus 
like calling up the President to contest the suspension of your child 
from school. You may have a valid fix for a bug, but the system simply 
doesn't work if everyone decides to bypass local authority and go right 
to the top.

> The fact that 2.5 has "pre" releases seems suggestive of a change in 
> mindset.
> A patch now has to be widely used, tested, and recommended even to get 
> into
> 2.5, yet how does the patch get such testing before it's integrated 
> into a
> buildable kernel? Chicken and egg problem there, you want more testing 
> but
> without integration each testers can test fewer patches at once.

There is no chicken and egg problem. A patch gets wide testing by 
submitting it for peer review exactly like what goes on day in and day 
out on linux-kernel and various other subsystem mailing lists. The patch 
gets integrated into various forks of the tree for one reason or another 
and obtains even wider testing and sooner or later, it makes it into the 
kernel.

The patch tracking problem does exist. Maintainers loose track of 
changes. People integrate changes into their forks and the patch creator 
assumes that because someone is using their patch, their work is done. 
Worst of all, the people maintaining these forked trees forget where the 
patches they applied came from and bug fixes and public comments get 
missed.

There has been a lot of discussion about augmenting the kernel 
development process with automated tools of one sort of another, but 
tools are only as good as the people using them.

* Linus's email is obviously overloaded so the first thing that should 
be done is to get him a mailing list where only people he trusts can 
post patches they have reviewed. As a matter of policy, every other 
patch should simply be ignored by Linus.

* A hierarchical structure should be drawn out that shows who owns what 
and the flow of patches. This exists somewhat in the MAINTAINERS file, 
but I have a feeling that in some cases, the chain of command or is 
mostly unknown as not all maintainers should have direct contact with 
Linus. A real benefit would be to place the maintainers email address or 
associated mailing list in the top of every file they own (which exists 
mostly, but I don't believe it is part of a formal process.)

* A system of documentation for patches should be put in place. Each 
patch should have associated information about why it exists, who has 
approved it, what priority it is, change lists and of course, who wrote 
it. When a comment is made, it needs to be sent back to the author of 
the patch otherwise the author is just going to become frustrated and 
attempt to escalate the issue himself.

* A system by which maintainers can prioritize the patches they submit 
to Linus. Some patches are simply more important than others and 
developers are just going to have to deal with the fact that submitting 
a patch doesn't mean getting it integrated the next day.

Now this may sound like a lot of bureaucracy, but if the number of 
people Linus trusts can be expanded and the hierarchy kept shallow, the 
faster patches can be accepted or denied.

A final word. One thing that I have noticed over the years is because of 
the frantic pace of development, people tend to get a bit impatient when 
submitting patches. This mentality has also infected the average user 
who seem to upgrade to the bleeding edge for no apparent reason. What 
people need to remember is the 'stable' label given to the 2.4 tree 
refers to the type and frequency of changes made to the code base, not 
the stability of a running kernel. As developers, we need to be more 
patient when submitting patches. The world doesn't end because a patch 
doesn't make it into the bleeding edge kernels. If distributions need 
the patch, they'll integrate it, but that fact alone doesn't justify the 
patch making it into the kernel.

Of course that's just my opinion, I could be wrong.

Jordan

Re: A modest proposal -- We need a patch penguin

[Ragnar Hojland Espinosa]

On Tue, Jan 29, 2002 at 11:46:55AM -0800, Jordan Mendelson wrote:
> Having one person who looks after patches is just going to create 
> another bottleneck. The real problem is not Linus not accepting patches, 
> but the process by which the patches are sent to him and the general 
> impatience of the developer community.

Which in turn is quite natural when the developer community sees valuable
obvious patches not applied for no reason at all.  Linus, would you accept
some modifications to your MUA so that keys besides deleting the patch(es)
mail canned messages to the submitters?
-- 
____/|  Ragnar Højland      Freedom - Linux - OpenGL |    Brainbench MVP
\ o.O|  PGP94C4B2F0D27DE025BE2302C104B78C56 B72F0822 | for Unix Programming
 =(_)=  "Thou shalt not follow the NULL pointer for  | (www.brainbench.com)
   U     chaos and madness await thee at its end."

Coding style - a non-issue

[Peter Waltenberg]

The problem was solved years ago.

"man indent"

Someone who cares, come up with an indentrc for the kernel code, and get it
into Documentation/CodingStyle
If the maintainers run all new code through indent with that indentrc
before checkin, the problem goes away.
The only one who'll incur any pain then is a code submitter who didn't
follow the rules. (Exactly the person we want to be in pain ;)).


Then we can all get on with doing useful things.

Cheers
Peter

Re: Coding style - a non-issue

[Russell King]

On Thu, Nov 29, 2001 at 09:29:26AM +1000, Peter Waltenberg wrote:
> Someone who cares, come up with an indentrc for the kernel code, and get it
> into Documentation/CodingStyle
> If the maintainers run all new code through indent with that indentrc
> before checkin, the problem goes away.
> The only one who'll incur any pain then is a code submitter who didn't
> follow the rules. (Exactly the person we want to be in pain ;)).

See: linux/scripts/Lindent

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: Coding style - a non-issue

[Alan Cox]

> The problem was solved years ago.
> 
> "man indent"
> Someone who cares, come up with an indentrc for the kernel code, and get it
> into Documentation/CodingStyle

The problem was solved years ago.

scripts/Lindent

Re: Coding style - a non-issue

[Robert Love]

On Wed, 2001-11-28 at 18:29, Peter Waltenberg wrote:

> Someone who cares, come up with an indentrc for the kernel code, and get it
> into Documentation/CodingStyle
> If the maintainers run all new code through indent with that indentrc
> before checkin, the problem goes away.
> The only one who'll incur any pain then is a code submitter who didn't
> follow the rules. (Exactly the person we want to be in pain ;)).

See scripts/lindent in your source tree ... does just this.

	Robert Love

Re: Coding style - a non-issue

[Alexander Viro]

On Thu, 29 Nov 2001, Peter Waltenberg wrote:

> The problem was solved years ago.
> 
> "man indent"
> 
> Someone who cares, come up with an indentrc for the kernel code, and get it
> into Documentation/CodingStyle
> If the maintainers run all new code through indent with that indentrc
> before checkin, the problem goes away.
> The only one who'll incur any pain then is a code submitter who didn't
> follow the rules. (Exactly the person we want to be in pain ;)).

indent does _not_ solve the problem of:
	* buggers who think that MyVariableIsBiggerThanYourVariable is a
good name
	* buggers who define a function with 42 arguments and body being
	return (foo == bar) ? TRUE : FALSE;
	* buggers who add 1001st broken implementation of memcmp(), call it
FooTurdCompare and prepend it with 20x80 block comment.
	* buggers who use typedefs like WORD, DWORD, BYTE, IMANIDIOTSHOOTME
and other crap from the same source (OK, they don't write the last one
explicitly - not that it wasn't obvious from the rest of their, ahem, code).
	* buggers who use Hungarian notation for no good reason and come up
with structure fields that sound like street names from R'Lyeh
	* buggers who introduce wrappers for standard kernel stuff - like,
say it, typedef int Int32; and sprinkle their crap with per-architecture
ifdefs.
	* buggers who think that cpp is Just The Thing and produce turds that
would make srb cringe in disgust.

Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
wankers (both individual and coprorat ones) responsible, their code and
commentary on said code...

Re: Coding style - a non-issue

[Larry McVoy]

> Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
> wankers (both individual and coprorat ones) responsible, their code and
> commentary on said code...

Please, please, please, I'm begging you, please do this.  It's the only way
people learn quickly.  Being nice is great, but nothing works faster than 
a cold shower of public humiliation :-)
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Davide Libenzi]

On Wed, 28 Nov 2001, Larry McVoy wrote:

> > Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
> > wankers (both individual and coprorat ones) responsible, their code and
> > commentary on said code...
>
> Please, please, please, I'm begging you, please do this.  It's the only way
> people learn quickly.  Being nice is great, but nothing works faster than
> a cold shower of public humiliation :-)

Well, I don't know about "Hall of Shame" but having an "Hall of Flame"
without you two guys is like going at the OktoberFest being abstemious :)



- Davide

Re: Coding style - a non-issue

[Henning P. Schmiedehausen]

Larry McVoy <lm@bitmover.com> writes:

>> Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
>> wankers (both individual and coprorat ones) responsible, their code and
>> commentary on said code...

>Please, please, please, I'm begging you, please do this.  It's the only way
>people learn quickly.  Being nice is great, but nothing works faster than 
>a cold shower of public humiliation :-)

Cool. I can really see it. "foo inc." releases a GPL driver for Linux
and gets publically humiliated in the "linux source code hall of
shame". Perfect. I can hear the laughter from Redmond over here (and
it is 12000km away).

Press release:

"If you support Linux, you may get flamed and humiliated in public for
doing so. Don't do it."

Grow up. There is more to life than coding style that is not "Al Viro
compatible (TM)".

Sigh, sometimes, sometimes, I _really_ understand the BSD folks when
they call the Linux community "a bunch of unkempt nerds that need to
get a life". If they only would use sane init scripts. ;-) 

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Larry McVoy]

On Fri, Nov 30, 2001 at 10:00:00AM +0000, Henning P. Schmiedehausen wrote:
> Larry McVoy <lm@bitmover.com> writes:
> 
> >> Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
> >> wankers (both individual and coprorat ones) responsible, their code and
> >> commentary on said code...
> 
> >Please, please, please, I'm begging you, please do this.  It's the only way
> >people learn quickly.  Being nice is great, but nothing works faster than 
> >a cold shower of public humiliation :-)
> 
> Cool. I can really see it. "foo inc." releases a GPL driver for Linux
> and gets publically humiliated in the "linux source code hall of
> shame". Perfect. I can hear the laughter from Redmond over here (and
> it is 12000km away).
> 
> Sigh, sometimes, sometimes, I _really_ understand the BSD folks when
> they call the Linux community "a bunch of unkempt nerds that need to
> get a life". 

Perhaps it would be illuminating to know that I was BSD hacker, 
and that I learned the value of this particular technique from
Sun's kernel group, who once upon a time were the best BSD group
on the planet.  It might also be illuminating to consider that 
this technique of getting people to listen is still in use at
Sun to this day.

Perhaps Sun's professionalism is not what you'd like to see here.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Henning Schmiedehausen]

On Fri, 2001-11-30 at 16:26, Larry McVoy wrote:

Hi,

> Perhaps it would be illuminating to know that I was BSD hacker, 
> and that I learned the value of this particular technique from

I know. You tell us this again and again. Please stop patronizing.

> Sun's kernel group, who once upon a time were the best BSD group
> on the planet.  It might also be illuminating to consider that 
> this technique of getting people to listen is still in use at
> Sun to this day.

It might be enlightening to you that a closed users group of SUN coders
is not compareable to a worldwide distributed environment of thousands
of people and companies.

> 
> Perhaps Sun's professionalism is not what you'd like to see here.#

You tell me, that SUN treated _CUSTOMERS_ and companies that wanted to
support SunOS 4.1.x like that? If yes, then I definitely know why they
went SysV. Surely noone wanted BSD any longer.

I would consider the internal development groups in SUN that treated
each other like this also "in need of a change". :-)

	Regards
		Henning

-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Jeff Garzik]

The security community has shown us time and again that public shaming
is often the only way to motivate vendors into fixing security
problems.  Yes, even BSD security guys do this :)

A "Top 10 ugliest Linux kernel drivers" list would probably provide
similar motivation.

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Coding style - a non-issue

[Henning Schmiedehausen]

On Fri, 2001-11-30 at 17:47, Jeff Garzik wrote:

Hi,

> The security community has shown us time and again that public shaming
> is often the only way to motivate vendors into fixing security
> problems.  Yes, even BSD security guys do this :)
> 
> A "Top 10 ugliest Linux kernel drivers" list would probably provide
> similar motivation.

A security issue is an universal accepted problem that most of the time
has a reason and a solution.

Coding style, however, is a very personal thing that start with "shall
we use TABs or not? (Jakarta: No. Linux: Yes ...) and goes on to "Is a
preprocessor macro a good thing or not" until variable names (Al Viro:
Names with more than five letters suck. :-) Java: Non-selfdescriptive
names suck. Microsoft: Non-hungarian names suck) and so on.

And you really want to judge code just because someone likes to wrap
code in preprocessor macros or use UPPERCASE variable names? 

Come on. That's a _fundamental_ different issue than dipping vendors in
their own shit if they messed up and their box/program has a security
issue. Code that you consider ugly as hell may be seen as "easily
understandable and maintainable" by the author. If it works and has no
bugs, so what? Just because it is hard for you and me to understand (cf.
"mindboggling unwind routines in the NTFS" (I thing Jeff Merkey stated
it like this). It still seems to work quite well.

Are you willing to judge "ugliness" of kernel drivers? What is ugly? Are
Donald Beckers' drivers ugly just because they use (at least on 2.2)
their own pci helper library? Is the aic7xxx driver ugly because it
needs libdb ? Or is ugly defined as "Larry and Al don't like them"? :-)

Flaming about coding style is about as pointless as flaming someone
because he supports another sports team. There is no universal accepted
coding style. Not even in C.

	Regards
		Henning

 
-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Martin Dalecki]

Henning Schmiedehausen wrote:
> Flaming about coding style is about as pointless as flaming someone
> because he supports another sports team. There is no universal accepted
> coding style. Not even in C.

Bull shit: indent -kr is the way to go. It ever was...

Re: Coding style - a non-issue

[Larry McVoy]

On Fri, Nov 30, 2001 at 06:15:28PM +0100, Henning Schmiedehausen wrote:
> On Fri, 2001-11-30 at 17:47, Jeff Garzik wrote:
> > The security community has shown us time and again that public shaming
> > is often the only way to motivate vendors into fixing security
> > problems.  Yes, even BSD security guys do this :)
> > 
> > A "Top 10 ugliest Linux kernel drivers" list would probably provide
> > similar motivation.
> 
> A security issue is an universal accepted problem that most of the time
> has a reason and a solution.
> 
> And you really want to judge code just because someone likes to wrap
> code in preprocessor macros or use UPPERCASE variable names? 

Henning, in any long lived source base, coding style is crucial.  People
who think that coding style is personal are just wrong.  Let's compare,
shall we?

Professional: the coding style at this job looks like XYZ, ok, I will now
    make my code look like XYZ.

Amateur: my coding style is better than yours.

I think that if you ask around, you'll find that the pros use a coding 
style that isn't theirs, even when writing new code.  They have evolved
to use the prevailing style in their environment.  I know that's true for
me, my original style was 4 space tabs, curly braces on their own line,
etc.  I now code the way Bill Joy codes, fondly known as Bill Joy normal
form.

Anyway, if you think any coding style is better than another, you completely
miss the point.  The existing style, whatever it is, is the style you use.
I personally despise the GNU coding style but when I make changes there,
that's what I use because it is their source base, not mine.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Daniel Phillips]

Hi Larry,

On November 30, 2001 06:27 pm, Larry McVoy wrote:
> I think that if you ask around, you'll find that the pros use a coding 
> style that isn't theirs, even when writing new code.  They have evolved
> to use the prevailing style in their environment.  I know that's true for
> me, my original style was 4 space tabs, curly braces on their own line,
> etc.  I now code the way Bill Joy codes, fondly known as Bill Joy normal
> form.

Err, because you're still working at Sun?  I'll just ignore that last little 
non sequitur, your comment about professionals adapting to the prevailing 
standards is right on.

On the other hand, the idea of a coding style hall of shame - publicly 
humiliating kernel contributers - is immature and just plain silly.  It's 
good to have a giggle thinking about it, but that's where it should stop.

--
Daniel

Re: Coding style - a non-issue

[Alexander Viro]

On Fri, 30 Nov 2001, Daniel Phillips wrote:

> On the other hand, the idea of a coding style hall of shame - publicly 
> humiliating kernel contributers - is immature and just plain silly.  It's 
> good to have a giggle thinking about it, but that's where it should stop.

Damnit, Daniel, are you deliberately trying to tempt me into doing that?

Re: Coding style - a non-issue

[Larry McVoy]

On Fri, Nov 30, 2001 at 06:49:11PM +0100, Daniel Phillips wrote:
> On the other hand, the idea of a coding style hall of shame - publicly 
> humiliating kernel contributers - is immature and just plain silly.  It's 
> good to have a giggle thinking about it, but that's where it should stop.

If you've got a more effective way of getting people to do the right thing,
lets hear it.  Remember, the goal is to protect the source base, not your,
my, or another's ego.

I used to think Sun's approach was pretty brutal, and I still do actually.
But I haven't found anything else which works as well.  I don't use that
technique at BitMover, instead I rewrite code that I find offensive.  That's
less annoying to the engineers but it is also a lot more costly in terms of
my time.  Since we're a small company, I can keep up.  When we double in
size, I won't be able to do so and I suspect we'll revert to the Sun way.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Daniel Phillips]

On November 30, 2001 07:13 pm, Larry McVoy wrote:
> On Fri, Nov 30, 2001 at 06:49:11PM +0100, Daniel Phillips wrote:
> > On the other hand, the idea of a coding style hall of shame - publicly 
> > humiliating kernel contributers - is immature and just plain silly.  It's 
> > good to have a giggle thinking about it, but that's where it should stop.
> 
> If you've got a more effective way of getting people to do the right thing,
> lets hear it.  Remember, the goal is to protect the source base, not your,
> my, or another's ego.

Yes, lead by example, it's at least as effective.  Maybe humiliation works at 
Sun, when you're getting a paycheck, but in the world of volunteer 
development it just makes people walk.

--
Daniel

Re: Coding style - a non-issue

[Larry McVoy]

On Fri, Nov 30, 2001 at 07:43:01PM +0100, Daniel Phillips wrote:
> On November 30, 2001 07:13 pm, Larry McVoy wrote:
> > On Fri, Nov 30, 2001 at 06:49:11PM +0100, Daniel Phillips wrote:
> > > On the other hand, the idea of a coding style hall of shame - publicly 
> > > humiliating kernel contributers - is immature and just plain silly.  It's 
> > > good to have a giggle thinking about it, but that's where it should stop.
> > 
> > If you've got a more effective way of getting people to do the right thing,
> > lets hear it.  Remember, the goal is to protect the source base, not your,
> > my, or another's ego.
> 
> Yes, lead by example, it's at least as effective.  

I'd like to see some data which backs up that statement.  My experience is
that that is an unsupportable statement.  You'd need to know how effective
the Sun way is, have seen multiple development organizations using that 
way and other ways, and have watched the progress.

I'm in a somewhat unique position in that I have a lot of ex-Sun engineers
using BitKeeper and I have a pretty good idea how fast they make changes.
It's a lot faster and a lot more consistent than the Linux effort, in fact,
there is no comparison.

I'm not claiming that the coding style is the source of their speed, but
it is part of the culture which is the source of their speed.

As far as I can tell, you are just asserting that leading by example is
more effective.  Am I incorrect?  Do you have data?  I have piles which
shows the opposite.

> Maybe humiliation works at 
> Sun, when you're getting a paycheck, but in the world of volunteer 
> development it just makes people walk.

Huh.  Not sure I agree with that either.  It's definitely a dicey area
but go through the archives (or your memory if it is better than mine)
and look at how the various leaders here respond to bad choices.  It's
basically public humiliation.  Linus is especially inclined to speak
his mind when he sees something bad.  And people stick around.

I think the thing you are missing is that what I am describing is a lot
like boot camp.  Someone with more knowledge and experience than you 
yells at your every mistake, you hate it for a while, and you emerge
from boot camp a stronger person with more skills and good habits as
well as a sense of pride.  If there was a way to "lead by example" and
accomplish the same goals in the same time, don't you think someone 
would have figured that out by now?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Daniel Phillips]

On November 30, 2001 08:05 pm, Larry McVoy wrote:
> Huh.  Not sure I agree with that either.  It's definitely a dicey area
> but go through the archives (or your memory if it is better than mine)
> and look at how the various leaders here respond to bad choices.  It's
> basically public humiliation.  Linus is especially inclined to speak
> his mind when he sees something bad.  And people stick around.

There's an additional pattern, you'll notice that the guys who end up wearing 
the dung are the ones with full time Linux programming jobs, who basically 
have no option but to stick around.  Do that to every newbie and after a 
while we'll have a smoking hole in the ground where Linux used to be.

A simple rule to remember is: when code is bad, criticize the code, not the 
coder.

> I think the thing you are missing is that what I am describing is a lot
> like boot camp.  Someone with more knowledge and experience than you 
> yells at your every mistake, you hate it for a while, and you emerge
> from boot camp a stronger person with more skills and good habits as
> well as a sense of pride.

Thanks, but I'll spend my summer in some other kind of camp ;-)  I'm sure it 
works for some people, but mutual respect is more what I'm used to and prefer.

> If there was a way to "lead by example" and
> accomplish the same goals in the same time, don't you think someone 
> would have figured that out by now?

Somebody did, and as hard as it is for some to fit it into their own model of 
the universe, there is somebody leading by example, not running a command 
economy but a self-organizing meritocracy.  Do we achieve the same goals in 
the same time?  Sometimes it doesn't seem like it, but because this thing 
just keeps crawling relentlessly forward on a thousand fronts, in the end we 
accomplish even more than Sun does.

--
Daniel

Re: Coding style - a non-issue

[Larry McVoy]

This is my last post on this topic, I don't think I can say more than I have.

On Fri, Nov 30, 2001 at 10:54:39PM +0100, Daniel Phillips wrote:
> On November 30, 2001 08:05 pm, Larry McVoy wrote:
> > Huh.  Not sure I agree with that either.  It's definitely a dicey area
> > but go through the archives (or your memory if it is better than mine)
> > and look at how the various leaders here respond to bad choices.  It's
> > basically public humiliation.  Linus is especially inclined to speak
> > his mind when he sees something bad.  And people stick around.
> 
> There's an additional pattern, you'll notice that the guys who end up wearing 
> the dung are the ones with full time Linux programming jobs, who basically 
> have no option but to stick around.  Do that to every newbie and after a 
> while we'll have a smoking hole in the ground where Linux used to be.
> 
> A simple rule to remember is: when code is bad, criticize the code, not the 
> coder.

Your priorities are upside down.  The code is more important than the
coder, it will outlive the coder's interest in that code.  Besides,
this isn't some touchy feely love fest, it's code.  It's suppose to
work and work well and be maintainable.  You don't get that by being
"nice", you get that by insisting on quality.  If being nice worked,
we wouldn't be having this conversation.

> > I think the thing you are missing is that what I am describing is a lot
> > like boot camp.  Someone with more knowledge and experience than you 
> > yells at your every mistake, you hate it for a while, and you emerge
> > from boot camp a stronger person with more skills and good habits as
> > well as a sense of pride.
> 
> Thanks, but I'll spend my summer in some other kind of camp ;-)  I'm sure it 
> works for some people, but mutual respect is more what I'm used to and prefer.

The problem here is that you are assuming that yelling at someone means
that you don't respect that someone.  Nothing could be further from the
truth.  If you didn't respect them enough to think you could get good 
results from them, I doubt you'd be yelling at them in the first place.
Don't confuse intense demands for excellence with a lack of respect,
that's not the case.

> > If there was a way to "lead by example" and
> > accomplish the same goals in the same time, don't you think someone 
> > would have figured that out by now?
> 
> Somebody did, and as hard as it is for some to fit it into their own model of 
> the universe, there is somebody leading by example, not running a command 
> economy but a self-organizing meritocracy.  Do we achieve the same goals in 
> the same time?  Sometimes it doesn't seem like it, but because this thing 
> just keeps crawling relentlessly forward on a thousand fronts, in the end we 
> accomplish even more than Sun does.

Bah.  Daniel, you are forgetting that I know what Sun has done first hand
and I know what Linux has done first hand.  If you think that Linux is
at the same level as Sun's OS or ever will be, you're kidding yourself.
Linux is really cool, I love it, and I use it every day.  But it's not
comparable to Solaris, sorry, not even close.  I'm not exactly known for
my love of Solaris, you know, in fact I really dislike it.  But I respect
it, it can take a licking and keep on ticking.  Linux isn't there yet
and unless the development model changes somewhat, I'll stand behind my
belief that it is unlikely to ever get there.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Andrew Morton]

Larry McVoy wrote:
> 
> Linux isn't there yet
> and unless the development model changes somewhat, I'll stand behind my
> belief that it is unlikely to ever get there.

I am (genuinely) interested in what changes you think are needed.

-

Re: Coding style - a non-issue

[rddunlap]

On Fri, 30 Nov 2001, Andrew Morton wrote:

| Larry McVoy wrote:
| >
| > Linux isn't there yet
| > and unless the development model changes somewhat, I'll stand behind my
| > belief that it is unlikely to ever get there.
|
| I am (genuinely) interested in what changes you think are needed.

Same here, regarding both development model and OS functionality,
reliability, etc.
-- 
~Randy

Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Fri, Nov 30, 2001 at 02:17:33PM -0800, Andrew Morton wrote:
> Larry McVoy wrote:
> > 
> > Linux isn't there yet
> > and unless the development model changes somewhat, I'll stand behind my
> > belief that it is unlikely to ever get there.
> 
> I am (genuinely) interested in what changes you think are needed.

Well I have an opinion, not sure if it will be well received or not,
but here goes.

There is a choice which needs to be made up front, and that is this:

    do you want to try and turn the Linux kernel hackers into Sun style
    hackers or do you want to try something else?

This assumes we have agreement that there is a difference between the
two camps.  I'd rather not get into a "this way is better than that way"
discussion, let's just postulate that the Sun way has some pros/cons
and so do the Linux way.

If you want to try and make Linux people work like Sun people, I think
that's going to be tough.  First of all, Sun has a pretty small kernel
group, they work closely with each other, and they are full time,
highly paid, professionals working with a culture that is intolerant of
anything but the best.  It's a cool place to be, to learn, but I think
it is virtually impossible to replicate in a distributed team, with way
more people, who are not paid the same or working in the same way.

Again, let's not argue the point, let's postulate for the time being
that the Linux guys aren't going to work like the Sun guys any time soon.

So what's the problem?  The Sun guys fix more bugs, handle more corner
cases, and scale up better (Linux is still better on the uniprocessors
but the gap has narrowed considerably; Sun is getting better faster than
Linux is getting better, performance wise.  That's a bit unfair because
Linux had, and has, better absolute numbers so there was less room to
improve, but the point is that Sun is catching up fast.)

As the source base increases in size, handles more devices, runs on more
platforms, etc., it gets harder and harder to make the OS be reliable.
Anyone can make a small amount of code work well, it's exponentially
more difficult to make a large amount of code work well.  There are lots
of studies which show this to be true, the mythical man month is a good
starting place.

OK, so it sounds like I'm saying that the Linux guys are lame, Sun is
great, and there isn't any chance that Linux is going to be as good
as Solaris.  That's not quite what I'm saying.  *If* you want to play
by the same rules as Sun, i.e., develop and build things the same way,
then that is what I'm saying.  The Linux team will never be as good
as the Sun team unless the Sun team gets a lot worse.  I think that's
a fact of life, Sun has 100s of millions of dollars a year going into
software development.  ESR can spout off all he likes, but there is no
way a team of people working for fun is going to compete with that.

On the other hand, there is perhaps a way Linux could be better.  But it
requires changing the rules quite a bit.

Instead of trying to make the Linux hackers compete with the Sun hackers,
what would happen if you architected your way around the problem?
For example, suppose I said we need to have a smaller, faster, more
reliable uniprocessor kernel.  Suppose I could wave a magic wand and
make SMP go away (I can't, but bear with me for a second).  The problem
space just got at least an order of magnitude less complex than what Sun
deals with.  I think they are up to 32-64 way SMP and you can imagine,
I hope, the additional complexity that added.  OK, so *if* uniprocessor
was the only thing we had to worry about, or say 2-4 way SMP with just
a handful of locks, then can we agree that it is a lot more likely that
we could produce a kernel which was in every way as good or better than
Sun's kernel, on the same platform?  If the answer is yes, keep reading,
if the answer is no, then we're done because I don't know what to do if
we can't get that far.

For the sake of discussion, let's assume that you buy what I am saying
so far.  And let's say that we agree that if you were to toss the SMP
stuff then we have a good chance at making a reliable kernel, albeit
an uninteresting one when compared to big boxes.  If you want me to go
into what I think it would take to do that, I will.

The problem is that we can't ignore the SMP issues, it drives hardware
sales, gets press, it's cool, etc.  We have to have both but the problem
is that if we have both we really need Sun's level of professionalism
to make it work, and it isn't realistic to expect that from a bunch of
underpaid (or not at all paid) Linux hackers.

Here's how you get both.  Fork the development efforts into the SMP part
and the uniprocessor part.  The uniprocessor focus is on reliability,
stability, performance.  The SMP part is a whole new development effort,
which is architected in such a way as to avoid the complexity of a
traditional SMP implementation.

The uniprocessor team owns the core architecture of the system.  The
abstractions provided, the baseline code, etc., that's all uni.  The
focus there is a small, fast, stable kernel.

The SMP team doesn't get to touch the core code, or at least there is 
a very strong feedback loop against.  In private converstations, we've
started talking about the "punch in the nose" feedback loop, which means
while it may be necessary to touch generic code for the benefit of SMP,
someone has to feel strongly enough about it that they well sacrifice
their nose.

It would seem like I haven't solved anything here, just painted a nice
but impossible picture.  Maybe.  I've actually thought long and hard 
about the approach needed to scale up without touching all the code
and it is radically different from the traditional way (i.e., how
Sun, SGI, Sequent, etc., did it).  If you are interested in that, I'll
talk about it but I'll wait to see if anyone cares.

The summary over all of this is:

    If you want to solve all the problems that Sun does, run on the same
    range of UP to big SMP, Linux is never going to be as reliable as 
    Solaris.  My opinion, of course, but one that is starting to gain
    some traction as the OS becomes more complex.

    That leaves you with a choice: either give up on some things,
    magically turn the Linux hackers into Sun hackers, or
    architect your way around the problem.

My vote is the last one, it fits better with the Linux effort, the answer
is way more cool than anything Sun or anyone else has done, and it lets
you have a simple mainline kernel source base.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Davide Libenzi]

On Fri, 30 Nov 2001, Larry McVoy wrote:

[ A lot of stuff Pro-Sun ]

Wait a minute.
Wasn't it you that were screaming against Sun, leaving their team because
their SMP decisions about scaling sucked, because their OS was bloated
like hell with sync spinlocks, saying that they tried to make it scale but
they failed miserably ?
What is changed now to make Solaris, a fairly vanishing OS, to be the
reference OS/devmodel for every kernel developer ?
Wasn't it you that were saying that Linux will never scale with more than
2 CPUs ?
Isn't Linux SMP improved from the very first implementation ?
Isn't Linux SMP improved from the very first implementation w/out losing
reliability ?
Why you don't try to compare 2.0.36 with 2.4.x let's say on a 8 way SMP ?
Why should it stop improving ?
Because you know that adding fine grained spinlocks will make the OS
complex to maintain and bloated ... like it was Solaris before you
suddendly changed your mind.


<YOUR QUOTE>
>     Then people want more performance.  So they thread some more and now
>     the locks aren't 1:1 to the objects.  What a lock covers starts to
>     become fuzzy.  Thinks break down quickly after this because what
>     happens is that it becomes unclear if you are covered or not and
>     it's too much work to figure it out, so each time a thing is added
>     to the kernel, it comes with a lock.  Before long, your 10 or 20
>     locks are 3000 or more like what Solaris has.  This is really bad,
>     it hurts performance in far reaching ways and it is impossible to
>     undo.
</YOUR QUOTE>

I kindly agree with this, just curious to understand which kind of amazing
architectural solution Solaris took to be a reference for SMP
development/scaling.




- Davide

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Fri, Nov 30, 2001 at 05:13:38PM -0800, Davide Libenzi wrote:
> On Fri, 30 Nov 2001, Larry McVoy wrote:
> Wait a minute.
> Wasn't it you that were screaming against Sun, leaving their team because
> their SMP decisions about scaling sucked, because their OS was bloated
> like hell with sync spinlocks, saying that they tried to make it scale but
> they failed miserably ?

Yup, that's me, guilty on all charges.

> What is changed now to make Solaris, a fairly vanishing OS, to be the
> reference OS/devmodel for every kernel developer ?

It's not.  I never said that we should solve the same problems the same
way that Sun did, go back and read the posting.

> Wasn't it you that were saying that Linux will never scale with more than
> 2 CPUs ?

No, that wasn't me.  I said it shouldn't scale beyond 4 cpus.  I'd be pretty
lame if I said it couldn't scale with more than 2.  Should != could.

> Because you know that adding fine grained spinlocks will make the OS
> complex to maintain and bloated ... like it was Solaris before you
> suddendly changed your mind.

Sorry it came out like that, I haven't changed my mind one bit.

> <YOUR QUOTE>
> >     Then people want more performance.  So they thread some more and now
> >     the locks aren't 1:1 to the objects.  What a lock covers starts to
> >     become fuzzy.  Thinks break down quickly after this because what
> >     happens is that it becomes unclear if you are covered or not and
> >     it's too much work to figure it out, so each time a thing is added
> >     to the kernel, it comes with a lock.  Before long, your 10 or 20
> >     locks are 3000 or more like what Solaris has.  This is really bad,
> >     it hurts performance in far reaching ways and it is impossible to
> >     undo.
> </YOUR QUOTE>
> 
> I kindly agree with this, just curious to understand which kind of amazing
> architectural solution Solaris took to be a reference for SMP
> development/scaling.

OK, so you got the wrong message.  I do _not_ like the approach Sun took,
it's a minor miracle that they are able to make Solaris work as well as
it works given the design decisions they made.

What I do like is Sun's engineering culture.  They work hard, they don't
back away from the corner cases, they have high standards.  All of which
and more are, in my opinion, a requirement to try and solve the problems
the way they solved them.

So the problem I've been stewing on is how you go about scaling the OS
in a way that doesn't require all those hot shot sun engineers to make
it work and maintain it.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Davide Libenzi]

On Fri, 30 Nov 2001, Larry McVoy wrote:

> On Fri, Nov 30, 2001 at 05:13:38PM -0800, Davide Libenzi wrote:
> > On Fri, 30 Nov 2001, Larry McVoy wrote:
> > Wait a minute.
> > Wasn't it you that were screaming against Sun, leaving their team because
> > their SMP decisions about scaling sucked, because their OS was bloated
> > like hell with sync spinlocks, saying that they tried to make it scale but
> > they failed miserably ?
>
> Yup, that's me, guilty on all charges.
>
> > What is changed now to make Solaris, a fairly vanishing OS, to be the
> > reference OS/devmodel for every kernel developer ?
>
> It's not.  I never said that we should solve the same problems the same
> way that Sun did, go back and read the posting.

This is your quote Larry :

<>
If you want to try and make Linux people work like Sun people, I think
that's going to be tough.  First of all, Sun has a pretty small kernel
group, they work closely with each other, and they are full time,
highly paid, professionals working with a culture that is intolerant of
anything but the best.  It's a cool place to be, to learn, but I think
it is virtually impossible to replicate in a distributed team, with way
more people, who are not paid the same or working in the same way.
<>

So, if these guys are smart, work hard and are professionals, why did they
take bad design decisions ?
Why didn't they implemented different solutions like, let's say "multiple
independent OSs running on clusters of 4 CPUs" ?
What we really have to like about Sun ?
Me personally, if I've to choose, I'll take the logo.




- Davide

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Fri, Nov 30, 2001 at 06:17:43PM -0800, Davide Libenzi wrote:
> > It's not.  I never said that we should solve the same problems the same
> > way that Sun did, go back and read the posting.
> 
> This is your quote Larry :
> 
> <>
> If you want to try and make Linux people work like Sun people, I think
> that's going to be tough.  First of all, Sun has a pretty small kernel
> group, they work closely with each other, and they are full time,
> highly paid, professionals working with a culture that is intolerant of
> anything but the best.  It's a cool place to be, to learn, but I think
> it is virtually impossible to replicate in a distributed team, with way
> more people, who are not paid the same or working in the same way.
> <>

I'm sorry, I'm lost.  You are quoting what I said, that's what I said
said, so what's the point?  yes, for the third time, that's what I said
and that's what I meant.

> So, if these guys are smart, work hard and are professionals, why did they
> take bad design decisions ?
> Why didn't they implemented different solutions like, let's say "multiple
> independent OSs running on clusters of 4 CPUs" ?

Because, just like the prevailing wisdom in the Linux hackers, they thought
it would be relatively straightforward to get SMP to work.  They started at
2, went to 4, etc., etc.  Noone ever asked them to go from 1 to 100 in one
shot.  It was always incremental.

I recently talked over the approach I have in mind with the architect of
Sun's multithreaded kernel, the guy who started and guided that effort at
Sun.  He agreed that if he had thought of my approach he would have done
it that way.  We both agreed it was unlikely that anyone would think of
that approach without first having done it the hard way.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Rik van Riel]

On Fri, 30 Nov 2001, Larry McVoy wrote:
> On Fri, Nov 30, 2001 at 06:17:43PM -0800, Davide Libenzi wrote:

> > So, if these guys are smart, work hard and are professionals, why did they
> > take bad design decisions ?
> > Why didn't they implemented different solutions like, let's say "multiple
> > independent OSs running on clusters of 4 CPUs" ?
>
> Because, just like the prevailing wisdom in the Linux hackers, they
> thought it would be relatively straightforward to get SMP to work.
> They started at 2, went to 4, etc., etc.  Noone ever asked them to go
> from 1 to 100 in one shot.  It was always incremental.

Incremental stuff always breaks and misses out on the corner
cases. The same seems to be true for stuff coming out of
random mutation and biological selection ... natural selection
really doesn't care about corner cases.

This, for example, has always resulted in a VM subsystem which
works nicely under low load but falls apart under high load.
Any efforts to fix that corner case have been met with nothing
but resistance.

Lets face it, if you want to deal with corner cases you don't
want to deal with Linus.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:

[...]

> Because, just like the prevailing wisdom in the Linux hackers, they thought
> it would be relatively straightforward to get SMP to work.  They started at
> 2, went to 4, etc., etc.  Noone ever asked them to go from 1 to 100 in one
> shot.  It was always incremental.

Maybe that is because 128 CPU machines aren't exactly common... just as
SPARC, m68k, S/390 development lags behind ia32 just because there are
many, many more of the later around.

Just as Linus said, the development is shaped by its environment.
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Sat, Dec 01, 2001 at 08:05:59PM -0300, Horst von Brand wrote:
> Larry McVoy <lm@bitmover.com> said:
> 
> [...]
> 
> > Because, just like the prevailing wisdom in the Linux hackers, they thought
> > it would be relatively straightforward to get SMP to work.  They started at
> > 2, went to 4, etc., etc.  Noone ever asked them to go from 1 to 100 in one
> > shot.  It was always incremental.
> 
> Maybe that is because 128 CPU machines aren't exactly common... just as
> SPARC, m68k, S/390 development lags behind ia32 just because there are
> many, many more of the later around.
> 
> Just as Linus said, the development is shaped by its environment.

Really?  So then people should be designing for 128 CPU machines, right?
So why is it that 100% of the SMP patches are incremental?  Linux is 
following exactly the same path taken by every other OS, 1->2, then 2->4,
then 4->8, etc.  By your logic, someone should be sitting down and saying
here is how you get to 128.  Other than myself, noone is doing that and
I'm not really a Linux kernel hack, so I don't count.  

So why is it that the development is just doing what has been done before?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Rik van Riel]

On Sun, 2 Dec 2001, Larry McVoy wrote:

> So why is it that the development is just doing what has been done before?

No matter how often you try to reinvent the wheel, you'll
end up converging to a round shape.

The sad thing is that many of the people here have to try
the octagonal wheel first (because it's "simpler to build
and more maintainable" ?) and then get confused when they
run into problems.

cheers,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Eric W. Biederman]

Larry McVoy <lm@bitmover.com> writes:

> On Sat, Dec 01, 2001 at 08:05:59PM -0300, Horst von Brand wrote:
> > Larry McVoy <lm@bitmover.com> said:
> > 
> > [...]
> > 
> > > Because, just like the prevailing wisdom in the Linux hackers, they thought
> > > it would be relatively straightforward to get SMP to work.  They started at
> > > 2, went to 4, etc., etc.  Noone ever asked them to go from 1 to 100 in one
> > > shot.  It was always incremental.
> > 
> > Maybe that is because 128 CPU machines aren't exactly common... just as
> > SPARC, m68k, S/390 development lags behind ia32 just because there are
> > many, many more of the later around.
> > 
> > Just as Linus said, the development is shaped by its environment.
> 
> Really?  So then people should be designing for 128 CPU machines, right?
> So why is it that 100% of the SMP patches are incremental?  Linux is 
> following exactly the same path taken by every other OS, 1->2, then 2->4,
> then 4->8, etc.  By your logic, someone should be sitting down and saying
> here is how you get to 128.  Other than myself, noone is doing that and
> I'm not really a Linux kernel hack, so I don't count.  
> 
> So why is it that the development is just doing what has been done before?

Hmm.  Meanwhile linux appears to be the platform of choice for new
clusters.  And if you are working on a cluster you can by incremental
improvements reach your design Larry.  In truth a 128 CPU machine is a
modest cluster.  The typical goal for a large system I have seen is a
1000 Node cluster.  Which could mean anywhere from 1000 to 4000
processors.  And I have seen a few serious proposals for even larger
systems.

So with a cluster you start by looking for something that scales, on a
relatively slow interconnect.  Which is simply lots, and lots of
kernels, that don't know each other at all.  And then you figure out
how to get them all talking to each other.

As has been pointed out exporting an interface that looks to the
programmer like a giant SMP.  Tends to make programs share too much
data, and thus get high degrees of lock contention.

The next incremental step is to get some good distributed and parallel
file systems.  So you can share one filesystem across the cluster.
And there is some work going on in those areas.  luster, gfs,
intermezzo.

Eric

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> The next incremental step is to get some good distributed and parallel
> file systems.  So you can share one filesystem across the cluster.
> And there is some work going on in those areas.  luster, gfs,
> intermezzo.

gfs went proprietary - you want opengfs

A lot of good work on the rest of that multi-node clustering is going on
already - take a look at the compaq open source site.

cccluster is more for numa boxes, but it needs the management and SSI views
that the compaq stuff offers simply because most programmers won't program
for a cccluster or manage one.

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Eric W. Biederman]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> > The next incremental step is to get some good distributed and parallel
> > file systems.  So you can share one filesystem across the cluster.
> > And there is some work going on in those areas.  luster, gfs,
> > intermezzo.
> 
> gfs went proprietary - you want opengfs

Right.
 
> A lot of good work on the rest of that multi-node clustering is going on
> already - take a look at the compaq open source site.

Basically my point.
 
> cccluster is more for numa boxes, but it needs the management and SSI views
> that the compaq stuff offers simply because most programmers won't program
> for a cccluster or manage one.

I've seen a fair number of mpi programs, and if you have a program
that takes weeks to run on a single system.  There is a lot of
incentive to work it out.  Plus I have read about a lot of web sites
that are running on a farm of servers.  Admittedly the normal
architecture has one fat database server behind the web servers, but
that brings me back to needing a good distributed storage techniques.

And I really don't care if most programmers won't program for a
cccluster.  Most programmers don't have one or a problem that needs
one to solve.  So you really only need those people interested in the
problem to work on it.

But single system image type projects are useful, but need to be
watched.  You really need to standardize on how a cluster is put
together (software wise), and making things easier always helps.  But
you also need to be very careful because you can easily write code
that does not scale.  And people doing cluster have wild notions of
scaling o.k. 64 Nodes worked let's try a thousand...

As far as I can tell the only real difference between a numa box, and
a normal cluster of machines running connected with fast ethernet is
that a numa interconnect is a blazingly fast interconnect.  So if you
can come up with a single system image solution over fast ethernet a
ccNuma machine just magically works.

Eric

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

> As far as I can tell the only real difference between a numa box, and
> a normal cluster of machines running connected with fast ethernet is
> that a numa interconnect is a blazingly fast interconnect.  

Plus some fairly hairy cache coherency hardware.

> So if you
> can come up with a single system image solution over fast ethernet a
> ccNuma machine just magically works.

it's not cc if you just use fast ethernet.

Martin.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> > can come up with a single system image solution over fast ethernet a
> > ccNuma machine just magically works.
> 
> it's not cc if you just use fast ethernet.

Thats a matter for handwaving and distributed shared memory algorithms. The
general point is still true - if you assume your NUMA interconnects are
utter crap when performance and latency issues come up - you'll get the
right results.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Davide Libenzi]

On Sun, 2 Dec 2001, Larry McVoy wrote:

> On Sat, Dec 01, 2001 at 08:05:59PM -0300, Horst von Brand wrote:
> > Larry McVoy <lm@bitmover.com> said:
> >
> > [...]
> >
> > > Because, just like the prevailing wisdom in the Linux hackers, they thought
> > > it would be relatively straightforward to get SMP to work.  They started at
> > > 2, went to 4, etc., etc.  Noone ever asked them to go from 1 to 100 in one
> > > shot.  It was always incremental.
> >
> > Maybe that is because 128 CPU machines aren't exactly common... just as
> > SPARC, m68k, S/390 development lags behind ia32 just because there are
> > many, many more of the later around.
> >
> > Just as Linus said, the development is shaped by its environment.
>
> Really?  So then people should be designing for 128 CPU machines, right?
> So why is it that 100% of the SMP patches are incremental?  Linux is
> following exactly the same path taken by every other OS, 1->2, then 2->4,
> then 4->8, etc.  By your logic, someone should be sitting down and saying
> here is how you get to 128.  Other than myself, noone is doing that and
> I'm not really a Linux kernel hack, so I don't count.
>
> So why is it that the development is just doing what has been done before?

That's exactly the Linus point: no long term preventive design.
That means short term design on what we actually have or on what we can
think available in a very near future.
Because, again, if you start designing now an SMP solution for 128 CPUs,
you're going to model it on the current technology that won't probably fit
in future SMP architectures and you'll be screwed up.
There should be a reason why software evolves in this way, think about it.
Either everyone else is fool or you're the prophet.
And if you're the prophet and you think that the future of multiprocessing
is UP on clusters, why instead of spreading your word between us poor
kernel fans don't you pull out money from your pocket ( or investors ) and
start a new Co. that will have that solution has primary and unique goal ?
You could be the J.C. Maxwell of the next century.



- Davide

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Davide Libenzi]

On Sun, 2 Dec 2001, Davide Libenzi wrote:

> That's exactly the Linus point: no long term preventive design.

And now for the ones that don't speak Italish :

s/preventive/prior/




- Davide

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Andrew Morton]

Larry McVoy wrote:
> 
> Really?  So then people should be designing for 128 CPU machines, right?

Linux only supports 99 CPUs.  At 100, "ksoftirqd_CPU100" overflows
task_struct.comm[].

Just thought I'd sneak in that helpful observation.

-

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Dave Jones]

On Sun, 2 Dec 2001, Andrew Morton wrote:

> > Really?  So then people should be designing for 128 CPU machines, right?
> Linux only supports 99 CPUs.  At 100, "ksoftirqd_CPU100" overflows
> task_struct.comm[].
> Just thought I'd sneak in that helpful observation.

Wasn't someone looking at fixing that problem so it didn't need a per-cpu
thread ? 128 kernel threads sitting around waiting for a problem that
rarely happens seems a little.. strange. (for want of a better word).

Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Andrew Morton]

Dave Jones wrote:
> 
> On Sun, 2 Dec 2001, Andrew Morton wrote:
> 
> > > Really?  So then people should be designing for 128 CPU machines, right?
> > Linux only supports 99 CPUs.  At 100, "ksoftirqd_CPU100" overflows
> > task_struct.comm[].
> > Just thought I'd sneak in that helpful observation.
> 
> Wasn't someone looking at fixing that problem so it didn't need a per-cpu
> thread ?

Not to my knowledge.

> 128 kernel threads sitting around waiting for a problem that
> rarely happens seems a little.. strange. (for want of a better word).

I've kinda lost the plot on ksoftirqd.  Never really understood
why a thread was needed for this, nor why it runs at nice +20.
But things seem to be working now.

-

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Jamie Lokier]

Andrew Morton wrote:
> > 128 kernel threads sitting around waiting for a problem that
> > rarely happens seems a little.. strange. (for want of a better word).
> 
> I've kinda lost the plot on ksoftirqd.  Never really understood
> why a thread was needed for this, nor why it runs at nice +20.
> But things seem to be working now.

Me no idea either.  It wasn't to work around the problem of losing
softirqs on syscall return was it?  Because there was a patch for that
in the low-latency set that fixed that without a thread, and without a
delay...

-- Jamie

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

>> Really?  So then people should be designing for 128 CPU machines, right?
> 
> Linux only supports 99 CPUs.  At 100, "ksoftirqd_CPU100" overflows
> task_struct.comm[].
> 
> Just thought I'd sneak in that helpful observation.

For machines that are 99bit architectures or more, maybe. For 32 bit machines,
your limit is 32, for 64 bit, 64.

M.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> > Just as Linus said, the development is shaped by its environment.
> 
> Really?  So then people should be designing for 128 CPU machines, right?

Linux environment is a single athlon/pii type processor with 128-256Mb of
RAM, because quite simply thats what most developers have.

> So why is it that 100% of the SMP patches are incremental?  Linux is 
> following exactly the same path taken by every other OS, 1->2, then 2->4,
> then 4->8, etc.  By your logic, someone should be sitting down and saying
> here is how you get to 128.  Other than myself, noone is doing that and
> I'm not really a Linux kernel hack, so I don't count.  

The problem being that unless you happen to have a load of people with 128
processor boxes you can't verify your work is even useful or correct. 

You can look at Linux development and the areas which have been heavily
funded rather than written for fun/need by people and you'll see a heavy
slant to the enterprise end. J Random Hacker doesn't have an IA64, an 8 way
SMP box or 2Tb of disk so J Random Hacker is much more interested in making
sure his/her webcam works (which is why we pee all over Solaris X86 on device 
support).

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Stephan von Krawczynski]

> On Sat, Dec 01, 2001 at 08:05:59PM -0300, Horst von Brand wrote:    
> > Just as Linus said, the development is shaped by its environment. 
>                                                                     
> Really?  So then people should be designing for 128 CPU machines,   
right?                                                                
> So why is it that 100% of the SMP patches are incremental?  Linux is
                                                                      
> following exactly the same path taken by every other OS, 1->2, then 
2->4,                                                                 
> then 4->8, etc.  By your logic, someone should be sitting down and  
saying                                                                
> here is how you get to 128.  Other than myself, noone is doing that 
and                                                                   
> I'm not really a Linux kernel hack, so I don't count.               
>                                                                     
> So why is it that the development is just doing what has been done  
before?                                                               
                                                                      
Please Larry, have a look at the environment: nobody here owns a box  
with 128 CPUs. Most of the people here take care of things they either
- own themselves                                                      
- have their hands own at work                                        
- get paid for                                                        
                                                                      
You will not find _any_ match with 128 CPUs here.                     
                                                                      
_Obviously_ you are completely right if this were a company _building_
these boxes. Then your question is the right one, as they would get   
paid for the job.                                                     
But this is a different environment. As long as you cannot buy these  
boxes at some local store for a buck and a bit, you will have no      
chance to find willing people for your approach. Therefore it is      
absolutely clear, that it will (again) walk the line from 1,2,4,8 ... 
CPUs, because the boxes will be available along this line.            
                                                                      
I give you this advice: if you _really_ want to move something in this
area, find someone who should care about this specific topic, and has 
the money _and_ the will to pay for development of critical GPL code  
like this.                                                            
Take the _first_ step: create the environment. _Then_ people will come
and follow your direction.                                            
                                                                      
Regards,                                                              
Stephan                                                               
                                                                      
                                                                      

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

> Please Larry, have a look at the environment: nobody here owns a box  
> with 128 CPUs. Most of the people here take care of things they either
> - own themselves                                                      
> - have their hands own at work                                        
> - get paid for                                                        
>                                                                       
> You will not find _any_ match with 128 CPUs here.                     

Nor will you find any match with 4 or 8 CPU systems, except in very rare
cases.  Yet changes go into the system for 8 way and 16 way performance.
That's a mistake.

I'd be ecstatic if the hackers limited themselves to what was commonly 
available, that is essentially what I'm arguing for.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:
> skraw@ithnet.com on Sun, Dec 02, 2001 at 11:52:32PM +0100 said:

[...]

> > You will not find _any_ match with 128 CPUs here.                     
> 
> Nor will you find any match with 4 or 8 CPU systems, except in very rare
> cases.  Yet changes go into the system for 8 way and 16 way performance.
> That's a mistake.

And you are proposing a fork for handling exactly this? I don't get it...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Henning P. Schmiedehausen]

Horst von Brand <vonbrand@inf.utfsm.cl> writes:

Hi,

>> Nor will you find any match with 4 or 8 CPU systems, except in very rare
>> cases.  Yet changes go into the system for 8 way and 16 way performance.
>> That's a mistake.

>And you are proposing a fork for handling exactly this? I don't get it...

[Ah, and I've sworn that I won't participate in this thread again...]

99.99% of your userbase have one or two processors. 99.999% of your
userbase have less than 8 processors.

It simply doesn't make sense to design something for 128+ Processors
if noone uses it. Especially if it puts a penalty on the user base
with one or two processors. Then such a design is actively hurting
your user base. And this is nonsense. That's why the design goes
1->2->4->8. We got SMP once a core developer got a sponsored SMP
board. Not because Linus designed his OS with "I want to scale to lots
of processors I can only dream of". Along that road lies madness (and
maybe GNU Hurd ;-) ).

I actually have five (at a customers site) eight-way machines. 
Unfortunately now they all run Windows 2000 (Datacenter Edition
(TM)). But if the "eight-way" support would hurt the "two way"
machines, I'd advocate to put that code at least under a compile
switch.

Actually I do like the idea in Larrys' paper. It doesn't hurt beyond
eight way, because the next eight processors are their own
"system". Just because it doesn't hurt the smaller boxes (much).

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Over 4-way systems considered harmful :-)

[M. Edward Borasky]

I'm going to weigh in here in favor of limiting effort on SMP development by
the core Linux team to systems with 4 processors and under. And not just
because I'd like to see those developers freed up to work on my M-Audio
Delta 66 :-). The economics of massively parallel MIMD machines just aren't
there. Sure, the military guys would *love* to have a petaflop engine, but
they're gonna build 'em anyway and quite probably not bother to contribute
their kernel source on this mailing list. *Commercial* applications for
supercomputers of this level are few and far between. I'm happy with my
GFlop-level UP Athlon Thunderbird. And if Moore's Law (or the AMD equivalent
:-) still holds, in 12 months I'll have something twice as fast (I've had it
for six months already :-).
--
Take Your Trading to the Next Level!
M. Edward Borasky, Meta-Trading Coach

znmeb@borasky-research.net
http://www.meta-trading-coach.com
http://groups.yahoo.com/group/meta-trading-coach

Re: Over 4-way systems considered harmful :-)

[Martin J. Bligh]

> I'm going to weigh in here in favor of limiting effort on SMP development by
> the core Linux team to systems with 4 processors and under. And not just
> because I'd like to see those developers freed up to work on my M-Audio
> Delta 66 :-). The economics of massively parallel MIMD machines just aren't
> there. Sure, the military guys would *love* to have a petaflop engine, but
> they're gonna build 'em anyway and quite probably not bother to contribute
> their kernel source on this mailing list. *Commercial* applications for
> supercomputers of this level are few and far between. I'm happy with my
> GFlop-level UP Athlon Thunderbird. And if Moore's Law (or the AMD equivalent
> :-) still holds, in 12 months I'll have something twice as fast (I've had it
> for six months already :-).

Two things.

1) If a company (say, IBM) pays people to work on 8 / 16 way scalability
because that's what they want out of Linux, then stopping development
on that isn't going to get effort redirected to fixing your soundcard (yes,
I realise you were being flippant, but the point's the same), the headcount
is just going to disappear. AKA your choice isn't "patches for 8 way
scalablilty, or patches for subsystem X that you're more interested in",
your choice is "patches for 8-way scalabity, or no patches". Provided that
those patches don't break anything else, you still win overall by getting them.

2) Working on scalability for 8 / 16 way machines will show up races,
performance problems et al that exist on 2 / 4 way machines but don't
show up as often, or as obviously. I have a 16 way box that shows up
races in the Linux kernel that might take you years to find on a 2 way.

What I'm trying to say is that you still win. Not as much as maybe you'd
like, but, hey, it's work you're getting for free, so don't complain too 
much about it. The maintainers are very good at beating the message
into us that we can't make small systems any worse performing whilst
making the big systems better.

Martin.

RE: Over 4-way systems considered harmful :-)

[M. Edward Borasky]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Martin J. Bligh
> Sent: Tuesday, December 04, 2001 9:41 AM
> To: M. Edward Borasky; linux-kernel@vger.kernel.org
> Subject: Re: Over 4-way systems considered harmful :-)
>
> Two things.
>
> 1) If a company (say, IBM) pays people to work on 8 / 16 way scalability
> because that's what they want out of Linux, then stopping development
> on that isn't going to get effort redirected to fixing your
> soundcard (yes,
> I realise you were being flippant, but the point's the same), the
> headcount
> is just going to disappear. AKA your choice isn't "patches for 8 way
> scalablilty, or patches for subsystem X that you're more interested in",
> your choice is "patches for 8-way scalabity, or no patches". Provided that
> those patches don't break anything else, you still win overall by
> getting them.

I don't see how this is a win for me. And it is a win for IBM only if it
gives them some advantage in serving their customers. I can certainly
*conceive* of workloads bursty enough to justify an 8-processor server, but
do they exist in the real world? And if they do, is a single 8-processor
server better than a pair of 4-processor servers when you take graceful
handling of faults into account? IBM has been building high-availability
systems for *decades*, preferring to field *slightly* slower but
*significantly* more reliable gear, which, legend has it, no one has ever
been fired for purchasing. :-)

> 2) Working on scalability for 8 / 16 way machines will show up races,
> performance problems et al that exist on 2 / 4 way machines but don't
> show up as often, or as obviously. I have a 16 way box that shows up
> races in the Linux kernel that might take you years to find on a 2 way.

Perhaps effort should be placed into software development processes and
tools that deny race conditions the right to be born, rather than depending
on testing on a 16-processor system to find them expeditiously :-). And
there is a whole discipline of software performance engineering to build
performance in from the start. Advances like that would be a *huge* win for
the Linux community, given our (relative) freedom from corporate-world
limitations like deadlines, sales quotas, programmer salaries, and
full-color brochures.

> What I'm trying to say is that you still win. Not as much as maybe you'd
> like, but, hey, it's work you're getting for free, so don't complain too
> much about it. The maintainers are very good at beating the message
> into us that we can't make small systems any worse performing whilst
> making the big systems better.

No, but we can release partly-baked VM schemes that have a shelf life on the
order of days :-). Seriously, though, I don't like stepping on someone
else's dream, especially since *my* dream -- a GFLOP dedicated to computer
music -- has been fulfilled for about $1500 US. When I bought that machine,
I thought I was going to need two processors -- one to run the OS and
service the keyboard, mouse and monitor and a second dedicated to generating
audio samples in real time. I had no idea how powerful these chips were
until I started shopping around. And when I loaded the Atlas linear algebra
library up on my Athlon and saw the speeds it was getting, I was in shock
for almost a week!

However, as I said in another post: "Moore's Law: good. Amdahl's Law: bad."
I guess the new generation has to discover Amdahl's Law for itself, and
*this* distinguished but elderly scientist is eager to be proven wrong :-).
--
Take Your Trading to the Next Level!
M. Edward Borasky, Meta-Trading Coach

znmeb@borasky-research.net
http://www.meta-trading-coach.com
http://groups.yahoo.com/group/meta-trading-coach

RE: Over 4-way systems considered harmful :-)

[Martin J. Bligh]

> I don't see how this is a win for me. And it is a win for IBM only if it
> gives them some advantage in serving their customers. I can certainly
> *conceive* of workloads bursty enough to justify an 8-processor server, but
> do they exist in the real world? And if they do, is a single 8-processor
> server better than a pair of 4-processor servers when you take graceful
> handling of faults into account? IBM has been building high-availability
> systems for *decades*, preferring to field *slightly* slower but
> *significantly* more reliable gear, which, legend has it, no one has ever
> been fired for purchasing. :-)

We (Sequent, now bought by IBM) sold 64 processor servers to people 
who needed them (the main market was large databases) so, yes, there 
is definitely a market for larger systems. It's cheaper to build a big processor 
farm out of a Beowulf style cluster, but it's not always easy / possible to 
split up your application over something like that. If you're generating 
fractals, it's easy, for other applications, it's not.
 
> Perhaps effort should be placed into software development processes and
> tools that deny race conditions the right to be born, rather than depending
> on testing on a 16-processor system to find them expeditiously :-). And

I wish you good fortune, sir. When you've acheived that, come back and
tell me, and we'll stop testing stuff ;-) It's always better to not code bugs in
the first place, but that's not the world we live in ...

> there is a whole discipline of software performance engineering to build
> performance in from the start. Advances like that would be a *huge* win for
> the Linux community, given our (relative) freedom from corporate-world
> limitations like deadlines, sales quotas, programmer salaries, and
> full-color brochures.

I'm all for encouraging good programming practices. In a way that's actually
harder in the diverse Linux community with hundreds, nay, thousands of
engineers putting code in. But we should still try - keeping the infrastructure
simple helps, for example. Trying to bug fix badly designed / written code is 
pissing into the wind.

M.

Re: Over 4-way systems considered harmful :-)

[Matthew Fredrickson]

On Tue, Dec 04, 2001 at 09:07:14PM -0800, M. Edward Borasky wrote:
> I don't see how this is a win for me. And it is a win for IBM only if it

I think what you don't seem to realize is that most developers don't
develop because they think it's good for you, or anyone else for that
matter.  Every developer has his own motivation, but I doubt that there
are a great majority of them that do it just to please others.

We don't do it for the "greater good", we do it because "it fixes this bug
that keeps hard locking my kernel whenever I play track 2 of the new cd
that came out", or "I guess nobody has written a driver for my new
seti@home-on-pci card processing board, looks like if I want it to work,
I'll have to write one".

I'm not trying to sham the greater good mentality, but I'm just trying to
tell you that most developers are primarily in it for themselves.

> Perhaps effort should be placed into software development processes and
> tools that deny race conditions the right to be born, rather than depending
> on testing on a 16-processor system to find them expeditiously :-). And

You have gcc, no?  write some tools.  I like doing driver development,
I'll keep doing that until I need a tool to help me out with something,
and then I'll write one.  Maybe I'll get lucky and someone else has
already written what I need.

Matthew Fredrickson

Re: Over 4-way systems considered harmful :-)

[Pavel Machek]

Hi!

> > I'm going to weigh in here in favor of limiting effort on SMP development by
> > the core Linux team to systems with 4 processors and under. And not just
> > because I'd like to see those developers freed up to work on my M-Audio
> > Delta 66 :-). The economics of massively parallel MIMD machines just aren't
> > there. Sure, the military guys would *love* to have a petaflop engine, but
> > they're gonna build 'em anyway and quite probably not bother to contribute
> > their kernel source on this mailing list. *Commercial* applications for
> > supercomputers of this level are few and far between. I'm happy with my
> > GFlop-level UP Athlon Thunderbird. And if Moore's Law (or the AMD equivalent
> > :-) still holds, in 12 months I'll have something twice as fast (I've had it
> > for six months already :-).
> 
> Two things.
> 
> 1) If a company (say, IBM) pays people to work on 8 / 16 way scalability
> because that's what they want out of Linux, then stopping development
> on that isn't going to get effort redirected to fixing your soundcard (yes,
> I realise you were being flippant, but the point's the same), the headcount
> is just going to disappear. AKA your choice isn't "patches for 8 way
> scalablilty, or patches for subsystem X that you're more interested in",
> your choice is "patches for 8-way scalabity, or no patches". Provided that
> those patches don't break anything else, you still win overall by getting them.
> 
> 2) Working on scalability for 8 / 16 way machines will show up races,
> performance problems et al that exist on 2 / 4 way machines but don't
> show up as often, or as obviously. I have a 16 way box that shows up
> races in the Linux kernel that might take you years to find on a 2 way.
> 
> What I'm trying to say is that you still win. Not as much as maybe you'd
> like, but, hey, it's work you're getting for free, so don't complain too 
> much about it. The maintainers are very good at beating the message
> into us that we can't make small systems any worse performing whilst
> making the big systems better.

Making it perform better, while not hurting up, and *not making code
messier* is good thing. Messiness of code might be as importnat as
performance, or even more important.
								Pavel
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: Over 4-way systems considered harmful :-)

[Todd Underwood]

folx,

although i agree with the sentiment expressed below, i beg to differ...

On Tue, 4 Dec 2001, M. Edward Borasky wrote:

> Sure, the military guys would *love* to have a petaflop engine, but
> they're gonna build 'em anyway and quite probably not bother to contribute
> their kernel source on this mailing list. *Commercial* applications for
> supercomputers of this level are few and far between. 

the CPlant (http://www.cs.sandia.gov/cplant/) system software (currently
built on top of Linux 2.2 but it's working now on 2.4 as well) is Open
Source (GPL).  CPlant is built by Sandia National Labs (which could be
interpreted as "the military guys" is is currently one of the largest
Linux-based supercomputers in the world.  The source for it is publicly
available and gives some interesting insight to what happens when you try
to scale beyond the "traditional" 32 or 64 processor cluster.

Additionally, the same systems software is being used in at least one big
commercial application (for processing genetic information).

my point here is not that building huge SMP machine makes sense
(obviously, for reasons that have been repeatedly reexplained by others
including Larry McVoy, it doesn't).  my point is just that many parts of
the national security aparatus are playing well in the Linux kernel
development space these days and it's foolish to write them off.

todd underwood, vp & cto
oso grande technologies, inc.
todd@osogrande.com

"Those who give up essential liberties for temporary safety deserve
neither liberty nor safety." - Benjamin Franklin

RE: Over 4-way systems considered harmful :-)

[M. Edward Borasky]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Todd Underwood
> Sent: Tuesday, December 04, 2001 12:49 PM
> To: M. Edward Borasky
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: Over 4-way systems considered harmful :-)
>
> folx,
>
> although i agree with the sentiment expressed below, i beg to differ...
>
> the CPlant (http://www.cs.sandia.gov/cplant/) system software (currently
> built on top of Linux 2.2 but it's working now on 2.4 as well) is Open
> Source (GPL).  CPlant is built by Sandia National Labs (which could be
> interpreted as "the military guys" is is currently one of the largest
> Linux-based supercomputers in the world.  The source for it is publicly
> available and gives some interesting insight to what happens when you try
> to scale beyond the "traditional" 32 or 64 processor cluster.
>
> Additionally, the same systems software is being used in at least one big
> commercial application (for processing genetic information).
>
> my point here is not that building huge SMP machine makes sense
> (obviously, for reasons that have been repeatedly reexplained by others
> including Larry McVoy, it doesn't).  my point is just that many parts of
> the national security aparatus are playing well in the Linux kernel
> development space these days and it's foolish to write them off.
>
> todd underwood, vp & cto
> oso grande technologies, inc.
> todd@osogrande.com
>
> "Those who give up essential liberties for temporary safety deserve
> neither liberty nor safety." - Benjamin Franklin

Yes ... but ... it is interesting how quickly the priorities and policies of
the National Laboratories can be "re-focused" with the change of the person
residing at 1600 Pennsylvania Avenue and "other factors" :-). Been there ...
done that ... got the scars :-). In short: "Moore's Law: good. Amdahl's Law:
bad" :-)
--
Take Your Trading to the Next Level!
M. Edward Borasky, Meta-Trading Coach

znmeb@borasky-research.net
http://www.meta-trading-coach.com
http://groups.yahoo.com/group/meta-trading-coach

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

>> Please Larry, have a look at the environment: nobody here owns a box  
>> with 128 CPUs. Most of the people here take care of things they either
>> - own themselves                                                      
>> - have their hands own at work                                        
>> - get paid for                                                        
>>                                                                       
>> You will not find _any_ match with 128 CPUs here.                     
> 
> Nor will you find any match with 4 or 8 CPU systems, except in very rare
> cases.  Yet changes go into the system for 8 way and 16 way performance.
> That's a mistake.
> 
> I'd be ecstatic if the hackers limited themselves to what was commonly 
> available, that is essentially what I'm arguing for.  

We need a *little* bit of foresight. If 4 ways are common now, and 8 ways 
and 16 ways are available, then in a year or two 8 ways (at least) will be
commonplace. On the other hand 128 cpu machines are a way off, and 
I'd agree we shouldn't spend too much time on them right now.

Martin.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Pavel Machek]

Hi!

> > cases.  Yet changes go into the system for 8 way and 16 way performance.
> > That's a mistake.
> > 
> > I'd be ecstatic if the hackers limited themselves to what was commonly 
> > available, that is essentially what I'm arguing for.  
> 
> We need a *little* bit of foresight. If 4 ways are common now, and 8 ways 
> and 16 ways are available, then in a year or two 8 ways (at least) will be
> commonplace. On the other hand 128 cpu machines are a way off, and 
> I'd agree we shouldn't spend too much time on them right now.

90% developers have more than one machine to play with... So maybe there's
time for mosix to be merged...?

Then we can create memnet (netdevice over shared memory), and Larry's dream
can come true...
								Pavel
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

> Then we can create memnet (netdevice over shared memory), and Larry's dream
> can come true...

I'm hoping, but my dreams do not include shared memory over a network.
That's just way too slow.  It's been done a pile of times, every time
people say that the caching will make it fast enough and those people
are wrong every time.

People who think DSM is a good idea are the same people who think a
millisecond is OK for a cache miss (current cache miss times are well
under .0002 milliseconds).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Wilson]

----- Original Message -----
From: "Larry McVoy" <lm@bitmover.com>
To: "Pavel Machek" <pavel@suse.cz>
Cc: "Martin J. Bligh" <Martin.Bligh@us.ibm.com>; "Larry McVoy"
<lm@bitmover.com>; "Stephan von Krawczynski" <skraw@ithnet.com>; "Horst von
Brand" <vonbrand@sleipnir.valparaiso.cl>; "lkml"
<linux-kernel@vger.kernel.org>
Sent: Thursday, December 06, 2001 3:50 PM
Subject: Re: Linux/Pro [was Re: Coding style - a non-issue]


> > Then we can create memnet (netdevice over shared memory), and Larry's
dream
> > can come true...
>
> I'm hoping, but my dreams do not include shared memory over a network.

He's talking about net over shared memory, not shared memory over net.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Stefan Smietanowski]

Larry McVoy wrote:

>>Please Larry, have a look at the environment: nobody here owns a box  
>>with 128 CPUs. Most of the people here take care of things they either
>>- own themselves                                                      
>>- have their hands own at work                                        
>>- get paid for                                                        
>>                                                                      
>>You will not find _any_ match with 128 CPUs here.                     
>>
> 
> Nor will you find any match with 4 or 8 CPU systems, except in very rare
> cases.  Yet changes go into the system for 8 way and 16 way performance.
> That's a mistake.
> 
> I'd be ecstatic if the hackers limited themselves to what was commonly 
> available, that is essentially what I'm arguing for.  

"There are only black cars, we don't make any other color. Noone has 
ordered a car with a different color cause we don't accept those orders. 
And since noone orders why add colors? Unnecessary cruft."

In essence, People don't run big boxes due to scalability issues, fixing 
those might get someone to install a 16-Way.

They sure as hell aren't gonna buy one and then wait around 3 years for 
Linux to support it.

// Stefan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> In essence, People don't run big boxes due to scalability issues, fixing 
> those might get someone to install a 16-Way.

Hackers don't run Linux on 16 way boxes because they cost $100,000 each

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Stefan Smietanowski]

Alan Cox wrote:

>>In essence, People don't run big boxes due to scalability issues, fixing 
>>those might get someone to install a 16-Way.
>>
> 
> Hackers don't run Linux on 16 way boxes because they cost $100,000 each
> 
> Alan
> 


And here I thought all linux hackers were millionaires :)

// Stefan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Henning P. Schmiedehausen]

Larry McVoy <lm@bitmover.com> writes:

>then 4->8, etc.  By your logic, someone should be sitting down and saying
>here is how you get to 128.  Other than myself, noone is doing that and
>I'm not really a Linux kernel hack, so I don't count.  

"No one that you know". I'm always surprised that you're able to speak
with such confidence. There may be lots of things going on that don't
daily report to you.

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Mon, Dec 03, 2001 at 11:01:24PM +0000, Henning P. Schmiedehausen wrote:
> Larry McVoy <lm@bitmover.com> writes:
> 
> >then 4->8, etc.  By your logic, someone should be sitting down and saying
> >here is how you get to 128.  Other than myself, noone is doing that and
> >I'm not really a Linux kernel hack, so I don't count.  
> 
> "No one that you know". I'm always surprised that you're able to speak
> with such confidence. There may be lots of things going on that don't
> daily report to you.

Right you are, but...  There is also piles of information that I absorb
on a daily basis, and I'm always willing to be educated.  For example,
you could educate me about all those 128 processor Linux boxes in the
world and fill in a hole in my knowledge.  I'm listening...
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

>> Larry McVoy <lm@bitmover.com> writes:
>> 
>> > then 4->8, etc.  By your logic, someone should be sitting down and saying
>> > here is how you get to 128.  Other than myself, noone is doing that and
>> > I'm not really a Linux kernel hack, so I don't count.  
>> 
>> "No one that you know". I'm always surprised that you're able to speak
>> with such confidence. There may be lots of things going on that don't
>> daily report to you.
> 
> Right you are, but...  There is also piles of information that I absorb
> on a daily basis, and I'm always willing to be educated.  For example,
> you could educate me about all those 128 processor Linux boxes in the
> world and fill in a hole in my knowledge.  I'm listening...

SGI has machines bigger than 128 (if memory serves, 1024??) that I thought
had booted Linux. The Sequent/IBM NUMA-Q archictecture now supports Linux 
and would go to 64 procs if we removed a few bitmap limitations from the kernel 
that have been patched out before (well, actually 60 is easy, 64 would require
some more apic modifications). 

Anyway, bigger than 8 way is no pipedream. I'll admit few people could afford
such machines, but they do exist.

Martin.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> you could educate me about all those 128 processor Linux boxes in the
> world and fill in a hole in my knowledge.  I'm listening...

There are at least two sets of people working on NUMA machines of that
order, as well as the IBM work on smaller NUMA systems.

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Lars Brinkhoff]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
> > you could educate me about all those 128 processor Linux boxes in the
> > world and fill in a hole in my knowledge.  I'm listening...
> There are at least two sets of people working on NUMA machines of that
> order, as well as the IBM work on smaller NUMA systems.

Are you NUMA people listening?  What do you think of Larry's ideas?

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

>> > you could educate me about all those 128 processor Linux boxes in the
>> > world and fill in a hole in my knowledge.  I'm listening...
>> There are at least two sets of people working on NUMA machines of that
>> order, as well as the IBM work on smaller NUMA systems.
> 
> Are you NUMA people listening?  What do you think of Larry's ideas?

I presume we're talking about what he calls ccClusters or SMP clusters.
I did a little background reading and found a couple of his old posts.
I'm not an expert on this, though I've done some work in the NUMA area. 
So I'll stick my neck out for people to chop off - I'm not sure I'd agree with 
some of his premises:

> Premise 1: SMP scaling is a bad idea beyond a very small number processors. 
>    The reasoning for this is that when you start out threading a kernel, 
>    it's just a few locks. That quickly evolves into more locks, and 
>    for a short time, there is a 1:1 mapping between each sort of object 
>    in the system (file, file system, device, process, etc) and a lock. 
>    So there can be a lot of locks, but there is only one reader/writer 
>    lock per object instance. This is a pretty nice place to be - it's 
>    understandable, explainable, and maintainable. 
>
>   Then people want more performance. So they thread some more and now 
>    the locks aren't 1:1 to the objects. What a lock covers starts to 
>    become fuzzy. Thinks break down quickly after this because what 
>    happens is that it becomes unclear if you are covered or not and 
>    it's too much work to figure it out, so each time a thing is added 
>    to the kernel, it comes with a lock. Before long, your 10 or 20 
>    locks are 3000 or more like what Solaris has. This is really bad, 
>    it hurts performance in far reaching ways and it is impossible to 
>    undo. 

OK, apart from the fact that there's some leaps of faith here (mostly
due to a lack of detail, I need to go and read some more of his papers),
the obvious objection to this is that just because he's seen it done badly
before, even that it's easy to do badly, it doesn't mean it's impossible to
do it well (it being scalability to many processors).

We will try to make it scale without breaking the low end systems. If we
can, all well and good. If we can't then our patches will get rejected
and we'll all be publicly flogged. Fair enough. And, yes, it's hard. And,
yes, it'll take a while. 

But whilst we gradually make scalability better, NUMA boxes will still
work in the meantime - just not quite as fast. I currently have a NUMA 
box that thinks it an SMP box ... it still works, just not particularly efficiently.
It will get better.

> Premise 3: it is far easier to take a bunch of operating system images 
>    and make them share the parts they need to share (i.e., the page 
>    cache), than to take a single image and pry it apart so that it 
>    runs well on N processors. 

Of course it's easier. But it seems like you're left with much more work to 
reiterate in each application you write to run on this thing. Do you want to 
do the work once in the kernel, or repeatedly in each application? I'd say
it's a damned sight easier to make an application work well on one big
machine than on a cluster.

I like Linus' opinion on the subject, which I'd boil down to "implement
both, see what works". We must have the most massivly parallel 
software engineering team for any OS ever - let's use it ;-)

Martin.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Rik van Riel]

On Tue, 4 Dec 2001, Martin J. Bligh wrote:

> > Premise 3: it is far easier to take a bunch of operating system images
> >    and make them share the parts they need to share (i.e., the page
> >    cache), than to take a single image and pry it apart so that it
> >    runs well on N processors.
>
> Of course it's easier. But it seems like you're left with much more
> work to reiterate in each application you write to run on this thing.
> Do you want to do the work once in the kernel, or repeatedly in each
> application?

There seems to be a little misunderstanding here; from what
I gathered when talking to Larry, the idea behind ccClusters
is that they provide a single system image in a NUMA box, but
with separated operating system kernels.

Of course, this is close to what a "single" NUMA kernel often
ends up doing with much ugliness, so I think Larry's idea to
construct NUMA OSes by putting individual kernels of nodes to
work together makes a lot of sense.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin J. Bligh]

>> > Premise 3: it is far easier to take a bunch of operating system images
>> >    and make them share the parts they need to share (i.e., the page
>> >    cache), than to take a single image and pry it apart so that it
>> >    runs well on N processors.
>> 
>> Of course it's easier. But it seems like you're left with much more
>> work to reiterate in each application you write to run on this thing.
>> Do you want to do the work once in the kernel, or repeatedly in each
>> application?
> 
> There seems to be a little misunderstanding here; from what
> I gathered when talking to Larry, the idea behind ccClusters
> is that they provide a single system image in a NUMA box, but
> with separated operating system kernels.

OK, then I've partially misunderstood this ... can people provide some 
more reference material? Please email to me, and I'll collate the results
back to the list (should save some traffic).

I have already the following:

http://www.bitmover.com/talks/cliq/slide01.html 
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0001.2/1172.html 
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0001.3/0236.html 
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0007.3/1222.html 

Thanks,

Martin.

SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

On Tue, Dec 04, 2001 at 03:37:37PM -0800, Martin J. Bligh wrote:
> >> > Premise 3: it is far easier to take a bunch of operating system images
> >> >    and make them share the parts they need to share (i.e., the page
> >> >    cache), than to take a single image and pry it apart so that it
> >> >    runs well on N processors.
> >> 
> >> Of course it's easier. But it seems like you're left with much more
> >> work to reiterate in each application you write to run on this thing.
> >> Do you want to do the work once in the kernel, or repeatedly in each
> >> application?
> > 
> > There seems to be a little misunderstanding here; from what
> > I gathered when talking to Larry, the idea behind ccClusters
> > is that they provide a single system image in a NUMA box, but
> > with separated operating system kernels.

Right except NUMA is orthogonal, ccClusters work fine on a regular SMP 
box.

> OK, then I've partially misunderstood this ... can people provide some 
> more reference material? Please email to me, and I'll collate the results
> back to the list (should save some traffic).

I'll try and type in a small explanation, I apologize in advance for the
bervity, I'm under a lot of pressure on the BK front these days...

The most recent set of slides are here:

    http://www.bitmover.com/ml/slide01.html

A couple of useful papers are at

    http://www.bitmover.com/llnl/smp.pdf
    http://www.bitmover.com/llnl/labs.pdf

The first explains why I think fine grained multi threading is a mistake
and the second is a paper I wrote to try and get LLNL to push for what
I called SMP clusters (which are not a cluster of SMPs, they are a 
cluster of operating system instances on a single SMP).

The basic idea is this: if you consider the usefulness of an SMP versus a
cluster, the main thing in favor of the SMP is

    all processes/processors can share the same memory at memory speeds.
    I typically describe this as "all processes can mmap the same data".
    A cluster loses here, even if it provides DSM over a high speed
    link, it isn't going to have 200 ns caches misses, it's orders of
    magnitude slower.  For a lot of MPI apps that doesn't matter, but
    there are apps for which high performance shared memory is required.

There are other issues like having a big fast bus, load balancing, etc.,
but the main thing is that you can share data quickly and coherently.
If you don't need that performance/coherency and you can afford to 
replicate the data, a traditional cluster is a *much* cheaper and 
easier answer.  Many problems, such as web server farms, are better
done on Beowulf style clusters than an SMP, they will actually scale
better.

OK, so suppose we focus on the SMP problem space.  It's a requirement
that all the processes on all the processors need to be able to access
memory coherently.  DSM and/or MPI isn't an answer for this problem 
space.

The traditional way to use an SMP is to take a single OS image and 
"thread" it such that all the CPUs can be in the OS at the same time.
Pretty much all the data structures need to get a lock and each CPU
takes the lock before it uses the data structure.  The limit of the
ratio of locks to cache lines is 1:1, i.e., each cache line will need
a lock in order to get 100% of the scaling on the system (yes, I know
this isn't quite true but it is close and you get the idea).

Go read the "smp.pdf" paper for my reasons on why this is a bad approach,
I'll assume for now you are willing to agree that it is for the purposes
of discussion.

If we want to get the most use out of big SMP boxes but we also want to
do the least amount of "damage" in the form of threading complexity in
the source base.  This is a "have your cake and eat it too" goal, one
that I think is eminently reachable.

So how I propose we do this is by booting multiple Linux images on
a single box.  Each OS image owns part of the machine, 1-4 CPUs, 0 or
more devices such as disk, ethernet, etc., part of memory.  In addition,
all OS images share, as a page cache, part of main memory, typically
the bulk of main memory.

The first thing to understand that the *only* way to share data is in
memory, in the globally shared page cache.  You do not share devices,
devices are proxied.  So if I want data from your disk or file system,
I ask you to put it in memory and then I mmap it.  In fact, you really
only share files and you only share them via mmap (yeah, read and write
as well but that's the uninteresting case).

This sharing gets complex because now we have more than one OS image
which is managing the same set of pages.  One could argue that the 
code complexity is just as bad as a fine grained multi threaded OS
image but that's simply incorrect.  I would hide almost 100% of this
code in a file system, with some generic changes (as few as possible)
in the VM system.  There are some changes in the process layer as well,
but we'll talk about them later.

If you're sitting here thinking about all the complexity involved in
sharing pages, it is really helpful to think about this in the following
way (note you would not actually implement it like this in the long
run but you could start this way):

Imagine that for any given file system there is one server OS image and N
client os images.  Imagine that for each client, there is a proxy process
running on behalf of the client on the server.  Sort of like NFS biods.
Each time the client OS wants to do an mmap() it asks the proxy to do
the mmap().  There are some corner cases but if you think about it, by
having the proxies do the mmaps, we *know* that all the server OS data
structures are correct.  As far as the server is concerned, the remote
OS clients are no different than the local proxy process.  This is from
the correctness point of view, not the performance point of view.

OK, so we've handled setting up the page tables, but we haven't handled
page faults or pageouts.  Let's punt on pageouts for the time being,
we can come back to that.  Let's figure out a pagefault path that will
give correct, albeit slow, behaviour.  Suppose that when the client faults
on a page, the client side file system sends a pagefault message to the
proxy, the proxy faults in the page, calls a new vtop() system call to
get the physical page, and passes that page descriptor back to the client
side.  The client side loads up the TLB & page tables and away we go.
Whoops, no we don't, because the remote OS could page out the page and
the client OS will get the wrong data (think about a TLB shootdown that
_didn't_ happen when it should have; bad bad bad).  Again, thinking 
just from the correctness point of view, suppose the proxy mlock()ed
the page into memory.  Now we know it is OK to load it up and use it.
This is why I said skip pageout for now, we're not going to do them 
to start with anyway.

OK, so start throwing stones at this.  Once we have a memory model that
works, I'll go through the process model.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[erich]

Larry McVoy <lm@bitmover.com> wrote:

> > > There seems to be a little misunderstanding here; from what
> > > I gathered when talking to Larry, the idea behind ccClusters
> > > is that they provide a single system image in a NUMA box, but
> > > with separated operating system kernels.
> 
> Right except NUMA is orthogonal, ccClusters work fine on a regular SMP 
> box.

...[details omitted for the moment]...

The basic idea seems a sound one, though maybe consider going from
a simple/robust cluster solution back to NUMA boxen.  A transparent
virtual computer image is kind of the goal long-term, right?

That is the plan I have for a different OS/kernel I'm working on,
and it seems valid so far.


Uni-proc design plus simple SMP only in the core kernel code fixes
SO many little SMP brain-deadnesses.

For example, there should be no reason that most drivers or any other
random kernel module should know anything about SMP.  Under Linux, it
annoys me to no end that I have to ever know (and yes, I count compiling
against "SMP" configuration having to know)...  more and more sources of
error.


Then, as long as you make the simple cluster solution handle "hot-swap/
bring-up/tear-down" of nodes from the beginning, you can easily do
things like bring new processor modules, machines, etc. online or out
of your system.


This even argues that you should try working from something like a
Mosix cluster as a starting point, to test it out.


--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: SMP/cc Cluster description [was Linux/Pro]

[Alan Cox]

> For example, there should be no reason that most drivers or any other
> random kernel module should know anything about SMP.  Under Linux, it
> annoys me to no end that I have to ever know (and yes, I count compiling
> against "SMP" configuration having to know)...  more and more sources of
> error.

Unfortunately the progression with processors seems to be going the
wrong way. Spin locks are getting more not less expensive. That makes
CONFIG_SMP=n a meaningful optimisation for the 99%+ of people not running
SMP


Alan

Re: SMP/cc Cluster description [was Linux/Pro]

[Jeff Merkey]

----- Original Message -----
From: "Alan Cox" <alan@lxorguk.ukuu.org.uk>
To: <erich@uruk.org>
Cc: "Larry McVoy" <lm@bitmover.com>; <linux-kernel@vger.kernel.org>
Sent: Wednesday, December 05, 2001 2:09 AM
Subject: Re: SMP/cc Cluster description [was Linux/Pro]


> > For example, there should be no reason that most drivers or any other
> > random kernel module should know anything about SMP.  Under Linux, it
> > annoys me to no end that I have to ever know (and yes, I count compiling
> > against "SMP" configuration having to know)...  more and more sources of
> > error.
>
> Unfortunately the progression with processors seems to be going the
> wrong way. Spin locks are getting more not less expensive. That makes
> CONFIG_SMP=n a meaningful optimisation for the 99%+ of people not running
> SMP

Amen.

Jeff

>
>
> Alan
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Loadable drivers [was SMP/cc Cluster description ]

[erich]

Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > For example, there should be no reason that most drivers or any other
> > random kernel module should know anything about SMP.  Under Linux, it
> > annoys me to no end that I have to ever know (and yes, I count compiling
> > against "SMP" configuration having to know)...  more and more sources of
> > error.
> 
> Unfortunately the progression with processors seems to be going the
> wrong way. Spin locks are getting more not less expensive. That makes
> CONFIG_SMP=n a meaningful optimisation for the 99%+ of people not running
> SMP

I appreciate the costs (having worked in the Intel x86 arch team) and don't
like that part either, but the question at hand is whether you need the SMP
locks in most of the compiled drivers/modules themselves.  That is in part
an artifact of the old architecture before they were loadable.


This really goes into a side-topic, but plainly:

The general driver/random module framework in Linux really needs to get
separated from the core kernel, and made so that it doesn't need to be
recompiled between minor kernel versions.  Perhaps even pulling the
drivers in general apart from each other, just aggregating releases for
convenience.

MS with Windows (and even DOS) went the right direction here.  In fact,
they have been hurting themselves by what lack of driver interoperability
there is even between Windows NT/2K/XP.  Admittedly they didn't have much
of a choice with their closed-source scheme, but it still is a better
solution from a usability and stability point of view in general.


Don't get me wrong, I only run MS Software on 2 of my 8 machines (and
have been trying to remove one of those with Wine/WineX), but I appreciate
a better overall solution when I see one.


I will go so far as to say, for the long term this is necessary for the
survival of Linux, and would help it's health even in the server arena.

For example, we need the ability to easily:
  --  Install random drivers on top of your kernel version and not
	have them disappear when changing kernels unless that is desired.

	This is just a royal pain for most people I've ever dealt with
	who are not kernel hackers, and rightly so.

  --  Install random kernels while retaining the same drivers for
	testing/stability purposes.

	I know that on my Linux servers, it bugs me that it's hard/nigh
	impossible sometimes to change the core kernel and still trust
	the drivers haven't drifted.


Those can usually be done by someone who is willing to/capable of
performing sleazy tricks or detailed hacking, but it's a burden.  It
gets prohibitive often enough to be very frustrating, and I'm more patient
than most I know with random kernel hacking.

Stability/control/compatibility is in general of far more concern to most
than tweaked performance of the core kernel against the drivers.  That
wouldn't be given up either, it just becomes an option.


A lesser solution, but moderately workable in the short-term:

Build a framework for an external "drivers" source/binary tree that can be
(via a single command) rebuilt for a different Linux kernel version.  This
is arguably a Linux distribution thing.


--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: Loadable drivers [was SMP/cc Cluster description ]

[erich]

I wrote:

> This really goes into a side-topic, but plainly:
> 
> The general driver/random module framework in Linux really needs to get
> separated from the core kernel, and made so that it doesn't need to be
> recompiled between minor kernel versions.  Perhaps even pulling the
> drivers in general apart from each other, just aggregating releases for
> convenience.

...and licensing issues aside (probably only GPL for the moment), this
kind of thing could form the basis for an open-source kernel/OS-independent
set of drivers.

I know that I have been intending on doing a Linux driver compatibility
layer of sorts for my OS project (going to be GPL'd) I'm working on.

Maybe there are some others interested in this kind of project?

--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: Loadable drivers [was SMP/cc Cluster description ]

[Stephan von Krawczynski]

On Wed, 05 Dec 2001 11:40:07 -0800
erich@uruk.org wrote:

> This really goes into a side-topic, but plainly:
> 
> The general driver/random module framework in Linux really needs to get
> separated from the core kernel, and made so that it doesn't need to be
> recompiled between minor kernel versions.  Perhaps even pulling the
> drivers in general apart from each other, just aggregating releases for
> convenience.

You have just expressed my wildest nightmares. Just go ahead ...

> MS with Windows (and even DOS) went the right direction here.  In fact,
> they have been hurting themselves by what lack of driver interoperability
> there is even between Windows NT/2K/XP.  Admittedly they didn't have much
> of a choice with their closed-source scheme, but it still is a better
> solution from a usability and stability point of view in general.

You can only be plain kidding with this statement. If you split the drivers
from the rest of the kernel, you managed to get rid of this nice (yes, I really
meant nice) monolithic design, where I only need a simple config file to
_update_ to a new kernel revision (_up_, not _down_) and be happy (including
all the drivers). Obviously you just prove yourself wrong - mentioning not
working driver interoperability between NT/2K/XP whatever - with the idea that
it is indeed possible to make major new kernel versions (which should be
getting better btw) without _any_ changes in the driver framework that will
break your nice and stable but _old_ drivers. What's the use of this? You are
not talking about releasing driver-plugin-/framework-patches or the like just
to load _old_ drivers into _new_ kernel-environment?
Is this what they do at MS? Well if, they have not come that far.

> Don't get me wrong, I only run MS Software on 2 of my 8 machines (and
> have been trying to remove one of those with Wine/WineX), but I appreciate
> a better overall solution when I see one.

This brings up memories about reading Peter Nortons' latest hardcore books some
years ago, brrrr.

Reading between your lines, I can well see that you are most probably talking
about closed-source linux-drivers breaking with permanently released new kernel
revisions. But, in fact, this is the closed-source phenomenon, and _not_ linux.

> I will go so far as to say, for the long term this is necessary for the
> survival of Linux, and would help it's health even in the server arena.

You just invented the circle with edges: you are talking about "long-term" and
"MS drivers solution" at the same time. Remember, this company is currently
saying the product cycle is 3 years. I know pretty big companies that haven't
even managed to get the NT drivers really working under W2K, and were just shot
down by XP release.
I tend to believe we could just wait another two MS cycles to have even the
biggest MS-fans converted to kernel-hackers, only because of being real fed up
with the brilliant, long term driver design.
 
Regards,
Stephan

Re: Loadable drivers [was SMP/cc Cluster description ]

[erich]

Stephan von Krawczynski <skraw@ithnet.com> wrote:

...
> You can only be plain kidding with this statement. If you split the
> drivers from the rest of the kernel, you managed to get rid of this
> nice (yes, I really meant nice) monolithic design, where I only need
> a simple config file to _update_ to a new kernel revision (_up_, not
> _down_) and be happy (including all the drivers).

Hmm.  There is little fundamentally incompatible with having splits in
the core kernel and sets of drivers, and getting most of what you want
here.

However, the comment about being "happy (including all the drivers)"
seems a bit naive in my experience.  That makes the assumption that the
drivers in the new/old/whatever kernel you change to are necessarily of
the same caliber as the ones you came from, and that is not always true
by any means.

Also, in a world where one values stability, being able to rev
backward is quite important also.  The lack of software package tools
like rpm being able to "downgrade" software easily is a serious one in
my mind.  What do you do when you install something major and it
breaks, but it has interdependencies on multiple other things?

I don't object to producing well-tested full sets of driver source that
go with kernel versions, I just don't want them to be tied if I have
a need to pull something apart (a driver from one version and from
another are the only ones that run stably), which frankly happens too
often for my taste.  Even if it didn't I'd still want it as much as
possibly for the fall-back options it provides.


...
> working driver interoperability between NT/2K/XP whatever - with the
> idea that it is indeed possible to make major new kernel versions (which
> should be getting better btw) without _any_ changes in the driver
> framework that will break your nice and stable but _old_ drivers. What's
> the use of this? You are not talking about releasing driver-plugin-/
> framework-patches or the like just to load _old_ drivers into _new_
> kernel-environment?
> Is this what they do at MS? Well if, they have not come that far.

I didn't go far enough with my comment.  I should have said that the
lack of a common driver framework base that works between *all* (or nearly
all, some obsolescence is ok) their Windows versions is a problem.

In the consumer line (win3.1/95/98/me) where they were keeping very good
compatibility across some kinds of drivers (it had it's problems, to be
sure), they were trying the hardest because they recognized that the
most important thing was to just have things work in the first place.

NOTE:  I'm not endorsing their overall API convenience (driver or
Win16/Win32), stability, suitability, merchantability, whatever, I'm
just talking about drivers and their distribution model at the moment.


...
> Reading between your lines, I can well see that you are most probably
> talking about closed-source linux-drivers breaking with permanently
> released new kernel revisions. But, in fact, this is the closed-source
> phenomenon, and _not_ linux.

No, though I don't object to closed-source in general per se.  I hate it
for myself and businesses I've worked for because I like to be able to
fix/improve/whatever code, but I recognize that the majority of users
out there would never touch code.

My general feeling is that binary drivers are ok/should be supported well
across versions since that is the thing you load in at boot/bring-system-
up time.  Having separate (and usually many) step(s) to getting a driver
and having it load on your system is a major and I'm thinking artificial
pain.


In general my argument stems from the same basis that the kernel/user-level
interface does:  keeping interfaces stable and removing packaging/bundling
requirements across major boundaries almost always yield a win somewhere.

In the case of MS and drivers, the win they have in convenience to end-users
of all types, and the ability to mix and match drivers forward or backward
up to some limitations in API revisions.


...
> I tend to believe we could just wait another two MS cycles to have even
> the biggest MS-fans converted to kernel-hackers, only because of being
> real fed up with the brilliant, long term driver design.

Most people will never touch code or a compiler, and just want a simple
obvious formula or even to have the system automagically do the Right
Things for you.  Even many programmers have limitations of curiousity or
energy, and it isn't a bad thing to organize even core system things to
be easy.

MS may be stupid/annoying in other ways, but they know this and have
moved toward it, albeit sluggishly at times.


--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: Loadable drivers [was SMP/cc Cluster description ]

[Stephan von Krawczynski]

On Wed, 05 Dec 2001 13:17:47 -0800
erich@uruk.org wrote:

> Stephan von Krawczynski <skraw@ithnet.com> wrote:
> > You can only be plain kidding with this statement. If you split the
> > drivers from the rest of the kernel, you managed to get rid of this
> > nice (yes, I really meant nice) monolithic design, where I only need
> > a simple config file to _update_ to a new kernel revision (_up_, not
> > _down_) and be happy (including all the drivers).
> 
> Hmm.  There is little fundamentally incompatible with having splits in
> the core kernel and sets of drivers, and getting most of what you want
> here.

There is. You double the administrational work. Most updates are performed
because you want the latest bugfixes. If you get more bugfixes than you
intended - lucky.
Currently you draw the latest kernel-tree (or the latest _you_ like to use),
use your old .config and it works out - as long as you do not use closed-source
drivers.
If you split things up, you draw _two_ archives, compile and install both.

> However, the comment about being "happy (including all the drivers)"
> seems a bit naive in my experience.  That makes the assumption that the
> drivers in the new/old/whatever kernel you change to are necessarily of
> the same caliber as the ones you came from, and that is not always true
> by any means.

Oops. Then you have a severe problem in understanding how linux _should_ be
worked with. Obviously you can go and buy some distribution and it may work out
pretty well. But being a serious administrator you _will_ do kernel updates
apart from the distro (sooner or later). If there are drivers in a newer kernel
release, that do not work any longer, compared to an older release, you should
say so. Best place would be the listed maintainer. Because if it doesn't do the
job for your configuration any longer, chances are high others were hit, too.
The maintainer cannot know, if you do not tell him.
I guess we have the mutual agreement here, that maintained drivers should get
better not worse through the revisions. This means configurations once
successful are not meant to break later on.
This basically is what I intended to say with "being happy" :-)

> Also, in a world where one values stability, being able to rev
> backward is quite important also.  The lack of software package tools
> like rpm being able to "downgrade" software easily is a serious one in
> my mind.  What do you do when you install something major and it
> breaks, but it has interdependencies on multiple other things?

This isn't kernel-related, is it? You can always boot the previous kernel (and
drivers), if you updated correctly. I am talking about revision-updates, not
major updates (like from 2.0 to 2.2). Major update will break mostly. But -
frankly spoken - it should. If you are using a bunch of very old binaries, you
should be driven to update these anyway.

> I don't object to producing well-tested full sets of driver source that
> go with kernel versions, I just don't want them to be tied if I have
> a need to pull something apart (a driver from one version and from
> another are the only ones that run stably), which frankly happens too
> often for my taste.  Even if it didn't I'd still want it as much as
> possibly for the fall-back options it provides.

Can you give a striking example for this? (old driver ok, new driver broken,
both included in kernel releases)

> I didn't go far enough with my comment.  I should have said that the
> lack of a common driver framework base that works between *all* (or nearly
> all, some obsolescence is ok) their Windows versions is a problem.

Which means: it does not work there.

> In the consumer line (win3.1/95/98/me) where they were keeping very good
> compatibility across some kinds of drivers (it had it's problems, to be
> sure), they were trying the hardest because they recognized that the
> most important thing was to just have things work in the first place.

I honour they were _trying_, only it was not successful. I wrote drivers for
all noted versions above and can tell you: it is a _mess_.

> NOTE:  I'm not endorsing their overall API convenience (driver or
> Win16/Win32), stability, suitability, merchantability, whatever, I'm
> just talking about drivers and their distribution model at the moment.

Ok, lets put it this way: they use a completely split up model of driver
maintenance to get the most out of the market (anybody can release anything at
anytime and any quality and interoperability), and therefore everything tends
to be severly broken and complex in administration. You have to draw the latest
drivers for your graphics card, scsi card, scanner, printer, USB equipment,
even monitor from the respective source (manufacturer), install it seperately
and pray it does not shoot your last driver installation from different
hardware component - which it does in a significant percentage. And if you
upgrade from (e.g.) win95 to win98, you will draw all drivers again and
completely reinstall everything.
To tell the pure truth: nobody cares about anything on w*indoze.

> > Reading between your lines, I can well see that you are most probably
> > talking about closed-source linux-drivers breaking with permanently
> > released new kernel revisions. But, in fact, this is the closed-source
> > phenomenon, and _not_ linux.
> 
> No, though I don't object to closed-source in general per se.  I hate it
> for myself and businesses I've worked for because I like to be able to
> fix/improve/whatever code, but I recognize that the majority of users
> out there would never touch code.

They _should_ not do that. And they _need_ not do that today. The distros
really got very far on the path to the real lusers, that don't know much and
don't want to learn anything. This is ok.
I mean have you had a look at the latest distros, I found it amazing how far
things have already come. You can install client systems under linux in 20% of
the time you would need for _any_ windows.

> My general feeling is that binary drivers are ok/should be supported well
> across versions since that is the thing you load in at boot/bring-system-
> up time.  Having separate (and usually many) step(s) to getting a driver
> and having it load on your system is a major and I'm thinking artificial
> pain.

I have learned something over the recent years: I guess RMS pointed in the
right direction. I _don't_ think binary drivers are ok. I want to control my
environment, and don't let _anybody_ control it _for_ me. And if something goes
wrong, I have a look. And if I am too dumb, I can ask somebody who isn't. And
there may be a lot of those.

> In general my argument stems from the same basis that the kernel/user-level
> interface does:  keeping interfaces stable and removing packaging/bundling
> requirements across major boundaries almost always yield a win somewhere.

Carl Sagan "The Demon Haunted World":
"If there is a chain of argument every link in the chain must work."

> In the case of MS and drivers, the win they have in convenience to end-users
> of all types, and the ability to mix and match drivers forward or backward
> up to some limitations in API revisions.

Ah, yes. Indeed I am one of those thinking that a driver should _work_, I do
not measure its quality on the number of versions available - and loadable in
my environment. If I have to this, the driver is _broken_.
But yes, you are right: 99% of all w-users obviously think internet is designed
for downloading the latest w-drivers, and it is a definitive must to have all
revisions to find one working.

> > I tend to believe we could just wait another two MS cycles to have even
> > the biggest MS-fans converted to kernel-hackers, only because of being
> > real fed up with the brilliant, long term driver design.
> 
> Most people will never touch code or a compiler, and just want a simple
> obvious formula or even to have the system automagically do the Right
> Things for you.  Even many programmers have limitations of curiousity or
> energy, and it isn't a bad thing to organize even core system things to
> be easy.

I talked about the developers, but even talking about users, I do believe that
...

> MS may be stupid/annoying in other ways, but they know this and have
> moved toward it, albeit sluggishly at times.

... it is not their ultimate goal to let MS control their lives. You think this
is over-estimation? They are already handing out electronic _passports_, sorry,
but in my understanding of democracy this is not done by commercial companies.
This is what a state is all about - knowing and identifying its citizens - and
no one _else_.

MS is not stupid. I won't tell you what it is though, find out yourself.

Regards,
Stephan

Re: Loadable drivers [was SMP/cc Cluster description ]

[Kai Henningsen]

skraw@ithnet.com (Stephan von Krawczynski)  wrote on 06.12.01 in <20011206173455.104b6a02.skraw@ithnet.com>:

> I have learned something over the recent years: I guess RMS pointed in the
> right direction. I _don't_ think binary drivers are ok. I want to control my
> environment, and don't let _anybody_ control it _for_ me. And if something
> goes wrong, I have a look. And if I am too dumb, I can ask somebody who
> isn't. And there may be a lot of those.

And it is absolutely amazing what you *can* do, if you have the soure,  
what you never expected to be able to do, but because you *do* have the  
source, you just *have* to look at the problem area ... and poof! there's  
something that certainly doesn't look right, maybe if you just try to  
change this little bit ...

And thus you learn something new which you wouldn't even have tried with  
closed source.

It's not only this. I remember when I first tried to get PPP to work - no,  
wait, back then it was SLIP. (Slightly before Linux 1.0, I think.) It just  
wouldn't work, and I had no idea why. Obviously I was doing something  
wrong, but what?

So I looked at the relevant kernel part. Still rather unclear, but the  
data has to go through *here* ... now suppose I insert a printk there, and  
one there, and then reboot and retry and watch syslog ... aha! (Well,  
actually, it took me several passes, and I don't remember what the problem  
turned out to be except it wasn't a kernel bug.)

MfG Kai

Re: Loadable drivers [was SMP/cc Cluster description ]

[erich]

Stephan von Krawczynski <skraw@ithnet.com> wrote:

> erich@uruk.org wrote:
> 
> > Hmm.  There is little fundamentally incompatible with having splits in
> > the core kernel and sets of drivers, and getting most of what you want
> > here.
> 
> There is. You double the administrational work. Most updates are
> performed because you want the latest bugfixes. If you get more bugfixes
> than you intended - lucky.
> Currently you draw the latest kernel-tree (or the latest _you_ like to
> use), use your old .config and it works out - as long as you do not use
> closed-source drivers.

And if something breaks out of it - unlucky?

I again think your statement that "...and it works out" is naive.

See below for more.


> If you split things up, you draw _two_ archives, compile and install
> both.

Actually, that's something of my point.

I think the control to install one or the other, certainly for the
systems I want to be most reliable (firewalls/other servers), is of
high benefit, hence my proposing this in the first place.

All these things tied together means they all stand or fall together
for most people, it's only people like many on this list or other
kernel hackers that end up being able to do anything about it.

There are many important reasons why those in charge of so many
production computing environments around the world try to, when
faced with some broken component, want to *only* change that one
and not all or even a large subset of them.  It is risky.

Even if you want to upgrade wholesale, then the ability to rev
backward something that doesn't work in the new set, but you know
works in the old form, is very valuable.


> Oops. Then you have a severe problem in understanding how linux _should_
> be worked with. Obviously you can go and buy some distribution and it
> may work out pretty well. But being a serious administrator you _will_
> do kernel updates apart from the distro (sooner or later). If there are
> drivers in a newer kernel release, that do not work any longer, compared
> to an older release, you should say so. Best place would be the listed
> maintainer. Because if it doesn't do the job for your configuration any
> longer, chances are high others were hit, too.
> The maintainer cannot know, if you do not tell him.
> I guess we have the mutual agreement here, that maintained drivers
> should get better not worse through the revisions. This means
> configurations once successful are not meant to break later on.
> This basically is what I intended to say with "being happy" :-)

OK, you're telling me that people "should fix it", but my point, and
one I haven't heard any contradiction to yet, is that often enough
people don't.

So, the response has been to hold the drivers close in so when
interfaces change/a bug is found, they can all be changed together.
I think there is some merit to that when you find a bug, but
otherwise this becomes a statistical shooting gallery.

My reasoning?  Each time driver code is touched, in particular without
testing done afterward, there is some probability of another bug
being induced.


> > I don't object to producing well-tested full sets of driver source that
> > go with kernel versions, I just don't want them to be tied if I have
> > a need to pull something apart (a driver from one version and from
> > another are the only ones that run stably), which frankly happens too
> > often for my taste.  Even if it didn't I'd still want it as much as
> > possibly for the fall-back options it provides.
> 
> Can you give a striking example for this? (old driver ok, new driver
> broken, both included in kernel releases)

Not really striking per se, but 2 off of the top of my head I've had to
deal with:

  --  integrated 10/100 MBit Enet network in a SiS735 motherboard,
      effectively SiS900 chipset.  Worked fine in later RH 7.1
      kernel (2.4.3-rhat), but not with 7.2 kernel(s) (2.4.7-rhat &
      2.4.9-rhat).
  --  drivers for the Lucent pcmcia Orinoco series of cards work/
      don't work across different revisions from 2.4.0-ish -> present.


Yes, I've reported various problems with other bits before, and
debugged some of my own, submitted patches, etc.

You can make the argument that some of them are young drivers/etc., but
that just proves my point that I want the control, not just for me, but
for others to easily load across drivers across different kernel
versions.

My issue with the current process is that it's developer-centric, and
though I've been able to work past these problems, none of the non-
kernel hackers I've known would generally care to.

In fact, several others who I know with similar hardware just gave up
when I told them they had to install a kernel/hacked driver patches.


> Ok, lets put it this way: they use a completely split up model of
> driver maintenance to get the most out of the market (anybody can
...
> hardware component - which it does in a significant percentage. And if
> you upgrade from (e.g.) win95 to win98, you will draw all drivers again
> and completely reinstall everything.
> 
> To tell the pure truth: nobody cares about anything on w*indoze.

I can't believe you just said that.  Maybe nobody *on this list*, but
in general, this makes it hard to take you at all seriously.


...
> > My general feeling is that binary drivers are ok/should be supported well
> > across versions since that is the thing you load in at boot/bring-system-
> > up time.  Having separate (and usually many) step(s) to getting a driver
> > and having it load on your system is a major and I'm thinking artificial
> > pain.
> 
> I have learned something over the recent years: I guess RMS pointed in the
> right direction. I _don't_ think binary drivers are ok. I want to control
> my environment, and don't let _anybody_ control it _for_ me. And if
> something goes wrong, I have a look. And if I am too dumb, I can ask
> somebody who isn't. And there may be a lot of those.

Huh?  I didn't ask you to use them.

I'm just pointing out that binary drivers are the end-product (you don't
"insmod driver.c"), so it's kind of hard to just hand-wave them out of
existence.  Not everyone has a fully self-hosted system with compilers
just sitting around to do your bidding.

Also, the model you present above is only suitable for a small fraction
of users.  It is irrelevant to most users if the code is fixable.  If
it doesn't work as is and there is no easy other thing they can try,
it is effectively broken.


--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: Loadable drivers [was SMP/cc Cluster description ]

[Stephan von Krawczynski]

On Thu, 06 Dec 2001 16:37:36 -0800
erich@uruk.org wrote:

> > If you split things up, you draw _two_ archives, compile and install
> > both.
> 
> Actually, that's something of my point.
> 
> I think the control to install one or the other, certainly for the
> systems I want to be most reliable (firewalls/other servers), is of
> high benefit, hence my proposing this in the first place.

Only in an environment with little evolution going on. But whats the use of
updating anyway, then?
Another thing that really jumps right into my face from your arguments:
you say "administrator needs to have a way to change core kernel, without
changing any drivers, and is not able to read and understand the source of both
core and drivers, so it is best for him to change as little as possible."
But wait: how does _he_ know then, that a standard update (kernel _and_
drivers, as currently) will not work. I mean, you trust the work of some people
building one set of core and drivers, but do not trust the same people to give
you a newer set of core and drivers? And all this, although you are not able or
willing to have a look at the _sources_ to really make sure.
I am definitely not fond of the proposed MS idea, that anybody can administrate
any server, he only needs to know how to perform mouse-clicks. If you can't
drive a car, you should probably not take the seat at the drivers side.

> All these things tied together means they all stand or fall together
> for most people, it's only people like many on this list or other
> kernel hackers that end up being able to do anything about it.

What is wrong with this idea? If you are joe-user, then take your preferred
distro and live with it (quite well nowadays). If you are an experienced user,
then tune your setup according to your needs. If you are really good at it,
then get yourself a job as an administrator for servers people work and rely on
- and read sources. And last, if you have some ideas, go ahead and write source
and give it to the public to verify and tune its quality. And if your real good
it may well be, that god will take your code and you suddenly become one of the
kernel-driver-maintainers. This is a pretty simple and straight-forward world.
And there is noone coming round the corner and tell you: "btw, we changed the
world just yesterday, everything is easy, only you should throw away your old
code."

> There are many important reasons why those in charge of so many
> production computing environments around the world try to, when
> faced with some broken component, want to *only* change that one
> and not all or even a large subset of them.  It is risky.

I really don't get it: if your system works, why upgrade? if your system does
not work: why did you give it to the people in the first place?

> Even if you want to upgrade wholesale, then the ability to rev
> backward something that doesn't work in the new set, but you know
> works in the old form, is very valuable.

Ever heard of lilo? It can boot multiple kernel-images. And guess what: they
are all finding their corresponding modules if you do it right.

> OK, you're telling me that people "should fix it", but my point, and
> one I haven't heard any contradiction to yet, is that often enough
> people don't.

Interesting. I in fact never heard of anyone complaining about a driver (that
is maintained) that does not get fixed if you tell the people about a problem.
If they don't have the hardware to test, simply give it to them.

> My reasoning?  Each time driver code is touched, in particular without
> testing done afterward, there is some probability of another bug
> being induced.

There is always another bug. This is fundamental in software development. This
is why it is really important to talk about bugs you came across. If you do not
use new code, no code will ever become better.

> > Can you give a striking example for this? (old driver ok, new driver
> > broken, both included in kernel releases)
> 
> Not really striking per se, but 2 off of the top of my head I've had to
> deal with:
> 
>   --  integrated 10/100 MBit Enet network in a SiS735 motherboard,
>       effectively SiS900 chipset.  Worked fine in later RH 7.1
>       kernel (2.4.3-rhat), but not with 7.2 kernel(s) (2.4.7-rhat &
>       2.4.9-rhat).

Okay, this is my personal tip for administrators: do not use distro-kernels, do
use linus-kernels, and do not update to very fresh ones. If you wanna be
brilliant, read LKML to know whats going on. This is no hint for joe-user.

>   --  drivers for the Lucent pcmcia Orinoco series of cards work/
>       don't work across different revisions from 2.4.0-ish -> present.

Haha. This is a real good example for real life not matching my ideas, perhaps
david hinds and linus should find some solution for it. If you want to have a
working set of lucent pcmcia you need to draw the latest pcmcia-package and
tools from david and install them on top of standard kernel. This has worked
since the civil war, believe me, I use it everyday with _lots_ of different
kernels and hardware setups. Only annoying part: it is not part of the kernel.
Therefore it is pretty interesting to hear you complaining about a _split away_
drivers' issue. :-)

> My issue with the current process is that it's developer-centric, and
> though I've been able to work past these problems, none of the non-
> kernel hackers I've known would generally care to.

Tell you what, this happened about 2 months ago:
I build a nice and simple hardware for a desktop. It contained Asus Board,
PIII-500, 512MB, ATI graphics card, IDE-10G-hd and ATAPI cdrom, keyboard,
PS/2-mouse and soundblaster . It didn't look all that non-standard.
I tried to install:

win95-orig: 
install fails right away, win95 cannot CD-boot

win98:
doesn't install with 512 MB, install fails.

win ME:
doesn't even get recognised by the BIOS as bootable CD

win 2K:
hangs in hardware detection cycle, I switched it off after 3 hours of continous
hd seeking at about 50%.

SuSE 7.2 (Linux):
install takes about 15 minutes, everything works.

What do we learn of this: win scales everybody down to joe-user-level, and you
cannot do a thing against it. I guess it is definitely not developer-centric. 

> > To tell the pure truth: nobody cares about anything on w*indoze.
> 
> I can't believe you just said that.  Maybe nobody *on this list*, but
> in general, this makes it hard to take you at all seriously.

I have seen to many things on windows to come to a different conclusion.

> Also, the model you present above is only suitable for a small fraction
> of users.  It is irrelevant to most users if the code is fixable.  If
> it doesn't work as is and there is no easy other thing they can try,
> it is effectively broken.

Again, you are talking about joe-user here. If joe-user has a problem, he
should call up the distro-hotline he bought, they will help him with
installation problems (mostly for free). He does not need a compiler or
anything.

And the experienced guy (or girl, hey there are some :-) will have a large
number of options available, including the source and his (her) brain.

Regards,
Stephan

Re: Loadable drivers [was SMP/cc Cluster description ]

[Keith Owens]

On Wed, 05 Dec 2001 11:40:07 -0800, 
erich@uruk.org wrote:
>Build a framework for an external "drivers" source/binary tree that can be
>(via a single command) rebuilt for a different Linux kernel version.  This
>is arguably a Linux distribution thing.

kbuild 2.5.  Already done.

Re: Loadable drivers [was SMP/cc Cluster description ]

[erich]

Keith Owens <kaos@ocs.com.au> wrote:

> On Wed, 05 Dec 2001 11:40:07 -0800, 
> erich@uruk.org wrote:
> >Build a framework for an external "drivers" source/binary tree that can be
> >(via a single command) rebuilt for a different Linux kernel version.  This
> >is arguably a Linux distribution thing.
> 
> kbuild 2.5.  Already done.

Cool.  Thanks for the good work guys!

[dang, now I have to go off and learn all about kbuild 2.5...]

--
    Erich Stefan Boleyn     <erich@uruk.org>     http://www.uruk.org/
"Reality is truly stranger than fiction; Probably why fiction is so popular"

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Tue, 4 Dec 2001 16:36:46 -0800
   
   OK, so start throwing stones at this.  Once we have a memory model that
   works, I'll go through the process model.

What is the difference between your messages and spin locks?
Both seem to shuffle between cpus anytime anything interesting
happens.

In the spinlock case, I can thread out the locks in the page cache
hash table so that the shuffling is reduced.  In the message case, I
always have to talk to someone.

Re: SMP/cc Cluster description

[Larry McVoy]

On Tue, Dec 04, 2001 at 06:36:01PM -0800, David S. Miller wrote:
>    From: Larry McVoy <lm@bitmover.com>
>    Date: Tue, 4 Dec 2001 16:36:46 -0800
>    
>    OK, so start throwing stones at this.  Once we have a memory model that
>    works, I'll go through the process model.
> 
> What is the difference between your messages and spin locks?
> Both seem to shuffle between cpus anytime anything interesting
> happens.
> 
> In the spinlock case, I can thread out the locks in the page cache
> hash table so that the shuffling is reduced.  In the message case, I
> always have to talk to someone.

Two points: 

a) if you haven't already, go read the Solaris doors paper.  Now think about
   a cross OS instead of a cross address space doors call.  They are very
   similar.  Think about a TLB shootdown, it's sort of a doors style cross
   call already, not exactly the same, but do you see what I mean?

b) I am not claiming that you will have less threading in the page cache.
   I suspect it will be virtually identical except that in my case 90%+
   of the threading is in the ccCluster file system, not the generic code.
   I do not want to get into a "my threading is better than your threading"
   discussion.  I'll even go so far as to say that this approach will have
   more threading if that makes you happy, as long as we agree it is outside
   of the generic part of the kernel.  So unless I have CONFIG_CC_CLUSTER,
   you pay nothing or very, very little.

   Where this approach wins big is everywhere except the page cache.  Every
   single data structure in the system becomes N-way more parallel -- with
   no additional locks -- when you boot up N instances of the OS.  That's
   a powerful statement.  Sure, I freely admit that you'll add a few locks
   to deal with cross OS synchronization but all of those are configed out
   unless you are running a ccCluster.  There is absolutely zero chance that
   you can get the same level of scaling with the same number of locks using
   the traditional approach.  I'll go out on a limb and predict it is at
   least 100x as many locks.  Think about it, there are a lot of things that
   are threaded simply because of some corner case that happens to show up
   in some benchmark or workload.  In the ccCluster case, those by and large
   go away.  Less locks, less deadlocks, less memory barriers, more reliable,
   less filling, tastes great and Mikey likes it :-)
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Tue, 4 Dec 2001 19:23:17 -0800

      There is absolutely zero chance that you can get the same level
      of scaling with the same number of locks using the traditional
      approach.  I'll go out on a limb and predict it is at least 100x as many locks.

Coming from the background of having threaded from scratch a complete
networking stack (ie. my shit doesn't stink and I'm here to tell you
about it :-))) I think your claims are pretty much out of whack.

Starting from initial implementation to having all the locking
disappear from the kernel profiles during any given load was composed
of three tasks:

	1) Is this object almost entirely reader based (or the corrolary)?
	   Use special locking that exploits this.  See linux/brlock.h
	   for one such "special locking" we invented to handle these
	   cases optimally.

	   This transformation results in ZERO shared dirty cache
	   lines if the initial analysis is correct.

	2) Can we "fan out" the locking so that people touching
           seperate objects %99 of the time touch different cache
	   lines?

	   This doesn't mean "more per-object locking", it means more
	   like "more per-zone locking".  Per-hashchain locking falls
	   into this category and is very effective for any load I
	   have ever seen.

	3) Is this really a per-cpu "thing"?  The per-cpu SKB header
	   caches are an example of this.  The per-cpu SLAB magazines
	   are yet another.

Another source of scalability problems has nothing to do with whether
you do some kind of clustering or not.  You are still going to get
into situations where multiple cpus want (for example) page 3 of
libc.so :-)  (what I'm trying to say is that it is a hardware issue
in some classes of situations)

Frankly, after applying #1 and/or #2 and/or #3 above to what shows up
to have contention (I claim the ipv4 stack to have had this done for
it) there isn't much you are going to get back.  I see zero reasons to
add any more locks to ipv4, and I don't think we've overdone it in the
networking either.

Even more boldly, I claim that Linux's current ipv4 scales further
than anything coming out of Sun engineering.  From my perspective
Sun's scalability efforts are more in line with "rubber-stamp"
per-object locking when things show up in the profiles than anything
else.  Their networking is really big and fat.  For example the
Solaris per-socket TCP information is nearly 4 times the size of that
in Linux (last time I checked their headers).  And as we all know
their stuff sits on top of some thick infrastructure (ie. STREAMS)
(OK, here is where someone pops up a realistic networking benchmark
where we scale worse than Solaris.  I would welcome such a retort
because it'd probably end up being a really simple thing to fix.)

My main point: I think we currently scale as far as we could in the
places we've done the work (which would include networking) and it
isn't "too much locking".

The problem areas of scalability, for which no real solution is
evident yet, involve the file name lookup tree data structures,
ie. the dcache under Linux.  All accesses here are tree based, and
everyone starts from similar roots.  So you can't per-node or
per-branch lock as everyone traverses the same paths.  Furthermore you
can't use "special locks" as in #1 since this data structure is
neither heavy reader nor heavy writer.

But the real point here is that SMP/cc clusters are not going to solve
this name lookup scaling problem.

The dcache_lock shows up heavily on real workloads under current
Linux.  And it will show up just as badly on a SMP/cc cluster.  SMP/cc
clusters talk a lot about "put it into a special filesystem and scale
that all you want" but I'm trying to show that frankly thats where the
"no solution evident" scaling problems actually are today.

If LLNL was not too jazzed up about your proposal, I right now don't
blame them.  Because with the information I have right now, I think
your claims about it's potential are bogus.

I really want to be shown wrong, simply because the name path locking
issue is one that has been giving me mental gas for quite some time.

Another thing I've found is that SMP scalability changes that help
the "8, 16, 32, 64" cpu case almost never harm the "4, 2" cpu
cases.  Rather, they tend to improve the smaller cpu number cases.
Finally, as I think Ingo pointed out recently, some of the results of
our SMP work has even improved the uniprocessor cases.

Franks a lot,
David S. Miller
davem@redhat.com

Re: SMP/cc Cluster description

[Jeff Merkey]

----- Original Message -----
From: "David S. Miller" <davem@redhat.com>
To: <lm@bitmover.com>
Cc: <Martin.Bligh@us.ibm.com>; <riel@conectiva.com.br>;
<lars.spam@nocrew.org>; <alan@lxorguk.ukuu.org.uk>; <hps@intermeta.de>;
<linux-kernel@vger.kernel.org>
Sent: Tuesday, December 04, 2001 11:05 PM
Subject: Re: SMP/cc Cluster description


>    From: Larry McVoy <lm@bitmover.com>
>    Date: Tue, 4 Dec 2001 19:23:17 -0800
>
> Even more boldly, I claim that Linux's current ipv4 scales further
> than anything coming out of Sun engineering.  From my perspective
> Sun's scalability efforts are more in line with "rubber-stamp"
> per-object locking when things show up in the profiles than anything
> else.  Their networking is really big and fat.  For example the
> Solaris per-socket TCP information is nearly 4 times the size of that
> in Linux (last time I checked their headers).  And as we all know
> their stuff sits on top of some thick infrastructure (ie. STREAMS)
> (OK, here is where someone pops up a realistic networking benchmark
> where we scale worse than Solaris.  I would welcome such a retort
> because it'd probably end up being a really simple thing to fix.)

David,

The job you did on ipv4 is quite excellent.  I multi-threaded the ODI layer
in NetWare,
and I have reviewed your work and it's as good as anything out there.  The
fewer locks,
the better.  Also, I agree that Sun's "hot air" regarding their SMP is just
that.  Sure, they
have a greart priority inheritenace model, but big f_cking deal.  sleep
locks and their behaviors
have little to do with I./O scaling on interrupt paths, other than to
increase the background
transaction activity on the memory bus.

Your ipv4 work is not perfect, but it's certainly good enough.  We found
with NetWare that SMP
scaling was tough to achieve since the processor was never the bottleneck --
the I/O bus was.
Uniprocessor NetWare 3.12 still runs circles around Linux or anything else,
and it's not
multithreaded, just well optimaized (and hand coded in assembler).

There are a few optimizations you could still to do to make it even faster,
but these are off line
discussions.  :-)

Jeff
.
>
> Franks a lot,
> David S. Miller
> davem@redhat.com
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: SMP/cc Cluster description

[Rusty Russell]

On Tue, 04 Dec 2001 22:05:11 -0800 (PST)
"David S. Miller" <davem@redhat.com> wrote:

> The problem areas of scalability, for which no real solution is
> evident yet, involve the file name lookup tree data structures,
> ie. the dcache under Linux.  All accesses here are tree based, and
> everyone starts from similar roots.  So you can't per-node or
> per-branch lock as everyone traverses the same paths.  Furthermore you
> can't use "special locks" as in #1 since this data structure is
> neither heavy reader nor heavy writer.

Yes.  dbench on 4-way was showing d_lookup hurting us, but replacing
dcache_lock with a rw_lock (Anton Blanchard provided an atomic_dec_and_wlock)
and a separate lock for the unused list DIDN'T HELP ONE BIT.

Why?  Because there's no contention on the lock!  The problem is almost
entirely moving the cacheline around (which is the same for a rw lock).

I'd love to say that I can solve this with RCU, but it's vastly non-trivial
and I haven't got code, so I'm not going to say that. 8)

Rusty.
-- 
  Anyone who quotes me is an idiot. -- Rusty Russell.

Re: SMP/cc Cluster description

[Davide Libenzi]

On Thu, 6 Dec 2001, Rusty Russell wrote:

> I'd love to say that I can solve this with RCU, but it's vastly non-trivial
> and I haven't got code, so I'm not going to say that. 8)

Lockless algos could help if we're able to have "good" quiescent point
inside the kernel. Or better have a good quiescent infrastructure to have
lockless code to plug in.



- Davide

Re: SMP/cc Cluster description

[David S. Miller]

   From: Davide Libenzi <davidel@xmailserver.org>
   Date: Wed, 5 Dec 2001 19:19:19 -0800 (PST)

   On Thu, 6 Dec 2001, Rusty Russell wrote:
   
   > I'd love to say that I can solve this with RCU, but it's vastly non-trivial
   > and I haven't got code, so I'm not going to say that. 8)
   
   Lockless algos could help if we're able to have "good" quiescent point
   inside the kernel. Or better have a good quiescent infrastructure to have
   lockless code to plug in.

Lockless algorithms don't get rid of the shared cache lines.

I used to once think that lockless algorithms were the SMP holy-grail,
but this was undone when I realized they had the same fundamental
overhead spinlocks do.

These lockless algorithms, instructions like CAS, DCAS, "infinite
consensus number", it's all crap.  You have to seperate out the access
areas amongst different cpus so they don't collide, and none of these
mechanisms do that.

That is, unless some lockless algorithm involving %100 local dirty
state has been invented while I wasn't looking :-)

Re: SMP/cc Cluster description

[Larry McVoy]

On Wed, Dec 05, 2001 at 11:56:17PM -0800, David S. Miller wrote:
> These lockless algorithms, instructions like CAS, DCAS, "infinite
> consensus number", it's all crap.  You have to seperate out the access
> areas amongst different cpus so they don't collide, and none of these
> mechanisms do that.

Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
all that seperation for every data structure for free.  Think about
it a bit.  Aren't you going to feel a little bit stupid if you do all
this work, one object at a time, and someone can come along and do the
whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
but it is much easier than the route you are taking.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 00:02:16 -0800
   
   Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
   all that seperation for every data structure for free.  Think about
   it a bit.  Aren't you going to feel a little bit stupid if you do all
   this work, one object at a time, and someone can come along and do the
   whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
   but it is much easier than the route you are taking.  

How does ccClusters avoid the file system namespace locking issues?
How do all the OS nodes see a consistent FS tree?

All the talk is about the "magic filesystem, thread it as much as you
want" and I'm telling you that is the fundamental problem, the
filesystem name space locking.

Re: SMP/cc Cluster description

[Jeff V. Merkey]

Guys,

I am the maintaner of SCI, the ccNUMA technology standard.  I know
alot about this stuff, and have been involved with SCI since 
1994.  I work with it every day and the Dolphin guys on some huge 
supercomputer accounts, like Los Alamos and Sandia Labs in NM.  
I will tell you this from what I know.

A shared everything approach is a programmers dream come true,
but you can forget getting reasonable fault tolerance with it.  The 
shared memory zealots want everyone to believe ccNUMA is better 
than sex, but it does not scale when compared to Shared-Nothing
programming models.  There's also a lot of tough issues for dealing 
with failed nodes, and how you recover when peoples memory is 
all over the place across a nuch of machines.  

SCI scales better in ccNUMA and all NUMA technoogies scale very 
well when they are used with "Explicit Coherence" instead of 
"Implicit Coherence" which is what you get with SMP systems.  
Years of research by Dr. Justin Rattner at Intel's High 
performance labs demonstrated that shared nothing models scaled
into the thousands of nodes, while all these shared everything
"Super SMP" approaches hit the wall at 64 processors generally.

SCI is the fastest shared nothing interface out there, and it also
can do ccNUMA.  Sequent, Sun, DG and a host of other NUMA providers
use Dolphin's SCI technology and have for years.   ccNUMA is useful 
for applications that still assume a shared nothing approach but that
use the ccNUMA and NUMA capabilities for better optimization.

Forget trying to recreate the COMA architecture of Kendall-Square.  
The name was truly descriptive of what happened in this architecture
when a node fails -- goes into a "COMA".  This whole discussion I have
lived through before and you will find that ccNUMA is virtually 
unimplementable on most general purpose OSs.  And yes, there are 
a lot of products and software out there, but when you look under 
the cover (like ServerNet) you discover their coherence models 
for the most part relay on push/pull explicit coherence models.

My 2 cents.

Jeff 



On Thu, Dec 06, 2001 at 12:09:32AM -0800, David S. Miller wrote:
>    From: Larry McVoy <lm@bitmover.com>
>    Date: Thu, 6 Dec 2001 00:02:16 -0800
>    
>    Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
>    all that seperation for every data structure for free.  Think about
>    it a bit.  Aren't you going to feel a little bit stupid if you do all
>    this work, one object at a time, and someone can come along and do the
>    whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
>    but it is much easier than the route you are taking.  
> 
> How does ccClusters avoid the file system namespace locking issues?
> How do all the OS nodes see a consistent FS tree?
> 
> All the talk is about the "magic filesystem, thread it as much as you
> want" and I'm telling you that is the fundamental problem, the
> filesystem name space locking.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: SMP/cc Cluster description

[Jeff V. Merkey]

On Thu, Dec 06, 2001 at 11:27:31AM -0700, Jeff V. Merkey wrote:

And also, if you download the SCI drivers in my area, and order
some SCI adapters from Dolphin in Albquerque, NM, you can set up 
a ccNUMA system with standard PCs.  Dolphin has 66Mhz versions (and
a 133Mhz coming in the future) that run at almost a gigabyte per 
second node-2-node over a parallel fabric.  The cross-sectional
SCI fabric bandwidth scales at (O)(2N) as you add nodes.  

If you want to play around with ccNUMA with Standard PCs, these 
cards are relatively inepxensive, and allow you to setup some 
powerful cc/SMP systems with explicit coherence.  The full 
ccNUMA boxes from DG are expensive, however.  That way, instead
of everyone talking about it, you guys could get some cool 
hardware and experiment with some of your rather forward 
looking and interesting ideas.

:-)

Jeff



> 
> 
> Guys,
> 
> I am the maintaner of SCI, the ccNUMA technology standard.  I know
> alot about this stuff, and have been involved with SCI since 
> 1994.  I work with it every day and the Dolphin guys on some huge 
> supercomputer accounts, like Los Alamos and Sandia Labs in NM.  
> I will tell you this from what I know.
> 
> A shared everything approach is a programmers dream come true,
> but you can forget getting reasonable fault tolerance with it.  The 
> shared memory zealots want everyone to believe ccNUMA is better 
> than sex, but it does not scale when compared to Shared-Nothing
> programming models.  There's also a lot of tough issues for dealing 
> with failed nodes, and how you recover when peoples memory is 
> all over the place across a nuch of machines.  
> 
> SCI scales better in ccNUMA and all NUMA technoogies scale very 
> well when they are used with "Explicit Coherence" instead of 
> "Implicit Coherence" which is what you get with SMP systems.  
> Years of research by Dr. Justin Rattner at Intel's High 
> performance labs demonstrated that shared nothing models scaled
> into the thousands of nodes, while all these shared everything
> "Super SMP" approaches hit the wall at 64 processors generally.
> 
> SCI is the fastest shared nothing interface out there, and it also
> can do ccNUMA.  Sequent, Sun, DG and a host of other NUMA providers
> use Dolphin's SCI technology and have for years.   ccNUMA is useful 
> for applications that still assume a shared nothing approach but that
> use the ccNUMA and NUMA capabilities for better optimization.
> 
> Forget trying to recreate the COMA architecture of Kendall-Square.  
> The name was truly descriptive of what happened in this architecture
> when a node fails -- goes into a "COMA".  This whole discussion I have
> lived through before and you will find that ccNUMA is virtually 
> unimplementable on most general purpose OSs.  And yes, there are 
> a lot of products and software out there, but when you look under 
> the cover (like ServerNet) you discover their coherence models 
> for the most part relay on push/pull explicit coherence models.
> 
> My 2 cents.
> 
> Jeff 
> 
> 
> 
> On Thu, Dec 06, 2001 at 12:09:32AM -0800, David S. Miller wrote:
> >    From: Larry McVoy <lm@bitmover.com>
> >    Date: Thu, 6 Dec 2001 00:02:16 -0800
> >    
> >    Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
> >    all that seperation for every data structure for free.  Think about
> >    it a bit.  Aren't you going to feel a little bit stupid if you do all
> >    this work, one object at a time, and someone can come along and do the
> >    whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
> >    but it is much easier than the route you are taking.  
> > 
> > How does ccClusters avoid the file system namespace locking issues?
> > How do all the OS nodes see a consistent FS tree?
> > 
> > All the talk is about the "magic filesystem, thread it as much as you
> > want" and I'm telling you that is the fundamental problem, the
> > filesystem name space locking.
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/

Re: SMP/cc Cluster description

[Martin J. Bligh]

> If you want to play around with ccNUMA with Standard PCs, these 
> cards are relatively inepxensive, and allow you to setup some 
> powerful cc/SMP systems with explicit coherence.  The full 
> ccNUMA boxes from DG are expensive, however.  That way, instead
> of everyone talking about it, you guys could get some cool 
> hardware and experiment with some of your rather forward 
> looking and interesting ideas.

Or you could just book some time on the 16x NUMA-Q that's publicly
available in the OSDL for free, and running Linux already.

Martin.

Re: SMP/cc Cluster description

[Jeff V. Merkey]

On Thu, Dec 06, 2001 at 10:36:57AM -0800, Martin J. Bligh wrote:
> > If you want to play around with ccNUMA with Standard PCs, these 
> > cards are relatively inepxensive, and allow you to setup some 
> > powerful cc/SMP systems with explicit coherence.  The full 
> > ccNUMA boxes from DG are expensive, however.  That way, instead
> > of everyone talking about it, you guys could get some cool 
> > hardware and experiment with some of your rather forward 
> > looking and interesting ideas.
> 
> Or you could just book some time on the 16x NUMA-Q that's publicly
> available in the OSDL for free, and running Linux already.
> 
> Martin.

Could even have some shipped off to you guys if you to play 
around with them.  :-)

Jeff

Re: SMP/cc Cluster description

[Davide Libenzi]

On Thu, 6 Dec 2001, Jeff V. Merkey wrote:

> Guys,
>
> I am the maintaner of SCI, the ccNUMA technology standard.  I know
> alot about this stuff, and have been involved with SCI since
> 1994.  I work with it every day and the Dolphin guys on some huge
> supercomputer accounts, like Los Alamos and Sandia Labs in NM.
> I will tell you this from what I know.
>
> A shared everything approach is a programmers dream come true,
> but you can forget getting reasonable fault tolerance with it.  The
> shared memory zealots want everyone to believe ccNUMA is better
> than sex, but it does not scale when compared to Shared-Nothing
> programming models.  There's also a lot of tough issues for dealing
> with failed nodes, and how you recover when peoples memory is
> all over the place across a nuch of machines.

If you can afford rewriting/rearchitecting your application it's pretty
clear that the share-nothing model is the winner one.
But if you can rewrite your application using a share-nothing model you
don't need any fancy clustering architectures since beowulf like cluster
would work for you and they'll give you a great scalability over the
number of nodes.
The problem arises when you've to choose between a new architecture
( share nothing ) using conventional clusters and a
share-all/keep-all-your-application-as-is one.
The share nothing is cheap and gives you a very nice scalability, these
are the two mayor pros for this solution.
On the other side you've a vary bad scalability and a very expensive
solution.
But you've to consider :

1) rewriting is risky

2) good developers to rewrite your stuff are expensive ( $100K up to $150K
	in my area )

These are the reason that let me think that conventional SMP machines will
have a future in addition to my believing that technology will help a lot
to improve scalability.




- Davide

Re: SMP/cc Cluster description

[Jeff V. Merkey]

On Thu, Dec 06, 2001 at 11:11:27AM -0800, Davide Libenzi wrote:
> On Thu, 6 Dec 2001, Jeff V. Merkey wrote:
> 
> > Guys,
> >
> > I am the maintaner of SCI, the ccNUMA technology standard.  I know
> > alot about this stuff, and have been involved with SCI since
> > 1994.  I work with it every day and the Dolphin guys on some huge
> > supercomputer accounts, like Los Alamos and Sandia Labs in NM.
> > I will tell you this from what I know.
> >
> > A shared everything approach is a programmers dream come true,
> > but you can forget getting reasonable fault tolerance with it.  The
> > shared memory zealots want everyone to believe ccNUMA is better
> > than sex, but it does not scale when compared to Shared-Nothing
> > programming models.  There's also a lot of tough issues for dealing
> > with failed nodes, and how you recover when peoples memory is
> > all over the place across a nuch of machines.
> 
> If you can afford rewriting/rearchitecting your application it's pretty
> clear that the share-nothing model is the winner one.
> But if you can rewrite your application using a share-nothing model you
> don't need any fancy clustering architectures since beowulf like cluster
> would work for you and they'll give you a great scalability over the
> number of nodes.
> The problem arises when you've to choose between a new architecture
> ( share nothing ) using conventional clusters and a
> share-all/keep-all-your-application-as-is one.
> The share nothing is cheap and gives you a very nice scalability, these
> are the two mayor pros for this solution.
> On the other side you've a vary bad scalability and a very expensive
> solution.
> But you've to consider :
> 
> 1) rewriting is risky
> 
> 2) good developers to rewrite your stuff are expensive ( $100K up to $150K
> 	in my area )
> 
> These are the reason that let me think that conventional SMP machines will
> have a future in addition to my believing that technology will help a lot
> to improve scalability.
> 

There's a way through the fog.  Shared Nothing with explicit coherence.
You are correct, applications need to be rewritten to exploit it.  It 
is possible to run existing SMP apps process -> process across nodes
with ccNUMA, and this works, but you don't get the scaling as shared
nothing.

Jeff

Jeff


> 
> 
> 
> - Davide
> 
> 
> 

Re: SMP/cc Cluster description

[David Lang]

also some applications (i.e. databases) are such that nobody has really
been able to rewrite them into the shared nothing model (although oracle
has attempted it, from what I hear it has problems)

David Lang

 On Thu, 6 Dec 2001,
Jeff V. Merkey wrote:

> Date: Thu, 6 Dec 2001 12:34:48 -0700
> From: Jeff V. Merkey <jmerkey@vger.timpanogas.org>
> To: Davide Libenzi <davidel@xmailserver.org>
> Cc: David S. Miller <davem@redhat.com>, lm@bitmover.com,
>      rusty@rustcorp.com.au, Martin J. Bligh <Martin.Bligh@us.ibm.com>,
>      Rik vav Riel <riel@conectiva.com.br>, lars.spam@nocrew.org,
>      Alan Cox <alan@lxorguk.ukuu.org.uk>, hps@intermeta.de,
>      lkml <linux-kernel@vger.kernel.org>, jmerkey@timpanogas.org
> Subject: Re: SMP/cc Cluster description
>
> On Thu, Dec 06, 2001 at 11:11:27AM -0800, Davide Libenzi wrote:
> > On Thu, 6 Dec 2001, Jeff V. Merkey wrote:
> >
> > > Guys,
> > >
> > > I am the maintaner of SCI, the ccNUMA technology standard.  I know
> > > alot about this stuff, and have been involved with SCI since
> > > 1994.  I work with it every day and the Dolphin guys on some huge
> > > supercomputer accounts, like Los Alamos and Sandia Labs in NM.
> > > I will tell you this from what I know.
> > >
> > > A shared everything approach is a programmers dream come true,
> > > but you can forget getting reasonable fault tolerance with it.  The
> > > shared memory zealots want everyone to believe ccNUMA is better
> > > than sex, but it does not scale when compared to Shared-Nothing
> > > programming models.  There's also a lot of tough issues for dealing
> > > with failed nodes, and how you recover when peoples memory is
> > > all over the place across a nuch of machines.
> >
> > If you can afford rewriting/rearchitecting your application it's pretty
> > clear that the share-nothing model is the winner one.
> > But if you can rewrite your application using a share-nothing model you
> > don't need any fancy clustering architectures since beowulf like cluster
> > would work for you and they'll give you a great scalability over the
> > number of nodes.
> > The problem arises when you've to choose between a new architecture
> > ( share nothing ) using conventional clusters and a
> > share-all/keep-all-your-application-as-is one.
> > The share nothing is cheap and gives you a very nice scalability, these
> > are the two mayor pros for this solution.
> > On the other side you've a vary bad scalability and a very expensive
> > solution.
> > But you've to consider :
> >
> > 1) rewriting is risky
> >
> > 2) good developers to rewrite your stuff are expensive ( $100K up to $150K
> > 	in my area )
> >
> > These are the reason that let me think that conventional SMP machines will
> > have a future in addition to my believing that technology will help a lot
> > to improve scalability.
> >
>
> There's a way through the fog.  Shared Nothing with explicit coherence.
> You are correct, applications need to be rewritten to exploit it.  It
> is possible to run existing SMP apps process -> process across nodes
> with ccNUMA, and this works, but you don't get the scaling as shared
> nothing.
>
> Jeff
>
> Jeff
>
>
> >
> >
> >
> > - Davide
> >
> >
> >
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: SMP/cc Cluster description

[Jeff V. Merkey]

On Thu, Dec 06, 2001 at 03:16:13PM -0800, David Lang wrote:
> also some applications (i.e. databases) are such that nobody has really
> been able to rewrite them into the shared nothing model (although oracle
> has attempted it, from what I hear it has problems)
> 
> David Lang

OPS (Oracle Parallel Server) is shared nothing.  

Jeff


> 
>  On Thu, 6 Dec 2001,
> Jeff V. Merkey wrote:
> 
> > Date: Thu, 6 Dec 2001 12:34:48 -0700
> > From: Jeff V. Merkey <jmerkey@vger.timpanogas.org>
> > To: Davide Libenzi <davidel@xmailserver.org>
> > Cc: David S. Miller <davem@redhat.com>, lm@bitmover.com,
> >      rusty@rustcorp.com.au, Martin J. Bligh <Martin.Bligh@us.ibm.com>,
> >      Rik vav Riel <riel@conectiva.com.br>, lars.spam@nocrew.org,
> >      Alan Cox <alan@lxorguk.ukuu.org.uk>, hps@intermeta.de,
> >      lkml <linux-kernel@vger.kernel.org>, jmerkey@timpanogas.org
> > Subject: Re: SMP/cc Cluster description
> >
> > On Thu, Dec 06, 2001 at 11:11:27AM -0800, Davide Libenzi wrote:
> > > On Thu, 6 Dec 2001, Jeff V. Merkey wrote:
> > >
> > > > Guys,
> > > >
> > > > I am the maintaner of SCI, the ccNUMA technology standard.  I know
> > > > alot about this stuff, and have been involved with SCI since
> > > > 1994.  I work with it every day and the Dolphin guys on some huge
> > > > supercomputer accounts, like Los Alamos and Sandia Labs in NM.
> > > > I will tell you this from what I know.
> > > >
> > > > A shared everything approach is a programmers dream come true,
> > > > but you can forget getting reasonable fault tolerance with it.  The
> > > > shared memory zealots want everyone to believe ccNUMA is better
> > > > than sex, but it does not scale when compared to Shared-Nothing
> > > > programming models.  There's also a lot of tough issues for dealing
> > > > with failed nodes, and how you recover when peoples memory is
> > > > all over the place across a nuch of machines.
> > >
> > > If you can afford rewriting/rearchitecting your application it's pretty
> > > clear that the share-nothing model is the winner one.
> > > But if you can rewrite your application using a share-nothing model you
> > > don't need any fancy clustering architectures since beowulf like cluster
> > > would work for you and they'll give you a great scalability over the
> > > number of nodes.
> > > The problem arises when you've to choose between a new architecture
> > > ( share nothing ) using conventional clusters and a
> > > share-all/keep-all-your-application-as-is one.
> > > The share nothing is cheap and gives you a very nice scalability, these
> > > are the two mayor pros for this solution.
> > > On the other side you've a vary bad scalability and a very expensive
> > > solution.
> > > But you've to consider :
> > >
> > > 1) rewriting is risky
> > >
> > > 2) good developers to rewrite your stuff are expensive ( $100K up to $150K
> > > 	in my area )
> > >
> > > These are the reason that let me think that conventional SMP machines will
> > > have a future in addition to my believing that technology will help a lot
> > > to improve scalability.
> > >
> >
> > There's a way through the fog.  Shared Nothing with explicit coherence.
> > You are correct, applications need to be rewritten to exploit it.  It
> > is possible to run existing SMP apps process -> process across nodes
> > with ccNUMA, and this works, but you don't get the scaling as shared
> > nothing.
> >
> > Jeff
> >
> > Jeff
> >
> >
> > >
> > >
> > >
> > > - Davide
> > >
> > >
> > >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >

Re: SMP/cc Cluster description

[David Lang]

On Thu, 6 Dec 2001, Jeff V. Merkey wrote:

> On Thu, Dec 06, 2001 at 03:16:13PM -0800, David Lang wrote:
> > also some applications (i.e. databases) are such that nobody has really
> > been able to rewrite them into the shared nothing model (although oracle
> > has attempted it, from what I hear it has problems)
> >
> > David Lang
>
> OPS (Oracle Parallel Server) is shared nothing.
>

correct, and from what I have been hearing from my local database folks
it's significantly less efficiant then a large SMP machine (up intil you
hit the point where you just can't buy a machine big enough :-)

I'm interested in hearing more if you have had better experiances with it.

David Lang

Re: SMP/cc Cluster description

[Jeff V. Merkey]

On Thu, Dec 06, 2001 at 08:23:36PM -0800, David Lang wrote:
> On Thu, 6 Dec 2001, Jeff V. Merkey wrote:
> 
> > On Thu, Dec 06, 2001 at 03:16:13PM -0800, David Lang wrote:
> > > also some applications (i.e. databases) are such that nobody has really
> > > been able to rewrite them into the shared nothing model (although oracle
> > > has attempted it, from what I hear it has problems)
> > >
> > > David Lang
> >
> > OPS (Oracle Parallel Server) is shared nothing.
> >
> 
> correct, and from what I have been hearing from my local database folks
> it's significantly less efficiant then a large SMP machine (up intil you
> hit the point where you just can't buy a machine big enough :-)
> 
> I'm interested in hearing more if you have had better experiances with it.
> 
> David Lang

I worked with the OPS code years back.  It came from DEC originally 
and is a very old technology.  It grew out of disk pinging, where 
the messages would be pinged across a shared disk.  Some cool features,
but I never saw it scale well beyond 16 nodes.  SQL queries are a lot 
like HTML requests, so similair approaches work well with them.  The code
was not so good, or readable.

Databases are "structured data" applications and present unique problems,
but most data stored on planet earth is "unstructured data", word files, 
emails, spreadsheets, etc.  The problem of scaling involves different 
approaches for these two categories, and the unstructured data problem 
is easily solvable and scalable.  

I ported Oracle to NetWare SMP in 1995 with Van Okamura (a very fun 
project), and SMP scaling was much better.  In those days, shared 
SCSI was what was around.  Writing an SOSD layer for Oracle 
was a lot of fun.  Working on OPS was also fun, but the code 
was not in such good shape.  Their method of dealing with 
deadlocks was not to.  Their approach assumed deadlocks were 
infrequent events (which they were) and they used a mathod that would 
detect them after the fact rather than before and deal with them 
then.  

I saw some impressive numbers for TPC-C come out of OPS 4 way clusters,
but more nodes than this (except the N-cube implementation) seemed to 
not do so well.

Jeff 

Re: SMP/cc Cluster description

[Daniel Phillips]

On December 6, 2001 09:02 am, Larry McVoy wrote:
> On Wed, Dec 05, 2001 at 11:56:17PM -0800, David S. Miller wrote:
> > These lockless algorithms, instructions like CAS, DCAS, "infinite
> > consensus number", it's all crap.  You have to seperate out the access
> > areas amongst different cpus so they don't collide, and none of these
> > mechanisms do that.
> 
> Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
> all that seperation for every data structure for free.  Think about
> it a bit.  Aren't you going to feel a little bit stupid if you do all
> this work, one object at a time, and someone can come along and do the
> whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
> but it is much easier than the route you are taking.  

What I don't get after looking at your material, is how you intend to do the 
locking.  Sharing a mmap across OS instances is fine, but how do processes on 
the two different OS's avoid stepping on each other when they access the same 
file?

--
Daniel

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 08:42:05PM +0100, Daniel Phillips wrote:
> On December 6, 2001 09:02 am, Larry McVoy wrote:
> > On Wed, Dec 05, 2001 at 11:56:17PM -0800, David S. Miller wrote:
> > > These lockless algorithms, instructions like CAS, DCAS, "infinite
> > > consensus number", it's all crap.  You have to seperate out the access
> > > areas amongst different cpus so they don't collide, and none of these
> > > mechanisms do that.
> > 
> > Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
> > all that seperation for every data structure for free.  Think about
> > it a bit.  Aren't you going to feel a little bit stupid if you do all
> > this work, one object at a time, and someone can come along and do the
> > whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
> > but it is much easier than the route you are taking.  
> 
> What I don't get after looking at your material, is how you intend to do the 
> locking.  Sharing a mmap across OS instances is fine, but how do processes on 
> the two different OS's avoid stepping on each other when they access the same 
> file?

Exactly the same way they would if they were two processes on a traditional
SMP OS.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Daniel Phillips]

On December 6, 2001 08:53 pm, Larry McVoy wrote:
> On Thu, Dec 06, 2001 at 08:42:05PM +0100, Daniel Phillips wrote:
> > On December 6, 2001 09:02 am, Larry McVoy wrote:
> > > On Wed, Dec 05, 2001 at 11:56:17PM -0800, David S. Miller wrote:
> > > > These lockless algorithms, instructions like CAS, DCAS, "infinite
> > > > consensus number", it's all crap.  You have to seperate out the access
> > > > areas amongst different cpus so they don't collide, and none of these
> > > > mechanisms do that.
> > > 
> > > Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
> > > all that seperation for every data structure for free.  Think about
> > > it a bit.  Aren't you going to feel a little bit stupid if you do all
> > > this work, one object at a time, and someone can come along and do the
> > > whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
> > > but it is much easier than the route you are taking.  
> > 
> > What I don't get after looking at your material, is how you intend to do the 
> > locking.  Sharing a mmap across OS instances is fine, but how do processes on 
> > the two different OS's avoid stepping on each other when they access the same 
> > file?
> 
> Exactly the same way they would if they were two processes on a traditional
> SMP OS.

They'd use locks internal to the VFS and fs, plus Posix-style locks and
assorted other userland serializers, which don't come for free.  As davem 
said, you'll have to present a coherent namespace, that's just one of the 
annoying details.  So far you haven't said much about how such things are 
going to be handled.

--
Daniel

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 09:10:51PM +0100, Daniel Phillips wrote:
> On December 6, 2001 08:53 pm, Larry McVoy wrote:
> > On Thu, Dec 06, 2001 at 08:42:05PM +0100, Daniel Phillips wrote:
> > > On December 6, 2001 09:02 am, Larry McVoy wrote:
> > > > On Wed, Dec 05, 2001 at 11:56:17PM -0800, David S. Miller wrote:
> > > > > These lockless algorithms, instructions like CAS, DCAS, "infinite
> > > > > consensus number", it's all crap.  You have to seperate out the access
> > > > > areas amongst different cpus so they don't collide, and none of these
> > > > > mechanisms do that.
> > > > 
> > > > Err, Dave, that's *exactly* the point of the ccCluster stuff.  You get
> > > > all that seperation for every data structure for free.  Think about
> > > > it a bit.  Aren't you going to feel a little bit stupid if you do all
> > > > this work, one object at a time, and someone can come along and do the
> > > > whole OS in one swoop?  Yeah, I'm spouting crap, it isn't that easy,
> > > > but it is much easier than the route you are taking.  
> > > 
> > > What I don't get after looking at your material, is how you intend to do the 
> > > locking.  Sharing a mmap across OS instances is fine, but how do processes on 
> > > the two different OS's avoid stepping on each other when they access the same 
> > > file?
> > 
> > Exactly the same way they would if they were two processes on a traditional
> > SMP OS.
> 
> They'd use locks internal to the VFS and fs, plus Posix-style locks and
> assorted other userland serializers, which don't come for free.  As davem 
> said, you'll have to present a coherent namespace, that's just one of the 
> annoying details.  So far you haven't said much about how such things are 
> going to be handled.

Huh?  Of course not, they'd use mutexes in a mmap-ed file, which uses
the hardware's coherency.  No locks in the vfs or fs, that's all done
in the mmap/page fault path for sure, but once the data is mapped you
aren't dealing with the file system at all.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 12:10:04 -0800

   Huh?  Of course not, they'd use mutexes in a mmap-ed file, which uses
   the hardware's coherency.  No locks in the vfs or fs, that's all done
   in the mmap/page fault path for sure, but once the data is mapped you
   aren't dealing with the file system at all.

We're talking about two things.

Once the data is MMAP'd, sure things are coherent just like on any
other SMP, for the user.

But HOW DID YOU GET THERE?  That is the question you are avoiding.
How do I look up "/etc/passwd" in the filesystem on a ccCluster?
How does OS image 1 see the same "/etc/passwd" as OS image 2?

If you aren't getting rid of this locking, what is the point?
That is what we are trying to talk about.

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 12:15:54PM -0800, David S. Miller wrote:
>    From: Larry McVoy <lm@bitmover.com>
>    Date: Thu, 6 Dec 2001 12:10:04 -0800
> 
>    Huh?  Of course not, they'd use mutexes in a mmap-ed file, which uses
>    the hardware's coherency.  No locks in the vfs or fs, that's all done
>    in the mmap/page fault path for sure, but once the data is mapped you
>    aren't dealing with the file system at all.
> 
> We're talking about two things.
> 
> Once the data is MMAP'd, sure things are coherent just like on any
> other SMP, for the user.
> 
> But HOW DID YOU GET THERE?  That is the question you are avoiding.
> How do I look up "/etc/passwd" in the filesystem on a ccCluster?
> How does OS image 1 see the same "/etc/passwd" as OS image 2?
> 
> If you aren't getting rid of this locking, what is the point?
> That is what we are trying to talk about.

The points are:

a) you have to thread the entire kernel, every data structure which is a
   problem.  Scheduler, networking, device drivers, everything.  That's
   thousands of locks and uncountable bugs, not to mention the impact on
   uniprocessor performance.

b) I have to thread a file system.

So I'm not saying that I'll thread less in the file system (actually I am,
but let's skip that for now and assume I have to do everything you have
to do).  All I'm saying is that I don't have to worry about the rest of
the kernel which is a huge savings.

You tell me - which is easier, multithreading the networking stack to 
64 way SMP or running 64 distinct networking stacks?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 12:21:16 -0800
   
   You tell me - which is easier, multithreading the networking stack to 
   64 way SMP or running 64 distinct networking stacks?

We've done %90 of the "other stuff" already, why waste the work?
We've done the networking, we've done the scheduler, and the
networking/block drivers are there too.

I was actually pretty happy with how easy (relatively) the networking
was to thread nicely.

The point is, you have to make a captivating argument for ccClusters,
what does it buy us now that we've done a lot of the work you are
telling us it will save?

Re: SMP/cc Cluster description

[Benjamin LaHaise]

On Thu, Dec 06, 2001 at 01:02:02PM -0800, David S. Miller wrote:
> We've done %90 of the "other stuff" already, why waste the work?
> We've done the networking, we've done the scheduler, and the
> networking/block drivers are there too.

The scheduler doesn't scale too well...

> I was actually pretty happy with how easy (relatively) the networking
> was to thread nicely.
> 
> The point is, you have to make a captivating argument for ccClusters,
> what does it buy us now that we've done a lot of the work you are
> telling us it will save?

The most captivating arguments are along the following lines:

	- scales perfectly across NUMA fabrics: there are a number of 
	  upcoming architechures (hammer, power4, others) where the 
	  latency costs on remote memory are significantly higher.  By
	  making the entire kernel local, we'll see optimal performance 
	  for local operations, with good performance for the remote 
	  actions (the ccClusterFS should be very low overhead).
	- opens up a number of possibilities in terms of serviceability: 
	  if a chunk of the system is taken offline, only the one kernel 
	  group has to go away.  Useful in containing failures.
	- lower overhead for SMP systems.  We can use UP kernels local 
	  to each CPU.  Should make kernel compiles faster. ;-)

At the very least it is well worth investigating.  Bootstrapping the 
ccCluster work shouldn't take more than a week or so, which will let 
us attach some hard numbers to the kind of impact it has on purely 
cpu local tasks.

		-ben
-- 
Fish.

Re: SMP/cc Cluster description

[Alan Cox]

> On Thu, Dec 06, 2001 at 01:02:02PM -0800, David S. Miller wrote:
> > We've done %90 of the "other stuff" already, why waste the work?
> > We've done the networking, we've done the scheduler, and the
> > networking/block drivers are there too.
> 
> The scheduler doesn't scale too well...

Understatement. However retrofitting a real scheduler doesn't break the
scalability of the system IMHO.

Re: SMP/cc Cluster description

[David S. Miller]

   From: Benjamin LaHaise <bcrl@redhat.com>
   Date: Thu, 6 Dec 2001 17:27:08 -0500

   	- lower overhead for SMP systems.  We can use UP kernels local 
   	  to each CPU.  Should make kernel compiles faster. ;-)
   
Actually, this isn't what is being proposed.  Something like
"4 cpu" SMP kernels.

   At the very least it is well worth investigating.  Bootstrapping the 
   ccCluster work shouldn't take more than a week or so, which will let 
   us attach some hard numbers to the kind of impact it has on purely 
   cpu local tasks.
   
I think it is worth considering too, but I don't know if a week
estimate is sane or not :-)

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 03:08:47PM -0800, David S. Miller wrote:
>    From: Benjamin LaHaise <bcrl@redhat.com>
>    Date: Thu, 6 Dec 2001 17:27:08 -0500
> 
>    	- lower overhead for SMP systems.  We can use UP kernels local 
>    	  to each CPU.  Should make kernel compiles faster. ;-)
>    
> Actually, this isn't what is being proposed.  Something like
> "4 cpu" SMP kernels.

I personally want to cluster small SMP OS images because I don't want to
do the process migration crap anywhere except at exec time, it simplifies
a lot.  So I need a second order load balancing term that I can get
from 2 or 4 way smp nodes.  If you are willing to process migration to
handle load imbalances, then you could do uniprocessor only.  I think
the complexity tradeoff is in favor of the small SMP OS clusters, we
already have them.  Process migration is a rats nest.

>    At the very least it is well worth investigating.  Bootstrapping the 
>    ccCluster work shouldn't take more than a week or so, which will let 
>    us attach some hard numbers to the kind of impact it has on purely 
>    cpu local tasks.
>    
> I think it is worth considering too, but I don't know if a week
> estimate is sane or not :-)

Yeah, it's possible that you could get something booting in a week but I
think it's a bit harder than that too.  One idea that was kicked around
was to use Jeff's UML work and "boot" multiple UML's on top of a virtual
SMP.  You get things to work there and then do a "port" to real hardware.
Kind of a cool idea if you ask me.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Adam Keys]

On December 06, 2001 05:26, Larry McVoy wrote: > Yeah, it's possible that you 
could get something booting in a week but I > think it's a bit harder than 
that too. One idea that was kicked around > was to use Jeff's UML work and 
"boot" multiple UML's on top of a virtual > SMP. You get things to work there 
and then do a "port" to real hardware. > Kind of a cool idea if you ask me.

Point me in the right direction. After reading over your slides and SMP paper 
(still have the labs.pdf on my queue), it seemed to me that you could easily 
simulate what you want with lots of UML's talking to each other. I think you 
would need to create some kind of device that uses a file or a shared memory 
segment as the cluster's memory. Actually, I think that (shared memory) is 
how Jeff had intended on implementing SMP in UML anyway. At this point I 
don't think UML supports SMP though I know of at least one person who was 
attempting it.

Once said device would implemented, you could start working on the unique 
challenges ccClusters present. I guess this would be what you consider 
"bootstrapping", although I don't really know what that would entail at this 
point. Then you just need some bored college student :) to hack it out.

I've been negligent in following this mammoth link...cluebat me if you 
mentioned it somewhere upthread.

-- akk~

Re: SMP/cc Cluster description

[Jeff Dike]

akeys@post.cis.smu.edu said:
>  it seemed to me that you could easily  simulate what you want with
> lots of UML's talking to each other. I think you  would need to create
> some kind of device that uses a file or a shared memory  segment as
> the cluster's memory. 

Yeah, there is already support for mapping in a random file and using that
as UML memory, so that would be used for the cluster interconnect for any
cluster emulations you wanted to run with UML.

> Actually, I think that (shared memory) is  how
> Jeff had intended on implementing SMP in UML anyway.

No, at least not any shared memory that's not already there.  UML uses a 
host process for each UML process, and UML kernel data and text are
shared between all these host processes.  SMP just means having more than
one host process runnable at a time.  Each runnable process on the host
is a virtual processor.

> At this point I
> don't think UML supports SMP though I know of at least one person who
> was  attempting it.

It doesn't yet.  Someone is (or was), but I haven't heard a peep from him in
at least a month.  So this is starting to look like another little project
which got quickly going but just as quickly abandoned.

				Jeff

Re: SMP/cc Cluster description

[Daniel Phillips]

On December 6, 2001 09:21 pm, Larry McVoy wrote:
> On Thu, Dec 06, 2001 at 12:15:54PM -0800, David S. Miller wrote:
> > If you aren't getting rid of this locking, what is the point?
> > That is what we are trying to talk about.
> 
> The points are:
> 
> a) you have to thread the entire kernel, every data structure which is a
>    problem.  Scheduler, networking, device drivers, everything.  That's
>    thousands of locks and uncountable bugs, not to mention the impact on
>    uniprocessor performance.
> 
> b) I have to thread a file system.

OK, this is your central point.  It's a little more than just a mmap, no?
We're pressing you on your specific ideas on how to handle the 'peripheral' 
details.

--
Daniel

Re: SMP/cc Cluster description

[Henning Schmiedehausen]

On Thu, 2001-12-06 at 22:30, Daniel Phillips wrote:
> On December 6, 2001 09:21 pm, Larry McVoy wrote:
> > On Thu, Dec 06, 2001 at 12:15:54PM -0800, David S. Miller wrote:
> > > If you aren't getting rid of this locking, what is the point?
> > > That is what we are trying to talk about.
> > 
> > The points are:
> > 
> > a) you have to thread the entire kernel, every data structure which is a
> >    problem.  Scheduler, networking, device drivers, everything.  That's
> >    thousands of locks and uncountable bugs, not to mention the impact on
> >    uniprocessor performance.
> > 
> > b) I have to thread a file system.
> 
> OK, this is your central point.  It's a little more than just a mmap, no?
> We're pressing you on your specific ideas on how to handle the 'peripheral' 
> details.

How about creating one node as "master" and write a "cluster network
filesystem" which uses shared memory as its "network layer". 

Then boot all other nodes diskless from these cluster network
filesystems.

You can still have shared mmap (which I believe is Larry's toy point)
between the nodes but you avoid all of the filesystem locking issues,
because you're going over (a hopefully superfast) memory network
filesystem.

Or go iSCSI and attach a network device to each of the cluster node. Or
go 802.1q, attach a virtual network device to each cluster node, pull
all of them out over GigE and let some Cisco outside sort these out
again. :-)

What I don't like about the approach is the fact that all nodes should
share the same file system. One (at least IMHO) does not want this for
at least /etc. The "all nodes the same FS" works fine for your number
cruncher clusters where every node runs more or less the same software. 

It does not work for cluster boxes like the Starfire or 390. There you
want to boot totally different OS images on the nodes. No sense in
implementing a "threaded file system" that is not used.

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: SMP/cc Cluster description

[Larry McVoy]

> How about creating one node as "master" and write a "cluster network
> filesystem" which uses shared memory as its "network layer". 

Right.

> Then boot all other nodes diskless from these cluster network
> filesystems.

Wrong.  Give each node its own private boot fs.  Then mount /data.

> You can still have shared mmap (which I believe is Larry's toy point)
> between the nodes but you avoid all of the filesystem locking issues,
> because you're going over (a hopefully superfast) memory network
> filesystem.

There is no network, unless you consider the memory interconnect a 
network (I think the hardware guys would raise their eyebrows at 
that name).

> What I don't like about the approach is the fact that all nodes should
> share the same file system. One (at least IMHO) does not want this for
> at least /etc. 

Read through my other postings, I said that things are private by
default.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Martin J. Bligh]

I'm taking mercy on people and trimming down the cc: list ...

>> What I don't like about the approach is the fact that all nodes should
>> share the same file system. One (at least IMHO) does not want this for
>> at least /etc. 
> 
> Read through my other postings, I said that things are private by
> default.

So if I understand you correctly, you're saying you have a private /etc for 
each instance of the sub-OS. Doesn't this make management of the system 
a complete pig? And require modifying many user level tools?

M.

Re: SMP/cc Cluster description

[Larry McVoy]

On Fri, Dec 07, 2001 at 08:44:03AM -0800, Martin J. Bligh wrote:
> I'm taking mercy on people and trimming down the cc: list ...
> 
> >> What I don't like about the approach is the fact that all nodes should
> >> share the same file system. One (at least IMHO) does not want this for
> >> at least /etc. 
> > 
> > Read through my other postings, I said that things are private by
> > default.
> 
> So if I understand you correctly, you're saying you have a private /etc for 
> each instance of the sub-OS. Doesn't this make management of the system 
> a complete pig? And require modifying many user level tools?

My pay job is developing a distributed source management system which works
by replication.  We already have users who put all the etc files in it and
manage them that way.  Works great.  It's like rdist except it never screws
up and it has merging.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Martin J. Bligh]

> My pay job is developing a distributed source management system which works
> by replication.  We already have users who put all the etc files in it and
> manage them that way.  Works great.  It's like rdist except it never screws
> up and it has merging.

So would that mean I would need bitkeeper installed in order to change my
password? 

And IIRC, bitkeeper is not free either?

M.

Re: SMP/cc Cluster description

[Larry McVoy]

On Fri, Dec 07, 2001 at 10:04:11AM -0800, Martin J. Bligh wrote:
> > My pay job is developing a distributed source management system which works
> > by replication.  We already have users who put all the etc files in it and
> > manage them that way.  Works great.  It's like rdist except it never screws
> > up and it has merging.
> 
> So would that mean I would need bitkeeper installed in order to change my
> password? 

No, that's just one way to solve the problem.  Another way would be to have
a master/slave relationship between the replicas sort of like CVS.  In fact,
you could use CVS.

> And IIRC, bitkeeper is not free either?

Actually it is for this purpose.  You can either do open logging (probably
not what you want) or run it in single user mode which doesn't log, you
just lose the audit trail (all checkins look like they are made by root).

If I could figure out a way to allow the use of BK for /etc with out any
restrictions at all, and at the same time prevent people from just putting
all their source in /etc and shutting down our commercial revenue, I'd
do it in a heartbeat.  I'd *love it* if when I did an upgrade from Red Hat,
the config files were part of a BK repository and I just did a pull/merge
to join my local changes with whatever they've done.  That would be a huge
step in making sys admin a lot less problematic.  But this is more than a
bit off topic...
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Martin J. Bligh]

>> So would that mean I would need bitkeeper installed in order to change my
>> password? 
> 
> No, that's just one way to solve the problem.  Another way would be to have
> a master/slave relationship between the replicas sort of like CVS.  In fact,
> you could use CVS.

I'm not sure that's any less vomitworthy. 

Keeping things simple that users and/or sysadmins have to deal with is a 
Good Thing (tm). I'd have the complexity in the kernel, where complexity 
is pushed to the kernel developers, thanks.
 
>> And IIRC, bitkeeper is not free either?
>
> (... some slighty twisted concept of free snipped.)
>
>  But this is more than a bit off topic...

No it's not that far off topic, my point is that you're shifting the complexity 
problems to other areas (eg. system mangement / the application level / 
filesystems / scheduler load balancing) rather than solving them.

Martin.

Re: SMP/cc Cluster description

[Larry McVoy]

On Fri, Dec 07, 2001 at 10:42:00AM -0800, Martin J. Bligh wrote:
> >> So would that mean I would need bitkeeper installed in order to change my
> >> password? 
> > 
> > No, that's just one way to solve the problem.  Another way would be to have
> > a master/slave relationship between the replicas sort of like CVS.  In fact,
> > you could use CVS.
> 
> I'm not sure that's any less vomitworthy. 

You're right, it's so much better to manage all machines independently
so that they can get out of sync with each other.

Did you even consider that this is virtually identical to the problem
that a network of workstations or servers has?  Did it occur to you that
people have solved this problem in many different ways?  Or did you just
want to piss into the wind and enjoy the spray?

> Keeping things simple that users and/or sysadmins have to deal with is a 
> Good Thing (tm). I'd have the complexity in the kernel, where complexity 
> is pushed to the kernel developers, thanks.

Yeah, that's what I want, my password file management in the kernel.  
Brilliant.  Why didn't I think of that?

> >> And IIRC, bitkeeper is not free either?
> >
> > (... some slighty twisted concept of free snipped.)
> >
> >  But this is more than a bit off topic...
> 
> No it's not that far off topic, my point is that you're shifting the complexity 
> problems to other areas (eg. system mangement / the application level / 
> filesystems / scheduler load balancing) rather than solving them.

Whoops, you are so right, in order to work on OS scaling I'd better solve
password file management or the OS ideas are meaningless.  Uh huh.  I'll
get right on that, thanks for setting me straight here.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Martin J. Bligh]

> You're right, it's so much better to manage all machines independently
> so that they can get out of sync with each other.

No it's much better to just have one machine running one instance of
the OS so that it can't get out of sync with itself.
 
>> Keeping things simple that users and/or sysadmins have to deal with is a 
>> Good Thing (tm). I'd have the complexity in the kernel, where complexity 
>> is pushed to the kernel developers, thanks.
> 
> Yeah, that's what I want, my password file management in the kernel.  
> Brilliant.  Why didn't I think of that?

No, I want my password file management to be in a one file for the whole
machine. Where it is now. Without requiring syncronisation. If we put the 
complexity in the kernel to make the system scale running one OS we 
don't have the problem that you're creating at all.
 
>> No it's not that far off topic, my point is that you're shifting the complexity 
>> problems to other areas (eg. system mangement / the application level / 
>> filesystems / scheduler load balancing) rather than solving them.
> 
> Whoops, you are so right, in order to work on OS scaling I'd better solve
> password file management or the OS ideas are meaningless.  Uh huh.  I'll
> get right on that, thanks for setting me straight here.

If you don't chop the OS up into multiple instances, you don't have these
problems. If you create the problems, I expect you to solve them. 

You're not making the system as a whole scale, you're just pushing the
problems out somewhere else.

Martin.

Re: SMP/cc Cluster description

[Daniel Bergman]

> My pay job is developing a distributed source management system which works
> by replication.  We already have users who put all the etc files in it and
> manage them that way.  Works great.  It's like rdist except it never screws
> up and it has merging.

I'm just curious, what about security? Is this done in clear-text? 
Sounds dangerous to put /etc/shadow, for example, in clear-text on the
cable.

Sorry for getting off-topic.

Regards,
Daniel

--
Daniel Bergman
Phone: 08 - 55064278
Mobile: 08 - 6311430 
Fax: 08 - 59827056
Email: d-b@home.se

Re: SMP/cc Cluster description

[Larry McVoy]

On Fri, Dec 07, 2001 at 08:00:32PM +0100, Daniel Bergman wrote:
> > My pay job is developing a distributed source management system which works
> > by replication.  We already have users who put all the etc files in it and
> > manage them that way.  Works great.  It's like rdist except it never screws
> > up and it has merging.
> 
> I'm just curious, what about security? Is this done in clear-text? 
> Sounds dangerous to put /etc/shadow, for example, in clear-text on the
> cable.

BitKeeper can, and typically does, use ssh as a transport.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Pavel Machek]

Hi!

> > My pay job is developing a distributed source management system which works
> > by replication.  We already have users who put all the etc files in it and
> > manage them that way.  Works great.  It's like rdist except it never screws
> > up and it has merging.
> 
> I'm just curious, what about security? Is this done in clear-text? 
> Sounds dangerous to put /etc/shadow, for example, in clear-text on the
> cable.

This is going over System Area Network. You don't encrypt your PCI, either.
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: SMP/cc Cluster description

[Alan Cox]

>    problem.  Scheduler, networking, device drivers, everything.  That's
>    thousands of locks and uncountable bugs, not to mention the impact on
>    uniprocessor performance.

Most of my block drivers in Linux have one lock. The block queuing layer
has one lock which is often the same lock.

> You tell me - which is easier, multithreading the networking stack to 
> 64 way SMP or running 64 distinct networking stacks?

Which is easier. Managing 64 routers or managing 1 router ?

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 10:37:18PM +0000, Alan Cox wrote:
> >    problem.  Scheduler, networking, device drivers, everything.  That's
> >    thousands of locks and uncountable bugs, not to mention the impact on
> >    uniprocessor performance.
> 
> Most of my block drivers in Linux have one lock. The block queuing layer
> has one lock which is often the same lock.

Hooray!  That's great and that's the way I'd like to keep it.  Do you think
you can do that on a 64 way SMP?  Not much chance, right?

> > You tell me - which is easier, multithreading the networking stack to 
> > 64 way SMP or running 64 distinct networking stacks?
> 
> Which is easier. Managing 64 routers or managing 1 router ?

That's a red herring, there are not 64 routers in either picture, there
are 64 ethernet interfaces in both pictures.  So let me rephrase the
question: given 64 ethernets, 64 CPUs, on one machine, what's easier,
1 multithreaded networking stack or 64 independent networking stacks?
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Alan Cox]

> > Most of my block drivers in Linux have one lock. The block queuing layer
> > has one lock which is often the same lock.
> 
> Hooray!  That's great and that's the way I'd like to keep it.  Do you think
> you can do that on a 64 way SMP?  Not much chance, right?

It wouldn't be a big problem to keep it that way on the well designed
hardware. The badly designed stuff (here Im thinking the NCR5380 I debugged
today since its fresh in my mind) I'd probably want 2 locks, one for queue
locking, one for request management.

> > Which is easier. Managing 64 routers or managing 1 router ?
> That's a red herring, there are not 64 routers in either picture, there
> are 64 ethernet interfaces in both pictures.  So let me rephrase the
> question: given 64 ethernets, 64 CPUs, on one machine, what's easier,
> 1 multithreaded networking stack or 64 independent networking stacks?

I think you miss the point. If I have to program the system as 64
independant stacks from the app level I'm going to go slowly mad

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 10:54:03PM +0000, Alan Cox wrote:
> > That's a red herring, there are not 64 routers in either picture, there
> > are 64 ethernet interfaces in both pictures.  So let me rephrase the
> > question: given 64 ethernets, 64 CPUs, on one machine, what's easier,
> > 1 multithreaded networking stack or 64 independent networking stacks?
> 
> I think you miss the point. If I have to program the system as 64
> independant stacks from the app level I'm going to go slowly mad

Well, that depends.  Suppose the application is a webserver.  Not your
simple static page web server, that one is on a shared nothing cluster
already.  It's a webserver that has a big honkin' database, with lots
of data being updated all time, the classic sort of thing that a big
SMP can handle but a cluster could not.  Fair enough?

Now imagine that the system is a collection of little OS images, each
with their own file system, etc.  Except /home/httpd is mounted on 
a globally shared file system.  Each os image has its own set of 
interfaces, one or more, and its own http server.  Which updates 
data in /home/httpd.

Can you see that this is a non-issue?  For this application, the ccCluster
model works great.  The data is all in a shared file system, nice and 
coherent, the apps don't actually know there is another OS banging on the 
data, it all just works.

Wait, I'll admit this means that the apps have to be thread safe, but that's
true for the traditional SMP as well.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 18:34:51 -0800

   The data is all in a shared file system, nice and coherent, the
   apps don't actually know there is another OS banging on the data,
   it all just works.

Larry1: "One way to get the ccCluster scalability is by un-globalizing
         the filesystem"

Larry2: "Let me tell you about this great application of ccClusters,
	 it involves using a shared file system.  It all just works."

Either you're going to replicate everyone's content or you're going to
use a shared filesystem.  In one case you'll go fast but have the same
locking problems as a traditional SMP, in the other case you'll go
slow because you'll be replicating all the time.

Which is it :-)

What I suppose is coming up is some example application that really
doesn't need a shared filesystem, which I bet will be a quite obscure
one or at least obscure enough that it can't justify ccCluster all on
it's own.

Re: SMP/cc Cluster description

[Alan Cox]

> the hardware's coherency.  No locks in the vfs or fs, that's all done
> in the mmap/page fault path for sure, but once the data is mapped you
> aren't dealing with the file system at all.

ftruncate

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 10:38:14PM +0000, Alan Cox wrote:
> > the hardware's coherency.  No locks in the vfs or fs, that's all done
> > in the mmap/page fault path for sure, but once the data is mapped you
> > aren't dealing with the file system at all.
> 
> ftruncate

I'm not sure what the point is.  We've already agreed that the multiple OS
instances will have synchonization to do for file operations, ftruncate
being one of them.

I thought the question was how N user processes do locking and my answer
stands: exactly like they'd do it on an SMP, with mutex_enter()/exit() on
some portion of the mapped file.  The mapped file is just a chunk of cache
coherent memory.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[Alexander Viro]

On Thu, 6 Dec 2001, Larry McVoy wrote:

> I'm not sure what the point is.  We've already agreed that the multiple OS
> instances will have synchonization to do for file operations, ftruncate
> being one of them.

That's nice.  But said operation involves serious wanking with metadata
and _that_ would better have exclusion with write(2) and some warranties
about pageouts.

You can do lockless get_block() and truncate().  And it will be a hive
of races always ready to break out.  We used to try that and it was
a fscking mess of unbelievable proportions - worse than that in full-blown
rename() support.

Re: SMP/cc Cluster description

[Alan Cox]

> > ftruncate
> 
> I'm not sure what the point is.  We've already agreed that the multiple OS
> instances will have synchonization to do for file operations, ftruncate
> being one of them.
> 
> I thought the question was how N user processes do locking and my answer
> stands: exactly like they'd do it on an SMP, with mutex_enter()/exit() on
> some portion of the mapped file.  The mapped file is just a chunk of cache

ftrucate invalidates that memory under you, on all nodes. That means you do
end up needing cross node locking and your file operations simply won't lie
down and scale cleanly

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 10:55:37PM +0000, Alan Cox wrote:
> > > ftruncate
> > 
> > I'm not sure what the point is.  We've already agreed that the multiple OS
> > instances will have synchonization to do for file operations, ftruncate
> > being one of them.
> > 
> > I thought the question was how N user processes do locking and my answer
> > stands: exactly like they'd do it on an SMP, with mutex_enter()/exit() on
> > some portion of the mapped file.  The mapped file is just a chunk of cache
> 
> ftrucate invalidates that memory under you, on all nodes. That means you do
> end up needing cross node locking and your file operations simply won't lie
> down and scale cleanly

Wait a second, you're missing something.  If you are going to make a single
OS image work on a 64 way (or whatever) SMP, you have all of these issues,
right?  I'm not introducing additional locking problems with the design.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 15:15:04 -0800
   
   Wait a second, you're missing something.  If you are going to make a single
   OS image work on a 64 way (or whatever) SMP, you have all of these issues,
   right?  I'm not introducing additional locking problems with the design.

And you're not taking any of them away from the VFS layer.  That is
where all the real fundamental current scaling problems are in the
Linux kernel.

That is why I spent so much timing describing the filesystem name path
locking problem, those are the major hurdles we have to go over.
Networking is old hat, people have done work to improve the scheduler
scaling, it's just these hand full of VFS layer issues that are
dogging us.

So my point is, if you're going to promote some "locking complexity"
advantage, I don't think that's where a ccCluster even makes a dent in
the viability spectrum.

Where it does have advantages are for things like offlining node
clusters in a NUMA system.  High availability et al.

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 03:19:45PM -0800, David S. Miller wrote:
>    From: Larry McVoy <lm@bitmover.com>
>    Date: Thu, 6 Dec 2001 15:15:04 -0800
>    
>    Wait a second, you're missing something.  If you are going to make a single
>    OS image work on a 64 way (or whatever) SMP, you have all of these issues,
>    right?  I'm not introducing additional locking problems with the design.
> 
> And you're not taking any of them away from the VFS layer.  That is
> where all the real fundamental current scaling problems are in the
> Linux kernel.

Sure I am, but I haven't told you how.  Suppose that your current
VFS can handle N cpus byt you have N*M cpus.  Take a look at
http://bitmover.com/lm/papers/bigfoot.ps and imagine applying that
technique here.  To summarize what I'm proposing, the locking problems are
because too many cpus want at the same data structures at the same time.
One way to solve that is to fine grain thread the data structures, and
that is a pain in the ass.  Another way to solve it may be to "stripe"
the file "servers".  Imagine each CPU serving up a part of a bigfoot
file system.  I've just reduced the scaling problems by a factor of M.

And, the ccCluster approach moves most of the nasty locking
problems into a ccCluster specific filesystem rather than buggering up
the generic paths.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 15:32:57 -0800
   
   And, the ccCluster approach moves most of the nasty locking
   problems into a ccCluster specific filesystem rather than buggering
   up the generic paths.

I still don't believe this, you are still going to need a lot of
generic VFS threading.  This is why myself and others keep talking
about ftruncate(), namei() et al.

If I look up "/etc" on bigfoot, littletoe, or whatever fancy name you
want to call the filesystem setup, SOMETHING has to control access to
the path name components (ie. there has to be locking).

You are not "N*M scaling" lookups on filesystem path components.
In fact, bigfoot sounds like it would make path name traversal more
heavyweight than it is now because these stripes need to coordinate
with each other somehow.

You keep saying "it'll be in the filesystem" over and over.  And the
point I'm trying to make is that this is not going to do away with the
fundamental problems.  They are still there with a ccCluster, they are
still there with bigfoot, and you are not getting N*M scaling on
filesystem name component walks.

Re: SMP/cc Cluster description

[Larry McVoy]

On Thu, Dec 06, 2001 at 03:47:35PM -0800, David S. Miller wrote:
>    From: Larry McVoy <lm@bitmover.com>
>    Date: Thu, 6 Dec 2001 15:32:57 -0800
>    
>    And, the ccCluster approach moves most of the nasty locking
>    problems into a ccCluster specific filesystem rather than buggering
>    up the generic paths.
> 
> I still don't believe this, you are still going to need a lot of
> generic VFS threading.  This is why myself and others keep talking
> about ftruncate(), namei() et al.
> 
> If I look up "/etc" on bigfoot, littletoe, or whatever fancy name you
> want to call the filesystem setup, SOMETHING has to control access to
> the path name components (ie. there has to be locking).

Sure, but you are assuming one file system, which is global.  That's
certainly one way to do it, but not the only way, and not the way that
I've suggested.  I'm not sure if you remember this, but I always advocated
partially shared and partially private.  /, for example, is private.
In fact, so is /proc, /tmp, and /dev.  There are /gproc, /gtmp, and /gdev
which are in the global namespace and do for the cluster what /<xxx>
does for a regular machine.

We can go around and around on this and the end result will be that I will
have narrowed the locking problem down to the point that only the processes
which are actually using the resource have to participate in the locking.
In a traditional SMP OS, all processes have to participate.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 16:17:44 -0800
   
   There are /gproc, /gtmp, and /gdev
   which are in the global namespace and do for the cluster what /<xxx>
   does for a regular machine.
   
And /getc, which is where my /getc/passwd is going to be.

   We can go around and around on this and the end result will be that
   I will have narrowed the locking problem down to the point that
   only the processes which are actually using the resource have to
   participate in the locking.  In a traditional SMP OS, all processes
   have to participate.

We can split up name spaces today with Al Viro's namespace
infrastructure.

But frankly for the cases where scalability matters, like a http
server, they are all going at the same files in a global file
space.

I still think ccClusters don't solve any new problems in the
locking space.  "I get rid of it by putting people on different
filesystems" is not an answer which is unique to ccClusters, current
systems can do that.

Re: SMP/cc Cluster description

[Larry McVoy]

> I still think ccClusters don't solve any new problems in the
> locking space.  "I get rid of it by putting people on different
> filesystems" is not an answer which is unique to ccClusters, current
> systems can do that.

If your point is that it doesn't solve any locking problems in the filesystem,
I'll almost grant you that.  Not quite because ccClusters open the door to 
different ways of solving problems that a traditional SMP doesn't.

However, where it wins big is on everything else.  Please explain to me how
you are going to make a scheduler that works for 64 CPUS that doesn't suck?
And explain to me how that will perform as well as N different scheduler
queues which I get for free.  Just as an example.  We can then go down the
path of every device driver, the networking stack, the process interfaces,
signals, etc.  

There is a hell of a lot of threading that has to go on to get to
64 cpus and it screws the heck out of the uniprocessor performance.
I think you want to prove how studly you are at threading, David,
but what you are really doing is proving that you are buried in the
trees and can't see the forest.  Pop up 50,000 feet and think about it.
Let's go have some beers and talk about it off line.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description

[David S. Miller]

   From: Larry McVoy <lm@bitmover.com>
   Date: Thu, 6 Dec 2001 18:43:27 -0800
   
   However, where it wins big is on everything else.  Please explain to me how
   you are going to make a scheduler that works for 64 CPUS that doesn't suck?

What stops me from basically doing a scheduler which ends up doing
what ccCluster does, groups of 4 cpu nodes?  Absolutely nothing of
course.  How is ccCluster unique in this regard then?

The scheduler is a mess right now only because Linus hates making
major changes to it.

   We can then go down the path of ... the networking stack ...
   signals ...

Done and done.  Device drivers are mostly done, and what was your
other category... oh process interfaces, those are done too.
   
In fact character devices are the only ugly area in 2.5.x, and
who really cares if TTYs scale to 64 cpus :-)  But this will get
mostly done anyways to kill off the global kernel lock completely.

   There is a hell of a lot of threading that has to go on to get to
   64 cpus and it screws the heck out of the uniprocessor performance.

Not with CONFIG_SMP turned off.  None of the interesting SMP overhead
hits the uniprocessor case.

Why do you keep talking about uniprocessor being screwed?  This is why
we have CONFIG_SMP, to nop the bulk of it out.

   I think you want to prove how studly you are at threading, David,

No, frankly I don't.

What I want is for you to show what is really unique and new about
ccClusters and what incredible doors are openned up by it.  So far I
have been shown ONE, and that is the high availability aspect.

To me, it is far from the holy grail you portray it to be.

   Let's go have some beers and talk about it off line.

How about posting some compelling arguments online first? :-)

It all boils down to the same shit currently.  "ccClusters lets you do
this", and this is leading to "but we can do that already today".

Franks a lot,
David S. Miller
davem@redhat.com

Re: SMP/cc Cluster description

[Martin J. Bligh]

> However, where it wins big is on everything else.  Please explain to me how
> you are going to make a scheduler that works for 64 CPUS that doesn't suck?

Modifying the current scheduler to use multiple scheduler queues is not 
particularly hard.  It's been done already. See http://lse.sourceforge.net or 
Davide's work.

Please explain how you're going to load balance the scheduler queues across 
your system in a way that doesn't suck.

> And explain to me how that will perform as well as N different scheduler
> queues which I get for free.  Just as an example. 

I dispute that the work that you have to do up front to get ccClusters to work
is "free". It's free after you've done the work already. You may think it's 
*easier*, but that's different.

So we're going to do our work one step at a time in many small chunks, and
you're going to do it all at once in one big chunk (and not end up with as
cohesive a system out of the end of it) .... not necessarily a huge difference
in overall effort (though I know you think it is, the logic you're using to demonstrate
your point is fallacious).

Your objection to the "more traditional" way of scaling things (which seems
to be based around what you call the locking cliff) seems to be that it greatly
increases the complexity of the OS. I would say that splitting the system into
multiple kernels then trying to glue it all back together also greatly increases
the complexity of the OS. Oh, and the complexity of the applications that run 
on it too if they have to worry about seperate bits of the FS for each instance 
of the sub-OS.

Martin.

Re: SMP/cc Cluster description

[Rik van Riel]

On Wed, 5 Dec 2001, Davide Libenzi wrote:
> On Thu, 6 Dec 2001, Rusty Russell wrote:
>
> > I'd love to say that I can solve this with RCU, but it's vastly non-trivial
> > and I haven't got code, so I'm not going to say that. 8)
>
> Lockless algos could help if we're able to have "good" quiescent point
> inside the kernel. Or better have a good quiescent infrastructure to
> have lockless code to plug in.

Machines get dragged down by _uncontended_ locks, simply
due to cache line ping-pong effects.

regards,

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/

Re: SMP/cc Cluster description

[Davide Libenzi]

On Thu, 6 Dec 2001, Rik van Riel wrote:

> On Wed, 5 Dec 2001, Davide Libenzi wrote:
> > On Thu, 6 Dec 2001, Rusty Russell wrote:
> >
> > > I'd love to say that I can solve this with RCU, but it's vastly non-trivial
> > > and I haven't got code, so I'm not going to say that. 8)
> >
> > Lockless algos could help if we're able to have "good" quiescent point
> > inside the kernel. Or better have a good quiescent infrastructure to
> > have lockless code to plug in.
>
> Machines get dragged down by _uncontended_ locks, simply
> due to cache line ping-pong effects.

Rik, i think you're confused about lockless algos.
It's not an rwlock where the reader has to dirty a cacheline in any case,
the reader simply does _not_ write any cache line accessing the
list/hash/tree or whatever you use.
These algo uses barries and all changes are done when the system walk
through a quiescent state by flushing a list-of-changes.
Drawback, you've to be able to tollerate stale data.



- Davide

Re: SMP/cc Cluster description

[Rik van Riel]

On Thu, 6 Dec 2001, Davide Libenzi wrote:

> > Machines get dragged down by _uncontended_ locks, simply
> > due to cache line ping-pong effects.
>
> Rik, i think you're confused about lockless algos.
> It's not an rwlock where the reader has to dirty a cacheline in any case,
> the reader simply does _not_ write any cache line accessing the
> list/hash/tree or whatever you use.

Hmmm indeed, so the cache lines can be shared as long
as the data is mostly read-only. I think I see it now.

However, this would only work for data which is mostly
read-only, not for anything else...

Rik
-- 
DMCA, SSSCA, W3C?  Who cares?  http://thefreeworld.net/

http://www.surriel.com/		http://distro.conectiva.com/

Re: SMP/cc Cluster description

[Davide Libenzi]

On Thu, 6 Dec 2001, Rik van Riel wrote:

> On Thu, 6 Dec 2001, Davide Libenzi wrote:
>
> > > Machines get dragged down by _uncontended_ locks, simply
> > > due to cache line ping-pong effects.
> >
> > Rik, i think you're confused about lockless algos.
> > It's not an rwlock where the reader has to dirty a cacheline in any case,
> > the reader simply does _not_ write any cache line accessing the
> > list/hash/tree or whatever you use.
>
> Hmmm indeed, so the cache lines can be shared as long
> as the data is mostly read-only. I think I see it now.
>
> However, this would only work for data which is mostly
> read-only, not for anything else...

yes of course, but in such case these methods could help solving cache
issues over traditional rwlocks.



- Davide

Re: SMP/cc Cluster description

[Momchil Velikov]

>>>>> "Larry" == Larry McVoy <lm@bitmover.com> writes:
Larry>    Where this approach wins big is everywhere except the page cache.  Every
Larry>    single data structure in the system becomes N-way more parallel -- with
Larry>    no additional locks -- when you boot up N instances of the OS.  That's

I was wondering about multiple OS instances in their own address
space. What's the need for separate address spaces for the kernels ?

It looks more natural to me to _actually_ have N instances of kernel
data structures in the _same address space_, i.e. turning
each global variable into an array_ indexed by an "instance id",
much in the same way as we have now per-CPU structures. Well,
I don't actually think it would be as simple as stated above, I'm just
proposing it as a general approach towards ccclustering.

(btw, there was some discussion on #kernelnewbies, on Nov 12th and
21st, you can find the logs here
http://vengeance.et.tudelft.nl/~smoke/log/kernelnewbies/2001-11/)


Regards,
-velco

Re: SMP/cc Cluster description

[Davide Libenzi]

On Tue, 4 Dec 2001, David S. Miller wrote:

>    From: Larry McVoy <lm@bitmover.com>
>    Date: Tue, 4 Dec 2001 16:36:46 -0800
>
>    OK, so start throwing stones at this.  Once we have a memory model that
>    works, I'll go through the process model.
>
> What is the difference between your messages and spin locks?
> Both seem to shuffle between cpus anytime anything interesting
> happens.
>
> In the spinlock case, I can thread out the locks in the page cache
> hash table so that the shuffling is reduced.  In the message case, I
> always have to talk to someone.

Time ago I read an interesting article that implemented shared memory over
network ( ATM in that case ) reproducing in large scale the
cache/memory/bus computer architecture.
Shared memory on each node was the equivalent of the CPU cache, a "generic
shared memory repository" was the equivalent of the main memory and the
snooping traffic was running on the network.
I think I picked it up from ACM but I can't find it right now.




- Davide

Re: SMP/cc Cluster description

[Stephen Satchell]

At 06:36 PM 12/4/01 -0800, David S. Miller wrote:
>What is the difference between your messages and spin locks?
>Both seem to shuffle between cpus anytime anything interesting
>happens.
>
>In the spinlock case, I can thread out the locks in the page cache
>hash table so that the shuffling is reduced.  In the message case, I
>always have to talk to someone.

While what I'm about to say has little bearing on the SMP/cc case:  one 
significant advantage of messages over spinlocks is being able to assign 
priority with low overhead in the quick-response real-time multi-CPU 
arena.  I worked with a cluster of up to 14 CPUs using something very much 
like NUMA in which task scheduling used a set of prioritized message 
queues.  The system I worked on was designed to break transaction-oriented 
tasks into a string of "work units" each of which could be processed very 
quickly -- on the order of three milliseconds or less.  (The limit of 14 
CPUs was set by the hardware used to implement the main system bus.)

I bring this up only because I have never seen a spinlock system that dealt 
with priority issues very well when under heavy load.

OK, I've said my piece, now I'll sit back and continue to watch your 
discussion.

Satch

Re: SMP/cc Cluster description [was Linux/Pro]

[Rob Landley]

On Tuesday 04 December 2001 07:36 pm, Larry McVoy wrote:

> The most recent set of slides are here:
>
>     http://www.bitmover.com/ml/slide01.html
>
> A couple of useful papers are at
>
>     http://www.bitmover.com/llnl/smp.pdf
>     http://www.bitmover.com/llnl/labs.pdf
>
> The first explains why I think fine grained multi threading is a mistake
> and the second is a paper I wrote to try and get LLNL to push for what
> I called SMP clusters (which are not a cluster of SMPs, they are a
> cluster of operating system instances on a single SMP).

This sounds a bit like the shared memory cluster work.  (See last month's big 
thread I had with Martin.  There's URLs on the web but my laptop's offline 
right now.  Google would know.)

If you take a beowulf style cluster, add in shared memory that can page fault 
across the network (just for explicitly shared mappings, like a networked 
shm), and a networkable sempahore implementation, you can program it a lot 
like NUMA without needing special hardware.  (Gigabit ethernet covers a lot 
of sins, and Myrinet's still relatively cheap compared to most 
supercomputers.)  You can even do a cheesy implementation of everything in 
userspace if you're not trying to scale it very far (although that largely 
wastes the point of myrinet, at least).

Some people have even been working on migrating processes from one node to 
another to automatically load balance, although that's a bit fancier than 
I've ever bothered with.  The hard part of it all is management of the 
cluster, and that's something people have been putting a LOT of effort into 
from all directions...

> The basic idea is this: if you consider the usefulness of an SMP versus a
> cluster, the main thing in favor of the SMP is
>
>     all processes/processors can share the same memory at memory speeds.
>     I typically describe this as "all processes can mmap the same data".
>     A cluster loses here, even if it provides DSM over a high speed
>     link, it isn't going to have 200 ns caches misses, it's orders of
>     magnitude slower.  For a lot of MPI apps that doesn't matter, but
>     there are apps for which high performance shared memory is required.
>
> There are other issues like having a big fast bus, load balancing, etc.,
> but the main thing is that you can share data quickly and coherently.

10gig ethernet is supposed to be coming out in 2003, and some people have 
prototypes already.  I'm fairly certain this is faster than the current 
generations of PCI bus.  Of course throughput != latency, but Grace Hopper 
was font of carrying around 11 centimeters of wire in her knitting bag and 
calling it a nanosecond, which is where the big bucks NUMA systems have their 
problems too. :)

> If you don't need that performance/coherency and you can afford to
> replicate the data, a traditional cluster is a *much* cheaper and
> easier answer.

The amount of ram a copy of the OS takes up these days is CHEAP.  A well 
tuned system needs what, an extra 16 megs per node?  (And that includes a lot 
of squishiness for buffers you probably need per-node anyway.)  If you're 
worried about that expense, you're obviously not paying for high-end hardware 
anyway...

I've seen people trying to do spinlocks across a numa system.  Why?  Don't Do 
That Then.  Purely OS internal abstractions don't need to be shared across 
the cluster.  I can share a printer through the network right now by just 
having my app talk to the server handling it, yet people seem to be trying to 
make part of the driver for a device live on the other side of the NUMA 
machine.  Why?  (Because we can!  Doesn't make it a good idea...)

> If we want to get the most use out of big SMP boxes but we also want to
> do the least amount of "damage" in the form of threading complexity in
> the source base.  This is a "have your cake and eat it too" goal, one
> that I think is eminently reachable.
>
> So how I propose we do this is by booting multiple Linux images on
> a single box.  Each OS image owns part of the machine, 1-4 CPUs, 0 or
> more devices such as disk, ethernet, etc., part of memory.  In addition,
> all OS images share, as a page cache, part of main memory, typically
> the bulk of main memory.

SMP is symmetrical.  On a truly SMP machine, there's no point in having 
multiple OS images because they're all equal cost to talk to main memory, so 
they might as well share a kernel image.  (It's read-only, there's no 
contention, they just cache it.  There may be per-CPU data structures, but 
we've got those already.)

SMP is likely to increase as die sizes shrink, because you can put 4 
processors on a chip today on PowerPC.  (This is just an alternative to 
insanely long pipelines that just start increasing latency of a pipeline 
flush.  Pentium 4 already has stages that do nothing more than transport data 
from one side of the chip to the other, that's just nuts.)  Plus more 
execution cores: An athlon currently has 3 cores, as do iTanic and Crusoe.  
Either you decouple them so they execute different stuff, or they run NOPs.  
So either way, you wind up with SMP on one die.  (You can't make chips TOO 
much smaller because it becomes less economical to cut the wafer up that 
small.  Harvesting and connecting the chips at finer granularity increases 
their cost...)

If die sizes shrink another 4 or 5 times before we hit atomic limits, we can 
fit at least 32 processors on a chip.  And then we just start increasing the 
number of layers and make 3D circuitry assuming we can deal with the heat 
problem (which people are working on: heat sinks in the middle of the chip, 
just wire around it).

THIS is why many-way SMP is interesting.  Crusoe and strongarm have the 
northbridge on die which makes this kind of thing easier (getting into shared 
L1 cache is bound to be fun), and then there's having the motherboard do SMP 
as well.  Assuming the motherboards can handle 8-way, dedicated memory 
bandwidth interconnects (like ev6), if each chip has just 8 processors, we're 
talking 64-way SMP for under $10k in a few years, meaning it'll be $2k a 
couple years after that.

There are three main reasons we haven't seen SMP take off in the past 15 
years, despite the fact there were SMP 486 motherboards back in the 80's.  
The first is that nothing microsoft has ever put out could gracefully handle 
it (they didn't even pretend to multitask one CPU until the 90's).  The 
second is that most programmers (outside of Unix and OS/2) didn't know what a 
semaphore was before about 1998, and are just now thinking about breaking 
stuff up so portions of the program can advance asynchronously.  Third was 
that low volume is high cost, so there was a chicken and egg problem on the 
hardware side.

Now the programming's gradually starting to get there, and we've got our 
first SMP athlon boards priced so a hobbyist can save up for one.  I don't 
think it's going to decrease in the future...

> OK, so we've handled setting up the page tables, but we haven't handled
> page faults or pageouts.  Let's punt on pageouts for the time being,
> we can come back to that.  Let's figure out a pagefault path that will
> give correct, albeit slow, behaviour.  Suppose that when the client faults
> on a page, the client side file system sends a pagefault message to the
> proxy, the proxy faults in the page, calls a new vtop() system call to
> get the physical page, and passes that page descriptor back to the client
> side.  The client side loads up the TLB & page tables and away we go.
> Whoops, no we don't, because the remote OS could page out the page and
> the client OS will get the wrong data (think about a TLB shootdown that
> _didn't_ happen when it should have; bad bad bad).  Again, thinking
> just from the correctness point of view, suppose the proxy mlock()ed
> the page into memory.  Now we know it is OK to load it up and use it.
> This is why I said skip pageout for now, we're not going to do them
> to start with anyway.
>
> OK, so start throwing stones at this.  Once we have a memory model that
> works, I'll go through the process model.

If you only worry about explicitly shared memory (a multi-process model vs a 
multi-thread model), you can cheese your way out of this by mounting a 
modified network filesystem in /dev/shm if you don't mind hideously high 
latency and a central point of failure.  (The filesystem has to be able to 
initiate page invalidations on mmaped areas, but I suspect this is a problem 
somebody's already dealt with.  Haven't played with it in a while...)

Rob

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

> I've seen people trying to do spinlocks across a numa system.  Why?  Don't Do 

Because they work well enough for low-contention locks. We have numa
aware locks too. Or just make the resource node-local.

M.

Re: SMP/cc Cluster description [was Linux/Pro]

[Alan Cox]

> So either way, you wind up with SMP on one die.  (You can't make chips TOO 
> much smaller because it becomes less economical to cut the wafer up that 
> small.  Harvesting and connecting the chips at finer granularity increases 

Except for VIA (where they cant really make them much smaller) the
mainstream x86 dies are _gigantic_

Re: SMP/cc Cluster description [was Linux/Pro]

[Rob Landley]

On Wednesday 05 December 2001 07:34 pm, Alan Cox wrote:
> > So either way, you wind up with SMP on one die.  (You can't make chips
> > TOO much smaller because it becomes less economical to cut the wafer up
> > that small.  Harvesting and connecting the chips at finer granularity
> > increases
>
> Except for VIA (where they cant really make them much smaller) the
> mainstream x86 dies are _gigantic_

Yeah.  Exactly.  Especially Intel's own.  Itanium's enormous, and I mentioned 
how Pentium 4 had pushed pipelining slightly beyond where you really get all 
that much benefit from it.

They've grown to take advantage of larger transistor budgets, but that's 
starting to hit limits on how big you can usefully make them before you get 
serious diminishing returns and problems with clock skew and signal 
propogation delays and other such fun.  (Hence the experiments with clockless 
processors, etc.)  The longer the wire, the longer it takes a signal to go 
down it.  You want contained modules so you can clock them fast (hence 
pipeline stages).

Athlon hasn't gone pipeline happy the way P4 has, so they don't suffer as 
badly from pipeline stalls, but it already has three execution cores (and 
requesite monstrous front-end decoding and scheduling stuff to those cores), 
with no plans to add a fourth core because the third isn't busy enough that 
they think it would help.

Yet manufacturing is going to continue to shrink the density, giving you more 
area to work with and a higher transistor budget, which is about 80% of 
Moore's Law.  Despite gloom and doom predictions since at least the early 
1980's has got four or five more doublings to go before we even have to worry 
about increasing the number of layers to get extra space.  So where's that 
extra transistor budget going to go?  Bigger L1 cache?  Fun, but the main 
benefit of a really huge cache isn't on a UP system, but on SMP.  The benefit 
of an L1 cache is one of them "integral of somevalue/x" functions, the 
increase in which falls off pretty rapidly the bigger it gets: the more bytes 
of cache the greater percentage chance your next piece of data will be in 
cache, but also the more transistors are guaranteed to be sitting idle 
because they do NOT contain data you need this cycle.  A point of diminishing 
returns definitely exists here.

You also can't make the chips infinitely small because it's not worth the 
money (manufacturing expense).  Beyond a certain point, more chips per wafer 
aren't that much cheaper because the taxidermy to test, cut, connect, mount, 
package, and ship them becomes a bigger percentage of the cost.  And you 
still want to amortize your factory, so reducing per-chip manufacturing 
expense won't reduce cost noticeably anyway as long as new manufacturing 
processes require billions to get the new line up and running. 

Intel is trying to transition to VLIW as a way to use more linked execution 
cores to do SOMETHING useful with the extra transistor budget, and 
Transmeta's even trying to get VLIW to do something USEFUL, but it's a 
different programming model that'll take a while to transition to, and it 
requires intelligent compilers finding paralellism in the chip, which isn't 
easy.

Or, you could use the execution units to run independent threads, which Intel 
is ALREADY experimenting with (SMT instead of SMP), but that's really just a 
way of backfitting SMP-on-a-die onto the existing linked-core design without 
having to redo your process counter and cache circuitry.  And again this 
requires compilers to catch up, which won't happen for a while, and even then 
a programmer could really do a better job of telling the computer what to do.

So the logical thing to do is SMP on a single die (which IBM at least has 
been attempting).  Not only does it convert transistors into execution speed 
efficiently, it allows you to have a flaming big L1 and L2 cache in a way 
that it's more likely to accomplish something useful.

And THAT'S what makes SMP interesting in the future.  To me anyway.  I could 
be wrong...

Rob

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

>> > There seems to be a little misunderstanding here; from what
>> > I gathered when talking to Larry, the idea behind ccClusters
>> > is that they provide a single system image in a NUMA box, but
>> > with separated operating system kernels.
> 
> Right except NUMA is orthogonal, ccClusters work fine on a regular SMP 
> box.

Can you clarify exactly what hardware you need for ccClusters? I've heard
from some people that you need some hardware support to do remote
memory accesses, but you seem to be saying you don't ?

If I give you 16 SMP systems, each with 4 processors and a gigabit
ethernet card, and connect those ethers through a switch, would that
be sufficient hardware?
 
>> OK, then I've partially misunderstood this ... can people provide some 
>> more reference material? Please email to me, and I'll collate the results
>> back to the list (should save some traffic).
> 
> I'll try and type in a small explanation, I apologize in advance for the
> bervity, I'm under a lot of pressure on the BK front these days...
> 
> The most recent set of slides are here:
> 
>     http://www.bitmover.com/ml/slide01.html
> 
> A couple of useful papers are at
> 
>     http://www.bitmover.com/llnl/smp.pdf
>     http://www.bitmover.com/llnl/labs.pdf
> 
> The first explains why I think fine grained multi threading is a mistake
> and the second is a paper I wrote to try and get LLNL to push for what
> I called SMP clusters (which are not a cluster of SMPs, they are a 
> cluster of operating system instances on a single SMP).

OK, I went and read those, and I think I now understand the general 
concept of what you're advocating. Before we get into too many low-level
technical discussions, if we can step back for a second and take the
50,000 ft view ....

(I've said "we" quite a bit below, where maybe I should sometimes say "I". 
I think my views are very roughly representative of *some* other NUMA people, 
but I have no right whatsoever to claim that, and I'm not really that sure of it 
either. Lots of things below shamelessly stolen from people who've been kind
enough to explain such concepts to me in the past).

I think what you're trying to get to is actually quite similar to where we're
trying to get to, you're just going up the mountain from the other side than
the point we chose to start at. I'll leave aside the discussion for the moment
of which face is easier to scale, I just want to point out that we're actually
going to more or less the same place.

We're both trying to present a single system image to the application 
by coupling a bunch of SMP systems together. To take an example of
a NUMA architecture (the one I know reasonably well) the Sequent / IBM 
NUMA-Q hardware architecture is basically a bunch of 4 way PCs with a big 
fat  interconnect slung down the back of them (I'm sure I'll get hung, drawn
and quartered by marketing for that. They're *really* nice PCs. The interconnect
is *very* quick. And there's some really nice cache coherency hardware, etc.
But basically they're just 4 way SMPs + interconnect). 

(I'm going to call each of the SMP units a node) You might choose to to do some 
of the memory transfers between nodes in software - we chose to do it in
hardware. You might choose to have distinct physical memory maps, we chose
to have a unified one. That's more of a detail - the concept's still the same.

We started out with the concept of a single OS image from a single instance of
the kernel (like a really big SMP box), and we break things down as need be. 
As I understand what you're advocating, you want to start out with multiple
instances of the kernel and build them back into one single OS image.

If you'll forgive me being rather woolly and simplistic for the minute (and don't
try to analyse this analogy to death), imagine the OS as two layers:

1) an upper layer doing things like system calls, etc. that provide a user level API
2) a lower layer doing things like page cache, disk IO, etc.

As I understand this, both of us will have the upper layer as a single instance
across the whole system. We'll start with a single instance of the lower layer,
you'll start with multiple instances. You'll presumably have to glue some bits
of that lower layer together, we'll presumably break some apart.

>From what I've read of your documents, you seem to think the only way we
can make a single OS instance scale is by introducing billions of fine-grained 
locks. I'm sure that's not *exactly* what you meant, but that's kind of how it 
comes across. Yes, there will defintely need to be some splitting up of locks.
But there are lots of other tools in the toolbox. Some things we might well do
are introduce a per-node page cache, maybe a per node dcache / inode
cache. One example we've already is have a scheduler that can be set up
with a seperate runqueue per node. And we'll replicate a copy of kernel
text to every node - processes on that node will use a local copy of the
kernel text. 

By the time we've split up a bunch of subsystems to work on a per-node
basis (with some global interactions between them) like this, it starts to look
a lot more like what you're doing with ccClusters.

Let's take an example of a bunch of statistics counters (say 200) that 
start off with a single lock covering all of them. We could break up the 
lock into 200 locks, one for each stats counter, or we could break up 
the counters into one counter per CPU with atomic update with no locking.
If you want to know the result of the counters, you read all of them and
add them together. No locking at all. In the case of many writes and few
reads, this would work much better. There are many ways to crack a nut
(or a lock).

I don't think fine-grained locking necessarily makes things slower. Done
badly, it certainly could. It could easily make things more complex, especially
if done badly. It will probably make things a little more complex, even if done
well. Personally, I believe that we can keep that managable, especially if
we're careful to actually document stuff. This means, for instance, things 
like Rick Lindsley's locking document, and actually putting some comments
against data structures saying what they're locked by! (Each is one index
into the x-map). We should avoid getting too carried away, and creating
*too* many locks - that is a danger, I agree with you there.

> The basic idea is this: if you consider the usefulness of an SMP versus a
> cluster, the main thing in favor of the SMP is
> 
>     all processes/processors can share the same memory at memory speeds.
>     I typically describe this as "all processes can mmap the same data".
>     A cluster loses here, even if it provides DSM over a high speed
>     link, it isn't going to have 200 ns caches misses, it's orders of
>     magnitude slower.  For a lot of MPI apps that doesn't matter, but
>     there are apps for which high performance shared memory is required.

I think a far more valid / interesting comparison would be NUMA vs
ccClusters rather than SMP - they're much more comparible. I don't
think anyone's advocating a 128 way SMP system (or at least I'm not).

> There are other issues like having a big fast bus, load balancing, etc.,
> but the main thing is that you can share data quickly and coherently.
> If you don't need that performance/coherency and you can afford to 
> replicate the data, a traditional cluster is a *much* cheaper and 
> easier answer.  Many problems, such as web server farms, are better
> done on Beowulf style clusters than an SMP, they will actually scale
> better.

Absolutely. For for some problems, the Beowulf style approach is much
cheaper and easier.
 
> OK, so suppose we focus on the SMP problem space.  It's a requirement
> that all the processes on all the processors need to be able to access
> memory coherently.  DSM and/or MPI isn't an answer for this problem 
> space.
> 
> The traditional way to use an SMP is to take a single OS image and 
> "thread" it such that all the CPUs can be in the OS at the same time.
> Pretty much all the data structures need to get a lock and each CPU
> takes the lock before it uses the data structure.  The limit of the
> ratio of locks to cache lines is 1:1, i.e., each cache line will need
> a lock in order to get 100% of the scaling on the system (yes, I know
> this isn't quite true but it is close and you get the idea).
> 
> Go read the "smp.pdf" paper for my reasons on why this is a bad approach,
> I'll assume for now you are willing to agree that it is for the purposes
> of discussion.
> 
> If we want to get the most use out of big SMP boxes but we also want to
> do the least amount of "damage" in the form of threading complexity in
> the source base.  This is a "have your cake and eat it too" goal, one
> that I think is eminently reachable.
> 
> So how I propose we do this is by booting multiple Linux images on
> a single box.  Each OS image owns part of the machine, 1-4 CPUs, 0 or
> more devices such as disk, ethernet, etc., part of memory.  In addition,
> all OS images share, as a page cache, part of main memory, typically
> the bulk of main memory.
> 
> The first thing to understand that the *only* way to share data is in
> memory, in the globally shared page cache.  You do not share devices,
> devices are proxied.  So if I want data from your disk or file system,
> I ask you to put it in memory and then I mmap it.  In fact, you really
> only share files and you only share them via mmap (yeah, read and write
> as well but that's the uninteresting case).
> 
> This sharing gets complex because now we have more than one OS image
> which is managing the same set of pages.  One could argue that the 
> code complexity is just as bad as a fine grained multi threaded OS
> image but that's simply incorrect.  I would hide almost 100% of this
> code in a file system, with some generic changes (as few as possible)
> in the VM system.  There are some changes in the process layer as well,
> but we'll talk about them later.
> 
> If you're sitting here thinking about all the complexity involved in
> sharing pages, it is really helpful to think about this in the following
> way (note you would not actually implement it like this in the long
> run but you could start this way):
> 
> Imagine that for any given file system there is one server OS image and N
> client os images.  Imagine that for each client, there is a proxy process
> running on behalf of the client on the server.  Sort of like NFS biods.
> Each time the client OS wants to do an mmap() it asks the proxy to do
> the mmap().  There are some corner cases but if you think about it, by
> having the proxies do the mmaps, we *know* that all the server OS data
> structures are correct.  As far as the server is concerned, the remote
> OS clients are no different than the local proxy process.  This is from
> the correctness point of view, not the performance point of view.
> 
> OK, so we've handled setting up the page tables, but we haven't handled
> page faults or pageouts.  Let's punt on pageouts for the time being,
> we can come back to that.  Let's figure out a pagefault path that will
> give correct, albeit slow, behaviour.  Suppose that when the client faults
> on a page, the client side file system sends a pagefault message to the
> proxy, the proxy faults in the page, calls a new vtop() system call to
> get the physical page, and passes that page descriptor back to the client
> side.  The client side loads up the TLB & page tables and away we go.
> Whoops, no we don't, because the remote OS could page out the page and
> the client OS will get the wrong data (think about a TLB shootdown that
> _didn't_ happen when it should have; bad bad bad).  Again, thinking 
> just from the correctness point of view, suppose the proxy mlock()ed
> the page into memory.  Now we know it is OK to load it up and use it.
> This is why I said skip pageout for now, we're not going to do them 
> to start with anyway.

That doesn't seem a million miles away from creating yourself a local memory
buffer, and then setting up the DMA engine of an interface card on a remote 
node to transfer you the data into that local buffer. Not exactly the same, 
but the concept's not all that dissimilar.

Martin.

Re: SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

> If I give you 16 SMP systems, each with 4 processors and a gigabit
> ethernet card, and connect those ethers through a switch, would that
> be sufficient hardware?

You've completely misunderstood the message, sorry, I must not have been clear.
What I am proposing is to cluster *OS* images on a *single* SMP as a way of
avoiding most of the locks necessary to scale up a single OS image on the 
same number of CPUs.

It has nothing to do with clustering more than one system, it's not that kind
of clustering.  It's clustering OS images.  

To make it easy, let's imagine you have a 16 way SMP box and an OS image that
runs well on one CPU.  Then a ccCluster would be 16 OS images, each running
on a different CPU, all on the same hardware.

DEC has done this, Sun has done this, IBM has really done this, but what none
of them have done is make mmap() work across OS boundaries.  If all OS images
could share the same page cache, that's a first order approximation of an 
16-way SMP OS with out all the locking.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[Rob Landley]

On Wednesday 05 December 2001 02:11 pm, Larry McVoy wrote:
> > If I give you 16 SMP systems, each with 4 processors and a gigabit
> > ethernet card, and connect those ethers through a switch, would that
> > be sufficient hardware?
>
> You've completely misunderstood the message, sorry, I must not have been
> clear. What I am proposing is to cluster *OS* images on a *single* SMP as a
> way of avoiding most of the locks necessary to scale up a single OS image
> on the same number of CPUs.
>
> It has nothing to do with clustering more than one system, it's not that
> kind of clustering.  It's clustering OS images.

So basically, you're just turning the per-processor data into a tree?

> To make it easy, let's imagine you have a 16 way SMP box and an OS image
> that runs well on one CPU.  Then a ccCluster would be 16 OS images, each
> running on a different CPU, all on the same hardware.
>
> DEC has done this, Sun has done this, IBM has really done this, but what
> none of them have done is make mmap() work across OS boundaries.

The shared memory clustering people are basically trying to make 
mmap+semaphores work across a high speed LAN.  Why?  Because it's cheap, and 
the programming model's familiar.

Approaching it all from a different direction, though.  Probably not of help 
to you...

Rob

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

>> If I give you 16 SMP systems, each with 4 processors and a gigabit
>> ethernet card, and connect those ethers through a switch, would that
>> be sufficient hardware?
> 
> You've completely misunderstood the message, sorry, I must not have been clear.

Oops ... that's twice now ;-) Maybe I'm reading it with too many preconceived
notions of what you're doing from conversations I've had with other people
about ccClusters. I'll try to do a mental reset, and start from a clean slate.

> What I am proposing is to cluster *OS* images on a *single* SMP as a way of
> avoiding most of the locks necessary to scale up a single OS image on the 
> same number of CPUs.

Which, to me, makes the whole thing much less interesting, since there aren't
SMP systems about that are really large that I know of anyway. Scaling to 
the size of current SMP systems is a much less difficult problem than scaling
to the size of NUMA systems.

BUT ... much of the rest of the message I sent you still applies anyway.
You can create virtual "pools" or "resource domains" within an SMP system
in the same way nodes exist on NUMA and work from the starting point of
a single OS image, instead of multiple.

The main advantage of starting with a single OS image, as I see it, is
that you have a system that works fine, but performs badly, from the
outset. Makes it easier to do development on - that's where I have the
NUMA-Q platform at the moment - it thinks it's an SMP box.

Martin.

Re: SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

> > What I am proposing is to cluster *OS* images on a *single* SMP as a way of
> > avoiding most of the locks necessary to scale up a single OS image on the 
> > same number of CPUs.
> 
> Which, to me, makes the whole thing much less interesting, since there aren't
> SMP systems about that are really large that I know of anyway. Scaling to 
> the size of current SMP systems is a much less difficult problem than scaling
> to the size of NUMA systems.

We don't agree on any of these points.  Scaling to a 16 way SMP pretty much 
ruins the source base, even when it is done by very careful people.

> The main advantage of starting with a single OS image, as I see it, is
> that you have a system that works fine, but performs badly, from the
> outset. 

Hey, I can make one of those :-)

Seriously, I went through this at SGI, that's exactly what they did, and it
was a huge mistake and it never worked.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

> We don't agree on any of these points.  Scaling to a 16 way SMP pretty much 
> ruins the source base, even when it is done by very careful people.

I'd say that the normal current limit on SMP machines is 8 way.
But you're right, we don't agree.  Time will tell who was right.
When I say I'm interested in 16 way scalability, I'm not talking about
SMP, so perhaps we're talking at slightly cross purposes.
 
> Seriously, I went through this at SGI, that's exactly what they did, and it
> was a huge mistake and it never worked.

You seem to make this odd logical deduction quite a bit - you (or company X)
has tried concept X before. Their implementation didn't work. Therefore the
concept is bad. It's not particularly convincing as an argument style to others.
But I understand that it makes *you* want to try something else.

Martin.

Re: SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

On Wed, Dec 05, 2001 at 01:14:45PM -0800, Martin J. Bligh wrote:
> > Seriously, I went through this at SGI, that's exactly what they did, and it
> > was a huge mistake and it never worked.
> 
> You seem to make this odd logical deduction quite a bit - you (or company X)
> has tried concept X before. Their implementation didn't work. Therefore the
> concept is bad. It's not particularly convincing as an argument style to others.

I think you'll find that a common theme amongst people with experience.  
I also will point out that what you are saying is exactly what I and 
every other young hotshot said in our twenties.  It's funny how when 
you let 15 years go by the people that you argued with in the past 
suddenly become right.  It's certainly been a pattern in my life that
when I argue with people who are older and more experienced, 9 times out
of 10, I let some years pass and I find myself arguing their position.
It's also interesting to note that these days virtually 100% of that
sort of discussion is with someone younger.  Hardly conclusive, but
you can see a pattern emerging.

You are right in suggesting that there are other answers, but what you
miss is that they are not very likely to work.  The field of operating
systems is well explored, in fact, I challenge you to name 5 things that
Linux has done which have not been done before.  The point being that
the graph of choices is well pruned.  Feel free to ignore the pruning,
there is always a chance that the old farts have pruned off a fruitful
branch, or times have changed soas to invalidate the reasoning; just be
warned that the chances are low.

What I'm trying to do is avoid having Linux go down some paths that I 
have seen other people go down because those paths have *all* resulted
in a kernel that none of us would want.  You can assert all you like 
that you'll not make those mistakes, but having seen those same assertions
a half a dozen times before from a half a dozen different OS efforts, all
of which were staffed with talented and careful people, you'll forgive
my skepticism.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

> What I'm trying to do is avoid having Linux go down some paths that I 
> have seen other people go down because those paths have *all* resulted
> in a kernel that none of us would want.  You can assert all you like 
> that you'll not make those mistakes, but having seen those same assertions
> a half a dozen times before from a half a dozen different OS efforts, all
> of which were staffed with talented and careful people, you'll forgive
> my skepticism.

You're right to point out the pitfalls that others have found in such paths
before - it's informative, and it's interesting. Personally I think I'd rather 
walk down that path, trying to avoid known pitfalls, than try your new
path, but I'll be delighted to see how your concept works out in practice. 
Good luck to you.

I still think (see my previous email) that we're actually heading to more or
less the same place from different directions, which was actually my main
point.

Time will tell ;-)

Martin.

Re: SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

> I still think (see my previous email) that we're actually heading to more or
> less the same place from different directions, which was actually my main
> point.

Oh, I agree with that.  There is no doubt of that, I thought that was fairly
apparent.  We aren't arguing about "what" we are arguing about "how".  You
are saying "I can take the path explored before and do it better" and I'm
saying "Maybe, but extremely unlikely given history.  It's far more likely
that you'll repeat history by ignoring it, a time honored tradition, albeit
ill-advised."
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[Davide Libenzi]

On Wed, 5 Dec 2001, Larry McVoy wrote:

> > I still think (see my previous email) that we're actually heading to more or
> > less the same place from different directions, which was actually my main
> > point.
>
> Oh, I agree with that.  There is no doubt of that, I thought that was fairly
> apparent.  We aren't arguing about "what" we are arguing about "how".  You
> are saying "I can take the path explored before and do it better" and I'm
> saying "Maybe, but extremely unlikely given history.  It's far more likely
> that you'll repeat history by ignoring it, a time honored tradition, albeit
> ill-advised."

Larry, I'd like to remember You that science progress has been made by
people that forced common knowledge, starting from Platone spherical earth
up to Einstain theory of relativity.
A lot of failures sure, but a single success is enough to pay out.
The _main_ difference with the two examples is that earth has always been
spherical and sub-atomic mechanics is always been in that way, while
technology progress could make someone else to succeed where you failed
time ago.




- Davide

Re: SMP/cc Cluster description [was Linux/Pro]

[Larry McVoy]

On Wed, Dec 05, 2001 at 02:13:04PM -0800, Davide Libenzi wrote:
> On Wed, 5 Dec 2001, Larry McVoy wrote:
> > Oh, I agree with that.  There is no doubt of that, I thought that was fairly
> > apparent.  We aren't arguing about "what" we are arguing about "how".  You
> > are saying "I can take the path explored before and do it better" and I'm
> > saying "Maybe, but extremely unlikely given history.  It's far more likely
> > that you'll repeat history by ignoring it, a time honored tradition, albeit
> > ill-advised."
> 
> Larry, I'd like to remember You that science progress has been made by
> people that forced common knowledge, starting from Platone spherical earth
> up to Einstain theory of relativity.
> A lot of failures sure, but a single success is enough to pay out.

How about you go read Einstein's history and show me where he made
discoveries by retracing paths which had been shown time and again to
be fruitless?  You'll look for quite a while.

Great leaps forward typically come from a fresh approach, not from 
retrying the same tired out ideas which have failed in the past.

On the other hand, as an old boss pointed out to me once, "don't worry
about the people wasting their time doing it wrong, it keeps them out of
your way".  So maybe I should just bow out of this part of the discussion.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: SMP/cc Cluster description [was Linux/Pro]

[Davide Libenzi]

On Wed, 5 Dec 2001, Larry McVoy wrote:

> On Wed, Dec 05, 2001 at 02:13:04PM -0800, Davide Libenzi wrote:
> > On Wed, 5 Dec 2001, Larry McVoy wrote:
> > > Oh, I agree with that.  There is no doubt of that, I thought that was fairly
> > > apparent.  We aren't arguing about "what" we are arguing about "how".  You
> > > are saying "I can take the path explored before and do it better" and I'm
> > > saying "Maybe, but extremely unlikely given history.  It's far more likely
> > > that you'll repeat history by ignoring it, a time honored tradition, albeit
> > > ill-advised."
> >
> > Larry, I'd like to remember You that science progress has been made by
> > people that forced common knowledge, starting from Platone spherical earth
> > up to Einstain theory of relativity.
> > A lot of failures sure, but a single success is enough to pay out.
>
> How about you go read Einstein's history and show me where he made
> discoveries by retracing paths which had been shown time and again to
> be fruitless?  You'll look for quite a while.

You're a funny guy Larry, suggesting other people to go for reading
something that, until now, I gave for sure that You've read before
suggesting someone else to read.
At that time there was a bunch of Phisics gurus saying that conventional
law of mechanins would apply even for sub-atomical particles.
Thanks god Albert decided to prove that maybe these guys where wrong.




- Davide

Re: SMP/cc Cluster description [was Linux/Pro]

[Henning Schmiedehausen]

On Wed, 2001-12-05 at 22:14, Martin J. Bligh wrote:
> > We don't agree on any of these points.  Scaling to a 16 way SMP pretty much 
> > ruins the source base, even when it is done by very careful people.
> 
> I'd say that the normal current limit on SMP machines is 8 way.
> But you're right, we don't agree.  Time will tell who was right.
> When I say I'm interested in 16 way scalability, I'm not talking about
> SMP, so perhaps we're talking at slightly cross purposes.

Well I do remember those Sequent Symmetry machines from university which
scaled to 24 processors and more. If I remember correctly they ran 16x
386 and 8x 486 in a single box (under Dynix?)

Wasn't too much interested in that then. I was to busy toying with my
Amiga. ;-)

	Regards
		Henning
 
-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: SMP/cc Cluster description [was Linux/Pro]

[David Lang]

one other thing to keep in mind on this entire discussion is that there
has not been a big incentive for the chip manufacturers to make a good
locking primitive (after all if 99% of the market is UP boxes why design
in extra stuff for the SMP modes)

This is changing as SMP boxes become mor popular.

has anyone pointed out this problem to the CPU vendors? it may be that
they would be interested in putting in some extra locking operations in a
future chip (for example I can easily see AMD implementing a 1
bit-per-lock set of locks that are arbatrated directly with the CPUs and
the chipset similar to the way cache coherancy works now rather then
the current model of useing a cacheline of memory that has to be moved
back and forth and then checked for status)

the big drawback would be that it would be non-standard commands which
each vendor would do differently, but the potential of implementing a
significant number of locks at the hardware level rather then just in
software is a very interesting thought.

David Lang



 On 6 Dec
2001, Henning Schmiedehausen wrote:

> Date: 06 Dec 2001 10:50:43 +0100
> From: Henning Schmiedehausen <hps@intermeta.de>
> To: Martin J. Bligh <Martin.Bligh@us.ibm.com>
> Cc: Larry McVoy <lm@bitmover.com>, Rik van Riel <riel@conectiva.com.br>,
>      Lars Brinkhoff <lars.spam@nocrew.org>,
>      Alan Cox <alan@lxorguk.ukuu.org.uk>, linux-kernel@vger.kernel.org
> Subject: Re: SMP/cc Cluster description [was Linux/Pro]
>
> On Wed, 2001-12-05 at 22:14, Martin J. Bligh wrote:
> > > We don't agree on any of these points.  Scaling to a 16 way SMP pretty much
> > > ruins the source base, even when it is done by very careful people.
> >
> > I'd say that the normal current limit on SMP machines is 8 way.
> > But you're right, we don't agree.  Time will tell who was right.
> > When I say I'm interested in 16 way scalability, I'm not talking about
> > SMP, so perhaps we're talking at slightly cross purposes.
>
> Well I do remember those Sequent Symmetry machines from university which
> scaled to 24 processors and more. If I remember correctly they ran 16x
> 386 and 8x 486 in a single box (under Dynix?)
>
> Wasn't too much interested in that then. I was to busy toying with my
> Amiga. ;-)
>
> 	Regards
> 		Henning
>
> --
> Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
> INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de
>
> Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
> D-91054 Buckenhof     Fax.: 09131 / 50654-20
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: SMP/cc Cluster description [was Linux/Pro]

[Martin J. Bligh]

>> > We don't agree on any of these points.  Scaling to a 16 way SMP pretty much 
>> > ruins the source base, even when it is done by very careful people.
>> 
>> I'd say that the normal current limit on SMP machines is 8 way.
>> But you're right, we don't agree.  Time will tell who was right.
>> When I say I'm interested in 16 way scalability, I'm not talking about
>> SMP, so perhaps we're talking at slightly cross purposes.
> 
> Well I do remember those Sequent Symmetry machines from university which
> scaled to 24 processors and more. If I remember correctly they ran 16x
> 386 and 8x 486 in a single box (under Dynix?)

Actually they'd scale to about 30 Pentiums. But then again, note that I said
normal and current. Our Symmetry boxes fit neither of those criteria ;-)

M.

Re: SMP/cc Cluster description [was Linux/Pro]

[Gerrit Huizenga]

In message <1007632244.24677.1.camel@forge>, > : Henning Schmiedehausen writes:
> On Wed, 2001-12-05 at 22:14, Martin J. Bligh wrote:
> > > We don't agree on any of these points.  Scaling to a 16 way SMP pretty
> much
> > > ruins the source base, even when it is done by very careful people.
> >
> > I'd say that the normal current limit on SMP machines is 8 way.
> > But you're right, we don't agree.  Time will tell who was right.
> > When I say I'm interested in 16 way scalability, I'm not talking about
> > SMP, so perhaps we're talking at slightly cross purposes.
> 
> Well I do remember those Sequent Symmetry machines from university which
> scaled to 24 processors and more. If I remember correctly they ran 16x
> 386 and 8x 486 in a single box (under Dynix?)
> 
> Wasn't too much interested in that then. I was to busy toying with my
> Amiga. ;-)
> 
>      Regards
>           Henning

Actually, up to 30 cpus, mixing and matching pairs of 12 or 16 MHz 80386,
25 or 50 MHz 80486, or 60 MHz Pentiums on the same SMP bus (known as
the Symmetry 2000 series).  

NUMA-Q stuff went to 64 processors - with quad building blocks (4
identical processors) where you could mix and match Pentium Pro, Xeon,
Pentium III quads at various clock rates.

The internals of the OS looked a lot like any standard OS, with a few
pieces of secret sauce (e.g. the read-copy-update stuff - see
 sf.net/projects/lse, multiprocessor counters, etc.)  Code complexity
was not orders of magnitude higher to support these high end machines
with near linear scalability.  It had some areas that were more complex,
but you train your staff on the complexitities, and it becomes standard
practice.  Just like Linux is enormously more complex than the first DOS
OS's, DYNIX/ptx is somewhat more complex than Linux.  But is Linux already
so complex that no one can contribute?  I don't think so.  Could DOS
engineers contribute to Linux?  I think they have - with some updated
training and knowledge about the additional complexities.

I really don't think Larry's fears can only be solved by another new
OS rewrite.  In some cases, just awareness of how to use the next
level of complex technology is enough to evolve the programmer to the
next stage.  It is happening all the time.  Doesn't mean that a fresh
approach is bad, either.  But the premise that engineers will never be
smart enough to program for high count SMP is rediculous.  The level
of complexity just isn't that much higher than where we are today.

gerrit

Re: SMP/cc Cluster description [was Linux/Pro]

[Nigel Gamble]

On Wed, 5 Dec 2001, Larry McVoy wrote:
> Seriously, I went through this at SGI, that's exactly what they did, and it
> was a huge mistake and it never worked.

In what sense do you think it "never worked"?  It worked for a huge
range of systems and applications.

But then, you left SGI before the process was really finished didn't
you?  (IRIX 6.5 was the first release that had all the pieces fully in
place.)

Nigel Gamble                                    nigel@nrg.org
Mountain View, CA, USA.                         http://www.nrg.org/

Re: SMP/cc Cluster description [was Linux/Pro]

[Alan Cox]

> If I give you 16 SMP systems, each with 4 processors and a gigabit
> ethernet card, and connect those ethers through a switch, would that
> be sufficient hardware?

Take a 16 CPU numa box thats really 4x4 + numa glue and run it as if it
was the 4 processors, 4 nodes with gige, only allowing for extra operations
"take access to remote page" "release access to remote page"

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Pavel Machek]

Hi!

> > Premise 3: it is far easier to take a bunch of operating system images 
> >    and make them share the parts they need to share (i.e., the page 
> >    cache), than to take a single image and pry it apart so that it 
> >    runs well on N processors. 
> 
> Of course it's easier. But it seems like you're left with much more work to 
> reiterate in each application you write to run on this thing. Do you want to 
> do the work once in the kernel, or repeatedly in each application? I'd say
> it's a damned sight easier to make an application work well on one big
> machine than on a cluster.

With mosix, it is exactly as hard. You just run it. I can do that today.
Larry's ideas should look same w.r.t. user.. Or maybe you'll see 128x
boot messages, but that's it.

								Pavel 
-- 
Philips Velo 1: 1"x4"x8", 300gram, 60, 12MB, 40bogomips, linux, mutt,
details at http://atrey.karlin.mff.cuni.cz/~pavel/velo/index.html.

Your patch for CS432x sound driver

[szonyi calin]

Hi
My box is: Cyrix 486, 12Meg RAM, CS4239 ISA sound
card,
no pci controler
I tested your patch both with kernel 2.4.14 +
 preemptible patches + ide patches , and with
kernel 2.4.16 clean.
The patch applied cleanly on both kernels.
The play command is working like it was working 
without this patch.
But when I was running mtv (http://www.mpegtv.com) 
the only thing I can hear is noise.
Whithout your patch mtv is playing the sound with
interrupts.
I didn't done extended testing because i don't have
 sound files on my computer (just movies) but if you
 want something specific tested please let me know 
and i'll try to help
Bye

=====
,-----.
                       ," ^   ^ ",
                       |  @   @  |
             ----OOOO---------------OOOO----

___________________________________________________________
Nokia 5510 Drôle de look... et quel son !
Cliquez sur http://fr.promotions.yahoo.com/nokia/ 
Découvrez-le et tentez votre chance pour en gagner un ! 
Fin du concours le 16 décembre.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> > Wasn't it you that were saying that Linux will never scale with more than
> > 2 CPUs ?
> 
> No, that wasn't me.  I said it shouldn't scale beyond 4 cpus.  I'd be pretty
> lame if I said it couldn't scale with more than 2.  Should != could.

Question: What happens when people stick 8 threads of execution on a die with
a single L2 cache ?

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Gérard Roudier]

On Sat, 1 Dec 2001, Alan Cox wrote:

> > > Wasn't it you that were saying that Linux will never scale with more than
> > > 2 CPUs ?
> >
> > No, that wasn't me.  I said it shouldn't scale beyond 4 cpus.  I'd be pretty
> > lame if I said it couldn't scale with more than 2.  Should != could.
>
> Question: What happens when people stick 8 threads of execution on a die with
> a single L2 cache ?

As long as we will not have clean asynchronous mechanisms available from
user land, some applications will have to use more threads of execution
than needed, even with programmers that aren't thread-maniac.

Response to your question: If the problem is to optimize IOs against 8
slow devices using synchronous IO APIs , you will get far better
performances. :-)

  Gérard.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Davide Libenzi]

On Sat, 1 Dec 2001, Alan Cox wrote:

> > > Wasn't it you that were saying that Linux will never scale with more than
> > > 2 CPUs ?
> >
> > No, that wasn't me.  I said it shouldn't scale beyond 4 cpus.  I'd be pretty
> > lame if I said it couldn't scale with more than 2.  Should != could.
>
> Question: What happens when people stick 8 threads of execution on a die with
> a single L2 cache ?

Or for example the new HP PARISC design ( Mako ) with two on-die cores,
3-4 Mb L1 I-Cache per core, 3-4 Mb L1 D-Cache per core and up to 32 Mb
external L2 ( with on-die tagging ).
Anyway it's still better 8 on-die threads of execution sharing an L2
instead of 8 CPU with 1 thread of execution shring through the external
bus.




- Davide

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin Dalecki]

Alan Cox wrote:
> 
> > > Wasn't it you that were saying that Linux will never scale with more than
> > > 2 CPUs ?
> >
> > No, that wasn't me.  I said it shouldn't scale beyond 4 cpus.  I'd be pretty
> > lame if I said it couldn't scale with more than 2.  Should != could.
> 
> Question: What happens when people stick 8 threads of execution on a die with
> a single L2 cache ?

That had been already researched. Gogin bejoind 2 threads on a single
CPU
engine doesn't give you very much... The first step is giving about 25%
the second only about 5%. There are papers in the IBM research magazine
on
this topic in context of the PowerPC.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> > Question: What happens when people stick 8 threads of execution on a die with
> > a single L2 cache ?
> 
> That had been already researched. Gogin bejoind 2 threads on a single
> CPU
> engine doesn't give you very much... The first step is giving about 25%
> the second only about 5%. There are papers in the IBM research magazine
> on

The IBM papers make certain architectural assumptions. With some of the
tiny modern CPU cores its going to perfectly viable to put 4 or 8 of them
on one die. At that point cccluster still has to have cluster nodes scaling
to 8 way

Re: Linux/Pro [was Re: Coding style - a non-issue]

[jeff millar]

----- Original Message -----
From: "Alan Cox" <alan@lxorguk.ukuu.org.uk>
To: <dalecki@evision.ag>
Cc: "Alan Cox" <alan@lxorguk.ukuu.org.uk>; "Larry McVoy" <lm@bitmover.com>;
"Davide Libenzi" <davidel@xmailserver.org>; "Andrew Morton"
<akpm@zip.com.au>; "Daniel Phillips" <phillips@bonn-fries.net>; "Henning
Schmiedehausen" <hps@intermeta.de>; "Jeff Garzik"
<jgarzik@mandrakesoft.com>; <linux-kernel@vger.kernel.org>
Sent: Sunday, December 02, 2001 11:42 AM
Subject: Re: Linux/Pro [was Re: Coding style - a non-issue]


> > > Question: What happens when people stick 8 threads of execution on a
die with
> > > a single L2 cache ?
> >
> > That had been already researched. Gogin bejoind 2 threads on a single
> > CPU
> > engine doesn't give you very much... The first step is giving about 25%
> > the second only about 5%. There are papers in the IBM research magazine
> > on
>
> The IBM papers make certain architectural assumptions. With some of the
> tiny modern CPU cores its going to perfectly viable to put 4 or 8 of them
> on one die. At that point cccluster still has to have cluster nodes
scaling
> to 8 way

Semiconductor technology will push this way because it's no longer possible
to propagate a signal across the die in one clock cycle.  This means
pipeline
interlocking becomes vastly more complicated.  The simple solution puts
several CPU cores on the die, each able to interlock in one clock but
sharing
memory over several clocks.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Andrew Morton]

Davide Libenzi wrote:
> 
> On Fri, 30 Nov 2001, Larry McVoy wrote:
> 
> [ A lot of stuff Pro-Sun ]
> 
> Wait a minute.

umm..   Let's try to keep moving forward here.

Larry appears to be referring to the proposal sometimes
known as ccClusters.  I'd ask him a couple of followup questions:

Is there any precedent for the ccCluster idea, which would
increase confidence that it'll actually work?

Will it run well on existing hardware, or are changes needed?

You're assuming that our current development processes are
sufficient for development of a great 1-to-4-way kernel, and
that the biggest force against that is the introduction of
fine-grained locking.   Are you sure about this?  Do you see
ways in which the uniprocessor team can improve?

My take is that there's a sort of centralism in the kernel where
key people get atracted into mm/*.c, fs/*.c, net/most_everything
and kernel/*.c while other great wilderness of the tree (with
honourable exceptions) get moldier.  How to address that?

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> sufficient for development of a great 1-to-4-way kernel, and
> that the biggest force against that is the introduction of
> fine-grained locking.   Are you sure about this?  Do you see
> ways in which the uniprocessor team can improve?

ccCluster seems a sane idea to me. I don't by Larry's software engineering
thesis but ccCluster makes sense simply because when you want an efficient
system in computing you get it by not pretending one thing is another.
SMP works best when the processors are not doing anything that interacts
with another CPU.

> key people get atracted into mm/*.c, fs/*.c, net/most_everything
> and kernel/*.c while other great wilderness of the tree (with
> honourable exceptions) get moldier.  How to address that?

Actually there are lots of people who work on the driver code nowdays
notably the janitors. The biggest problem there IMHO is that when it comes
to driver code Linus has no taste, so he keeps accepting driver patches
which IMHO are firmly at the hamburger end of "taste"

Another thing for 2.5 is going to be to weed out the unused and unmaintained
drivers, and either someone fixes them or they go down the comsic toilet and
we pull the flush handle before 2.6 comes out.

Thankfully the scsi layer breakage is going to help no end in the area of 
clockwork 8 bit scsi controllers, which is major culprit number 1. number 2
is probably the audio which is hopefully going to go away with ALSA based
code.

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Victor Yodaiken]

On Sat, Dec 01, 2001 at 10:05:55AM +0000, Alan Cox wrote:
> > sufficient for development of a great 1-to-4-way kernel, and
> > that the biggest force against that is the introduction of
> > fine-grained locking.   Are you sure about this?  Do you see
> > ways in which the uniprocessor team can improve?
> 
> ccCluster seems a sane idea to me. I don't by Larry's software engineering
> thesis but ccCluster makes sense simply because when you want an efficient
> system in computing you get it by not pretending one thing is another.
> SMP works best when the processors are not doing anything that interacts
> with another CPU.

Careful Alan. That sounds suspiciously like a "design principle", and
true macho Linux developers don't need no theoretical stuff like that.
They just slop that code together and see what explodes - pulling their
alchemists hats over their eyes for protection.

> 
> > key people get atracted into mm/*.c, fs/*.c, net/most_everything
> > and kernel/*.c while other great wilderness of the tree (with
> > honourable exceptions) get moldier.  How to address that?
> 
> Actually there are lots of people who work on the driver code nowdays
> notably the janitors. The biggest problem there IMHO is that when it comes
> to driver code Linus has no taste, so he keeps accepting driver patches
> which IMHO are firmly at the hamburger end of "taste"

"Taste" ? Now you want aesthetics as well as theory. I'm horrified.

Technical content: does anyone know the max spinlock depth in Linux 2.5
?

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Martin Dalecki]

Alan Cox wrote:
> Another thing for 2.5 is going to be to weed out the unused and unmaintained
> drivers, and either someone fixes them or they go down the comsic toilet and
> we pull the flush handle before 2.6 comes out.
> 
> Thankfully the scsi layer breakage is going to help no end in the area of
> clockwork 8 bit scsi controllers, which is major culprit number 1. number 2
> is probably the audio which is hopefully going to go away with ALSA based
> code.

Please consider the following wipe out candidates as well:

1. ftape <- it should really really go away
2. proprietary CD-ROM
3. xd.c (ridiculous isn't it?)
4. old ide driver...
5. old directory reading syscalls.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> Please consider the following wipe out candidates as well:
> 
> 2. proprietary CD-ROM
> 3. xd.c (ridiculous isn't it?)
> 4. old ide driver...

I know people using all 3 of those, while bugs in some of the old scsi 8bit
drivers went unnoticed for a year.

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Oliver Xymoron]

On Sun, 2 Dec 2001, Alan Cox wrote:

> > Please consider the following wipe out candidates as well:
> >
> > 2. proprietary CD-ROM
> > 3. xd.c (ridiculous isn't it?)
> > 4. old ide driver...
>
> I know people using all 3 of those, while bugs in some of the old scsi 8bit
> drivers went unnoticed for a year.

We need a 'prompt for unmaintained drivers' trailing-edge option in the
build process so people will know when something's been orphaned and pick
it up.

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Jeff Garzik]

Oliver Xymoron wrote:
> 
> On Sun, 2 Dec 2001, Alan Cox wrote:
> 
> > > Please consider the following wipe out candidates as well:
> > >
> > > 2. proprietary CD-ROM
> > > 3. xd.c (ridiculous isn't it?)
> > > 4. old ide driver...
> >
> > I know people using all 3 of those, while bugs in some of the old scsi 8bit
> > drivers went unnoticed for a year.
> 
> We need a 'prompt for unmaintained drivers' trailing-edge option in the
> build process so people will know when something's been orphaned and pick
> it up.

There's already CONFIG_OBSOLETE...

-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Oliver Xymoron]

On Sun, 2 Dec 2001, Jeff Garzik wrote:

> Oliver Xymoron wrote:
> >
> > On Sun, 2 Dec 2001, Alan Cox wrote:
> >
> > > > Please consider the following wipe out candidates as well:
> > > >
> > > > 2. proprietary CD-ROM
> > > > 3. xd.c (ridiculous isn't it?)
> > > > 4. old ide driver...
> > >
> > > I know people using all 3 of those, while bugs in some of the old scsi 8bit
> > > drivers went unnoticed for a year.
> >
> > We need a 'prompt for unmaintained drivers' trailing-edge option in the
> > build process so people will know when something's been orphaned and pick
> > it up.
>
> There's already CONFIG_OBSOLETE...

And it's practically obsolete itself, outside of the ARM directory. What
I'm proposing is something in the Code Maturity menu that's analogous to
EXPERIMENTAL along with a big (UNMAINTAINED) marker next to unmaintained
drivers. Obsolete and unmaintained and deprecated all mean slightly
different things, by the way. So the config option would probably say
'Show obsolete, unmaintained, or deprecated items?' and mark each item
appropriately. Anything that no one made a fuss about by 2.7 would be
candidates for removal.

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Jeff Garzik]

Oliver Xymoron wrote:
> 
> On Sun, 2 Dec 2001, Jeff Garzik wrote:
> 
> > Oliver Xymoron wrote:
> > >
> > > On Sun, 2 Dec 2001, Alan Cox wrote:
> > >
> > > > > Please consider the following wipe out candidates as well:
> > > > >
> > > > > 2. proprietary CD-ROM
> > > > > 3. xd.c (ridiculous isn't it?)
> > > > > 4. old ide driver...
> > > >
> > > > I know people using all 3 of those, while bugs in some of the old scsi 8bit
> > > > drivers went unnoticed for a year.
> > >
> > > We need a 'prompt for unmaintained drivers' trailing-edge option in the
> > > build process so people will know when something's been orphaned and pick
> > > it up.
> >
> > There's already CONFIG_OBSOLETE...
> 
> And it's practically obsolete itself, outside of the ARM directory. What
> I'm proposing is something in the Code Maturity menu that's analogous to
> EXPERIMENTAL along with a big (UNMAINTAINED) marker next to unmaintained
> drivers. Obsolete and unmaintained and deprecated all mean slightly
> different things, by the way. So the config option would probably say
> 'Show obsolete, unmaintained, or deprecated items?' and mark each item
> appropriately. Anything that no one made a fuss about by 2.7 would be
> candidates for removal.

The idea behind CONFIG_OBSOLETE is supposed to be that it does not
actually appear as a Y/N option.  You enclose a Config.in option with
that, and it disappears 

But I'm all for removing old stuff.  There is no reason to keep
something that flat out doesn't work and hasn't for a long long time... 
if somebody wants to pick it up they can grab linux-2.2 or linux-2.0
from any FTP mirror.

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Oliver Xymoron]

On Sun, 2 Dec 2001, Jeff Garzik wrote:

> Oliver Xymoron wrote:
> >
> > And it's practically obsolete itself, outside of the ARM directory. What
> > I'm proposing is something in the Code Maturity menu that's analogous to
> > EXPERIMENTAL along with a big (UNMAINTAINED) marker next to unmaintained
> > drivers. Obsolete and unmaintained and deprecated all mean slightly
> > different things, by the way. So the config option would probably say
> > 'Show obsolete, unmaintained, or deprecated items?' and mark each item
> > appropriately. Anything that no one made a fuss about by 2.7 would be
> > candidates for removal.
>
> The idea behind CONFIG_OBSOLETE is supposed to be that it does not
> actually appear as a Y/N option.  You enclose a Config.in option with
> that, and it disappears

Which works for stuff that is really known broken. It doesn't work for
stuff that you'd like to get rid of but you suspect people may still be
using (sbpcd) or stuff that you want to phase out (initrd).

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Linux/Pro [was Re: Coding style - a non-issue]

[[MOc]cda*mirabilos]

> But I'm all for removing old stuff.  There is no reason to keep
> something that flat out doesn't work and hasn't for a long long
time...
> if somebody wants to pick it up they can grab linux-2.2 or linux-2.0
> from any FTP mirror.

By the way, what happened to xiafs?
Back to 2.0.33 it even didn't work (complaints after newfs).

Just an interest...
Thorsten

Re: Linux/Pro [was Re: Coding style - a non-issue]

[H. Peter Anvin]

Followup to:  <000b01c17b68$2ff846e0$30d8fea9@ecce>
By author:    "[MOc]cda*mirabilos" <mirabilos@netcologne.de>
In newsgroup: linux.dev.kernel
> 
> By the way, what happened to xiafs?
> Back to 2.0.33 it even didn't work (complaints after newfs).
> 

It got ripped out because the vfs changed and noone ported it.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> And it's practically obsolete itself, outside of the ARM directory. What

Thats just history. Doesn't mean it won't do the job. Probably when we get
to say 2.5.4 or so someone should do a build all as modules and anything
that doesn't build gets an obsolete tag until someone fixes it.

Anything not fixed for 2.6 will then be nicely labelled

RE: Linux/Pro [was Re: Coding style - a non-issue]

[M. Edward Borasky]

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org
> [mailto:linux-kernel-owner@vger.kernel.org]On Behalf Of Alan Cox
> Sent: Saturday, December 01, 2001 2:06 AM
> To: Andrew Morton
> Cc: Davide Libenzi; Larry McVoy; Daniel Phillips; Henning
> Schmiedehausen; Jeff Garzik; linux-kernel@vger.kernel.org
> Subject: Re: Linux/Pro [was Re: Coding style - a non-issue]

> Another thing for 2.5 is going to be to weed out the unused and
> unmaintained
> drivers, and either someone fixes them or they go down the comsic
> toilet and
> we pull the flush handle before 2.6 comes out.
>
> Thankfully the scsi layer breakage is going to help no end in the area of
> clockwork 8 bit scsi controllers, which is major culprit number
> 1. number 2
> is probably the audio which is hopefully going to go away with ALSA based
> code.

I am *completely* underwhelmed by ALSA, *especially* in the areas of
usability and documentation! I have an M-Audio Delta 66 sound card and I was
unable to get ALSA to work with it. At the time I attempted this, I did not
have the free time to do c-code-level integration work; I needed something
that was plug-and-play with usable documentation. I ended up buying the
closed-source OSS/Linux driver, which at least installs and boots. Their
documentation isn't much better, and I finally was forced to dual-boot
Windows 2000 to get a usable audio tool set for this device.

My point here is that just because a composer is *capable* of doing
integration work and building or repairing tools (and I am) does *not* mean
he (or she :-) has either the time or the willingness to do so (and I
don't).
--
Take Your Trading to the Next Level!
M. Edward Borasky, Meta-Trading Coach

znmeb@borasky-research.net
http://www.meta-trading-coach.com
http://groups.yahoo.com/group/meta-trading-coach

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Horst von Brand]

"M. Edward Borasky" <znmeb@aracnet.com> said:

[...]

> My point here is that just because a composer is *capable* of doing
> integration work and building or repairing tools (and I am) does *not* mean
> he (or she :-) has either the time or the willingness to do so (and I
> don't).

So band together with some others with your same problem, and pay somebody
to fix it. What you saved on propietary OS lease should make up for it.
Amply.

Oh wait, you are just a troll, right?
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

RE: Linux/Pro [was Re: Coding style - a non-issue]

[M. Edward Borasky]

> -----Original Message-----
> From: Horst von Brand [mailto:vonbrand@sleipnir.valparaiso.cl]
> Sent: Sunday, December 02, 2001 7:23 PM
> To: M. Edward Borasky
> Cc: linux-kernel@vger.kernel.org
> Subject: Re: Linux/Pro [was Re: Coding style - a non-issue]
>
>
> "M. Edward Borasky" <znmeb@aracnet.com> said:
>
> [...]
>
> > My point here is that just because a composer is *capable* of doing
> > integration work and building or repairing tools (and I am)
> does *not* mean
> > he (or she :-) has either the time or the willingness to do so (and I
> > don't).
>
> So band together with some others with your same problem, and pay somebody
> to fix it. What you saved on propietary OS lease should make up for it.
> Amply.

What I spent on Windows 2000 is $300 US. This converted my $400
top-of-the-line sound card from a useless space-taker on my desk to a
functioning musical device. As for banding together with some others, well,
they are even *more* frustrated than I am, because most of them are *purely*
musicians and *can't* program. Nor do they have the money to spend on
programmers. I'm on a number of musical mailing lists, and their
overwhelming complaint is that they spend most of their time being system
administrators rather than musicians/composers. And these are people using
*commercial* tools -- some *quite* expensive -- on Windows and Macs.

> Oh wait, you are just a troll, right?

Not really ... if you'd like I can be, though. Eventually, when I run out of
other projects, I'll sit down and force ALSA to work with my sound card if
someone hasn't done it already. Of course, now that I have the sound card
running and Windows 2000, why would I need to? So much of Linux is
plug-and-play right now, at least the Red Hat Linux that I'm using. I bought
a sound card unsupported by Red Hat because I knew of two drivers for it --
OSS/Linux and ALSA. I tried ALSA first and gave up on it after a week of
agony on the ALSA mailing list. Then I bought OSS/Linux, which installed
fine but didn't generate any sound. When I sent e-mail to the support desk,
I got a very fast response -- RTFM. The FM in this case consists of a single
page ASCII document which is less than helpful.

What I'm trying to establish here is that if ALSA is to become the
main-stream Linux sound driver set, it's going to need to support -- *fully*
support -- the top-of-the-line sound cards like my M-Audio Delta 66. It
isn't enough to just support the Envy chip inside -- it has to support the
whole card with interfaces to all the sound tools that come with KDE and
Gnome! It has to install flawlessly, boot flawlessly and understand
everything that is in the card. I haven't checked recently to see if the
ALSA situation has changed any -- too busy making music on my Windows
machine :-).
--
Take Your Trading to the Next Level!
M. Edward Borasky, Meta-Trading Coach

znmeb@borasky-research.net
http://www.meta-trading-coach.com
http://groups.yahoo.com/group/meta-trading-coach

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> What I'm trying to establish here is that if ALSA is to become the
> main-stream Linux sound driver set, it's going to need to support -- *fully*
> support -- the top-of-the-line sound cards like my M-Audio Delta 66. It

Not really. The number of people who actually care about such cards is close
to nil. What matters is that the API can cleanly express what the Delta66
can do, and that you can write a driver for it under ALSA without hacking up
the ALSA core.

I'm happy both of those are true.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[David Weinehall]

On Tue, Dec 04, 2001 at 09:28:49AM +0000, Alan Cox wrote:
> On Mon, Dec 03, 2001 at 06:31:38AM -0800, M. Edward Borasky wrote:
> > What I'm trying to establish here is that if ALSA is to become the
> > main-stream Linux sound driver set, it's going to need to support --
> > *fully* support -- the top-of-the-line sound cards like my M-Audio
> > Delta 66.
> 
> Not really. The number of people who actually care about such cards is
> close to nil. What matters is that the API can cleanly express what
> the Delta66 can do, and that you can write a driver for it under ALSA
> without hacking up the ALSA core.

Indeed. And I'm sure the ALSA-team would be delighted and fully willing
to write a working driver, if mr Borasky donated an M-Audio Delta 66
together with full documentation to them...


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Dan Hollis]

On Tue, 4 Dec 2001, David Weinehall wrote:
> Indeed. And I'm sure the ALSA-team would be delighted and fully willing
> to write a working driver, if mr Borasky donated an M-Audio Delta 66
> together with full documentation to them...

ALSA already has a working driver...!

-Dan
-- 
[-] Omae no subete no kichi wa ore no mono da. [-]

Re: Linux/Pro [was Re: Coding style - a non-issue]

[David Weinehall]

On Tue, Dec 04, 2001 at 11:35:11AM -0800, Dan Hollis wrote:
> On Tue, 4 Dec 2001, David Weinehall wrote:
> > Indeed. And I'm sure the ALSA-team would be delighted and fully willing
> > to write a working driver, if mr Borasky donated an M-Audio Delta 66
> > together with full documentation to them...
> 
> ALSA already has a working driver...!

The point I was trying to make was just "stop complaining about lack
of drivers, contribute one or help someone else create one. I wasn't
criticizing ALSA, rather the opposite. Now, if I could just find
someone willing to program a driver for that old 8-bit, totally sucky,
IBM ACPA/A I have (the only MCA sound adapter I have managed to get
hold of...)


/David
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Dan Hollis]

On Tue, 4 Dec 2001, Alan Cox wrote:
> > What I'm trying to establish here is that if ALSA is to become the
> > main-stream Linux sound driver set, it's going to need to support -- *fully*
> > support -- the top-of-the-line sound cards like my M-Audio Delta 66. It
> Not really. The number of people who actually care about such cards is close
> to nil. What matters is that the API can cleanly express what the Delta66
> can do, and that you can write a driver for it under ALSA without hacking up
> the ALSA core.
> I'm happy both of those are true.

ALSA has supported ice1712 chipset for some time now.

BTW Delta 66 isnt top of the line -- Delta 1010 is. And ALSA supports it
too.

-Dan
-- 
[-] Omae no subete no kichi wa ore no mono da. [-]

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Pavel Machek]

Hi!

> Another thing for 2.5 is going to be to weed out the unused and unmaintained
> drivers, and either someone fixes them or they go down the comsic toilet and
> we pull the flush handle before 2.6 comes out.

Hey, I still have 8-bit isa scsi card somewhere.... Last time I fixed
that was just before 2.4 because that was when I got it... Don't flush
drivers too fast, please... If you kill drivers during 2.5, people
will not notice, and even interesting drivers will get killed. Killing
them during 2.6.2 might be better.
								Pavel
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Alan Cox]

> that was just before 2.4 because that was when I got it... Don't flush
> drivers too fast, please... If you kill drivers during 2.5, people
> will not notice, and even interesting drivers will get killed. Killing
> them during 2.6.2 might be better.

They need to die before 2.6. I'm all for leaving the code present and the
ability to select

	Expert
		Drivers that need fixing
			Clockwork scsi controller, windup mark 2

in 2.6 so that people do fix them

Alan

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Stephan von Krawczynski]

On Fri, 30 Nov 2001 15:57:40 -0800
Larry McVoy <lm@bitmover.com> wrote:

> [...]
> Here's how you get both.  Fork the development efforts into the SMP part
> and the uniprocessor part.  The uniprocessor focus is on reliability,
> stability, performance.  The SMP part is a whole new development effort,
> which is architected in such a way as to avoid the complexity of a
> traditional SMP implementation.
> 
> The uniprocessor team owns the core architecture of the system.  The
> abstractions provided, the baseline code, etc., that's all uni.  The
> focus there is a small, fast, stable kernel.
> 
> The SMP team doesn't get to touch the core code, or at least there is 
> a very strong feedback loop against.  In private converstations, we've
> started talking about the "punch in the nose" feedback loop, which means
> while it may be necessary to touch generic code for the benefit of SMP,
> someone has to feel strongly enough about it that they well sacrifice
> their nose.

Hi Larry,

let me first tell you this: I am only stating my very personal opinion here and
am probably no guy of your experience or insight in high-tech delevopment
groups. But I saw the whole business for quite some years now, so my thinking:

Your proposal is the wrong way, because:
1) You split up "the crew". Whatever you may call "the crew" here, they all
have one thing in common: they are working on the _same_ project _and_ (very
important either) _one_ man has the final decision. If you look at _any_ other
OS developed by quite a number of completely different people you have to admit
one thing: everything was busted when they began to split up in different
"branches". That simply does not work out. I am only referring to simple human
interaction and communication here, nothing to do with the technical issues.
One of the basic reasons for the success of Linux so far is the collaborated
work of a damn lot of people on the _same_ project.

2) I cannot see the _need_ for such a "team-splitup". If you say one team (core
team) has the "last word" about the baseline code, what do you think will
happen if "necessary" code changes for the second team will be refused? Simple:
they will fork a completely new linux-tree. And this is _the_ end. If you were
to write a driver, what tree will you choose after that? I mean you are dealing
with the main reason why people like linux, here. And not: SCO Unix 3.x,
Unixware, Solaris, Minix, AT&T Unix, Xenix, HPUX, AIX, BSD, NetBSD, BSDi,
Solaris-for-Intel, make-my-day ... (all registered trademark of their
respective owners, which have all low interaction capabilities)
I don't want that, do we want that?

3) Your SMP team (you are talking of true high scaled SMP here) has a very big
problem: it has _very_ few users (compared to UP) and even less developers with
the _hardware_ you need for something like that. I mean you are not talking
about buying an Asus Board and 2 PIII here, you talk about serious, expensive
hardware here. How many people have this at home, or at work for free playing?
Well, see what I mean?

4) Warning, this is the hard stuff!
Ok, so you are fond of SUN. Well, me too. But I am not completely blind, not
yet :-) So I must tell you, if Solaris were the real big hit, then why its
Intel-Version is virtualy been eaten up on the market (the _buying_ market out
there) by linux? The last time I met a guy seriously talking about Solaris
(x86) being "better" than Linux was about three years ago. And btw, this was
_the_ last guy talking about it at all. So lets simply assume this: the Solaris
UP market is already gone, if you are talking about the _mass_ market. This
parrot is _deceased_! It is plain dead.
Now you are dealing with another problem: SUN (being kind of the last resort of
a widespread and really working commercial unix) will have a lot of work to do
in the future to keep up with the simple mass of software and application
coming in for linux, only because it is even _more_ widespread today than
Solaris has ever been. And it is _real_ cheap, and you get it _everywhere_. And
everybody owns a box on which you can use it.
This is going to get very hard for SUN, if they do not enter a stage of
complete rethinking what is going on out there.
To make that clear: _nobody_ here _fights_ against SUN or Solaris. Quite a few
of us like it very much. But this is a commercial product. It needs to be sold
to survive - and that is going to be a future problem. SUN will not survive
only building the high-power low-mass computer. CRAY did not either. So maybe
the real good choice would be this: let the good parts of Solaris (and maybe
its SMP features) migrate into linux. This does not mean they should lay off
the staff, just on contrary: these are brilliant people, let them do what they
can do best, but keep an eye on the market. We are in the state of: the market
_demands_ linux. It has already become a well-known trademark, I tend to
believe better-known than Solaris. Somehow one has the feeling they indeed know
whats coming (fs), but have not come to the final (and hard to take)
conclusion.
And to clarify: I am not taking any drugs. This is serious. I have the strong
feeling, that there is already a big player out there, that has learnt a hard
lesson: IBM - and the lesson named OS/2.
When OS/2 came out, I told the people: if they are not going to give it away
for free, they will not make it. And they didn't. Indeed I did not expect them
to learn at all (simply because big companies are mostly not quick-movers), but
they do really astonishing things nowadays. I have the strong feeling the
management is at least as brilliant as the Solaris developpers and worth every
buck, too.

But this is only my small voice, and quite possibly only few are listening, if
any ...

Regards,
Stephan

PS: Is this really a topic for a kernel-mailing-list?

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Mike Fedyk]

On Sat, Dec 01, 2001 at 02:21:57AM +0100, Stephan von Krawczynski wrote:
> _the_ last guy talking about it at all. So lets simply assume this: the Solaris
> UP market is already gone, if you are talking about the _mass_ market. This
> parrot is _deceased_! It is plain dead.

Not completely.  Many people who *need* to develop for solaris on sun
hardware will run solaris x86 at home (or maybe on a corporate laptop) to be
able to work at home and test the software there too.  I know one such
developer myself, there have to be more.

> So maybe
> the real good choice would be this: let the good parts of Solaris (and maybe
> its SMP features) migrate into linux. 

Before 2.3 and 2.4 there probably would've been much more contention against
something like this.  Even now with large SMP scalability goals, I think it
would be hard to get something like this to be accepted into Linux.

mf

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Kai Henningsen]

mfedyk@matchmail.com (Mike Fedyk)  wrote on 30.11.01 in <20011130210155.B489@mikef-linux.matchmail.com>:

> On Sat, Dec 01, 2001 at 02:21:57AM +0100, Stephan von Krawczynski wrote:

> > So maybe
> > the real good choice would be this: let the good parts of Solaris (and
> > maybe its SMP features) migrate into linux.
>
> Before 2.3 and 2.4 there probably would've been much more contention against
> something like this.  Even now with large SMP scalability goals, I think it
> would be hard to get something like this to be accepted into Linux.

It works for SGI/Irix/XFS, why would it not work for SUN/Solaris/whatever?

MfG Kai

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Mike Fedyk]

On Sun, Dec 02, 2001 at 12:05:00AM +0200, Kai Henningsen wrote:
> mfedyk@matchmail.com (Mike Fedyk)  wrote on 30.11.01 in <20011130210155.B489@mikef-linux.matchmail.com>:
> 
> > On Sat, Dec 01, 2001 at 02:21:57AM +0100, Stephan von Krawczynski wrote:
> 
> > > So maybe
> > > the real good choice would be this: let the good parts of Solaris (and
> > > maybe its SMP features) migrate into linux.
> >
> > Before 2.3 and 2.4 there probably would've been much more contention against
> > something like this.  Even now with large SMP scalability goals, I think it
> > would be hard to get something like this to be accepted into Linux.
> 
> It works for SGI/Irix/XFS, why would it not work for SUN/Solaris/whatever?
>

I meant actually transplanting Solaris's SMP into Linux.  You don't see SGI
doing anything like this...

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Mark Frazer]

Stephan von Krawczynski <skraw@ithnet.com> [01/11/30 20:27]:
> 4) Warning, this is the hard stuff!
> Ok, so you are fond of SUN. Well, me too. But I am not completely blind, not
> yet :-) So I must tell you, if Solaris were the real big hit, then why its
> Intel-Version is virtualy been eaten up on the market (the _buying_ market out
> there) by linux?

I can't say for the O/S buying market.  But I do embedded (pretty large
embedded systems but embedded nonetheless) development and we walked away
from Solaris after comparing the complexity of our first network drivers.

STREAMS:  just say no.

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Larry McVoy]

On Sat, Dec 01, 2001 at 11:04:30AM -0500, Mark Frazer wrote:
> Stephan von Krawczynski <skraw@ithnet.com> [01/11/30 20:27]:
> > 4) Warning, this is the hard stuff!
> > Ok, so you are fond of SUN. Well, me too. But I am not completely blind, not
> > yet :-) So I must tell you, if Solaris were the real big hit, then why its
> > Intel-Version is virtualy been eaten up on the market (the _buying_ market out
> > there) by linux?
> 
> I can't say for the O/S buying market.  But I do embedded (pretty large
> embedded systems but embedded nonetheless) development and we walked away
> from Solaris after comparing the complexity of our first network drivers.
> 
> STREAMS:  just say no.

Amen to that.  STREAMS would be one of the strongest arguments in favor
of Linus' theory that evolution takes care of it.  STREAMS were done at
Sun by some "architects" who thought they would be better than sockets.
Linus is dead right, on this sort of issue, Linux responds quickly and 
weeds out the crap.  We still have some room for discussion on the 
design issues, but yeah, Sun blew it on this one and refused to admit it.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Linux/Pro [was Re: Coding style - a non-issue]

[Mike Fedyk]

On Fri, Nov 30, 2001 at 03:57:40PM -0800, Larry McVoy wrote:
> Here's how you get both.  Fork the development efforts into the SMP part
> and the uniprocessor part.  

Basically with linux, and enough #ifdef's you end up with both in one.  IIUC

What would be nice is UP only drivers for initial release. Write a driver
module that says "I'm made for UP and haven't been tested with SMP/HighMEM"
so if you try to compile it with those features it would break with a
helpful error message.

What would be interesting would be SMP with support for UP.  The UP only
module would be inserted into a SMP kernel, but would only work inside one
processor, and would have source compatibility with both UP ans SMP kernels.
No non-UP locking required.

Is something like this possible?

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Andrew Morton wrote:
> Larry McVoy wrote:
> > Linux isn't there yet
> > and unless the development model changes somewhat, I'll stand behind my
> > belief that it is unlikely to ever get there.
>
> I am (genuinely) interested in what changes you think are needed.

I'm very interested too, though I'll have to agree with Larry
that Linux really isn't going anywhere in particular and seems
to be making progress through sheer luck.

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Daniel Phillips]

Hi Rik,

On December 1, 2001 01:35 am, Rik van Riel wrote:
> On Fri, 30 Nov 2001, Andrew Morton wrote:
> > Larry McVoy wrote:
> > > Linux isn't there yet
> > > and unless the development model changes somewhat, I'll stand behind my
> > > belief that it is unlikely to ever get there.
> >
> > I am (genuinely) interested in what changes you think are needed.
> 
> I'm very interested too, though I'll have to agree with Larry
> that Linux really isn't going anywhere in particular and seems
> to be making progress through sheer luck.

You just reminded me of Minnesota Fats most famous quote:

  "The more I practice, the luckier I get"

--
Daniel

Re: Coding style - a non-issue

[Linus Torvalds]

On Fri, 30 Nov 2001, Rik van Riel wrote:
>
> I'm very interested too, though I'll have to agree with Larry
> that Linux really isn't going anywhere in particular and seems
> to be making progress through sheer luck.

Hey, that's not a bug, that's a FEATURE!

You know what the most complex piece of engineering known to man in the
whole solar system is?

Guess what - it's not Linux, it's not Solaris, and it's not your car.

It's you. And me.

And think about how you and me actually came about - not through any
complex design.

Right. "sheer luck".

Well, sheer luck, AND:
 - free availability and _crosspollination_ through sharing of "source
   code", although biologists call it DNA.
 - a rather unforgiving user environment, that happily replaces bad
   versions of us with better working versions and thus culls the herd
   (biologists often call this "survival of the fittest")
 - massive undirected parallel development ("trial and error")

I'm deadly serious: we humans have _never_ been able to replicate
something more complicated than what we ourselves are, yet natural
selection did it without even thinking.

Don't underestimate the power of survival of the fittest.

And don't EVER make the mistake that you can design something better than
what you get from ruthless massively parallel trial-and-error with a
feedback cycle. That's giving your intelligence _much_ too much credit.

Quite frankly, Sun is doomed. And it has nothing to do with their
engineering practices or their coding style.

		Linus

Re: Coding style - a non-issue

[Mike Castle]

On Fri, Nov 30, 2001 at 04:50:34PM -0800, Linus Torvalds wrote:
> Well, sheer luck, AND:
>  - free availability and _crosspollination_ through sharing of "source
>    code", although biologists call it DNA.
>  - a rather unforgiving user environment, that happily replaces bad
>    versions of us with better working versions and thus culls the herd
>    (biologists often call this "survival of the fittest")
>  - massive undirected parallel development ("trial and error")

Linux is one big genetic algorithms project?

mrc
-- 
     Mike Castle      dalgoda@ix.netcom.com      www.netcom.com/~dalgoda/
    We are all of us living in the shadow of Manhattan.  -- Watchmen
fatal ("You are in a maze of twisty compiler features, all different"); -- gcc

Re: Coding style - a non-issue

[Davide Libenzi]

On Fri, 30 Nov 2001, Mike Castle wrote:

> On Fri, Nov 30, 2001 at 04:50:34PM -0800, Linus Torvalds wrote:
> > Well, sheer luck, AND:
> >  - free availability and _crosspollination_ through sharing of "source
> >    code", although biologists call it DNA.
> >  - a rather unforgiving user environment, that happily replaces bad
> >    versions of us with better working versions and thus culls the herd
> >    (biologists often call this "survival of the fittest")
> >  - massive undirected parallel development ("trial and error")
>
> Linux is one big genetic algorithms project?

It is more subtle, look better inside :)



- Davide

Re: Coding style - a non-issue

[Jamie Lokier]

Mike Castle wrote:
> Linux is one big genetic algorithms project?

No but I'm sure the VM layer is :-)

-- Jamie

Re: Coding style - a non-issue

[Rik van Riel]

On Sat, 1 Dec 2001, Jamie Lokier wrote:
> Mike Castle wrote:
> > Linux is one big genetic algorithms project?
>
> No but I'm sure the VM layer is :-)

I guess we now know the reason Linus purposefully
makes sure no comment ever matches the code ;/

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Daniel Phillips]

On December 1, 2001 05:27 pm, Rik van Riel wrote:
> On Sat, 1 Dec 2001, Jamie Lokier wrote:
> > Mike Castle wrote:
> > > Linux is one big genetic algorithms project?
> >
> > No but I'm sure the VM layer is :-)
> 
> I guess we now know the reason Linus purposefully
> makes sure no comment ever matches the code ;/

2.5 is now open, I'm sure Linus will take patches to fix comment bugs now.

--
Daniel

Re: Coding style - a non-issue

[Petko Manolov]

:-)) This made my day..  May be my week. :-))

100% agree (better not play gods) and i think this is the end of
the discussion.


		Petko


Linus Torvalds wrote:

> On Fri, 30 Nov 2001, Rik van Riel wrote:
> 
>>I'm very interested too, though I'll have to agree with Larry
>>that Linux really isn't going anywhere in particular and seems
>>to be making progress through sheer luck.
>>
> 
> Hey, that's not a bug, that's a FEATURE!
> 
> You know what the most complex piece of engineering known to man in the
> whole solar system is?
> 
> Guess what - it's not Linux, it's not Solaris, and it's not your car.
> 
> It's you. And me.
> 
> And think about how you and me actually came about - not through any
> complex design.
> 
> Right. "sheer luck".
> 
> Well, sheer luck, AND:
>  - free availability and _crosspollination_ through sharing of "source
>    code", although biologists call it DNA.
>  - a rather unforgiving user environment, that happily replaces bad
>    versions of us with better working versions and thus culls the herd
>    (biologists often call this "survival of the fittest")
>  - massive undirected parallel development ("trial and error")
> 
> I'm deadly serious: we humans have _never_ been able to replicate
> something more complicated than what we ourselves are, yet natural
> selection did it without even thinking.
> 
> Don't underestimate the power of survival of the fittest.
> 
> And don't EVER make the mistake that you can design something better than
> what you get from ruthless massively parallel trial-and-error with a
> feedback cycle. That's giving your intelligence _much_ too much credit.
> 
> Quite frankly, Sun is doomed. And it has nothing to do with their
> engineering practices or their coding style.
> 
> 		Linus

Re: Coding style - a non-issue

[Tim Hockin]

> I'm deadly serious: we humans have _never_ been able to replicate
> something more complicated than what we ourselves are, yet natural
> selection did it without even thinking.

a very interesting argument, but not very pertinent - we don't have 10's of
thousands of year or even really 10's of years.  We have to use intellect
to root out the obviously bad ideas, and even more importantly the
bad-but-not-obviously-bad ideas.

> Quite frankly, Sun is doomed. And it has nothing to do with their
> engineering practices or their coding style.

I'd love to hear your thoughts on why.

Re: Coding style - a non-issue

[Linus Torvalds]

On Fri, 30 Nov 2001, Tim Hockin wrote:
>
> > I'm deadly serious: we humans have _never_ been able to replicate
> > something more complicated than what we ourselves are, yet natural
> > selection did it without even thinking.
>
> a very interesting argument, but not very pertinent - we don't have 10's of
> thousands of year or even really 10's of years.  We have to use intellect
> to root out the obviously bad ideas, and even more importantly the
> bad-but-not-obviously-bad ideas.

Directed evolution - ie evolution that has more specific goals, and faster
penalties for perceived failure, works on the scale of tens or hundreds of
years, not tens of thousands. Look at dog breeding, but look even more at
livestock breeding, where just a few decades have made a big difference.

The belief that evolution is necessarily slow is totally unfounded.

HOWEVER, the belief that _too_ much direction is bad is certainly not
unfounded: it tends to show up major design problems that do not show up
with less control. Again, see overly aggressive breeding of some dogs
causing bad characteristics, and especially the poultry industry.

And you have to realize that the above is with entities that are much more
complex than your random software project, and where historically you have
not been able to actually influence anything but selection itself.

Being able to influence not just selection, but actually influencing the
_mutations_ that happen directly obviously cuts down the time by another
large piece.

In short, your comment about "not pertinent" only shows that you are
either not very well informed about biological changes, or, more likely,
it's just a gut reaction without actually _thinking_ about it.

Biological evolution is alive and well, and does not take millions of
years to make changes. In fact, most paleolontologists consider some of
the changes due to natural disasters to have happened susprisingly fast,
even in the _absense_ of "intelligent direction".

Of course, at the same time evolution _does_ heavily tend to favour
"stable" life-forms (sharks and many amphibians have been around for
millions of years). That's not because evolution is slow, but simply
because good lifeforms work well in their environments, and if the
environment doesn't change radically they have very few pressures to
change.

There is no inherent "goodness" in change. In fact, there are a lot of
reasons _against_ change, something we often forget in technology. The
fact that evolution is slow when there is no big reason to evolve is a
_goodness_, not a strike against it.

> > Quite frankly, Sun is doomed. And it has nothing to do with their
> > engineering practices or their coding style.
>
> I'd love to hear your thoughts on why.

You heard them above. Sun is basically inbreeding. That tends to be good
to bring out specific characteristics of a breed, and tends to be good for
_specialization_. But it's horrible for actual survival, and generates a
very one-sided system that does not adapt well to change.

Microsoft, for all the arguments against them, is better off simply
because of the size of its population - they have a much wider consumer
base, which in turn has caused them largely to avoid specialization. As a
result, Microsoft has a much wider appeal - and suddenly most of the
niches that Sun used to have are all gone, and its fighting for its life
in many of its remaining ones.

Why do you think Linux ends up being the most widely deployed Unix? It's
avoided niches, it's avoided inbreeding, and not being too directed means
that it doesn't get the problems you see with unbalanced systems.

Face it, being one-sided is a BAD THING. Unix was dying because it was
becoming much too one-sided.

Try to prove me wrong.

		Linus

Re: Coding style - a non-issue

[Horst von Brand]

Tim Hockin <thockin@hockin.org> said:
> "Linus Torvalds" at Nov 30, 2001 04:50:34 PM said:
> > I'm deadly serious: we humans have _never_ been able to replicate
> > something more complicated than what we ourselves are, yet natural
> > selection did it without even thinking.
> 
> a very interesting argument, but not very pertinent - we don't have 10's of
> thousands of year or even really 10's of years.  We have to use intellect
> to root out the obviously bad ideas, and even more importantly the
> bad-but-not-obviously-bad ideas.

If you look at what people in the evolutionistic algorithms camp are doing,
you'll see them consider "mutations" that force movement in the "best"
direction (steepest ascent), not just at random, and similarly other
"intelligent" genetic operators.

In any case, what Linus is saying isn't just "go and change stuff at
random" either..., this is clearly a metaphor. An apt one.
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Coding style - a non-issue

[Victor Yodaiken]

On Fri, Nov 30, 2001 at 04:50:34PM -0800, Linus Torvalds wrote:
> 
> You know what the most complex piece of engineering known to man in the
> whole solar system is?
> 
> Guess what - it's not Linux, it's not Solaris, and it's not your car.
> 
> It's you. And me.
> 
> And think about how you and me actually came about - not through any
> complex design.
> 
> Right. "sheer luck".

Somehow this does not give me a warm and fuzzy feeling about
the length of the next Linux kernel development cycle. 

> And don't EVER make the mistake that you can design something better than
> what you get from ruthless massively parallel trial-and-error with a
> feedback cycle. That's giving your intelligence _much_ too much credit.

Linux is what it is because of design, not accident. And you know
that better than anyone.
If mindless rooting about could make a good OS,  then we'd all be using 
[ in a rare moment of good sense I don't finish this sentence ]

The question is whether Linux can still be designed at
current scale.




> Quite frankly, Sun is doomed. And it has nothing to do with their
> engineering practices or their coding style.

The San Andreas fault? 

Re: Coding style - a non-issue

[Linus Torvalds]

On Fri, 30 Nov 2001, Victor Yodaiken wrote:
>
> > And don't EVER make the mistake that you can design something better than
> > what you get from ruthless massively parallel trial-and-error with a
> > feedback cycle. That's giving your intelligence _much_ too much credit.
>
> Linux is what it is because of design, not accident. And you know
> that better than anyone.

Let's just be honest, and admit that it wasn't designed.

Sure, there's design too - the design of UNIX made a scaffolding for the
system, and more importantly it made it easier for people to communicate
because people had a mental _model_ for what the system was like, which
means that it's much easier to discuss changes.

But that's like saying that you know that you're going to build a car with
four wheels and headlights - it's true, but the real bitch is in the
details.

And I know better than most that what I envisioned 10 years ago has
_nothing_ in common with what Linux is today. There was certainly no
premeditated design there.

And I will claim that nobody else "designed" Linux any more than I did,
and I doubt I'll have many people disagreeing. It grew. It grew with a lot
of mutations - and because the mutations were less than random, they were
faster and more directed than alpha-particles in DNA.

> The question is whether Linux can still be designed at
> current scale.

Trust me, it never was.

And I will go further and claim that _no_ major software project that has
been successful in a general marketplace (as opposed to niches) has ever
gone through those nice lifecycles they tell you about in CompSci classes.
Have you _ever_ heard of a project that actually started off with trying
to figure out what it should do, a rigorous design phase, and a
implementation phase?

Dream on.

Software evolves. It isn't designed. The only question is how strictly you
_control_ the evolution, and how open you are to external sources of
mutations.

And too much control of the evolution will kill you. Inevitably, and
without fail. Always. In biology, and in software.

Amen.

		Linus

Re: Coding style - a non-issue

[Larry McVoy]

On Fri, Nov 30, 2001 at 07:15:55PM -0800, Linus Torvalds wrote:
> > The question is whether Linux can still be designed at
> > current scale.
> 
> Trust me, it never was.

Yeah, right, Linus.  We should all go home and turn loose the monkeys and
let them pound on the keyboard.  They'll just as good a job, in fact, by
your reasoning they'll get there faster, they aren't so likely to waste
time trying to design it.

I can't believe the crap you are spewing on this one and I don't think you
do either.  If you do, you need a break.  I'm all for letting people explore,
let software evolve, that's all good.  But somebody needs to keep an eye on
it.

If that's not true, Linus, then bow out.   You aren't needed and *you*
just proved it.  You can't have it both ways.  Either you are here
for a reason or you aren't.  So which is it?  You're arguing that you
don't matter.  I personally don't agree, I think Linux would be a pile
of dog doo without you.  If you don't agree, back off and see what happens.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Linus Torvalds]

On Fri, 30 Nov 2001, Larry McVoy wrote:
>
> I can't believe the crap you are spewing on this one and I don't think you
> do either.  If you do, you need a break.  I'm all for letting people explore,
> let software evolve, that's all good.  But somebody needs to keep an eye on
> it.

Like somebody had to keep an eye on our evolution so that you had a chance
to be around?

Who's naive?

> If that's not true, Linus, then bow out.   You aren't needed and *you*
> just proved it.

Oh, absolutely.

I wish more people realized it. Some people realize it only when they get
really pissed off at me and say "Go screw yourself, I can do this on my
own". And you know what? They are right too, even if they come to that
conclusion for what I consider the wrong reasons.

The reason I'm doing Linux is not because I think I'm "needed". It's
because I enjoy it, and because I happen to believe that I'm better than
most at it. Not necessarily better than everybody else around there, but
good enough, and with the social ties to make me unbeatable right now.

But "indispensable"? Grow up, Larry. You give me too much credit.

And why should I bow out just because I'm not indispenable? Are you
indispensable for the continued well-being of humanity? I believe not,
although you are of course free to disagree. Should you thus "bow out" of
your life just because you're strictly speaking not really needed?

Do I direct some stuff? Yes. But, quite frankly, so do many others. Alan
Cox, Al Viro, David Miller, even you. And a lot of companies, which are
part of the evolution whether they realize it or not. And all the users,
who end up being part of the "fitness testing".

And yes, I actually do believe in what I'm saying.

		Linus

Re: Coding style - a non-issue

[Daniel Phillips]

On December 1, 2001 04:30 am, Larry McVoy wrote:
> I can't believe the crap you are spewing on this one and I don't think you
> do either.  If you do, you need a break.  I'm all for letting people 
> explore, let software evolve, that's all good.  But somebody needs to keep 
> an eye on it.
> 
> If that's not true, Linus, then bow out.   You aren't needed and *you*
> just proved it.  You can't have it both ways.  Either you are here
> for a reason or you aren't.  So which is it?  You're arguing that you
> don't matter.  I personally don't agree, I think Linux would be a pile
> of dog doo without you.  If you don't agree, back off and see what happens.

If you've been involved in any design sessions with Linus - if you could even 
grace them with that name - you know he relies way more on intuition than 
process.  Actually, far from taking the role of the omniscient creator, he 
tends to act more like the survival test.  Not that he's short of the 
necessary skills to do it your way.  I think he does it the way he does it 
because it's fun and interesting and, oh yes, effective.

--
Daniel

Re: Coding style - a non-issue

[Victor Yodaiken]

On Fri, Nov 30, 2001 at 07:15:55PM -0800, Linus Torvalds wrote:
> And I will claim that nobody else "designed" Linux any more than I did,
> and I doubt I'll have many people disagreeing. It grew. It grew with a lot
> of mutations - and because the mutations were less than random, they were
> faster and more directed than alpha-particles in DNA.

Ok. There was no design, just "less than random mutations". 
Deep. 

There was a overall architecture, from Dennis and Ken. There
where a couple of good sound design principles, and there were a 
couple of people with some sense of how it should work together.
None of that is incompatible with lots of trial and error and learn
by doing.

Here's a characteristic good Linux design method ,( or call it "less than random
mutation method" if that makes you feel happy): read the literature,
think hard, try something, implement
it, find it doesn't do what was hoped and tear the whole thing down.
That's design. Undesigned systems use the method of: implement some crap and then try to
engineer the consequences away. 

> 
> > The question is whether Linux can still be designed at
> > current scale.
> 
> Trust me, it never was.

Trust you? Ha.

> And I will go further and claim that _no_ major software project that has
> been successful in a general marketplace (as opposed to niches) has ever
> gone through those nice lifecycles they tell you about in CompSci classes.

That's classic:
	A) "trust me"
	B) now here's a monster bit of misdirection for you to choke on.

Does anyone believe in those stupid software lifcycles?
No.
So does it follow that this has anything to do with design?
No.


> Have you _ever_ heard of a project that actually started off with trying
> to figure out what it should do, a rigorous design phase, and a
> implementation phase?
> 
> Dream on.

I've seen better arguments in slashdot. 

There was no puppet master - ok.
There was no step by step recipe that showed how it should all work - ok
There was no design involved - nope.

Re: Coding style - a non-issue

[Linus Torvalds]

On Fri, 30 Nov 2001, Victor Yodaiken wrote:
>
> Ok. There was no design, just "less than random mutations".
> Deep.

I'm not claiming to be deep, I'm claiming to do it for fun.

I _am_ claiming that the people who think you "design" software are
seriously simplifying the issue, and don't actually realize how they
themselves work.

> There was a overall architecture, from Dennis and Ken.

Ask them. I'll bet you five bucks they'll agree with me, not with you.
I've talked to both, but not really about this particular issue, so I
might lose, but I think I've got the much better odds.

If you want to see a system that was more thoroughly _designed_, you
should probably point not to Dennis and Ken, but to systems like L4 and
Plan-9, and people like Jochen Liedtk and Rob Pike.

And notice how they aren't all that popular or well known? "Design" is
like a religion - too much of it makes you inflexibly and unpopular.

The very architecture of UNIX has very much been an evolution. Sure, there
are some basic ideas, but basic ideas do not make a system.

When they say that the devil is in the details, they are trying to tell
you that the details MATTER. In fact, the details matter quite a lot more
than the design ever does.

> Here's a characteristic good Linux design method ,( or call it "less than random
> mutation method" if that makes you feel happy): read the literature,
> think hard, try something, implement

Hah.

I don't think I've seen very many examples of that particular design
methodology.

It's impressive that you think this is how stuff gets done, but from
personal experience I would say that it's certainly not true to any
noticeable degree. The impressive part is that Linux development could
_look_ to anybody like it is that organized.

Yes, people read literature too, but that tends to be quite spotty. t's
done mainly for details like TCP congestion control timeouts etc - they
are _important_ details, but at the same time we're talking about a few
hundred lines out of 20 _million_.

And no, I'm no tclaiming that the rest is "random". But I _am_ claiming
that there is no common goal, and that most development ends up being done
for fairly random reasons - one persons particular interest or similar.

It's "directed mutation" on a microscopic level, but there is very little
macroscopic direction. There are lots of individuals with some generic
feeling about where they want to take the system (and I'm obviously one of
them), but in the end we're all a bunch of people with not very good
vision.

And that is GOOD.

A strong vision and a sure hand sound like good things on paper. It's just
that I have never _ever_ met a technical person (including me) whom I
would trust to know what is really the right thing to do in the long run.

Too strong a strong vision can kill you - you'll walk right over the edge,
firm in the knowledge of the path in front of you.

I'd much rather have "brownian motion", where a lot of microscopic
directed improvements end up pushing the system slowly in a direction that
none of the individual developers really had the vision to see on their
own.

And I'm a firm believer that in order for this to work _well_, you have to
have a development group that is fairly strange and random.

To get back to the original claim - where Larry idolizes the Sun
engineering team for their singlemindedness and strict control - and the
claim that Linux seems ot get better "by luck": I really believe this is
important.

The problem with "singlemindedness and strict control" (or "design") is
that it sure gets you from point A to point B in a much straighter line,
and with less expenditure of energy, but how the HELL are you going to
consistently know where you actually want to end up? It's not like we know
that B is our final destination.

In fact, most developers don't know even what the right _intermediate_
destinations are, much less the final one. And having somebody who shows
you the "one true path" may be very nice for getting a project done, but I
have this strong belief that while the "one true path" sometimes ends up
being the right one (and with an intelligent leader it may _mostly_ be the
right one), every once in a while it's definitely the wrong thing to do.

And if you only walk in single file, and in the same direction, you only
need to make one mistake to die.

In contrast, if you walk in all directions at once, and kind of feel your
way around, you may not get to the point you _thought_ you wanted, but you
never make really bad mistakes, because you always ended up having to
satisfy a lot of _different_ opinions. You get a more balanced system.

This is what I meant by inbreeding, and the problem that UNIX
traditionally had with companies going for one niche.

(Linux companies also tend to aim for a niche, but they tend to aim for
_different_ niches, so the end result is the very "many different
directions" that I think is what you _want_ to have to survive).

> > And I will go further and claim that _no_ major software project that has
> > been successful in a general marketplace (as opposed to niches) has ever
> > gone through those nice lifecycles they tell you about in CompSci classes.
>
> That's classic:
> 	A) "trust me"
> 	B) now here's a monster bit of misdirection for you to choke on.
>
> Does anyone believe in those stupid software lifcycles?
> No.
> So does it follow that this has anything to do with design?
> No.

Hey, the whole _notion_ of "design" is that you know what you're going to
write, and you design for it.

Or do you have some other meaning of the word "design" that I haven't
heard about.

		Linus

Re: Coding style - a non-issue

[Daniel Phillips]

On December 1, 2001 06:15 am, Linus Torvalds wrote:
> On Fri, 30 Nov 2001, Victor Yodaiken wrote:
> > Here's a characteristic good Linux design method ,( or call it "less than 
> > random mutation method" if that makes you feel happy): read the 
> > literature, think hard, try something, implement
> 
> Hah.
> 
> I don't think I've seen very many examples of that particular design
> methodology.

I do it a little differently: think hard, try something, implement, read the 
literature, repeat as necessary.

--
Daniel

Re: Coding style - a non-issue

[Victor Yodaiken]

On Sat, Dec 01, 2001 at 07:13:55AM +0100, Daniel Phillips wrote:
> On December 1, 2001 06:15 am, Linus Torvalds wrote:
> > On Fri, 30 Nov 2001, Victor Yodaiken wrote:
> > > Here's a characteristic good Linux design method ,( or call it "less than 
> > > random mutation method" if that makes you feel happy): read the 
> > > literature, think hard, try something, implement
> > 
> > Hah.
> > 
> > I don't think I've seen very many examples of that particular design
> > methodology.
> 
> I do it a little differently: think hard, try something, implement, read the 
> literature, repeat as necessary.

Ordering is not key in this recipe.

Re: Coding style - a non-issue

[Daniel Phillips]

On December 1, 2001 09:17 pm, Victor Yodaiken wrote:
> On Sat, Dec 01, 2001 at 07:13:55AM +0100, Daniel Phillips wrote:
> > On December 1, 2001 06:15 am, Linus Torvalds wrote:
> > > On Fri, 30 Nov 2001, Victor Yodaiken wrote:
> > > > Here's a characteristic good Linux design method ,( or call it "less than 
> > > > random mutation method" if that makes you feel happy): read the 
> > > > literature, think hard, try something, implement
> > > 
> > > Hah.
> > > 
> > > I don't think I've seen very many examples of that particular design
> > > methodology.
> > 
> > I do it a little differently: think hard, try something, implement, read the 
> > literature, repeat as necessary.
> 
> Ordering is not key in this recipe.

Right, I'm just saying that's how I typically do it.  A decade ago I'd 
probably have put the 'try something' first, and a decade before that,
'read the litature'.

--
Daniel

Re: Coding style - a non-issue

[Victor Yodaiken]

On Fri, Nov 30, 2001 at 09:15:50PM -0800, Linus Torvalds wrote:
> 
> On Fri, 30 Nov 2001, Victor Yodaiken wrote:
> >
> > Ok. There was no design, just "less than random mutations".
> > Deep.
> 
> I'm not claiming to be deep, I'm claiming to do it for fun.
> 
> I _am_ claiming that the people who think you "design" software are
> seriously simplifying the issue, and don't actually realize how they
> themselves work.

Just to make sure we are speaking the same language, here is what the
Oxford English Dictionary sez
	Design: (1) a plan or scheme conceived in the mind; a project.
	        ...
		(2) a purpose, an intention, an aim
		...
		(3) an end in view, a goal
		...
		(4) A preliminary sketch, a plan or pattern

For the verb we get things like: "draw, sketch, outline, delineate"

> 
> > There was a overall architecture, from Dennis and Ken.
> 
> Ask them. I'll bet you five bucks they'll agree with me, not with you.
> I've talked to both, but not really about this particular issue, so I
> might lose, but I think I've got the much better odds.

You're on.  Send me the $5.
Here's what Dennis Ritchie wrote in his preface to the re-issued Lions
book:
	"you will see in the code an underlying structure that has
	lasted a long time and has managed to accomodate vast changes
	in the computing environment"


> If you want to see a system that was more thoroughly _designed_, you
> should probably point not to Dennis and Ken, but to systems like L4 and
> Plan-9, and people like Jochen Liedtk and Rob Pike.

You appear to be using "design" to mean "complete specification". 
See above.

> 
> And notice how they aren't all that popular or well known? "Design" is
> like a religion - too much of it makes you inflexibly and unpopular.

Memory fades with age, as I know from sad experience, but try to
remember who wrote things like:

	|
	|However, I still would not call "pthreads" designed. 
	|
	|Engineered. Even well done for what it tried to do. But not "Designed".
	|
	|This is like VMS. It was good, solid, engineering. Design? Who needs
	|design? It _worked_.
	|
	|But that's not how UNIX is or should be.  There was more than just
	|engineering in UNIX.  There was Design with a capital "D".  Notions of
	|"process" vs "file", and things that transcend pure engineering.
	|Minimalism.
	|
	|In the end, it comes down to aesthetics.  pthreads is "let's solve a
	|problem".  But it's not answering the big questions in the universe. 
	|It's not asking itself "what is the underlying _meaning_ of threads?".
	|"What is the big picture?".

Some academic twit, no doubt, with no understanding or experience in
actually making a blue collar OS really work.
The same fool once wrote:

> Think about WHY our system call latency beats everybody else on the
> planet. Think about WHY Linux is fast. It's because it's designed
> right.


Please send the $5 soon.

Re: Coding style - a non-issue

[Kai Henningsen]

torvalds@transmeta.com (Linus Torvalds)  wrote on 30.11.01 in <Pine.LNX.4.33.0111302048200.1459-100000@penguin.transmeta.com>:

> On Fri, 30 Nov 2001, Victor Yodaiken wrote:

> > Here's a characteristic good Linux design method ,( or call it "less than
> > random mutation method" if that makes you feel happy): read the
> > literature, think hard, try something, implement
>
> Hah.
>
> I don't think I've seen very many examples of that particular design
> methodology.

Recently, that seems to get popular in gcc. Sometimes, on gc@gcc.gnu.org,  
you'll see a whole thread where people throw one literature reference  
after another at each other.

Maybe a whole 3% of the traffic.


MfG Kai

Re: Coding style - a non-issue

[Davide Libenzi]

On Fri, 30 Nov 2001, Linus Torvalds wrote:

>
> On Fri, 30 Nov 2001, Victor Yodaiken wrote:
> >
> > Ok. There was no design, just "less than random mutations".
> > Deep.
>
> I'm not claiming to be deep, I'm claiming to do it for fun.
>
> I _am_ claiming that the people who think you "design" software are
> seriously simplifying the issue, and don't actually realize how they
> themselves work.
>
> > There was a overall architecture, from Dennis and Ken.
>
> Ask them. I'll bet you five bucks they'll agree with me, not with you.
> I've talked to both, but not really about this particular issue, so I
> might lose, but I think I've got the much better odds.
>
> If you want to see a system that was more thoroughly _designed_, you
> should probably point not to Dennis and Ken, but to systems like L4 and
> Plan-9, and people like Jochen Liedtk and Rob Pike.
>
> And notice how they aren't all that popular or well known? "Design" is
> like a religion - too much of it makes you inflexibly and unpopular.

The most successful software have born from fixing/patching an
initial/simple implementation while the greatest software failures
have born from deep planning and design.
And for successful I mean widely used/apreciated/longlived.
You start with a very limited ( in features ) initial version, people
happen to like it, you sell/distribute it and they're going to ask for new
features.
Some of these new features will fit the initial design some other will
make you patching it in an undesigned way, but you think that you'll fix
it later.
But people are going to like it even more and become more hungry of new
features, some of them will fit in the initial implementation design some
other will introduce crappy patching, but later time is fix time.
And so on.
What is the deep thought in this, it's that if you've time for deep
planning/design it means that nobody is actually using your software and
when your cutting edge, well designed implementation will see the light,
noone is gonna use it because they've already embraced something else.
Not cutting edge, not really deeply designed but a fu*king working real
software that is solving real problems for the 99% of its users.




- Davide

Re: Coding style - a non-issue

[Lars Brinkhoff]

Davide Libenzi <davidel@xmailserver.org> writes:
> What is the deep thought in this, it's that if you've time for deep
> planning/design it means that nobody is actually using your software
> and when your cutting edge, well designed implementation will see
> the light, noone is gonna use it because they've already embraced
> something else.

Also known as "Worse is Better":

  http://www.ai.mit.edu/docs/articles/good-news/subsection3.2.1.html

-- 
Lars Brinkhoff          http://lars.nocrew.org/     Linux, GCC, PDP-10
Brinkhoff Consulting    http://www.brinkhoff.se/    programming

Re: Coding style - a non-issue

[Ingo Molnar]

On Fri, 30 Nov 2001, Linus Torvalds wrote:

> It's "directed mutation" on a microscopic level, but there is very
> little macroscopic direction. There are lots of individuals with some
> generic feeling about where they want to take the system (and I'm
> obviously one of them), but in the end we're all a bunch of people
> with not very good vision.

the fundamental reasons why this happens are the following, special
conditions of our planet:

 - the human brain has a limited capacity, both in terms of 'storage'
   and 'computational power'.

 - the world itself, at least how we understand it currently, is
   inherently random. (Such randomness is the main driving force of
   'structure' as well, it seems.)

 - this planet has limited resources, so 'survival of the fittest' is
   still the main driving force of 'life'.

 - we do not know and understand the rules of our world yet, and chip
   technology (which drives and enables operating systems) is right at the
   edge (and often beyond the edge) of what physics is capable of
   explaining today. We simply do not know what breakthrough (or brick
   wall of performance) might happen in 1, 2 or 5 years. We did not know
   50 years ago that something like 'SMP' would happen and be commonplace
   these days. I mean, often we dont even know what might happen in the
   next minute.

due to these fundamental issues the development of 'technology' becomes
very, very unpredictable. We only have 5 billion brain cells, and there's
no upgrade path for the time being. Software projects such as Linux are
already complex enough to push this limit. And the capacity limits of the
human brain, plus the striving towards something better (driven by human
needs) cause thousands of ambitios people working on parts of Linux in
parallel.

a few things are clear:

- if we knew the rules of this world and knew how to apply them to every
  detail then we'd be building the 'perfect chip' by this christmas, and
  there would probably be no need for anything else, ever.

- if the human brain (and human fingers) had no limits then we'd be
  designing the 'perfect OS' from grounds up and write it in 2 minutes.

- and if the world wasnt random then there would probably be nothing
  on this planet, ever.

but the reality is that we humans have severe limits, and we do not
understand this random world yet, so we'll unevitably have lots of fun
writing random pieces of Linux (or other) code in many many years to come.

in fact, most computer science books are a glaring example of how limited
the human brain is, and how small and useless part of the world we are
able to explain exactly ;-)

and frankly, i'd *very* disappointed if it was possible to predict the
future beyond our lifetime, and if it was possible to design a perfect
enough OS that does not need any changing in the foreseable future. I'd be
a pretty disappointed and bored person, because someone would predict what
happens and we'd only be dumbly following that grand plan. But the reality
is that such grand plan does not exist. One of the exciting things about
developing an OS is that we almost never know what's around the next
corner.

This whole effort called Linux might look structured on the micro-level
(this world *does* appear to have some rules apart of chaos), but it
simply *cannot* be anything better than random on the macro (longterm)
level. So we better admit this to ourselves and adapt to it.

And anyone who knows better knows something that is worth a dozen Nobel
prizes.

	Ingo

Re: Coding style - a non-issue

[Rik van Riel]

On Sun, 2 Dec 2001, Ingo Molnar wrote:

> This whole effort called Linux might look structured on the micro-level
> (this world *does* appear to have some rules apart of chaos), but it
> simply *cannot* be anything better than random on the macro (longterm)
> level. So we better admit this to ourselves and adapt to it.

Note that this screams for some minimal kind of modularity
on the source level, trying to limit the "magic" to as small
a portion of the code base as possible.

Also, natural selection tends to favour the best return/effort
ratio, not the best end result. Letting a kernel take shape
due to natural selection pressure could well result in a system
which is relatively simple, works well for 95% of the population,
has the old cruft level at the upper limit of what's deemed
acceptable and completely breaks for the last 5% of the population.

That is, unless we give those last 5% of the population baseball
bats and Linus his house address, in which case there is some
biological pressure to merge the fixes needed for those folks ;)

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Ingo Molnar]

On Sun, 2 Dec 2001, Rik van Riel wrote:

> Note that this screams for some minimal kind of modularity on the
> source level, trying to limit the "magic" to as small a portion of the
> code base as possible.

Linux is pretty modular. It's not dogmatically so, nor does it attempt to
guarantee absolute or externally visible modularity, but most parts of it
are pretty modular.

> Also, natural selection tends to favour the best return/effort ratio,
> not the best end result. [...]

there is no 'effort' involved in evolution. Nature does not select along
the path we went. It's exactly this property why it took 5 billion years
to get here, while Linux took just 10 years to be built from grounds up.
The fact is that bacteria took pretty random paths for 2 billion years to
get to the next level. That's alot of 'effort'. So *once* we have
something that is better, it does not matter how long it took to get
there.

( This kind of 'development effort' is not the same as 'robustness', ie.
the amount of effort needed to keep it at the complexity level it is,
against small perturbations in the environment - but that is a different
kind of effort. )

> [...] Letting a kernel take shape due to natural selection pressure
> could well result in a system which is relatively simple, works well
> for 95% of the population, has the old cruft level at the upper limit
> of what's deemed acceptable and completely breaks for the last 5% of
> the population.

no. An insect that is 95.1% effective digesting banana leafs in the jungle
will completely eradicate a competing insect that is 95.0% effective
digesting banana leaves, within a few hundred generations. (provided both
insects have exactly the same parameters otherwise.) And it does not
matter whether it took 100 million years to get to 95.1%, or just one
lucky set of alpha particles hitting a specific DNA part of the original
insect.

	Ingo

Re: Coding style - a non-issue

[Daniel Phillips]

On December 2, 2001 09:12 pm, Ingo Molnar wrote:
> On Sun, 2 Dec 2001, Rik van Riel wrote:
> > Also, natural selection tends to favour the best return/effort ratio,
> > not the best end result. [...]
> 
> there is no 'effort' involved in evolution. Nature does not select along
> the path we went. It's exactly this property why it took 5 billion years
> to get here, while Linux took just 10 years to be built from grounds up.
> The fact is that bacteria took pretty random paths for 2 billion years to
> get to the next level. That's alot of 'effort'.

One fact that is often missed by armchair evolutionists is that evolution is 
not random.  It's controlled by a mechanism (most obviously: gene shuffling) 
and the mechanism *itself* evolves.  That is why evolution speeds up over 
time.  There's a random element, yes, but it's not the principle element.

The fact that Linux has evolved from nothing to what it is in a mere 10 years 
- 30 if you count the 20 years of Unix that came before it - is due entirely 
to the fact that Nature has evolved a very efficient mechanism (us) to guide 
Linux's evolution.

> So *once* we have something that is better, it does not matter how long it 
> took to get there.

Sure, once you are better than the other guy you're going to eat his lunch.  
But time does matter: a critter that fails to get its evolutionary tail in 
gear before somebody eats its lunch isn't going to get a second chance.

--
Daniel

Re: Coding style - a non-issue

[Stephan von Krawczynski]

On Sun, 2 Dec 2001 19:41:26 +0100
Daniel Phillips <phillips@bonn-fries.net> wrote:

> One fact that is often missed by armchair evolutionists is that evolution is 
> not random.  It's controlled by a mechanism (most obviously: gene shuffling) 
> and the mechanism *itself* evolves.  That is why evolution speeds up over 
> time.  There's a random element, yes, but it's not the principle element.

You mean "controlled" up to the point where your small environment got randomly
hit by a smaller sized stone coming right from the nowhere corner of the
universe, or not?

Regards,
Stephan

Re: Coding style - a non-issue

[Daniel Phillips]

On December 2, 2001 08:04 pm, Stephan von Krawczynski wrote:
> On Sun, 2 Dec 2001 19:41:26 +0100
> Daniel Phillips <phillips@bonn-fries.net> wrote:
> 
> > One fact that is often missed by armchair evolutionists is that evolution is 
> > not random.  It's controlled by a mechanism (most obviously: gene shuffling) 
> > and the mechanism *itself* evolves.  That is why evolution speeds up over 
> > time.  There's a random element, yes, but it's not the principal element.
> 
> You mean "controlled" up to the point where your small environment got randomly
> hit by a smaller sized stone coming right from the nowhere corner of the
> universe, or not?

See "principal" above.  There's a random element in the game of bridge, too,
but it's not the principal element.

--
Daniel

Re: Coding style - a non-issue

[Stephan von Krawczynski]

On Sun, 2 Dec 2001 20:17:33 +0100
Daniel Phillips <phillips@bonn-fries.net> wrote:

> > You mean "controlled" up to the point where your small environment got
randomly
> > hit by a smaller sized stone coming right from the nowhere corner of the
> > universe, or not?
> 
> See "principal" above.  There's a random element in the game of bridge, too,
> but it's not the principal element.

Please name another planet where your controlling principle is proven to even
exist.

For a being that is located on a "very small stage in a vast cosmic arena"
(Carl Sagan) you have a very strong opinion about the big picture. How come you
think you are able to _see_ it at all? 
Wouldn't it be more accurate to simply admit, we _can't_ see it (at least
currently) and therefore it must be referred to as _random_? Obviously nobody
is hindered to try to find more pixels of the big picture. But shouldn't one
keep in mind that the picture is possibly _very_ big, compared to oneself and
the levels of complexity we are adjusted to.

A dropped stone is possibly only falling _down_ relative to _your_ personal
point of view.

Regards,
Stephan

Re: Coding style - a non-issue

[Ingo Molnar]

On Sun, 2 Dec 2001, Daniel Phillips wrote:

> One fact that is often missed by armchair evolutionists is that
> evolution is not random. It's controlled by a mechanism (most
> obviously: gene shuffling) and the mechanism *itself* evolves. That is
> why evolution speeds up over time. There's a random element, yes, but
> it's not the principle element.

claiming that the randomness is not the principle element of evolution is
grossly incorrect.

there are two components to the evolution of species: random mutations and
a search of the *existing* gene space called gene shuffling. (to be more
exact gene shuffling is only possible for species that actually do it -
bacteria dont.)

In fact gene shuffling in modern species is designed to 'search for useful
features rewarded by the environment to combine them in one specimen'. Ie.
the combination of useful features such as 'feathers' or 'wings',
introduced as random mutations of dinosaurs. Gene shuffling does not
result in radically new features.

gene shuffling is just the following rule: 'combine two successful DNA
chains more or less randomly to find out whether we can get the better
genes of the two.'. Since most species reproduce more than once, random
gene shuffling has a chance of combining the best possible genes. Risking
oversimplification, i'd say that genes are in essence invariant 'modules'
of a species' genetic plan, which can be intermixed between entities
without harming basic functionality. A requirement of the gene shuffling
process is that the resulting entity has to remain compatible enough with
the source entities to be able to reproduce itself and intermix its genes
with the original gene pool.

in terms of Linux, various new genes are similar to various patches that
improve the kernel. Some of them produce a kernel that crashes trivially,
those are obviously incorrect. Some of them might or might not be useful
in the future. We dont know how the environment will evolve in the future,
so we better keep all our options open and have a big enough 'gene pool'.
The development of individual patches is 'directed' and 'engineered' in
the sense that they produce a working kernel and they are derived from
some past experience and expectations of future. They might be correct or
incorrect, but judging them is always a function of the 'environment'.
Some patches become 'correct' over time. Eg. the preemptable kernel
patches are gaining significance these days - it was clearly a no-no 3
years ago. This is because the environment has changed, and certain
patches predicted or guessed the direction of technology environment
correctly.

if you look at patches on the micro-level, it has lots of structure, and
most of it is 'engineered'. If you look at it on the macro-level, the
Linux kernel as a whole has

(and gene shuffling itself has another random component as well, it's the
imperfectness of it that is one of the sources of random mutations.)

saying that the randomness of evolution is not the principle element is
like claiming that the current Linux code is sufficient and we only have
to shuffle around existing functions to make it better.

> > So *once* we have something that is better, it does not matter how long it
> > took to get there.
>
> Sure, once you are better than the other guy you're going to eat his
> lunch.  But time does matter: a critter that fails to get its
> evolutionary tail in gear before somebody eats its lunch isn't going
> to get a second chance.

this is another interesting detail: the speed of being able to adopt to a
changing environment does matter.

But the original claim which i replied to was that the cost of developing
a new 'feature' matters. Which i said is not true - evolution does not
care about time of development if the environment is relatively stable, or
is changing slowly. The speed of evolution/development only matters once
the environment changes fast.

So to draw the analogy with Linux - as long as the environment (chip
technology, etc.) changes rapidly, what matters most is the ability to
evolve. But once the environment 'cools down' a bit, we can freely search
for the most perfect features in a stable environment, and we'll end up
being 99.9% perfect (or better). [ The original claim which i replied to
said that we'll end up being 95% perfect and stop there, because further
development is too expensive - this claim i took issue with. ]

In fact this happened a number of times during Linux's lifetime. Eg. the
prospect of SMP unsettled the codebase alot and the (relative) quality of
uniprocessor Linux perhaps even decreased. Once the external environment
has settled down, other aspects of Linux caught up as well.

believe me, there was no 'grand plan'. Initially (5 years ago) Linus said
that SMP does not matter much at the moment, and that nothing should be
done in SMP space that hurts uniprocessors. Today it's exactly the other
way around. And i think it's perfectly possible that there will be a new
paradigm in 5 years.

	Ingo

Re: Coding style - a non-issue

[Pavel Machek]

Hi!

> believe me, there was no 'grand plan'. Initially (5 years ago) Linus said
> that SMP does not matter much at the moment, and that nothing should be
> done in SMP space that hurts uniprocessors. Today it's exactly the other
> way around. 

I hope uniprocessors still matter... SMP is nice, but 90% machines are
still uniprocessors... [Or someone get me SMP ;)]
								Pavel
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: Coding style - a non-issue

[Ingo Molnar]

On Sun, 2 Dec 2001, Rik van Riel wrote:

> I think you've pretty much proven how well random
> development works.

i think it's fair to say that we should not increase entropy artificially,
eg. we should not apply randomly generated patches to the kernel tree.

the point is, we should accept the fact that while this world appears to
be governed by rules to a certain degree, the world is also chaotic to a
large degree, and that a Grand Plan That Explains Everything does not
exist. And even if it existed, we are very far away from achieving it, and
even if some friendly alien dropped it on our head, we'd very likely be
unable to get our 5 billion brain cells into a state that is commonly
referred to as 'fully grokking it'.

and having accepted these limitations, we should observe the unevitable
effects of them: that any human prediction of future technology
development beyond 5 years is very, very hypothetical, and thus we must
have some fundamental way of dealing with this unpredictability. (such as
trying to follow Nature's Smart Way Of Not Understanding Much But Still
Getting Some Work Done - called evolution.)

	Ingo

Re: Coding style - a non-issue

[Alan Cox]

> Here's a characteristic good Linux design method ,( or call it "less than random
> mutation method" if that makes you feel happy): read the literature,
> think hard, try something, implement it

That assumes computer science is a functional engineering discipline. Its
not, at best we are at the alchemy stage of progression. You put two things
together it goes bang and you try to work out why.

In many of these fields there is no formal literature. The scientific paper
system in computer science is based on publishing things people already
believe. Much of the rest of the knowledge is unwritten or locked away in
labs as a trade secret, and wil probably never be reused.

Take TCP for example. The TCP protocol is specified in a series of
documents. If you make a formally correct implementation of the base TCP RFC
you won't even make connections. Much of the flow control behaviour, the
queueing and the detail is learned only by being directly part of the
TCP implementing community. You can read  all the scientific papers you
like, it will not make you a good TCP implementor. 

Alan

Re: Coding style - a non-issue

[Victor Yodaiken]

On Sat, Dec 01, 2001 at 08:57:17AM +0000, Alan Cox wrote:
> > Here's a characteristic good Linux design method ,( or call it "less than random
> > mutation method" if that makes you feel happy): read the literature,
> > think hard, try something, implement it
> 
> That assumes computer science is a functional engineering discipline. Its
> not, at best we are at the alchemy stage of progression. You put two things
> together it goes bang and you try to work out why.

Recently, our correspondent from Wales wrote:
	... the changes have been done and
	tested one at a time as they are merged. Real engineering process is the
	only way to get this sort of thing working well.

I really dislike this "alchemy" stuff. It's demeaning and misleading.
All the alchemists ever managed to create were cases of mercury
poisoning.


> In many of these fields there is no formal literature. The scientific paper
> system in computer science is based on publishing things people already
> believe. Much of the rest of the knowledge is unwritten or locked away in
> labs as a trade secret, and wil probably never be reused.
> 
> Take TCP for example. The TCP protocol is specified in a series of
> documents. If you make a formally correct implementation of the base TCP RFC
> you won't even make connections. Much of the flow control behaviour, the
> queueing and the detail is learned only by being directly part of the
> TCP implementing community. You can read  all the scientific papers you
> like, it will not make you a good TCP implementor. 

And you can hack away all you want, you'll never get TCP to work either.
This stuff is a mixture of theory and practice and
whether your theory is picked up from years of experience, boozy
arguments, and thinking, or from a strictly supervised
tour of the library it's theory all the same. 
CS is like any other skilled field. There's a difference between a guy
who knows how to hammer and a master carpenter. 

Re: Coding style - a non-issue

[Alan Cox]

> Recently, our correspondent from Wales wrote:
> 	... the changes have been done and
> 	tested one at a time as they are merged. Real engineering process is the
> 	only way to get this sort of thing working well.

Which doesn't conflict. Engineering does not require science. Science helps
a lot but people built perfectly good brick walls long before they knew why
cement works.

> All the alchemists ever managed to create were cases of mercury
> poisoning.

and chemistry, eventually. You take it as far more demeaning than its meant.

But right now given two chunks of code, I find out what happens by putting
them together not by formal methods. In the case of alchemy v chemistry the
chemists know whether it will probably go bang before they try it (and the
chemical engineers still duck anyway)

Alan

Re: Coding style - a non-issue

[Victor Yodaiken]

On Sat, Dec 01, 2001 at 01:38:11PM +0000, Alan Cox wrote:
> Which doesn't conflict. Engineering does not require science. Science helps
> a lot but people built perfectly good brick walls long before they knew why
> cement works.
> 
> > All the alchemists ever managed to create were cases of mercury
> > poisoning.
> 
> and chemistry, eventually. You take it as far more demeaning than its meant.
> 
> But right now given two chunks of code, I find out what happens by putting
> them together not by formal methods. In the case of alchemy v chemistry the
> chemists know whether it will probably go bang before they try it (and the
> chemical engineers still duck anyway)

Oh come on Alan. You look at a patch and can discard 99% of the really
bad ones _before_ you try them out. How do you do it? Divination? 
Nobody uses formal methods for anything other than generating papers
with more authors than readers. It's true that the academicians
have made a fetish out of the elaborate typesetting that they call
"theory", but in the real world, the distinction between science and
engineering is nothing more than some class snobbery and a rough
categorization of who pays for the work.

If there is a point here, which now seems unlikely, it's that there are
design and engineering skills needed to make real software work.
Neither slashdot nor the formal methods research fellows at Oxford
are capable of generating Linux. 

Re: Coding style - a non-issue

[Stephen Satchell]

[cc list trimmed]

At 07:30 PM 11/30/01 -0800, Larry McVoy wrote:
>Yeah, right, Linus.  We should all go home and turn loose the monkeys and
>let them pound on the keyboard.  They'll just as good a job, in fact, by
>your reasoning they'll get there faster, they aren't so likely to waste
>time trying to design it.

You are confusing the production of Shakespeare with the production of good 
OS code.

The high-level design aspect is that there is a problem to be solved or a 
feature to be provided.  That represents a goal.  Some goals are good and 
some goals are bad.  In many cases, you learn which when you actually do 
the code to implement the goal, and determine whether it helps, hinders, or 
just bloats the OS.

The lower-level design aspect is planning how to achieve the 
goal.  Implementation of the lower-level design in code to achieve the goal 
can contain flaws, flaws that don't appear on paper but raise ugly warts 
when you actually try the implementation out.  In this sense, this is like 
a mutation to achieve a specific effect -- blue eyes, say -- that has a 
disastrous side effect -- it causes the heart to have a single chamber 
instead of four.

This assumes that your, um, mutation even lets the organism live.  Many 
don't.  We call it debugging.


>I can't believe the crap you are spewing on this one and I don't think you
>do either.  If you do, you need a break.  I'm all for letting people explore,
>let software evolve, that's all good.  But somebody needs to keep an eye on
>it.

I don't know you, so I don't know how long you have been in the 
industry.  I've watched Unix evolve from the beginning.  AT&T streams 
versus Berkeley sockets was a wonderful war, and we are all for the better 
for the experimentation because we got the best of both -- especially as I 
was involved in ARPAnet in the beginning and saw the influence of what 
turned into TCP/IP in both environments.  We have different styles of 
system initialization, with some being better for manual management and 
some being better for package management -- and we have both living in the 
world today.  The development of X-terminals was fun, too, to try to 
divorce the requirements for a screen from the system that feeds it, and 
yet today the two different processes run together in a single box without 
too much problem.

These were the products of evolution, of system designers trying to meet a 
need, and the process of getting there was painful and left a lot of 
corpses behind it -- piles of dead code, for sure, but also dead companies 
and burned-out development people.  That's part of the business, 
particularly in the commercial world.

"But someone has to keep an eye on it."  Very true.  After all, in this 
process of natural selection, how do we weed out the mutations that don't 
work?  In the Linux development environment, we have several levels of 
weeding going on.  First, there is peer testing -- people downloading 
patches and trying out new things, which weeds out the worst of the 
mutations.  Then, there are the maintainers who sit in judgement as the 
patches roll by, picking out the deformed ones bodily and making sure that 
two patches that change the same code play nicely with each other.  We then 
have the pre-releases, which for the 2.4 tree were patch-inspected and 
maintained by Linux and by Alan Cox, which people can download and try on 
playpen systems with applications to see if anything subtly nasty crept in 
-- this weeds out a few of the mutations that harm the organism but doesn't 
kill it outright.  Finally, we have a production release into which people 
throw all the varied and wacky applications -- and sometimes an ugly (like 
the VM problem) finally is exposed to the light.

Interestingly, you see a similar development process at commercial software 
vendors and with the successful Open Source projects.  Some of the details 
differ (especially the details on the review process) but the overall 
process is similar.  Why?  It works.

I suggest you check out this site http://www.swebok.org/ and specifically 
download this document 
http://www.swebok.org/documents/stoneman095/Trial_Version_0_95.pdf AND READ 
IT before responding further.  While the Software Engineering Body Of 
Knowledge does not use the exact concepts that Linus used in describing how 
things are done, you will find interesting correlations between what is 
described by this document and the idea that you have called "crap."

Pay particular attention to the section on Validation and Verification.

Stephen Satchell

Re: Coding style - a non-issue

[Zilvinas Valinskas]

On Fri, Nov 30, 2001 at 10:31:24PM -0800, Stephen Satchell wrote:
> [cc list trimmed]
> 
> You are confusing the production of Shakespeare with the production of good 
> OS code.
> 
> The high-level design aspect is that there is a problem to be solved or a 
> feature to be provided.  That represents a goal.  Some goals are good and 
> some goals are bad.  In many cases, you learn which when you actually do 
> the code to implement the goal, and determine whether it helps, hinders, or 
> just bloats the OS.
> 

Sometimes design only _works_ it's way around problems ... 

-- 
Zilvinas Valinskas

Re: Coding style - a non-issue

[Kai Henningsen]

torvalds@transmeta.com (Linus Torvalds)  wrote on 30.11.01 in <Pine.LNX.4.33.0111301907010.1296-100000@penguin.transmeta.com>:

> Have you _ever_ heard of a project that actually started off with trying
> to figure out what it should do, a rigorous design phase, and a
> implementation phase?

Yes.

That one gave rise to _The_Mythical_Man_Month_.

Or in other words, it demonstrated how this was broken enough to create  
one of the more important classics in the field.

MfG Kai

Re: Coding style - a non-issue

[Andreas Dilger]

On Nov 30, 2001  20:02 -0700, Victor Yodaiken wrote:
> On Fri, Nov 30, 2001 at 04:50:34PM -0800, Linus Torvalds wrote:
> > Quite frankly, Sun is doomed. And it has nothing to do with their
> > engineering practices or their coding style.
> 
> The San Andreas fault? 

I wish people would stop saying that, I have never been to California. ;-)

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: Coding style - a non-issue

[Horst von Brand]

Victor Yodaiken <yodaiken@fsmlabs.com> said:

[...]

> Linux is what it is because of design, not accident. And you know
> that better than anyone.

I'd say it is better because the mutations themselves (individual patches)
go through a _very_ harsh evaluation before being applied in the "official"
sources. There is a population of kernels out there (each developer has a
few, distributions check out others, ...), only what survives there has got
a chance to be considered for inclusion.

Sure, this is not the only way Linux moves forward. But it is a large
factor in the success of "Release early. Release often. Take patches from
everywhere."
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Coding style - a non-issue

[Larry McVoy]

On Sat, Dec 01, 2001 at 08:18:06PM -0300, Horst von Brand wrote:
> Victor Yodaiken <yodaiken@fsmlabs.com> said:
> > Linux is what it is because of design, not accident. And you know
> > that better than anyone.
> 
> I'd say it is better because the mutations themselves (individual patches)
> go through a _very_ harsh evaluation before being applied in the "official"
> sources. 

Which is exactly Victor's point.  That evaluation is the design.  If the 
mutation argument held water then Linus would apply *ALL* patches and then
remove the bad ones.  But he doesn't.  Which just goes to show that on this
mutation nonsense, he's just spouting off.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:
> vonbrand@sleipnir.valparaiso.cl on Sat, Dec 01, 2001 at 08:18:06PM -0300

[...]

> > I'd say it is better because the mutations themselves (individual patches)
> > go through a _very_ harsh evaluation before being applied in the "official"
> > sources. 

> Which is exactly Victor's point.  That evaluation is the design.  If the 
> mutation argument held water then Linus would apply *ALL* patches and then
> remove the bad ones.  But he doesn't.  Which just goes to show that on this
> mutation nonsense, he's just spouting off.

Who is to say that bad mutations can't be weeded out _before_ a full
organism is built? It seems not to happen openly in nature's evolution
(then again, there are non-viable embryos, various DNA repair mechanisms
that seem to go wrong all the time in certain parts of the genome, parts
that mutate very fast while others don't change, ...), but this is just a
metaphor, not a slavish following. We certainly (at least think we) can do
better than just random typing.

In your reading, the environment (which evaluates individuals) is the
design. Right (in the sense that you end up with individuals fit to that
environment), but also very wrong (as many quite different layouts will
work).
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Coding style - a non-issue

[Daniel Phillips]

On December 2, 2001 09:25 pm, Larry McVoy wrote:
> On Sat, Dec 01, 2001 at 08:18:06PM -0300, Horst von Brand wrote:
> > Victor Yodaiken <yodaiken@fsmlabs.com> said:
> > > Linux is what it is because of design, not accident. And you know
> > > that better than anyone.
> > 
> > I'd say it is better because the mutations themselves (individual patches)
> > go through a _very_ harsh evaluation before being applied in the 
"official"
> > sources. 
> 
> Which is exactly Victor's point.  That evaluation is the design.

Nope, that isn't design, that's reacting.

> If the mutation argument held water then Linus would apply *ALL* patches 
> and then remove the bad ones.  But he doesn't.  Which just goes to show
> that on this mutation nonsense, he's just spouting off.

Hogwash ;)  Please see my post above where I point out 'evolution isn't 
random'.  Your genes have a plan, if only a vague one.  It goes something 
like this: "we'll allow random variations, but only along certain lines, 
within limits, and in certain combinations, and we'll try to stick to 
variations that haven't killed us in the past."

Sounds a lot like how Linus does things, huh?

I'm sure Linus does have quite considerable talent for design, but I haven't 
seen him execise it much.  Mostly he acts as a kind of goodness daemon, 
sitting in his little pinhole and letting what he considers 'good' stuff pass 
into the box.  There's no doubt about it, it's different from the way you 
like to develop, you and me both.  Equally clearly, it works pretty well.

--
Daniel

Re: Coding style - a non-issue

[Victor Yodaiken]

On Mon, Dec 03, 2001 at 01:55:08AM +0100, Daniel Phillips wrote:
> I'm sure Linus does have quite considerable talent for design, but I haven't 
> seen him execise it much.  Mostly he acts as a kind of goodness daemon, 
> sitting in his little pinhole and letting what he considers 'good' stuff pass 
> into the box.  There's no doubt about it, it's different from the way you 
> like to develop, you and me both.  Equally clearly, it works pretty well.

This is a good explanation of why Linux may fail as a project, but it is
pure fantasy as to how it has so far succeded as a project. 

The tiny part of system I wrote directly and the larger part that
I got to see up close involved a great deal of design, old fashioned 
careful engineering, and even aesthetic principles of what wasgood
design. 

Don't drink the cool aid. Go back and look in the kernel archives and 
you will see extensive design discussions among all the core developers.
Linus has a point about the development of Linux not being in
accord with some master plan (at least not one anyone admits to) , but 
that's about as far as it goes.

Re: Coding style - a non-issue

[Daniel Phillips]

On December 3, 2001 01:04 pm, Victor Yodaiken wrote:
> On Mon, Dec 03, 2001 at 01:55:08AM +0100, Daniel Phillips wrote:
> > I'm sure Linus does have quite considerable talent for design, but I haven't 
> > seen him exercise it much.  Mostly he acts as a kind of goodness daemon, 
> > sitting in his little pinhole and letting what he considers 'good' stuff pass 
> > into the box.  There's no doubt about it, it's different from the way you 
> > like to develop, you and me both.  Equally clearly, it works pretty well.
> 
> This is a good explanation of why Linux may fail as a project, but it is
> pure fantasy as to how it has so far succeeded as a project. 
> 
> The tiny part of system I wrote directly and the larger part that
      ^^^^^^^^^
> I got to see up close involved a great deal of design, old fashioned 
> careful engineering, and even aesthetic principles of what wasgood
> design. 

You're just supporting the point of view that Linus has been espousing, and 
I personally support:  Linux is engineered at a micro level[1] but evolves
on its own at a macro level.

Sure, Linux evolves with help from Linus, but he acts as a filter, not a 
designer.  When Linus does on rare occasions forget himself and actually 
design something, its micro-engineering like you or I would do.  So if Linux 
is designed, who does do the designing, can you name him?  I can tell you for 
sure it's not Linus.

> Don't drink the cool aid. Go back and look in the kernel archives and 
> you will see extensive design discussions among all the core developers.
> Linus has a point about the development of Linux not being in
> accord with some master plan (at least not one anyone admits to) , but 
> that's about as far as it goes.

Don't worry about me drinking the cool aid, first I already drank it and 
second I'm personally already fully devoted to the notion of design process, 
including all the usual steps:  blue sky, discussion, requirements, data 
design, detail design, prototype, etc. etc.  You'll find the 'paper trails' 
in the archives if you've got the patience to go spelunking, and you'll have 
a hard time finding one of those designs that became a dead end.  This design 
thing does work for me.  It doesn't change the fact that what I'm doing is 
micro-engineering.

I'll get really worried if Linus wakes up one day and decides that from now 
on he's going to properly engineer every aspect of the Linux kernel.  The 
same way I'd feel if Linux got taken over by a committee.

--
Daniel

[1] In places.  All those little warts and occasional pools of sewage are 
clearly not 'engineered'.

Re: Coding style - a non-issue

[Victor Yodaiken]

On Mon, Dec 03, 2001 at 02:20:38PM +0100, Daniel Phillips wrote:
> You're just supporting the point of view that Linus has been espousing, and 
> I personally support:  Linux is engineered at a micro level[1] but evolves
> on its own at a macro level.

If this becomes true, then Linux fails. If the coreLinux designers did
not "get" the UNIX design and understand what was good about Plan9, then
Linux would look like IRIX without quality control. And the engineer
formerly known as Linus has admitted as much in  past .e.g. with the 
comment about UNIX being about "Desgin with a capital D".

> I'll get really worried if Linus wakes up one day and decides that from now 
> on he's going to properly engineer every aspect of the Linux kernel.  The 
> same way I'd feel if Linux got taken over by a committee.

I'm at a loss here. I state that an emphasis on design has been critical
to Linux  and you respond that it would be bad if Linus wanted to
personally engineer every aspect of the Linux kernel. 
In the english language, the word "design" does not have the same
semantics as "control every detail". 

Anyways, enough.



> 
> --
> Daniel
> 
> [1] In places.  All those little warts and occasional pools of sewage are 
> clearly not 'engineered'.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Coding style - a non-issue

[Ingo Molnar]

On Mon, 3 Dec 2001, Daniel Phillips wrote:

> [...] Please see my post above where I point out 'evolution isn't
> random'.  Your genes have a plan, if only a vague one.  It goes
> something like this: "we'll allow random variations, but only along
> certain lines, within limits, and in certain combinations, and we'll
> try to stick to variations that haven't killed us in the past."

so what you say in essence is that "evolution isnt random, it's random"
;-) The fact that the brownean motion is 'vaguely directed' (ie. evolution
has a limited amount of 'memory' of past experience coded into the DNA)
does not make it less random. Randomness does not have to be completely
undirected - perhaps you know a different definition for 'random'. Just
the fact that we got from bacteria to humans and from bacteria to trees
shows that it's not only random, it's also unstable and chaotic. (the same
initial conditions resulted in multiple, wildly different and almost
completely unrelated set of end results.)

and nobody claimed Linux development was totally (chryptographically)
random. We just claim that Linux development has a fair dose of randomness
and unpredictability besides having a fair dose of structure, and that its
development model is much closer to evolution than to the formal methods
of software development.

at which point i think we finally agree?

	Ingo

Re: Coding style - a non-issue

[David L. Parsley]

Larry McVoy wrote:


> Which is exactly Victor's point.  That evaluation is the design.  If the 
> mutation argument held water then Linus would apply *ALL* patches and then
> remove the bad ones.  But he doesn't.  Which just goes to show that on this
> mutation nonsense, he's just spouting off.


Eh, come on Larry.  You're too smart for this crap (as are others, your 
straw just broke the camel's back).  Linus was just using an analogy to 
illustrate some very valid points.  All analogies break down when 
applied to the nth degree.  Insulting Linus because you've found a spot 
where the analogy breaks is just ludicrous.

regards,
	David

Re: Coding style - a non-issue

[Larry McVoy]

On Sun, Dec 02, 2001 at 08:34:09PM -0500, David L. Parsley wrote:
> Larry McVoy wrote:
> > Which is exactly Victor's point.  That evaluation is the design.  If the 
> > mutation argument held water then Linus would apply *ALL* patches and then
> > remove the bad ones.  But he doesn't.  Which just goes to show that on this
> > mutation nonsense, he's just spouting off.
> 
> Eh, come on Larry.  You're too smart for this crap (as are others, your 
> straw just broke the camel's back).  Linus was just using an analogy to 
> illustrate some very valid points.  All analogies break down when 
> applied to the nth degree.  Insulting Linus because you've found a spot 
> where the analogy breaks is just ludicrous.

This whole mutation crap is ludicrous and if you read through the archives
you can find numerous examples where Linus himself says so.  I have no idea
why he is saying what he is, but that's neither here nor there.  Nonsense
is nonsense, regardless of who says it or why they say it.

Doesn't it strike you the least bit strange that when I challenge Linus to
bow out because he asserts that he isn't needed, this is just some grand
experiment in genetics which is working fine, he says everything would be
fine if he left but he isn't going to because he's having fun?  Isn't that
just a tad convenient?  It's a crock of crap too.  Linus has excellent taste,
better than any OS guy I've run into in 20 years, and if he bowed out a ton
of crap would make it into the kernel that doesn't now.  Genetic mutation
my ass.  If you want an experiment in evolution, then let *everything* into
the kernel.  That's how evolution works, it tries everything, it doesn't
prescreen.  Go read Darwin, go think, there isn't any screening going on,
evolution *is* the screening.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Horst von Brand]

Larry McVoy <lm@bitmover.com> said:

[...]

> Doesn't it strike you the least bit strange that when I challenge Linus to
> bow out because he asserts that he isn't needed, this is just some grand
> experiment in genetics which is working fine, he says everything would be
> fine if he left but he isn't going to because he's having fun?

Hummm... Linux works because there is a harsh evaluation of patches that
try to make it into the stock kernel. There Linus _is_ needed, if nothing
else as a final authority almost everybody respects.

If he enjoys what he is doing, why shouldn't he go on doing it? If he seems
to be doing it right, irrespective of whatever reason (right or terribly
wrong) he gives for it?

>                                                                 Isn't that
> just a tad convenient?  It's a crock of crap too.  Linus has excellent taste,
> better than any OS guy I've run into in 20 years, and if he bowed out a ton
> of crap would make it into the kernel that doesn't now.  Genetic mutation
> my ass.

Accept patches and ideas from all over the place, coupled with evaluation
of said patches and suggestions which weeds out the bad ones before
integration, others when they show wanting after integration is a form of
evolution. No overall direction, no grand design a la OS/360, just design
(sort of) in rather limited areas at a time. Not 100% Darwinian evolution,
with some taints of "acquired traits are inherited", and some (limited)
sense of direction, and also cross movement of traits form others (be it
BSD, or different kernel lines).

>         If you want an experiment in evolution, then let *everything* into
> the kernel.  That's how evolution works, it tries everything, it doesn't
> prescreen.  Go read Darwin, go think, there isn't any screening going on,
> evolution *is* the screening.

Why does the screening have to be at the level of full organisms? It
_looks_ that way because you don't see the busted sperm or broken eggs, or
the stillborn embryos which make up the "preliminary checks show it won't
work" in nature. The process is (hopefully) much more efficient here than
in nature, and visible, that is all.
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Coding style - a non-issue

[Ragnar Hojland Espinosa]

On Mon, Dec 03, 2001 at 10:39:08PM -0300, Horst von Brand wrote:
> >         If you want an experiment in evolution, then let *everything* into
> > the kernel.  That's how evolution works, it tries everything, it doesn't
> > prescreen.  Go read Darwin, go think, there isn't any screening going on,
> > evolution *is* the screening.
> 
> Why does the screening have to be at the level of full organisms? It
> _looks_ that way because you don't see the busted sperm or broken eggs, or
> the stillborn embryos which make up the "preliminary checks show it won't
> work" in nature. The process is (hopefully) much more efficient here than
> in nature, and visible, that is all.

And I'd add something more along those lines..

Evolution and selection is about species, not individuals as its commonly
considered, so what might be bad for an individual (getting "screened" at
early ages) might be good for (reproduction of) the species (since it
ensures a better reproduction material quality)  Darwinian evolution doesnt
fit too well in the kernel.

On the other hand we can think of developers' minds as a copy-on-write DNA.
DNA knows when something wont work, so it doesn't try it.   Screening :)

-- 
____/|  Ragnar Højland      Freedom - Linux - OpenGL |    Brainbench MVP
\ o.O|  PGP94C4B2F0D27DE025BE2302C104B78C56 B72F0822 | for Unix Programming
 =(_)=  "Thou shalt not follow the NULL pointer for  | (www.brainbench.com)
   U     chaos and madness await thee at its end."

Re: Coding style - a non-issue

[Oliver Xymoron]

On Sun, 2 Dec 2001, Larry McVoy wrote:

> If you want an experiment in evolution, then let *everything* into
> the kernel.  That's how evolution works, it tries everything, it doesn't
> prescreen.  Go read Darwin, go think, there isn't any screening going on,
> evolution *is* the screening.

So-called 'natural selection' is only a subset of things that can quite
legitimately be called evolution. And there certainly is screening in
nature, it's called sexual selection.

Linus's point is mainly about parallelism. Many more changes get tried in
the Linux space than could ever happen in a traditional software
development environment.

--
 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

Re: Coding style - a non-issue

[Stephen Satchell]

[cc list trimmed]

At 06:02 PM 11/30/01 -0800, Tim Hockin wrote:
> > Linux sez:
> > I'm deadly serious: we humans have _never_ been able to replicate
> > something more complicated than what we ourselves are, yet natural
> > selection did it without even thinking.
>
>a very interesting argument, but not very pertinent - we don't have 10's of
>thousands of year or even really 10's of years.  We have to use intellect
>to root out the obviously bad ideas, and even more importantly the
>bad-but-not-obviously-bad ideas.

Disagree with your position strongly.  It's very pertinent.

Most of the bad-but-not-obviously-bad ideas get rooted out by people trying 
them and finding them to be wanting.  Take, for example, the VM flap in the 
2.4.* tree:  an astonishing side effect of the operation of the VM system 
caused people to come up with one that wasn't so astonishing.  We're not 
sure why the original VM caused such problems.  We fixed it anyway.  (No, I 
played no part in that particular adventure, I was just viewing from the 
sidelines.)

The "Linux Way" as I understand it is to release early and release 
often.  That means that we go through a "generation" of released code every 
few weeks, and a "generation" of beta candidates just about daily...and if 
you include the patches that appear here during every 24 hours, the 
generation cycle is even faster than that.  That means that any mutations 
that are detrimental to the organism are exposed within days -- sometimes 
even hours -- of their introduction into the code base.

When we have a development tree open (as 2.5 is now freshly open) there are 
even more generations of code, which further makes natural selection viable 
as a weeding process for good and bad code.  The difference is that the 
number of people affected by the weeding process is smaller, and the 
probability of killing production systems with mutations becomes 
smaller.  The population of the organism is thus healthier because 
mutations affect a smaller fraction of the population, and the chances of 
expensive illness is reduced.

Beneficial mutations are "back-ported" into the 2.4 and even the 2.2 code 
trees, mutations that have proven their worth by extensive experimentation 
and experience.  Unlike the biological equivalent, this selective spreading 
of mutations further improves the health of the population of organisms.

Now that I've stretched the analogy as far as I care to, I will stop 
now.  Please consider the life-cycle of the kernel when thinking about what 
Linus said.

Just my pair-o-pennies(tm).

Stephen Satchell

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Linus Torvalds wrote:

> And don't EVER make the mistake that you can design something better than
> what you get from ruthless massively parallel trial-and-error with a
> feedback cycle. That's giving your intelligence _much_ too much credit.

So, are we going to take out the appendix in 2.5 or will
we continue hoping our kernel doesn't catch an illness
without actually doing anything preventive ?

Biological selection does nothing except removing the weak
ones, it cannot automatically create systems which work well.

In short, I believe the biological selection is just that,
selection. The creation of stuff will need some direction.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Ingo Oeser]

On Sat, Dec 01, 2001 at 09:18:54AM -0200, Rik van Riel wrote:
> Biological selection does nothing except removing the weak
> ones, it cannot automatically create systems which work well.
> 
> In short, I believe the biological selection is just that,
> selection. The creation of stuff will need some direction.

Creation is as simple as: 

1. If you encounter a problem, try to solve it.
2. If you cannot solve it, mark/document/publish it and try to
   work around it for now.
3. If you cannot work around it, leave it to other people and
   offer help.

Which is pretty much what this list here is for ;-)

Regards

Ingo Oeser
-- 
Science is what we can tell a computer. Art is everything else. --- D.E.Knuth

Re: Coding style - a non-issue

[Rik van Riel]

On Sat, 1 Dec 2001, Ingo Oeser wrote:
> On Sat, Dec 01, 2001 at 09:18:54AM -0200, Rik van Riel wrote:
> > Biological selection does nothing except removing the weak
> > ones, it cannot automatically create systems which work well.
> >
> > In short, I believe the biological selection is just that,
> > selection. The creation of stuff will need some direction.
>
> Creation is as simple as:
>
> 1. If you encounter a problem, try to solve it.
> 2. If you cannot solve it, mark/document/publish it and try to
>    work around it for now.
> 3. If you cannot work around it, leave it to other people and
>    offer help.
>
> Which is pretty much what this list here is for ;-)

4. If the problem cannot be solved with a quick hack by
   one person, give up and switch to another OS.

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Martin Dalecki]

Rik van Riel wrote:
> 
> On Fri, 30 Nov 2001, Linus Torvalds wrote:
> 
> > And don't EVER make the mistake that you can design something better than
> > what you get from ruthless massively parallel trial-and-error with a
> > feedback cycle. That's giving your intelligence _much_ too much credit.
> 
> So, are we going to take out the appendix in 2.5 or will
> we continue hoping our kernel doesn't catch an illness
> without actually doing anything preventive ?
> 
> Biological selection does nothing except removing the weak
> ones, it cannot automatically create systems which work well.
> 
> In short, I believe the biological selection is just that,
> selection. The creation of stuff will need some direction.
> 
> regards,

One thing Linus doesn't realize is that the most successfull
results of the biological selection are not the high mamals but
simle archaic bacterias. This is the point where his
analogy breaks badly.

One one thing he simple appears to have forgotten: Operating 
systems have a *purpose*. Therefore the stuff one wan't to do with
a system is what is driving it's true developement - this is most
pertinent in the developement of a driver - one does it to
use some kind of hardware it's that simple.

However I agree with him fully that most of the so called 
software engineering methods do not have much in common with
the reality out there and are basically a self exercise.

Re: Coding style - a non-issue

[Stephan von Krawczynski]

On Sun, 02 Dec 2001 17:25:54 +0100
Martin Dalecki <dalecki@evision-ventures.com> wrote:

> One one thing he [Linus] simple appears to have forgotten: Operating 
> systems have a *purpose*.

He didn't. The simple purpose of all being is equal: survival
As it is equal to all, talking about it is pretty redundant.

Regards,
Stephan

Re: Coding style - a non-issue

[Ingo Molnar]

On Sun, 2 Dec 2001, Martin Dalecki wrote:

> One thing Linus doesn't realize is that the most successfull results
> of the biological selection are not the high mamals but simle archaic
> bacterias. This is the point where his analogy breaks badly.

the fact that simple forms of life still exist shows only one thing: often
the environment is so harsh (ie. so *random*) that only the simplest can
survive. This is not 'success', it's a bow towards chaos.

the other reason is that simple forms of life are often just a tool for
higher forms of lifes to exist, or do not matter to the existence of
higher forms of life. Lets note that the human race *did* eradicate
certain strains of bacteria from this planet completely (except some more
or less safe places of storage), such as smallpocks, which pretty much
shows who has 'succeeded'.

	Ingo

Re: Coding style - a non-issue

[Horst von Brand]

Linus Torvalds <torvalds@transmeta.com> said:
> On Fri, 30 Nov 2001, Rik van Riel wrote:
> > I'm very interested too, though I'll have to agree with Larry
> > that Linux really isn't going anywhere in particular and seems
> > to be making progress through sheer luck.

In any case, Larry's suggestion would fracture Linux. Also, Solaris might
be a fine piece of work, but what is commonly called Linux (i.e.,
distributions) is more flexible, easier to use, and friendly
overall. Besides, the kernel handles an insane selection of hardware, and
is able to interoperate with allmost everything else there is. No mean
feat, that one.

> Hey, that's not a bug, that's a FEATURE!

Bingo!
-- 
Horst von Brand                             vonbrand@sleipnir.valparaiso.cl
Casilla 9G, Vin~a del Mar, Chile                               +56 32 672616

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Linus Torvalds wrote:
> On Fri, 30 Nov 2001, Rik van Riel wrote:
> >
> > I'm very interested too, though I'll have to agree with Larry
> > that Linux really isn't going anywhere in particular and seems
> > to be making progress through sheer luck.
>
> Hey, that's not a bug, that's a FEATURE!

Don't forget the fact that 2.4 is the first kernel you
managed to get stable under high load since 1.2.

Both 2.0 and 2.2 didn't get stable until Alan took over
and Alan's 2.4 fork got stable some 4 months before your
2.4 tree got stable.

I think you've pretty much proven how well random
development works.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[H. Peter Anvin]

Followup to:  <20011130140613.F14710@work.bitmover.com>
By author:    Larry McVoy <lm@bitmover.com>
In newsgroup: linux.dev.kernel
> > 
> > A simple rule to remember is: when code is bad, criticize the code, not the 
> > coder.
> 
> Your priorities are upside down.  The code is more important than the
> coder, it will outlive the coder's interest in that code.  Besides,
> this isn't some touchy feely love fest, it's code.  It's suppose to
> work and work well and be maintainable.  You don't get that by being
> "nice", you get that by insisting on quality.  If being nice worked,
> we wouldn't be having this conversation.
> 

So the sensible thing to do is, again, to criticize the code, not the
coder.

There are multiple reasons for that:

a) The code is what counts.
b) People take personal attacks, well, personally.  It causes
   unnecessary bad blood.
c) There are people who will produce beautiful code one minute, and
   complete shitpiles the next one.

If a certain piece of code is a shitpile, go ahead and say so.  Please
do, however, explain why that is, and please give the maintainer a
chance to listen before being flamed publically.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Coding style - a non-issue

[Henning Schmiedehausen]

On Fri, 2001-11-30 at 19:13, Larry McVoy wrote:

> But I haven't found anything else which works as well.  I don't use that
> technique at BitMover, instead I rewrite code that I find offensive.  That's

Sounds like the thing that Mr. Gates did when Microsoft was small. Maybe
there _is_ a point in this.

	Regards
		Henning


-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Henning Schmiedehausen]

On Fri, 2001-11-30 at 18:27, Larry McVoy wrote:

Larry

> I think that if you ask around, you'll find that the pros use a coding 
> style that isn't theirs, even when writing new code.  They have evolved
> to use the prevailing style in their environment.  I know that's true for
> me, my original style was 4 space tabs, curly braces on their own line,
> etc.  I now code the way Bill Joy codes, fondly known as Bill Joy normal
> form.

I don't have to ask around. You may want to know that I work in this
industry for about 15 years and write commercial code (that is "code
that ships") since about that time (and I don't run around telling
everybody each and every time about it and what I've already done). I've
written code in a good two dozen languages (including things that really
deserve to die like Comal) and I've worked in projects from "one man
show" to "lots of people in Elbonia also work on this". So, please,
please, Larry, _STOP THE FUCK PATRONIZING OTHERS_. Actually, I try to
consider myself a "pro" in some languages and a bloody amateur in others
(like Lisp or Python). That is "pro" as in "pays his bills with writing
software".

> 
> Anyway, if you think any coding style is better than another, you completely
> miss the point.  The existing style, whatever it is, is the style you use.
> I personally despise the GNU coding style but when I make changes there,
> that's what I use because it is their source base, not mine.

We're in violent agreement here. Unfortunatly we do not agree whether it
is good to force a driver writer (which is, IMHO, most of the time a
well defined entity of one, maybe two or three source code files that
uses a well established (though sometimes changing) API to talk to the
kernel) to convert his style to the linux kernel ("our style is better
than yours, you must convert") or allow him to continue working (and
maintaining) his driver in the kernel tree in his own style. Even if his
variable names contain Uppercase letters.

The question now is, what is "amateur behaviour": Forcing this driver
writer to change or to tolerate his style in his driver? We're still
talking about a driver, not about the VM subsystem or the networking
core.

And will "putting up a list of the ten most ugly drivers with author
names" really persuade this author to change? Or simply to drop out and
write a driver for an OS that may be more tolerant?

That the core components of a large software project must adhere to a
style guide (and the Linux style guide is pretty good, because it is
_SHORT_), there is no disagreement between you and me.

	Regards
		Henning


 
-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Larry McVoy]

Henning, perhaps you can explain to me how the following isn't a 

	"I don't do XYZ"

	"XYZ"

statement?

On Fri, Nov 30, 2001 at 06:53:05PM +0100, Henning Schmiedehausen wrote:
> You may want to know that I work in this
> industry for about 15 years and write commercial code (that is "code
> that ships") since about that time (and I don't run around telling
> everybody each and every time about it and what I've already done). 

That would be the "I don't do XYZ..."

> I've
> written code in a good two dozen languages (including things that really
> deserve to die like Comal) and I've worked in projects from "one man
> show" to "lots of people in Elbonia also work on this".

And this would be the "XYZ".

If you are going to yell at people for a particular behaviour, it's really
more effective if you don't show that behaviour in the midst of your yelling,
wouldn't you agree?  Just a friendly thought.

> So, please, please, Larry, _STOP THE FUCK PATRONIZING OTHERS_.

It would appear that you find everything I say patronizing, regardless of
what it is or how it is said.  I'm sorry about that, but I'm not going
to change how I speak to suit you.  Yell all you want.  I'd suggest
that if you find my emails offensive, you add me to your kill file.

> The question now is, what is "amateur behaviour": Forcing this driver
> writer to change or to tolerate his style in his driver? We're still
> talking about a driver, not about the VM subsystem or the networking
> core.

Your approach to this whole topic is a good example of why I run my own
company and I have absolute authority to fire people at will.  If you 
worked here and you persisted with this approach, you would be fired,
without question.  I don't know how to say it more clearly, I don't
say it lightly, hiring someone, training them, all of that is a huge
investment.  One which I would discard if you couldn't fit in.  Coding
style is part of fitting in, it isn't optional in any code I pay for.

You may have a different view, that's cool, you're entitled.  But realize
that any professional software organization is unlikely to agree with you.
For whatever that is worth.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Coding style - a non-issue

[Mike Fedyk]

On Fri, Nov 30, 2001 at 10:07:29AM -0800, Larry McVoy wrote:
> Henning, perhaps you can explain to me how the following isn't a 
> 
> 	"I don't do XYZ"
> 
> 	"XYZ"
> 
> statement?
> 
> On Fri, Nov 30, 2001 at 06:53:05PM +0100, Henning Schmiedehausen wrote:
> > You may want to know that I work in this
> > industry for about 15 years and write commercial code (that is "code
> > that ships") since about that time (and I don't run around telling
> > everybody each and every time about it and what I've already done). 
> 
> That would be the "I don't do XYZ..."
> 
> > I've
> > written code in a good two dozen languages (including things that really
> > deserve to die like Comal) and I've worked in projects from "one man
> > show" to "lots of people in Elbonia also work on this".
> 
> And this would be the "XYZ".
> 
> If you are going to yell at people for a particular behaviour, it's really
> more effective if you don't show that behaviour in the midst of your yelling,
> wouldn't you agree?  Just a friendly thought.
>

He's basically complaining that you like to point out what you have done in
the past a lot.  Then goes to say that he has qualifications to prove that
his opinion should be listened to.

Not that I've been reading lkml long enough to notice...

> > So, please, please, Larry, _STOP THE FUCK PATRONIZING OTHERS_.
> 
> It would appear that you find everything I say patronizing, regardless of
> what it is or how it is said.  I'm sorry about that, but I'm not going
> to change how I speak to suit you.  Yell all you want.  I'd suggest
> that if you find my emails offensive, you add me to your kill file.
>

I don't know about you, but I wouldn't want to be in anyone's killfile.

> > The question now is, what is "amateur behaviour": Forcing this driver
> > writer to change or to tolerate his style in his driver? We're still
> > talking about a driver, not about the VM subsystem or the networking
> > core.
> 
> Your approach to this whole topic is a good example of why I run my own
> company and I have absolute authority to fire people at will.  If you 
> worked here and you persisted with this approach, you would be fired,
> without question.  I don't know how to say it more clearly, I don't
> say it lightly, hiring someone, training them, all of that is a huge
> investment.  One which I would discard if you couldn't fit in.  Coding
> style is part of fitting in, it isn't optional in any code I pay for.
>

Key words: "pay for".

This is Linux-Kernel.  Each developer is on their own on how they pay the
their bills.  The question is... Why not accept a *driver* that *works* but
the source doesn't look so good?  Maintainability? Yes.  Take the code, and
encourage better code.  Or even convert said initial submission to the
kernel style.

What really needs to happen...

Accept the driver, but also accept future submissions that *only* clean up
the comments.  It has been said that patches with comments and without code
have been notoriously droped.

mf

Re: Coding style - a non-issue

[Alexander Viro]

On Fri, 30 Nov 2001, Mike Fedyk wrote:

> This is Linux-Kernel.  Each developer is on their own on how they pay the
> their bills.  The question is... Why not accept a *driver* that *works* but
> the source doesn't look so good?

Because this "works" may very well include exploitable buffer overruns in
kernel mode.  I had seen that - ioctl() assuming that nobody would pass
it deliberately incorrect arguments and doing something like
	copy_from_user(&foo, arg, sizeof(foo));
	copy_from_user(bar, foo.addr, foo.len);

The problem being, seeing what really happens required half an hour of
wading through the shitload of #defines.  _After_ seeing copy_from_user()
in a macro during greap over the tree - that's what had triggered the
further search.
 
> What really needs to happen...
> 
> Accept the driver, but also accept future submissions that *only* clean up
> the comments.  It has been said that patches with comments and without code
> have been notoriously droped.

Commented pile of shit is still nothing but a pile of shit.  If you comment
Netscape to hell and back it will still remain a festering dungpile.  Same
for NT, GNOME, yodda, yodda...

Re: Coding style - a non-issue

[Rusty Russell]

On Sat, 1 Dec 2001 00:14:54 -0500 (EST)
Alexander Viro <viro@math.psu.edu> wrote:

> 
> 
> On Fri, 30 Nov 2001, Mike Fedyk wrote:
> 
> > This is Linux-Kernel.  Each developer is on their own on how they pay the
> > their bills.  The question is... Why not accept a *driver* that *works* but
> > the source doesn't look so good?
> 
> Because this "works" may very well include exploitable buffer overruns in
> kernel mode.  I had seen that - ioctl() assuming that nobody would pass

And: bad code spreads.  Anyone who has done infrastructure change in the
kernel sees this: people copy (presumed) working code.

Hence I now lean towards "change EVERYTHING" rather than "wrap old source, add
new", and "fix even if not broken", eg. my "set_bit needs a long" patch which
also changed x86-specific code (where it doesn't matter).

Cheers,
Rusty.
-- 
  Anyone who quotes me is an idiot. -- Rusty Russell.

Re: Coding style - a non-issue

[Alan Cox]

> Flaming about coding style is about as pointless as flaming someone
> because he supports another sports team. There is no universal accepted
> coding style. Not even in C.

The kernel has an accepted coding style, both the documented and the
tradition part of it. Using that makes life a lot lot easier for maintaining
the code. Enforcing it there is a good idea, except for special cases
(headers shared with NT has been one example of that).

There are also some nice tools around that will do the first stage import of
a Hungarian NT'ese driver and linuxise it. 

Alan

Re: Coding style - a non-issue

[Alexander Viro]

On 30 Nov 2001, Henning Schmiedehausen wrote:

> issue. Code that you consider ugly as hell may be seen as "easily
> understandable and maintainable" by the author. If it works and has no
> bugs, so what? Just because it is hard for you and me to understand (cf.

... it goes without peer review for years.  And that means bugs.

Fact of life: we all suck at reviewing our own code.  You, me, Ken Thompson,
anybody - we tend to overlook bugs in the code we'd written.  Depending on
the skill we can compensate - there are technics for that, but it doesn't
change the fact that review by clued people who didn't write the thing
tends to show bugs we'd missed for years.

If you really don't know that by your own experience - you don't _have_
experience.  There is a damn good reason for uniform style within a
project: peer review helps.  I've lost the count of bugs in the drivers
that I'd found just grepping the tree.  Even on that level review catches
tons of bugs.  And I have no reason to doubt that authors of respective
drivers would fix them as soon as they'd see said bugs.

"It's my code and I don't care if nobody else can read it" is an immediate
firing offense in any sane place.  It may be OK in academentia, but in the
real life it's simply unacceptable.

It's all nice and dandy to shed tears for poor, abused, well-meaning company
that had made everyone happy by correct but unreadable code and now gets
humiliated by mean ingrates.  Nice image, but in reality the picture is
quite different.  Code _is_ buggy.  That much is a given, regardless of
the origin of that code.  The only question is how soon are these bugs
fixed.  And that directly depends on the amount of efforts required to
read through that code.

Sigh...  Ironic that _you_ recommend somebody to grow up - I would expect
the level of naivety you'd demonstrated from a CS grad who'd never worked
on anything beyond his toy project.  Not from somebody adult.

Re: Coding style - a non-issue

[Henning Schmiedehausen]

On Fri, 2001-11-30 at 18:55, Alexander Viro wrote:

Hi,

> On 30 Nov 2001, Henning Schmiedehausen wrote:
> 
> > issue. Code that you consider ugly as hell may be seen as "easily
> > understandable and maintainable" by the author. If it works and has no
> > bugs, so what? Just because it is hard for you and me to understand (cf.
> 
> ... it goes without peer review for years.  And that means bugs.

That's right. And I didn't say, that _this is a good thing_. The
question was (and is IMHO), "do we put such code into a "hall of driver
writer shame" or do we either just reject the code from the kernel tree
or do we help "the poor misunderstood vendor" to convert.

Simply flaming them down will definitely not help. As someone with your
experience should know, too.

> Sigh...  Ironic that _you_ recommend somebody to grow up - I would expect
> the level of naivety you'd demonstrated from a CS grad who'd never worked
> on anything beyond his toy project.  Not from somebody adult.

You're welcome.

I'm willing to give you a bet: You put up such a "hall of shame" and we
will see what comes first:

a) media echo that "linux core developers start insulting code
committers"

or

b) vendors start cleaning up their code.



	Regards
		Henning

 
-- 
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen       -- Geschaeftsfuehrer
INTERMETA - Gesellschaft fuer Mehrwertdienste mbH     hps@intermeta.de

Am Schwabachgrund 22  Fon.: 09131 / 50654-0   info@intermeta.de
D-91054 Buckenhof     Fax.: 09131 / 50654-20   

Re: Coding style - a non-issue

[Pavel Machek]

Hi!

> > Sigh...  Ironic that _you_ recommend somebody to grow up - I would expect
> > the level of naivety you'd demonstrated from a CS grad who'd never worked
> > on anything beyond his toy project.  Not from somebody adult.
> 
> You're welcome.
> 
> I'm willing to give you a bet: You put up such a "hall of shame" and we
> will see what comes first:
> 
> a) media echo that "linux core developers start insulting code
> committers"
> 
> or
> 
> b) vendors start cleaning up their code.

Question is... what hurts us more. Bad press or bad code? I guess bad
code is worse...
								Pavel
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: Coding style - a non-issue

[Alan Cox]

> > a) media echo that "linux core developers start insulting code
> > committers"
> > 
> > b) vendors start cleaning up their code.
> 
> Question is... what hurts us more. Bad press or bad code? I guess bad
> code is worse...

What would be much much more constructive isnt quite a hall of shame - its
to build a set of pages that take problem drivers and quote chunks of them
with an explanation of _why_ it is wrong, what should be used instead and
possible the "after" code if it also gets cleaned up.

That way people coming along actually learn something from it. Anyone can
be a critic, its rather harder and much more valuable to be a critic that
actually has positive impacts on what you criticize


Alan

Re: Coding style - a non-issue

[Dave Jones]

On Sun, 2 Dec 2001, Alan Cox wrote:

> What would be much much more constructive isnt quite a hall of shame - its
> to build a set of pages that take problem drivers and quote chunks of them
> with an explanation of _why_ it is wrong, what should be used instead and
> possible the "after" code if it also gets cleaned up.

I planned to do something like this for the kernel-janitor project.
The janitor todo-list is aparently too terse for some people, and
having a perl script to point out problems wasn't enough. Maybe having the
script point to a webpage explaining /why/ xxxx needs changing would be
more useful.

The current TODO list there goes halfway toward this, but needs
expanding, more explanations etc.. Any contributions more than
welcomed.

regards,
Dave.

-- 
| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Alexander Viro wrote:

> Fact of life: we all suck at reviewing our own code.  You, me, Ken
> Thompson, anybody - we tend to overlook bugs in the code we'd written.
> Depending on the skill we can compensate - there are technics for
> that, but it doesn't change the fact that review by clued people who
> didn't write the thing tends to show bugs we'd missed for years.

Absolutely agreed.  Note that this goes hand in hand with
another issue, no matter how scary it may sound to other
people ... <drum roll>

	DOCUMENTATION

Because, without documentation we can only see what code
does and not what it's supposed to do.

This in turn means other people cannot identify bugs in
the code, simply because they're not sure what the code
is supposed to do.

regards,

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Jeff Garzik]

Diverse coding styles in the Linux kernel create long term maintenance
problems.  End of story.

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Coding style - a non-issue

[Keith Owens]

On 30 Nov 2001 18:15:28 +0100, 
Henning Schmiedehausen <hps@intermeta.de> wrote:
>Are you willing to judge "ugliness" of kernel drivers? What is ugly?
>... Is the aic7xxx driver ugly because it needs libdb ? ...

Yes, and no, mainly yes.  Requiring libdb, lex and yacc to to generate
the firmware is not ugly, user space programs can use any tools that
the developer needs.  I have no opinion either way about the driver
code, from what I can tell aic7xxx is a decent SCSI driver, once it is
built.

What is ugly in aic7xxx is :-

* Kludging BSD makefile style into aix7ccc/aicasm/Makefile.  It is not
  compatible with the linux kernel makefile style.

* Using a manual flag (CONFIG_AIC7XXX_BUILD_FIRMWARE) instead of
  automatically detecting when the firmware needs to be rebuilt.  Users
  who set that flag by mistake but do not have libdb, lex and yacc
  cannot compile a kernel.

* Not checking that the db.h file it picked will actually compile and
  link.

* Butchering the modules_install rule to add a special case for aic7xxx
  instead of using the same method that every other module uses.

* Including endian.h in the aic7xxx driver, but endian.h is a user
  space include.  Code that is linked into the kernel or a module
  MUST NOT include user space headers.

* Not correctly defining the dependencies between generated headers and
  the code that includes those headers.  Generated headers require
  explicit dependencies, the only reason it works is because aic7xxx ...

* Ships generated files and overwrites them under the same name.
  Shipping generated files is bad enough but is sometime necessary when
  the end user might not have the tools to build the files (libdb, lex,
  yacc).  Overwriting the shipped files under the same name is asking
  for problem with source repositories and generating spurious diffs.

All of the above problems are caused by a developer who insists on
doing his own makefile style instead of following the kernel standards
for makefiles.  Developers with their own standards are BAD!

BTW, I have made repeated offers to rewrite the aic7xx makefiles for
2.4 but the aic7xxx maintainer refuses to do so.  I _will_ rewrite them
in 2.5, as part of the kernel build 2.5 redesign.

Keith Owens, kernel build maintainer.

Re: Coding style - a non-issue

[Gérard Roudier]

Hi Keith,

When I have had to prepare a Makefile for sym-2 as a sub-directory of
drivers/scsi (sym53c8xx_2), it didn't seem to me that a non-ugly way to do
so was possible. I mean that using sub-directory for scsi drivers wasn't
expected by the normal kernel build procedure. Looking into some network
parts that wanted to do so, I only discovered hacky stuff. This left me in
the situation I had to do this in an ugly way.

As you cannot ignore the scsi driver directory is a mess since years due
to too many sources files in an single directory. Will such ugly-ness be
cleaned up in linux-2.5?

By the way, in my opinion, a software that is as ugly as you describe but
not more looks excellentware to me. :-)

  Gérard.


On Sat, 1 Dec 2001, Keith Owens wrote:

> On 30 Nov 2001 18:15:28 +0100,
> Henning Schmiedehausen <hps@intermeta.de> wrote:
> >Are you willing to judge "ugliness" of kernel drivers? What is ugly?
> >... Is the aic7xxx driver ugly because it needs libdb ? ...
>
> Yes, and no, mainly yes.  Requiring libdb, lex and yacc to to generate
> the firmware is not ugly, user space programs can use any tools that
> the developer needs.  I have no opinion either way about the driver
> code, from what I can tell aic7xxx is a decent SCSI driver, once it is
> built.
>
> What is ugly in aic7xxx is :-
>
> * Kludging BSD makefile style into aix7ccc/aicasm/Makefile.  It is not
>   compatible with the linux kernel makefile style.
>
> * Using a manual flag (CONFIG_AIC7XXX_BUILD_FIRMWARE) instead of
>   automatically detecting when the firmware needs to be rebuilt.  Users
>   who set that flag by mistake but do not have libdb, lex and yacc
>   cannot compile a kernel.
>
> * Not checking that the db.h file it picked will actually compile and
>   link.
>
> * Butchering the modules_install rule to add a special case for aic7xxx
>   instead of using the same method that every other module uses.
>
> * Including endian.h in the aic7xxx driver, but endian.h is a user
>   space include.  Code that is linked into the kernel or a module
>   MUST NOT include user space headers.
>
> * Not correctly defining the dependencies between generated headers and
>   the code that includes those headers.  Generated headers require
>   explicit dependencies, the only reason it works is because aic7xxx ...
>
> * Ships generated files and overwrites them under the same name.
>   Shipping generated files is bad enough but is sometime necessary when
>   the end user might not have the tools to build the files (libdb, lex,
>   yacc).  Overwriting the shipped files under the same name is asking
>   for problem with source repositories and generating spurious diffs.
>
> All of the above problems are caused by a developer who insists on
> doing his own makefile style instead of following the kernel standards
> for makefiles.  Developers with their own standards are BAD!
>
> BTW, I have made repeated offers to rewrite the aic7xx makefiles for
> 2.4 but the aic7xxx maintainer refuses to do so.  I _will_ rewrite them
> in 2.5, as part of the kernel build 2.5 redesign.
>
> Keith Owens, kernel build maintainer.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Clean up drivers/scsi (was: Coding style - a non-issue)

[Keith Owens]

Subject changed and cc trimmed.

On Sat, 1 Dec 2001 09:54:48 +0100 (CET), 
=?ISO-8859-1?Q?G=E9rard_Roudier?= <groudier@free.fr> wrote:
>When I have had to prepare a Makefile for sym-2 as a sub-directory of
>drivers/scsi (sym53c8xx_2), it didn't seem to me that a non-ugly way to do
>so was possible. I mean that using sub-directory for scsi drivers wasn't
>expected by the normal kernel build procedure. Looking into some network
>parts that wanted to do so, I only discovered hacky stuff. This left me in
>the situation I had to do this in an ugly way.

kbuild 2.5 is designed to cope with components that are spread over
several directories, as well as drivers in their own sub directory.
Supporting sym53c8xx_2 was just one line in drivers/scsi/Makefile.in.

select(CONFIG_SCSI_NCR53C7xx            53c7,8xx.o)
link_subdirs(sym53c8xx_2)               # HBA in its own directory
select(CONFIG_SCSI_SYM53C8XX            sym53c8xx.o)

plus drivers/scsi/sym53c8xx_2/Makefile.in.

objlink(sym53c8xx.o sym_fw.o sym_glue.o sym_hipd.o sym_malloc.o sym_misc.o
        sym_nvram.o)
select(CONFIG_SCSI_SYM53C8XX_2 sym53c8xx.o)

>As you cannot ignore the scsi driver directory is a mess since years due
>to too many sources files in an single directory. Will such ugly-ness be
>cleaned up in linux-2.5?

At some time in the 2.5 series I hope to split drivers/scsi into core,
arch independent drivers, arch specific drivers and peripheral drivers
(tape, cdrom etc.), with a top level scsi/Makefile.in that contains:

link_subdirs(core)
link_subdirs(hba)
link_subdirs(acorn)
# link_subdirs(any other arch specific scsi directories here)
link_subdirs(peripheral)

At the moment, adding a new driver means changing the middle of the
Makefile which is nasty.  It also prevents kbuild 2.5 from supporting
add on SCSI drivers in an easy manner.  With a clean split like the
above, testing a new driver outside the kernel source tree only
requires drivers/scsi/hba/Makefile.in.append containing just one line
for the new object.

Whether the drivers are all in one directory or are futher sub divided
is up to the SCSI or individual driver maintainers.  Obviously drivers
built from multiple files should be in their own sub directory.  That
still leaves a large number of drivers built from single source files,
how should they be split, if at all?  One directory per driver is
overkill, grouping by supplier breaks when the supplier is taken over,
grouping by chipset might be useful.

I must emphasise that this is just my wishlist.  It needs broad
agreement from the SCSI maintainers before we start moving the files
around.  Until there is agreement, linux-2.5 will have the same
cluttered scsi makefile, converted to kbuild 2.5 style.

Re: Coding style - a non-issue

[David S. Miller]

   From: Keith Owens <kaos@ocs.com.au>
   Date: Sat, 01 Dec 2001 12:17:03 +1100
   
   What is ugly in aic7xxx is :-

You missed:

* #undef's "current"

Re: Coding style - a non-issue

[Keith Owens]

On Sun, 02 Dec 2001 15:21:57 -0800 (PST), 
"David S. Miller" <davem@redhat.com> wrote:
>   From: Keith Owens <kaos@ocs.com.au>
>   Date: Sat, 01 Dec 2001 12:17:03 +1100
>   
>   What is ugly in aic7xxx is :-
>
>You missed:
>
>* #undef's "current"

Where?  fgrep -ir current 2.4.17-pre2/drivers/scsi/aic7xxx did not find it.

Re: Coding style - a non-issue

[Gérard Roudier]

On Mon, 3 Dec 2001, Keith Owens wrote:

> On Sun, 02 Dec 2001 15:21:57 -0800 (PST),
> "David S. Miller" <davem@redhat.com> wrote:
> >   From: Keith Owens <kaos@ocs.com.au>
> >   Date: Sat, 01 Dec 2001 12:17:03 +1100
> >
> >   What is ugly in aic7xxx is :-
> >
> >You missed:
> >
> >* #undef's "current"
>
> Where?  fgrep -ir current 2.4.17-pre2/drivers/scsi/aic7xxx did not find it.

What is ugly is "David S. Miller" ?

The 'Z' in the first name and the 'K' in the family name. :-)

  Gérard.

Re: Coding style - a non-issue

[Gérard Roudier]

On Tue, 4 Dec 2001, Gérard Roudier wrote:

>
> On Mon, 3 Dec 2001, Keith Owens wrote:
>
> > On Sun, 02 Dec 2001 15:21:57 -0800 (PST),
> > "David S. Miller" <davem@redhat.com> wrote:
> > >   From: Keith Owens <kaos@ocs.com.au>
> > >   Date: Sat, 01 Dec 2001 12:17:03 +1100
> > >
> > >   What is ugly in aic7xxx is :-
> > >
> > >You missed:
> > >
> > >* #undef's "current"
> >
> > Where?  fgrep -ir current 2.4.17-pre2/drivers/scsi/aic7xxx did not find it.
>
> What is ugly is "David S. Miller" ?
               ^^
Amusing mistake, I wanted to write 'in' instead of 'is'. :-)

>
> The 'Z' in the first name and the 'K' in the family name. :-)
>
>   Gérard.
>
>

Re: Coding style - a non-issue

[David S. Miller]

   
   On Mon, 3 Dec 2001, Keith Owens wrote:
   
   > On Sun, 02 Dec 2001 15:21:57 -0800 (PST),
   > "David S. Miller" <davem@redhat.com> wrote:
   > >   From: Keith Owens <kaos@ocs.com.au>
   > >   Date: Sat, 01 Dec 2001 12:17:03 +1100
   > >
   > >   What is ugly in aic7xxx is :-
   > >
   > >You missed:
   > >
   > >* #undef's "current"
   >
   > Where?  fgrep -ir current 2.4.17-pre2/drivers/scsi/aic7xxx did not find it.

OMFG, it's been fixed... this is amazing.  I am honestly shocked.

Re: Coding style - a non-issue

[Martin Dalecki]

Jeff Garzik wrote:
> 
> The security community has shown us time and again that public shaming
> is often the only way to motivate vendors into fixing security
> problems.  Yes, even BSD security guys do this :)
> 
> A "Top 10 ugliest Linux kernel drivers" list would probably provide
> similar motivation.

Yehh.... However some of the uglinesses results from ignorance
on behalf of the overall kernel maintainers, who don't care
to apply "cosmetic" changes to drivers, just to don't
irritate the oftes so called "maintainer". Two expierences:
ftape and mcd I'm through.... 

BTW.> ftape (for the pascal emulation) and DAC960 
(for the silly ICantReadThisCasing) 
are my personal "top ranks" in regard
of the contest for the most ugly code in the kernel...
serial.c is another one for the whole multiport support which
may be used by maybe 0.1% of the Linux users thrown on them all
and some "magic" number silliness as well...

Re: Coding style - a non-issue

[Russell King]

On Fri, Nov 30, 2001 at 06:20:40PM +0100, Martin Dalecki wrote:
> serial.c is another one for the whole multiport support which
> may be used by maybe 0.1% of the Linux users thrown on them all
> and some "magic" number silliness as well...

Magic number silliness is something that's gone with my replacement
serial drivers.  If multiport is such a problem, it can easily be
cleaned up.  I'll add that to my to do list for serial stuff.

Thanks.

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: Coding style - a non-issue

[Martin Dalecki]

Russell King wrote:
> 
> On Fri, Nov 30, 2001 at 06:20:40PM +0100, Martin Dalecki wrote:
> > serial.c is another one for the whole multiport support which
> > may be used by maybe 0.1% of the Linux users thrown on them all
> > and some "magic" number silliness as well...
> 
> Magic number silliness is something that's gone with my replacement
> serial drivers.  If multiport is such a problem, it can easily be
> cleaned up.  I'll add that to my to do list for serial stuff.

Well sombeody really cares apparently! Thank's. Any pointers
where to ahve a look at it? BTW> Did you consider ther misc device
idea? (Hooking serial at to the misc device stuff).

Re: Coding style - a non-issue

[Russell King]

On Fri, Nov 30, 2001 at 06:49:01PM +0100, Martin Dalecki wrote:
> Well sombeody really cares apparently! Thank's.

Currently its a restructuring of serial.c to allow different uart type
ports to be driven without duplicating serial.c many times over.  (the
generic_serial didn't hack it either).

> Any pointers where to ahve a look at it?

I should probably put this on a web page somewhere 8)

  :pserver:cvs@pubcvs.arm.linux.org.uk:/mnt/src/cvsroot, module serial
  (no password)

> BTW> Did you consider ther misc device idea? (Hooking serial at to the
> misc device stuff).

I just caught that comment - I'll await your reply.

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: Coding style - a non-issue

[Martin Dalecki]

Russell King wrote:
> 
> On Fri, Nov 30, 2001 at 06:49:01PM +0100, Martin Dalecki wrote:
> > Well sombeody really cares apparently! Thank's.
> 
> Currently its a restructuring of serial.c to allow different uart type
> ports to be driven without duplicating serial.c many times over.  (the
> generic_serial didn't hack it either).
> 
> > Any pointers where to ahve a look at it?
> 
> I should probably put this on a web page somewhere 8)
> 
>   :pserver:cvs@pubcvs.arm.linux.org.uk:/mnt/src/cvsroot, module serial
>   (no password)
> 
> > BTW> Did you consider ther misc device idea? (Hooking serial at to the
> > misc device stuff).
> 
> I just caught that comment - I'll await your reply.

I'm just right now looking at the code found above.
First of all: GREAT WORK! And then you are right a bit, I just don't
see too much code duplication between the particular drivers there.
However I still don't see the need to carrige the whole tty stuff just
to drive a mouse... but I don't see a solution right now.
I wouldn't be good to introduce a scatter heap of "micro" 
driver modules like the ALSA people did as well too..

However in serial/linux/drivers/serial/serial_clps711x.c
the following wonders me a bit:

#ifdef CONFIG_DEVFS_FS
	normal_name:		SERIAL_CLPS711X_NAME,
	callout_name:		CALLOUT_CLPS711X_NAME,
#else
	normal_name:		SERIAL_CLPS711X_NAME,
	callout_name:		CALLOUT_CLPS711X_NAME,
#endif

What is the ifdef supposed to be good for?

 
One other suggestion serial_code.c could be just serial.c or code.c
or main.c, since _xxxx.c should distinguish between particulart devices.
It was a bit clumy to find the entry-point for me...
And then we have in uart_register_driver:

	normal->open		= uart_open;
	normal->close		= uart_close;
	normal->write		= uart_write;
	normal->put_char	= uart_put_char;
	normal->flush_chars	= uart_flush_chars;
	normal->write_room	= uart_write_room;
	normal->chars_in_buffer	= uart_chars_in_buffer;
	normal->flush_buffer	= uart_flush_buffer;
	normal->ioctl		= uart_ioctl;
	normal->throttle	= uart_throttle;
	normal->unthrottle	= uart_unthrottle;
	normal->send_xchar	= uart_send_xchar;
	normal->set_termios	= uart_set_termios;
	normal->stop		= uart_stop;
	normal->start		= uart_start;
	normal->hangup		= uart_hangup;
	normal->break_ctl	= uart_break_ctl;
	normal->wait_until_sent	= uart_wait_until_sent;

And so on....

Why not do:

{
     static strcut tty_driver _normal = {
     open: uart_open,
     close: uart_close,
     ...
     };

     ...

     *normal = _normal;
}
...
for the static stuff first and only initialize the
dynamically calculated addresses dynamically later, like
the double refferences...
This would save already *quite a bit* of .text ;-).

Yeah I know I'm a bit perverse about optimizations....


You already do it for the callout device nearly this
way:

	/*
	 * The callout device is just like the normal device except for
	 * the major number and the subtype code.
	 */
	*callout		= *normal;
	callout->name		= drv->callout_name;
	callout->major		= drv->callout_major;
	callout->subtype	= SERIAL_TYPE_CALLOUT;
	callout->read_proc	= NULL;
	callout->proc_entry	= NULL;

Reagrds.

Re: Coding style - a non-issue

[Alan Cox]

> irritate the oftes so called "maintainer". Two expierences:
> ftape and mcd I'm through.... 

I timed the mcd maintainer out and tidied it anyway. I figured since it
wasnt being maintained nobody would scream too loudly - nobody has

> BTW.> ftape (for the pascal emulation) and DAC960 

ftape is an awkward one. Really the newer ftape4 wants merging into the
kernel but that should have happened a long time ago

> serial.c is another one for the whole multiport support which
> may be used by maybe 0.1% of the Linux users thrown on them all
> and some "magic" number silliness as well...

serial.c is a good example of the "ugly" that actually matters more, as is
floppy.c. Clean well formatted code that is stil opaque. 

Re: Coding style - a non-issue

[Martin Dalecki]

Alan Cox wrote:
> 
> > irritate the oftes so called "maintainer". Two expierences:
> > ftape and mcd I'm through....
> 
> I timed the mcd maintainer out and tidied it anyway. I figured since it
> wasnt being maintained nobody would scream too loudly - nobody has

Wenn sorry for the missconception I wanted to insult *you*, my expierenc
in regard to this is even older...

> > BTW.> ftape (for the pascal emulation) and DAC960
> 
> ftape is an awkward one. Really the newer ftape4 wants merging into the
> kernel but that should have happened a long time ago

It diverged too much from what's in the kernel since about already 3-4
years.
And I don't think that it's that much better in terms of implementation
style...
Fortunately all those floppy interface iomega streamers are 
physically obsolete by now. Plese notice that ftape4 is using a
different storage format, well this is due to the fact that the
ftape inside the kernel wasn't up to the corresponding standard
(QIO-80)...

> > serial.c is another one for the whole multiport support which
> > may be used by maybe 0.1% of the Linux users thrown on them all
> > and some "magic" number silliness as well...
> 
> serial.c is a good example of the "ugly" that actually matters more, as is
> floppy.c. Clean well formatted code that is stil opaque.

floppy.c is indeed one of the best compiler test cases around there.
But personally I would excuse floppy.c a bit becouse it's dealing with
a really awkward controller interface ;-).
serial.c should be hooked at the misc char device interface sooner or
later.
But somehow this never happened becouse nobody dared to care enough.

Re: Coding style - a non-issue

[Russell King]

On Fri, Nov 30, 2001 at 06:42:17PM +0100, Martin Dalecki wrote:
> serial.c should be hooked at the misc char device interface sooner or
> later.

Please explain.  Especially contentrate on justifing why serial interfaces
aren't a tty device.

Thanks.

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: Coding style - a non-issue

[Martin Dalecki]

Russell King wrote:
> 
> On Fri, Nov 30, 2001 at 06:42:17PM +0100, Martin Dalecki wrote:
> > serial.c should be hooked at the misc char device interface sooner or
> > later.
> 
> Please explain.  Especially contentrate on justifing why serial interfaces
> aren't a tty device.

No problem ;-).

There is the hardware: in esp. the serial controller itself - this
belongs
to misc, becouse a mouse for example doesn't have to interpret any tty
stuff
This animal belongs to the same cage as the PS/2 variant of it.
And then there is one abstraction level above it: the tty interface -
this belongs to a line discipline.

We have this split anyway already there /dev/ttyS0 and /dev/cua0 somehow
emulated on one level.

Understood?

Re: Coding style - a non-issue

[Maciej W. Rozycki]

On Fri, 30 Nov 2001, Martin Dalecki wrote:

> > Please explain.  Especially contentrate on justifing why serial interfaces
> > aren't a tty device.
> 
> No problem ;-).
> 
> There is the hardware: in esp. the serial controller itself - this
> belongs
> to misc, becouse a mouse for example doesn't have to interpret any tty
> stuff

 The same applies to the console keyboard, which is hooked to a standard
UART on certain systems as well. 

-- 
+  Maciej W. Rozycki, Technical University of Gdansk, Poland   +
+--------------------------------------------------------------+
+        e-mail: macro@ds2.pg.gda.pl, PGP key available        +

Re: Coding style - a non-issue

[Russell King]

On Fri, Nov 30, 2001 at 07:40:29PM +0100, Maciej W. Rozycki wrote:
> The same applies to the console keyboard, which is hooked to a standard
> UART on certain systems as well. 

This particular point is up for discussion between myself and James Simmons
(and other interested parties).  We're getting there...

--
Russell King (rmk@arm.linux.org.uk)                The developer of ARM Linux
             http://www.arm.linux.org.uk/personal/aboutme.html

Re: Coding style - a non-issue

[antirez]

On Fri, Nov 30, 2001 at 11:47:28AM -0500, Jeff Garzik wrote:
> The security community has shown us time and again that public shaming
> is often the only way to motivate vendors into fixing security
> problems.  Yes, even BSD security guys do this :)

It's a bit different. Usually the security community uses it
when there isn't a way to fix the code (i.e. non-free code)
or when the maintaner of the code refused to fix the issue.
Also to "expose" the security problem isn't the same as
to flame.

While not a good idea, something like a long name
for a local var isn't a big problem like a completly
broken software with huge security holes used by milions of people
every day.

The quality of the code should be ensured in a different
way. If there is a standard coding-style you should either:

1) refuse to include broken code before the programmer fix it
2) fix it yourself before the inclusion

Flames aren't a good solution imho.

-- 
Salvatore Sanfilippo <antirez@invece.org>
http://www.kyuzz.org/antirez
finger antirez@tella.alicom.com for PGP key
28 52 F5 4A 49 65 34 29 - 1D 1B F6 DA 24 C7 12 BF

Re: Coding style - a non-issue

[Paul G. Allen]

antirez wrote:
> 
> On Fri, Nov 30, 2001 at 11:47:28AM -0500, Jeff Garzik wrote:
> > The security community has shown us time and again that public shaming
> > is often the only way to motivate vendors into fixing security
> > problems.  Yes, even BSD security guys do this :)
> 
> It's a bit different. Usually the security community uses it
> when there isn't a way to fix the code (i.e. non-free code)
> or when the maintaner of the code refused to fix the issue.
> Also to "expose" the security problem isn't the same as
> to flame.
> 
> While not a good idea, something like a long name
> for a local var isn't a big problem like a completly
> broken software with huge security holes used by milions of people
> every day.
> 

A variable/function name should ALWAYS be descriptive of the
variable/function purpose. If it takes a long name, so be it. At least
the next guy looking at it will know what it is for.

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[antirez]

On Fri, Nov 30, 2001 at 10:20:43AM -0800, Paul G. Allen wrote:
> antirez wrote:
> A variable/function name should ALWAYS be descriptive of the
> variable/function purpose. If it takes a long name, so be it. At least
> the next guy looking at it will know what it is for.

I agree, but descriptive != long

for (mydearcountr = 0; mydearcounter < n; mydearcounter++)

and it was just an example. Read it as "bad coding style".

-- 
Salvatore Sanfilippo <antirez@invece.org>
http://www.kyuzz.org/antirez
finger antirez@tella.alicom.com for PGP key
28 52 F5 4A 49 65 34 29 - 1D 1B F6 DA 24 C7 12 BF

Re: Coding style - a non-issue

[Paul G. Allen]

antirez wrote:
> 
> On Fri, Nov 30, 2001 at 10:20:43AM -0800, Paul G. Allen wrote:
> > antirez wrote:
> > A variable/function name should ALWAYS be descriptive of the
> > variable/function purpose. If it takes a long name, so be it. At least
> > the next guy looking at it will know what it is for.
> 
> I agree, but descriptive != long
> 
> for (mydearcountr = 0; mydearcounter < n; mydearcounter++)
> 
> and it was just an example. Read it as "bad coding style".
> 

Well if you're counting dear in the kernel, I'd say you have more
problems than your coding style. ;)

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Jeff Garzik]

"Paul G. Allen" wrote:
> A variable/function name should ALWAYS be descriptive of the
> variable/function purpose. If it takes a long name, so be it. At least
> the next guy looking at it will know what it is for.

That's complete crap.  Human beings know and understand context, and can
use it effectively.  'idx' or 'i' or 'bh' may make perfect sense in
context.  There is absolutely no need for
JournalBHThatIsStoredAndSyncedWithSuperblock.

Kernel code like DAC960 proves that long variable names -decrease- code
readability.

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Coding style - a non-issue

[John Kodis]

On Fri, Nov 30, 2001 at 02:00:18PM -0500, Jeff Garzik wrote:

> Human beings know and understand context, and can use it
> effectively.  'idx' or 'i' or 'bh' may make perfect sense in
> context.  There is absolutely no need for
> JournalBHThatIsStoredAndSyncedWithSuperblock.

Mathematics has a rich tradition of using short variable names such as 
"pi" rather than something like "circle-circumference-to-diameter-ratio".
They keep formulas from becoming unreadably long, and have a meaning 
which is well understood in context.  While the long version may more
self-explainatory, it's the short form which is universally preferred.

-- John Kodis.

Re: Coding style - a non-issue

[Paul G. Allen]

John Kodis wrote:
> 
> On Fri, Nov 30, 2001 at 02:00:18PM -0500, Jeff Garzik wrote:
> 
> > Human beings know and understand context, and can use it
> > effectively.  'idx' or 'i' or 'bh' may make perfect sense in
> > context.  There is absolutely no need for
> > JournalBHThatIsStoredAndSyncedWithSuperblock.

JounalBH would be far better than i or idx.

> 
> Mathematics has a rich tradition of using short variable names such as
> "pi" rather than something like "circle-circumference-to-diameter-ratio".
> They keep formulas from becoming unreadably long, and have a meaning
> which is well understood in context.  While the long version may more
> self-explainatory, it's the short form which is universally preferred.
> 

While 'pi', 'e', 'theta', 'phi', etc. are universally understood, things
like 'i', 'a', and 'idx' are not. I can use these for anything I want
and even for more than one thing, and they say nothing about what they
are for. 'i', 'j', etc. are fine as loop counters and array indexes
where their meaning is apparent by context, but are _not_ fine in other
situations. You (or the person that wrote the code) may think that the
name is perfectly fine, but someone else that thinks a bit different may
not.

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Kai Henningsen]

pgallen@randomlogic.com (Paul G. Allen)  wrote on 30.11.01 in <3C07EBB9.CF5EB85E@randomlogic.com>:

> John Kodis wrote:

> > Mathematics has a rich tradition of using short variable names such as
> > "pi" rather than something like "circle-circumference-to-diameter-ratio".
> > They keep formulas from becoming unreadably long, and have a meaning
> > which is well understood in context.  While the long version may more
> > self-explainatory, it's the short form which is universally preferred.

> While 'pi', 'e', 'theta', 'phi', etc. are universally understood, things
> like 'i', 'a', and 'idx' are not.

I'd certainly call 'i' well understood in both math and computing. In  
math, 'i' is what engineers call 'j' (i*i == -1), and in computing, 'i'  
('j', 'k', ...) is a counter for loops (some variant of int) that don't  
exceed about a screenful.

> I can use these for anything I want
> and even for more than one thing,

Of course, if you use them differently from what the convention is, *then*  
you are in trouble.

> and they say nothing about what they
> are for. 'i', 'j', etc. are fine as loop counters and array indexes
> where their meaning is apparent by context, but are _not_ fine in other
> situations. You (or the person that wrote the code) may think that the
> name is perfectly fine, but someone else that thinks a bit different may
> not.

Yup.


MfG Kai

Re: Coding style - a non-issue

[john slee]

On Fri, Nov 30, 2001 at 12:27:37PM -0800, Paul G. Allen wrote:
> While 'pi', 'e', 'theta', 'phi', etc. are universally understood, things
> like 'i', 'a', and 'idx' are not. I can use these for anything I want
> and even for more than one thing, and they say nothing about what they
> are for. 'i', 'j', etc. are fine as loop counters and array indexes
> where their meaning is apparent by context, but are _not_ fine in other

the meaning of 'i', 'j', 'k', is universal.  show me a real world
program (or programming book!) not from redmond that doesn't use these
names for loop counters.

j.

-- 
R N G G   "Well, there it goes again... And we just sit 
 I G G G   here without opposable thumbs." -- gary larson

Re: Coding style - a non-issue

[Paul G. Allen]

john slee wrote:
> 
> On Fri, Nov 30, 2001 at 12:27:37PM -0800, Paul G. Allen wrote:
> > While 'pi', 'e', 'theta', 'phi', etc. are universally understood, things
> > like 'i', 'a', and 'idx' are not. I can use these for anything I want
> > and even for more than one thing, and they say nothing about what they
> > are for. 'i', 'j', etc. are fine as loop counters and array indexes
> > where their meaning is apparent by context, but are _not_ fine in other
> 
> the meaning of 'i', 'j', 'k', is universal.  show me a real world
> program (or programming book!) not from redmond that doesn't use these
> names for loop counters.
> 

If you'd been paying attention, I discounted simple loop counters in an
early post.

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Pavel Machek]

Hi!

> >> Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
> >> wankers (both individual and coprorat ones) responsible, their code and
> >> commentary on said code...
> 
> >Please, please, please, I'm begging you, please do this.  It's the only way
> >people learn quickly.  Being nice is great, but nothing works faster than 
> >a cold shower of public humiliation :-)
> 
> Cool. I can really see it. "foo inc." releases a GPL driver for Linux
> and gets publically humiliated in the "linux source code hall of
> shame". Perfect. I can hear the laughter from Redmond over here (and
> it is 12000km away).
> 
> Press release:
> 
> "If you support Linux, you may get flamed and humiliated in public for
> doing so. Don't do it."

And what? Bad driver is better than proprietary hardware, still good
driver is better than good driver.

And yes, I'd like to see that hall of shame.

								Pavel
-- 
"I do not steal MS software. It is not worth it."
                                -- Pavel Kankovsky

Re: Coding style - a non-issue

[David S. Miller]

   From: Alexander Viro <viro@math.psu.edu>
   Date: Wed, 28 Nov 2001 19:17:42 -0500 (EST)
   
   indent does _not_ solve the problem of:

Al, I think you just described the Intel e100 driver.
:-)

Re: Coding style - a non-issue

[Petko Manolov]

Alexander Viro wrote:

> 
>>Someone who cares, come up with an indentrc for the kernel code, and get it
>>into Documentation/CodingStyle

...

> indent does _not_ solve the problem of:
...


I quite liked it, applies perfectly for some people i know... :-)

Anyway, in Lindent you'll see "-psl" (--procname-start-lines) option
which contradict with what is written in CodingStyle i.e:

int my_proc(void)
{
...
}

which I personally prefer.


		Petko

Re: Coding style - a non-issue

[Matthias Andree]

On Wed, 28 Nov 2001, Alexander Viro wrote:

> Al, -><- close to setting up a Linux Kernel Hall of Shame - one with names of
> wankers (both individual and coprorat ones) responsible, their code and
> commentary on said code...

Oh, can I vote for someone spoiling outside the Kernel? I have a
candidate for that one.

Seriously, don't waste your *p_time on that except people resist any
hints to CodingStyleIsForLameAssHackersIWantMyEDIT.COM for an extended
amount of *p_time.

-- 
Matthias Andree

"They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety."         Benjamin Franklin

Re: Coding style - a non-issue

[Paul G. Allen]

Peter Waltenberg wrote:
> 
> The problem was solved years ago.
> 
> "man indent"
> 
> Someone who cares, come up with an indentrc for the kernel code, and get it
> into Documentation/CodingStyle
> If the maintainers run all new code through indent with that indentrc
> before checkin, the problem goes away.
> The only one who'll incur any pain then is a code submitter who didn't
> follow the rules. (Exactly the person we want to be in pain ;)).
> 
> Then we can all get on with doing useful things.
> 

IMEO, there is but one source as reference for coding style: A book by
the name of "Code Complete". (Sorry, I can't remember the author and I
no longer have a copy. Maybe my Brother will chime in here and fill in
the blanks since he still has his copy.)

Outside of that, every place I have worked as a programmer, with a team
of programmers, had a style that was adhered to almost religiously. In
many cases the style closely followed "Code Complete". In the case of
the kernel, as Alan and others have mentioned, there IS a Linux kernel
coding style.

In 99% of the Linux code I have seen, the style does indeed "suck". Why?
Consider a new coder coming in for any given task. S/he knows nothing
about the kernel and needs to get up to speed quickly. S/he starts
browsing the source - the ONLY definitive explanation of what it does
and how it works - and finds:

 - Single letter variable names that aren't simple loop counters and
must ask "What the h*** are these for?"
 - No function/file comment headers explaining what the purpose of the
function/file is.
 - Very few comments at all, which is not necessarily bad except...
 - The code is not self documenting and without comments it takes an
hour to figure out what function Foo() does.
 - Opening curly braces at the end of a the first line of a large code
block making it extremely difficult to find where the code block begins
or ends.
 - Short variable/function names that someone thinks is descriptive but
really isn't.
 - Inconsistent coding style from one file to the next.
 - Other problems.

After all, the kernel must be maintained by a number of people and those
people will come and go. The only real way to keep bugs at a minimum,
efficiency at a maximum, and the learning curve for new coders
acceptable is consistent coding style and code that is easily
maintained. The things I note above are not a means to that end. Sure,
maybe Bob, the designer and coder of bobsdriver.o knows the code inside
and out without need of a single comment or descriptive
function/variable name, but what happens when Bob can no longer maintain
it? It's 10,000 lines of code, the kernel is useless without it, it
broke with kernel 2.6.0, and Joe, the new maintainer of bobsdriver.o, is
having a hell of a time figuring out what the damn thing does.

An extreme case? Maybe, but how many times does someone come in to
development and have to spend more hours than necessary trying to figure
out how things work (or are supposed to work) instead of actually
writing useful code?

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[John H. Robinson, IV]

On Fri, Nov 30, 2001 at 10:15:41AM -0800, Paul G. Allen wrote:
> 
> IMEO, there is but one source as reference for coding style: A book by
> the name of "Code Complete". (Sorry, I can't remember the author and I
> no longer have a copy. Maybe my Brother will chime in here and fill in
> the blanks since he still has his copy.)

Code Complete: A Practical Handbook of Software Construction.
Redmond, Wa.: Microsoft Press, 880 pages, 1993.
Retail price: $35.
ISBN: 1-55615-484-4.  

Steve McConnell

source: http://www.construx.com/stevemcc/cc.htm

-john

Re: Coding style - a non-issue

[Paul G. Allen]

"John H. Robinson, IV" wrote:
> 
> On Fri, Nov 30, 2001 at 10:15:41AM -0800, Paul G. Allen wrote:
> >
> > IMEO, there is but one source as reference for coding style: A book by
> > the name of "Code Complete". (Sorry, I can't remember the author and I
> > no longer have a copy. Maybe my Brother will chime in here and fill in
> > the blanks since he still has his copy.)
> 
> Code Complete: A Practical Handbook of Software Construction.
> Redmond, Wa.: Microsoft Press, 880 pages, 1993.
> Retail price: $35.
> ISBN: 1-55615-484-4.
> 

Thanks John. You beat my bro. to it. Of course, he's probably still in
bed since it's not even noon yet. :)

(Note to self: Order a new copy of the book. I should have done it last
night when I ordered 3 other programming books. :/)

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Nestor Florez]

Book     : Code Complete
Author   : Steve McConnell
Publisher: Microsoft Press

Nestor :-)

----- Original Message -----
From: "Paul G. Allen" <pgallen@randomlogic.com>
To: "Linux kernel developer's mailing list" <linux-kernel@vger.kernel.org>;
<kplug-list@kernel-panic.org>; <kplug-lpsg@kernel-panic.org>
Sent: Friday, November 30, 2001 10:15 AM
Subject: Re: Coding style - a non-issue


> Peter Waltenberg wrote:
> >
> > The problem was solved years ago.
> >
> > "man indent"
> >
> > Someone who cares, come up with an indentrc for the kernel code, and get
it
> > into Documentation/CodingStyle
> > If the maintainers run all new code through indent with that indentrc
> > before checkin, the problem goes away.
> > The only one who'll incur any pain then is a code submitter who didn't
> > follow the rules. (Exactly the person we want to be in pain ;)).
> >
> > Then we can all get on with doing useful things.
> >
>
> IMEO, there is but one source as reference for coding style: A book by
> the name of "Code Complete". (Sorry, I can't remember the author and I
> no longer have a copy. Maybe my Brother will chime in here and fill in
> the blanks since he still has his copy.)
>
> Outside of that, every place I have worked as a programmer, with a team
> of programmers, had a style that was adhered to almost religiously. In
> many cases the style closely followed "Code Complete". In the case of
> the kernel, as Alan and others have mentioned, there IS a Linux kernel
> coding style.
>
> In 99% of the Linux code I have seen, the style does indeed "suck". Why?
> Consider a new coder coming in for any given task. S/he knows nothing
> about the kernel and needs to get up to speed quickly. S/he starts
> browsing the source - the ONLY definitive explanation of what it does
> and how it works - and finds:
>
>  - Single letter variable names that aren't simple loop counters and
> must ask "What the h*** are these for?"
>  - No function/file comment headers explaining what the purpose of the
> function/file is.
>  - Very few comments at all, which is not necessarily bad except...
>  - The code is not self documenting and without comments it takes an
> hour to figure out what function Foo() does.
>  - Opening curly braces at the end of a the first line of a large code
> block making it extremely difficult to find where the code block begins
> or ends.
>  - Short variable/function names that someone thinks is descriptive but
> really isn't.
>  - Inconsistent coding style from one file to the next.
>  - Other problems.
>
> After all, the kernel must be maintained by a number of people and those
> people will come and go. The only real way to keep bugs at a minimum,
> efficiency at a maximum, and the learning curve for new coders
> acceptable is consistent coding style and code that is easily
> maintained. The things I note above are not a means to that end. Sure,
> maybe Bob, the designer and coder of bobsdriver.o knows the code inside
> and out without need of a single comment or descriptive
> function/variable name, but what happens when Bob can no longer maintain
> it? It's 10,000 lines of code, the kernel is useless without it, it
> broke with kernel 2.6.0, and Joe, the new maintainer of bobsdriver.o, is
> having a hell of a time figuring out what the damn thing does.
>
> An extreme case? Maybe, but how many times does someone come in to
> development and have to spend more hours than necessary trying to figure
> out how things work (or are supposed to work) instead of actually
> writing useful code?
>
> PGA
> --
> Paul G. Allen
> UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
> Akamai Technologies, Inc.
> www.akamai.com

Re: Coding style - a non-issue

[Jeff Garzik]

"Paul G. Allen" wrote:
> IMEO, there is but one source as reference for coding style: A book by
> the name of "Code Complete". (Sorry, I can't remember the author and I
> no longer have a copy. Maybe my Brother will chime in here and fill in
> the blanks since he still has his copy.)

Hungarian notation???

That was developed by programmers with apparently no skill to
see/remember how a variable is defined.  IMHO in the Linux community
it's widely considered one of the worst coding styles possible.


> Outside of that, every place I have worked as a programmer, with a team
> of programmers, had a style that was adhered to almost religiously.

yes

> In 99% of the Linux code I have seen, the style does indeed "suck". Why?
> Consider a new coder coming in for any given task. S/he knows nothing
> about the kernel and needs to get up to speed quickly. S/he starts
> browsing the source - the ONLY definitive explanation of what it does
> and how it works - and finds:

99% is far and above the level of suck defined by most :)


>  - Single letter variable names that aren't simple loop counters and
> must ask "What the h*** are these for?"
>  - No function/file comment headers explaining what the purpose of the
> function/file is.
>  - Very few comments at all, which is not necessarily bad except...
>  - The code is not self documenting and without comments it takes an
> hour to figure out what function Foo() does.

We could definitely use a ton more comments... patches accepted.


>  - Opening curly braces at the end of a the first line of a large code
> block making it extremely difficult to find where the code block begins
> or ends.

use a decent editor


>  - Short variable/function names that someone thinks is descriptive but
> really isn't.

not all variable names need their purpose obvious to complete newbies. 
sometimes it takes time to understand the code's purpose, in which case
the variable names become incredibly descriptive.


> After all, the kernel must be maintained by a number of people and those
> people will come and go. The only real way to keep bugs at a minimum,
> efficiency at a maximum, and the learning curve for new coders
> acceptable is consistent coding style and code that is easily
> maintained. The things I note above are not a means to that end. Sure,
> maybe Bob, the designer and coder of bobsdriver.o knows the code inside
> and out without need of a single comment or descriptive
> function/variable name, but what happens when Bob can no longer maintain
> it? It's 10,000 lines of code, the kernel is useless without it, it
> broke with kernel 2.6.0, and Joe, the new maintainer of bobsdriver.o, is
> having a hell of a time figuring out what the damn thing does.

yes

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Coding style - a non-issue

[Paul G. Allen]

Jeff Garzik wrote:
> 
> 
> We could definitely use a ton more comments... patches accepted.
> 

I've actually thought of doing just that. :)

Alas, I'm too busy coding other things right now, so kernel stuff
(unless I need something to make the other project I'm working on
work/work better) will just have to wait. Hell, I'm even behind 2 or 3
kernel versions on the documentation I've been putting on my web site.
:(

> >  - Opening curly braces at the end of a the first line of a large code
> > block making it extremely difficult to find where the code block begins
> > or ends.
> 
> use a decent editor

A person shouldn't _need_ a decent editor to pick out the beginning/end
of a code block (or anything else for that matter). The problem is
exacerbated when such a block contains other blocks and quickly picking
out where each begins/ends becomes tiresome. I _do_ have excellent
editors, IDEs, and source code browsers and have used many different
kinds in many different jobs. They still can not replace what the human
eye and mind perceive.

> 
> >  - Short variable/function names that someone thinks is descriptive but
> > really isn't.
> 
> not all variable names need their purpose obvious to complete newbies.
> sometimes it takes time to understand the code's purpose, in which case
> the variable names become incredibly descriptive.

It should not take "time" to discover the purpose of _any_ variable or
function, or file, whether newbie or not. The point of coding is to
solve a problem, not perform an excersise in deductive or logical
reasoning before the problem (which usually involves further logical
reasoning) can be solved.

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Jeff Garzik]

"Paul G. Allen" wrote:
> It should not take "time" to discover the purpose of _any_ variable or
> function, or file, whether newbie or not. The point of coding is to
> solve a problem, not perform an excersise in deductive or logical
> reasoning before the problem (which usually involves further logical
> reasoning) can be solved.

Part of the problem being solved by Linux kernel code is efficient long
term maintenance and efficient peer review.  Overly verbose code does
not solve these problems.

	Jeff


-- 
Jeff Garzik      | Only so many songs can be sung
Building 1024    | with two lips, two lungs, and one tongue.
MandrakeSoft     |         - nomeansno

Re: Coding style - a non-issue

[David Weinehall]

On Fri, Nov 30, 2001 at 12:06:43PM -0800, Paul G. Allen wrote:

[snip]
> A person shouldn't _need_ a decent editor to pick out the beginning/end
> of a code block (or anything else for that matter). The problem is
> exacerbated when such a block contains other blocks and quickly picking
> out where each begins/ends becomes tiresome. I _do_ have excellent
> editors, IDEs, and source code browsers and have used many different
> kinds in many different jobs. They still can not replace what the human
> eye and mind perceive.

Uhhhm, knowing when a code block begins? Usually you'll notice this from
the indentation. It's quite hard not to notice a tabsized shift
to the right...

[snip]


/David Weinehall
  _                                                                 _
 // David Weinehall <tao@acc.umu.se> /> Northern lights wander      \\
//  Maintainer of the v2.0 kernel   //  Dance across the winter sky //
\>  http://www.acc.umu.se/~tao/    </   Full colour fire           </

Re: Coding style - a non-issue

[Paul G. Allen]

David Weinehall wrote:
> 
> On Fri, Nov 30, 2001 at 12:06:43PM -0800, Paul G. Allen wrote:
> 
> [snip]
> > A person shouldn't _need_ a decent editor to pick out the beginning/end
> > of a code block (or anything else for that matter). The problem is
> > exacerbated when such a block contains other blocks and quickly picking
> > out where each begins/ends becomes tiresome. I _do_ have excellent
> > editors, IDEs, and source code browsers and have used many different
> > kinds in many different jobs. They still can not replace what the human
> > eye and mind perceive.
> 
> Uhhhm, knowing when a code block begins? Usually you'll notice this from
> the indentation. It's quite hard not to notice a tabsized shift
> to the right...
> 

Whitespace can be placed almost anywhere and the program will still
compile. It can even be removed altogether. The only thing that keeps a
program legible is proper formatting. It's real damn easy to miss a
brace when the formatting is poor. And real easy to spend an hour trying
to figure out where that missing brace goes, that is after the hour you
spent figuring out that it was missing in the first place.

I guess some people Just Don't Get It. Some people just do not know how
to write maintainable code.

PGA
-- 
Paul G. Allen
UNIX Admin II ('til Dec. 3)/FlUnKy At LaRgE (forever!)
Akamai Technologies, Inc.
www.akamai.com

Re: Coding style - a non-issue

[Tracy R Reed]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Sat, Dec 01, 2001 at 01:29:14PM -0800, Paul G. Allen wrote:
> compile. It can even be removed altogether. The only thing that keeps a
> program legible is proper formatting. It's real damn easy to miss a
> brace when the formatting is poor. And real easy to spend an hour trying

And this is why Python has chosen the format that it has.

-- 
Tracy Reed      http://www.ultraviolet.org
"Every artist is a cannibal, every poet is a thief.
 They all kill their inspiration, and sing about the grief." - U2

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: Coding style - a non-issue

[Mike Fedyk]

On Sat, Dec 01, 2001 at 01:29:14PM -0800, Paul G. Allen wrote:
> David Weinehall wrote:
> > 
> > On Fri, Nov 30, 2001 at 12:06:43PM -0800, Paul G. Allen wrote:
> > 
> > [snip]
> > > A person shouldn't _need_ a decent editor to pick out the beginning/end
> > > of a code block (or anything else for that matter). The problem is
> > > exacerbated when such a block contains other blocks and quickly picking
> > > out where each begins/ends becomes tiresome. I _do_ have excellent
> > > editors, IDEs, and source code browsers and have used many different
> > > kinds in many different jobs. They still can not replace what the human
> > > eye and mind perceive.
> > 
> > Uhhhm, knowing when a code block begins? Usually you'll notice this from
> > the indentation. It's quite hard not to notice a tabsized shift
> > to the right...
> > 
> 
> Whitespace can be placed almost anywhere and the program will still
> compile. It can even be removed altogether. The only thing that keeps a
> program legible is proper formatting. It's real damn easy to miss a
> brace when the formatting is poor. And real easy to spend an hour trying
> to figure out where that missing brace goes, that is after the hour you
> spent figuring out that it was missing in the first place.
>

Then when you get your hands on code like this you have two bugs to fix:

1) The origional problem you wanted to code up

2) Fix the formatting.

I suggest (yes, more work) that you run the code through the code normalizer
for that project and then look at the code produced from that.

That way, you will be able to see the code blocks in a standard way.  As you
look at the formatted code, you can program in the old indentation format.

When you're done, you have two patches.  One with the origional fix, and
another with the formatting improvements.

You'll probably want to get a concensus of what the other developers in the
project have agreed upon before you submit the formatting patch.  Much the
way this thread may turn out...

mf

Re: Coding style - a non-issue

[H. Peter Anvin]

Followup to:  <3C07D669.6C234598@mandrakesoft.com>
By author:    Jeff Garzik <jgarzik@mandrakesoft.com>
In newsgroup: linux.dev.kernel
>
> "Paul G. Allen" wrote:
> > IMEO, there is but one source as reference for coding style: A book by
> > the name of "Code Complete". (Sorry, I can't remember the author and I
> > no longer have a copy. Maybe my Brother will chime in here and fill in
> > the blanks since he still has his copy.)
> 
> Hungarian notation???
> 
> That was developed by programmers with apparently no skill to
> see/remember how a variable is defined.  IMHO in the Linux community
> it's widely considered one of the worst coding styles possible.
> 

Indeed.  What can you say for a technique which basically promotes
*reducing* abstraction and information hiding?

There is a reason why the Win64 ABI uses the same "int" and "long" as
Win32... (both are 32 bits.)  They added meaningless abstractions, and
didn't add abstractions where they needed to...

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Coding style - a non-issue

[Kai Henningsen]

hpa@zytor.com (H. Peter Anvin)  wrote on 30.11.01 in <9u8qtu$u2b$1@cesium.transmeta.com>:

> By author:    Jeff Garzik <jgarzik@mandrakesoft.com>

> > "Paul G. Allen" wrote:
> > > IMEO, there is but one source as reference for coding style: A book by
> > > the name of "Code Complete". (Sorry, I can't remember the author and I
> > > no longer have a copy. Maybe my Brother will chime in here and fill in
> > > the blanks since he still has his copy.)
> >
> > Hungarian notation???
> >
> > That was developed by programmers with apparently no skill to
> > see/remember how a variable is defined.  IMHO in the Linux community
> > it's widely considered one of the worst coding styles possible.
> >
>
> Indeed.  What can you say for a technique which basically promotes
> *reducing* abstraction and information hiding?
>
> There is a reason why the Win64 ABI uses the same "int" and "long" as
> Win32... (both are 32 bits.)  They added meaningless abstractions, and
> didn't add abstractions where they needed to...

Well, that doesn't necessarily make the idea itself completely crap just  
because the standard implementation is.

Sure, calling a variable "I point to a char! And by the way, I'm named  
Fritz" may be a bad idea, but in some situations it may well make sense to  
say that a certain variable is a container and another is a lock.

And it seems that sometimes, the kernel does just that. (pagecache_lock,  
page_list, just picking two fast examples from 2.4.0 mm)

But lpfilename is, indeed, fucking insane.


MfG Kai

Re: Coding style - a non-issue

[Andrew Morton]

Jeff Garzik wrote:
> 
> We could definitely use a ton more comments... patches accepted.
> 

Too late.  Useful comments go in during, or even before the code.

While we're on the coding style topic: ext2_new_block.

-

Re: Coding style - a non-issue

[Alexander Viro]

On Fri, 30 Nov 2001, Andrew Morton wrote:

> Jeff Garzik wrote:
> > 
> > We could definitely use a ton more comments... patches accepted.
> > 
> 
> Too late.  Useful comments go in during, or even before the code.
> 
> While we're on the coding style topic: ext2_new_block.

Yes.  I hope that new variant (see balloc.c,v) gets the thing into
better form, but then I'm obviously biased...

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Andrew Morton wrote:
> Jeff Garzik wrote:
> >
> > We could definitely use a ton more comments... patches accepted.
>
> Too late.  Useful comments go in during, or even before the code.

Indeed, patches with comment updates tend to get
discarded by Linus. ;(

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Re: Coding style - a non-issue

[Rik van Riel]

On Fri, 30 Nov 2001, Jeff Garzik wrote:
> "Paul G. Allen" wrote:
> > IMEO, there is but one source as reference for coding style: A book by
> > the name of "Code Complete". (Sorry, I can't remember the author and I
> > no longer have a copy. Maybe my Brother will chime in here and fill in
> > the blanks since he still has his copy.)
>
> Hungarian notation???
>
> That was developed by programmers with apparently no skill to
> see/remember how a variable is defined.  IMHO in the Linux community
> it's widely considered one of the worst coding styles possible.

If your functions are so large that you need hungarian
notation to figure out the type of each variable, chances
are forgetting the variable type isn't the biggest of your
problems ;)

Rik
-- 
Shortwave goes a long way:  irc.starchat.net  #swl

http://www.surriel.com/		http://distro.conectiva.com/

Wow! Is memory ever cheap!

[Larry McVoy]

This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS.  The
DIMMS are not ECC, but we use BitKeeper here and it tells us when we
have bad DIMMS.

Guess what the memory cost?  $396.58 shipped to my door, second day air,
with a lifetime warranty.  I got it at www.memory4less.com which I found
using www.pricewatch.com.  I have no association with either of those
places other than being a customer (i.e., this isn't advertising spam).

I'm burning it in right now, I wrote a little program which fills it
with different test patterns and then reads them back to make sure they
don't lose any bits.  Seems to be working, it's done about 30 passes.

1.5GB for $400.  Amazing.  No more whining from you guys that BitKeeper
uses too much memory :-)

$ hinv
Main memory size: 1535.9375 Mbytes
1 AuthenticAMD  processor
1 1.44M floppy drive
1 vga+ graphics device
1 keyboard
IDE devices:
    /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
SCSI devices:
    /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
PCI bus devices:
    Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
    PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
    ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
    IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
    Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    RAID storage controller: Unknown vendor Unknown device (rev 18).
    VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[Matthew Jacob]

I'll frickin' whine if I want to :-). I still use bitkeeper on a Solaris 2.6
machine with 32MB of memory.


On Sat, 5 May 2001, Larry McVoy wrote:

> This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS.  The
> DIMMS are not ECC, but we use BitKeeper here and it tells us when we
> have bad DIMMS.
> 
> Guess what the memory cost?  $396.58 shipped to my door, second day air,
> with a lifetime warranty.  I got it at www.memory4less.com which I found
> using www.pricewatch.com.  I have no association with either of those
> places other than being a customer (i.e., this isn't advertising spam).
> 
> I'm burning it in right now, I wrote a little program which fills it
> with different test patterns and then reads them back to make sure they
> don't lose any bits.  Seems to be working, it's done about 30 passes.
> 
> 1.5GB for $400.  Amazing.  No more whining from you guys that BitKeeper
> uses too much memory :-)
> 
> $ hinv
> Main memory size: 1535.9375 Mbytes
> 1 AuthenticAMD  processor
> 1 1.44M floppy drive
> 1 vga+ graphics device
> 1 keyboard
> IDE devices:
>     /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
> SCSI devices:
>     /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
> PCI bus devices:
>     Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
>     PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
>     ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
>     IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
>     Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
>     Ethernet controller: 3Com 3C905B 100bTX (rev 48).
>     Ethernet controller: 3Com 3C905B 100bTX (rev 48).
>     Ethernet controller: 3Com 3C905B 100bTX (rev 48).
>     Ethernet controller: 3Com 3C905B 100bTX (rev 48).
>     RAID storage controller: Unknown vendor Unknown device (rev 18).
>     VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
> -- 
> ---
> Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Wow! Is memory ever cheap!

[Chris Wedgwood]

    I'm burning it in right now, I wrote a little program which fills
    it with different test patterns and then reads them back to make
    sure they don't lose any bits.  Seems to be working, it's done
    about 30 passes.

I wrote something similar to test an Alpha with a flakey L2 cache; it
didn't find anything. However, a script that did kernel compiles in a
loop soon finds errors.

I don't know much about memory testing, other than it is hard, really
hard -- and there is some magic in the way gcc access memory that
seems to trigger nasties.

It has been suggested that a good thesis would be to distill whatever
magic gcc has for testing memory and study that :)
    
    1.5GB for $400.  Amazing.  No more whining from you guys that
    BitKeeper uses too much memory :-)

1.5GB without ECC? Seems like a disater waiting to happen? Is ECC
memory much more expensive?


  --cw

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Sun, May 06, 2001 at 02:20:43PM +1200, Chris Wedgwood wrote:
> 1.5GB without ECC? Seems like a disater waiting to happen? Is ECC
> memory much more expensive?

Almost twice as expensive for 512MB dimms.

I used to be a die hard ECC fan but that changed since what we do here is
BitKeeper and BitKeeper checksums everything.  It tells us right away when
we have problems (to date it has found bad memory dimms, NFS corruption,
and a SPARC/Linux cache aliasing bug).  So I've given up in ECC, we don't
need it.

On the other hand, if your apps don't have built in integrity checks then
ECC is pretty much a requirement.

By the way, the integrity checks don't need to be complicated, BK uses
a horrible 16 bit ignore the overflow checksum to be compat with SCCS
and it seems to have caught everything that much more sophisticated and
CPU intensive checksums have caught.  In other words, anything is much
much better than nothing.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Followup to:  <20010505194536.D14127@work.bitmover.com>
By author:    Larry McVoy <lm@bitmover.com>
In newsgroup: linux.dev.kernel
>
> On Sun, May 06, 2001 at 02:20:43PM +1200, Chris Wedgwood wrote:
> > 1.5GB without ECC? Seems like a disater waiting to happen? Is ECC
> > memory much more expensive?
> 
> Almost twice as expensive for 512MB dimms.
> 
> I used to be a die hard ECC fan but that changed since what we do here is
> BitKeeper and BitKeeper checksums everything.  It tells us right away when
> we have problems (to date it has found bad memory dimms, NFS corruption,
> and a SPARC/Linux cache aliasing bug).  So I've given up in ECC, we don't
> need it.
> 
> On the other hand, if your apps don't have built in integrity checks then
> ECC is pretty much a requirement.
> 

Isn't this pretty much saying "if you're willing to dedicate your
system to running nothing but Bitkeeper, you can run it really fast?"

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Mon, May 07, 2001 at 11:47:34AM -0700, H. Peter Anvin wrote:
> Followup to:  <20010505194536.D14127@work.bitmover.com>
> By author:    Larry McVoy <lm@bitmover.com>
> In newsgroup: linux.dev.kernel
> >
> > On Sun, May 06, 2001 at 02:20:43PM +1200, Chris Wedgwood wrote:
> > > 1.5GB without ECC? Seems like a disater waiting to happen? Is ECC
> > > memory much more expensive?
> > 
> > Almost twice as expensive for 512MB dimms.
> > 
> > I used to be a die hard ECC fan but that changed since what we do here is
> > BitKeeper and BitKeeper checksums everything.  It tells us right away when
> > we have problems (to date it has found bad memory dimms, NFS corruption,
> > and a SPARC/Linux cache aliasing bug).  So I've given up in ECC, we don't
> > need it.
> > 
> > On the other hand, if your apps don't have built in integrity checks then
> > ECC is pretty much a requirement.
> 
> Isn't this pretty much saying "if you're willing to dedicate your
> system to running nothing but Bitkeeper, you can run it really fast?"

A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;
B) As I said above, "if your apps don't have built in integrity checks then
   ECC is pretty much a requirement";
C) As I said above, we use our systems for BK development, so this choice
   makes sense for us.

I think the point you are really missing is that it is not an either/or
choice.  All you really need in practice is one application which is
both heavily used and has integrity checks.  It could be BitKeeper or
something else, all that matters is that it will detect memory problems.

That application will flush out your memory problems.  Yeah, you could
get burned before that app finds them and if you are worried about that,
then run ECC.  I think it's an interesting data point, however, that I
care deeply about data integrity and I've transitioned from insisting 
on ECC to not caring.  If my choice wasn't working for me, I would 
still be using ECC.  In other words, I'm a lot closer to your way of
thinking than you might expect.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Larry McVoy wrote:
> > >
> > > On the other hand, if your apps don't have built in integrity checks then
> > > ECC is pretty much a requirement.
> >
> > Isn't this pretty much saying "if you're willing to dedicate your
> > system to running nothing but Bitkeeper, you can run it really fast?"
> 
> A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;

"It" meaning BitKeeper.

> B) As I said above, "if your apps don't have built in integrity checks then
>    ECC is pretty much a requirement";
> C) As I said above, we use our systems for BK development, so this choice
>    makes sense for us.
> 
> I think the point you are really missing is that it is not an either/or
> choice.  All you really need in practice is one application which is
> both heavily used and has integrity checks.  It could be BitKeeper or
> something else, all that matters is that it will detect memory problems.

This is not true in my experience.  YES, it will detect bad memory
configurations, but with over 2^34 memory cells to worry about -- each of
them carrying a charge of a few dozen electrons only -- you WILL have
random failures, especially if the memory is allowed to remain stale for
extended periods of time, as is very likely in a configuration like this
(think disk cache.)

Bad memory configurations is bad.  However, good memory configurations
are not necessarily perfect.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Mon, May 07, 2001 at 12:01:50PM -0700, H. Peter Anvin wrote:
> Larry McVoy wrote:
> > > Isn't this pretty much saying "if you're willing to dedicate your
> > > system to running nothing but Bitkeeper, you can run it really fast?"
> > 
> > A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;
> 
> "It" meaning BitKeeper.

What does BitKeeper have to do with this conversation?  

s/BitKeeper/any_app_which_has_integrity_checks/

Whether that app runs fast or not has nothing to do with ECC/non-ECC, right?
And while whether that app runs fast or not may have something to do with
other apps that you run along side of it, that's true for all apps, right?
So why the focus on BitKeeper?  Am I missing something?

> This is not true in my experience.  YES, it will detect bad memory
> configurations, but with over 2^34 memory cells to worry about -- each of
> them carrying a charge of a few dozen electrons only -- you WILL have
> random failures, especially if the memory is allowed to remain stale for
> extended periods of time, as is very likely in a configuration like this
> (think disk cache.)

BitKeeper, at least, runs the integrity checks every time it accesses the
data.  So it doesn't matter if it is in the disk cache or not.  The same
could be true of any other application.

> Bad memory configurations is bad.  However, good memory configurations
> are not necessarily perfect.

No, they most certainly aren't.  You can have all the ECC you want and if 
the disk or the bus or some or the part of the path corrupts the data then
you are hosed.

Dave Clark made the point that you _MUST_ have end to end checks if you 
care about your data.  He would argue, correctly, in my opinion, that 
it doesn't matter if you have ECC, something else can screw you.

And in fact, while we were having this discussion I was running a disk
scrubber to see if I had bad disks or not:

[root@disks /u1]# df -m .
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/hdg2                17099         0     17099   0% /u1
[root@disks /u1]# ~lm/tmp/a.out 17000
000000000000000001111111111
BAD @ 1045602304 3e50b000:3e52a000
[root@disks /u1]# 

and from dmesg:

hdh: irq timeout: status=0x50 { DriveReady SeekComplete }
hdg: timeout waiting for DMA
ide_dmaproc: chipset supported ide_dma_timeout func only: 14

But my application was NEVER notified that the drive subsystem was hosed,
the operating system (this is 2.4.4, by the way, steaming hot bits), never
told the application that the data was probably bad.  And it isn't a memory
problem, I ran a memory scrubber and the system memory is just fine, it's
the disk subsystem that went out to lunch.  Without telling me, by the way.
If this happened inside of SUN the guy reponsible would be getting a new
orifice courtesy of systems group.  Not cool to pass the data up when it
is bad.

So explain to me how ECC is enough?  It's clearly not.  People have made
compelling arguments for end to end checks for at least 25 years, the
internet works largely because of these end to end checks (turn off 
checksums and find out if I'm right or not, you'll see), ECC isn't end to
end, so what's the point?  Yeah, it's better than nothing but to argue
that it even remotely approaches enough is just flat out wrong, and it
is is _inherently_ unable to be part of a solution which is correct, it's
simply one place that the data lands.  
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Larry McVoy wrote:
> > >
> > > A) Fast has nothing to do with it, ECC runs at the same speed as non-ECC;
> >
> > "It" meaning BitKeeper.
> 
> What does BitKeeper have to do with this conversation?
> 
> s/BitKeeper/any_app_which_has_integrity_checks/
> 
> Whether that app runs fast or not has nothing to do with ECC/non-ECC, right?
> And while whether that app runs fast or not may have something to do with
> other apps that you run along side of it, that's true for all apps, right?
> So why the focus on BitKeeper?  Am I missing something?
> 

Because your original post was "yeah, Bitkeeper is a memory hog but you
can get really cheap non-ECC RAM so just stuff your system with crappy
RAM and be happy."  Doing so dedicates my system to running a small set
of applications, which I am utterly uninterested in.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Mon, May 07, 2001 at 12:21:28PM -0700, H. Peter Anvin wrote:
> Larry McVoy wrote:
> > What does BitKeeper have to do with this conversation?
> 
> Because your original post was "yeah, Bitkeeper is a memory hog but you
> can get really cheap non-ECC RAM so just stuff your system with crappy
> RAM and be happy."  Doing so dedicates my system to running a small set
> of applications, which I am utterly uninterested in.

.. BitKeeper isn't a memory hog, the kernel is bloated.  Over 100MB of
   source last I checked.  BitKeeper is incredibly good at _NOT_ being
   a memory hog, it uses the page cache as its memory pool.  If things
   fit in the cache, they go fast, if they don't, they don't.  BitKeeper
   is just like diff in that respect.  If you think BitKeeper is a memory
   hog, then you must hate diff too.  How about netscape?  Don't run that
   either?  Give me a break.

.. It's great that you aren't interested in running that set of small 
   applications, I'm sure the entire kernel list is happy to learn that.

.. You can get really cheap ECC ram as well, even if it were 2x as expensive,
   that's still really cheap, less than 50 cents a megabyte, so what's your
   problem?  Go get some ECC memory and be happy.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Larry McVoy wrote:
> 
> On Mon, May 07, 2001 at 12:21:28PM -0700, H. Peter Anvin wrote:
> > Larry McVoy wrote:
> > > What does BitKeeper have to do with this conversation?
> >
> > Because your original post was "yeah, Bitkeeper is a memory hog but you
> > can get really cheap non-ECC RAM so just stuff your system with crappy
> > RAM and be happy."  Doing so dedicates my system to running a small set
> > of applications, which I am utterly uninterested in.
> 
> .. BitKeeper isn't a memory hog, the kernel is bloated.  Over 100MB of
>    source last I checked.  BitKeeper is incredibly good at _NOT_ being
>    a memory hog, it uses the page cache as its memory pool.  If things
>    fit in the cache, they go fast, if they don't, they don't.  BitKeeper
>    is just like diff in that respect.  If you think BitKeeper is a memory
>    hog, then you must hate diff too.  How about netscape?  Don't run that
>    either?  Give me a break.
> 

I wasn't the one who said it, you did.  I don't have any evidence either
way.

> .. It's great that you aren't interested in running that set of small
>    applications, I'm sure the entire kernel list is happy to learn that.

I believe the same is true for most people, with the major exceptions
being the embedded systems and server farm people.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Mon, May 07, 2001 at 12:33:57PM -0700, H. Peter Anvin wrote:
> Larry McVoy wrote:
> > > Because your original post was "yeah, Bitkeeper is a memory hog but you
> > > can get really cheap non-ECC RAM so just stuff your system with crappy
> > > RAM and be happy."  

> I wasn't the one who said it, you did.  I don't have any evidence either
> way.

Err, Peter, it's starting to sound like you have some ax to grind that I
don't know about.  So I'll bow out of this conversation.

For the record, however, I never stated that BitKeeper is a memory hog,
that's a conclusion you drew.  Somehow, if I had said "look, for very
little money you can fit all of the kernel source, revision history,
and objects in memory", would you have translated that into "BitKeeper
is a memory hog"?  It seems that way.  

BitKeeper has nothing to do with it, it's all about how big the data
set is that the application is working on.  BitKeeper is better than
most applications, it mmaps the data and uses the page cache so that it
doesn't cache it twice.  Contrast that with most other apps, you'll see
they have their own internal cache of data when they should just use mmap.

Anyway, I think we've beaten this to death, so let's move on.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Larry McVoy wrote:
> 
> On Mon, May 07, 2001 at 12:33:57PM -0700, H. Peter Anvin wrote:
> > Larry McVoy wrote:
> > > > Because your original post was "yeah, Bitkeeper is a memory hog but you
> > > > can get really cheap non-ECC RAM so just stuff your system with crappy
> > > > RAM and be happy."
> 
> > I wasn't the one who said it, you did.  I don't have any evidence either
> > way.
> 
> Err, Peter, it's starting to sound like you have some ax to grind that I
> don't know about.  So I'll bow out of this conversation.
> 

The only axe I have to grind was the obvious application myopia of your
original post... "my application is the only one that matters."  That's
all.

	-hpa

-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Wow! Is memory ever cheap!

[Ben Ford]

[-- Attachment #1: Type: text/plain, Size: 2831 bytes --]

H. Peter Anvin wrote:

>Larry McVoy wrote:
>
>>On Mon, May 07, 2001 at 12:33:57PM -0700, H. Peter Anvin wrote:
>>
>>>Larry McVoy wrote:
>>>
>>>>>Because your original post was "yeah, Bitkeeper is a memory hog but you
>>>>>can get really cheap non-ECC RAM so just stuff your system with crappy
>>>>>RAM and be happy."
>>>>>
>>>I wasn't the one who said it, you did.  I don't have any evidence either
>>>way.
>>>
>>Err, Peter, it's starting to sound like you have some ax to grind that I
>>don't know about.  So I'll bow out of this conversation.
>>
>
>The only axe I have to grind was the obvious application myopia of your
>original post... "my application is the only one that matters."  That's
>all.
>
>	-hpa
>
<quote>

This is a 750Mhz K7 system with 1.5GB of memory in 3 512MB DIMMS.  The
DIMMS are not ECC, but we use BitKeeper here and it tells us when we
have bad DIMMS.

Guess what the memory cost?  $396.58 shipped to my door, second day air,
with a lifetime warranty.  I got it at www.memory4less.com <http://www.memory4less.com> which I found
using www.pricewatch.com <http://www.pricewatch.com>.  I have no association with either of those
places other than being a customer (i.e., this isn't advertising spam).

I'm burning it in right now, I wrote a little program which fills it
with different test patterns and then reads them back to make sure they
don't lose any bits.  Seems to be working, it's done about 30 passes.

1.5GB for $400.  Amazing.  No more whining from you guys that BitKeeper
uses too much memory  [:-)] 

$ hinv
Main memory size: 1535.9375 Mbytes
1 AuthenticAMD  processor
1 1.44M floppy drive
1 vga+ graphics device
1 keyboard
IDE devices:
    /dev/hda is a ST310211A, 9541MB w/512kB Cache, CHS=1216/255/63
SCSI devices:
    /dev/sda is a 3ware disk, model 3w-xxxx 74.541 GB
PCI bus devices:
    Host bridge: VIA Technologies VT 82C691 Apollo Pro (rev 2).
    PCI bridge: VIA Technologies VT 82C598 Apollo MVP3 AGP (rev 0).
    ISA bridge: VIA Technologies VT 82C686 Apollo Super (rev 34).
    IDE interface: VIA Technologies VT 82C586 Apollo IDE (rev 16).
    Host bridge: VIA Technologies VT 82C686 Apollo Super ACPI (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    Ethernet controller: 3Com 3C905B 100bTX (rev 48).
    RAID storage controller: Unknown vendor Unknown device (rev 18).quote
    VGA compatible controller: Matrox Matrox G200 AGP (rev 1).
-- --- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm

</quote>

Lets move on now.

-- 
I'd rather listen to Newton than to Mundie [MS flunkie who made a speech on
the evil-ness of open source]. He may have been dead for almost three
hundred years, but despite that he stinks up the room less.

Linus

Re: Wow! Is memory ever cheap!

[Marty Leisner]

I'm confused by the "lets not use ECC and use bk" talk.

My understanding is suns big machines stopped using ecc and they
started to have "random" problems running big-iron applications
that took them a while to figure out (and a lot of bad press) and can
only be rectified in the big cycle (this was last year so its probably solved 
now).

I thought one of the primary reasons to have ecc is to catch
wierd things before they become catostrophic...and at least
know WHY weirdness is happening...


marty

Re: Wow! Is memory ever cheap!

[Larry McVoy]

On Wed, May 09, 2001 at 12:24:25AM -0400, Marty Leisner wrote:
> I'm confused by the "lets not use ECC and use bk" talk.

I'll take a pass at unconfusing you, I can see how you might be.  I wish
I had never mentioned BK, that was never the point.  End to end was the
point, BK was just an example and now I'm getting accused of bringing
up the whole thread as a BK advertisement.  Which completely misses
the point.  Please go read

http://www.google.com/search?q=cache:web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf+clark+end+to+end&hl=en

which is a text version of the paper I mentioned before.  The basic
message of the paper is that it really doesn't help much to have things
like ECC unless you can be sure that 100% of the rest of your system
has similar checks.

The point was made again, but apparently missed here, when I pointed
out that Linux's disk subsystem passes up bad data when it knows there
may be a problem.  ECC will not help you in this case, the data was bad
before it hit memory.  So now you have carefully error corrected BAD DATA.
See the point?  ECC doesn't help unless every other component is equally
careful; those components include software and hardware.  You can fix
that chunk of software and then I'll go find a rogue disk controller
that breaks the datapath, there are plenty to choose from.

Just to make sure you understand:  I think ECC is a fine thing.  If I'm
running systems with no other integrity checks, I'll take ECC and like it.
However, having ECC does not mean that I trust that my data is safe,
that is most certainly not a true statement.  The bus, the disks, the
disk controller, the disk driver, the buffer cache, etc, can all corrupt
the data.   Oh, yeah, let's not forget NFS.  I have seen each and every
one of those things corrupt data.

As to the BitKeeper stuff, those of you who think this is a BitKeeper
discussion are off wacking in the weeds.  The point isn't that BitKeeper
is good because it has integrity checks, the point is that integrity
checks are a good thing.  Period.   BitKeeper was just an example.
If there was a Linux filesystem that had built in integrity checks (and
I knew about it, for all I know there is one), then I would have used
that as the example.  I used BitKeeper as an example because I know it
and I can point to numerous cases where it exposed problems that ECC
would not have caught.  Ask Dave Miller about the mmap/read sparc linux
cache aliasing bug that BK exposed, that one was nasty.

Let's review:  ECC is nice, but it doesn't solve all data corruption
problems.  Applications which do their own end to end data integrity
checks will catch many more error cases than what ECC catches.  My efforts
in this thread had nothing to do with BitKeeper, they were trying to
get people to realize that end to end is good, and ECC isn't end to end.

Examples of end to end applications, which I should have thought of
sooner, are the md5sums on ftp.kernel.org, the integrity checks in rpms,
crcs in cpio.  I'm sure you can think of lots of others, this is an
old problem.

> My understanding is suns big machines stopped using ecc and they

The SUN problem was a cache problem and there is no way that I believe
that SUN would turn of ECC in the cache.  There are good reasons for
not doing so.  If you think through the end to end argument, you will
see that you have no way to do checks on the data path into/out of the
processor.  If that part of the datapath is not checked then no amount
of checking elsewhere does any good, the processor can be corrupting
your data and never know it.  If SUN was so stupid as to remove this,
then it is a dramatically different place.  I heard that there was a
bug in the cache controller, I never heard that they had removed ECC.
If you really want to know I can ask, I know at least one of the guys
who works on that stuff there.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: Wow! Is memory ever cheap!

[Dan Hollis]

On Tue, 8 May 2001, Larry McVoy wrote:
> which is a text version of the paper I mentioned before.  The basic
> message of the paper is that it really doesn't help much to have things
> like ECC unless you can be sure that 100% of the rest of your system
> has similar checks.

UDMA has crc, scsi has parity, pci has (i think) parity, tcpip has crc,
your cpu l1 and l2 have ecc...

Looks like similar checks are already there.

-Dan

Re: Wow! Is memory ever cheap!

[Gérard Roudier]

On Tue, 8 May 2001, Dan Hollis wrote:

> On Tue, 8 May 2001, Larry McVoy wrote:
> > which is a text version of the paper I mentioned before.  The basic
> > message of the paper is that it really doesn't help much to have things
> > like ECC unless you can be sure that 100% of the rest of your system
> > has similar checks.
> 
> UDMA has crc, scsi has parity, pci has (i think) parity, tcpip has crc,
> your cpu l1 and l2 have ecc...

SCSI Ultra-160 has CRC.

PCI has parity (btw, you think right), but only a few drivers make sure
PCI parity checking is enabled. On the other hand, a PCI parity error
should be considered as extremally serious and the system should be
stopped when such happens.

Btw, it seems (read at the pci list) that the original PCI hadn't parity.
After all, PCI had been designed for PC machines... :)

> Looks like similar checks are already there.
> 
> -Dan
> 
  Gérard.

Re: Wow! Is memory ever cheap!

[Matthew Jacob]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset=X-UNKNOWN, Size: 1316 bytes --]



On Wed, 9 May 2001, [ISO-8859-1] Gérard Roudier wrote:

> 
> 
> On Tue, 8 May 2001, Dan Hollis wrote:
> 
> > On Tue, 8 May 2001, Larry McVoy wrote:
> > > which is a text version of the paper I mentioned before.  The basic
> > > message of the paper is that it really doesn't help much to have things
> > > like ECC unless you can be sure that 100% of the rest of your system
> > > has similar checks.
> > 
> > UDMA has crc, scsi has parity, pci has (i think) parity, tcpip has crc,
> > your cpu l1 and l2 have ecc...
> 
> SCSI Ultra-160 has CRC.
> 
> PCI has parity (btw, you think right), but only a few drivers make sure
> PCI parity checking is enabled. On the other hand, a PCI parity error

Sun's panic if they get SERR or PERR.

> should be considered as extremally serious and the system should be
> stopped when such happens.
> 
> Btw, it seems (read at the pci list) that the original PCI hadn't parity.
> After all, PCI had been designed for PC machines... :)
> 
> > Looks like similar checks are already there.
> > 
> > -Dan
> > 
>   Gérard.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Wow! Is memory ever cheap!

[John Alvord]

On Tue, 8 May 2001 22:22:10 -0700, Larry McVoy <lm@bitmover.com>
wrote:
>
>Just to make sure you understand:  I think ECC is a fine thing.  If I'm
>running systems with no other integrity checks, I'll take ECC and like it.
>However, having ECC does not mean that I trust that my data is safe,
>that is most certainly not a true statement.  The bus, the disks, the
>disk controller, the disk driver, the buffer cache, etc, can all corrupt
>the data.   Oh, yeah, let's not forget NFS.  I have seen each and every
>one of those things corrupt data.

This is an interesting observation of a truth that was well known in
the second generation computers of the 1950s and 1960s. I first worked
at John Hancock... they had a bunch of 7074 machines. All those
systems made use of programmed checksums in each tape block and in
each full file. The reason was that those machines did not have ECC...
they did have parity checking if I remember right. With IBM's third
generation computers (S/360s) and probably other manufacturers, ECC
became a standard feature. Parity checking was added through different
data paths such as channel memory, buffer memory, etc. There was so
much protection added that the programmed checksums became
superfluous.

There were still odd moments. I remember working on an Amdahl computer
problem where some internal data paths... where the contents of one
register moved to an internal storage area... and the path did not
have parity. There was a machine fault... the path was electrically
open, so the contents of the register always became zero. But since it
wasn't parity checked, there was no machine check. I remember another
problem on the IBM 3033. Cosmic rays (really) caused one bit errors in
channel memory. That was parity but not ECC so you got a weird channel
check. Back at the diagnosis ranch, the board looked good. It was only
when someone noticed that the rate of such problems was proportional
to the height above sea level that the light bulb went on.

The lesson is that when paths are not checked, hardware or software,
data being held or transformed can change. Old lesson but a good one
to know.

john alvord

Re: Wow! Is memory ever cheap!

[Malcolm Beattie]

Larry McVoy writes:
> On Wed, May 09, 2001 at 12:24:25AM -0400, Marty Leisner wrote:
> > My understanding is suns big machines stopped using ecc and they
> 
> The SUN problem was a cache problem and there is no way that I believe
> that SUN would turn of ECC in the cache.  There are good reasons for
> not doing so.  If you think through the end to end argument, you will
> see that you have no way to do checks on the data path into/out of the
> processor.  If that part of the datapath is not checked then no amount
> of checking elsewhere does any good, the processor can be corrupting
> your data and never know it.  If SUN was so stupid as to remove this,
> then it is a dramatically different place.  I heard that there was a
> bug in the cache controller, I never heard that they had removed ECC.

There are issues with error detection/correction/recovery with
different designs of L1 and L2 caches. There's a good paper:

    IBM S/390 storage hierarchy - G5 and G6 performance considerations
    IBM Journal of Research and Development
    Vol 43 No. 5/6
available at
    http://www.research.ibm.com/journal/rd/435/jackson.html

which covers IBM's choice of L1 and L2 design for S/390. The section on
"S/390 reliability and performance implications" is relevant here. In
particular, they use a solution which isn't best from the performance
point of view but ensures you don't discover "too late" about an error.
I heard a rumour (now I get to the unsubstantiated part :-) that Sun
chose a higher-performing design for their cache subsystem but which has
a nastier failure mode in the case of cache errors.

--Malcolm

-- 
Malcolm Beattie <mbeattie@sable.ox.ac.uk>
Unix Systems Programmer
Oxford University Computing Services

Re: Wow! Is memory ever cheap!

[Edgar Toernig]

Larry McVoy wrote:
> 
> Let's review:  ECC is nice, but it doesn't solve all data corruption
> problems.  Applications which do their own end to end data integrity
> checks will catch many more error cases than what ECC catches.

I think you have a wrong idea why the ECC is there.  ECC deals with
the inherit shortcommings of DRAM.

DRAMs are not perfect.  They have a probability to lose a bit.
Normally this probability is low enough to live with it.  Lets say
you have a system with 1MByte and let's say the probability for a
single bit error is around 1 error in 100 years.  Good enough.
Now put 1GByte in the system. You'll get a probability of 10 errors
per year.  Maybe good enough for a Windows box but not acceptable
for your server.  So you put in ECC to bring this probability back
into reasonable numbers.  ECC can correct the single bit errors.
You only have to deal with double bit errors.  Chance for them is
much much lower.

Sure, it doesn't solve all data corruption problems - only simple
errors in DRAMs.  But it makes systems with huge amount of RAM staying
up alive much longer.  And btw, your integrity checks over data will
not protect against a corrupted kernel or application...

Ciao, ET.

PS: Just let your app run long enough.  I'm sure it will detect a
checksum error some day ;-)

Re: Wow! Is memory ever cheap!

[H. Peter Anvin]

Followup to:  <3AF9B0D2.F2E330F9@gmx.de>
By author:    Edgar Toernig <froese@gmx.de>
In newsgroup: linux.dev.kernel
> 
> I think you have a wrong idea why the ECC is there.  ECC deals with
> the inherit shortcommings of DRAM.
> 
> DRAMs are not perfect.  They have a probability to lose a bit.
> Normally this probability is low enough to live with it.  Lets say
> you have a system with 1MByte and let's say the probability for a
> single bit error is around 1 error in 100 years.  Good enough.
> Now put 1GByte in the system. You'll get a probability of 10 errors
> per year.  Maybe good enough for a Windows box but not acceptable
> for your server.  So you put in ECC to bring this probability back
> into reasonable numbers.  ECC can correct the single bit errors.
> You only have to deal with double bit errors.  Chance for them is
> much much lower.
> 

Yes, ECC, unlike parity, is a technique for reducing the error rate,
with the side benefit of intercepting an error when it happens.

I am not disagreeing with Larry that integrity checks are a Good
Thing[TM], and in general are a hallmark of good engineering.
However, they are not a replacement for ECC for the purpose of driving
the failure rate down into an acceptable probability range.

It is of course a very nice thing that DRAM prices have come down into
the range where buying them in gigabyte quantities are reasonable :)

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt