is the weeks before -rc1 the time to really be working on -next?

is the weeks before -rc1 the time to really be working on -next?

[Greg KH]

Just wondering, I know that -next is failing right now, and is a major
pain to produce, but that seems to be primarily due to all of the
subsystems merging with Linus right now.

So does it even make sense to try to create a -next during the 2 weeks
of the major merge window?  It seems to just cause you a whole lot of
work, that in the end, is mostly unecessary as all of the subsystem
maintainers are doing the merging themselves as trees move into Linus's
tree?

thanks,

greg k-h

Re: is the weeks before -rc1 the time to really be working on -next?

[Geert Uytterhoeven]

On Wed, 15 Oct 2008, Greg KH wrote:
> Just wondering, I know that -next is failing right now, and is a major
> pain to produce, but that seems to be primarily due to all of the
> subsystems merging with Linus right now.
> 
> So does it even make sense to try to create a -next during the 2 weeks
> of the major merge window?  It seems to just cause you a whole lot of
> work, that in the end, is mostly unecessary as all of the subsystem
> maintainers are doing the merging themselves as trees move into Linus's
> tree?

I don't know if it's worth the effort...

However, we definitely missed linux-next during the last few weeks, seeing
things getting into mainline that don't even survive simple compile tests.

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: is the weeks before -rc1 the time to really be working on -next?

[Greg KH]

On Wed, Oct 15, 2008 at 10:04:17PM +0200, Geert Uytterhoeven wrote:
> On Wed, 15 Oct 2008, Greg KH wrote:
> > Just wondering, I know that -next is failing right now, and is a major
> > pain to produce, but that seems to be primarily due to all of the
> > subsystems merging with Linus right now.
> > 
> > So does it even make sense to try to create a -next during the 2 weeks
> > of the major merge window?  It seems to just cause you a whole lot of
> > work, that in the end, is mostly unecessary as all of the subsystem
> > maintainers are doing the merging themselves as trees move into Linus's
> > tree?
> 
> I don't know if it's worth the effort...
> 
> However, we definitely missed linux-next during the last few weeks, seeing
> things getting into mainline that don't even survive simple compile tests.

Oh I agree, I missed it a lot the past few weeks, it's amazing how much
we already started to rely on it in such a short time.  That shows how
valuable the effort is.

Keep it up, I just don't want to see Stephen burn out during the pre-rc1
time period when it might not make as much sense to be trying to create
the tree.

thanks,

greg k-h

Re: is the weeks before -rc1 the time to really be working on -next?

[Andrew Morton]

On Wed, 15 Oct 2008 10:40:15 -0700
Greg KH <greg@kroah.com> wrote:

> Just wondering, I know that -next is failing right now, and is a major
> pain to produce, but that seems to be primarily due to all of the
> subsystems merging with Linus right now.

No, I'd say it's primarily due to subsystem maintainers losing
discipline and changing (or compile-time and runtime breaking) other
people's stuff.  This problem appears to have become much worse since
linux-next started.  I suspect Stephen is cleaning up others' trash so
they're producing more of it.

This situation has totally screwed me over, because my tree is so
dependent upon the composite everyone-else tree.  And a large reason
for that dependency is not that I'm carrying patches against other
people's code - it's that they're changing (or breaking) things which
lie outside their area of responsibility.

During Stephen's absence I was forced to try to assemble a
linux-next-like tree locally and that has become much much harder than
it was before linux-next, because all those trees have gone so rampant.

On one day of last week it took me from 10:00AM until 4:00PM just to
get all the patches applied and partially compiling, despite the fact
that I had them all applied and compiling 24 hours beforehand.

> So does it even make sense to try to create a -next during the 2 weeks
> of the major merge window?  It seems to just cause you a whole lot of
> work, that in the end, is mostly unecessary as all of the subsystem
> maintainers are doing the merging themselves as trees move into Linus's
> tree?
> 

It's very useful to me, because my tree is based on everyone else's. 
With no linux-next I'd need to either go back to pulling everyone
else's junk or I'd need to rebase on mainline.

The latter is sorely tempting.  It would save me vast amounts of time
and hair-tearing.  I'd base my tree on mainline and dammit I'd merge
first.  So everyone who has been changing stuff which is outside their
area of responsibility and breaking other people's stuff would get to
see the consequences of their actions instead of Stephen and I bearing
the brunt of it all the time.


I fear we've reached the stage now where people are merrily merging
bright-and-shiny things into their local trees without giving much
thought at all to the consequences for others.

Re: is the weeks before -rc1 the time to really be working on -next?

[Rafael J. Wysocki]

On Wednesday, 15 of October 2008, Andrew Morton wrote:
> On Wed, 15 Oct 2008 10:40:15 -0700
> Greg KH <greg@kroah.com> wrote:
> 
> > Just wondering, I know that -next is failing right now, and is a major
> > pain to produce, but that seems to be primarily due to all of the
> > subsystems merging with Linus right now.
> 
> No, I'd say it's primarily due to subsystem maintainers losing
> discipline and changing (or compile-time and runtime breaking) other
> people's stuff.  This problem appears to have become much worse since
> linux-next started.  I suspect Stephen is cleaning up others' trash so
> they're producing more of it.
> 
> This situation has totally screwed me over, because my tree is so
> dependent upon the composite everyone-else tree.  And a large reason
> for that dependency is not that I'm carrying patches against other
> people's code - it's that they're changing (or breaking) things which
> lie outside their area of responsibility.
> 
> During Stephen's absence I was forced to try to assemble a
> linux-next-like tree locally and that has become much much harder than
> it was before linux-next, because all those trees have gone so rampant.
> 
> On one day of last week it took me from 10:00AM until 4:00PM just to
> get all the patches applied and partially compiling, despite the fact
> that I had them all applied and compiling 24 hours beforehand.
> 
> > So does it even make sense to try to create a -next during the 2 weeks
> > of the major merge window?  It seems to just cause you a whole lot of
> > work, that in the end, is mostly unecessary as all of the subsystem
> > maintainers are doing the merging themselves as trees move into Linus's
> > tree?
> > 
> 
> It's very useful to me, because my tree is based on everyone else's. 
> With no linux-next I'd need to either go back to pulling everyone
> else's junk or I'd need to rebase on mainline.
> 
> The latter is sorely tempting.  It would save me vast amounts of time
> and hair-tearing.  I'd base my tree on mainline and dammit I'd merge
> first.  So everyone who has been changing stuff which is outside their
> area of responsibility and breaking other people's stuff would get to
> see the consequences of their actions instead of Stephen and I bearing
> the brunt of it all the time.
> 
> 
> I fear we've reached the stage now where people are merrily merging
> bright-and-shiny things into their local trees without giving much
> thought at all to the consequences for others.

Not to mention trying to compile their trees or to test them. :-(

Re: is the weeks before -rc1 the time to really be working on -next?

[Alan Cox]

> first.  So everyone who has been changing stuff which is outside their
> area of responsibility and breaking other people's stuff would get to
> see the consequences of their actions instead of Stephen and I bearing
> the brunt of it all the time.
> 
> 
> I fear we've reached the stage now where people are merrily merging
> bright-and-shiny things into their local trees without giving much
> thought at all to the consequences for others.

Its not that simple. There are areas where the tree divides don't fit the
coding divides. ttydev was horribly entwined with USB and there really
wasn't a way to break that up at the time in question. Because -next
tries to assemble all the trees based on Linus tree it can't cope with
that case well at all and in fact there was some neccessary rule bending
to make next work out where some of the ttydev work was done with ttydev
on top of other subtrees.

So at times it would be very helpful with -next to be able to do limited
tree ordering. If I could have flagged tty as requiring USB first for
example there would have been a lot less pain involved.

Alan

Re: is the weeks before -rc1 the time to really be working on -next?

[Andrew Morton]

On Wed, 15 Oct 2008 23:36:07 +0100
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > first.  So everyone who has been changing stuff which is outside their
> > area of responsibility and breaking other people's stuff would get to
> > see the consequences of their actions instead of Stephen and I bearing
> > the brunt of it all the time.
> > 
> > 
> > I fear we've reached the stage now where people are merrily merging
> > bright-and-shiny things into their local trees without giving much
> > thought at all to the consequences for others.
> 
> Its not that simple. There are areas where the tree divides don't fit the
> coding divides. ttydev was horribly entwined with USB and there really
> wasn't a way to break that up at the time in question. Because -next
> tries to assemble all the trees based on Linus tree it can't cope with
> that case well at all and in fact there was some neccessary rule bending
> to make next work out where some of the ttydev work was done with ttydev
> on top of other subtrees.

Yes, sometimes there are nasty special cases and I of course understand
that and I did allow for it in making that assertion.  There's quite a
lot of gratuitous lazy stuff happening too.

> So at times it would be very helpful with -next to be able to do limited
> tree ordering. If I could have flagged tty as requiring USB first for
> example there would have been a lot less pain involved.

Well there's an alternative here: you base the ttydev tree on top of
linux-next.  That way we spread the load around a bit.

But the problem here is that once linux-next merges your patches, you
no longer have a tree on which to base your patches!  You need to get
your hands on "linux-next without my stuff" to maintain them.


I'm in the same situation with -mm and I do have a scheme planned for
that, but it'll involve asking Stephen to add a "mm starts here" tag,
so I can extract "linux-next without my stuff".  (I still haven't got
around to making this happen - I've become a fulltime reject fixer).


But the problem with that scheme is that it'd be hard to generalise for
other trees.  Or maybe not - we can just say "Alan goes first, then
-mm".  Then you piggyback on top of the infrastructure which I use.


Then again, the ttydev problem was hopefully a once-off, so we don't
need to do anything now.

Re: is the weeks before -rc1 the time to really be working on -next?

[David Miller]

From: Andrew Morton <akpm@linux-foundation.org>
Date: Wed, 15 Oct 2008 15:53:53 -0700

> But the problem here is that once linux-next merges your patches, you
> no longer have a tree on which to base your patches!  You need to get
> your hands on "linux-next without my stuff" to maintain them.

I know this doesn't work for you, but if you ran -mm just like any
other GIT tree it might mesh a whole lot better.

And in reality that kind of situation isn't a big deal in the
context of -next.  People are rebasing their trees all the time
there, and it mostly seems to work itself out.

It's a lot more work for a contributor to do work against -mm,
since the response to "which -mm should I work against and where
do I get it from" is a bit more involved that just "pull from
this GIT tree and do your work on top of that."

And just like networking we could have Stephen treat the -mm
GIT tree as "important" which roughly means that other conflicting
trees will be knocked out of a -next release in deference to -mm.

Those people will have to fix their stuff, not you.  And you'll
always therefore get coverage in -next.

Unlike the general sentiment expressed here, I think -next is helping.
Even if only because Stephen pokes people with trees causing problems
on a daily basis.

Re: is the weeks before -rc1 the time to really be working on -next?

[Andrew Morton]

On Wed, 15 Oct 2008 16:08:21 -0700 (PDT)
David Miller <davem@davemloft.net> wrote:

> From: Andrew Morton <akpm@linux-foundation.org>
> Date: Wed, 15 Oct 2008 15:53:53 -0700
> 
> > But the problem here is that once linux-next merges your patches, you
> > no longer have a tree on which to base your patches!  You need to get
> > your hands on "linux-next without my stuff" to maintain them.
> 
> I know this doesn't work for you, but if you ran -mm just like any
> other GIT tree it might mesh a whole lot better.

A lot of -mm is "small random subsystem trees", maybe 100 in total. 
They'd work fine as git trees.

There are also 100-odd "trees" in -mm (many of which have zero length)
which are "things to bug a subsystem maintainer with".  Alas, subsystem
maintainers like to fumble the patches I send them, then merge other
stuff which breaks the patches which I'm maintaining for them.  So in
reality each of those 100-odd trees is based on and tracks a separate
git tree.  Or, of course, on linux-next..  Sometimes I end up
maintaining these for a *long* time - years.

Then there are the nasty ones - patches which weren't factored into
"core patch followed by per-maintainer patches" and which need to go in
as a single hit.  Fortuntely these are relatively rare and we _could_
push harder to break them into core-plus-per-maintainer form.  Or I
could just lose the emails ;) They often tend to not be terribly
important.

> And in reality that kind of situation isn't a big deal in the
> context of -next.  People are rebasing their trees all the time
> there, and it mostly seems to work itself out.
> 
> It's a lot more work for a contributor to do work against -mm,
> since the response to "which -mm should I work against and where
> do I get it from" is a bit more involved that just "pull from
> this GIT tree and do your work on top of that."
> 
> And just like networking we could have Stephen treat the -mm
> GIT tree as "important" which roughly means that other conflicting
> trees will be knocked out of a -next release in deference to -mm.
> 
> Those people will have to fix their stuff, not you.  And you'll
> always therefore get coverage in -next.

Yes, but then people would end up being based on linux-next, and that's
a pretty rubbery target with all the rebasing and trees getting
dropped, etc.  And they'd accidentally end up having to actually
compile and run linux-next, shock-horror-oh-the-humanity.

> Unlike the general sentiment expressed here, I think -next is helping.
> Even if only because Stephen pokes people with trees causing problems
> on a daily basis.

yup.  Plus the runtime testing.

I doubt if the world would end if we just stopped trying to run any of
these uber-trees.  Everyone bases their work on mainline and then
everything goes smash/bang/curse during the merge window.  It wouldn't
be pretty, but it'd sure make people merge their trees promptly ;)

Re: is the weeks before -rc1 the time to really be working on -next?

[Tony Luck]

> Yes, but then people would end up being based on linux-next, and that's
> a pretty rubbery target with all the rebasing and trees getting
> dropped, etc.  And they'd accidentally end up having to actually
> compile and run linux-next, shock-horror-oh-the-humanity.

I built and booted linux-next a couple of times a week or so
for ia64.  Most of the time it worked just fine.  I found a few
build problems, but only three run-time problems. My boot
log shows 49 successful boots of linux-next based kernels
from 2.6.25-rc2 to 2.6.27-rc6

But perhaps less people are working on breaking ia64 :-)

-Tony

Re: is the weeks before -rc1 the time to really be working on -next?

[David Miller]

From: Andrew Morton <akpm@linux-foundation.org>
Date: Wed, 15 Oct 2008 16:41:39 -0700

> On Wed, 15 Oct 2008 16:08:21 -0700 (PDT)
> David Miller <davem@davemloft.net> wrote:
> 
> > And just like networking we could have Stephen treat the -mm
> > GIT tree as "important" which roughly means that other conflicting
> > trees will be knocked out of a -next release in deference to -mm.
> > 
> > Those people will have to fix their stuff, not you.  And you'll
> > always therefore get coverage in -next.
> 
> Yes, but then people would end up being based on linux-next, and that's
> a pretty rubbery target with all the rebasing and trees getting
> dropped, etc.  And they'd accidentally end up having to actually
> compile and run linux-next, shock-horror-oh-the-humanity.

That's not the idea actually.

The idea is that you run your tree strictly just like any other
subsystem maintainer.  Your tree is a clone of Linus's and you
put -mm stuff in on top of that.  Full stop.

So anyone can work on -mm independent of -next.  Just like networking,
PCI, INPUT, or any other subsystem.  There is no reason in my mind to
treat -mm specially, anything else is just pain for the people that
could contribute and help.

> I doubt if the world would end if we just stopped trying to run any of
> these uber-trees.  Everyone bases their work on mainline and then
> everything goes smash/bang/curse during the merge window.  It wouldn't
> be pretty, but it'd sure make people merge their trees promptly ;)

I think this aspect is interesting actually.

For folks that use stable GIT trees, we can cross polinate ourselves
to resolve merge problems quite simply, and I have seen this happen
already in the most recent cycle.

Things go smoothly for people that maintain stable GIT trees.  The
more stuff you touch the more important this is.

The "I rebase my tree ever 3 hours" stuff only works if you are
strictly working in a certain area and have no contributors you care
about.

But I even question this situation, no piece of code is an island in
the kernel and even well contained drivers want to modify some
infrastructure from time to time.