RE: New Release Process

RE: New Release Process

[Ian Pratt]

> > I was hoping you could clarify what the decisions were for the new 
> > release process that you proposed at the Winter XenSummit.
> 
> We decided to try to aim for ~6 week intervals for 3.0.x 
> releases, stablizing the tree in -unstable then doing the 
> release and sweeping the code into 3.0-testing. We'll then 
> try and backport critical fixes from -unstable into 
> 3.0-testing and spin new 3.0.x-y build numbers as required. 
> Any similarity to the Linux process is purely intentional :)

Here's my thoughts on how we should kick-off with the new release
process:

It's been over 6 weeks since the 3.0.0 release, and the -unstable tree
is actually looking pretty good right now -- two of the bugs I mentioned
yesterday are now fixed. 

My current inclination is to call a 3.0.1 release Friday/Saturday and
sweep the tree into -testing. Monday morning we'd then incorporate hvm
and the 2.6.15 tree and work flat out to get that fully tested and
stabilized ASAP, so SuSE can pick it up for SLES10.

SuSE have said they are actually going to base their release off 2.6.16,
even though we're still likely to be on 2.6.16-rcX by their freeze date.
One thing we could do to help them is to break with tradition and to
check the 2.6.16-rcX into the tree rather than the most recent stable
release, 2.6.15. This would help get 2.6.16 stabilized quicker, which
would certainly help them. 2.6.16 is already at rc1, which means that
many of the 'rough edges' should have been found, so I doubt we'll be
hurting ourselves too much. This is -unstable, after all.

What do other developers feel about trying to help SuSE out like this?
No doubt we might have to end up doing something similar for RH come the
RHEL5 freeze date. My feeling is that its in the xen community's
interest to have the best possible vendor releases, as the users always
end up coming to our mailing lists to complain :) 

What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?

Any reason not to call 3.0.1 now? There are a load of bug fixes and
improvements over 3.0.0.

Ian



  

Re: RE: New Release Process

[Mark Williamson]

> Here's my thoughts on how we should kick-off with the new release
> process:
>
> It's been over 6 weeks since the 3.0.0 release, and the -unstable tree
> is actually looking pretty good right now -- two of the bugs I mentioned
> yesterday are now fixed.
>
> My current inclination is to call a 3.0.1 release Friday/Saturday and
> sweep the tree into -testing.

It sounds like it'd be a net win.  Many people are only going to use official 
releases, so it should aid a lot of newcomers.

> SuSE have said they are actually going to base their release off 2.6.16,
> even though we're still likely to be on 2.6.16-rcX by their freeze date.
> One thing we could do to help them is to break with tradition and to
> check the 2.6.16-rcX into the tree rather than the most recent stable
> release, 2.6.15. This would help get 2.6.16 stabilized quicker, which
> would certainly help them. 2.6.16 is already at rc1, which means that
> many of the 'rough edges' should have been found, so I doubt we'll be
> hurting ourselves too much. This is -unstable, after all.
>
> What do other developers feel about trying to help SuSE out like this?

We've been having reasonably large delays between releases anyhow.  If 3.0.2, 
featuring 2.6.16 has to wait for 2.6.16 itself to be released it's still 
unlikely to take any longer than the last cycle.  And in the meantime it does 
help SuSE out.

> No doubt we might have to end up doing something similar for RH come the
> RHEL5 freeze date. My feeling is that its in the xen community's
> interest to have the best possible vendor releases, as the users always
> end up coming to our mailing lists to complain :)

I think better co-ordination with vendors - particularly whilst Xen 
installations aren't quite so widespread - sounds like a fairly solid idea.  
It's much better all round if people can get solid releases from distro 
packages without having to rebuild or install stuff from XenSource.

Right now, being flexible with the release dates is still viable and gets us 
concrete benefits in some cases - there's no reason to be excessively rigid.

> What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?

Hey, 2.6.16-rc1 is gonna be an improvement over 2.6.12 anyhow!

It'd also be nice from a development PoV to have some of the more recent 
kernel features / APIs available in our official release.  And of course, we 
*do* need to keep up with mainline if we want our patches merged.

$0.02,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: RE: New Release Process

[Nivedita Singhvi]

Ian Pratt wrote:

> It's been over 6 weeks since the 3.0.0 release, and the -unstable tree
> is actually looking pretty good right now -- two of the bugs I mentioned
> yesterday are now fixed. 
> 
> My current inclination is to call a 3.0.1 release Friday/Saturday and
> sweep the tree into -testing. Monday morning we'd then incorporate hvm
> and the 2.6.15 tree and work flat out to get that fully tested and
> stabilized ASAP, so SuSE can pick it up for SLES10.

Agreed, I think this is fine.

> SuSE have said they are actually going to base their release off 2.6.16,
> even though we're still likely to be on 2.6.16-rcX by their freeze date.
> One thing we could do to help them is to break with tradition and to
> check the 2.6.16-rcX into the tree rather than the most recent stable
> release, 2.6.15. This would help get 2.6.16 stabilized quicker, which
> would certainly help them. 2.6.16 is already at rc1, which means that

Probably keeps us current for longer, so sounds reasonable.

> many of the 'rough edges' should have been found, so I doubt we'll be
> hurting ourselves too much. This is -unstable, after all.
> 
> What do other developers feel about trying to help SuSE out like this?
> No doubt we might have to end up doing something similar for RH come the
> RHEL5 freeze date. My feeling is that its in the xen community's
> interest to have the best possible vendor releases, as the users always
> end up coming to our mailing lists to complain :) 
> 
> What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?

I think going to 2.6.16-rc1 is a good idea..

> Any reason not to call 3.0.1 now? There are a load of bug fixes and
> improvements over 3.0.0.

Doing that now is fine. The emphasis as soon as that is done should
be to stabilize the new stuff incoming asap, so that a stable hvm,
upgraded linux kernel etc can be picked up by SLES10..i.e.
hopefully we can get a 3.0.2 out soon, too..

Our focus will continue to be hitting on xen-unstable as hard as
we can and fixing existing showstopper bugs there..

thanks,
Nivedita

Re: RE: New Release Process

[Matt Ayres]

Ian Pratt wrote:
>>> I was hoping you could clarify what the decisions were for the new 
>>> release process that you proposed at the Winter XenSummit.
>> We decided to try to aim for ~6 week intervals for 3.0.x 
>> releases, stablizing the tree in -unstable then doing the 
>> release and sweeping the code into 3.0-testing. We'll then 
>> try and backport critical fixes from -unstable into 
>> 3.0-testing and spin new 3.0.x-y build numbers as required. 
>> Any similarity to the Linux process is purely intentional :)
> 
> Here's my thoughts on how we should kick-off with the new release
> process:
> 
> It's been over 6 weeks since the 3.0.0 release, and the -unstable tree
> is actually looking pretty good right now -- two of the bugs I mentioned
> yesterday are now fixed. 
> 
> My current inclination is to call a 3.0.1 release Friday/Saturday and
> sweep the tree into -testing. Monday morning we'd then incorporate hvm
> and the 2.6.15 tree and work flat out to get that fully tested and
> stabilized ASAP, so SuSE can pick it up for SLES10.
> 

Most of the bugs I have encountered have been fixed and -unstable is 
running fairly stable.  I still experience the bug in bugzilla id # 487 
(No space left on device).

> 
> What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?
> 

If my vote counts, I say 2.6.16-rc1 :)

> Any reason not to call 3.0.1 now? There are a load of bug fixes and
> improvements over 3.0.0.

I'd say 3.0.1 is required as -unstable has essentially become 
3.0-testing over the past few weeks.  I'd like to see a tree where 
-unstable is truly unstable and not the most stable.

Thank you,
Matt Ayres

Re: RE: New Release Process

[Keir Fraser]

On 26 Jan 2006, at 18:45, Matt Ayres wrote:

>> Any reason not to call 3.0.1 now? There are a load of bug fixes and
>> improvements over 3.0.0.
>
> I'd say 3.0.1 is required as -unstable has essentially become 
> 3.0-testing over the past few weeks.  I'd like to see a tree where 
> -unstable is truly unstable and not the most stable.

Upgrading to 2.6.16-rc will probably make your wish come true. ;-)

  -- Keir

Re: RE: New Release Process

[Matt Ayres]

Keir Fraser wrote:
> 
> On 26 Jan 2006, at 18:45, Matt Ayres wrote:
> 
>>> Any reason not to call 3.0.1 now? There are a load of bug fixes and
>>> improvements over 3.0.0.
>>
>> I'd say 3.0.1 is required as -unstable has essentially become 
>> 3.0-testing over the past few weeks.  I'd like to see a tree where 
>> -unstable is truly unstable and not the most stable.
> 
> Upgrading to 2.6.16-rc will probably make your wish come true. ;-)
> 

Current -unstable will become 3.0.1, correct?  I am wondering as the 
Novell guys' CFQ scheduler is turning out to be very useful and with no 
troubles so far.  It'd be nice to get this included in 3.0.1.

Thank you,
Matt Ayres

Re: New Release Process

[Anthony Liguori]

Ian Pratt wrote:

>My current inclination is to call a 3.0.1 release Friday/Saturday and
>sweep the tree into -testing. Monday morning we'd then incorporate hvm
>and the 2.6.15 tree and work flat out to get that fully tested and
>stabilized ASAP, so SuSE can pick it up for SLES10.
>  
>
Yeah, this is a good idea.  Many of us were hoping at the Summit that 
3.0.1 would be released ASAP.

>What do other developers feel about trying to help SuSE out like this?
>  
>
In my mind, switching to a higher kernel version is always a good thing 
especially since it gets us closer to being able to start dropping 
patches for upstream merge.

>No doubt we might have to end up doing something similar for RH come the
>RHEL5 freeze date. My feeling is that its in the xen community's
>interest to have the best possible vendor releases, as the users always
>end up coming to our mailing lists to complain :) 
>  
>
Frequently releases are a very good thing.  If we have frequent 
releases, and stay close to the upstream kernel, we shouldn't have to 
worry much about the distro releases.

Regards,

Anthony Liguori

>What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?
>
>Any reason not to call 3.0.1 now? There are a load of bug fixes and
>improvements over 3.0.0.
>
>Ian
>
>
>
>  
>
>
>
>
>
>
>
>
>  
>

Re: RE: New Release Process

[Gerd Hoffmann]

  Hi,

> SuSE have said they are actually going to base their release off 2.6.16,
> even though we're still likely to be on 2.6.16-rcX by their freeze date.
> One thing we could do to help them is to break with tradition and to
> check the 2.6.16-rcX into the tree rather than the most recent stable
> release, 2.6.15. This would help get 2.6.16 stabilized quicker, which
> would certainly help them. 2.6.16 is already at rc1, which means that
> many of the 'rough edges' should have been found, so I doubt we'll be
> hurting ourselves too much. This is -unstable, after all.

Question:  Which tree(s) are run through the XenRT test suite?  Is this
only xen-unstable.hg?  If so, then merging 2.6.16-rc1 into unstable
would be very helpful.  Not only for us, but also to make it stable for
the mainline merge.  If the linux-2.6-xen.hg and ext/linux-2.6-merge.hg
trees are regression-tested as well it would be less important.  But
would probably still give us more people testing the code on different
hardware ...

cheers,

  Gerd

-- 
Gerd 'just married' Hoffmann <kraxel@suse.de>
I'm the hacker formerly known as Gerd Knorr.
http://www.suse.de/~kraxel/just-married.jpeg

Re: RE: New Release Process

[Vincent Hanquez]

On Fri, Jan 27, 2006 at 11:56:18AM +0100, Gerd Hoffmann wrote:
> Question:  Which tree(s) are run through the XenRT test suite?  Is this
> only xen-unstable.hg? 

only xen-unstable.hg get automatic testing.
The unstable+subarch tree got some manual testing from time to time.

-- 
Vincent Hanquez

Re: RE: New Release Process

[Paul Larson]

Ian Pratt wrote:
> What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?
>   
I agree that skipping 2.6.15 and going straight to the 2.6.16 latest rc 
kernels would be a good thing, however I'm confused about one thing (or 
more likely I'm just dense).  Are you looking at pulling in one of the 
alternate kernel hg trees on xenbits, or just manually updating the 
sparse tree to 2.6.16-rc1.  We have the 2.6.16 subarch merge tree, the 
2.6.16 merge tree under ext, and the hvm tree under ext also.  Is one of 
these going to become the new basis for the sparse tree?

Another question this brings up: what is the process for syncing 
relevant changes and fixes between sparse, 2.6 merge, 2.6 subarch, and 
the hvm tree?  Is there any plan to consolidate these?

Thanks,
Paul Larson

RE: RE: New Release Process

[Ian Pratt]

> Current -unstable will become 3.0.1, correct?  I am wondering 
> as the Novell guys' CFQ scheduler is turning out to be very 
> useful and with no troubles so far.  It'd be nice to get this 
> included in 3.0.1.

Yes. There's a lot of regression tests running on current -unstable at
the moment, but if users could give it a thorough kick-about too that
would be great.

We may sweep -unstable into -testing a day or two ahead of the 3.0.1
release so that work on -unstable can continue while we wait for a
couple of bug fix patches to get tested. (I'd really like to see the ip
checksum issue that effects some setups fixed in 3.0.1) 

Ian

RE: RE: New Release Process

[Ian Pratt]

> I agree that skipping 2.6.15 and going straight to the 2.6.16 
> latest rc kernels would be a good thing, however I'm confused 
> about one thing (or more likely I'm just dense).  Are you 
> looking at pulling in one of the alternate kernel hg trees on 
> xenbits, or just manually updating the sparse tree to 
> 2.6.16-rc1.  We have the 2.6.16 subarch merge tree, the
> 2.6.16 merge tree under ext, and the hvm tree under ext also. 
>  Is one of these going to become the new basis for the sparse tree?

We'd generate a sparse tree for -unstable from the linux-2.6-xen.hg.
It's easy to keep the two trees in sync using a simple script. Having
the sparse tree is still useful for the moment as it enables us to tie
together xen and tools versions with a particular dom0.   
 
> Another question this brings up: what is the process for 
> syncing relevant changes and fixes between sparse, 2.6 merge, 
> 2.6 subarch, and the hvm tree?  Is there any plan to 
> consolidate these?

hvm will be checked into -unstable, and effectively becomes obsolete.
linux-2.6-xen.hg will be kept in sync with -unstable using scripts.
ext/linux-2.6-merge.hg is where all the linux merge tidyup work is going
on, and will be updated manualy.

Ian 

RE: RE: New Release Process

[Wahlig, Elsie]

Ian, 

I agree with your thoughts here.  The changes are orthogonal.  
I think you can go straight to 2.6.16-rcX.

Elsie  

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Ian Pratt
> Sent: Thursday, January 26, 2006 5:34 PM
> To: Nakajima, Jun; Anthony Liguori; xen-devel
> Subject: RE: [Xen-devel] RE: New Release Process
> 
>  
> > I think it would be better if we incorporate them one by one, 
> > not them together on the _same_ day (I doubt you are doing 
> > that, though), because we can debug effectively focusing on 
> > fewer problems. For example, 1. hvm, 2. sanity testing (a day 
> > or two), 3. 2.6.15 or 2.6.16-rcX
> 
> Normally I'd totally agree, but these changes are actually quite
> orthogonal: hvm basically touches just xen, and the linux tree upgrade
> is self contained. Those doing hvm testing could carry on 
> using a 2.6.12
> dom0 kernel from this week, just to keep things isolated.
> 
> Whether we should go straight to 2.6.16-rc1, or whether we 
> should go via
> 2.6.12-subarchxen and 2.6.15 is less clear. 2.6.12-subarch and 2.6.15
> both seem pretty stable on 32b, but x86_64 needs more testing. I'd
> certainly be inclined to check-in each of those trees, even 
> if we didn't
> let them mature at the tip for very long. People could then at least
> roll-back for 'binary chop' purposes.
> 
> views?
> 
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 

RE: RE: New Release Process

[Nakajima, Jun]

Ian Pratt wrote:
>> I think it would be better if we incorporate them one by one,
>> not them together on the _same_ day (I doubt you are doing
>> that, though), because we can debug effectively focusing on
>> fewer problems. For example, 1. hvm, 2. sanity testing (a day
>> or two), 3. 2.6.15 or 2.6.16-rcX
> 
> Normally I'd totally agree, but these changes are actually quite
> orthogonal: hvm basically touches just xen, and the linux tree upgrade
> is self contained. Those doing hvm testing could carry on using a
> 2.6.12 dom0 kernel from this week, just to keep things isolated.

The hvm stuff heavily depends on dom0, tools, and xen. If the developers
don't know if the hvm is broken or other things are broken, they won't
move to the new tree until the things get stable, i.e. fewer people will
be working on the unstable tree. Given the number (easily >20) of people
working on hvm (not just Intel), to accelerate the process I think a day
of checkpointing would be worth it (there can be potential merge errors,
too). That way we can get more people debugging problems with the new
linux side in the unstable because those will be the blocking issues for
them eventually.

> 
> Whether we should go straight to 2.6.16-rc1, or whether we should go
> via 
> 2.6.12-subarchxen and 2.6.15 is less clear. 2.6.12-subarch and 2.6.15
> both seem pretty stable on 32b, but x86_64 needs more testing. I'd
> certainly be inclined to check-in each of those trees, even if we
> didn't let them mature at the tip for very long. People could then at
> least roll-back for 'binary chop' purposes.
> 
> views?

One way would be to have multiple linux trees in the unstable, and make
it possible to choose which linux tree to build (at make world time, for
example). If one is more stable than the other one, we can debug more
efficiently because we have more data points. 

I think having both 2.6.15 and 2.6.16-rc1 trees is the shortest way to
get 2.6.16 ; presumably 2.6.15 is much closer to 2.6.16 (than
2.6.12-subarch). As we get 2.6.15 stable, we would duplicate the fixes
to 2.6.16-rcX. 

> 
> Ian


Jun
---
Intel Open Source Technology Center

RE: RE: New Release Process

[Ian Pratt]

 
> I think it would be better if we incorporate them one by one, 
> not them together on the _same_ day (I doubt you are doing 
> that, though), because we can debug effectively focusing on 
> fewer problems. For example, 1. hvm, 2. sanity testing (a day 
> or two), 3. 2.6.15 or 2.6.16-rcX

Normally I'd totally agree, but these changes are actually quite
orthogonal: hvm basically touches just xen, and the linux tree upgrade
is self contained. Those doing hvm testing could carry on using a 2.6.12
dom0 kernel from this week, just to keep things isolated.

Whether we should go straight to 2.6.16-rc1, or whether we should go via
2.6.12-subarchxen and 2.6.15 is less clear. 2.6.12-subarch and 2.6.15
both seem pretty stable on 32b, but x86_64 needs more testing. I'd
certainly be inclined to check-in each of those trees, even if we didn't
let them mature at the tip for very long. People could then at least
roll-back for 'binary chop' purposes.

views?

Ian

RE: RE: New Release Process

[Nakajima, Jun]

Ian Pratt wrote:
>>> I was hoping you could clarify what the decisions were for the new
>>> release process that you proposed at the Winter XenSummit.
>> 
>> We decided to try to aim for ~6 week intervals for 3.0.x
>> releases, stablizing the tree in -unstable then doing the
>> release and sweeping the code into 3.0-testing. We'll then
>> try and backport critical fixes from -unstable into
>> 3.0-testing and spin new 3.0.x-y build numbers as required.
>> Any similarity to the Linux process is purely intentional :)
> 
> Here's my thoughts on how we should kick-off with the new release
> process:
> 
> It's been over 6 weeks since the 3.0.0 release, and the -unstable tree
> is actually looking pretty good right now -- two of the bugs I
> mentioned yesterday are now fixed.
> 
> My current inclination is to call a 3.0.1 release Friday/Saturday and
> sweep the tree into -testing. Monday morning we'd then incorporate hvm
> and the 2.6.15 tree and work flat out to get that fully tested and
> stabilized ASAP, so SuSE can pick it up for SLES10.

I think it would be better if we incorporate them one by one, not them
together on the _same_ day (I doubt you are doing that, though), because
we can debug effectively focusing on fewer problems. For example,
1. hvm, 2. sanity testing (a day or two), 3. 2.6.15 or 2.6.16-rcX

This way we can get more hvm people focusing on 2.6.15 or 2.6.16-rcX.

> 
> SuSE have said they are actually going to base their release off
> 2.6.16, even though we're still likely to be on 2.6.16-rcX by their
> freeze date. One thing we could do to help them is to break with
> tradition and to check the 2.6.16-rcX into the tree rather than the
> most recent stable release, 2.6.15. This would help get 2.6.16
> stabilized quicker, which would certainly help them. 2.6.16 is
> already at rc1, which means that many of the 'rough edges' should
> have been found, so I doubt we'll be hurting ourselves too much. This
> is -unstable, after all. 

I like the idea of prefetching. 

> 
> What do other developers feel about trying to help SuSE out like this?
> No doubt we might have to end up doing something similar for RH come
> the RHEL5 freeze date. My feeling is that its in the xen community's
> interest to have the best possible vendor releases, as the users
> always end up coming to our mailing lists to complain :)
> 
> What do you think? Should we stick with 2.6.15 or go to 2.6.16-rc1 ?

Having the best possible vendor releases will definitely make our life
easier. I think we should go with 2.6.16-rc1. 

> 
> Any reason not to call 3.0.1 now? There are a load of bug fixes and
> improvements over 3.0.0.
> 
> Ian
> 
I think it's a good idea to release 3.0.1 at this point. 

Jun
---
Intel Open Source Technology Center

RE: New Release Process

[Ian Pratt]

 
> I was hoping you could clarify what the decisions were for 
> the new release process that you proposed at the Winter XenSummit.

We decided to try to aim for ~6 week intervals for 3.0.x releases,
stablizing the tree in -unstable then doing the release and sweeping the
code into 3.0-testing. We'll then try and backport critical fixes from
-unstable into 3.0-testing and spin new 3.0.x-y build numbers as
required. Any similarity to the Linux process is purely intentional :)

We've got a bunch of mergeing to do this week and then plenty of
testing... 

There are a couple of bugs I'd really like to see fixed ASAP: 
 * gnttab_transfer: out-of-range or xen frame xxxxx001
 * the various checksum offload issues
 * investigate the reported memory leak with some routed network configs
 * save/restore problem with block devices under load

Ian

Re: RE: New Release Process

[Matt Ayres]

>  * investigate the reported memory leak with some routed network configs

Where can I read more on this (ie. bug #, thread title, etc) ? I'd like 
to see how my routed setup relates to the others.

Re: RE: New Release Process

[Patrick Scharrenberg]

Hi Ian!


> > There are a couple of bugs I'd really like to see fixed ASAP: 
> > [...]
> >  * investigate the reported memory leak with some routed network configs
> > [...]
>   
This seems to be fixed. I run this server for ~6 days now with the
unstable code from ~6days ago and the same config.
Since I couln't see the memory leak anymore I thought you fixed it earlier.

Patrick