3.0.3 freeze

3.0.3 freeze

[Ian Pratt]

Folks,

I think its time to declare a 3.0.3 feature freeze. We've got all the
'must have' feature patches in -unstable and many of the 'would be nice'
variety too.

Just to summarize, we have: the new scheduler, blktap with file-based VM
storage, upgraded device emulation, new shadow pagetable code, PV
drivers for HVM guests, networking support for segmentation offload,
support for the Power architecture, misc performance optimizations and
bug fixes.

There were a number of patches that haven't quite made the cut off due
to various outstanding issues or lack of review time: The NUMA allocator
patch has been observed to cause problems on at least one system. The PV
framebuffer could do with a few interface tweaks and a code cleanup. The
kexec/kdump patch just needs more testing [I feel bad about this one --
maybe we can retrofit it]. The xend lifecycle management patches will be
held-over to the next release so it can become part of a larger set of
control tool changes.

Now the tree is in 'freeze' state, please can everyone get ready to do
some serious testing. We're still working on a handful of known issues
we can reproduce, so this message isn't quite a full call to arms for
testing from the user community yet, but it would be great if developers
could start giving it a workout.  

Thanks,
Ian

Re: 3.0.3 freeze

[Sean Dague]

[-- Attachment #1.1: Type: text/plain, Size: 1867 bytes --]

On Tue, Aug 22, 2006 at 05:03:27PM +0100, Ian Pratt wrote:
> 
> Folks,
> 
> I think its time to declare a 3.0.3 feature freeze. We've got all the
> 'must have' feature patches in -unstable and many of the 'would be nice'
> variety too.
> 
> Just to summarize, we have: the new scheduler, blktap with file-based VM
> storage, upgraded device emulation, new shadow pagetable code, PV
> drivers for HVM guests, networking support for segmentation offload,
> support for the Power architecture, misc performance optimizations and
> bug fixes.
> 
> There were a number of patches that haven't quite made the cut off due
> to various outstanding issues or lack of review time: The NUMA allocator
> patch has been observed to cause problems on at least one system. The PV
> framebuffer could do with a few interface tweaks and a code cleanup. The
> kexec/kdump patch just needs more testing [I feel bad about this one --
> maybe we can retrofit it]. The xend lifecycle management patches will be
> held-over to the next release so it can become part of a larger set of
> control tool changes.

If at all possible, it would be really nice to get the dm-userspace patches
in that came out last night.  They have been heavily tested by the IBM team,
and shouldn't have any negative effect on the rest of the code base.

> Now the tree is in 'freeze' state, please can everyone get ready to do
> some serious testing. We're still working on a handful of known issues
> we can reproduce, so this message isn't quite a full call to arms for
> testing from the user community yet, but it would be great if developers
> could start giving it a workout.  
> 
> Thanks,
> Ian

	-Sean

-- 
Sean Dague
IBM Linux Technology Center                     email: japh@us.ibm.com
Open Hypervisor Team                           alt: sldague@us.ibm.com

[-- Attachment #1.2: Type: application/pgp-signature, Size: 191 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: 3.0.3 freeze

[Ryan Harper]

* Ian Pratt <m+Ian.Pratt@cl.cam.ac.uk> [2006-08-22 11:33]:
> 
> Folks,
> 
> I think its time to declare a 3.0.3 feature freeze. We've got all the
> 'must have' feature patches in -unstable and many of the 'would be nice'
> variety too.
> 
> Just to summarize, we have: the new scheduler, blktap with file-based VM
> storage, upgraded device emulation, new shadow pagetable code, PV
> drivers for HVM guests, networking support for segmentation offload,
> support for the Power architecture, misc performance optimizations and
> bug fixes.
> 
> There were a number of patches that haven't quite made the cut off due
> to various outstanding issues or lack of review time: The NUMA allocator
> patch has been observed to cause problems on at least one system. The PV

While I'm waiting to see if Raj got xentop to work (I'm confident that
the issue is addressed by the latest set of patches), his latest reply
indicates that the major issue (dom0 slowdown) is no longer present.

Raj, please reply with where you are at.  

It would be nice to include the NUMA patches if there are no outstanding
issues.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

RE: 3.0.3 freeze

[Subrahmanian, Raj]

Ryan,
> While I'm waiting to see if Raj got xentop to work (I'm 
> confident that the issue is addressed by the latest set of 
> patches), his latest reply indicates that the major issue 
> (dom0 slowdown) is no longer present.
> Raj, please reply with where you are at.  
> It would be nice to include the NUMA patches if there are no 
> outstanding issues.
My machine has been pre-empted again, so I have not had a chance to try
that out yet. I will try this as soon as I can. 
I was having other responsiveness issues as well. I was ssh-ed into the
system and the response would be quite erratic. I doubt it was the
network(since everything else on the network was running fine), so I
would like to try this patch a little more before I sign off on it.
I really appreciate the patience shown by all concerned.
Thanks
Raj
Xen Development Team
Unisys Corp.

Re: 3.0.3 freeze

[Tuan Van]

Ian Pratt wrote:
> Folks,
>
> I think its time to declare a 3.0.3 feature freeze. We've got all the
> 'must have' feature patches in -unstable and many of the 'would be nice'
> variety too.
>
>   
would someone please take a look at
http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=741 before
release 3.0.3 ?

Thanks,
Tuan

Re: 3.0.3 freeze

[Daniel P. Berrange]

On Tue, Aug 22, 2006 at 05:03:27PM +0100, Ian Pratt wrote:
> I think its time to declare a 3.0.3 feature freeze. We've got all the
> 'must have' feature patches in -unstable and many of the 'would be nice'
> variety too.

[snip]

> Now the tree is in 'freeze' state, please can everyone get ready to do
> some serious testing. We're still working on a handful of known issues
> we can reproduce, so this message isn't quite a full call to arms for
> testing from the user community yet, but it would be great if developers
> could start giving it a workout.  

Is the current bug fixing leading upto 3.0.3 still taking place on the
xen-unstable.hg tree, or has it branched & moved to another tree now ?

The 3.0 tree listed on xenbits doesn't seem to have any recent updates

  http://xenbits.xensource.com/xen-3.0-testing.hg

But xen-unstable.hg is receiving major changes that we really were not
expecting for a tree that is in freeze state. For example, the following
changeset that went into xen-unstable at the weekend will likely break
compatabilty for libvirt in quite a major way

  http://xenbits.xensource.com/xen-unstable.hg?cs=86d26e6ec89b

  "Replace dom0_ops hypercall with three new hypercalls:
    1. platform_op -- used by dom0 kernel to perform actions on the
       hardware platform (e.g., MTRR access, microcode update, platform
       quirks, ...)
    2. domctl -- used by management tools to control a specified domain
    3. sysctl -- used by management tools for system-wide actions"

Is this change going into 3.0.3, or is xen-unstable now reflecting the
development for 3.0.4 ?  We've got the means to support this new format
for hypercalls in libvirt, while keeping compatability for old API 
(detectable at runtime), but if its not going to 3.0.3 we can postpone 
this dev work... 

Regards,
Dan
--
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.3 freeze

[Keir Fraser]

On 28/8/06 3:58 pm, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> Is this change going into 3.0.3, or is xen-unstable now reflecting the
> development for 3.0.4 ?  We've got the means to support this new format
> for hypercalls in libvirt, while keeping compatability for old API
> (detectable at runtime), but if its not going to 3.0.3 we can postpone
> this dev work... 

All bug fixing is going on in the unstable tree: there has been no fork. I'm
now done with major refactorings (domctl/sysctl on Friday; shadow2->shadow
today). I wanted those in before 3.0.3 because, although large, they are
unlikely to break anything, and it is a pain to move patches across branches
after a fork when only one branch has a major code reorg applied to it. I
hope you'll agree that the hypercall refactoring has resulted in a cleaner
interface than the old dom0_ops, and that it is a useful goal to allow the
dom0 kernel and tools interfaces to evolve separately from each other.

 -- Keir

Re: 3.0.3 freeze

[Daniel P. Berrange]

On Mon, Aug 28, 2006 at 04:08:00PM +0100, Keir Fraser wrote:
> On 28/8/06 3:58 pm, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > Is this change going into 3.0.3, or is xen-unstable now reflecting the
> > development for 3.0.4 ?  We've got the means to support this new format
> > for hypercalls in libvirt, while keeping compatability for old API
> > (detectable at runtime), but if its not going to 3.0.3 we can postpone
> > this dev work... 
> 
> All bug fixing is going on in the unstable tree: there has been no fork. I'm
> now done with major refactorings (domctl/sysctl on Friday; shadow2->shadow
> today). I wanted those in before 3.0.3 because, although large, they are
> unlikely to break anything, and it is a pain to move patches across branches
> after a fork when only one branch has a major code reorg applied to it. I
> hope you'll agree that the hypercall refactoring has resulted in a cleaner
> interface than the old dom0_ops, and that it is a useful goal to allow the
> dom0 kernel and tools interfaces to evolve separately from each other.

Yes, the re-factoring does look like a good idea from a long term maintenance
POV - it just caught us a little by surprise ! Thanks for clarifying the
current branch situation.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.3 freeze

[Daniel Veillard]

On Mon, Aug 28, 2006 at 04:08:00PM +0100, Keir Fraser wrote:
> 
> 
> 
> On 28/8/06 3:58 pm, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > Is this change going into 3.0.3, or is xen-unstable now reflecting the
> > development for 3.0.4 ?  We've got the means to support this new format
> > for hypercalls in libvirt, while keeping compatability for old API
> > (detectable at runtime), but if its not going to 3.0.3 we can postpone
> > this dev work... 
> 
> All bug fixing is going on in the unstable tree: there has been no fork. I'm
> now done with major refactorings (domctl/sysctl on Friday; shadow2->shadow
> today). I wanted those in before 3.0.3 because, although large, they are
> unlikely to break anything,

  Huh ? libvirt uses dom0 syscalls. And libvirt uses its own code to do it
since the existing libraries for dom0 calls are GPL'ed which would not be 
compatible with libvirt own licencing (LGPL). The headers used by libvirt
are simply removed, the ioctl entry points are changed, etc ... You really
expected that to 'not break anything' ?

> and it is a pain to move patches across branches
> after a fork when only one branch has a major code reorg applied to it. I
> hope you'll agree that the hypercall refactoring has resulted in a cleaner
> interface than the old dom0_ops,

 +#define XEN_DOMCTL_getdomaininfo     5

 +#define XEN_SYSCTL_getdomaininfolist 6

Can you explain why listing one domain info is a control operation (subject
to changes) and listing multiple domain info in a nearly identical way is a 
system operation (and supposedly slightly more stable from now on).
This patch is close to 9500 lines long, I am trying to understand what 
it contains, some of the HV calls reuse the same ioctl numbers, some have
been moved to two other calls. The structures have changed, some probably
have different size and content but it's hard to tell just by looking at
the patch. It seems that some calls have the same entry point but not the
same data to be passed down, which makes autodetection of the underlying
HV version especially tricky.
Was any HV call removed, or did they were all split between the 3 new sets ?
I will have a lot of work digesting this and making sure libvirt work 
with both old and new hypervisors.

> and that it is a useful goal to allow the
> dom0 kernel and tools interfaces to evolve separately from each other.

  The proper way in my opinion is to not break the hypercalls, if you need
changes in extensions and semantic, create new calls ! It is possible to
make graceful evolution of APIs, otherwise none of the software you are
using right now to read this mail would simply be possible.

  Yes I guess the change is useful, maybe this was neededi now, but an advance
warning would be nice before commiting something which break binary
compatibility, or at least a mail on xen-devel stating the fact that this
was changed if you really don't want any prior discussion to your commit.

  Thanks in advance,

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

Re: 3.0.3 freeze

[Keir Fraser]

On 28/8/06 4:49 pm, "Daniel Veillard" <veillard@redhat.com> wrote:

>> All bug fixing is going on in the unstable tree: there has been no fork. I'm
>> now done with major refactorings (domctl/sysctl on Friday; shadow2->shadow
>> today). I wanted those in before 3.0.3 because, although large, they are
>> unlikely to break anything,
> 
>   Huh ? libvirt uses dom0 syscalls. And libvirt uses its own code to do it
> since the existing libraries for dom0 calls are GPL'ed which would not be
> compatible with libvirt own licencing (LGPL). The headers used by libvirt
> are simply removed, the ioctl entry points are changed, etc ... You really
> expected that to 'not break anything' ?

API/ABI compatibility is not guaranteed at the low-level control-plane
interfaces: that is instead provided at the XML-RPC level (and, practically
speaking, the libxenctrl/libxenguest interface is pretty stable too). By 'no
breakage' I mean that the new interfaces work fine with a matching set of
the supported libraries libxenctrl and libxenguest.

>  +#define XEN_DOMCTL_getdomaininfo     5
> 
>  +#define XEN_SYSCTL_getdomaininfolist 6
> 
> Can you explain why listing one domain info is a control operation (subject
> to changes) and listing multiple domain info in a nearly identical way is a
> system operation (and supposedly slightly more stable from now on).

Neither of the above interfaces (sysctl/domctl) has guaranteed ongoing API
or ABI compatibility. The new platform_op interface, used by dom0 kernel, is
the only new hypercall for which we will guarantee ongoing stability.

The two commands you refer to are in different hypercalls because one is
specific to a particular domain (hence domctl) while the other is a request
for 'system-wide' info about all domains (hence sysctl).

> Was any HV call removed, or did they were all split between the 3 new sets ?

None were removed. They've simply been reassigned as sysctls or domctls to
rationalise the API.

>   The proper way in my opinion is to not break the hypercalls, if you need
> changes in extensions and semantic, create new calls ! It is possible to
> make graceful evolution of APIs, otherwise none of the software you are
> using right now to read this mail would simply be possible.

We guarantee compatibility for *all* our other interfaces. The Xen
management API is also being stabilised, but at the XML-RPC level.

 -- Keir

Re: 3.0.3 freeze

[Rik van Riel]

Keir Fraser wrote:

> API/ABI compatibility is not guaranteed at the low-level control-plane
> interfaces: that is instead provided at the XML-RPC level 

Which XML-RPC layer?

The one that's not been written yet, or the one we were
told would go away? :)

-- 
What is important?  What you want to be true, or what is true?

Re: 3.0.3 freeze

[Keir Fraser]

On 28/8/06 5:47 pm, "Rik van Riel" <riel@redhat.com> wrote:

>> API/ABI compatibility is not guaranteed at the low-level control-plane
>> interfaces: that is instead provided at the XML-RPC level
> 
> Which XML-RPC layer?
> 
> The one that's not been written yet, or the one we were
> told would go away? :)

http://wiki.xensource.com/xenwiki/XenApi
http://lists.xensource.com/archives/html/xen-api/

I.e., the one that's being drafted and discussed right now, on the xen wiki
and on the xen-api mailing list.

 -- Keir

Re: 3.0.3 freeze

[Rik van Riel]

Keir Fraser wrote:
> On 28/8/06 5:47 pm, "Rik van Riel" <riel@redhat.com> wrote:
> 
>>> API/ABI compatibility is not guaranteed at the low-level control-plane
>>> interfaces: that is instead provided at the XML-RPC level
>> Which XML-RPC layer?
>>
>> The one that's not been written yet, or the one we were
>> told would go away? :)
> 
> http://wiki.xensource.com/xenwiki/XenApi
> http://lists.xensource.com/archives/html/xen-api/
> 
> I.e., the one that's being drafted and discussed right now, on the xen wiki
> and on the xen-api mailing list.

Are you seriously suggesting that people rely on the stability
of an API that hasn't even been created yet, and that people
should not be surprised when the current API breaks, because
they should have been using the one that doesn't exist yet?

Words fail me.  At least, words I'd want to write down :)

-- 
What is important?  What you want to be true, or what is true?

Re: 3.0.3 freeze

[Keir Fraser]

On 28/8/06 6:12 pm, "Rik van Riel" <riel@redhat.com> wrote:

>> http://wiki.xensource.com/xenwiki/XenApi
>> http://lists.xensource.com/archives/html/xen-api/
>> 
>> I.e., the one that's being drafted and discussed right now, on the xen wiki
>> and on the xen-api mailing list.
> 
> Are you seriously suggesting that people rely on the stability
> of an API that hasn't even been created yet,

That is the stabilisation point for VM management. That's where effort
should be expended if you want to get to a stable, supported management
interface asap.

> and that people
> should not be surprised when the current API breaks, because
> they should have been using the one that doesn't exist yet?

There is no current ongoing supported API. We have never claimed there was.
The closest is libxenctrl/libxenguest, which hid just about all the
underlying hypervisor changes (except for bits that it doesn't fully
support, like trace-buffer access and perf counters), or the xm command
(which is stable, but it is stretching it a bit to call it an API).

 -- Keir

Re: 3.0.3 freeze

[Daniel Veillard]

On Mon, Aug 28, 2006 at 05:32:01PM +0100, Keir Fraser wrote:
> On 28/8/06 4:49 pm, "Daniel Veillard" <veillard@redhat.com> wrote:
> 
> >> All bug fixing is going on in the unstable tree: there has been no fork. I'm
> >> now done with major refactorings (domctl/sysctl on Friday; shadow2->shadow
> >> today). I wanted those in before 3.0.3 because, although large, they are
> >> unlikely to break anything,
> > 
> >   Huh ? libvirt uses dom0 syscalls. And libvirt uses its own code to do it
> > since the existing libraries for dom0 calls are GPL'ed which would not be
> > compatible with libvirt own licencing (LGPL). The headers used by libvirt
> > are simply removed, the ioctl entry points are changed, etc ... You really
> > expected that to 'not break anything' ?
> 
> API/ABI compatibility is not guaranteed at the low-level control-plane
> interfaces: that is instead provided at the XML-RPC level (and, practically

  Seriously, which one ? The one about to be deprecated or the one which
doesn't exist yet ? There is in practice no API/ABI here, only the sxp 
of xend can be considered a stable ABI so far, but that's slated for removal.
Stable doesn't mean a 6 month time span for us, really.

> speaking, the libxenctrl/libxenguest interface is pretty stable too). By 'no
> breakage' I mean that the new interfaces work fine with a matching set of
> the supported libraries libxenctrl and libxenguest.

  Which I can't use due to Licence issues, they are still GPL if I'm not 
mistaken. And anyway they won't even work on 3.0.2 on x86_64 because the
HV ABI broke in the meantime.

> >  +#define XEN_DOMCTL_getdomaininfo     5
> > 
> >  +#define XEN_SYSCTL_getdomaininfolist 6
> > 
> > Can you explain why listing one domain info is a control operation (subject
> > to changes) and listing multiple domain info in a nearly identical way is a
> > system operation (and supposedly slightly more stable from now on).
> 
> Neither of the above interfaces (sysctl/domctl) has guaranteed ongoing API
> or ABI compatibility. The new platform_op interface, used by dom0 kernel, is
> the only new hypercall for which we will guarantee ongoing stability.

  Okay, that mean we will have to work around the changes for the foreseeable
future. You don't care about being stable there, we do, at least now I 
complained publicly about it. I have reason why I care for those, we explained
them. I don't know why you don't want to garantee anything at that level.
I know that non stable kernel interfaces can change, that's a fact of life,
but things can be handled gracefully and nicely, it's also possible to be
uncooperative on purpose, I hope it is not the case.

> The two commands you refer to are in different hypercalls because one is
> specific to a particular domain (hence domctl) while the other is a request
> for 'system-wide' info about all domains (hence sysctl).

  Okay

> > Was any HV call removed, or did they were all split between the 3 new sets ?
> 
> None were removed. They've simply been reassigned as sysctls or domctls to
> rationalise the API.

  Okay thanks, it's not easy to track down because that was dispatched on
multiple files now.

> >   The proper way in my opinion is to not break the hypercalls, if you need
> > changes in extensions and semantic, create new calls ! It is possible to
> > make graceful evolution of APIs, otherwise none of the software you are
> > using right now to read this mail would simply be possible.
> 
> We guarantee compatibility for *all* our other interfaces. The Xen
> management API is also being stabilised, but at the XML-RPC level.

  And the level of CPU consumption used just for monitoring was not
acceptable last we checked, only the hypercalls allowed decent
performances.

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/

Re: 3.0.3 freeze

[Anthony Wright]

Hi Ian,

It's three weeks since you posted this message. I wondered if you could 
post an update?

Thanks,
Tony Wright.

Ian Pratt wrote:
> Folks,
>
> I think its time to declare a 3.0.3 feature freeze. We've got all the
> 'must have' feature patches in -unstable and many of the 'would be nice'
> variety too.
>
> Just to summarize, we have: the new scheduler, blktap with file-based VM
> storage, upgraded device emulation, new shadow pagetable code, PV
> drivers for HVM guests, networking support for segmentation offload,
> support for the Power architecture, misc performance optimizations and
> bug fixes.
>
> There were a number of patches that haven't quite made the cut off due
> to various outstanding issues or lack of review time: The NUMA allocator
> patch has been observed to cause problems on at least one system. The PV
> framebuffer could do with a few interface tweaks and a code cleanup. The
> kexec/kdump patch just needs more testing [I feel bad about this one --
> maybe we can retrofit it]. The xend lifecycle management patches will be
> held-over to the next release so it can become part of a larger set of
> control tool changes.
>
> Now the tree is in 'freeze' state, please can everyone get ready to do
> some serious testing. We're still working on a handful of known issues
> we can reproduce, so this message isn't quite a full call to arms for
> testing from the user community yet, but it would be great if developers
> could start giving it a workout.  
>
> Thanks,
> Ian
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>   

RE: 3.0.3 freeze

[Ian Pratt]

> It's three weeks since you posted this message. I wondered if you
could
> post an update?

I think the unstable tree is looking in pretty good shape right now.
There are a few issues being worked on with HVM, but stability for most
configurations is very good.

The more testing it gets over the next week or so, the better... Please
give it a go!

Thanks,
Ian


> Ian Pratt wrote:
> > Folks,
> >
> > I think its time to declare a 3.0.3 feature freeze. We've got all
the
> > 'must have' feature patches in -unstable and many of the 'would be
nice'
> > variety too.
> >
> > Just to summarize, we have: the new scheduler, blktap with
file-based VM
> > storage, upgraded device emulation, new shadow pagetable code, PV
> > drivers for HVM guests, networking support for segmentation offload,
> > support for the Power architecture, misc performance optimizations
and
> > bug fixes.
> >
> > There were a number of patches that haven't quite made the cut off
due
> > to various outstanding issues or lack of review time: The NUMA
allocator
> > patch has been observed to cause problems on at least one system.
The PV
> > framebuffer could do with a few interface tweaks and a code cleanup.
The
> > kexec/kdump patch just needs more testing [I feel bad about this one
--
> > maybe we can retrofit it]. The xend lifecycle management patches
will be
> > held-over to the next release so it can become part of a larger set
of
> > control tool changes.
> >
> > Now the tree is in 'freeze' state, please can everyone get ready to
do
> > some serious testing. We're still working on a handful of known
issues
> > we can reproduce, so this message isn't quite a full call to arms
for
> > testing from the user community yet, but it would be great if
developers
> > could start giving it a workout.
> >
> > Thanks,
> > Ian
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> >

Re: 3.0.3 freeze

[Anthony Wright]

Ian Pratt wrote:
>> It's three weeks since you posted this message. I wondered if you could post an update?
>>     
> I think the unstable tree is looking in pretty good shape right now.
> There are a few issues being worked on with HVM, but stability for most
> configurations is very good.
>
> The more testing it gets over the next week or so, the better... Please
> give it a go!
>
> Thanks,
> Ian
>   
Are you intending to produce an intermediate xen-3.0-testing 
pre-release, or will you be going straight from xen-unstable? (I'd 
rather be testing against a stable code base).

Also are you intending to go through bugzilla and deal with some of the 
issues there? I've noticed the reports have been steadily growing 
recently (I've got a few, but suspect some of them may have been fixed).

Thanks,
Tony.

Re: 3.0.3 freeze

[Matt Ayres]

Anthony Wright wrote:
> Ian Pratt wrote:
>>> It's three weeks since you posted this message. I wondered if you 
>>> could post an update?
>>>     
>> I think the unstable tree is looking in pretty good shape right now.
>> There are a few issues being worked on with HVM, but stability for most
>> configurations is very good.
>>
>> The more testing it gets over the next week or so, the better... Please
>> give it a go!
>>
>> Thanks,
>> Ian
>>   
> Are you intending to produce an intermediate xen-3.0-testing 
> pre-release, or will you be going straight from xen-unstable? (I'd 
> rather be testing against a stable code base).
> 

Same here.  I wasn't planning on doing production level testing until 
there was a mass merge to -testing and/or there was a release candidate.

Regards,
Matt

Re: 3.0.3 freeze

[John Lenz]

On 09/12/06 08:45, Matt Ayres wrote:
> 
> Same here.  I wasn't planning on doing production level testing until
> there was a mass merge to -testing and/or there was a release candidate.
> 

Me too.  I was waiting for an RC or -testing release.  But if that isn't
going to happen, I will test the latest unstable tree.  Just try and
give a date when you think unstable is not in a broken state?

Thanks,
John

Re: 3.0.3 freeze

[Nicholas Lee]

[-- Attachment #1.1: Type: text/plain, Size: 152 bytes --]

As probably a lot of people, I'm waiting for a release into testing.

Especially with the HVM issues I wont be testing the current unstable.


Nicholas

[-- Attachment #1.2: Type: text/html, Size: 167 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: 3.0.3 freeze

[Anthony Wright]

Ian Pratt wrote:
> I think the unstable tree is looking in pretty good shape right now.
> There are a few issues being worked on with HVM, but stability for most
> configurations is very good.
>
> The more testing it gets over the next week or so, the better... Please
> give it a go!
>
> Thanks,
> Ian
>   

Hi Ian,

You sent this a week ago, so I wondered if you'd mine bring us up to date?

It would be really helpful to know what you're waiting for before doing 
the freeze, whether there will be a release candidate process and 
whether you'll be do a run through bugzilla to make sure the significant 
bugs have been fixed?

Thanks,

Tony.

Re: 3.0.3 freeze

[Keir Fraser]

On 19/9/06 12:21, "Anthony Wright" <anthony@communitymesh.com> wrote:

> You sent this a week ago, so I wondered if you'd mine bring us up to date?
> 
> It would be really helpful to know what you're waiting for before doing
> the freeze, whether there will be a release candidate process and
> whether you'll be do a run through bugzilla to make sure the significant
> bugs have been fixed?

We'll be forking a 3.0.3-testing tree as soon as we have upgraded the Linux
sparse tree to 2.6.16.29 (which among other things fixes a number of
security vulnerabilities). Hopefully this will happen later today. This tree
can be considered a 'rolling' release candidate which we'd like to get
tested as widely as possible. We'll likely make a final release of 3.0.3-0
in the next couple of weeks.

Although we keep tabs on bugzilla, we'd very much like to receive updates,
on xen-devel and/or by updating bugzilla entries, on what bugs are still
live in the 3.0.3-testing so we can focus our efforts.

 -- Keir

RE: 3.0.3 freeze

[Ian Pratt]

> But xen-unstable.hg is receiving major changes that we really 
> were not expecting for a tree that is in freeze state. For 
> example, the following changeset that went into xen-unstable 
> at the weekend will likely break compatabilty for libvirt in 
> quite a major way
> 
>   http://xenbits.xensource.com/xen-unstable.hg?cs=86d26e6ec89b
> 
>   "Replace dom0_ops hypercall with three new hypercalls:
>     1. platform_op -- used by dom0 kernel to perform actions on the
>        hardware platform (e.g., MTRR access, microcode 
> update, platform
>        quirks, ...)
>     2. domctl -- used by management tools to control a 
> specified domain
>     3. sysctl -- used by management tools for system-wide actions"

I didn't think we were going to get this patch in time for 3.0.3, but
when it arrived we decided it was low risk (it's mostly just a code
refactoring) and dropped it in. It then got stuck in the staging tree
for a couple of days behind some SMP HVM-related malaise.

The big advantage this patch gives us is that it should now be possible
to maintain the dom0 to hypervisor ABI in a backward compatible fashion
--- we've previously only ever guaranteed this for domU's. The
hypervisor and control stack will still need to be a matched set for the
time being, but using a any 3.0.3 compliant kernel as a dom0 should now
work on a 3.0.4 hypervisor etc.

Ian

RE: 3.0.3 freeze

[Ian Pratt]

> > All bug fixing is going on in the unstable tree: there has been no 
> > fork. I'm now done with major refactorings (domctl/sysctl 
> on Friday; 
> > shadow2->shadow today). I wanted those in before 3.0.3 because, 
> > although large, they are unlikely to break anything,
> 
>   Huh ? libvirt uses dom0 syscalls. And libvirt uses its own 
> code to do it since the existing libraries for dom0 calls are 
> GPL'ed which would not be compatible with libvirt own 
> licencing (LGPL). The headers used by libvirt are simply 
> removed, the ioctl entry points are changed, etc ... You 
> really expected that to 'not break anything' ?

We were only thinking in terms of the internal consistency of the tree
and the public APIs and libraries -- I didn't realise libvirt was going
directly at the dom0_op ABI rather than using the libxenguest or
libxenctrl libraries.

If your reason for doing this was just the GPL'ness of the libraries
then I assume you'd support the email I posted a few weeks ago
suggesting we try to change the library license to LGPL? Would you then
start using the libraries?

Thanks,
Ian  

Re: 3.0.3 freeze

[Daniel P. Berrange]

On Mon, Aug 28, 2006 at 05:48:56PM +0100, Ian Pratt wrote:
> > > All bug fixing is going on in the unstable tree: there has been no 
> > > fork. I'm now done with major refactorings (domctl/sysctl 
> > on Friday; 
> > > shadow2->shadow today). I wanted those in before 3.0.3 because, 
> > > although large, they are unlikely to break anything,
> > 
> >   Huh ? libvirt uses dom0 syscalls. And libvirt uses its own 
> > code to do it since the existing libraries for dom0 calls are 
> > GPL'ed which would not be compatible with libvirt own 
> > licencing (LGPL). The headers used by libvirt are simply 
> > removed, the ioctl entry points are changed, etc ... You 
> > really expected that to 'not break anything' ?
> 
> We were only thinking in terms of the internal consistency of the tree
> and the public APIs and libraries -- I didn't realise libvirt was going
> directly at the dom0_op ABI rather than using the libxenguest or
> libxenctrl libraries.
> 
> If your reason for doing this was just the GPL'ness of the libraries
> then I assume you'd support the email I posted a few weeks ago
> suggesting we try to change the library license to LGPL? Would you then
> start using the libraries?

I don't think the libraries match our use case since they only ever
support the current version of the HV api. libvirt meanwhile supports
*all* historic versions of the HV that it needs. So if someone upgrades
libvirt to a newer release, but doesn't upgrade their HV, their management
layer keeps working reliably - libvirt checks the HV version at runtime and
decides the calls to use. So if we switched to the libraries we'd loose
this compatability with different HV versions

Regards,
Dan
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.3 freeze

[Daniel Veillard]

On Mon, Aug 28, 2006 at 05:48:56PM +0100, Ian Pratt wrote:
> > > All bug fixing is going on in the unstable tree: there has been no 
> > > fork. I'm now done with major refactorings (domctl/sysctl 
> > on Friday; 
> > > shadow2->shadow today). I wanted those in before 3.0.3 because, 
> > > although large, they are unlikely to break anything,
> > 
> >   Huh ? libvirt uses dom0 syscalls. And libvirt uses its own 
> > code to do it since the existing libraries for dom0 calls are 
> > GPL'ed which would not be compatible with libvirt own 
> > licencing (LGPL). The headers used by libvirt are simply 
> > removed, the ioctl entry points are changed, etc ... You 
> > really expected that to 'not break anything' ?
> 
> We were only thinking in terms of the internal consistency of the tree
> and the public APIs and libraries -- I didn't realise libvirt was going
> directly at the dom0_op ABI rather than using the libxenguest or
> libxenctrl libraries.

  The problem is that considering the state of a snapshot at one point in
time is not sufficient. You cannot assume everything will be upgraded as
one shot and restarted clean. There isn't any reason a Dom0 kernel and the
hypervisor should not run for years even if the DomU and userland Dom0
are upgraded independantly to cover bug fixes and so on. We all understand
it is not the kind of garantee that you feel okay to provide, that's
normal and fine, but we need to cooperate to ultimately provide this
level of garantee to users. If I were using a libxenctrl library from
beginning of July libvirt would have stopped working for everybody mid-July
and if linking against a library from 3 weeks ago, with changeset 86d26e6ec89b
this would break again. We really want to get the xen bits tested, we 
(especially Juan) are working hard to get early testing, but if the whole
stack breaks down (libvirt failing means the guest installer fails) too
often, this takes a lot of effort from us, and this turns off the early
testers. This is a loosing situation. I do understand your statement of
not willing to guarantee HV call stability, because technically things
will have to evolve for example as part of getting in the upstream Linux
kernel, some things certainly will change, I'm fine with it, but I'm not
happy with unilateral changes which are not discuted or at least announced.
I'm firmly convinced this reduces early testing and generate more error
than we should see if thing were planned and introduced smoothly. 

> If your reason for doing this was just the GPL'ness of the libraries
> then I assume you'd support the email I posted a few weeks ago
> suggesting we try to change the library license to LGPL? Would you then
> start using the libraries?

  That would be a step in the right direction, I would still need to copy
some of this code instead of rewriting it when there are change, since as
Daniel Berrange pointed out too, we need the stack to work for a longuer
timeframe, which means detecting the HV version and using the right code.
That would reduce the work needed when the hypervisor call changes but 
I won't be able to use it as-is. 
  What is sad is that changeset 86d26e6ec89b is also a step in the right
direction, it's just the way update to this very sensible part happened
which generated troubles, it's also annoying that it won't solve the problem
for people on ppc64, maybe this could have been fixed if that had been 
worked on the xen-devel list for some time. Obviously this change is not
new, that 9500 line patch certainly tooks some time to make, is there
anything preventing discussion about it ? Is that just lack of time ?
This kind of changes is not the average addition of a new feature, it
highly affects perceived stability for every testers, and I hope that by
discussing those in advance we could improve the speed of development,
testing and ease deployments.

  Yours,

Daniel

-- 
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard      | virtualization library  http://libvirt.org/
veillard@redhat.com  | libxml GNOME XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine  http://rpmfind.net/