New release candidate for Xen 4.0.0 (RC9)

New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

Folks,

4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg

This is almost certainly the last RC, and will become the 4.0.0 release
around the middle of next week. Please test!

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 314 bytes --]

On 03/31/2010 11:27 AM, Keir Fraser wrote:
> Folks,
> 
> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
> 
> This is almost certainly the last RC, and will become the 4.0.0 release
> around the middle of next week. Please test!
> 
BTW, any estimates for the Xen 3.4.3?

j.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 11:07, "Joanna Rutkowska" <joanna@invisiblethingslab.com>
wrote:

> On 03/31/2010 11:27 AM, Keir Fraser wrote:
>> Folks,
>> 
>> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
>> 
>> This is almost certainly the last RC, and will become the 4.0.0 release
>> around the middle of next week. Please test!
>> 
> BTW, any estimates for the Xen 3.4.3?

Later in April.

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Joanna Rutkowska]

[-- Attachment #1.1: Type: text/plain, Size: 986 bytes --]

On 03/31/2010 12:54 PM, Keir Fraser wrote:
> On 31/03/2010 11:07, "Joanna Rutkowska" <joanna@invisiblethingslab.com>
> wrote:
> 
>> On 03/31/2010 11:27 AM, Keir Fraser wrote:
>>> Folks,
>>>
>>> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
>>>
>>> This is almost certainly the last RC, and will become the 4.0.0 release
>>> around the middle of next week. Please test!
>>>
>> BTW, any estimates for the Xen 3.4.3?
> 
> Later in April.
BTW, on this page:

http://xenbits.xensource.com/

one can see linux-2.6.18-xen.hg repo as a sub-repo of the just-created
xen-4.0-testing.hg... I was under impression that 4.0 would be using
pvops0 kernel *only* and that you would not support 2.6.18 anymore for
this hypervisor...

Can you shed some light on this issue -- why is this kernel repo there,
and what kernel will *really* be the official and stable option for Xen
4? What will be part of the next week release?

Thanks,
joanna.


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 226 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 12:00, "Joanna Rutkowska" <joanna@invisiblethingslab.com>
wrote:

> BTW, on this page:
> 
> http://xenbits.xensource.com/
> 
> one can see linux-2.6.18-xen.hg repo as a sub-repo of the just-created
> xen-4.0-testing.hg... I was under impression that 4.0 would be using
> pvops0 kernel *only* and that you would not support 2.6.18 anymore for
> this hypervisor...
> 
> Can you shed some light on this issue -- why is this kernel repo there,
> and what kernel will *really* be the official and stable option for Xen
> 4? What will be part of the next week release?

The Xen source repo will build pv_ops by default. You can use whatever
kernel you like as dom0/domU: that's not really an aspect of the release.

 -- Keir

RE: New release candidate for Xen 4.0.0 (RC9)

[Ian Pratt]

> http://xenbits.xensource.com/

That page seriously needs fixing to correctly reflect the active trees.
 
> one can see linux-2.6.18-xen.hg repo as a sub-repo of the just-created
> xen-4.0-testing.hg... I was under impression that 4.0 would be using
> pvops0 kernel *only* and that you would not support 2.6.18 anymore for
> this hypervisor...

I don't think it makes sense for anyone to really be using 2.6.18 anymore.

The 2.6.27 tree is probably the most widely tested right now, and is shipping in commercial xen distros, and xen.org's Xen Cloud Platform and XCI. 

pvops is certainly where the development effort is currently focussed, and hopefully what the commercial distros will all be using later this year[*].

Ian

[*] Novell also have a 2.6.32 'classic xen patch port', not to be confused with pvops



> Can you shed some light on this issue -- why is this kernel repo there,
> and what kernel will *really* be the official and stable option for Xen 4?
> What will be part of the next week release?
> 
> Thanks,
> joanna.

RE: New release candidate for Xen 4.0.0 (RC9)

[Dan Magenheimer]

> From: Ian Pratt [mailto:Ian.Pratt@eu.citrix.com]
> To: Joanna Rutkowska; Keir Fraser
> Subject: RE: [Xen-devel] New release candidate for Xen 4.0.0 (RC9)
> 
> > http://xenbits.xensource.com/
> 
> That page seriously needs fixing to correctly reflect the active trees.
> 
> > one can see linux-2.6.18-xen.hg repo as a sub-repo of the just-
> created
> > xen-4.0-testing.hg... I was under impression that 4.0 would be using
> > pvops0 kernel *only* and that you would not support 2.6.18 anymore
> for
> > this hypervisor...
> 
> I don't think it makes sense for anyone to really be using 2.6.18
> anymore.

Sorry, I have to disagree.  The only real issue with 2.6.18 is
old drivers.  If your machine works fine with 2.6.18 dom0, it
is still very likely the most stable and fully-functional dom0
bits and has been tested with many previous releases of Xen
and was the default for pre-4.0 until a few months ago.
One need only look at the changelog to see it is still
actively used: http://xenbits.xensource.com/linux-2.6.18-xen.hg 
I have used this tree for all of my 4.0 development and testing.

You will likely have problems with it on newer hardware...
though I have been developing/testing fine on a Nehalem box.
 
> The 2.6.27 tree is probably the most widely tested right now, and is
> shipping in commercial xen distros, and xen.org's Xen Cloud Platform
> and XCI.

I don't want to get into a numbers argument, but all Oracle VM
shipments are using a 2.6.18-based dom0.  And I'd venture to
guess that, since most enterprise shops rarely move immediately to
the leading edge release, the vast majority of commercial Xen users
are using a 2.6.18-based dom0.

> pvops is certainly where the development effort is currently focussed,
> and hopefully what the commercial distros will all be using later this
> year[*].

Yes, agreed.  This is definitely the direction.  But I don't
think it's time to throw away linux-2.6.18-xen.hg quite yet
(the suggestion of which is what triggered my allergic reaction
here :-).
 
> > Can you shed some light on this issue -- why is this kernel repo
> there,
> > and what kernel will *really* be the official and stable option for
> Xen 4?
> > What will be part of the next week release?

Keir already answered this tersely, but let me explain further
(and others can correct me if I am wrong).  There IS NO official
dom0 for Xen 4.0 from xen.org.  Xen 4.0 is a hypervisor not
a virtualization distro.  Similarly, Linux 2.6.33 is a kernel release,
not an OS distro.  Distros of both are free to choose whatever other
components work best for their customers.

So IMHO the http://xenbits.xensource.com/linux-2.6.18-xen.hg tree
is still maintained because it is still useful for a significant
number of developers.

P.S. I *do* plan to switch to pvops... but I've been saying that
for over a year now ;-)

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 15:20, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

>>> Can you shed some light on this issue -- why is this kernel repo
>> there,
>>> and what kernel will *really* be the official and stable option for
>> Xen 4?
>>> What will be part of the next week release?
> 
> Keir already answered this tersely, but let me explain further
> (and others can correct me if I am wrong).  There IS NO official
> dom0 for Xen 4.0 from xen.org.  Xen 4.0 is a hypervisor not
> a virtualization distro.  Similarly, Linux 2.6.33 is a kernel release,
> not an OS distro.  Distros of both are free to choose whatever other
> components work best for their customers.

Yes, that's pretty much the situation. Basically, when moving to 4.0 there
is no need to change any dom0 or domU kernel (unless you want to use certain
new features like Remus or blktap2, and your current dom0 doesn't have the
required bits). Of course that doesn't mean we abdicate responsibility for
producing high-quality kernels for running on Xen -- or at least HQ
Xen-specific bits that can then be pulled into HQ kernel releases by
distros. I think trying to publish "kernel releases" for end users would
actually be a dangerous thing for us to do, unless we were also prepared to
deal with things like security updates in a timely fashion, and we haven't
got a channel for distributing that kind of thing in the same effective way
that distros have.

 -- Keir

RE: New release candidate for Xen 4.0.0 (RC9)

[Ian Pratt]

> > I don't think it makes sense for anyone to really be using 2.6.18
> > anymore.
> 
> Sorry, I have to disagree.  The only real issue with 2.6.18 is
> old drivers.  If your machine works fine with 2.6.18 dom0, it
> is still very likely the most stable and fully-functional dom0
> bits and has been tested with many previous releases of Xen
> and was the default for pre-4.0 until a few months ago.

That's a very big "if". The 2.6.18 available for download from xenbits has not been actively kept up to date with new drivers etc. 

If you really want to use 2.6.18, at least use an actively maintained version e.g. from XenServer 5.5 or OracleVM, CentOS etc.

> > The 2.6.27 tree is probably the most widely tested right now, and is
> > shipping in commercial xen distros, and xen.org's Xen Cloud Platform
> > and XCI.
> 
> I don't want to get into a numbers argument, but all Oracle VM
> shipments are using a 2.6.18-based dom0.  And I'd venture to
> guess that, since most enterprise shops rarely move immediately to
> the leading edge release, the vast majority of commercial Xen users
> are using a 2.6.18-based dom0.

2.6.27 is hardly leading edge for use with Xen. Novell released SLES11 over a year ago. 

Over the last year, there will have been a lot more test resources focussed on the combination of modern xen and 2.6.27 than there have been on 2.6.18.

You may be right that there is more 2.6.18 in enterprise use, but the vast majority of that will be in combination with older versions of Xen.

Ian



> > pvops is certainly where the development effort is currently focussed,
> > and hopefully what the commercial distros will all be using later this
> > year[*].
> 
> Yes, agreed.  This is definitely the direction.  But I don't
> think it's time to throw away linux-2.6.18-xen.hg quite yet
> (the suggestion of which is what triggered my allergic reaction
> here :-).
> 
> > > Can you shed some light on this issue -- why is this kernel repo
> > there,
> > > and what kernel will *really* be the official and stable option for
> > Xen 4?
> > > What will be part of the next week release?
> 
> Keir already answered this tersely, but let me explain further
> (and others can correct me if I am wrong).  There IS NO official
> dom0 for Xen 4.0 from xen.org.  Xen 4.0 is a hypervisor not
> a virtualization distro.  Similarly, Linux 2.6.33 is a kernel release,
> not an OS distro.  Distros of both are free to choose whatever other
> components work best for their customers.
> 
> So IMHO the http://xenbits.xensource.com/linux-2.6.18-xen.hg tree
> is still maintained because it is still useful for a significant
> number of developers.
> 
> P.S. I *do* plan to switch to pvops... but I've been saying that
> for over a year now ;-)

Re: New release candidate for Xen 4.0.0 (RC9)

[Grant McWilliams]

[-- Attachment #1.1: Type: text/plain, Size: 840 bytes --]

Grant McWilliams

Some people, when confronted with a problem, think "I know, I'll use
Windows."
Now they have two problems.


> pvops is certainly where the development effort is currently focussed,
> > and hopefully what the commercial distros will all be using later this
> > year[*].
>
> Yes, agreed.  This is definitely the direction.  But I don't
> think it's time to throw away linux-2.6.18-xen.hg quite yet
> (the suggestion of which is what triggered my allergic reaction
> here :-).
>
>
Do we yet have any real numbers on the performance hit with the move to
pvops? We HAVE to at some point move but I think part of the decision on
WHEN to move will be based on this. I've seen various messages over the
years about "we have to sit down and figure out how big the hit is and what
to do about it".

Just curious.

Grant McWilliams

[-- Attachment #1.2: Type: text/html, Size: 1191 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: New release candidate for Xen 4.0.0 (RC9)

[Bastian Blank]

On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg

This still lacks the fix for IO-APIC setup.

Bastian

-- 
Emotions are alien to me.  I'm a scientist.
		-- Spock, "This Side of Paradise", stardate 3417.3

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 16:02, "Bastian Blank" <waldi@debian.org> wrote:

> On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
>> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
> 
> This still lacks the fix for IO-APIC setup.

What fix would that be?

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Bastian Blank]

On Wed, Mar 31, 2010 at 04:09:15PM +0100, Keir Fraser wrote:
> On 31/03/2010 16:02, "Bastian Blank" <waldi@debian.org> wrote:
> > On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
> >> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
> > This still lacks the fix for IO-APIC setup.
> What fix would that be?

<20100324111736.GA9327@wavehammer.waldi.eu.org>

Bastian

-- 
Military secrets are the most fleeting of all.
		-- Spock, "The Enterprise Incident", stardate 5027.4

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 16:44, "Bastian Blank" <waldi@debian.org> wrote:

> On Wed, Mar 31, 2010 at 04:09:15PM +0100, Keir Fraser wrote:
>> On 31/03/2010 16:02, "Bastian Blank" <waldi@debian.org> wrote:
>>> On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
>>>> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
>>> This still lacks the fix for IO-APIC setup.
>> What fix would that be?
> 
> <20100324111736.GA9327@wavehammer.waldi.eu.org>

I don't feel any the wiser.

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Konrad Rzeszutek Wilk]

On Wed, Mar 31, 2010 at 05:38:05PM +0100, Keir Fraser wrote:
> On 31/03/2010 16:44, "Bastian Blank" <waldi@debian.org> wrote:
> 
> > On Wed, Mar 31, 2010 at 04:09:15PM +0100, Keir Fraser wrote:
> >> On 31/03/2010 16:02, "Bastian Blank" <waldi@debian.org> wrote:
> >>> On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
> >>>> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
> >>> This still lacks the fix for IO-APIC setup.
> >> What fix would that be?
> > 
> > <20100324111736.GA9327@wavehammer.waldi.eu.org>

 http://mid.gmane.org/20100324111736.GA9327@wavehammer.waldi.eu.org

> 
> I don't feel any the wiser.
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 19:03, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:

> On Wed, Mar 31, 2010 at 05:38:05PM +0100, Keir Fraser wrote:
>> On 31/03/2010 16:44, "Bastian Blank" <waldi@debian.org> wrote:
>> 
>>> On Wed, Mar 31, 2010 at 04:09:15PM +0100, Keir Fraser wrote:
>>>> On 31/03/2010 16:02, "Bastian Blank" <waldi@debian.org> wrote:
>>>>> On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
>>>>>> 4.0.0-rc9 is now tagged in
>>>>>> http://xenbits.xensource.com/xen-4.0-testing.hg
>>>>> This still lacks the fix for IO-APIC setup.
>>>> What fix would that be?
>>> 
>>> <20100324111736.GA9327@wavehammer.waldi.eu.org>
> 
>  http://mid.gmane.org/20100324111736.GA9327@wavehammer.waldi.eu.org

Ah right. Well, looks a bit scary this close to release and not clear to me
that it's a regression. I suggest re-submit to xen-devel for inclusion in
xen-unstable and possible backport for 4.0.1.

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Jeremy Fitzhardinge]

On 03/31/2010 11:18 AM, Keir Fraser wrote:
> Ah right. Well, looks a bit scary this close to release and not clear to me
> that it's a regression. I suggest re-submit to xen-devel for inclusion in
> xen-unstable and possible backport for 4.0.1.
>    

I think it's fairly safe.  From a quick poke about, it appears to only 
affect guests which are using PHYSDEVOP_setup_gsi, which is only new 
2.6.32+-based ones.

     J

Re: New release candidate for Xen 4.0.0 (RC9)

[Konrad Rzeszutek Wilk]

On Wed, Mar 31, 2010 at 10:27:21AM +0100, Keir Fraser wrote:
> Folks,
> 
> 4.0.0-rc9 is now tagged in http://xenbits.xensource.com/xen-4.0-testing.hg
> 
> This is almost certainly the last RC, and will become the 4.0.0 release
> around the middle of next week. Please test!

I don't have a fix, but I did this (this is with xen/stable-2.6.32.x
latest and also rolled back to git commit 47bd9cad11b1a723c6e5d1127aebe570a3e34f34
with a cherry pick of d3b9e35a81e31ff4edb4666f9c36eb5b2032f4da - just in case
the problem was due to the PAT changes):


-sh-3.1# 
-sh-3.1# cat /proc/mtrr
reg00: base=0x000000000 (    0MB), size= 2048MB, count=1: write-back
reg01: base=0x080000000 ( 2048MB), size= 1024MB, count=1: write-back
reg02: base=0x0c0000000 ( 3072MB), size=  256MB, count=1: write-back
-sh-3.1# 
-sh-3.1# echo "disable=2" > /proc/mtrr
(XEN) Assertion '((lock)->lock <= 0)' failed at /home/konrad/git/tip/nex-VI/xen/xen/include/asm/sp:18
(XEN) ----[ Xen-4.1-100331c50   rcx: 00000000000006f0
(XEN) rdx: ffff82c480278a18   rsi: 000000000000000f   rdi: ffff82c480244ee0
(XEN) rbp: ffff82c48037fc98   rsp: ffff82c48037fc98   r8:  ffff82c48037fc48
(XEN) r9:  ffff82c48037fc40   r10: 0000000000000002   r11: 0000ffff0000ffff
(XEN) r12: 0000000000000000   r13: 0000000000000002   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 00000002245d3000   cr2: 00007f5c26627b78
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48037fc98:
(XEN)    ffff82c48037fca8 ffff82c48019cce5 ffff82c48037fce8 ffff82c48019ce57
(XEN)    0000000000000086 ffff82c48037fd08 0000000000000002 0000000000000246
(XEN)    0000000000000000 ffff82c48037fd0c ffff82c48037fd58 ffff82c48019d24b
(XEN)    0000000000218129 ffff82c480170eee ffff82c48037fd18 ffff82c480170fe7
(XEN)    ffff83022fee0000 000000028011f675 0000000000000002 ffff8300cfee8000
(XEN)    ffff83022fee0000 0000000000000002 0000000000000000 ffff880008130020
(XEN)    ffff82c48037fdc8 ffff82c48019d525 0000000000000008 ffff82c48037fdb0
(XEN)    0000000000000000 ffff82c48027e080 0000000000010000 00000000000c0000
(XEN)    06ff8300cfdbe000 fffffffffffffff3 ffff880008128df8 ffff880008128df8
(XEN)    0000000000000000 ffff880008130020 ffff82c48037ff08 ffff82c480153fc3
(XEN)    0000000000006000 0000000000000056 ffff82c48037fe00 ffff82c48037fe30
(XEN)    ffff82c4801746d4 0000000000000001 0000000000000000 ffff82c48037fe40
(XEN)    ffff8300cfee8000 ffff8300cfdbe000 00000015a25fa88c 0000000000000001
(XEN)    ffff82c480278080 ffff82c48037fec0 0300000100000020 0000000200000000
(XEN)    ffff880008139d00 ffff880008139d00 0000000000000000 ffff880008130020
(XEN)    ffff880008128e38 ffffffff810115c6 ffff880008128e88 ffffffff8101186c
(XEN)    0000000000000000 00000015a25fa88c 0000000218c6f130 0000000002753c6a
(XEN)    00000012ba64a0c1 00000000c3b44ffa 0000000000000000 ffff82c48037ff28
(XEN)    ffff82c480279280 ffff8300cfee8000 00000000ffff00ff ffff880006aebd6c
(XEN)    0000000000000000 ffff880008130020 00007d3b7fc800b7 ffff82c4801fb1dc
(XEN) Xen call trace:
(XEN)    [<ffff82c48011fa40>] _spin_unlock+0xc/0x22
(XEN)    [<ffff82c48019cce5>] post_set+0x70/0x72
(XEN)    [<ffff82c48019ce57>] generic_set_mtrr+0xd9/0xf3
(XEN)    [<ffff82c48019d24b>] set_mtrr+0xda/0x121
(XEN)    [<ffff82c48019d525>] mtrr_del_page+0x172/0x199
(XEN)    [<ffff82c480153fc3>] do_platform_op+0x283/0x1af8
(XEN)    [<ffff82c4801fb1dc>] syscall_enter+0x10c/0x166
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion '((lock)->lock <= 0)' failed at /home/konrad/git/tip/nex-VI/xen/xen/include/asm/sp:18
(XEN) ****************************************
(XEN) 

I tested the same exact kernel on baremetal, and had no trouble with the
above mentioned command. The machine is a Dell T105 AMD 1352 w/ 8GB of
RAM.

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 20:10, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:

>> This is almost certainly the last RC, and will become the 4.0.0 release
>> around the middle of next week. Please test!
> 
> I don't have a fix, but I did this (this is with xen/stable-2.6.32.x
> latest and also rolled back to git commit
> 47bd9cad11b1a723c6e5d1127aebe570a3e34f34
> with a cherry pick of d3b9e35a81e31ff4edb4666f9c36eb5b2032f4da - just in case
> the problem was due to the PAT changes):

Is it a regression?

 -- Keir

Re: New release candidate for Xen 4.0.0 (RC9)

[Konrad Rzeszutek Wilk]

On Wed, Mar 31, 2010 at 08:33:13PM +0100, Keir Fraser wrote:
> On 31/03/2010 20:10, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:
> 
> >> This is almost certainly the last RC, and will become the 4.0.0 release
> >> around the middle of next week. Please test!
> > 
> > I don't have a fix, but I did this (this is with xen/stable-2.6.32.x
> > latest and also rolled back to git commit
> > 47bd9cad11b1a723c6e5d1127aebe570a3e34f34
> > with a cherry pick of d3b9e35a81e31ff4edb4666f9c36eb5b2032f4da - just in case
> > the problem was due to the PAT changes):
> 
> Is it a regression?

I haven't tried this type of operation before so I can't comment. I can
spin out a Xen hypervisor - what version back do you want me to compare
to?


Interestingly, on Intel it makes the machine crawl and gets CPU#0 stuck,
while on AMD I get that nice crash.



> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 20:52, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:

>>> I don't have a fix, but I did this (this is with xen/stable-2.6.32.x
>>> latest and also rolled back to git commit
>>> 47bd9cad11b1a723c6e5d1127aebe570a3e34f34
>>> with a cherry pick of d3b9e35a81e31ff4edb4666f9c36eb5b2032f4da - just in
>>> case
>>> the problem was due to the PAT changes):
>> 
>> Is it a regression?
> 
> I haven't tried this type of operation before so I can't comment. I can
> spin out a Xen hypervisor - what version back do you want me to compare
> to?

Last stable release (3.4.2). Or even 3.4-testing tip.

I'm only interested in fixing regressions right now for 4.0.0. If this bug
has existed since at least 3.4 then the fix (when we have one!) can wait a
bit longer, for 4.0.1. My guess is that the MTRR interface is getting little
use and rotted some time ago.

 -- Keir

> 
> Interestingly, on Intel it makes the machine crawl and gets CPU#0 stuck,
> while on AMD I get that nice crash.

Re: New release candidate for Xen 4.0.0 (RC9)

[Konrad Rzeszutek Wilk]

On Wed, Mar 31, 2010 at 09:07:31PM +0100, Keir Fraser wrote:
> On 31/03/2010 20:52, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:
> 
> >>> I don't have a fix, but I did this (this is with xen/stable-2.6.32.x
> >>> latest and also rolled back to git commit
> >>> 47bd9cad11b1a723c6e5d1127aebe570a3e34f34
> >>> with a cherry pick of d3b9e35a81e31ff4edb4666f9c36eb5b2032f4da - just in
> >>> case
> >>> the problem was due to the PAT changes):
> >> 
> >> Is it a regression?
> > 
> > I haven't tried this type of operation before so I can't comment. I can
> > spin out a Xen hypervisor - what version back do you want me to compare
> > to?
> 
> Last stable release (3.4.2). Or even 3.4-testing tip.
> 
> I'm only interested in fixing regressions right now for 4.0.0. If this bug
> has existed since at least 3.4 then the fix (when we have one!) can wait a
> bit longer, for 4.0.1. My guess is that the MTRR interface is getting little

Yeah. I get the same failure with 3.4 from xen-3.4-testing:


echo "disable=2" > /proc/mtrr
(XEN) Assertion '((lock)->lock <= 0)' failed at /home/konrad/git/tip/nex-VI/xen/xen/include/asm/sp:18
(XEN) ----[ Xen-3.4-100331000000000000000f   rdi: ffff828c8020ce20
(XEN) rbp: ffff828c8026fcb8   rsp: ffff828c8026fcb8   r8:  ffff828c8026fc68
(XEN) r9:  ffff828c8026fc60   r10: 0000000000000000   r11: 00000000f0000800
(XEN) r12: 0000000000000000   r13: 0000000000000002   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000006f0
(XEN) cr3: 00000000cad0e000   cr2: 00007fe07ea35385
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff828c8026fcb8:
(XEN)    ffff828c8026fcc8 ffff828c80184265 ffff828c8026fd08 ffff828c801843d7
(XEN)    0000000000000086 ffff828c8026fd28 0000000000000002 0000000000000246
(XEN)    ffff828c8026fd2c 80000000bd4d7165 0000000000000000 0000000000000002
(XEN)    ffff8800bf5efc80 80100000be263065 0000000800000000 0000000000000000
(XEN)    00000000000be263 ffff820000000002 0000000000000006 0000000000000002
(XEN)    0000000000000008 0000000000000002 0000000000000000 0000000000000000
(XEN)    ffff828c8026fde8 ffff828c80184a19 0000000000000000 ffff8300cfada000
(XEN)    0000000000000000 0000000000000001 0000000000010000 00000000000c0000
(XEN)    06ff828c80140877 fffffffffffffff3 ffff880007dcade8 ffff8800062f5d6c
(XEN)    00000000000004fd 0000000000000000 ffff828c8026ff08 ffff828c801428e3
(XEN)    0000000000000000 0000000000000001 0000000000000282 ffff828c8026fe40
(XEN)    ffff8300cfada000 ffff8300cfcce000 0000002b4d860d0b 0000000000000001
(XEN)    ffff828c80244100 ffff828c8026fec0 0300000100000020 0000000200000000
(XEN)    0000002b17ecba31 0000002ac27e0b7b ffff880007ddce40 ffff880007ddce40
(XEN)    0000000000000000 ffff880007dd2020 ffff880007dcae38 ffffffff810117da
(XEN)    ffff880007dcae88 ffffffff81011a80 0000000000000000 0000002b4d860d0b
(XEN)    00000002afcbb5c2 000000000167cfd4 000000287eb6586a ffff828c8026ff28
(XEN)    ffff828c80244d30 ffff8300cfada000 00000000ffff00ff ffff8800062f5d6c
(XEN)    00000000000004fd 0000000000000000 00007d737fd900b7 ffff828c801d61dc
(XEN)    ffffffff810090ea 0000000000000007 0000000000000000 00000000000004fd
(XEN) Xen call trace:
(XEN)    [<ffff828c8011bd14>] _spin_unlock+0xc/0x22
(XEN)    [<ffff828c80184265>] post_set+0x70/0x72
(XEN)    [<ffff828c801843d7>] generic_set_mtrr+0xd9/0xf3
(XEN)    [<0000000000000002>] ???
(XEN)    
(XEN) 
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion '((lock)->lock <= 0)' failed at /home/konrad/git/tip/nex-VI/xen/xen/include/asm/sp:18
(XEN) ****************************************
(XEN) 

Re: New release candidate for Xen 4.0.0 (RC9)

[Keir Fraser]

On 31/03/2010 22:07, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:

>> I'm only interested in fixing regressions right now for 4.0.0. If this bug
>> has existed since at least 3.4 then the fix (when we have one!) can wait a
>> bit longer, for 4.0.1. My guess is that the MTRR interface is getting little
> 
> Yeah. I get the same failure with 3.4 from xen-3.4-testing:

I didn't reproduce this with my setup, but the crash below is because
spin_unlock() is finding that the lock is not locked on entry. This is the
set_atomicity_lock as acquired/released by prepare_set/post_set. Looking at
the code they seem to be used in matched pairs and I can't see any memory
corruption issues or anything. If you want to get this fixed then you'll
have to add some tracing to find out what's going on (e.g, dump the lock
state at the end of prepare_set and start of post_set). If you can reproduce
this deterministically with your setup then should be pretty easy to nail
this bug.

 -- Keir

> (XEN)    [<ffff828c8011bd14>] _spin_unlock+0xc/0x22
> (XEN)    [<ffff828c80184265>] post_set+0x70/0x72
> (XEN)    [<ffff828c801843d7>] generic_set_mtrr+0xd9/0xf3
> (XEN)    [<0000000000000002>] ???
> (XEN)    
> (XEN) 
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Assertion '((lock)->lock <= 0)' failed at
> /home/konrad/git/tip/nex-VI/xen/xen/include/asm/sp:18
> (XEN) ****************************************
> (XEN) 

Re: New release candidate for Xen 4.0.0 (RC9)

[Konrad Rzeszutek Wilk]

On Wed, May 19, 2010 at 01:10:36PM +0100, Keir Fraser wrote:
> On 31/03/2010 22:07, "Konrad Rzeszutek Wilk" <konrad.wilk@oracle.com> wrote:
> 
> >> I'm only interested in fixing regressions right now for 4.0.0. If this bug
> >> has existed since at least 3.4 then the fix (when we have one!) can wait a
> >> bit longer, for 4.0.1. My guess is that the MTRR interface is getting little
> > 
> > Yeah. I get the same failure with 3.4 from xen-3.4-testing:
> 
> I didn't reproduce this with my setup, but the crash below is because
> spin_unlock() is finding that the lock is not locked on entry. This is the
> set_atomicity_lock as acquired/released by prepare_set/post_set. Looking at
> the code they seem to be used in matched pairs and I can't see any memory
> corruption issues or anything. If you want to get this fixed then you'll
> have to add some tracing to find out what's going on (e.g, dump the lock
> state at the end of prepare_set and start of post_set). If you can reproduce
> this deterministically with your setup then should be pretty easy to nail
> this bug.

OK. Thanks.