E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

[Mark van Dijk]

Hi Xen developers and enthousiasts,

Recently I have been having a lot of problems with HVM hosts on a dual
Xeon E5606 box. Problem is that HVM hosts don't work on it at all and
crash the server.

Today I got the logging to work over an IPMI console so then I compiled
a fresh xen-unstable. Apart from the message I posted in the subject
the call trace reads as follows:

(XEN) Xen call trace:
(XEN)    [<ffff82c4801e1be9>] ept_get_entry+0x13a/0x28f
(XEN)    [<ffff82c4801da274>] __get_gfn_type_access+0x175/0x256
(XEN)    [<ffff82c4801b5a55>] hvm_hap_nested_page_fault+0x133/0x422
(XEN)    [<ffff82c4801d3c8e>] vmx_vmexit_handler+0x136b/0x1614

The entire log from boot to crash can be viewed at the following link:
http://pastebin.com/5wcH7GWR

Here's the kernel's config
http://pastebin.com/E51S61Qk

And although I'm not sure if knowing BIOS settings is useful it doesn't
hurt to share the many settings it has. So I captured them and posted
the images:
http://imgur.com/a/wGBVS/all#0

I hope you can help me with getting this fixed. It's been a nightmare
despite the lack of sleep :) but then again I also learned much along
the way. I triple checked the consistency of the above posts to make
sure they are all from the same build. The problem has been occuring
for a fairly long time now but I had to make sure there was no issue on
my side. I'm still not sure but I'm also fresh out of ideas, so thanks
up front for your help or suggestions.

As always please CC me when you reply to the list, I am not receiving
xen-devel emails because most are way outside my comfort zone...

-- 
Thank you,
Mark van Dijk.               ,--------------------------------
----------------------------'        Wed Aug 15 20:38 UTC 2012
Today is Boomtime, the 8th day of Bureaucracy in the YOLD 3178

Re: E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

[Jan Beulich]

[-- Attachment #1: Type: text/plain, Size: 1166 bytes --]

>>> On 15.08.12 at 22:39, Mark van Dijk <lists+xen@internecto.net> wrote:
> (XEN) Xen call trace:
> (XEN)    [<ffff82c4801e1be9>] ept_get_entry+0x13a/0x28f
> (XEN)    [<ffff82c4801da274>] __get_gfn_type_access+0x175/0x256
> (XEN)    [<ffff82c4801b5a55>] hvm_hap_nested_page_fault+0x133/0x422
> (XEN)    [<ffff82c4801d3c8e>] vmx_vmexit_handler+0x136b/0x1614
> 
> The entire log from boot to crash can be viewed at the following link:
> http://pastebin.com/5wcH7GWR 

Unfortunately quite a few of the registers contain values that
could sensibly be "i". Could you either disassemble the
instructions around the place yourself to find out which one it
is, or make the xen-syms file corresponding to this run available?

I'm suspecting "i" to actually be 2, and the code not having got
updated when 1Gb page support got added to PoD. You could
hence alternatively also try the attached debugging patch,
which - if my guess is right - may at once fix your problem.

However, because of this outright claiming that HVM doesn't
work seems a little harsh - did you try running your guests
without the use of PoD (i.e. with memory == maxmem)?

Jan


[-- Attachment #2: ept-pod-1Gb-assert.patch --]
[-- Type: text/plain, Size: 699 bytes --]

--- a/xen/arch/x86/mm/p2m-ept.c
+++ b/xen/arch/x86/mm/p2m-ept.c
@@ -521,13 +521,13 @@ static mfn_t ept_get_entry(struct p2m_do
             }
 
             /* Populate this superpage */
-            ASSERT(i == 1);
+if(i >= 2) printk("PoD[%lx] level=%d\n", gfn, i);//temp
+            ASSERT(i <= 2);
 
             index = gfn_remainder >> ( i * EPT_TABLE_ORDER);
             ept_entry = table + index;
 
-            if ( !p2m_pod_demand_populate(p2m, gfn, 
-                                            PAGE_ORDER_2M, q) )
+            if ( !p2m_pod_demand_populate(p2m, gfn, i * EPT_TABLE_ORDER, q) )
                 goto retry;
             else
                 goto out;

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

[Mark van Dijk]

On Thu, 16 Aug 2012 09:11:50 +0100
"Jan Beulich" <JBeulich@suse.com> wrote:

> >>> On 15.08.12 at 22:39, Mark van Dijk <lists+xen@internecto.net>
> >>> wrote:
> > (XEN) Xen call trace:
> > (XEN)    [<ffff82c4801e1be9>] ept_get_entry+0x13a/0x28f
> > (XEN)    [<ffff82c4801da274>] __get_gfn_type_access+0x175/0x256
> > (XEN)    [<ffff82c4801b5a55>] hvm_hap_nested_page_fault+0x133/0x422
> > (XEN)    [<ffff82c4801d3c8e>] vmx_vmexit_handler+0x136b/0x1614
> > 
> > The entire log from boot to crash can be viewed at the following
> > link: http://pastebin.com/5wcH7GWR 
> 
> Unfortunately quite a few of the registers contain values that
> could sensibly be "i". Could you either disassemble the
> instructions around the place yourself to find out which one it
> is, or make the xen-syms file corresponding to this run available?

Hi Jan, I'm not capable to disassemble anything because my experience
with that is zero. So it's probably easier to give you the xen-syms
file, I posted it to a pastebin with a little detour:

curl http://sprunge.us/cGeL | openssl enc -a -d | \
 xz -d > xen-syms-4.2.0-rc3-pre

md5sum aa27f5aeea45f72cab88848e5996080e

> 
> I'm suspecting "i" to actually be 2, and the code not having got
> updated when 1Gb page support got added to PoD. You could
> hence alternatively also try the attached debugging patch,
> which - if my guess is right - may at once fix your problem.

Sure, I'll give that a go and let you know, thanks.

> However, because of this outright claiming that HVM doesn't
> work seems a little harsh - did you try running your guests
> without the use of PoD (i.e. with memory == maxmem)?

If it sounded like I meant to complain that HVM is broken then I
apologise, that's not what I meant. I meant it didn't work on my system
and I had no idea that this relates to maxmem. :)

I'll admit that I didn't try the hvm without the maxmem setting but the
VM has settings 'memory = 1024; maxmem = 4096'.

Mark

Re: E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

[Jan Beulich]

>>> On 16.08.12 at 10:48, "Mark van Dijk" <lists+xen@internecto.net> wrote:
> On Thu, 16 Aug 2012 09:11:50 +0100
> "Jan Beulich" <JBeulich@suse.com> wrote:
> 
>> >>> On 15.08.12 at 22:39, Mark van Dijk <lists+xen@internecto.net>
>> >>> wrote:
>> > (XEN) Xen call trace:
>> > (XEN)    [<ffff82c4801e1be9>] ept_get_entry+0x13a/0x28f
>> > (XEN)    [<ffff82c4801da274>] __get_gfn_type_access+0x175/0x256
>> > (XEN)    [<ffff82c4801b5a55>] hvm_hap_nested_page_fault+0x133/0x422
>> > (XEN)    [<ffff82c4801d3c8e>] vmx_vmexit_handler+0x136b/0x1614
>> > 
>> > The entire log from boot to crash can be viewed at the following
>> > link: http://pastebin.com/5wcH7GWR 
>> 
>> Unfortunately quite a few of the registers contain values that
>> could sensibly be "i". Could you either disassemble the
>> instructions around the place yourself to find out which one it
>> is, or make the xen-syms file corresponding to this run available?
> 
> Hi Jan, I'm not capable to disassemble anything because my experience
> with that is zero. So it's probably easier to give you the xen-syms
> file, I posted it to a pastebin with a little detour:
> 
> curl http://sprunge.us/cGeL | openssl enc -a -d | \
>  xz -d > xen-syms-4.2.0-rc3-pre

Thanks. That confirms that "i" (in r12d) is indeed 2. So there are
good chances that the patch I sent earlier will help. Please let us
know.

Jan

Re: E5606 with no HVM; Assertion 'i == 1' failed at p2m-ept.c:524

[Mark van Dijk]

On Thu, 16 Aug 2012 12:18:33 +0100
"Jan Beulich" <JBeulich@suse.com> wrote:

> >>> On 16.08.12 at 10:48, "Mark van Dijk" <lists+xen@internecto.net>
> >>> wrote:
> > On Thu, 16 Aug 2012 09:11:50 +0100
> > "Jan Beulich" <JBeulich@suse.com> wrote:
> > 
> >> >>> On 15.08.12 at 22:39, Mark van Dijk <lists+xen@internecto.net>
> >> >>> wrote:
> >> > (XEN) Xen call trace:
> >> > (XEN)    [<ffff82c4801e1be9>] ept_get_entry+0x13a/0x28f
> >> > (XEN)    [<ffff82c4801da274>] __get_gfn_type_access+0x175/0x256
> >> > (XEN)    [<ffff82c4801b5a55>]
> >> > hvm_hap_nested_page_fault+0x133/0x422 (XEN)
> >> > [<ffff82c4801d3c8e>] vmx_vmexit_handler+0x136b/0x1614
> >> > 
> >> > The entire log from boot to crash can be viewed at the following
> >> > link: http://pastebin.com/5wcH7GWR 
> >> 
> >> Unfortunately quite a few of the registers contain values that
> >> could sensibly be "i". Could you either disassemble the
> >> instructions around the place yourself to find out which one it
> >> is, or make the xen-syms file corresponding to this run available?
> > 
> > Hi Jan, I'm not capable to disassemble anything because my
> > experience with that is zero. So it's probably easier to give you
> > the xen-syms file, I posted it to a pastebin with a little detour:
> > 
> > curl http://sprunge.us/cGeL | openssl enc -a -d | \
> >  xz -d > xen-syms-4.2.0-rc3-pre
> 
> Thanks. That confirms that "i" (in r12d) is indeed 2. So there are
> good chances that the patch I sent earlier will help. Please let us
> know.

Good news: the VM boots. Thanks Jan!

My console reads:

...
(XEN) irq.c:270: Dom1 PCI link 3 changed 5 -> 0
(XEN) PoD[bfc00] level=2
(XEN) PoD[7fc00] level=2

So that's one more confirmation. :)

Mark

-- 
Stay in touch,
Mark van Dijk.               ,---------------------------------
----------------------------'         Thu Aug 16 11:57 UTC 2012
Today is Pungenday, the 9th day of Bureaucracy in the YOLD 3178