2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Xen: 3.2.1-rc5 64bit
Dom0: 2.6.18.8 (at cs 524) 32-pae
DomU: 2.6.26-rc8 32-pae

root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
Using config file "/linodes/xencaker/xen.conf".
Started domain xencaker
                        root@newark13:~#

root@newark13:~# xm dmesg
...
(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 
0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e019:[<00000000c0103b55>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
(XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 00000000deadbeef
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000c06c1000
(XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000002d351b000   cr2: 0000000000000000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c0667f08:
(XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
c065c580
(XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
00000000
(XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
00000000
(XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
1f898975
(XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
00000007
(XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
00000000
(XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
00040800
(XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Xen: 3.2.1-rc5 64bit
> Dom0: 2.6.18.8 (at cs 524) 32-pae
> DomU: 2.6.26-rc8 32-pae
>
> root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
> Using config file "/linodes/xencaker/xen.conf".
> Started domain xencaker

Is this consistent or sporadic?  Does it depend on how much guest memory 
you give it?  What was the last kernel version which worked?

>                        root@newark13:~#
>
> root@newark13:~# xm dmesg
> ...

Were there any other errors/warnings printed here?

> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
> VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

> (XEN) RFLAGS: 0000000000000282   CONTEXT: guest
> (XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 
> 00000000deadbeef
> (XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 
> 00000000c06c1000
> (XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  
> 0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
> 0000000000000000
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
> 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 
> 00000000000026b0
> (XEN) cr3: 00000002d351b000   cr2: 0000000000000000
> (XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
> (XEN) Guest stack trace from esp=c0667f08:
> (XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
> c065c580
> (XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
> 00000000
> (XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
> 00000000
> (XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
> 1f898975
> (XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
> 00000007
> (XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
> 00000000
> (XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
> 00040800
> (XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000
>
> -Chris

Thanks,
    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
> Is this consistent or sporadic?  

Consistently fails to boot, across a few test boxes, slightly different 
Xen versions -- all around Xen 3.2.1, same config.

> Does it depend on how much guest memory you give it?

No.  Tried with 128, 360, 5000.  They all fail in the same way.

> What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

> Were there any other errors/warnings printed here?

Nope.  Just some whitespace.

>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>> VCPU 0 [ec=0000]
>> (XEN) domain_crash_sync called from entry.S
>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e019:[<00000000c0103b55>]
> 
> Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the 
files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks,
-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>> Is this consistent or sporadic?  
>
> Consistently fails to boot, across a few test boxes, slightly 
> different Xen versions -- all around Xen 3.2.1, same config.
>
>> Does it depend on how much guest memory you give it?
>
> No.  Tried with 128, 360, 5000.  They all fail in the same way.
>
>> What was the last kernel version which worked?
>
> I built a 2.6.25.10 pv_ops this morning and it boots fine.
>
>> Were there any other errors/warnings printed here?
>
> Nope.  Just some whitespace.

Are you sure?  Could you send a full xm dmesg log output?  The crash is 
from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
with debug enabled, it should have printed something about why it was 
failing the hypercall.

>>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>>> VCPU 0 [ec=0000]
>>> (XEN) domain_crash_sync called from entry.S
>>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>>> (XEN) CPU:    0
>>> (XEN) RIP:    e019:[<00000000c0103b55>]
>>
>> Could you decode this to see which function this corresponds to?
>
> I'd love to know how for future reference, but for now I've placed the 
> files here:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks, that's very interesting.  I had a very similar crash in a 
current -rc kernel when running it under the Citrix XenServer product 
version of Xen, and I wasn't sure where to point fingers.

For reference:

$ gdb 2.6.26-rc8-linode11 
GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55						<- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:	ud2a   
(gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:	push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:	mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:	push   %edi
...

Thanks,
	J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>>> Were there any other errors/warnings printed here?
>>
>> Nope.  Just some whitespace.
> 
> Are you sure?  Could you send a full xm dmesg log output?  The crash is 
> from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
> with debug enabled, it should have printed something about why it was 
> failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create 
run.  Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

> For reference:
> 
> $ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu"...
> (no debugging symbols found)
> (gdb) x/i 0x00000000c0103b55                        <- note: need to add 
> 0x to start
> 0xc0103b55 <make_lowmem_page_readonly+85>:    ud2a   (gdb) disassemble 
> make_lowmem_page_readonly
> Dump of assembler code for function make_lowmem_page_readonly:
> 0xc0103b00 <make_lowmem_page_readonly+0>:    push   %ebp
> 0xc0103b01 <make_lowmem_page_readonly+1>:    mov    %eax,%ebp
> 0xc0103b03 <make_lowmem_page_readonly+3>:    push   %edi

Cool, thanks :)

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>>>> Were there any other errors/warnings printed here?
>>>
>>> Nope.  Just some whitespace.
>>
>> Are you sure?  Could you send a full xm dmesg log output?  The crash 
>> is from a BUG() caused by a failing hypercall.  Since you have 
>> compiled Xen with debug enabled, it should have printed something 
>> about why it was failing the hypercall.
>
> Unfortunately, I included the entire dmesg output from that xm create 
> run.  Here's more of the transcript in case it's useful:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/log.txt
>
> Although, looking at it again, I did miss this line:
>
> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
> 000d2e50

Good, that's what I was hoping to see.  (Well, actually not that 
*particular* message, which is strange, but some explanatory message at 
least.)

    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>> Although, looking at it again, I did miss this line:
>>
>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>> 000d2e50
> 
> Good, that's what I was hoping to see.  (Well, actually not that 
> *particular* message, which is strange, but some explanatory message at 
> least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
further if that'd be useful to you.

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>   
>>> Although, looking at it again, I did miss this line:
>>>
>>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>>> 000d2e50
>>>       
>> Good, that's what I was hoping to see.  (Well, actually not that 
>> *particular* message, which is strange, but some explanatory message at 
>> least.)
>>     
>
> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
> further if that'd be useful to you.
>   

Yes, please.

    J