2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Xen: 3.2.1-rc5 64bit
Dom0: 2.6.18.8 (at cs 524) 32-pae
DomU: 2.6.26-rc8 32-pae

root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
Using config file "/linodes/xencaker/xen.conf".
Started domain xencaker
                        root@newark13:~#

root@newark13:~# xm dmesg
...
(XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on VCPU 
0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 332 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e019:[<00000000c0103b55>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
(XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 00000000deadbeef
(XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 00000000c06c1000
(XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000002d351b000   cr2: 0000000000000000
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c0667f08:
(XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
c065c580
(XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
00000000
(XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
00000000
(XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
1f898975
(XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
00000007
(XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
00000000
(XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
00040800
(XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Xen: 3.2.1-rc5 64bit
> Dom0: 2.6.18.8 (at cs 524) 32-pae
> DomU: 2.6.26-rc8 32-pae
>
> root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
> Using config file "/linodes/xencaker/xen.conf".
> Started domain xencaker

Is this consistent or sporadic?  Does it depend on how much guest memory 
you give it?  What was the last kernel version which worked?

>                        root@newark13:~#
>
> root@newark13:~# xm dmesg
> ...

Were there any other errors/warnings printed here?

> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
> VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

> (XEN) RFLAGS: 0000000000000282   CONTEXT: guest
> (XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 
> 00000000deadbeef
> (XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 
> 00000000c06c1000
> (XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  
> 0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
> 0000000000000000
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
> 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 
> 00000000000026b0
> (XEN) cr3: 00000002d351b000   cr2: 0000000000000000
> (XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
> (XEN) Guest stack trace from esp=c0667f08:
> (XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
> c065c580
> (XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
> 00000000
> (XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
> 00000000
> (XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
> 1f898975
> (XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
> 00000007
> (XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
> 00000000
> (XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
> 00040800
> (XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000
>
> -Chris

Thanks,
    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Xen: 3.2.1-rc5 64bit
> Dom0: 2.6.18.8 (at cs 524) 32-pae
> DomU: 2.6.26-rc8 32-pae
>
> root@newark13:~# xm create -f /linodes/xencaker/xen.conf -c
> Using config file "/linodes/xencaker/xen.conf".
> Started domain xencaker

Is this consistent or sporadic?  Does it depend on how much guest memory 
you give it?  What was the last kernel version which worked?

>                        root@newark13:~#
>
> root@newark13:~# xm dmesg
> ...

Were there any other errors/warnings printed here?

> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
> VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e019:[<00000000c0103b55>]

Could you decode this to see which function this corresponds to?

> (XEN) RFLAGS: 0000000000000282   CONTEXT: guest
> (XEN) rax: 00000000ffffffea   rbx: 00000000deadbeef   rcx: 
> 00000000deadbeef
> (XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 
> 00000000c06c1000
> (XEN) rbp: 00000000c06c0000   rsp: 00000000c0667f08   r8:  
> 0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
> 0000000000000000
> (XEN) r12: 0000000000000000   r13: 0000000000000000   r14: 
> 0000000000000000
> (XEN) r15: 0000000000000000   cr0: 000000008005003b   cr4: 
> 00000000000026b0
> (XEN) cr3: 00000002d351b000   cr2: 0000000000000000
> (XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
> (XEN) Guest stack trace from esp=c0667f08:
> (XEN)   c0103b55 0001e019 00010082 00000001 006c0000 c0756d50 c06c1000 
> c065c580
> (XEN)   c067e4dc c0667f44 c0678ac9 c0667f38 00100000 c0756d50 00000000 
> 00000000
> (XEN)   00040800 00010676 c0667fb8 c06a3b00 c065c680 c065c580 c0676452 
> 00000000
> (XEN)   00000000 00000000 c065c580 00000000 c0667fb0 c0667fa4 c0667fac 
> 1f898975
> (XEN)   40000000 c0667fdc f5800000 00000007 40000000 c0667fdc f5800000 
> 00000007
> (XEN)   c066e771 c0537060 f5800000 00000007 c012500f 40000000 c06753a2 
> 00000000
> (XEN)   c069c5c0 c069c5dc c069c698 c069c6b0 00000000 1f898975 00080201 
> 00040800
> (XEN)   00010676 f5800000 00000000 c0752000 00000000 00000000
>
> -Chris

Thanks,
    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
> Is this consistent or sporadic?  

Consistently fails to boot, across a few test boxes, slightly different 
Xen versions -- all around Xen 3.2.1, same config.

> Does it depend on how much guest memory you give it?

No.  Tried with 128, 360, 5000.  They all fail in the same way.

> What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

> Were there any other errors/warnings printed here?

Nope.  Just some whitespace.

>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>> VCPU 0 [ec=0000]
>> (XEN) domain_crash_sync called from entry.S
>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e019:[<00000000c0103b55>]
> 
> Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the 
files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks,
-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
> Is this consistent or sporadic?  

Consistently fails to boot, across a few test boxes, slightly different 
Xen versions -- all around Xen 3.2.1, same config.

> Does it depend on how much guest memory you give it?

No.  Tried with 128, 360, 5000.  They all fail in the same way.

> What was the last kernel version which worked?

I built a 2.6.25.10 pv_ops this morning and it boots fine.

> Were there any other errors/warnings printed here?

Nope.  Just some whitespace.

>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>> VCPU 0 [ec=0000]
>> (XEN) domain_crash_sync called from entry.S
>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>> (XEN) CPU:    0
>> (XEN) RIP:    e019:[<00000000c0103b55>]
> 
> Could you decode this to see which function this corresponds to?

I'd love to know how for future reference, but for now I've placed the 
files here:

http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks,
-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>> Is this consistent or sporadic?  
>
> Consistently fails to boot, across a few test boxes, slightly 
> different Xen versions -- all around Xen 3.2.1, same config.
>
>> Does it depend on how much guest memory you give it?
>
> No.  Tried with 128, 360, 5000.  They all fail in the same way.
>
>> What was the last kernel version which worked?
>
> I built a 2.6.25.10 pv_ops this morning and it boots fine.
>
>> Were there any other errors/warnings printed here?
>
> Nope.  Just some whitespace.

Are you sure?  Could you send a full xm dmesg log output?  The crash is 
from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
with debug enabled, it should have printed something about why it was 
failing the hypercall.

>>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>>> VCPU 0 [ec=0000]
>>> (XEN) domain_crash_sync called from entry.S
>>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>>> (XEN) CPU:    0
>>> (XEN) RIP:    e019:[<00000000c0103b55>]
>>
>> Could you decode this to see which function this corresponds to?
>
> I'd love to know how for future reference, but for now I've placed the 
> files here:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks, that's very interesting.  I had a very similar crash in a 
current -rc kernel when running it under the Citrix XenServer product 
version of Xen, and I wasn't sure where to point fingers.

For reference:

$ gdb 2.6.26-rc8-linode11 
GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55						<- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:	ud2a   
(gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:	push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:	mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:	push   %edi
...

Thanks,
	J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>> Is this consistent or sporadic?  
>
> Consistently fails to boot, across a few test boxes, slightly 
> different Xen versions -- all around Xen 3.2.1, same config.
>
>> Does it depend on how much guest memory you give it?
>
> No.  Tried with 128, 360, 5000.  They all fail in the same way.
>
>> What was the last kernel version which worked?
>
> I built a 2.6.25.10 pv_ops this morning and it boots fine.
>
>> Were there any other errors/warnings printed here?
>
> Nope.  Just some whitespace.

Are you sure?  Could you send a full xm dmesg log output?  The crash is 
from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
with debug enabled, it should have printed something about why it was 
failing the hypercall.

>>> (XEN) traps.c:413:d332 Unhandled invalid opcode fault/trap [#6] on 
>>> VCPU 0 [ec=0000]
>>> (XEN) domain_crash_sync called from entry.S
>>> (XEN) Domain 332 (vcpu#0) crashed on cpu#0:
>>> (XEN) ----[ Xen-3.2.1-rc5  x86_64  debug=y  Not tainted ]----
>>> (XEN) CPU:    0
>>> (XEN) RIP:    e019:[<00000000c0103b55>]
>>
>> Could you decode this to see which function this corresponds to?
>
> I'd love to know how for future reference, but for now I've placed the 
> files here:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/

Thanks, that's very interesting.  I had a very similar crash in a 
current -rc kernel when running it under the Citrix XenServer product 
version of Xen, and I wasn't sure where to point fingers.

For reference:

$ gdb 2.6.26-rc8-linode11 
GNU gdb Fedora (6.8-10.fc9)
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
(no debugging symbols found)
(gdb) x/i 0x00000000c0103b55						<- note: need to add 0x to start
0xc0103b55 <make_lowmem_page_readonly+85>:	ud2a   
(gdb) disassemble make_lowmem_page_readonly
Dump of assembler code for function make_lowmem_page_readonly:
0xc0103b00 <make_lowmem_page_readonly+0>:	push   %ebp
0xc0103b01 <make_lowmem_page_readonly+1>:	mov    %eax,%ebp
0xc0103b03 <make_lowmem_page_readonly+3>:	push   %edi
...

Thanks,
	J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>>> Were there any other errors/warnings printed here?
>>
>> Nope.  Just some whitespace.
> 
> Are you sure?  Could you send a full xm dmesg log output?  The crash is 
> from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
> with debug enabled, it should have printed something about why it was 
> failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create 
run.  Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

> For reference:
> 
> $ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu"...
> (no debugging symbols found)
> (gdb) x/i 0x00000000c0103b55                        <- note: need to add 
> 0x to start
> 0xc0103b55 <make_lowmem_page_readonly+85>:    ud2a   (gdb) disassemble 
> make_lowmem_page_readonly
> Dump of assembler code for function make_lowmem_page_readonly:
> 0xc0103b00 <make_lowmem_page_readonly+0>:    push   %ebp
> 0xc0103b01 <make_lowmem_page_readonly+1>:    mov    %eax,%ebp
> 0xc0103b03 <make_lowmem_page_readonly+3>:    push   %edi

Cool, thanks :)

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>>> Were there any other errors/warnings printed here?
>>
>> Nope.  Just some whitespace.
> 
> Are you sure?  Could you send a full xm dmesg log output?  The crash is 
> from a BUG() caused by a failing hypercall.  Since you have compiled Xen 
> with debug enabled, it should have printed something about why it was 
> failing the hypercall.

Unfortunately, I included the entire dmesg output from that xm create 
run.  Here's more of the transcript in case it's useful:

http://www.theshore.net/~caker/xen/pvops-bug1/log.txt

Although, looking at it again, I did miss this line:

(XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 000d2e50

> For reference:
> 
> $ gdb 2.6.26-rc8-linode11 GNU gdb Fedora (6.8-10.fc9)
> Copyright (C) 2008 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later 
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "i386-redhat-linux-gnu"...
> (no debugging symbols found)
> (gdb) x/i 0x00000000c0103b55                        <- note: need to add 
> 0x to start
> 0xc0103b55 <make_lowmem_page_readonly+85>:    ud2a   (gdb) disassemble 
> make_lowmem_page_readonly
> Dump of assembler code for function make_lowmem_page_readonly:
> 0xc0103b00 <make_lowmem_page_readonly+0>:    push   %ebp
> 0xc0103b01 <make_lowmem_page_readonly+1>:    mov    %eax,%ebp
> 0xc0103b03 <make_lowmem_page_readonly+3>:    push   %edi

Cool, thanks :)

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>>>> Were there any other errors/warnings printed here?
>>>
>>> Nope.  Just some whitespace.
>>
>> Are you sure?  Could you send a full xm dmesg log output?  The crash 
>> is from a BUG() caused by a failing hypercall.  Since you have 
>> compiled Xen with debug enabled, it should have printed something 
>> about why it was failing the hypercall.
>
> Unfortunately, I included the entire dmesg output from that xm create 
> run.  Here's more of the transcript in case it's useful:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/log.txt
>
> Although, looking at it again, I did miss this line:
>
> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
> 000d2e50

Good, that's what I was hoping to see.  (Well, actually not that 
*particular* message, which is strange, but some explanatory message at 
least.)

    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>>>> Were there any other errors/warnings printed here?
>>>
>>> Nope.  Just some whitespace.
>>
>> Are you sure?  Could you send a full xm dmesg log output?  The crash 
>> is from a BUG() caused by a failing hypercall.  Since you have 
>> compiled Xen with debug enabled, it should have printed something 
>> about why it was failing the hypercall.
>
> Unfortunately, I included the entire dmesg output from that xm create 
> run.  Here's more of the transcript in case it's useful:
>
> http://www.theshore.net/~caker/xen/pvops-bug1/log.txt
>
> Although, looking at it again, I did miss this line:
>
> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
> 000d2e50

Good, that's what I was hoping to see.  (Well, actually not that 
*particular* message, which is strange, but some explanatory message at 
least.)

    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>> Although, looking at it again, I did miss this line:
>>
>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>> 000d2e50
> 
> Good, that's what I was hoping to see.  (Well, actually not that 
> *particular* message, which is strange, but some explanatory message at 
> least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
further if that'd be useful to you.

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Christopher S. Aker]

Jeremy Fitzhardinge wrote:
>> Although, looking at it again, I did miss this line:
>>
>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>> 000d2e50
> 
> Good, that's what I was hoping to see.  (Well, actually not that 
> *particular* message, which is strange, but some explanatory message at 
> least.)

FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
further if that'd be useful to you.

-Chris

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>   
>>> Although, looking at it again, I did miss this line:
>>>
>>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>>> 000d2e50
>>>       
>> Good, that's what I was hoping to see.  (Well, actually not that 
>> *particular* message, which is strange, but some explanatory message at 
>> least.)
>>     
>
> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
> further if that'd be useful to you.
>   

Yes, please.

    J

Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Christopher S. Aker wrote:
> Jeremy Fitzhardinge wrote:
>   
>>> Although, looking at it again, I did miss this line:
>>>
>>> (XEN) mm.c:645:d341 Non-privileged (341) attempt to map I/O space 
>>> 000d2e50
>>>       
>> Good, that's what I was hoping to see.  (Well, actually not that 
>> *particular* message, which is strange, but some explanatory message at 
>> least.)
>>     
>
> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it down 
> further if that'd be useful to you.
>   

Yes, please.

    J

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Guillaume Rousse]

Jeremy Fitzhardinge a écrit :
>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it 
>> down further if that'd be useful to you.
>>   
> 
> Yes, please.
Was there any final conclusion there ? Because I'm currently trying a 
2.6.27.5 with the same kind of issue:
(XEN) mm.c:1482:d31 Bad L1 flags 800000
(XEN) traps.c:437:d31 Unhandled invalid opcode fault/trap [#6] on VCPU 0 
[ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 31 (vcpu#0) crashed on cpu#2:
(XEN) ----[ Xen-3.3.0  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    2
(XEN) RIP:    e033:[<ffffffff8020b3ed>]
(XEN) RFLAGS: 0000000000000282   EM: 1   CONTEXT: pv guest
(XEN) rax: 00000000ffffffea   rbx: 000000000000077f   rcx: 0000000000000054
(XEN) rdx: 0000000000000000   rsi: 8000000331df5061   rdi: ffffffff8077f000
(XEN) rbp: ffffffff806e1f88   rsp: ffffffff806e1f28   r8:  0000000000000010
(XEN) r9:  0000000000000010   r10: ffffffff806e1e88   r11: ffffffff80827f60
(XEN) r12: ffffffff8077f000   r13: 8000000000000161   r14: 8000000331df5061
(XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000033c35c000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff806e1f28:
(XEN)    0000000000000054 ffffffff80827f60 ffffffff8020b3ed 000000010000e030
(XEN)    0000000000010082 ffffffff806e1f68 000000000000e02b ffffffff8020b3e9
(XEN)    0000000000000200 000000000000000c 000000000000000c 0000000000001800
(XEN)    ffffffff806e1ff8 ffffffff806f99cf 0000000000000000 0000000000001000
(XEN)    0000000000001800 ffffffff80e48000 0000000000020000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff80d45000
(XEN)    ffffffff80d46000 ffffffff80d47000 ffffffff80d48000 ffffffff80d49000
(XEN)    ffffffff80d4a000 ffffffff80d4b000 ffffffff80d4c000 ffffffff80d4d000
(XEN)    ffffffff80d4e000 ffffffff80d4f000 ffffffff80d50000 ffffffff80d51000
(XEN)    ffffffff80d52000 ffffffff80d53000 ffffffff80d54000 ffffffff80d55000
(XEN)    ffffffff80d56000 ffffffff80d57000 ffffffff80d58000 ffffffff80d59000
(XEN)    ffffffff80d5a000 ffffffff80d5b000 ffffffff80d5c000 ffffffff80d5d000
(XEN)    ffffffff80d5e000 ffffffff80d5f000 ffffffff80d60000 ffffffff80d61000
(XEN)    ffffffff80d62000 ffffffff80d63000 ffffffff80d64000 ffffffff80d65000
(XEN)    ffffffff80d66000 ffffffff80d67000 ffffffff80d68000 ffffffff80d69000
(XEN)    ffffffff80d6a000 ffffffff80d6b000 ffffffff80d6c000 ffffffff80d6d000
(XEN)    ffffffff80d6e000 ffffffff80d6f000 ffffffff80d70000 ffffffff80d71000
(XEN)    ffffffff80d72000 ffffffff80d73000 ffffffff80d74000 ffffffff80d75000
(XEN)    ffffffff80d76000 ffffffff80d77000 ffffffff80d78000 ffffffff80d79000
-- 
Guillaume Rousse
Service des Moyens Informatiques
INRIA Saclay - Ile de France
Tel: 01 69 35 69 62

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Guillaume Rousse wrote:
> Jeremy Fitzhardinge a écrit :
>>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it 
>>> down further if that'd be useful to you.
>>>   
>>
>> Yes, please.
> Was there any final conclusion there ? Because I'm currently trying a 
> 2.6.27.5 with the same kind of issue:

I don't know of any outstanding bugs which cause this kind of symptom.  
What's happening on the Linux side when the domain crashes this way?

    J

> (XEN) mm.c:1482:d31 Bad L1 flags 800000
> (XEN) traps.c:437:d31 Unhandled invalid opcode fault/trap [#6] on VCPU 
> 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 31 (vcpu#0) crashed on cpu#2:
> (XEN) ----[ Xen-3.3.0  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    2
> (XEN) RIP:    e033:[<ffffffff8020b3ed>]
> (XEN) RFLAGS: 0000000000000282   EM: 1   CONTEXT: pv guest
> (XEN) rax: 00000000ffffffea   rbx: 000000000000077f   rcx: 
> 0000000000000054
> (XEN) rdx: 0000000000000000   rsi: 8000000331df5061   rdi: 
> ffffffff8077f000
> (XEN) rbp: ffffffff806e1f88   rsp: ffffffff806e1f28   r8:  
> 0000000000000010
> (XEN) r9:  0000000000000010   r10: ffffffff806e1e88   r11: 
> ffffffff80827f60
> (XEN) r12: ffffffff8077f000   r13: 8000000000000161   r14: 
> 8000000331df5061
> (XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 
> 00000000000026b0
> (XEN) cr3: 000000033c35c000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
> (XEN) Guest stack trace from rsp=ffffffff806e1f28:
> (XEN)    0000000000000054 ffffffff80827f60 ffffffff8020b3ed 
> 000000010000e030
> (XEN)    0000000000010082 ffffffff806e1f68 000000000000e02b 
> ffffffff8020b3e9
> (XEN)    0000000000000200 000000000000000c 000000000000000c 
> 0000000000001800
> (XEN)    ffffffff806e1ff8 ffffffff806f99cf 0000000000000000 
> 0000000000001000
> (XEN)    0000000000001800 ffffffff80e48000 0000000000020000 
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 
> 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 
> ffffffff80d45000
> (XEN)    ffffffff80d46000 ffffffff80d47000 ffffffff80d48000 
> ffffffff80d49000
> (XEN)    ffffffff80d4a000 ffffffff80d4b000 ffffffff80d4c000 
> ffffffff80d4d000
> (XEN)    ffffffff80d4e000 ffffffff80d4f000 ffffffff80d50000 
> ffffffff80d51000
> (XEN)    ffffffff80d52000 ffffffff80d53000 ffffffff80d54000 
> ffffffff80d55000
> (XEN)    ffffffff80d56000 ffffffff80d57000 ffffffff80d58000 
> ffffffff80d59000
> (XEN)    ffffffff80d5a000 ffffffff80d5b000 ffffffff80d5c000 
> ffffffff80d5d000
> (XEN)    ffffffff80d5e000 ffffffff80d5f000 ffffffff80d60000 
> ffffffff80d61000
> (XEN)    ffffffff80d62000 ffffffff80d63000 ffffffff80d64000 
> ffffffff80d65000
> (XEN)    ffffffff80d66000 ffffffff80d67000 ffffffff80d68000 
> ffffffff80d69000
> (XEN)    ffffffff80d6a000 ffffffff80d6b000 ffffffff80d6c000 
> ffffffff80d6d000
> (XEN)    ffffffff80d6e000 ffffffff80d6f000 ffffffff80d70000 
> ffffffff80d71000
> (XEN)    ffffffff80d72000 ffffffff80d73000 ffffffff80d74000 
> ffffffff80d75000
> (XEN)    ffffffff80d76000 ffffffff80d77000 ffffffff80d78000 
> ffffffff80d79000

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Guillaume Rousse]

Jeremy Fitzhardinge a écrit :
> Guillaume Rousse wrote:
>> Jeremy Fitzhardinge a écrit :
>>>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it 
>>>> down further if that'd be useful to you.
>>>>   
>>>
>>> Yes, please.
>> Was there any final conclusion there ? Because I'm currently trying a 
>> 2.6.27.5 with the same kind of issue:
> 
> I don't know of any outstanding bugs which cause this kind of symptom.  
> What's happening on the Linux side when the domain crashes this way?
Immediate crash, without further information.

I've opened a but report on mandriva bugzilla:
https://qa.mandriva.com/show_bug.cgi?id=46470

And I've yet to try with vanilla linus kernel.
-- 
Guillaume Rousse
Moyens Informatiques - INRIA Futurs
Tel: 01 69 35 69 62

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Guillaume Rousse wrote:
> Jeremy Fitzhardinge a écrit :
>> Guillaume Rousse wrote:
>>> Jeremy Fitzhardinge a écrit :
>>>>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it 
>>>>> down further if that'd be useful to you.
>>>>>   
>>>>
>>>> Yes, please.
>>> Was there any final conclusion there ? Because I'm currently trying 
>>> a 2.6.27.5 with the same kind of issue:
>>
>> I don't know of any outstanding bugs which cause this kind of 
>> symptom.  What's happening on the Linux side when the domain crashes 
>> this way?
> Immediate crash, without further information.
>
> I've opened a but report on mandriva bugzilla:
> https://qa.mandriva.com/show_bug.cgi?id=46470

Does earlyprintk=xen reveal anything more?  What does the rip 
(ffffffff8020b3ed) map to in the kernel?

> And I've yet to try with vanilla linus kernel.


Please do.  If possible, build a linus kernel with the mandriva config.

    J

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Guillaume Rousse]

Jeremy Fitzhardinge a écrit :
> Guillaume Rousse wrote:
>> Jeremy Fitzhardinge a écrit :
>>> Guillaume Rousse wrote:
>>>> Jeremy Fitzhardinge a écrit :
>>>>>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow it 
>>>>>> down further if that'd be useful to you.
>>>>>>   
>>>>>
>>>>> Yes, please.
>>>> Was there any final conclusion there ? Because I'm currently trying 
>>>> a 2.6.27.5 with the same kind of issue:
>>>
>>> I don't know of any outstanding bugs which cause this kind of 
>>> symptom.  What's happening on the Linux side when the domain crashes 
>>> this way?
>> Immediate crash, without further information.
>>
>> I've opened a but report on mandriva bugzilla:
>> https://qa.mandriva.com/show_bug.cgi?id=46470
> 
> Does earlyprintk=xen reveal anything more?  What does the rip 
> (ffffffff8020b3ed) map to in the kernel?
I guess earlyprintk=xen is a dom0 option, because passing it to domU 
didn't change anything. The given adress map to set_page_prot, according 
to the System.map file.

BTW, it only happens with ISCSI-stored hosts, whereas it works OK with 
LVM-stored hosts (tough on slightly different hardware). On other hosts, 
the guest kernel boots OK, and then fails to mount underlying FS (but 
that's a different issue). I updated upstream report with details.
-- 
Guillaume Rousse
Service des Moyens Informatiques
INRIA Saclay - Ile de France
Tel: 01 69 35 69 62

Re: Re: 2.6.26-rc8 pv_ops causes Unhandled invalid opcode fault/trap

[Jeremy Fitzhardinge]

Guillaume Rousse wrote:
> Jeremy Fitzhardinge a écrit :
>> Guillaume Rousse wrote:
>>> Jeremy Fitzhardinge a écrit :
>>>> Guillaume Rousse wrote:
>>>>> Jeremy Fitzhardinge a écrit :
>>>>>>> FWIW, I just tried 2.6.26-rc1 and it booted fine.  I can narrow 
>>>>>>> it down further if that'd be useful to you.
>>>>>>>   
>>>>>>
>>>>>> Yes, please.
>>>>> Was there any final conclusion there ? Because I'm currently 
>>>>> trying a 2.6.27.5 with the same kind of issue:
>>>>
>>>> I don't know of any outstanding bugs which cause this kind of 
>>>> symptom.  What's happening on the Linux side when the domain 
>>>> crashes this way?
>>> Immediate crash, without further information.
>>>
>>> I've opened a but report on mandriva bugzilla:
>>> https://qa.mandriva.com/show_bug.cgi?id=46470
>>
>> Does earlyprintk=xen reveal anything more?  What does the rip 
>> (ffffffff8020b3ed) map to in the kernel?
> I guess earlyprintk=xen is a dom0 option, because passing it to domU 
> didn't change anything.

It will be most helpful if you're running a debug=y build of Xen, so the 
domU's early console output appears on the Xen console.

> The given adress map to set_page_prot, according to the System.map file.

That's interesting.

> BTW, it only happens with ISCSI-stored hosts, whereas it works OK with 
> LVM-stored hosts (tough on slightly different hardware). 

Uh, that's very mysterious.  The underlying storage shouldn't have any 
effect on whether the kernel can boot - it's either loaded or not.
> On other hosts, the guest kernel boots OK, and then fails to mount 
> underlying FS (but that's a different issue). I updated upstream 
> report with details.


I'm on vacation at the moment, but I'll give this a closer look when I 
return.

    J