Re: 2.6.28 64-bit domU not booting

[Valtteri Kiviniemi <valtteri.kiviniemi@dataproof.fi>]

Hi,

This is indeed a very strange problem. I just disabled _everything_ in 
the kernel menuconfig. No drivers, no filesystems, no nothing. I even 
disabled all in processor types and features.

Still the same error message. So it's probably a bug or something 
because its not just working no matter what options are enabled/disabled 
in kernel.

- Valtteri Kiviniemi

Valtteri Kiviniemi kirjoitti:
> Hi,
> 
> Nice, I dont think that i have the necessary skillz for that :) but thnx 
> anyway
> 
> - Valtteri Kiviniemi
> 
> Ian Campbell kirjoitti:
>> On Wed, 2009-01-07 at 15:28 +0200, Valtteri Kiviniemi wrote:
>>> Hi,
>>>
>>> I have now upgraded to the latest Xen with debug=y and I dont still 
>>> get any specific error messages or how do I use the debug?
>>
>> Looks like it is crashing really very early before even earlyprintk. I
>> guess you need to decode the RIP and any likely looking return addresses
>> on the stack manually as I described in a previous mail.
>>
>> Ian.
>>> In xm dmesg I have now this:
>>>
>>> root@xen-1:/# xm dmesg
>>> (XEN) mm.c:1482:d5 Bad L1 flags 800000
>>> (XEN) traps.c:437:d5 Unhandled invalid opcode fault/trap [#6] on VCPU 
>>> 0 [ec=0000]
>>> (XEN) domain_crash_sync called from entry.S
>>> (XEN) Domain 5 (vcpu#0) crashed on cpu#2:
>>> (XEN) ----[ Xen-3.3.1  x86_64  debug=y  Not tainted ]----
>>> (XEN) CPU:    2
>>> (XEN) RIP:    e033:[<ffffffff8020a7aa>]
>>> (XEN) RFLAGS: 0000000000000282   EM: 1   CONTEXT: pv guest
>>> (XEN) rax: 00000000ffffffea   rbx: 8000000000000161   rcx: 
>>> 0000000000000008
>>> (XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 
>>> 00000000deadbeef
>>> (XEN) rbp: 0000000001000000   rsp: ffffffff80599f58   r8:  
>>> 0000000000000000
>>> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
>>> 0000000000000000
>>> (XEN) r12: ffffffff80622000   r13: 0000000000000200   r14: 
>>> 0000000000001000
>>> (XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 
>>> 00000000000026b0
>>> (XEN) cr3: 000000021a1bc000   cr2: 0000000000000000
>>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
>>> (XEN) Guest stack trace from rsp=ffffffff80599f58:
>>> (XEN)    0000000000000008 0000000000000000 ffffffff8020a7aa 
>>> 000000010000e030
>>> (XEN)    0000000000010082 ffffffff80599f98 000000000000e02b 
>>> ffffffff8020a7a6
>>> (XEN)    ffffffff8020c18c 0000000000000200 0000000000000008 
>>> ffffffff805c4d3f
>>> (XEN)    ffffffff80710000 0000000000010000 0000000000000008 
>>> 0000000000000000
>>> (XEN)    0000000000000000 0000000000000000 0000000000000000 
>>> 0000000000000000
>>> (XEN)    0000000000000000 ffffffff8068d000 ffffffff8068e000 
>>> ffffffff8068f000
>>> (XEN)    ffffffff80690000 ffffffff80691000 ffffffff80692000 
>>> ffffffff80693000
>>> (XEN)    ffffffff80694000 ffffffff80695000 ffffffff80696000 
>>> ffffffff80697000
>>> (XEN)    ffffffff80698000 ffffffff80699000 ffffffff8069a000 
>>> ffffffff8069b000
>>> (XEN)    ffffffff8069c000 ffffffff8069d000 ffffffff8069e000 
>>> ffffffff8069f000
>>> (XEN)    ffffffff806a0000 ffffffff806a1000 ffffffff806a2000 
>>> ffffffff806a3000
>>> (XEN)    ffffffff806a4000 ffffffff806a5000 ffffffff806a6000 
>>> ffffffff806a7000
>>> (XEN)    ffffffff806a8000 ffffffff806a9000 ffffffff806aa000 
>>> ffffffff806ab000
>>> (XEN)    ffffffff806ac000 ffffffff806ad000 ffffffff806ae000 
>>> ffffffff806af000
>>> (XEN)    ffffffff806b0000 ffffffff806b1000 ffffffff806b2000 
>>> ffffffff806b3000
>>> (XEN)    ffffffff806b4000 ffffffff806b5000 ffffffff806b6000 
>>> ffffffff806b7000
>>> (XEN)    ffffffff806b8000 ffffffff806b9000 ffffffff806ba000 
>>> ffffffff806bb000
>>> (XEN)    ffffffff806bc000 ffffffff806bd000 ffffffff806be000 
>>> ffffffff806bf000
>>> (XEN)    ffffffff806c0000 ffffffff806c1000 ffffffff806c2000 
>>> ffffffff806c3000
>>> (XEN)    ffffffff806c4000 ffffffff806c5000 ffffffff806c6000 
>>> ffffffff806c7000
>>> (XEN) mm.c:1482:d6 Bad L1 flags 800000
>>> (XEN) traps.c:437:d6 Unhandled invalid opcode fault/trap [#6] on VCPU 
>>> 0 [ec=0000]
>>> (XEN) domain_crash_sync called from entry.S
>>> (XEN) Domain 6 (vcpu#0) crashed on cpu#3:
>>> (XEN) ----[ Xen-3.3.1  x86_64  debug=y  Not tainted ]----
>>> (XEN) CPU:    3
>>> (XEN) RIP:    e033:[<ffffffff8020a7aa>]
>>> (XEN) RFLAGS: 0000000000000282   EM: 1   CONTEXT: pv guest
>>> (XEN) rax: 00000000ffffffea   rbx: 8000000000000161   rcx: 
>>> 0000000000000008
>>> (XEN) rdx: 00000000deadbeef   rsi: 00000000deadbeef   rdi: 
>>> 00000000deadbeef
>>> (XEN) rbp: 0000000001000000   rsp: ffffffff80599f58   r8:  
>>> 0000000000000000
>>> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
>>> 0000000000000000
>>> (XEN) r12: ffffffff80622000   r13: 0000000000000200   r14: 
>>> 0000000000001000
>>> (XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 
>>> 00000000000026b0
>>> (XEN) cr3: 0000000219dbc000   cr2: 0000000000000000
>>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
>>> (XEN) Guest stack trace from rsp=ffffffff80599f58:
>>> (XEN)    0000000000000008 0000000000000000 ffffffff8020a7aa 
>>> 000000010000e030
>>> (XEN)    0000000000010082 ffffffff80599f98 000000000000e02b 
>>> ffffffff8020a7a6
>>> (XEN)    ffffffff8020c18c 0000000000000200 0000000000000008 
>>> ffffffff805c4d3f
>>> (XEN)    ffffffff80710000 0000000000010000 0000000000000008 
>>> 0000000000000000
>>> (XEN)    0000000000000000 0000000000000000 0000000000000000 
>>> 0000000000000000
>>> (XEN)    0000000000000000 ffffffff8068d000 ffffffff8068e000 
>>> ffffffff8068f000
>>> (XEN)    ffffffff80690000 ffffffff80691000 ffffffff80692000 
>>> ffffffff80693000
>>> (XEN)    ffffffff80694000 ffffffff80695000 ffffffff80696000 
>>> ffffffff80697000
>>> (XEN)    ffffffff80698000 ffffffff80699000 ffffffff8069a000 
>>> ffffffff8069b000
>>> (XEN)    ffffffff8069c000 ffffffff8069d000 ffffffff8069e000 
>>> ffffffff8069f000
>>> (XEN)    ffffffff806a0000 ffffffff806a1000 ffffffff806a2000 
>>> ffffffff806a3000
>>> (XEN)    ffffffff806a4000 ffffffff806a5000 ffffffff806a6000 
>>> ffffffff806a7000
>>> (XEN)    ffffffff806a8000 ffffffff806a9000 ffffffff806aa000 
>>> ffffffff806ab000
>>> (XEN)    ffffffff806ac000 ffffffff806ad000 ffffffff806ae000 
>>> ffffffff806af000
>>> (XEN)    ffffffff806b0000 ffffffff806b1000 ffffffff806b2000 
>>> ffffffff806b3000
>>> (XEN)    ffffffff806b4000 ffffffff806b5000 ffffffff806b6000 
>>> ffffffff806b7000
>>> (XEN)    ffffffff806b8000 ffffffff806b9000 ffffffff806ba000 
>>> ffffffff806bb000
>>> (XEN)    ffffffff806bc000 ffffffff806bd000 ffffffff806be000 
>>> ffffffff806bf000
>>> (XEN)    ffffffff806c0000 ffffffff806c1000 ffffffff806c2000 
>>> ffffffff806c3000
>>> (XEN)    ffffffff806c4000 ffffffff806c5000 ffffffff806c6000 
>>> ffffffff806c7000
>>>
>>>
>>> - Valtteri Kiviniemi
>>>
>>> Valtteri Kiviniemi kirjoitti:
>>>> Hi,
>>>>
>>>> Yeah I'll upgrade to the newest Xen with debug enabled later, so 
>>>> I'll will post the results then.
>>>>
>>>>
>>>> - Valtteri Kiviniemi
>>>>
>>>> Ian Campbell kirjoitti:
>>>>> On Tue, 2009-01-06 at 16:37 +0200, Valtteri Kiviniemi wrote:
>>>>>> Hi,
>>>>>>
>>>>>> No it does not print anything, just disappears immediately.
>>>>>>
>>>>>> This is what I get in xm dmesg after trying to launch the domU 
>>>>>> with 2.6.28
>>>>>>
>>>>>> http://pastebin.com/m6ab14265
>>>>> Since they aren't huge please just paste the messages inline or attach
>>>>> them rather than sending me off to some website.
>>>>>
>>>>> Looks like a hypercall has failed and the kernel has done a BUG():
>>>>>
>>>>> (XEN) mm.c:1482:d71 Bad L1 flags 800000
>>>>> (XEN) traps.c:437:d71 Unhandled invalid opcode fault/trap [#6] on 
>>>>> VCPU 0 [ec=0000]
>>>>> (XEN) domain_crash_sync called from entry.S
>>>>> (XEN) Domain 71 (vcpu#0) crashed on cpu#0:
>>>>> (XEN) ----[ Xen-3.3.1-rc1-pre  x86_64  debug=n  Not tainted ]----
>>>>> (XEN) CPU:    0
>>>>> (XEN) RIP:    e033:[<ffffffff8020a7aa>]
>>>>> (XEN) RFLAGS: 0000000000000282   EM: 1   CONTEXT: pv guest
>>>>> (XEN) rax: 00000000ffffffea   rbx: 8000000000000161   rcx: 
>>>>> 0000000000000008
>>>>> (XEN) rdx: 0000000000000000   rsi: 80000001d9ddd061   rdi: 
>>>>> ffffffff80622000
>>>>> (XEN) rbp: 0000000001000000   rsp: ffffffff80599f58   r8:  
>>>>> 0000000000000000
>>>>> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 
>>>>> 0000000000000000
>>>>> (XEN) r12: ffffffff80622000   r13: 0000000000000200   r14: 
>>>>> 0000000000001000
>>>>> (XEN) r15: 0000000000000800   cr0: 000000008005003b   cr4: 
>>>>> 00000000000026b0
>>>>> (XEN) cr3: 00000001d9cef000   cr2: 0000000000000000
>>>>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
>>>>> (XEN) Guest stack trace from rsp=ffffffff80599f58:
>>>>> (XEN)    0000000000000008 0000000000000000 ffffffff8020a7aa 
>>>>> 000000010000e030
>>>>> (XEN)    0000000000010082 ffffffff80599f98 000000000000e02b 
>>>>> ffffffff8020a7a6
>>>>> (XEN)    ffffffff8020c18c 0000000000000200 0000000000000008 
>>>>> ffffffff805c4d3f
>>>>> (XEN)    ffffffff80710000 0000000000010000 0000000000000008 
>>>>> 0000000000000000
>>>>> (XEN)    0000000000000000 0000000000000000 0000000000000000 
>>>>> 0000000000000000
>>>>> (XEN)    0000000000000000 ffffffff8068d000 ffffffff8068e000 
>>>>> ffffffff8068f000
>>>>> (XEN)    ffffffff80690000 ffffffff80691000 ffffffff80692000 
>>>>> ffffffff80693000
>>>>> (XEN)    ffffffff80694000 ffffffff80695000 ffffffff80696000 
>>>>> ffffffff80697000
>>>>> (XEN)    ffffffff80698000 ffffffff80699000 ffffffff8069a000 
>>>>> ffffffff8069b000
>>>>> (XEN)    ffffffff8069c000 ffffffff8069d000 ffffffff8069e000 
>>>>> ffffffff8069f000
>>>>> (XEN)    ffffffff806a0000 ffffffff806a1000 ffffffff806a2000 
>>>>> ffffffff806a3000
>>>>> (XEN)    ffffffff806a4000 ffffffff806a5000 ffffffff806a6000 
>>>>> ffffffff806a7000
>>>>> (XEN)    ffffffff806a8000 ffffffff806a9000 ffffffff806aa000 
>>>>> ffffffff806ab000
>>>>> (XEN)    ffffffff806ac000 ffffffff806ad000 ffffffff806ae000 
>>>>> ffffffff806af000
>>>>> (XEN)    ffffffff806b0000 ffffffff806b1000 ffffffff806b2000 
>>>>> ffffffff806b3000
>>>>> (XEN)    ffffffff806b4000 ffffffff806b5000 ffffffff806b6000 
>>>>> ffffffff806b7000
>>>>> (XEN)    ffffffff806b8000 ffffffff806b9000 ffffffff806ba000 
>>>>> ffffffff806bb000
>>>>> (XEN)    ffffffff806bc000 ffffffff806bd000 ffffffff806be000 
>>>>> ffffffff806bf000
>>>>> (XEN)    ffffffff806c0000 ffffffff806c1000 ffffffff806c2000 
>>>>> ffffffff806c3000
>>>>> (XEN)    ffffffff806c4000 ffffffff806c5000 ffffffff806c6000 
>>>>> ffffffff806c7000
>>>>>
>>>>> What does ffffffff8020a7aa correspond to in your kernel? Do any of the
>>>>> other values on the stack correspond to interesting functions? I 
>>>>> usually
>>>>> pipe such stack traces into "| gdb-disas vmlinux" (attached, needs
>>>>> kernel with debug info), of course it would be better to update to 
>>>>> 3.3.1
>>>>> final and build with debugging enabled to see the actual stack trace
>>>>> from the domU kernel.
>>>>>
>>>>>
>>>>>> - Valtteri Kiviniemi
>>>>>>
>>>>>> Ian Campbell kirjoitti:
>>>>>>> On Mon, 2009-01-05 at 20:11 +0200, Valtteri Kiviniemi wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I don't have the opportunity of rebuilding my hypervisor right 
>>>>>>>> now, but
>>>>>>>> here is the .config:
>>>>>>>>
>>>>>>>> http://pastebin.com/m181cf183
>>>>>>>>
>>>>>>>> Is there something wrong with the config or am I missing 
>>>>>>>> something there?
>>>>>>> No I think your config is ok, I successfully built and booted 
>>>>>>> 2.6.28 as
>>>>>>> a 64 bit domU on my test system using it. My command line was
>>>>>>> "root=/dev/xvda1 ro 4 console=hvc0 earlyprintk=xen"
>>>>>>>
>>>>>>> Do you get anything on the Xen console (xm dmesg) when you try to 
>>>>>>> start
>>>>>>> the guest?
>>>>>>>
>>>>>>>
>>>>>>>> - Valtteri Kiviniemi
>>>>>>>>
>>>>>>>> Ian Campbell kirjoitti:
>>>>>>>>> On Mon, 2009-01-05 at 14:19 +0200, Valtteri Kiviniemi wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Im using vanilla 2.6.28 kernel from kernel.org and when I try 
>>>>>>>>>> to boot my   domU with it, it just disappears without any 
>>>>>>>>>> error or warning.
>>>>>>>>> Can you build a debug=y hypervisor and add earlyprintk=xen to 
>>>>>>>>> the domU
>>>>>>>>> command line?
>>>>>>>>>
>>>>>>>>>> I have double checked that I have all the necessary options 
>>>>>>>>>> enabled on my kernel config.
>>>>>>>>> Can you supply the config pls.
>>>>>>>>>
>>>>>>>>> Ian.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Xen-devel mailing list
>>>>>>>>> Xen-devel@lists.xensource.com
>>>>>>>>> http://lists.xensource.com/xen-devel
>>>>>>>> _______________________________________________
>>>>>>>> Xen-devel mailing list
>>>>>>>> Xen-devel@lists.xensource.com
>>>>>>>> http://lists.xensource.com/xen-devel
>>>>>>> _______________________________________________
>>>>>>> Xen-devel mailing list
>>>>>>> Xen-devel@lists.xensource.com
>>>>>>> http://lists.xensource.com/xen-devel
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel