2.6.32-rc not booting with Xen-3.2 (bisected)

2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

Hi,

I'm running a Xen-3.2 Dom0 (Ubuntu 8.04 LTS, kernel 2.6.24) which is 
hosting several DomUs. One of them is a debian/unstable and I'm trying to 
run the latest -git version of Linus' tree. However, I wasn't able to test 
for a while and I noticed just now that I cannot boot 2.6.32-rc1 any more, 
2.6.31 (release) was booting fine. Unfortunately, "xm create sid.cfg -c" 
just says "Domain started", but then "xm list" just shows the DomU in "p" 
or in "b" state, and the console ("-c") isn't attached either, so I don't 
really know where the DomU stops (or panics). However, I was able to 
bisect it, I hope the bisection makes any sense:

------------------------------------
sid$ git bisect log
git bisect start
# good: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31
git bisect good 74fca6a42863ffacaf7ba6f1936a9f228950f657
# bad: [17d857be649a21ca90008c6dc425d849fa83db5c] Linux 2.6.32-rc1
git bisect bad 17d857be649a21ca90008c6dc425d849fa83db5c
# good: [5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3] Staging: vt6656: Integrate vt6656 into build system.
git bisect good 5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3
# bad: [c720f5655df159a630fa0290a0bd67c93e92b0bf] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6
git bisect bad c720f5655df159a630fa0290a0bd67c93e92b0bf
# good: [a03fdb7612874834d6847107198712d18b5242c7] Merge branch 'timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect good a03fdb7612874834d6847107198712d18b5242c7
# good: [3530c1886291df061e3972c55590777ef1cb67f8] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect good 3530c1886291df061e3972c55590777ef1cb67f8
# bad: [84d6ae431f315e8973aac3c3fe1d550fc9240ef3] V4L/DVB (13033): pt1: Don't use a deprecated DMA_BIT_MASK macro
git bisect bad 84d6ae431f315e8973aac3c3fe1d550fc9240ef3
# bad: [068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8] V4L/DVB (12939): SAA7164: Removed a duplicate call to address any PCI quirks.
git bisect bad 068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8
# bad: [f4e96deb4513d044653027d4921fd7592195503a] V4L/DVB (12756): af9015: fix typo in register compare
git bisect bad f4e96deb4513d044653027d4921fd7592195503a
# good: [dd0a70c8f921708eba29ef9f30dde1f14a74af05] x86: Move tsc_init to late_time_init
git bisect good dd0a70c8f921708eba29ef9f30dde1f14a74af05
# bad: [53e712d0844e99b8d8720327470b86ef401fb727] V4L/DVB (12734): cx25821: Fix some compilation troubles
git bisect bad 53e712d0844e99b8d8720327470b86ef401fb727
# good: [7bd867dfb4e0357e06a3211ab2bd0e714110def3] x86: Move get/set_wallclock to x86_platform_ops
git bisect good 7bd867dfb4e0357e06a3211ab2bd0e714110def3
# bad: [78f28b7c555359c67c2a0d23f7436e915329421e] Merge branch 'x86-platform-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 78f28b7c555359c67c2a0d23f7436e915329421e
# good: [9c88b25bcfde51eccf2cca13d933a2948d0ca4eb] frv: clean up linker script using new linker script macros.
git bisect good 9c88b25bcfde51eccf2cca13d933a2948d0ca4eb
-----------------------------------

Version numbers and more for both the domU and dom0 are on:
http://nerdbynature.de/bits/2.6.32/dom0/dpkg-l.txt

Only the "Merge branch x86-platform-for-linus" pops out 
where a lot of x86 changes have been merged, but I have no
idea how to bisect within a "single" change.

Please advise.

Thanks,
Christian.
-- 
BOFH excuse #308:

CD-ROM server needs recalibration

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Mon, 26 Oct 2009 at 15:02, Christian Kujau wrote:
> really know where the DomU stops (or panics). However, I was able to 
> bisect it, I hope the bisection makes any sense:

My bisection was one iteration short, here's the full one, which confirms 
my suspicion of 78f28b7c555359c67c2a0d23f7436e915329421e being the 
culprit:

domU$ git bisect start
# good: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31
git bisect good 74fca6a42863ffacaf7ba6f1936a9f228950f657
# bad: [17d857be649a21ca90008c6dc425d849fa83db5c] Linux 2.6.32-rc1
git bisect bad 17d857be649a21ca90008c6dc425d849fa83db5c
# good: [5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3] Staging: vt6656: Integrate vt6656 into build system.
git bisect good 5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3
# bad: [c720f5655df159a630fa0290a0bd67c93e92b0bf] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6
git bisect bad c720f5655df159a630fa0290a0bd67c93e92b0bf
# good: [a03fdb7612874834d6847107198712d18b5242c7] Merge branch 'timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect good a03fdb7612874834d6847107198712d18b5242c7
# good: [3530c1886291df061e3972c55590777ef1cb67f8] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
git bisect good 3530c1886291df061e3972c55590777ef1cb67f8
# bad: [84d6ae431f315e8973aac3c3fe1d550fc9240ef3] V4L/DVB (13033): pt1: Don't use a deprecated DMA_BIT_MASK macro
git bisect bad 84d6ae431f315e8973aac3c3fe1d550fc9240ef3
# bad: [068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8] V4L/DVB (12939): SAA7164: Removed a duplicate call to address any PCI quirks.
git bisect bad 068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8
# bad: [f4e96deb4513d044653027d4921fd7592195503a] V4L/DVB (12756): af9015: fix typo in register compare
git bisect bad f4e96deb4513d044653027d4921fd7592195503a
# good: [dd0a70c8f921708eba29ef9f30dde1f14a74af05] x86: Move tsc_init to late_time_init
git bisect good dd0a70c8f921708eba29ef9f30dde1f14a74af05
# bad: [53e712d0844e99b8d8720327470b86ef401fb727] V4L/DVB (12734): cx25821: Fix some compilation troubles
git bisect bad 53e712d0844e99b8d8720327470b86ef401fb727
# good: [7bd867dfb4e0357e06a3211ab2bd0e714110def3] x86: Move get/set_wallclock to x86_platform_ops
git bisect good 7bd867dfb4e0357e06a3211ab2bd0e714110def3
# bad: [78f28b7c555359c67c2a0d23f7436e915329421e] Merge branch 'x86-platform-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git bisect bad 78f28b7c555359c67c2a0d23f7436e915329421e
# good: [9c88b25bcfde51eccf2cca13d933a2948d0ca4eb] frv: clean up linker script using new linker script macros.
git bisect good 9c88b25bcfde51eccf2cca13d933a2948d0ca4eb
# good: [3240a77b515f4a15e24d8a3a2135bc87b02cf1d6] sparc: Clean up linker script using new linker script macros.
git bisect good 3240a77b515f4a15e24d8a3a2135bc87b02cf1d6


> Version numbers and more for both the domU and dom0 are on:
> http://nerdbynature.de/bits/2.6.32/dom0/dpkg-l.txt

That should read: http://nerdbynature.de/bits/2.6.32/

Thanks,
Christian.
-- 
BOFH excuse #19:

floating point processor overflow

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Jeremy Fitzhardinge]

On 10/26/09 15:17, Christian Kujau wrote:
> On Mon, 26 Oct 2009 at 15:02, Christian Kujau wrote:
>   
>> really know where the DomU stops (or panics). However, I was able to 
>> bisect it, I hope the bisection makes any sense:
>>     
> My bisection was one iteration short, here's the full one, which confirms 
> my suspicion of 78f28b7c555359c67c2a0d23f7436e915329421e being the 
> culprit:
>   

Yes, that is indeed a likely suspect.  Is Xen compiled with DEBUG=y? 
Does anything appear on the Xen console (xm dmesg)?

    J

> domU$ git bisect start
> # good: [74fca6a42863ffacaf7ba6f1936a9f228950f657] Linux 2.6.31
> git bisect good 74fca6a42863ffacaf7ba6f1936a9f228950f657
> # bad: [17d857be649a21ca90008c6dc425d849fa83db5c] Linux 2.6.32-rc1
> git bisect bad 17d857be649a21ca90008c6dc425d849fa83db5c
> # good: [5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3] Staging: vt6656: Integrate vt6656 into build system.
> git bisect good 5d1fe0c98f2aef99fb57aaf6dd25e793c186cea3
> # bad: [c720f5655df159a630fa0290a0bd67c93e92b0bf] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6
> git bisect bad c720f5655df159a630fa0290a0bd67c93e92b0bf
> # good: [a03fdb7612874834d6847107198712d18b5242c7] Merge branch 'timers-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
> git bisect good a03fdb7612874834d6847107198712d18b5242c7
> # good: [3530c1886291df061e3972c55590777ef1cb67f8] Merge branch 'for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4
> git bisect good 3530c1886291df061e3972c55590777ef1cb67f8
> # bad: [84d6ae431f315e8973aac3c3fe1d550fc9240ef3] V4L/DVB (13033): pt1: Don't use a deprecated DMA_BIT_MASK macro
> git bisect bad 84d6ae431f315e8973aac3c3fe1d550fc9240ef3
> # bad: [068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8] V4L/DVB (12939): SAA7164: Removed a duplicate call to address any PCI quirks.
> git bisect bad 068ed40b8fc14cd3d16b5cf2db59ecd735a68ca8
> # bad: [f4e96deb4513d044653027d4921fd7592195503a] V4L/DVB (12756): af9015: fix typo in register compare
> git bisect bad f4e96deb4513d044653027d4921fd7592195503a
> # good: [dd0a70c8f921708eba29ef9f30dde1f14a74af05] x86: Move tsc_init to late_time_init
> git bisect good dd0a70c8f921708eba29ef9f30dde1f14a74af05
> # bad: [53e712d0844e99b8d8720327470b86ef401fb727] V4L/DVB (12734): cx25821: Fix some compilation troubles
> git bisect bad 53e712d0844e99b8d8720327470b86ef401fb727
> # good: [7bd867dfb4e0357e06a3211ab2bd0e714110def3] x86: Move get/set_wallclock to x86_platform_ops
> git bisect good 7bd867dfb4e0357e06a3211ab2bd0e714110def3
> # bad: [78f28b7c555359c67c2a0d23f7436e915329421e] Merge branch 'x86-platform-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
> git bisect bad 78f28b7c555359c67c2a0d23f7436e915329421e
> # good: [9c88b25bcfde51eccf2cca13d933a2948d0ca4eb] frv: clean up linker script using new linker script macros.
> git bisect good 9c88b25bcfde51eccf2cca13d933a2948d0ca4eb
> # good: [3240a77b515f4a15e24d8a3a2135bc87b02cf1d6] sparc: Clean up linker script using new linker script macros.
> git bisect good 3240a77b515f4a15e24d8a3a2135bc87b02cf1d6
>
>
>   
>> Version numbers and more for both the domU and dom0 are on:
>> http://nerdbynature.de/bits/2.6.32/dom0/dpkg-l.txt
>>     
> That should read: http://nerdbynature.de/bits/2.6.32/
>
> Thanks,
> Christian.
>   

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Mon, 26 Oct 2009 at 15:55, Jeremy Fitzhardinge wrote:
> Yes, that is indeed a likely suspect.

Is there any way I can bisect this any further, i.e. which change of this 
merge is causing this?

> Is Xen compiled with DEBUG=y? 

Hm, I doubt that the "stock" Ubuntu Xen packages are compiled with DEBUG 
flags. I've put the .config of the running Dom0 kernel here:

http://nerdbynature.de/bits/2.6.32/dom0/

> Does anything appear on the Xen console (xm dmesg)?

Oh, I totally forgot about "xm dmesg", it's on the URL above too. I don't 
see any timestamps in "xm dmesg", so I guess I'll have to try again with 
"xm dmesg --clear" inbetween boots.

Thanks,
Christian.
-- 
BOFH excuse #210:

We didn't pay the Internet bill and it's been cut off.

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Mon, 26 Oct 2009 at 18:14, Christian Kujau wrote:
> Oh, I totally forgot about "xm dmesg", it's on the URL above too. I don't 
> see any timestamps in "xm dmesg", so I guess I'll have to try again with 
> "xm dmesg --clear" inbetween boots.

Did that - when I boot a working kernel (2.6.31), "xm dmesg" prints 
nothing but a newline. When I boot 2.6.31-rc1, this happens:

$ xm create sid.cfg -c
Using config file "/etc/xen/sid.cfg".
Started domain sid

$ xm dmesg
(XEN) mm.c:665:d889 Error getting mfn 127f (pfn 7403) from L1 entry 801000000127f065 for dom889
(XEN) traps.c:413:d889 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
(XEN) domain_crash_sync called from entry.S
(XEN) Domain 889 (vcpu#0) crashed on cpu#0:
(XEN) ----[ Xen-3.2.1-rc1-pre  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff812c50ad>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: guest
(XEN) rax: 00000000ffffffea   rbx: 800000000127f061   rcx: 000000000008885a
(XEN) rdx: 0000000000000000   rsi: 800000000127f061   rdi: ffffffff8127f000
(XEN) rbp: ffffffff81279fb8   rsp: ffffffff81279f18   r8:  0000000000000000
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: ffffffff8127f000   r13: 0000000000000000   r14: 0000000000000080
(XEN) r15: ffffffff81279fc8   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000000aa991000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffffffff81279f18:
(XEN)    000000000008885a 0000000000000000 ffffffff812c50ad 000000010000e030
(XEN)    0000000000010082 ffffffff81279f58 000000000000e02b ffffffff812c504c
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff81279f60 0000008000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff810197bd ffff8127f000007f 000000000000ffff
(XEN)    0000000000000000 ffffffff812c4d19 0000000000000000 0000000000000000
(XEN)    0000000000000000 ffffffff81385000 ffffffff81386000 ffffffff81387000
(XEN)    ffffffff81388000 ffffffff81389000 ffffffff8138a000 ffffffff8138b000
(XEN)    ffffffff8138c000 ffffffff8138d000 ffffffff8138e000 ffffffff8138f000
(XEN)    ffffffff81390000 ffffffff81391000 ffffffff81392000 ffffffff81393000
(XEN)    ffffffff81394000 ffffffff81395000 ffffffff81396000 ffffffff81397000
(XEN)    ffffffff81398000 ffffffff81399000 ffffffff8139a000 ffffffff8139b000
(XEN)    ffffffff8139c000 ffffffff8139d000 ffffffff8139e000 ffffffff8139f000
(XEN)    ffffffff813a0000 ffffffff813a1000 ffffffff813a2000 ffffffff813a3000
(XEN)    ffffffff813a4000 ffffffff813a5000 ffffffff813a6000 ffffffff813a7000
(XEN)    ffffffff813a8000 ffffffff813a9000 ffffffff813aa000 ffffffff813ab000
(XEN)    ffffffff813ac000 ffffffff813ad000 ffffffff813ae000 ffffffff813af000
(XEN)    ffffffff813b0000 ffffffff813b1000 ffffffff813b2000 ffffffff813b3000
(XEN)    ffffffff813b4000 ffffffff813b5000 ffffffff813b6000 ffffffff813b7000


Also, "xm list" takes very long to complete now:

$ time xm list sid
Name                  ID   Mem VCPUs      State   Time(s)
sid                  906   512     1     --p---     0.0
real    0m10.677s user    0m0.108s sys     0m0.028s


HTH,
Christian.
-- 
BOFH excuse #43:

boss forgot system password

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Jeremy Fitzhardinge]

On 10/26/09 18:26, Christian Kujau wrote:
> On Mon, 26 Oct 2009 at 18:14, Christian Kujau wrote:
>   
>> Oh, I totally forgot about "xm dmesg", it's on the URL above too. I don't 
>> see any timestamps in "xm dmesg", so I guess I'll have to try again with 
>> "xm dmesg --clear" inbetween boots.
>>     
> Did that - when I boot a working kernel (2.6.31), "xm dmesg" prints 
> nothing but a newline. When I boot 2.6.31-rc1, this happens:
>
> $ xm create sid.cfg -c
> Using config file "/etc/xen/sid.cfg".
> Started domain sid
>
> $ xm dmesg
> (XEN) mm.c:665:d889 Error getting mfn 127f (pfn 7403) from L1 entry 801000000127f065 for dom889
> (XEN) traps.c:413:d889 Unhandled invalid opcode fault/trap [#6] on VCPU 0 [ec=0000]
> (XEN) domain_crash_sync called from entry.S
> (XEN) Domain 889 (vcpu#0) crashed on cpu#0:
> (XEN) ----[ Xen-3.2.1-rc1-pre  x86_64  debug=n  Not tainted ]----
> (XEN) CPU:    0
> (XEN) RIP:    e033:[<ffffffff812c50ad>]
>   

What does this map to in the kernel symbol table?

> (XEN) RFLAGS: 0000000000000282   CONTEXT: guest
> (XEN) rax: 00000000ffffffea   rbx: 800000000127f061   rcx: 000000000008885a
> (XEN) rdx: 0000000000000000   rsi: 800000000127f061   rdi: ffffffff8127f000
> (XEN) rbp: ffffffff81279fb8   rsp: ffffffff81279f18   r8:  0000000000000000
> (XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
> (XEN) r12: ffffffff8127f000   r13: 0000000000000000   r14: 0000000000000080
> (XEN) r15: ffffffff81279fc8   cr0: 000000008005003b   cr4: 00000000000026b0
> (XEN) cr3: 00000000aa991000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
> (XEN) Guest stack trace from rsp=ffffffff81279f18:
> (XEN)    000000000008885a 0000000000000000 ffffffff812c50ad 000000010000e030
> (XEN)    0000000000010082 ffffffff81279f58 000000000000e02b ffffffff812c504c
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 ffffffff81279f60 0000008000000000 0000000000000000
> (XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
> (XEN)    0000000000000000 ffffffff810197bd ffff8127f000007f 000000000000ffff
> (XEN)    0000000000000000 ffffffff812c4d19 0000000000000000 0000000000000000
> (XEN)    0000000000000000 ffffffff81385000 ffffffff81386000 ffffffff81387000
> (XEN)    ffffffff81388000 ffffffff81389000 ffffffff8138a000 ffffffff8138b000
> (XEN)    ffffffff8138c000 ffffffff8138d000 ffffffff8138e000 ffffffff8138f000
> (XEN)    ffffffff81390000 ffffffff81391000 ffffffff81392000 ffffffff81393000
> (XEN)    ffffffff81394000 ffffffff81395000 ffffffff81396000 ffffffff81397000
> (XEN)    ffffffff81398000 ffffffff81399000 ffffffff8139a000 ffffffff8139b000
> (XEN)    ffffffff8139c000 ffffffff8139d000 ffffffff8139e000 ffffffff8139f000
> (XEN)    ffffffff813a0000 ffffffff813a1000 ffffffff813a2000 ffffffff813a3000
> (XEN)    ffffffff813a4000 ffffffff813a5000 ffffffff813a6000 ffffffff813a7000
> (XEN)    ffffffff813a8000 ffffffff813a9000 ffffffff813aa000 ffffffff813ab000
> (XEN)    ffffffff813ac000 ffffffff813ad000 ffffffff813ae000 ffffffff813af000
> (XEN)    ffffffff813b0000 ffffffff813b1000 ffffffff813b2000 ffffffff813b3000
> (XEN)    ffffffff813b4000 ffffffff813b5000 ffffffff813b6000 ffffffff813b7000
>
>
> Also, "xm list" takes very long to complete now:
>
> $ time xm list sid
> Name                  ID   Mem VCPUs      State   Time(s)
> sid                  906   512     1     --p---     0.0
> real    0m10.677s user    0m0.108s sys     0m0.028s
>   

Hm, that's odd.  Never seen that.

    J

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Mon, 26 Oct 2009 at 21:20, Jeremy Fitzhardinge wrote:
> What does this map to in the kernel symbol table?

Hm, I'm not sure how to answer this? Got a hint, please?

Christian.

> > Also, "xm list" takes very long to complete now:
> >
> > $ time xm list sid
> > Name                  ID   Mem VCPUs      State   Time(s)
> > sid                  906   512     1     --p---     0.0
> > real    0m10.677s user    0m0.108s sys     0m0.028s
> Hm, that's odd.  Never seen that.

The load of the Dom0 with the panicked DomU "running" higher than 
usual (3-4 instead of ~0) and xenstored is spinning and waiting for 
(causing?) disk I/O. This is not the case when the DomU is runing just 
fine. I didn't strace xenstored yet, because I don't want to risk
killing the whole Xen box.

Christian.
-- 
BOFH excuse #73:

Daemons did it

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Jeremy Fitzhardinge]

On 10/26/09 22:01, Christian Kujau wrote:
> On Mon, 26 Oct 2009 at 21:20, Jeremy Fitzhardinge wrote:
>   
>> What does this map to in the kernel symbol table?
>>     
> Hm, I'm not sure how to answer this? Got a hint, please?
>   

$ gdb vmlinux
(gdb) x/i 0xffffffff812c50ad

>>> Also, "xm list" takes very long to complete now:
>>>
>>> $ time xm list sid
>>> Name                  ID   Mem VCPUs      State   Time(s)
>>> sid                  906   512     1     --p---     0.0
>>> real    0m10.677s user    0m0.108s sys     0m0.028s
>>>       
>> Hm, that's odd.  Never seen that.
>>     
> The load of the Dom0 with the panicked DomU "running" higher than 
> usual (3-4 instead of ~0) and xenstored is spinning and waiting for 
> (causing?) disk I/O. This is not the case when the DomU is runing just 
> fine. I didn't strace xenstored yet, because I don't want to risk
> killing the whole Xen box.
>   

Hm, sounds like a xenstored bug, or something is just pounding on it. 
Are there any other domains running, or other busy processes?

    J

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Tue, 27 Oct 2009 at 11:18, Jeremy Fitzhardinge wrote:
> $ gdb vmlinux
> (gdb) x/i 0xffffffff812c50ad

The value is always ffffffff812c50ad, gdb says:

(gdb) x/i 0xffffffff812c50ad
0xffffffff812c50ad <xen_load_gdt_boot+252>:     ud2a  

> Hm, sounds like a xenstored bug, or something is just pounding on it. 
> Are there any other domains running, or other busy processes?

Yes, there's another domU running. But the high load and xenstored 
spinning is only happening when I try to start the domain with this 
2.6.32-rc kernel.

Christian.
-- 
BOFH excuse #145:

Flat tire on station wagon with tapes.  ("Never underestimate the bandwidth of a station wagon full of tapes hurling down the highway" Andrew S. Tannenbaum)

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Jeremy Fitzhardinge]

On 10/27/09 12:01, Christian Kujau wrote:
> On Tue, 27 Oct 2009 at 11:18, Jeremy Fitzhardinge wrote:
>   
>> $ gdb vmlinux
>> (gdb) x/i 0xffffffff812c50ad
>>     
> The value is always ffffffff812c50ad, gdb says:
>
> (gdb) x/i 0xffffffff812c50ad
> 0xffffffff812c50ad <xen_load_gdt_boot+252>:     ud2a  
>   

OK, I can repro this here and I'm looking into it.   This particular
crash has been reported before on other kernels, but it seems more
easily triggered in this kernel.

    J

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Jeremy Fitzhardinge]

On 10/27/09 12:01, Christian Kujau wrote:
> On Tue, 27 Oct 2009 at 11:18, Jeremy Fitzhardinge wrote:
>   
>> $ gdb vmlinux
>> (gdb) x/i 0xffffffff812c50ad
>>     
> The value is always ffffffff812c50ad, gdb says:
>
> (gdb) x/i 0xffffffff812c50ad
> 0xffffffff812c50ad <xen_load_gdt_boot+252>:     ud2a  
>   

Please try the patch below.

>> Hm, sounds like a xenstored bug, or something is just pounding on it. 
>> Are there any other domains running, or other busy processes?
>>     
> Yes, there's another domU running. But the high load and xenstored 
> spinning is only happening when I try to start the domain with this 
> 2.6.32-rc kernel.
>   

That seems unlikely to be related (or a secondary effect of having a
domain crash).

    J

Subject: [PATCH] xen: set up mmu_ops before trying to set any ptes

xen_setup_stackprotector() ends up trying to set page protections,
so we need to have vm_mmu_ops set up before trying to do so.
Failing to do so causes an early boot crash.

[ Impact: Fix early crash under Xen. ]

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 3439616..23a4d80 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -1075,6 +1075,8 @@ asmlinkage void __init xen_start_kernel(void)
 	 * Set up some pagetable state before starting to set any ptes.
 	 */
 
+	xen_init_mmu_ops();
+
 	/* Prevent unwanted bits from being set in PTEs. */
 	__supported_pte_mask &= ~_PAGE_GLOBAL;
 	if (!xen_initial_domain())
@@ -1099,7 +1101,6 @@ asmlinkage void __init xen_start_kernel(void)
 	 */
 	xen_setup_stackprotector();
 
-	xen_init_mmu_ops();
 	xen_init_irq_ops();
 	xen_init_cpuid_mask();
 

Re: 2.6.32-rc not booting with Xen-3.2 (bisected)

[Christian Kujau]

On Tue, 27 Oct 2009 at 19:10, Jeremy Fitzhardinge wrote:
> Please try the patch below.

I see the patch made it into mainline already[0], so I just tried the 
latest git checkout from Linus' tree and the DomU is booting now, with no 
other ill effects afaict.

Thanks so much for the quick fix!

Christian.

[0] 2e2ec952350f25242f2e0539db16b1e46f9eb01b
-- 
BOFH excuse #319:

Your computer hasn't been returning all the bits it gets from the Internet.