crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[James Harper]

I just upgraded one of my servers to Xen 4.0.1-rc1-pre (from
http://xenbits.xensource.com/xen-4.0-testing.hg) and kernel
2.6.31.13-ga3e7c7b (from
git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git) I get
this:

[    0.026808] Performance Counters: AMD PMU driver.
[    0.028001] ------------[ cut here ]------------
[    0.028001] WARNING: at arch/x86/xen/enlighten.c:722
init_hw_perf_counters+0x2fe/0x39e()
[    0.028001] Hardware name: ProLiant DL385 G2
[    0.028001] Modules linked in:
[    0.028001] Pid: 0, comm: swapper Not tainted 2.6.31.13-ga3e7c7b #1
[    0.028001] Call Trace:
[    0.028001]  [<ffffffff81465fc4>] ? init_hw_perf_counters+0x2fe/0x39e
[    0.028001]  [<ffffffff81465fc4>] ? init_hw_perf_counters+0x2fe/0x39e
[    0.028001]  [<ffffffff810494b6>] ? warn_slowpath_common+0x77/0xa3
[    0.028001]  [<ffffffff81465fc4>] ? init_hw_perf_counters+0x2fe/0x39e
[    0.028001]  [<ffffffff8129938f>] ? identify_cpu+0x2ff/0x308
[    0.028001]  [<ffffffff81465c6f>] ? identify_boot_cpu+0x15/0x3e
[    0.028001]  [<ffffffff81465ca1>] ? check_bugs+0x9/0x2e
[    0.028001]  [<ffffffff8145d854>] ? start_kernel+0x370/0x386
[    0.028001]  [<ffffffff8146031f>] ? xen_start_kernel+0x625/0x62b
[    0.028001] ---[ end trace 4eaa2a86a8e2da22 ]---
[    0.028001] ... version:                 0
[    0.028001] ... bit width:               48
[    0.028001] ... generic counters:        4
[    0.028001] ... value mask:              0000ffffffffffff
[    0.028001] ... max period:              00007fffffffffff
[    0.028001] ... fixed-purpose counters:  0
[    0.028001] ... counter mask:            000000000000000f
[    0.028008] SMP alternatives: switching to UP code

Which appears to be non-terminal, but then this:

[    0.076001] PCI: HP ProLiant DL385 G2 detected, enabling pci=bfsort.
[    0.076086] bio: create slab <bio-0> at 0
[    0.081343] BUG: unable to handle kernel NULL pointer dereference at
(null)
[    0.084001] IP: [<ffffffff81028051>] add_pin_to_irq_node+0x14/0x74
[    0.084001] PGD 0
[    0.084001] Oops: 0000 [#1] SMP
[    0.084001] last sysfs file:
[    0.084001] CPU 0
[    0.084001] Modules linked in:
[    0.084001] Pid: 1, comm: swapper Tainted: G        W
2.6.31.13-ga3e7c7b #1 ProLiant DL385 G2
[    0.084001] RIP: e030:[<ffffffff81028051>]  [<ffffffff81028051>]
add_pin_to_irq_node+0x14/0x74
[    0.084001] RSP: e02b:ffff88001fd43e00  EFLAGS: 00010286
[    0.084001] RAX: ffffffff813ab900 RBX: 0000000000000009 RCX:
0000000000000009
[    0.084001] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
0000000000000000
[    0.084001] RBP: 0000000000000000 R08: 0000000000000001 R09:
007400726f727265
[    0.084001] R10: ffff88001fe3dd70 R11: 0000000000000000 R12:
0000000000000009
[    0.084001] R13: 0000000000000000 R14: ffffffff813ab900 R15:
0000000000000009
[    0.084001] FS:  0000000000000000(0000) GS:ffffc90000000000(0000)
knlGS:0000000000000000
[    0.084001] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    0.084001] CR2: 0000000000000000 CR3: 0000000001001000 CR4:
0000000000000660
[    0.084001] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[    0.084001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[    0.084001] Process swapper (pid: 1, threadinfo ffff88001fd42000,
task ffff88001fd41480)
[    0.084001] Stack:
[    0.084001]  ffff88001fe3dd70 0000000000000009 0000000000000000
0000000000000001
[    0.084001] <0> 0000000000000009 ffffffff81028e34 0000000000000000
0000000000000009
[    0.084001] <0> ffffffff81197b0b ffff88001fe3a3a0 ffffffff814430a0
ffffffff81492ab0
[    0.084001] Call Trace:
[    0.084001]  [<ffffffff81028e34>] ?
setup_IO_APIC_irq_extra+0xde/0x1f6
[    0.084001]  [<ffffffff81197b0b>] ?
acpi_ev_sci_xrupt_handler+0x0/0x22
[    0.084001]  [<ffffffff81024691>] ? acpi_gsi_to_irq+0x14/0x1b
[    0.084001]  [<ffffffff81189eba>] ?
acpi_os_install_interrupt_handler+0x26/0x99
[    0.084001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.084001]  [<ffffffff8119716c>] ?
acpi_ev_install_xrupt_handlers+0x6/0x50
[    0.084001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.084001]  [<ffffffff81482aa0>] ? acpi_init+0x77/0x25f
[    0.084001]  [<ffffffff81480b20>] ? fbmem_init+0x62/0x92
[    0.084001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.084001]  [<ffffffff8100a051>] ? do_one_initcall+0x50/0x151
[    0.084001]  [<ffffffff8145d25a>] ? kernel_init+0x15b/0x1b1
[    0.084001]  [<ffffffff81013b6a>] ? child_rip+0xa/0x20
[    0.084001]  [<ffffffff81012d27>] ? int_ret_from_sys_call+0x7/0x1b
[    0.084001]  [<ffffffff810134dd>] ? retint_restore_args+0x5/0x6
[    0.084001]  [<ffffffff81013b60>] ? child_rip+0x0/0x20
[    0.084001] Code: 7f 02 81 ff 13 48 83 c3 08 48 83 3b 00 eb d8 5b 48
89 e8 5d 41 5c c3 41 55 49 89 fd 89 f7 41 54 41 89 cc 55 89 d5 53 48 83
ec 08 <49> 8b 5d 00 48 85 db 75 36 e8 7e ff ff ff 48 85 c0 75 19 41 5b
[    0.084001] RIP  [<ffffffff81028051>] add_pin_to_irq_node+0x14/0x74
[    0.084001]  RSP <ffff88001fd43e00>
[    0.084001] CR2: 0000000000000000
[    0.084001] ---[ end trace 4eaa2a86a8e2da23 ]---
[    0.084014] Kernel panic - not syncing: Attempted to kill init!
[    0.086664] Pid: 1, comm: swapper Tainted: G      D W
2.6.31.13-ga3e7c7b #1
[    0.088001] Call Trace:
[    0.088001]  [<ffffffff8129f2db>] ? panic+0x86/0x13a
[    0.088001]  [<ffffffff8100e6bd>] ? xen_force_evtchn_callback+0x9/0xa
[    0.088001]  [<ffffffff8100ed72>] ? check_events+0x12/0x20
[    0.088001]  [<ffffffff8100e6bd>] ? xen_force_evtchn_callback+0x9/0xa
[    0.088001]  [<ffffffff8100ed72>] ? check_events+0x12/0x20
[    0.088001]  [<ffffffff812a151d>] ? _write_lock_irq+0x7/0x16
[    0.088001]  [<ffffffff810523fd>] ? exit_ptrace+0xa7/0x126
[    0.088001]  [<ffffffff8104c45a>] ? do_exit+0x6a/0x67f
[    0.088001]  [<ffffffff8100ed5f>] ? xen_restore_fl_direct_end+0x0/0x1
[    0.088001]  [<ffffffff81016a15>] ? oops_end+0xae/0xb3
[    0.088001]  [<ffffffff8102f98b>] ? no_context+0x1ec/0x1fb
[    0.088001]  [<ffffffff81159992>] ? idr_get_empty_slot+0x169/0x24a
[    0.088001]  [<ffffffff8102fb27>] ?
__bad_area_nosemaphore+0x18d/0x1b1
[    0.088001]  [<ffffffff81159b68>] ? ida_get_new_above+0xf5/0x1b3
[    0.088001]  [<ffffffff8100ed72>] ? check_events+0x12/0x20
[    0.088001]  [<ffffffff8102fd50>] ? do_page_fault+0x9e/0x27b
[    0.088001]  [<ffffffff812a18f5>] ? page_fault+0x25/0x30
[    0.088001]  [<ffffffff81028051>] ? add_pin_to_irq_node+0x14/0x74
[    0.088001]  [<ffffffff81028e34>] ?
setup_IO_APIC_irq_extra+0xde/0x1f6
[    0.088001]  [<ffffffff81197b0b>] ?
acpi_ev_sci_xrupt_handler+0x0/0x22
[    0.088001]  [<ffffffff81024691>] ? acpi_gsi_to_irq+0x14/0x1b
[    0.088001]  [<ffffffff81189eba>] ?
acpi_os_install_interrupt_handler+0x26/0x99
[    0.088001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.088001]  [<ffffffff8119716c>] ?
acpi_ev_install_xrupt_handlers+0x6/0x50
[    0.088001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.088001]  [<ffffffff81482aa0>] ? acpi_init+0x77/0x25f
[    0.088001]  [<ffffffff81480b20>] ? fbmem_init+0x62/0x92
[    0.088001]  [<ffffffff81482a29>] ? acpi_init+0x0/0x25f
[    0.088001]  [<ffffffff8100a051>] ? do_one_initcall+0x50/0x151
[    0.088001]  [<ffffffff8145d25a>] ? kernel_init+0x15b/0x1b1
[    0.088001]  [<ffffffff81013b6a>] ? child_rip+0xa/0x20
[    0.088001]  [<ffffffff81012d27>] ? int_ret_from_sys_call+0x7/0x1b
[    0.088001]  [<ffffffff810134dd>] ? retint_restore_args+0x5/0x6
[    0.088001]  [<ffffffff81013b60>] ? child_rip+0x0/0x20
</>hpiLO->

... which is terminal and hangs hard. This is on a HP DL385G2 which I'm
accessing remotely via an iLO virtual serial port.

Any suggestions? 

Thanks

James

Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[Keir Fraser]

On 16/05/2010 14:58, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I just upgraded one of my servers to Xen 4.0.1-rc1-pre (from
> http://xenbits.xensource.com/xen-4.0-testing.hg) and kernel
> 2.6.31.13-ga3e7c7b (from
> git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git) I get
> this:

Does it work any better with Xen 4.0.0 (tag RELEASE-4.0.0 in the
xen-4.0-testing.hg repo)?

 -- Keir

RE: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[James Harper]

> 
> On 16/05/2010 14:58, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > I just upgraded one of my servers to Xen 4.0.1-rc1-pre (from
> > http://xenbits.xensource.com/xen-4.0-testing.hg) and kernel
> > 2.6.31.13-ga3e7c7b (from
> > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git) I get
> > this:
> 
> Does it work any better with Xen 4.0.0 (tag RELEASE-4.0.0 in the
> xen-4.0-testing.hg repo)?
> 

I finally got a chance to test this and no, it still crashes.

Is the git URL above the best pvops kernel to be using? I see references
to 2.6.32 around the place and wonder if I should be using that instead?

James

Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[Pasi Kärkkäinen]

On Wed, May 19, 2010 at 10:37:13PM +1000, James Harper wrote:
> > 
> > On 16/05/2010 14:58, "James Harper" <james.harper@bendigoit.com.au>
> wrote:
> > 
> > > I just upgraded one of my servers to Xen 4.0.1-rc1-pre (from
> > > http://xenbits.xensource.com/xen-4.0-testing.hg) and kernel
> > > 2.6.31.13-ga3e7c7b (from
> > > git://git.kernel.org/pub/scm/linux/kernel/git/jeremy/xen.git) I get
> > > this:
> > 
> > Does it work any better with Xen 4.0.0 (tag RELEASE-4.0.0 in the
> > xen-4.0-testing.hg repo)?
> > 
> 
> I finally got a chance to test this and no, it still crashes.
> 
> Is the git URL above the best pvops kernel to be using? I see references
> to 2.6.32 around the place and wonder if I should be using that instead?
> 

I think you want to use xen/stable-2.6.32.x branch from xen.git,
which is the long-term maintained branch.

changelog: http://git.kernel.org/?p=linux/kernel/git/jeremy/xen.git;a=shortlog;h=xen/stable-2.6.32.x
More information: http://wiki.xensource.com/xenwiki/XenParavirtOps

-- Pasi

Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[Keir Fraser]

On 19/05/2010 13:37, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Does it work any better with Xen 4.0.0 (tag RELEASE-4.0.0 in the
>> xen-4.0-testing.hg repo)?
>> 
> 
> I finally got a chance to test this and no, it still crashes.
> 
> Is the git URL above the best pvops kernel to be using? I see references
> to 2.6.32 around the place and wonder if I should be using that instead?

Yes, but it has multiple branches. If you want to give 2.6.32 a go then that
is branch xen/stable-2.6.32.x

 K.

Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[HePeng]

I had the same situation. I think you'd better check your udev 
version. Mine is 095, and is too old for the kernel 2.6.31.13. 

So I turn on the CONFIG_SYSFS_DEPRECATED_V2 options in kernel 
config. and rebuilt the kernel. 

And It's OK now.

RE: Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[James Harper]

> 
> I had the same situation. I think you'd better check your udev
> version. Mine is 095, and is too old for the kernel 2.6.31.13.
> 
> So I turn on the CONFIG_SYSFS_DEPRECATED_V2 options in kernel
> config. and rebuilt the kernel.
> 
> And It's OK now.
> 

Did that cause the same crash I described?

James

Re: Re: crash on boot Xen 4.0.1-rc1-pre & kernel 2.6.31.13-ga3e7c7b

[Konrad Rzeszutek Wilk]

On Wed, May 19, 2010 at 10:36:05PM +1000, James Harper wrote:
> > 
> > I had the same situation. I think you'd better check your udev
> > version. Mine is 095, and is too old for the kernel 2.6.31.13.
> > 
> > So I turn on the CONFIG_SYSFS_DEPRECATED_V2 options in kernel
> > config. and rebuilt the kernel.
> > 
> > And It's OK now.
> > 
> 
> Did that cause the same crash I described?

Hm, that crash looked to be happening before initrd is read (where the
SYSFS_DEPRE.. is required).

Do try the xen/stable-2.6.32.x. There were fixes put in there to fix
ACPI SCI interrupt handler problem. Can you also attach the full serial log?
I am curious under what IRQ the ACPI SCI is installed..