PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Paweł Zuzelski]

Hello,

kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining of the
boot process.

I have bisected it to a single commit, and the first bad commit is:
[fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts for IPIs and VIRQs

kernel v2.6.35.5 with reverted this commit works for me.

Here are the kernel configs I was using:
http://carme.pld-linux.org/~pawelz/kernel-2.6.35.5-domU-config
http://carme.pld-linux.org/~pawelz/kernel-2.6.32.22-domU-config
As you can see they are stripped down configs, intended to run in domU only.

I was testing it with the very simple domU configuration:

kernel  = '/srv/xen/bzImage'
memory  = '128'
vcpus = 2
name    = 'test'
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Here is the full output of kernel 2.6.35.5:

Using config file "/etc/xen/test".
Started domain test
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: 
[    0.000000] PID hash table entries: 512 (order: 0, 4096 bytes)
[    0.000000] Subtract (33 early reservations)
[    0.000000]   #1 [0001976000 - 0001987000]  XEN PAGETABLES
[    0.000000]   #2 [0001000000 - 00019125f8]   TEXT DATA BSS
[    0.000000]   #3 [0001933000 - 0001976000]  XEN START INFO
[    0.000000]   #4 [0000010000 - 0000012000]      TRAMPOLINE
[    0.000000]   #5 [0000012000 - 0000040000]         PGTABLE
[    0.000000]   #6 [0001912600 - 0001917600]       NODE_DATA
[    0.000000]   #7 [0001917600 - 0001918600]         BOOTMEM
[    0.000000]   #8 [0001918600 - 0001918618]         BOOTMEM
[    0.000000]   #9 [0001919000 - 000191a000]         BOOTMEM
[    0.000000]   #10 [000191a000 - 000191b000]         BOOTMEM
[    0.000000]   #11 [000191b000 - 000191c000]         BOOTMEM
[    0.000000]   #12 [0002200000 - 00023c0000]        MEMMAP 0
[    0.000000]   #13 [0001918640 - 00019187c0]         BOOTMEM
[    0.000000]   #14 [000191c000 - 000191cc00]         BOOTMEM
[    0.000000]   #15 [000191d000 - 000191e000]         BOOTMEM
[    0.000000]   #16 [000191e000 - 000191f000]         BOOTMEM
[    0.000000]   #17 [000191f000 - 0001920000]         BOOTMEM
[    0.000000]   #18 [00019187c0 - 00019188a0]         BOOTMEM
[    0.000000]   #19 [00019188c0 - 0001918928]         BOOTMEM
[    0.000000]   #20 [0001918940 - 00019189a8]         BOOTMEM
[    0.000000]   #21 [00019189c0 - 0001918a28]         BOOTMEM
[    0.000000]   #22 [0001918a40 - 0001918a41]         BOOTMEM
[    0.000000]   #23 [0001918a80 - 0001918a81]         BOOTMEM
[    0.000000]   #24 [0001987000 - 00019c1000]         BOOTMEM
[    0.000000]   #25 [0001918ac0 - 0001918ac8]         BOOTMEM
[    0.000000]   #26 [0001918b00 - 0001918b08]         BOOTMEM
[    0.000000]   #27 [0001918b40 - 0001918b48]         BOOTMEM
[    0.000000]   #28 [0001918b80 - 0001918b90]         BOOTMEM
[    0.000000]   #29 [0001918bc0 - 0001918cc0]         BOOTMEM
[    0.000000]   #30 [0001918cc0 - 0001918d08]         BOOTMEM
[    0.000000]   #31 [0001918d40 - 0001918d88]         BOOTMEM
[    0.000000]   #32 [0001920000 - 0001921000]         BOOTMEM
[    0.000000] Memory: 118724k/131072k available (3327k kernel code, 448k absent, 11900k reserved, 3931k data, 440k init)
[    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000] 	RCU-based detection of stalled CPUs is disabled.
[    0.000000] 	Verbose stalled-CPUs detection is disabled.
[    0.000000] NR_IRQS:2304
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] installing Xen timer for CPU 0
[    0.000000] BUG: unable to handle kernel NULL pointer dereference at (null)
[    0.000000] IP: [<(null)>] (null)
[    0.000000] PGD 0 
[    0.000000] Oops: 0010 [#1] SMP 
[    0.000000] last sysfs file: 
[    0.000000] CPU 0 
[    0.000000] Modules linked in:
[    0.000000] 
[    0.000000] Pid: 0, comm: swapper Not tainted 2.6.35.5 #1 /
[    0.000000] RIP: e030:[<0000000000000000>]  [<(null)>] (null)
[    0.000000] RSP: e02b:ffffffff81601d70  EFLAGS: 00010082
[    0.000000] RAX: ffffffff818fdb50 RBX: 0000000000000000 RCX: 0000000000000000
[    0.000000] RDX: 0000000000000000 RSI: ffffffff818c7958 RDI: 0000000000000000
[    0.000000] RBP: ffffffff81601d88 R08: ffffea00001b22d8 R09: 000000000000001a
[    0.000000] R10: 0000000000000000 R11: 0000000000006477 R12: ffffffff81623280
[    0.000000] R13: 0000000000000000 R14: 00000000ffffffea R15: 0000000000000000
[    0.000000] FS:  0000000000000000(0000) GS:ffff880001987000(0000) knlGS:0000000000000000
[    0.000000] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    0.000000] CR2: 0000000000000000 CR3: 00000000016b9000 CR4: 0000000000002620
[    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    0.000000] Process swapper (pid: 0, threadinfo ffffffff81600000, task ffffffff816c1020)
[    0.000000] Stack:
[    0.000000]  ffffffff8107c849 0000000000000000 ffff880007c0d000 ffffffff81601da8
[    0.000000] <0> ffffffff8107c829 ffffffff8133d3fb ffffffff81623280 ffffffff81601df8
[    0.000000] <0> ffffffff8107c033 ffffffff816202e4 ffffffff816232e4 ffffffff8100572f
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff8107c849>] ? default_enable+0x1a/0x28
[    0.000000]  [<ffffffff8107c829>] default_startup+0x19/0x1f
[    0.000000]  [<ffffffff8133d3fb>] ? _raw_spin_lock_irqsave+0xd/0x24
[    0.000000]  [<ffffffff8107c033>] __setup_irq+0x1ab/0x2d8
[    0.000000]  [<ffffffff8100572f>] ? xen_restore_fl_direct_end+0x0/0x1
[    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
[    0.000000]  [<ffffffff8107c278>] request_threaded_irq+0x118/0x146
[    0.000000]  [<ffffffff8125dda5>] bind_virq_to_irqhandler+0x146/0x168
[    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
[    0.000000]  [<ffffffff810055ee>] xen_setup_timer+0x59/0x9d
[    0.000000]  [<ffffffff81731b68>] xen_time_init+0x7b/0x89
[    0.000000]  [<ffffffff81731ead>] x86_late_time_init+0xa/0x11
[    0.000000]  [<ffffffff8172eb49>] start_kernel+0x30b/0x38d
[    0.000000]  [<ffffffff8172e2c6>] x86_64_start_reservations+0xb1/0xb5
[    0.000000]  [<ffffffff817313e8>] xen_start_kernel+0x508/0x50f
[    0.000000] Code:  Bad RIP value.
[    0.000000] RIP  [<(null)>] (null)
[    0.000000]  RSP <ffffffff81601d70>
[    0.000000] CR2: 0000000000000000
[    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
[    0.000000] Kernel panic - not syncing: Attempted to kill the idle task!
[    0.000000] Pid: 0, comm: swapper Tainted: G      D     2.6.35.5 #1
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff8133bae9>] panic+0x86/0xfa
[    0.000000]  [<ffffffff8103c5c6>] do_exit+0x6d/0x77e
[    0.000000]  [<ffffffff8133d43f>] ? _raw_spin_unlock_irqrestore+0x11/0x13
[    0.000000]  [<ffffffff8103ac55>] ? kmsg_dump+0x11e/0x139
[    0.000000]  [<ffffffff8100b936>] oops_end+0x8f/0x94
[    0.000000]  [<ffffffff810239cc>] no_context+0x1f4/0x203
[    0.000000]  [<ffffffff81023b65>] __bad_area_nosemaphore+0x18a/0x1ad
[    0.000000]  [<ffffffff81023b96>] bad_area_nosemaphore+0xe/0x10
[    0.000000]  [<ffffffff81023f10>] do_page_fault+0x115/0x229
[    0.000000]  [<ffffffff8133daf5>] page_fault+0x25/0x30
[    0.000000]  [<ffffffff8107c849>] ? default_enable+0x1a/0x28
[    0.000000]  [<ffffffff8107c829>] default_startup+0x19/0x1f
[    0.000000]  [<ffffffff8133d3fb>] ? _raw_spin_lock_irqsave+0xd/0x24
[    0.000000]  [<ffffffff8107c033>] __setup_irq+0x1ab/0x2d8
[    0.000000]  [<ffffffff8100572f>] ? xen_restore_fl_direct_end+0x0/0x1
[    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
[    0.000000]  [<ffffffff8107c278>] request_threaded_irq+0x118/0x146
[    0.000000]  [<ffffffff8125dda5>] bind_virq_to_irqhandler+0x146/0x168
[    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
[    0.000000]  [<ffffffff810055ee>] xen_setup_timer+0x59/0x9d
[    0.000000]  [<ffffffff81731b68>] xen_time_init+0x7b/0x89
[    0.000000]  [<ffffffff81731ead>] x86_late_time_init+0xa/0x11
[    0.000000]  [<ffffffff8172eb49>] start_kernel+0x30b/0x38d
[    0.000000]  [<ffffffff8172e2c6>] x86_64_start_reservations+0xb1/0xb5
[    0.000000]  [<ffffffff817313e8>] xen_start_kernel+0x508/0x50f

Additional info:
  arch: x86_64
  cpu:  Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz
  xen:  3.2.1
  dom0: Linux version 2.6.26-2-xen-amd64 (Debian 2.6.26-24lenny1)
		(dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease)
		(Debian 4.1.2-25)) #1 SMP Thu Aug 19 01:12:45 UTC 2010

-- 
Regards,
Paweł Zuzelski

Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[James Dingwall]

(Sorry, I don't have the original mail to reply to)

I was experiencing the same immediate crash with a null pointer dereference (log below) on boot with 2.6.35.5.  Reverting fb412a178502dc498430723b082a932f797e4763 also resolved the problem for me.

If my .config or other information about my setup will help let me know.

James


Linux version 2.6.35.5 (root@behemoth) (gcc version 4.4.3 (Gentoo 4.4.3-r2 p1.2) ) #6 SMP Wed Sep 22 10:19:10 BST 2010
Command line: root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/xvda1 udev doscsi dolvm dodmraid
ACPI in unprivileged domain disabled
BIOS-provided physical RAM map:
 Xen: 0000000000000000 - 00000000000a0000 (usable)
 Xen: 00000000000a0000 - 0000000000100000 (reserved)
 Xen: 0000000000100000 - 0000000020000000 (usable)
NX (Execute Disable) protection: active
DMI not present or invalid.
No AGP bridge found
last_pfn = 0x20000 max_arch_pfn = 0x400000000
init_memory_mapping: 0000000000000000-0000000020000000
RAMDISK: 01713000 - 01f20000
No NUMA configuration found
Faking a node at 0000000000000000-0000000020000000
Initmem setup node 0 0000000000000000-0000000020000000
  NODE_DATA [00000000016f24c0 - 00000000016f74bf]
Zone PFN ranges:
  DMA      0x00000001 -> 0x00001000
  DMA32    0x00001000 -> 0x00100000
  Normal   empty
Movable zone start PFN for each node
early_node_map[2] active PFN ranges
    0: 0x00000001 -> 0x000000a0
    0: 0x00000100 -> 0x00020000
SMP: Allowing 1 CPUs, 0 hotplug CPUs
No local APIC present
APIC: disable apic facility
APIC: switched to apic NOOP
Allocating PCI resources starting at 20000000 (gap: 20000000:e0000000)
Booting paravirtualized kernel on Xen
Xen version: 4.0.0 (preserve-AD)
setup_percpu: NR_CPUS:8 nr_cpumask_bits:8 nr_cpu_ids:1 nr_node_ids:1
PERCPU: Embedded 27 pages/cpu @ffff880002068000 s81856 r8192 d20544 u110592
pcpu-alloc: s81856 r8192 d20544 u110592 alloc=27*4096
pcpu-alloc: [0] 0
Xen: using vcpu_info placement
Built 1 zonelists in Node order, mobility grouping on.  Total pages: 129183
Policy zone: DMA32
Kernel command line: root=/dev/ram0 init=/linuxrc ramdisk=8192 real_root=/dev/xvda1 udev doscsi dolvm dodmraid
PID hash table entries: 2048 (order: 2, 16384 bytes)
Checking aperture...
No AGP bridge found
Subtract (37 early reservations)
  #1 [0002023000 - 0002038000]  XEN PAGETABLES
  #2 [0001000000 - 00016f2484]   TEXT DATA BSS
  #3 [0001713000 - 0001f20000]         RAMDISK
  #4 [0001f20000 - 0002023000]  XEN START INFO
  #5 [0000001000 - 0000003000]      TRAMPOLINE
  #6 [0000100000 - 00001ea000]         PGTABLE
  #7 [00016f24c0 - 00016f74c0]       NODE_DATA
  #8 [00016f74c0 - 00016f84c0]         BOOTMEM
  #9 [0000003000 - 0000003040]         BOOTMEM
  #10 [00016f9000 - 00016fa000]         BOOTMEM
  #11 [00016fa000 - 00016fb000]         BOOTMEM
  #12 [00016fb000 - 00016fc000]         BOOTMEM
  #13 [00016fc000 - 00016fd000]         BOOTMEM
  #14 [00016fd000 - 00016fe000]         BOOTMEM
  #15 [00016fe000 - 00016ff000]         BOOTMEM
  #16 [0002a00000 - 0003100000]        MEMMAP 0
  #17 [0002038000 - 0002050000]         BOOTMEM
  #18 [0002050000 - 0002068000]         BOOTMEM
  #19 [00016ff000 - 0001700000]         BOOTMEM
  #20 [0001700000 - 0001701000]         BOOTMEM
  #21 [0001701000 - 0001702000]         BOOTMEM
  #22 [00016f84c0 - 00016f85a0]         BOOTMEM
  #23 [00016f85c0 - 00016f8628]         BOOTMEM
  #24 [00016f8640 - 00016f86a8]         BOOTMEM
  #25 [00016f86c0 - 00016f8728]         BOOTMEM
  #26 [00016f8740 - 00016f879b]         BOOTMEM
  #27 [00016f87c0 - 00016f881b]         BOOTMEM
  #28 [0002068000 - 0002083000]         BOOTMEM
  #29 [00016f8840 - 00016f8848]         BOOTMEM
  #30 [00016f8880 - 00016f8888]         BOOTMEM
  #31 [00016f88c0 - 00016f88c4]         BOOTMEM
  #32 [00016f8900 - 00016f8908]         BOOTMEM
  #33 [00016f8940 - 00016f8a40]         BOOTMEM
  #34 [00016f8a40 - 00016f8a88]         BOOTMEM
  #35 [00016f8ac0 - 00016f8b08]         BOOTMEM
  #36 [0001702000 - 0001706000]         BOOTMEM
Memory: 498928k/524288k available (3822k kernel code, 388k absent, 24972k reserved, 2017k data, 540k init)
Hierarchical RCU implementation.
        RCU-based detection of stalled CPUs is disabled.
        Verbose stalled-CPUs detection is disabled.
NR_IRQS:4352 nr_irqs:256
Console: colour dummy device 80x25
console [tty0] enabled
console [hvc0] enabled
installing Xen timer for CPU 0
BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<(null)>] (null)
PGD 0
Oops: 0010 [#1] SMP
last sysfs file:
CPU 0
Modules linked in:

Pid: 0, comm: swapper Not tainted 2.6.35.5 #6 /
RIP: e030:[<0000000000000000>]  [<(null)>] (null)
RSP: e02b:ffffffff8151dd70  EFLAGS: 00010046
RAX: ffffffff816d6180 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: ffffffff8151dd88 R08: ffff88001fc016a0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8153f200
R13: 0000000000000000 R14: 00000000ffffffda R15: ffffffff8153f264
FS:  0000000000000000(0000) GS:ffff880002068000(0000) knlGS:0000000000000000
CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001541000 CR4: 0000000000000660
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper (pid: 0, threadinfo ffffffff8151c000, task ffffffff81549020)
Stack:
 ffffffff81067804 0000000000000000 ffff88001fc04e40 ffffffff8151dda8
<0> ffffffff810677e4 0000000000000040 ffffffff8153f200 ffffffff8151ddf8
<0> ffffffff81066cc7 0000000000000001 0000000000000200 ffffffff8100599f
Call Trace:
 [<ffffffff81067804>] ? default_enable+0x1a/0x28
 [<ffffffff810677e4>] default_startup+0x19/0x1f
 [<ffffffff81066cc7>] __setup_irq+0x186/0x2c1
 [<ffffffff8100599f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810056c0>] ? xen_timer_interrupt+0x0/0x17b
 [<ffffffff81066ee4>] request_threaded_irq+0xe2/0x129
 [<ffffffff81067b90>] ? handle_percpu_irq+0x0/0x69
 [<ffffffff812603bb>] bind_virq_to_irqhandler+0x18a/0x1b7
 [<ffffffff81217d26>] ? kvasprintf+0x5e/0x6e
 [<ffffffff810056c0>] ? xen_timer_interrupt+0x0/0x17b
 [<ffffffff8100567c>] xen_setup_timer+0x59/0x9d
 [<ffffffff815ce759>] xen_time_init+0xa1/0xad
 [<ffffffff815cef4e>] x86_late_time_init+0xa/0x11
 [<ffffffff815cad01>] start_kernel+0x35a/0x3d2
 [<ffffffff815ca2c8>] x86_64_start_reservations+0xb3/0xb7
 [<ffffffff815cdbfc>] xen_start_kernel+0x52b/0x52f
Code:  Bad RIP value.
RIP  [<(null)>] (null)
 RSP <ffffffff8151dd70>
CR2: 0000000000000000
---[ end trace a7919e7f17c0a725 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
Pid: 0, comm: swapper Tainted: G      D     2.6.35.5 #6
Call Trace:
 [<ffffffff813b6bd1>] panic+0x86/0xfa
 [<ffffffff810059b2>] ? check_events+0x12/0x20
 [<ffffffff8103e375>] do_exit+0x6d/0x6e8
 [<ffffffff813b8e8a>] ? _raw_spin_unlock_irqrestore+0x19/0x1c
 [<ffffffff8103bd78>] ? kmsg_dump+0x126/0x140
 [<ffffffff8100bad9>] oops_end+0x97/0x9c
 [<ffffffff810290a9>] no_context+0x1f7/0x206
 [<ffffffff81005361>] ? xen_force_evtchn_callback+0xd/0xf
 [<ffffffff810059b2>] ? check_events+0x12/0x20
 [<ffffffff81029237>] __bad_area_nosemaphore+0x17f/0x1a2
 [<ffffffff8100599f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff81029268>] bad_area_nosemaphore+0xe/0x10
 [<ffffffff81029532>] do_page_fault+0x168/0x30d
 [<ffffffff8100599f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff813b94b5>] page_fault+0x25/0x30
 [<ffffffff81067804>] ? default_enable+0x1a/0x28
 [<ffffffff810677e4>] default_startup+0x19/0x1f
 [<ffffffff81066cc7>] __setup_irq+0x186/0x2c1
 [<ffffffff8100599f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff810056c0>] ? xen_timer_interrupt+0x0/0x17b
 [<ffffffff81066ee4>] request_threaded_irq+0xe2/0x129
 [<ffffffff81067b90>] ? handle_percpu_irq+0x0/0x69
 [<ffffffff812603bb>] bind_virq_to_irqhandler+0x18a/0x1b7
 [<ffffffff81217d26>] ? kvasprintf+0x5e/0x6e
 [<ffffffff810056c0>] ? xen_timer_interrupt+0x0/0x17b
 [<ffffffff8100567c>] xen_setup_timer+0x59/0x9d
 [<ffffffff815ce759>] xen_time_init+0xa1/0xad
 [<ffffffff815cef4e>] x86_late_time_init+0xa/0x11
 [<ffffffff815cad01>] start_kernel+0x35a/0x3d2
 [<ffffffff815ca2c8>] x86_64_start_reservations+0xb3/0xb7
 [<ffffffff815cdbfc>] xen_start_kernel+0x52b/0x52f

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp

RE: Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[James Dingwall]

> I was experiencing the same immediate crash with a null pointer
> dereference (log below) on boot with 2.6.35.5.  Reverting
> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> me.

Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.

--- drivers/xen/events.c.orig   2010-09-23 16:10:34.257124313 +0100
+++ drivers/xen/events.c        2010-09-23 16:10:44.326194462 +0100
@@ -935,7 +935,7 @@
 	.retrigger		= retrigger_dynirq,
 };

-static struct irq_chip en_percpu_chip __read_mostly = {
+static struct irq_chip xen_percpu_chip __read_mostly = {
 	.name			= "xen-percpu",

 	.disable		= disable_dynirq,

James
This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp

Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Paweł Zuzelski]

On Thu, 23 Sep 2010, James Dingwall wrote:

> > I was experiencing the same immediate crash with a null pointer
> > dereference (log below) on boot with 2.6.35.5.  Reverting
> > fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> > me.
>
> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.

I confirm. This patch works for me. Thanks.

-- 
Regards,
Paweł Zuzelski

RE: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Dan Magenheimer]

Jeremy --

FYI, I think I've also seen this problem, or something similar,
but ONLY on a Nehalem box (and only intermittently), not on my
Core 2 Duo boxen.  The
Nehalem box is an SDP so I had assumed that it was something
to do with that, but maybe not.  Maybe some feature is
"leaking through" to the guest.  Hyperthreading? MSI? EPT?

Anyway, if you have a newer box, you might try reproducing
on that, rather than your usual development box (which IIRC
was a Core 2 Duo laptop?)

> -----Original Message-----
> From: Jeremy Fitzhardinge [mailto:jeremy@goop.org]
> Sent: Thursday, September 23, 2010 6:02 PM
> To: Pawel Zuzelski
> Cc: virtualization@lists.osdl.org; Jeremy Fitzhardinge; xen-
> devel@lists.xensource.com; lkml
> Subject: Re: [Xen-devel] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics
> just after the boot
> 
>  On 09/21/2010 12:05 PM, Paweł Zuzelski wrote:
> > Hello,
> >
> > kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining
> of the
> > boot process.
> >
> > I have bisected it to a single commit, and the first bad commit is:
> > [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts
> for IPIs and VIRQs
> >
> > kernel v2.6.35.5 with reverted this commit works for me.
> 
> Thanks very much for doing that.  I'll have to work out what's going on
> (obviously it doesn't do it for me).
> 
>     J
> 
> > Here are the kernel configs I was using:
> > http://carme.pld-linux.org/~pawelz/kernel-2.6.35.5-domU-config
> > http://carme.pld-linux.org/~pawelz/kernel-2.6.32.22-domU-config
> > As you can see they are stripped down configs, intended to run in
> domU only.
> >
> > I was testing it with the very simple domU configuration:
> >
> > kernel  = '/srv/xen/bzImage'
> > memory  = '128'
> > vcpus = 2
> > name    = 'test'
> > on_poweroff = 'destroy'
> > on_reboot   = 'restart'
> > on_crash    = 'restart'
> >
> > Here is the full output of kernel 2.6.35.5:
> >
> > Using config file "/etc/xen/test".
> > Started domain test
> > [    0.000000] Policy zone: DMA32
> > [    0.000000] Kernel command line:
> > [    0.000000] PID hash table entries: 512 (order: 0, 4096 bytes)
> > [    0.000000] Subtract (33 early reservations)
> > [    0.000000]   #1 [0001976000 - 0001987000]  XEN PAGETABLES
> > [    0.000000]   #2 [0001000000 - 00019125f8]   TEXT DATA BSS
> > [    0.000000]   #3 [0001933000 - 0001976000]  XEN START INFO
> > [    0.000000]   #4 [0000010000 - 0000012000]      TRAMPOLINE
> > [    0.000000]   #5 [0000012000 - 0000040000]         PGTABLE
> > [    0.000000]   #6 [0001912600 - 0001917600]       NODE_DATA
> > [    0.000000]   #7 [0001917600 - 0001918600]         BOOTMEM
> > [    0.000000]   #8 [0001918600 - 0001918618]         BOOTMEM
> > [    0.000000]   #9 [0001919000 - 000191a000]         BOOTMEM
> > [    0.000000]   #10 [000191a000 - 000191b000]         BOOTMEM
> > [    0.000000]   #11 [000191b000 - 000191c000]         BOOTMEM
> > [    0.000000]   #12 [0002200000 - 00023c0000]        MEMMAP 0
> > [    0.000000]   #13 [0001918640 - 00019187c0]         BOOTMEM
> > [    0.000000]   #14 [000191c000 - 000191cc00]         BOOTMEM
> > [    0.000000]   #15 [000191d000 - 000191e000]         BOOTMEM
> > [    0.000000]   #16 [000191e000 - 000191f000]         BOOTMEM
> > [    0.000000]   #17 [000191f000 - 0001920000]         BOOTMEM
> > [    0.000000]   #18 [00019187c0 - 00019188a0]         BOOTMEM
> > [    0.000000]   #19 [00019188c0 - 0001918928]         BOOTMEM
> > [    0.000000]   #20 [0001918940 - 00019189a8]         BOOTMEM
> > [    0.000000]   #21 [00019189c0 - 0001918a28]         BOOTMEM
> > [    0.000000]   #22 [0001918a40 - 0001918a41]         BOOTMEM
> > [    0.000000]   #23 [0001918a80 - 0001918a81]         BOOTMEM
> > [    0.000000]   #24 [0001987000 - 00019c1000]         BOOTMEM
> > [    0.000000]   #25 [0001918ac0 - 0001918ac8]         BOOTMEM
> > [    0.000000]   #26 [0001918b00 - 0001918b08]         BOOTMEM
> > [    0.000000]   #27 [0001918b40 - 0001918b48]         BOOTMEM
> > [    0.000000]   #28 [0001918b80 - 0001918b90]         BOOTMEM
> > [    0.000000]   #29 [0001918bc0 - 0001918cc0]         BOOTMEM
> > [    0.000000]   #30 [0001918cc0 - 0001918d08]         BOOTMEM
> > [    0.000000]   #31 [0001918d40 - 0001918d88]         BOOTMEM
> > [    0.000000]   #32 [0001920000 - 0001921000]         BOOTMEM
> > [    0.000000] Memory: 118724k/131072k available (3327k kernel code,
> 448k absent, 11900k reserved, 3931k data, 440k init)
> > [    0.000000] SLUB: Genslabs=14, HWalign=64, Order=0-3,
> MinObjects=0, CPUs=2, Nodes=1
> > [    0.000000] Hierarchical RCU implementation.
> > [    0.000000] 	RCU-based detection of stalled CPUs is disabled.
> > [    0.000000] 	Verbose stalled-CPUs detection is disabled.
> > [    0.000000] NR_IRQS:2304
> > [    0.000000] Console: colour dummy device 80x25
> > [    0.000000] console [tty0] enabled
> > [    0.000000] console [hvc0] enabled
> > [    0.000000] installing Xen timer for CPU 0
> > [    0.000000] BUG: unable to handle kernel NULL pointer dereference
> at (null)
> > [    0.000000] IP: [<(null)>] (null)
> > [    0.000000] PGD 0
> > [    0.000000] Oops: 0010 [#1] SMP
> > [    0.000000] last sysfs file:
> > [    0.000000] CPU 0
> > [    0.000000] Modules linked in:
> > [    0.000000]
> > [    0.000000] Pid: 0, comm: swapper Not tainted 2.6.35.5 #1 /
> > [    0.000000] RIP: e030:[<0000000000000000>]  [<(null)>] (null)
> > [    0.000000] RSP: e02b:ffffffff81601d70  EFLAGS: 00010082
> > [    0.000000] RAX: ffffffff818fdb50 RBX: 0000000000000000 RCX:
> 0000000000000000
> > [    0.000000] RDX: 0000000000000000 RSI: ffffffff818c7958 RDI:
> 0000000000000000
> > [    0.000000] RBP: ffffffff81601d88 R08: ffffea00001b22d8 R09:
> 000000000000001a
> > [    0.000000] R10: 0000000000000000 R11: 0000000000006477 R12:
> ffffffff81623280
> > [    0.000000] R13: 0000000000000000 R14: 00000000ffffffea R15:
> 0000000000000000
> > [    0.000000] FS:  0000000000000000(0000) GS:ffff880001987000(0000)
> knlGS:0000000000000000
> > [    0.000000] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
> > [    0.000000] CR2: 0000000000000000 CR3: 00000000016b9000 CR4:
> 0000000000002620
> > [    0.000000] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> > [    0.000000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> > [    0.000000] Process swapper (pid: 0, threadinfo ffffffff81600000,
> task ffffffff816c1020)
> > [    0.000000] Stack:
> > [    0.000000]  ffffffff8107c849 0000000000000000 ffff880007c0d000
> ffffffff81601da8
> > [    0.000000] <0> ffffffff8107c829 ffffffff8133d3fb ffffffff81623280
> ffffffff81601df8
> > [    0.000000] <0> ffffffff8107c033 ffffffff816202e4 ffffffff816232e4
> ffffffff8100572f
> > [    0.000000] Call Trace:
> > [    0.000000]  [<ffffffff8107c849>] ? default_enable+0x1a/0x28
> > [    0.000000]  [<ffffffff8107c829>] default_startup+0x19/0x1f
> > [    0.000000]  [<ffffffff8133d3fb>] ?
> _raw_spin_lock_irqsave+0xd/0x24
> > [    0.000000]  [<ffffffff8107c033>] __setup_irq+0x1ab/0x2d8
> > [    0.000000]  [<ffffffff8100572f>] ?
> xen_restore_fl_direct_end+0x0/0x1
> > [    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
> > [    0.000000]  [<ffffffff8107c278>] request_threaded_irq+0x118/0x146
> > [    0.000000]  [<ffffffff8125dda5>]
> bind_virq_to_irqhandler+0x146/0x168
> > [    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
> > [    0.000000]  [<ffffffff810055ee>] xen_setup_timer+0x59/0x9d
> > [    0.000000]  [<ffffffff81731b68>] xen_time_init+0x7b/0x89
> > [    0.000000]  [<ffffffff81731ead>] x86_late_time_init+0xa/0x11
> > [    0.000000]  [<ffffffff8172eb49>] start_kernel+0x30b/0x38d
> > [    0.000000]  [<ffffffff8172e2c6>]
> x86_64_start_reservations+0xb1/0xb5
> > [    0.000000]  [<ffffffff817313e8>] xen_start_kernel+0x508/0x50f
> > [    0.000000] Code:  Bad RIP value.
> > [    0.000000] RIP  [<(null)>] (null)
> > [    0.000000]  RSP <ffffffff81601d70>
> > [    0.000000] CR2: 0000000000000000
> > [    0.000000] ---[ end trace 4eaa2a86a8e2da22 ]---
> > [    0.000000] Kernel panic - not syncing: Attempted to kill the idle
> task!
> > [    0.000000] Pid: 0, comm: swapper Tainted: G      D     2.6.35.5
> #1
> > [    0.000000] Call Trace:
> > [    0.000000]  [<ffffffff8133bae9>] panic+0x86/0xfa
> > [    0.000000]  [<ffffffff8103c5c6>] do_exit+0x6d/0x77e
> > [    0.000000]  [<ffffffff8133d43f>] ?
> _raw_spin_unlock_irqrestore+0x11/0x13
> > [    0.000000]  [<ffffffff8103ac55>] ? kmsg_dump+0x11e/0x139
> > [    0.000000]  [<ffffffff8100b936>] oops_end+0x8f/0x94
> > [    0.000000]  [<ffffffff810239cc>] no_context+0x1f4/0x203
> > [    0.000000]  [<ffffffff81023b65>]
> __bad_area_nosemaphore+0x18a/0x1ad
> > [    0.000000]  [<ffffffff81023b96>] bad_area_nosemaphore+0xe/0x10
> > [    0.000000]  [<ffffffff81023f10>] do_page_fault+0x115/0x229
> > [    0.000000]  [<ffffffff8133daf5>] page_fault+0x25/0x30
> > [    0.000000]  [<ffffffff8107c849>] ? default_enable+0x1a/0x28
> > [    0.000000]  [<ffffffff8107c829>] default_startup+0x19/0x1f
> > [    0.000000]  [<ffffffff8133d3fb>] ?
> _raw_spin_lock_irqsave+0xd/0x24
> > [    0.000000]  [<ffffffff8107c033>] __setup_irq+0x1ab/0x2d8
> > [    0.000000]  [<ffffffff8100572f>] ?
> xen_restore_fl_direct_end+0x0/0x1
> > [    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
> > [    0.000000]  [<ffffffff8107c278>] request_threaded_irq+0x118/0x146
> > [    0.000000]  [<ffffffff8125dda5>]
> bind_virq_to_irqhandler+0x146/0x168
> > [    0.000000]  [<ffffffff810052bb>] ? xen_timer_interrupt+0x0/0x17a
> > [    0.000000]  [<ffffffff810055ee>] xen_setup_timer+0x59/0x9d
> > [    0.000000]  [<ffffffff81731b68>] xen_time_init+0x7b/0x89
> > [    0.000000]  [<ffffffff81731ead>] x86_late_time_init+0xa/0x11
> > [    0.000000]  [<ffffffff8172eb49>] start_kernel+0x30b/0x38d
> > [    0.000000]  [<ffffffff8172e2c6>]
> x86_64_start_reservations+0xb1/0xb5
> > [    0.000000]  [<ffffffff817313e8>] xen_start_kernel+0x508/0x50f
> >
> > Additional info:
> >   arch: x86_64
> >   cpu:  Intel(R) Xeon(R) CPU           E5345  @ 2.33GHz
> >   xen:  3.2.1
> >   dom0: Linux version 2.6.26-2-xen-amd64 (Debian 2.6.26-24lenny1)
> > 		(dannf@debian.org) (gcc version 4.1.3 20080704 (prerelease)
> > 		(Debian 4.1.2-25)) #1 SMP Thu Aug 19 01:12:45 UTC 2010
> >
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Paweł Zuzelski]

On Thu, 23 Sep 2010, Jeremy Fitzhardinge wrote:

>  On 09/21/2010 12:05 PM, Paweł Zuzelski wrote:
> > Hello,
> >
> > kernels 2.6.35.5 and 2.6.32.22 xen domU panics at the very begining of the
> > boot process.
> >
> > I have bisected it to a single commit, and the first bad commit is:
> > [fb412a178502dc498430723b082a932f797e4763] xen: use percpu interrupts for IPIs and VIRQs
> >
> > kernel v2.6.35.5 with reverted this commit works for me.
> 
> Thanks very much for doing that.  I'll have to work out what's going on
> (obviously it doesn't do it for me).

Have you seen the message from James Dingwall?

He had found the typo in fb412a178502dc498430723b082a932f797e4763
commit and sent the patch to lkml that fixes it.

Please, see https://patchwork.kernel.org/patch/202282/ if you have
not received James' mail. That patch resolves the problem for me and
for James.

-- 
Regards,
Paweł Zuzelski

Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Jeremy Fitzhardinge]

 On 09/23/2010 08:23 AM, James Dingwall wrote:
>> I was experiencing the same immediate crash with a null pointer
>> dereference (log below) on boot with 2.6.35.5.  Reverting
>> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
>> me.
> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.

Yes, that would do it.  Upstream is OK, so it looks like this crept in
when being cherry-picked into stable-2.6.32/master and stable-2.6.35/master.

    J

> --- drivers/xen/events.c.orig   2010-09-23 16:10:34.257124313 +0100
> +++ drivers/xen/events.c        2010-09-23 16:10:44.326194462 +0100
> @@ -935,7 +935,7 @@
>  	.retrigger		= retrigger_dynirq,
>  };
>
> -static struct irq_chip en_percpu_chip __read_mostly = {
> +static struct irq_chip xen_percpu_chip __read_mostly = {
>  	.name			= "xen-percpu",
>
>  	.disable		= disable_dynirq,
>
> James
> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
> you may review at http://www.amdocs.com/email_disclaimer.asp
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Re: PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Kyle McMartin]

On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
> > I was experiencing the same immediate crash with a null pointer
> > dereference (log below) on boot with 2.6.35.5.  Reverting
> > fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> > me.
> 
> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
> 

Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
James here points out. Adding you to CC since it seems to have missed
stable 2.6.35.6...

Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.

--Kyle

> --- drivers/xen/events.c.orig   2010-09-23 16:10:34.257124313 +0100
> +++ drivers/xen/events.c        2010-09-23 16:10:44.326194462 +0100
> @@ -935,7 +935,7 @@
>  	.retrigger		= retrigger_dynirq,
>  };
> 
> -static struct irq_chip en_percpu_chip __read_mostly = {
> +static struct irq_chip xen_percpu_chip __read_mostly = {
>  	.name			= "xen-percpu",
> 
>  	.disable		= disable_dynirq,
> 
> James
> This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
> you may review at http://www.amdocs.com/email_disclaimer.asp
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Greg KH]

On Sun, Sep 26, 2010 at 10:40:44PM -0400, Kyle McMartin wrote:
> On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
> > > I was experiencing the same immediate crash with a null pointer
> > > dereference (log below) on boot with 2.6.35.5.  Reverting
> > > fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> > > me.
> > 
> > Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
> > 
> 
> Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
> James here points out. Adding you to CC since it seems to have missed
> stable 2.6.35.6...
> 
> Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.

I don't understand, where did the error come from?  Was it in a patch I
applied, or did I mess up some patch?  Is the same problem upstream, and
if not, why not?

And finally, can you send this in a format that I can apply it?

thanks,

greg k-h

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Kyle McMartin]

On Sun, Sep 26, 2010 at 11:13:36PM -0700, Greg KH wrote:
> > Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
> > James here points out. Adding you to CC since it seems to have missed
> > stable 2.6.35.6...
> > 
> > Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
> 
> I don't understand, where did the error come from?  Was it in a patch I
> applied, or did I mess up some patch?  Is the same problem upstream, and
> if not, why not?

It's correct upstream, and wrong in the patches you added to
stable-queue.git for .32 and .35, no idea how it got introduced. :/

> 
> And finally, can you send this in a format that I can apply it?
> 

I'm just the messenger... James, can you send a patch to Greg with your
signed-off-by and such?

--Kyle

> thanks,
> 
> greg k-h

RE: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot [PATCH]

[James Dingwall]

> On Sun, Sep 26, 2010 at 11:13:36PM -0700, Greg KH wrote:
> > > Greg, a spelling mistake cropped into this patch during 2.6.35.5
> that
> > > James here points out. Adding you to CC since it seems to have
> missed
> > > stable 2.6.35.6...
> > >
> > > Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
> >
> > I don't understand, where did the error come from?  Was it in a patch
> I
> > applied, or did I mess up some patch?  Is the same problem upstream,
> and
> > if not, why not?
> 
> It's correct upstream, and wrong in the patches you added to
> stable-queue.git for .32 and .35, no idea how it got introduced. :/
> 
> >
> > And finally, can you send this in a format that I can apply it?
> >
> 
> I'm just the messenger... James, can you send a patch to Greg with your
> signed-off-by and such?
> 
> --Kyle
> 
> > thanks,
> >
> > greg k-h

Hi,

My first contribution so I hope I didn't miss any important steps in formatting the patch and that Outlook keeps it intact...  If it needs any fixing up let me know.

James

Correctly name the irq_chip structure to fix an immediate failure when booting as a xen pv_ops guest with a NULL pointer exception. The missing 'x' was introduced in commit [fb412a178502dc498430723b082a932f797e4763] applied to 2.6.3[25]-stable trees.  The commit to mainline was [aaca49642b92c8a57d3ca5029a5a94019c7af69f] which did not have the problem.

Signed-off-by: James Dingwall <james@dingwall.me.uk>
Reported-by: Paweł Zuzelski <pawelz@pld-linux.org>
Tested-by: Paweł Zuzelski <pawelz@pld-linux.org>

--- drivers/xen/events.c.orig	2010-09-27 09:10:09.245652829 +0100
+++ drivers/xen/events.c	2010-09-27 09:10:27.522304658 +0100
@@ -935,7 +935,7 @@ static struct irq_chip xen_dynamic_chip 
 	.retrigger	= retrigger_dynirq,
 };
 
-static struct irq_chip en_percpu_chip __read_mostly = {
+static struct irq_chip xen_percpu_chip __read_mostly = {
 	.name		= "xen-percpu",
 
 	.disable	= disable_dynirq,

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Jeremy Fitzhardinge]

 On 09/26/2010 11:13 PM, Greg KH wrote:
> On Sun, Sep 26, 2010 at 10:40:44PM -0400, Kyle McMartin wrote:
>> On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
>>>> I was experiencing the same immediate crash with a null pointer
>>>> dereference (log below) on boot with 2.6.35.5.  Reverting
>>>> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
>>>> me.
>>> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
>>>
>> Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
>> James here points out. Adding you to CC since it seems to have missed
>> stable 2.6.35.6...
>>
>> Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
> I don't understand, where did the error come from?  Was it in a patch I
> applied, or did I mess up some patch?  Is the same problem upstream, and
> if not, why not?

I don't know.  It looks like an 'x' was deleted due to an editor
fat-finger or something.  The original upstream patches were cc:d to
stable, so I think you would have got them that way (ie, I don't think I
sent them to you specifically).  Is there an opportunity in your
workflow where this error could have crept in?  The patches should have
been a pretty clean 'git cherry-pick' with no conflicts.

Thanks,
    J

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Greg KH]

On Mon, Sep 27, 2010 at 11:08:43AM -0700, Jeremy Fitzhardinge wrote:
>  On 09/26/2010 11:13 PM, Greg KH wrote:
> > On Sun, Sep 26, 2010 at 10:40:44PM -0400, Kyle McMartin wrote:
> >> On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
> >>>> I was experiencing the same immediate crash with a null pointer
> >>>> dereference (log below) on boot with 2.6.35.5.  Reverting
> >>>> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> >>>> me.
> >>> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
> >>>
> >> Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
> >> James here points out. Adding you to CC since it seems to have missed
> >> stable 2.6.35.6...
> >>
> >> Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
> > I don't understand, where did the error come from?  Was it in a patch I
> > applied, or did I mess up some patch?  Is the same problem upstream, and
> > if not, why not?
> 
> I don't know.  It looks like an 'x' was deleted due to an editor
> fat-finger or something.  The original upstream patches were cc:d to
> stable, so I think you would have got them that way (ie, I don't think I
> sent them to you specifically).  Is there an opportunity in your
> workflow where this error could have crept in?  The patches should have
> been a pretty clean 'git cherry-pick' with no conflicts.

Sorry, I don't know what happened, using my scripts, it does drop to an
editor so I can add the "signed-off-by" line, so perhaps I accidentally
did this in there.

It's now in the -stable queue and I'll push out new releases in a few
hours.

thanks,

greg k-h

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot [PATCH]

[Greg KH]

On Mon, Sep 27, 2010 at 09:37:17AM +0100, James Dingwall wrote:
> My first contribution so I hope I didn't miss any important steps in
> formatting the patch and that Outlook keeps it intact...  If it needs
> any fixing up let me know.

Looks good, I've now queued it up, thanks so much for this.

greg k-h

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Jeremy Fitzhardinge]

 On 09/28/2010 11:29 AM, Greg KH wrote:
> On Mon, Sep 27, 2010 at 11:08:43AM -0700, Jeremy Fitzhardinge wrote:
>>  On 09/26/2010 11:13 PM, Greg KH wrote:
>>> On Sun, Sep 26, 2010 at 10:40:44PM -0400, Kyle McMartin wrote:
>>>> On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
>>>>>> I was experiencing the same immediate crash with a null pointer
>>>>>> dereference (log below) on boot with 2.6.35.5.  Reverting
>>>>>> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
>>>>>> me.
>>>>> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
>>>>>
>>>> Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
>>>> James here points out. Adding you to CC since it seems to have missed
>>>> stable 2.6.35.6...
>>>>
>>>> Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
>>> I don't understand, where did the error come from?  Was it in a patch I
>>> applied, or did I mess up some patch?  Is the same problem upstream, and
>>> if not, why not?
>> I don't know.  It looks like an 'x' was deleted due to an editor
>> fat-finger or something.  The original upstream patches were cc:d to
>> stable, so I think you would have got them that way (ie, I don't think I
>> sent them to you specifically).  Is there an opportunity in your
>> workflow where this error could have crept in?  The patches should have
>> been a pretty clean 'git cherry-pick' with no conflicts.
> Sorry, I don't know what happened, using my scripts, it does drop to an
> editor so I can add the "signed-off-by" line, so perhaps I accidentally
> did this in there.
>
> It's now in the -stable queue and I'll push out new releases in a few
> hours.
>

I should have pointed out it is the same in the stable-2.6.32 tree as well.

    J

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Greg KH]

On Wed, Sep 29, 2010 at 08:48:07AM -0700, Jeremy Fitzhardinge wrote:
>  On 09/28/2010 11:29 AM, Greg KH wrote:
> > On Mon, Sep 27, 2010 at 11:08:43AM -0700, Jeremy Fitzhardinge wrote:
> >>  On 09/26/2010 11:13 PM, Greg KH wrote:
> >>> On Sun, Sep 26, 2010 at 10:40:44PM -0400, Kyle McMartin wrote:
> >>>> On Thu, Sep 23, 2010 at 04:23:55PM +0100, James Dingwall wrote:
> >>>>>> I was experiencing the same immediate crash with a null pointer
> >>>>>> dereference (log below) on boot with 2.6.35.5.  Reverting
> >>>>>> fb412a178502dc498430723b082a932f797e4763 also resolved the problem for
> >>>>>> me.
> >>>>> Checking the diff from this commit it looks like there is a typo in the patch.  With the following (probably mangled by my client) I get 2.6.35.5 to boot.  Suggest it should be a candidate for .32 and .35 stable trees.
> >>>>>
> >>>> Greg, a spelling mistake cropped into this patch during 2.6.35.5 that
> >>>> James here points out. Adding you to CC since it seems to have missed
> >>>> stable 2.6.35.6...
> >>>>
> >>>> Was in xen-use-percpu-interrupts-for-ipis-and-virqs.patch.
> >>> I don't understand, where did the error come from?  Was it in a patch I
> >>> applied, or did I mess up some patch?  Is the same problem upstream, and
> >>> if not, why not?
> >> I don't know.  It looks like an 'x' was deleted due to an editor
> >> fat-finger or something.  The original upstream patches were cc:d to
> >> stable, so I think you would have got them that way (ie, I don't think I
> >> sent them to you specifically).  Is there an opportunity in your
> >> workflow where this error could have crept in?  The patches should have
> >> been a pretty clean 'git cherry-pick' with no conflicts.
> > Sorry, I don't know what happened, using my scripts, it does drop to an
> > editor so I can add the "signed-off-by" line, so perhaps I accidentally
> > did this in there.
> >
> > It's now in the -stable queue and I'll push out new releases in a few
> > hours.
> >
> 
> I should have pointed out it is the same in the stable-2.6.32 tree as well.

Yes, sorry, I know, I only had the chance to get the .35 release out
yesterday, I'll do the .32 one now.

thanks,

greg k-h

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Jeremy Fitzhardinge]

 On 09/29/2010 05:06 PM, Greg KH wrote:
> On Wed, Sep 29, 2010 at 08:48:07AM -0700, Jeremy Fitzhardinge wrote:
>>
>> I should have pointed out it is the same in the stable-2.6.32 tree as well.
> Yes, sorry, I know, I only had the chance to get the .35 release out
> yesterday, I'll do the .32 one now.
>

Ping?  Looks like the fix hasn't hit stable-2.6.32/master yet.

Thanks,
    J

Re: [stable] PROBLEM: [BISECTED] 2.6.35.5 xen domU panics just after the boot

[Greg KH]

On Fri, Oct 01, 2010 at 09:46:56AM -0700, Jeremy Fitzhardinge wrote:
>  On 09/29/2010 05:06 PM, Greg KH wrote:
> > On Wed, Sep 29, 2010 at 08:48:07AM -0700, Jeremy Fitzhardinge wrote:
> >>
> >> I should have pointed out it is the same in the stable-2.6.32 tree as well.
> > Yes, sorry, I know, I only had the chance to get the .35 release out
> > yesterday, I'll do the .32 one now.
> >
> 
> Ping?  Looks like the fix hasn't hit stable-2.6.32/master yet.

No it hasn't, sorry.  Turns out the Embedded Linux kernel "summit" was
too interesting to get a stable kernel out during it, and I've been in a
train/plane/train/ferry/taxi/bed since then.

Should now be out in a few hours...

thanks,

greg k-h