x86_64 DOM0 SMP oops

x86_64 DOM0 SMP oops

[Ryan Harper]

While working/testing the hotplug patch for x86_64, I encountered an
[1]oops.  When the system was idle or minor load, I didn't have any
problems, but as I kicked off an parallel build of the kernel, I hit the
oops.  

changeset:   6315:531ad4bde8f249c318588f41bdbb72d0b0e0ea9c
tag:         tip
parent:      6314:473af43ac65b1a872b058469d01acc6cfa21505d
parent:      6313:d8fd24b4308078f6d20aa8a306e540c5833ca5a2
user:        cl349@firebug.cl.cam.ac.uk
date:        Mon Aug 22 05:21:18 2005
summary:     merge?

1. 
Unable to handle kernel NULL pointer dereference at 000000000000001c RIP:
<ffffffff80112c1a>{retint_kernel+0}
PGD aae067 PUD aaf067 PMD 0
Oops: 0000 [1] PREEMPT SMP
CPU 0
Modules linked in: container video thermal processor fan button battery ac amd64_agp agpgart
Pid: 0, comm: swapper Not tainted 2.6.12-xen0-smp
RIP: e030:[<ffffffff80112c1a>] <ffffffff80112c1a>{retint_kernel+0}
RSP: e02b:ffffffff805e5f08  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000000ffffffff RCX: 0000000000000000
RDX: ffffffffff577000 RSI: ffff8800059bff50 RDI: ffffffff804be220
RBP: ffffffff804bcf80 R08: 00000000ffffffff R09: ffffffff805e5bb8
R10: 0000000000000002 R11: 0000000000000212 R12: ffffffff80100038
R13: 00000000ffffffff R14: 0000ffffffff8010 R15: ffffffff805e5fb0
FS:  00002aaaaaac1250(0000) GS:ffffffff805cfe00(0000) knlGS:ffffffff805cfe00
CS:  e033 DS: 0000 ES: 0000
Process swapper (pid: 0, threadinfo ffffffff805e4000, task ffffffff804be220)
Stack: 0000000000000286 0000000000000008 000000010003b14e 000000010003b14e
       0000000000000000 ffffffff801102a1 0000000000000000 000000010003b150
       0000000000000001 0000000000000011
Call Trace:<ffffffff801102a1>{xen_idle+193} <ffffffff801102a1>{xen_idle+193}
       <ffffffff801102ef>{cpu_idle+63} <ffffffff806068ae>{start_kernel+542}
       <ffffffff806061b1>{x86_64_start_kernel+337}
Code: 83 79 1c 00 0f 85 58 fe ff ff 0f ba 61 10 03 0f 83 4d fe ff
RIP <ffffffff80112c1a>{retint_kernel+0} RSP <ffffffff805e5f08>
CR2: 000000000000001c
 <0>Kernel panic - not syncing: Attempted to kill the idle task!
 stop_this_cpu disable_local_APIC
stop_this_cpu disable_local_APIC
<0>general protection fault: 0000 [2] PREEMPT SMP
CPU 1
Modules linked in: container video thermal processor fan button battery ac amd64_agp agpgart
Pid: 0, comm: swapper Not tainted 2.6.12-xen0-smp
RIP: e030:[<ffffffff801174e0>] <ffffffff801174e0>{smp_really_stop_cpu+16}
RSP: e02b:ffff880000a53db8  EFLAGS: 00010292
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffff8011745d
RDX: ffffffffff577000 RSI: 0000000000000021 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000008 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000246 R12: ffff880000a53e98
R13: 0000000000000107 R14: 0000000000000000 R15: ffff880000a53e98
FS:  00002aaaaaac1250(0000) GS:ffffffff805cfe80(0000) knlGS:ffffffff805cfe80
CS:  e033 DS: 002b ES: 002b
Process swapper (pid: 0, threadinfo ffff880000a52000, task ffff880000a4cdd0)
Stack: ffffffff80653f98 ffffffff8011762e ffff880000a5df00 ffffffff8015621c
       0000000000008380 0000000000000107 ffffffff805d9a00 ffffffff805d9a28
       ffff880000a5df00 ffffffff80156348
Call Trace:<ffffffff8011762e>{smp_call_function_interrupt+62} <ffffffff8015621c>{handle_IRQ_event+140}
       <ffffffff80156348>{__do_IRQ+200} <ffffffff80116484>{do_IRQ+52}
       <ffffffff8010e46a>{evtchn_do_upcall+170} <ffffffff80112f71>{do_hypervisor_callback+17}
       <ffffffff801102a1>{xen_idle+193} <ffffffff801102a1>{xen_idle+193}
       <ffffffff801102ef>{cpu_idle+63}

Code: f4 eb fd 66 66 66 90 66 66 90 66 66 90 66 66 90 53 48 c7 c7
RIP <ffffffff801174e0>{smp_really_stop_cpu+16} RSP <ffff880000a53db8>
 <0>Kernel panic - not syncing: Aiee, killing interrupt handler!
 stop_this_cpu disable_local_APIC
Badness in xen_send_IPI_mask at arch/xen/x86_64/kernel/genapic_xen.c:116

Call Trace:<ffffffff801187bd>{xen_send_IPI_mask+141} <ffffffff8012e430>{recalc_task_prio+320}
       <ffffffff801172dd>{smp_send_reschedule+29} <ffffffff8012eaa7>{try_to_wake_up+759}
       <ffffffff8014c679>{autoremove_wake_function+9} <ffffffff801309a3>{__wake_up_common+67}
       <ffffffff80130a13>{__wake_up+67} <ffffffff80186b9b>{pipe_writev+1211}
       <ffffffff8014c670>{autoremove_wake_function+0} <ffffffff80186c0a>{pipe_write+26}
       <ffffffff80179a50>{vfs_write+192} <ffffffff80179ba3>{sys_write+83}
       <ffffffff8011254a>{system_call+134} <ffffffff801124c4>{system_call+0}

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: x86_64 DOM0 SMP oops

[Chris Wright]

* Ryan Harper (ryanh@us.ibm.com) wrote:
> While working/testing the hotplug patch for x86_64, I encountered an
> [1]oops.  When the system was idle or minor load, I didn't have any
> problems, but as I kicked off an parallel build of the kernel, I hit the
> oops.  
> 
> changeset:   6315:531ad4bde8f249c318588f41bdbb72d0b0e0ea9c
> tag:         tip
> parent:      6314:473af43ac65b1a872b058469d01acc6cfa21505d
> parent:      6313:d8fd24b4308078f6d20aa8a306e540c5833ca5a2
> user:        cl349@firebug.cl.cam.ac.uk
> date:        Mon Aug 22 05:21:18 2005
> summary:     merge?
> 
> 1. 
> Unable to handle kernel NULL pointer dereference at 000000000000001c RIP:
> <ffffffff80112c1a>{retint_kernel+0}
> PGD aae067 PUD aaf067 PMD 0
> Oops: 0000 [1] PREEMPT SMP

Not surprising, w/ PREEMPT.  Is it stable w/out PREEMPT?

thanks,
-chris

Re: x86_64 DOM0 SMP oops

[Ryan Harper]

* Chris Wright <chrisw@osdl.org> [2005-08-22 14:13]:
> * Ryan Harper (ryanh@us.ibm.com) wrote:
> > While working/testing the hotplug patch for x86_64, I encountered an
> > [1]oops.  When the system was idle or minor load, I didn't have any
> > problems, but as I kicked off an parallel build of the kernel, I hit the
> > oops.  
> > 
> > changeset:   6315:531ad4bde8f249c318588f41bdbb72d0b0e0ea9c
> > tag:         tip
> > parent:      6314:473af43ac65b1a872b058469d01acc6cfa21505d
> > parent:      6313:d8fd24b4308078f6d20aa8a306e540c5833ca5a2
> > user:        cl349@firebug.cl.cam.ac.uk
> > date:        Mon Aug 22 05:21:18 2005
> > summary:     merge?
> > 
> > 1. 
> > Unable to handle kernel NULL pointer dereference at 000000000000001c RIP:
> > <ffffffff80112c1a>{retint_kernel+0}
> > PGD aae067 PUD aaf067 PMD 0
> > Oops: 0000 [1] PREEMPT SMP
> 
> Not surprising, w/ PREEMPT.  Is it stable w/out PREEMPT?

Really?  I thought CONFIG_PREEMPT was tested since it was on for x86_32
SMP.  I'll give that a shot and let you know.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com

Re: x86_64 DOM0 SMP oops

[Chris Wright]

* Ryan Harper (ryanh@us.ibm.com) wrote:
> > > Unable to handle kernel NULL pointer dereference at 000000000000001c RIP:
> > > <ffffffff80112c1a>{retint_kernel+0}
> > > PGD aae067 PUD aaf067 PMD 0
> > > Oops: 0000 [1] PREEMPT SMP
> > 
> > Not surprising, w/ PREEMPT.  Is it stable w/out PREEMPT?
> 
> Really?  I thought CONFIG_PREEMPT was tested since it was on for x86_32
> SMP.

No, I expect it to be broken (and you seem to be able to easily prove
that ;-)

> I'll give that a shot and let you know.

Great, thanks,
-chris

Re: x86_64 DOM0 SMP oops

[Ryan Harper]

* Chris Wright <chrisw@osdl.org> [2005-08-22 14:26]:
> * Ryan Harper (ryanh@us.ibm.com) wrote:
> > > > Unable to handle kernel NULL pointer dereference at 000000000000001c RIP:
> > > > <ffffffff80112c1a>{retint_kernel+0}
> > > > PGD aae067 PUD aaf067 PMD 0
> > > > Oops: 0000 [1] PREEMPT SMP
> > > 
> > > Not surprising, w/ PREEMPT.  Is it stable w/out PREEMPT?
> > 
> > Really?  I thought CONFIG_PREEMPT was tested since it was on for x86_32
> > SMP.
> 
> No, I expect it to be broken (and you seem to be able to easily prove
> that ;-)
> 
> > I'll give that a shot and let you know.

That seems to have done it.  Built several kernels back-to-back, no
oops.

Now if I could just get this to go away:

Timer ISR/1: Time went backwards: delta=-169458624 cpu_delta=1110558976
shadow=57067465250 off=133093726 processed=57370017600
cpu_processed=56090000000
 0: 57370017600
  1: 56090000000

Anyone else with a dual-opteron see this?  It shows up on everything
(x86_64 UP, x86_32 UP/SMP).

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com