Re: 2.6.24-rc1-gb4f5550 oops

Re: 2.6.24-rc1-gb4f5550 oops

[Rafael J. Wysocki]

On Thursday, 8 of November 2007, Grant Wilson wrote:
> On Thu, 8 Nov 2007 01:06:21 +0100
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > Hi,
> > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > 
> > (1) Is this reproducible?
> > (2) Did it happen previously on your system?
> >
> > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> 
> This has now happened twice - the second time was last night when
> running 2.6.24-rc2.
> 
> Here's that second occurrence:
> 
> [ 7224.233621] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP:
> [ 7224.233631]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> [ 7224.233644] PGD 0
> [ 7224.233650] Oops: 0000 [1] PREEMPT SMP
> [ 7224.233660] CPU 0
> [ 7224.233716] Modules linked in:
> [ 7224.233722] Pid: 7622, comm: ssh Not tainted 2.6.24-rc2 #1
> [ 7224.233726] RIP: 0010:[<ffffffff8023572e>]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> [ 7224.233735] RSP: 0018:ffff81000aafba78  EFLAGS: 00010006
> [ 7224.233738] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000005745
> [ 7224.233743] RDX: ffff81000442fbf0 RSI: ffff810006c96860 RDI: ffff810004438b80
> [ 7224.233748] RBP: ffff81000aafbaa8 R08: 0000007e8e25cf71 R09: 0000000000000000
> [ 7224.233752] R10: ffff81000442fbf0 R11: 0000000000000001 R12: ffff810007479d80
> [ 7224.233756] R13: ffff810006c96860 R14: ffff810009924860 R15: ffff81000442b8e0
> [ 7224.233760] FS:  00002b8bf089d350(0000) GS:ffffffff808d7000(0000) knlGS:0000000000000000
> [ 7224.233764] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 7224.233768] CR2: 0000000000000120 CR3: 000000000ab3f000 CR4: 00000000000006e0
> [ 7224.233771] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 7224.233775] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 7224.233779] Process ssh (pid: 7622, threadinfo ffff81000aafa000, task ffff81000abc2000)
> [ 7224.233782] Stack:  ffff810004438b80 0000000000000001 ffff810006c96860 ffff810004438b80
> [ 7224.233796]  0000000000000000 ffff81000442b8e0 ffff81000aafbb38 ffffffff8023061e
> [ 7224.233807]  0000000000000400 ffff81000442fb80 0000000000000000 0000000100000000
> [ 7224.233816] Call Trace:
> [ 7224.233823]  [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
> [ 7224.233828]  [<ffffffff802306cd>] default_wake_function+0xd/0x10
> [ 7224.233833]  [<ffffffff8022daca>] __wake_up_common+0x5a/0x90
> [ 7224.233839]  [<ffffffff8023095a>] __wake_up_sync+0x4a/0x70
> [ 7224.233845]  [<ffffffff80602fbf>] unix_write_space+0x8f/0xa0
> [ 7224.233850]  [<ffffffff805936f9>] sock_wfree+0x49/0x50
> [ 7224.233854]  [<ffffffff80595579>] __kfree_skb+0x69/0xe0
> [ 7224.233859]  [<ffffffff80595607>] kfree_skb+0x17/0x30
> [ 7224.233863]  [<ffffffff806016c7>] unix_stream_recvmsg+0x267/0x610
> [ 7224.233869]  [<ffffffff8058e457>] sock_aio_read+0x107/0x110
> [ 7224.233875]  [<ffffffff802928f1>] do_sync_read+0xf1/0x130
> [ 7224.233882]  [<ffffffff802a4f20>] __d_free+0x30/0x40
> [ 7224.233887]  [<ffffffff80251830>] autoremove_wake_function+0x0/0x40
> [ 7224.233892]  [<ffffffff80293246>] vfs_read+0x156/0x160
> [ 7224.233897]  [<ffffffff80293650>] sys_read+0x50/0x90
> [ 7224.233901]  [<ffffffff8020bd7e>] system_call+0x7e/0x83
> [ 7224.233904]
> [ 7224.233907]
> [ 7224.233907] Code: 48 8b 90 20 01 00 00 48 39 93 20 01 00 00 75 e2 48 81 3b 00
> [ 7224.233951] RIP  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> [ 7224.233957]  RSP <ffff81000aafba78>
> [ 7224.233961] CR2: 0000000000000120
> [ 7224.233967] note: ssh[7622] exited with preempt_count 3

Hmm.

Please run "gdb vmlinux" and see what code corresponds to
check_preempt_wakeup+0x6e in your kernel.

Re: 2.6.24-rc1-gb4f5550 oops

[Grant Wilson]

On Thu, 8 Nov 2007 16:53:10 +0100
"Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Thursday, 8 of November 2007, Grant Wilson wrote:
> > On Thu, 8 Nov 2007 01:06:21 +0100
> > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > 
> > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > Hi,
> > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > 
> > > (1) Is this reproducible?
> > > (2) Did it happen previously on your system?
> > >
> > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > 
> > This has now happened twice - the second time was last night when
> > running 2.6.24-rc2.
> > 
> > Here's that second occurrence:
> > 
[snip]
> 
> Hmm.
> 
> Please run "gdb vmlinux" and see what code corresponds to
> check_preempt_wakeup+0x6e in your kernel.

Dump of assembler code for function check_preempt_wakeup:
0xffffffff80232940 <check_preempt_wakeup+0>:    push   %rbp
0xffffffff80232941 <check_preempt_wakeup+1>:    mov    %rsp,%rbp
0xffffffff80232944 <check_preempt_wakeup+4>:    sub    $0x30,%rsp
0xffffffff80232948 <check_preempt_wakeup+8>:    mov    %r13,-0x18(%rbp)
0xffffffff8023294c <check_preempt_wakeup+12>:   mov    %rbx,-0x28(%rbp)
0xffffffff80232950 <check_preempt_wakeup+16>:   mov    %rsi,%r13
0xffffffff80232953 <check_preempt_wakeup+19>:   mov    %r12,-0x20(%rbp)
0xffffffff80232957 <check_preempt_wakeup+23>:   mov    %r14,-0x10(%rbp)
0xffffffff8023295b <check_preempt_wakeup+27>:   mov    %r15,-0x8(%rbp)
0xffffffff8023295f <check_preempt_wakeup+31>:   cmpl   $0x63,0x20(%rsi)
0xffffffff80232963 <check_preempt_wakeup+35>:   mov    0x750(%rdi),%r14
0xffffffff8023296a <check_preempt_wakeup+42>:   mov    0x168(%r14),%r12
0xffffffff80232971 <check_preempt_wakeup+49>:   jle    0xffffffff80232a1c <check_preempt_wakeup+220>
0xffffffff80232977 <check_preempt_wakeup+55>:   cmpl   $0x3,0x17c(%rsi)
0xffffffff8023297e <check_preempt_wakeup+62>:   je     0xffffffff802329f8 <check_preempt_wakeup+184>
0xffffffff80232980 <check_preempt_wakeup+64>:   testb  $0x10,0x593cb9(%rip)        # 0xffffffff807c6640 <sysctl_sched_features>
0xffffffff80232987 <check_preempt_wakeup+71>:   je     0xffffffff802329f8 <check_preempt_wakeup+184>
0xffffffff80232989 <check_preempt_wakeup+73>:   cmp    0x168(%rsi),%r12
0xffffffff80232990 <check_preempt_wakeup+80>:   lea    0x48(%r14),%rbx
0xffffffff80232994 <check_preempt_wakeup+84>:   lea    0x48(%rsi),%rax
0xffffffff80232998 <check_preempt_wakeup+88>:   je     0xffffffff802329be <check_preempt_wakeup+126>
0xffffffff8023299a <check_preempt_wakeup+90>:   nopw   0x0(%rax,%rax,1)
0xffffffff802329a0 <check_preempt_wakeup+96>:   mov    0x118(%rbx),%rbx
0xffffffff802329a7 <check_preempt_wakeup+103>:  mov    0x118(%rax),%rax
0xffffffff802329ae <check_preempt_wakeup+110>:  mov    0x120(%rax),%rdx
0xffffffff802329b5 <check_preempt_wakeup+117>:  cmp    %rdx,0x120(%rbx)
0xffffffff802329bc <check_preempt_wakeup+124>:  jne    0xffffffff802329a0 <check_preempt_wakeup+96>
0xffffffff802329be <check_preempt_wakeup+126>:  cmpq   $0x400,(%rbx)
0xffffffff802329c5 <check_preempt_wakeup+133>:  mov    0x40(%rbx),%r12
0xffffffff802329c9 <check_preempt_wakeup+137>:  mov    0x40(%rax),%r15
0xffffffff802329cd <check_preempt_wakeup+141>:  mov    0x593c81(%rip),%edi        # 0xffffffff807c6654 <sysctl_sched_wakeup_granularity>
0xffffffff802329d3 <check_preempt_wakeup+147>:  jne    0xffffffff80232a37 <check_preempt_wakeup+247>
0xffffffff802329d5 <check_preempt_wakeup+149>:  sub    %r15,%r12
0xffffffff802329d8 <check_preempt_wakeup+152>:  cmp    %r12,%rdi
0xffffffff802329db <check_preempt_wakeup+155>:  jge    0xffffffff802329f8 <check_preempt_wakeup+184>
0xffffffff802329dd <check_preempt_wakeup+157>:  testb  $0x20,0x593c5c(%rip)        # 0xffffffff807c6640 <sysctl_sched_features>

Cheers,
Grant
-- 
Running Linux 2.6.24-rc2 on x86_64

Re: 2.6.24-rc1-gb4f5550 oops

[Rafael J. Wysocki]

On Thursday, 8 of November 2007, Grant Wilson wrote:
> On Thu, 8 Nov 2007 16:53:10 +0100
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > On Thu, 8 Nov 2007 01:06:21 +0100
> > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > 
> > > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > > Hi,
> > > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > > 
> > > > (1) Is this reproducible?
> > > > (2) Did it happen previously on your system?
> > > >
> > > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > > 
> > > This has now happened twice - the second time was last night when
> > > running 2.6.24-rc2.
> > > 
> > > Here's that second occurrence:
> > > 
> [snip]
> > 
> > Hmm.
> > 
> > Please run "gdb vmlinux" and see what code corresponds to
> > check_preempt_wakeup+0x6e in your kernel.
>
> 
> Dump of assembler code for function check_preempt_wakeup:

Well, thanks, but I meant the source code.  Please do "gdb vmlinux" and then
"l *check_preempt_wakeup+0x6e" in gdb.

Thanks,
Rafael

Re: 2.6.24-rc1-gb4f5550 oops

[Grant Wilson]

On Thu, 8 Nov 2007 22:42:21 +0100
"Rafael J. Wysocki" <rjw@sisk.pl> wrote:

> On Thursday, 8 of November 2007, Grant Wilson wrote:
> > On Thu, 8 Nov 2007 16:53:10 +0100
> > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > 
> > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > On Thu, 8 Nov 2007 01:06:21 +0100
> > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > 
> > > > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > > > Hi,
> > > > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > > > 
> > > > > (1) Is this reproducible?
> > > > > (2) Did it happen previously on your system?
> > > > >
> > > > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > > > 
> > > > This has now happened twice - the second time was last night when
> > > > running 2.6.24-rc2.
> > > > 
> > > > Here's that second occurrence:
> > > > 
> > [snip]
> > > 
> > > Hmm.
> > > 
> > > Please run "gdb vmlinux" and see what code corresponds to
> > > check_preempt_wakeup+0x6e in your kernel.
> >
> > 
> > Dump of assembler code for function check_preempt_wakeup:
> 
> Well, thanks, but I meant the source code.  Please do "gdb vmlinux" and then
> "l *check_preempt_wakeup+0x6e" in gdb.

Here's the requested output:

(gdb) l *check_preempt_wakeup+0x6e
0xffffffff802329ae is in check_preempt_wakeup (kernel/sched_fair.c:668).
663
664     /* Do the two (enqueued) entities belong to the same group ? */
665     static inline int
666     is_same_group(struct sched_entity *se, struct sched_entity *pse)
667     {
668             if (se->cfs_rq == pse->cfs_rq)
669                     return 1;
670
671             return 0;
672     }

Grant

Re: 2.6.24-rc1-gb4f5550 oops

[Rafael J. Wysocki]

On Thursday, 8 of November 2007, Grant Wilson wrote:
> On Thu, 8 Nov 2007 22:42:21 +0100
> "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> 
> > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > On Thu, 8 Nov 2007 16:53:10 +0100
> > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > 
> > > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > > On Thu, 8 Nov 2007 01:06:21 +0100
> > > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > > 
> > > > > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > > > > Hi,
> > > > > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > > > > 
> > > > > > (1) Is this reproducible?
> > > > > > (2) Did it happen previously on your system?
> > > > > >
> > > > > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > > > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > > > > 
> > > > > This has now happened twice - the second time was last night when
> > > > > running 2.6.24-rc2.
> > > > > 
> > > > > Here's that second occurrence:
> > > > > 
> > > [snip]
> > > > 
> > > > Hmm.
> > > > 
> > > > Please run "gdb vmlinux" and see what code corresponds to
> > > > check_preempt_wakeup+0x6e in your kernel.
> > >
> > > 
> > > Dump of assembler code for function check_preempt_wakeup:
> > 
> > Well, thanks, but I meant the source code.  Please do "gdb vmlinux" and then
> > "l *check_preempt_wakeup+0x6e" in gdb.
> 
> Here's the requested output:
> 
> (gdb) l *check_preempt_wakeup+0x6e
> 0xffffffff802329ae is in check_preempt_wakeup (kernel/sched_fair.c:668).
> 663
> 664     /* Do the two (enqueued) entities belong to the same group ? */
> 665     static inline int
> 666     is_same_group(struct sched_entity *se, struct sched_entity *pse)
> 667     {
> 668             if (se->cfs_rq == pse->cfs_rq)
> 669                     return 1;
> 670
> 671             return 0;
> 672     }

Well, it looks like either se or pse is NULL.

Ingo, can you please have a look?

Thanks,
Rafael

Re: 2.6.24-rc1-gb4f5550 oops

[Peter Zijlstra]

On Thu, 2007-11-08 at 23:49 +0100, Rafael J. Wysocki wrote:
> On Thursday, 8 of November 2007, Grant Wilson wrote:
> > On Thu, 8 Nov 2007 22:42:21 +0100
> > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > 
> > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > On Thu, 8 Nov 2007 16:53:10 +0100
> > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > 
> > > > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > > > On Thu, 8 Nov 2007 01:06:21 +0100
> > > > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > > > 
> > > > > > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > > > > > Hi,
> > > > > > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > > > > > 
> > > > > > > (1) Is this reproducible?
> > > > > > > (2) Did it happen previously on your system?
> > > > > > >
> > > > > > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > > > > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > > > > > 
> > > > > > This has now happened twice - the second time was last night when
> > > > > > running 2.6.24-rc2.
> > > > > > 
> > > > > > Here's that second occurrence:
> > > > > > 
> > > > [snip]
> > > > > 
> > > > > Hmm.
> > > > > 
> > > > > Please run "gdb vmlinux" and see what code corresponds to
> > > > > check_preempt_wakeup+0x6e in your kernel.
> > > >
> > > > 
> > > > Dump of assembler code for function check_preempt_wakeup:
> > > 
> > > Well, thanks, but I meant the source code.  Please do "gdb vmlinux" and then
> > > "l *check_preempt_wakeup+0x6e" in gdb.
> > 
> > Here's the requested output:
> > 
> > (gdb) l *check_preempt_wakeup+0x6e
> > 0xffffffff802329ae is in check_preempt_wakeup (kernel/sched_fair.c:668).
> > 663
> > 664     /* Do the two (enqueued) entities belong to the same group ? */
> > 665     static inline int
> > 666     is_same_group(struct sched_entity *se, struct sched_entity *pse)
> > 667     {
> > 668             if (se->cfs_rq == pse->cfs_rq)
> > 669                     return 1;
> > 670
> > 671             return 0;
> > 672     }
> 
> Well, it looks like either se or pse is NULL.
> 
> Ingo, can you please have a look?

Most puzzling this, it should be guaranteed that the top sched_entities
are of the same group, therefore avoiding this loop into NULL. Obviously
something has gone wrong.

Grant, is there anything specific you can tell us about how to reproduce
this?

Re: 2.6.24-rc1-gb4f5550 oops

[Grant Wilson]

On Mon, 12 Nov 2007 19:05:49 +0100
Peter Zijlstra <peterz@infradead.org> wrote:

> 
> On Thu, 2007-11-08 at 23:49 +0100, Rafael J. Wysocki wrote:
> > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > On Thu, 8 Nov 2007 22:42:21 +0100
> > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > 
> > > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > > On Thu, 8 Nov 2007 16:53:10 +0100
> > > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > > 
> > > > > > On Thursday, 8 of November 2007, Grant Wilson wrote:
> > > > > > > On Thu, 8 Nov 2007 01:06:21 +0100
> > > > > > > "Rafael J. Wysocki" <rjw@sisk.pl> wrote:
> > > > > > > 
> > > > > > > > On Monday, 5 of November 2007, Grant Wilson wrote:
> > > > > > > > > Hi,
> > > > > > > > > I got this oops on 2.6.24-rc1-641-gb4f5550:
> > > > > > > > 
> > > > > > > > (1) Is this reproducible?
> > > > > > > > (2) Did it happen previously on your system?
> > > > > > > >
> > > > > > > > [18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: 
> > > > > > > > [18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
> > > > > > > 
> > > > > > > This has now happened twice - the second time was last night when
> > > > > > > running 2.6.24-rc2.
> > > > > > > 
> > > > > > > Here's that second occurrence:
> > > > > > > 
> > > > > [snip]
> > > > > > 
> > > > > > Hmm.
> > > > > > 
> > > > > > Please run "gdb vmlinux" and see what code corresponds to
> > > > > > check_preempt_wakeup+0x6e in your kernel.
> > > > >
> > > > > 
> > > > > Dump of assembler code for function check_preempt_wakeup:
> > > > 
> > > > Well, thanks, but I meant the source code.  Please do "gdb vmlinux" and then
> > > > "l *check_preempt_wakeup+0x6e" in gdb.
> > > 
> > > Here's the requested output:
> > > 
> > > (gdb) l *check_preempt_wakeup+0x6e
> > > 0xffffffff802329ae is in check_preempt_wakeup (kernel/sched_fair.c:668).
> > > 663
> > > 664     /* Do the two (enqueued) entities belong to the same group ? */
> > > 665     static inline int
> > > 666     is_same_group(struct sched_entity *se, struct sched_entity *pse)
> > > 667     {
> > > 668             if (se->cfs_rq == pse->cfs_rq)
> > > 669                     return 1;
> > > 670
> > > 671             return 0;
> > > 672     }
> > 
> > Well, it looks like either se or pse is NULL.
> > 
> > Ingo, can you please have a look?
> 
> Most puzzling this, it should be guaranteed that the top sched_entities
> are of the same group, therefore avoiding this loop into NULL. Obviously
> something has gone wrong.
> 
> Grant, is there anything specific you can tell us about how to reproduce
> this?

I'm afraid not.  It has only happened twice and both times I was away
from the box in question at the time of failure, so it wasn't doing a
great deal.

I'm running 2.6.24-rc2 on two boxes and both times it happened on the
box running a quad core processor.

Grant