From: Fengguang Wu <fengguang.wu@intel.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: xen-devel@lists.xenproject.org,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Date: Mon, 7 Oct 2013 10:11:18 +0800 [thread overview]
Message-ID: <20131007021118.GA27927@localhost> (raw)
In-Reply-To: <CA+55aFxk2FoG7+BpHJAysR2bz-WQXpje6buMd8Sp6-SBqHexZg@mail.gmail.com>
On Sun, Oct 06, 2013 at 10:26:24AM -0700, Linus Torvalds wrote:
> On Sun, Oct 6, 2013 at 1:23 AM, Fengguang Wu <fengguang.wu@intel.com> wrote:
> >
> > I got the below dmesg and the first bad commit is commit cf39c8e5352b:
> > Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
>
> Ugh. How reliable is the double fault? Because bisecting it to the
> merge that didn't even have any conflicts in it as far as I can
> remember means that there's something really subtle going on wrt some
> semantic conflict or other. Or, alternatively, it means that the
> bisect failed because the double fault isn't 100% reliable..
Oops, it's not a reliable bisect...
The "first" bad commit cf39c8e5352b4fb9efedfe7e9acb566a85ed847c runs
and produces 25 good dmesgs and 3530 bad dmesgs, however only 1 of the
bad boots has "double fault:" in its dmesg.
Looking into all the 3530 bad dmesgs, I find all kinds of bug messages:
$ grep_crash_head -h dmesg-* | sed 's/^[^a-zA-Z]*//' | sort | uniq -c | sort -nr
3086 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
3047 BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
3046 Kernel panic - not syncing: Fatal exception in interrupt
2969 BUG: kernel boot oops
374 BUG: kernel test oops
255 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
167 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011!
167 invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
148 Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
48 INFO: lockdep is turned off.
43 BUG: unable to handle kernel
33 BUG: kernel boot crashed
30 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20
27 general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
17 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
13 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
13 BUG: unable to handle kernel NULL pointer dereference at (null)
12 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
11 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
11 Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
9 INFO: trying to register non-static key.
9 BUG: scheduling while atomic: init/136/0x10000002
8 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
8 BUG: unable to handle kernel NULL pointer dereference
6 Oops: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
5 BUG: unable to handle kernel paging request at ffffffffffffffa8
5 BUG: Bad page map in process init pte:ffffffffffffffff pmd:06d9e067
5 BUG: Bad page map in process init pte:ffffffff pmd:06d9e067
4 Oops: 0002 [#1]
4 Kernel panic - not syncing: Attempted to kill the idle task!
4 BUG: unable to handle kernel paging request at ffff88000cd94000
3 invalid opcode: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 WARNING: CPU: 0 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 BUG: kernel boot hang
3 BUG: Bad page map in process init pte:ffffffff81f0fa00 pmd:06d9e067
3 BUG: Bad page map in process init pte:ffffffff81b52e93 pmd:06d9e067
3 BUG: Bad page map in process init pte:dead4ead00000000 pmd:06d9e067
2 kernel BUG at /c/wfg/linux-drm/include/linux/mm.h:286!
2 general protection fault: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
2 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 Oops: 0002 [#1] PREEMPT
2 Oops: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
2 BUG: unable to handle kernel paging request at 0000000000010012
2 BUG: unable to handle kernel NULL pointer dereference at 00000000000008e0
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000017
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
2 BUG: scheduling while atomic: rc.local/134/0x10000002
2 BUG: scheduling while atomic: S99-rc.local/210/0x10000002
2 BUG: Bad page map in process init pte:ffffffff81b6e50a pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e290 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e208 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e180 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e0f8 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e080 pmd:06d9e067
2 BUG: Bad page map in process init pte:ffff880006d9e070 pmd:06d9e067
2 BUG: Bad page map in process init pte:00000001 pmd:06d9e067
1 tail/181 is trying to release lock (&sig->cred_guard_mutex) at:
1 kernel BUG at /c/wfg/linux-drm/mm/slab.c:463!
1 general protection fault: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 double fault: 0000 [#9] PREEMPT SMP DEBUG_PAGEALLOC
1 WARNING: CPU: 1 PID: 95 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 222 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/include/linux/kref.h:47 kref_get+0x2d/0x36()
1 WARNING: CPU: 1 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_io.c:1411 tty_open+0x215/0x42b()
1 WARNING: CPU: 1 PID: 129 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 117 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 114 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 8 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 129 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 125 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 Oops: 0010 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] e PorR dEirEecMtoPryT
1 Oops: 0002 [#1] dPonRe.E
1 Oops: 0002 [#1] PREEMPT SMP puDlaEteB-vUolGat_ilPe.AGEALLOC
1 Oops: 0002 [#1] PREEMPT SMP
1 Oops: 0002 [#1] PREEMPT /bSinM/sPh: /procD/sEBUG_PAGEALLOC
1 Oops: 0000 [#4] PREEMPT SMP DEBUG_PAGEALLOC
1 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
1 INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.349 msecs
1 BUG: unable to handle kernel paging request at ffff88000cd6c800
1 BUG: unable to handle kernel paging request at 00002939a1ff0088
1 BUG: unable to handle kernel paging request at 0000200000000000
1 BUG: unable to handle kernel paging request at 000000010000003f
1 BUG: unable to handle kernel paging request at 000000007fffffff
1 BUG: unable to handle kernel paging request at 0000000003ffffc0
1 BUG: unable to handle kernel ULL pointer dereference at 0000000000000008
1 BUG: unable to handle kernel ULL pointer dereference
1 BUG: unable to handle kernel PleaNse wULait: bLoo pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULL pointerereference at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000027
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000014
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
1 BUG: unable to handle kernel NULL poinr deference at 0000000000000008
1 BUG: unable to 9: No suchandle kernel NULL pointer dereference at 0000000000000008
1 BUG: spinlock lockup suspected on CPU#1, migration/1/11
1 BUG: spinlock lockup suspected on CPU#1, kworker/1:1/20
1 BUG: spinlock lockup suspected on CPU#0, sh/126
1 BUG: spinlock lockup suspected on CPU#0, sh/115
1 BUG: scheduling while atomic: sh/130/0x10000002
1 BUG: scheduling while atomic: reboot/222/0x10000002
1 BUG: scheduling while atomic: rc/212/0x10000002
1 BUG: scheduling while atomic: init/212/0x10000002
1 BUG: scheduling while atomic: init/138/0x10000002
1 BUG: Bad rss-counter state mm:ffff88000ca206c0 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9d4080 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff880006a106c0 idx:1 val:2
1 BUG: Bad page map in process init pte:ffffffff82863660 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632c0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632b8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632b0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff828632a0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82863280 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8275cf00 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82314d30 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff82310d10 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff823102f0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8230e7f0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8ab2a pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aaf9 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aabc pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff81b8aaa9 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffffffff8191eba0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff8800076d53c0 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e1c8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e140 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d9e0b8 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff880006d90000 pmd:06d9e067
1 BUG: Bad page map in process init pte:ffff88000692ec40 pmd:06d9e067
1 BUG: Bad page map in process init pte:dead4ead003c003c pmd:06d9e067
1 BUG: Bad page map in process init pte:dead4ead002c002c pmd:06d9e067
1 BUG: Bad page map in process init pte:200005401 pmd:06d9e067
It's not really the "first" bad commit because its parent commit 3398d252a4d
("Merge tag 'modules-next-for-linus' of git://git.kernel.org/pub/scm/linux/kerne l/git/rusty/linux")
has a similar 495 good dmesgs and 14587 bad dmesgs with error messages:
13847 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
13792 BUG: unable to handle kernel NULL pointer dereference at 0000000000000008
13760 Kernel panic - not syncing: Fatal exception in interrupt
10011 BUG: kernel boot oops
4280 BUG: kernel test oops
382 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
316 kernel BUG at /c/wfg/linux-drm/mm/slab.c:3011!
301 invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
225 Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b
178 INFO: lockdep is turned off.
84 BUG: sleeping function called from invalid context at /c/wfg/linux-drm/kernel/rwsem.c:20
65 BUG: unable to handle kernel
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
52 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
42 WARNING: CPU: 0 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
40 general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
38 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
29 BUG: kernel boot crashed
27 WARNING: CPU: 1 PID: 0 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
22 BUG: scheduling while atomic: init/136/0x10000002
21 BUG: scheduling while atomic: S99-rc.local/210/0x10000002
17 invalid opcode: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
15 Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
14 Oops: 0010 [#1] PREEMPT SMP DEBUG_PAGEALLOC
13 Oops: 0000 [#2] PREEMPT SMP DEBUG_PAGEALLOC
13 BUG: unable to handle kernel NULL pointer dereference at (null)
13 BUG: unable to handle kernel NULL pointer dereference
10 BUG: unable to handle kernel paging request at ffffffffffffffa8
8 Oops: 0002 [#1]
8 BUG: scheduling while atomic: init/214/0x10000002
8 BUG: scheduling while atomic: init/212/0x10000002
7 BUG: scheduling while atomic: init/213/0x10000002
6 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
6 BUG: unable to handle kernel paging request at 0000000000010012
6 BUG: unable to handle kernel NULL pointer dereference at 0000000000000001
6 BUG: scheduling while atomic: rc.local/134/0x10000002
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
5 WARNING: CPU: 1 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
4 BUG: unable to handle kernel NULL pointer dereference at 00000000000008e0
4 BUG: scheduling while atomic: ifconfig/230/0x10000002
4 BUG: kernel boot hang
3 WARNING: CPU: 1 PID: 117 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 WARNING: CPU: 1 PID: 116 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
3 WARNING: CPU: 0 PID: 1 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
3 INFO: trying to register non-static key.
3 BUG: unable to handle kernel paging request at ffff88000cd9f000
3 BUG: unable to handle kernel N
3 BUG: scheduling while atomic: rc/212/0x10000002
2 kernel BUG at /c/wfg/linux-drm/mm/slab.c:463!
2 WARNING: CPU: 1 PID: 121 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 1 PID: 118 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 22 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
2 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
2 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
2 WARNING: CPU: 0 PID: 124 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 WARNING: CPU: 0 PID: 123 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
2 Oops: 0002 [#2] PREEMPT SMP DEBUG_PAGEALLOC
2 Oops: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
2 BUG: unable to handle kernel ULL pointer dereference at 0000000000000008
2 BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
2 BUG: scheduling while atomic: reboot/210/0x10000002
2 BUG: scheduling while atomic: rc/136/0x10000002
2 BUG: scheduling while atomic: S20sendsigs/141/0x10000002
1 t[ri ni ty: n ot f1o.668013] Kernel panic - not syncing: Fatal exception in interrupt
1 sh: /[pr oc /s 4.771591] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 proc o[n /proc fa 2.048043] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 n/modp[ro be : 2.087974] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 kernel BUG at /c/wfg/linux-drm/kernel/cred.c:758!
1 invalid opcode: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 gr[ep : /p ro c/2fi.le087655] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCsy
1 general protection fault: 0000 [#3] PREEMPT SMP DEBUG_PAGEALLOC
1 et c/rc5 .d1.667815] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC/S99-rc.loca
1 bi[n/ sh : / pr oc4/s.179326] Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 96 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 221 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 1 PID: 22 at /c/wfg/linux-drm/kernel/lockdep.c:960 __bfs+0xf1/0x1b3()
1 WARNING: CPU: 1 PID: 138 at /c/wfg/linux-drm/drivers/tty/tty_mutex.c:23 tty_lock_nested+0x34/0x83()
1 WARNING: CPU: 1 PID: 122 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 120 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 1 PID: 12 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 1 PID: 119 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 115 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 1 PID: 113 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 221 at /c/wfg/linux-drm/include/linux/kref.h:47 kobject_get+0x37/0x44()
1 WARNING: CPU: 0 PID: 200 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 20 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 199 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 196 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 191 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 187 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 168 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 162 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 147 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 144 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 132 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/include/linux/kref.h:47 kref_get+0x2d/0x36()
1 WARNING: CPU: 0 PID: 130 at /c/wfg/linux-drm/drivers/tty/tty_io.c:1411 tty_open+0x215/0x42b()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/lib/debugobjects.c:260 debug_print_object+0x7c/0x8b()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/kernel/workqueue.c:458 work_fixup_activate+0x6a/0x6f()
1 WARNING: CPU: 0 PID: 128 at /c/wfg/linux-drm/kernel/workqueue.c:1379 __queue_work+0x1a1/0x1ee()
1 WARNING: CPU: 0 PID: 126 at /c/wfg/linux-drm/kernel/workqueue.c:591 set_work_data+0x33/0x50()
1 Oops: 0010 [#2] PREEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] iled: NPRo suchE deEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] grPRep: /Eproc/fileEMPT SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] bPooRtlEogEd.M
1 Oops: 0002 [#1] PRe EorE dMirPecTto ry
1 Oops: 0002 [#1] PREmount:E moMPTunt ing SMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT nSe M15P: /triniDtyE:BUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT SmoMunPt: mountDinEg BUG_PAGEALLOC
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCs/
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCm
1 Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOCgrep: /proc/file
1 Oops: 0002 [#1] PREEMPT SMP
1 Oops: 0002 [#1] PREEMPT
1 Oops: 0002 [#1] CPonRfiEguEriMngP nTet wSMP DEBUG_PAGEALLOC
1 Oops: 0002 [#1] /PbiRn/EshE: M/pProTc/ sSMP eDlfE/fBd/U9:G N_o PsuAcGEALLOC
1 Oops: 0000 [#4] PREEMPT SMP DEBUG_PAGEALLOC
1 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000009
1 BUG: unable to handle kernel paging request at ffffffffe7e9eb38
1 BUG: unable to handle kernel paging request at ffff880006db9808
1 BUG: unable to handle kernel paging request at ffff880006db9800
1 BUG: unable to handle kernel paging request at ffff880006db3008
1 BUG: unable to handle kernel paging request at ffff880006db3000
1 BUG: unable to handle kernel paging request at 00002939a1ff0088
1 BUG: unable to handle kernel paging request at 0000200000000080
1 BUG: unable to handle kernel paging request at 00000000c1dcacfe
1 BUG: unable to handle kernel paging request at 000000007fffffff
1 BUG: unable to handle kernel paging request at 0000000003ffffc0
1 BUG: unable to handle kernel gNULL pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULproc on /proc Lfa pointer dereference at 0000000000000008
1 BUG: unable to handle kernel NULgrep: /proc/cmdline: No such file or directory
1 BUG: unable to handle kernel NULRuLnni ngp poositinnstter /detec/rrpem-fpoestrinensctse/100.. .at 00
1 BUG: unable to handle kernel NULL pointer ereference
1 BUG: unable to handle kernel NULL pointer derefrence at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereferenceStarting Bootlog at 0000000000000008
1 BUG: unable to handle kernel NULL pointer dereference atmo 00unt: 0moun00ti0ng0 00pro0c on /proc fa00ile000d: No such de8
1 BUG: unable to handle kernel NULL pointer dereference at moun0000t: 0moun0ting0 000000p0roc 0on /8proc
1 BUG: unable to handle kernel NULL pointer dereference at gr0e0p:0 0/p0ro00000c0/cm0dl0008in
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000017
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
1 BUG: unable to handle kernel NULL pointer dereference at 0000000000000004
1 BUG: unable to handle kernel NULL pointer dereference a/bt in0000/sh: 00/p0ro0c/0000s000elf/fd/9:8
1 BUG: unable to handle kernel NULL pointer dereferce at 0000000000000008
1 BUG: unable to handle kernel NULL pointer derefence at 0000000000000008
1 BUG: unable to handle kernel NULL pointe dereference at 0000000000000008
1 BUG: unable to handle kernel NULL poiner dereference at 0000000000000008
1 BUG: unable to handle kernel NULL p
1 BUG: unable to handle kernel NUL/etc/rcS.d/S00fbLsetup : pointer lidne 3:e reference at 0000000000000008
1 BUG: unable to handle kernel NUL/biLn/s ph: /poiroc/nsterel f/fd/9: dNo sucereh file feror edirectnorcy
1 BUG: unable to handle kernel NU/bin/LLsh: / procp/soelf/fdi/9: No sucnteh frile or dire ctdoryeref
1 BUG: unable to handle kernel NU
1 BUG: unable to handle kernel /kNerUneLl-Lte stpointer dereference at 0000000000000008
1 BUG: unable to handle kernel -> 0:9
1 BUG: spinlock lockup suspected on CPU#0, swapper/0/0
1 BUG: scheduling while atomic: stty/188/0x10000002
1 BUG: scheduling while atomic: sleep/130/0x00000002
1 BUG: scheduling while atomic: sleep/129/0x10000002
1 BUG: scheduling while atomic: sleep/129/0x00000002
1 BUG: scheduling while atomic: reboot/221/0x10000002
1 BUG: scheduling while atomic: reboot/134/0x10000002
1 BUG: scheduling while atomic: rc/214/0x10000002
1 BUG: scheduling while atomic: rc/213/0x10000002
1 BUG: scheduling while atomic: rc.local/135/0x10000002
1 BUG: Bad rss-counter state mm:ffff88000cab4d00 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000caaf340 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca33340 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca1d980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000ca1b980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9e1980 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9de080 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9d06c0 idx:1 val:2
1 BUG: Bad rss-counter state mm:ffff88000c9be6c0 idx:1 val:2
Sorry I should have did more analyzes before reporting it out. The Xen tree is
clean: 10000 good boots w/o a single bad dmesg. Which is also indicated by the
below bisect log:
git bisect bad cf39c8e5352b # 18:34 727- Merge tag 'stable/for-linus-3.12-rc0-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
...
git bisect good 23b7eaf82207 # 03:19 5000+ hvc_xen: Remove unnecessary __GFP_ZERO from kzalloc
git bisect good 3398d252a4da # 13:35 10000+ Merge tag 'modules-next-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/rusty/linux
git bisect good 23b7eaf82207 # 14:55 10000+ hvc_xen: Remove unnecessary __GFP_ZERO from kzalloc
git bisect bad 14951f22f1cc # 14:55 0- Revert "perf symbols: Demangle cloned functions"
git bisect bad e62063d69911 # 15:28 1444- Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs
git bisect bad a0cf1abc25ac # 15:28 0- Add linux-next specific files for 20130927
git bisect bad b31801966611 # 15:56 728- Merge branch 'tools/kvm'
Where the 5000+/10000+ etc. are the number of boots before claiming the kernel GOOD;
the 727-/0-/728- are the number of boots (minus 1) before finding the bug message in dmesg.
In this bisect, the bug message is "double fault:" and the bisect works in this way:
loop {
if grep "double fault:" dmesg; then
report FAILURE
else if booted N times
report GOOD and break
}
I'll check and improve the bisect script to verify the bisect result
for working with this situation.
Thanks,
Fengguang
next prev parent reply other threads:[~2013-10-07 2:11 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-10-06 8:23 [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC Fengguang Wu
2013-10-06 17:26 ` Linus Torvalds
2013-10-07 2:11 ` Fengguang Wu [this message]
2013-10-07 5:10 ` Fengguang Wu
2013-10-07 8:12 ` Linus Torvalds
2013-10-07 8:12 ` Linus Torvalds
2013-10-07 8:35 ` Fengguang Wu
2013-10-07 22:14 ` Linus Torvalds
2013-10-07 22:29 ` Russell King - ARM Linux
2013-10-07 22:29 ` Russell King - ARM Linux
2013-10-07 23:33 ` Russell King - ARM Linux
2013-10-07 23:33 ` Russell King - ARM Linux
2013-10-07 23:47 ` Linus Torvalds
2013-10-07 23:47 ` Linus Torvalds
2013-10-08 2:09 ` Fengguang Wu
2013-10-08 2:09 ` Fengguang Wu
2013-10-08 2:14 ` Fengguang Wu
2013-10-08 2:36 ` Fengguang Wu
2013-10-08 3:07 ` Linus Torvalds
2013-10-08 3:07 ` Linus Torvalds
2013-10-08 2:36 ` Fengguang Wu
2013-10-08 2:51 ` Linus Torvalds
2013-10-08 2:51 ` Linus Torvalds
2013-10-08 3:11 ` Fengguang Wu
2013-10-08 3:29 ` Linus Torvalds
2013-10-08 3:29 ` Linus Torvalds
2013-10-08 3:35 ` Fengguang Wu
2013-10-08 4:35 ` [DRM_CURRUS_QEMU/timer] WARNING: CPU: 0 PID: 1 at debug_print_object() Fengguang Wu
2013-10-08 4:35 ` Fengguang Wu
2013-10-08 3:35 ` [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC Fengguang Wu
2013-10-08 12:17 ` Fengguang Wu
2013-10-08 22:14 ` Russell King - ARM Linux
2013-10-08 22:48 ` Greg Kroah-Hartman
2013-10-09 0:45 ` Linus Torvalds
2013-10-09 1:18 ` Dave Jones
2013-10-09 1:26 ` [PATCH] kobject: show debug info on delayed kobject release Fengguang Wu
2013-10-09 14:47 ` Russell King - ARM Linux
2013-10-09 14:47 ` Russell King - ARM Linux
2013-10-09 1:26 ` Fengguang Wu
2013-10-09 1:18 ` [xen] double fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC Dave Jones
2013-10-09 14:12 ` Josh Boyer
2013-10-09 14:12 ` Josh Boyer
2013-10-09 0:45 ` Linus Torvalds
2013-10-09 0:46 ` Fengguang Wu
2013-10-09 0:46 ` Fengguang Wu
2013-10-08 22:48 ` Greg Kroah-Hartman
2013-10-08 22:14 ` Russell King - ARM Linux
2013-10-08 12:17 ` Fengguang Wu
2013-10-08 3:11 ` Fengguang Wu
2013-10-08 8:06 ` Russell King - ARM Linux
2013-10-08 8:06 ` Russell King - ARM Linux
2013-10-10 2:23 ` Dave Airlie
2013-10-10 2:38 ` Linus Torvalds
2013-10-10 2:38 ` Linus Torvalds
2013-10-10 9:19 ` Russell King - ARM Linux
2013-10-10 10:53 ` Russell King - ARM Linux
2013-10-11 3:55 ` Dave Airlie
2013-10-11 3:55 ` Dave Airlie
2013-10-11 4:28 ` Dave Airlie
2013-10-11 4:28 ` Dave Airlie
2013-10-11 5:14 ` Fengguang Wu
2013-10-23 11:20 ` Xiong Zhou
2013-10-23 11:20 ` Xiong Zhou
2013-10-11 5:14 ` Fengguang Wu
2013-10-10 10:53 ` Russell King - ARM Linux
2013-10-10 9:19 ` Russell King - ARM Linux
2013-10-10 2:23 ` Dave Airlie
2013-10-08 2:14 ` Fengguang Wu
2013-10-08 2:43 ` Linus Torvalds
2013-10-08 2:43 ` Linus Torvalds
2013-10-08 7:58 ` Ingo Molnar
2013-10-08 8:20 ` Fengguang Wu
2013-10-08 8:20 ` Fengguang Wu
2013-10-08 8:26 ` Ingo Molnar
2013-10-08 9:34 ` Ingo Molnar
2013-10-08 9:34 ` Ingo Molnar
2013-10-08 8:26 ` Ingo Molnar
2013-10-08 7:58 ` Ingo Molnar
2013-10-07 22:14 ` Linus Torvalds
2013-10-07 5:10 ` Fengguang Wu
2013-10-07 2:11 ` Fengguang Wu
2013-10-06 17:26 ` Linus Torvalds
-- strict thread matches above, loose matches on Subject: below --
2013-10-06 22:14 Boris Ostrovsky
2013-10-06 22:14 Boris Ostrovsky
2013-10-06 8:23 Fengguang Wu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131007021118.GA27927@localhost \
--to=fengguang.wu@intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.