On Tue, Nov 12, 2013 at 04:29:56PM +0100, Peter Zijlstra wrote: > On Tue, Nov 12, 2013 at 10:15:41AM -0500, Vivek Goyal wrote: > > I see that we allocate per cpu stats but don't do any initializations. > > > > static void tg_stats_alloc_fn(struct work_struct *work) > > { > > static struct tg_stats_cpu *stats_cpu; /* this fn is non-reentrant */ > > struct delayed_work *dwork = to_delayed_work(work); > > bool empty = false; > > > > alloc_stats: > > if (!stats_cpu) { > > stats_cpu = alloc_percpu(struct tg_stats_cpu); > > if (!stats_cpu) { > > /* allocation failed, try again after some time */ > > schedule_delayed_work(dwork, msecs_to_jiffies(10)); > > return; > > } > > } > > > > spin_lock_irq(&tg_stats_alloc_lock); > > Absolutely! > > Something like this perhaps? Did I miss more blkg_[rw]stats? If I read > the git grep output right, this was the last one. It changed the error into another one: /kernel/i386-randconfig-j7-11082318/9d0d532888f6e77970016c0c270ddececfab8f9e +---------------------------------------------------------+-------+--------------+--------------+--------------+ | | v3.12 | 838cc7b488f8 | d3516a7318f8 | 9d0d532888f6 | +---------------------------------------------------------+-------+--------------+--------------+--------------+ | boot_successes | 129 | 0 | 0 | 0 | | boot_failures | 1 | 100 | 100 | 100 | | BUG:kernel_early_hang_without_any_printk_output | 1 | | | | | INFO:trying_to_register_non-static_key | 0 | 100 | 100 | | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 0 | 0 | 0 | 98 | | Oops:SMP | 0 | 0 | 0 | 100 | | BUG:unable_to_handle_kernel_paging_request_at | 0 | 0 | 0 | 100 | | BUG:unable_to_handle_kernel | 0 | 0 | 0 | 2 | +---------------------------------------------------------+-------+--------------+--------------+--------------+ Here are 3 of the call traces. Attached 1 full dmesg. dmesg-quantal-ant-3:20131113183320:i386-randconfig-j7-11082318:3.12.0-00187-g9d0d532:1 [ 52.578884] SCSI Media Changer driver v0.25 [ 52.781957] scsi_debug: host protection [ 52.783001] scsi0 : scsi_debug, version 1.82 [20100324], dev_size_mb=8, opts=0x0 [ 52.846181] BUG: unable to handle kernel NULL pointer dereference at 00000008 [ 52.846293] IP: [<78287084>] lockdep_init_map+0xd/0x544 [ 52.846293] *pde = 00000000 [ 52.846293] Oops: 0002 [#1] SMP [ 52.846293] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 3.12.0-00187-g9d0d532 #1 [ 52.846293] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 52.846293] Workqueue: events tg_stats_alloc_fn [ 52.846293] task: 854f0070 ti: 854f6000 task.ti: 854f6000 [ 52.846293] EIP: 0060:[<78287084>] EFLAGS: 00000296 CPU: 0 [ 52.846293] EIP is at lockdep_init_map+0xd/0x544 [ 52.846293] EAX: 00000004 EBX: 79d614b0 ECX: 79d614b0 EDX: 793fba74 [ 52.846293] ESI: 00000004 EDI: 79d614b8 EBP: 854f7e90 ESP: 854f7e78 [ 52.846293] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 52.846293] CR0: 8005003b CR2: 00000008 CR3: 018a4000 CR4: 000006d0 [ 52.846293] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 52.846293] DR6: 00000000 DR7: 00000000 [ 52.846293] Stack: [ 52.846293] 0000014a 0000014b 854bd740 00000000 00000000 79d614b8 854f7eac 785fae1d [ 52.846293] 00000000 795aea60 854bd740 00000000 00000000 854f7ef8 7826bae8 00000000 [ 52.846293] 00000001 00000000 7826ba8f 854a6a00 85bf0700 795aea60 85bef380 795aea60 [ 52.846293] Call Trace: [ 52.846293] [<785fae1d>] tg_stats_alloc_fn+0xe3/0x155 [ 52.846293] [<7826bae8>] process_one_work+0x1ef/0x349 [ 52.846293] [<7826ba8f>] ? process_one_work+0x196/0x349 [ 52.846293] [<7826c039>] worker_thread+0x1a3/0x274 [ 52.846293] [<7826be96>] ? rescuer_thread+0x233/0x233 [ 52.846293] [<78270649>] kthread+0x8a/0x8f [ 52.846293] [<78270000>] ? SyS_timer_gettime+0x43/0x9e [ 52.846293] [<78fbac7b>] ret_from_kernel_thread+0x1b/0x30 [ 52.846293] [<782705bf>] ? __kthread_parkme+0x50/0x50 [ 52.846293] Code: f0 8b 7d e8 64 ff 0d 68 d7 74 79 89 70 10 89 78 14 e9 9b fe ff ff 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 89 c6 53 89 cb 83 ec 0c 40 04 00 00 00 00 c7 40 08 00 00 00 00 64 a1 10 d0 74 79 89 [ 52.846293] EIP: [<78287084>] lockdep_init_map+0xd/0x544 SS:ESP 0068:854f7e78 [ 52.846293] CR2: 0000000000000008 [ 52.870492] ---[ end trace cee670421f9807b7 ]--- [ 52.871070] kworker/0:0 (4) used greatest stack depth: 6464 bytes left dmesg-quantal-ant-3:20131113183450:i386-randconfig-j7-11082318:3.12.0-00187-g9d0d532:1 [ 60.495607] scsi_debug: host protection [ 60.496526] scsi0 : scsi_debug, version 1.82 [20100324], dev_size_mb=8, opts=0x0 [ 60.549921] scsi 0:0:0:0: Direct-Access Linux scsi_debug 0004 PQ: 0 ANSI: 5 [ 60.574257] BUG: unable to handle kernel NULL pointer dereference at 00000008 [ 60.574284] IP: [<78287084>] lockdep_init_map+0xd/0x544 [ 60.574284] *pde = 00000000 [ 60.574284] Oops: 0002 [#1] SMP [ 60.574284] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 3.12.0-00187-g9d0d532 #1 [ 60.574284] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 60.574284] Workqueue: events tg_stats_alloc_fn [ 60.574284] task: 854f0070 ti: 854f6000 task.ti: 854f6000 [ 60.574284] EIP: 0060:[<78287084>] EFLAGS: 00000296 CPU: 0 [ 60.574284] EIP is at lockdep_init_map+0xd/0x544 [ 60.574284] EAX: 00000004 EBX: 79d614b0 ECX: 79d614b0 EDX: 793fba74 [ 60.574284] ESI: 00000004 EDI: 79d614b8 EBP: 854f7e90 ESP: 854f7e78 [ 60.574284] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 60.574284] CR0: 8005003b CR2: 00000008 CR3: 018a4000 CR4: 000006d0 [ 60.574284] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 60.574284] DR6: 00000000 DR7: 00000000 [ 60.574284] Stack: [ 60.574284] 0000014a 0000014b 854bd740 00000000 00000000 79d614b8 854f7eac 785fae1d [ 60.574284] 00000000 795aea60 854bd740 00000000 00000000 854f7ef8 7826bae8 00000000 [ 60.574284] 00000001 00000000 7826ba8f 854a6a00 85bf0700 795aea60 85bef380 795aea60 [ 60.574284] Call Trace: [ 60.574284] [<785fae1d>] tg_stats_alloc_fn+0xe3/0x155 [ 60.574284] [<7826bae8>] process_one_work+0x1ef/0x349 [ 60.574284] [<7826ba8f>] ? process_one_work+0x196/0x349 [ 60.574284] [<7826c039>] worker_thread+0x1a3/0x274 [ 60.574284] [<7826be96>] ? rescuer_thread+0x233/0x233 [ 60.574284] [<78270649>] kthread+0x8a/0x8f [ 60.574284] [<78270000>] ? SyS_timer_gettime+0x43/0x9e [ 60.574284] [<78fbac7b>] ret_from_kernel_thread+0x1b/0x30 [ 60.574284] [<782705bf>] ? __kthread_parkme+0x50/0x50 [ 60.574284] Code: f0 8b 7d e8 64 ff 0d 68 d7 74 79 89 70 10 89 78 14 e9 9b fe ff ff 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 89 c6 53 89 cb 83 ec 0c 40 04 00 00 00 00 c7 40 08 00 00 00 00 64 a1 10 d0 74 79 89 [ 60.574284] EIP: [<78287084>] lockdep_init_map+0xd/0x544 SS:ESP 0068:854f7e78 [ 60.574284] CR2: 0000000000000008 [ 60.598000] ---[ end trace 2de24a556554a2aa ]--- [ 60.598976] kworker/0:0 (4) used greatest stack depth: 6464 bytes left dmesg-quantal-ant-6:20131113183449:i386-randconfig-j7-11082318:3.12.0-00187-g9d0d532:1 [ 54.292005] SCSI Media Changer driver v0.25 [ 54.459617] scsi_debug: host protection [ 54.460597] scsi0 : scsi_debug, version 1.82 [20100324], dev_size_mb=8, opts=0x0 [ 54.519306] BUG: unable to handle kernel NULL pointer dereference at 00000008 [ 54.519328] IP: [<78287084>] lockdep_init_map+0xd/0x544 [ 54.519328] *pde = 00000000 [ 54.519328] Oops: 0002 [#1] SMP [ 54.519328] CPU: 0 PID: 4 Comm: kworker/0:0 Not tainted 3.12.0-00187-g9d0d532 #1 [ 54.519328] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 54.519328] Workqueue: events tg_stats_alloc_fn [ 54.519328] task: 854f0070 ti: 854f6000 task.ti: 854f6000 [ 54.519328] EIP: 0060:[<78287084>] EFLAGS: 00000296 CPU: 0 [ 54.519328] EIP is at lockdep_init_map+0xd/0x544 [ 54.519328] EAX: 00000004 EBX: 79d614b0 ECX: 79d614b0 EDX: 793fba74 [ 54.519328] ESI: 00000004 EDI: 79d614b8 EBP: 854f7e90 ESP: 854f7e78 [ 54.519328] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 [ 54.519328] CR0: 8005003b CR2: 00000008 CR3: 018a4000 CR4: 000006d0 [ 54.519328] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 54.519328] DR6: 00000000 DR7: 00000000 [ 54.519328] Stack: [ 54.519328] 0000014a 0000014b 854bd740 00000000 00000000 79d614b8 854f7eac 785fae1d [ 54.519328] 00000000 795aea60 854bd740 00000000 00000000 854f7ef8 7826bae8 00000000 [ 54.519328] 00000001 00000000 7826ba8f 854a6a00 85bf0700 795aea60 85bef380 795aea60 [ 54.519328] Call Trace: [ 54.519328] [<785fae1d>] tg_stats_alloc_fn+0xe3/0x155 [ 54.519328] [<7826bae8>] process_one_work+0x1ef/0x349 [ 54.519328] [<7826ba8f>] ? process_one_work+0x196/0x349 [ 54.519328] [<7826c039>] worker_thread+0x1a3/0x274 [ 54.519328] [<7826be96>] ? rescuer_thread+0x233/0x233 [ 54.519328] [<78270649>] kthread+0x8a/0x8f [ 54.519328] [<78270000>] ? SyS_timer_gettime+0x43/0x9e [ 54.519328] [<78fbac7b>] ret_from_kernel_thread+0x1b/0x30 [ 54.519328] [<782705bf>] ? __kthread_parkme+0x50/0x50 [ 54.519328] Code: f0 8b 7d e8 64 ff 0d 68 d7 74 79 89 70 10 89 78 14 e9 9b fe ff ff 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 57 56 89 c6 53 89 cb 83 ec 0c 40 04 00 00 00 00 c7 40 08 00 00 00 00 64 a1 10 d0 74 79 89 [ 54.519328] EIP: [<78287084>] lockdep_init_map+0xd/0x544 SS:ESP 0068:854f7e78 [ 54.519328] CR2: 0000000000000008 [ 54.543176] ---[ end trace ddf2d30616f27d7a ]--- [ 54.544356] kworker/0:0 (4) used greatest stack depth: 6464 bytes left Thanks, Fengguang