From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sasha Levin Subject: blk/dm: Kernel crash on 3.5-rc2 Date: Mon, 11 Jun 2012 19:18:10 +0200 Message-ID: <1339435090.4999.76.camel@lappy> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: Sender: linux-raid-owner@vger.kernel.org To: neilb@suse.de, axboe Cc: "linux-kernel@vger.kernel.org" , linux-raid@vger.kernel.org, Dave Jones List-Id: linux-raid.ids Hi all, I was fuzzing with trinity inside a KVM tools guest running on 3.5-rc2, when I hit the following: [ 1848.195822] INFO: trying to register non-static key. [ 1848.196040] the code is fine but needs lockdep annotation. [ 1848.196040] turning off the locking correctness validator. [ 1848.196040] Pid: 2901, comm: kworker/4:1 Tainted: G W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1848.196040] Call Trace: [ 1848.196040] [] register_lock_class+0x538/0x540 [ 1848.196040] [] ? sched_clock+0x1d/0x30 [ 1848.196040] [] __lock_acquire+0xf7/0x4b0 [ 1848.196040] [] lock_acquire+0x18a/0x1e0 [ 1848.196040] [] ? __wake_up+0x2d/0x70 [ 1848.196040] [] ? get_lock_stats+0x2a/0x60 [ 1848.196040] [] _raw_spin_lock_irqsave+0x81/0xc0 [ 1848.196040] [] ? __wake_up+0x2d/0x70 [ 1848.196040] [] __wake_up+0x2d/0x70 [ 1848.196040] [] blk_drain_queue+0x116/0x160 [ 1848.196040] [] blk_cleanup_queue+0x17c/0x1b0 [ 1848.196040] [] md_free+0x53/0x70 [ 1848.196040] [] kobject_cleanup+0x145/0x190 [ 1848.196040] [] kobject_release+0xd/0x10 [ 1848.196040] [] kobject_put+0x4c/0x60 [ 1848.196040] [] mddev_delayed_delete+0x2f/0x40 [ 1848.196040] [] process_one_work+0x366/0x5f0 [ 1848.196040] [] ? process_one_work+0x240/0x5f0 [ 1848.196040] [] ? bb_show+0x20/0x20 [ 1848.196040] [] worker_thread+0x211/0x3a0 [ 1848.196040] [] ? manage_workers+0x130/0x130 [ 1848.196040] [] kthread+0xb2/0xc0 [ 1848.196040] [] kernel_thread_helper+0x4/0x10 [ 1848.196040] [] ? retint_restore_args+0x13/0x13 [ 1848.196040] [] ? __init_kthread_worker+0x70/0x70 [ 1848.196040] [] ? gs_change+0x13/0x13 Which was followed by: [ 1848.196040] BUG: unable to handle kernel NULL pointer dereference at (null) [ 1848.196040] IP: [] __wake_up_common+0x2c/0x90 [ 1848.196040] PGD 0 [ 1848.196040] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 1848.196040] CPU 4 [ 1848.196040] Pid: 2901, comm: kworker/4:1 Tainted: G W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1848.196040] RIP: 0010:[] [] __wake_up_common+0x2c/0x90 [ 1848.196040] RSP: 0018:ffff880040df5bd0 EFLAGS: 00010086 [ 1848.196040] RAX: ffffffffffffffe8 RBX: ffff8800644a0040 RCX: 0000000000000000 [ 1848.196040] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8800644a0040 [ 1848.196040] RBP: ffff880040df5c10 R08: 0000000000000000 R09: 0000000000000001 [ 1848.196040] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000086 [ 1848.196040] R13: ffff8800644a0088 R14: 0000000000000000 R15: 0000000000000000 [ 1848.196040] FS: 0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000 [ 1848.196040] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1848.196040] CR2: 0000000000000000 CR3: 0000000004a25000 CR4: 00000000000406e0 [ 1848.196040] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1848.196040] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1848.196040] Process kworker/4:1 (pid: 2901, threadinfo ffff880040df4000, task ffff8800415ab000) [ 1848.196040] Stack: [ 1848.196040] ffff880040df5c10 0000000300000000 ffffffff8111533d ffff8800644a0040 [ 1848.196040] 0000000000000086 0000000000000003 0000000000000000 0000000000000000 [ 1848.196040] ffff880040df5c50 ffffffff81115353 ffff880040df5c50 ffff8800644a0000 [ 1848.196040] Call Trace: [ 1848.196040] [] ? __wake_up+0x2d/0x70 [ 1848.196040] [] __wake_up+0x43/0x70 [ 1848.196040] [] blk_drain_queue+0x116/0x160 [ 1848.196040] [] blk_cleanup_queue+0x17c/0x1b0 [ 1848.196040] [] md_free+0x53/0x70 [ 1848.196040] [] kobject_cleanup+0x145/0x190 [ 1848.196040] [] kobject_release+0xd/0x10 [ 1848.196040] [] kobject_put+0x4c/0x60 [ 1848.196040] [] mddev_delayed_delete+0x2f/0x40 [ 1848.196040] [] process_one_work+0x366/0x5f0 [ 1848.196040] [] ? process_one_work+0x240/0x5f0 [ 1848.196040] [] ? bb_show+0x20/0x20 [ 1848.196040] [] worker_thread+0x211/0x3a0 [ 1848.196040] [] ? manage_workers+0x130/0x130 [ 1848.196040] [] kthread+0xb2/0xc0 [ 1848.196040] [] kernel_thread_helper+0x4/0x10 [ 1848.196040] [] ? retint_restore_args+0x13/0x13 [ 1848.196040] [] ? __init_kthread_worker+0x70/0x70 [ 1848.196040] [] ? gs_change+0x13/0x13 [ 1848.196040] Code: 48 89 e5 41 57 41 89 cf 41 56 4d 89 c6 41 55 4c 8d 6f 48 41 54 53 48 83 ec 18 89 75 cc 89 55 c8 48 8b 57 48 48 8d 42 e8 49 39 d5 <48> 8b 58 18 74 4e 48 83 eb 18 66 2e 0f 1f 84 00 00 00 00 00 44 [ 1848.196040] RIP [] __wake_up_common+0x2c/0x90 [ 1848.196040] RSP [ 1848.196040] CR2: 0000000000000000 [ 1848.196040] ---[ end trace 6d450e935ee18982 ]--- Once the noise from the OOPS has stopped (there were 4-5 follow-up dumps - mostly things angry the oopsing itself), I've tried sysrq-t to see what's up in the system, but that was interrupted half-way by: [ 1848.862010] BUG: spinlock lockup on CPU#4, kworker/4:1/2901 [ 1848.862010] lock: 0xffff8800419d6140, .magic: dead4ead, .owner: kworker/4:1/2901, .owner_cpu: 4 [ 1848.862010] Pid: 2901, comm: kworker/4:1 Tainted: G D W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1848.862010] Call Trace: [ 1848.862010] [] spin_dump+0x78/0xc0 [ 1848.862010] [] do_raw_spin_lock+0xfe/0x140 [ 1848.862010] [] _raw_spin_lock_irq+0x81/0xa0 [ 1848.862010] [] ? __schedule+0x1a0/0x880 [ 1848.862010] [] __schedule+0x1a0/0x880 [ 1848.862010] [] schedule+0x55/0x60 [ 1848.862010] [] do_exit+0x107/0x4f0 [ 1848.862010] [] oops_end+0x108/0x120 [ 1848.862010] [] no_context+0x1bf/0x1e0 [ 1848.862010] [] ? cfq_put_queue+0x1f9/0x2a0 [ 1848.862010] [] __bad_area_nosemaphore+0x20d/0x250 [ 1848.862010] [] ? free_debug_processing+0x2bf/0x350 [ 1848.862010] [] bad_area_nosemaphore+0xe/0x10 [ 1848.862010] [] do_page_fault+0x209/0x450 [ 1848.862010] [] ? __lock_acquired+0xda/0x2e0 [ 1848.862010] [] ? debug_object_activate+0x6d/0x1b0 [ 1848.862010] [] ? print_lock_contention_bug+0x22/0x110 [ 1848.862010] [] ? __lock_acquired+0xda/0x2e0 [ 1848.862010] [] ? __queue_work+0x250/0x500 [ 1848.862010] [] do_async_page_fault+0x31/0xb0 [ 1848.862010] [] async_page_fault+0x25/0x30 [ 1848.862010] [] ? kthread_data+0xb/0x20 [ 1848.862010] [] wq_worker_sleeping+0x10/0xb0 [ 1848.862010] [] __schedule+0x226/0x880 [ 1848.862010] [] schedule+0x55/0x60 [ 1848.862010] [] do_exit+0x4c9/0x4f0 [ 1848.862010] [] oops_end+0x108/0x120 [ 1848.862010] [] no_context+0x1bf/0x1e0 [ 1848.862010] [] __bad_area_nosemaphore+0x20d/0x250 [ 1848.862010] [] bad_area_nosemaphore+0xe/0x10 [ 1848.862010] [] do_page_fault+0x209/0x450 [ 1848.862010] [] ? dump_stack+0x77/0x82 [ 1848.862010] [] ? register_lock_class+0x538/0x540 [ 1848.862010] [] ? sched_clock+0x1d/0x30 [ 1848.862010] [] do_async_page_fault+0x31/0xb0 [ 1848.862010] [] async_page_fault+0x25/0x30 [ 1848.862010] [] ? __wake_up_common+0x2c/0x90 [ 1848.862010] [] ? __wake_up+0x2d/0x70 [ 1848.862010] [] __wake_up+0x43/0x70 [ 1848.862010] [] blk_drain_queue+0x116/0x160 [ 1848.862010] [] blk_cleanup_queue+0x17c/0x1b0 [ 1848.862010] [] md_free+0x53/0x70 [ 1848.862010] [] kobject_cleanup+0x145/0x190 [ 1848.862010] [] kobject_release+0xd/0x10 [ 1848.862010] [] kobject_put+0x4c/0x60 [ 1848.862010] [] mddev_delayed_delete+0x2f/0x40 [ 1848.862010] [] process_one_work+0x366/0x5f0 [ 1848.862010] [] ? process_one_work+0x240/0x5f0 [ 1848.862010] [] ? bb_show+0x20/0x20 [ 1848.862010] [] worker_thread+0x211/0x3a0 [ 1848.862010] [] ? manage_workers+0x130/0x130 [ 1848.862010] [] kthread+0xb2/0xc0 [ 1848.862010] [] kernel_thread_helper+0x4/0x10 [ 1848.862010] [] ? retint_restore_args+0x13/0x13 [ 1848.862010] [] ? __init_kthread_worker+0x70/0x70 [ 1848.862010] [] ? gs_change+0x13/0x13 [ 1848.862010] sending NMI to all CPUs: [ 1919.331706] NMI backtrace for cpu 0 [ 1919.331706] CPU 0 [ 1919.331706] Pid: 0, comm: swapper/0 Tainted: G D W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1919.331706] RIP: 0010:[] [] delay_tsc+0x29/0x140 [ 1919.331706] RSP: 0018:ffff88000d803a38 EFLAGS: 00000002 [ 1919.331706] RAX: 0000000000000000 RBX: ffff8800419d6140 RCX: 000000000000e778 [ 1919.331706] RDX: 00000000e778e777 RSI: 0000000000000000 RDI: 0000000000000001 [ 1919.331706] RBP: ffff88000d803a68 R08: 0000000000010006 R09: 0000000000000000 [ 1919.331706] R10: 0000000000000000 R11: 0000000000000001 R12: 000000004fd88e78 [ 1919.331706] R13: 0000000000000001 R14: 0000000000000001 R15: 00000012c96bbf75 [ 1919.331706] FS: 0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000 [ 1919.331706] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1919.331706] CR2: 00007fdb780a2600 CR3: 0000000033ecf000 CR4: 00000000000406f0 [ 1919.331706] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1919.331706] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1919.331706] Process swapper/0 (pid: 0, threadinfo ffffffff84a00000, task ffffffff84a2d400) [ 1919.331706] Stack: [ 1919.331706] ffff88000d803ab8 ffff8800419d6140 000000004fd88e78 00000000948a9c80 [ 1919.331706] 0000000000000001 00000012c96bbf75 ffff88000d803a78 ffffffff8196736a [ 1919.331706] ffff88000d803ab8 ffffffff8197e07d ffff88000d803ab8 0000000000000086 [ 1919.331706] Call Trace: [ 1919.331706] [ 1919.331706] [] __delay+0xa/0x10 [ 1919.331706] [] do_raw_spin_lock+0xdd/0x140 [ 1919.331706] [] _raw_spin_lock_irqsave+0xa1/0xc0 [ 1919.331706] [] ? print_cfs_rq+0x128/0x700 [ 1919.331706] [] print_cfs_rq+0x128/0x700 [ 1919.331706] [] ? unthrottle_offline_cfs_rqs+0x100/0x100 [ 1919.331706] [] ? __lock_acquired+0x3d/0x2e0 [ 1919.331706] [] print_cfs_stats+0xf9/0x1f0 [ 1919.331706] [] ? unthrottle_offline_cfs_rqs+0x100/0x100 [ 1919.331706] [] ? print_cpu+0x59c/0x6b0 [ 1919.331706] [] print_cpu+0x5aa/0x6b0 [ 1919.331706] [] ? print_cpu+0x590/0x6b0 [ 1919.331706] [] sched_debug_show+0x5e2/0x680 [ 1919.331706] [] sysrq_sched_debug_show+0xd/0x10 [ 1919.331706] [] show_state_filter+0x1aa/0x230 [ 1919.331706] [] ? account_system_time+0x2a0/0x2a0 [ 1919.331706] [] sysrq_handle_showstate+0xb/0x10 [ 1919.331706] [] __handle_sysrq+0xb7/0x180 [ 1919.331706] [] handle_sysrq+0x21/0x30 [ 1919.331706] [] serial8250_rx_chars+0x179/0x210 [ 1919.331706] [] ? _raw_spin_lock_irqsave+0xad/0xc0 [ 1919.331706] [] ? serial8250_handle_irq+0x23/0x80 [ 1919.331706] [] serial8250_handle_irq+0x46/0x80 [ 1919.331706] [] serial8250_default_handle_irq+0x1e/0x30 [ 1919.331706] [] serial8250_interrupt+0x4e/0xd0 [ 1919.331706] [] handle_irq_event_percpu+0x151/0x3e0 [ 1919.331706] [] handle_irq_event+0x43/0x70 [ 1919.331706] [] ? handle_level_irq+0x19/0xe0 [ 1919.331706] [] handle_level_irq+0xa8/0xe0 [ 1919.331706] [] handle_irq+0x164/0x180 [ 1919.331706] [] ? srcu_notifier_call_chain+0x20/0x20 [ 1919.331706] [] do_IRQ+0x58/0xd0 [ 1919.331706] [] common_interrupt+0x6f/0x6f [ 1919.331706] [ 1919.331706] [] ? default_idle+0x1a8/0x390 [ 1919.331706] [] ? native_safe_halt+0x6/0x10 [ 1919.331706] [] ? trace_hardirqs_on+0xd/0x10 [ 1919.331706] [] default_idle+0x1c5/0x390 [ 1919.331706] [] cpu_idle+0x11f/0x180 [ 1919.331706] [] rest_init+0x131/0x140 [ 1919.331706] [] ? csum_partial_copy_generic+0x170/0x170 [ 1919.331706] [] start_kernel+0x3c9/0x3d6 [ 1919.331706] [] ? kernel_init+0x28a/0x28a [ 1919.331706] [] x86_64_start_reservations+0xff/0x104 [ 1919.331706] [] x86_64_start_kernel+0x155/0x164 [ 1919.331706] Code: 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 <48> 83 3d 37 d0 0d 03 00 75 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 [ 1867.123342] NMI backtrace for cpu 1 [ 1867.123342] CPU 1 [ 1867.123342] Pid: 0, comm: swapper/1 Tainted: G D W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1867.123342] RIP: 0010:[] [] native_read_tsc+0xc/0x20 [ 1867.123342] RSP: 0018:ffff88001b803d70 EFLAGS: 00000082 [ 1867.123342] RAX: 000000000000049e RBX: 000000006b283299 RCX: 000000006b2832ce [ 1867.123342] RDX: 000000000000049e RSI: 0000000000000000 RDI: 0000000000000001 [ 1867.123342] RBP: ffff88001b803da8 R08: 0000000000010002 R09: 0000000000000000 [ 1867.123342] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000d5aa010 [ 1867.123342] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001 [ 1867.123342] FS: 0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000 [ 1867.123342] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1867.123342] CR2: 00007f73f1bc1a40 CR3: 0000000004a25000 CR4: 00000000000406e0 [ 1867.123342] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1867.123342] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1867.123342] Process swapper/1 (pid: 0, threadinfo ffff88000d5aa000, task ffff88000d5b0000) [ 1867.123342] Stack: [ 1867.123342] ffffffff81967425 ffff88001b803df8 ffff8800419d6140 0000000091d6599d [ 1867.123342] 00000000948a9c80 0000000000000001 0000000000000001 ffff88001b803db8 [ 1867.123342] ffffffff8196736a ffff88001b803df8 ffffffff8197e07d ffff88001b803df8 [ 1867.123342] Call Trace: [ 1867.123342] [ 1867.123342] [] ? delay_tsc+0x85/0x140 [ 1867.123342] [] __delay+0xa/0x10 [ 1867.123342] [] do_raw_spin_lock+0xdd/0x140 [ 1867.123342] [] _raw_spin_lock+0x5b/0x70 [ 1867.123342] [] ? do_sched_rt_period_timer+0x7e/0x320 [ 1867.123342] [] ? _raw_spin_unlock+0x30/0x60 [ 1867.123342] [] do_sched_rt_period_timer+0x7e/0x320 [ 1867.123342] [] sched_rt_period_timer+0x45/0x70 [ 1867.123342] [] __run_hrtimer+0x206/0x330 [ 1867.123342] [] ? do_sched_rt_period_timer+0x320/0x320 [ 1867.123342] [] hrtimer_interrupt+0x118/0x230 [ 1867.123342] [] smp_apic_timer_interrupt+0x85/0xa0 [ 1867.123342] [] apic_timer_interrupt+0x6f/0x80 [ 1867.123342] [ 1867.123342] [] ? default_idle+0x1a8/0x390 [ 1867.123342] [] ? native_safe_halt+0x6/0x10 [ 1867.123342] [] ? trace_hardirqs_on+0xd/0x10 [ 1867.123342] [] default_idle+0x1c5/0x390 [ 1867.123342] [] cpu_idle+0x11f/0x180 [ 1867.123342] [] ? setup_APIC_timer+0x88/0x8d [ 1867.123342] [] start_secondary+0xe1/0xe8 [ 1867.123342] Code: 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 c9 48 89 d0 <89> c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 00 00 00 00 00 55 [ 1848.862010] NMI backtrace for cpu 4 [ 1848.862010] CPU 4 [ 1848.862010] Pid: 2901, comm: kworker/4:1 Tainted: G D W 3.5.0-rc2-sasha-00015-g7b268f7 #374 [ 1848.862010] RIP: 0010:[] [] native_write_msr_safe+0xa/0x10 [ 1848.862010] RSP: 0018:ffff880040df5348 EFLAGS: 00000086 [ 1848.862010] RAX: 0000000000000400 RBX: 0000000000000004 RCX: 0000000000000830 [ 1848.862010] RDX: 0000000000000004 RSI: 0000000000000400 RDI: 0000000000000830 [ 1848.862010] RBP: ffff880040df5348 R08: ffffffff851259c0 R09: 0000000000000000 [ 1848.862010] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004 [ 1848.862010] R13: ffffffff851259c0 R14: 0000000000000004 R15: 000000000000ef40 [ 1848.862010] FS: 0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000 [ 1848.862010] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1848.862010] CR2: fffffffffffffff8 CR3: 0000000004a25000 CR4: 00000000000406e0 [ 1848.862010] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1848.862010] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 1848.862010] Process kworker/4:1 (pid: 2901, threadinfo ffff880040df4000, task ffff8800415ab000) [ 1848.862010] Stack: [ 1848.862010] ffff880040df53b8 ffffffff81093d2d 0000000000000000 0000000000000000 [ 1848.862010] ffff880040df53c8 0000000000000086 0000000000000002 0000000200080000 [ 1848.862010] ffff880040df5398 0000000000000000 00000000948a9c80 00000000948a9c80 [ 1848.862010] Call Trace: [ 1848.862010] [] __x2apic_send_IPI_mask+0xdd/0x1b0 [ 1848.862010] [] x2apic_send_IPI_all+0x17/0x20 [ 1848.862010] [] arch_trigger_all_cpu_backtrace+0x64/0xb0 [ 1848.862010] [] do_raw_spin_lock+0x103/0x140 [ 1848.862010] [] _raw_spin_lock_irq+0x81/0xa0 [ 1848.862010] [] ? __schedule+0x1a0/0x880 [ 1848.862010] [] __schedule+0x1a0/0x880 [ 1848.862010] [] schedule+0x55/0x60 [ 1848.862010] [] do_exit+0x107/0x4f0 [ 1848.862010] [] oops_end+0x108/0x120 [ 1848.862010] [] no_context+0x1bf/0x1e0 [ 1848.862010] [] ? cfq_put_queue+0x1f9/0x2a0 [ 1848.862010] [] __bad_area_nosemaphore+0x20d/0x250 [ 1848.862010] [] ? free_debug_processing+0x2bf/0x350 [ 1848.862010] [] bad_area_nosemaphore+0xe/0x10 [ 1848.862010] [] do_page_fault+0x209/0x450 [ 1848.862010] [] ? __lock_acquired+0xda/0x2e0 [ 1848.862010] [] ? debug_object_activate+0x6d/0x1b0 [ 1848.862010] [] ? print_lock_contention_bug+0x22/0x110 [ 1848.862010] [] ? __lock_acquired+0xda/0x2e0 [ 1848.862010] [] ? __queue_work+0x250/0x500 [ 1848.862010] [] do_async_page_fault+0x31/0xb0 [ 1848.862010] [] async_page_fault+0x25/0x30 [ 1848.862010] [] ? kthread_data+0xb/0x20 [ 1848.862010] [] wq_worker_sleeping+0x10/0xb0 [ 1848.862010] [] __schedule+0x226/0x880 [ 1848.862010] [] schedule+0x55/0x60 [ 1848.862010] [] do_exit+0x4c9/0x4f0 [ 1848.862010] [] oops_end+0x108/0x120 [ 1848.862010] [] no_context+0x1bf/0x1e0 [ 1848.862010] [] __bad_area_nosemaphore+0x20d/0x250 [ 1848.862010] [] bad_area_nosemaphore+0xe/0x10 [ 1848.862010] [] do_page_fault+0x209/0x450 [ 1848.862010] [] ? dump_stack+0x77/0x82 [ 1848.862010] [] ? register_lock_class+0x538/0x540 [ 1848.862010] [] ? sched_clock+0x1d/0x30 [ 1848.862010] [] do_async_page_fault+0x31/0xb0 [ 1848.862010] [] async_page_fault+0x25/0x30 [ 1848.862010] [] ? __wake_up_common+0x2c/0x90 [ 1848.862010] [] ? __wake_up+0x2d/0x70 [ 1848.862010] [] __wake_up+0x43/0x70 [ 1848.862010] [] blk_drain_queue+0x116/0x160 [ 1848.862010] [] blk_cleanup_queue+0x17c/0x1b0 [ 1848.862010] [] md_free+0x53/0x70 [ 1848.862010] [] kobject_cleanup+0x145/0x190 [ 1848.862010] [] kobject_release+0xd/0x10 [ 1848.862010] [] kobject_put+0x4c/0x60 [ 1848.862010] [] mddev_delayed_delete+0x2f/0x40 [ 1848.862010] [] process_one_work+0x366/0x5f0 [ 1848.862010] [] ? process_one_work+0x240/0x5f0 [ 1848.862010] [] ? bb_show+0x20/0x20 [ 1848.862010] [] worker_thread+0x211/0x3a0 [ 1848.862010] [] ? manage_workers+0x130/0x130 [ 1848.862010] [] kthread+0xb2/0xc0 [ 1848.862010] [] kernel_thread_helper+0x4/0x10 [ 1848.862010] [] ? retint_restore_args+0x13/0x13 [ 1848.862010] [] ? __init_kthread_worker+0x70/0x70 [ 1848.862010] [] ? gs_change+0x13/0x13 [ 1848.862010] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 And this little loop went on until everything choked and the vm just hang there.