rcu,sched: spinlock recursion on 3.5-rc2

rcu,sched: spinlock recursion on 3.5-rc2

[Sasha Levin]

Hey all,

I've got the following splat while fuzzing with trinity in a KVM tools guest on 3.5-rc2:

[ 8110.274070] BUG: spinlock recursion on CPU#0, rcu_torture_rea/2658
[ 8110.275014]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8110.275014] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8110.275014] Call Trace:
[ 8110.275014]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8110.275014]  [<ffffffff8197deab>] spin_bug+0x2b/0x40
[ 8110.275014]  [<ffffffff8197dfee>] do_raw_spin_lock+0x4e/0x140
[ 8110.275014]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.275014]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8110.275014]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8110.275014]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8110.275014]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8110.275014]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8110.275014]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8110.275014]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8110.275014]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8110.275014]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.275014]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8110.275014]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8110.275014]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8110.275014]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8110.275014]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8110.275014]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8110.275014]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8110.275014]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.275014]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.275014]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.275014]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.275014]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.275014]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] BUG: spinlock lockup on CPU#1, rcu_torture_rea/2653
[ 8110.306059]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ>  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8110.306059]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI>  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] sending NMI to all CPUs:
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8110.306059] RSP: 0018:ffff88001b803bd0  EFLAGS: 00000086
[ 8110.306059] RAX: 0000000000000400 RBX: 0000000000000001 RCX: 0000000000000830
[ 8110.306059] RDX: 0000000000000001 RSI: 0000000000000400 RDI: 0000000000000830
[ 8110.306059] RBP: ffff88001b803bd0 R08: ffffffff851259c0 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 8110.306059] R13: ffffffff851259c0 R14: 0000000000000001 R15: 000000000000ef40
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c40 ffffffff81093d2d 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c50 0000000000000086 0000000000000002 0000000200080000
[ 8110.306059]  ffff88001b803c20 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8110.306059]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8110.306059]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8110.306059]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff81116887>]  [<ffffffff81116887>] sub_preempt_count+0x87/0xb0
[ 8111.347769] RSP: 0018:ffff88000cf959e8  EFLAGS: 00000002
[ 8111.347769] RAX: ffff88000cf95fd8 RBX: 000000004eacf13e RCX: 0000000000000000
[ 8111.347769] RDX: 00000000000012db RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf959e8 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf94010
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000000 R15: 000000004eacf17b
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a28 ffffffff819674aa ffff88000cf95a18 ffff88000d9d6140
[ 8111.347769]  000000008abefb96 00000000948a9c80 0000000000000001 0000000000000086
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 01 f0 04 00 75 3e be 4b 0c 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff <eb> 17 0f 1f 80 00 00 00 00 81 ff fe 00 00 00 77 e0 eb aa 66 0f 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff819674a0>]  [<ffffffff819674a0>] delay_tsc+0x100/0x140
[ 8111.351463] RSP: 0018:ffff88000d575c90  EFLAGS: 00000006
[ 8111.351463] RAX: 0000000000000029 RBX: 000000006bcb86d7 RCX: 000000006bcb8700
[ 8111.351463] RDX: 00000000000012db RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575cc0 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000d574010
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: 000000006bcb8700
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575d10 ffff88000d9d6140 0000000092cd334a 00000000948a9c80
[ 8111.351463]  0000000000000001 ffff88000d9d6140 ffff88000d575cd0 ffffffff8196736a
[ 8111.351463]  ffff88000d575d10 ffffffff8197e07d ffff88000d575d10 0000000000000086
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: cf 0d 03 00 75 04 0f 0b eb fe 44 29 fb 41 89 c6 41 01 dd e8 f3 91 70 ff 66 90 89 c3 e9 6a ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 <bf> 01 00 00 00 e8 56 f3 7a ff 65 48 8b 04 25 88 d9 00 00 48 8b 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff819673c9>]  [<ffffffff819673c9>] delay_tsc+0x29/0x140
[ 8111.011063] RSP: 0018:ffff880041803d78  EFLAGS: 00000002
[ 8111.011063] RAX: 0000000000000004 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.011063] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: 000000008c9addbe
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803df8 ffff88000d9d6140 000000008c9addbe 00000000948a9c80
[ 8111.011063]  0000000000000001 0000000000000001 ffff880041803db8 ffffffff8196736a
[ 8111.011063]  ffff880041803df8 ffffffff8197e07d ffff880041803df8 0000000000000086
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 <48> 83 3d 37 d0 0d 03 00 75 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff819673c9>]  [<ffffffff819673c9>] delay_tsc+0x29/0x140
[ 8112.229101] RSP: 0018:ffff8800354d7ca0  EFLAGS: 00000006
[ 8112.229101] RAX: 0000000000000003 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8112.229101] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000090092bc0
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7d20 ffff88000d9d6140 0000000090092bc0 00000000948a9c80
[ 8112.229101]  0000000000000001 ffff88000d9d6140 ffff8800354d7ce0 ffffffff8196736a
[ 8112.229101]  ffff8800354d7d20 ffffffff8197e07d ffff8800354d7d20 0000000000000086
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 00 00 55 48 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 <48> 83 3d 37 d0 0d 03 00 75 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 
[ 8111.351463] BUG: spinlock lockup on CPU#2, rcub/5/11
[ 8111.351463]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.351463]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] sending NMI to all CPUs:
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.351463] RSP: 0018:ffff88000d575c30  EFLAGS: 00000086
[ 8111.351463] RAX: 0000000000000400 RBX: 0000000000000002 RCX: 0000000000000830
[ 8111.351463] RDX: 0000000000000002 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.351463] RBP: ffff88000d575c30 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000002
[ 8111.351463] R13: ffffffff851259c0 R14: 0000000000000002 R15: 000000000000ef40
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575ca0 ffffffff81093d2d 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cb0 0000000000000086 0000000000000002 0000000200080000
[ 8111.351463]  ffff88000d575c80 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.351463]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.351463]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.351463]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff81116890>]  [<ffffffff81116890>] sub_preempt_count+0x90/0xb0
[ 8110.306059] RSP: 0018:ffff88001b803c20  EFLAGS: 00000006
[ 8110.306059] RAX: ffff88000cf73fd8 RBX: 00000000c3c0e658 RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c20 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: 00000000c3c0e67c
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c60 ffffffff819674aa 0000000000000000 ffff88000d9d6140
[ 8110.306059]  0000000001978cb7 00000000948a9c80 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c70 ffffffff8196736a ffff88001b803cb0 ffffffff8197e07d
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff eb 17 0f 1f 80 00 00 00 00 <81> ff fe 00 00 00 77 e0 eb aa 66 0f 1f 44 00 00 c9 0f 1f 80 00 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff819673d1>]  [<ffffffff819673d1>] delay_tsc+0x31/0x140
[ 8111.347769] RSP: 0018:ffff88000cf959f8  EFLAGS: 00000082
[ 8111.347769] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf95a28 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 000000008bfc4879
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a18 ffff88000d9d6140 000000008bfc4879 00000000948a9c80
[ 8111.347769]  0000000000000001 0000000000000086 ffff88000cf95a38 ffffffff8196736a
[ 8111.347769]  ffff88000cf95a78 ffffffff8197e07d ffff88000cf95a78 0000000000000086
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 e8 f2 f4 7a ff e8 8d 84 01 00 0f 1f 00 0f ae e8 48 83 3d 37 d0 0d 03 00 <75> 07 0f 0b 0f 1f 00 eb fe 41 89 c6 e8 9e 92 70 ff 66 90 65 4c 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8112.229101] RSP: 0018:ffff8800354d7c90  EFLAGS: 00000082
[ 8112.229101] RAX: 00000000cfcf63b2 RBX: 00000000cfcf6383 RCX: 00000000cfcf6383
[ 8112.229101] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7c90 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800354d6010
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cd0 ffffffff81967425 ffff8800354d7d20 ffff88000d9d6140
[ 8112.229101]  00000000913d633e 00000000948a9c80 0000000000000001 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7ce0 ffffffff8196736a ffff8800354d7d20 ffffffff8197e07d
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff81967425>] delay_tsc+0x85/0x140
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8111.011063] RSP: 0018:ffff880041803d78  EFLAGS: 00000082
[ 8111.011063] RAX: 000012dccfa8fa68 RBX: 00000000cfa8fa68 RCX: 00000000cfa8fa68
[ 8111.011063] RDX: 00000000000012dc RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: ffff88000cf74010
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000004 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803df8 ffff88000d9d6140 000000008daabef7 00000000948a9c80
[ 8111.011063]  0000000000000001 0000000000000001 ffff880041803db8 ffffffff8196736a
[ 8111.011063]  ffff880041803df8 ffffffff8197e07d ffff880041803df8 0000000000000086
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 
[ 8112.229101] BUG: spinlock lockup on CPU#3, watchdog/3/27
[ 8112.229101]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8112.229101]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] sending NMI to all CPUs:
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff819673a1>]  [<ffffffff819673a1>] delay_tsc+0x1/0x140
[ 8111.347769] RSP: 0018:ffff88000cf95a28  EFLAGS: 00000002
[ 8111.347769] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf95a38 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 000000008f1a16ab
[ 8111.347769] R13: 00000000948a9c80 R14: 0000000000000001 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769]  ffff88000cf95a78 0000000000000086 ffff88000d9d6140 ffff88000d9d6158
[ 8111.347769]  00000000001d6140 000000000000008b ffff88000cf95aa8 ffffffff837c0b2b
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8196736a>] ? __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 00 00 00 48 89 e5 65 48 8b 14 25 58 60 1d 00 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a 01 ff 15 08 0c 47 03 c9 c3 66 0f 1f 44 00 00 55 <48> 89 e5 41 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8112.229101] RSP: 0018:ffff8800354d7c40  EFLAGS: 00000086
[ 8112.229101] RAX: 0000000000000400 RBX: 0000000000000003 RCX: 0000000000000830
[ 8112.229101] RDX: 0000000000000003 RSI: 0000000000000400 RDI: 0000000000000830
[ 8112.229101] RBP: ffff8800354d7c40 R08: ffffffff851259c0 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 8112.229101] R13: ffffffff851259c0 R14: 0000000000000003 R15: 000000000000ef40
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cb0 ffffffff81093d2d 0000000000000000 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7cc0 0000000000000086 0000000000000002 0000000200080000
[ 8112.229101]  ffff8800354d7c90 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8112.229101]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8112.229101]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8112.229101]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.351463] RSP: 0018:ffff88000d575c80  EFLAGS: 00000082
[ 8111.351463] RAX: 00000000e96c109c RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.351463] RDX: 00000000000012df RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575c80 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000033fccab
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575cc0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8111.351463]  00000000033fccab 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cd0 ffffffff8196736a ffff88000d575d10 ffffffff8197e07d
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8196736a>]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059] RSP: 0018:ffff88001b803c70  EFLAGS: 00000082
[ 8110.306059] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012df RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c70 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000058864be
[ 8110.306059] R13: 00000000948a9c80 R14: 0000000000000000 R15: ffff88001b803d70
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803cb0 ffffffff8197e07d ffff88001b803cb0 0000000000000086
[ 8110.306059]  ffff88000d9d6140 ffff88000d9d6158 0000000000000287 0000000000000000
[ 8110.306059]  ffff88001b803ce0 ffffffff837c0b2b ffffffff81126620 000000000000fff8
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 55 48 c7 05 54 0c 47 03 a0 73 96 81 48 89 e5 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 ff 15 36 0c 47 03 <c9> c3 0f 1f 40 00 55 48 8d 04 bd 00 00 00 00 48 89 e5 65 48 8b 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff81070688>]  [<ffffffff81070688>] native_read_tsc+0x8/0x20
[ 8111.011063] RSP: 0018:ffff880041803d68  EFLAGS: 00000082
[ 8111.011063] RAX: 0000000007eddbe7 RBX: ffff88000d9d6140 RCX: 0000000007eddbe7
[ 8111.011063] RDX: 00000000000012e0 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803d68 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000091460d14
[ 8111.011063] R13: 0000000000000001 R14: 0000000000000004 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803da8 ffffffff819673e2 ffff880041803df8 ffff88000d9d6140
[ 8111.011063]  0000000091460d14 00000000948a9c80 0000000000000001 0000000000000001
[ 8111.011063]  ffff880041803db8 ffffffff8196736a ffff880041803df8 ffffffff8197e07d
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 <c9> 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 00 00 
[ 8111.011063] BUG: spinlock lockup on CPU#4, rcu_torture_rea/2654
[ 8111.011063]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ>  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.011063]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI>  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] sending NMI to all CPUs:
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.011063] RSP: 0018:ffff880041803d18  EFLAGS: 00000086
[ 8111.011063] RAX: 0000000000000400 RBX: 0000000000000004 RCX: 0000000000000830
[ 8111.011063] RDX: 0000000000000004 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.011063] RBP: ffff880041803d18 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000004
[ 8111.011063] R13: ffffffff851259c0 R14: 0000000000000004 R15: 000000000000ef40
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803d88 ffffffff81093d2d 0000000000000000 0000000000000001
[ 8111.011063]  ffff880041803d98 0000000000000086 0000000000000002 0000000200080000
[ 8111.011063]  ffff880041803d68 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.011063]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.011063]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.011063]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.347769] RSP: 0018:ffff88000cf959e8  EFLAGS: 00000082
[ 8111.347769] RAX: 0000000069137f7d RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.347769] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.347769] RBP: ffff88000cf959e8 R08: 0000000010000004 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000932d7753
[ 8111.347769] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000086
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a28 ffffffff819673e2 ffff88000cf95a18 ffff88000d9d6140
[ 8111.347769]  00000000932d7753 00000000948a9c80 0000000000000001 0000000000000086
[ 8111.347769]  ffff88000cf95a38 ffffffff8196736a ffff88000cf95a78 ffffffff8197e07d
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.347769]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.347769]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8110.306059] RSP: 0018:ffff88001b803c30  EFLAGS: 00000082
[ 8110.306059] RAX: 000012e369144e60 RBX: 0000000069144e60 RCX: 0000000069144e60
[ 8110.306059] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c60 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: ffff88001b803d70
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  0000000000000000 ffff88000d9d6140 000000000a12546a 00000000948a9c80
[ 8110.306059]  0000000000000000 ffff88001b803d70 ffff88001b803c70 ffffffff8196736a
[ 8110.306059]  ffff88001b803cb0 ffffffff8197e07d ffff88001b803cb0 0000000000000086
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff81070688>]  [<ffffffff81070688>] native_read_tsc+0x8/0x20
[ 8112.229101] RSP: 0018:ffff8800354d7c90  EFLAGS: 00000082
[ 8112.229101] RAX: 0000000069147143 RBX: ffff88000d9d6140 RCX: 0000000069147143
[ 8112.229101] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7c90 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000003e7b190
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  ffff8800354d7cd0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8112.229101]  0000000003e7b190 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8112.229101]  ffff8800354d7ce0 ffffffff8196736a ffff8800354d7d20 ffffffff8197e07d
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 <c9> 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 00 00 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff81070686>]  [<ffffffff81070686>] native_read_tsc+0x6/0x20
[ 8111.351463] RSP: 0018:ffff88000d575c80  EFLAGS: 00000082
[ 8111.351463] RAX: 00000000686f9b15 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.351463] RDX: 00000000000012e3 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575c80 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000006a0401f
[ 8111.351463] R13: 0000000000000001 R14: 0000000000000002 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffff88000d575cc0 ffffffff819673e2 0000000000000000 ffff88000d9d6140
[ 8111.351463]  0000000006a0401f 00000000948a9c80 0000000000000000 ffff88000d9d6140
[ 8111.351463]  ffff88000d575cd0 ffffffff8196736a ffff88000d575d10 ffffffff8197e07d
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff819673e2>] delay_tsc+0x42/0x140
[ 8111.351463]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 08 ff 15 ae be 9c 03 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 <89> c1 c9 48 89 d0 89 c9 48 c1 e0 20 48 09 c8 c3 66 2e 0f 1f 84 
[ 8111.347769] BUG: spinlock lockup on CPU#0, rcu_torture_rea/2658
[ 8111.347769]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
[ 8111.347769]  [<ffffffff8197e09e>] do_raw_spin_lock+0xfe/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] sending NMI to all CPUs:
[ 8111.347769] NMI backtrace for cpu 0
[ 8111.347769] CPU 0 
[ 8111.347769] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.347769] RIP: 0010:[<ffffffff8109a5aa>]  [<ffffffff8109a5aa>] native_write_msr_safe+0xa/0x10
[ 8111.347769] RSP: 0018:ffff88000cf95998  EFLAGS: 00000086
[ 8111.347769] RAX: 0000000000000400 RBX: 0000000000000000 RCX: 0000000000000830
[ 8111.347769] RDX: 0000000000000000 RSI: 0000000000000400 RDI: 0000000000000830
[ 8111.347769] RBP: ffff88000cf95998 R08: ffffffff851259c0 R09: 0000000000000000
[ 8111.347769] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[ 8111.347769] R13: ffffffff851259c0 R14: 0000000000000000 R15: 000000000000ef40
[ 8111.347769] FS:  0000000000000000(0000) GS:ffff88000d800000(0000) knlGS:0000000000000000
[ 8111.347769] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.347769] CR2: 0000000000000ff8 CR3: 000000005eb81000 CR4: 00000000000406f0
[ 8111.347769] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.347769] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.347769] Process rcu_torture_rea (pid: 2658, threadinfo ffff88000cf94000, task ffff88000cfa0000)
[ 8111.347769] Stack:
[ 8111.347769]  ffff88000cf95a08 ffffffff81093d2d 0000000000000000 0000000000000086
[ 8111.347769]  ffff88000cf95a18 0000000000000086 0000000000000002 0000000200080000
[ 8111.347769]  ffff88000cf959e8 0000000000000000 00000000948a9c80 00000000948a9c80
[ 8111.347769] Call Trace:
[ 8111.347769]  [<ffffffff81093d2d>] __x2apic_send_IPI_mask+0xdd/0x1b0
[ 8111.347769]  [<ffffffff81093e17>] x2apic_send_IPI_all+0x17/0x20
[ 8111.347769]  [<ffffffff8108f184>] arch_trigger_all_cpu_backtrace+0x64/0xb0
[ 8111.347769]  [<ffffffff8197e0a3>] do_raw_spin_lock+0x103/0x140
[ 8111.347769]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.347769]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
[ 8111.347769]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
[ 8111.347769]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
[ 8111.347769]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
[ 8111.347769]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
[ 8111.347769]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
[ 8111.347769]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
[ 8111.347769]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.347769]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
[ 8111.347769]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
[ 8111.347769]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
[ 8111.347769]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
[ 8111.347769]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
[ 8111.347769]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.347769]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.347769]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.347769]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.347769]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.347769]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.347769]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.347769]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.347769] Code: 00 55 89 f9 48 89 e5 0f 32 45 31 c0 89 c7 44 89 06 48 89 d0 89 f9 48 c1 e0 20 c9 48 09 c8 c3 66 90 55 89 f0 48 89 e5 89 f9 0f 30 <31> c0 c9 c3 66 90 55 89 f9 48 89 e5 0f 33 89 c7 c9 48 89 d0 89 
[ 8110.306059] NMI backtrace for cpu 1
[ 8110.306059] CPU 1 
[ 8110.306059] Pid: 2653, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8110.306059] RIP: 0010:[<ffffffff81116890>]  [<ffffffff81116890>] sub_preempt_count+0x90/0xb0
[ 8110.306059] RSP: 0018:ffff88001b803c20  EFLAGS: 00000006
[ 8110.306059] RAX: ffff88000cf73fd8 RBX: 0000000046560e3e RCX: 0000000000000000
[ 8110.306059] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8110.306059] RBP: ffff88001b803c20 R08: 0000000000000105 R09: 0000000000000000
[ 8110.306059] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000cf72010
[ 8110.306059] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000046560e6d
[ 8110.306059] FS:  0000000000000000(0000) GS:ffff88001b800000(0000) knlGS:0000000000000000
[ 8110.306059] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8110.306059] CR2: 0000000000000ffc CR3: 000000005f1ee000 CR4: 00000000000406e0
[ 8110.306059] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8110.306059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8110.306059] Process rcu_torture_rea (pid: 2653, threadinfo ffff88000cf72000, task ffff88000cf6b000)
[ 8110.306059] Stack:
[ 8110.306059]  ffff88001b803c60 ffffffff819674aa 0000000000000000 ffff88000d9d6140
[ 8110.306059]  000000000c21d807 00000000948a9c80 0000000000000000 ffff88001b803d70
[ 8110.306059]  ffff88001b803c70 ffffffff8196736a ffff88001b803cb0 ffffffff8197e07d
[ 8110.306059] Call Trace:
[ 8110.306059]  <IRQ> 
[ 8110.306059]  [<ffffffff819674aa>] delay_tsc+0x10a/0x140
[ 8110.306059]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8110.306059]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8110.306059]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8110.306059]  [<ffffffff81126620>] ? load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126620>] load_balance+0x410/0x820
[ 8110.306059]  [<ffffffff81126d38>] ? update_shares+0x158/0x240
[ 8110.306059]  [<ffffffff81126fdd>] rebalance_domains+0x1bd/0x290
[ 8110.306059]  [<ffffffff81126e20>] ? update_shares+0x240/0x240
[ 8110.306059]  [<ffffffff8106f5fd>] ? sched_clock+0x1d/0x30
[ 8110.306059]  [<ffffffff811272c3>] run_rebalance_domains+0x43/0x60
[ 8110.306059]  [<ffffffff810e2c01>] __do_softirq+0x221/0x460
[ 8110.306059]  [<ffffffff8109a516>] ? kvm_clock_read+0x46/0x80
[ 8110.306059]  [<ffffffff837c3aec>] call_softirq+0x1c/0x30
[ 8110.306059]  [<ffffffff81069215>] do_softirq+0x75/0x120
[ 8110.306059]  [<ffffffff810e1c0b>] irq_exit+0x5b/0xf0
[ 8110.306059]  [<ffffffff8108e99a>] smp_apic_timer_interrupt+0x8a/0xa0
[ 8110.306059]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8110.306059]  <EOI> 
[ 8110.306059]  [<ffffffff837c164b>] ? _raw_spin_unlock_irq+0x2b/0x80
[ 8110.306059]  [<ffffffff837c166c>] ? _raw_spin_unlock_irq+0x4c/0x80
[ 8110.306059]  [<ffffffff811171bd>] finish_task_switch+0x8d/0x110
[ 8110.306059]  [<ffffffff81117180>] ? finish_task_switch+0x50/0x110
[ 8110.306059]  [<ffffffff837bf943>] __schedule+0x7e3/0x880
[ 8110.306059]  [<ffffffff8118dfc0>] ? rcu_torture_reader+0x380/0x380
[ 8110.306059]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8110.306059]  [<ffffffff8118dee2>] rcu_torture_reader+0x2a2/0x380
[ 8110.306059]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8110.306059]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8110.306059]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8110.306059]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8110.306059]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8110.306059]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8110.306059]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8110.306059] Code: 00 00 48 c7 c7 6f a7 40 84 e8 2d 21 fc ff eb 2b 0f 1f 00 65 48 8b 04 25 88 d9 00 00 29 b8 44 e0 ff ff eb 17 0f 1f 80 00 00 00 00 <81> ff fe 00 00 00 77 e0 eb aa 66 0f 1f 44 00 00 c9 0f 1f 80 00 
[ 8111.351463] NMI backtrace for cpu 2
[ 8111.351463] CPU 2 
[ 8111.351463] Pid: 11, comm: rcub/5 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.351463] RIP: 0010:[<ffffffff819674d1>]  [<ffffffff819674d1>] delay_tsc+0x131/0x140
[ 8111.351463] RSP: 0018:ffff88000d575cc8  EFLAGS: 00000082
[ 8111.351463] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 0000000000000000
[ 8111.351463] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.351463] RBP: ffff88000d575cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8111.351463] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000008a00398
[ 8111.351463] R13: 00000000948a9c80 R14: 0000000000000000 R15: ffff88000d9d6140
[ 8111.351463] FS:  0000000000000000(0000) GS:ffff880029800000(0000) knlGS:0000000000000000
[ 8111.351463] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.351463] CR2: 00007f155b06eff9 CR3: 00000000389c4000 CR4: 00000000000406e0
[ 8111.351463] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.351463] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.351463] Process rcub/5 (pid: 11, threadinfo ffff88000d574000, task ffff88000d578000)
[ 8111.351463] Stack:
[ 8111.351463]  ffffffff8196736a ffff88000d575d10 ffffffff8197e07d ffff88000d575d10
[ 8111.351463]  0000000000000086 ffff88000d9d6140 ffff88000d9d6158 0000000000000002
[ 8111.351463]  0000000000000000 ffff88000d575d40 ffffffff837c0b2b ffffffff8112bfec
[ 8111.351463] Call Trace:
[ 8111.351463]  [<ffffffff8196736a>] ? __delay+0xa/0x10
[ 8111.351463]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.351463]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.351463]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8111.351463]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8111.351463]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8111.351463]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8111.351463]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8111.351463]  [<ffffffff81190e50>] rcu_boost_kthread+0x90/0x120
[ 8111.351463]  [<ffffffff81190dc0>] ? trace_rcu_utilization+0xd0/0xd0
[ 8111.351463]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.351463]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.351463]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.351463]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.351463]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.351463] Code: 56 f3 7a ff 65 48 8b 04 25 88 d9 00 00 48 8b 80 38 e0 ff ff a8 08 74 05 e8 bd 85 e5 01 48 83 c4 08 5b 41 5c 41 5d 41 5e 41 5f c9 <c3> 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 8d 04 bf 48 
[ 8111.011063] NMI backtrace for cpu 4
[ 8111.011063] CPU 4 
[ 8111.011063] Pid: 2654, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8111.011063] RIP: 0010:[<ffffffff819673a4>]  [<ffffffff819673a4>] delay_tsc+0x4/0x140
[ 8111.011063] RSP: 0018:ffff880041803da8  EFLAGS: 00000002
[ 8111.011063] RAX: 0000000000000000 RBX: ffff88000d9d6140 RCX: 000000000000b6d3
[ 8111.011063] RDX: 00000000b6d3b6d2 RSI: 0000000000000000 RDI: 0000000000000001
[ 8111.011063] RBP: ffff880041803da8 R08: 0000000000010001 R09: 0000000000000000
[ 8111.011063] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000191edad
[ 8111.011063] R13: 00000000948a9c80 R14: 0000000000000000 R15: 0000000000000001
[ 8111.011063] FS:  0000000000000000(0000) GS:ffff880041800000(0000) knlGS:0000000000000000
[ 8111.011063] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8111.011063] CR2: 00007f93d6ff5021 CR3: 000000000809d000 CR4: 00000000000406e0
[ 8111.011063] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8111.011063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8111.011063] Process rcu_torture_rea (pid: 2654, threadinfo ffff88000cf74000, task ffff88000cf78000)
[ 8111.011063] Stack:
[ 8111.011063]  ffff880041803db8 ffffffff8196736a ffff880041803df8 ffffffff8197e07d
[ 8111.011063]  ffff880041803df8 0000000000000086 ffff88000d9d6140 ffff88000d9d6158
[ 8111.011063]  0000000000000001 0000000000000000 ffff880041803e28 ffffffff837c0b2b
[ 8111.011063] Call Trace:
[ 8111.011063]  <IRQ> 
[ 8111.011063]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8111.011063]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8111.011063]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8111.011063]  [<ffffffff8112c8fe>] ? do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112c8fe>] do_sched_rt_period_timer+0x7e/0x320
[ 8111.011063]  [<ffffffff8112cbe5>] sched_rt_period_timer+0x45/0x70
[ 8111.011063]  [<ffffffff8110c4a6>] __run_hrtimer+0x206/0x330
[ 8111.011063]  [<ffffffff8112cba0>] ? do_sched_rt_period_timer+0x320/0x320
[ 8111.011063]  [<ffffffff8110c898>] hrtimer_interrupt+0x118/0x230
[ 8111.011063]  [<ffffffff8111641f>] ? account_system_vtime+0xff/0x140
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8108e995>] smp_apic_timer_interrupt+0x85/0xa0
[ 8111.011063]  [<ffffffff837c30ef>] apic_timer_interrupt+0x6f/0x80
[ 8111.011063]  <EOI> 
[ 8111.011063]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
[ 8111.011063]  [<ffffffff8114f01d>] ? lock_release+0x17d/0x1a0
[ 8111.011063]  [<ffffffff8118dc38>] rcu_torture_read_unlock+0x58/0x60
[ 8111.011063]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
[ 8111.011063]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
[ 8111.011063]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
[ 8111.011063]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8111.011063]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8111.011063]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8111.011063]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8111.011063]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8111.011063] Code: 48 89 e5 65 48 8b 14 25 58 60 1d 00 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a 01 ff 15 08 0c 47 03 c9 c3 66 0f 1f 44 00 00 55 48 89 e5 <41> 57 41 56 41 55 49 89 fd 41 54 bf 01 00 00 00 53 48 83 ec 08 
[ 8112.229101] NMI backtrace for cpu 3
[ 8112.229101] CPU 3 
[ 8112.229101] Pid: 27, comm: watchdog/3 Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376  
[ 8112.229101] RIP: 0010:[<ffffffff8196740e>]  [<ffffffff8196740e>] delay_tsc+0x6e/0x140
[ 8112.229101] RSP: 0018:ffff8800354d7ca0  EFLAGS: 00000082
[ 8112.229101] RAX: 000012e54536e761 RBX: 000000004536e761 RCX: 000000004536e761
[ 8112.229101] RDX: 00000000000012e5 RSI: 0000000000000000 RDI: 0000000000000001
[ 8112.229101] RBP: ffff8800354d7cd0 R08: 0000000000000002 R09: 0000000000000000
[ 8112.229101] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800354d6010
[ 8112.229101] R13: 0000000000000001 R14: 0000000000000003 R15: ffff88000d9d6140
[ 8112.229101] FS:  0000000000000000(0000) GS:ffff880035a00000(0000) knlGS:0000000000000000
[ 8112.229101] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 8112.229101] CR2: 00007fe1aee26600 CR3: 000000000b42d000 CR4: 00000000000406e0
[ 8112.229101] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 8112.229101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 8112.229101] Process watchdog/3 (pid: 27, threadinfo ffff8800354d6000, task ffff8800354d8000)
[ 8112.229101] Stack:
[ 8112.229101]  0000000000000000 ffff88000d9d6140 0000000005bcbf34 00000000948a9c80
[ 8112.229101]  0000000000000000 ffff88000d9d6140 ffff8800354d7ce0 ffffffff8196736a
[ 8112.229101]  ffff8800354d7d20 ffffffff8197e07d ffff8800354d7d20 0000000000000086
[ 8112.229101] Call Trace:
[ 8112.229101]  [<ffffffff8196736a>] __delay+0xa/0x10
[ 8112.229101]  [<ffffffff8197e07d>] do_raw_spin_lock+0xdd/0x140
[ 8112.229101]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
[ 8112.229101]  [<ffffffff8112bfec>] ? pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff837c1530>] ? _raw_spin_unlock+0x30/0x60
[ 8112.229101]  [<ffffffff8112bfec>] pull_rt_task+0xec/0x270
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff8112c224>] pre_schedule_rt+0x14/0x20
[ 8112.229101]  [<ffffffff837bf3b9>] __schedule+0x259/0x880
[ 8112.229101]  [<ffffffff81121785>] ? sched_clock_local+0x25/0x90
[ 8112.229101]  [<ffffffff81183620>] ? watchdog_enable+0x110/0x110
[ 8112.229101]  [<ffffffff837bfb45>] schedule+0x55/0x60
[ 8112.229101]  [<ffffffff8118369a>] watchdog+0x7a/0xd0
[ 8112.229101]  [<ffffffff81106e32>] kthread+0xb2/0xc0
[ 8112.229101]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
[ 8112.229101]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
[ 8112.229101]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
[ 8112.229101]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
[ 8112.229101] Code: 90 65 4c 8b 24 25 88 d9 00 00 89 c3 49 81 ec c8 1f 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 0f ae e8 48 83 3d fa cf 0d 03 00 <75> 10 0f 0b 66 0f 1f 44 00 00 eb fe 66 0f 1f 44 00 00 e8 5b 92 

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Thomas Gleixner]

On Tue, 12 Jun 2012, Sasha Levin wrote:

> Hey all,
> 
> I've got the following splat while fuzzing with trinity in a KVM tools guest on 3.5-rc2:
> 
> [ 8110.274070] BUG: spinlock recursion on CPU#0, rcu_torture_rea/2658
> [ 8110.275014]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
> [ 8110.275014] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
> [ 8110.275014] Call Trace:
> [ 8110.275014]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
> [ 8110.275014]  [<ffffffff8197deab>] spin_bug+0x2b/0x40
> [ 8110.275014]  [<ffffffff8197dfee>] do_raw_spin_lock+0x4e/0x140
> [ 8110.275014]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
> [ 8110.275014]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
> [ 8110.275014]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
> [ 8110.275014]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
> [ 8110.275014]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
> [ 8110.275014]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
> [ 8110.275014]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
> [ 8110.275014]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
> [ 8110.275014]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
> [ 8110.275014]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
> [ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
> [ 8110.275014]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
> [ 8110.275014]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
> [ 8110.275014]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
> [ 8110.275014]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
> [ 8110.275014]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
> [ 8110.275014]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
> [ 8110.275014]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
> [ 8110.275014]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
> [ 8110.275014]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
> [ 8110.275014]  [<ffffffff81106e32>] kthread+0xb2/0xc0
> [ 8110.275014]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
> [ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
> [ 8110.275014]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
> [ 8110.275014]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13

Ok, that's nasty.

The torture thread got preempted. rcu_preempt_note_context_switch()
tries to unlock the boosting rt mutex.

Though rcu_preempt_note_context_switch() is called with rq lock
held. So it's not a surprise that the code will dead lock.

My brain hurts already from looking, so Paul to the rescue!

Thanks,

	tglx

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Paul E. McKenney]

On Tue, Jun 12, 2012 at 03:40:13PM +0200, Thomas Gleixner wrote:
> On Tue, 12 Jun 2012, Sasha Levin wrote:
> 
> > Hey all,
> > 
> > I've got the following splat while fuzzing with trinity in a KVM tools guest on 3.5-rc2:
> > 
> > [ 8110.274070] BUG: spinlock recursion on CPU#0, rcu_torture_rea/2658
> > [ 8110.275014]  lock: 0xffff88000d9d6140, .magic: dead4ead, .owner: rcu_torture_rea/2658, .owner_cpu: 0
> > [ 8110.275014] Pid: 2658, comm: rcu_torture_rea Tainted: G        W    3.5.0-rc2-sasha-00019-gbd68491 #376
> > [ 8110.275014] Call Trace:
> > [ 8110.275014]  [<ffffffff8197de38>] spin_dump+0x78/0xc0
> > [ 8110.275014]  [<ffffffff8197deab>] spin_bug+0x2b/0x40
> > [ 8110.275014]  [<ffffffff8197dfee>] do_raw_spin_lock+0x4e/0x140
> > [ 8110.275014]  [<ffffffff837c0b2b>] _raw_spin_lock+0x5b/0x70
> > [ 8110.275014]  [<ffffffff8111d051>] ? rt_mutex_setprio+0x81/0x2c0
> > [ 8110.275014]  [<ffffffff8111d051>] rt_mutex_setprio+0x81/0x2c0
> > [ 8110.275014]  [<ffffffff81155160>] __rt_mutex_adjust_prio+0x20/0x30
> > [ 8110.275014]  [<ffffffff837c0164>] rt_mutex_slowunlock+0x104/0x130
> > [ 8110.275014]  [<ffffffff837c0199>] rt_mutex_unlock+0x9/0x10
> > [ 8110.275014]  [<ffffffff81193e30>] rcu_read_unlock_special+0x350/0x400
> > [ 8110.275014]  [<ffffffff8114841a>] ? get_lock_stats+0x2a/0x60
> > [ 8110.275014]  [<ffffffff811941aa>] rcu_preempt_note_context_switch+0x22a/0x300
> > [ 8110.275014]  [<ffffffff837bf8ca>] __schedule+0x76a/0x880
> > [ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
> > [ 8110.275014]  [<ffffffff8118dc20>] ? rcu_torture_read_unlock+0x40/0x60
> > [ 8110.275014]  [<ffffffff837bff64>] preempt_schedule_irq+0x94/0xd0
> > [ 8110.275014]  [<ffffffff837c1da6>] retint_kernel+0x26/0x30
> > [ 8110.275014]  [<ffffffff81193ae1>] ? rcu_read_unlock_special+0x1/0x400
> > [ 8110.275014]  [<ffffffff81193f2d>] ? __rcu_read_unlock+0x4d/0xa0
> > [ 8110.275014]  [<ffffffff8118dc3d>] rcu_torture_read_unlock+0x5d/0x60
> > [ 8110.275014]  [<ffffffff8118dedd>] rcu_torture_reader+0x29d/0x380
> > [ 8110.275014]  [<ffffffff8118ca50>] ? T.865+0x50/0x50
> > [ 8110.275014]  [<ffffffff8118dc40>] ? rcu_torture_read_unlock+0x60/0x60
> > [ 8110.275014]  [<ffffffff81106e32>] kthread+0xb2/0xc0
> > [ 8110.275014]  [<ffffffff837c39f4>] kernel_thread_helper+0x4/0x10
> > [ 8110.275014]  [<ffffffff837c1c74>] ? retint_restore_args+0x13/0x13
> > [ 8110.275014]  [<ffffffff81106d80>] ? __init_kthread_worker+0x70/0x70
> > [ 8110.275014]  [<ffffffff837c39f0>] ? gs_change+0x13/0x13
> 
> Ok, that's nasty.
> 
> The torture thread got preempted. rcu_preempt_note_context_switch()
> tries to unlock the boosting rt mutex.
> 
> Though rcu_preempt_note_context_switch() is called with rq lock
> held. So it's not a surprise that the code will dead lock.
> 
> My brain hurts already from looking, so Paul to the rescue!

My brain hurts from beating my head on my desk.  It seems that attempts
to enhance PREEMPT_RCU's read-side performance require even more paranoia
than I normally bring to bear.  :-/

Please see below for what I expect is the relevant revert.

							Thanx, Paul

------------------------------------------------------------------------

Revert "rcu: Move PREEMPT_RCU preemption to switch_to() invocation"

This reverts commit 616c310e83b872024271c915c1b9ab505b9efad9
(Move PREEMPT_RCU preemption to switch_to() invocation) which can
result in runqueue deadlock.

Reported-by: Sasha Levin <levinsasha928@gmail.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

diff --git a/arch/um/drivers/mconsole_kern.c b/arch/um/drivers/mconsole_kern.c
index 88e466b..43b39d6 100644
--- a/arch/um/drivers/mconsole_kern.c
+++ b/arch/um/drivers/mconsole_kern.c
@@ -705,7 +705,6 @@ static void stack_proc(void *arg)
 	struct task_struct *from = current, *to = arg;
 
 	to->thread.saved_task = from;
-	rcu_switch_from(from);
 	switch_to(from, to, from);
 }
 
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index 26d1a47..9cac722 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -184,7 +184,6 @@ static inline int rcu_preempt_depth(void)
 /* Internal to kernel */
 extern void rcu_sched_qs(int cpu);
 extern void rcu_bh_qs(int cpu);
-extern void rcu_preempt_note_context_switch(void);
 extern void rcu_check_callbacks(int cpu, int user);
 struct notifier_block;
 extern void rcu_idle_enter(void);
diff --git a/include/linux/rcutiny.h b/include/linux/rcutiny.h
index adb5e5a..148c6db 100644
--- a/include/linux/rcutiny.h
+++ b/include/linux/rcutiny.h
@@ -87,6 +87,10 @@ static inline void kfree_call_rcu(struct rcu_head *head,
 
 #ifdef CONFIG_TINY_RCU
 
+static inline void rcu_preempt_note_context_switch(void)
+{
+}
+
 static inline int rcu_needs_cpu(int cpu)
 {
 	return 0;
@@ -94,6 +98,7 @@ static inline int rcu_needs_cpu(int cpu)
 
 #else /* #ifdef CONFIG_TINY_RCU */
 
+void rcu_preempt_note_context_switch(void);
 int rcu_preempt_needs_cpu(void);
 
 static inline int rcu_needs_cpu(int cpu)
@@ -106,6 +111,7 @@ static inline int rcu_needs_cpu(int cpu)
 static inline void rcu_note_context_switch(int cpu)
 {
 	rcu_sched_qs(cpu);
+	rcu_preempt_note_context_switch();
 }
 
 /*
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 4059c0f..06a4c5f 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1871,22 +1871,12 @@ static inline void rcu_copy_process(struct task_struct *p)
 	INIT_LIST_HEAD(&p->rcu_node_entry);
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-	if (prev->rcu_read_lock_nesting != 0)
-		rcu_preempt_note_context_switch();
-}
-
 #else
 
 static inline void rcu_copy_process(struct task_struct *p)
 {
 }
 
-static inline void rcu_switch_from(struct task_struct *prev)
-{
-}
-
 #endif
 
 #ifdef CONFIG_SMP
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index 0da7b88..d994530 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -201,6 +201,7 @@ void rcu_note_context_switch(int cpu)
 {
 	trace_rcu_utilization("Start context switch");
 	rcu_sched_qs(cpu);
+	rcu_preempt_note_context_switch(cpu);
 	trace_rcu_utilization("End context switch");
 }
 EXPORT_SYMBOL_GPL(rcu_note_context_switch);
diff --git a/kernel/rcutree.h b/kernel/rcutree.h
index 7f5d138..bd220db 100644
--- a/kernel/rcutree.h
+++ b/kernel/rcutree.h
@@ -430,6 +430,7 @@ DECLARE_PER_CPU(char, rcu_cpu_has_work);
 /* Forward declarations for rcutree_plugin.h */
 static void rcu_bootup_announce(void);
 long rcu_batches_completed(void);
+static void rcu_preempt_note_context_switch(int cpu);
 static int rcu_preempt_blocked_readers_cgp(struct rcu_node *rnp);
 #ifdef CONFIG_HOTPLUG_CPU
 static void rcu_report_unblock_qs_rnp(struct rcu_node *rnp,
diff --git a/kernel/rcutree_plugin.h b/kernel/rcutree_plugin.h
index 2411000..fb6e07e 100644
--- a/kernel/rcutree_plugin.h
+++ b/kernel/rcutree_plugin.h
@@ -153,7 +153,7 @@ static void rcu_preempt_qs(int cpu)
  *
  * Caller must disable preemption.
  */
-void rcu_preempt_note_context_switch(void)
+static void rcu_preempt_note_context_switch(int cpu)
 {
 	struct task_struct *t = current;
 	unsigned long flags;
@@ -164,7 +164,7 @@ void rcu_preempt_note_context_switch(void)
 	    (t->rcu_read_unlock_special & RCU_READ_UNLOCK_BLOCKED) == 0) {
 
 		/* Possibly blocking in an RCU read-side critical section. */
-		rdp = __this_cpu_ptr(rcu_preempt_state.rda);
+		rdp = per_cpu_ptr(rcu_preempt_state.rda, cpu);
 		rnp = rdp->mynode;
 		raw_spin_lock_irqsave(&rnp->lock, flags);
 		t->rcu_read_unlock_special |= RCU_READ_UNLOCK_BLOCKED;
@@ -228,7 +228,7 @@ void rcu_preempt_note_context_switch(void)
 	 * means that we continue to block the current grace period.
 	 */
 	local_irq_save(flags);
-	rcu_preempt_qs(smp_processor_id());
+	rcu_preempt_qs(cpu);
 	local_irq_restore(flags);
 }
 
@@ -1002,6 +1002,14 @@ void rcu_force_quiescent_state(void)
 EXPORT_SYMBOL_GPL(rcu_force_quiescent_state);
 
 /*
+ * Because preemptible RCU does not exist, we never have to check for
+ * CPUs being in quiescent states.
+ */
+static void rcu_preempt_note_context_switch(int cpu)
+{
+}
+
+/*
  * Because preemptible RCU does not exist, there are never any preempted
  * RCU readers.
  */
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index d5594a4..eaead2d 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -2081,7 +2081,6 @@ context_switch(struct rq *rq, struct task_struct *prev,
 #endif
 
 	/* Here we just switch the register state and the stack. */
-	rcu_switch_from(prev);
 	switch_to(prev, next, prev);
 
 	barrier();

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Thomas Gleixner]

On Tue, 12 Jun 2012, Paul E. McKenney wrote:
> On Tue, Jun 12, 2012 at 03:40:13PM +0200, Thomas Gleixner wrote:
> > The torture thread got preempted. rcu_preempt_note_context_switch()
> > tries to unlock the boosting rt mutex.
> > 
> > Though rcu_preempt_note_context_switch() is called with rq lock
> > held. So it's not a surprise that the code will dead lock.
> > 
> > My brain hurts already from looking, so Paul to the rescue!
> 
> My brain hurts from beating my head on my desk.  It seems that attempts
> to enhance PREEMPT_RCU's read-side performance require even more paranoia
> than I normally bring to bear.  :-/
> 
> Please see below for what I expect is the relevant revert.
> 
> 							Thanx, Paul
> 
> ------------------------------------------------------------------------
> 
> Revert "rcu: Move PREEMPT_RCU preemption to switch_to() invocation"
> 
> This reverts commit 616c310e83b872024271c915c1b9ab505b9efad9
> (Move PREEMPT_RCU preemption to switch_to() invocation) which can
> result in runqueue deadlock.

Hmm, not sure. The deadlock was not triggered in switch_to. It was
just at the beginning of __schedule()

need_resched:
	preempt_disable();
        cpu = smp_processor_id();
	rq = cpu_rq(cpu);
        rcu_note_context_switch(cpu);
	
	Which ends up in rcu_read_unlock_special() which tries to
	unlock the rtmutex.

So that code is still there ....

Thanks,

	tglx

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Peter Zijlstra]

On Tue, 2012-06-12 at 17:07 +0200, Thomas Gleixner wrote:
> Hmm, not sure. The deadlock was not triggered in switch_to. It was
> just at the beginning of __schedule()
> 
> 
How can you tell? switch_to() is a macro not a function, it won't ever
show up on a stack-trace.. but I think you meant context_switch() but
that typically gets inlined, similar problem.

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Thomas Gleixner]

On Tue, 12 Jun 2012, Peter Zijlstra wrote:

> On Tue, 2012-06-12 at 17:07 +0200, Thomas Gleixner wrote:
> > Hmm, not sure. The deadlock was not triggered in switch_to. It was
> > just at the beginning of __schedule()
> > 
> > 
> How can you tell? switch_to() is a macro not a function, it won't ever
> show up on a stack-trace.. but I think you meant context_switch() but
> that typically gets inlined, similar problem.

Bah, I misread the trace and the rcu code. :(

So, yes the revert should solve the problem. Looks like I need more
coffee, not Paul :)

Thanks,

	tglx

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Paul E. McKenney]

On Tue, Jun 12, 2012 at 05:07:37PM +0200, Thomas Gleixner wrote:
> On Tue, 12 Jun 2012, Paul E. McKenney wrote:
> > On Tue, Jun 12, 2012 at 03:40:13PM +0200, Thomas Gleixner wrote:
> > > The torture thread got preempted. rcu_preempt_note_context_switch()
> > > tries to unlock the boosting rt mutex.
> > > 
> > > Though rcu_preempt_note_context_switch() is called with rq lock
> > > held. So it's not a surprise that the code will dead lock.
> > > 
> > > My brain hurts already from looking, so Paul to the rescue!
> > 
> > My brain hurts from beating my head on my desk.  It seems that attempts
> > to enhance PREEMPT_RCU's read-side performance require even more paranoia
> > than I normally bring to bear.  :-/
> > 
> > Please see below for what I expect is the relevant revert.
> > 
> > 							Thanx, Paul
> > 
> > ------------------------------------------------------------------------
> > 
> > Revert "rcu: Move PREEMPT_RCU preemption to switch_to() invocation"
> > 
> > This reverts commit 616c310e83b872024271c915c1b9ab505b9efad9
> > (Move PREEMPT_RCU preemption to switch_to() invocation) which can
> > result in runqueue deadlock.
> 
> Hmm, not sure. The deadlock was not triggered in switch_to. It was
> just at the beginning of __schedule()
> 
> need_resched:
> 	preempt_disable();
>         cpu = smp_processor_id();
> 	rq = cpu_rq(cpu);
>         rcu_note_context_switch(cpu);
> 	
> 	Which ends up in rcu_read_unlock_special() which tries to
> 	unlock the rtmutex.
> 
> So that code is still there ....

???

rcu_note_context_switch() does not call rcu_preempt_note_context_switch()
unless the patch is reverted.  No rcu_preempt_note_context_switch(),
no rcu_read_unlock_special().

Besides which, unless I am even more confused than normal, __schedule()
doesn't hold any runqueue locks at this point.  However, those locks
really are held across the call to switch_to() in context_switch().

I bet that gcc inlined context_switch().

							Thanx, Paul

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Paul E. McKenney]

On Tue, Jun 12, 2012 at 05:20:10PM +0200, Thomas Gleixner wrote:
> On Tue, 12 Jun 2012, Peter Zijlstra wrote:
> 
> > On Tue, 2012-06-12 at 17:07 +0200, Thomas Gleixner wrote:
> > > Hmm, not sure. The deadlock was not triggered in switch_to. It was
> > > just at the beginning of __schedule()
> > > 
> > > 
> > How can you tell? switch_to() is a macro not a function, it won't ever
> > show up on a stack-trace.. but I think you meant context_switch() but
> > that typically gets inlined, similar problem.
> 
> Bah, I misread the trace and the rcu code. :(
> 
> So, yes the revert should solve the problem. Looks like I need more
> coffee, not Paul :)

Well, given that the commit that I reverted was my own, I definitely
needed more of something or another...  :-)

							Thanx, Paul

Re: rcu,sched: spinlock recursion on 3.5-rc2

[Sasha Levin]

On Tue, 2012-06-12 at 17:15 +0200, Peter Zijlstra wrote:
> On Tue, 2012-06-12 at 17:07 +0200, Thomas Gleixner wrote:
> > Hmm, not sure. The deadlock was not triggered in switch_to. It was
> > just at the beginning of __schedule()
> > 
> > 
> How can you tell? switch_to() is a macro not a function, it won't ever
> show up on a stack-trace.. but I think you meant context_switch() but
> that typically gets inlined, similar problem.

addr2line -i confirmed it was pointing to switch_to(), if it matters.

btw, wouldn't it be nice if there were a short script that would take an
oops and convert all those kernel addresses into file+line numbers based
on a given vmlinux? It would make our lives so much easier.