From: Nikolay Borisov <kernel@kyup.com>
To: "Linux-Kernel@Vger. Kernel. Org" <linux-kernel@vger.kernel.org>
Cc: oleg@redhat.com, riel@redhat.com,
Peter Zijlstra <peterz@infradead.org>,
Steven Rostedt <rostedt@goodmis.org>
Subject: HARD LOCKUP: Strange hard lock up on spin_lock(&sighand->siglock);
Date: Mon, 27 Jul 2015 15:54:01 +0300 [thread overview]
Message-ID: <55B629E9.6020207@kyup.com> (raw)
Hello,
I have a machine running kernel 3.13.3, in fact I have 3 such machines
which are connected via corosync in a cluster. On one of the machines
I observed the following lock-up:
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 6 PID: 30236 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 6
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc06bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc06c08 ffff88087fc06bf8 ffffffff81044202 ffff88087fc06bd8
Jul 26 06:01:14 shrek kernel: ffff880869940000 0000000000000000 ffff88087fc06d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84d ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 9 PID: 30215 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 9
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc66bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc66c08 ffff88087fc66bf8 ffffffff81044202 ffff88087fc66b10
Jul 26 06:01:14 shrek kernel: ffff880869990000 0000000000000000 ffff88087fc66d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d460>] ? hw_nmi_get_sample_period+0x20/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15f>] ? _raw_spin_lock+0x1f/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84e ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 10 PID: 30390 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 10
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88087fc86bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88087fc86c08 ffff88087fc86bf8 ffffffff81044202 ffff88087fc86bd8
Jul 26 06:01:14 shrek kernel: ffff8808699b0000 0000000000000000 ffff88087fc86d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed84f ]---
Jul 26 06:01:14 shrek kernel: ------------[ cut here ]------------
Jul 26 06:01:14 shrek kernel: WARNING: CPU: 2 PID: 30144 at kernel/watchdog.c:245 watchdog_overflow_callback+0x98/0xc0()
Jul 26 06:01:14 shrek kernel: Watchdog detected hard LOCKUP on cpu 2
Jul 26 06:01:14 shrek kernel: Modules linked in: dm_snapshot netconsole configfs fuse bonding ipv6 xt_state iptable_mangle xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw dm_mirror dm_region_hash dm_log
lpc_ich mfd_core ioapic ioatdma ipmi_devintf ipmi_si ipmi_msghandler igb(O) dca
Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: 00000000000000f5 ffff88046fc46bb8 ffffffff815a6c8e 00000000000000f5
Jul 26 06:01:14 shrek kernel: ffff88046fc46c08 ffff88046fc46bf8 ffffffff81044202 ffff880867bd2510
Jul 26 06:01:14 shrek kernel: ffff880469de0000 0000000000000000 ffff88046fc46d48 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <NMI> [<ffffffff815a6c8e>] dump_stack+0x49/0x5b
Jul 26 06:01:14 shrek kernel: [<ffffffff81044202>] warn_slowpath_common+0x82/0xb0
Jul 26 06:01:14 shrek kernel: [<ffffffff810442d1>] warn_slowpath_fmt+0x41/0x50
Jul 26 06:01:14 shrek kernel: [<ffffffff810b7898>] watchdog_overflow_callback+0x98/0xc0
Jul 26 06:01:14 shrek kernel: [<ffffffff810c00c8>] __perf_event_overflow+0x98/0x240
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d460>] ? hw_nmi_get_sample_period+0x20/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810c07a4>] perf_event_overflow+0x14/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8101c14c>] intel_pmu_handle_irq+0x1cc/0x3b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8131e770>] ? apei_read+0x60/0x90
Jul 26 06:01:14 shrek kernel: [<ffffffff810160c4>] perf_event_nmi_handler+0x34/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff81006e46>] nmi_handle+0x76/0x110
Jul 26 06:01:14 shrek kernel: [<ffffffff81006fdd>] do_nmi+0xfd/0x360
Jul 26 06:01:14 shrek kernel: [<ffffffff815abc2a>] end_repeat_nmi+0x1e/0x32
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab15c>] ? _raw_spin_lock+0x1c/0x30
Jul 26 06:01:14 shrek kernel: <<EOE>> [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: ---[ end trace 13447e4cb40ed850 ]---
Jul 26 06:01:14 shrek kernel: INFO: rcu_sched detected stalls on CPUs/tasks: { 8 18} (detected by 12, t=14712 jiffies, g=115985071, c=115985070, q=186073)
Jul 26 06:01:14 shrek kernel: sending NMI to all CPUs:
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 1
Jul 26 06:01:14 shrek kernel: CPU: 1 PID: 30726 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff88013d551260 ti: ffff88013d5517b8 task.ti: ffff88013d5517b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880300eb5b08 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000a RSI: ffff880300eb5c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880300eb5b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880300eb5c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000034a18fe5700(0000) GS:ffff88046fc20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000019bfc18 CR3: 00000000015ba000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880300eb5b38 ffffffff81053474 ffff880300ff2e10 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880300eb5c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 0
Jul 26 06:01:14 shrek kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffffffff81a0a920 ti: ffffffff81a0ae78 task.ti: ffffffff81a0ae78
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffffffff81a01ec8 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffffffff81a0ae80 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc0c200
Jul 26 06:01:14 shrek kernel: RBP: ffffffff81a01ec8 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81a0ae80
Jul 26 06:01:14 shrek kernel: R13: ffffffff81a0ae80 R14: ffffffff81afa5c8 R15: ffffffff81a0ae80
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002dd81723ae0 CR3: 00000000015b8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffffffff81a01ed8 ffffffff8100c6b6 ffffffff81a01f48 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000286 ffffffff81a0ae78 ffffffff81a0ae80 ffffffff81a0ae78
Jul 26 06:01:14 shrek kernel: ffffffff81885200 77e46a1d0cffc667 00000000000937e0 ffffffff81c940c0
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8159ea42>] rest_init+0x72/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81c11069>] start_kernel+0x3f5/0x3fc
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10aa9>] ? repair_env_string+0x62/0x62
Jul 26 06:01:14 shrek kernel: [<ffffffff815a5eac>] ? memblock_reserve+0x46/0x4b
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10120>] ? early_idt_handlers+0x120/0x120
Jul 26 06:01:14 shrek kernel: [<ffffffff81c104ed>] x86_64_start_reservations+0x2a/0x2c
Jul 26 06:01:14 shrek kernel: [<ffffffff81c10643>] x86_64_start_kernel+0x154/0x15f
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 2
Jul 26 06:01:14 shrek kernel: CPU: 2 PID: 30144 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867824980 ti: ffff880867824ed8 task.ti: ffff880867824ed8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88063e911bc8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000004 RSI: ffff88063e911cf0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff88063e911bc8 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff88063e911cf0 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002d271e24700(0000) GS:ffff88046fc40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000010da108 CR3: 00000000015bc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88063e911bf8 ffffffff81053474 ffff88037e541ce8 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff88063e911d58 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 3
Jul 26 06:01:14 shrek kernel: CPU: 3 PID: 1270 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8806883ad2b0 ti: ffff8806883ad808 task.ti: ffff8806883ad808
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15a>] [<ffffffff815ab15a>] ffffffff815ab15a
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8805b3375b48 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000b RSI: ffff8805b3375c70 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8805b3375b48 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8805b3375c70 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000029bd8f09700(0000) GS:ffff88046fc60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000003ca0498 CR3: 00000000015be000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8805b3375b78 ffffffff81053474 ffff88010261a258 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8805b3375cd8 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: b6 03 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 <f3> 90 0f b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 4
Jul 26 06:01:14 shrek kernel: CPU: 4 PID: 0 Comm: swapper/4 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d70930 ti: ffff880469d70e88 task.ti: ffff880469d70e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d79e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d70e90 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fc8c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d79e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d70e90
Jul 26 06:01:14 shrek kernel: R13: ffff880469d70e90 R14: ffffffff81afa5c8 R15: ffff880469d70e90
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fc80000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002e423d730a0 CR3: 00000000015c0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d79e98 ffffffff8100c6b6 ffff880469d79f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000004 ffff880469d70e88 ffff880469d70e90 ffff880469d70e88
Jul 26 06:01:14 shrek kernel: 000000000043fd9f b6726a4c2c676c66 ffff880469d79ef8 0000000000000004
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 5
Jul 26 06:01:14 shrek kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d71260 ti: ffff880469d717b8 task.ti: ffff880469d717b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d7be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d717c0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcac200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d7be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d717c0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d717c0 R14: ffffffff81afa5c8 R15: ffff880469d717c0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fca0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015c2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d7be98 ffffffff8100c6b6 ffff880469d7bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000005 ffff880469d717b8 ffff880469d717c0 ffff880469d717b8
Jul 26 06:01:14 shrek kernel: 0000000000000206 a8d39d71d1e5d619 ffff880469d7bef8 0000000000000005
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 6
Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 45.273 msecs
Jul 26 06:01:14 shrek kernel: CPU: 6 PID: 30236 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8806a1f78930 ti: ffff8806a1f78e88 task.ti: ffff8806a1f78e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880801077be8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000008 RSI: ffff880801077d10 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880801077be8 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880801077d10 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002fb353c6700(0000) GS:ffff88087fc00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000021c82a8 CR3: 00000000015c4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880801077c18 ffffffff81053474 ffff88012c217b58 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880801077d78 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: INFO: NMI handler (ffffffff8102d460) took too long to run: 52.833 msecs
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 7
Jul 26 06:01:14 shrek kernel: CPU: 7 PID: 29973 Comm: ps Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867821b90 ti: ffff8808678220e8 task.ti: ffff8808678220e8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed2fdb08 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000002 RSI: ffff8807ed2fdc30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed2fdb08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807ed2fdc30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 00000344c1e647a0(0000) GS:ffff88087fc20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000344c1e011dc CR3: 00000000015c6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807ed2fdb38 ffffffff81053474 ffff880122b384b0 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807ed2fdc98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 8
Jul 26 06:01:14 shrek kernel: CPU: 8 PID: 29979 Comm: sg-queue Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880622796e40 ti: ffff880622797398 task.ti: ffff880622797398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab0bd>] [<ffffffff815ab0bd>] ffffffff815ab0bd
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88067e6a1e68 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: ffff880622796e40
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: ffff88067e6a1eb8 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff88067e6a1e68 R08: ffff88067e6a1eb8 R09: 000000000000751a
Jul 26 06:01:14 shrek kernel: R10: 0000000000000008 R11: 0000000000000246 R12: ffff88067e6a1e98
Jul 26 06:01:14 shrek kernel: R13: ffff88067e6a1eb8 R14: 0000000002d7c680 R15: 0000000002d7ccb0
Jul 26 06:01:14 shrek kernel: FS: 0000037e10229700(0000) GS:ffff88087fc40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000037e0f8978d0 CR3: 00000000015c8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88067e6a1e88 ffffffff81053620 0000000000000000 0000000000000002
Jul 26 06:01:14 shrek kernel: ffff88067e6a1ea8 ffffffff8105369a 0000000000000000 0000000002d7ccb0
Jul 26 06:01:14 shrek kernel: ffff88067e6a1ef8 ffffffff81056fcb 0000000000000000 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053620>] __set_current_blocked+0x30/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8105369a>] sigprocmask+0x4a/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81056fcb>] SyS_rt_sigprocmask+0xbb/0xd0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ab6bd>] ? retint_swapgs+0xe/0x11
Jul 26 06:01:14 shrek kernel: Code: 48 83 c4 08 b8 01 00 00 00 5b c9 c3 66 90 55 48 89 e5 fa b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 55 48
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 9
Jul 26 06:01:14 shrek kernel: CPU: 9 PID: 30215 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff88072c811260 ti: ffff88072c8117b8 task.ti: ffff88072c8117b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807ed29fbc8 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000007 RSI: ffff8807ed29fd00 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807ed29fbc8 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807ed29fd00 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 000002e88ef8a700(0000) GS:ffff88087fc60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000000000459c008 CR3: 00000000015ca000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807ed29fbf8 ffffffff81053474 ffff88010a1a6640 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8807ed29fd58 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 10
Jul 26 06:01:14 shrek kernel: CPU: 10 PID: 30390 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8804b4c60930 ti: ffff8804b4c60e88 task.ti: ffff8804b4c60e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880594293b08 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000009 RSI: ffff880594293c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880594293b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880594293c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000031905858700(0000) GS:ffff88087fc80000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 00000000009fc3c8 CR3: 00000000015cc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880594293b38 ffffffff81053474 ffff880101077450 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff880594293c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 11
Jul 26 06:01:14 shrek kernel: CPU: 11 PID: 30213 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff8804a95b1260 ti: ffff8804a95b17b8 task.ti: ffff8804a95b17b8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff880696bf3ba8 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000006 RSI: ffff880696bf3ce0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff880696bf3ba8 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff880696bf3ce0 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 000002bb0925b700(0000) GS:ffff88087fca0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000000be3018 CR3: 00000000015ce000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880696bf3bd8 ffffffff81053474 ffff88007993a3e8 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff880696bf3d38 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 12
Jul 26 06:01:14 shrek kernel: CPU: 12 PID: 0 Comm: swapper/12 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d752b0 ti: ffff880469d75808 task.ti: ffff880469d75808
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff812ab2c4>] [<ffffffff812ab2c4>] ffffffff812ab2c4
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff88046fcc3d38 EFLAGS: 00000046
Jul 26 06:01:14 shrek kernel: RAX: 00000000a7bd4e81 RBX: 0000000000000000 RCX: 00000000a7bd4e5d
Jul 26 06:01:14 shrek kernel: RDX: 000000000000000c RSI: 000000000000000c RDI: 0000000000033452
Jul 26 06:01:14 shrek kernel: RBP: ffff88046fcc3d38 R08: ffffffff81afa5c8 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000001848 R11: 000000002b300043 R12: 0000000000001000
Jul 26 06:01:14 shrek kernel: R13: ffffffff81afa5c8 R14: 0000000000000400 R15: 0000000000000096
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fcc0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff88046fcc3d48 ffffffff812ab289 ffff88046fcc3d68 ffffffff8102ca62
Jul 26 06:01:14 shrek kernel: 0000000000000002 000000000000a024 ffff88046fcc3db8 ffffffff8102d428
Jul 26 06:01:14 shrek kernel: ffffffff00000003 000000000000000d ffff88046fcc3d98 0000000000000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: <IRQ>
Jul 26 06:01:14 shrek kernel: [<ffffffff812ab289>] __const_udelay+0x29/0x30
Jul 26 06:01:14 shrek kernel: [<ffffffff8102ca62>] native_safe_apic_wait_icr_idle+0x22/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d428>] default_send_IPI_mask_sequence_phys+0xb8/0xd0
Jul 26 06:01:14 shrek kernel: [<ffffffff810306c2>] physflat_send_IPI_all+0x12/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff8102d537>] arch_trigger_all_cpu_backtrace+0x47/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff81091e87>] rcu_check_callbacks+0x617/0x640
Jul 26 06:01:14 shrek kernel: [<ffffffff810515c3>] update_process_times+0x43/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8109c2a1>] tick_sched_timer+0x51/0x130
Jul 26 06:01:14 shrek kernel: [<ffffffff8106980f>] __run_hrtimer+0x5f/0x130
Jul 26 06:01:14 shrek kernel: [<ffffffff81069cc6>] hrtimer_interrupt+0x106/0x250
Jul 26 06:01:14 shrek kernel: [<ffffffff8102c1f6>] local_apic_timer_interrupt+0x36/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff8102cade>] smp_apic_timer_interrupt+0x3e/0x60
Jul 26 06:01:14 shrek kernel: [<ffffffff815acba3>] apic_timer_interrupt+0x83/0x90
Jul 26 06:01:14 shrek kernel: <EOI>
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c456>] ? default_idle+0x6/0x10
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: a0 00 00 0f 1f 00 0f ae e8 0f 31 89 c1 eb 16 0f 1f 84 00 00 00 00 00 f3 90 65 8b 14 25 20 a0 00 00 39 d6 75 12 0f 1f 00 0f ae e8 <0f> 31 89 c2 29 ca 39 fa 72 e2 c9 c3 29 c1 89 d6 01 cf 0f 1f 00
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 13
Jul 26 06:01:14 shrek kernel: CPU: 13 PID: 0 Comm: swapper/13 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d75be0 ti: ffff880469d76138 task.ti: ffff880469d76138
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76140 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fcec200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76140
Jul 26 06:01:14 shrek kernel: R13: ffff880469d76140 R14: ffffffff81afa5c8 R15: ffff880469d76140
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fce0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8be98 ffffffff8100c6b6 ffff880469d8bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000d ffff880469d76138 ffff880469d76140 ffff880469d76138
Jul 26 06:01:14 shrek kernel: 0043fd9fa00c6708 5e3321e5c10994cc ffff880469d8bef8 000000000000000d
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 14
Jul 26 06:01:14 shrek kernel: CPU: 14 PID: 0 Comm: swapper/14 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d76510 ti: ffff880469d76a68 task.ti: ffff880469d76a68
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8de88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d76a70 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd0c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8de88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d76a70
Jul 26 06:01:14 shrek kernel: R13: ffff880469d76a70 R14: ffffffff81afa5c8 R15: ffff880469d76a70
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000008b70820 CR3: 00000000015d4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8de98 ffffffff8100c6b6 ffff880469d8df08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000e ffff880469d76a68 ffff880469d76a70 ffff880469d76a68
Jul 26 06:01:14 shrek kernel: 0043fd9fa1d85012 617db263e885420d ffff880469d8def8 000000000000000e
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 15
Jul 26 06:01:14 shrek kernel: CPU: 15 PID: 0 Comm: swapper/15 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d76e40 ti: ffff880469d77398 task.ti: ffff880469d77398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d8fe88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d773a0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd2c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d8fe88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d773a0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d773a0 R14: ffffffff81afa5c8 R15: ffff880469d773a0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015d6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d8fe98 ffffffff8100c6b6 ffff880469d8ff08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 000000000000000f ffff880469d77398 ffff880469d773a0 ffff880469d77398
Jul 26 06:01:14 shrek kernel: 0043fd9fa3a15351 a19db38607e2294d ffff880469d8fef8 000000000000000f
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 16
Jul 26 06:01:14 shrek kernel: CPU: 16 PID: 0 Comm: swapper/16 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d90000 ti: ffff880469d90558 task.ti: ffff880469d90558
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d99e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90560 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd4c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d99e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90560
Jul 26 06:01:14 shrek kernel: R13: ffff880469d90560 R14: ffffffff81afa5c8 R15: ffff880469d90560
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001758000 CR3: 00000000015d8000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d99e98 ffffffff8100c6b6 ffff880469d99f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000010 ffff880469d90558 ffff880469d90560 ffff880469d90558
Jul 26 06:01:14 shrek kernel: 0043fd9fa56a1bdf 2e70ed886688fb1c ffff880469d99ef8 0000000000000010
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 17
Jul 26 06:01:14 shrek kernel: CPU: 17 PID: 0 Comm: swapper/17 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d90930 ti: ffff880469d90e88 task.ti: ffff880469d90e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9be88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d90e90 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88046fd6c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9be88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d90e90
Jul 26 06:01:14 shrek kernel: R13: ffff880469d90e90 R14: ffffffff81afa5c8 R15: ffff880469d90e90
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88046fd60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000034fa60bf398 CR3: 00000000015da000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d9be98 ffffffff8100c6b6 ffff880469d9bf08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000011 ffff880469d90e88 ffff880469d90e90 ffff880469d90e88
Jul 26 06:01:14 shrek kernel: ffff880469d9beb2 05735dbd652beb34 ffff880469d9bef8 0000000000000011
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 18
Jul 26 06:01:14 shrek kernel: CPU: 18 PID: 29972 Comm: ps Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880622792df0 ti: ffff880622793348 task.ti: ffff880622793348
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8806db9e3b38 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000001 RSI: ffff8806db9e3c60 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8806db9e3b38 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8806db9e3c60 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 000002969e6097a0(0000) GS:ffff88087fcc0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 000002969e5e01dc CR3: 00000000015dc000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8806db9e3b68 ffffffff81053474 ffff880129ded2c0 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8806db9e3cc8 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 19
Jul 26 06:01:14 shrek kernel: CPU: 19 PID: 0 Comm: swapper/19 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d91b90 ti: ffff880469d920e8 task.ti: ffff880469d920e8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469d9fe88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d920f0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fcec200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469d9fe88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d920f0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d920f0 R14: ffffffff81afa5c8 R15: ffff880469d920f0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fce0000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001816cf8 CR3: 00000000015de000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469d9fe98 ffffffff8100c6b6 ffff880469d9ff08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000013 ffff880469d920e8 ffff880469d920f0 ffff880469d920e8
Jul 26 06:01:14 shrek kernel: 0043fd9faac3c713 e2b091c5f11a3308 ffff880469d9fef8 0000000000000013
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 20
Jul 26 06:01:14 shrek kernel: CPU: 20 PID: 0 Comm: swapper/20 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d924c0 ti: ffff880469d92a18 task.ti: ffff880469d92a18
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da1e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d92a20 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd0c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469da1e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d92a20
Jul 26 06:01:14 shrek kernel: R13: ffff880469d92a20 R14: ffffffff81afa5c8 R15: ffff880469d92a20
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd00000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e0000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469da1e98 ffffffff8100c6b6 ffff880469da1f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000014 ffff880469d92a18 ffff880469d92a20 ffff880469d92a18
Jul 26 06:01:14 shrek kernel: 0043fd9fac8b8e6d 0fa0aa77f886bbaa ffff880469da1ef8 0000000000000014
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 21
Jul 26 06:01:14 shrek kernel: CPU: 21 PID: 30214 Comm: pkill Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880867a2ee40 ti: ffff880867a2f398 task.ti: ffff880867a2f398
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15c>] [<ffffffff815ab15c>] ffffffff815ab15c
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8804b4dc9b68 EFLAGS: 00000097
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000004
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000005 RSI: ffff8804b4dc9ca0 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8804b4dc9b68 R08: 0000000000000001 R09: 000000000000fffd
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000006 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8804b4dc9ca0 R14: ffff880622797420 R15: 0000000000000000
Jul 26 06:01:14 shrek kernel: FS: 0000032a2ff54700(0000) GS:ffff88087fd20000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000001319008 CR3: 00000000015e2000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8804b4dc9b98 ffffffff81053474 ffff8800794e9c20 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff880488a07500 ffffffff81ab4400 ffff8804b4dc9cf8 ffffffff8119ecca
Jul 26 06:01:14 shrek kernel: ffffffff0000751a ffffffff00000000 ffffffff00000000 ffffffff00000000
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119ecca>] proc_pid_status+0x3ea/0x7c0
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: 38 c2 75 f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 <0f> b6 07 38 c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 22
Jul 26 06:01:14 shrek kernel: CPU: 22 PID: 30017 Comm: pidof Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880490800930 ti: ffff880490800e88 task.ti: ffff880490800e88
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff815ab15f>] [<ffffffff815ab15f>] ffffffff815ab15f
Jul 26 06:01:14 shrek kernel: RSP: 0018:ffff8807f1d03b08 EFLAGS: 00000093
Jul 26 06:01:14 shrek kernel: RAX: 000000000000002e RBX: ffff880622796e40 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000003 RSI: ffff8807f1d03c30 RDI: ffff880621f44a08
Jul 26 06:01:14 shrek kernel: RBP: ffff8807f1d03b08 R08: 0000000000000001 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000022 R11: 0000000000000001 R12: ffff880621f44200
Jul 26 06:01:14 shrek kernel: R13: ffff8807f1d03c30 R14: ffff880622797420 R15: ffff8801fd476300
Jul 26 06:01:14 shrek kernel: FS: 0000025ccaa4c700(0000) GS:ffff88087fd40000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: 0000000003f2a4f8 CR3: 00000000015e4000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff8807f1d03b38 ffffffff81053474 ffff88013bb36190 ffff880622796e40
Jul 26 06:01:14 shrek kernel: ffff8806d3259180 ffffffff81ab4ac0 ffff8807f1d03c98 ffffffff8119dd02
Jul 26 06:01:14 shrek kernel: ffffffffffff4111 ffffffffffff4111 ffffffffffff4111 ffffffffffff4111
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff81053474>] __lock_task_sighand+0x44/0x80
Jul 26 06:01:14 shrek kernel: [<ffffffff8119dd02>] do_task_stat+0x102/0xbb0
Jul 26 06:01:14 shrek kernel: [<ffffffff8119e7bf>] proc_tgid_stat+0xf/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff81199f2f>] proc_single_show+0x5f/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff81153a52>] seq_read+0x192/0x670
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ebc5>] vfs_read+0xd5/0x230
Jul 26 06:01:14 shrek kernel: [<ffffffff8112ee1a>] SyS_read+0x5a/0xa0
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac028>] system_call_fastpath+0x16/0x1b
Jul 26 06:01:14 shrek kernel: [<ffffffff815ac058>] ? sysret_check+0x26/0x65
Jul 26 06:01:14 shrek kernel: Code: f7 eb ee 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 b8 00 01 00 00 f0 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 75 04 c9 c3 f3 90 0f b6 07 <38> c2 75 f7 c9 c3 90 90 90 90 90 90 90 90 90 90 90 55 48 89 e5
Jul 26 06:01:14 shrek kernel: NMI backtrace for cpu 23
Jul 26 06:01:14 shrek kernel: CPU: 23 PID: 0 Comm: swapper/23 Tainted: W O 3.13.3-sg1 #6
Jul 26 06:01:14 shrek kernel: Hardware name: Supermicro X9DRD-7LN4F(-JBOD)/X9DRD-EF/X9DRD-7LN4F, BIOS 3.0a 12/05/2013
Jul 26 06:01:14 shrek kernel: task: ffff880469d94050 ti: ffff880469d945a8 task.ti: ffff880469d945a8
Jul 26 06:01:14 shrek kernel: RIP: 0010:[<ffffffff8100c456>] [<ffffffff8100c456>] ffffffff8100c456
Jul 26 06:01:14 shrek kernel: RSP: 0000:ffff880469da7e88 EFLAGS: 00000286
Jul 26 06:01:14 shrek kernel: RAX: 00000000ffffffed RBX: ffff880469d945b0 RCX: 0000000000000000
Jul 26 06:01:14 shrek kernel: RDX: 0000000000000000 RSI: 0140000000000000 RDI: ffff88087fd6c200
Jul 26 06:01:14 shrek kernel: RBP: ffff880469da7e88 R08: 0000000000000000 R09: 0000000000000000
Jul 26 06:01:14 shrek kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880469d945b0
Jul 26 06:01:14 shrek kernel: R13: ffff880469d945b0 R14: ffffffff81afa5c8 R15: ffff880469d945b0
Jul 26 06:01:14 shrek kernel: FS: 0000000000000000(0000) GS:ffff88087fd60000(0000) knlGS:0000000000000000
Jul 26 06:01:14 shrek kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 26 06:01:14 shrek kernel: CR2: ffffffffff600400 CR3: 00000000015e6000 CR4: 00000000001607f0
Jul 26 06:01:14 shrek kernel: Stack:
Jul 26 06:01:14 shrek kernel: ffff880469da7e98 ffffffff8100c6b6 ffff880469da7f08 ffffffff810897b9
Jul 26 06:01:14 shrek kernel: 0000000000000017 ffff880469d945a8 ffff880469d945b0 ffff880469d945a8
Jul 26 06:01:14 shrek kernel: 0043fd9fb1e28232 a8f790adc213c71a ffff880469da7ef8 0000000000000017
Jul 26 06:01:14 shrek kernel: Call Trace:
Jul 26 06:01:14 shrek kernel: [<ffffffff8100c6b6>] arch_cpu_idle+0x16/0x20
Jul 26 06:01:14 shrek kernel: [<ffffffff810897b9>] cpu_startup_entry+0xc9/0x1b0
Jul 26 06:01:14 shrek kernel: [<ffffffff8102a928>] start_secondary+0x238/0x350
Jul 26 06:01:14 shrek kernel: Code: 31 c0 eb d7 90 0f 20 e0 48 83 e0 fb 0f 22 e0 31 c0 eb c8 66 90 55 48 89 e5 ff 15 16 b0 87 00 c9 c3 0f 1f 40 00 55 48 89 e5 fb f4 <c9> c3 0f 1f 84 00 00 00 00 00 55 8b 05 61 f6 ae 00 48 89 e5 65
My investigation showed up that the CPUs are in the following state:
CPUS 1, 2, 3, 6, 7, 8, 9, 10, 11, 18, 21, 22 are locked on the spinlock
inside __lock_task_sighand, in a rcu_read_lock() section. Whereas
CPUS: 4, 5, 13, 14, 15, 16, 17, 19, 20, 23 are staying idle.
>From the logs I gather that 2 different subsystems detected that:
the NMI watchdog and the rcu stall detector (cpu's 8 and 18).
While looking at the code and googling around I discovered the
following commit from PeterZ: https://lkml.org/lkml/2008/11/13/157
Based on that I think could be happening is that the sighand itself is
being freed while we are in the grace period inside __lock_task_sighand
but the slab page itself is not freed as per the semantics of
SLAB_DESTROY_BY_RCU. I looked up the source of this function in the
latest kernels and saw that Oleg had put a comment clarifying the
semantics but I'm still not convinced that it is safe. What if
we are trying to lock the spinlock before this particular slab is
initialised with sighand_ctor? Looking at the code and reading
Peter's explanation of SLAB_RCU I cannot identify which portion
of __lock_task_sighand implements the following logic:
if (!try_get_ref(obj)) // might fail for free objects
goto again;
Also what's more peculiar was the fact that when "Shrek" experienced
this issue, resources were migrated to the 2nd machine and it
also experienced the same behavior - CPUs locked on the sighand
spinlock.
That's the first time I've seen this particular issue. And I'm not
really able to come up with a way where a task's sighand could be
corrupted in a such a way as to prevent acquiring the lock.
Regards,
Nikolay
next reply other threads:[~2015-07-27 12:54 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-07-27 12:54 Nikolay Borisov [this message]
2015-08-03 16:49 ` HARD LOCKUP: Strange hard lock up on spin_lock(&sighand->siglock); Steven Rostedt
2015-08-03 17:04 ` Oleg Nesterov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=55B629E9.6020207@kyup.com \
--to=kernel@kyup.com \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.