From: Sasha Levin <sasha.levin@oracle.com>
To: Peter Zijlstra <peterz@infradead.org>, Ingo Molnar <mingo@kernel.org>
Cc: LKML <linux-kernel@vger.kernel.org>,
Dave Jones <davej@codemonkey.org.uk>,
Davidlohr Bueso <dave@stgolabs.net>,
jason.low2@hp.com
Subject: Re: sched: softlockups in multi_cpu_stop
Date: Fri, 06 Mar 2015 06:27:27 -0500 [thread overview]
Message-ID: <54F98F1F.3080107@oracle.com> (raw)
In-Reply-To: <54F41516.6060608@oracle.com>
I've bisected this to "locking/rwsem: Check for active lock before bailing on spinning". Relevant parties Cc'ed.
Thanks,
Sasha
On 03/02/2015 02:45 AM, Sasha Levin wrote:
> Hi all,
>
> I'm seeing the following lockup pretty often while fuzzing with trinity:
>
> [ 880.960250] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 447s! [migration/1:14]
> [ 880.960700] Modules linked in:
> [ 880.960700] irq event stamp: 380954
> [ 880.960700] hardirqs last enabled at (380953): restore_args (arch/x86/kernel/entry_64.S:780)
> [ 880.960700] hardirqs last disabled at (380954): apic_timer_interrupt (arch/x86/kernel/entry_64.S:920)
> [ 880.960700] softirqs last enabled at (380952): __do_softirq (./arch/x86/include/asm/preempt.h:22 kernel/softirq.c:300)
> [ 880.960700] softirqs last disabled at (380947): irq_exit (kernel/softirq.c:350 kernel/softirq.c:391)
> [ 880.960700] CPU: 1 PID: 14 Comm: migration/1 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880052cc8000 ti: ffff880052cd0000 task.ti: ffff880052cd0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880052cd7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: 0000000042506841 RCX: 1ffff1000a5992d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880052cd7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff9315e211
> [ 880.960700] R13: ffff880052cd7ae8 R14: ffffffff93220213 R15: ffff880052cd7ad8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880053200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002d88fd8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8805d4f67728 0000000000000282
> [ 880.960700] ffff880000000000 00ff880052cc8000 ffff880052cd7bd8 dffffc0000000000
> [ 880.960700] ffff880053218440 ffff8805d4f67778 ffff8805d4f67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] sending NMI to other CPUs:
> [ 881.290044] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 1.336 msecs
> [ 881.292380] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 2.083 msecs
> [ 881.297603] INFO: NMI handler (arch_trigger_all_cpu_backtrace_handler) took too long to run: 7.423 msecs
> [ 880.960700] NMI backtrace for cpu 0
> [ 880.960700] CPU: 0 PID: 11 Comm: migration/0 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88000d6e3000 ti: ffff88000d6f0000 task.ti: ffff88000d6f0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88000d6f7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0007c21ee5 RCX: 1ffff10001adc8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88000d6f7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88003e10f738 R14: 0000000000000001 R15: ffff88003e10f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880029000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002d31fb0 CR3: 000000002082c000 CR4: 00000000000007b0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88003e10f728 0000000000000282
> [ 880.960700] ffff880000000000 00ff88000d6e3000 ffff88000d6f7bd8 dffffc0000000000
> [ 880.960700] ffff880029018440 ffff88003e10f778 ffff88003e10f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 2
> [ 880.960700] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802ccca0000 ti: ffff88000dee8000 task.ti: ffff88000dee8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000deefd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000deefd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000deeffd8 R14: 0000000000000000 R15: ffff88000deeffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88007d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f6baacd10f8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000deefd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000deeffd8 0000000000000000 ffff88000deefd78 ffffffff931629ef
> [ 880.960700] ffff88000deefe68 ffffffff934018e0 ffff88000deefde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 3
> [ 880.960700] CPU: 3 PID: 32 Comm: migration/3 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8800a6cc0000 ti: ffff8800a6cc8000 task.ti: ffff8800a6cc8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8800a6ccfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0067e09ee5 RCX: 1ffff10014d982d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8800a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88033f04f738 R14: 0000000000000001 R15: ffff88033f04f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800a7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001490fe8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88033f04f728 0000000000000282
> [ 880.960700] ffff880000000000 00ff8800a6cc0000 ffff8800a6ccfbd8 dffffc0000000000
> [ 880.960700] ffff8800a7218440 ffff88033f04f778 ffff88033f04f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 4
> [ 880.960700] CPU: 4 PID: 41 Comm: migration/4 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8800cacb3000 ti: ffff8800cacc0000 task.ti: ffff8800cacc0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8800cacc7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed006e615ee5 RCX: 1ffff100195968d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8800cacc7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8803730af738 R14: 0000000000000001 R15: ffff8803730af718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8800cf200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000189efd8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803730af728 0000000000000282
> [ 880.960700] ffff880000000000 00ff8800cacb3000 ffff8800cacc7bd8 dffffc0000000000
> [ 880.960700] ffff8800cf218440 ffff8803730af778 ffff8803730af718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 5
> [ 880.960700] CPU: 5 PID: 50 Comm: migration/5 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880128c6b000 ti: ffff880128c88000 task.ti: ffff880128c88000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880128c8fb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008c0bbee5 RCX: 1ffff1002518d8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880128c8fbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8804605df738 R14: 0000000000000001 R15: ffff8804605df718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880129000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002bf8ff0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8804605df728 0000000000000282
> [ 880.960700] ffff880100000000 00ff880128c6b000 ffff880128c8fbd8 dffffc0000000000
> [ 880.960700] ffff880129018440 ffff8804605df778 ffff8804605df718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 6
> [ 880.960700] CPU: 6 PID: 59 Comm: migration/6 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880152cb3000 ti: ffff880152cc0000 task.ti: ffff880152cc0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880152cc7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0063d8cee5 RCX: 1ffff1002a5968d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880152cc7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88031ec67738 R14: 0000000000000001 R15: ffff88031ec67718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880153200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002b54fa0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88031ec67728 0000000000000282
> [ 880.960700] ffff880100000000 00ff880152cb3000 ffff880152cc7bd8 dffffc0000000000
> [ 880.960700] ffff880153218440 ffff88031ec67778 ffff88031ec67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 7
> [ 880.960700] CPU: 7 PID: 0 Comm: swapper/7 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8808dd0b0000 ti: ffff88000e2d0000 task.ti: ffff88000e2d0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0000:ffff88000e2d7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e2d7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e2d7fd8 R14: 0000000000000000 R15: ffff88000e2d7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88017d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000000000 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e2d7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e2d7fd8 0000000000000000 ffff88000e2d7d78 ffffffff931629ef
> [ 880.960700] ffff88000e2d7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 8
> [ 880.960700] CPU: 8 PID: 77 Comm: migration/8 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801a6cc0000 ti: ffff8801a6cc8000 task.ti: ffff8801a6cc8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8801a6ccfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed003abaaee5 RCX: 1ffff10034d982d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8801a6ccfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8801d5d57738 R14: 0000000000000001 R15: ffff8801d5d57718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801a7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000e69fc8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8801d5d57728 0000000000000282
> [ 880.960700] ffff880100000000 00ff8801a6cc0000 ffff8801a6ccfbd8 dffffc0000000000
> [ 880.960700] ffff8801a7218440 ffff8801d5d57778 ffff8801d5d57718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 9
> [ 880.960700] CPU: 9 PID: 86 Comm: migration/9 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801d0c88000 ti: ffff8801d0c90000 task.ti: ffff8801d0c90000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8801d0c97b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed007245bee5 RCX: 1ffff1003a1912d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8801d0c97bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8803922df738 R14: 0000000000000001 R15: ffff8803922df718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801d1000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029a8fb0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8803922df728 0000000000000282
> [ 880.960700] ffff880100000000 00ff8801d0c88000 ffff8801d0c97bd8 dffffc0000000000
> [ 880.960700] ffff8801d1018440 ffff8803922df778 ffff8803922df718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 10
> [ 880.960700] CPU: 10 PID: 0 Comm: swapper/10 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880128c68000 ti: ffff88000e6c0000 task.ti: ffff88000e6c0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6c7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6c7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6c7fd8 R14: 0000000000000000 R15: ffff88000e6c7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8801fb200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000636668 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6c7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6c7fd8 0000000000000000 ffff88000e6c7d78 ffffffff931629ef
> [ 880.960700] ffff88000e6c7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 11
> [ 880.960700] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801a6ca8000 ti: ffff88000e6c8000 task.ti: ffff88000e6c8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6cfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6cfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6cffd8 R14: 0000000000000000 R15: ffff88000e6cffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880225200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000030db000 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6cfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6cffd8 0000000000000000 ffff88000e6cfd78 ffffffff931629ef
> [ 880.960700] ffff88000e6cfe68 ffffffff934018e0 ffff88000e6cfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 12
> [ 880.960700] CPU: 12 PID: 113 Comm: migration/12 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88024ecd0000 ti: ffff88024ecd8000 task.ti: ffff88024ecd8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88024ecdfb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0073a32ee5 RCX: 1ffff10049d9a2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88024ecdfbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88039d197738 R14: 0000000000000001 R15: ffff88039d197718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88024f200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002648fe0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039d197728 0000000000000282
> [ 880.960700] ffff880200000000 00ff88024ecd0000 ffff88024ecdfbd8 dffffc0000000000
> [ 880.960700] ffff88024f218440 ffff88039d197778 ffff88039d197718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 13
> [ 880.960700] CPU: 13 PID: 0 Comm: swapper/13 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802f6cc0000 ti: ffff88000e6d8000 task.ti: ffff88000e6d8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000e6dfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000e6dfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000e6dffd8 R14: 0000000000000000 R15: ffff88000e6dffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880279000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000063a69c CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000e6dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000e6dffd8 0000000000000000 ffff88000e6dfd78 ffffffff931629ef
> [ 880.960700] ffff88000e6dfe68 ffffffff934018e0 ffff88000e6dfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 14
> [ 880.960700] CPU: 14 PID: 131 Comm: migration/14 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802a2cdb000 ti: ffff8802a2ce8000 task.ti: ffff8802a2ce8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802a2cefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0059614ee5 RCX: 1ffff1005459b8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802a2cefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8802cb0a7738 R14: 0000000000000001 R15: ffff8802cb0a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802a3200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002cc8ff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802cb0a7728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802a2cdb000 ffff8802a2cefbd8 dffffc0000000000
> [ 880.960700] ffff8802a3218440 ffff8802cb0a7778 ffff8802cb0a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 15
> [ 880.960700] CPU: 15 PID: 140 Comm: migration/15 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802ccca3000 ti: ffff8802cccd0000 task.ti: ffff8802cccd0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802cccd7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0023834ee5 RCX: 1ffff100599948d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802cccd7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88011c1a7738 R14: 0000000000000001 R15: ffff88011c1a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802cd200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001ad0a08 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88011c1a7728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802ccca3000 ffff8802cccd7bd8 dffffc0000000000
> [ 880.960700] ffff8802cd218440 ffff88011c1a7778 ffff88011c1a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 16
> [ 880.960700] CPU: 16 PID: 149 Comm: migration/16 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802f6ce8000 ti: ffff8802f6cf0000 task.ti: ffff8802f6cf0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8802f6cf7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149d5ee5 RCX: 1ffff1005ed9d2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8802f6cf7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8800a4eaf738 R14: 0000000000000001 R15: ffff8800a4eaf718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8802f7200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001f6afe8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4eaf728 0000000000000282
> [ 880.960700] ffff880200000000 00ff8802f6ce8000 ffff8802f6cf7bd8 dffffc0000000000
> [ 880.960700] ffff8802f7218440 ffff8800a4eaf778 ffff8800a4eaf718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 17
> [ 880.960700] CPU: 17 PID: 158 Comm: migration/17 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880320c7b000 ti: ffff880320c88000 task.ti: ffff880320c88000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880320c8fb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed000b451ee5 RCX: 1ffff1006418f8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880320c8fbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88005a28f738 R14: 0000000000000001 R15: ffff88005a28f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880321000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 000000000161bff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88005a28f728 0000000000000282
> [ 880.960700] ffff880300000000 00ff880320c7b000 ffff880320c8fbd8 dffffc0000000000
> [ 880.960700] ffff880321018440 ffff88005a28f778 ffff88005a28f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 18
> [ 880.960700] CPU: 18 PID: 0 Comm: swapper/18 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8806ad0b0000 ti: ffff88000eac0000 task.ti: ffff88000eac0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eac7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eac7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eac7fd8 R14: 0000000000000000 R15: ffff88000eac7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88034b200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000001692ac0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eac7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eac7fd8 0000000000000000 ffff88000eac7d78 ffffffff931629ef
> [ 880.960700] ffff88000eac7e68 ffffffff934018e0 ffff88000eac7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 19
> [ 880.960700] CPU: 19 PID: 0 Comm: swapper/19 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8807ed0c8000 ti: ffff88000eac8000 task.ti: ffff88000eac8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eacfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eacfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eacffd8 R14: 0000000000000000 R15: ffff88000eacffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880375200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007ffe1ae26d00 CR3: 000000000660a000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eacfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eacffd8 0000000000000000 ffff88000eacfd78 ffffffff931629ef
> [ 880.960700] ffff88000eacfe68 ffffffff934018e0 ffff88000eacfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 20
> [ 880.960700] CPU: 20 PID: 185 Comm: migration/20 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88039ece0000 ti: ffff88039ece8000 task.ti: ffff88039ece8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88039ecefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008ddccee5 RCX: 1ffff10073d9c2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88039ecefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88046ee67738 R14: 0000000000000001 R15: ffff88046ee67718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88039f200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029aeff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ee67728 0000000000000282
> [ 880.960700] ffff880300000000 00ff88039ece0000 ffff88039ecefbd8 dffffc0000000000
> [ 880.960700] ffff88039f218440 ffff88046ee67778 ffff88046ee67718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 21
> [ 880.960700] CPU: 21 PID: 194 Comm: migration/21 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803c8ca8000 ti: ffff8803c8cb0000 task.ti: ffff8803c8cb0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8803c8cb7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed00149caee5 RCX: 1ffff100791952d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8803c8cb7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8800a4e57738 R14: 0000000000000001 R15: ffff8800a4e57718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803c9000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002914fb0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8800a4e57728 0000000000000282
> [ 880.960700] ffff880300000000 00ff8803c8ca8000 ffff8803c8cb7bd8 dffffc0000000000
> [ 880.960700] ffff8803c9018440 ffff8800a4e57778 ffff8800a4e57718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 22
> [ 880.960700] CPU: 22 PID: 203 Comm: migration/22 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803f2cdb000 ti: ffff8803f2ce8000 task.ti: ffff8803f2ce8000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8803f2cefb78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed008dde3ee5 RCX: 1ffff1007e59b8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8803f2cefbd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88046ef1f738 R14: 0000000000000001 R15: ffff88046ef1f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8803f3200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000002e38b80 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88046ef1f728 0000000000000282
> [ 880.960700] ffff880300000000 00ff8803f2cdb000 ffff8803f2cefbd8 dffffc0000000000
> [ 880.960700] ffff8803f3218440 ffff88046ef1f778 ffff88046ef1f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 23
> [ 880.960700] CPU: 23 PID: 212 Comm: migration/23 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88041cc7b000 ti: ffff88041cce0000 task.ti: ffff88041cce0000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff88041cce7b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed006926eee5 RCX: 1ffff1008398f8d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff88041cce7bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff880349377738 R14: 0000000000000001 R15: ffff880349377718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88041d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f9c055599d0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff880349377728 0000000000000282
> [ 880.960700] ffff880400000000 00ff88041cc7b000 ffff88041cce7bd8 dffffc0000000000
> [ 880.960700] ffff88041d218440 ffff880349377778 ffff880349377718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 24
> [ 880.960700] CPU: 24 PID: 0 Comm: swapper/24 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8801d0c68000 ti: ffff88000eed0000 task.ti: ffff88000eed0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eed7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eed7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eed7fd8 R14: 0000000000000000 R15: ffff88000eed7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880447200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029e4220 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eed7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eed7fd8 0000000000000000 ffff88000eed7d78 ffffffff931629ef
> [ 880.960700] ffff88000eed7e68 ffffffff934018e0 ffff88000eed7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 25
> [ 880.960700] CPU: 25 PID: 12438 Comm: trinity-c68 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8802cb318000 ti: ffff8802cb3a8000 task.ti: ffff8802cb3a8000
> [ 880.960700] RIP: rwsem_down_write_failed (./arch/x86/include/asm/processor.h:658 ./arch/x86/include/asm/processor.h:663 kernel/locking/rwsem-xadd.c:413 kernel/locking/rwsem-xadd.c:442)
> [ 880.960700] RSP: 0018:ffff8802cb3afc08 EFLAGS: 00000286
> [ 880.960700] RAX: ffffffff00000001 RBX: ffffffffa18e12a8 RCX: ffff8802cb318000
> [ 880.960700] RDX: fffffffe00000001 RSI: fffffffeffffffff RDI: 0000000100000000
> [ 880.960700] RBP: ffff8802cb3afd98 R08: 0000000000000000 R09: ffffffff00000001
> [ 880.960700] R10: 0000000000000001 R11: 0000000000000001 R12: ffff8803f10c3000
> [ 880.960700] R13: ffffffffa18e12a0 R14: ffff8802cb3afd68 R15: ffffffffa18e1250
> [ 880.960700] FS: 00007f9c05559700(0000) GS:ffff880471000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 880.960700] CR2: 00007f9c0549b614 CR3: 00000002cb385000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffff9dbf3ced ffff8802cb318d00 0000000000000000 ffff8802cb318000
> [ 880.960700] 00000000000004ff ffffffffa3ebf6b0 1ffff10059675f89 1ffff10059675f8d
> [ 880.960700] 0000000041b58ab3 ffff8802cb3a8010 1ffff10059675002 ffff8802cb3a8000
> [ 880.960700] Call Trace:
> [ 880.960700] ? rwsem_down_write_failed (include/linux/rcupdate.h:912 kernel/locking/rwsem-xadd.c:306 kernel/locking/rwsem-xadd.c:381 kernel/locking/rwsem-xadd.c:442)
> [ 880.960700] ? rwsem_down_read_failed (kernel/locking/rwsem-xadd.c:433)
> [ 880.960700] ? trace_hardirqs_off (kernel/locking/lockdep.c:2647)
> [ 880.960700] ? __acct_update_integrals (kernel/tsacct.c:125)
> [ 880.960700] ? taskstats_exit (kernel/tsacct.c:125)
> [ 880.960700] call_rwsem_down_write_failed (arch/x86/lib/rwsem.S:104)
> [ 880.960700] ? down_write (kernel/locking/rwsem.h:4 kernel/locking/rwsem.c:52)
> [ 880.960700] ? ipcget (ipc/util.c:349 ipc/util.c:646)
> [ 880.960700] ipcget (ipc/util.c:349 ipc/util.c:646)
> [ 880.960700] ? trace_hardirqs_on (kernel/locking/lockdep.c:2609)
> [ 880.960700] ? syscall_trace_enter_phase1 (include/linux/context_tracking.h:27 arch/x86/kernel/ptrace.c:1486)
> [ 880.960700] SyS_semget (ipc/sem.c:591 ipc/sem.c:572)
> [ 880.960700] ? sem_exit_ns (ipc/sem.c:572)
> [ 880.960700] ? trace_hardirqs_on_thunk (arch/x86/lib/thunk_64.S:42)
> [ 880.960700] tracesys_phase2 (arch/x86/kernel/entry_64.S:347)
> [ 880.960700] Code: 8d 0c 0a 48 89 d0 f0 49 0f b1 0f 48 39 c2 0f 84 2b 04 00 00 48 89 c2 48 8d 04 3a 48 85 f0 74 df 4d 85 e4 0f 84 50 04 00 00 f3 90 <e9> 57 ff ff ff e8 02 62 87 f5 84 c0 0f 85 f5 fd ff ff 48 c7 c2
> All code
> ========
> 0: 8d 0c 0a lea (%rdx,%rcx,1),%ecx
> 3: 48 89 d0 mov %rdx,%rax
> 6: f0 49 0f b1 0f lock cmpxchg %rcx,(%r15)
> b: 48 39 c2 cmp %rax,%rdx
> e: 0f 84 2b 04 00 00 je 0x43f
> 14: 48 89 c2 mov %rax,%rdx
> 17: 48 8d 04 3a lea (%rdx,%rdi,1),%rax
> 1b: 48 85 f0 test %rsi,%rax
> 1e: 74 df je 0xffffffffffffffff
> 20: 4d 85 e4 test %r12,%r12
> 23: 0f 84 50 04 00 00 je 0x479
> 29: f3 90 pause
> 2b:* e9 57 ff ff ff jmpq 0xffffffffffffff87 <-- trapping instruction
> 30: e8 02 62 87 f5 callq 0xfffffffff5876237
> 35: 84 c0 test %al,%al
> 37: 0f 85 f5 fd ff ff jne 0xfffffffffffffe32
> 3d: 48 rex.W
> 3e: c7 .byte 0xc7
> 3f: c2 .byte 0xc2
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: e9 57 ff ff ff jmpq 0xffffffffffffff5c
> 5: e8 02 62 87 f5 callq 0xfffffffff587620c
> a: 84 c0 test %al,%al
> c: 0f 85 f5 fd ff ff jne 0xfffffffffffffe07
> 12: 48 rex.W
> 13: c7 .byte 0xc7
> 14: c2 .byte 0xc2
> ...
> [ 880.960700] NMI backtrace for cpu 26
> [ 880.960700] CPU: 26 PID: 0 Comm: swapper/26 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8803c8c78000 ti: ffff88000eee0000 task.ti: ffff88000eee0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eee7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eee7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eee7fd8 R14: 0000000000000000 R15: ffff88000eee7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88049b200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f0f2bc1d5a8 CR3: 000000000678f000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eee7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eee7fd8 0000000000000000 ffff88000eee7d78 ffffffff931629ef
> [ 880.960700] ffff88000eee7e68 ffffffff934018e0 ffff88000eee7de8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 27
> [ 880.960700] CPU: 27 PID: 248 Comm: migration/27 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff8804c4cf8000 ti: ffff8804c3b00000 task.ti: ffff8804c3b00000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff8804c3b07b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed0072423ee5 RCX: 1ffff1009899f2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff8804c3b07bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff88039211f738 R14: 0000000000000001 R15: ffff88039211f718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804c5200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00000000029c4ff8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff88039211f728 0000000000000282
> [ 880.960700] ffff880400000000 00ff8804c4cf8000 ffff8804c3b07bd8 dffffc0000000000
> [ 880.960700] ffff8804c5218440 ffff88039211f778 ffff88039211f718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 28
> [ 880.960700] CPU: 28 PID: 0 Comm: swapper/28 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88065d0c0000 ti: ffff88000eef0000 task.ti: ffff88000eef0000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eef7d38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eef7d38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eef7fd8 R14: 0000000000000000 R15: ffff88000eef7fd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff8804ef200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 0000000000b34fb8 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eef7d68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eef7fd8 0000000000000000 ffff88000eef7d78 ffffffff931629ef
> [ 880.960700] ffff88000eef7e68 ffffffff934018e0 0000000000000000 0000000000000000
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 29
> [ 880.960700] CPU: 29 PID: 0 Comm: swapper/29 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88079d0b0000 ti: ffff88000eef8000 task.ti: ffff88000eef8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0018:ffff88000eeffd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000eeffd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000eefffd8 R14: 0000000000000000 R15: ffff88000eefffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880519000000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f0f2b8e5210 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000eeffd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000eefffd8 0000000000000000 ffff88000eeffd78 ffffffff931629ef
> [ 880.960700] ffff88000eeffe68 ffffffff934018e0 ffff88000eeffde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
> [ 880.960700] NMI backtrace for cpu 30
> [ 880.960700] CPU: 30 PID: 269 Comm: migration/30 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff880542cf8000 ti: ffff880541b00000 task.ti: ffff880541b00000
> [ 880.960700] RIP: multi_cpu_stop (kernel/stop_machine.c:192)
> [ 880.960700] RSP: 0000:ffff880541b07b78 EFLAGS: 00000246
> [ 880.960700] RAX: 0000000000000000 RBX: ffffed005ea54ee5 RCX: 1ffff100a859f2d5
> [ 880.960700] RDX: 0000000000000000 RSI: ffffffff9e370ae0 RDI: ffffffffa31716b4
> [ 880.960700] RBP: ffff880541b07bd8 R08: 0000000000000001 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: dffffc0000000000
> [ 880.960700] R13: ffff8802f52a7738 R14: 0000000000000001 R15: ffff8802f52a7718
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff880543200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007f42552112c0 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffffffffa15a9540 ffffffff93532e90 ffff8802f52a7728 0000000000000282
> [ 880.960700] ffff880500000000 00ff880542cf8000 ffff880541b07bd8 dffffc0000000000
> [ 880.960700] ffff880543218440 ffff8802f52a7778 ffff8802f52a7718 ffffffff935335b0
> [ 880.960700] Call Trace:
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? queue_stop_cpus_work (kernel/stop_machine.c:170)
> [ 880.960700] cpu_stopper_thread (kernel/stop_machine.c:474)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] ? cpu_stop_should_run (include/linux/list.h:189 kernel/stop_machine.c:443)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? ikconfig_read_current (kernel/stop_machine.c:437)
> [ 880.960700] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2554 kernel/locking/lockdep.c:2601)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? cpu_stop_create (kernel/stop_machine.c:449)
> [ 880.960700] smpboot_thread_fn (kernel/smpboot.c:161)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? schedule (./arch/x86/include/asm/bitops.h:311 (discriminator 1) kernel/sched/core.c:2824 (discriminator 1))
> [ 880.960700] ? __kthread_parkme (kernel/kthread.c:164)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] ? sort_range (kernel/smpboot.c:105)
> [ 880.960700] kthread (kernel/kthread.c:207)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] ret_from_fork (arch/x86/kernel/entry_64.S:565)
> [ 880.960700] ? flush_kthread_work (kernel/kthread.c:176)
> [ 880.960700] Code: eb 27 0f 1f 00 41 83 fe 03 75 0a 80 7d cf 00 0f 85 50 01 00 00 f0 41 ff 4f 24 74 71 41 83 fe 04 0f 84 d7 00 00 00 45 89 f0 f3 90 <4c> 89 e8 48 c1 e8 03 42 0f b6 04 20 84 c0 74 08 3c 03 0f 8e f5
> All code
> ========
> 0: eb 27 jmp 0x29
> 2: 0f 1f 00 nopl (%rax)
> 5: 41 83 fe 03 cmp $0x3,%r14d
> 9: 75 0a jne 0x15
> b: 80 7d cf 00 cmpb $0x0,-0x31(%rbp)
> f: 0f 85 50 01 00 00 jne 0x165
> 15: f0 41 ff 4f 24 lock decl 0x24(%r15)
> 1a: 74 71 je 0x8d
> 1c: 41 83 fe 04 cmp $0x4,%r14d
> 20: 0f 84 d7 00 00 00 je 0xfd
> 26: 45 89 f0 mov %r14d,%r8d
> 29: f3 90 pause
> 2b:* 4c 89 e8 mov %r13,%rax <-- trapping instruction
> 2e: 48 c1 e8 03 shr $0x3,%rax
> 32: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> 37: 84 c0 test %al,%al
> 39: 74 08 je 0x43
> 3b: 3c 03 cmp $0x3,%al
> 3d: 0f .byte 0xf
> 3e: 8e f5 mov %ebp,%?
> ...
>
> Code starting with the faulting instruction
> ===========================================
> 0: 4c 89 e8 mov %r13,%rax
> 3: 48 c1 e8 03 shr $0x3,%rax
> 7: 42 0f b6 04 20 movzbl (%rax,%r12,1),%eax
> c: 84 c0 test %al,%al
> e: 74 08 je 0x18
> 10: 3c 03 cmp $0x3,%al
> 12: 0f .byte 0xf
> 13: 8e f5 mov %ebp,%?
> ...
> [ 880.960700] NMI backtrace for cpu 31
> [ 880.960700] CPU: 31 PID: 0 Comm: swapper/31 Not tainted 4.0.0-rc1-next-20150227-sasha-00039-gae4a688-dirty #1971
> [ 880.960700] task: ffff88000f2d0000 ti: ffff88000f2d8000 task.ti: ffff88000f2d8000
> [ 880.960700] RIP: native_safe_halt (./arch/x86/include/asm/irqflags.h:50)
> [ 880.960700] RSP: 0000:ffff88000f2dfd38 EFLAGS: 00000282
> [ 880.960700] RAX: dffffc0000000000 RBX: dffffc0000000000 RCX: 0000000000000000
> [ 880.960700] RDX: 1ffffffff4115884 RSI: 0000000000000001 RDI: ffffffffa08ac420
> [ 880.960700] RBP: ffff88000f2dfd38 R08: 0000000000000000 R09: 0000000000000000
> [ 880.960700] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa31716b4
> [ 880.960700] R13: ffff88000f2dffd8 R14: 0000000000000000 R15: ffff88000f2dffd8
> [ 880.960700] FS: 0000000000000000(0000) GS:ffff88056d200000(0000) knlGS:0000000000000000
> [ 880.960700] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 880.960700] CR2: 00007fff144e2980 CR3: 000000002082c000 CR4: 00000000000007a0
> [ 880.960700] Stack:
> [ 880.960700] ffff88000f2dfd68 ffffffff93160774 dffffc0000000000 ffffffffa31716b4
> [ 880.960700] ffff88000f2dffd8 0000000000000000 ffff88000f2dfd78 ffffffff931629ef
> [ 880.960700] ffff88000f2dfe68 ffffffff934018e0 ffff88000f2dfde8 ffffffff9dbf7ec5
> [ 880.960700] Call Trace:
> [ 880.960700] default_idle (./arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:314)
> [ 880.960700] arch_cpu_idle (arch/x86/kernel/process.c:306)
> [ 880.960700] cpu_startup_entry (./arch/x86/include/asm/thread_info.h:162 include/linux/sched.h:2880 kernel/sched/idle.c:189 kernel/sched/idle.c:250 kernel/sched/idle.c:298)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/paravirt.h:809 include/linux/spinlock_api_smp.h:162 kernel/locking/spinlock.c:191)
> [ 880.960700] ? arch_cpu_idle_prepare (??:?)
> [ 880.960700] ? _raw_spin_unlock_irqrestore (./arch/x86/include/asm/preempt.h:95 include/linux/spinlock_api_smp.h:163 kernel/locking/spinlock.c:191)
> [ 880.960700] ? clockevents_register_device (kernel/time/clockevents.c:481)
> [ 880.960700] start_secondary (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] ? set_cpu_sibling_map (arch/x86/kernel/smpboot.c:219)
> [ 880.960700] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 00 add %al,(%rax)
> 4: 00 55 48 add %dl,0x48(%rbp)
> 7: 89 e5 mov %esp,%ebp
> 9: fa cli
> a: 5d pop %rbp
> b: c3 retq
> c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 13: 00 00
> 15: 55 push %rbp
> 16: 48 89 e5 mov %rsp,%rbp
> 19: fb sti
> 1a: 5d pop %rbp
> 1b: c3 retq
> 1c: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1)
> 23: 00 00
> 25: 55 push %rbp
> 26: 48 89 e5 mov %rsp,%rbp
> 29: fb sti
> 2a: f4 hlt
> 2b:* 5d pop %rbp <-- trapping instruction
> 2c: c3 retq
> 2d: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 34: 00
> 35: 55 push %rbp
> 36: 48 89 e5 mov %rsp,%rbp
> 39: f4 hlt
> 3a: 5d pop %rbp
> 3b: c3 retq
> 3c: 66 data16
> 3d: 0f .byte 0xf
> 3e: 1f (bad)
> 3f: 84 00 test %al,(%rax)
>
> Code starting with the faulting instruction
> ===========================================
> 0: 5d pop %rbp
> 1: c3 retq
> 2: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1)
> 9: 00
> a: 55 push %rbp
> b: 48 89 e5 mov %rsp,%rbp
> e: f4 hlt
> f: 5d pop %rbp
> 10: c3 retq
> 11: 66 data16
> 12: 0f .byte 0xf
> 13: 1f (bad)
> 14: 84 00 test %al,(%rax)
>
>
> Thanks,
> Sasha
>
next prev parent reply other threads:[~2015-03-06 11:28 UTC|newest]
Thread overview: 44+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-02 7:45 sched: softlockups in multi_cpu_stop Sasha Levin
[not found] ` <CAMiJ5CVWvUhGK=MWYB_CTNs901p=jsT4i5gkWTaHih7qdQdkFQ@mail.gmail.com>
2015-03-04 5:44 ` Rafael David Tinoco
2015-03-06 11:27 ` Sasha Levin [this message]
2015-03-06 12:32 ` Ingo Molnar
2015-03-06 14:34 ` Rafael David Tinoco
2015-03-06 14:45 ` Sasha Levin
2015-03-06 15:46 ` Sasha Levin
2015-03-06 17:19 ` Davidlohr Bueso
2015-03-06 18:02 ` Sasha Levin
2015-03-06 21:59 ` Sasha Levin
2015-03-06 18:57 ` Jason Low
2015-03-06 19:05 ` Linus Torvalds
2015-03-06 19:20 ` Davidlohr Bueso
2015-03-06 19:32 ` Linus Torvalds
2015-03-06 19:45 ` Davidlohr Bueso
2015-03-06 19:55 ` Davidlohr Bueso
2015-03-06 20:00 ` Davidlohr Bueso
2015-03-06 21:42 ` Linus Torvalds
2015-03-06 19:29 ` Jason Low
2015-03-06 21:12 ` Jason Low
2015-03-06 21:24 ` Linus Torvalds
2015-03-07 1:53 ` Jason Low
2015-03-06 22:15 ` Davidlohr Bueso
2015-03-07 1:55 ` Ming Lei
2015-03-07 2:07 ` Davidlohr Bueso
2015-03-07 2:10 ` Ming Lei
2015-03-07 2:26 ` Davidlohr Bueso
2015-03-07 2:29 ` Davidlohr Bueso
2015-03-07 2:55 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:19 ` Ming Lei
2015-03-07 3:41 ` Davidlohr Bueso
2015-03-07 2:56 ` Jason Low
2015-03-07 3:08 ` Ming Lei
2015-03-07 3:10 ` Davidlohr Bueso
2015-03-07 3:17 ` Jason Low
2015-03-07 3:39 ` Ming Lei
2015-03-07 3:53 ` Jason Low
2015-03-07 1:58 ` Jason Low
2015-03-07 4:31 ` Jason Low
2015-03-07 4:44 ` Davidlohr Bueso
2015-03-07 6:45 ` Jason Low
2015-03-07 5:54 ` Ming Lei
2015-03-07 6:57 ` Jason Low
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=54F98F1F.3080107@oracle.com \
--to=sasha.levin@oracle.com \
--cc=dave@stgolabs.net \
--cc=davej@codemonkey.org.uk \
--cc=jason.low2@hp.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.