[4.9-rc5] kernel BUG at kernel/sched/rt.c:764!

[CAI Qian <caiqian@redhat.com>]

Occasionally, this machine hit it during boot with this config.

http://people.redhat.com/qcai/tmp/config-god-4.9rc2

[   18.125103] x2apic enabled
[   18.128182] Switched APIC routing to cluster x2apic.
[   18.137063] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[   18.153805] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[   18.165021] Performance Events: PEBS fmt2+, Broadwell events, 16-deep LBR, full-width counters, Intel PMU driver.
[   18.176595] ... version:                3
[   18.181074] ... bit width:              48
[   18.185647] ... generic registers:      4
[   18.190124] ... value mask:             0000ffffffffffff
[   18.196055] ... max period:             0000ffffffffffff
[   18.201986] ... fixed-purpose events:   3
[   18.206453] ... event mask:             000000070000000f
[   20.648609] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[   20.702972] x86: Booting SMP configuration:
[   20.712720] .... node  #0, CPUs:        #1[   20.847790]   #2[   20.974935]   #3[   21.109503]   #4[   21.245976]   #5[   21.383743]   #6[   21.554680]   #7[   21.703806]   #8[   21.864885]   #9[   22.018063]  #10[   22.154530]  #11[   22.345902]  #12[   22.523560]  #13[   22.661047]  #14[   22.821751]  #15[   22.999171]  #16[   23.142056]  #17[   23.315885]  #18[   23.450304]  #19[   23.642422]  #20[   23.816793]  #21[   23.955838] .... node  #1, CPUs:   #22[   24.166606]  #23[   24.340859]  #24[   24.501884]  #25[   24.679949]  #26[   24.839650]  #27[   25.014436]  #28[   25.179093]  #29[   25.319094]  #30[   25.482463]  #31[   25.636126]  #32[   25.820521]  #33[   25.983310]  #34[   26.162576]  #35[   26.326769]  #36[   26.508100]  #37[   26.672296]  #38[   26.847331]  #39[   27.011591]  #40[   27.154124]  #41[   27.314263]  #42[   27.472972]  #43[   27.661509] .... node  #0, CPUs:   #44[   27.827697]  #45[   28.009301]  #46[   28.173231]  #47[   28.353749]  #48[   28.517099]  #49[   28.686097]  #50[   28.850425]  #51[   28.928408]  #52[   29.006194]  #53[   29.084035]  #54[   29.161891]  #55[   29.239825]  #56[   29.317658]  #57[   29.395585]  #58[   29.473428]  #59[   29.551326]  #60[   29.629383]  #61[   29.707235]  #62[   29.785018]  #63[   29.862918]  #64[   29.940800]  #65[   30.018569] .... node  #1, CPUs:   #66[   30.098050]  #67[   30.175751]  #68[   30.253356]  #69[   30.331126]  #70[   30.408855]  #71[   30.486657]  #72[   30.564568]  #73[   30.642370]  #74[   30.720135]  #75[   30.798071]  #76[   30.875768]  #77[   30.953472]  #78[   31.031228]  #79[   31.109028]  #80[   31.186751]  #81[   31.264504]  #82[   31.342254]  #83[   31.420027]  #84[   31.497807]  #85[   31.575565]  #86[   31.653323]  #87[   31.720946] x86: Booted up 2 nodes, 88 CPUs
[   31.725672] ----------------
[   31.728884] | NMI testsuite:
[   31.732102] --------------------
[   31.735706]   remote IPI:  ok  |
[   31.749619]    local IPI:  ok  |
[   31.765645] --------------------
[   31.769257] Good, all   2 testcases passed! |
[   31.774148] ---------------------------------
[   31.779019] smpboot: Total of 88 processors activated (391240.60 BogoMIPS)
[   32.277215] perf: interrupt took too long (7702 > 6366), lowering kernel.perf_event_max_sample_rate to 25000
[   32.277237] INFO: NMI handler (perf_event_nmi_handler) took too long to run: 1.174 msecs
[   32.316901] ------------[ cut here ]------------
[   32.322058] kernel BUG at kernel/sched/rt.c:764!
[   32.327210] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC KASAN
[   32.334593] Modules linked in:
[   32.338013] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.9.0-rc5+ #2
[   32.345008] Hardware name: Intel Corporation S2600WTT/S2600WTT, BIOS GRRFSDP1.86B.0271.R00.1510301446 10/30/2015
[   32.356367] task: ffff880e3f278000 task.stack: ffff880848280000
[   32.362973] RIP: 0010:[<ffffffff812ff2b5>]  [<ffffffff812ff2b5>] rq_offline_rt+0x6b5/0xda0
[   32.372217] RSP: 0000:ffff8808482878a8  EFLAGS: 00010082
[   32.378144] RAX: 0000000000000007 RBX: fffffffffd050f80 RCX: 1ffffffff0675688
[   32.386109] RDX: 0000000000000000 RSI: ffffffff833ab440 RDI: ffff880e3f278cd4
[   32.394073] RBP: ffff880848287958 R08: 0000000000000000 R09: ffff880e575e2950
[   32.402037] R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000058
[   32.410001] R13: ffffffff857390c4 R14: ffffffff85f60100 R15: dffffc0000000000
[   32.417965] FS:  0000000000000000(0000) GS:ffff88085a600000(0000) knlGS:0000000000000000
[   32.426997] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   32.433409] CR2: ffff881077fff000 CR3: 0000000003610000 CR4: 00000000003406f0
[   32.441373] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   32.449337] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   32.457301] Stack:
[   32.459544]  ffffffff85f607f8 ffffffff85f60210 ffff880e575e2a40 ffff880e575e2aa0
[   32.467842]  ffffffff85f60100 ffff880e575e2930 ffffffff85f60218 ffff880e575e2040
[   32.476140]  ffff880e575e2938 ffff880e575e2040 ffff880e575e2aa0 ffff880e575e2040
[   32.484436] Call Trace:
[   32.487169]  [<ffffffff81283331>] set_rq_offline+0xa1/0x290
[   32.493391]  [<ffffffff81285dee>] rq_attach_root+0x2fe/0x6d0
[   32.499709]  [<ffffffff8128668a>] cpu_attach_domain+0x4ca/0x1be0
[   32.506413]  [<ffffffff812861c0>] ? rq_attach_root+0x6d0/0x6d0
[   32.512925]  [<ffffffff81df6a97>] ? debug_smp_processor_id+0x17/0x20
[   32.520020]  [<ffffffff813abc25>] ? rcu_is_watching+0x15/0x130
[   32.526532]  [<ffffffff812a8f56>] build_sched_domains+0x2e86/0x5340
[   32.533531]  [<ffffffff812a60d0>] ? build_sched_domain+0x1620/0x1620
[   32.540624]  [<ffffffff81802c65>] ? __kmalloc_node+0x175/0x400
[   32.547136]  [<ffffffff81d4b411>] ? alloc_cpumask_var_node+0x51/0x100
[   32.554318]  [<ffffffff81d4b4ee>] ? alloc_cpumask_var+0xe/0x10
[   32.560831]  [<ffffffff85984cf1>] sched_init_smp+0xbf3/0xdb0
[   32.567149]  [<ffffffff859840fe>] ? trace_event_define_fields_sched_process_template+0x91/0x91
[   32.576766]  [<ffffffff8592bf91>] kernel_init_freeable+0x402/0x757
[   32.583668]  [<ffffffff813413a0>] ? trace_hardirqs_on_caller+0x520/0x720
[   32.591148]  [<ffffffff8592bb8f>] ? start_kernel+0x772/0x772
[   32.597466]  [<ffffffff8127e62e>] ? preempt_count_sub+0x5e/0xe0
[   32.604076]  [<ffffffff81076eb0>] ? compat_start_thread+0xa0/0xa0
[   32.610881]  [<ffffffff82c9ad20>] ? rest_init+0x190/0x190
[   32.616906]  [<ffffffff82c9ad33>] kernel_init+0x13/0x140
[   32.622835]  [<ffffffff82c9ad20>] ? rest_init+0x190/0x190
[   32.628863]  [<ffffffff82cb5957>] ret_from_fork+0x27/0x40
[   32.634889] Code: e8 b1 5a 9b 01 48 89 df e8 79 e9 ff ff e9 aa fa ff ff 48 8b 7d 90 4c 8b b5 70 ff ff ff e8 64 55 9b 01 48 85 db 0f 84 6d ff ff ff <0f> 0b 48 c7 c7 e0 e3 a2 83 e8 63 2b b7 00 4c 8b b5 70 ff ff ff
[   32.656589] RIP  [<ffffffff812ff2b5>] rq_offline_rt+0x6b5/0xda0
[   32.663207]  RSP <ffff8808482878a8>
[   32.667148] ---[ end trace 331ae5ac79a63af2 ]---
[   32.672301] Kernel panic - not syncing: Fatal exception
[   33.752196] Shutting down cpus with NMI
[   33.756508] ---[ end Kernel panic - not syncing: Fatal exception