Re: [RFC][PATCH 00/16] sched: Core scheduling

Subhra Mazumdar
2019-03-01T02:56:59Z

On 2/18/19 8:56 AM, Peter Zijlstra wrote:
> A much 'demanded' feature: core-scheduling :-(
>
> I still hate it with a passion, and that is part of why it took a little
> longer than 'promised'.
>
> While this one doesn't have all the 'features' of the previous (never
> published) version and isn't L1TF 'complete', I tend to like the structure
> better (relatively speaking: I hate it slightly less).
>
> This one is sched class agnostic and therefore, in principle, doesn't horribly
> wreck RT (in fact, RT could 'ab'use this by setting 'task->core_cookie = task'
> to force-idle siblings).
>
> Now, as hinted by that, there are semi sane reasons for actually having this.
> Various hardware features like Intel RDT - Memory Bandwidth Allocation, work
> per core (due to SMT fundamentally sharing caches) and therefore grouping
> related tasks on a core makes it more reliable.
>
> However; whichever way around you turn this cookie; it is expensive and nasty.
>
I am seeing the following hard lockup frequently now. Following is full
kernel output:

[ 5846.412296] drop_caches (8657): drop_caches: 3
[ 5846.624823] drop_caches (8658): drop_caches: 3
[ 5850.604641] hugetlbfs: oracle (8671): Using mlock ulimits for SHM_HUGETL
B is deprecated
[ 5962.930812] NMI watchdog: Watchdog detected hard LOCKUP on cpu 32
[ 5962.930814] Modules linked in: drbd lru_cache autofs4 cpufreq_powersave
ipv6 crc_ccitt mxm_wmi iTCO_wdt iTCO_vendor_support btrfs raid6_pq
zstd_compress zstd_decompress xor pcspkr i2c_i801 lpc_ich mfd_core ioatdma
ixgbe dca mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler wmi
pcc_cpufreq acpi_pad ext4 fscrypto jbd2 mbcache sd_mod ahci libahci nvme
nvme_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[ 5962.930828] CPU: 32 PID: 10333 Comm: oracle_10333_tp Not tainted
5.0.0-rc7core_sched #1
[ 5962.930828] Hardware name: Oracle Corporation ORACLE SERVER
X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5962.930829] RIP: 0010:try_to_wake_up+0x98/0x470
[ 5962.930830] Code: 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 8b 43
3c 8b 73 60 85 f6 0f 85 a6 01 00 00 8b 43 38 85 c0 74 09 f3 90 8b 43 38
<85> c0 75 f7 48 8b 43 10 a8 02 b8 00 00 00 00 0f 85 d5 01 00 00 0f
[ 5962.930831] RSP: 0018:ffffc9000f4dbcb8 EFLAGS: 00000002
[ 5962.930832] RAX: 0000000000000001 RBX: ffff88dfb4af1680 RCX:
0000000000000041
[ 5962.930832] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
ffff88dfb4af214c
[ 5962.930833] RBP: 0000000000000000 R08: 0000000000000001 R09:
ffffc9000f4dbd80
[ 5962.930833] R10: ffff888000000000 R11: ffffea00f0003d80 R12:
ffff88dfb4af214c
[ 5962.930834] R13: 0000000000000001 R14: 0000000000000046 R15:
0000000000000001
[ 5962.930834] FS:  00007ff4fabd9ae0(0000) GS:ffff88dfbe280000(0000)
knlGS:0000000000000000
[ 5962.930834] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5962.930835] CR2: 0000000f4cc84000 CR3: 0000003b93d36002 CR4:
00000000003606e0
[ 5962.930835] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5962.930836] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 5962.930836] Call Trace:
[ 5962.930837]  ? __switch_to_asm+0x34/0x70
[ 5962.930837]  ? __switch_to_asm+0x40/0x70
[ 5962.930838]  ? __switch_to_asm+0x34/0x70
[ 5962.930838]  autoremove_wake_function+0x11/0x50
[ 5962.930838]  __wake_up_common+0x8f/0x160
[ 5962.930839]  ? __switch_to_asm+0x40/0x70
[ 5962.930839]  __wake_up_common_lock+0x7c/0xc0
[ 5962.930840]  pipe_write+0x24e/0x3f0
[ 5962.930840]  __vfs_write+0x127/0x1b0
[ 5962.930840]  vfs_write+0xb3/0x1b0
[ 5962.930841]  ksys_write+0x52/0xc0
[ 5962.930841]  do_syscall_64+0x5b/0x170
[ 5962.930842]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5962.930842] RIP: 0033:0x3b5900e7b0
[ 5962.930843] Code: 97 20 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90
90 90 90 90 90 90 90 90 83 3d f1 db 20 00 00 75 10 b8 01 00 00 00 0f 05
<48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 5e fa ff ff 48 89 04 24
[ 5962.930843] RSP: 002b:00007ffedbcd93a8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 5962.930844] RAX: ffffffffffffffda RBX: 00007ff4faa86e24 RCX:
0000003b5900e7b0
[ 5962.930845] RDX: 000000000000028f RSI: 00007ff4faa9688e RDI:
000000000000000a
[ 5962.930845] RBP: 00007ffedbcd93c0 R08: 00007ffedbcd9458 R09:
0000000000000020
[ 5962.930846] R10: 0000000000000000 R11: 0000000000000246 R12:
00007ffedbcd9458
[ 5962.930847] R13: 00007ff4faa9688e R14: 00007ff4faa89cc8 R15:
00007ff4faa86bd0
[ 5962.930847] Kernel panic - not syncing: Hard LOCKUP
[ 5962.930848] CPU: 32 PID: 10333 Comm: oracle_10333_tp Not tainted
5.0.0-rc7core_sched #1
[ 5962.930848] Hardware name: Oracle Corporation ORACLE
SERVER X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5962.930849] Call Trace:
[ 5962.930849]  <NMI>
[ 5962.930849]  dump_stack+0x5c/0x7b
[ 5962.930850]  panic+0xfe/0x2b2
[ 5962.930850]  nmi_panic+0x35/0x40
[ 5962.930851]  watchdog_overflow_callback+0xef/0x100
[ 5962.930851]  __perf_event_overflow+0x5a/0xe0
[ 5962.930852]  handle_pmi_common+0x1d1/0x280
[ 5962.930852]  ? __set_pte_vaddr+0x32/0x50
[ 5962.930852]  ? __set_pte_vaddr+0x32/0x50
[ 5962.930853]  ? set_pte_vaddr+0x3c/0x60
[ 5962.930853]  ? intel_pmu_handle_irq+0xad/0x170
[ 5962.930854]  intel_pmu_handle_irq+0xad/0x170
[ 5962.930854]  perf_event_nmi_handler+0x2e/0x50
[ 5962.930854]  nmi_handle+0x6f/0x120
[ 5962.930855]  default_do_nmi+0xee/0x110
[ 5962.930855]  do_nmi+0xe5/0x130
[ 5962.930856]  end_repeat_nmi+0x16/0x50
[ 5962.930856] RIP: 0010:try_to_wake_up+0x98/0x470
[ 5962.930857] Code: 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 44 00 00 8b 43
3c 8b 73 60 85 f6 0f 85 a6 01 00 00 8b 43 38 85 c0 74 09 f3 90 8b 43 38
<85> c0 75 f7 48 8b 43 10 a8 02 b8 00 00 00 00 0f 85 d5 01 00 00 0f
[ 5962.930857] RSP: 0018:ffffc9000f4dbcb8 EFLAGS: 00000002
[ 5962.930858] RAX: 0000000000000001 RBX: ffff88dfb4af1680 RCX:
0000000000000041
[ 5962.930859] RDX: 0000000000000001 RSI: 0000000000000000 RDI:
ffff88dfb4af214c
[ 5962.930859] RBP: 0000000000000000 R08: 0000000000000001 R09:
ffffc9000f4dbd80
[ 5962.930859] R10: ffff888000000000 R11: ffffea00f0003d80 R12:
ffff88dfb4af214c
[ 5962.930860] R13: 0000000000000001 R14: 0000000000000046 R15:
0000000000000001
[ 5962.930860]  ? try_to_wake_up+0x98/0x470
[ 5962.930861]  ? try_to_wake_up+0x98/0x470
[ 5962.930861]  </NMI>
[ 5962.930862]  ? __switch_to_asm+0x34/0x70
[ 5962.930862]  ? __switch_to_asm+0x40/0x70
[ 5962.930862]  ? __switch_to_asm+0x34/0x70
[ 5962.930863]  autoremove_wake_function+0x11/0x50
[ 5962.930863]  __wake_up_common+0x8f/0x160
[ 5962.930864]  ? __switch_to_asm+0x40/0x70
[ 5962.930864]  __wake_up_common_lock+0x7c/0xc0
[ 5962.930864]  pipe_write+0x24e/0x3f0
[ 5962.930865]  __vfs_write+0x127/0x1b0
[ 5962.930865]  vfs_write+0xb3/0x1b0
[ 5962.930866]  ksys_write+0x52/0xc0
[ 5962.930866]  do_syscall_64+0x5b/0x170
[ 5962.930866]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5962.930867] RIP: 0033:0x3b5900e7b0
[ 5962.930868] Code: 97 20 00 31 d2 48 29 c2 64 89 11 48 83 c8 ff eb ea 90
90 90 90 90 90 90 90 90 83 3d f1 db 20 00 00 75 10 b8 01 00 00 00 0f 05
<48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 5e fa ff ff 48 89 04 24
[ 5962.930868] RSP: 002b:00007ffedbcd93a8 EFLAGS: 00000246 ORIG_RAX:
0000000000000001
[ 5962.930869] RAX: ffffffffffffffda RBX: 00007ff4faa86e24 RCX:
0000003b5900e7b0
[ 5962.930869] RDX: 000000000000028f RSI: 00007ff4faa9688e RDI:
000000000000000a
[ 5962.930870] RBP: 00007ffedbcd93c0 R08: 00007ffedbcd9458 R09:
0000000000000020
[ 5962.930870] R10: 0000000000000000 R11: 0000000000000246 R12:
00007ffedbcd9458
[ 5962.930871] R13: 00007ff4faa9688e R14: 00007ff4faa89cc8 R15:
00007ff4faa86bd0
[ 5963.987766] NMI watchdog: Watchdog detected hard LOCKUP on cpu 11
[ 5963.987767] Modules linked in: drbd lru_cache autofs4 cpufreq_powersave
ipv6 crc_ccitt mxm_wmi iTCO_wdt iTCO_vendor_support btrfs raid6_pq
zstd_compress zstd_decompress xor pcspkr i2c_i801 lpc_ich mfd_core ioatdma
ixgbe dca mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler wmi
pcc_cpufreq acpi_pad ext4 fscrypto jbd2 mbcache sd_mod ahci libahci nvme
nvme_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[ 5963.987775] CPU: 11 PID: 8805 Comm: ora_lg02_tpcc1 Not tainted
5.0.0-rc7core_sched #1
[ 5963.987775] Hardware name: Oracle Corporation ORACLE SERVER
X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5963.987776] RIP: 0010:native_queued_spin_lock_slowpath+0x183/0x1e0
[ 5963.987777] Code: 48 c1 ee 0b 83 e8 01 83 e6 60 48 98 48 81 c6 00 3a 02
00 48 03 34 c5 20 98 13 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08
<85> c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 8b 07 66
[ 5963.987777] RSP: 0018:ffffc90023003760 EFLAGS: 00000046
[ 5963.987778] RAX: 0000000000000000 RBX: 0000000000000002 RCX:
0000000000300000
[ 5963.987778] RDX: ffff88afbf2e3a00 RSI: ffff88dfbeae3a00 RDI:
ffff88dfbe1a2d40
[ 5963.987779] RBP: ffff88dfbe1a2d40 R08: 0000000000300000 R09:
00000fffffc00000
[ 5963.987779] R10: ffffc90023003778 R11: ffff88afb77b3340 R12:
000000000000001c
[ 5963.987779] R13: 0000000000022d40 R14: 0000000000000000 R15:
000000000000001c
[ 5963.987780] FS:  00007f4e14e73ae0(0000) GS:ffff88afbf2c0000(0000)
knlGS:0000000000000000
[ 5963.987780] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5963.987780] CR2: 00007fe503647850 CR3: 0000000d1b1ae002 CR4:
00000000003606e0
[ 5963.987781] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5963.987781] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 5963.987781] Call Trace:
[ 5963.987781]  _raw_spin_lock_irqsave+0x39/0x40
[ 5963.987782]  update_blocked_averages+0x32/0x610
[ 5963.987782]  update_nohz_stats+0x4d/0x60
[ 5963.987782]  update_sd_lb_stats+0x2e5/0x7d0
[ 5963.987783]  find_busiest_group+0x3e/0x5b0
[ 5963.987783]  load_balance+0x18c/0xc00
[ 5963.987783]  newidle_balance+0x278/0x490
[ 5963.987783]  __schedule+0xd16/0x1060
[ 5963.987784]  ? lock_timer_base+0x66/0x80
[ 5963.987784]  schedule+0x32/0x70
[ 5963.987784]  schedule_timeout+0x16d/0x360
[ 5963.987785]  ? __next_timer_interrupt+0xc0/0xc0
[ 5963.987785]  do_semtimedop+0x966/0x1180
[ 5963.987785]  ? xas_load+0x9/0x80
[ 5963.987786]  ? find_get_entry+0x5d/0x1e0
[ 5963.987786]  ? pagecache_get_page+0x1b4/0x2d0
[ 5963.987786]  ? __vfs_getxattr+0x2a/0x70
[ 5963.987786]  ? enqueue_task_rt+0x98/0xb0
[ 5963.987787]  ? check_preempt_curr+0x50/0x90
[ 5963.987787]  ? push_rt_tasks+0x20/0x20
[ 5963.987787]  ? ttwu_do_wakeup+0x5e/0x160
[ 5963.987788]  ? try_to_wake_up+0x54/0x470
[ 5963.987788]  ? wake_up_q+0x2d/0x70
[ 5963.987788]  ? semctl_setval+0x26d/0x400
[ 5963.987788]  ? ksys_semtimedop+0x52/0x80
[ 5963.987789]  ksys_semtimedop+0x52/0x80
[ 5963.987789]  do_syscall_64+0x5b/0x170
[ 5963.987789]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5963.987789] RIP: 0033:0x3b58ceb28a
[ 5963.987790] Code: 73 01 c3 48 8b 0d 3e 2d 2a 00 31 d2 48 29 c2 64 89 11
48 83 c8 ff eb ea 90 90 90 90 90 90 90 90 49 89 ca b8 dc 00 00 00 0f 05
<48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 0e 2d 2a 00 31 d2 48 29 c2 64
[ 5963.987790] RSP: 002b:00007fff79282ed8 EFLAGS: 00000206 ORIG_RAX:
00000000000000dc
[ 5963.987791] RAX: ffffffffffffffda RBX: ffffffffffd23940 RCX:
0000003b58ceb28a
[ 5963.987791] RDX: 0000000000000001 RSI: 00007fff792830b8 RDI:
0000000000058002
[ 5963.987791] RBP: 00007fff792830e0 R08: 0000000000000000 R09:
0000000171327788
[ 5963.987792] R10: 00007fff79283068 R11: 0000000000000206 R12:
00007fff792833c8
[ 5963.987792] R13: 0000000000058002 R14: 0000000168dc0770 R15:
0000000000000000
[ 5963.987796] Shutting down cpus with NMI
[ 5963.987796] Kernel Offset: disabled
[ 5963.987797] NMI watchdog: Watchdog detected hard LOCKUP on cpu 33
[ 5963.987797] Modules linked in: drbd lru_cache autofs4 cpufreq_powersave
ipv6 crc_ccitt mxm_wmi iTCO_wdt iTCO_vendor_support btrfs raid6_pq
zstd_compress zstd_decompress xor pcspkr i2c_i801 lpc_ich mfd_core ioatdma
ixgbe dca mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler wmi
pcc_cpufreq acpi_pad ext4 fscrypto jbd2 mbcache sd_mod ahci libahci nvme
nvme_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[ 5963.987805] CPU: 33 PID: 10303 Comm: oracle_10303_tp Not tainted
5.0.0-rc7core_sched #1
[ 5963.987806] Hardware name: Oracle Corporation ORACLE
SERVER X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5963.987806] RIP: 0010:native_queued_spin_lock_slowpath+0x180/0x1e0
[ 5963.987807] Code: c1 e8 12 48 c1 ee 0b 83 e8 01 83 e6 60 48 98 48 81
c6 00 3a 02 00 48 03 34 c5 20 98 13 82 48 89 16 8b 42 08 85 c0 75 09 f3 90
<8b> 42 08 85 c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90
[ 5963.987807] RSP: 0018:ffffc90024833980 EFLAGS: 00000046
[ 5963.987808] RAX: 0000000000000000 RBX: 0000000000000002 RCX:
0000000000880000
[ 5963.987808] RDX: ffff88dfbe2e3a00 RSI: ffff88dfbe763a00 RDI:
ffff88dfbe3e2d40
[ 5963.987809] RBP: ffff88dfbe3e2d40 R08: 0000000000880000 R09:
0000002000000000
[ 5963.987809] R10: 0000000000000004 R11: ffff88dfb6ffd2c0 R12:
0000000000000025
[ 5963.987809] R13: 0000000000022d40 R14: 0000000000000000 R15:
0000000000000025
[ 5963.987810] FS:  00007f0b7e5feae0(0000) GS:ffff88dfbe2c0000(0000)
knlGS:0000000000000000
[ 5963.987810] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5963.987810] CR2: 000000007564d0e7 CR3: 0000003debf6a001 CR4:
00000000003606e0
[ 5963.987811] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5963.987811] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 5963.987811] Call Trace:
[ 5963.987812]  _raw_spin_lock_irqsave+0x39/0x40
[ 5963.987812]  update_blocked_averages+0x32/0x610
[ 5963.987812]  update_nohz_stats+0x4d/0x60
[ 5963.987812]  update_sd_lb_stats+0x2e5/0x7d0
[ 5963.987813]  find_busiest_group+0x3e/0x5b0
[ 5963.987813]  load_balance+0x18c/0xc00
[ 5963.987813]  ? __switch_to_asm+0x40/0x70
[ 5963.987813]  ? __switch_to_asm+0x34/0x70
[ 5963.987814]  newidle_balance+0x278/0x490
[ 5963.987814]  __schedule+0xd16/0x1060
[ 5963.987814]  ? enqueue_hrtimer+0x3a/0x90
[ 5963.987814]  schedule+0x32/0x70
[ 5963.987815]  do_nanosleep+0x81/0x180
[ 5963.987815]  hrtimer_nanosleep+0xce/0x1f0
[ 5963.987815]  ? __hrtimer_init+0xb0/0xb0
[ 5963.987816]  __x64_sys_nanosleep+0x8d/0xa0
[ 5963.987816]  do_syscall_64+0x5b/0x170
[ 5963.987816]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 5963.987816] RIP: 0033:0x3b5900eff0
[ 5963.987817] Code: 73 01 c3 48 8b 0d b8 8f 20 00 31 d2 48 29 c2 64 89 11
48 83 c8 ff eb ea 90 90 83 3d b1 d3 20 00 00 75 10 b8 23 00 00 00 0f 05
<48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 1e f2 ff ff 48 89 04 24
[ 5963.987817] RSP: 002b:00007ffe359b5158 EFLAGS: 00000246 ORIG_RAX:
0000000000000023
[ 5963.987818] RAX: ffffffffffffffda RBX: 0000000000169d10 RCX:
0000003b5900eff0
[ 5963.987818] RDX: 0000000000000000 RSI: 00007ffe359b5170 RDI:
00007ffe359b5160
[ 5963.987818] RBP: 00007ffe359b51c0 R08: 0000000000000000 R09:
0000000000000000
[ 5963.987819] R10: 00000001512241f8 R11: 0000000000000246 R12:
0000000000000000
[ 5963.987819] R13: 00007ffe359b5280 R14: 00000000000005ca R15:
0000000000000000
[ 5963.987827] NMI watchdog: Watchdog detected hard LOCKUP on cpu 75
[ 5963.987828] Modules linked in: drbd lru_cache autofs4 cpufreq_powersave
ipv6 crc_ccitt mxm_wmi iTCO_wdt iTCO_vendor_support btrfs raid6_pq
zstd_compress zstd_decompress xor pcspkr i2c_i801 lpc_ich mfd_core ioatdma
ixgbe dca mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler wmi
pcc_cpufreq acpi_pad ext4 fscrypto jbd2 mbcache sd_mod ahci libahci nvme
nvme_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[ 5963.987835] CPU: 75 PID: 0 Comm: swapper/75 Not tainted
5.0.0-rc7core_sched #1
[ 5963.987836] Hardware name: Oracle Corporation
ORACLE SERVER X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5963.987836] RIP: 0010:native_queued_spin_lock_slowpath+0x5e/0x1e0
[ 5963.987837] Code: ff 75 40 f0 0f ba 2f 08 0f 82 e7 00 00 00 8b 07 30 e4
09 c6 f7 c6 00 ff ff ff 75 1b 85 f6 74 0e 8b 07 84 c0 74 08 f3 90 8b 07
<84> c0 75 f8 b8 01 00 00 00 66 89 07 c3 81 e6 00 ff 00 00 75 04 c6
[ 5963.987837] RSP: 0000:ffffc9000c77bd78 EFLAGS: 00000002
[ 5963.987838] RAX: 0000000001240101 RBX: 000000000000004b RCX:
0000000000000001
[ 5963.987838] RDX: ffff88dfbe522d40 RSI: 0000000000000001 RDI:
ffff88dfbe262d40
[ 5963.987838] RBP: 0000000000022d40 R08: 0000000000000000 R09:
0000000000000001
[ 5963.987839] R10: 0000000000000001 R11: 0000000000000001 R12:
ffff88dfb4e40000
[ 5963.987839] R13: ffff88dfbe7e2d40 R14: 000000000000002a R15:
ffff88dfbe522d40
[ 5963.987840] FS:  0000000000000000(0000) GS:ffff88dfbe7c0000(0000)
knlGS:0000000000000000
[ 5963.987840] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5963.987840] CR2: 0000000f4b348000 CR3: 0000003c45ed6001 CR4:
00000000003606e0
[ 5963.987841] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5963.987841] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 5963.987841] Call Trace:
[ 5963.987842]  _raw_spin_lock+0x24/0x30
[ 5963.987842]  sched_core_balance+0x15c/0x4f0
[ 5963.987842]  __balance_callback+0x49/0xa0
[ 5963.987843]  __schedule+0xdc0/0x1060
[ 5963.987843]  schedule_idle+0x28/0x40
[ 5963.987843]  do_idle+0x164/0x260
[ 5963.987843]  cpu_startup_entry+0x19/0x20
[ 5963.987844]  start_secondary+0x17d/0x1d0
[ 5963.987844]  secondary_startup_64+0xa4/0xb0
[ 5963.987845] NMI watchdog: Watchdog detected hard LOCKUP on cpu 81
[ 5963.987845] Modules linked in: drbd lru_cache autofs4 cpufreq_powersave
ipv6 crc_ccitt mxm_wmi iTCO_wdt iTCO_vendor_support btrfs raid6_pq
zstd_compress zstd_decompress xor pcspkr i2c_i801 lpc_ich mfd_core ioatdma
ixgbe dca mdio sg ipmi_ssif i2c_core ipmi_si ipmi_msghandler wmi
pcc_cpufreq acpi_pad ext4 fscrypto jbd2 mbcache sd_mod ahci libahci nvme
nvme_core megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[ 5963.987853] CPU: 81 PID: 0 Comm: swapper/81 Not tainted
5.0.0-rc7core_sched #1
[ 5963.987854] Hardware name: Oracle Corporation ORACLE SERVER
X6-2L/ASM,MOBO TRAY,2U, BIOS 39050100 08/30/2016
[ 5963.987854] RIP: 0010:native_queued_spin_lock_slowpath+0x183/0x1e0
[ 5963.987854] Code: 48 c1 ee 0b 83 e8 01 83 e6 60 48 98 48 81 c6 00 3a 02
00 48 03 34 c5 20 98 13 82 48 89 16 8b 42 08 85 c0 75 09 f3 90 8b 42 08
<85> c0 74 f7 48 8b 32 48 85 f6 74 07 0f 0d 0e eb 02 f3 90 8b 07 66
[ 5963.987855] RSP: 0000:ffffc9000c7abd78 EFLAGS: 00000046
[ 5963.987855] RAX: 0000000000000000 RBX: 0000000000000051 RCX:
0000000001480000
[ 5963.987856] RDX: ffff88dfbe963a00 RSI: ffff88dfbe523a00 RDI:
ffff88dfbe522d40
[ 5963.987856] RBP: 0000000000022d40 R08: 0000000001480000 R09:
0000000000000001
[ 5963.987856] R10: ffff88dfb7a41680 R11: 0000000000000001 R12:
0000000000000001
[ 5963.987857] R13: ffff88dfbe962d40 R14: 0000000000000056 R15:
ffff88dfbeaa2d40
[ 5963.987857] FS:  0000000000000000(0000) GS:ffff88dfbe940000(0000)
knlGS:0000000000000000
[ 5963.987857] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5963.987857] CR2: 0000000f4a7a8000 CR3: 0000003b67e2a002 CR4:
00000000003606e0
[ 5963.987858] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 5963.987858] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 5963.987858] Call Trace:
[ 5963.987859]  _raw_spin_lock+0x24/0x30
[ 5963.987859]  sched_core_balance+0x15c/0x4f0
[ 5963.987859]  __balance_callback+0x49/0xa0
[ 5963.987859]  __schedule+0xdc0/0x1060
[ 5963.987860]  schedule_idle+0x28/0x40
[ 5963.987860]  do_idle+0x164/0x260
[ 5963.987860]  cpu_startup_entry+0x19/0x20
[ 5963.987860]  start_secondary+0x17d/0x1d0
[ 5963.987861]  secondary_startup_64+0xa4/0xb0
[ 5983.129164] ---[ end Kernel panic - not syncing: Hard LOCKUP ]---