Hard lockups with VMs on 4.19

[<gamanakis@gmail.com>]

From: gamanakis@gmail.com <gamanakis@gmail.com> 
Sent: Thursday, November 22, 2018 7:48 PM
To: peterz@infradead.org; linux-kernel@vger.kernel.org
Subject: Hard lockups with VMs on 4.19

I am seeing a lot of hard lockups when I fire up a KVM (Windows 10 with
VFIO), on 4.19.2 and 4.19.3 (tested from linux-stable).
System is a Xeon E5-2667v2 on Supermicro X9SRA. 
Kernel command line: “consoleblank=0 intel_iommu=on rcu_nocbs=1-15
nohz_full=1-15 transparent_hugepage=never audit=0 pcie_aspm=off”
Doing a git bisect revealed as the culprit: 

commit 9cf57731b63e37ed995b46690adc604891a9a28f
Author: Peter Zijlstra <mailto:peterz@infradead.org>
Date:   Thu Jun 7 10:52:03 2018 +0200

    watchdog/softlockup: Replace "watchdog/%u" threads with cpu_stop_work

    Oleg suggested to replace the "watchdog/%u" threads with
    cpu_stop_work. That removes one thread per CPU while at the same time
    fixes softlockup vs SCHED_DEADLINE.

    But more importantly, it does away with the single
    smpboot_update_cpumask_percpu_thread() user, which allows
    cleanups/shrinkage of the smpboot interface.

    Suggested-by: Oleg Nesterov <mailto:oleg@redhat.com>
    Signed-off-by: Peter Zijlstra (Intel) <mailto:peterz@infradead.org>
    Cc: Linus Torvalds <mailto:torvalds@linux-foundation.org>
    Cc: Peter Zijlstra <mailto:peterz@infradead.org>
    Cc: Thomas Gleixner <mailto:tglx@linutronix.de>
    Cc: mailto:linux-kernel@vger.kernel.org
    Signed-off-by: Ingo Molnar <mailto:mingo@kernel.org>

-----------------8<-----------------

NMI watchdog: Watchdog detected hard LOCKUP on cpu 10
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 10 PID: 12844 Comm: fixdep Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0010:__rcu_read_lock+0xe/0x20
Code: df 48 85 d2 40 0f 94 c5 e8 2f 2f 73 00 89 e8 5b 5d c3 66 2e 0f 1f 84
00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 40 5c 01 00 <83> 80 90 03 00 00
01 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
RSP: 0018:ffff9f0721e67c40 EFLAGS: 00000246
RAX: ffff8e13ef389e40 RBX: ffff8e13ef389e40 RCX: 0000000000000000
RDX: 0000000000000041 RSI: 00000000006080c0 RDI: ffff8e13e2373300
RBP: ffff8e13e2373300 R08: 0000000000000003 R09: 0000000000000000
R10: 0000000000000fe0 R11: 0000000000000000 R12: 00000000006080c0
R13: ffffffffaf4766aa R14: ffff8e1cbf57ea00 R15: ffff8e1cbf57ea00
FS:  00007fc6b422a540(0000) GS:ffff8e1cbfa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555bdbfdc5f8 CR3: 0000001732010004 CR4: 00000000001606e0
Call Trace:
get_mem_cgroup_from_mm+0x16/0xf0
memcg_kmem_get_cache+0x6a/0x2a0
kmem_cache_alloc+0xf7/0x1c0
__alloc_file+0x2a/0xe0
? ext4_file_mmap+0xa0/0xa0 [ext4]
alloc_empty_file+0x43/0xc0
? terminate_walk+0x91/0x100
path_openat+0x4c/0x14e0
? current_time+0x4f/0x90
? atime_needs_update+0x77/0xe0
? touch_atime+0x33/0xe0
do_filp_open+0x93/0x100
? __check_object_size+0x102/0x189
? _raw_spin_unlock+0x16/0x30
do_sys_open+0x186/0x210
do_syscall_64+0x5b/0x170
entry_SYSCALL_64_after_hwframe+0x44/0xa9
RIP: 0033:0x7fc6b41514c2
Code: 25 00 00 41 00 3d 00 00 41 00 74 4c 48 8d 05 85 70 0d 00 8b 00 85 c0
75 6d 89 f2 b8 01 01 00 00 48 89 fe bf 9c ff ff ff 0f 05 <48> 3d 00 f0 ff ff
0f 87 a2 00 00 00 48 8b 4c 24 28 64 48 33 0c 25
RSP: 002b:00007fffe98405b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000555bdbfd28b3 RCX: 00007fc6b41514c2
RDX: 0000000000000000 RSI: 0000555bdbfd28b3 RDI: 00000000ffffff9c
RBP: 0000555bdbfd28b3 R08: 0000555bdbfd28b3 R09: 00007fffe98401d0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fffe9841c54 R14: 0000000000000015 R15: 00007fffe9841c54
NMI watchdog: Watchdog detected hard LOCKUP on cpu 12
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 12 PID: 13040 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x6ae07f
Code: 85 46 ff ff ff 66 0f 1f 44 00 00 4c 8b 77 18 49 89 fd 4d 85 f6 0f 84
d0 00 00 00 49 8b 06 48 85 c0 0f 84 f4 00 00 00 48 8b 10 <48> 8b 48 08 48 85
d2 0f 84 84 01 00 00 49 89 16 48 89 4a 08 66 0f
RSP: 002b:00007fff1afce640 EFLAGS: 00000206
RAX: 00000000025baba0 RBX: 0000000000000000 RCX: 0000000000000007
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 000000000268d210
RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000025f51a0
R10: 0000000000000001 R11: 00000000024f0ca8 R12: 0000000000000080
R13: 000000000268d210 R14: 00000000024874b0 R15: 00000000024cbe40
FS:  00007f8fddc7dac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 6
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 6 PID: 14001 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x12b0328
Code: a3 ff ff 48 8b 45 38 48 8b 70 10 8b 03 8d 78 01 eb cf 41 57 41 56 49
89 f6 48 83 c6 14 41 55 41 54 55 53 48 89 fb 48 83 ec 38 <48> 8b af c0 01 00
00 e8 7c ff ff ff 48 89 df 48 89 83 c0 01 00 00
RSP: 002b:00007fffc9ea1ce0 EFLAGS: 00000202
RAX: 0000000002cb67a0 RBX: 0000000002c45590 RCX: 0000000000000001
RDX: 0000000000000035 RSI: 00007f7ed3d61df4 RDI: 0000000002c45590
RBP: 0000000000000000 R08: 000000000000023c R09: 00007f7ed515c000
R10: 000000000dd86d60 R11: 0000000000000000 R12: 00007f7ed3d3c3d8
R13: 0000000000000000 R14: 00007f7ed3d61de0 R15: 00007f7ed3d61de0
FS:  00007f7ed4688ac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 14
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 14 PID: 14175 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x7fe6e03faba0
Code: 5d 41 5c 41 5d 41 5e 41 5f c3 49 89 fd e8 48 87 fe ff 49 83 c5 02 45
0f b7 65 fe 49 89 c7 4c 8d 70 fe 48 85 db 75 10 eb cf 90 <48> 83 c5 08 48 8b
5d 00 48 85 db 74 c1 66 44 3b 23 75 ed 48 8d 7b
RSP: 002b:00007fffbc0666a0 EFLAGS: 00000283
RAX: 0000000000000008 RBX: 00007fffbc06ea62 RCX: 00000000000003e7
RDX: 0000000000004900 RSI: 0000000002a807c1 RDI: 00007fe6e05453e7
RBP: 00007fffbc06b958 R08: 0000000000000000 R09: 0000000000000000
R10: 000000001a6677ae R11: 0000000000000000 R12: 000000000000414c
R13: 00007fe6e05453e9 R14: 0000000000000006 R15: 0000000000000008
FS:  00007fe6e03beac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 2
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 2 PID: 14846 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x12aa6d1
Code: e8 64 42 30 ff 48 89 df 48 85 db 75 ec 5b c3 66 2e 0f 1f 84 00 00 00
00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 55 <53> 48 89 f3 48 83
ec 08 48 8b 57 40 48 8b 42 10 48 8b 4a 18 48 29
RSP: 002b:00007ffe4da64ca0 EFLAGS: 00000202
RAX: 00000000031b2300 RBX: 0000000000000001 RCX: 0000000000000000
RDX: 00007ffe4da64d3c RSI: 0000000000000002 RDI: 00000000030336a0
RBP: 0000000003033d70 R08: 000000000303a110 R09: 0000000000000000
R10: 000000000303a110 R11: 0000000000000000 R12: 00000000031b23de
R13: 0000000001a83040 R14: 00000000030336a0 R15: 00007ffe4da64d3c
FS:  00007f78a8eecac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 13
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 13 PID: 15862 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x12b71a1
Code: 4c 24 08 e8 41 21 03 00 49 8b 4d 18 4c 8b 4c 24 08 e9 63 ff ff ff 41
80 4d 50 02 eb 92 0f 1f 40 00 39 55 08 0f 85 f5 fc ff ff <48> 8b 7d 00 89 4c
24 20 44 89 5c 24 28 4c 89 44 24 18 48 89 54 24
RSP: 002b:00007ffdd11d70c0 EFLAGS: 00000246
RAX: 00000000000042f0 RBX: 00000000000042f0 RCX: 00000000f630c2f0
RDX: 0000000000000005 RSI: 0000000002beb4d7 RDI: 00000000029e4a70
RBP: 00007fd9671b4108 R08: 0000000002b12c30 R09: 0000000000000005
R10: 0000000000008000 R11: 0000000000008000 R12: 0000000002a43c58
R13: 00000000029e4a70 R14: 00007ffdd11d71bc R15: 0000000000007fff
FS:  00007fd967691ac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 11
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 11 PID: 16002 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x5e8bae
Code: c1 29 c3 89 d8 4d 8d 2c c7 49 8b 45 00 48 85 c0 0f 84 02 01 00 00 89
d5 49 89 f6 49 89 fc 48 83 f8 01 74 4e 48 8b 36 4c 8b 00 <4c> 89 54 24 18 89
4c 24 10 48 63 46 08 48 8b 36 4c 89 c7 44 89 4c
RSP: 002b:00007ffd42b14d60 EFLAGS: 00000206
RAX: 000000000143ee48 RBX: 00000000000000ce RCX: 0000000000000007
RDX: 0000000000166e6e RSI: 00007ffd42b14de0 RDI: 0000000002a7e810
RBP: 0000000000166e6e R08: 000000000143d459 R09: 00000000000000fb
R10: 0000000000000005 R11: 0000000000166e6e R12: 0000000002a7e810
R13: 0000000002af7e70 R14: 00007ffd42b14dd8 R15: 0000000002af7800
FS:  00007ff5800a6ac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 3
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 3 PID: 16417 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0xcdba17
Code: 00 00 00 48 89 f2 89 f5 bf 02 00 00 00 48 83 fa 01 0f 87 7d fd ff ff
eb 9a 66 2e 0f 1f 84 00 00 00 00 00 48 8b 05 71 c5 13 01 <89> d5 8b b8 ac 01
00 00 8d 47 ff 48 98 48 39 d0 0f 82 54 fd ff ff
RSP: 002b:00007ffcd5b770e0 EFLAGS: 00000287
RAX: 0000000002256390 RBX: 00007fa6e85d10a8 RCX: 0000000000000080
RDX: 0000000000000008 RSI: 0000000000000008 RDI: 0000000000000080
RBP: 000000002ca727c4 R08: 0000000000000080 R09: 0000000000000001
R10: 0000000000000008 R11: 0000000000000001 R12: 00007fa6e7d09d20
R13: 0000000000000001 R14: 0000000000000002 R15: 00007fa6e7d0c488
FS:  00007fa6e8a9cac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 15
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 15 PID: 16522 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x129dfe0
Code: 84 c0 0f 84 23 02 00 00 48 8b 4d 08 48 8d 15 a7 2f 7c 00 48 89 df 31
c0 be 06 00 00 00 e8 98 24 00 00 e9 04 02 00 00 0f 1f 00 <48> 8b 45 08 0f b6
40 10 a8 01 0f 84 80 00 00 00 d0 e8 0f b6 d0 83
RSP: 002b:00007fff2c18a540 EFLAGS: 00000246
RAX: 0000000000000035 RBX: 00000000029bf6a0 RCX: 0000000000000000
RDX: 0b5a02400b5a01c0 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 00000000029bfc48 R08: 00000000000001b3 R09: 00007f9025d4c000
R10: 000000000b5a01c0 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 00000000029bf6a0
FS:  00007f902527eac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 7
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 7 PID: 16495 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x12b2248
Code: 41 59 e9 63 ff ff ff 0f 1f 00 8b b4 24 a8 00 00 00 c6 84 24 93 00 00
00 00 85 f6 0f 84 49 ff ff ff 90 45 0f b6 9e 85 04 00 00 <3c> 50 75 8c 41 8b
4d 08 41 0f b7 55 06 8d 41 ff 48 8d 04 40 48 c1
RSP: 002b:00007ffff6bead70 EFLAGS: 00000246
RAX: 0000000000000035 RBX: 0000000000000028 RCX: 00000000023f2dc0
RDX: 00007f16bd10a160 RSI: 0000000000000000 RDI: 00007f16bbfdb7f8
RBP: 0000000002440000 R08: 0008000000000000 R09: 00007f16bdbecc10
R10: 0000000002205010 R11: 0000000000000002 R12: 0000000000000000
R13: 00007f16bd4ce090 R14: 00000000022796a0 R15: 0000000000000000
FS:  00007f16bda2bac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 5
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 5 PID: 15214 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0xa2f900
Code: 0c 48 8b 51 08 49 89 14 ef 48 83 c5 01 48 8b 5c c3 08 48 85 db 75 a8
0f 1f 00 48 85 ed 74 6b 48 83 ed 01 49 8b 1c ef eb 91 90 <4c> 8b 53 08 41 0f
b7 02 83 3c 85 40 b2 4a 01 0c 48 89 c1 75 86 83
RSP: 002b:00007fffd880d650 EFLAGS: 00000246
RAX: 0000000000000054 RBX: 00007f91fbe22ae0 RCX: 00007f91fbe22b00
RDX: 0000000000000000 RSI: 0000000000725f70 RDI: 00007f91fbe22af8
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000078
R10: 0000000000000078 R11: 0000000000000000 R12: 0000000000000001
R13: 00007fffd880d760 R14: 0000000000725f70 R15: 00007fffd880d660
FS:  00007f92050edac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 8
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 8 PID: 19549 Comm: genksyms Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x7f00f83a1b50
Code: 15 ad f7 11 00 0f 83 e2 00 00 00 48 39 f7 72 0f 74 12 4c 8d 0c 16 4c
39 cf 0f 82 63 01 00 00 48 89 d1 f3 a4 c3 80 fa 08 73 12 <80> fa 04 73 1e 80
fa 01 77 26 72 05 0f b6 0e 88 0f c3 48 8b 4c 16
RSP: 002b:00007ffdcfd167f8 EFLAGS: 00000297
RAX: 000055b0cd5eef90 RBX: 000055b0cd5eeb40 RCX: 000055b0ccd58010
RDX: 0000000000000002 RSI: 000055b0ccd585f7 RDI: 000055b0cd5eef90
RBP: 000000000000003a R08: 000055b0ccbe5860 R09: 000055b0ccd585f7
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000002
R13: 0000000000000001 R14: 000055b0ccbe5fc0 R15: 000055b0ccbe61a0
FS:  00007f00f84c3540 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 9
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 9 PID: 20039 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x67b7d2
Code: 83 c4 08 31 c0 5b 5d 41 5c 41 5d c3 0f 1f 40 00 bf 88 95 42 01 31 c0
e8 6c e6 c0 00 eb dc 66 2e 0f 1f 84 00 00 00 00 00 41 57 <41> 56 41 55 41 54
55 53 48 81 ec 88 00 00 00 48 89 0c 24 64 48 8b
RSP: 002b:00007ffc28206fb0 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000004 RCX: 00007ffc282070c8
RDX: 00007f86a1e84680 RSI: 0000000000000004 RDI: 0000000000000000
RBP: 00007f86a1e84680 R08: 0000000000000000 R09: 00007ffc282070c8
R10: 0000000007571fe8 R11: 0000000000000008 R12: 0000000007571fe8
R13: 00007f86a1f237e0 R14: 00007f86a1f31c80 R15: 00007f86a1e84680
FS:  00007f86a26d6ac0 GS:  0000000000000000
NMI watchdog: Watchdog detected hard LOCKUP on cpu 1
Modules linked in: vhost_net vhost tap tun devlink veth cfg80211 rfkill
8021q garp mrp bridge stp llc nct6775 hwmon_vid msr intel_rapl sb_edac
x86_pkg_temp_thermal intel_powerclamp snd_hda_codec_hdmi coretemp mousedev
snd_usb_audio kvm_intel snd_usbmidi_lib kvm snd_rawmidi hid_logitech_hidpp
snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul snd_hda_intel
crc32_pclmul snd_seq_device ghash_clmulni_intel pcbc snd_hda_codec
input_leds aesni_intel hid_logitech_dj snd_hda_core snd_hwdep aes_x86_64
hid_generic crypto_simd cryptd snd_pcm snd_timer snd usblp soundcore
glue_helper iTCO_wdt iTCO_vendor_support intel_cstate intel_uncore
intel_rapl_perf e1000e psmouse pcspkr ioatdma pcc_cpufreq dca i2c_i801
lpc_ich wmi evdev mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc sg
crypto_user
ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 fscrypto sd_mod
usbhid hid serio_raw atkbd libps2 isci libsas ahci scsi_transport_sas
libahci xhci_pci crc32c_intel libata xhci_hcd ehci_pci ehci_hcd i8042 serio
uas usb_storage scsi_mod vfio_pci irqbypass vfio_virqfd vfio_iommu_type1
vfio
CPU: 1 PID: 20175 Comm: cc1 Not tainted 4.19.3-9-custom19 #1
Hardware name: Supermicro X9SRA/X9SRA-3/X9SRA/X9SRA-3, BIOS 3.2 01/16/2015
RIP: 0033:0x12ae100
Code: 87 c3 0f 1f 84 00 00 00 00 00 48 63 47 28 4c 8b 4f 18 44 8b 47 24 48
89 c6 48 8d 04 80 49 8d 04 c1 39 10 76 45 44 39 c6 73 78 <42> 8d 04 06 d1 e8
89 c1 48 8d 0c 89 41 39 14 c9 76 1a eb 44 0f 1f
RSP: 002b:00007ffc4825b420 EFLAGS: 00000297
RAX: 000000000000508f RBX: 000000007ffb0bec RCX: 000000000000503f
RDX: 000000007ffb14e4 RSI: 0000000000005040 RDI: 00007fbdd9a4e000
RBP: 000000007ffb0bc2 R08: 000000000000508f R09: 00007fbdd82c7000
R10: 000000007ffb14e4 R11: 00007fbdd9a4e000 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: 000000007ffb0bea
FS:  00007fbdd8f72ac0 GS:  0000000000000000
perf: interrupt took too long (2503 > 2500), lowering
kernel.perf_event_max_sample_rate to 79800
logitech-hidpp-device 0003:046D:101B.000A: HID++ 1.0 device connected.
perf: interrupt took too long (3139 > 3128), lowering
kernel.perf_event_max_sample_rate to 63600
br0: port 2(vethPC49WT) entered disabled state
device vethPC49WT left promiscuous mode
br0: port 2(vethPC49WT) entered disabled state
e1000e: eno2 NIC Link is Down
br0: port 1(eno2) entered disabled state
device eno2 left promiscuous mode
br0: port 1(eno2) entered disabled state
systemd-shutdow: 50 output lines suppressed due to ratelimiting
shutdown[1]: Syncing filesystems and block devices.
shutdown[1]: Sending SIGTERM to remaining processes...

-----------------8<-----------------