* Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues
@ 2024-06-17 11:33 Jaroslav Pulchart
2024-06-17 13:46 ` David Sterba
0 siblings, 1 reply; 5+ messages in thread
From: Jaroslav Pulchart @ 2024-06-17 11:33 UTC (permalink / raw)
To: linux-btrfs; +Cc: Igor Raits, Jan Cipa, Zdenek Pesek, Daniel Secik
[-- Attachment #1: Type: text/plain, Size: 664 bytes --]
Hello,
We recently upgraded part of our production environment to kernel
6.9.y. Since then, we've been encountering random kernel "NULL pointer
dereference" and "soft lockup" errors when using BTRFS. These issues
occur sporadically, sometimes after several days, and I haven't been
able to reproduce them consistently. Due to this unpredictability,
bisecting is not a feasible option.
Attached are console logs from some instances of these issues:
* "NULL pointer dereference" in "btrfs.issue.1.log"
* "soft lockup" in "btrfs.issue.1.log"
Any assistance with investigating and resolving these problems would
be greatly appreciated.
Thank you,
Jaroslav Pulchart
[-- Attachment #2: btrfs.issue.1.log --]
[-- Type: text/x-log, Size: 5733 bytes --]
[1072053.323243] BUG: kernel NULL pointer dereference, address: 000000000000002c
[1072053.324629] #PF: supervisor read access in kernel mode
[1072053.325570] #PF: error_code(0x0000) - not-present page
[1072053.326495] PGD f85a13067 P4D f85a13067 PUD fdb9c0067 PMD 0
[1072053.327442] Oops: 0000 [#1] PREEMPT SMP NOPTI
[1072053.328255] CPU: 15 PID: 2354438 Comm: kworker/u195:18 Tainted: G E 6.9.3-1.gdc.el9.x86_64 #1
[1072053.329411] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[1072053.330448] Workqueue: events_unbound btrfs_preempt_reclaim_metadata_space [btrfs]
[1072053.331475] RIP: 0010:attach_eb_folio_to_filemap+0xd0/0x200 [btrfs]
[1072053.332398] Code: 86 99 00 00 00 48 8b 7d 08 48 89 ee 40 f6 c7 01 0f 85 08 01 00 00 0f 1f 44 00 00 48 89 ef 48 8b 07 f6 c4 80 74 78 48 8b 57 28 <8b> 42 2c 85 c0 74 39 8d 48 01 f0 0f b1 4a 2c 75 f2 48 85 d2 74 5e
[1072053.334584] RSP: 0018:ffffb9864f5db6f0 EFLAGS: 00010282
[1072053.335386] RAX: 01d7ffffd400822d RBX: 0000000018f75728 RCX: 0000000000000000
[1072053.336548] RDX: 0000000000000000 RSI: ffffe622db57b7c0 RDI: ffffe622db57b7c0
[1072053.337502] RBP: ffffe622db57b7c0 R08: 0000000000000001 R09: ffffe6217aeaed48
[1072053.338438] R10: ffff91507ffd6e00 R11: ffffb9864f5db690 R12: ffff91418353a5a0
[1072053.339356] R13: ffff91418353a5a0 R14: ffff916f43fb0768 R15: ffff918c87d7b000
[1072053.340276] FS: 0000000000000000(0000) GS:ffff915043b80000(0000) knlGS:0000000000000000
[1072053.341274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1072053.342103] CR2: 000000000000002c CR3: 0000000fe4938002 CR4: 0000000000770ef0
[1072053.343028] PKRU: 55555554
[1072053.343616] Call Trace:
[1072053.344177] <TASK>
[1072053.344708] ? __die+0x20/0x70
[1072053.345351] ? page_fault_oops+0x75/0x170
[1072053.346242] ? exc_page_fault+0x6c/0x160
[1072053.346911] ? asm_exc_page_fault+0x22/0x30
[1072053.347559] ? attach_eb_folio_to_filemap+0xd0/0x200 [btrfs]
[1072053.348328] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[1072053.349305] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[1072053.350022] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[1072053.350742] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[1072053.351441] btrfs_cow_block+0xc7/0x280 [btrfs]
[1072053.352099] btrfs_search_slot+0x768/0xbd0 [btrfs]
[1072053.352764] ? btrfs_del_csums+0x53/0x3d0 [btrfs]
[1072053.353487] btrfs_del_csums+0x10b/0x3d0 [btrfs]
[1072053.354176] do_free_extent_accounting+0x3c/0x340 [btrfs]
[1072053.354877] __btrfs_free_extent.isra.0+0xb5f/0x10a0 [btrfs]
[1072053.355584] run_delayed_data_ref+0x80/0x1d0 [btrfs]
[1072053.356224] btrfs_run_delayed_refs_for_head+0x311/0x420 [btrfs]
[1072053.356943] ? kmem_cache_alloc+0x12f/0x330
[1072053.357496] __btrfs_run_delayed_refs+0x9c/0x1a0 [btrfs]
[1072053.358157] btrfs_run_delayed_refs+0x7b/0x130 [btrfs]
[1072053.358789] flush_space+0x23c/0x250 [btrfs]
[1072053.359360] btrfs_preempt_reclaim_metadata_space+0xfb/0x330 [btrfs]
[1072053.360076] process_one_work+0x193/0x3d0
[1072053.360597] worker_thread+0x2fc/0x410
[1072053.361088] ? __pfx_worker_thread+0x10/0x10
[1072053.361611] kthread+0xdc/0x110
[1072053.362043] ? __pfx_kthread+0x10/0x10
[1072053.362507] ret_from_fork+0x2d/0x50
[1072053.362975] ? __pfx_kthread+0x10/0x10
[1072053.363438] ret_from_fork_asm+0x1a/0x30
[1072053.363929] </TASK>
[1072053.364280] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) tcp_diag(E) udp_diag(E) inet_diag(E) tls(E) isofs(E) binfmt_misc(E) sunrpc(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) virtio_net(E) i2c_i801(E) virtio_gpu(E) net_failover(E) vfat(E) i2c_smbus(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fat(E) fuse(E) zram(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) polyval_clmulni(E) libata(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[1072053.364311] Unloaded tainted modules: amd_atl(E):3 edac_mce_amd(E):1 padlock_aes(E):3
[1072053.372039] CR2: 000000000000002c
[1072053.372487] ---[ end trace 0000000000000000 ]---
[1072053.373039] RIP: 0010:attach_eb_folio_to_filemap+0xd0/0x200 [btrfs]
[1072053.373932] Code: 86 99 00 00 00 48 8b 7d 08 48 89 ee 40 f6 c7 01 0f 85 08 01 00 00 0f 1f 44 00 00 48 89 ef 48 8b 07 f6 c4 80 74 78 48 8b 57 28 <8b> 42 2c 85 c0 74 39 8d 48 01 f0 0f b1 4a 2c 75 f2 48 85 d2 74 5e
[1072053.375971] RSP: 0018:ffffb9864f5db6f0 EFLAGS: 00010282
[1072053.376574] RAX: 01d7ffffd400822d RBX: 0000000018f75728 RCX: 0000000000000000
[1072053.377305] RDX: 0000000000000000 RSI: ffffe622db57b7c0 RDI: ffffe622db57b7c0
[1072053.378044] RBP: ffffe622db57b7c0 R08: 0000000000000001 R09: ffffe6217aeaed48
[1072053.378776] R10: ffff91507ffd6e00 R11: ffffb9864f5db690 R12: ffff91418353a5a0
[1072053.379501] R13: ffff91418353a5a0 R14: ffff916f43fb0768 R15: ffff918c87d7b000
[1072053.380244] FS: 0000000000000000(0000) GS:ffff915043b80000(0000) knlGS:0000000000000000
[1072053.381057] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1072053.381698] CR2: 000000000000002c CR3: 0000000fe4938002 CR4: 0000000000770ef0
[1072053.382434] PKRU: 55555554
[1072053.382875] Kernel panic - not syncing: Fatal exception
[1072053.384036] Kernel Offset: 0x5a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[1072053.385076] ---[ end Kernel panic - not syncing: Fatal exception ]---
[-- Attachment #3: btrfs.issue.2.log --]
[-- Type: text/x-log, Size: 282568 bytes --]
[290646.447265] watchdog: BUG: soft lockup - CPU#22 stuck for 82s! [kworker/u100:5:3816805]
[290646.448507] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290646.448557] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290646.458705] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290646.459906] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290646.460981] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290646.461926] RIP: 0010:xas_start+0x66/0xd0
[290646.462631] Code: d2 75 29 48 c7 47 18 00 00 00 00 c3 cc cc cc cc 48 c1 fa 02 85 d2 74 c7 31 c0 c3 cc cc cc cc 0f b6 48 fe 48 d3 ea 48 83 fa 3f <76> d7 48 c7 47 18 01 00 00 00 31 c0 c3 cc cc cc cc 48 85 d2 74 44
[290646.464942] RSP: 0018:ffffb956e260f808 EFLAGS: 00000293
[290646.465736] RAX: ffff9074f9f11ff2 RBX: 0000000000000000 RCX: 0000000000000012
[290646.466692] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[290646.467662] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290646.468627] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290646.469593] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290646.470600] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290646.471648] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290646.472493] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290646.473470] PKRU: 55555554
[290646.474064] Call Trace:
[290646.474621] <IRQ>
[290646.475194] ? watchdog_timer_fn+0x1d7/0x250
[290646.475899] ? __pfx_watchdog_timer_fn+0x10/0x10
[290646.476646] ? __hrtimer_run_queues+0x10f/0x2b0
[290646.477387] ? hrtimer_interrupt+0xff/0x240
[290646.478109] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290646.478894] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290646.479674] </IRQ>
[290646.480214] <TASK>
[290646.480733] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290646.481544] ? xas_start+0x66/0xd0
[290646.482235] xas_load+0xa/0x50
[290646.482839] filemap_get_entry+0x6a/0x140
[290646.483536] __filemap_get_folio+0x33/0x2d0
[290646.484290] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290646.485170] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290646.485996] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290646.486809] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290646.487651] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290646.488489] btrfs_cow_block+0xc7/0x280 [btrfs]
[290646.489319] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290646.490138] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290646.490926] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290646.491764] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290646.492576] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290646.493440] btrfs_work_helper+0xba/0x190 [btrfs]
[290646.494251] process_one_work+0x193/0x3d0
[290646.494916] worker_thread+0x2fc/0x410
[290646.495569] ? __pfx_worker_thread+0x10/0x10
[290646.496261] kthread+0xdc/0x110
[290646.496824] ? __pfx_kthread+0x10/0x10
[290646.497443] ret_from_fork+0x2d/0x50
[290646.498084] ? __pfx_kthread+0x10/0x10
[290646.498675] ret_from_fork_asm+0x1a/0x30
[290646.499291] </TASK>
[290662.446179] watchdog: BUG: soft lockup - CPU#12 stuck for 75s! [migration/12:55]
[290662.446964] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290662.446997] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290662.454286] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290662.455267] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290662.456171] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290662.456953] RIP: 0010:stop_machine_yield+0x2/0x10
[290662.457534] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290662.459548] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290662.460249] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290662.461121] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290662.461995] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290662.462851] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290662.463719] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290662.464576] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290662.465505] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290662.466279] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290662.467174] PKRU: 55555554
[290662.467646] Call Trace:
[290662.468101] <IRQ>
[290662.468508] ? watchdog_timer_fn+0x1d7/0x250
[290662.469137] ? __pfx_watchdog_timer_fn+0x10/0x10
[290662.469781] ? __hrtimer_run_queues+0x10f/0x2b0
[290662.470413] ? hrtimer_interrupt+0xff/0x240
[290662.471025] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290662.471709] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290662.472372] </IRQ>
[290662.472804] <TASK>
[290662.473226] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290662.473937] ? stop_machine_yield+0x2/0x10
[290662.474520] multi_cpu_stop+0x94/0x100
[290662.475125] ? __pfx_multi_cpu_stop+0x10/0x10
[290662.475732] cpu_stopper_thread+0x90/0x140
[290662.476324] ? __pfx_smpboot_thread_fn+0x10/0x10
[290662.476973] smpboot_thread_fn+0xd4/0x1d0
[290662.477548] kthread+0xdc/0x110
[290662.477968] ? __pfx_kthread+0x10/0x10
[290662.478388] ret_from_fork+0x2d/0x50
[290662.478830] ? __pfx_kthread+0x10/0x10
[290662.479249] ret_from_fork_asm+0x1a/0x30
[290662.479686] </TASK>
[290882.445964] watchdog: BUG: soft lockup - CPU#22 stuck for 302s! [kworker/u100:5:3816805]
[290882.447245] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290882.447296] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290882.457475] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290882.458676] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290882.459754] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290882.460710] RIP: 0010:xas_start+0x5f/0xd0
[290882.461398] Code: 10 00 00 77 21 48 85 d2 75 29 48 c7 47 18 00 00 00 00 c3 cc cc cc cc 48 c1 fa 02 85 d2 74 c7 31 c0 c3 cc cc cc cc 0f b6 48 fe <48> d3 ea 48 83 fa 3f 76 d7 48 c7 47 18 01 00 00 00 31 c0 c3 cc cc
[290882.463728] RSP: 0018:ffffb956e260f808 EFLAGS: 00000282
[290882.464542] RAX: ffff9074f9f11ff2 RBX: 0000000000000000 RCX: 0000000000000012
[290882.465501] RDX: 0000000000022de8 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[290882.466470] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290882.467424] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290882.468385] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290882.469352] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290882.470413] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290882.471268] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290882.472257] PKRU: 55555554
[290882.472869] Call Trace:
[290882.473419] <IRQ>
[290882.473949] ? watchdog_timer_fn+0x1d7/0x250
[290882.474675] ? __pfx_watchdog_timer_fn+0x10/0x10
[290882.475415] ? __hrtimer_run_queues+0x10f/0x2b0
[290882.476159] ? hrtimer_interrupt+0xff/0x240
[290882.476884] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290882.477678] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290882.478459] </IRQ>
[290882.478981] <TASK>
[290882.479507] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290882.480304] ? xas_start+0x5f/0xd0
[290882.480957] xas_load+0xa/0x50
[290882.481574] filemap_get_entry+0x6a/0x140
[290882.482262] __filemap_get_folio+0x33/0x2d0
[290882.482991] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290882.483884] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290882.484711] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290882.485542] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290882.486378] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290882.487219] btrfs_cow_block+0xc7/0x280 [btrfs]
[290882.488025] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290882.488846] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290882.489649] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290882.490500] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290882.491300] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290882.492167] btrfs_work_helper+0xba/0x190 [btrfs]
[290882.492982] process_one_work+0x193/0x3d0
[290882.493676] worker_thread+0x2fc/0x410
[290882.494313] ? __pfx_worker_thread+0x10/0x10
[290882.495016] kthread+0xdc/0x110
[290882.495594] ? __pfx_kthread+0x10/0x10
[290882.496203] ret_from_fork+0x2d/0x50
[290882.496813] ? __pfx_kthread+0x10/0x10
[290882.497409] ret_from_fork_asm+0x1a/0x30
[290882.498019] </TASK>
[290886.444940] watchdog: BUG: soft lockup - CPU#12 stuck for 284s! [migration/12:55]
[290886.445654] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290886.445686] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290886.451383] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290886.452057] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290886.452705] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290886.453244] RIP: 0010:stop_machine_yield+0x2/0x10
[290886.453742] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290886.455283] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290886.455784] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290886.456357] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290886.456936] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290886.457519] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290886.458091] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290886.458694] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290886.459317] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290886.459843] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290886.460434] PKRU: 55555554
[290886.460791] Call Trace:
[290886.461130] <IRQ>
[290886.461459] ? watchdog_timer_fn+0x1d7/0x250
[290886.461897] ? __pfx_watchdog_timer_fn+0x10/0x10
[290886.462354] ? __hrtimer_run_queues+0x10f/0x2b0
[290886.462811] ? hrtimer_interrupt+0xff/0x240
[290886.463237] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290886.463765] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290886.464267] </IRQ>
[290886.464618] <TASK>
[290886.464940] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290886.465437] ? stop_machine_yield+0x2/0x10
[290886.465863] multi_cpu_stop+0x94/0x100
[290886.466267] ? __pfx_multi_cpu_stop+0x10/0x10
[290886.466730] cpu_stopper_thread+0x90/0x140
[290886.467152] ? __pfx_smpboot_thread_fn+0x10/0x10
[290886.467632] smpboot_thread_fn+0xd4/0x1d0
[290886.468049] kthread+0xdc/0x110
[290886.468436] ? __pfx_kthread+0x10/0x10
[290886.468843] ret_from_fork+0x2d/0x50
[290886.469238] ? __pfx_kthread+0x10/0x10
[290886.469672] ret_from_fork_asm+0x1a/0x30
[290886.470084] </TASK>
[290914.444784] watchdog: BUG: soft lockup - CPU#12 stuck for 310s! [migration/12:55]
[290914.445512] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290914.445543] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290914.451443] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290914.452112] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290914.452767] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290914.453329] RIP: 0010:stop_machine_yield+0x2/0x10
[290914.453773] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290914.455160] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290914.455658] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290914.456239] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290914.456836] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290914.457460] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290914.458036] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290914.458645] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290914.459282] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290914.459793] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290914.460399] PKRU: 55555554
[290914.460757] Call Trace:
[290914.461100] <IRQ>
[290914.461448] ? watchdog_timer_fn+0x1d7/0x250
[290914.461886] ? __pfx_watchdog_timer_fn+0x10/0x10
[290914.462354] ? __hrtimer_run_queues+0x10f/0x2b0
[290914.462805] ? hrtimer_interrupt+0xff/0x240
[290914.463241] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290914.463738] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290914.464214] </IRQ>
[290914.464547] <TASK>
[290914.464867] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290914.465366] ? stop_machine_yield+0x2/0x10
[290914.465794] multi_cpu_stop+0x94/0x100
[290914.466204] ? __pfx_multi_cpu_stop+0x10/0x10
[290914.466656] cpu_stopper_thread+0x90/0x140
[290914.467079] ? __pfx_smpboot_thread_fn+0x10/0x10
[290914.467563] smpboot_thread_fn+0xd4/0x1d0
[290914.467981] kthread+0xdc/0x110
[290914.468381] ? __pfx_kthread+0x10/0x10
[290914.468791] ret_from_fork+0x2d/0x50
[290914.469196] ? __pfx_kthread+0x10/0x10
[290914.469611] ret_from_fork_asm+0x1a/0x30
[290914.470024] </TASK>
[291110.443706] watchdog: BUG: soft lockup - CPU#12 stuck for 492s! [migration/12:55]
[291110.444432] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291110.444463] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291110.450115] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291110.450784] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291110.451463] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291110.452000] RIP: 0010:stop_machine_yield+0x2/0x10
[291110.452472] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291110.453836] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291110.454334] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291110.454907] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291110.455513] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291110.456083] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291110.456705] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291110.457295] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291110.457919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291110.458449] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291110.459025] PKRU: 55555554
[291110.459409] Call Trace:
[291110.459751] <IRQ>
[291110.460070] ? watchdog_timer_fn+0x1d7/0x250
[291110.460524] ? __pfx_watchdog_timer_fn+0x10/0x10
[291110.460973] ? __hrtimer_run_queues+0x10f/0x2b0
[291110.461458] ? hrtimer_interrupt+0xff/0x240
[291110.461886] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291110.462390] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291110.462857] </IRQ>
[291110.463184] <TASK>
[291110.463513] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291110.463996] ? stop_machine_yield+0x2/0x10
[291110.464445] multi_cpu_stop+0x94/0x100
[291110.464850] ? __pfx_multi_cpu_stop+0x10/0x10
[291110.465311] cpu_stopper_thread+0x90/0x140
[291110.465735] ? __pfx_smpboot_thread_fn+0x10/0x10
[291110.466186] smpboot_thread_fn+0xd4/0x1d0
[291110.466609] kthread+0xdc/0x110
[291110.466981] ? __pfx_kthread+0x10/0x10
[291110.467416] ret_from_fork+0x2d/0x50
[291110.467813] ? __pfx_kthread+0x10/0x10
[291110.468225] ret_from_fork_asm+0x1a/0x30
[291110.468638] </TASK>
[291146.444510] watchdog: BUG: soft lockup - CPU#22 stuck for 548s! [kworker/u100:5:3816805]
[291146.445468] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291146.445501] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291146.451613] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291146.452384] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291146.453047] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291146.453633] RIP: 0010:xas_descend+0x7/0x80
[291146.454084] Code: 48 c1 e8 20 48 89 57 08 c3 cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 <48> d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89
[291146.455516] RSP: 0018:ffffb956e260f808 EFLAGS: 00000297
[291146.456015] RAX: 0000000000022de8 RBX: ffff9074f9f11ff0 RCX: 0000000000000012
[291146.456618] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[291146.457241] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291146.457825] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291146.458451] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291146.459058] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291146.459693] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291146.460245] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291146.460827] PKRU: 55555554
[291146.461230] Call Trace:
[291146.461577] <IRQ>
[291146.461900] ? watchdog_timer_fn+0x1d7/0x250
[291146.462347] ? __pfx_watchdog_timer_fn+0x10/0x10
[291146.462796] ? __hrtimer_run_queues+0x10f/0x2b0
[291146.463280] ? hrtimer_interrupt+0xff/0x240
[291146.463711] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291146.464221] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291146.464691] </IRQ>
[291146.465020] <TASK>
[291146.465357] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291146.465841] ? xas_descend+0x7/0x80
[291146.466266] xas_load+0x3d/0x50
[291146.466641] filemap_get_entry+0x6a/0x140
[291146.467073] __filemap_get_folio+0x33/0x2d0
[291146.467505] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291146.468045] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291146.468541] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291146.469056] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291146.469565] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291146.470089] btrfs_cow_block+0xc7/0x280 [btrfs]
[291146.470561] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291146.471062] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291146.471546] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291146.472067] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291146.472556] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291146.473092] btrfs_work_helper+0xba/0x190 [btrfs]
[291146.473578] process_one_work+0x193/0x3d0
[291146.473992] worker_thread+0x2fc/0x410
[291146.474399] ? __pfx_worker_thread+0x10/0x10
[291146.474807] kthread+0xdc/0x110
[291146.475192] ? __pfx_kthread+0x10/0x10
[291146.475560] ret_from_fork+0x2d/0x50
[291146.475923] ? __pfx_kthread+0x10/0x10
[291146.476300] ret_from_fork_asm+0x1a/0x30
[291146.476666] </TASK>
[291159.528813] rcu: INFO: rcu_preempt self-detected stall on CPU
[291159.529456] rcu: 22-....: (599152 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=175113
[291159.530214] rcu: (t=600123 jiffies g=38283501 q=341293 ncpus=24)
[291159.530696] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291159.531416] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291159.532044] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291159.532579] RIP: 0010:filemap_get_entry+0x6a/0x140
[291159.533033] Code: 00 00 48 c7 44 24 28 00 00 00 00 48 c7 44 24 30 00 00 00 00 e8 97 bf e5 ff 48 c7 44 24 18 03 00 00 00 48 89 e7 e8 16 b0 98 00 <48> 89 c7 48 3d 06 04 00 00 74 e4 48 3d 02 04 00 00 74 dc 48 85 c0
[291159.534352] RSP: 0018:ffffb956e260f828 EFLAGS: 00000246
[291159.534798] RAX: ffffe4bc7bfa2580 RBX: 0000000000000000 RCX: 0000000000000028
[291159.535420] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291159.535980] RBP: ffff905787aeb9a8 R08: 000000000004ebe5 R09: 0000000000000995
[291159.536523] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291159.537119] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291159.537664] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291159.538290] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291159.538764] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291159.539353] PKRU: 55555554
[291159.539678] Call Trace:
[291159.540021] <IRQ>
[291159.540311] ? rcu_dump_cpu_stacks+0xcb/0x110
[291159.540718] ? print_cpu_stall+0x111/0x260
[291159.541150] ? check_cpu_stall+0x10f/0x220
[291159.541538] ? rcu_pending+0x29/0x190
[291159.541917] ? rcu_sched_clock_irq+0xb8/0x350
[291159.542341] ? update_process_times+0x69/0xa0
[291159.542747] ? tick_nohz_handler+0xc5/0x130
[291159.543220] ? __pfx_tick_nohz_handler+0x10/0x10
[291159.543636] ? __hrtimer_run_queues+0x10f/0x2b0
[291159.544110] ? hrtimer_interrupt+0xff/0x240
[291159.544504] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291159.544973] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291159.545406] </IRQ>
[291159.545693] <TASK>
[291159.546008] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291159.546462] ? filemap_get_entry+0x6a/0x140
[291159.546862] ? filemap_get_entry+0x6a/0x140
[291159.547270] __filemap_get_folio+0x33/0x2d0
[291159.547660] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291159.548193] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291159.548650] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291159.549146] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291159.549616] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291159.550115] btrfs_cow_block+0xc7/0x280 [btrfs]
[291159.550550] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291159.551034] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291159.551481] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291159.551968] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291159.552420] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291159.552903] btrfs_work_helper+0xba/0x190 [btrfs]
[291159.553367] process_one_work+0x193/0x3d0
[291159.553745] worker_thread+0x2fc/0x410
[291159.554148] ? __pfx_worker_thread+0x10/0x10
[291159.554530] kthread+0xdc/0x110
[291159.554859] ? __pfx_kthread+0x10/0x10
[291159.555229] ret_from_fork+0x2d/0x50
[291159.555567] ? __pfx_kthread+0x10/0x10
[291159.555908] ret_from_fork_asm+0x1a/0x30
[291159.556273] </TASK>
[291166.443402] watchdog: BUG: soft lockup - CPU#12 stuck for 544s! [migration/12:55]
[291166.444160] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291166.444194] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291166.449857] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291166.450524] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291166.451159] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291166.451694] RIP: 0010:stop_machine_yield+0x2/0x10
[291166.452202] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291166.453590] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291166.454103] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291166.454677] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291166.455277] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291166.455856] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291166.456440] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291166.457054] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291166.457677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291166.458218] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291166.458799] PKRU: 55555554
[291166.459167] Call Trace:
[291166.459514] <IRQ>
[291166.459844] ? watchdog_timer_fn+0x1d7/0x250
[291166.460298] ? __pfx_watchdog_timer_fn+0x10/0x10
[291166.460756] ? __hrtimer_run_queues+0x10f/0x2b0
[291166.461245] ? hrtimer_interrupt+0xff/0x240
[291166.461676] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291166.462198] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291166.462671] </IRQ>
[291166.463024] <TASK>
[291166.463352] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291166.463856] ? stop_machine_yield+0x2/0x10
[291166.464294] multi_cpu_stop+0x94/0x100
[291166.464704] ? __pfx_multi_cpu_stop+0x10/0x10
[291166.465169] cpu_stopper_thread+0x90/0x140
[291166.465595] ? __pfx_smpboot_thread_fn+0x10/0x10
[291166.466079] smpboot_thread_fn+0xd4/0x1d0
[291166.466501] kthread+0xdc/0x110
[291166.466882] ? __pfx_kthread+0x10/0x10
[291166.467297] ret_from_fork+0x2d/0x50
[291166.467696] ? __pfx_kthread+0x10/0x10
[291166.468164] ret_from_fork_asm+0x1a/0x30
[291166.468581] </TASK>
[291519.589891] rcu: INFO: rcu_preempt self-detected stall on CPU
[291519.590536] rcu: 22-....: (958767 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=340224
[291519.591311] rcu: (t=960186 jiffies g=38283501 q=546668 ncpus=24)
[291519.591796] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291519.592511] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291519.593137] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291519.593670] RIP: 0010:xas_descend+0x1b/0x80
[291519.594094] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc
[291519.595426] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291519.595872] RAX: ffff9056ce8d4922 RBX: ffff9074f9f11ff0 RCX: 0000000000000000
[291519.596456] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[291519.597004] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291519.597560] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291519.598137] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291519.598679] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291519.599310] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291519.599781] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291519.600361] PKRU: 55555554
[291519.600686] Call Trace:
[291519.601197] <IRQ>
[291519.601485] ? rcu_dump_cpu_stacks+0xcb/0x110
[291519.601893] ? print_cpu_stall+0x111/0x260
[291519.602315] ? check_cpu_stall+0x10f/0x220
[291519.602702] ? rcu_pending+0x29/0x190
[291519.603104] ? rcu_sched_clock_irq+0xb8/0x350
[291519.603509] ? update_process_times+0x69/0xa0
[291519.603916] ? tick_nohz_handler+0xc5/0x130
[291519.604329] ? __pfx_tick_nohz_handler+0x10/0x10
[291519.604757] ? __hrtimer_run_queues+0x10f/0x2b0
[291519.605205] ? hrtimer_interrupt+0xff/0x240
[291519.605598] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291519.606060] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291519.606494] </IRQ>
[291519.606779] <TASK>
[291519.607105] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291519.607555] ? xas_descend+0x1b/0x80
[291519.607921] xas_load+0x3d/0x50
[291519.608279] filemap_get_entry+0x6a/0x140
[291519.608659] __filemap_get_folio+0x33/0x2d0
[291519.609074] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291519.609568] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291519.610042] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291519.610506] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291519.610978] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291519.611460] btrfs_cow_block+0xc7/0x280 [btrfs]
[291519.611917] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291519.612390] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291519.612832] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291519.613340] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291519.613787] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291519.614299] btrfs_work_helper+0xba/0x190 [btrfs]
[291519.614741] process_one_work+0x193/0x3d0
[291519.615152] worker_thread+0x2fc/0x410
[291519.615511] ? __pfx_worker_thread+0x10/0x10
[291519.615896] kthread+0xdc/0x110
[291519.616236] ? __pfx_kthread+0x10/0x10
[291519.616581] ret_from_fork+0x2d/0x50
[291519.616923] ? __pfx_kthread+0x10/0x10
[291519.617276] ret_from_fork_asm+0x1a/0x30
[291519.617619] </TASK>
[291530.441459] watchdog: BUG: soft lockup - CPU#12 stuck for 883s! [migration/12:55]
[290702.446963] watchdog: BUG: soft lockup - CPU#22 stuck for 134s! [kworker/u100:5:3816805]
[290702.448225] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290702.448273] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290702.459122] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290702.460380] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290702.461489] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290702.462471] RIP: 0010:xas_descend+0x16/0x80
[290702.463219] Code: cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 <48> 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88
[290702.465691] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[290702.466516] RAX: 000000000000003b RBX: ffff9066c9ceada0 RCX: 0000000000000037
[290702.467515] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[290702.468505] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290702.469499] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290702.470498] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290702.471491] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290702.472581] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290702.473445] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290702.474442] PKRU: 55555554
[290702.475045] Call Trace:
[290702.475624] <IRQ>
[290702.476149] ? watchdog_timer_fn+0x1d7/0x250
[290702.476895] ? __pfx_watchdog_timer_fn+0x10/0x10
[290702.477659] ? __hrtimer_run_queues+0x10f/0x2b0
[290702.478413] ? hrtimer_interrupt+0xff/0x240
[290702.479146] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290702.479978] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290702.480787] </IRQ>
[290702.481335] <TASK>
[290702.481883] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290702.482719] ? xas_descend+0x16/0x80
[290702.483381] xas_load+0x3d/0x50
[290702.484020] filemap_get_entry+0x6a/0x140
[290702.484795] __filemap_get_folio+0x33/0x2d0
[290702.485512] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290702.486407] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290702.487258] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290702.488118] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290702.489005] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290702.489871] btrfs_cow_block+0xc7/0x280 [btrfs]
[290702.490651] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290702.491443] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290702.492244] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290702.493100] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290702.493922] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290702.494787] btrfs_work_helper+0xba/0x190 [btrfs]
[290702.495580] process_one_work+0x193/0x3d0
[290702.496246] worker_thread+0x2fc/0x410
[290702.496900] ? __pfx_worker_thread+0x10/0x10
[290702.497580] kthread+0xdc/0x110
[290702.498145] ? __pfx_kthread+0x10/0x10
[290702.498965] ret_from_fork+0x2d/0x50
[290702.499557] ? __pfx_kthread+0x10/0x10
[290702.500337] ret_from_fork_asm+0x1a/0x30
[290702.500954] </TASK>
[290718.445878] watchdog: BUG: soft lockup - CPU#12 stuck for 127s! [migration/12:55]
[290718.446594] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290718.446625] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290718.452269] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290718.452940] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290718.453590] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290718.454127] RIP: 0010:stop_machine_yield+0x2/0x10
[290718.454592] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290718.455950] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290718.456436] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290718.457011] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290718.457614] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290718.458182] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290718.458788] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290718.459370] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290718.460005] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290718.460536] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290718.461118] PKRU: 55555554
[290718.461506] Call Trace:
[290718.461849] <IRQ>
[290718.462168] ? watchdog_timer_fn+0x1d7/0x250
[290718.462674] ? __pfx_watchdog_timer_fn+0x10/0x10
[290718.463129] ? __hrtimer_run_queues+0x10f/0x2b0
[290718.463607] ? hrtimer_interrupt+0xff/0x240
[290718.464035] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290718.464537] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290718.465003] </IRQ>
[290718.465333] <TASK>
[290718.465665] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290718.466152] ? stop_machine_yield+0x2/0x10
[290718.466607] multi_cpu_stop+0x94/0x100
[290718.467014] ? __pfx_multi_cpu_stop+0x10/0x10
[290718.467464] cpu_stopper_thread+0x90/0x140
[290718.467889] ? __pfx_smpboot_thread_fn+0x10/0x10
[290718.468341] smpboot_thread_fn+0xd4/0x1d0
[290718.468768] kthread+0xdc/0x110
[290718.469143] ? __pfx_kthread+0x10/0x10
[290718.469581] ret_from_fork+0x2d/0x50
[290718.469974] ? __pfx_kthread+0x10/0x10
[290718.470378] ret_from_fork_asm+0x1a/0x30
[290718.470805] </TASK>
[291186.444293] watchdog: BUG: soft lockup - CPU#22 stuck for 585s! [kworker/u100:5:3816805]
[291186.445171] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291186.445203] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291186.451435] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291186.452242] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291186.452923] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291186.453512] RIP: 0010:xas_load+0x30/0x50
[291186.453978] Code: 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe 48 8d 58 fe 38 4d 10 <77> ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b 00 75 c8 5b 5d c3 cc
[291186.455406] RSP: 0018:ffffb956e260f810 EFLAGS: 00000297
[291186.455933] RAX: ffff9066c9ceada2 RBX: ffff9066c9ceada0 RCX: 0000000000000006
[291186.456518] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[291186.457183] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291186.457769] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291186.458369] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291186.458985] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291186.459614] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291186.460161] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291186.460749] PKRU: 55555554
[291186.461126] Call Trace:
[291186.461468] <IRQ>
[291186.461793] ? watchdog_timer_fn+0x1d7/0x250
[291186.462247] ? __pfx_watchdog_timer_fn+0x10/0x10
[291186.462703] ? __hrtimer_run_queues+0x10f/0x2b0
[291186.463168] ? hrtimer_interrupt+0xff/0x240
[291186.463599] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291186.464120] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291186.464591] </IRQ>
[291186.464948] <TASK>
[291186.465269] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291186.465760] ? xas_load+0x30/0x50
[291186.466159] filemap_get_entry+0x6a/0x140
[291186.466579] __filemap_get_folio+0x33/0x2d0
[291186.467056] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291186.467594] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291186.468137] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291186.468641] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291186.469308] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291186.469829] btrfs_cow_block+0xc7/0x280 [btrfs]
[291186.470307] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291186.470797] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291186.471294] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291186.471805] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291186.472309] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291186.472846] btrfs_work_helper+0xba/0x190 [btrfs]
[291186.473340] process_one_work+0x193/0x3d0
[291186.473762] worker_thread+0x2fc/0x410
[291186.474178] ? __pfx_worker_thread+0x10/0x10
[291186.474595] kthread+0xdc/0x110
[291186.474993] ? __pfx_kthread+0x10/0x10
[291186.475370] ret_from_fork+0x2d/0x50
[291186.475737] ? __pfx_kthread+0x10/0x10
[291186.476119] ret_from_fork_asm+0x1a/0x30
[291186.476489] </TASK>
[291194.443249] watchdog: BUG: soft lockup - CPU#12 stuck for 570s! [migration/12:55]
[291194.444005] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291194.444034] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291194.449875] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291194.450544] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291194.451195] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291194.451750] RIP: 0010:stop_machine_yield+0x2/0x10
[291194.452203] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291194.453577] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291194.454082] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291194.454666] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291194.455254] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291194.455862] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291194.456441] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291194.457055] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291194.457688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291194.458208] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291194.458802] PKRU: 55555554
[291194.459168] Call Trace:
[291194.459515] <IRQ>
[291194.459870] ? watchdog_timer_fn+0x1d7/0x250
[291194.460313] ? __pfx_watchdog_timer_fn+0x10/0x10
[291194.460785] ? __hrtimer_run_queues+0x10f/0x2b0
[291194.461245] ? hrtimer_interrupt+0xff/0x240
[291194.461690] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291194.462188] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291194.462666] </IRQ>
[291194.463002] <TASK>
[291194.463327] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291194.463832] ? stop_machine_yield+0x2/0x10
[291194.464263] multi_cpu_stop+0x94/0x100
[291194.464678] ? __pfx_multi_cpu_stop+0x10/0x10
[291194.465130] cpu_stopper_thread+0x90/0x140
[291194.465553] ? __pfx_smpboot_thread_fn+0x10/0x10
[291194.466041] smpboot_thread_fn+0xd4/0x1d0
[291194.466464] kthread+0xdc/0x110
[291194.466903] ? __pfx_kthread+0x10/0x10
[291194.467311] ret_from_fork+0x2d/0x50
[291194.467720] ? __pfx_kthread+0x10/0x10
[291194.468138] ret_from_fork_asm+0x1a/0x30
[291194.468554] </TASK>
[291214.444140] watchdog: BUG: soft lockup - CPU#22 stuck for 611s! [kworker/u100:5:3816805]
[291214.445023] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291214.445053] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291214.451169] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291214.451929] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291214.452573] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291214.453166] RIP: 0010:xas_descend+0x22/0x80
[291214.453611] Code: 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 <83> e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc cc cc 48 83 fa 02 75
[291214.455037] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291214.455621] RAX: ffff9056ce8d4922 RBX: ffff9074f9f11ff0 RCX: 0000000000000000
[291214.456498] RDX: ffff9056ce8d4922 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[291214.457147] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291214.457769] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291214.458344] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291214.458963] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291214.459594] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291214.460122] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291214.460718] PKRU: 55555554
[291214.461075] Call Trace:
[291214.461416] <IRQ>
[291214.461762] ? watchdog_timer_fn+0x1d7/0x250
[291214.462192] ? __pfx_watchdog_timer_fn+0x10/0x10
[291214.462642] ? __hrtimer_run_queues+0x10f/0x2b0
[291214.463104] ? hrtimer_interrupt+0xff/0x240
[291214.463535] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291214.464031] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291214.464496] </IRQ>
[291214.464850] <TASK>
[291214.465169] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291214.465656] ? xas_descend+0x22/0x80
[291214.466065] xas_load+0x3d/0x50
[291214.466437] filemap_get_entry+0x6a/0x140
[291214.466894] __filemap_get_folio+0x33/0x2d0
[291214.467318] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291214.467880] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291214.468374] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291214.468901] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291214.469405] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291214.469940] btrfs_cow_block+0xc7/0x280 [btrfs]
[291214.470407] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291214.470923] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291214.471405] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291214.471949] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291214.472431] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291214.473047] btrfs_work_helper+0xba/0x190 [btrfs]
[291214.473681] process_one_work+0x193/0x3d0
[291214.474091] worker_thread+0x2fc/0x410
[291214.474527] ? __pfx_worker_thread+0x10/0x10
[291214.474965] kthread+0xdc/0x110
[291214.475306] ? __pfx_kthread+0x10/0x10
[291214.475693] ret_from_fork+0x2d/0x50
[291214.476050] ? __pfx_kthread+0x10/0x10
[291214.476407] ret_from_fork_asm+0x1a/0x30
[291214.476805] </TASK>
[291222.443098] watchdog: BUG: soft lockup - CPU#12 stuck for 597s! [migration/12:55]
[291222.443872] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291222.443905] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291222.449681] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291222.450351] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291222.451000] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291222.451551] RIP: 0010:stop_machine_yield+0x2/0x10
[291222.452007] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291222.453381] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291222.453897] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291222.454473] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291222.455070] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291222.455657] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291222.456238] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291222.456848] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291222.457473] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291222.458004] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291222.458594] PKRU: 55555554
[291222.458964] Call Trace:
[291222.459311] <IRQ>
[291222.459648] ? watchdog_timer_fn+0x1d7/0x250
[291222.460093] ? __pfx_watchdog_timer_fn+0x10/0x10
[291222.460556] ? __hrtimer_run_queues+0x10f/0x2b0
[291222.461016] ? hrtimer_interrupt+0xff/0x240
[291222.461448] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291222.461965] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291222.462437] </IRQ>
[291222.462813] <TASK>
[291222.463144] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291222.463647] ? stop_machine_yield+0x2/0x10
[291222.464079] multi_cpu_stop+0x94/0x100
[291222.464496] ? __pfx_multi_cpu_stop+0x10/0x10
[291222.464947] cpu_stopper_thread+0x90/0x140
[291222.465369] ? __pfx_smpboot_thread_fn+0x10/0x10
[291222.465854] smpboot_thread_fn+0xd4/0x1d0
[291222.466271] kthread+0xdc/0x110
[291222.466659] ? __pfx_kthread+0x10/0x10
[291222.467066] ret_from_fork+0x2d/0x50
[291222.467465] ? __pfx_kthread+0x10/0x10
[291222.467895] ret_from_fork_asm+0x1a/0x30
[291222.468308] </TASK>
[291418.442055] watchdog: BUG: soft lockup - CPU#12 stuck for 779s! [migration/12:55]
[291418.442835] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291418.442868] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291418.450393] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291418.451097] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291418.451770] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291418.452311] RIP: 0010:stop_machine_yield+0x2/0x10
[291418.452779] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291418.454143] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291418.454631] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291418.455207] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291418.455816] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291418.456387] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291418.456993] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291418.457579] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291418.458206] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291418.458752] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291418.459334] PKRU: 55555554
[291418.459725] Call Trace:
[291418.460065] <IRQ>
[291418.460384] ? watchdog_timer_fn+0x1d7/0x250
[291418.460853] ? __pfx_watchdog_timer_fn+0x10/0x10
[291418.461305] ? __hrtimer_run_queues+0x10f/0x2b0
[291418.461783] ? hrtimer_interrupt+0xff/0x240
[291418.462214] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291418.462731] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291418.463201] </IRQ>
[291418.463531] <TASK>
[291418.463863] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291418.464349] ? stop_machine_yield+0x2/0x10
[291418.464798] multi_cpu_stop+0x94/0x100
[291418.465207] ? __pfx_multi_cpu_stop+0x10/0x10
[291418.465674] cpu_stopper_thread+0x90/0x140
[291418.466095] ? __pfx_smpboot_thread_fn+0x10/0x10
[291418.466552] smpboot_thread_fn+0xd4/0x1d0
[291418.466976] kthread+0xdc/0x110
[291418.467349] ? __pfx_kthread+0x10/0x10
[291418.467782] ret_from_fork+0x2d/0x50
[291418.468177] ? __pfx_kthread+0x10/0x10
[291418.468594] ret_from_fork_asm+0x1a/0x30
[291418.469012] </TASK>
[291422.443033] watchdog: BUG: soft lockup - CPU#22 stuck for 805s! [kworker/u100:5:3816805]
[291422.444009] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291422.444043] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291422.451107] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291422.451912] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291422.452619] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291422.453232] RIP: 0010:filemap_get_entry+0x6a/0x140
[291422.453767] Code: 00 00 48 c7 44 24 28 00 00 00 00 48 c7 44 24 30 00 00 00 00 e8 97 bf e5 ff 48 c7 44 24 18 03 00 00 00 48 89 e7 e8 16 b0 98 00 <48> 89 c7 48 3d 06 04 00 00 74 e4 48 3d 02 04 00 00 74 dc 48 85 c0
[291422.455280] RSP: 0018:ffffb956e260f828 EFLAGS: 00000246
[291422.455842] RAX: ffffe4bc7bfa2580 RBX: 0000000000000000 RCX: 0000000000000028
[291422.456464] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291422.457107] RBP: ffff905787aeb9a8 R08: 000000000004ebe5 R09: 0000000000000995
[291422.457767] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291422.458386] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291422.459034] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291422.459766] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291422.460313] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291422.460997] PKRU: 55555554
[291422.461373] Call Trace:
[291422.461767] <IRQ>
[291422.462103] ? watchdog_timer_fn+0x1d7/0x250
[291422.462575] ? __pfx_watchdog_timer_fn+0x10/0x10
[291422.463057] ? __hrtimer_run_queues+0x10f/0x2b0
[291422.463540] ? hrtimer_interrupt+0xff/0x240
[291422.464007] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291422.464520] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291422.465035] </IRQ>
[291422.465379] <TASK>
[291422.465751] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291422.466258] ? filemap_get_entry+0x6a/0x140
[291422.466747] ? filemap_get_entry+0x6a/0x140
[291422.467195] __filemap_get_folio+0x33/0x2d0
[291422.467680] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291422.468241] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291422.468788] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291422.469306] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291422.469886] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291422.470414] btrfs_cow_block+0xc7/0x280 [btrfs]
[291422.470928] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291422.471437] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291422.471958] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291422.472490] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291422.473015] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291422.473581] btrfs_work_helper+0xba/0x190 [btrfs]
[291422.474094] process_one_work+0x193/0x3d0
[291422.474528] worker_thread+0x2fc/0x410
[291422.474961] ? __pfx_worker_thread+0x10/0x10
[291422.475395] kthread+0xdc/0x110
[291422.475792] ? __pfx_kthread+0x10/0x10
[291422.476183] ret_from_fork+0x2d/0x50
[291422.476577] ? __pfx_kthread+0x10/0x10
[291422.476962] ret_from_fork_asm+0x1a/0x30
[291422.477349] </TASK>
[291446.441906] watchdog: BUG: soft lockup - CPU#12 stuck for 805s! [migration/12:55]
[291446.442671] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291446.442704] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291446.448455] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291446.449114] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291446.449764] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291446.450310] RIP: 0010:stop_machine_yield+0x2/0x10
[291446.450755] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291446.452147] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291446.452647] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291446.453214] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291446.453817] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291446.454390] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291446.454972] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291446.455578] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291446.456201] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291446.456733] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291446.457314] PKRU: 55555554
[291446.457678] Call Trace:
[291446.458018] <IRQ>
[291446.458340] ? watchdog_timer_fn+0x1d7/0x250
[291446.458782] ? __pfx_watchdog_timer_fn+0x10/0x10
[291446.459234] ? __hrtimer_run_queues+0x10f/0x2b0
[291446.459715] ? hrtimer_interrupt+0xff/0x240
[291446.460145] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291446.460664] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291446.461131] </IRQ>
[291446.461466] <TASK>
[291446.461791] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291446.462275] ? stop_machine_yield+0x2/0x10
[291446.462718] multi_cpu_stop+0x94/0x100
[291446.463122] ? __pfx_multi_cpu_stop+0x10/0x10
[291446.463595] cpu_stopper_thread+0x90/0x140
[291446.464016] ? __pfx_smpboot_thread_fn+0x10/0x10
[291446.464479] smpboot_thread_fn+0xd4/0x1d0
[291446.464901] kthread+0xdc/0x110
[291446.465273] ? __pfx_kthread+0x10/0x10
[291446.465699] ret_from_fork+0x2d/0x50
[291446.466093] ? __pfx_kthread+0x10/0x10
[291446.466526] ret_from_fork_asm+0x1a/0x30
[291446.466940] </TASK>
[291530.442227] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291530.442260] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291530.448177] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291530.448849] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291530.449480] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291530.450050] RIP: 0010:stop_machine_yield+0x2/0x10
[291530.450498] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291530.451887] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291530.452378] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291530.452967] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291530.453546] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291530.454154] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291530.454730] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291530.455343] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291530.455980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291530.456492] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291530.457103] PKRU: 55555554
[291530.457460] Call Trace:
[291530.457802] <IRQ>
[291530.458192] ? watchdog_timer_fn+0x1d7/0x250
[291530.458632] ? __pfx_watchdog_timer_fn+0x10/0x10
[291530.459120] ? __hrtimer_run_queues+0x10f/0x2b0
[291530.459577] ? hrtimer_interrupt+0xff/0x240
[291530.460026] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291530.460512] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291530.460998] </IRQ>
[291530.461329] <TASK>
[291530.461653] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291530.462176] ? stop_machine_yield+0x2/0x10
[291530.462603] multi_cpu_stop+0x94/0x100
[291530.463028] ? __pfx_multi_cpu_stop+0x10/0x10
[291530.463475] cpu_stopper_thread+0x90/0x140
[291530.463907] ? __pfx_smpboot_thread_fn+0x10/0x10
[291530.464370] smpboot_thread_fn+0xd4/0x1d0
[291530.464785] kthread+0xdc/0x110
[291530.465192] ? __pfx_kthread+0x10/0x10
[291530.465599] ret_from_fork+0x2d/0x50
[291530.466011] ? __pfx_kthread+0x10/0x10
[291530.466418] ret_from_fork_asm+0x1a/0x30
[291530.466834] </TASK>
[290582.447608] watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kworker/u100:5:3816805]
[290582.448732] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290582.448763] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290582.456791] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G E 6.9.3-3.gdc.el9.x86_64 #1
[290582.457813] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290582.458737] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290582.459583] RIP: 0010:xas_load+0xd/0x50
[290582.460230] Code: 48 8b 40 08 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc
[290582.462126] RSP: 0018:ffffb956e260f810 EFLAGS: 00000206
[290582.462845] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000022
[290582.463681] RDX: ffff9066c9ceada2 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[290582.464506] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290582.465314] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290582.466106] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290582.466902] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290582.467760] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290582.468492] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290582.469270] PKRU: 55555554
[290582.469811] Call Trace:
[290582.470331] <IRQ>
[290582.470809] ? watchdog_timer_fn+0x1d7/0x250
[290582.471453] ? __pfx_watchdog_timer_fn+0x10/0x10
[290582.472080] ? __hrtimer_run_queues+0x10f/0x2b0
[290582.472788] ? hrtimer_interrupt+0xff/0x240
[290582.473403] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290582.474025] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290582.474636] </IRQ>
[290582.475075] <TASK>
[290582.475512] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290582.476114] ? xas_load+0xd/0x50
[290582.476610] filemap_get_entry+0x6a/0x140
[290582.477159] __filemap_get_folio+0x33/0x2d0
[290582.477688] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290582.478387] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290582.478975] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290582.479590] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290582.480185] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290582.480772] btrfs_cow_block+0xc7/0x280 [btrfs]
[290582.481339] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290582.481871] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290582.482437] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290582.483010] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290582.483550] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290582.484100] btrfs_work_helper+0xba/0x190 [btrfs]
[290582.484693] process_one_work+0x193/0x3d0
[290582.485113] worker_thread+0x2fc/0x410
[290582.485534] ? __pfx_worker_thread+0x10/0x10
[290582.485950] kthread+0xdc/0x110
[290582.486468] ? __pfx_kthread+0x10/0x10
[290582.486859] ret_from_fork+0x2d/0x50
[290582.487273] ? __pfx_kthread+0x10/0x10
[290582.487658] ret_from_fork_asm+0x1a/0x30
[290582.488035] </TASK>
[290610.447458] watchdog: BUG: soft lockup - CPU#22 stuck for 49s! [kworker/u100:5:3816805]
[290610.448665] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290610.448711] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290610.458649] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290610.459831] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290610.460876] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290610.461794] RIP: 0010:xas_load+0x30/0x50
[290610.462475] Code: 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe 48 8d 58 fe 38 4d 10 <77> ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b 00 75 c8 5b 5d c3 cc
[290610.464753] RSP: 0018:ffffb956e260f810 EFLAGS: 00000293
[290610.465546] RAX: ffff9056ce8d4922 RBX: ffff9056ce8d4920 RCX: 000000000000000c
[290610.466478] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[290610.467420] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290610.468373] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290610.469310] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290610.470280] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290610.471288] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290610.472104] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290610.473043] PKRU: 55555554
[290610.473609] Call Trace:
[290610.474135] <IRQ>
[290610.474634] ? watchdog_timer_fn+0x1d7/0x250
[290610.475320] ? __pfx_watchdog_timer_fn+0x10/0x10
[290610.476030] ? __hrtimer_run_queues+0x10f/0x2b0
[290610.476754] ? hrtimer_interrupt+0xff/0x240
[290610.477524] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290610.478297] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290610.479020] </IRQ>
[290610.479533] <TASK>
[290610.480027] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290610.480807] ? xas_load+0x30/0x50
[290610.481428] filemap_get_entry+0x6a/0x140
[290610.482087] __filemap_get_folio+0x33/0x2d0
[290610.482772] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290610.483609] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290610.484421] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290610.485234] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290610.486032] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290610.486841] btrfs_cow_block+0xc7/0x280 [btrfs]
[290610.487652] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290610.488508] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290610.489299] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290610.490261] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290610.491032] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290610.491867] btrfs_work_helper+0xba/0x190 [btrfs]
[290610.492637] process_one_work+0x193/0x3d0
[290610.493302] worker_thread+0x2fc/0x410
[290610.493932] ? __pfx_worker_thread+0x10/0x10
[290610.494605] kthread+0xdc/0x110
[290610.495910] ? __pfx_kthread+0x10/0x10
[290610.496776] ret_from_fork+0x2d/0x50
[290610.497359] ? __pfx_kthread+0x10/0x10
[290610.497933] ret_from_fork_asm+0x1a/0x30
[290610.498542] </TASK>
[290619.410786] rcu: INFO: rcu_preempt self-detected stall on CPU
[290619.411428] rcu: 22-....: (59909 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=13294
[290619.412266] rcu: (t=60002 jiffies g=38283501 q=46200 ncpus=24)
[290619.412812] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290619.413551] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290619.414236] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290619.415116] RIP: 0010:xas_load+0x30/0x50
[290619.415720] Code: 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe 48 8d 58 fe 38 4d 10 <77> ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b 00 75 c8 5b 5d c3 cc
[290619.417983] RSP: 0018:ffffb956e260f810 EFLAGS: 00000297
[290619.418650] RAX: ffff9074f9f11ff2 RBX: ffff9074f9f11ff0 RCX: 0000000000000012
[290619.419524] RDX: 0000000000000002 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[290619.420391] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290619.421237] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290619.422068] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290619.422892] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290619.423979] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290619.424684] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290619.425368] PKRU: 55555554
[290619.425714] Call Trace:
[290619.426220] <IRQ>
[290619.426600] ? rcu_dump_cpu_stacks+0xcb/0x110
[290619.427233] ? print_cpu_stall+0x111/0x260
[290619.427789] ? check_cpu_stall+0x10f/0x220
[290619.428355] ? rcu_pending+0x29/0x190
[290619.428872] ? rcu_sched_clock_irq+0xb8/0x350
[290619.429457] ? update_process_times+0x69/0xa0
[290619.430055] ? tick_nohz_handler+0xc5/0x130
[290619.430621] ? __pfx_tick_nohz_handler+0x10/0x10
[290619.431254] ? __hrtimer_run_queues+0x10f/0x2b0
[290619.431832] ? hrtimer_interrupt+0xff/0x240
[290619.432396] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290619.433046] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290619.433672] </IRQ>
[290619.434079] <TASK>
[290619.434459] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290619.435152] ? xas_load+0x30/0x50
[290619.435631] filemap_get_entry+0x6a/0x140
[290619.436206] __filemap_get_folio+0x33/0x2d0
[290619.436754] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290619.437472] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290619.438163] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290619.438817] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290619.439504] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290619.440211] btrfs_cow_block+0xc7/0x280 [btrfs]
[290619.440822] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290619.441467] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290619.442112] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290619.442773] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290619.443425] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290619.444148] btrfs_work_helper+0xba/0x190 [btrfs]
[290619.444773] process_one_work+0x193/0x3d0
[290619.445330] worker_thread+0x2fc/0x410
[290619.445838] ? __pfx_worker_thread+0x10/0x10
[290619.446407] kthread+0xdc/0x110
[290619.446861] ? __pfx_kthread+0x10/0x10
[290619.447365] ret_from_fork+0x2d/0x50
[290619.447869] ? __pfx_kthread+0x10/0x10
[290619.448364] ret_from_fork_asm+0x1a/0x30
[290619.448865] </TASK>
[290634.446330] watchdog: BUG: soft lockup - CPU#12 stuck for 49s! [migration/12:55]
[290854.446125] watchdog: BUG: soft lockup - CPU#22 stuck for 276s! [kworker/u100:5:3816805]
[290854.447354] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290854.447402] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290854.457326] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290854.458495] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290854.459536] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290854.460455] RIP: 0010:xas_descend+0x10/0x80
[290854.461169] Code: cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f <89> c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48
[290854.463415] RSP: 0018:ffffb956e260f808 EFLAGS: 00000206
[290854.464214] RAX: 0000000000000022 RBX: ffff9056ce8d4920 RCX: 0000000000000022
[290854.465154] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[290854.466093] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290854.467026] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290854.467973] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290854.468914] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290854.469930] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290854.470750] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290854.471682] PKRU: 55555554
[290854.472247] Call Trace:
[290854.472792] <IRQ>
[290854.473295] ? watchdog_timer_fn+0x1d7/0x250
[290854.474001] ? __pfx_watchdog_timer_fn+0x10/0x10
[290854.474730] ? __hrtimer_run_queues+0x10f/0x2b0
[290854.475443] ? hrtimer_interrupt+0xff/0x240
[290854.476145] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290854.476928] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290854.477681] </IRQ>
[290854.478194] <TASK>
[290854.478705] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290854.479475] ? xas_descend+0x10/0x80
[290854.480133] xas_load+0x3d/0x50
[290854.480739] filemap_get_entry+0x6a/0x140
[290854.481418] __filemap_get_folio+0x33/0x2d0
[290854.482182] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290854.483045] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290854.483850] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290854.484655] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290854.485465] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290854.486280] btrfs_cow_block+0xc7/0x280 [btrfs]
[290854.487060] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290854.487849] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290854.488627] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290854.489434] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290854.490234] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290854.491085] btrfs_work_helper+0xba/0x190 [btrfs]
[290854.491871] process_one_work+0x193/0x3d0
[290854.492522] worker_thread+0x2fc/0x410
[290854.493167] ? __pfx_worker_thread+0x10/0x10
[290854.493843] kthread+0xdc/0x110
[290854.494390] ? __pfx_kthread+0x10/0x10
[290854.495025] ret_from_fork+0x2d/0x50
[290854.495802] ? __pfx_kthread+0x10/0x10
[290854.496376] ret_from_fork_asm+0x1a/0x30
[290854.496982] </TASK>
[290858.445102] watchdog: BUG: soft lockup - CPU#12 stuck for 258s! [migration/12:55]
[290858.445893] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290858.445927] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290858.451688] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290858.452350] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290858.452994] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290858.453538] RIP: 0010:multi_cpu_stop+0x8c/0x100
[290858.453976] Code: d0 0f 1f 00 41 89 c6 f0 ff 4d 00 75 0f 8b 43 20 8b 4b 10 83 c0 01 89 4b 24 89 43 20 e8 9d ef f9 ff 41 83 ff 04 74 2e 45 89 fc <4c> 89 ef e8 4c ff ff ff 44 8b 7b 20 45 39 fc 75 aa 41 83 ff 01 76
[290858.455331] RSP: 0018:ffffb956c8303e48 EFLAGS: 00000246
[290858.455863] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290858.456432] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffff9065c3203400
[290858.457073] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290858.457658] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290858.458230] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290858.458832] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290858.459449] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290858.459977] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290858.460558] PKRU: 55555554
[290858.460924] Call Trace:
[290858.461261] <IRQ>
[290858.461590] ? watchdog_timer_fn+0x1d7/0x250
[290858.462026] ? __pfx_watchdog_timer_fn+0x10/0x10
[290858.462474] ? __hrtimer_run_queues+0x10f/0x2b0
[290858.462933] ? hrtimer_interrupt+0xff/0x240
[290858.463356] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290858.463865] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290858.464332] </IRQ>
[290858.464662] <TASK>
[290858.464983] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290858.465464] ? multi_cpu_stop+0x8c/0x100
[290858.465894] ? multi_cpu_stop+0xb2/0x100
[290858.466302] ? __pfx_multi_cpu_stop+0x10/0x10
[290858.466762] cpu_stopper_thread+0x90/0x140
[290858.467181] ? __pfx_smpboot_thread_fn+0x10/0x10
[290858.467638] smpboot_thread_fn+0xd4/0x1d0
[290858.468055] kthread+0xdc/0x110
[290858.468425] ? __pfx_kthread+0x10/0x10
[290858.468853] ret_from_fork+0x2d/0x50
[290858.469244] ? __pfx_kthread+0x10/0x10
[290858.469657] ret_from_fork_asm+0x1a/0x30
[290858.470069] </TASK>
[290938.445650] watchdog: BUG: soft lockup - CPU#22 stuck for 354s! [kworker/u100:5:3816805]
[290938.446532] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290938.446565] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290938.452654] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290938.453413] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290938.454055] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290938.454659] RIP: 0010:xas_descend+0x29/0x80
[290938.455102] Code: 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 <74> 0e 88 4f 12 c3 cc cc cc cc 48 83 fa 02 75 f2 48 3d fd 00 00 00
[290938.456538] RSP: 0018:ffffb956e260f808 EFLAGS: 00000246
[290938.457028] RAX: ffff904df9a5c492 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[290938.457631] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[290938.458230] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290938.458813] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290938.459426] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290938.460007] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290938.460677] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290938.461212] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290938.461992] PKRU: 55555554
[290938.462391] Call Trace:
[290938.462910] <IRQ>
[290938.463263] ? watchdog_timer_fn+0x1d7/0x250
[290938.463701] ? __pfx_watchdog_timer_fn+0x10/0x10
[290938.464169] ? __hrtimer_run_queues+0x10f/0x2b0
[290938.464623] ? hrtimer_interrupt+0xff/0x240
[290938.465057] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290938.465560] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290938.466033] </IRQ>
[290938.466375] <TASK>
[290938.466696] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290938.467196] ? xas_descend+0x29/0x80
[290938.467597] xas_load+0x3d/0x50
[290938.467973] filemap_get_entry+0x6a/0x140
[290938.468445] __filemap_get_folio+0x33/0x2d0
[290938.468874] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290938.469450] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290938.469946] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290938.470487] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290938.470993] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290938.471519] btrfs_cow_block+0xc7/0x280 [btrfs]
[290938.471989] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290938.472495] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290938.472976] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290938.473563] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290938.474055] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290938.474591] btrfs_work_helper+0xba/0x190 [btrfs]
[290938.475079] process_one_work+0x193/0x3d0
[290938.475506] worker_thread+0x2fc/0x410
[290938.475892] ? __pfx_worker_thread+0x10/0x10
[290938.476334] kthread+0xdc/0x110
[290938.476690] ? __pfx_kthread+0x10/0x10
[290938.477060] ret_from_fork+0x2d/0x50
[290938.477435] ? __pfx_kthread+0x10/0x10
[290938.477794] ret_from_fork_asm+0x1a/0x30
[290938.478179] </TASK>
[290942.444628] watchdog: BUG: soft lockup - CPU#12 stuck for 336s! [migration/12:55]
[290942.445392] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290942.445423] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290942.451567] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290942.452291] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290942.452940] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290942.453590] RIP: 0010:stop_machine_yield+0x2/0x10
[290942.454050] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290942.455498] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290942.455992] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290942.456612] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290942.457267] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290942.457870] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290942.458498] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290942.459106] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290942.459773] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290942.460325] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290942.460934] PKRU: 55555554
[290942.461325] Call Trace:
[290942.461678] <IRQ>
[290942.462006] ? watchdog_timer_fn+0x1d7/0x250
[290942.462471] ? __pfx_watchdog_timer_fn+0x10/0x10
[290942.462939] ? __hrtimer_run_queues+0x10f/0x2b0
[290942.463437] ? hrtimer_interrupt+0xff/0x240
[290942.463881] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290942.464409] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290942.464894] </IRQ>
[290942.465253] <TASK>
[290942.465584] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290942.466103] ? stop_machine_yield+0x2/0x10
[290942.466556] multi_cpu_stop+0x94/0x100
[290942.466973] ? __pfx_multi_cpu_stop+0x10/0x10
[290942.467457] cpu_stopper_thread+0x90/0x140
[290942.467894] ? __pfx_smpboot_thread_fn+0x10/0x10
[290942.468403] smpboot_thread_fn+0xd4/0x1d0
[290942.468837] kthread+0xdc/0x110
[290942.469251] ? __pfx_kthread+0x10/0x10
[290942.469685] ret_from_fork+0x2d/0x50
[290942.470100] ? __pfx_kthread+0x10/0x10
[290942.470527] ret_from_fork_asm+0x1a/0x30
[290942.470955] </TASK>
[290998.444317] watchdog: BUG: soft lockup - CPU#12 stuck for 388s! [migration/12:55]
[290998.445076] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290998.445108] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290998.450775] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290998.451436] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290998.452079] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290998.452614] RIP: 0010:stop_machine_yield+0x2/0x10
[290998.453114] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290998.454490] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290998.454992] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290998.455564] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290998.456173] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290998.456751] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290998.457333] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290998.457935] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290998.458555] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290998.459090] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290998.459668] PKRU: 55555554
[290998.460061] Call Trace:
[290998.460404] <IRQ>
[290998.460731] ? watchdog_timer_fn+0x1d7/0x250
[290998.461183] ? __pfx_watchdog_timer_fn+0x10/0x10
[290998.461635] ? __hrtimer_run_queues+0x10f/0x2b0
[290998.462116] ? hrtimer_interrupt+0xff/0x240
[290998.462546] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290998.463060] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290998.463525] </IRQ>
[290998.463860] <TASK>
[290998.464185] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290998.464666] ? stop_machine_yield+0x2/0x10
[290998.465119] multi_cpu_stop+0x94/0x100
[290998.465523] ? __pfx_multi_cpu_stop+0x10/0x10
[290998.465991] cpu_stopper_thread+0x90/0x140
[290998.466413] ? __pfx_smpboot_thread_fn+0x10/0x10
[290998.466874] smpboot_thread_fn+0xd4/0x1d0
[290998.467293] kthread+0xdc/0x110
[290998.467667] ? __pfx_kthread+0x10/0x10
[290998.468140] ret_from_fork+0x2d/0x50
[290998.468535] ? __pfx_kthread+0x10/0x10
[290998.468967] ret_from_fork_asm+0x1a/0x30
[290998.469381] </TASK>
[291006.445272] watchdog: BUG: soft lockup - CPU#22 stuck for 417s! [kworker/u100:5:3816805]
[291006.446151] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291006.446183] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291006.452285] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291006.453041] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291006.453682] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291006.454278] RIP: 0010:xas_descend+0x16/0x80
[291006.454720] Code: cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 <48> 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88
[291006.456198] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291006.456694] RAX: 000000000000003b RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291006.457287] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291006.457902] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291006.458482] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291006.459100] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291006.459688] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291006.460335] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291006.460875] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291006.461463] PKRU: 55555554
[291006.461836] Call Trace:
[291006.462184] <IRQ>
[291006.462502] ? watchdog_timer_fn+0x1d7/0x250
[291006.462987] ? __pfx_watchdog_timer_fn+0x10/0x10
[291006.463439] ? __hrtimer_run_queues+0x10f/0x2b0
[291006.463921] ? hrtimer_interrupt+0xff/0x240
[291006.464354] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291006.464856] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291006.465330] </IRQ>
[291006.465656] <TASK>
[291006.465994] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291006.466477] ? xas_descend+0x16/0x80
[291006.466914] xas_load+0x3d/0x50
[291006.467293] filemap_get_entry+0x6a/0x140
[291006.467719] __filemap_get_folio+0x33/0x2d0
[291006.468167] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291006.468704] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291006.469221] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291006.469729] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291006.470256] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291006.470765] btrfs_cow_block+0xc7/0x280 [btrfs]
[291006.471253] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291006.471743] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291006.472241] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291006.472756] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291006.473261] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291006.473798] btrfs_work_helper+0xba/0x190 [btrfs]
[291006.474284] process_one_work+0x193/0x3d0
[291006.474700] worker_thread+0x2fc/0x410
[291006.475105] ? __pfx_worker_thread+0x10/0x10
[291006.475514] kthread+0xdc/0x110
[291006.475901] ? __pfx_kthread+0x10/0x10
[291006.476270] ret_from_fork+0x2d/0x50
[291006.476627] ? __pfx_kthread+0x10/0x10
[291006.477024] ret_from_fork_asm+0x1a/0x30
[291006.477393] </TASK>
[291062.444967] watchdog: BUG: soft lockup - CPU#22 stuck for 470s! [kworker/u100:5:3816805]
[291062.445920] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291062.445953] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291062.452824] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291062.453596] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291062.454261] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291062.454907] RIP: 0010:xas_descend+0x1b/0x80
[291062.455369] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc
[291062.456822] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291062.457329] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000022
[291062.457999] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[291062.458633] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291062.459224] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291062.459850] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291062.460449] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291062.461105] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291062.461662] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291062.462254] PKRU: 55555554
[291062.462655] Call Trace:
[291062.463008] <IRQ>
[291062.463334] ? watchdog_timer_fn+0x1d7/0x250
[291062.463805] ? __pfx_watchdog_timer_fn+0x10/0x10
[291062.464268] ? __hrtimer_run_queues+0x10f/0x2b0
[291062.464766] ? hrtimer_interrupt+0xff/0x240
[291062.465215] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291062.465751] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291062.466232] </IRQ>
[291062.466594] <TASK>
[291062.466925] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291062.467422] ? xas_descend+0x1b/0x80
[291062.467840] xas_load+0x3d/0x50
[291062.468217] filemap_get_entry+0x6a/0x140
[291062.468679] __filemap_get_folio+0x33/0x2d0
[291062.469114] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291062.469683] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291062.470185] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291062.470758] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291062.471270] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291062.471815] btrfs_cow_block+0xc7/0x280 [btrfs]
[291062.472288] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291062.472810] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291062.473297] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291062.473845] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291062.474334] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291062.474877] btrfs_work_helper+0xba/0x190 [btrfs]
[291062.475371] process_one_work+0x193/0x3d0
[291062.475799] worker_thread+0x2fc/0x410
[291062.476190] ? __pfx_worker_thread+0x10/0x10
[291062.476637] kthread+0xdc/0x110
[291062.476985] ? __pfx_kthread+0x10/0x10
[291062.477362] ret_from_fork+0x2d/0x50
[291062.477743] ? __pfx_kthread+0x10/0x10
[291062.478107] ret_from_fork_asm+0x1a/0x30
[291062.478498] </TASK>
[291082.443859] watchdog: BUG: soft lockup - CPU#12 stuck for 466s! [migration/12:55]
[290733.509182] INFO: task systemd:1 blocked for more than 122 seconds.
[290733.510057] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.510869] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.511764] task:systemd state:D stack:0 pid:1 tgid:1 ppid:0 flags:0x00000002
[290733.512738] Call Trace:
[290733.513176] <TASK>
[290733.513582] __schedule+0x222/0x670
[290733.514081] schedule+0x2c/0xb0
[290733.514585] schedule_timeout+0x146/0x160
[290733.515111] __wait_for_common+0x8f/0x1d0
[290733.515669] ? __pfx_schedule_timeout+0x10/0x10
[290733.516242] __wait_rcu_gp+0x116/0x120
[290733.516753] synchronize_rcu+0xf8/0x110
[290733.517262] ? __pfx_call_rcu_hurry+0x10/0x10
[290733.517810] ? __pfx_wakeme_after_rcu+0x10/0x10
[290733.518383] rcu_sync_enter+0x54/0xf0
[290733.518877] ? _kstrtoull+0x3b/0x90
[290733.519372] percpu_down_write+0x1d/0x180
[290733.519888] cgroup_procs_write_start+0x68/0x140
[290733.520475] __cgroup_procs_write+0x5a/0x150
[290733.521007] cgroup_procs_write+0x13/0x20
[290733.521543] kernfs_fop_write_iter+0x128/0x1c0
[290733.522084] vfs_write+0x308/0x420
[290733.522580] ksys_write+0x5f/0xe0
[290733.523038] do_syscall_64+0x5d/0x170
[290733.523548] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[290733.524134] RIP: 0033:0x7f9a3bcfda8f
[290733.524655] RSP: 002b:00007ffc8e19aa60 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[290733.525464] RAX: ffffffffffffffda RBX: 0000000000000008 RCX: 00007f9a3bcfda8f
[290733.526223] RDX: 0000000000000008 RSI: 00007ffc8e19ac5a RDI: 0000000000000030
[290733.526980] RBP: 00007ffc8e19ac5a R08: 0000000000000000 R09: 00007ffc8e19aae0
[290733.527755] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000008
[290733.528528] R13: 000055fb7e3fba20 R14: 0000000000000008 R15: 00007f9a3bdf69e0
[290733.529283] </TASK>
[290733.529700] INFO: task khugepaged:201 blocked for more than 122 seconds.
[290733.530442] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.531160] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.531992] task:khugepaged state:D stack:0 pid:201 tgid:201 ppid:2 flags:0x00004002
[290733.532948] Call Trace:
[290733.533380] <TASK>
[290733.533770] __schedule+0x222/0x670
[290733.534267] schedule+0x2c/0xb0
[290733.534743] schedule_timeout+0x146/0x160
[290733.535276] ? kvm_sched_clock_read+0xd/0x20
[290733.535838] ? sched_clock+0xc/0x30
[290733.536346] ? sched_clock_cpu+0xb/0x190
[290733.536869] ? __smp_call_single_queue+0xae/0x120
[290733.537472] __wait_for_common+0x8f/0x1d0
[290733.537997] ? __pfx_schedule_timeout+0x10/0x10
[290733.538593] __flush_work.isra.0+0x15f/0x270
[290733.539139] ? __pfx_wq_barrier_func+0x10/0x10
[290733.539725] __lru_add_drain_all+0x14f/0x200
[290733.540275] khugepaged+0x73/0x1e0
[290733.540759] ? __pfx_autoremove_wake_function+0x10/0x10
[290733.541382] ? __pfx_khugepaged+0x10/0x10
[290733.541901] kthread+0xdc/0x110
[290733.542372] ? __pfx_kthread+0x10/0x10
[290733.542870] ret_from_fork+0x2d/0x50
[290733.543373] ? __pfx_kthread+0x10/0x10
[290733.543868] ret_from_fork_asm+0x1a/0x30
[290733.544411] </TASK>
[290733.544811] INFO: task btrfs-transacti:18817 blocked for more than 122 seconds.
[290733.545612] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.546350] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.547176] task:btrfs-transacti state:D stack:0 pid:18817 tgid:18817 ppid:2 flags:0x00004000
[290733.548117] Call Trace:
[290733.548564] <TASK>
[290733.548953] __schedule+0x222/0x670
[290733.549467] schedule+0x2c/0xb0
[290733.549933] schedule_preempt_disabled+0x11/0x20
[290733.550530] rwsem_down_read_slowpath+0x260/0x4f0
[290733.551112] down_read+0x45/0xa0
[290733.551614] __btrfs_tree_read_lock+0x19/0x80 [btrfs]
[290733.552290] btrfs_read_lock_root_node+0x34/0x90 [btrfs]
[290733.552982] btrfs_search_slot+0x127/0xbd0 [btrfs]
[290733.553648] ? kmem_cache_alloc+0x24d/0x330
[290733.554209] ? kmem_cache_alloc+0x12f/0x330
[290733.554786] ? merge_next_state+0x1a/0x90 [btrfs]
[290733.555437] ? btrfs_del_csums+0x53/0x3d0 [btrfs]
[290733.556057] btrfs_del_csums+0x10b/0x3d0 [btrfs]
[290733.556695] cleanup_ref_head+0x27d/0x2a0 [btrfs]
[290733.557325] __btrfs_run_delayed_refs+0xb9/0x1a0 [btrfs]
[290733.557984] btrfs_run_delayed_refs+0x7b/0x130 [btrfs]
[290733.558657] btrfs_commit_transaction+0x68/0xb80 [btrfs]
[290733.559333] ? start_transaction+0xd3/0x830 [btrfs]
[290733.559962] transaction_kthread+0x155/0x1d0 [btrfs]
[290733.560617] ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[290733.561296] kthread+0xdc/0x110
[290733.561749] ? __pfx_kthread+0x10/0x10
[290733.562247] ret_from_fork+0x2d/0x50
[290733.562738] ? __pfx_kthread+0x10/0x10
[290733.563239] ret_from_fork_asm+0x1a/0x30
[290733.563760] </TASK>
[290733.564134] INFO: task in:imjournal:18867 blocked for more than 122 seconds.
[290733.564901] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.565647] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.566476] task:in:imjournal state:D stack:0 pid:18867 tgid:18865 ppid:1 flags:0x00004002
[290733.567418] Call Trace:
[290733.567827] <TASK>
[290733.568220] __schedule+0x222/0x670
[290733.568723] schedule+0x2c/0xb0
[290733.569195] schedule_timeout+0x146/0x160
[290733.569737] __wait_for_common+0x8f/0x1d0
[290733.570269] ? __pfx_schedule_timeout+0x10/0x10
[290733.570844] stop_two_cpus+0x177/0x1b0
[290733.571370] ? __pfx_migrate_swap_stop+0x10/0x10
[290733.571946] ? __pfx_multi_cpu_stop+0x10/0x10
[290733.572522] ? migrate_swap+0xb4/0x130
[290733.573029] ? __pfx_multi_cpu_stop+0x10/0x10
[290733.573607] ? migrate_swap+0xb4/0x130
[290733.574113] migrate_swap+0xb4/0x130
[290733.574633] task_numa_migrate.isra.0+0x65a/0x960
[290733.575215] task_numa_fault+0x28c/0x340
[290733.575734] do_numa_page+0x450/0x510
[290733.576232] __handle_mm_fault+0x300/0x720
[290733.576757] handle_mm_fault+0x105/0x270
[290733.577269] do_user_addr_fault+0x215/0x690
[290733.577804] ? syscall_exit_work+0xff/0x130
[290733.578346] exc_page_fault+0x6c/0x160
[290733.578840] asm_exc_page_fault+0x22/0x30
[290733.579367] RIP: 0033:0x7f37eac964bf
[290733.579849] RSP: 002b:00007f37e97ffa80 EFLAGS: 00010206
[290733.580475] RAX: 0000000000005860 RBX: 0000000000000110 RCX: 0000000000005861
[290733.581229] RDX: 00007f37dc0007a0 RSI: 000000003e380000 RDI: 00007f37dc07f500
[290733.581988] RBP: 00007f37dc000080 R08: 0000000000000003 R09: 0000000000000000
[290733.582807] R10: 0000000000000200 R11: 00007f37dc000020 R12: 0000000000005860
[290733.583586] R13: 0000000000000100 R14: 0000000000005860 R15: 00007f37dc07f500
[290733.584347] </TASK>
[290733.584729] INFO: task rs:main Q:Reg:18869 blocked for more than 122 seconds.
[290733.585505] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.586230] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.587050] task:rs:main Q:Reg state:D stack:0 pid:18869 tgid:18865 ppid:1 flags:0x00000002
[290733.588009] Call Trace:
[290733.588449] <TASK>
[290733.588833] __schedule+0x222/0x670
[290733.589340] schedule+0x2c/0xb0
[290733.589807] wait_extent_bit.constprop.0+0x1c7/0x210 [btrfs]
[290733.590527] ? __pfx_autoremove_wake_function+0x10/0x10
[290733.591144] lock_extent+0x88/0xe0 [btrfs]
[290733.591740] lock_and_cleanup_extent_if_need+0x77/0x340 [btrfs]
[290733.592484] btrfs_buffered_write+0x2f1/0x890 [btrfs]
[290733.593140] btrfs_do_write_iter+0x197/0x1e0 [btrfs]
[290733.593805] vfs_write+0x308/0x420
[290733.594301] ksys_write+0x5f/0xe0
[290733.594782] do_syscall_64+0x5d/0x170
[290733.595284] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[290733.595889] RIP: 0033:0x7f37eacfda8f
[290733.596403] RSP: 002b:00007f37e8bff890 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[290733.597194] RAX: ffffffffffffffda RBX: 00007f37d0027360 RCX: 00007f37eacfda8f
[290733.597963] RDX: 0000000000000049 RSI: 00007f37d0027360 RDI: 000000000000000a
[290733.598785] RBP: 00007f37d0014080 R08: 0000000000000000 R09: 0000000000000000
[290733.599570] R10: 000000000000001d R11: 0000000000000293 R12: 0000000000000000
[290733.600344] R13: 000055b293636be0 R14: 00007f37d002e200 R15: 0000000000000049
[290733.601107] </TASK>
[290733.601522] INFO: task event_loop:18961 blocked for more than 122 seconds.
[290733.602270] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.603012] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.603896] task:event_loop state:D stack:0 pid:18961 tgid:18899 ppid:1 flags:0x00000002
[290733.604865] Call Trace:
[290733.605298] <TASK>
[290733.605705] __schedule+0x222/0x670
[290733.606222] schedule+0x2c/0xb0
[290733.606854] wait_extent_bit.constprop.0+0x1c7/0x210 [btrfs]
[290733.607592] ? __pfx_autoremove_wake_function+0x10/0x10
[290733.608235] lock_extent+0x88/0xe0 [btrfs]
[290733.608848] lock_and_cleanup_extent_if_need+0x77/0x340 [btrfs]
[290733.609613] btrfs_buffered_write+0x2f1/0x890 [btrfs]
[290733.610292] ? update_load_avg+0x7e/0x7a0
[290733.610849] btrfs_do_write_iter+0x197/0x1e0 [btrfs]
[290733.611532] vfs_write+0x308/0x420
[290733.612048] ksys_write+0x5f/0xe0
[290733.612576] do_syscall_64+0x5d/0x170
[290733.613095] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[290733.613751] RIP: 0033:0x7fab292fda8f
[290733.614277] RSP: 002b:00007fab152fd7f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000001
[290733.615120] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fab292fda8f
[290733.615940] RDX: 0000000000000414 RSI: 00007faad82c55a0 RDI: 0000000000000035
[290733.616764] RBP: 0000000000000414 R08: 0000000000000000 R09: 0000000000000000
[290733.617574] R10: 00005635a3812060 R11: 0000000000000293 R12: 0000000000000000
[290733.618365] R13: 0000000000000414 R14: 00007faad82c55a0 R15: 00007fab152fda00
[290733.619146] </TASK>
[290733.619783] INFO: task 9_dirty_io_sche:52907 blocked for more than 122 seconds.
[290733.620595] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.621354] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.622197] task:9_dirty_io_sche state:D stack:0 pid:52907 tgid:52842 ppid:1 flags:0x00000002
[290733.623159] Call Trace:
[290733.623625] <TASK>
[290733.624036] __schedule+0x222/0x670
[290733.624579] schedule+0x2c/0xb0
[290733.625066] wait_extent_bit.constprop.0+0x1c7/0x210 [btrfs]
[290733.625814] ? __pfx_autoremove_wake_function+0x10/0x10
[290733.626492] lock_extent+0x88/0xe0 [btrfs]
[290733.627088] lock_and_cleanup_extent_if_need+0x77/0x340 [btrfs]
[290733.627859] btrfs_buffered_write+0x2f1/0x890 [btrfs]
[290733.628567] btrfs_do_write_iter+0x197/0x1e0 [btrfs]
[290733.629242] ? __futex_wait+0x142/0x1b0
[290733.629791] do_iter_readv_writev+0x10a/0x200
[290733.630387] vfs_writev+0x124/0x2e0
[290733.630902] ? rseq_get_rseq_cs+0x1d/0x240
[290733.631522] ? rseq_ip_fixup+0x6d/0x1d0
[290733.632056] ? do_futex+0x138/0x1d0
[290733.632598] ? do_writev+0x6b/0x110
[290733.633107] do_writev+0x6b/0x110
[290733.633626] do_syscall_64+0x5d/0x170
[290733.634136] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[290733.634786] RIP: 0033:0x7f6356f0368d
[290733.635299] RSP: 002b:00007f62ff0d69f0 EFLAGS: 00000293 ORIG_RAX: 0000000000000014
[290733.636109] RAX: ffffffffffffffda RBX: 00007f62ff0d75a0 RCX: 00007f6356f0368d
[290733.636897] RDX: 0000000000000001 RSI: 00007f62ff0d6a98 RDI: 0000000000000018
[290733.637690] RBP: 0000000080000000 R08: 0000000000000000 R09: 00007f62f265cf50
[290733.638487] R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000000
[290733.639265] R13: 0000000000000001 R14: 00007f62ff0d6a98 R15: 00005564dc7adee0
[290733.640045] </TASK>
[290733.640475] INFO: task varnishlog:2301077 blocked for more than 123 seconds.
[290733.641246] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.641998] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.642847] task:varnishlog state:D stack:0 pid:2301077 tgid:2301077 ppid:1 flags:0x00000002
[290733.643871] Call Trace:
[290733.644322] <TASK>
[290733.644731] __schedule+0x222/0x670
[290733.645250] schedule+0x2c/0xb0
[290733.645740] wait_extent_bit.constprop.0+0x1c7/0x210 [btrfs]
[290733.646480] ? __pfx_autoremove_wake_function+0x10/0x10
[290733.647125] lock_extent+0x88/0xe0 [btrfs]
[290733.647741] lock_and_cleanup_extent_if_need+0x77/0x340 [btrfs]
[290733.648512] btrfs_buffered_write+0x2f1/0x890 [btrfs]
[290733.649187] ? from_kgid_munged+0xe/0x20
[290733.649734] btrfs_do_write_iter+0x197/0x1e0 [btrfs]
[290733.650406] vfs_write+0x308/0x420
[290733.650911] ksys_write+0x5f/0xe0
[290733.651427] do_syscall_64+0x5d/0x170
[290733.651941] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[290733.652620] RIP: 0033:0x7f57722fda57
[290733.653125] RSP: 002b:00007ffdc5711d98 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[290733.653959] RAX: ffffffffffffffda RBX: 00000000000001a6 RCX: 00007f57722fda57
[290733.654787] RDX: 00000000000001a6 RSI: 000055ef840de970 RDI: 0000000000000003
[290733.655579] RBP: 000055ef840de970 R08: 0000000000000002 R09: 00007f57723b14e0
[290733.656370] R10: 00007f57723b13e0 R11: 0000000000000246 R12: 00000000000001a6
[290733.657143] R13: 000055ef840d45a0 R14: 00000000000001a6 R15: 00007f57723f69e0
[290733.657939] </TASK>
[290733.658601] INFO: task kworker/u97:2:3833788 blocked for more than 123 seconds.
[290733.659408] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.660161] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.661006] task:kworker/u97:2 state:D stack:0 pid:3833788 tgid:3833788 ppid:2 flags:0x00004000
[290733.661999] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290733.662850] Call Trace:
[290733.663296] <TASK>
[290733.663718] __schedule+0x222/0x670
[290733.664241] ? leaf_space_used+0xad/0xd0 [btrfs]
[290733.664894] schedule+0x2c/0xb0
[290733.665396] schedule_preempt_disabled+0x11/0x20
[290733.666001] rwsem_down_read_slowpath+0x260/0x4f0
[290733.666641] down_read+0x45/0xa0
[290733.667141] __btrfs_tree_read_lock+0x19/0x80 [btrfs]
[290733.667838] btrfs_read_lock_root_node+0x34/0x90 [btrfs]
[290733.668571] btrfs_search_slot+0x127/0xbd0 [btrfs]
[290733.669237] ? btrfs_alloc_reserved_file_extent+0x8c/0xf0 [btrfs]
[290733.670001] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290733.670678] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290733.671374] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290733.672030] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290733.672762] btrfs_work_helper+0xba/0x190 [btrfs]
[290733.673441] process_one_work+0x193/0x3d0
[290733.673989] worker_thread+0x2fc/0x410
[290733.674541] ? __pfx_worker_thread+0x10/0x10
[290733.675124] kthread+0xdc/0x110
[290733.675634] ? __pfx_kthread+0x10/0x10
[290733.676148] ret_from_fork+0x2d/0x50
[290733.676684] ? __pfx_kthread+0x10/0x10
[290733.677207] ret_from_fork_asm+0x1a/0x30
[290733.677744] </TASK>
[290733.678141] INFO: task kworker/u97:6:3860155 blocked for more than 123 seconds.
[290733.678947] Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290733.679709] ""echo 0 > /proc/sys/kernel/hung_task_timeout_secs"" disables this message.
[290733.680563] task:kworker/u97:6 state:D stack:0 pid:3860155 tgid:3860155 ppid:2 flags:0x00004000
[290733.681548] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290733.682305] Call Trace:
[290733.682748] <TASK>
[290733.683151] __schedule+0x222/0x670
[290733.683683] ? leaf_space_used+0xad/0xd0 [btrfs]
[290733.684332] schedule+0x2c/0xb0
[290733.684820] schedule_preempt_disabled+0x11/0x20
[290733.685435] rwsem_down_read_slowpath+0x260/0x4f0
[290733.686027] down_read+0x45/0xa0
[290733.686540] __btrfs_tree_read_lock+0x19/0x80 [btrfs]
[290733.687211] btrfs_read_lock_root_node+0x34/0x90 [btrfs]
[290733.687901] btrfs_search_slot+0x127/0xbd0 [btrfs]
[290733.688565] ? btrfs_alloc_reserved_file_extent+0x8c/0xf0 [btrfs]
[290733.689321] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290733.689973] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290733.690717] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290733.691368] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290733.692045] btrfs_work_helper+0xba/0x190 [btrfs]
[290733.692745] process_one_work+0x193/0x3d0
[290733.693277] worker_thread+0x2fc/0x410
[290733.693794] ? __pfx_worker_thread+0x10/0x10
[290733.694359] kthread+0xdc/0x110
[290733.694821] ? __pfx_kthread+0x10/0x10
[290733.695338] ret_from_fork+0x2d/0x50
[290733.695833] ? __pfx_kthread+0x10/0x10
[290733.696361] ret_from_fork_asm+0x1a/0x30
[290733.696882] </TASK>
[290733.697277] Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[290746.445726] watchdog: BUG: soft lockup - CPU#12 stuck for 153s! [migration/12:55]
[290746.446435] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290746.446468] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290746.452222] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290746.452906] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290746.453564] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290746.454131] RIP: 0010:stop_machine_yield+0x2/0x10
[290746.454597] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290746.455995] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290746.456527] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290746.457121] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290746.457720] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290746.458318] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290746.458906] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290746.459526] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290746.460160] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290746.460682] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290746.461280] PKRU: 55555554
[290746.461644] Call Trace:
[290746.462001] <IRQ>
[290746.462357] ? watchdog_timer_fn+0x1d7/0x250
[290746.462799] ? __pfx_watchdog_timer_fn+0x10/0x10
[290746.463268] ? __hrtimer_run_queues+0x10f/0x2b0
[290746.463722] ? hrtimer_interrupt+0xff/0x240
[290746.464159] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290746.464655] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290746.465134] </IRQ>
[290746.465475] <TASK>
[290746.465798] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290746.466298] ? stop_machine_yield+0x2/0x10
[290746.466726] multi_cpu_stop+0x94/0x100
[290746.467147] ? __pfx_multi_cpu_stop+0x10/0x10
[290746.467599] cpu_stopper_thread+0x90/0x140
[290746.468024] ? __pfx_smpboot_thread_fn+0x10/0x10
[290746.468512] smpboot_thread_fn+0xd4/0x1d0
[290746.468931] kthread+0xdc/0x110
[290746.469336] ? __pfx_kthread+0x10/0x10
[290746.469745] ret_from_fork+0x2d/0x50
[290746.470150] ? __pfx_kthread+0x10/0x10
[290746.470569] ret_from_fork_asm+0x1a/0x30
[290746.470986] </TASK>
[290966.445495] watchdog: BUG: soft lockup - CPU#22 stuck for 380s! [kworker/u100:5:3816805]
[290966.446379] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290966.446412] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290966.452628] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290966.453400] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290966.454057] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290966.454637] RIP: 0010:xas_load+0x30/0x50
[290966.455077] Code: 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe 48 8d 58 fe 38 4d 10 <77> ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b 00 75 c8 5b 5d c3 cc
[290966.456502] RSP: 0018:ffffb956e260f810 EFLAGS: 00000246
[290966.457014] RAX: ffff904df9a5c492 RBX: ffff904df9a5c490 RCX: 0000000000000000
[290966.457598] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[290966.458209] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290966.458788] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290966.459429] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290966.460029] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290966.460659] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290966.461201] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290966.461780] PKRU: 55555554
[290966.462173] Call Trace:
[290966.462514] <IRQ>
[290966.462828] ? watchdog_timer_fn+0x1d7/0x250
[290966.463343] ? __pfx_watchdog_timer_fn+0x10/0x10
[290966.463797] ? __hrtimer_run_queues+0x10f/0x2b0
[290966.464284] ? hrtimer_interrupt+0xff/0x240
[290966.464886] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290966.465383] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290966.465849] </IRQ>
[290966.466203] <TASK>
[290966.466522] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290966.467017] ? xas_load+0x30/0x50
[290966.467403] filemap_get_entry+0x6a/0x140
[290966.467821] __filemap_get_folio+0x33/0x2d0
[290966.468274] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290966.468802] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290966.469328] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290966.469825] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290966.470359] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290966.470858] btrfs_cow_block+0xc7/0x280 [btrfs]
[290966.471347] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290966.471830] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290966.472349] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290966.472855] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290966.473362] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290966.473887] btrfs_work_helper+0xba/0x190 [btrfs]
[290966.474391] process_one_work+0x193/0x3d0
[290966.474803] worker_thread+0x2fc/0x410
[290966.475231] ? __pfx_worker_thread+0x10/0x10
[290966.475645] kthread+0xdc/0x110
[290966.476018] ? __pfx_kthread+0x10/0x10
[290966.476395] ret_from_fork+0x2d/0x50
[290966.476755] ? __pfx_kthread+0x10/0x10
[290966.477153] ret_from_fork_asm+0x1a/0x30
[290966.477525] </TASK>
[291082.444589] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291082.444621] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291082.450372] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291082.451041] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291082.451692] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291082.452240] RIP: 0010:rcu_preempt_deferred_qs+0x11/0x40
[291082.452722] Code: ff ff ff c3 cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 65 8a 05 ca ab c7 46 84 c0 75 0a 8b 87 38 08 00 00 <85> c0 74 1a 65 48 8b 05 23 8e c7 46 8b 80 34 08 00 00 85 c0 75 08
[291082.454092] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291082.454601] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291082.455181] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffff9065c3203400
[291082.455778] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291082.456361] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291082.456950] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291082.457555] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291082.458183] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291082.458721] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291082.459308] PKRU: 55555554
[291082.459677] Call Trace:
[291082.460024] <IRQ>
[291082.460355] ? watchdog_timer_fn+0x1d7/0x250
[291082.460806] ? __pfx_watchdog_timer_fn+0x10/0x10
[291082.461265] ? __hrtimer_run_queues+0x10f/0x2b0
[291082.461727] ? hrtimer_interrupt+0xff/0x240
[291082.462158] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291082.462678] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291082.463149] </IRQ>
[291082.463500] <TASK>
[291082.463825] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291082.464321] ? rcu_preempt_deferred_qs+0x11/0x40
[291082.464786] multi_cpu_stop+0xb2/0x100
[291082.465194] ? __pfx_multi_cpu_stop+0x10/0x10
[291082.465661] cpu_stopper_thread+0x90/0x140
[291082.466084] ? __pfx_smpboot_thread_fn+0x10/0x10
[291082.466561] smpboot_thread_fn+0xd4/0x1d0
[291082.466982] kthread+0xdc/0x110
[291082.467363] ? __pfx_kthread+0x10/0x10
[291082.467780] ret_from_fork+0x2d/0x50
[291082.468178] ? __pfx_kthread+0x10/0x10
[291082.468617] ret_from_fork_asm+0x1a/0x30
[291082.469034] </TASK>
[291090.444815] watchdog: BUG: soft lockup - CPU#22 stuck for 496s! [kworker/u100:5:3816805]
[291090.445657] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291090.445690] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291090.451927] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291090.452698] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291090.453367] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291090.453961] RIP: 0010:xas_descend+0x1b/0x80
[291090.454438] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc
[291090.455868] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291090.456382] RAX: ffffe4bc7bfa2580 RBX: ffff904df9a5c490 RCX: 0000000000000028
[291090.456965] RDX: 0000000000000002 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291090.457622] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291090.458209] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291090.458804] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291090.459403] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291090.460035] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291090.460582] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291090.461168] PKRU: 55555554
[291090.461562] Call Trace:
[291090.461909] <IRQ>
[291090.462233] ? watchdog_timer_fn+0x1d7/0x250
[291090.462686] ? __pfx_watchdog_timer_fn+0x10/0x10
[291090.463134] ? __hrtimer_run_queues+0x10f/0x2b0
[291090.463620] ? hrtimer_interrupt+0xff/0x240
[291090.464049] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291090.464568] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291090.465038] </IRQ>
[291090.465380] <TASK>
[291090.465703] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291090.466188] ? xas_descend+0x1b/0x80
[291090.466609] xas_load+0x3d/0x50
[291090.466986] filemap_get_entry+0x6a/0x140
[291090.467445] __filemap_get_folio+0x33/0x2d0
[291090.467875] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291090.468443] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291090.468944] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291090.469476] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291090.469984] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291090.470519] btrfs_cow_block+0xc7/0x280 [btrfs]
[291090.470990] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291090.471507] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291090.471989] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291090.472533] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291090.473019] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291090.473575] btrfs_work_helper+0xba/0x190 [btrfs]
[291090.474057] process_one_work+0x193/0x3d0
[291090.474497] worker_thread+0x2fc/0x410
[291090.474885] ? __pfx_worker_thread+0x10/0x10
[291090.475299] kthread+0xdc/0x110
[291090.475662] ? __pfx_kthread+0x10/0x10
[291090.476028] ret_from_fork+0x2d/0x50
[291090.476419] ? __pfx_kthread+0x10/0x10
[291090.476781] ret_from_fork_asm+0x1a/0x30
[291090.477146] </TASK>
[291118.444663] watchdog: BUG: soft lockup - CPU#22 stuck for 522s! [kworker/u100:5:3816805]
[291118.445598] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291118.445630] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291118.451776] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291118.452549] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291118.453213] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291118.453803] RIP: 0010:xas_descend+0x1b/0x80
[291118.454276] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc
[291118.455711] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291118.456225] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000022
[291118.456812] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[291118.457607] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291118.458208] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291118.458794] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291118.459443] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291118.460334] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291118.461014] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291118.461832] PKRU: 55555554
[291118.462297] Call Trace:
[291118.462713] <IRQ>
[291118.463099] ? watchdog_timer_fn+0x1d7/0x250
[291118.463670] ? __pfx_watchdog_timer_fn+0x10/0x10
[291118.464282] ? __hrtimer_run_queues+0x10f/0x2b0
[291118.464856] ? hrtimer_interrupt+0xff/0x240
[291118.465433] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291118.466070] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291118.466698] </IRQ>
[291118.467091] <TASK>
[291118.467497] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291118.468135] ? xas_descend+0x1b/0x80
[291118.468647] xas_load+0x3d/0x50
[291118.469119] filemap_get_entry+0x6a/0x140
[291118.469668] __filemap_get_folio+0x33/0x2d0
[291118.470233] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291118.470919] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291118.471566] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291118.472223] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291118.472879] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291118.473451] btrfs_cow_block+0xc7/0x280 [btrfs]
[291118.473923] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291118.474481] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291118.474965] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291118.475517] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291118.476007] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291118.476553] btrfs_work_helper+0xba/0x190 [btrfs]
[291118.477040] process_one_work+0x193/0x3d0
[291118.477474] worker_thread+0x2fc/0x410
[291118.477864] ? __pfx_worker_thread+0x10/0x10
[291118.478312] kthread+0xdc/0x110
[291118.478661] ? __pfx_kthread+0x10/0x10
[291118.479030] ret_from_fork+0x2d/0x50
[291118.479413] ? __pfx_kthread+0x10/0x10
[291118.479773] ret_from_fork_asm+0x1a/0x30
[291118.480144] </TASK>
[291138.443554] watchdog: BUG: soft lockup - CPU#12 stuck for 518s! [migration/12:55]
[291138.444386] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291138.444420] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291138.451410] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291138.452104] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291138.452715] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291138.453298] RIP: 0010:stop_machine_yield+0x2/0x10
[291138.453733] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291138.455138] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291138.455618] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291138.456220] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291138.456793] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291138.457419] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291138.458006] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291138.458594] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291138.459249] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291138.459752] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291138.460362] PKRU: 55555554
[291138.460716] Call Trace:
[291138.461063] <IRQ>
[291138.461393] ? watchdog_timer_fn+0x1d7/0x250
[291138.461826] ? __pfx_watchdog_timer_fn+0x10/0x10
[291138.462305] ? __hrtimer_run_queues+0x10f/0x2b0
[291138.462750] ? hrtimer_interrupt+0xff/0x240
[291138.463213] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291138.463693] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291138.464188] </IRQ>
[291138.464510] <TASK>
[291138.464829] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291138.465349] ? stop_machine_yield+0x2/0x10
[291138.465770] multi_cpu_stop+0x94/0x100
[291138.466199] ? __pfx_multi_cpu_stop+0x10/0x10
[291138.466633] cpu_stopper_thread+0x90/0x140
[291138.467059] ? __pfx_smpboot_thread_fn+0x10/0x10
[291138.467516] smpboot_thread_fn+0xd4/0x1d0
[291138.467933] kthread+0xdc/0x110
[291138.468319] ? __pfx_kthread+0x10/0x10
[291138.468719] ret_from_fork+0x2d/0x50
[291138.469128] ? __pfx_kthread+0x10/0x10
[291138.469532] ret_from_fork_asm+0x1a/0x30
[291138.469948] </TASK>
[291242.443991] watchdog: BUG: soft lockup - CPU#22 stuck for 637s! [kworker/u100:5:3816805]
[291242.444875] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291242.444906] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291242.451406] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291242.452178] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291242.453096] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291242.453931] RIP: 0010:xas_descend+0x29/0x80
[291242.454391] Code: 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 <74> 0e 88 4f 12 c3 cc cc cc cc 48 83 fa 02 75 f2 48 3d fd 00 00 00
[291242.455870] RSP: 0018:ffffb956e260f808 EFLAGS: 00000246
[291242.456379] RAX: ffff904df9a5c492 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291242.456999] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291242.457638] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291242.458234] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291242.458873] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291242.459478] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291242.460141] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291242.460709] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291242.461307] PKRU: 55555554
[291242.461712] Call Trace:
[291242.462062] <IRQ>
[291242.462391] ? watchdog_timer_fn+0x1d7/0x250
[291242.462852] ? __pfx_watchdog_timer_fn+0x10/0x10
[291242.463315] ? __hrtimer_run_queues+0x10f/0x2b0
[291242.463812] ? hrtimer_interrupt+0xff/0x240
[291242.464255] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291242.464795] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291242.465276] </IRQ>
[291242.465644] <TASK>
[291242.465973] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291242.466478] ? xas_descend+0x29/0x80
[291242.466900] xas_load+0x3d/0x50
[291242.467283] filemap_get_entry+0x6a/0x140
[291242.467768] __filemap_get_folio+0x33/0x2d0
[291242.468211] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291242.468795] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291242.469303] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291242.469864] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291242.470396] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291242.470933] btrfs_cow_block+0xc7/0x280 [btrfs]
[291242.471421] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291242.471939] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291242.472442] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291242.472989] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291242.473494] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291242.474045] btrfs_work_helper+0xba/0x190 [btrfs]
[291242.474560] process_one_work+0x193/0x3d0
[291242.474984] worker_thread+0x2fc/0x410
[291242.475387] ? __pfx_worker_thread+0x10/0x10
[291242.475832] kthread+0xdc/0x110
[291242.476188] ? __pfx_kthread+0x10/0x10
[291242.476609] ret_from_fork+0x2d/0x50
[291242.476978] ? __pfx_kthread+0x10/0x10
[291242.477348] ret_from_fork_asm+0x1a/0x30
[291242.477767] </TASK>
[291250.442948] watchdog: BUG: soft lockup - CPU#12 stuck for 623s! [migration/12:55]
[291250.443683] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291250.443717] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291250.449502] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291250.450163] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291250.450808] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291250.451354] RIP: 0010:stop_machine_yield+0x2/0x10
[291250.451801] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291250.453166] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291250.453677] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291250.454246] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291250.454887] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291250.455467] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291250.456049] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291250.456655] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291250.457275] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291250.457817] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291250.458399] PKRU: 55555554
[291250.458764] Call Trace:
[291250.459104] <IRQ>
[291250.459431] ? watchdog_timer_fn+0x1d7/0x250
[291250.459880] ? __pfx_watchdog_timer_fn+0x10/0x10
[291250.460333] ? __hrtimer_run_queues+0x10f/0x2b0
[291250.460794] ? hrtimer_interrupt+0xff/0x240
[291250.461221] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291250.461768] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291250.462235] </IRQ>
[291250.462626] <TASK>
[291250.462945] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291250.463439] ? stop_machine_yield+0x2/0x10
[291250.463871] multi_cpu_stop+0x94/0x100
[291250.464274] ? __pfx_multi_cpu_stop+0x10/0x10
[291250.464747] cpu_stopper_thread+0x90/0x140
[291250.465169] ? __pfx_smpboot_thread_fn+0x10/0x10
[291250.465646] smpboot_thread_fn+0xd4/0x1d0
[291250.466062] kthread+0xdc/0x110
[291250.466442] ? __pfx_kthread+0x10/0x10
[291250.466858] ret_from_fork+0x2d/0x50
[291250.467252] ? __pfx_kthread+0x10/0x10
[291250.467687] ret_from_fork_asm+0x1a/0x30
[291250.468101] </TASK>
[291450.442884] watchdog: BUG: soft lockup - CPU#22 stuck for 831s! [kworker/u100:5:3816805]
[291450.443823] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291450.443856] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291450.450232] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291450.450999] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291450.451693] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291450.452286] RIP: 0010:xas_load+0xd/0x50
[291450.452731] Code: 48 8b 40 08 c3 cc cc cc cc 0f 1f 40 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 <83> e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc
[291450.454182] RSP: 0018:ffffb956e260f810 EFLAGS: 00000206
[291450.454727] RAX: ffff904df9a5c492 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291450.455332] RDX: ffff904df9a5c492 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291450.456031] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291450.456655] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291450.457251] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291450.457865] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291450.458532] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291450.459051] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291450.459682] PKRU: 55555554
[291450.460046] Call Trace:
[291450.460413] <IRQ>
[291450.460740] ? watchdog_timer_fn+0x1d7/0x250
[291450.461182] ? __pfx_watchdog_timer_fn+0x10/0x10
[291450.461682] ? __hrtimer_run_queues+0x10f/0x2b0
[291450.462133] ? hrtimer_interrupt+0xff/0x240
[291450.462605] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291450.463089] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291450.463642] </IRQ>
[291450.463968] <TASK>
[291450.464300] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291450.464805] ? xas_load+0xd/0x50
[291450.465183] filemap_get_entry+0x6a/0x140
[291450.465639] __filemap_get_folio+0x33/0x2d0
[291450.466065] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291450.466633] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291450.467132] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291450.467669] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291450.468172] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291450.468886] btrfs_cow_block+0xc7/0x280 [btrfs]
[291450.469361] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291450.469860] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291450.470346] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291450.470865] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291450.471352] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291450.471887] btrfs_work_helper+0xba/0x190 [btrfs]
[291450.472382] process_one_work+0x193/0x3d0
[291450.472824] worker_thread+0x2fc/0x410
[291450.473218] ? __pfx_worker_thread+0x10/0x10
[291450.473702] kthread+0xdc/0x110
[291450.474052] ? __pfx_kthread+0x10/0x10
[291450.474447] ret_from_fork+0x2d/0x50
[291450.474854] ? __pfx_kthread+0x10/0x10
[291450.475217] ret_from_fork_asm+0x1a/0x30
[291450.475619] </TASK>
[291474.441757] watchdog: BUG: soft lockup - CPU#12 stuck for 831s! [migration/12:55]
[291474.442500] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291474.442533] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291474.448277] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291474.448938] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291474.449577] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291474.450118] RIP: 0010:stop_machine_yield+0x2/0x10
[291474.450584] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291474.451944] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291474.452445] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291474.453013] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291474.453615] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291474.454196] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291474.454780] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291474.455381] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291474.456004] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291474.456550] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291474.457127] PKRU: 55555554
[291474.457501] Call Trace:
[291474.457842] <IRQ>
[291474.458169] ? watchdog_timer_fn+0x1d7/0x250
[291474.458617] ? __pfx_watchdog_timer_fn+0x10/0x10
[291474.459067] ? __hrtimer_run_queues+0x10f/0x2b0
[291474.459549] ? hrtimer_interrupt+0xff/0x240
[291474.459976] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291474.460487] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291474.460956] </IRQ>
[291474.461289] <TASK>
[291474.461612] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291474.462097] ? stop_machine_yield+0x2/0x10
[291474.462561] multi_cpu_stop+0x94/0x100
[291474.462964] ? __pfx_multi_cpu_stop+0x10/0x10
[291474.463439] cpu_stopper_thread+0x90/0x140
[291474.463860] ? __pfx_smpboot_thread_fn+0x10/0x10
[291474.464322] smpboot_thread_fn+0xd4/0x1d0
[291474.464741] kthread+0xdc/0x110
[291474.465115] ? __pfx_kthread+0x10/0x10
[291474.465549] ret_from_fork+0x2d/0x50
[291474.465943] ? __pfx_kthread+0x10/0x10
[291474.466376] ret_from_fork_asm+0x1a/0x30
[291474.466792] </TASK>
[290730.446813] watchdog: BUG: soft lockup - CPU#22 stuck for 160s! [kworker/u100:5:3816805]
[290730.448037] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290730.448087] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290730.458017] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290730.459187] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290730.460216] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290730.461148] RIP: 0010:xas_load+0x29/0x50
[290730.461837] Code: 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe <48> 8d 58 fe 38 4d 10 77 ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b
[290730.464453] RSP: 0018:ffffb956e260f810 EFLAGS: 00000282
[290730.465234] RAX: ffff904df9a5c492 RBX: ffff9066c9ceada0 RCX: 0000000000000000
[290730.466172] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[290730.467118] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290730.468054] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290730.469005] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290730.469948] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290730.470966] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290730.471801] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290730.472750] PKRU: 55555554
[290730.473312] Call Trace:
[290730.473866] <IRQ>
[290730.474375] ? watchdog_timer_fn+0x1d7/0x250
[290730.475068] ? __pfx_watchdog_timer_fn+0x10/0x10
[290730.475804] ? __hrtimer_run_queues+0x10f/0x2b0
[290730.476530] ? hrtimer_interrupt+0xff/0x240
[290730.477218] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290730.478001] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290730.478779] </IRQ>
[290730.479290] <TASK>
[290730.479810] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290730.480604] ? xas_load+0x29/0x50
[290730.481211] filemap_get_entry+0x6a/0x140
[290730.481895] __filemap_get_folio+0x33/0x2d0
[290730.482594] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290730.483440] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290730.484229] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290730.485042] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290730.485874] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290730.486701] btrfs_cow_block+0xc7/0x280 [btrfs]
[290730.487462] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290730.488252] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290730.489058] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290730.489905] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290730.490707] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290730.491602] btrfs_work_helper+0xba/0x190 [btrfs]
[290730.492392] process_one_work+0x193/0x3d0
[290730.493091] worker_thread+0x2fc/0x410
[290730.493757] ? __pfx_worker_thread+0x10/0x10
[290730.494443] kthread+0xdc/0x110
[290730.495026] ? __pfx_kthread+0x10/0x10
[290730.495663] ret_from_fork+0x2d/0x50
[290730.496237] ? __pfx_kthread+0x10/0x10
[290730.496831] ret_from_fork_asm+0x1a/0x30
[290730.497439] </TASK>
[290786.446508] watchdog: BUG: soft lockup - CPU#22 stuck for 212s! [kworker/u100:5:3816805]
[290786.447763] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290786.447813] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290786.457403] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290786.458584] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290786.459641] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290786.460581] RIP: 0010:xas_descend+0x3/0x80
[290786.461283] Code: 10 48 89 07 48 c1 e8 20 48 89 57 08 c3 cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e <48> 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44
[290786.463534] RSP: 0018:ffffb956e260f808 EFLAGS: 00000297
[290786.464393] RAX: ffff9066c9ceada2 RBX: ffff9066c9ceada0 RCX: 0000000000000006
[290786.465346] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[290786.466291] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290786.467236] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290786.468168] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290786.469106] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290786.470123] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290786.470943] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290786.471883] PKRU: 55555554
[290786.472463] Call Trace:
[290786.473014] <IRQ>
[290786.473521] ? watchdog_timer_fn+0x1d7/0x250
[290786.474231] ? __pfx_watchdog_timer_fn+0x10/0x10
[290786.474946] ? __hrtimer_run_queues+0x10f/0x2b0
[290786.475681] ? hrtimer_interrupt+0xff/0x240
[290786.476390] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290786.477168] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290786.477919] </IRQ>
[290786.478446] <TASK>
[290786.478955] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290786.479746] ? xas_descend+0x3/0x80
[290786.480392] xas_load+0x3d/0x50
[290786.481002] filemap_get_entry+0x6a/0x140
[290786.481676] __filemap_get_folio+0x33/0x2d0
[290786.482378] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290786.483238] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290786.484047] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290786.484850] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290786.485672] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290786.486499] btrfs_cow_block+0xc7/0x280 [btrfs]
[290786.487311] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290786.488115] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290786.488896] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290786.489721] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290786.490525] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290786.491368] btrfs_work_helper+0xba/0x190 [btrfs]
[290786.492147] process_one_work+0x193/0x3d0
[290786.492801] worker_thread+0x2fc/0x410
[290786.493445] ? __pfx_worker_thread+0x10/0x10
[290786.494109] kthread+0xdc/0x110
[290786.494661] ? __pfx_kthread+0x10/0x10
[290786.495273] ret_from_fork+0x2d/0x50
[290786.495843] ? __pfx_kthread+0x10/0x10
[290786.496441] ret_from_fork_asm+0x1a/0x30
[290786.497044] </TASK>
[290799.451812] rcu: INFO: rcu_preempt self-detected stall on CPU
[290799.452732] rcu: 22-....: (239606 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=55331
[290799.453936] rcu: (t=240045 jiffies g=38283501 q=131195 ncpus=24)
[290799.454732] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290799.455853] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290799.456818] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290799.457682] RIP: 0010:xas_load+0x29/0x50
[290799.458317] Code: 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe <48> 8d 58 fe 38 4d 10 77 ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b
[290799.460423] RSP: 0018:ffffb956e260f810 EFLAGS: 00000286
[290799.461160] RAX: ffff9056ce8d4922 RBX: ffff9074f9f11ff0 RCX: 000000000000000c
[290799.462036] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[290799.462910] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290799.463769] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290799.464679] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290799.465566] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290799.466548] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290799.467329] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290799.468227] PKRU: 55555554
[290799.468739] Call Trace:
[290799.469255] <IRQ>
[290799.469704] ? rcu_dump_cpu_stacks+0xcb/0x110
[290799.470421] ? print_cpu_stall+0x111/0x260
[290799.471055] ? check_cpu_stall+0x10f/0x220
[290799.471674] ? rcu_pending+0x29/0x190
[290799.472281] ? rcu_sched_clock_irq+0xb8/0x350
[290799.472933] ? update_process_times+0x69/0xa0
[290799.473574] ? tick_nohz_handler+0xc5/0x130
[290799.474220] ? __pfx_tick_nohz_handler+0x10/0x10
[290799.474875] ? __hrtimer_run_queues+0x10f/0x2b0
[290799.475543] ? hrtimer_interrupt+0xff/0x240
[290799.476201] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290799.476926] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290799.477618] </IRQ>
[290799.478078] <TASK>
[290799.478533] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290799.479266] ? xas_load+0x29/0x50
[290799.479822] filemap_get_entry+0x6a/0x140
[290799.480452] __filemap_get_folio+0x33/0x2d0
[290799.481089] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290799.481876] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290799.482621] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290799.483380] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290799.484154] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290799.484894] btrfs_cow_block+0xc7/0x280 [btrfs]
[290799.485600] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290799.486345] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290799.487062] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290799.487822] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290799.488553] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290799.489347] btrfs_work_helper+0xba/0x190 [btrfs]
[290799.490075] process_one_work+0x193/0x3d0
[290799.490684] worker_thread+0x2fc/0x410
[290799.491287] ? __pfx_worker_thread+0x10/0x10
[290799.491912] kthread+0xdc/0x110
[290799.492433] ? __pfx_kthread+0x10/0x10
[290799.493000] ret_from_fork+0x2d/0x50
[290799.493541] ? __pfx_kthread+0x10/0x10
[290799.494100] ret_from_fork_asm+0x1a/0x30
[290799.494652] </TASK>
[290802.445421] watchdog: BUG: soft lockup - CPU#12 stuck for 205s! [migration/12:55]
[290802.446167] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290802.446200] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290802.452374] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290802.453081] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290802.453936] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290802.454662] RIP: 0010:stop_machine_yield+0x2/0x10
[290802.455285] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290802.457167] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290802.457788] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290802.458576] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290802.459298] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290802.460115] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290802.460904] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290802.461677] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290802.462570] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290802.463284] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290802.464103] PKRU: 55555554
[290802.464536] Call Trace:
[290802.464959] <IRQ>
[290802.465338] ? watchdog_timer_fn+0x1d7/0x250
[290802.465908] ? __pfx_watchdog_timer_fn+0x10/0x10
[290802.466492] ? __hrtimer_run_queues+0x10f/0x2b0
[290802.467080] ? hrtimer_interrupt+0xff/0x240
[290802.467572] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290802.468082] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290802.468551] </IRQ>
[290802.468880] <TASK>
[290802.469209] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290802.469693] ? stop_machine_yield+0x2/0x10
[290802.470147] multi_cpu_stop+0x94/0x100
[290802.470552] ? __pfx_multi_cpu_stop+0x10/0x10
[290802.471010] cpu_stopper_thread+0x90/0x140
[290802.471435] ? __pfx_smpboot_thread_fn+0x10/0x10
[290802.471895] smpboot_thread_fn+0xd4/0x1d0
[290802.472315] kthread+0xdc/0x110
[290802.472690] ? __pfx_kthread+0x10/0x10
[290802.473131] ret_from_fork+0x2d/0x50
[290802.473524] ? __pfx_kthread+0x10/0x10
[290802.473938] ret_from_fork_asm+0x1a/0x30
[290802.474354] </TASK>
[290970.444473] watchdog: BUG: soft lockup - CPU#12 stuck for 362s! [migration/12:55]
[290970.445287] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290970.445320] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290970.451183] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290970.451860] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290970.452496] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290970.453074] RIP: 0010:stop_machine_yield+0x2/0x10
[290970.453657] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290970.455463] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290970.455946] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290970.456536] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290970.457152] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290970.457737] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290970.458487] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290970.459331] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290970.460231] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290970.460915] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290970.461707] PKRU: 55555554
[290970.462164] Call Trace:
[290970.462582] <IRQ>
[290970.462984] ? watchdog_timer_fn+0x1d7/0x250
[290970.463552] ? __pfx_watchdog_timer_fn+0x10/0x10
[290970.464190] ? __hrtimer_run_queues+0x10f/0x2b0
[290970.464771] ? hrtimer_interrupt+0xff/0x240
[290970.465349] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290970.465999] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290970.466614] </IRQ>
[290970.467017] <TASK>
[290970.467405] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290970.468062] ? stop_machine_yield+0x2/0x10
[290970.468608] multi_cpu_stop+0x94/0x100
[290970.469179] ? __pfx_multi_cpu_stop+0x10/0x10
[290970.469742] cpu_stopper_thread+0x90/0x140
[290970.470301] ? __pfx_smpboot_thread_fn+0x10/0x10
[290970.470892] smpboot_thread_fn+0xd4/0x1d0
[290970.471435] kthread+0xdc/0x110
[290970.471907] ? __pfx_kthread+0x10/0x10
[290970.472428] ret_from_fork+0x2d/0x50
[290970.472935] ? __pfx_kthread+0x10/0x10
[290970.473455] ret_from_fork_asm+0x1a/0x30
[290970.473995] </TASK>
[290979.497797] rcu: INFO: rcu_preempt self-detected stall on CPU
[290979.498506] rcu: 22-....: (419345 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=105681
[290979.499310] rcu: (t=420091 jiffies g=38283501 q=236595 ncpus=24)
[290979.499855] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290979.500592] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290979.501275] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290979.501853] RIP: 0010:xas_start+0x0/0xd0
[290979.502276] Code: 00 00 c3 cc cc cc cc 49 83 c1 01 4c 89 4f 08 c3 cc cc cc cc 4c 89 c0 eb a5 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <48> 8b 57 18 48 89 d0 83 e0 03 74 6b 48 83 f8 02 75 09 48 81 fa 05
[290979.503650] RSP: 0018:ffffb956e260f808 EFLAGS: 00000246
[290979.504158] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000028
[290979.504721] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[290979.505313] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290979.505879] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290979.506453] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290979.507045] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290979.507644] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290979.508163] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290979.508712] PKRU: 55555554
[290979.509071] Call Trace:
[290979.509383] <IRQ>
[290979.509671] ? rcu_dump_cpu_stacks+0xcb/0x110
[290979.510116] ? print_cpu_stall+0x111/0x260
[290979.510505] ? check_cpu_stall+0x10f/0x220
[290979.510895] ? rcu_pending+0x29/0x190
[290979.511279] ? rcu_sched_clock_irq+0xb8/0x350
[290979.511675] ? update_process_times+0x69/0xa0
[290979.512113] ? tick_nohz_handler+0xc5/0x130
[290979.512504] ? __pfx_tick_nohz_handler+0x10/0x10
[290979.512932] ? __hrtimer_run_queues+0x10f/0x2b0
[290979.513345] ? hrtimer_interrupt+0xff/0x240
[290979.513735] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290979.514216] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290979.514645] </IRQ>
[290979.514951] <TASK>
[290979.515235] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290979.515682] ? __pfx_xas_start+0x10/0x10
[290979.516091] xas_load+0xa/0x50
[290979.516425] filemap_get_entry+0x6a/0x140
[290979.516809] __filemap_get_folio+0x33/0x2d0
[290979.517214] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290979.517703] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290979.518191] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290979.518651] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290979.519151] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290979.519613] btrfs_cow_block+0xc7/0x280 [btrfs]
[290979.520075] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290979.520537] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290979.521014] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290979.521485] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290979.521948] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290979.522427] btrfs_work_helper+0xba/0x190 [btrfs]
[290979.522873] process_one_work+0x193/0x3d0
[290979.523261] worker_thread+0x2fc/0x410
[290979.523616] ? __pfx_worker_thread+0x10/0x10
[290979.524032] kthread+0xdc/0x110
[290979.524355] ? __pfx_kthread+0x10/0x10
[290979.524699] ret_from_fork+0x2d/0x50
[290979.525066] ? __pfx_kthread+0x10/0x10
[290979.525402] ret_from_fork_asm+0x1a/0x30
[290979.525745] </TASK>
[291026.444163] watchdog: BUG: soft lockup - CPU#12 stuck for 414s! [migration/12:55]
[291026.444990] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291026.445023] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291026.451080] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291026.451780] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291026.452399] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291026.453026] RIP: 0010:stop_machine_yield+0x2/0x10
[291026.453470] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291026.454866] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291026.455350] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291026.455964] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291026.456549] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291026.457144] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291026.457733] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291026.458315] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291026.458980] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291026.459491] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291026.460115] PKRU: 55555554
[291026.460476] Call Trace:
[291026.460852] <IRQ>
[291026.461177] ? watchdog_timer_fn+0x1d7/0x250
[291026.461622] ? __pfx_watchdog_timer_fn+0x10/0x10
[291026.462091] ? __hrtimer_run_queues+0x10f/0x2b0
[291026.462548] ? hrtimer_interrupt+0xff/0x240
[291026.462999] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291026.463485] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291026.463989] </IRQ>
[291026.464315] <TASK>
[291026.464646] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291026.465151] ? stop_machine_yield+0x2/0x10
[291026.465590] multi_cpu_stop+0x94/0x100
[291026.466014] ? __pfx_multi_cpu_stop+0x10/0x10
[291026.466455] cpu_stopper_thread+0x90/0x140
[291026.466914] ? __pfx_smpboot_thread_fn+0x10/0x10
[291026.467371] smpboot_thread_fn+0xd4/0x1d0
[291026.467828] kthread+0xdc/0x110
[291026.468212] ? __pfx_kthread+0x10/0x10
[291026.468631] ret_from_fork+0x2d/0x50
[291026.469047] ? __pfx_kthread+0x10/0x10
[291026.469455] ret_from_fork_asm+0x1a/0x30
[291026.469906] </TASK>
[291034.445119] watchdog: BUG: soft lockup - CPU#22 stuck for 443s! [kworker/u100:5:3816805]
[291034.446056] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291034.446086] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291034.452416] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291034.453196] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291034.453888] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291034.454475] RIP: 0010:xas_descend+0x7/0x80
[291034.454949] Code: 48 c1 e8 20 48 89 57 08 c3 cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 <48> d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89
[291034.456589] RSP: 0018:ffffb956e260f808 EFLAGS: 00000297
[291034.457107] RAX: 0000000000022de8 RBX: ffff9074f9f11ff0 RCX: 0000000000000012
[291034.457718] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[291034.458313] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291034.458945] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291034.459543] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291034.460154] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291034.460820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291034.461340] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291034.461967] PKRU: 55555554
[291034.462336] Call Trace:
[291034.462702] <IRQ>
[291034.463033] ? watchdog_timer_fn+0x1d7/0x250
[291034.463480] ? __pfx_watchdog_timer_fn+0x10/0x10
[291034.464210] ? __hrtimer_run_queues+0x10f/0x2b0
[291034.464778] ? hrtimer_interrupt+0xff/0x240
[291034.465218] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291034.465851] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291034.466334] </IRQ>
[291034.466860] <TASK>
[291034.467192] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291034.467704] ? xas_descend+0x7/0x80
[291034.468106] xas_load+0x3d/0x50
[291034.468486] filemap_get_entry+0x6a/0x140
[291034.468936] __filemap_get_folio+0x33/0x2d0
[291034.469369] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291034.469973] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291034.470477] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291034.471008] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291034.471522] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291034.472048] btrfs_cow_block+0xc7/0x280 [btrfs]
[291034.472743] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291034.473238] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291034.473772] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291034.474285] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291034.475003] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291034.475533] btrfs_work_helper+0xba/0x190 [btrfs]
[291034.476036] process_one_work+0x193/0x3d0
[291034.476450] worker_thread+0x2fc/0x410
[291034.476880] ? __pfx_worker_thread+0x10/0x10
[291034.477296] kthread+0xdc/0x110
[291034.477662] ? __pfx_kthread+0x10/0x10
[291034.478051] ret_from_fork+0x2d/0x50
[291034.478409] ? __pfx_kthread+0x10/0x10
[291034.478996] ret_from_fork_asm+0x1a/0x30
[291034.479367] </TASK>
[291390.442204] watchdog: BUG: soft lockup - CPU#12 stuck for 753s! [migration/12:55]
[291390.442968] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291390.443001] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291390.448751] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291390.449412] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291390.450057] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291390.450603] RIP: 0010:stop_machine_yield+0x2/0x10
[291390.451051] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291390.452415] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291390.452962] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291390.453532] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291390.454135] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291390.454715] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291390.455298] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291390.455902] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291390.456523] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291390.457060] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291390.457647] PKRU: 55555554
[291390.458011] Call Trace:
[291390.458351] <IRQ>
[291390.458676] ? watchdog_timer_fn+0x1d7/0x250
[291390.459121] ? __pfx_watchdog_timer_fn+0x10/0x10
[291390.459572] ? __hrtimer_run_queues+0x10f/0x2b0
[291390.460039] ? hrtimer_interrupt+0xff/0x240
[291390.460465] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291390.460981] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291390.461448] </IRQ>
[291390.461786] <TASK>
[291390.462109] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291390.462600] ? stop_machine_yield+0x2/0x10
[291390.463034] multi_cpu_stop+0x94/0x100
[291390.463449] ? __pfx_multi_cpu_stop+0x10/0x10
[291390.463916] cpu_stopper_thread+0x90/0x140
[291390.464339] ? __pfx_smpboot_thread_fn+0x10/0x10
[291390.464817] smpboot_thread_fn+0xd4/0x1d0
[291390.465235] kthread+0xdc/0x110
[291390.465620] ? __pfx_kthread+0x10/0x10
[291390.466034] ret_from_fork+0x2d/0x50
[291390.466429] ? __pfx_kthread+0x10/0x10
[291390.466861] ret_from_fork_asm+0x1a/0x30
[291390.467277] </TASK>
[291394.443220] watchdog: BUG: soft lockup - CPU#22 stuck for 779s! [kworker/u100:5:3816805]
[291394.444280] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291394.444313] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291394.450758] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291394.451496] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291394.452213] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291394.452863] RIP: 0010:xas_descend+0x29/0x80
[291394.453435] Code: 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 <74> 0e 88 4f 12 c3 cc cc cc cc 48 83 fa 02 75 f2 48 3d fd 00 00 00
[291394.455416] RSP: 0018:ffffb956e260f808 EFLAGS: 00000293
[291394.456095] RAX: ffffe4bc7bfa2580 RBX: ffff904df9a5c490 RCX: 0000000000000028
[291394.456923] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291394.457634] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291394.458247] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291394.458873] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291394.459465] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291394.460140] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291394.460680] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291394.461295] PKRU: 55555554
[291394.461667] Call Trace:
[291394.462027] <IRQ>
[291394.462351] ? watchdog_timer_fn+0x1d7/0x250
[291394.462826] ? __pfx_watchdog_timer_fn+0x10/0x10
[291394.463284] ? __hrtimer_run_queues+0x10f/0x2b0
[291394.463753] ? hrtimer_interrupt+0xff/0x240
[291394.464192] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291394.464681] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291394.465164] </IRQ>
[291394.465491] <TASK>
[291394.465854] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291394.466341] ? xas_descend+0x29/0x80
[291394.466755] xas_load+0x3d/0x50
[291394.467135] filemap_get_entry+0x6a/0x140
[291394.467557] __filemap_get_folio+0x33/0x2d0
[291394.468020] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291394.468552] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291394.469070] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291394.469575] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291394.470099] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291394.470604] btrfs_cow_block+0xc7/0x280 [btrfs]
[291394.471094] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291394.471584] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291394.472083] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291394.472593] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291394.473097] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291394.473622] btrfs_work_helper+0xba/0x190 [btrfs]
[291394.474121] process_one_work+0x193/0x3d0
[291394.474530] worker_thread+0x2fc/0x410
[291394.474957] ? __pfx_worker_thread+0x10/0x10
[291394.475365] kthread+0xdc/0x110
[291394.475735] ? __pfx_kthread+0x10/0x10
[291394.476109] ret_from_fork+0x2d/0x50
[291394.476464] ? __pfx_kthread+0x10/0x10
[291394.476861] ret_from_fork_asm+0x1a/0x30
[291394.477226] </TASK>
[291478.442735] watchdog: BUG: soft lockup - CPU#22 stuck for 857s! [kworker/u100:5:3816805]
[291478.443639] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291478.443669] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291478.449898] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291478.450677] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291478.451345] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291478.451940] RIP: 0010:xas_load+0x30/0x50
[291478.452436] Code: 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe 48 8d 58 fe 38 4d 10 <77> ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b 00 75 c8 5b 5d c3 cc
[291478.454413] RSP: 0018:ffffb956e260f810 EFLAGS: 00000246
[291478.455061] RAX: ffff904df9a5c492 RBX: ffff904df9a5c490 RCX: 0000000000000000
[291478.455892] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291478.456710] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291478.457524] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291478.458340] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291478.459140] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291478.460029] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291478.460753] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291478.461593] PKRU: 55555554
[291478.462038] Call Trace:
[291478.462452] <IRQ>
[291478.462777] ? watchdog_timer_fn+0x1d7/0x250
[291478.463223] ? __pfx_watchdog_timer_fn+0x10/0x10
[291478.463701] ? __hrtimer_run_queues+0x10f/0x2b0
[291478.464162] ? hrtimer_interrupt+0xff/0x240
[291478.464619] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291478.465106] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291478.465604] </IRQ>
[291478.465929] <TASK>
[291478.466276] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291478.466767] ? xas_load+0x30/0x50
[291478.467158] filemap_get_entry+0x6a/0x140
[291478.467597] __filemap_get_folio+0x33/0x2d0
[291478.468024] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291478.468597] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291478.469097] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291478.469628] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291478.470146] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291478.470671] btrfs_cow_block+0xc7/0x280 [btrfs]
[291478.471156] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291478.471663] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291478.472153] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291478.472680] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291478.473172] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291478.473719] btrfs_work_helper+0xba/0x190 [btrfs]
[291478.474220] process_one_work+0x193/0x3d0
[291478.474654] worker_thread+0x2fc/0x410
[291478.475048] ? __pfx_worker_thread+0x10/0x10
[291478.475508] kthread+0xdc/0x110
[291478.475863] ? __pfx_kthread+0x10/0x10
[291478.476267] ret_from_fork+0x2d/0x50
[291478.476633] ? __pfx_kthread+0x10/0x10
[291478.476996] ret_from_fork_asm+0x1a/0x30
[291478.477416] </TASK>
[291054.444011] watchdog: BUG: soft lockup - CPU#12 stuck for 440s! [migration/12:55]
[291054.444771] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291054.444804] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291054.450477] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291054.451144] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291054.451786] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291054.452326] RIP: 0010:stop_machine_yield+0x2/0x10
[291054.452793] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291054.454162] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291054.454670] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291054.455243] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291054.455847] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291054.456428] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291054.457013] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291054.457606] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291054.458230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291054.458764] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291054.459343] PKRU: 55555554
[291054.459730] Call Trace:
[291054.460074] <IRQ>
[291054.460398] ? watchdog_timer_fn+0x1d7/0x250
[291054.460842] ? __pfx_watchdog_timer_fn+0x10/0x10
[291054.461292] ? __hrtimer_run_queues+0x10f/0x2b0
[291054.461777] ? hrtimer_interrupt+0xff/0x240
[291054.462209] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291054.462718] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291054.463189] </IRQ>
[291054.463526] <TASK>
[291054.463856] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291054.464344] ? stop_machine_yield+0x2/0x10
[291054.464795] multi_cpu_stop+0x94/0x100
[291054.465198] ? __pfx_multi_cpu_stop+0x10/0x10
[291054.465662] cpu_stopper_thread+0x90/0x140
[291054.466086] ? __pfx_smpboot_thread_fn+0x10/0x10
[291054.466547] smpboot_thread_fn+0xd4/0x1d0
[291054.466970] kthread+0xdc/0x110
[291054.467344] ? __pfx_kthread+0x10/0x10
[291054.467782] ret_from_fork+0x2d/0x50
[291054.468180] ? __pfx_kthread+0x10/0x10
[291054.468601] ret_from_fork_asm+0x1a/0x30
[291054.469017] </TASK>
[291362.442353] watchdog: BUG: soft lockup - CPU#12 stuck for 727s! [migration/12:55]
[291362.443104] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291362.443135] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291362.449074] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291362.449764] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291362.450411] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291362.451003] RIP: 0010:stop_machine_yield+0x2/0x10
[291362.451454] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291362.453115] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291362.453617] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291362.454270] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291362.454891] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291362.455496] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291362.456179] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291362.456794] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291362.457466] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291362.458033] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291362.458645] PKRU: 55555554
[291362.459080] Call Trace:
[291362.459436] <IRQ>
[291362.459777] ? watchdog_timer_fn+0x1d7/0x250
[291362.460246] ? __pfx_watchdog_timer_fn+0x10/0x10
[291362.460720] ? __hrtimer_run_queues+0x10f/0x2b0
[291362.461216] ? hrtimer_interrupt+0xff/0x240
[291362.461666] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291362.462215] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291362.462708] </IRQ>
[291362.463147] <TASK>
[291362.463555] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291362.464285] ? stop_machine_yield+0x2/0x10
[291362.464893] multi_cpu_stop+0x94/0x100
[291362.465455] ? __pfx_multi_cpu_stop+0x10/0x10
[291362.466121] cpu_stopper_thread+0x90/0x140
[291362.466705] ? __pfx_smpboot_thread_fn+0x10/0x10
[291362.467366] smpboot_thread_fn+0xd4/0x1d0
[291362.467967] kthread+0xdc/0x110
[291362.468472] ? __pfx_kthread+0x10/0x10
[291362.469047] ret_from_fork+0x2d/0x50
[291362.469587] ? __pfx_kthread+0x10/0x10
[291362.470172] ret_from_fork_asm+0x1a/0x30
[291362.470741] </TASK>
[291366.443331] watchdog: BUG: soft lockup - CPU#22 stuck for 753s! [kworker/u100:5:3816805]
[291366.444259] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291366.444292] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291366.450926] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291366.451662] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291366.452352] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291366.452986] RIP: 0010:xas_descend+0x1b/0x80
[291366.453435] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 <48> 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88 4f 12 c3 cc cc
[291366.454909] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291366.455413] RAX: ffff904df9a5c492 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291366.456039] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291366.456627] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291366.457252] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291366.457843] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291366.458560] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291366.459489] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291366.460199] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291366.461037] PKRU: 55555554
[291366.461480] Call Trace:
[291366.461917] <IRQ>
[291366.462301] ? watchdog_timer_fn+0x1d7/0x250
[291366.462877] ? __pfx_watchdog_timer_fn+0x10/0x10
[291366.463467] ? __hrtimer_run_queues+0x10f/0x2b0
[291366.464066] ? hrtimer_interrupt+0xff/0x240
[291366.464618] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291366.465285] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291366.465913] </IRQ>
[291366.466306] <TASK>
[291366.466692] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291366.467356] ? xas_descend+0x1b/0x80
[291366.467874] xas_load+0x3d/0x50
[291366.468341] filemap_get_entry+0x6a/0x140
[291366.468894] __filemap_get_folio+0x33/0x2d0
[291366.469446] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291366.470163] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291366.470812] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291366.471398] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291366.471923] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291366.472431] btrfs_cow_block+0xc7/0x280 [btrfs]
[291366.472924] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291366.473412] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291366.473912] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291366.474423] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291366.474930] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291366.475481] btrfs_work_helper+0xba/0x190 [btrfs]
[291366.476114] process_one_work+0x193/0x3d0
[291366.476638] worker_thread+0x2fc/0x410
[291366.477157] ? __pfx_worker_thread+0x10/0x10
[291366.477691] kthread+0xdc/0x110
[291366.478148] ? __pfx_kthread+0x10/0x10
[291366.478618] ret_from_fork+0x2d/0x50
[291366.479106] ? __pfx_kthread+0x10/0x10
[291366.479579] ret_from_fork_asm+0x1a/0x30
[291366.480091] </TASK>
[291502.441608] watchdog: BUG: soft lockup - CPU#12 stuck for 857s! [migration/12:55]
[291502.442599] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291502.442629] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291502.451186] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291502.452187] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291502.453134] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291502.453917] RIP: 0010:stop_machine_yield+0x2/0x10
[291502.454535] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291502.456525] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291502.457010] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291502.457597] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291502.458192] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291502.458769] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291502.459374] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291502.459952] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291502.460639] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291502.461185] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291502.461770] PKRU: 55555554
[291502.462138] Call Trace:
[291502.462485] <IRQ>
[291502.462805] ? watchdog_timer_fn+0x1d7/0x250
[291502.463270] ? __pfx_watchdog_timer_fn+0x10/0x10
[291502.463732] ? __hrtimer_run_queues+0x10f/0x2b0
[291502.464203] ? hrtimer_interrupt+0xff/0x240
[291502.464637] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291502.465138] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291502.465610] </IRQ>
[291502.465932] <TASK>
[291502.466281] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291502.466768] ? stop_machine_yield+0x2/0x10
[291502.467222] multi_cpu_stop+0x94/0x100
[291502.467631] ? __pfx_multi_cpu_stop+0x10/0x10
[291502.468072] cpu_stopper_thread+0x90/0x140
[291502.468509] ? __pfx_smpboot_thread_fn+0x10/0x10
[291502.468960] smpboot_thread_fn+0xd4/0x1d0
[291502.469408] kthread+0xdc/0x110
[291502.469782] ? __pfx_kthread+0x10/0x10
[291502.470212] ret_from_fork+0x2d/0x50
[291502.470613] ? __pfx_kthread+0x10/0x10
[291502.471024] ret_from_fork_asm+0x1a/0x30
[291502.471457] </TASK>
[291506.442586] watchdog: BUG: soft lockup - CPU#22 stuck for 883s! [kworker/u100:5:3816805]
[291506.443539] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291506.443572] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291506.449892] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291506.450666] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291506.451357] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291506.451947] RIP: 0010:xas_descend+0x10/0x80
[291506.452421] Code: cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f <89> c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48
[291506.453855] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291506.454395] RAX: 00000000000008b7 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291506.454992] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291506.455638] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291506.456305] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291506.456894] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291506.457533] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291506.458185] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291506.458719] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291506.459353] PKRU: 55555554
[291506.459718] Call Trace:
[291506.460072] <IRQ>
[291506.460411] ? watchdog_timer_fn+0x1d7/0x250
[291506.460852] ? __pfx_watchdog_timer_fn+0x10/0x10
[291506.461346] ? __hrtimer_run_queues+0x10f/0x2b0
[291506.461800] ? hrtimer_interrupt+0xff/0x240
[291506.462271] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291506.462770] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291506.463282] </IRQ>
[291506.463627] <TASK>
[291506.463954] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291506.464463] ? xas_descend+0x10/0x80
[291506.464859] xas_load+0x3d/0x50
[291506.465274] filemap_get_entry+0x6a/0x140
[291506.465696] __filemap_get_folio+0x33/0x2d0
[291506.466143] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291506.466675] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291506.467208] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291506.467708] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291506.468251] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291506.468752] btrfs_cow_block+0xc7/0x280 [btrfs]
[291506.469260] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291506.469745] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291506.470262] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291506.470770] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291506.471320] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291506.471862] btrfs_work_helper+0xba/0x190 [btrfs]
[291506.472385] process_one_work+0x193/0x3d0
[291506.472816] worker_thread+0x2fc/0x410
[291506.473243] ? __pfx_worker_thread+0x10/0x10
[291506.473653] kthread+0xdc/0x110
[291506.474002] ? __pfx_kthread+0x10/0x10
[291506.474379] ret_from_fork+0x2d/0x50
[291506.474735] ? __pfx_kthread+0x10/0x10
[291506.475110] ret_from_fork_asm+0x1a/0x30
[291506.475479] </TASK>
[290634.447122] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290634.447157] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290634.454230] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290634.454968] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290634.455612] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290634.456252] RIP: 0010:stop_machine_yield+0x2/0x10
[290634.456718] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290634.458155] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290634.458650] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290634.459271] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290634.459867] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290634.460470] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290634.461090] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290634.461683] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290634.462352] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290634.462881] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290634.463487] PKRU: 55555554
[290634.463858] Call Trace:
[290634.464221] <IRQ>
[290634.464552] ? watchdog_timer_fn+0x1d7/0x250
[290634.465031] ? __pfx_watchdog_timer_fn+0x10/0x10
[290634.465501] ? __hrtimer_run_queues+0x10f/0x2b0
[290634.465979] ? hrtimer_interrupt+0xff/0x240
[290634.466432] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290634.466951] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290634.467463] </IRQ>
[290634.467909] <TASK>
[290634.468324] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290634.469021] ? stop_machine_yield+0x2/0x10
[290634.469595] multi_cpu_stop+0x94/0x100
[290634.470163] ? __pfx_multi_cpu_stop+0x10/0x10
[290634.470765] cpu_stopper_thread+0x90/0x140
[290634.471344] ? __pfx_smpboot_thread_fn+0x10/0x10
[290634.472007] smpboot_thread_fn+0xd4/0x1d0
[290634.472585] kthread+0xdc/0x110
[290634.473110] ? __pfx_kthread+0x10/0x10
[290634.473658] ret_from_fork+0x2d/0x50
[290634.474232] ? __pfx_kthread+0x10/0x10
[290634.474785] ret_from_fork_asm+0x1a/0x30
[290634.475357] </TASK>
[290674.447114] watchdog: BUG: soft lockup - CPU#22 stuck for 108s! [kworker/u100:5:3816805]
[290674.448456] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290674.448504] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290674.459122] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290674.460330] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290674.461407] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290674.462425] RIP: 0010:xas_load+0x29/0x50
[290674.463141] Code: 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe <48> 8d 58 fe 38 4d 10 77 ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b
[290674.465456] RSP: 0018:ffffb956e260f810 EFLAGS: 00000282
[290674.466317] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000006
[290674.467286] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[290674.468258] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290674.469227] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290674.470205] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290674.471179] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290674.472230] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290674.473093] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290674.474077] PKRU: 55555554
[290674.474656] Call Trace:
[290674.475223] <IRQ>
[290674.475750] ? watchdog_timer_fn+0x1d7/0x250
[290674.476461] ? __pfx_watchdog_timer_fn+0x10/0x10
[290674.477220] ? __hrtimer_run_queues+0x10f/0x2b0
[290674.477975] ? hrtimer_interrupt+0xff/0x240
[290674.478675] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290674.479481] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290674.480264] </IRQ>
[290674.480796] <TASK>
[290674.481322] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290674.482132] ? xas_load+0x29/0x50
[290674.482775] filemap_get_entry+0x6a/0x140
[290674.483461] __filemap_get_folio+0x33/0x2d0
[290674.484189] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290674.485112] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290674.485987] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290674.486813] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290674.487645] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290674.488500] btrfs_cow_block+0xc7/0x280 [btrfs]
[290674.489293] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290674.490116] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290674.490967] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290674.491814] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290674.492613] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290674.493477] btrfs_work_helper+0xba/0x190 [btrfs]
[290674.494287] process_one_work+0x193/0x3d0
[290674.494992] worker_thread+0x2fc/0x410
[290674.495639] ? __pfx_worker_thread+0x10/0x10
[290674.496337] kthread+0xdc/0x110
[290674.496943] ? __pfx_kthread+0x10/0x10
[290674.497554] ret_from_fork+0x2d/0x50
[290674.498176] ? __pfx_kthread+0x10/0x10
[290674.498795] ret_from_fork_asm+0x1a/0x30
[290674.499408] </TASK>
[290690.446029] watchdog: BUG: soft lockup - CPU#12 stuck for 101s! [migration/12:55]
[290690.446977] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290690.447009] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290690.454784] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290690.455493] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290690.456144] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290690.456738] RIP: 0010:stop_machine_yield+0x2/0x10
[290690.457193] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290690.458623] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290690.459113] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290690.459739] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290690.460330] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290690.460952] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290690.461569] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290690.462180] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290690.462853] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290690.463374] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290690.464001] PKRU: 55555554
[290690.464370] Call Trace:
[290690.464754] <IRQ>
[290690.465087] ? watchdog_timer_fn+0x1d7/0x250
[290690.465543] ? __pfx_watchdog_timer_fn+0x10/0x10
[290690.466020] ? __hrtimer_run_queues+0x10f/0x2b0
[290690.466483] ? hrtimer_interrupt+0xff/0x240
[290690.467024] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290690.467543] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290690.468039] </IRQ>
[290690.468368] <TASK>
[290690.468773] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290690.469279] ? stop_machine_yield+0x2/0x10
[290690.469748] multi_cpu_stop+0x94/0x100
[290690.470182] ? __pfx_multi_cpu_stop+0x10/0x10
[290690.470654] cpu_stopper_thread+0x90/0x140
[290690.471100] ? __pfx_smpboot_thread_fn+0x10/0x10
[290690.471568] smpboot_thread_fn+0xd4/0x1d0
[290690.472009] kthread+0xdc/0x110
[290690.472394] ? __pfx_kthread+0x10/0x10
[290690.472831] ret_from_fork+0x2d/0x50
[290690.473236] ? __pfx_kthread+0x10/0x10
[290690.473681] ret_from_fork_asm+0x1a/0x30
[290690.474108] </TASK>
[290758.446662] watchdog: BUG: soft lockup - CPU#22 stuck for 186s! [kworker/u100:5:3816805]
[290758.447858] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290758.447906] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290758.457709] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290758.458893] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290758.459937] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290758.460873] RIP: 0010:xas_descend+0x3/0x80
[290758.461582] Code: 10 48 89 07 48 c1 e8 20 48 89 57 08 c3 cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e <48> 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44
[290758.463850] RSP: 0018:ffffb956e260f808 EFLAGS: 00000297
[290758.464670] RAX: ffff9074f9f11ff2 RBX: ffff9074f9f11ff0 RCX: 0000000000000012
[290758.465624] RDX: 0000000000000002 RSI: ffff9074f9f11ff0 RDI: ffffb956e260f828
[290758.466564] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290758.467509] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290758.468446] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290758.469383] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290758.470400] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290758.471223] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290758.472167] PKRU: 55555554
[290758.472737] Call Trace:
[290758.473292] <IRQ>
[290758.473800] ? watchdog_timer_fn+0x1d7/0x250
[290758.474509] ? __pfx_watchdog_timer_fn+0x10/0x10
[290758.475244] ? __hrtimer_run_queues+0x10f/0x2b0
[290758.475963] ? hrtimer_interrupt+0xff/0x240
[290758.476672] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290758.477454] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290758.478227] </IRQ>
[290758.478745] <TASK>
[290758.479268] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290758.480049] ? xas_descend+0x3/0x80
[290758.480693] xas_load+0x3d/0x50
[290758.481306] filemap_get_entry+0x6a/0x140
[290758.481978] __filemap_get_folio+0x33/0x2d0
[290758.482689] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290758.483552] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290758.484352] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290758.485162] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290758.485973] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290758.486803] btrfs_cow_block+0xc7/0x280 [btrfs]
[290758.487580] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290758.488379] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290758.489165] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290758.489983] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290758.490788] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290758.491640] btrfs_work_helper+0xba/0x190 [btrfs]
[290758.492456] process_one_work+0x193/0x3d0
[290758.493107] worker_thread+0x2fc/0x410
[290758.493735] ? __pfx_worker_thread+0x10/0x10
[290758.494438] kthread+0xdc/0x110
[290758.494986] ? __pfx_kthread+0x10/0x10
[290758.495626] ret_from_fork+0x2d/0x50
[290758.496204] ? __pfx_kthread+0x10/0x10
[290758.496781] ret_from_fork_asm+0x1a/0x30
[290758.497386] </TASK>
[290774.445574] watchdog: BUG: soft lockup - CPU#12 stuck for 179s! [migration/12:55]
[290774.446282] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290774.446312] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290774.451997] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290774.452688] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290774.453339] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290774.453874] RIP: 0010:stop_machine_yield+0x2/0x10
[290774.454339] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290774.455699] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290774.456205] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290774.456780] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290774.457457] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290774.458033] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290774.458620] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290774.459221] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290774.459844] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290774.460380] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290774.460958] PKRU: 55555554
[290774.461327] Call Trace:
[290774.461669] <IRQ>
[290774.461992] ? watchdog_timer_fn+0x1d7/0x250
[290774.462441] ? __pfx_watchdog_timer_fn+0x10/0x10
[290774.462890] ? __hrtimer_run_queues+0x10f/0x2b0
[290774.463403] ? hrtimer_interrupt+0xff/0x240
[290774.463832] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290774.464342] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290774.464811] </IRQ>
[290774.465148] <TASK>
[290774.465470] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290774.465958] ? stop_machine_yield+0x2/0x10
[290774.466394] multi_cpu_stop+0x94/0x100
[290774.466797] ? __pfx_multi_cpu_stop+0x10/0x10
[290774.467265] cpu_stopper_thread+0x90/0x140
[290774.467684] ? __pfx_smpboot_thread_fn+0x10/0x10
[290774.468148] smpboot_thread_fn+0xd4/0x1d0
[290774.468568] kthread+0xdc/0x110
[290774.468943] ? __pfx_kthread+0x10/0x10
[290774.469366] ret_from_fork+0x2d/0x50
[290774.469762] ? __pfx_kthread+0x10/0x10
[290774.470226] ret_from_fork_asm+0x1a/0x30
[290774.470637] </TASK>
[290910.445806] watchdog: BUG: soft lockup - CPU#22 stuck for 328s! [kworker/u100:5:3816805]
[290910.447000] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290910.447048] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290910.456929] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290910.458119] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290910.459210] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290910.460136] RIP: 0010:xas_load+0x29/0x50
[290910.460826] Code: 90 55 48 89 fd 53 e8 16 ff ff ff 48 89 c2 83 e2 03 48 83 fa 02 75 08 48 3d 00 10 00 00 77 07 5b 5d c3 cc cc cc cc 0f b6 48 fe <48> 8d 58 fe 38 4d 10 77 ec 48 89 de 48 89 ef e8 c3 fd ff ff 80 3b
[290910.463069] RSP: 0018:ffffb956e260f810 EFLAGS: 00000282
[290910.463883] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000006
[290910.464830] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[290910.465771] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290910.466712] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290910.467660] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290910.468610] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290910.469635] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290910.470454] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290910.471392] PKRU: 55555554
[290910.471960] Call Trace:
[290910.472511] <IRQ>
[290910.473016] ? watchdog_timer_fn+0x1d7/0x250
[290910.473746] ? __pfx_watchdog_timer_fn+0x10/0x10
[290910.474472] ? __hrtimer_run_queues+0x10f/0x2b0
[290910.475187] ? hrtimer_interrupt+0xff/0x240
[290910.475881] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290910.476671] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290910.477420] </IRQ>
[290910.477935] <TASK>
[290910.478454] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290910.479231] ? xas_load+0x29/0x50
[290910.479863] filemap_get_entry+0x6a/0x140
[290910.480552] __filemap_get_folio+0x33/0x2d0
[290910.481234] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290910.482086] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290910.482898] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290910.483717] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290910.484546] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290910.485367] btrfs_cow_block+0xc7/0x280 [btrfs]
[290910.486125] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290910.486934] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290910.487752] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290910.488624] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290910.489426] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290910.490262] btrfs_work_helper+0xba/0x190 [btrfs]
[290910.491054] process_one_work+0x193/0x3d0
[290910.491738] worker_thread+0x2fc/0x410
[290910.492388] ? __pfx_worker_thread+0x10/0x10
[290910.493072] kthread+0xdc/0x110
[290910.493666] ? __pfx_kthread+0x10/0x10
[290910.494275] ret_from_fork+0x2d/0x50
[290910.494873] ? __pfx_kthread+0x10/0x10
[290910.495472] ret_from_fork_asm+0x1a/0x30
[290910.496065] </TASK>
[291270.443842] watchdog: BUG: soft lockup - CPU#22 stuck for 663s! [kworker/u100:5:3816805]
[291270.444714] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291270.444748] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291270.451046] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291270.451819] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291270.452501] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291270.453085] RIP: 0010:xas_descend+0x10/0x80
[291270.453572] Code: cc cc cc cc cc cc cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f <89> c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48
[291270.455008] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291270.455534] RAX: 00000000000008b7 RBX: ffff9066c9ceada0 RCX: 0000000000000037
[291270.456113] RDX: 0000000000000002 RSI: ffff9066c9ceada0 RDI: ffffb956e260f828
[291270.456738] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291270.457321] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291270.457915] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291270.458538] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291270.459164] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291270.459715] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291270.460301] PKRU: 55555554
[291270.460677] Call Trace:
[291270.461017] <IRQ>
[291270.461337] ? watchdog_timer_fn+0x1d7/0x250
[291270.461785] ? __pfx_watchdog_timer_fn+0x10/0x10
[291270.462243] ? __hrtimer_run_queues+0x10f/0x2b0
[291270.462708] ? hrtimer_interrupt+0xff/0x240
[291270.463135] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291270.463667] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291270.464132] </IRQ>
[291270.464500] <TASK>
[291270.464820] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291270.465307] ? xas_descend+0x10/0x80
[291270.465724] xas_load+0x3d/0x50
[291270.466097] filemap_get_entry+0x6a/0x140
[291270.466553] __filemap_get_folio+0x33/0x2d0
[291270.466981] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291270.467549] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291270.468044] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291270.468626] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291270.469136] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291270.469676] btrfs_cow_block+0xc7/0x280 [btrfs]
[291270.470146] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291270.470671] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291270.471153] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291270.471708] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291270.472198] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291270.472741] btrfs_work_helper+0xba/0x190 [btrfs]
[291270.473224] process_one_work+0x193/0x3d0
[291270.473657] worker_thread+0x2fc/0x410
[291270.474044] ? __pfx_worker_thread+0x10/0x10
[291270.474512] kthread+0xdc/0x110
[291270.474857] ? __pfx_kthread+0x10/0x10
[291270.475228] ret_from_fork+0x2d/0x50
[291270.475609] ? __pfx_kthread+0x10/0x10
[291270.475968] ret_from_fork_asm+0x1a/0x30
[291270.476337] </TASK>
[291278.442800] watchdog: BUG: soft lockup - CPU#12 stuck for 649s! [migration/12:55]
[291278.443789] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291278.443823] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291278.450983] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291278.451704] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291278.452333] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291278.452881] RIP: 0010:rcu_momentary_dyntick_idle+0x1d/0x40
[291278.453376] Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 b8 08 00 00 00 65 c6 05 c7 ab c7 46 00 65 48 8b 15 03 e6 c5 46 f0 0f c1 82 28 01 03 00 <a8> 04 74 0d 65 48 8b 3d 17 8d c7 46 e9 d2 fe ff ff 0f 0b eb ef 66
[291278.454753] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000292
[291278.455241] RAX: 00000000bd8a1b4c RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291278.455824] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291278.456428] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291278.457004] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291278.457617] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291278.458204] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291278.458838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291278.459359] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291278.459942] PKRU: 55555554
[291278.460306] Call Trace:
[291278.460663] <IRQ>
[291278.460987] ? watchdog_timer_fn+0x1d7/0x250
[291278.461460] ? __pfx_watchdog_timer_fn+0x10/0x10
[291278.461919] ? __hrtimer_run_queues+0x10f/0x2b0
[291278.462388] ? hrtimer_interrupt+0xff/0x240
[291278.462823] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291278.463315] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291278.463797] </IRQ>
[291278.464122] <TASK>
[291278.464513] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291278.465006] ? rcu_momentary_dyntick_idle+0x1d/0x40
[291278.465500] multi_cpu_stop+0xb2/0x100
[291278.465909] ? __pfx_multi_cpu_stop+0x10/0x10
[291278.466368] cpu_stopper_thread+0x90/0x140
[291278.466797] ? __pfx_smpboot_thread_fn+0x10/0x10
[291278.467256] smpboot_thread_fn+0xd4/0x1d0
[291278.467686] kthread+0xdc/0x110
[291278.468062] ? __pfx_kthread+0x10/0x10
[291278.468529] ret_from_fork+0x2d/0x50
[291278.468928] ? __pfx_kthread+0x10/0x10
[291278.469346] ret_from_fork_asm+0x1a/0x30
[291278.469767] </TASK>
[291298.443693] watchdog: BUG: soft lockup - CPU#22 stuck for 689s! [kworker/u100:5:3816805]
[291298.444552] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291298.444585] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291298.450909] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291298.451691] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291298.452383] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291298.452974] RIP: 0010:xas_start+0x0/0xd0
[291298.453438] Code: 00 00 c3 cc cc cc cc 49 83 c1 01 4c 89 4f 08 c3 cc cc cc cc 4c 89 c0 eb a5 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <48> 8b 57 18 48 89 d0 83 e0 03 74 6b 48 83 f8 02 75 09 48 81 fa 05
[291298.454873] RSP: 0018:ffffb956e260f808 EFLAGS: 00000246
[291298.455402] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000028
[291298.455987] RDX: 0000000000000000 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291298.456616] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291298.457205] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291298.457801] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291298.458421] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291298.459053] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291298.459609] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291298.460197] PKRU: 55555554
[291298.460574] Call Trace:
[291298.460918] <IRQ>
[291298.461260] ? watchdog_timer_fn+0x1d7/0x250
[291298.461701] ? __pfx_watchdog_timer_fn+0x10/0x10
[291298.462156] ? __hrtimer_run_queues+0x10f/0x2b0
[291298.462623] ? hrtimer_interrupt+0xff/0x240
[291298.463051] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291298.463579] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291298.464049] </IRQ>
[291298.464413] <TASK>
[291298.464735] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291298.465240] ? __pfx_xas_start+0x10/0x10
[291298.465660] xas_load+0xa/0x50
[291298.466031] filemap_get_entry+0x6a/0x140
[291298.466489] __filemap_get_folio+0x33/0x2d0
[291298.466919] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291298.467490] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291298.467988] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291298.468520] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291298.469025] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291298.469562] btrfs_cow_block+0xc7/0x280 [btrfs]
[291298.470034] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291298.470552] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291298.471034] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291298.471565] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291298.472049] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291298.472592] btrfs_work_helper+0xba/0x190 [btrfs]
[291298.473086] process_one_work+0x193/0x3d0
[291298.473530] worker_thread+0x2fc/0x410
[291298.473918] ? __pfx_worker_thread+0x10/0x10
[291298.474374] kthread+0xdc/0x110
[291298.474720] ? __pfx_kthread+0x10/0x10
[291298.475096] ret_from_fork+0x2d/0x50
[291298.475476] ? __pfx_kthread+0x10/0x10
[291298.475833] ret_from_fork_asm+0x1a/0x30
[291298.476204] </TASK>
[291306.442651] watchdog: BUG: soft lockup - CPU#12 stuck for 675s! [migration/12:55]
[291306.443456] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291306.443489] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291306.449455] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291306.450139] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291306.450772] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291306.451355] RIP: 0010:stop_machine_yield+0x2/0x10
[291306.451802] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291306.453202] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291306.453687] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291306.454294] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291306.454871] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291306.455480] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291306.456067] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291306.456660] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291306.457323] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291306.457833] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291306.458448] PKRU: 55555554
[291306.458806] Call Trace:
[291306.459163] <IRQ>
[291306.459500] ? watchdog_timer_fn+0x1d7/0x250
[291306.459937] ? __pfx_watchdog_timer_fn+0x10/0x10
[291306.460433] ? __hrtimer_run_queues+0x10f/0x2b0
[291306.460882] ? hrtimer_interrupt+0xff/0x240
[291306.461348] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291306.461836] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291306.462341] </IRQ>
[291306.462668] <TASK>
[291306.462992] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291306.463515] ? stop_machine_yield+0x2/0x10
[291306.463939] multi_cpu_stop+0x94/0x100
[291306.464384] ? __pfx_multi_cpu_stop+0x10/0x10
[291306.464822] cpu_stopper_thread+0x90/0x140
[291306.465279] ? __pfx_smpboot_thread_fn+0x10/0x10
[291306.465731] smpboot_thread_fn+0xd4/0x1d0
[291306.466157] kthread+0xdc/0x110
[291306.466548] ? __pfx_kthread+0x10/0x10
[291306.466954] ret_from_fork+0x2d/0x50
[291306.467382] ? __pfx_kthread+0x10/0x10
[291306.467786] ret_from_fork_asm+0x1a/0x30
[291306.468217] </TASK>
[291326.443544] watchdog: BUG: soft lockup - CPU#22 stuck for 715s! [kworker/u100:5:3816805]
[291326.444414] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291326.444446] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291326.450595] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291326.451363] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291326.452006] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291326.452614] RIP: 0010:xas_descend+0x16/0x80
[291326.453060] Code: cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 <48> 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 74 0e 88
[291326.454493] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291326.454993] RAX: 000000000000002c RBX: ffff904df9a5c490 RCX: 0000000000000028
[291326.455591] RDX: 0000000000000002 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291326.456206] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291326.456785] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291326.457405] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291326.457992] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291326.458636] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291326.459183] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291326.459762] PKRU: 55555554
[291326.460137] Call Trace:
[291326.460480] <IRQ>
[291326.460796] ? watchdog_timer_fn+0x1d7/0x250
[291326.461264] ? __pfx_watchdog_timer_fn+0x10/0x10
[291326.461715] ? __hrtimer_run_queues+0x10f/0x2b0
[291326.462201] ? hrtimer_interrupt+0xff/0x240
[291326.462632] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291326.463133] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291326.463605] </IRQ>
[291326.463930] <TASK>
[291326.464268] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291326.464751] ? xas_descend+0x16/0x80
[291326.465187] xas_load+0x3d/0x50
[291326.465561] filemap_get_entry+0x6a/0x140
[291326.465987] __filemap_get_folio+0x33/0x2d0
[291326.466430] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291326.466967] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291326.467480] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291326.468003] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291326.468530] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291326.469037] btrfs_cow_block+0xc7/0x280 [btrfs]
[291326.469522] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291326.470009] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291326.470509] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291326.471021] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291326.471525] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291326.472048] btrfs_work_helper+0xba/0x190 [btrfs]
[291326.472545] process_one_work+0x193/0x3d0
[291326.472959] worker_thread+0x2fc/0x410
[291326.473362] ? __pfx_worker_thread+0x10/0x10
[291326.473768] kthread+0xdc/0x110
[291326.474166] ? __pfx_kthread+0x10/0x10
[291326.474533] ret_from_fork+0x2d/0x50
[291326.474889] ? __pfx_kthread+0x10/0x10
[291326.475289] ret_from_fork_asm+0x1a/0x30
[291326.475654] </TASK>
[291334.442502] watchdog: BUG: soft lockup - CPU#12 stuck for 701s! [migration/12:55]
[291334.443248] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[291334.443279] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[291334.448960] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291334.449634] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291334.450283] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[291334.450823] RIP: 0010:stop_machine_yield+0x2/0x10
[291334.451293] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[291334.452658] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[291334.453163] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[291334.453733] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[291334.454330] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[291334.454911] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[291334.455497] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[291334.456086] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[291334.456708] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291334.457243] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[291334.457820] PKRU: 55555554
[291334.458210] Call Trace:
[291334.458554] <IRQ>
[291334.458875] ? watchdog_timer_fn+0x1d7/0x250
[291334.459336] ? __pfx_watchdog_timer_fn+0x10/0x10
[291334.459787] ? __hrtimer_run_queues+0x10f/0x2b0
[291334.460271] ? hrtimer_interrupt+0xff/0x240
[291334.460702] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291334.461211] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291334.461677] </IRQ>
[291334.462007] <TASK>
[291334.462338] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291334.462825] ? stop_machine_yield+0x2/0x10
[291334.463286] multi_cpu_stop+0x94/0x100
[291334.463691] ? __pfx_multi_cpu_stop+0x10/0x10
[291334.464157] cpu_stopper_thread+0x90/0x140
[291334.464578] ? __pfx_smpboot_thread_fn+0x10/0x10
[291334.465038] smpboot_thread_fn+0xd4/0x1d0
[291334.465455] kthread+0xdc/0x110
[291334.465827] ? __pfx_kthread+0x10/0x10
[291334.466302] ret_from_fork+0x2d/0x50
[291334.466699] ? __pfx_kthread+0x10/0x10
[291334.467136] ret_from_fork_asm+0x1a/0x30
[291334.467549] </TASK>
[291339.557850] rcu: INFO: rcu_preempt self-detected stall on CPU
[291339.558581] rcu: 22-....: (778963 ticks this GP) idle=59ec/1/0x4000000000000000 softirq=9778443/9778443 fqs=257775
[291339.559428] rcu: (t=780153 jiffies g=38283501 q=444221 ncpus=24)
[291339.559990] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[291339.560740] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[291339.561449] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[291339.562051] RIP: 0010:xas_start+0x26/0xd0
[291339.562473] Code: 90 90 90 90 48 8b 57 18 48 89 d0 83 e0 03 74 6b 48 83 f8 02 75 09 48 81 fa 05 c0 ff ff 77 31 48 8b 07 48 8b 57 08 48 8b 40 08 <48> 89 c1 83 e1 03 48 83 f9 02 75 08 48 3d 00 10 00 00 77 21 48 85
[291339.563871] RSP: 0018:ffffb956e260f808 EFLAGS: 00000202
[291339.564365] RAX: ffff9074f9f11ff2 RBX: 0000000000000000 RCX: 0000000000000028
[291339.564939] RDX: 0000000000022de8 RSI: ffff904df9a5c490 RDI: ffffb956e260f828
[291339.565525] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[291339.566129] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[291339.566699] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[291339.567311] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[291339.567924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[291339.568431] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[291339.568989] PKRU: 55555554
[291339.569338] Call Trace:
[291339.569657] <IRQ>
[291339.569959] ? rcu_dump_cpu_stacks+0xcb/0x110
[291339.570398] ? print_cpu_stall+0x111/0x260
[291339.570795] ? check_cpu_stall+0x10f/0x220
[291339.571231] ? rcu_pending+0x29/0x190
[291339.571604] ? rcu_sched_clock_irq+0xb8/0x350
[291339.572031] ? update_process_times+0x69/0xa0
[291339.572439] ? tick_nohz_handler+0xc5/0x130
[291339.573081] ? __pfx_tick_nohz_handler+0x10/0x10
[291339.573505] ? __hrtimer_run_queues+0x10f/0x2b0
[291339.573929] ? hrtimer_interrupt+0xff/0x240
[291339.574349] ? __sysvec_apic_timer_interrupt+0x51/0x150
[291339.574796] ? sysvec_apic_timer_interrupt+0x6d/0x90
[291339.575271] </IRQ>
[291339.575562] <TASK>
[291339.575850] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[291339.576332] ? xas_start+0x26/0xd0
[291339.576687] xas_load+0xa/0x50
[291339.577055] filemap_get_entry+0x6a/0x140
[291339.577444] __filemap_get_folio+0x33/0x2d0
[291339.577837] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[291339.578361] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[291339.578820] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[291339.579305] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[291339.579966] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[291339.580451] btrfs_cow_block+0xc7/0x280 [btrfs]
[291339.580889] btrfs_search_slot+0x768/0xbd0 [btrfs]
[291339.581357] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[291339.581800] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[291339.582313] ? btrfs_global_root+0x4d/0x60 [btrfs]
[291339.582758] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[291339.583273] btrfs_work_helper+0xba/0x190 [btrfs]
[291339.583717] process_one_work+0x193/0x3d0
[291339.584134] worker_thread+0x2fc/0x410
[291339.584494] ? __pfx_worker_thread+0x10/0x10
[291339.584879] kthread+0xdc/0x110
[291339.585222] ? __pfx_kthread+0x10/0x10
[291339.585567] ret_from_fork+0x2d/0x50
[291339.585905] ? __pfx_kthread+0x10/0x10
[291339.586264] ret_from_fork_asm+0x1a/0x30
[291339.586610] </TASK>
[290606.446481] watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [migration/12:55]
[290606.447505] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290606.447549] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290606.456281] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290606.457292] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290606.458252] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290606.459071] RIP: 0010:stop_machine_yield+0x2/0x10
[290606.459743] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290606.461789] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290606.462517] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290606.463387] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290606.464255] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290606.465109] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290606.465972] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290606.466839] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290606.467783] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290606.468552] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290606.469430] PKRU: 55555554
[290606.469964] Call Trace:
[290606.470489] <IRQ>
[290606.470970] ? watchdog_timer_fn+0x1d7/0x250
[290606.471634] ? __pfx_watchdog_timer_fn+0x10/0x10
[290606.472330] ? __hrtimer_run_queues+0x10f/0x2b0
[290606.472999] ? hrtimer_interrupt+0xff/0x240
[290606.473650] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290606.474381] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290606.475081] </IRQ>
[290606.475571] <TASK>
[290606.476052] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290606.476780] ? stop_machine_yield+0x2/0x10
[290606.477447] multi_cpu_stop+0x94/0x100
[290606.478056] ? __pfx_multi_cpu_stop+0x10/0x10
[290606.478721] cpu_stopper_thread+0x90/0x140
[290606.479363] ? __pfx_smpboot_thread_fn+0x10/0x10
[290606.480032] smpboot_thread_fn+0xd4/0x1d0
[290606.480664] kthread+0xdc/0x110
[290606.481243] ? __pfx_kthread+0x10/0x10
[290606.481842] ret_from_fork+0x2d/0x50
[290606.482445] ? __pfx_kthread+0x10/0x10
[290606.483041] ret_from_fork_asm+0x1a/0x30
[290606.483663] </TASK>
[290826.446288] watchdog: BUG: soft lockup - CPU#22 stuck for 250s! [kworker/u100:5:3816805]
[290826.447545] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290826.447596] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290826.457539] CPU: 22 PID: 3816805 Comm: kworker/u100:5 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290826.458727] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290826.459776] Workqueue: btrfs-endio-write btrfs_work_helper [btrfs]
[290826.460694] RIP: 0010:xas_descend+0x29/0x80
[290826.461403] Code: 90 0f b6 0e 48 8b 47 08 48 d3 e8 48 89 c1 83 e1 3f 89 c8 48 83 c0 04 48 8b 44 c6 08 48 89 77 18 48 89 c2 83 e2 03 48 83 fa 02 <74> 0e 88 4f 12 c3 cc cc cc cc 48 83 fa 02 75 f2 48 3d fd 00 00 00
[290826.463675] RSP: 0018:ffffb956e260f808 EFLAGS: 00000246
[290826.464470] RAX: ffff9066c9ceada2 RBX: ffff9056ce8d4920 RCX: 0000000000000022
[290826.465438] RDX: 0000000000000002 RSI: ffff9056ce8d4920 RDI: ffffb956e260f828
[290826.466394] RBP: ffffb956e260f828 R08: 000000000004ebe5 R09: 0000000000000995
[290826.467342] R10: 0000000000000000 R11: 000000000000000d R12: 0000000000022de8
[290826.468300] R13: 0000000000000002 R14: ffff905787aeb9a8 R15: ffffe4bc7bfa2580
[290826.469251] FS: 0000000000000000(0000) GS:ffff908383c00000(0000) knlGS:0000000000000000
[290826.470274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290826.471112] CR2: 000055fb7e4e3d98 CR3: 0000000fb3f5e006 CR4: 0000000000770ef0
[290826.472067] PKRU: 55555554
[290826.472639] Call Trace:
[290826.473210] <IRQ>
[290826.473708] ? watchdog_timer_fn+0x1d7/0x250
[290826.474417] ? __pfx_watchdog_timer_fn+0x10/0x10
[290826.475159] ? __hrtimer_run_queues+0x10f/0x2b0
[290826.475882] ? hrtimer_interrupt+0xff/0x240
[290826.476573] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290826.477419] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290826.478181] </IRQ>
[290826.478692] <TASK>
[290826.479220] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290826.480017] ? xas_descend+0x29/0x80
[290826.480653] xas_load+0x3d/0x50
[290826.481276] filemap_get_entry+0x6a/0x140
[290826.481954] __filemap_get_folio+0x33/0x2d0
[290826.482642] attach_eb_folio_to_filemap+0x68/0x200 [btrfs]
[290826.483501] alloc_extent_buffer+0x2c4/0x530 [btrfs]
[290826.484322] btrfs_init_new_buffer+0x38/0x270 [btrfs]
[290826.485138] btrfs_alloc_tree_block+0x12e/0x310 [btrfs]
[290826.485963] btrfs_force_cow_block+0x11b/0x7d0 [btrfs]
[290826.486777] btrfs_cow_block+0xc7/0x280 [btrfs]
[290826.487541] btrfs_search_slot+0x768/0xbd0 [btrfs]
[290826.488351] btrfs_lookup_csum+0x5c/0x150 [btrfs]
[290826.489148] btrfs_csum_file_blocks+0x182/0x6b0 [btrfs]
[290826.489974] ? btrfs_global_root+0x4d/0x60 [btrfs]
[290826.490772] btrfs_finish_one_ordered+0x6e9/0xa80 [btrfs]
[290826.491607] btrfs_work_helper+0xba/0x190 [btrfs]
[290826.492397] process_one_work+0x193/0x3d0
[290826.493066] worker_thread+0x2fc/0x410
[290826.493693] ? __pfx_worker_thread+0x10/0x10
[290826.494367] kthread+0xdc/0x110
[290826.494933] ? __pfx_kthread+0x10/0x10
[290826.495528] ret_from_fork+0x2d/0x50
[290826.496127] ? __pfx_kthread+0x10/0x10
[290826.496702] ret_from_fork_asm+0x1a/0x30
[290826.497306] </TASK>
[290830.445264] watchdog: BUG: soft lockup - CPU#12 stuck for 231s! [migration/12:55]
[290830.445999] Modules linked in: mptcp_diag(E) xsk_diag(E) raw_diag(E) unix_diag(E) af_packet_diag(E) netlink_diag(E) tcp_diag(E) udp_diag(E) inet_diag(E) rpcsec_gss_krb5(E) auth_rpcgss(E) nfsv4(E) dns_resolver(E) nfs(E) lockd(E) grace(E) netfs(E) sunrpc(E) nf_conntrack(E) nf_defrag_ipv6(E) nf_defrag_ipv4(E) binfmt_misc(E) zram(E) tls(E) isofs(E) intel_rapl_msr(E) intel_rapl_common(E) kvm_amd(E) ccp(E) kvm(E) i2c_i801(E) vfat(E) virtio_net(E) virtio_gpu(E) i2c_smbus(E) fat(E) net_failover(E) failover(E) virtio_balloon(E) virtio_dma_buf(E) fuse(E) ext4(E) mbcache(E) jbd2(E) sr_mod(E) cdrom(E) sg(E) ahci(E) libahci(E) crct10dif_pclmul(E) crc32_pclmul(E) libata(E) polyval_clmulni(E) polyval_generic(E) ghash_clmulni_intel(E) sha512_ssse3(E) virtio_blk(E) serio_raw(E) btrfs(E) xor(E) zstd_compress(E) raid6_pq(E) libcrc32c(E) crc32c_intel(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E)
[290830.446032] Unloaded tainted modules: edac_mce_amd(E):1 amd_atl(E):2 padlock_aes(E):3
[290830.451772] CPU: 12 PID: 55 Comm: migration/12 Tainted: G EL 6.9.3-3.gdc.el9.x86_64 #1
[290830.452434] Hardware name: RDO OpenStack Compute/RHEL, BIOS edk2-20240214-2.el9 02/14/2024
[290830.453074] Stopper: multi_cpu_stop+0x0/0x100 <- migrate_swap+0xb4/0x130
[290830.453613] RIP: 0010:stop_machine_yield+0x2/0x10
[290830.454073] Code: cc cc cc b8 fe ff ff ff eb df e8 29 04 ad 00 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 90 <c3> cc cc cc cc 66 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90
[290830.455438] RSP: 0018:ffffb956c8303e40 EFLAGS: 00000246
[290830.455945] RAX: 0000000000000000 RBX: ffffb956cee6fb58 RCX: ffff907483a25190
[290830.456513] RDX: ffff907483a00000 RSI: ffffb956cee6fb80 RDI: ffffffffba232600
[290830.457122] RBP: ffffb956cee6fb7c R08: ffffb956cee6fb80 R09: ffffb956cee6fb80
[290830.457697] R10: 0000000000000043 R11: 00000000000db5e1 R12: 0000000000000001
[290830.458277] R13: ffffffffba232600 R14: 0000000000000000 R15: 0000000000000001
[290830.458914] FS: 0000000000000000(0000) GS:ffff907483a00000(0000) knlGS:0000000000000000
[290830.459533] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[290830.460071] CR2: 00007f37dc084d60 CR3: 00000029a7820006 CR4: 0000000000770ef0
[290830.460650] PKRU: 55555554
[290830.461017] Call Trace:
[290830.461355] <IRQ>
[290830.461679] ? watchdog_timer_fn+0x1d7/0x250
[290830.462126] ? __pfx_watchdog_timer_fn+0x10/0x10
[290830.462574] ? __hrtimer_run_queues+0x10f/0x2b0
[290830.463077] ? hrtimer_interrupt+0xff/0x240
[290830.463505] ? __sysvec_apic_timer_interrupt+0x51/0x150
[290830.464017] ? sysvec_apic_timer_interrupt+0x6d/0x90
[290830.464482] </IRQ>
[290830.464814] <TASK>
[290830.465137] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[290830.465620] ? stop_machine_yield+0x2/0x10
[290830.466068] multi_cpu_stop+0x94/0x100
[290830.466471] ? __pfx_multi_cpu_stop+0x10/0x10
[290830.466935] cpu_stopper_thread+0x90/0x140
[290830.467355] ? __pfx_smpboot_thread_fn+0x10/0x10
[290830.467814] smpboot_thread_fn+0xd4/0x1d0
[290830.468231] kthread+0xdc/0x110
[290830.468601] ? __pfx_kthread+0x10/0x10
[290830.469036] ret_from_fork+0x2d/0x50
[290830.469431] ? __pfx_kthread+0x10/0x10
[290830.469850] ret_from_fork_asm+0x1a/0x30
[290830.470267] </TASK>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues
2024-06-17 11:33 Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues Jaroslav Pulchart
@ 2024-06-17 13:46 ` David Sterba
2024-06-17 14:55 ` Jaroslav Pulchart
0 siblings, 1 reply; 5+ messages in thread
From: David Sterba @ 2024-06-17 13:46 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: linux-btrfs, Igor Raits, Jan Cipa, Zdenek Pesek, Daniel Secik
Hi,
On Mon, Jun 17, 2024 at 01:33:59PM +0200, Jaroslav Pulchart wrote:
> Hello,
>
> We recently upgraded part of our production environment to kernel
> 6.9.y. Since then, we've been encountering random kernel "NULL pointer
> dereference" and "soft lockup" errors when using BTRFS. These issues
> occur sporadically, sometimes after several days, and I haven't been
> able to reproduce them consistently. Due to this unpredictability,
> bisecting is not a feasible option.
>
> Attached are console logs from some instances of these issues:
> * "NULL pointer dereference" in "btrfs.issue.1.log"
> * "soft lockup" in "btrfs.issue.1.log"
> Any assistance with investigating and resolving these problems would
> be greatly appreciated.
thanks for the report, the symptoms match the problem that was fixed
recently by commit f3a5367c679d ("btrfs: protect folio::private when
attaching extent buffer folios").
> [1072053.328255] CPU: 15 PID: 2354438 Comm: kworker/u195:18 Tainted: G E 6.9.3-1.gdc.el9.x86_64 #1
6.9.3 does not have the fix yet (unless you're using a manually patched
kernel), it's in 6.9.5.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues
2024-06-17 13:46 ` David Sterba
@ 2024-06-17 14:55 ` Jaroslav Pulchart
2024-07-01 11:39 ` Jaroslav Pulchart
0 siblings, 1 reply; 5+ messages in thread
From: Jaroslav Pulchart @ 2024-06-17 14:55 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, Igor Raits, Jan Cipa, Zdenek Pesek, Daniel Secik
>
> Hi,
>
> On Mon, Jun 17, 2024 at 01:33:59PM +0200, Jaroslav Pulchart wrote:
> > Hello,
> >
> > We recently upgraded part of our production environment to kernel
> > 6.9.y. Since then, we've been encountering random kernel "NULL pointer
> > dereference" and "soft lockup" errors when using BTRFS. These issues
> > occur sporadically, sometimes after several days, and I haven't been
> > able to reproduce them consistently. Due to this unpredictability,
> > bisecting is not a feasible option.
> >
> > Attached are console logs from some instances of these issues:
> > * "NULL pointer dereference" in "btrfs.issue.1.log"
> > * "soft lockup" in "btrfs.issue.1.log"
> > Any assistance with investigating and resolving these problems would
> > be greatly appreciated.
>
> thanks for the report, the symptoms match the problem that was fixed
> recently by commit f3a5367c679d ("btrfs: protect folio::private when
> attaching extent buffer folios").
>
> > [1072053.328255] CPU: 15 PID: 2354438 Comm: kworker/u195:18 Tainted: G E 6.9.3-1.gdc.el9.x86_64 #1
>
> 6.9.3 does not have the fix yet (unless you're using a manually patched
> kernel), it's in 6.9.5.
Thanks, we will try the 6.9.5 asap and report results after a few days.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues
2024-06-17 14:55 ` Jaroslav Pulchart
@ 2024-07-01 11:39 ` Jaroslav Pulchart
2024-07-01 12:53 ` David Sterba
0 siblings, 1 reply; 5+ messages in thread
From: Jaroslav Pulchart @ 2024-07-01 11:39 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, Igor Raits, Jan Cipa, Zdenek Pesek, Daniel Secik
>
> >
> > Hi,
> >
> > On Mon, Jun 17, 2024 at 01:33:59PM +0200, Jaroslav Pulchart wrote:
> > > Hello,
> > >
> > > We recently upgraded part of our production environment to kernel
> > > 6.9.y. Since then, we've been encountering random kernel "NULL pointer
> > > dereference" and "soft lockup" errors when using BTRFS. These issues
> > > occur sporadically, sometimes after several days, and I haven't been
> > > able to reproduce them consistently. Due to this unpredictability,
> > > bisecting is not a feasible option.
> > >
> > > Attached are console logs from some instances of these issues:
> > > * "NULL pointer dereference" in "btrfs.issue.1.log"
> > > * "soft lockup" in "btrfs.issue.1.log"
> > > Any assistance with investigating and resolving these problems would
> > > be greatly appreciated.
> >
> > thanks for the report, the symptoms match the problem that was fixed
> > recently by commit f3a5367c679d ("btrfs: protect folio::private when
> > attaching extent buffer folios").
> >
> > > [1072053.328255] CPU: 15 PID: 2354438 Comm: kworker/u195:18 Tainted: G E 6.9.3-1.gdc.el9.x86_64 #1
> >
> > 6.9.3 does not have the fix yet (unless you're using a manually patched
> > kernel), it's in 6.9.5.
>
> Thanks, we will try the 6.9.5 asap and report results after a few days.
Hello,
we do not see the issue any more with >= 6.9.5
Best regards
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues
2024-07-01 11:39 ` Jaroslav Pulchart
@ 2024-07-01 12:53 ` David Sterba
0 siblings, 0 replies; 5+ messages in thread
From: David Sterba @ 2024-07-01 12:53 UTC (permalink / raw)
To: Jaroslav Pulchart
Cc: dsterba, linux-btrfs, Igor Raits, Jan Cipa, Zdenek Pesek,
Daniel Secik
On Mon, Jul 01, 2024 at 01:39:40PM +0200, Jaroslav Pulchart wrote:
> >
> > >
> > > Hi,
> > >
> > > On Mon, Jun 17, 2024 at 01:33:59PM +0200, Jaroslav Pulchart wrote:
> > > > Hello,
> > > >
> > > > We recently upgraded part of our production environment to kernel
> > > > 6.9.y. Since then, we've been encountering random kernel "NULL pointer
> > > > dereference" and "soft lockup" errors when using BTRFS. These issues
> > > > occur sporadically, sometimes after several days, and I haven't been
> > > > able to reproduce them consistently. Due to this unpredictability,
> > > > bisecting is not a feasible option.
> > > >
> > > > Attached are console logs from some instances of these issues:
> > > > * "NULL pointer dereference" in "btrfs.issue.1.log"
> > > > * "soft lockup" in "btrfs.issue.1.log"
> > > > Any assistance with investigating and resolving these problems would
> > > > be greatly appreciated.
> > >
> > > thanks for the report, the symptoms match the problem that was fixed
> > > recently by commit f3a5367c679d ("btrfs: protect folio::private when
> > > attaching extent buffer folios").
> > >
> > > > [1072053.328255] CPU: 15 PID: 2354438 Comm: kworker/u195:18 Tainted: G E 6.9.3-1.gdc.el9.x86_64 #1
> > >
> > > 6.9.3 does not have the fix yet (unless you're using a manually patched
> > > kernel), it's in 6.9.5.
> >
> > Thanks, we will try the 6.9.5 asap and report results after a few days.
>
>
> Hello,
>
> we do not see the issue any more with >= 6.9.5
Great, thanks for testing.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2024-07-01 12:54 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-06-17 11:33 Linux 6.9.y btrfs: "NULL pointer dereference in attach_eb_folio_to_filemap" and "BUG: soft lockup" issues Jaroslav Pulchart
2024-06-17 13:46 ` David Sterba
2024-06-17 14:55 ` Jaroslav Pulchart
2024-07-01 11:39 ` Jaroslav Pulchart
2024-07-01 12:53 ` David Sterba
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox