* `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
@ 2024-07-17 5:33 Paul Menzel
2024-07-27 21:15 ` Salvatore Bonaccorso
0 siblings, 1 reply; 16+ messages in thread
From: Paul Menzel @ 2024-07-17 5:33 UTC (permalink / raw)
To: Chuck Lever, Jeff Layton; +Cc: linux-nfs, it+linux-nfs
[-- Attachment #1: Type: text/plain, Size: 35506 bytes --]
Dear Linux folks,
Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
04/22/2021, a mount from another server hung. Linux logs:
```
$ dmesg -T
[Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476
(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU
Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
[Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0
init=/bin/systemd audit=0 random.trust_cpu=on
systemd.unified_cgroup_hierarchy
[…]
[Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS
2.11.2 04/22/2021
[…]
[Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid b6f12d96
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000056d1aff1 xid 6ad5584a
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000008075849 xid 406ed865
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000028481e8f xid 7f81b676
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 26099b1f
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid 7ed4dbf5
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid a930d2bf
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 5b099b1f
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b3d4dbf5
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid de30d2bf
[Tue Jul 16 11:20:21 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid 4431d2bf
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000007ce5d717 xid 2c364663
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid df31d2bf
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000000be8f11f xid acdab0f5
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000d6d182c4 xid 3d172cb9
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000976cd55a xid a6cb0a18
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000e11f40dd xid 35f006fd
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000042906e77 xid d9415db0
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000bc03be29 xid eed92785
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000056d1aff1 xid a1d6584a
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000008075849 xid 776fd865
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid edf22d96
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 000000009327f72c xid 12b9ab32
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000b55d160f xid 0e3dd152
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000976cd55a xid a7cb0a18
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000042906e77 xid da415db0
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000bc03be29 xid efd92785
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000008075849 xid 786fd865
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid eef22d96
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 9f91a3d2
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000060d5bb55 xid 3aea57c8
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 73a5017a
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 5d0a9b1f
[Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
[Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply:
calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
[Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP)
idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
[Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
[Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
[Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task
stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:36:40 2024] Call Trace:
[Tue Jul 16 11:36:40 2024] <IRQ>
[Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:36:40 2024] </IRQ>
[Tue Jul 16 11:36:40 2024] <TASK>
[Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01
c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00
00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f
1f 84 00 00
[Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e
RCX: 000000000000100d
[Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046
RDI: ffffffff82435600
[Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770
R09: ffffffffa012c788
[Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283
R12: 0000000000000000
[Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000
R15: ffff88909311c005
[Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
[Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
[Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:36:40 2024] </TASK>
[Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP)
idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
[Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
[Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
[Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task
stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:37:19 2024] Call Trace:
[Tue Jul 16 11:37:19 2024] <IRQ>
[Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:37:19 2024] </IRQ>
[Tue Jul 16 11:37:19 2024] <TASK>
[Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00
00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00
00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f
44 00 00 fa
[Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
[Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500
RCX: 0000000000000001
[Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500
RDI: ffffffffa012c700
[Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770
R09: ffffffffa012c788
[Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283
R12: ffff88997131a530
[Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000
R15: ffff88909311c005
[Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
[Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:37:19 2024] </TASK>
[Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[…]
```
This continues. Please find the output of `dmesg -T` attached.
Status of rpcio at 12:00:
```
@furoncles:/scratch/local2/20240716--forensics$ grep -e '/proc/328' -e
'proc/30413' forensics-00th_min.log
/proc/328/. 0:0 1720083010
/proc/328/cwd /
/proc/328/stat 328 (rpciod) I 2 0 0 0 -1 69238880 0 0 0 0 0 0 0 0 0 -20
1 0 1781 0 0 18446744073709551615 0 0 0 0 0 0 0 2147483647 0 1 0 0 17 1
0 0 0 0 0 0 0 0 0 0 0 0 0
/proc/328/statm 0 0 0 0 0 0 0
/proc/328/status Name: rpciod
/proc/328/status Umask: 0000
/proc/328/status State: I (idle)
/proc/328/status Tgid: 328
/proc/328/status Ngid: 0
/proc/328/status Pid: 328
/proc/328/status PPid: 2
/proc/328/status TracerPid: 0
/proc/328/status Uid: 0 0 0 0
/proc/328/status Gid: 0 0 0 0
/proc/328/status FDSize: 64
/proc/328/status Groups:
/proc/328/status NStgid: 328
/proc/328/status NSpid: 328
/proc/328/status NSpgid: 0
/proc/328/status NSsid: 0
/proc/328/status Threads: 1
/proc/328/status SigQ: 1/513101
/proc/328/status SigPnd: 0000000000000000
/proc/328/status ShdPnd: 0000000000000000
/proc/328/status SigBlk: 0000000000000000
/proc/328/status SigIgn: ffffffffffffffff
/proc/328/status SigCgt: 0000000000000000
/proc/328/status CapInh: 0000000000000000
/proc/328/status CapPrm: 000001ffffffffff
/proc/328/status CapEff: 000001ffffffffff
/proc/328/status CapBnd: 000001ffffffffff
/proc/328/status CapAmb: 0000000000000000
/proc/328/status NoNewPrivs: 0
/proc/328/status Seccomp: 0
/proc/328/status Seccomp_filters: 0
/proc/328/status Speculation_Store_Bypass: thread vulnerable
/proc/328/status SpeculationIndirectBranch: conditional enabled
/proc/328/status Cpus_allowed: ffff
/proc/328/status Cpus_allowed_list: 0-15
/proc/328/status Mems_allowed: 00000000,00000003
/proc/328/status Mems_allowed_list: 0-1
/proc/328/status voluntary_ctxt_switches: 2
/proc/328/status nonvoluntary_ctxt_switches: 0
/proc/328/io rchar: 0
/proc/328/io wchar: 0
/proc/328/io syscr: 0
/proc/328/io syscw: 0
/proc/328/io read_bytes: 0
/proc/328/io write_bytes: 0
/proc/328/io cancelled_write_bytes: 0
/proc/328/stack [<0>] rescuer_thread+0x2d4/0x390
/proc/328/stack [<0>] kthread+0x115/0x140
/proc/328/stack [<0>] ret_from_fork+0x1f/0x30
/proc/30413/. 0:0 1721118610
/proc/30413/cwd /
/proc/30413/stat 30413 (kworker/u34:2+rpciod) R 2 0 0 0 -1 69501024 0
937 0 0 0 142411 0 0 20 0 1 0 110072519 0 0 18446744073709551615 0 0 0 0
0 0 0 2147483647 0 0 0 0 17 7 0 0 0 0 0 0 0 0 0 0 0 0 0
/proc/30413/statm 0 0 0 0 0 0 0
/proc/30413/status Name: kworker/u34:2+rpciod
/proc/30413/status Umask: 0000
/proc/30413/status State: R (running)
/proc/30413/status Tgid: 30413
/proc/30413/status Ngid: 0
/proc/30413/status Pid: 30413
/proc/30413/status PPid: 2
/proc/30413/status TracerPid: 0
/proc/30413/status Uid: 0 0 0 0
/proc/30413/status Gid: 0 0 0 0
/proc/30413/status FDSize: 64
/proc/30413/status Groups:
/proc/30413/status NStgid: 30413
/proc/30413/status NSpid: 30413
/proc/30413/status NSpgid: 0
/proc/30413/status NSsid: 0
/proc/30413/status Threads: 1
/proc/30413/status SigQ: 1/513101
/proc/30413/status SigPnd: 0000000000000000
/proc/30413/status ShdPnd: 0000000000000000
/proc/30413/status SigBlk: 0000000000000000
/proc/30413/status SigIgn: ffffffffffffffff
/proc/30413/status SigCgt: 0000000000000000
/proc/30413/status CapInh: 0000000000000000
/proc/30413/status CapPrm: 000001ffffffffff
/proc/30413/status CapEff: 000001ffffffffff
/proc/30413/status CapBnd: 000001ffffffffff
/proc/30413/status CapAmb: 0000000000000000
/proc/30413/status NoNewPrivs: 0
/proc/30413/status Seccomp: 0
/proc/30413/status Seccomp_filters: 0
/proc/30413/status Speculation_Store_Bypass: thread vulnerable
/proc/30413/status SpeculationIndirectBranch: conditional enabled
/proc/30413/status Cpus_allowed: aaaa
/proc/30413/status Cpus_allowed_list: 1,3,5,7,9,11,13,15
/proc/30413/status Mems_allowed: 00000000,00000003
/proc/30413/status Mems_allowed_list: 0-1
/proc/30413/status voluntary_ctxt_switches: 17319
/proc/30413/status nonvoluntary_ctxt_switches: 141
/proc/30413/io rchar: 106624
/proc/30413/io wchar: 40157
/proc/30413/io syscr: 169
/proc/30413/io syscw: 89
/proc/30413/io read_bytes: 0
/proc/30413/io write_bytes: 118784
/proc/30413/io cancelled_write_bytes: 0
```
Can the CPU usage be deduced from the output above?
Debugging of hanging mount:
```
furoncles:~/# ps aux | grep mount
root 538 0.0 0.0 19776 1044 ? S 12:20 0:00
/usr/bin/mount -t nfs -s -o nosuid,sec=mariux
snugglebites:/amd/snugglebites/M/M8005/project/XYZ /project/XYZ
root 539 0.0 0.0 4352 1528 ? D 12:20 0:00
/sbin/mount.nfs snugglebites:/amd/snugglebites/M/M8005/project/XYZ
/project/XYZ -s -o rw,nosuid,sec=mariux
root 804 0.0 0.0 20044 2544 pts/1 S+ 12:36 0:00 grep mount
root 915 0.0 0.0 9272 7684 ? Ss Jul03 0:27
/usr/sbin/rpc.mountd --foreground --manage-gids
root 926 0.0 0.0 1274056 7904 ? Ssl Jul03 0:05
/usr/sbin/automount -v
root:furoncles:~/# more /proc/539/stack
[<0>] __wait_rcu_gp+0x12b/0x140
[<0>] synchronize_rcu+0x77/0xa0
[<0>] nfs_free_server+0xe/0xb0 [nfs]
[<0>] nfs_kill_super+0x2b/0x40 [nfs]
[<0>] deactivate_locked_super+0x2c/0x90
[<0>] cleanup_mnt+0xee/0x180
[<0>] task_work_run+0x54/0x90
[<0>] exit_to_user_mode_prepare+0x15a/0x160
[<0>] syscall_exit_to_user_mode+0x1d/0x40
[<0>] do_syscall_64+0x48/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
root:furoncles:~/# kill -9 539
root:furoncles:~/# more /proc/539/stack
[<0>] __wait_rcu_gp+0x12b/0x140
[<0>] synchronize_rcu+0x77/0xa0
[<0>] nfs_free_server+0xe/0xb0 [nfs]
[<0>] nfs_kill_super+0x2b/0x40 [nfs]
[<0>] deactivate_locked_super+0x2c/0x90
[<0>] cleanup_mnt+0xee/0x180
[<0>] task_work_run+0x54/0x90
[<0>] exit_to_user_mode_prepare+0x15a/0x160
[<0>] syscall_exit_to_user_mode+0x1d/0x40
[<0>] do_syscall_64+0x48/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
```
Rebooting the other server worked, but then a umount of another
directory hung:
root:furoncles:~/# ps aux | grep mount
root 915 0.0 0.0 9272 7684 ? Ss Jul03 0:27
/usr/sbin/rpc.mountd --foreground --manage-gids
root 926 0.0 0.0 1274056 7880 ? Ssl Jul03 0:05
/usr/sbin/automount -v
root 1586 0.0 0.0 19760 1100 ? D 13:16 0:00
/usr/bin/umount -c /scratch/local2
root 1645 0.0 0.0 20044 2508 pts/0 S+ 13:19 0:00
grep mount
root:furoncles:~/# sudo lsof | grep scratch
^C^C^C
This hung for a while. Then another mount hung:
root:furoncles:~/# ps aux | grep mount
root 915 0.0 0.0 9272 7684 ? Ss Jul03 0:27
/usr/sbin/rpc.mountd --foreground --manage-gids
root 926 0.0 0.0 1274056 7912 ? Ssl Jul03 0:05
/usr/sbin/automount -v
root 1683 0.0 0.0 19776 1112 ? S 13:21 0:00
/usr/bin/mount -t nfs -s -o nosuid,sec=mariux
handsomejack:/amd/handsomejack/2/home/edv/pmenzel /home/pmenzel
root 1684 0.0 0.0 4352 1612 ? D 13:21 0:00
/sbin/mount.nfs handsomejack:/amd/handsomejack/2/home/edv/pmenzel
/home/pmenzel -s -o rw,nosuid,sec=mariux
root 1744 0.0 0.0 20044 2544 pts/0 S+ 13:23 0:00
grep mount
Killing it did not help as expected for a process in uninterruptale sleep:
root:furoncles:~/# kill -9 1684
root:furoncles:~/# ps aux | grep mount
root 915 0.0 0.0 9272 7684 ? Ss Jul03 0:27
/usr/sbin/rpc.mountd --foreground --manage-gids
root 926 0.0 0.0 1274056 7912 ? Ssl Jul03 0:05
/usr/sbin/automount -v
root 1683 0.0 0.0 19776 1112 ? S 13:21 0:00
/usr/bin/mount -t nfs -s -o nosuid,sec=mariux
handsomejack:/amd/handsomejack/2/home/edv/pmenzel /home/pmenzel
root 1684 0.0 0.0 4352 1612 ? D 13:21 0:00
/sbin/mount.nfs handsomejack:/amd/handsomejack/2/home/edv/pmenzel
/home/pmenzel -s -o rw,nosuid,sec=mariux
root 1758 0.0 0.0 20044 2536 pts/0 S+ 13:23 0:00
grep mount
root:furoncles:~/# kill -9 1683
root:furoncles:~/# ps aux | grep mount
root 915 0.0 0.0 9272 7684 ? Ss Jul03 0:27
/usr/sbin/rpc.mountd --foreground --manage-gids
root 926 0.0 0.0 1274056 7912 ? Ssl Jul03 0:05
/usr/sbin/automount -v
root 1683 0.0 0.0 0 0 ? Z 13:21 0:00
[mount] <defunct>
root 1684 0.0 0.0 4352 1612 ? D 13:21 0:00
/sbin/mount.nfs handsomejack:/amd/handsomejack/2/home/edv/pmenzel
/home/pmenzel -s -o rw,nosuid,sec=mariux
root 1765 0.0 0.0 20044 2676 pts/0 S+ 13:23 0:00
grep mount
```
root:furoncles:~/# ps aux | grep D
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 9 0.0 0.0 0 0 ? D Jul03 0:04
[kworker/u32:0+nfsd4_callbacks]
root 101 0.0 0.0 0 0 ? DN Jul03 0:18
[khugepaged]
root 808 4.0 0.0 0 0 ? D Jul03 759:22 [md0_raid6]
root 964 0.0 0.0 6816 4932 ? Ss Jul03 0:04 sshd:
/usr/sbin/sshd -D -o ListenAddress 141.14.24.151 [listener] 1 of
4096-4096 startups
root 980 0.3 0.0 0 0 ? D Jul03 55:37 [nfsd]
root 981 0.3 0.0 0 0 ? D Jul03 59:34 [nfsd]
root 982 0.3 0.0 0 0 ? D Jul03 62:27 [nfsd]
root 983 0.3 0.0 0 0 ? D Jul03 66:04 [nfsd]
root 984 0.3 0.0 0 0 ? D Jul03 71:15 [nfsd]
root 985 0.4 0.0 0 0 ? D Jul03 80:47 [nfsd]
root 986 0.5 0.0 0 0 ? D Jul03 93:32 [nfsd]
root 987 0.6 0.0 0 0 ? D Jul03 121:26 [nfsd]
nobody 1422 0.0 0.0 17252 1976 ? Ss Jul03 0:11
/usr/sbin/lighttpd -D -f /etc/mxloadmonitor-lighttpd.conf
pmenzel 1632 0.0 0.0 15764 6392 ? Ds 13:18 0:00
sshd-session: pmenzel [priv]
root 1642 0.0 0.0 20624 3704 pts/0 Ds+ 13:19 0:00 -bash
root 1684 0.0 0.0 4352 1612 ? D 13:21 0:00
/sbin/mount.nfs handsomejack:/amd/handsomejack/2/home/edv/pmenzel
/home/pmenzel -s -o rw,nosuid,sec=mariux
root 1881 0.0 0.0 20044 2444 pts/2 S+ 13:28 0:00 grep D
root 30497 0.0 0.0 0 0 ? D 10:27 0:04
[kworker/u33:4+nfsd4]
root:furoncles:~/# ps aux | grep kworker
root 7 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/0:0H-kblockd]
root 9 0.0 0.0 0 0 ? D Jul03 0:04
[kworker/u32:0+nfsd4_callbacks]
root 21 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/1:0H-kblockd]
root 27 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/2:0H-kblockd]
root 32 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/3:0H-kblockd]
root 37 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/4:0H-kblockd]
root 42 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/5:0H-kblockd]
root 47 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/6:0H-kblockd]
root 52 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/7:0H-kblockd]
root 57 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/8:0H-kblockd]
root 62 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/9:0H-kblockd]
root 67 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/10:0H-events_highpri]
root 72 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/11:0H-kblockd]
root 77 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/12:0H-kblockd]
root 82 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/13:0H-kblockd]
root 87 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/14:0H-kblockd]
root 92 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/15:0H-kblockd]
root 142 0.3 0.0 0 0 ? I< Jul03 60:38
[kworker/10:1H-kblockd]
root 207 0.0 0.0 0 0 ? I Jul03 0:02
[kworker/u32:1-poll_mpt3sas0_statu]
root 208 0.3 0.0 0 0 ? I< Jul03 57:05
[kworker/3:1H-kblockd]
root 209 0.2 0.0 0 0 ? I< Jul03 52:16
[kworker/15:1H-kblockd]
root 210 0.2 0.0 0 0 ? I< Jul03 55:24
[kworker/1:1H-kblockd]
root 211 0.2 0.0 0 0 ? I< Jul03 55:34
[kworker/7:1H-kblockd]
root 212 0.2 0.0 0 0 ? I< Jul03 51:58
[kworker/13:1H-kblockd]
root 213 0.2 0.0 0 0 ? I< Jul03 55:15
[kworker/5:1H-kblockd]
root 214 0.2 0.0 0 0 ? I< Jul03 53:53
[kworker/9:1H-kblockd]
root 215 0.2 0.0 0 0 ? I< Jul03 53:31
[kworker/11:1H-kblockd]
root 218 0.3 0.0 0 0 ? I< Jul03 62:37
[kworker/4:1H-kblockd]
root 219 0.3 0.0 0 0 ? I< Jul03 57:36
[kworker/0:1H-kblockd]
root 273 0.3 0.0 0 0 ? I< Jul03 62:53
[kworker/2:1H-kblockd]
root 304 0.3 0.0 0 0 ? I< Jul03 62:06
[kworker/6:1H-kblockd]
root 329 0.0 0.0 0 0 ? I< Jul03 0:00
[kworker/u35:0]
root 380 0.0 0.0 0 0 ? I 12:10 0:00
[kworker/8:0-events]
root 482 0.3 0.0 0 0 ? I< Jul03 58:51
[kworker/14:1H-kblockd]
root 502 0.3 0.0 0 0 ? I< Jul03 59:07
[kworker/12:1H-kblockd]
root 519 0.3 0.0 0 0 ? I< Jul03 66:24
[kworker/8:1H-kblockd]
root 1011 0.0 0.0 0 0 ? I 12:46 0:00
[kworker/13:2]
root 1048 0.0 0.0 0 0 ? I 12:48 0:00
[kworker/4:2-mm_percpu_wq]
root 1063 0.0 0.0 0 0 ? I 12:48 0:00
[kworker/15:2-mm_percpu_wq]
root 1168 0.0 0.0 0 0 ? I 12:55 0:00
[kworker/1:3-events]
root 1339 0.0 0.0 0 0 ? I 13:03 0:00
[kworker/1:0-events]
root 1367 0.0 0.0 0 0 ? I 13:05 0:00
[kworker/0:1-events_power_efficient]
root 1384 0.0 0.0 0 0 ? I 13:05 0:00
[kworker/2:1-mm_percpu_wq]
root 1435 0.0 0.0 0 0 ? I 13:08 0:00
[kworker/10:0-events]
root 1436 0.0 0.0 0 0 ? I 13:08 0:00
[kworker/7:0-events]
root 1473 0.0 0.0 0 0 ? I 13:09 0:00
[kworker/u34:1-xfs-cil/sdaw1]
root 1524 0.0 0.0 0 0 ? I 13:12 0:00
[kworker/11:0-events]
root 1584 0.0 0.0 0 0 ? I 13:16 0:00
[kworker/13:0-events]
root 1588 0.0 0.0 0 0 ? I 13:16 0:00
[kworker/8:1-events]
root 1589 0.0 0.0 0 0 ? I 13:16 0:00
[kworker/6:2-events]
root 1599 0.0 0.0 0 0 ? I 13:16 0:00
[kworker/u34:0-xfs-cil/sdaw1]
root 1650 0.0 0.0 0 0 ? I 13:19 0:00
[kworker/u33:1-events_unbound]
root 1783 0.0 0.0 0 0 ? I 13:24 0:00
[kworker/u33:0]
root 1818 0.0 0.0 0 0 ? I 13:26 0:00
[kworker/u34:3-flush-67:0]
root 1829 0.0 0.0 0 0 ? I 13:27 0:00
[kworker/1:1-events]
root 1883 0.0 0.0 20044 2544 pts/2 S+ 13:28 0:00 grep
kworker
root 16164 0.0 0.0 0 0 ? I< Jul09 0:00
[kworker/u36:1-xprtiod]
root 20434 0.0 0.0 0 0 ? I< Jul09 0:00
[kworker/u36:0-xprtiod]
root 23362 0.0 0.0 0 0 ? I< Jul08 0:00
[kworker/u37:1-xprtiod]
root 30413 61.2 0.0 0 0 ? R 10:24 112:30
[kworker/u34:2+rpciod]
root 30432 0.0 0.0 0 0 ? I< Jul11 0:00
[kworker/u37:0-xprtiod]
root 30497 0.0 0.0 0 0 ? D 10:27 0:04
[kworker/u33:4+nfsd4]
root 31459 0.0 0.0 0 0 ? I 11:05 0:01
[kworker/15:1-events]
root 31511 0.0 0.0 0 0 ? I 11:08 0:02
[kworker/14:0-events]
root 31521 0.0 0.0 0 0 ? I 11:08 0:02
[kworker/12:1-events]
root 31587 0.0 0.0 0 0 ? I 11:11 0:02
[kworker/5:2-events]
root 31861 0.0 0.0 0 0 ? I 11:20 0:01
[kworker/6:0-events]
root 31962 0.0 0.0 0 0 ? I 11:25 0:00
[kworker/3:2-events]
root 31963 0.0 0.0 0 0 ? I 11:25 0:00
[kworker/5:0-mm_percpu_wq]
root 31965 0.0 0.0 0 0 ? I 11:25 0:00
[kworker/14:2-xfsalloc]
root 31978 0.0 0.0 0 0 ? R 11:26 0:01
[kworker/7:2-events_power_efficient]
root 31979 0.0 0.0 0 0 ? I 11:26 0:01
[kworker/9:1-mm_percpu_wq]
root 32016 0.0 0.0 0 0 ? I 11:28 0:02
[kworker/2:2-events]
root 32018 0.0 0.0 0 0 ? I 11:28 0:00
[kworker/4:1-mm_percpu_wq]
root 32045 0.0 0.0 0 0 ? I 11:29 0:01
[kworker/10:1-events]
root 32085 0.0 0.0 0 0 ? I 11:31 0:01
[kworker/0:2-events_power_efficient]
root 32088 0.0 0.0 0 0 ? I 11:32 0:00
[kworker/3:0-events]
root 32149 0.0 0.0 0 0 ? I 11:33 0:00
[kworker/9:2-events]
root 32205 0.0 0.0 0 0 ? I 11:36 0:00
[kworker/u32:2-poll_mpt3sas0_statu]
root 32617 0.0 0.0 0 0 ? I 12:00 0:00
[kworker/u33:2-events_unbound]
root 32618 0.0 0.0 0 0 ? I 12:00 0:00
[kworker/11:2-events]
root 32691 0.0 0.0 0 0 ? I 12:02 0:00
[kworker/12:0-events]
```
`top` shows rpciod using 100 % of one CPU/thread:
30413 root 20 0 0 0 0 R 100.0 0.0
121:58.21 kworker/u34:2+rpciod
Details for `mount.nfs` in uninterruptable sleep:
root 1684 0.0 0.0 4352 1612 ? D 13:21 0:00
/sbin/mount.nfs handsomejack:/amd/handsomejack/2/home/edv/pmenzel
/home/pmenzel -s -o rw,nosuid,sec=mariux
```
# cat /proc/1684/stack
[<0>] rcu_barrier+0x1a5/0x280
[<0>] cgroup_writeback_umount+0x1f/0x30
[<0>] generic_shutdown_super+0x2f/0x100
[<0>] nfs_kill_super+0x1b/0x40 [nfs]
[<0>] deactivate_locked_super+0x2c/0x90
[<0>] cleanup_mnt+0xee/0x180
[<0>] task_work_run+0x54/0x90
[<0>] exit_to_user_mode_prepare+0x15a/0x160
[<0>] syscall_exit_to_user_mode+0x1d/0x40
[<0>] do_syscall_64+0x48/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
# cat /proc/1684/stack
[<0>] synchronize_rcu_expedited+0x330/0x6b0
[<0>] bdi_unregister+0x76/0x220
[<0>] bdi_put+0x5a/0x60
[<0>] generic_shutdown_super+0xe9/0x100
[<0>] nfs_kill_super+0x1b/0x40 [nfs]
[<0>] deactivate_locked_super+0x2c/0x90
[<0>] cleanup_mnt+0xee/0x180
[<0>] task_work_run+0x54/0x90
[<0>] exit_to_user_mode_prepare+0x15a/0x160
[<0>] syscall_exit_to_user_mode+0x1d/0x40
[<0>] do_syscall_64+0x48/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
```
### Trying to reboot with `sync`
```
root:furoncles:~/# ps aux | grep sync
root 1907 0.0 0.0 0 0 ? D 13:30 0:00
[kworker/11:1+xfs-sync/md0]
root 2102 0.0 0.0 19040 980 pts/2 D+ 13:35 0:00 sync
root 2187 0.0 0.0 20044 2672 pts/1 S+ 13:37 0:00 grep sync
root:furoncles:~/# more /proc/2102/stack
[<0>] xlog_wait_on_iclog+0x11f/0x170
[<0>] xfs_fs_sync_fs+0x40/0xb0
[<0>] iterate_supers+0x6f/0xe0
[<0>] ksys_sync+0x60/0xa0
[<0>] __do_sys_sync+0xa/0x20
[<0>] do_syscall_64+0x3c/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x6c/0xd6
```
This stays like this for several minutes:
```
oot:furoncles:~/# top
top - 13:38:33 up 12 days, 20:58, 3 users, load average: 23.72, 22.93,
20.65
Tasks: 321 total, 5 running, 315 sleeping, 0 stopped, 1 zombie
%Cpu(s): 0.0 us, 6.3 sy, 0.0 ni, 87.5 id, 6.3 wa, 0.0 hi, 0.0 si,
0.0 st
MiB Mem : 128294.0 total, 3828.9 free, 17175.0 used, 107290.1 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 110074.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
30413 root 20 0 0 0 0 R 100.0 0.0 122:16.25
kworker/u34:2+rpciod
2201 root 20 0 20732 3300 2552 R 0.3 0.0 0:00.03
top
1 root 20 0 169092 8848 4384 S 0.0 0.0 0:50.57
systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.94
kthreadd
3 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
rcu_gp
4 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
rcu_par_gp
5 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
netns
7 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/0:0H-kblockd
9 root 20 0 0 0 0 D 0.0 0.0 0:04.74
kworker/u32:0+nfsd4_callbacks
10 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
mm_percpu_wq
11 root 20 0 0 0 0 S 0.0 0.0 0:00.00
rcu_tasks_rude_
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00
rcu_tasks_trace
13 root 20 0 0 0 0 S 0.0 0.0 1:16.12
ksoftirqd/0
14 root 20 0 0 0 0 I 0.0 0.0 7:54.74
rcu_sched
15 root rt 0 0 0 0 S 0.0 0.0 0:06.19
migration/0
16 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/0
17 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/1
18 root rt 0 0 0 0 S 0.0 0.0 0:07.83
migration/1
19 root 20 0 0 0 0 S 0.0 0.0 0:46.20
ksoftirqd/1
21 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/1:0H-kblockd
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/2
24 root rt 0 0 0 0 S 0.0 0.0 0:07.14
migration/2
25 root 20 0 0 0 0 S 0.0 0.0 0:49.00
ksoftirqd/2
27 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/2:0H-kblockd
28 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/3
29 root rt 0 0 0 0 S 0.0 0.0 0:06.03
migration/3
30 root 20 0 0 0 0 S 0.0 0.0 0:37.88
ksoftirqd/3
32 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/3:0H-kblockd
33 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/4
34 root rt 0 0 0 0 S 0.0 0.0 0:06.38
migration/4
35 root 20 0 0 0 0 S 0.0 0.0 0:44.86
ksoftirqd/4
37 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/4:0H-kblockd
38 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/5
39 root rt 0 0 0 0 S 0.0 0.0 0:05.54
migration/5
40 root 20 0 0 0 0 S 0.0 0.0 0:34.50
ksoftirqd/5
42 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/5:0H-kblockd
43 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/6
44 root rt 0 0 0 0 S 0.0 0.0 0:05.99
migration/6
45 root 20 0 0 0 0 S 0.0 0.0 0:40.42
ksoftirqd/6
47 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/6:0H-kblockd
48 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/7
49 root rt 0 0 0 0 S 0.0 0.0 0:05.14
migration/7
50 root 20 0 0 0 0 S 0.0 0.0 0:35.38
ksoftirqd/7
52 root 0 -20 0 0 0 I 0.0 0.0 0:00.00
kworker/7:0H-kblockd
53 root 20 0 0 0 0 S 0.0 0.0 0:00.00
cpuhp/8
54 root rt 0 0 0 0 S 0.0 0.0 0:03.81
migration/8
55 root 20 0 0 0 0 S 0.0 0.0 0:30.54
ksoftirqd/8
Connection to furoncles closed by remote host. 0.0 0.0 0:00.00
kworker/8:0H-kblockd
Connection to furoncles closed.
```
The system still hung and needed to be power cycled.
Does somebody have any insight?
Kind regards,
Paul
[-- Attachment #2: 20240716--furoncles--linux-5.15.160--dmesg-T.txt --]
[-- Type: text/plain, Size: 625239 bytes --]
[Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476 (root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
[Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
[Wed Jul 3 16:39:34 2024] BIOS-provided physical RAM map:
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x0000000000000100-0x000000000009ffff] usable
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x0000000000100000-0x00000000682fefff] usable
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x00000000682ff000-0x000000006ebfefff] reserved
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x000000006ebff000-0x000000006f9fefff] ACPI NVS
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x000000006f9ff000-0x000000006fffefff] ACPI data
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[Wed Jul 3 16:39:34 2024] BIOS-e820: [mem 0x0000000100000000-0x000000207fffffff] usable
[Wed Jul 3 16:39:34 2024] NX (Execute Disable) protection: active
[Wed Jul 3 16:39:34 2024] e820: update [mem 0x00100000-0x0010006f] usable ==> usable
[Wed Jul 3 16:39:34 2024] extended physical RAM map:
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x0000000000000100-0x000000000009ffff] usable
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x0000000000100000-0x000000000010006f] usable
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x0000000000100070-0x00000000682fefff] usable
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x00000000682ff000-0x000000006ebfefff] reserved
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x000000006ebff000-0x000000006f9fefff] ACPI NVS
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x000000006f9ff000-0x000000006fffefff] ACPI data
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x000000006ffff000-0x000000006fffffff] usable
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x0000000070000000-0x000000008fffffff] reserved
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x00000000fe000000-0x00000000fe010fff] reserved
[Wed Jul 3 16:39:34 2024] reserve setup_data: [mem 0x0000000100000000-0x000000207fffffff] usable
[Wed Jul 3 16:39:34 2024] efi: EFI v2.70 by Dell Inc.
[Wed Jul 3 16:39:34 2024] efi: ACPI=0x6fffe000 ACPI 2.0=0x6fffe014 SMBIOS=0x68e3b000 SMBIOS 3.0=0x68e39000 MEMATTR=0x653c4020
[Wed Jul 3 16:39:34 2024] SMBIOS 3.2.0 present.
[Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
[Wed Jul 3 16:39:34 2024] tsc: Detected 3800.000 MHz processor
[Wed Jul 3 16:39:34 2024] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[Wed Jul 3 16:39:34 2024] e820: remove [mem 0x000a0000-0x000fffff] usable
[Wed Jul 3 16:39:34 2024] last_pfn = 0x2080000 max_arch_pfn = 0x400000000
[Wed Jul 3 16:39:34 2024] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[Wed Jul 3 16:39:34 2024] e820: update [mem 0x70000000-0x73ffffff] usable ==> reserved
[Wed Jul 3 16:39:34 2024] e820: update [mem 0x80000000-0xffffffff] usable ==> reserved
[Wed Jul 3 16:39:34 2024] x2apic: enabled by BIOS, switching to x2apic ops
[Wed Jul 3 16:39:34 2024] last_pfn = 0x70000 max_arch_pfn = 0x400000000
[Wed Jul 3 16:39:34 2024] Using GB pages for direct mapping
[Wed Jul 3 16:39:34 2024] Secure boot could not be determined
[Wed Jul 3 16:39:34 2024] RAMDISK: [mem 0x207c1a8000-0x207cffffff]
[Wed Jul 3 16:39:34 2024] ACPI: Early table checksum verification disabled
[Wed Jul 3 16:39:34 2024] ACPI: RSDP 0x000000006FFFE014 000024 (v02 DELL )
[Wed Jul 3 16:39:34 2024] ACPI: XSDT 0x000000006FBFF188 0000F4 (v01 DELL PE_SC3 00000000 01000013)
[Wed Jul 3 16:39:34 2024] ACPI: FACP 0x000000006FFF8000 000114 (v06 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: DSDT 0x000000006FD00000 2E81B0 (v02 DELL PE_SC3 00000003 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: FACS 0x000000006F76E000 000040
[Wed Jul 3 16:39:34 2024] ACPI: SSDT 0x000000006FFFC000 00046C (v02 INTEL ADDRXLAT 00000001 INTL 20180508)
[Wed Jul 3 16:39:34 2024] ACPI: MCEJ 0x000000006FFFB000 000130 (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: WD__ 0x000000006FFFA000 000134 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: SLIC 0x000000006FFF9000 000024 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: HPET 0x000000006FFF7000 000038 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: APIC 0x000000006FFF5000 0016DE (v04 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: MCFG 0x000000006FFF4000 00003C (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: MIGT 0x000000006FFF3000 000040 (v01 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: MSCT 0x000000006FFF2000 000090 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: PCAT 0x000000006FFF1000 000088 (v02 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: PCCT 0x000000006FFF0000 00006E (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: RASF 0x000000006FFEF000 000030 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: SLIT 0x000000006FFEE000 00042C (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: SRAT 0x000000006FFEB000 002D30 (v03 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: SVOS 0x000000006FFEA000 000032 (v01 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: WSMT 0x000000006FFE9000 000028 (v01 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: OEM4 0x000000006FC52000 0AD1C1 (v02 INTEL CPU CST 00003000 INTL 20180508)
[Wed Jul 3 16:39:34 2024] ACPI: SSDT 0x000000006FC1A000 037465 (v02 INTEL SSDT PM 00004000 INTL 20180508)
[Wed Jul 3 16:39:34 2024] ACPI: SSDT 0x000000006FC00000 000C94 (v02 DELL PE_SC3 00000000 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: SSDT 0x000000006FC16000 00357F (v02 INTEL SpsNm 00000002 INTL 20180508)
[Wed Jul 3 16:39:34 2024] ACPI: DMAR 0x000000006FFFD000 0001A0 (v01 DELL PE_SC3 00000001 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: HEST 0x000000006FC15000 00017C (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: BERT 0x000000006FC14000 000030 (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: ERST 0x000000006FC13000 000230 (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: EINJ 0x000000006FC12000 000150 (v01 DELL PE_SC3 00000002 DELL 00000001)
[Wed Jul 3 16:39:34 2024] ACPI: Reserving FACP table memory at [mem 0x6fff8000-0x6fff8113]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving DSDT table memory at [mem 0x6fd00000-0x6ffe81af]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving FACS table memory at [mem 0x6f76e000-0x6f76e03f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SSDT table memory at [mem 0x6fffc000-0x6fffc46b]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving MCEJ table memory at [mem 0x6fffb000-0x6fffb12f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving WD__ table memory at [mem 0x6fffa000-0x6fffa133]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SLIC table memory at [mem 0x6fff9000-0x6fff9023]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving HPET table memory at [mem 0x6fff7000-0x6fff7037]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving APIC table memory at [mem 0x6fff5000-0x6fff66dd]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving MCFG table memory at [mem 0x6fff4000-0x6fff403b]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving MIGT table memory at [mem 0x6fff3000-0x6fff303f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving MSCT table memory at [mem 0x6fff2000-0x6fff208f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving PCAT table memory at [mem 0x6fff1000-0x6fff1087]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving PCCT table memory at [mem 0x6fff0000-0x6fff006d]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving RASF table memory at [mem 0x6ffef000-0x6ffef02f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SLIT table memory at [mem 0x6ffee000-0x6ffee42b]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SRAT table memory at [mem 0x6ffeb000-0x6ffedd2f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SVOS table memory at [mem 0x6ffea000-0x6ffea031]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving WSMT table memory at [mem 0x6ffe9000-0x6ffe9027]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving OEM4 table memory at [mem 0x6fc52000-0x6fcff1c0]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SSDT table memory at [mem 0x6fc1a000-0x6fc51464]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SSDT table memory at [mem 0x6fc00000-0x6fc00c93]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving SSDT table memory at [mem 0x6fc16000-0x6fc1957e]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving DMAR table memory at [mem 0x6fffd000-0x6fffd19f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving HEST table memory at [mem 0x6fc15000-0x6fc1517b]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving BERT table memory at [mem 0x6fc14000-0x6fc1402f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving ERST table memory at [mem 0x6fc13000-0x6fc1322f]
[Wed Jul 3 16:39:34 2024] ACPI: Reserving EINJ table memory at [mem 0x6fc12000-0x6fc1214f]
[Wed Jul 3 16:39:34 2024] Setting APIC routing to cluster x2apic.
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x0012 -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0022 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x000a -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0028 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x0004 -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0024 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x001a -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x003a -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x0013 -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0023 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x000b -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0029 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x0005 -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x0025 -> Node 1
[Wed Jul 3 16:39:34 2024] SRAT: PXM 0 -> APIC 0x001b -> Node 0
[Wed Jul 3 16:39:34 2024] SRAT: PXM 1 -> APIC 0x003b -> Node 1
[Wed Jul 3 16:39:34 2024] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0x7fffffff]
[Wed Jul 3 16:39:34 2024] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x107fffffff]
[Wed Jul 3 16:39:34 2024] ACPI: SRAT: Node 1 PXM 1 [mem 0x1080000000-0x207fffffff]
[Wed Jul 3 16:39:34 2024] NUMA: Initialized distance table, cnt=2
[Wed Jul 3 16:39:34 2024] NUMA: Node 0 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x107fffffff] -> [mem 0x00000000-0x107fffffff]
[Wed Jul 3 16:39:34 2024] NODE_DATA(0) allocated [mem 0x107fffb000-0x107fffffff]
[Wed Jul 3 16:39:34 2024] NODE_DATA(1) allocated [mem 0x207fffa000-0x207fffefff]
[Wed Jul 3 16:39:34 2024] Reserving 256MB of memory at 1360MB for crashkernel (System RAM: 130690MB)
[Wed Jul 3 16:39:34 2024] Zone ranges:
[Wed Jul 3 16:39:34 2024] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[Wed Jul 3 16:39:34 2024] DMA32 [mem 0x0000000001000000-0x00000000ffffffff]
[Wed Jul 3 16:39:34 2024] Normal [mem 0x0000000100000000-0x000000207fffffff]
[Wed Jul 3 16:39:34 2024] Device empty
[Wed Jul 3 16:39:34 2024] Movable zone start for each node
[Wed Jul 3 16:39:34 2024] Early memory node ranges
[Wed Jul 3 16:39:34 2024] node 0: [mem 0x0000000000001000-0x000000000009ffff]
[Wed Jul 3 16:39:34 2024] node 0: [mem 0x0000000000100000-0x00000000682fefff]
[Wed Jul 3 16:39:34 2024] node 0: [mem 0x000000006ffff000-0x000000006fffffff]
[Wed Jul 3 16:39:34 2024] node 0: [mem 0x0000000100000000-0x000000107fffffff]
[Wed Jul 3 16:39:34 2024] node 1: [mem 0x0000001080000000-0x000000207fffffff]
[Wed Jul 3 16:39:34 2024] Initmem setup node 0 [mem 0x0000000000001000-0x000000107fffffff]
[Wed Jul 3 16:39:34 2024] Initmem setup node 1 [mem 0x0000001080000000-0x000000207fffffff]
[Wed Jul 3 16:39:34 2024] On node 0, zone DMA: 1 pages in unavailable ranges
[Wed Jul 3 16:39:34 2024] On node 0, zone DMA: 96 pages in unavailable ranges
[Wed Jul 3 16:39:34 2024] On node 0, zone DMA32: 32000 pages in unavailable ranges
[Wed Jul 3 16:39:34 2024] ACPI: PM-Timer IO Port: 0x508
[Wed Jul 3 16:39:34 2024] ACPI: X2APIC_NMI (uid[0xffffffff] high level lint[0x1])
[Wed Jul 3 16:39:34 2024] ACPI: LAPIC_NMI (acpi_id[0xff] high level lint[0x1])
[Wed Jul 3 16:39:34 2024] IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
[Wed Jul 3 16:39:34 2024] IOAPIC[1]: apic_id 9, version 32, address 0xfec01000, GSI 24-31
[Wed Jul 3 16:39:34 2024] IOAPIC[2]: apic_id 10, version 32, address 0xfec08000, GSI 32-39
[Wed Jul 3 16:39:34 2024] IOAPIC[3]: apic_id 11, version 32, address 0xfec10000, GSI 40-47
[Wed Jul 3 16:39:34 2024] IOAPIC[4]: apic_id 12, version 32, address 0xfec18000, GSI 48-55
[Wed Jul 3 16:39:34 2024] IOAPIC[5]: apic_id 15, version 32, address 0xfec20000, GSI 72-79
[Wed Jul 3 16:39:34 2024] IOAPIC[6]: apic_id 16, version 32, address 0xfec28000, GSI 80-87
[Wed Jul 3 16:39:34 2024] IOAPIC[7]: apic_id 17, version 32, address 0xfec30000, GSI 88-95
[Wed Jul 3 16:39:34 2024] IOAPIC[8]: apic_id 18, version 32, address 0xfec38000, GSI 96-103
[Wed Jul 3 16:39:34 2024] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[Wed Jul 3 16:39:34 2024] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[Wed Jul 3 16:39:34 2024] ACPI: Using ACPI (MADT) for SMP configuration information
[Wed Jul 3 16:39:34 2024] ACPI: HPET id: 0x8086a701 base: 0xfed00000
[Wed Jul 3 16:39:34 2024] TSC deadline timer available
[Wed Jul 3 16:39:34 2024] smpboot: Allowing 16 CPUs, 0 hotplug CPUs
[Wed Jul 3 16:39:34 2024] [mem 0x90000000-0xfdffffff] available for PCI devices
[Wed Jul 3 16:39:34 2024] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[Wed Jul 3 16:39:34 2024] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:16 nr_node_ids:2
[Wed Jul 3 16:39:34 2024] percpu: Embedded 54 pages/cpu s183448 r8192 d29544 u262144
[Wed Jul 3 16:39:34 2024] pcpu-alloc: s183448 r8192 d29544 u262144 alloc=1*2097152
[Wed Jul 3 16:39:34 2024] pcpu-alloc: [0] 00 02 04 06 08 10 12 14 [1] 01 03 05 07 09 11 13 15
[Wed Jul 3 16:39:34 2024] Built 2 zonelists, mobility grouping on. Total pages: 32933874
[Wed Jul 3 16:39:34 2024] Policy zone: Normal
[Wed Jul 3 16:39:34 2024] Kernel command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
[Wed Jul 3 16:39:34 2024] audit: disabled (until reboot)
[Wed Jul 3 16:39:34 2024] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[Wed Jul 3 16:39:34 2024] Memory: 131353684K/133827196K available (14347K kernel code, 3475K rwdata, 3540K rodata, 1800K init, 2744K bss, 2473256K reserved, 0K cma-reserved)
[Wed Jul 3 16:39:34 2024] ftrace: allocating 41679 entries in 163 pages
[Wed Jul 3 16:39:34 2024] ftrace: allocated 163 pages with 4 groups
[Wed Jul 3 16:39:34 2024] rcu: Hierarchical RCU implementation.
[Wed Jul 3 16:39:34 2024] rcu: RCU event tracing is enabled.
[Wed Jul 3 16:39:34 2024] rcu: RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=16.
[Wed Jul 3 16:39:34 2024] Rude variant of Tasks RCU enabled.
[Wed Jul 3 16:39:34 2024] Tracing variant of Tasks RCU enabled.
[Wed Jul 3 16:39:34 2024] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[Wed Jul 3 16:39:34 2024] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=16
[Wed Jul 3 16:39:35 2024] NR_IRQS: 33024, nr_irqs: 1912, preallocated irqs: 16
[Wed Jul 3 16:39:35 2024] random: crng init done
[Wed Jul 3 16:39:35 2024] Console: colour EGA 80x25
[Wed Jul 3 16:39:35 2024] printk: console [tty0] enabled
[Wed Jul 3 16:39:36 2024] printk: console [ttyS0] enabled
[Wed Jul 3 16:39:36 2024] mempolicy: Disabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[Wed Jul 3 16:39:36 2024] ACPI: Core revision 20210730
[Wed Jul 3 16:39:36 2024] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 79635855245 ns
[Wed Jul 3 16:39:36 2024] APIC: Switch to symmetric I/O mode setup
[Wed Jul 3 16:39:36 2024] DMAR: Host address width 46
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000d37fc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar0: reg_base_addr d37fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000e0ffc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar1: reg_base_addr e0ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000ee7fc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar2: reg_base_addr ee7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar3: reg_base_addr fbffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000aaffc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar4: reg_base_addr aaffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000b87fc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar5: reg_base_addr b87fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x000000c5ffc000 flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: dmar6: reg_base_addr c5ffc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: DRHD base: 0x0000009d7fc000 flags: 0x1
[Wed Jul 3 16:39:36 2024] DMAR: dmar7: reg_base_addr 9d7fc000 ver 1:0 cap 8d2078c106f0466 ecap f020df
[Wed Jul 3 16:39:36 2024] DMAR: RMRR base: 0x0000006f760000 end: 0x0000006f762fff
[Wed Jul 3 16:39:36 2024] DMAR: ATSR flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR: ATSR flags: 0x0
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 12 under DRHD base 0xc5ffc000 IOMMU 6
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 11 under DRHD base 0xb87fc000 IOMMU 5
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 10 under DRHD base 0xaaffc000 IOMMU 4
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 18 under DRHD base 0xfbffc000 IOMMU 3
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 17 under DRHD base 0xee7fc000 IOMMU 2
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 16 under DRHD base 0xe0ffc000 IOMMU 1
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 15 under DRHD base 0xd37fc000 IOMMU 0
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 8 under DRHD base 0x9d7fc000 IOMMU 7
[Wed Jul 3 16:39:36 2024] DMAR-IR: IOAPIC id 9 under DRHD base 0x9d7fc000 IOMMU 7
[Wed Jul 3 16:39:36 2024] DMAR-IR: HPET id 0 under DRHD base 0x9d7fc000
[Wed Jul 3 16:39:36 2024] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar6 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar5 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar4 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar3 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar2 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar1 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar0 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: IRQ remapping was enabled on dmar7 but we are not in kdump mode
[Wed Jul 3 16:39:36 2024] DMAR-IR: Enabled IRQ remapping in x2apic mode
[Wed Jul 3 16:39:36 2024] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[Wed Jul 3 16:39:36 2024] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x6d8cb0e20f4, max_idle_ns: 881590468932 ns
[Wed Jul 3 16:39:36 2024] Calibrating delay loop (skipped), value calculated using timer frequency.. 7600.00 BogoMIPS (lpj=3800000)
[Wed Jul 3 16:39:36 2024] CPU0: Thermal monitoring enabled (TM1)
[Wed Jul 3 16:39:36 2024] process: using mwait in idle threads
[Wed Jul 3 16:39:36 2024] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[Wed Jul 3 16:39:36 2024] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[Wed Jul 3 16:39:36 2024] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[Wed Jul 3 16:39:36 2024] Spectre V2 : WARNING: Unprivileged eBPF is enabled with eIBRS on, data leaks possible via Spectre v2 BHB attacks!
[Wed Jul 3 16:39:36 2024] Spectre V2 : Spectre BHI mitigation: SW BHB clearing on vm exit
[Wed Jul 3 16:39:36 2024] Spectre V2 : Spectre BHI mitigation: SW BHB clearing on syscall
[Wed Jul 3 16:39:36 2024] Spectre V2 : Mitigation: Enhanced / Automatic IBRS
[Wed Jul 3 16:39:36 2024] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[Wed Jul 3 16:39:36 2024] Spectre V2 : Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT
[Wed Jul 3 16:39:36 2024] RETBleed: Mitigation: Enhanced IBRS
[Wed Jul 3 16:39:36 2024] Spectre V2 : mitigation: Enabling conditional Indirect Branch Prediction Barrier
[Wed Jul 3 16:39:36 2024] Speculative Store Bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp
[Wed Jul 3 16:39:36 2024] TAA: Mitigation: TSX disabled
[Wed Jul 3 16:39:36 2024] MMIO Stale Data: Mitigation: Clear CPU buffers
[Wed Jul 3 16:39:36 2024] GDS: Mitigation: Microcode
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[Wed Jul 3 16:39:36 2024] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[5]: 960, xstate_sizes[5]: 64
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]: 512
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[Wed Jul 3 16:39:36 2024] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]: 8
[Wed Jul 3 16:39:36 2024] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[Wed Jul 3 16:39:36 2024] Freeing SMP alternatives memory: 40K
[Wed Jul 3 16:39:36 2024] pid_max: default: 32768 minimum: 301
[Wed Jul 3 16:39:36 2024] Dentry cache hash table entries: 8388608 (order: 14, 67108864 bytes, vmalloc)
[Wed Jul 3 16:39:36 2024] Inode-cache hash table entries: 4194304 (order: 13, 33554432 bytes, vmalloc)
[Wed Jul 3 16:39:36 2024] Mount-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[Wed Jul 3 16:39:36 2024] Mountpoint-cache hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc)
[Wed Jul 3 16:39:36 2024] smpboot: Estimated ratio of average max frequency by base frequency (times 1024): 1050
[Wed Jul 3 16:39:36 2024] smpboot: CPU0: Intel(R) Xeon(R) Gold 5222 CPU @ 3.80GHz (family: 0x6, model: 0x55, stepping: 0x7)
[Wed Jul 3 16:39:36 2024] Performance Events: PEBS fmt3+, Skylake events, 32-deep LBR, full-width counters, Intel PMU driver.
[Wed Jul 3 16:39:36 2024] ... version: 4
[Wed Jul 3 16:39:36 2024] ... bit width: 48
[Wed Jul 3 16:39:36 2024] ... generic registers: 4
[Wed Jul 3 16:39:36 2024] ... value mask: 0000ffffffffffff
[Wed Jul 3 16:39:36 2024] ... max period: 00007fffffffffff
[Wed Jul 3 16:39:36 2024] ... fixed-purpose events: 3
[Wed Jul 3 16:39:36 2024] ... event mask: 000000070000000f
[Wed Jul 3 16:39:36 2024] signal: max sigframe size: 3632
[Wed Jul 3 16:39:36 2024] rcu: Hierarchical SRCU implementation.
[Wed Jul 3 16:39:36 2024] smp: Bringing up secondary CPUs ...
[Wed Jul 3 16:39:36 2024] x86: Booting SMP configuration:
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #1
[Wed Jul 3 16:39:35 2024] smpboot: CPU 1 Converting physical 0 to logical die 1
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #2
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #3
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #4
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #5
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #6
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #7
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #8
[Wed Jul 3 16:39:36 2024] MMIO Stale Data CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/processor_mmio_stale_data.html for more details.
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #9
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #10
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #11
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #12
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #13
[Wed Jul 3 16:39:36 2024] .... node #0, CPUs: #14
[Wed Jul 3 16:39:36 2024] .... node #1, CPUs: #15
[Wed Jul 3 16:39:36 2024] smp: Brought up 2 nodes, 16 CPUs
[Wed Jul 3 16:39:36 2024] smpboot: Max logical packages: 2
[Wed Jul 3 16:39:36 2024] smpboot: Total of 16 processors activated (121737.31 BogoMIPS)
[Wed Jul 3 16:39:36 2024] devtmpfs: initialized
[Wed Jul 3 16:39:36 2024] x86/mm: Memory block size: 2048MB
[Wed Jul 3 16:39:36 2024] ACPI: PM: Registering ACPI NVS region [mem 0x6ebff000-0x6f9fefff] (14680064 bytes)
[Wed Jul 3 16:39:36 2024] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[Wed Jul 3 16:39:36 2024] futex hash table entries: 4096 (order: 6, 262144 bytes, vmalloc)
[Wed Jul 3 16:39:36 2024] NET: Registered PF_NETLINK/PF_ROUTE protocol family
[Wed Jul 3 16:39:36 2024] thermal_sys: Registered thermal governor 'step_wise'
[Wed Jul 3 16:39:36 2024] thermal_sys: Registered thermal governor 'user_space'
[Wed Jul 3 16:39:36 2024] cpuidle: using governor menu
[Wed Jul 3 16:39:36 2024] Detected 1 PCC Subspaces
[Wed Jul 3 16:39:36 2024] Registering PCC driver as Mailbox controller
[Wed Jul 3 16:39:36 2024] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[Wed Jul 3 16:39:36 2024] ACPI: bus type PCI registered
[Wed Jul 3 16:39:36 2024] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[Wed Jul 3 16:39:36 2024] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
[Wed Jul 3 16:39:36 2024] pmd_set_huge: Cannot satisfy [mem 0x80000000-0x80200000] with a huge-page mapping due to MTRR override.
[Wed Jul 3 16:39:36 2024] PCI: Using configuration type 1 for base access
[Wed Jul 3 16:39:36 2024] PCI: Dell System detected, enabling pci=bfsort.
[Wed Jul 3 16:39:36 2024] kprobes: kprobe jump-optimization is enabled. All kprobes are optimized if possible.
[Wed Jul 3 16:39:36 2024] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[Wed Jul 3 16:39:36 2024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[Wed Jul 3 16:39:36 2024] raid6: avx512x4 gen() 18936 MB/s
[Wed Jul 3 16:39:36 2024] raid6: avx512x4 xor() 4659 MB/s
[Wed Jul 3 16:39:36 2024] raid6: avx512x2 gen() 19202 MB/s
[Wed Jul 3 16:39:36 2024] raid6: avx512x2 xor() 11364 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx512x1 gen() 17572 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx512x1 xor() 10110 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x4 gen() 11415 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x4 xor() 5109 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x2 gen() 13116 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x2 xor() 8345 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x1 gen() 11266 MB/s
[Wed Jul 3 16:39:37 2024] raid6: avx2x1 xor() 7278 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x4 gen() 4324 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x4 xor() 2830 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x2 gen() 4787 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x2 xor() 2879 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x1 gen() 4296 MB/s
[Wed Jul 3 16:39:37 2024] raid6: sse2x1 xor() 2197 MB/s
[Wed Jul 3 16:39:37 2024] raid6: using algorithm avx512x2 gen() 19202 MB/s
[Wed Jul 3 16:39:37 2024] raid6: .... xor() 11364 MB/s, rmw enabled
[Wed Jul 3 16:39:37 2024] raid6: using avx512x2 recovery algorithm
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Module Device)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Processor Device)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(3.0 _SCP Extensions)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Processor Aggregator Device)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Linux-Dell-Video)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[Wed Jul 3 16:39:37 2024] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[Wed Jul 3 16:39:37 2024] ACPI: 5 ACPI AML tables successfully acquired and loaded
[Wed Jul 3 16:39:37 2024] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
[Wed Jul 3 16:39:37 2024] ACPI: Dynamic OEM Table Load:
[Wed Jul 3 16:39:37 2024] ACPI: Interpreter enabled
[Wed Jul 3 16:39:37 2024] ACPI: PM: (supports S0 S5)
[Wed Jul 3 16:39:37 2024] ACPI: Using IOAPIC for interrupt routing
[Wed Jul 3 16:39:37 2024] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[Wed Jul 3 16:39:37 2024] ACPI: Enabled 5 GPEs in block 00 to 7F
[Wed Jul 3 16:39:37 2024] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00-16])
[Wed Jul 3 16:39:37 2024] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:37 2024] acpi PNP0A08:00: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:37 2024] acpi PNP0A08:00: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:37 2024] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:37 2024] PCI host bridge to bus 0000:00
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [io 0x1000-0x3fff window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [mem 0x000c4000-0x000c7fff window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [mem 0xfe010000-0xfe010fff window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [mem 0x90000000-0x9d7fffff window]
[Wed Jul 3 16:39:37 2024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x383fffffffff window]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:00: root bus resource [bus 00-16]
[Wed Jul 3 16:39:38 2024] pci 0000:00:00.0: [8086:2020] type 00 class 0x060000
[Wed Jul 3 16:39:38 2024] pci 0000:00:05.0: [8086:2024] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:00:05.2: [8086:2025] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:00:05.4: [8086:2026] type 00 class 0x080020
[Wed Jul 3 16:39:38 2024] pci 0000:00:05.4: reg 0x10: [mem 0x93020000-0x93020fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:08.0: [8086:2014] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:00:08.1: [8086:2015] type 00 class 0x110100
[Wed Jul 3 16:39:38 2024] pci 0000:00:08.2: [8086:2016] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.0: [8086:a1ec] type 00 class 0xff0000
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: [8086:a1d2] type 00 class 0x010601
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x10: [mem 0x93016000-0x93017fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x14: [mem 0x9301f000-0x9301f0ff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x18: [io 0x2068-0x206f]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x1c: [io 0x2074-0x2077]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x20: [io 0x2040-0x205f]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: reg 0x24: [mem 0x92f80000-0x92ffffff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:11.5: PME# supported from D3hot
[Wed Jul 3 16:39:38 2024] pci 0000:00:14.0: [8086:a1af] type 00 class 0x0c0330
[Wed Jul 3 16:39:38 2024] pci 0000:00:14.0: reg 0x10: [mem 0x93000000-0x9300ffff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:14.0: PME# supported from D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:00:14.2: [8086:a1b1] type 00 class 0x118000
[Wed Jul 3 16:39:38 2024] pci 0000:00:14.2: reg 0x10: [mem 0x9301c000-0x9301cfff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.0: [8086:a1ba] type 00 class 0x078000
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.0: reg 0x10: [mem 0x9301b000-0x9301bfff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.0: PME# supported from D3hot
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.1: [8086:a1bb] type 00 class 0x078000
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.1: reg 0x10: [mem 0x9301a000-0x9301afff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.1: PME# supported from D3hot
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.4: [8086:a1be] type 00 class 0x078000
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.4: reg 0x10: [mem 0x93019000-0x93019fff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:16.4: PME# supported from D3hot
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: [8086:a182] type 00 class 0x010601
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x10: [mem 0x93014000-0x93015fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x14: [mem 0x9301e000-0x9301e0ff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x18: [io 0x2060-0x2067]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x1c: [io 0x2070-0x2073]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x20: [io 0x2020-0x203f]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: reg 0x24: [mem 0x92f00000-0x92f7ffff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:17.0: PME# supported from D3hot
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: [8086:a190] type 01 class 0x060400
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.4: [8086:a194] type 01 class 0x060400
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.4: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.5: [8086:a195] type 01 class 0x060400
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.5: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.0: [8086:a1c1] type 00 class 0x060100
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.2: [8086:a1a1] type 00 class 0x058000
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.2: reg 0x10: [mem 0x93010000-0x93013fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.4: [8086:a1a3] type 00 class 0x0c0500
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.4: reg 0x10: [mem 0x93018000-0x930180ff 64bit]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.4: reg 0x20: [io 0x2000-0x201f]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.5: [8086:a1a4] type 00 class 0x0c8000
[Wed Jul 3 16:39:38 2024] pci 0000:00:1f.5: reg 0x10: [mem 0xfe010000-0xfe010fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: PCI bridge to [bus 01]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: bridge window [io 0x3000-0x3fff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: bridge window [mem 0x92a00000-0x92dfffff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.0: bridge window [mem 0x380000000000-0x3800001fffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:02:00.0: [1556:be00] type 01 class 0x060400
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.4: PCI bridge to [bus 02-03]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.4: bridge window [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.4: bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:03: extended config space not accessible
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: [102b:0536] type 00 class 0x030000
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: reg 0x10: [mem 0x91000000-0x91ffffff pref]
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: reg 0x14: [mem 0x92808000-0x9280bfff]
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: reg 0x18: [mem 0x92000000-0x927fffff]
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[Wed Jul 3 16:39:38 2024] pci 0000:03:00.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[Wed Jul 3 16:39:38 2024] pci 0000:02:00.0: PCI bridge to [bus 03]
[Wed Jul 3 16:39:38 2024] pci 0000:02:00.0: bridge window [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:38 2024] pci 0000:02:00.0: bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: [14e4:165f] type 00 class 0x020000
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: reg 0x10: [mem 0x92e30000-0x92e3ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: reg 0x18: [mem 0x92e40000-0x92e4ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: reg 0x20: [mem 0x92e50000-0x92e5ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: reg 0x30: [mem 0xfffc0000-0xffffffff pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.0: 4.000 Gb/s available PCIe bandwidth, limited by 5.0 GT/s PCIe x1 link at 0000:00:1c.5 (capable of 8.000 Gb/s with 5.0 GT/s PCIe x2 link)
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: [14e4:165f] type 00 class 0x020000
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: reg 0x10: [mem 0x92e00000-0x92e0ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: reg 0x18: [mem 0x92e10000-0x92e1ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: reg 0x20: [mem 0x92e20000-0x92e2ffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: reg 0x30: [mem 0xfffc0000-0xffffffff pref]
[Wed Jul 3 16:39:38 2024] pci 0000:04:00.1: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.5: PCI bridge to [bus 04]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.5: bridge window [mem 0x90000000-0x900fffff]
[Wed Jul 3 16:39:38 2024] pci 0000:00:1c.5: bridge window [mem 0x92e00000-0x92efffff 64bit pref]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:00: on NUMA node 0
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKA configured for IRQ 11
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKA disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKB configured for IRQ 6
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKB disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKC configured for IRQ 5
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKC disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKD configured for IRQ 11
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKD disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKE configured for IRQ 11
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKE disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKF configured for IRQ 6
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKF disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKG configured for IRQ 5
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKG disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKH configured for IRQ 11
[Wed Jul 3 16:39:38 2024] ACPI: PCI: Interrupt link LNKH disabled
[Wed Jul 3 16:39:38 2024] ACPI: PCI Root Bridge [PC01] (domain 0000 [bus 17-39])
[Wed Jul 3 16:39:38 2024] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:38 2024] acpi PNP0A08:01: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:38 2024] acpi PNP0A08:01: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:38 2024] acpi PNP0A08:01: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:38 2024] PCI host bridge to bus 0000:17
[Wed Jul 3 16:39:38 2024] pci_bus 0000:17: root bus resource [io 0x4000-0x5fff window]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:17: root bus resource [mem 0x9d800000-0xaaffffff window]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:17: root bus resource [mem 0x384000000000-0x387fffffffff window]
[Wed Jul 3 16:39:38 2024] pci_bus 0000:17: root bus resource [bus 17-39]
[Wed Jul 3 16:39:38 2024] pci 0000:17:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:38 2024] pci 0000:17:05.4: reg 0x10: [mem 0x9d800000-0x9d800fff]
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:08.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:09.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0a.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0b.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0b.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0b.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0b.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0e.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:38 2024] pci 0000:17:0e.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0e.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:0f.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:10.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:11.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:11.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:11.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:11.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1d.0: [8086:2054] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1d.1: [8086:2055] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1d.2: [8086:2056] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1d.3: [8086:2057] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.0: [8086:2080] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.1: [8086:2081] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.2: [8086:2082] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.3: [8086:2083] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.4: [8086:2084] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.5: [8086:2085] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:17:1e.6: [8086:2086] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci_bus 0000:17: on NUMA node 0
[Wed Jul 3 16:39:39 2024] ACPI: PCI Root Bridge [PC02] (domain 0000 [bus 3a-5c])
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:02: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:02: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:02: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:39 2024] PCI host bridge to bus 0000:3a
[Wed Jul 3 16:39:39 2024] pci_bus 0000:3a: root bus resource [io 0x6000-0x7fff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:3a: root bus resource [mem 0xab000000-0xb87fffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:3a: root bus resource [mem 0x388000000000-0x38bfffffffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:3a: root bus resource [bus 3a-5c]
[Wed Jul 3 16:39:39 2024] pci 0000:3a:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:39 2024] pci 0000:3a:05.4: reg 0x10: [mem 0xab000000-0xab000fff]
[Wed Jul 3 16:39:39 2024] pci 0000:3a:08.0: [8086:2066] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:09.0: [8086:2066] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.0: [8086:2040] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.1: [8086:2041] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.2: [8086:2042] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.3: [8086:2043] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.4: [8086:2044] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.5: [8086:2045] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.6: [8086:2046] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0a.7: [8086:2047] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0b.0: [8086:2048] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0b.1: [8086:2049] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0b.2: [8086:204a] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0b.3: [8086:204b] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.0: [8086:2040] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.1: [8086:2041] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.2: [8086:2042] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.3: [8086:2043] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.4: [8086:2044] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.5: [8086:2045] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.6: [8086:2046] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0c.7: [8086:2047] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0d.0: [8086:2048] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0d.1: [8086:2049] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0d.2: [8086:204a] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:3a:0d.3: [8086:204b] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci_bus 0000:3a: on NUMA node 0
[Wed Jul 3 16:39:39 2024] ACPI: PCI Root Bridge [PC03] (domain 0000 [bus 5d-7f])
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:03: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:03: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:03: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:39 2024] PCI host bridge to bus 0000:5d
[Wed Jul 3 16:39:39 2024] pci_bus 0000:5d: root bus resource [io 0x8000-0x9fff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:5d: root bus resource [mem 0xb8800000-0xc5ffffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:5d: root bus resource [mem 0x38c000000000-0x38ffffffffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:5d: root bus resource [bus 5d-7f]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:00.0: [8086:2030] type 01 class 0x060400
[Wed Jul 3 16:39:39 2024] pci 0000:5d:00.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:39 2024] pci 0000:5d:02.0: [8086:2032] type 01 class 0x060400
[Wed Jul 3 16:39:39 2024] pci 0000:5d:02.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:39 2024] pci 0000:5d:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:39 2024] pci 0000:5d:05.4: reg 0x10: [mem 0xbb300000-0xbb300fff]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:0e.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:0e.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:0f.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:0f.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:10.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:10.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:12.0: [8086:204c] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:12.1: [8086:204d] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:12.2: [8086:204e] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:12.4: [8086:204c] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:12.5: [8086:204d] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:15.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:15.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:16.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:16.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:16.4: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:16.5: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5d:17.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:5d:17.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: [1000:0097] type 00 class 0x010700
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x10: [io 0x8000-0x80ff]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x14: [mem 0xbb200000-0xbb20ffff 64bit]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x1c: [mem 0xbb100000-0xbb1fffff 64bit]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x30: [mem 0xfff00000-0xffffffff pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: supports D1 D2
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x174: [mem 0x00000000-0x0000ffff 64bit]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: VF(n) BAR0 space: [mem 0x00000000-0x000fffff 64bit] (contains BAR0 for 16 VFs)
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: reg 0x17c: [mem 0x00000000-0x000fffff 64bit]
[Wed Jul 3 16:39:39 2024] pci 0000:5e:00.0: VF(n) BAR2 space: [mem 0x00000000-0x00ffffff 64bit] (contains BAR2 for 16 VFs)
[Wed Jul 3 16:39:39 2024] pci 0000:5d:00.0: PCI bridge to [bus 5e]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:00.0: bridge window [io 0x8000-0x8fff]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:00.0: bridge window [mem 0xbb100000-0xbb2fffff]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.0: [8086:1572] type 00 class 0x020000
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.0: reg 0x10: [mem 0xba000000-0xbaffffff 64bit pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.0: reg 0x1c: [mem 0xbb008000-0xbb00ffff 64bit pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.1: [8086:1572] type 00 class 0x020000
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.1: reg 0x10: [mem 0xb9000000-0xb9ffffff 64bit pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.1: reg 0x1c: [mem 0xbb000000-0xbb007fff 64bit pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.1: reg 0x30: [mem 0xfff80000-0xffffffff pref]
[Wed Jul 3 16:39:39 2024] pci 0000:5f:00.1: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:39 2024] pci 0000:5d:02.0: PCI bridge to [bus 5f]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:02.0: bridge window [mem 0xb8800000-0xb88fffff]
[Wed Jul 3 16:39:39 2024] pci 0000:5d:02.0: bridge window [mem 0xb9000000-0xbb0fffff 64bit pref]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:5d: on NUMA node 0
[Wed Jul 3 16:39:39 2024] ACPI: PCI Root Bridge [PC06] (domain 0000 [bus 80-84])
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:06: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:06: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:06: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:06: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:39 2024] acpi PNP0A08:06: host bridge window [io 0x0000 window] (ignored, not CPU addressable)
[Wed Jul 3 16:39:39 2024] PCI host bridge to bus 0000:80
[Wed Jul 3 16:39:39 2024] pci_bus 0000:80: root bus resource [mem 0xc6000000-0xd37fffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:80: root bus resource [mem 0x390000000000-0x393fffffffff window]
[Wed Jul 3 16:39:39 2024] pci_bus 0000:80: root bus resource [bus 80-84]
[Wed Jul 3 16:39:39 2024] pci 0000:80:05.0: [8086:2024] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:80:05.2: [8086:2025] type 00 class 0x088000
[Wed Jul 3 16:39:39 2024] pci 0000:80:05.4: [8086:2026] type 00 class 0x080020
[Wed Jul 3 16:39:40 2024] pci 0000:80:05.4: reg 0x10: [mem 0xc6000000-0xc6000fff]
[Wed Jul 3 16:39:40 2024] pci 0000:80:08.0: [8086:2014] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:80:08.1: [8086:2015] type 00 class 0x110100
[Wed Jul 3 16:39:40 2024] pci 0000:80:08.2: [8086:2016] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci_bus 0000:80: on NUMA node 1
[Wed Jul 3 16:39:40 2024] ACPI: PCI Root Bridge [PC07] (domain 0000 [bus 85-ad])
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:07: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:07: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:07: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:07: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:40 2024] PCI host bridge to bus 0000:85
[Wed Jul 3 16:39:40 2024] pci_bus 0000:85: root bus resource [io 0xa000-0xbfff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:85: root bus resource [mem 0xd3800000-0xe0ffffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:85: root bus resource [mem 0x394000000000-0x397fffffffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:85: root bus resource [bus 85-ad]
[Wed Jul 3 16:39:40 2024] pci 0000:85:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:40 2024] pci 0000:85:05.4: reg 0x10: [mem 0xd3800000-0xd3800fff]
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:08.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:09.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.4: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.5: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.6: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0a.7: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0b.0: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0b.1: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0b.2: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0b.3: [8086:208d] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0e.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:0f.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.4: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.5: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.6: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:10.7: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:11.0: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:11.1: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:11.2: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:11.3: [8086:208e] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1d.0: [8086:2054] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1d.1: [8086:2055] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1d.2: [8086:2056] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1d.3: [8086:2057] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.0: [8086:2080] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.1: [8086:2081] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.2: [8086:2082] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.3: [8086:2083] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.4: [8086:2084] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.5: [8086:2085] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:85:1e.6: [8086:2086] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci_bus 0000:85: on NUMA node 1
[Wed Jul 3 16:39:40 2024] ACPI: PCI Root Bridge [PC08] (domain 0000 [bus ae-d6])
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:08: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:08: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:08: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:08: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:40 2024] PCI host bridge to bus 0000:ae
[Wed Jul 3 16:39:40 2024] pci_bus 0000:ae: root bus resource [io 0xc000-0xdfff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:ae: root bus resource [mem 0xe1000000-0xee7fffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:ae: root bus resource [mem 0x398000000000-0x39bfffffffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:ae: root bus resource [bus ae-d6]
[Wed Jul 3 16:39:40 2024] pci 0000:ae:00.0: [8086:2030] type 01 class 0x060400
[Wed Jul 3 16:39:40 2024] pci 0000:ae:00.0: PME# supported from D0 D3hot D3cold
[Wed Jul 3 16:39:40 2024] pci 0000:ae:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:40 2024] pci 0000:ae:05.4: reg 0x10: [mem 0xe1100000-0xe1100fff]
[Wed Jul 3 16:39:40 2024] pci 0000:ae:08.0: [8086:2066] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:09.0: [8086:2066] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.0: [8086:2040] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.1: [8086:2041] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.2: [8086:2042] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.3: [8086:2043] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.4: [8086:2044] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.5: [8086:2045] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.6: [8086:2046] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0a.7: [8086:2047] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0b.0: [8086:2048] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0b.1: [8086:2049] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0b.2: [8086:204a] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0b.3: [8086:204b] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.0: [8086:2040] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.1: [8086:2041] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.2: [8086:2042] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.3: [8086:2043] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.4: [8086:2044] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.5: [8086:2045] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.6: [8086:2046] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0c.7: [8086:2047] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0d.0: [8086:2048] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0d.1: [8086:2049] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0d.2: [8086:204a] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:ae:0d.3: [8086:204b] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: [9005:028f] type 00 class 0x010700
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: reg 0x10: [mem 0xe1000000-0xe1007fff 64bit]
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: reg 0x20: [io 0xc000-0xc0ff]
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: reg 0x30: [mem 0xffe00000-0xffffffff pref]
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: supports D1
[Wed Jul 3 16:39:40 2024] pci 0000:af:00.0: PME# supported from D0 D1 D3hot
[Wed Jul 3 16:39:40 2024] pci 0000:ae:00.0: PCI bridge to [bus af]
[Wed Jul 3 16:39:40 2024] pci 0000:ae:00.0: bridge window [io 0xc000-0xcfff]
[Wed Jul 3 16:39:40 2024] pci 0000:ae:00.0: bridge window [mem 0xe1000000-0xe10fffff]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:ae: on NUMA node 1
[Wed Jul 3 16:39:40 2024] ACPI: PCI Root Bridge [PC09] (domain 0000 [bus d7-ff])
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:09: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:09: _OSC: platform does not support [LTR]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:09: _OSC: OS now controls [PME PCIeCapability]
[Wed Jul 3 16:39:40 2024] acpi PNP0A08:09: FADT indicates ASPM is unsupported, using BIOS configuration
[Wed Jul 3 16:39:40 2024] PCI host bridge to bus 0000:d7
[Wed Jul 3 16:39:40 2024] pci_bus 0000:d7: root bus resource [io 0xe000-0xffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:d7: root bus resource [mem 0xee800000-0xfbffffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:d7: root bus resource [mem 0x39c000000000-0x39ffffffffff window]
[Wed Jul 3 16:39:40 2024] pci_bus 0000:d7: root bus resource [bus d7-ff]
[Wed Jul 3 16:39:40 2024] pci 0000:d7:05.0: [8086:2034] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:d7:05.2: [8086:2035] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:d7:05.4: [8086:2036] type 00 class 0x080020
[Wed Jul 3 16:39:40 2024] pci 0000:d7:05.4: reg 0x10: [mem 0xee800000-0xee800fff]
[Wed Jul 3 16:39:40 2024] pci 0000:d7:0e.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:40 2024] pci 0000:d7:0e.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:d7:0f.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:40 2024] pci 0000:d7:0f.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:40 2024] pci 0000:d7:10.0: [8086:2058] type 00 class 0x110100
[Wed Jul 3 16:39:40 2024] pci 0000:d7:10.1: [8086:2059] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:12.0: [8086:204c] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:12.1: [8086:204d] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:12.2: [8086:204e] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:12.4: [8086:204c] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:12.5: [8086:204d] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:15.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:15.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:16.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:16.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:16.4: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:16.5: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci 0000:d7:17.0: [8086:2018] type 00 class 0x088000
[Wed Jul 3 16:39:41 2024] pci 0000:d7:17.1: [8086:2088] type 00 class 0x110100
[Wed Jul 3 16:39:41 2024] pci_bus 0000:d7: on NUMA node 1
[Wed Jul 3 16:39:41 2024] iommu: Default domain type: Translated
[Wed Jul 3 16:39:41 2024] iommu: DMA domain TLB invalidation policy: lazy mode
[Wed Jul 3 16:39:41 2024] pci 0000:03:00.0: vgaarb: setting as boot VGA device
[Wed Jul 3 16:39:41 2024] pci 0000:03:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[Wed Jul 3 16:39:41 2024] pci 0000:03:00.0: vgaarb: bridge control possible
[Wed Jul 3 16:39:41 2024] vgaarb: loaded
[Wed Jul 3 16:39:41 2024] SCSI subsystem initialized
[Wed Jul 3 16:39:41 2024] libata version 3.00 loaded.
[Wed Jul 3 16:39:41 2024] ACPI: bus type USB registered
[Wed Jul 3 16:39:41 2024] usbcore: registered new interface driver usbfs
[Wed Jul 3 16:39:41 2024] usbcore: registered new interface driver hub
[Wed Jul 3 16:39:41 2024] usbcore: registered new device driver usb
[Wed Jul 3 16:39:41 2024] mc: Linux media interface: v0.10
[Wed Jul 3 16:39:41 2024] videodev: Linux video capture interface: v2.00
[Wed Jul 3 16:39:41 2024] pps_core: LinuxPPS API ver. 1 registered
[Wed Jul 3 16:39:41 2024] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[Wed Jul 3 16:39:41 2024] PTP clock support registered
[Wed Jul 3 16:39:41 2024] Registered efivars operations
[Wed Jul 3 16:39:41 2024] PCI: Using ACPI for IRQ routing
[Wed Jul 3 16:39:41 2024] PCI: pci_cache_line_size set to 64 bytes
[Wed Jul 3 16:39:41 2024] e820: reserve RAM buffer [mem 0x682ff000-0x6bffffff]
[Wed Jul 3 16:39:41 2024] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[Wed Jul 3 16:39:41 2024] hpet0: 8 comparators, 64-bit 24.000000 MHz counter
[Wed Jul 3 16:39:41 2024] clocksource: Switched to clocksource tsc-early
[Wed Jul 3 16:39:41 2024] VFS: Disk quotas dquot_6.6.0
[Wed Jul 3 16:39:41 2024] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[Wed Jul 3 16:39:41 2024] FS-Cache: Loaded
[Wed Jul 3 16:39:41 2024] CacheFiles: Loaded
[Wed Jul 3 16:39:41 2024] pnp: PnP ACPI init
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0500-0x05fe] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0400-0x047f] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0600-0x061f] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0ca0-0x0ca5] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0880-0x0883] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [io 0x0800-0x081f] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfed1c000-0xfed3ffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfed45000-0xfed8bfff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xff000000-0xffffffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfee00000-0xfeefffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfed12000-0xfed1200f] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfed12010-0xfed1201f] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:01: [mem 0xfed1b000-0xfed1bfff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfd000000-0xfdabffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfdad0000-0xfdadffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfdb00000-0xfdffffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfe000000-0xfe00ffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfe011000-0xfe01ffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfe036000-0xfe03bfff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfe03d000-0xfe3fffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:04: [mem 0xfe410000-0xfe7fffff] has been reserved
[Wed Jul 3 16:39:41 2024] system 00:05: [io 0x1000-0x10fe] has been reserved
[Wed Jul 3 16:39:41 2024] pnp: PnP ACPI: found 6 devices
[Wed Jul 3 16:39:41 2024] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[Wed Jul 3 16:39:41 2024] NET: Registered PF_INET protocol family
[Wed Jul 3 16:39:41 2024] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] tcp_listen_portaddr_hash hash table entries: 65536 (order: 8, 1048576 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] TCP established hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] TCP: Hash tables configured (established 524288 bind 65536)
[Wed Jul 3 16:39:41 2024] UDP hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] UDP-Lite hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc)
[Wed Jul 3 16:39:41 2024] pci 0000:04:00.0: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:04:00.1: can't claim BAR 6 [mem 0xfffc0000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: can't claim BAR 6 [mem 0xfff00000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:5f:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:5f:00.1: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:af:00.0: can't claim BAR 6 [mem 0xffe00000-0xffffffff pref]: no compatible bridge window
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.0: PCI bridge to [bus 01]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.0: bridge window [io 0x3000-0x3fff]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.0: bridge window [mem 0x92a00000-0x92dfffff]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.0: bridge window [mem 0x380000000000-0x3800001fffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci 0000:02:00.0: PCI bridge to [bus 03]
[Wed Jul 3 16:39:41 2024] pci 0000:02:00.0: bridge window [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:41 2024] pci 0000:02:00.0: bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.4: PCI bridge to [bus 02-03]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.4: bridge window [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.4: bridge window [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci 0000:04:00.0: BAR 6: assigned [mem 0x90000000-0x9003ffff pref]
[Wed Jul 3 16:39:41 2024] pci 0000:04:00.1: BAR 6: assigned [mem 0x90040000-0x9007ffff pref]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.5: PCI bridge to [bus 04]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.5: bridge window [mem 0x90000000-0x900fffff]
[Wed Jul 3 16:39:41 2024] pci 0000:00:1c.5: bridge window [mem 0x92e00000-0x92efffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 5 [io 0x1000-0x3fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 7 [mem 0x000c4000-0x000c7fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 8 [mem 0xfe010000-0xfe010fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 9 [mem 0x90000000-0x9d7fffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:00: resource 10 [mem 0x380000000000-0x383fffffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:01: resource 0 [io 0x3000-0x3fff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:01: resource 1 [mem 0x92a00000-0x92dfffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:01: resource 2 [mem 0x380000000000-0x3800001fffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:02: resource 1 [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:02: resource 2 [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:03: resource 1 [mem 0x92000000-0x928fffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:03: resource 2 [mem 0x91000000-0x91ffffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:04: resource 1 [mem 0x90000000-0x900fffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:04: resource 2 [mem 0x92e00000-0x92efffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:17: resource 4 [io 0x4000-0x5fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:17: resource 5 [mem 0x9d800000-0xaaffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:17: resource 6 [mem 0x384000000000-0x387fffffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:3a: resource 4 [io 0x6000-0x7fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:3a: resource 5 [mem 0xab000000-0xb87fffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:3a: resource 6 [mem 0x388000000000-0x38bfffffffff window]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 9: no space for [mem size 0x01000000 64bit]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 9: failed to assign [mem size 0x01000000 64bit]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 7: no space for [mem size 0x00100000 64bit]
[Wed Jul 3 16:39:41 2024] pci 0000:5e:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:00.0: PCI bridge to [bus 5e]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:00.0: bridge window [io 0x8000-0x8fff]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:00.0: bridge window [mem 0xbb100000-0xbb2fffff]
[Wed Jul 3 16:39:41 2024] pci 0000:5f:00.0: BAR 6: assigned [mem 0xb8800000-0xb887ffff pref]
[Wed Jul 3 16:39:41 2024] pci 0000:5f:00.1: BAR 6: assigned [mem 0xb8880000-0xb88fffff pref]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:02.0: PCI bridge to [bus 5f]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:02.0: bridge window [mem 0xb8800000-0xb88fffff]
[Wed Jul 3 16:39:41 2024] pci 0000:5d:02.0: bridge window [mem 0xb9000000-0xbb0fffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5d: Some PCI device resources are unassigned, try booting with pci=realloc
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5d: resource 4 [io 0x8000-0x9fff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5d: resource 5 [mem 0xb8800000-0xc5ffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5d: resource 6 [mem 0x38c000000000-0x38ffffffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5e: resource 0 [io 0x8000-0x8fff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5e: resource 1 [mem 0xbb100000-0xbb2fffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5f: resource 1 [mem 0xb8800000-0xb88fffff]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:5f: resource 2 [mem 0xb9000000-0xbb0fffff 64bit pref]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:80: resource 4 [mem 0xc6000000-0xd37fffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:80: resource 5 [mem 0x390000000000-0x393fffffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:85: resource 4 [io 0xa000-0xbfff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:85: resource 5 [mem 0xd3800000-0xe0ffffff window]
[Wed Jul 3 16:39:41 2024] pci_bus 0000:85: resource 6 [mem 0x394000000000-0x397fffffffff window]
[Wed Jul 3 16:39:41 2024] pci 0000:af:00.0: BAR 6: no space for [mem size 0x00200000 pref]
[Wed Jul 3 16:39:42 2024] pci 0000:af:00.0: BAR 6: failed to assign [mem size 0x00200000 pref]
[Wed Jul 3 16:39:42 2024] pci 0000:ae:00.0: PCI bridge to [bus af]
[Wed Jul 3 16:39:42 2024] pci 0000:ae:00.0: bridge window [io 0xc000-0xcfff]
[Wed Jul 3 16:39:42 2024] pci 0000:ae:00.0: bridge window [mem 0xe1000000-0xe10fffff]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:ae: resource 4 [io 0xc000-0xdfff window]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:ae: resource 5 [mem 0xe1000000-0xee7fffff window]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:ae: resource 6 [mem 0x398000000000-0x39bfffffffff window]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:af: resource 0 [io 0xc000-0xcfff]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:af: resource 1 [mem 0xe1000000-0xe10fffff]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:d7: resource 4 [io 0xe000-0xffff window]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:d7: resource 5 [mem 0xee800000-0xfbffffff window]
[Wed Jul 3 16:39:42 2024] pci_bus 0000:d7: resource 6 [mem 0x39c000000000-0x39ffffffffff window]
[Wed Jul 3 16:39:42 2024] pci 0000:17:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] pci 0000:3a:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] pci 0000:5d:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] pci 0000:85:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] pci 0000:ae:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] pci 0000:d7:05.0: disabled boot interrupts on device [8086:2034]
[Wed Jul 3 16:39:42 2024] PCI: CLS 0 bytes, default 64
[Wed Jul 3 16:39:42 2024] Trying to unpack rootfs image as initramfs...
[Wed Jul 3 16:39:42 2024] DMAR: No SATC found
[Wed Jul 3 16:39:42 2024] DMAR: dmar6: Using Queued invalidation
[Wed Jul 3 16:39:42 2024] DMAR: dmar2: Using Queued invalidation
[Wed Jul 3 16:39:42 2024] DMAR: dmar7: Using Queued invalidation
[Wed Jul 3 16:39:42 2024] pci 0000:00:00.0: Adding to iommu group 0
[Wed Jul 3 16:39:42 2024] pci 0000:00:05.0: Adding to iommu group 1
[Wed Jul 3 16:39:42 2024] pci 0000:00:05.2: Adding to iommu group 2
[Wed Jul 3 16:39:42 2024] pci 0000:00:05.4: Adding to iommu group 3
[Wed Jul 3 16:39:42 2024] pci 0000:00:08.0: Adding to iommu group 4
[Wed Jul 3 16:39:42 2024] pci 0000:00:08.1: Adding to iommu group 5
[Wed Jul 3 16:39:42 2024] pci 0000:00:08.2: Adding to iommu group 6
[Wed Jul 3 16:39:42 2024] pci 0000:00:11.0: Adding to iommu group 7
[Wed Jul 3 16:39:42 2024] pci 0000:00:11.5: Adding to iommu group 7
[Wed Jul 3 16:39:42 2024] pci 0000:00:14.0: Adding to iommu group 8
[Wed Jul 3 16:39:42 2024] pci 0000:00:14.2: Adding to iommu group 8
[Wed Jul 3 16:39:42 2024] pci 0000:00:16.0: Adding to iommu group 9
[Wed Jul 3 16:39:42 2024] pci 0000:00:16.1: Adding to iommu group 9
[Wed Jul 3 16:39:42 2024] pci 0000:00:16.4: Adding to iommu group 9
[Wed Jul 3 16:39:42 2024] pci 0000:00:17.0: Adding to iommu group 10
[Wed Jul 3 16:39:42 2024] pci 0000:00:1c.0: Adding to iommu group 11
[Wed Jul 3 16:39:42 2024] pci 0000:00:1c.4: Adding to iommu group 12
[Wed Jul 3 16:39:42 2024] pci 0000:00:1c.5: Adding to iommu group 13
[Wed Jul 3 16:39:42 2024] pci 0000:00:1f.0: Adding to iommu group 14
[Wed Jul 3 16:39:42 2024] pci 0000:00:1f.2: Adding to iommu group 14
[Wed Jul 3 16:39:42 2024] pci 0000:00:1f.4: Adding to iommu group 14
[Wed Jul 3 16:39:42 2024] pci 0000:00:1f.5: Adding to iommu group 14
[Wed Jul 3 16:39:42 2024] pci 0000:02:00.0: Adding to iommu group 15
[Wed Jul 3 16:39:42 2024] pci 0000:03:00.0: Adding to iommu group 15
[Wed Jul 3 16:39:42 2024] pci 0000:04:00.0: Adding to iommu group 16
[Wed Jul 3 16:39:42 2024] pci 0000:04:00.1: Adding to iommu group 16
[Wed Jul 3 16:39:42 2024] pci 0000:17:05.0: Adding to iommu group 17
[Wed Jul 3 16:39:42 2024] pci 0000:17:05.2: Adding to iommu group 18
[Wed Jul 3 16:39:42 2024] pci 0000:17:05.4: Adding to iommu group 19
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.0: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.1: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.2: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.3: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.4: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.5: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.6: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:08.7: Adding to iommu group 20
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.0: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.1: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.2: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.3: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.4: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.5: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.6: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:09.7: Adding to iommu group 21
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.0: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.1: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.2: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.3: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.4: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.5: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.6: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] pci 0000:17:0a.7: Adding to iommu group 22
[Wed Jul 3 16:39:42 2024] Freeing initrd memory: 14688K
[Wed Jul 3 16:39:42 2024] pci 0000:17:0b.0: Adding to iommu group 23
[Wed Jul 3 16:39:42 2024] pci 0000:17:0b.1: Adding to iommu group 23
[Wed Jul 3 16:39:42 2024] pci 0000:17:0b.2: Adding to iommu group 23
[Wed Jul 3 16:39:42 2024] pci 0000:17:0b.3: Adding to iommu group 23
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.0: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.1: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.2: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.3: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.4: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.5: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.6: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0e.7: Adding to iommu group 24
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.0: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.1: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.2: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.3: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.4: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.5: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.6: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:0f.7: Adding to iommu group 25
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.0: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.1: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.2: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.3: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.4: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.5: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.6: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:10.7: Adding to iommu group 26
[Wed Jul 3 16:39:42 2024] pci 0000:17:11.0: Adding to iommu group 27
[Wed Jul 3 16:39:42 2024] pci 0000:17:11.1: Adding to iommu group 27
[Wed Jul 3 16:39:42 2024] pci 0000:17:11.2: Adding to iommu group 27
[Wed Jul 3 16:39:42 2024] pci 0000:17:11.3: Adding to iommu group 27
[Wed Jul 3 16:39:42 2024] pci 0000:17:1d.0: Adding to iommu group 28
[Wed Jul 3 16:39:42 2024] pci 0000:17:1d.1: Adding to iommu group 28
[Wed Jul 3 16:39:42 2024] pci 0000:17:1d.2: Adding to iommu group 28
[Wed Jul 3 16:39:42 2024] pci 0000:17:1d.3: Adding to iommu group 28
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.0: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.1: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.2: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.3: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.4: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.5: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:17:1e.6: Adding to iommu group 29
[Wed Jul 3 16:39:42 2024] pci 0000:3a:05.0: Adding to iommu group 30
[Wed Jul 3 16:39:42 2024] pci 0000:3a:05.2: Adding to iommu group 31
[Wed Jul 3 16:39:42 2024] pci 0000:3a:05.4: Adding to iommu group 32
[Wed Jul 3 16:39:42 2024] pci 0000:3a:08.0: Adding to iommu group 33
[Wed Jul 3 16:39:42 2024] pci 0000:3a:09.0: Adding to iommu group 34
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.0: Adding to iommu group 35
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.1: Adding to iommu group 36
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.2: Adding to iommu group 37
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.3: Adding to iommu group 38
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.4: Adding to iommu group 39
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.5: Adding to iommu group 40
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.6: Adding to iommu group 41
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0a.7: Adding to iommu group 42
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0b.0: Adding to iommu group 43
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0b.1: Adding to iommu group 44
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0b.2: Adding to iommu group 45
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0b.3: Adding to iommu group 46
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.0: Adding to iommu group 47
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.1: Adding to iommu group 48
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.2: Adding to iommu group 49
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.3: Adding to iommu group 50
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.4: Adding to iommu group 51
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.5: Adding to iommu group 52
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.6: Adding to iommu group 53
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0c.7: Adding to iommu group 54
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0d.0: Adding to iommu group 55
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0d.1: Adding to iommu group 56
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0d.2: Adding to iommu group 57
[Wed Jul 3 16:39:42 2024] pci 0000:3a:0d.3: Adding to iommu group 58
[Wed Jul 3 16:39:42 2024] pci 0000:5d:00.0: Adding to iommu group 59
[Wed Jul 3 16:39:42 2024] pci 0000:5d:02.0: Adding to iommu group 60
[Wed Jul 3 16:39:42 2024] pci 0000:5d:05.0: Adding to iommu group 61
[Wed Jul 3 16:39:42 2024] pci 0000:5d:05.2: Adding to iommu group 62
[Wed Jul 3 16:39:42 2024] pci 0000:5d:05.4: Adding to iommu group 63
[Wed Jul 3 16:39:42 2024] pci 0000:5d:0e.0: Adding to iommu group 64
[Wed Jul 3 16:39:42 2024] pci 0000:5d:0e.1: Adding to iommu group 65
[Wed Jul 3 16:39:42 2024] pci 0000:5d:0f.0: Adding to iommu group 66
[Wed Jul 3 16:39:42 2024] pci 0000:5d:0f.1: Adding to iommu group 67
[Wed Jul 3 16:39:42 2024] pci 0000:5d:10.0: Adding to iommu group 68
[Wed Jul 3 16:39:42 2024] pci 0000:5d:10.1: Adding to iommu group 68
[Wed Jul 3 16:39:42 2024] pci 0000:5d:12.0: Adding to iommu group 69
[Wed Jul 3 16:39:42 2024] pci 0000:5d:12.1: Adding to iommu group 70
[Wed Jul 3 16:39:42 2024] pci 0000:5d:12.2: Adding to iommu group 70
[Wed Jul 3 16:39:42 2024] pci 0000:5d:12.4: Adding to iommu group 71
[Wed Jul 3 16:39:42 2024] pci 0000:5d:12.5: Adding to iommu group 70
[Wed Jul 3 16:39:42 2024] pci 0000:5d:15.0: Adding to iommu group 72
[Wed Jul 3 16:39:42 2024] pci 0000:5d:15.1: Adding to iommu group 72
[Wed Jul 3 16:39:42 2024] pci 0000:5d:16.0: Adding to iommu group 73
[Wed Jul 3 16:39:42 2024] pci 0000:5d:16.1: Adding to iommu group 73
[Wed Jul 3 16:39:42 2024] pci 0000:5d:16.4: Adding to iommu group 73
[Wed Jul 3 16:39:42 2024] pci 0000:5d:16.5: Adding to iommu group 73
[Wed Jul 3 16:39:42 2024] pci 0000:5d:17.0: Adding to iommu group 74
[Wed Jul 3 16:39:42 2024] pci 0000:5d:17.1: Adding to iommu group 74
[Wed Jul 3 16:39:42 2024] pci 0000:5e:00.0: Adding to iommu group 75
[Wed Jul 3 16:39:42 2024] pci 0000:5f:00.0: Adding to iommu group 76
[Wed Jul 3 16:39:42 2024] pci 0000:5f:00.1: Adding to iommu group 77
[Wed Jul 3 16:39:42 2024] pci 0000:80:05.0: Adding to iommu group 78
[Wed Jul 3 16:39:42 2024] pci 0000:80:05.2: Adding to iommu group 79
[Wed Jul 3 16:39:42 2024] pci 0000:80:05.4: Adding to iommu group 80
[Wed Jul 3 16:39:42 2024] pci 0000:80:08.0: Adding to iommu group 81
[Wed Jul 3 16:39:42 2024] pci 0000:80:08.1: Adding to iommu group 82
[Wed Jul 3 16:39:42 2024] pci 0000:80:08.2: Adding to iommu group 83
[Wed Jul 3 16:39:43 2024] pci 0000:85:05.0: Adding to iommu group 84
[Wed Jul 3 16:39:43 2024] pci 0000:85:05.2: Adding to iommu group 85
[Wed Jul 3 16:39:43 2024] pci 0000:85:05.4: Adding to iommu group 86
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.0: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.1: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.2: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.3: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.4: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.5: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.6: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:08.7: Adding to iommu group 87
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.0: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.1: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.2: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.3: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.4: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.5: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.6: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:09.7: Adding to iommu group 88
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.0: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.1: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.2: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.3: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.4: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.5: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.6: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0a.7: Adding to iommu group 89
[Wed Jul 3 16:39:43 2024] pci 0000:85:0b.0: Adding to iommu group 90
[Wed Jul 3 16:39:43 2024] pci 0000:85:0b.1: Adding to iommu group 90
[Wed Jul 3 16:39:43 2024] pci 0000:85:0b.2: Adding to iommu group 90
[Wed Jul 3 16:39:43 2024] pci 0000:85:0b.3: Adding to iommu group 90
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.0: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.1: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.2: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.3: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.4: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.5: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.6: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0e.7: Adding to iommu group 91
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.0: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.1: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.2: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.3: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.4: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.5: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.6: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:0f.7: Adding to iommu group 92
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.0: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.1: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.2: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.3: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.4: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.5: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.6: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:10.7: Adding to iommu group 93
[Wed Jul 3 16:39:43 2024] pci 0000:85:11.0: Adding to iommu group 94
[Wed Jul 3 16:39:43 2024] pci 0000:85:11.1: Adding to iommu group 94
[Wed Jul 3 16:39:43 2024] pci 0000:85:11.2: Adding to iommu group 94
[Wed Jul 3 16:39:43 2024] pci 0000:85:11.3: Adding to iommu group 94
[Wed Jul 3 16:39:43 2024] pci 0000:85:1d.0: Adding to iommu group 95
[Wed Jul 3 16:39:43 2024] pci 0000:85:1d.1: Adding to iommu group 95
[Wed Jul 3 16:39:43 2024] pci 0000:85:1d.2: Adding to iommu group 95
[Wed Jul 3 16:39:43 2024] pci 0000:85:1d.3: Adding to iommu group 95
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.0: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.1: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.2: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.3: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.4: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.5: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:85:1e.6: Adding to iommu group 96
[Wed Jul 3 16:39:43 2024] pci 0000:ae:00.0: Adding to iommu group 97
[Wed Jul 3 16:39:43 2024] pci 0000:ae:05.0: Adding to iommu group 98
[Wed Jul 3 16:39:43 2024] pci 0000:ae:05.2: Adding to iommu group 99
[Wed Jul 3 16:39:43 2024] pci 0000:ae:05.4: Adding to iommu group 100
[Wed Jul 3 16:39:43 2024] pci 0000:ae:08.0: Adding to iommu group 101
[Wed Jul 3 16:39:43 2024] pci 0000:ae:09.0: Adding to iommu group 102
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.0: Adding to iommu group 103
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.1: Adding to iommu group 104
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.2: Adding to iommu group 105
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.3: Adding to iommu group 106
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.4: Adding to iommu group 107
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.5: Adding to iommu group 108
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.6: Adding to iommu group 109
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0a.7: Adding to iommu group 110
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0b.0: Adding to iommu group 111
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0b.1: Adding to iommu group 112
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0b.2: Adding to iommu group 113
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0b.3: Adding to iommu group 114
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.0: Adding to iommu group 115
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.1: Adding to iommu group 116
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.2: Adding to iommu group 117
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.3: Adding to iommu group 118
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.4: Adding to iommu group 119
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.5: Adding to iommu group 120
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.6: Adding to iommu group 121
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0c.7: Adding to iommu group 122
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0d.0: Adding to iommu group 123
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0d.1: Adding to iommu group 124
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0d.2: Adding to iommu group 125
[Wed Jul 3 16:39:43 2024] pci 0000:ae:0d.3: Adding to iommu group 126
[Wed Jul 3 16:39:43 2024] pci 0000:af:00.0: Adding to iommu group 127
[Wed Jul 3 16:39:43 2024] pci 0000:d7:05.0: Adding to iommu group 128
[Wed Jul 3 16:39:43 2024] pci 0000:d7:05.2: Adding to iommu group 129
[Wed Jul 3 16:39:43 2024] pci 0000:d7:05.4: Adding to iommu group 130
[Wed Jul 3 16:39:43 2024] pci 0000:d7:0e.0: Adding to iommu group 131
[Wed Jul 3 16:39:43 2024] pci 0000:d7:0e.1: Adding to iommu group 132
[Wed Jul 3 16:39:43 2024] pci 0000:d7:0f.0: Adding to iommu group 133
[Wed Jul 3 16:39:43 2024] pci 0000:d7:0f.1: Adding to iommu group 134
[Wed Jul 3 16:39:43 2024] pci 0000:d7:10.0: Adding to iommu group 135
[Wed Jul 3 16:39:43 2024] pci 0000:d7:10.1: Adding to iommu group 135
[Wed Jul 3 16:39:43 2024] pci 0000:d7:12.0: Adding to iommu group 136
[Wed Jul 3 16:39:43 2024] pci 0000:d7:12.1: Adding to iommu group 137
[Wed Jul 3 16:39:43 2024] pci 0000:d7:12.2: Adding to iommu group 137
[Wed Jul 3 16:39:43 2024] pci 0000:d7:12.4: Adding to iommu group 138
[Wed Jul 3 16:39:43 2024] pci 0000:d7:12.5: Adding to iommu group 137
[Wed Jul 3 16:39:43 2024] pci 0000:d7:15.0: Adding to iommu group 139
[Wed Jul 3 16:39:43 2024] pci 0000:d7:15.1: Adding to iommu group 139
[Wed Jul 3 16:39:43 2024] pci 0000:d7:16.0: Adding to iommu group 140
[Wed Jul 3 16:39:43 2024] pci 0000:d7:16.1: Adding to iommu group 140
[Wed Jul 3 16:39:43 2024] pci 0000:d7:16.4: Adding to iommu group 140
[Wed Jul 3 16:39:43 2024] pci 0000:d7:16.5: Adding to iommu group 140
[Wed Jul 3 16:39:43 2024] pci 0000:d7:17.0: Adding to iommu group 141
[Wed Jul 3 16:39:43 2024] pci 0000:d7:17.1: Adding to iommu group 141
[Wed Jul 3 16:39:43 2024] DMAR: Intel(R) Virtualization Technology for Directed I/O
[Wed Jul 3 16:39:43 2024] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[Wed Jul 3 16:39:43 2024] software IO TLB: mapped [mem 0x0000000051000000-0x0000000055000000] (64MB)
[Wed Jul 3 16:39:43 2024] RAPL PMU: API unit is 2^-32 Joules, 2 fixed counters, 655360 ms ovfl timer
[Wed Jul 3 16:39:43 2024] RAPL PMU: hw unit of domain package 2^-14 Joules
[Wed Jul 3 16:39:43 2024] RAPL PMU: hw unit of domain dram 2^-16 Joules
[Wed Jul 3 16:39:43 2024] Initialise system trusted keyrings
[Wed Jul 3 16:39:43 2024] workingset: timestamp_bits=40 max_order=25 bucket_order=0
[Wed Jul 3 16:39:43 2024] zbud: loaded
[Wed Jul 3 16:39:43 2024] SGI XFS with ACLs, security attributes, realtime, quota, no debug enabled
[Wed Jul 3 16:39:43 2024] xor: automatically using best checksumming function avx
[Wed Jul 3 16:39:43 2024] Key type asymmetric registered
[Wed Jul 3 16:39:43 2024] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247)
[Wed Jul 3 16:39:43 2024] io scheduler mq-deadline registered
[Wed Jul 3 16:39:43 2024] io scheduler kyber registered
[Wed Jul 3 16:39:43 2024] io scheduler bfq registered
[Wed Jul 3 16:39:43 2024] pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
[Wed Jul 3 16:39:43 2024] pcieport 0000:00:1c.4: PME: Signaling with IRQ 25
[Wed Jul 3 16:39:43 2024] pcieport 0000:00:1c.5: PME: Signaling with IRQ 26
[Wed Jul 3 16:39:43 2024] pcieport 0000:5d:00.0: PME: Signaling with IRQ 28
[Wed Jul 3 16:39:43 2024] pcieport 0000:5d:02.0: PME: Signaling with IRQ 29
[Wed Jul 3 16:39:43 2024] pcieport 0000:ae:00.0: PME: Signaling with IRQ 31
[Wed Jul 3 16:39:43 2024] IPMI message handler: version 39.2
[Wed Jul 3 16:39:43 2024] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[Wed Jul 3 16:39:43 2024] ACPI: button: Power Button [PWRF]
[Wed Jul 3 16:39:43 2024] Monitor-Mwait will be used to enter C-1 state
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP00: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP00: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP01: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP01: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP02: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP02: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP03: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP03: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP04: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP04: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP05: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP05: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP06: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP06: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK0.CP07: Found 1 idle states
[Wed Jul 3 16:39:43 2024] ACPI: \_SB_.SCK1.CP07: Found 1 idle states
[Wed Jul 3 16:39:43 2024] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[Wed Jul 3 16:39:43 2024] 00:02: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
[Wed Jul 3 16:39:43 2024] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[Wed Jul 3 16:39:43 2024] lp: driver loaded but no devices found
[Wed Jul 3 16:39:43 2024] Linux agpgart interface v0.103
[Wed Jul 3 16:39:43 2024] brd: module loaded
[Wed Jul 3 16:39:43 2024] loop: module loaded
[Wed Jul 3 16:39:43 2024] drbd: initialized. Version: 8.4.11 (api:1/proto:86-101)
[Wed Jul 3 16:39:44 2024] drbd: built-in
[Wed Jul 3 16:39:44 2024] drbd: registered as block device major 147
[Wed Jul 3 16:39:44 2024] lpc_ich 0000:00:1f.0: I/O space for ACPI uninitialized
[Wed Jul 3 16:39:44 2024] lpc_ich 0000:00:1f.0: No MFD cells added
[Wed Jul 3 16:39:44 2024] Microchip SmartPQI Driver (v2.1.10-020)
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: Microchip Smart Family Controller found
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: Maximum Known Feature not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 0 Read Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 1 Read Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 5 Read Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 6 Read Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 0 Write Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 1 Write Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 5 Write Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID 6 Write Bypass not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID Bypass on encrypted logical volumes on NVMe not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: Unique WWID in Report Physical LUN not supported by controller
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: Online Firmware Activation enabled
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: Serial Management Protocol enabled
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: New Soft Reset Handshake enabled
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: RAID IU Timeout enabled
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: TMF IU Timeout enabled
[Wed Jul 3 16:39:44 2024] scsi host0: smartpqi
[Wed Jul 3 16:39:44 2024] scsi 0:0:0:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:0:0 5000c500d7904ce5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:1:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:1:0 5000c500d79aada9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:2:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:2:0 5000c500d7650145 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:3:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:3:0 5000c500d7905a05 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:4:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:4:0 5000c500d6e654c5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:5:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:5:0 5000c500cbcff725 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:6:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:6:0 5000c500d790f1d1 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:7:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:7:0 5000c500d790ea01 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:8:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:8:0 5000c500d79035a9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:9:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:9:0 5000c500d79db67d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:10:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:10:0 5000c500d78f6395 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:11:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:11:0 5000c500d78e4225 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:12:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:12:0 5000c500d78f13c9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:13:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:13:0 5000c500d790e7dd Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] tsc: Refined TSC clocksource calibration: 3791.094 MHz
[Wed Jul 3 16:39:44 2024] scsi 0:0:14:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x6d4af7910a1, max_idle_ns: 881590839817 ns
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:14:0 5000c500d78f1421 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] clocksource: Switched to clocksource tsc
[Wed Jul 3 16:39:44 2024] scsi 0:0:15:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:15:0 5000c500d78f0d49 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:16:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:16:0 5000c500d78e3215 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:17:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:17:0 5000c500d790d35d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:18:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:18:0 5000c500d78f285d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:19:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:19:0 5000c500d7905375 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:20:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:44 2024] smartpqi 0000:af:00.0: added 0:0:20:0 5000c500d78de735 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:44 2024] scsi 0:0:21:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:21:0 5000c500d75d98e5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:22:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:22:0 5000c500d7910ce5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:23:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:23:0 5000c500d7902079 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:24:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:24:0 5000c500cbde7d9d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:25:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:25:0 5000c500d6e59df5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:26:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:26:0 5000c500d6f1339d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:27:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:27:0 5000c500d78e5049 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:28:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:28:0 5000c500d6f4e291 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:29:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:29:0 5000c500d6f748b1 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:30:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:30:0 5000c500d6e57db5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:31:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:31:0 5000c500d78e4d6d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:32:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:32:0 5000c500d6f4a4d9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:33:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:33:0 5000c500d6f481a9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:34:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:34:0 5000c500d6e5f9d5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:35:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:35:0 5000c500d78e8d01 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:36:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:36:0 5000c500d6f4dca5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:37:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:37:0 5000c500d6e57f0d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:38:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:38:0 5000c500d6f4ea61 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:39:0: Direct-Access SEAGATE ST16000NM004J E001 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:39:0 5000c500d78e598d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:40:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:40:0 5000c500d6e5aea9 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:41:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:41:0 5000c500d6e1502d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:42:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:42:0 5000c500d6fe649d Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:43:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:43:0 5000c500d6eeee05 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:44:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:44:0 5000c500d6f17471 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:45:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:45:0 5000c500d6f638e5 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:46:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:46:0 5000c500d6e58279 Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:47:0: Direct-Access SEAGATE ST16000NM004J E002 PQ: 0 ANSI: 7
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:47:0 5000c500d6e57abd Direct-Access SEAGATE ST16000NM004J AIO+ qd=64
[Wed Jul 3 16:39:45 2024] scsi 0:0:48:0: Enclosure AIC 12G 4U24SAS3swap 0c01 PQ: 0 ANSI: 5
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:48:0 50015b21408b8c7d Enclosure AIC 12G 4U24SAS3swap AIO-
[Wed Jul 3 16:39:45 2024] scsi 0:0:49:0: Enclosure AIC 12G 4U24SAS3swap 0c01 PQ: 0 ANSI: 5
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:49:0 50015b21408b863d Enclosure AIC 12G 4U24SAS3swap AIO-
[Wed Jul 3 16:39:45 2024] scsi 0:0:50:0: Enclosure Adaptec Smart Adapter 2.93 PQ: 0 ANSI: 5
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:0:50:0 50000d1e0019e5d8 Enclosure Adaptec Smart Adapter AIO-
[Wed Jul 3 16:39:45 2024] scsi 0:2:0:0: RAID Adaptec 1100-8e 2.93 PQ: 0 ANSI: 5
[Wed Jul 3 16:39:45 2024] smartpqi 0000:af:00.0: added 0:2:0:0 0000000000000000 RAID Adaptec 1100-8e
[Wed Jul 3 16:39:45 2024] megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
[Wed Jul 3 16:39:45 2024] megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
[Wed Jul 3 16:39:45 2024] megasas: 07.717.02.00-rc1
[Wed Jul 3 16:39:45 2024] mpt3sas version 39.100.00.00 loaded
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: 63 BIT PCI BUS DMA ADDRESSING SUPPORTED, total mem (131368668 kB)
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: MSI-X vectors supported: 96
[Wed Jul 3 16:39:45 2024] no of cores: 16, max_msix_vectors: -1
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: 0 16 16
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: High IOPs queues : disabled
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix0: PCI-MSI-X enabled: IRQ 50
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix1: PCI-MSI-X enabled: IRQ 51
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix2: PCI-MSI-X enabled: IRQ 52
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix3: PCI-MSI-X enabled: IRQ 53
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix4: PCI-MSI-X enabled: IRQ 54
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix5: PCI-MSI-X enabled: IRQ 55
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix6: PCI-MSI-X enabled: IRQ 56
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix7: PCI-MSI-X enabled: IRQ 57
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix8: PCI-MSI-X enabled: IRQ 58
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix9: PCI-MSI-X enabled: IRQ 59
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix10: PCI-MSI-X enabled: IRQ 60
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix11: PCI-MSI-X enabled: IRQ 61
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix12: PCI-MSI-X enabled: IRQ 62
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix13: PCI-MSI-X enabled: IRQ 63
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix14: PCI-MSI-X enabled: IRQ 64
[Wed Jul 3 16:39:45 2024] mpt3sas0-msix15: PCI-MSI-X enabled: IRQ 65
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: iomem(0x00000000bb200000), mapped(0x0000000075a5e6fa), size(65536)
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: ioport(0x0000000000008000), size(256)
[Wed Jul 3 16:39:45 2024] mpt3sas_cm0: CurrentHostPageSize is 0: Setting default host page size to 4k
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: scatter gather: sge_in_main_msg(1), sge_per_chain(7), sge_per_io(128), chains_per_io(19)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: request pool(0x0000000065db5e87) - dma(0xffa00000): depth(9700), frame_size(128), pool_size(1212 kB)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: sense pool(0x00000000a0e33d29) - dma(0xfe100000): depth(9463), element_size(96), pool_size (887 kB)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: reply pool(0x00000000f61c3ac9) - dma(0xfde00000): depth(9764), frame_size(128), pool_size(1220 kB)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: config page(0x000000008b608c44) - dma(0xfddee000): size(512)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: Allocated physical memory: size(28267 kB)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: Current Controller Queue Depth(9460),Max Controller Queue Depth(9584)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: Scatter Gather Elements per IO(128)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: _base_display_fwpkg_version: complete
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: FW Package Ver(16.17.01.00)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: LSISAS3008: FWVersion(16.00.11.00), ChipRevision(0x02), BiosVersion(18.00.03.00)
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: Protocol=(Initiator,Target), Capabilities=(TLR,EEDP,Snapshot Buffer,Diag Trace Buffer,Task Set Full,NCQ)
[Wed Jul 3 16:39:46 2024] scsi host1: Fusion MPT SAS Host
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: sending port enable !!
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: Attached scsi generic sg0 type 0
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: hba_port entry: 00000000df6daeb7, port: 255 is added to hba_port list
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: Attached scsi generic sg1 type 0
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: host_add: handle(0x0001), sas_addr(0x52cea7f06c599e00), phys(8)
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: Attached scsi generic sg2 type 0
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: handle(0xa) sas_address(0x4433221106000000) port_type(0x1)
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: [sdy] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: [sdac] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: [sdag] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: [sdz] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: [sdai] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: [sdas] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: [sdap] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: Attached scsi generic sg3 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: [sdf] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: [sdaq] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: [sdau] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: [sdad] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: [sdao] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: [sdak] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: [sdah] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: [sdal] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: [sdar] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: [sdav] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: [sdaa] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: handle(0xb) sas_address(0x4433221107000000) port_type(0x1)
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: [sdam] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: [sdae] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] mpt3sas_cm0: port enable: SUCCESS
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: [sdat] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: Attached scsi generic sg4 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:5:0: Attached scsi generic sg5 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: Attached scsi generic sg6 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: Attached scsi generic sg7 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: Attached scsi generic sg8 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: Attached scsi generic sg9 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: Attached scsi generic sg10 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: Attached scsi generic sg11 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: Attached scsi generic sg12 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: Attached scsi generic sg13 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: Attached scsi generic sg14 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: Attached scsi generic sg15 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: Attached scsi generic sg16 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: Attached scsi generic sg17 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: Attached scsi generic sg18 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: Attached scsi generic sg19 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: Attached scsi generic sg20 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: Attached scsi generic sg21 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: Attached scsi generic sg22 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: Attached scsi generic sg23 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:24:0: Attached scsi generic sg24 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:25:0: Attached scsi generic sg25 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:26:0: Attached scsi generic sg26 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: Attached scsi generic sg27 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:28:0: Attached scsi generic sg28 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:29:0: Attached scsi generic sg29 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:30:0: Attached scsi generic sg30 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: Attached scsi generic sg31 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:32:0: Attached scsi generic sg32 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:33:0: Attached scsi generic sg33 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:34:0: Attached scsi generic sg34 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: Attached scsi generic sg35 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:36:0: Attached scsi generic sg36 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:37:0: Attached scsi generic sg37 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:38:0: Attached scsi generic sg38 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: Attached scsi generic sg39 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:40:0: Attached scsi generic sg40 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:41:0: Attached scsi generic sg41 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:42:0: Attached scsi generic sg42 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:43:0: Attached scsi generic sg43 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:44:0: Attached scsi generic sg44 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:45:0: Attached scsi generic sg45 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:46:0: Attached scsi generic sg46 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:47:0: Attached scsi generic sg47 type 0
[Wed Jul 3 16:39:46 2024] scsi 0:0:48:0: Attached scsi generic sg48 type 13
[Wed Jul 3 16:39:46 2024] scsi 0:0:49:0: Attached scsi generic sg49 type 13
[Wed Jul 3 16:39:46 2024] scsi 0:0:50:0: Attached scsi generic sg50 type 13
[Wed Jul 3 16:39:46 2024] scsi 0:2:0:0: Attached scsi generic sg51 type 12
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: Direct-Access ATA ST1000NM0018-2F2 EA04 PQ: 0 ANSI: 6
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: SATA: handle(0x000a), sas_addr(0x4433221106000000), phy(6), device_name(0x5000c500c623f3da)
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: enclosure logical id (0x52cea7f06c599e00), slot(0)
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: enclosure level(0x0001), connector name( )
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[Wed Jul 3 16:39:46 2024] scsi 1:0:0:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] end_device-1:0: add: handle(0x000a), sas_addr(0x4433221106000000)
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: Direct-Access ATA ST1000NM0018-2F2 EA04 PQ: 0 ANSI: 6
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: SATA: handle(0x000b), sas_addr(0x4433221107000000), phy(7), device_name(0x5000c500c6241352)
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: enclosure logical id (0x52cea7f06c599e00), slot(1)
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: enclosure level(0x0001), connector name( )
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[Wed Jul 3 16:39:46 2024] scsi 1:0:1:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] end_device-1:1: add: handle(0x000b), sas_addr(0x4433221107000000)
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: Attached scsi generic sg52 type 0
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: Attached scsi generic sg53 type 0
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: [sdax] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: [sdaw] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] 31251759104 512-byte logical blocks: (16.0 TB/14.6 TiB)
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] 4096-byte physical blocks
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] Mode Sense: df 00 10 08
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: [sdax] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: [sdaw] Write Protect is off
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: [sdax] Mode Sense: 9b 00 10 08
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: [sdax] Write cache: disabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: [sdaw] Mode Sense: 9b 00 10 08
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: [sdaw] Write cache: disabled, read cache: enabled, supports DPO and FUA
[Wed Jul 3 16:39:46 2024] sd 1:0:1:0: [sdax] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sdaw: sdaw1 sdaw2 sdaw3
[Wed Jul 3 16:39:46 2024] sd 0:0:31:0: [sdaf] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:1:0: [sdb] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:15:0: [sdp] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:22:0: [sdw] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:2:0: [sdc] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:6:0: [sdg] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:4:0: [sde] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:12:0: [sdm] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:7:0: [sdh] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:18:0: [sds] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:9:0: [sdj] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:13:0: [sdn] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:27:0: [sdab] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:39:0: [sdan] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:14:0: [sdo] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:10:0: [sdk] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:23:0: [sdx] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:35:0: [sdaj] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:0:0: [sda] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:21:0: [sdv] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:16:0: [sdq] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:11:0: [sdl] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 1:0:0:0: [sdaw] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:20:0: [sdu] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:8:0: [sdi] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:3:0: [sdd] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:17:0: [sdr] Attached SCSI disk
[Wed Jul 3 16:39:46 2024] sd 0:0:19:0: [sdt] Attached SCSI disk
[Wed Jul 3 16:39:48 2024] ses 0:0:48:0: Attached Enclosure device
[Wed Jul 3 16:39:48 2024] ses 0:0:49:0: Attached Enclosure device
[Wed Jul 3 16:39:48 2024] ses 0:0:50:0: Attached Enclosure device
[Wed Jul 3 16:39:48 2024] ahci 0000:00:11.5: version 3.0
[Wed Jul 3 16:39:48 2024] ahci 0000:00:11.5: AHCI 0001.0301 32 slots 6 ports 6 Gbps 0x3f impl SATA mode
[Wed Jul 3 16:39:48 2024] ahci 0000:00:11.5: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst
[Wed Jul 3 16:39:48 2024] scsi host2: ahci
[Wed Jul 3 16:39:48 2024] scsi host3: ahci
[Wed Jul 3 16:39:48 2024] scsi host4: ahci
[Wed Jul 3 16:39:48 2024] scsi host5: ahci
[Wed Jul 3 16:39:48 2024] scsi host6: ahci
[Wed Jul 3 16:39:48 2024] scsi host7: ahci
[Wed Jul 3 16:39:48 2024] ata1: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80100 irq 66
[Wed Jul 3 16:39:48 2024] ata2: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80180 irq 66
[Wed Jul 3 16:39:48 2024] ata3: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80200 irq 66
[Wed Jul 3 16:39:48 2024] ata4: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80280 irq 66
[Wed Jul 3 16:39:48 2024] ata5: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80300 irq 66
[Wed Jul 3 16:39:48 2024] ata6: SATA max UDMA/133 abar m524288@0x92f80000 port 0x92f80380 irq 66
[Wed Jul 3 16:39:48 2024] ahci 0000:00:17.0: AHCI 0001.0301 32 slots 8 ports 6 Gbps 0xff impl SATA mode
[Wed Jul 3 16:39:48 2024] ahci 0000:00:17.0: flags: 64bit ncq sntf pm led clo only pio slum part ems deso sadm sds apst
[Wed Jul 3 16:39:48 2024] scsi host8: ahci
[Wed Jul 3 16:39:48 2024] scsi host9: ahci
[Wed Jul 3 16:39:48 2024] scsi host10: ahci
[Wed Jul 3 16:39:48 2024] scsi host11: ahci
[Wed Jul 3 16:39:48 2024] scsi host12: ahci
[Wed Jul 3 16:39:48 2024] scsi host13: ahci
[Wed Jul 3 16:39:48 2024] scsi host14: ahci
[Wed Jul 3 16:39:48 2024] scsi host15: ahci
[Wed Jul 3 16:39:48 2024] ata7: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00100 irq 67
[Wed Jul 3 16:39:48 2024] ata8: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00180 irq 67
[Wed Jul 3 16:39:48 2024] ata9: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00200 irq 67
[Wed Jul 3 16:39:48 2024] ata10: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00280 irq 67
[Wed Jul 3 16:39:48 2024] ata11: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00300 irq 67
[Wed Jul 3 16:39:48 2024] ata12: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00380 irq 67
[Wed Jul 3 16:39:48 2024] ata13: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00400 irq 67
[Wed Jul 3 16:39:48 2024] ata14: SATA max UDMA/133 abar m524288@0x92f00000 port 0x92f00480 irq 67
[Wed Jul 3 16:39:48 2024] tun: Universal TUN/TAP device driver, 1.6
[Wed Jul 3 16:39:48 2024] Fusion MPT base driver 3.04.20
[Wed Jul 3 16:39:48 2024] Copyright (c) 1999-2008 LSI Corporation
[Wed Jul 3 16:39:48 2024] Fusion MPT SPI Host driver 3.04.20
[Wed Jul 3 16:39:48 2024] Fusion MPT FC Host driver 3.04.20
[Wed Jul 3 16:39:48 2024] Fusion MPT SAS Host driver 3.04.20
[Wed Jul 3 16:39:48 2024] Fusion MPT misc device (ioctl) driver 3.04.20
[Wed Jul 3 16:39:48 2024] mptctl: Registered with Fusion MPT base driver
[Wed Jul 3 16:39:48 2024] mptctl: /dev/mptctl @ (major,minor=10,220)
[Wed Jul 3 16:39:48 2024] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[Wed Jul 3 16:39:48 2024] ehci-pci: EHCI PCI platform driver
[Wed Jul 3 16:39:48 2024] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[Wed Jul 3 16:39:48 2024] ohci-pci: OHCI PCI platform driver
[Wed Jul 3 16:39:48 2024] uhci_hcd: USB Universal Host Controller Interface driver
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: xHCI Host Controller
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 1
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: hcc params 0x200077c1 hci version 0x100 quirks 0x0000000000009810
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: xHCI Host Controller
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: new USB bus registered, assigned bus number 2
[Wed Jul 3 16:39:48 2024] xhci_hcd 0000:00:14.0: Host supports USB 3.0 SuperSpeed
[Wed Jul 3 16:39:48 2024] hub 1-0:1.0: USB hub found
[Wed Jul 3 16:39:48 2024] hub 1-0:1.0: 16 ports detected
[Wed Jul 3 16:39:48 2024] hub 2-0:1.0: USB hub found
[Wed Jul 3 16:39:48 2024] hub 2-0:1.0: 10 ports detected
[Wed Jul 3 16:39:48 2024] usbcore: registered new interface driver usb-storage
[Wed Jul 3 16:39:48 2024] i8042: PNP: No PS/2 controller found.
[Wed Jul 3 16:39:48 2024] rtc_cmos 00:00: RTC can wake from S4
[Wed Jul 3 16:39:48 2024] rtc_cmos 00:00: registered as rtc0
[Wed Jul 3 16:39:48 2024] rtc_cmos 00:00: setting system clock to 2024-07-03T14:39:48 UTC (1720017588)
[Wed Jul 3 16:39:48 2024] rtc_cmos 00:00: alarms up to one month, y3k, 114 bytes nvram
[Wed Jul 3 16:39:48 2024] i801_smbus 0000:00:1f.4: SPD Write Disable is set
[Wed Jul 3 16:39:48 2024] ata2: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] i801_smbus 0000:00:1f.4: SMBus using PCI interrupt
[Wed Jul 3 16:39:48 2024] ata6: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] ata5: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] ata1: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] ata3: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] ata4: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] i2c i2c-0: 4/16 memory slots populated (from DMI)
[Wed Jul 3 16:39:48 2024] i2c i2c-0: Systems with more than 4 memory slots not supported yet, not instantiating SPD
[Wed Jul 3 16:39:48 2024] intel_pstate: Intel P-state driver initializing
[Wed Jul 3 16:39:48 2024] EFI Variables Facility v0.08 2004-May-17
[Wed Jul 3 16:39:48 2024] hid: raw HID events driver (C) Jiri Kosina
[Wed Jul 3 16:39:48 2024] Key type dns_resolver registered
[Wed Jul 3 16:39:48 2024] microcode: sig=0x50657, pf=0x80, revision=0x5003604
[Wed Jul 3 16:39:48 2024] microcode: Microcode Update Driver: v2.2.
[Wed Jul 3 16:39:48 2024] IPI shorthand broadcast: enabled
[Wed Jul 3 16:39:48 2024] ata12: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:48 2024] sched_clock: Marking stable (13509827570, 1489719769)->(15473915089, -474367750)
[Wed Jul 3 16:39:49 2024] ata11: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] registered taskstats version 1
[Wed Jul 3 16:39:49 2024] ata7: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] Loading compiled-in X.509 certificates
[Wed Jul 3 16:39:49 2024] ata9: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] zswap: loaded using pool lzo/zbud
[Wed Jul 3 16:39:49 2024] ata13: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] ata8: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] ata14: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] ata10: SATA link down (SStatus 0 SControl 300)
[Wed Jul 3 16:39:49 2024] usb 1-14: new high-speed USB device number 2 using xhci_hcd
[Wed Jul 3 16:39:49 2024] clk: Disabling unused clocks
[Wed Jul 3 16:39:49 2024] Freeing unused kernel image (initmem) memory: 1800K
[Wed Jul 3 16:39:49 2024] Write protecting the kernel read-only data: 20480k
[Wed Jul 3 16:39:49 2024] Freeing unused kernel image (text/rodata gap) memory: 2036K
[Wed Jul 3 16:39:49 2024] Freeing unused kernel image (rodata/data gap) memory: 556K
[Wed Jul 3 16:39:49 2024] Run /init as init process
[Wed Jul 3 16:39:49 2024] with arguments:
[Wed Jul 3 16:39:49 2024] /init
[Wed Jul 3 16:39:49 2024] with environment:
[Wed Jul 3 16:39:49 2024] HOME=/
[Wed Jul 3 16:39:49 2024] TERM=linux
[Wed Jul 3 16:39:49 2024] hub 1-14:1.0: USB hub found
[Wed Jul 3 16:39:49 2024] hub 1-14:1.0: 4 ports detected
[Wed Jul 3 16:39:49 2024] usb 1-14.1: new high-speed USB device number 3 using xhci_hcd
[Wed Jul 3 16:39:50 2024] hub 1-14.1:1.0: USB hub found
[Wed Jul 3 16:39:50 2024] hub 1-14.1:1.0: 4 ports detected
[Wed Jul 3 16:39:50 2024] usb 1-14.4: new high-speed USB device number 4 using xhci_hcd
[Wed Jul 3 16:39:50 2024] XFS (sdaw1): Mounting V5 Filesystem
[Wed Jul 3 16:39:50 2024] hub 1-14.4:1.0: USB hub found
[Wed Jul 3 16:39:50 2024] hub 1-14.4:1.0: 4 ports detected
[Wed Jul 3 16:39:50 2024] XFS (sdaw1): Ending clean mount
[Wed Jul 3 16:39:51 2024] systemd[1]: Inserted module 'autofs4'
[Wed Jul 3 16:39:51 2024] NET: Registered PF_INET6 protocol family
[Wed Jul 3 16:39:51 2024] Segment Routing with IPv6
[Wed Jul 3 16:39:51 2024] In-situ OAM (IOAM) with IPv6
[Wed Jul 3 16:39:51 2024] NET: Registered PF_UNIX/PF_LOCAL protocol family
[Wed Jul 3 16:39:51 2024] systemd[1]: Inserted module 'unix'
[Wed Jul 3 16:39:51 2024] systemd[1]: systemd 242 running in system mode. (+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[Wed Jul 3 16:39:51 2024] systemd[1]: Detected architecture x86-64.
[Wed Jul 3 16:39:51 2024] systemd[1]: Set hostname to <furoncles.molgen.mpg.de>.
[Wed Jul 3 16:39:52 2024] systemd[1]: Created slice system-serial\x2dlog.slice.
[Wed Jul 3 16:39:52 2024] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[Wed Jul 3 16:39:52 2024] systemd[1]: Listening on udev Kernel Socket.
[Wed Jul 3 16:39:52 2024] systemd[1]: Listening on initctl Compatibility Named Pipe.
[Wed Jul 3 16:39:52 2024] systemd[1]: Listening on udev Control Socket.
[Wed Jul 3 16:39:52 2024] systemd[1]: Listening on Journal Socket.
[Wed Jul 3 16:39:52 2024] systemd[1]: Condition check resulted in Load Kernel Modules being skipped.
[Wed Jul 3 16:39:53 2024] RPC: Registered named UNIX socket transport module.
[Wed Jul 3 16:39:53 2024] RPC: Registered udp transport module.
[Wed Jul 3 16:39:53 2024] RPC: Registered tcp transport module.
[Wed Jul 3 16:39:53 2024] RPC: Registered tcp NFSv4.1 backchannel transport module.
[Wed Jul 3 16:39:54 2024] systemd-journald[320]: Received request to flush runtime journal from PID 1
[Wed Jul 3 16:39:54 2024] ipmi_si: IPMI System Interface driver
[Wed Jul 3 16:39:54 2024] ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
[Wed Jul 3 16:39:54 2024] ipmi_platform: ipmi_si: SMBIOS: io 0xca8 regsize 1 spacing 4 irq 10
[Wed Jul 3 16:39:54 2024] ipmi_si: Adding SMBIOS-specified kcs state machine
[Wed Jul 3 16:39:54 2024] iTCO_vendor_support: vendor-support=0
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: ipmi_platform: probing via ACPI
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: ipmi_platform: [io 0x0ca8] regsize 1 spacing 4 irq 10
[Wed Jul 3 16:39:54 2024] ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI
[Wed Jul 3 16:39:54 2024] ipmi_si: Adding ACPI-specified kcs state machine
[Wed Jul 3 16:39:54 2024] ipmi_si: Trying ACPI-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 10
[Wed Jul 3 16:39:54 2024] wmi_bus wmi_bus-PNP0C14:00: WQBC data block query control method not found
[Wed Jul 3 16:39:54 2024] mgag200 0000:03:00.0: vgaarb: deactivate vga console
[Wed Jul 3 16:39:54 2024] iTCO_wdt iTCO_wdt: Found a Intel PCH TCO device (Version=4, TCOBASE=0x0400)
[Wed Jul 3 16:39:54 2024] Console: switching to colour dummy device 80x25
[Wed Jul 3 16:39:54 2024] iTCO_wdt iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0)
[Wed Jul 3 16:39:54 2024] [drm] Initialized mgag200 1.0.0 20110418 for 0000:03:00.0 on minor 0
[Wed Jul 3 16:39:54 2024] fbcon: mgag200drmfb (fb0) is primary device
[Wed Jul 3 16:39:54 2024] mgag200 0000:03:00.0: [drm] drm_plane_enable_fb_damage_clips() not called
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: The BMC does not support setting the recv irq bit, compensating, but the BMC needs to be fixed.
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: Using irq 10
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: IPMI message handler: Found new BMC (man_id: 0x0002a2, prod_id: 0x0100, dev_id: 0x20)
[Wed Jul 3 16:39:54 2024] ipmi_si IPI0001:00: IPMI kcs interface initialized
[Wed Jul 3 16:39:54 2024] pstore: Using crash dump compression: deflate
[Wed Jul 3 16:39:54 2024] pstore: Registered efi as persistent store backend
[Wed Jul 3 16:39:54 2024] Console: switching to colour frame buffer device 128x48
[Wed Jul 3 16:39:55 2024] mgag200 0000:03:00.0: [drm] fb0: mgag200drmfb frame buffer device
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.0 eth0: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 2c:ea:7f:67:63:0f
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.0 eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.0 eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.0 eth0: dma_rwctrl[00000001] dma_mask[64-bit]
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.1 eth1: Tigon3 [partno(BCM95720) rev 5720000] (PCI Express) MAC address 2c:ea:7f:67:63:10
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.1 eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.1 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] TSOcap[1]
[Wed Jul 3 16:39:55 2024] tg3 0000:04:00.1 eth1: dma_rwctrl[00000001] dma_mask[64-bit]
[Wed Jul 3 16:39:55 2024] i40e: Intel(R) Ethernet Connection XL710 Network Driver
[Wed Jul 3 16:39:55 2024] i40e: Copyright (c) 2013 - 2019 Intel Corporation.
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.0: fw 8.815.63341 api 1.12 nvm 8.15 0x800096c1 20.0.17 [8086:1572] [8086:0006]
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.0: MAC address: f8:f2:1e:bb:d4:b0
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.0 eth2: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.0: PCI-Express: Speed 8.0GT/s Width x8
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.0: Features: PF-id[0] VFs: 64 VSIs: 66 QP: 16 RSS FD_ATR FD_SB NTUPLE VxLAN Geneve PTP VEPA
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.1: fw 8.815.63341 api 1.12 nvm 8.15 0x800096c1 20.0.17 [8086:1572] [8086:0006]
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.1: MAC address: f8:f2:1e:bb:d4:b1
[Wed Jul 3 16:39:55 2024] i40e 0000:5f:00.1: PCI-Express: Speed 8.0GT/s Width x8
[Wed Jul 3 16:39:56 2024] i40e 0000:5f:00.1: Features: PF-id[1] VFs: 64 VSIs: 66 QP: 16 RSS FD_ATR FD_SB NTUPLE VxLAN Geneve PTP VEPA
[Wed Jul 3 16:39:57 2024] md: md0 stopped.
[Wed Jul 3 16:39:57 2024] md/raid:md0: device sdah operational as raid disk 0
[Wed Jul 3 16:39:57 2024] md/raid:md0: device sdaf operational as raid disk 11
[Wed Jul 3 16:39:57 2024] md/raid:md0: device sdab operational as raid disk 10
[Wed Jul 3 16:39:57 2024] md/raid:md0: device sdaj operational as raid disk 9
[Wed Jul 3 16:39:57 2024] md/raid:md0: device sdae operational as raid disk 8
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdaa operational as raid disk 7
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdai operational as raid disk 6
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdy operational as raid disk 5
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdac operational as raid disk 4
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdag operational as raid disk 3
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdad operational as raid disk 2
[Wed Jul 3 16:39:58 2024] md/raid:md0: device sdz operational as raid disk 1
[Wed Jul 3 16:39:58 2024] md/raid:md0: raid level 6 active with 12 out of 12 devices, algorithm 2
[Wed Jul 3 16:39:58 2024] md: md1 stopped.
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdap operational as raid disk 0
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdan operational as raid disk 11
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdav operational as raid disk 10
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdar operational as raid disk 9
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdam operational as raid disk 8
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdau operational as raid disk 7
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdaq operational as raid disk 6
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdak operational as raid disk 5
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdas operational as raid disk 4
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdao operational as raid disk 3
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdal operational as raid disk 2
[Wed Jul 3 16:39:58 2024] md/raid:md1: device sdat operational as raid disk 1
[Wed Jul 3 16:39:58 2024] md/raid:md1: raid level 6 active with 12 out of 12 devices, algorithm 2
[Wed Jul 3 16:39:58 2024] md: md2 stopped.
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdm operational as raid disk 0
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdx operational as raid disk 11
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdw operational as raid disk 10
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdv operational as raid disk 9
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdu operational as raid disk 8
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdt operational as raid disk 7
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sds operational as raid disk 6
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdr operational as raid disk 5
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdq operational as raid disk 4
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdp operational as raid disk 3
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdo operational as raid disk 2
[Wed Jul 3 16:39:58 2024] md/raid:md2: device sdn operational as raid disk 1
[Wed Jul 3 16:39:58 2024] md/raid:md2: raid level 6 active with 12 out of 12 devices, algorithm 2
[Wed Jul 3 16:39:58 2024] md: md3 stopped.
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdj operational as raid disk 0
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdd operational as raid disk 11
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdh operational as raid disk 10
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdl operational as raid disk 9
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdc operational as raid disk 8
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdg operational as raid disk 7
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdk operational as raid disk 6
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sda operational as raid disk 5
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sde operational as raid disk 4
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdi operational as raid disk 3
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdf operational as raid disk 2
[Wed Jul 3 16:39:58 2024] md/raid:md3: device sdb operational as raid disk 1
[Wed Jul 3 16:39:58 2024] md/raid:md3: raid level 6 active with 12 out of 12 devices, algorithm 2
[Wed Jul 3 16:39:58 2024] XFS (sdaw2): Mounting V5 Filesystem
[Wed Jul 3 16:39:59 2024] XFS (sdaw2): Ending clean mount
[Wed Jul 3 16:39:59 2024] XFS (md0): Mounting V5 Filesystem
[Wed Jul 3 16:39:59 2024] XFS (md0): Ending clean mount
[Wed Jul 3 16:39:59 2024] XFS (md1): Mounting V5 Filesystem
[Wed Jul 3 16:39:59 2024] XFS (md1): Ending clean mount
[Wed Jul 3 16:39:59 2024] XFS (md2): Mounting V5 Filesystem
[Wed Jul 3 16:40:04 2024] XFS (md2): Ending clean mount
[Wed Jul 3 16:40:04 2024] XFS (md3): Mounting V5 Filesystem
[Wed Jul 3 16:40:04 2024] XFS (md3): Ending clean mount
[Wed Jul 3 16:40:05 2024] tg3 0000:04:00.0 net00: renamed from eth0
[Wed Jul 3 16:40:05 2024] tg3 0000:04:00.1 net01: renamed from eth1
[Wed Jul 3 16:40:05 2024] i40e 0000:5f:00.0 net02: renamed from eth2
[Wed Jul 3 16:40:05 2024] i40e 0000:5f:00.1 net03: renamed from eth3
[Wed Jul 3 16:40:05 2024] 8021q: 802.1Q VLAN Support v1.8
[Wed Jul 3 16:40:05 2024] 8021q: adding VLAN 0 to HW filter on device net02
[Wed Jul 3 16:40:06 2024] NFSD: Using UMH upcall client tracking operations.
[Wed Jul 3 16:40:06 2024] NFSD: starting 90-second grace period (net f0000000)
[Wed Jul 3 16:40:07 2024] rpc-srv/tcp: nfsd: got error -32 when sending 20 bytes - shutting down socket
[Wed Jul 3 16:41:02 2024] hrtimer: interrupt took 3804 ns
[Wed Jul 3 16:53:31 2024] FS-Cache: Netfs 'nfs' registered for caching
[Wed Jul 3 16:53:31 2024] NFS: Registering the id_resolver key type
[Wed Jul 3 16:53:31 2024] Key type id_resolver registered
[Wed Jul 3 16:53:31 2024] Key type id_legacy registered
[Mon Jul 8 12:34:54 2024] NET: Registered PF_PACKET protocol family
[Wed Jul 10 06:14:26 2024] smartpqi 0000:af:00.0: resetting scsi 0:0:22:0 due to cmd 0xa3
[Wed Jul 10 06:14:26 2024] smartpqi 0000:af:00.0: reset of scsi 0:0:22:0: SUCCESS
[Wed Jul 10 06:14:26 2024] sd 0:0:22:0: Power-on or device reset occurred
[Thu Jul 11 06:14:21 2024] smartpqi 0000:af:00.0: resetting scsi 0:0:15:0 due to cmd 0x4d
[Thu Jul 11 06:14:21 2024] smartpqi 0000:af:00.0: reset of scsi 0:0:15:0: SUCCESS
[Thu Jul 11 06:14:21 2024] sd 0:0:15:0: Power-on or device reset occurred
[Sat Jul 13 01:59:46 2024] md: data-check of RAID array md0
[Sat Jul 13 01:59:46 2024] md: data-check of RAID array md1
[Sat Jul 13 01:59:46 2024] md: data-check of RAID array md2
[Sat Jul 13 01:59:46 2024] md: data-check of RAID array md3
[Sat Jul 13 05:59:47 2024] md: md0: data-check interrupted.
[Sat Jul 13 05:59:57 2024] md: md1: data-check interrupted.
[Sat Jul 13 06:00:07 2024] md: md2: data-check interrupted.
[Sat Jul 13 06:00:17 2024] md: md3: data-check interrupted.
[Sun Jul 14 01:59:43 2024] md: data-check of RAID array md0
[Sun Jul 14 01:59:43 2024] md: data-check of RAID array md1
[Sun Jul 14 01:59:43 2024] md: data-check of RAID array md2
[Sun Jul 14 01:59:43 2024] md: data-check of RAID array md3
[Sun Jul 14 05:59:44 2024] md: md0: data-check interrupted.
[Sun Jul 14 06:00:04 2024] md: md2: data-check interrupted.
[Sun Jul 14 06:00:14 2024] md: md3: data-check interrupted.
[Sun Jul 14 06:00:37 2024] md: md1: data-check interrupted.
[Mon Jul 15 01:59:42 2024] md: data-check of RAID array md0
[Mon Jul 15 01:59:42 2024] md: data-check of RAID array md1
[Mon Jul 15 01:59:42 2024] md: data-check of RAID array md2
[Mon Jul 15 01:59:42 2024] md: data-check of RAID array md3
[Mon Jul 15 05:59:42 2024] md: md0: data-check interrupted.
[Mon Jul 15 05:59:52 2024] md: md1: data-check interrupted.
[Mon Jul 15 06:00:02 2024] md: md2: data-check interrupted.
[Mon Jul 15 06:00:13 2024] md: md3: data-check interrupted.
[Tue Jul 16 01:59:39 2024] md: data-check of RAID array md0
[Tue Jul 16 01:59:39 2024] md: data-check of RAID array md1
[Tue Jul 16 01:59:39 2024] md: data-check of RAID array md2
[Tue Jul 16 01:59:39 2024] md: data-check of RAID array md3
[Tue Jul 16 02:45:53 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000051e1bd3f xid 87924d8e
[Tue Jul 16 05:59:40 2024] md: md0: data-check interrupted.
[Tue Jul 16 05:59:51 2024] md: md1: data-check interrupted.
[Tue Jul 16 06:00:01 2024] md: md2: data-check interrupted.
[Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid b6f12d96
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000056d1aff1 xid 6ad5584a
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000008075849 xid 406ed865
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid 7f81b676
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 26099b1f
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid 7ed4dbf5
[Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid a930d2bf
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 5b099b1f
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b3d4dbf5
[Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid de30d2bf
[Tue Jul 16 11:20:21 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid 4431d2bf
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000007ce5d717 xid 2c364663
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000001bba6d7e xid df31d2bf
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000000be8f11f xid acdab0f5
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d6d182c4 xid 3d172cb9
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000976cd55a xid a6cb0a18
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e11f40dd xid 35f006fd
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000042906e77 xid d9415db0
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000bc03be29 xid eed92785
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000056d1aff1 xid a1d6584a
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000008075849 xid 776fd865
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid edf22d96
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009327f72c xid 12b9ab32
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b55d160f xid 0e3dd152
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000976cd55a xid a7cb0a18
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000042906e77 xid da415db0
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000bc03be29 xid efd92785
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000008075849 xid 786fd865
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000aeeb49cf xid eef22d96
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 9f91a3d2
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000060d5bb55 xid 3aea57c8
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 73a5017a
[Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000155c8644 xid 5d0a9b1f
[Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
[Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
[Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
[Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
[Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
[Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:36:40 2024] Call Trace:
[Tue Jul 16 11:36:40 2024] <IRQ>
[Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:36:40 2024] </IRQ>
[Tue Jul 16 11:36:40 2024] <TASK>
[Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
[Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
[Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
[Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
[Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:36:40 2024] </TASK>
[Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
[Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
[Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
[Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:37:19 2024] Call Trace:
[Tue Jul 16 11:37:19 2024] <IRQ>
[Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:37:19 2024] </IRQ>
[Tue Jul 16 11:37:19 2024] <TASK>
[Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
[Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
[Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:37:19 2024] </TASK>
[Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:37:57 2024] rcu: 15-....: (20996 ticks this GP) idle=35b/1/0x4000000000000000 softirq=29720101/29720101 fqs=5251
[Tue Jul 16 11:37:57 2024] (t=21013 jiffies g=194959005 q=1748)
[Tue Jul 16 11:37:57 2024] Task dump for CPU 15:
[Tue Jul 16 11:37:57 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:37:57 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:37:57 2024] Call Trace:
[Tue Jul 16 11:37:57 2024] <IRQ>
[Tue Jul 16 11:37:57 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:37:57 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:37:57 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:37:57 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:37:57 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:37:57 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:37:57 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:37:57 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:37:57 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:37:57 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:37:57 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:37:57 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:37:57 2024] </IRQ>
[Tue Jul 16 11:37:57 2024] <TASK>
[Tue Jul 16 11:37:57 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:37:57 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 11:37:57 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 11:37:57 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 11:37:57 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002ba0000f
[Tue Jul 16 11:37:57 2024] RDX: 0000000141da4b69 RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 11:37:57 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:37:57 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 11:37:57 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:37:57 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:37:57 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:37:57 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:37:57 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:37:57 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:37:57 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:37:57 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:37:57 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:37:57 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:37:57 2024] kthread+0x115/0x140
[Tue Jul 16 11:37:57 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:37:57 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:37:57 2024] </TASK>
[Tue Jul 16 11:39:07 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:39:07 2024] rcu: 7-....: (20999 ticks this GP) idle=e2d/1/0x4000000000000000 softirq=29984497/29984498 fqs=4618
[Tue Jul 16 11:39:07 2024] (t=21017 jiffies g=194959009 q=3031)
[Tue Jul 16 11:39:07 2024] Task dump for CPU 7:
[Tue Jul 16 11:39:07 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:39:07 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:39:07 2024] Call Trace:
[Tue Jul 16 11:39:07 2024] <IRQ>
[Tue Jul 16 11:39:07 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:39:07 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:39:07 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:39:07 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:39:07 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:39:07 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:39:07 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:39:07 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:39:07 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:39:07 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:39:07 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:39:07 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:39:07 2024] </IRQ>
[Tue Jul 16 11:39:07 2024] <TASK>
[Tue Jul 16 11:39:07 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:39:07 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xc9/0x180 [sunrpc]
[Tue Jul 16 11:39:07 2024] Code: 38 41 83 45 4c 01 f0 80 4b 30 02 4c 89 63 28 49 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78 42 49 8b 45 50 4c 89 70 08 <48> 89 43 60 4c 89 7b 68 4d 89 75 50 48 83 c4 10 5b 5d 41 5c 41 5d
[Tue Jul 16 11:39:07 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000286
[Tue Jul 16 11:39:07 2024] RAX: ffffffffa012c750 RBX: ffff88997131a500 RCX: 000000002fa00007
[Tue Jul 16 11:39:07 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 11:39:07 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:39:07 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141db5f69
[Tue Jul 16 11:39:07 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffffffffa012c750
[Tue Jul 16 11:39:07 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:39:07 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:39:07 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:39:07 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:39:08 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:39:08 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:39:08 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:39:08 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:39:08 2024] kthread+0x115/0x140
[Tue Jul 16 11:39:08 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:39:08 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:39:08 2024] </TASK>
[Tue Jul 16 11:40:11 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:40:11 2024] rcu: 7-....: (84004 ticks this GP) idle=e2d/1/0x4000000000000000 softirq=29984497/29984498 fqs=19763
[Tue Jul 16 11:40:11 2024] (t=84254 jiffies g=194959009 q=7663)
[Tue Jul 16 11:40:11 2024] Task dump for CPU 7:
[Tue Jul 16 11:40:11 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:40:11 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:40:11 2024] Call Trace:
[Tue Jul 16 11:40:11 2024] <IRQ>
[Tue Jul 16 11:40:11 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:40:11 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:40:11 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:40:11 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:40:11 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:40:11 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:40:11 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:40:11 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:40:11 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:40:11 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:40:11 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:40:11 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:40:11 2024] </IRQ>
[Tue Jul 16 11:40:11 2024] <TASK>
[Tue Jul 16 11:40:11 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:40:11 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 11:40:11 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 11:40:11 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 11:40:11 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 11:40:11 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 11:40:11 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:40:11 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 11:40:11 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:40:11 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:40:11 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:40:11 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:40:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:40:11 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:40:11 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:40:11 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:40:11 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:40:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:40:11 2024] kthread+0x115/0x140
[Tue Jul 16 11:40:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:40:11 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:40:11 2024] </TASK>
[Tue Jul 16 11:41:14 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:41:14 2024] rcu: 7-....: (147009 ticks this GP) idle=e2d/1/0x4000000000000000 softirq=29984497/29984498 fqs=35388
[Tue Jul 16 11:41:14 2024] (t=147497 jiffies g=194959009 q=9471)
[Tue Jul 16 11:41:14 2024] Task dump for CPU 7:
[Tue Jul 16 11:41:14 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:41:14 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:41:14 2024] Call Trace:
[Tue Jul 16 11:41:14 2024] <IRQ>
[Tue Jul 16 11:41:14 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:41:14 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:41:14 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:41:14 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:41:14 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:41:14 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:41:14 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:41:14 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:41:14 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:41:14 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:41:14 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:41:14 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:41:14 2024] </IRQ>
[Tue Jul 16 11:41:14 2024] <TASK>
[Tue Jul 16 11:41:14 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:41:14 2024] RIP: 0010:rpc_exit_task+0x89/0x100 [sunrpc]
[Tue Jul 16 11:41:14 2024] Code: 1f 00 48 83 7b 20 00 74 39 48 89 df e8 60 11 ff ff 31 c0 66 81 a3 dc 00 00 00 df f7 66 89 83 de 00 00 00 0f b6 83 e2 00 00 00 <83> e0 c3 83 c8 28 88 83 e2 00 00 00 e8 76 7a 03 e1 48 89 83 d0 00
[Tue Jul 16 11:41:14 2024] RSP: 0018:ffffc900087cfe28 EFLAGS: 00000206
[Tue Jul 16 11:41:14 2024] RAX: 0000000000000029 RBX: ffff88997131a500 RCX: 0000000045600007
[Tue Jul 16 11:41:14 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88997131a500
[Tue Jul 16 11:41:14 2024] RBP: ffffffffa00d3230 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:41:14 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 11:41:14 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:41:14 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:41:14 2024] ? rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 11:41:14 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:41:14 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:41:14 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:41:14 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:41:14 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:41:14 2024] kthread+0x115/0x140
[Tue Jul 16 11:41:14 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:41:14 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:41:14 2024] </TASK>
[Tue Jul 16 11:42:17 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:42:17 2024] rcu: 7-....: (210010 ticks this GP) idle=e2d/1/0x4000000000000000 softirq=29984497/29984498 fqs=51078
[Tue Jul 16 11:42:17 2024] (t=210725 jiffies g=194959009 q=11007)
[Tue Jul 16 11:42:17 2024] Task dump for CPU 7:
[Tue Jul 16 11:42:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:42:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:42:17 2024] Call Trace:
[Tue Jul 16 11:42:17 2024] <IRQ>
[Tue Jul 16 11:42:17 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:42:17 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:42:17 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:42:17 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:42:17 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:42:17 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:42:17 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:42:17 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:42:17 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:42:17 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:42:17 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:42:17 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:42:17 2024] </IRQ>
[Tue Jul 16 11:42:17 2024] <TASK>
[Tue Jul 16 11:42:17 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:42:17 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 11:42:17 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 11:42:17 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:42:17 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 11:42:17 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 11:42:17 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:42:17 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141de3ca8
[Tue Jul 16 11:42:17 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:42:17 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 11:42:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:42:17 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:42:17 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:42:17 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:42:17 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:42:17 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:42:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:42:17 2024] kthread+0x115/0x140
[Tue Jul 16 11:42:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:42:17 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:42:17 2024] </TASK>
[Tue Jul 16 11:43:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:43:40 2024] rcu: 15-....: (21001 ticks this GP) idle=5eb/1/0x4000000000000000 softirq=29720106/29720107 fqs=5156
[Tue Jul 16 11:43:40 2024] (t=21014 jiffies g=194959013 q=12819)
[Tue Jul 16 11:43:40 2024] Task dump for CPU 15:
[Tue Jul 16 11:43:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:43:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:43:40 2024] Call Trace:
[Tue Jul 16 11:43:40 2024] <IRQ>
[Tue Jul 16 11:43:40 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:43:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:43:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:43:40 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:43:40 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:43:40 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:43:40 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:43:40 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:43:40 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:43:40 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:43:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:43:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:43:40 2024] </IRQ>
[Tue Jul 16 11:43:40 2024] <TASK>
[Tue Jul 16 11:43:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:43:40 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 11:43:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 11:43:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:43:40 2024] RAX: 00000000b710a28c RBX: 000000003f48d04a RCX: 000000000000100f
[Tue Jul 16 11:43:40 2024] RDX: 0000000000453736 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 11:43:40 2024] RBP: 0003ed6a9d5785a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:43:40 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 11:43:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:43:40 2024] ktime_get+0x38/0xa0
[Tue Jul 16 11:43:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:43:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:43:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:43:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:43:40 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:43:40 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:43:40 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:43:40 2024] kthread+0x115/0x140
[Tue Jul 16 11:43:40 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:43:40 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:43:40 2024] </TASK>
[Tue Jul 16 11:44:23 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:44:23 2024] rcu: 15-....: (20998 ticks this GP) idle=e03/1/0x4000000000000000 softirq=29720112/29720120 fqs=5251
[Tue Jul 16 11:44:23 2024] (t=21014 jiffies g=194959021 q=2335)
[Tue Jul 16 11:44:23 2024] Task dump for CPU 15:
[Tue Jul 16 11:44:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:44:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:44:23 2024] Call Trace:
[Tue Jul 16 11:44:23 2024] <IRQ>
[Tue Jul 16 11:44:23 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:44:23 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:44:23 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:44:23 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:44:23 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:44:23 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:44:23 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:44:23 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:44:23 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:44:23 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:44:23 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:44:23 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:44:23 2024] </IRQ>
[Tue Jul 16 11:44:23 2024] <TASK>
[Tue Jul 16 11:44:23 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:44:23 2024] RIP: 0010:try_to_grab_pending+0x15/0x150
[Tue Jul 16 11:44:23 2024] Code: 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 d5 53 48 89 fb 48 83 ec 08 9c 58 <fa> 48 89 02 40 84 f6 0f 85 9d 00 00 00 f0 48 0f ba 2b 00 72 0f 31
[Tue Jul 16 11:44:23 2024] RSP: 0018:ffffc900087cfdb0 EFLAGS: 00000282
[Tue Jul 16 11:44:23 2024] RAX: 0000000000000282 RBX: ffffffffa012c768 RCX: 0000000000000017
[Tue Jul 16 11:44:23 2024] RDX: ffffc900087cfdd8 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 11:44:23 2024] RBP: ffffc900087cfdd8 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:44:23 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 11:44:23 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:44:23 2024] __cancel_work+0x37/0xb0
[Tue Jul 16 11:44:23 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:44:23 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:44:23 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:44:23 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:44:23 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:44:23 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:44:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:44:23 2024] kthread+0x115/0x140
[Tue Jul 16 11:44:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:44:23 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:44:23 2024] </TASK>
[Tue Jul 16 11:44:23 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 21236 jiffies s: 21957 root: 0x8000/.
[Tue Jul 16 11:44:23 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 11:44:23 2024] Task dump for CPU 15:
[Tue Jul 16 11:44:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:44:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:44:23 2024] Call Trace:
[Tue Jul 16 11:44:23 2024] <TASK>
[Tue Jul 16 11:44:23 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:44:23 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 11:44:23 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 11:44:23 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:44:23 2024] ? nfsd4_cb_done+0x28c/0x380 [nfsd]
[Tue Jul 16 11:44:23 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 11:44:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:44:23 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:44:23 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 11:44:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:44:23 2024] ? kthread+0x115/0x140
[Tue Jul 16 11:44:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:44:23 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 11:44:23 2024] </TASK>
[Tue Jul 16 11:45:26 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:45:26 2024] rcu: 15-....: (84003 ticks this GP) idle=e03/1/0x4000000000000000 softirq=29720112/29720120 fqs=20992
[Tue Jul 16 11:45:26 2024] (t=84261 jiffies g=194959021 q=4189)
[Tue Jul 16 11:45:26 2024] Task dump for CPU 15:
[Tue Jul 16 11:45:26 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:45:26 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:45:26 2024] Call Trace:
[Tue Jul 16 11:45:26 2024] <IRQ>
[Tue Jul 16 11:45:26 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:45:26 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:45:26 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:45:26 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:45:26 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:45:26 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:45:26 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:45:26 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:45:26 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:45:26 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:45:26 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:45:26 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:45:26 2024] </IRQ>
[Tue Jul 16 11:45:26 2024] <TASK>
[Tue Jul 16 11:45:26 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:45:26 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 11:45:26 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 11:45:26 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:45:26 2024] RAX: 000000007f177804 RBX: 000000003f4c0b58 RCX: 000000000000100f
[Tue Jul 16 11:45:26 2024] RDX: 0000000000453794 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 11:45:26 2024] RBP: 0003ed834b6f29a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:45:26 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
[Tue Jul 16 11:45:26 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:45:26 2024] ktime_get+0x38/0xa0
[Tue Jul 16 11:45:26 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:45:26 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:45:26 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:45:26 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:45:26 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:45:26 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:45:26 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:45:26 2024] kthread+0x115/0x140
[Tue Jul 16 11:45:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:45:26 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:45:26 2024] </TASK>
[Tue Jul 16 11:45:26 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 84603 jiffies s: 21957 root: 0x8000/.
[Tue Jul 16 11:45:26 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 11:45:26 2024] Task dump for CPU 15:
[Tue Jul 16 11:45:26 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:45:26 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:45:26 2024] Call Trace:
[Tue Jul 16 11:45:26 2024] <TASK>
[Tue Jul 16 11:45:26 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 11:45:26 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:45:26 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 11:45:26 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 11:45:26 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 11:45:26 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 11:45:26 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:45:26 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:45:26 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:45:26 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:45:26 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:45:26 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:45:26 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 11:45:26 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:45:26 2024] ? kthread+0x115/0x140
[Tue Jul 16 11:45:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:45:26 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 11:45:26 2024] </TASK>
[Tue Jul 16 11:46:50 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:46:50 2024] rcu: 7-....: (21000 ticks this GP) idle=7cf/1/0x4000000000000000 softirq=29984507/29984507 fqs=5081
[Tue Jul 16 11:46:50 2024] (t=21016 jiffies g=194959025 q=6195)
[Tue Jul 16 11:46:50 2024] Task dump for CPU 7:
[Tue Jul 16 11:46:50 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:46:50 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:46:50 2024] Call Trace:
[Tue Jul 16 11:46:50 2024] <IRQ>
[Tue Jul 16 11:46:50 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:46:50 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:46:50 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:46:50 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:46:50 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:46:50 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:46:50 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:46:50 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:46:50 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:46:50 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:46:50 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:46:50 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:46:50 2024] </IRQ>
[Tue Jul 16 11:46:50 2024] <TASK>
[Tue Jul 16 11:46:50 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:46:50 2024] RIP: 0010:nfsd4_cb_done+0x294/0x380 [nfsd]
[Tue Jul 16 11:46:50 2024] Code: 00 00 8b 05 36 33 04 00 85 c0 0f 8f 93 00 00 00 48 8b 3b e9 6b fe ff ff 4c 89 e7 eb be e8 b4 e8 ee ff 85 c0 0f 84 a0 fd ff ff <5b> 48 89 ef be d0 07 00 00 5d 41 5c 41 5d e9 29 48 f0 ff 83 82 ac
[Tue Jul 16 11:46:50 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000202
[Tue Jul 16 11:46:50 2024] RAX: 0000000000000001 RBX: ffff888e83163d80 RCX: 0000000000000002
[Tue Jul 16 11:46:50 2024] RDX: ffff888473d77a00 RSI: ffff888e83163d80 RDI: ffff88997131a500
[Tue Jul 16 11:46:50 2024] RBP: ffff88997131a500 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:46:50 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff888696daa000
[Tue Jul 16 11:46:50 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:46:50 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:46:50 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:46:50 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:46:50 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:46:50 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:46:50 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:46:50 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:46:50 2024] kthread+0x115/0x140
[Tue Jul 16 11:46:50 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:46:50 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:46:50 2024] </TASK>
[Tue Jul 16 11:47:21 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:47:21 2024] rcu: 15-....: (20996 ticks this GP) idle=e29/1/0x4000000000000000 softirq=29720127/29720127 fqs=4705
[Tue Jul 16 11:47:21 2024] (t=21017 jiffies g=194959029 q=6554)
[Tue Jul 16 11:47:21 2024] Task dump for CPU 15:
[Tue Jul 16 11:47:21 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:47:21 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:47:21 2024] Call Trace:
[Tue Jul 16 11:47:21 2024] <IRQ>
[Tue Jul 16 11:47:21 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:47:21 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:47:21 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:47:21 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:47:21 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:47:21 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:47:21 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:47:21 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:47:21 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:47:21 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:47:21 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:47:21 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:47:21 2024] </IRQ>
[Tue Jul 16 11:47:21 2024] <TASK>
[Tue Jul 16 11:47:21 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:47:21 2024] RIP: 0010:try_to_grab_pending+0x14/0x150
[Tue Jul 16 11:47:21 2024] Code: 0f 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 d5 53 48 89 fb 48 83 ec 08 9c <58> fa 48 89 02 40 84 f6 0f 85 9d 00 00 00 f0 48 0f ba 2b 00 72 0f
[Tue Jul 16 11:47:21 2024] RSP: 0018:ffffc900087cfd50 EFLAGS: 00000292
[Tue Jul 16 11:47:21 2024] RAX: 0000000000000000 RBX: ffffffffa012c768 RCX: 00000000000007d0
[Tue Jul 16 11:47:21 2024] RDX: ffffc900087cfd80 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 11:47:21 2024] RBP: ffffc900087cfd80 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:47:21 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 11:47:21 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:47:21 2024] mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 11:47:21 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:47:21 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:47:21 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:47:21 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:47:21 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:47:21 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:47:21 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:47:21 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:47:21 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:47:21 2024] kthread+0x115/0x140
[Tue Jul 16 11:47:21 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:47:21 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:47:21 2024] </TASK>
[Tue Jul 16 11:48:24 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:48:24 2024] rcu: 15-....: (84001 ticks this GP) idle=e29/1/0x4000000000000000 softirq=29720127/29720127 fqs=20413
[Tue Jul 16 11:48:24 2024] (t=84262 jiffies g=194959029 q=8073)
[Tue Jul 16 11:48:24 2024] Task dump for CPU 15:
[Tue Jul 16 11:48:24 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:48:24 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:48:24 2024] Call Trace:
[Tue Jul 16 11:48:24 2024] <IRQ>
[Tue Jul 16 11:48:24 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:48:24 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:48:24 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:48:24 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:48:24 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:48:24 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:48:24 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:48:24 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:48:24 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:48:24 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:48:24 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:48:24 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:48:24 2024] </IRQ>
[Tue Jul 16 11:48:24 2024] <TASK>
[Tue Jul 16 11:48:24 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:48:24 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 11:48:24 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 11:48:24 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 11:48:24 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004fa0000f
[Tue Jul 16 11:48:24 2024] RDX: 0000000141e3de79 RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 11:48:24 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:48:24 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 11:48:24 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:48:24 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:48:24 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:48:24 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:48:24 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:48:24 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:48:24 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:48:24 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:48:24 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:48:24 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:48:24 2024] kthread+0x115/0x140
[Tue Jul 16 11:48:24 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:48:24 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:48:24 2024] </TASK>
[Tue Jul 16 11:49:25 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:49:25 2024] rcu: 15-....: (21000 ticks this GP) idle=659/1/0x4000000000000000 softirq=29720134/29720135 fqs=5251
[Tue Jul 16 11:49:25 2024] (t=21013 jiffies g=194959037 q=3685)
[Tue Jul 16 11:49:25 2024] Task dump for CPU 15:
[Tue Jul 16 11:49:25 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:49:25 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:49:25 2024] Call Trace:
[Tue Jul 16 11:49:25 2024] <IRQ>
[Tue Jul 16 11:49:25 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:49:25 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:49:25 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:49:25 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:49:25 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:49:25 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:49:25 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:49:25 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:49:25 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:49:25 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:49:25 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:49:25 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:49:25 2024] </IRQ>
[Tue Jul 16 11:49:25 2024] <TASK>
[Tue Jul 16 11:49:25 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:49:25 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 11:49:25 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 11:49:25 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 11:49:25 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002ba0000f
[Tue Jul 16 11:49:25 2024] RDX: 0000000141e4cb5a RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 11:49:25 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:49:25 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 11:49:25 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:49:25 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:49:25 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:49:25 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:49:25 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:49:25 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:49:25 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:49:25 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:49:25 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:49:25 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:49:25 2024] kthread+0x115/0x140
[Tue Jul 16 11:49:25 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:49:25 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:49:25 2024] </TASK>
[Tue Jul 16 11:49:46 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:49:46 2024] rcu: 7-....: (21000 ticks this GP) idle=0ab/1/0x4000000000000000 softirq=29984521/29984522 fqs=5251
[Tue Jul 16 11:49:46 2024] (t=21013 jiffies g=194959041 q=1938)
[Tue Jul 16 11:49:46 2024] Task dump for CPU 7:
[Tue Jul 16 11:49:46 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:49:46 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:49:46 2024] Call Trace:
[Tue Jul 16 11:49:46 2024] <IRQ>
[Tue Jul 16 11:49:46 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:49:46 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:49:46 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:49:46 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:49:46 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:49:46 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:49:46 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:49:46 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:49:46 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:49:46 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:49:46 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:49:46 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:49:46 2024] </IRQ>
[Tue Jul 16 11:49:46 2024] <TASK>
[Tue Jul 16 11:49:46 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:49:46 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 11:49:46 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 11:49:46 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 11:49:46 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000034200007
[Tue Jul 16 11:49:46 2024] RDX: 0000000141e51f5a RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 11:49:46 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:49:46 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 11:49:46 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:49:46 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:49:46 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:49:46 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:49:46 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:49:46 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:49:46 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:49:46 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:49:46 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:49:46 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:49:46 2024] kthread+0x115/0x140
[Tue Jul 16 11:49:46 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:49:46 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:49:46 2024] </TASK>
[Tue Jul 16 11:50:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:50:57 2024] rcu: 13-....: (21000 ticks this GP) idle=2fd/1/0x4000000000000000 softirq=31904984/31904984 fqs=5251
[Tue Jul 16 11:50:57 2024] (t=21013 jiffies g=194959049 q=5348)
[Tue Jul 16 11:50:57 2024] Task dump for CPU 13:
[Tue Jul 16 11:50:57 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:50:57 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:50:57 2024] Call Trace:
[Tue Jul 16 11:50:57 2024] <IRQ>
[Tue Jul 16 11:50:57 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:50:57 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:50:57 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:50:57 2024] ? trigger_load_balance+0x24e/0x300
[Tue Jul 16 11:50:57 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:50:57 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:50:57 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:50:57 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:50:57 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:50:57 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:50:57 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:50:57 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:50:57 2024] </IRQ>
[Tue Jul 16 11:50:57 2024] <TASK>
[Tue Jul 16 11:50:57 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:50:57 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 11:50:57 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 11:50:57 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 11:50:57 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000023a0000d
[Tue Jul 16 11:50:57 2024] RDX: 0000000141e6335a RSI: 0000000000000046 RDI: ffff88a07fd9b880
[Tue Jul 16 11:50:57 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:50:57 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 11:50:57 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 11:50:57 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:50:57 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:50:57 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:50:57 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:50:57 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:50:57 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:50:57 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:50:57 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:50:57 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:50:57 2024] kthread+0x115/0x140
[Tue Jul 16 11:50:57 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:50:57 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:50:57 2024] </TASK>
[Tue Jul 16 11:52:00 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:52:00 2024] rcu: 13-....: (84005 ticks this GP) idle=2fd/1/0x4000000000000000 softirq=31904984/31904984 fqs=20759
[Tue Jul 16 11:52:00 2024] (t=84259 jiffies g=194959049 q=6854)
[Tue Jul 16 11:52:00 2024] Task dump for CPU 13:
[Tue Jul 16 11:52:00 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:52:00 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:52:00 2024] Call Trace:
[Tue Jul 16 11:52:00 2024] <IRQ>
[Tue Jul 16 11:52:00 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:52:00 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:52:00 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:52:00 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:52:00 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:52:00 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:52:00 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:52:00 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:52:00 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:52:00 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:52:00 2024] </IRQ>
[Tue Jul 16 11:52:00 2024] <TASK>
[Tue Jul 16 11:52:00 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:52:00 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 11:52:00 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 11:52:00 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 11:52:00 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 11:52:00 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd9b880
[Tue Jul 16 11:52:00 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:52:00 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 11:52:00 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:52:00 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:52:00 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:52:00 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:52:00 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:52:00 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:52:00 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:52:00 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:52:00 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:52:00 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:52:00 2024] kthread+0x115/0x140
[Tue Jul 16 11:52:00 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:52:00 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:52:00 2024] </TASK>
[Tue Jul 16 11:53:11 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:53:11 2024] rcu: 5-....: (20999 ticks this GP) idle=78d/1/0x4000000000000000 softirq=30126105/30126105 fqs=5198
[Tue Jul 16 11:53:11 2024] (t=21016 jiffies g=194959053 q=4411)
[Tue Jul 16 11:53:11 2024] Task dump for CPU 5:
[Tue Jul 16 11:53:11 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:53:11 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:53:11 2024] Call Trace:
[Tue Jul 16 11:53:11 2024] <IRQ>
[Tue Jul 16 11:53:11 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:53:11 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:53:11 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:53:11 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:53:11 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:53:11 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:53:11 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:53:11 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:53:11 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:53:11 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:53:11 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:53:11 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:53:11 2024] </IRQ>
[Tue Jul 16 11:53:11 2024] <TASK>
[Tue Jul 16 11:53:11 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:53:11 2024] RIP: 0010:__rpc_do_wake_up_task_on_wq+0xb1/0x1a0 [sunrpc]
[Tue Jul 16 11:53:11 2024] Code: 48 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 48 89 46 40 48 83 c0 22 48 89 46 48 83 6b 4c 01 48 83 c4 08 48 89 ef 5b 5d <e9> ca c2 ff ff 48 85 c0 0f 84 ac 00 00 00 48 8b 4e 50 48 8d 56 50
[Tue Jul 16 11:53:11 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000282
[Tue Jul 16 11:53:11 2024] RAX: dead000000000122 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 11:53:11 2024] RDX: ffffffffa012c708 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 11:53:11 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:53:11 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 11:53:11 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:53:11 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:53:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:53:11 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:53:11 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:53:11 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:53:11 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:53:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:53:11 2024] kthread+0x115/0x140
[Tue Jul 16 11:53:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:53:11 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:53:11 2024] </TASK>
[Tue Jul 16 11:54:14 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:54:14 2024] rcu: 5-....: (84002 ticks this GP) idle=78d/1/0x4000000000000000 softirq=30126105/30126105 fqs=20952
[Tue Jul 16 11:54:14 2024] (t=84246 jiffies g=194959053 q=6066)
[Tue Jul 16 11:54:14 2024] Task dump for CPU 5:
[Tue Jul 16 11:54:14 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:54:14 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:54:14 2024] Call Trace:
[Tue Jul 16 11:54:14 2024] <IRQ>
[Tue Jul 16 11:54:14 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:54:14 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:54:14 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:54:14 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:54:14 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:54:14 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:54:14 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:54:14 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:54:14 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:54:14 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:54:14 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:54:14 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:54:14 2024] </IRQ>
[Tue Jul 16 11:54:14 2024] <TASK>
[Tue Jul 16 11:54:14 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:54:14 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0x89/0x180 [sunrpc]
[Tue Jul 16 11:54:14 2024] Code: 05 00 49 89 fd 85 c0 0f 8f d0 00 00 00 4c 8d 73 60 4c 89 73 60 4c 89 73 68 41 0f b6 45 48 84 c0 75 67 49 8b 45 10 48 8d 53 40 <49> 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b
[Tue Jul 16 11:54:14 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000246
[Tue Jul 16 11:54:14 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 11:54:14 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 11:54:14 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:54:14 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141e9365a
[Tue Jul 16 11:54:14 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 11:54:14 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:54:14 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:54:14 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:54:14 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:54:14 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:54:14 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:54:14 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:54:14 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:54:14 2024] kthread+0x115/0x140
[Tue Jul 16 11:54:14 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:54:15 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:54:15 2024] </TASK>
[Tue Jul 16 11:54:45 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:54:45 2024] rcu: 7-....: (20999 ticks this GP) idle=cf1/1/0x4000000000000000 softirq=29984537/29984537 fqs=5251
[Tue Jul 16 11:54:45 2024] (t=21013 jiffies g=194959057 q=6053)
[Tue Jul 16 11:54:45 2024] Task dump for CPU 7:
[Tue Jul 16 11:54:45 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:54:45 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:54:45 2024] Call Trace:
[Tue Jul 16 11:54:45 2024] <IRQ>
[Tue Jul 16 11:54:45 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:54:45 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:54:45 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:54:45 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:54:45 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:54:45 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:54:45 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:54:45 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:54:45 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:54:45 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:54:45 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:54:45 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:54:45 2024] </IRQ>
[Tue Jul 16 11:54:45 2024] <TASK>
[Tue Jul 16 11:54:45 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:54:45 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0x20/0x180 [sunrpc]
[Tue Jul 16 11:54:45 2024] Code: 1f 84 00 00 00 00 00 0f 1f 00 0f 1f 44 00 00 48 8b 46 30 48 d1 e8 83 e0 01 0f 85 20 01 00 00 41 57 41 56 41 55 41 54 49 89 d4 <55> 48 89 c5 53 48 89 f3 48 83 ec 10 48 c7 44 24 08 00 00 00 00 48
[Tue Jul 16 11:54:45 2024] RSP: 0018:ffffc900087cfde0 EFLAGS: 00000246
[Tue Jul 16 11:54:45 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 11:54:45 2024] RDX: 0000000141e9af5a RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 11:54:45 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:54:45 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141e9af5a
[Tue Jul 16 11:54:45 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:54:45 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:54:45 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:54:45 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:54:45 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:54:45 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:54:45 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:54:45 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:54:45 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:54:45 2024] kthread+0x115/0x140
[Tue Jul 16 11:54:45 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:54:45 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:54:45 2024] </TASK>
[Tue Jul 16 11:55:15 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:55:15 2024] rcu: 7-....: (21000 ticks this GP) idle=cd9/1/0x4000000000000000 softirq=29984541/29984541 fqs=4950
[Tue Jul 16 11:55:15 2024] (t=21017 jiffies g=194959065 q=1681)
[Tue Jul 16 11:55:15 2024] Task dump for CPU 7:
[Tue Jul 16 11:55:15 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:55:15 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:55:15 2024] Call Trace:
[Tue Jul 16 11:55:15 2024] <IRQ>
[Tue Jul 16 11:55:15 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:55:15 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:55:15 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:55:15 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:55:15 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:55:15 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:55:15 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:55:15 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:55:15 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:55:15 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:55:15 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:55:15 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:55:15 2024] </IRQ>
[Tue Jul 16 11:55:15 2024] <TASK>
[Tue Jul 16 11:55:15 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:55:15 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 11:55:15 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 11:55:15 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:55:15 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 11:55:15 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 11:55:15 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:55:15 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141ea1b89
[Tue Jul 16 11:55:15 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:55:15 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 11:55:15 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:55:15 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:55:15 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:55:15 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:55:15 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:55:15 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:55:15 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:55:15 2024] kthread+0x115/0x140
[Tue Jul 16 11:55:15 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:55:15 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:55:15 2024] </TASK>
[Tue Jul 16 11:55:36 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:55:36 2024] rcu: 15-....: (21000 ticks this GP) idle=997/1/0x4000000000000000 softirq=29720163/29720164 fqs=5245
[Tue Jul 16 11:55:36 2024] (t=21014 jiffies g=194959069 q=1395)
[Tue Jul 16 11:55:36 2024] Task dump for CPU 15:
[Tue Jul 16 11:55:36 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:55:36 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:55:36 2024] Call Trace:
[Tue Jul 16 11:55:36 2024] <IRQ>
[Tue Jul 16 11:55:36 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:55:36 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:55:37 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:55:37 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:55:37 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:55:37 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:55:37 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:55:37 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:55:37 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:55:37 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:55:37 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:55:37 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:55:37 2024] </IRQ>
[Tue Jul 16 11:55:37 2024] <TASK>
[Tue Jul 16 11:55:37 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:55:37 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 11:55:37 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 11:55:37 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 11:55:37 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 11:55:37 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 11:55:37 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:55:37 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141ea6f8a
[Tue Jul 16 11:55:37 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:55:37 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 11:55:37 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:55:37 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:55:37 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:55:37 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:55:37 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:55:37 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:55:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:55:37 2024] kthread+0x115/0x140
[Tue Jul 16 11:55:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:55:37 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:55:37 2024] </TASK>
[Tue Jul 16 11:56:32 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:56:32 2024] rcu: 15-....: (20997 ticks this GP) idle=1b1/1/0x4000000000000000 softirq=29720172/29720172 fqs=5251
[Tue Jul 16 11:56:32 2024] (t=21013 jiffies g=194959077 q=2071)
[Tue Jul 16 11:56:32 2024] Task dump for CPU 15:
[Tue Jul 16 11:56:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:56:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:56:32 2024] Call Trace:
[Tue Jul 16 11:56:32 2024] <IRQ>
[Tue Jul 16 11:56:32 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:56:32 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:56:32 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:56:32 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:56:32 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:56:32 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:56:32 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:56:32 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:56:32 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:56:32 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:56:32 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:56:32 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:56:32 2024] </IRQ>
[Tue Jul 16 11:56:32 2024] <TASK>
[Tue Jul 16 11:56:32 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:56:32 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 11:56:32 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 11:56:32 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 11:56:32 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 11:56:32 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 11:56:32 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:56:32 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 11:56:32 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:56:32 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:56:32 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:56:32 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:56:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:56:32 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:56:32 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:56:32 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:56:32 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:56:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:56:32 2024] kthread+0x115/0x140
[Tue Jul 16 11:56:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:56:32 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:56:32 2024] </TASK>
[Tue Jul 16 11:56:32 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 21236 jiffies s: 21961 root: 0x8000/.
[Tue Jul 16 11:56:32 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 11:56:32 2024] Task dump for CPU 15:
[Tue Jul 16 11:56:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:56:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:56:32 2024] Call Trace:
[Tue Jul 16 11:56:32 2024] <TASK>
[Tue Jul 16 11:56:32 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:56:32 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 11:56:32 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:56:32 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 11:56:32 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 11:56:32 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:56:32 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:56:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:56:32 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:56:32 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:56:32 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:56:32 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 11:56:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:56:32 2024] ? kthread+0x115/0x140
[Tue Jul 16 11:56:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:56:32 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 11:56:32 2024] </TASK>
[Tue Jul 16 11:57:35 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:57:35 2024] rcu: 15-....: (84002 ticks this GP) idle=1b1/1/0x4000000000000000 softirq=29720172/29720172 fqs=20741
[Tue Jul 16 11:57:35 2024] (t=84261 jiffies g=194959077 q=5125)
[Tue Jul 16 11:57:35 2024] Task dump for CPU 15:
[Tue Jul 16 11:57:35 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:57:35 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:57:35 2024] Call Trace:
[Tue Jul 16 11:57:35 2024] <IRQ>
[Tue Jul 16 11:57:35 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:57:35 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:57:35 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:57:35 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 11:57:35 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:57:35 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:57:35 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:57:35 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:57:35 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:57:35 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:57:35 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:57:35 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:57:35 2024] </IRQ>
[Tue Jul 16 11:57:35 2024] <TASK>
[Tue Jul 16 11:57:35 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:57:35 2024] RIP: 0010:read_tsc+0xc/0x20
[Tue Jul 16 11:57:35 2024] Code: 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 66 90 48 c1 e2 20 48 09 d0 <c3> cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 8b 05
[Tue Jul 16 11:57:35 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000202
[Tue Jul 16 11:57:35 2024] RAX: 00453a18096339ba RBX: 000000003f6239b2 RCX: 000000000000100f
[Tue Jul 16 11:57:35 2024] RDX: 00453a1800000000 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 11:57:35 2024] RBP: 0003ee2d073863a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:57:35 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 11:57:35 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 11:57:35 2024] ktime_get+0x38/0xa0
[Tue Jul 16 11:57:35 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:57:35 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:57:35 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:57:35 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:57:35 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:57:35 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:57:35 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:57:35 2024] kthread+0x115/0x140
[Tue Jul 16 11:57:35 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:57:35 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:57:35 2024] </TASK>
[Tue Jul 16 11:57:35 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 84603 jiffies s: 21961 root: 0x8000/.
[Tue Jul 16 11:57:35 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 11:57:35 2024] Task dump for CPU 15:
[Tue Jul 16 11:57:35 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:57:35 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:57:35 2024] Call Trace:
[Tue Jul 16 11:57:35 2024] <TASK>
[Tue Jul 16 11:57:35 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 11:57:35 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:57:35 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 11:57:35 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 11:57:35 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:57:35 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:57:35 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 11:57:35 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:57:35 2024] ? kthread+0x115/0x140
[Tue Jul 16 11:57:35 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:57:35 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 11:57:35 2024] </TASK>
[Tue Jul 16 11:58:38 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 11:58:38 2024] rcu: 15-....: (147007 ticks this GP) idle=1b1/1/0x4000000000000000 softirq=29720172/29720172 fqs=34858
[Tue Jul 16 11:58:38 2024] (t=147494 jiffies g=194959077 q=7026)
[Tue Jul 16 11:58:38 2024] Task dump for CPU 15:
[Tue Jul 16 11:58:38 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:58:38 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:58:38 2024] Call Trace:
[Tue Jul 16 11:58:38 2024] <IRQ>
[Tue Jul 16 11:58:38 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 11:58:38 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 11:58:38 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 11:58:38 2024] ? trigger_load_balance+0x24e/0x300
[Tue Jul 16 11:58:38 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 11:58:38 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 11:58:38 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 11:58:38 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 11:58:38 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 11:58:38 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 11:58:38 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 11:58:38 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 11:58:38 2024] </IRQ>
[Tue Jul 16 11:58:38 2024] <TASK>
[Tue Jul 16 11:58:38 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 11:58:38 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xae/0x180 [sunrpc]
[Tue Jul 16 11:58:38 2024] Code: 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 4c 89 63 28 <49> 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78 42 49 8b 45
[Tue Jul 16 11:58:38 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 11:58:38 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 11:58:38 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 11:58:38 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 11:58:38 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141ed3d68
[Tue Jul 16 11:58:38 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 11:58:38 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:58:38 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:58:38 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 11:58:38 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 11:58:38 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:58:38 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:58:38 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 11:58:38 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:58:38 2024] kthread+0x115/0x140
[Tue Jul 16 11:58:38 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:58:38 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 11:58:38 2024] </TASK>
[Tue Jul 16 11:58:41 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 150139 jiffies s: 21961 root: 0x8000/.
[Tue Jul 16 11:58:41 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 11:58:41 2024] Task dump for CPU 15:
[Tue Jul 16 11:58:41 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 11:58:41 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 11:58:41 2024] Call Trace:
[Tue Jul 16 11:58:41 2024] <TASK>
[Tue Jul 16 11:58:41 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 11:58:41 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:58:41 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 11:58:41 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 11:58:41 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 11:58:41 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 11:58:41 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 11:58:41 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 11:58:41 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 11:58:41 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 11:58:41 2024] ? kthread+0x115/0x140
[Tue Jul 16 11:58:41 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 11:58:41 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 11:58:41 2024] </TASK>
[Tue Jul 16 12:00:00 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:00:00 2024] rcu: 7-....: (20998 ticks this GP) idle=f6d/1/0x4000000000000000 softirq=29984552/29984552 fqs=5251
[Tue Jul 16 12:00:00 2024] (t=21013 jiffies g=194959081 q=12726)
[Tue Jul 16 12:00:00 2024] Task dump for CPU 7:
[Tue Jul 16 12:00:00 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:00:00 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:00:00 2024] Call Trace:
[Tue Jul 16 12:00:00 2024] <IRQ>
[Tue Jul 16 12:00:00 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:00:00 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:00:00 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:00:00 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:00:00 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:00:00 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:00:00 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:00:00 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:00:00 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:00:00 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:00:00 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:00:00 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:00:00 2024] </IRQ>
[Tue Jul 16 12:00:00 2024] <TASK>
[Tue Jul 16 12:00:00 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:00:00 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0x3c/0x180 [sunrpc]
[Tue Jul 16 12:00:00 2024] Code: 01 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 c5 53 48 89 f3 48 83 ec 10 48 c7 44 24 08 00 00 00 00 48 8b 05 14 6f 33 e2 <48> 39 d0 78 1a c7 46 04 92 ff ff ff 48 83 c4 10 5b 5d 41 5c 41 5d
[Tue Jul 16 12:00:00 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000286
[Tue Jul 16 12:00:00 2024] RAX: 0000000141ee739a RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:00:00 2024] RDX: 0000000141ee7b6a RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:00:00 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:00:00 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141ee7b6a
[Tue Jul 16 12:00:00 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:00:00 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:00:00 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:00:00 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:00:00 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:00:00 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:00:00 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:00:00 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:00:00 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:00:00 2024] kthread+0x115/0x140
[Tue Jul 16 12:00:00 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:00:00 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:00:00 2024] </TASK>
[Tue Jul 16 12:00:25 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:00:25 2024] rcu: 15-....: (20999 ticks this GP) idle=9df/1/0x4000000000000000 softirq=29720177/29720177 fqs=5251
[Tue Jul 16 12:00:25 2024] (t=21013 jiffies g=194959085 q=12997)
[Tue Jul 16 12:00:25 2024] Task dump for CPU 15:
[Tue Jul 16 12:00:25 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:00:25 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:00:25 2024] Call Trace:
[Tue Jul 16 12:00:25 2024] <IRQ>
[Tue Jul 16 12:00:25 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:00:25 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:00:25 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:00:25 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:00:25 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:00:25 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:00:25 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:00:25 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:00:25 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:00:25 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:00:25 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:00:25 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:00:25 2024] </IRQ>
[Tue Jul 16 12:00:25 2024] <TASK>
[Tue Jul 16 12:00:25 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:00:25 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xaa/0x180 [sunrpc]
[Tue Jul 16 12:00:25 2024] Code: 67 49 8b 45 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 <4c> 89 63 28 49 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78
[Tue Jul 16 12:00:25 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 12:00:25 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 12:00:25 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:00:25 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:00:25 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141eedf69
[Tue Jul 16 12:00:25 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 12:00:25 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:00:25 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:00:25 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:00:25 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:00:25 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:00:25 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:00:25 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:00:25 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:00:25 2024] kthread+0x115/0x140
[Tue Jul 16 12:00:25 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:00:25 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:00:25 2024] </TASK>
[Tue Jul 16 12:01:28 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:01:28 2024] rcu: 15-....: (84003 ticks this GP) idle=9df/1/0x4000000000000000 softirq=29720177/29720177 fqs=21006
[Tue Jul 16 12:01:28 2024] (t=84253 jiffies g=194959085 q=14713)
[Tue Jul 16 12:01:28 2024] Task dump for CPU 15:
[Tue Jul 16 12:01:28 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:01:28 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:01:29 2024] Call Trace:
[Tue Jul 16 12:01:29 2024] <IRQ>
[Tue Jul 16 12:01:29 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:01:29 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:01:29 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:01:29 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:01:29 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:01:29 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:01:29 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:01:29 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:01:29 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:01:29 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:01:29 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:01:29 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:01:29 2024] </IRQ>
[Tue Jul 16 12:01:29 2024] <TASK>
[Tue Jul 16 12:01:29 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:01:29 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xaa/0x180 [sunrpc]
[Tue Jul 16 12:01:29 2024] Code: 67 49 8b 45 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 <4c> 89 63 28 49 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78
[Tue Jul 16 12:01:29 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 12:01:29 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 12:01:29 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:01:29 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:01:29 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141efd66f
[Tue Jul 16 12:01:29 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 12:01:29 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:01:29 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:01:29 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:01:29 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:01:29 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:01:29 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:01:29 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:01:29 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:01:29 2024] kthread+0x115/0x140
[Tue Jul 16 12:01:29 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:01:29 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:01:29 2024] </TASK>
[Tue Jul 16 12:01:52 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:01:52 2024] rcu: 7-....: (20999 ticks this GP) idle=833/1/0x4000000000000000 softirq=29984560/29984562 fqs=4764
[Tue Jul 16 12:01:52 2024] (t=21017 jiffies g=194959089 q=14392)
[Tue Jul 16 12:01:52 2024] Task dump for CPU 7:
[Tue Jul 16 12:01:52 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:01:52 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:01:52 2024] Call Trace:
[Tue Jul 16 12:01:52 2024] <IRQ>
[Tue Jul 16 12:01:52 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:01:52 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:01:52 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:01:52 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:01:52 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:01:52 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:01:52 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:01:52 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:01:52 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:01:52 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:01:52 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:01:52 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:01:52 2024] </IRQ>
[Tue Jul 16 12:01:52 2024] <TASK>
[Tue Jul 16 12:01:52 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:01:52 2024] RIP: 0010:try_to_grab_pending+0x14/0x150
[Tue Jul 16 12:01:52 2024] Code: 0f 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 d5 53 48 89 fb 48 83 ec 08 9c <58> fa 48 89 02 40 84 f6 0f 85 9d 00 00 00 f0 48 0f ba 2b 00 72 0f
[Tue Jul 16 12:01:52 2024] RSP: 0018:ffffc900087cfd50 EFLAGS: 00000292
[Tue Jul 16 12:01:52 2024] RAX: 0000000000000000 RBX: ffffffffa012c768 RCX: 00000000000007d0
[Tue Jul 16 12:01:52 2024] RDX: ffffc900087cfd80 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 12:01:52 2024] RBP: ffffc900087cfd80 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:01:52 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:01:52 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:01:52 2024] mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:01:52 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:01:52 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:01:52 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:01:52 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:01:52 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:01:52 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:01:52 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:01:52 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:01:52 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:01:52 2024] kthread+0x115/0x140
[Tue Jul 16 12:01:52 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:01:53 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:01:53 2024] </TASK>
[Tue Jul 16 12:02:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:02:19 2024] rcu: 15-....: (21000 ticks this GP) idle=a33/1/0x4000000000000000 softirq=29720184/29720185 fqs=4610
[Tue Jul 16 12:02:19 2024] (t=21017 jiffies g=194959093 q=12378)
[Tue Jul 16 12:02:19 2024] Task dump for CPU 15:
[Tue Jul 16 12:02:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:02:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:02:19 2024] Call Trace:
[Tue Jul 16 12:02:19 2024] <IRQ>
[Tue Jul 16 12:02:19 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:02:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:02:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:02:19 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:02:19 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:02:19 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:02:19 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:02:19 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:02:19 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:02:19 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:02:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:02:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:02:19 2024] </IRQ>
[Tue Jul 16 12:02:19 2024] <TASK>
[Tue Jul 16 12:02:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:02:19 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:02:19 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:02:19 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:02:19 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002ba0000f
[Tue Jul 16 12:02:19 2024] RDX: 0000000141f09b5a RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 12:02:19 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:02:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:02:19 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:02:19 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:02:19 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:02:19 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:02:19 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:02:19 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:02:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:02:19 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:02:19 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:02:19 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:02:19 2024] kthread+0x115/0x140
[Tue Jul 16 12:02:19 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:02:19 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:02:19 2024] </TASK>
[Tue Jul 16 12:03:04 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:03:04 2024] rcu: 7-....: (20998 ticks this GP) idle=89d/1/0x4000000000000000 softirq=29984569/29984570 fqs=4560
[Tue Jul 16 12:03:04 2024] (t=21017 jiffies g=194959097 q=10822)
[Tue Jul 16 12:03:04 2024] Task dump for CPU 7:
[Tue Jul 16 12:03:04 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:03:04 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:03:04 2024] Call Trace:
[Tue Jul 16 12:03:04 2024] <IRQ>
[Tue Jul 16 12:03:04 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:03:04 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:03:04 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:03:04 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:03:04 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:03:04 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:03:04 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:03:04 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:03:04 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:03:04 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:03:04 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:03:04 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:03:04 2024] </IRQ>
[Tue Jul 16 12:03:04 2024] <TASK>
[Tue Jul 16 12:03:04 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:03:04 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:03:04 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:03:04 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:03:04 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 12:03:04 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 12:03:04 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:03:04 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141f14389
[Tue Jul 16 12:03:04 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:03:04 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 12:03:04 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:03:04 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:03:04 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:03:04 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:03:04 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:03:04 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:03:04 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:03:04 2024] kthread+0x115/0x140
[Tue Jul 16 12:03:04 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:03:04 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:03:04 2024] </TASK>
[Tue Jul 16 12:04:07 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:04:07 2024] rcu: 7-....: (84003 ticks this GP) idle=89d/1/0x4000000000000000 softirq=29984569/29984570 fqs=18997
[Tue Jul 16 12:04:07 2024] (t=84253 jiffies g=194959097 q=13959)
[Tue Jul 16 12:04:07 2024] Task dump for CPU 7:
[Tue Jul 16 12:04:07 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:04:07 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:04:07 2024] Call Trace:
[Tue Jul 16 12:04:07 2024] <IRQ>
[Tue Jul 16 12:04:07 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:04:07 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:04:07 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:04:07 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:04:07 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:04:07 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:04:07 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:04:07 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:04:07 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:04:07 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:04:07 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:04:07 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:04:07 2024] </IRQ>
[Tue Jul 16 12:04:07 2024] <TASK>
[Tue Jul 16 12:04:07 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:04:07 2024] RIP: 0010:try_to_grab_pending+0x15/0x150
[Tue Jul 16 12:04:07 2024] Code: 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 d5 53 48 89 fb 48 83 ec 08 9c 58 <fa> 48 89 02 40 84 f6 0f 85 9d 00 00 00 f0 48 0f ba 2b 00 72 0f 31
[Tue Jul 16 12:04:07 2024] RSP: 0018:ffffc900087cfd58 EFLAGS: 00000292
[Tue Jul 16 12:04:07 2024] RAX: 0000000000000292 RBX: ffffffffa012c768 RCX: 00000000000007d0
[Tue Jul 16 12:04:07 2024] RDX: ffffc900087cfd80 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 12:04:07 2024] RBP: ffffc900087cfd80 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:04:07 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:04:07 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:04:07 2024] mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:04:07 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:04:07 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:04:07 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:04:07 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:04:07 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:04:07 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:04:07 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:04:07 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:04:07 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:04:07 2024] kthread+0x115/0x140
[Tue Jul 16 12:04:07 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:04:07 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:04:07 2024] </TASK>
[Tue Jul 16 12:05:10 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:05:10 2024] rcu: 7-....: (147008 ticks this GP) idle=89d/1/0x4000000000000000 softirq=29984569/29984570 fqs=33184
[Tue Jul 16 12:05:10 2024] (t=147501 jiffies g=194959097 q=15967)
[Tue Jul 16 12:05:10 2024] Task dump for CPU 7:
[Tue Jul 16 12:05:10 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:05:10 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:05:10 2024] Call Trace:
[Tue Jul 16 12:05:10 2024] <IRQ>
[Tue Jul 16 12:05:10 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:05:10 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:05:10 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:05:10 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:05:10 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:05:10 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:05:10 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:05:10 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:05:10 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:05:10 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:05:10 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:05:11 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:05:11 2024] </IRQ>
[Tue Jul 16 12:05:11 2024] <TASK>
[Tue Jul 16 12:05:11 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:05:11 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:05:11 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:05:11 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:05:11 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004d200007
[Tue Jul 16 12:05:11 2024] RDX: 0000000141f3396d RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:05:11 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:05:11 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:05:11 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:05:11 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:05:11 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:05:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:05:11 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:05:11 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:05:11 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:05:11 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:05:11 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:05:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:05:11 2024] kthread+0x115/0x140
[Tue Jul 16 12:05:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:05:11 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:05:11 2024] </TASK>
[Tue Jul 16 12:06:14 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:06:14 2024] rcu: 7-....: (210013 ticks this GP) idle=89d/1/0x4000000000000000 softirq=29984569/29984570 fqs=46828
[Tue Jul 16 12:06:14 2024] (t=210745 jiffies g=194959097 q=17775)
[Tue Jul 16 12:06:14 2024] Task dump for CPU 7:
[Tue Jul 16 12:06:14 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:06:14 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:06:14 2024] Call Trace:
[Tue Jul 16 12:06:14 2024] <IRQ>
[Tue Jul 16 12:06:14 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:06:14 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:06:14 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:06:14 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:06:14 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:06:14 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:06:14 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:06:14 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:06:14 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:06:14 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:06:14 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:06:14 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:06:14 2024] </IRQ>
[Tue Jul 16 12:06:14 2024] <TASK>
[Tue Jul 16 12:06:14 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:06:14 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:06:14 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:06:14 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:06:14 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:06:14 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:06:14 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:06:14 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:06:14 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:06:14 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:06:14 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:06:14 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:06:14 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:06:14 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:06:14 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:06:14 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:06:14 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:06:14 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:06:14 2024] kthread+0x115/0x140
[Tue Jul 16 12:06:14 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:06:14 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:06:14 2024] </TASK>
[Tue Jul 16 12:07:21 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:07:21 2024] rcu: 3-....: (20996 ticks this GP) idle=0f9/1/0x4000000000000000 softirq=31469352/31469361 fqs=4623
[Tue Jul 16 12:07:21 2024] (t=21014 jiffies g=194959101 q=11723)
[Tue Jul 16 12:07:21 2024] Task dump for CPU 3:
[Tue Jul 16 12:07:21 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:07:21 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:07:21 2024] Call Trace:
[Tue Jul 16 12:07:21 2024] <IRQ>
[Tue Jul 16 12:07:21 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:07:21 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:07:21 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:07:21 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:07:21 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:07:21 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:07:21 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:07:21 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:07:21 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:07:21 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:07:21 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:07:21 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:07:21 2024] </IRQ>
[Tue Jul 16 12:07:21 2024] <TASK>
[Tue Jul 16 12:07:21 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:07:21 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:07:21 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:07:21 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:07:21 2024] RAX: 0000000052f6b2e8 RBX: 000000003f7415d8 RCX: 0000000000001003
[Tue Jul 16 12:07:21 2024] RDX: 0000000000453c1d RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:07:21 2024] RBP: 0003eeb5778ac7a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:07:21 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 12:07:21 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:07:21 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:07:21 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:07:21 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:07:21 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:07:21 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:07:21 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:07:21 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:07:21 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:07:21 2024] kthread+0x115/0x140
[Tue Jul 16 12:07:21 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:07:21 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:07:21 2024] </TASK>
[Tue Jul 16 12:08:24 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:08:24 2024] rcu: 3-....: (83999 ticks this GP) idle=0f9/1/0x4000000000000000 softirq=31469352/31469361 fqs=18902
[Tue Jul 16 12:08:24 2024] (t=84250 jiffies g=194959101 q=14119)
[Tue Jul 16 12:08:24 2024] Task dump for CPU 3:
[Tue Jul 16 12:08:24 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:08:24 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:08:24 2024] Call Trace:
[Tue Jul 16 12:08:24 2024] <IRQ>
[Tue Jul 16 12:08:24 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:08:24 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:08:24 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:08:24 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:08:24 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:08:24 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:08:24 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:08:24 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:08:24 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:08:24 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:08:24 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:08:24 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:08:24 2024] </IRQ>
[Tue Jul 16 12:08:24 2024] <TASK>
[Tue Jul 16 12:08:24 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:08:24 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:08:24 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:08:24 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:08:24 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000048e00003
[Tue Jul 16 12:08:24 2024] RDX: 0000000141f62eaa RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 12:08:24 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:08:24 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:08:24 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:08:24 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:08:24 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:08:24 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:08:24 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:08:24 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:08:24 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:08:24 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:08:24 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:08:24 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:08:24 2024] kthread+0x115/0x140
[Tue Jul 16 12:08:24 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:08:24 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:08:24 2024] </TASK>
[Tue Jul 16 12:09:27 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:09:28 2024] rcu: 3-....: (147004 ticks this GP) idle=0f9/1/0x4000000000000000 softirq=31469352/31469361 fqs=33397
[Tue Jul 16 12:09:28 2024] (t=147493 jiffies g=194959101 q=19580)
[Tue Jul 16 12:09:28 2024] Task dump for CPU 3:
[Tue Jul 16 12:09:28 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:09:28 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:09:28 2024] Call Trace:
[Tue Jul 16 12:09:28 2024] <IRQ>
[Tue Jul 16 12:09:28 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:09:28 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:09:28 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:09:28 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:09:28 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:09:28 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:09:28 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:09:28 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:09:28 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:09:28 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:09:28 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:09:28 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:09:28 2024] </IRQ>
[Tue Jul 16 12:09:28 2024] <TASK>
[Tue Jul 16 12:09:28 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:09:28 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:09:28 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:09:28 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:09:28 2024] RAX: 00000000f5c11296 RBX: 000000003f77f06a RCX: 0000000000001003
[Tue Jul 16 12:09:28 2024] RDX: 0000000000453c8c RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:09:28 2024] RBP: 0003eed2cdba33a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:09:28 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 12:09:28 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:09:28 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:09:28 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:09:28 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:09:28 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:09:28 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:09:28 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:09:28 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:09:28 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:09:28 2024] kthread+0x115/0x140
[Tue Jul 16 12:09:28 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:09:28 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:09:28 2024] </TASK>
[Tue Jul 16 12:10:37 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:10:37 2024] rcu: 11-....: (21000 ticks this GP) idle=373/1/0x4000000000000000 softirq=35298178/35298178 fqs=4514
[Tue Jul 16 12:10:37 2024] (t=21017 jiffies g=194959105 q=22079)
[Tue Jul 16 12:10:37 2024] Task dump for CPU 11:
[Tue Jul 16 12:10:37 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:10:37 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:10:37 2024] Call Trace:
[Tue Jul 16 12:10:37 2024] <IRQ>
[Tue Jul 16 12:10:37 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:10:37 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:10:37 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:10:37 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:10:37 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:10:37 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:10:37 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:10:37 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:10:37 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:10:37 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:10:37 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:10:37 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:10:37 2024] </IRQ>
[Tue Jul 16 12:10:37 2024] <TASK>
[Tue Jul 16 12:10:37 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:10:37 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:10:37 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:10:37 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:10:37 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000036a0000b
[Tue Jul 16 12:10:37 2024] RDX: 0000000141f83398 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 12:10:37 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:10:37 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:10:37 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:10:37 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:10:37 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:10:37 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:10:37 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:10:37 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:10:37 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:10:37 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:10:37 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:10:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:10:37 2024] kthread+0x115/0x140
[Tue Jul 16 12:10:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:10:37 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:10:37 2024] </TASK>
[Tue Jul 16 12:11:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:11:40 2024] rcu: 11-....: (84005 ticks this GP) idle=373/1/0x4000000000000000 softirq=35298178/35298178 fqs=18939
[Tue Jul 16 12:11:40 2024] (t=84261 jiffies g=194959105 q=27842)
[Tue Jul 16 12:11:40 2024] Task dump for CPU 11:
[Tue Jul 16 12:11:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:11:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:11:40 2024] Call Trace:
[Tue Jul 16 12:11:40 2024] <IRQ>
[Tue Jul 16 12:11:40 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:11:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:11:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:11:40 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:11:40 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:11:40 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:11:40 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:11:40 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:11:40 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:11:40 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:11:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:11:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:11:40 2024] </IRQ>
[Tue Jul 16 12:11:40 2024] <TASK>
[Tue Jul 16 12:11:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:11:40 2024] RIP: 0010:mod_delayed_work_on+0xa/0xa0
[Tue Jul 16 12:11:40 2024] Code: 0e 31 c0 80 e7 02 74 01 fb 5b c3 cc cc cc cc e8 4c ff ff ff b8 01 00 00 00 eb e8 0f 1f 44 00 00 0f 1f 44 00 00 41 56 49 89 ce <41> 55 49 89 f5 41 54 41 89 fc 55 48 89 d5 53 48 83 ec 10 65 48 8b
[Tue Jul 16 12:11:40 2024] RSP: 0018:ffffc900087cfdb0 EFLAGS: 00000287
[Tue Jul 16 12:11:40 2024] RAX: 0000000141f922d4 RBX: ffff88997131a500 RCX: 00000000000007d0
[Tue Jul 16 12:11:40 2024] RDX: ffffffffa012c768 RSI: ffff888107352000 RDI: 0000000000000200
[Tue Jul 16 12:11:40 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:11:40 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141f92aa4
[Tue Jul 16 12:11:40 2024] R13: ffffffffa012c700 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:11:40 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:11:40 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:11:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:11:40 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:11:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:11:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:11:40 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:11:40 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:11:40 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:11:40 2024] kthread+0x115/0x140
[Tue Jul 16 12:11:40 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:11:40 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:11:40 2024] </TASK>
[Tue Jul 16 12:12:11 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 21391 jiffies s: 21969 root: 0x800/.
[Tue Jul 16 12:12:11 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:12:11 2024] Task dump for CPU 11:
[Tue Jul 16 12:12:11 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:12:11 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:12:11 2024] Call Trace:
[Tue Jul 16 12:12:11 2024] <TASK>
[Tue Jul 16 12:12:11 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:12:11 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:12:11 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:12:11 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 12:12:11 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:12:11 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:12:11 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:12:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:12:11 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:12:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:12:11 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:12:11 2024] </TASK>
[Tue Jul 16 12:12:43 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:12:43 2024] rcu: 11-....: (147010 ticks this GP) idle=373/1/0x4000000000000000 softirq=35298178/35298178 fqs=33176
[Tue Jul 16 12:12:43 2024] (t=147504 jiffies g=194959105 q=30536)
[Tue Jul 16 12:12:43 2024] Task dump for CPU 11:
[Tue Jul 16 12:12:43 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:12:43 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:12:43 2024] Call Trace:
[Tue Jul 16 12:12:43 2024] <IRQ>
[Tue Jul 16 12:12:43 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:12:43 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:12:43 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:12:43 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:12:43 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:12:43 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:12:43 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:12:43 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:12:43 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:12:43 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:12:43 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:12:43 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:12:43 2024] </IRQ>
[Tue Jul 16 12:12:43 2024] <TASK>
[Tue Jul 16 12:12:43 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:12:43 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 12:12:43 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 12:12:43 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 12:12:43 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:12:43 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:12:43 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:12:43 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:12:43 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:12:43 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:12:43 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:12:43 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:12:43 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:12:43 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:12:43 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:12:43 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:12:43 2024] kthread+0x115/0x140
[Tue Jul 16 12:12:43 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:12:43 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:12:43 2024] </TASK>
[Tue Jul 16 12:13:17 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 87950 jiffies s: 21969 root: 0x800/.
[Tue Jul 16 12:13:17 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:13:17 2024] Task dump for CPU 11:
[Tue Jul 16 12:13:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:13:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:13:17 2024] Call Trace:
[Tue Jul 16 12:13:17 2024] <TASK>
[Tue Jul 16 12:13:17 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:13:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:13:17 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:13:17 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:13:17 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:13:17 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:13:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:13:17 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:13:17 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:13:17 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:13:17 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:13:18 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:13:18 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:13:18 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:13:18 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:13:18 2024] </TASK>
[Tue Jul 16 12:13:46 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:13:46 2024] rcu: 11-....: (210015 ticks this GP) idle=373/1/0x4000000000000000 softirq=35298178/35298178 fqs=47623
[Tue Jul 16 12:13:46 2024] (t=210740 jiffies g=194959105 q=32279)
[Tue Jul 16 12:13:46 2024] Task dump for CPU 11:
[Tue Jul 16 12:13:46 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:13:46 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:13:46 2024] Call Trace:
[Tue Jul 16 12:13:46 2024] <IRQ>
[Tue Jul 16 12:13:46 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:13:46 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:13:46 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:13:46 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:13:46 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:13:46 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:13:46 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:13:46 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:13:46 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:13:46 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:13:46 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:13:46 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:13:46 2024] </IRQ>
[Tue Jul 16 12:13:46 2024] <TASK>
[Tue Jul 16 12:13:46 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:13:46 2024] RIP: 0010:try_to_grab_pending+0x7/0x150
[Tue Jul 16 12:13:46 2024] Code: 0b e9 d6 fc ff ff 0f 0b e9 30 fd ff ff 0f 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 <55> 48 89 d5 53 48 89 fb 48 83 ec 08 9c 58 fa 48 89 02 40 84 f6 0f
[Tue Jul 16 12:13:46 2024] RSP: 0018:ffffc900087cfdc8 EFLAGS: 00000246
[Tue Jul 16 12:13:46 2024] RAX: 0000000000000000 RBX: ffffffffa012c768 RCX: 0000000000000017
[Tue Jul 16 12:13:46 2024] RDX: ffffc900087cfdd8 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 12:13:46 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:13:46 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:13:46 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:13:46 2024] __cancel_work+0x37/0xb0
[Tue Jul 16 12:13:46 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:13:46 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:13:47 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:13:47 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:13:47 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:13:47 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:13:47 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:13:47 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:13:47 2024] kthread+0x115/0x140
[Tue Jul 16 12:13:47 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:13:47 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:13:47 2024] </TASK>
[Tue Jul 16 12:14:23 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 153486 jiffies s: 21969 root: 0x800/.
[Tue Jul 16 12:14:23 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:14:23 2024] Task dump for CPU 11:
[Tue Jul 16 12:14:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:14:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:14:23 2024] Call Trace:
[Tue Jul 16 12:14:23 2024] <TASK>
[Tue Jul 16 12:14:23 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:14:23 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:14:23 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:14:23 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:14:23 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 12:14:23 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:14:23 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:14:23 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:14:23 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:14:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:14:23 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:14:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:14:23 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:14:23 2024] </TASK>
[Tue Jul 16 12:14:50 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:14:50 2024] rcu: 11-....: (273020 ticks this GP) idle=373/1/0x4000000000000000 softirq=35298178/35298178 fqs=62226
[Tue Jul 16 12:14:50 2024] (t=273984 jiffies g=194959105 q=34559)
[Tue Jul 16 12:14:50 2024] Task dump for CPU 11:
[Tue Jul 16 12:14:50 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:14:50 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:14:50 2024] Call Trace:
[Tue Jul 16 12:14:50 2024] <IRQ>
[Tue Jul 16 12:14:50 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:14:50 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:14:50 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:14:50 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:14:50 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:14:50 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:14:50 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:14:50 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:14:50 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:14:50 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:14:50 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:14:50 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:14:50 2024] </IRQ>
[Tue Jul 16 12:14:50 2024] <TASK>
[Tue Jul 16 12:14:50 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:14:50 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:14:50 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:14:50 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:14:50 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 12:14:50 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 12:14:50 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:14:50 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000141fc07ee
[Tue Jul 16 12:14:50 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:14:50 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 12:14:50 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:14:50 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:14:50 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:14:50 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:14:50 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:14:50 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:14:50 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:14:50 2024] kthread+0x115/0x140
[Tue Jul 16 12:14:50 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:14:50 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:14:50 2024] </TASK>
[Tue Jul 16 12:15:38 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:15:38 2024] rcu: 7-....: (20998 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=5184
[Tue Jul 16 12:15:38 2024] (t=21017 jiffies g=194959109 q=29082)
[Tue Jul 16 12:15:38 2024] Task dump for CPU 7:
[Tue Jul 16 12:15:38 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:15:38 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:15:38 2024] Call Trace:
[Tue Jul 16 12:15:38 2024] <IRQ>
[Tue Jul 16 12:15:38 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:15:38 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:15:38 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:15:38 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:15:38 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:15:38 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:15:38 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:15:38 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:15:38 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:15:38 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:15:38 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:15:38 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:15:38 2024] </IRQ>
[Tue Jul 16 12:15:38 2024] <TASK>
[Tue Jul 16 12:15:38 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:15:38 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:15:38 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:15:38 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:15:38 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:15:38 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:15:38 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:15:38 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:15:38 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:15:38 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:15:38 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:15:38 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:15:38 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:15:38 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:15:38 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:15:38 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:15:38 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:15:38 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:15:38 2024] kthread+0x115/0x140
[Tue Jul 16 12:15:38 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:15:38 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:15:38 2024] </TASK>
[Tue Jul 16 12:16:41 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:16:41 2024] rcu: 7-....: (84003 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=20202
[Tue Jul 16 12:16:41 2024] (t=84260 jiffies g=194959109 q=30815)
[Tue Jul 16 12:16:41 2024] Task dump for CPU 7:
[Tue Jul 16 12:16:41 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:16:41 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:16:41 2024] Call Trace:
[Tue Jul 16 12:16:41 2024] <IRQ>
[Tue Jul 16 12:16:41 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:16:41 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:16:41 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:16:41 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:16:41 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:16:41 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:16:41 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:16:41 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:16:41 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:16:41 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:16:41 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:16:41 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:16:41 2024] </IRQ>
[Tue Jul 16 12:16:41 2024] <TASK>
[Tue Jul 16 12:16:41 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:16:41 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:16:41 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:16:41 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:16:41 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:16:41 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:16:41 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:16:41 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:16:41 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:16:41 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:16:41 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:16:41 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:16:41 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:16:41 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:16:41 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:16:41 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:16:41 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:16:41 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:16:41 2024] kthread+0x115/0x140
[Tue Jul 16 12:16:41 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:16:41 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:16:41 2024] </TASK>
[Tue Jul 16 12:17:44 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:17:44 2024] rcu: 7-....: (147008 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=34406
[Tue Jul 16 12:17:44 2024] (t=147503 jiffies g=194959109 q=32348)
[Tue Jul 16 12:17:44 2024] Task dump for CPU 7:
[Tue Jul 16 12:17:44 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:17:44 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:17:44 2024] Call Trace:
[Tue Jul 16 12:17:44 2024] <IRQ>
[Tue Jul 16 12:17:44 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:17:44 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:17:44 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:17:44 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:17:44 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:17:44 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:17:44 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:17:44 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:17:44 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:17:44 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:17:44 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:17:44 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:17:44 2024] </IRQ>
[Tue Jul 16 12:17:44 2024] <TASK>
[Tue Jul 16 12:17:44 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:17:44 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:17:44 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:17:44 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000246
[Tue Jul 16 12:17:44 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000017
[Tue Jul 16 12:17:44 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:17:44 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:17:44 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:17:44 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:17:44 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:17:44 2024] rpc_wake_up_queued_task+0x1f/0x50 [sunrpc]
[Tue Jul 16 12:17:44 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:17:44 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:17:44 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:17:44 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:17:44 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:17:44 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:17:44 2024] kthread+0x115/0x140
[Tue Jul 16 12:17:44 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:17:44 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:17:44 2024] </TASK>
[Tue Jul 16 12:18:47 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:18:47 2024] rcu: 7-....: (210013 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=48840
[Tue Jul 16 12:18:47 2024] (t=210742 jiffies g=194959109 q=33991)
[Tue Jul 16 12:18:47 2024] Task dump for CPU 7:
[Tue Jul 16 12:18:47 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:18:47 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:18:47 2024] Call Trace:
[Tue Jul 16 12:18:47 2024] <IRQ>
[Tue Jul 16 12:18:47 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:18:47 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:18:47 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:18:47 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:18:47 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:18:47 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:18:47 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:18:47 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:18:47 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:18:47 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:18:47 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:18:47 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:18:47 2024] </IRQ>
[Tue Jul 16 12:18:47 2024] <TASK>
[Tue Jul 16 12:18:47 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:18:47 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:18:47 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:18:48 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:18:48 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 12:18:48 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 12:18:48 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:18:48 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000141ffa8f5
[Tue Jul 16 12:18:48 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:18:48 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 12:18:48 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:18:48 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:18:48 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:18:48 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:18:48 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:18:48 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:18:48 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:18:48 2024] kthread+0x115/0x140
[Tue Jul 16 12:18:48 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:18:48 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:18:48 2024] </TASK>
[Tue Jul 16 12:19:51 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:19:51 2024] rcu: 7-....: (273018 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=63475
[Tue Jul 16 12:19:51 2024] (t=273979 jiffies g=194959109 q=38437)
[Tue Jul 16 12:19:51 2024] Task dump for CPU 7:
[Tue Jul 16 12:19:51 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:19:51 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:19:51 2024] Call Trace:
[Tue Jul 16 12:19:51 2024] <IRQ>
[Tue Jul 16 12:19:51 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:19:51 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:19:51 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:19:51 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:19:51 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:19:51 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:19:51 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:19:51 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:19:51 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:19:51 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:19:51 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:19:51 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:19:51 2024] </IRQ>
[Tue Jul 16 12:19:51 2024] <TASK>
[Tue Jul 16 12:19:51 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:19:51 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:19:51 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:19:51 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:19:51 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000042e00007
[Tue Jul 16 12:19:51 2024] RDX: 000000014200a7ca RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:19:51 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:19:51 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:19:51 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:19:51 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:19:51 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:19:51 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:19:51 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:19:51 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:19:51 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:19:51 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:19:51 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:19:51 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:19:51 2024] kthread+0x115/0x140
[Tue Jul 16 12:19:51 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:19:51 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:19:51 2024] </TASK>
[Tue Jul 16 12:20:31 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 21091 jiffies s: 21973 root: 0x80/.
[Tue Jul 16 12:20:31 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:20:31 2024] Task dump for CPU 7:
[Tue Jul 16 12:20:31 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:20:31 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:20:31 2024] Call Trace:
[Tue Jul 16 12:20:31 2024] <TASK>
[Tue Jul 16 12:20:31 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:20:31 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:20:31 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:20:31 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:20:31 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:20:31 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:20:31 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:20:31 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:20:31 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:20:31 2024] ? __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 12:20:31 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:20:31 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:20:31 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:20:31 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:20:31 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:20:31 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:20:31 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:20:31 2024] </TASK>
[Tue Jul 16 12:20:54 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:20:54 2024] rcu: 7-....: (336022 ticks this GP) idle=ae3/1/0x4000000000000000 softirq=29984582/29984582 fqs=77727
[Tue Jul 16 12:20:54 2024] (t=337222 jiffies g=194959109 q=41938)
[Tue Jul 16 12:20:54 2024] Task dump for CPU 7:
[Tue Jul 16 12:20:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:20:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:20:54 2024] Call Trace:
[Tue Jul 16 12:20:54 2024] <IRQ>
[Tue Jul 16 12:20:54 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:20:54 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:20:54 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:20:54 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:20:54 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:20:54 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:20:54 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:20:54 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:20:54 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:20:54 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:20:54 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:20:54 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:20:54 2024] </IRQ>
[Tue Jul 16 12:20:54 2024] <TASK>
[Tue Jul 16 12:20:54 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:20:54 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:20:54 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:20:54 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:20:54 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:20:54 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:20:54 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:20:54 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:20:54 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:20:54 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:20:54 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:20:54 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:20:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:20:54 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:20:54 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:20:54 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:20:54 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:20:54 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:20:54 2024] kthread+0x115/0x140
[Tue Jul 16 12:20:54 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:20:54 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:20:54 2024] </TASK>
[Tue Jul 16 12:21:50 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:21:50 2024] rcu: 15-....: (21000 ticks this GP) idle=a3d/1/0x4000000000000000 softirq=29720202/29720202 fqs=4833
[Tue Jul 16 12:21:50 2024] (t=21017 jiffies g=194959113 q=29541)
[Tue Jul 16 12:21:50 2024] Task dump for CPU 15:
[Tue Jul 16 12:21:50 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:21:50 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:21:50 2024] Call Trace:
[Tue Jul 16 12:21:50 2024] <IRQ>
[Tue Jul 16 12:21:50 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:21:50 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:21:50 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:21:50 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:21:50 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:21:50 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:21:50 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:21:50 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:21:50 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:21:50 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:21:50 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:21:50 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:21:50 2024] </IRQ>
[Tue Jul 16 12:21:50 2024] <TASK>
[Tue Jul 16 12:21:50 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:21:50 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 12:21:50 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 12:21:51 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 12:21:51 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:21:51 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:21:51 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:21:51 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:21:51 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:21:51 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:21:51 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:21:51 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:21:51 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:21:51 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:21:51 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:21:51 2024] kthread+0x115/0x140
[Tue Jul 16 12:21:51 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:21:51 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:21:51 2024] </TASK>
[Tue Jul 16 12:21:51 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 21232 jiffies s: 21977 root: 0x8000/.
[Tue Jul 16 12:21:51 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:21:51 2024] Task dump for CPU 15:
[Tue Jul 16 12:21:51 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:21:51 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:21:51 2024] Call Trace:
[Tue Jul 16 12:21:51 2024] <TASK>
[Tue Jul 16 12:21:51 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:21:51 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:21:51 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:21:51 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:21:51 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:21:51 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:21:51 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:21:51 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:21:51 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:21:51 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:21:51 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:21:51 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:21:51 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:21:51 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:21:51 2024] </TASK>
[Tue Jul 16 12:22:54 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:22:54 2024] rcu: 15-....: (84005 ticks this GP) idle=a3d/1/0x4000000000000000 softirq=29720202/29720202 fqs=18973
[Tue Jul 16 12:22:54 2024] (t=84252 jiffies g=194959113 q=31194)
[Tue Jul 16 12:22:54 2024] Task dump for CPU 15:
[Tue Jul 16 12:22:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:22:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:22:54 2024] Call Trace:
[Tue Jul 16 12:22:54 2024] <IRQ>
[Tue Jul 16 12:22:54 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:22:54 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:22:54 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:22:54 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:22:54 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:22:54 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:22:54 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:22:54 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:22:54 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:22:54 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:22:54 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:22:54 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:22:54 2024] </IRQ>
[Tue Jul 16 12:22:54 2024] <TASK>
[Tue Jul 16 12:22:54 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:22:54 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:22:54 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:22:54 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:22:54 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 12:22:54 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 12:22:54 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:22:54 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000142036acb
[Tue Jul 16 12:22:54 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:22:54 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 12:22:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:22:54 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:22:54 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:22:54 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:22:54 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:22:54 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:22:54 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:22:54 2024] kthread+0x115/0x140
[Tue Jul 16 12:22:54 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:22:54 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:22:54 2024] </TASK>
[Tue Jul 16 12:22:55 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 85568 jiffies s: 21977 root: 0x8000/.
[Tue Jul 16 12:22:55 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:22:55 2024] Task dump for CPU 15:
[Tue Jul 16 12:22:55 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:22:55 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:22:55 2024] Call Trace:
[Tue Jul 16 12:22:55 2024] <TASK>
[Tue Jul 16 12:22:55 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:22:55 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:22:55 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:22:55 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:22:55 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:22:55 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:22:55 2024] ? nfsd4_cb_done+0x28c/0x380 [nfsd]
[Tue Jul 16 12:22:55 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:22:55 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:22:55 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:22:55 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:22:55 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:22:55 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:22:55 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:22:55 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:22:55 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:22:55 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:22:55 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:22:55 2024] </TASK>
[Tue Jul 16 12:24:08 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:24:08 2024] rcu: 1-....: (21000 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=4745
[Tue Jul 16 12:24:08 2024] (t=21017 jiffies g=194959117 q=20706)
[Tue Jul 16 12:24:08 2024] Task dump for CPU 1:
[Tue Jul 16 12:24:08 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:24:08 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:24:08 2024] Call Trace:
[Tue Jul 16 12:24:08 2024] <IRQ>
[Tue Jul 16 12:24:08 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:24:08 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:24:08 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:24:08 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:24:08 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:24:08 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:24:08 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:24:08 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:24:08 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:24:08 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:24:08 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:24:08 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:24:08 2024] </IRQ>
[Tue Jul 16 12:24:08 2024] <TASK>
[Tue Jul 16 12:24:08 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:24:08 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 12:24:08 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 12:24:08 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 12:24:08 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:24:08 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:24:08 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:24:08 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:24:08 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:24:08 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:24:08 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:24:08 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:24:08 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:24:08 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:24:08 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:24:08 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:24:08 2024] kthread+0x115/0x140
[Tue Jul 16 12:24:08 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:24:08 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:24:08 2024] </TASK>
[Tue Jul 16 12:25:11 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:25:11 2024] rcu: 1-....: (84005 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=19271
[Tue Jul 16 12:25:11 2024] (t=84251 jiffies g=194959117 q=22540)
[Tue Jul 16 12:25:11 2024] Task dump for CPU 1:
[Tue Jul 16 12:25:11 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:25:11 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:25:11 2024] Call Trace:
[Tue Jul 16 12:25:11 2024] <IRQ>
[Tue Jul 16 12:25:11 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:25:11 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:25:11 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:25:11 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:25:11 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:25:11 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:25:11 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:25:11 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:25:11 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:25:11 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:25:11 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:25:11 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:25:11 2024] </IRQ>
[Tue Jul 16 12:25:11 2024] <TASK>
[Tue Jul 16 12:25:11 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:25:11 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 12:25:11 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 12:25:11 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 12:25:11 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:25:11 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:25:11 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:25:11 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:25:11 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:25:11 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:25:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:25:11 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:25:11 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:25:11 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:25:11 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:25:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:25:11 2024] kthread+0x115/0x140
[Tue Jul 16 12:25:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:25:11 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:25:11 2024] </TASK>
[Tue Jul 16 12:26:14 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:26:14 2024] rcu: 1-....: (147010 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=33845
[Tue Jul 16 12:26:14 2024] (t=147485 jiffies g=194959117 q=24277)
[Tue Jul 16 12:26:14 2024] Task dump for CPU 1:
[Tue Jul 16 12:26:14 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:26:14 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:26:14 2024] Call Trace:
[Tue Jul 16 12:26:14 2024] <IRQ>
[Tue Jul 16 12:26:14 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:26:14 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:26:14 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:26:14 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:26:14 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:26:14 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:26:14 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:26:14 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:26:14 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:26:14 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:26:14 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:26:14 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:26:14 2024] </IRQ>
[Tue Jul 16 12:26:14 2024] <TASK>
[Tue Jul 16 12:26:14 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:26:14 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:26:14 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:26:14 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:26:14 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:26:14 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:26:14 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:26:14 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:26:14 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:26:14 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:26:14 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:26:14 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:26:14 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:26:14 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:26:14 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:26:14 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:26:14 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:26:14 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:26:14 2024] kthread+0x115/0x140
[Tue Jul 16 12:26:14 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:26:14 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:26:14 2024] </TASK>
[Tue Jul 16 12:27:17 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:27:17 2024] rcu: 1-....: (210015 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=48354
[Tue Jul 16 12:27:17 2024] (t=210726 jiffies g=194959117 q=26046)
[Tue Jul 16 12:27:17 2024] Task dump for CPU 1:
[Tue Jul 16 12:27:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:27:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:27:17 2024] Call Trace:
[Tue Jul 16 12:27:17 2024] <IRQ>
[Tue Jul 16 12:27:17 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:27:17 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:27:17 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:27:17 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:27:17 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:27:17 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:27:17 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:27:17 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:27:17 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:27:17 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:27:17 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:27:17 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:27:17 2024] </IRQ>
[Tue Jul 16 12:27:17 2024] <TASK>
[Tue Jul 16 12:27:17 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:27:17 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:27:17 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:27:17 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:27:17 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004e200001
[Tue Jul 16 12:27:17 2024] RDX: 00000001420778a7 RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:27:17 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:27:17 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:27:17 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:27:17 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:27:17 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:27:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:27:17 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:27:17 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:27:17 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:27:17 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:27:18 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:27:18 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:27:18 2024] kthread+0x115/0x140
[Tue Jul 16 12:27:18 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:27:18 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:27:18 2024] </TASK>
[Tue Jul 16 12:28:21 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:28:21 2024] rcu: 1-....: (273020 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=63713
[Tue Jul 16 12:28:21 2024] (t=273971 jiffies g=194959117 q=27551)
[Tue Jul 16 12:28:21 2024] Task dump for CPU 1:
[Tue Jul 16 12:28:21 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:28:21 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:28:21 2024] Call Trace:
[Tue Jul 16 12:28:21 2024] <IRQ>
[Tue Jul 16 12:28:21 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:28:21 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:28:21 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:28:21 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:28:21 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:28:21 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:28:21 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:28:21 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:28:21 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:28:21 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:28:21 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:28:21 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:28:21 2024] </IRQ>
[Tue Jul 16 12:28:21 2024] <TASK>
[Tue Jul 16 12:28:21 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:28:21 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:28:21 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:28:21 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
[Tue Jul 16 12:28:21 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:28:21 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:28:21 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:28:21 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:28:21 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:28:21 2024] __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 12:28:21 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:28:21 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:28:21 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:28:21 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:28:21 2024] kthread+0x115/0x140
[Tue Jul 16 12:28:21 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:28:21 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:28:21 2024] </TASK>
[Tue Jul 16 12:29:24 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:29:24 2024] rcu: 1-....: (336021 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=79231
[Tue Jul 16 12:29:24 2024] (t=337186 jiffies g=194959117 q=29030)
[Tue Jul 16 12:29:24 2024] Task dump for CPU 1:
[Tue Jul 16 12:29:24 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:29:24 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:29:24 2024] Call Trace:
[Tue Jul 16 12:29:24 2024] <IRQ>
[Tue Jul 16 12:29:24 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:29:24 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:29:24 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:29:24 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:29:24 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:29:24 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:29:24 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:29:24 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:29:24 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:29:24 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:29:24 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:29:24 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:29:24 2024] </IRQ>
[Tue Jul 16 12:29:24 2024] <TASK>
[Tue Jul 16 12:29:24 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:29:24 2024] RIP: 0010:__rpc_execute+0xa5/0x410 [sunrpc]
[Tue Jul 16 12:29:24 2024] Code: 00 00 48 8b 43 30 a8 40 0f 85 0f 01 00 00 48 8b 6b 38 48 89 ef e8 5b 58 a7 e1 48 8b 43 30 a8 02 75 72 c6 45 00 00 48 8b 6b 18 <48> 8b 53 20 48 85 ed 75 a9 48 85 d2 0f 84 47 01 00 00 8b 05 cb 79
[Tue Jul 16 12:29:24 2024] RSP: 0018:ffffc900087cfe38 EFLAGS: 00000246
[Tue Jul 16 12:29:24 2024] RAX: 0000000000000045 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:29:24 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:29:24 2024] RBP: ffffffffa00cb630 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:29:24 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:29:24 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:29:24 2024] ? rpc_wake_up_next_func+0x10/0x10 [sunrpc]
[Tue Jul 16 12:29:24 2024] ? __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 12:29:24 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:29:24 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:29:24 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:29:24 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:29:24 2024] kthread+0x115/0x140
[Tue Jul 16 12:29:24 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:29:24 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:29:24 2024] </TASK>
[Tue Jul 16 12:30:27 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:30:27 2024] rcu: 1-....: (399026 ticks this GP) idle=231/1/0x4000000000000000 softirq=30799232/30799241 fqs=94456
[Tue Jul 16 12:30:27 2024] (t=400419 jiffies g=194959117 q=33835)
[Tue Jul 16 12:30:27 2024] Task dump for CPU 1:
[Tue Jul 16 12:30:27 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:30:27 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:30:27 2024] Call Trace:
[Tue Jul 16 12:30:27 2024] <IRQ>
[Tue Jul 16 12:30:27 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:30:27 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:30:27 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:30:27 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:30:27 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:30:27 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:30:27 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:30:27 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:30:27 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:30:27 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:30:27 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:30:27 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:30:27 2024] </IRQ>
[Tue Jul 16 12:30:27 2024] <TASK>
[Tue Jul 16 12:30:27 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:30:27 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:30:27 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:30:27 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:30:27 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 12:30:27 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 12:30:27 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:30:27 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 00000001420a55d2
[Tue Jul 16 12:30:27 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:30:27 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 12:30:27 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:30:27 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:30:27 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:30:27 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:30:27 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:30:27 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:30:27 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:30:27 2024] kthread+0x115/0x140
[Tue Jul 16 12:30:27 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:30:27 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:30:27 2024] </TASK>
[Tue Jul 16 12:31:53 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:31:53 2024] rcu: 1-....: (21000 ticks this GP) idle=247/1/0x4000000000000000 softirq=30799475/30799479 fqs=4816
[Tue Jul 16 12:31:54 2024] (t=21017 jiffies g=194959129 q=1409)
[Tue Jul 16 12:31:54 2024] Task dump for CPU 1:
[Tue Jul 16 12:31:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:31:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:31:54 2024] Call Trace:
[Tue Jul 16 12:31:54 2024] <IRQ>
[Tue Jul 16 12:31:54 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:31:54 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:31:54 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:31:54 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:31:54 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:31:54 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:31:54 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:31:54 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:31:54 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:31:54 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:31:54 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:31:54 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:31:54 2024] </IRQ>
[Tue Jul 16 12:31:54 2024] <TASK>
[Tue Jul 16 12:31:54 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:31:54 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:31:54 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:31:54 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:31:54 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:31:54 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:31:54 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:31:54 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:31:54 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:31:54 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:31:54 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:31:54 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:31:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:31:54 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:31:54 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:31:54 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:31:54 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:31:54 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:31:54 2024] kthread+0x115/0x140
[Tue Jul 16 12:31:54 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:31:54 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:31:54 2024] </TASK>
[Tue Jul 16 12:32:37 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:32:37 2024] rcu: 9-....: (20994 ticks this GP) idle=a89/1/0x4000000000000000 softirq=65675730/65675730 fqs=5251
[Tue Jul 16 12:32:37 2024] (t=21013 jiffies g=194959133 q=1969)
[Tue Jul 16 12:32:37 2024] Task dump for CPU 9:
[Tue Jul 16 12:32:37 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:32:37 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:32:37 2024] Call Trace:
[Tue Jul 16 12:32:37 2024] <IRQ>
[Tue Jul 16 12:32:37 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:32:37 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:32:37 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:32:37 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:32:37 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:32:37 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:32:37 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:32:37 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:32:37 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:32:37 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:32:37 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:32:37 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:32:37 2024] </IRQ>
[Tue Jul 16 12:32:37 2024] <TASK>
[Tue Jul 16 12:32:37 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:32:37 2024] RIP: 0010:rpc_make_runnable+0x10/0x70 [sunrpc]
[Tue Jul 16 12:32:37 2024] Code: 66 2e 0f 1f 84 00 00 00 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 <0f> 92 c0 f0 80 66 30 fd 84 c0 75 49 f6 86 dc 00 00 00 01 74 33 48
[Tue Jul 16 12:32:37 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000283
[Tue Jul 16 12:32:37 2024] RAX: dead000000000122 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:32:37 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:32:37 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:32:37 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:32:37 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:32:37 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:32:37 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:32:37 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:32:37 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:32:37 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:32:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:32:37 2024] kthread+0x115/0x140
[Tue Jul 16 12:32:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:32:37 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:32:37 2024] </TASK>
[Tue Jul 16 12:32:37 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 9-... } 21225 jiffies s: 21985 root: 0x200/.
[Tue Jul 16 12:32:37 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:32:37 2024] Task dump for CPU 9:
[Tue Jul 16 12:32:37 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:32:37 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:32:37 2024] Call Trace:
[Tue Jul 16 12:32:37 2024] <TASK>
[Tue Jul 16 12:32:37 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:32:37 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:32:37 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:32:37 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:32:37 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 12:32:37 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:32:37 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:32:37 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:32:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:32:37 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:32:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:32:37 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:32:37 2024] </TASK>
[Tue Jul 16 12:33:49 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:33:49 2024] rcu: 1-....: (20997 ticks this GP) idle=a9f/1/0x4000000000000000 softirq=30799492/30799492 fqs=4652
[Tue Jul 16 12:33:49 2024] (t=21017 jiffies g=194959137 q=3246)
[Tue Jul 16 12:33:49 2024] Task dump for CPU 1:
[Tue Jul 16 12:33:49 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:33:49 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:33:49 2024] Call Trace:
[Tue Jul 16 12:33:49 2024] <IRQ>
[Tue Jul 16 12:33:49 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:33:49 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:33:49 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:33:49 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:33:49 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:33:49 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:33:49 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:33:49 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:33:49 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:33:49 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:33:49 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:33:49 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:33:49 2024] </IRQ>
[Tue Jul 16 12:33:49 2024] <TASK>
[Tue Jul 16 12:33:49 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:33:49 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:33:49 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:33:49 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:33:49 2024] RAX: 0000000031ad17aa RBX: 000000003fa47ca8 RCX: 0000000000001001
[Tue Jul 16 12:33:49 2024] RDX: 0000000000454197 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:33:49 2024] RBP: 0003f02733b7cfa3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:33:49 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 12:33:49 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:33:49 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:33:49 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:33:49 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:33:49 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:33:49 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:33:49 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:33:49 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:33:49 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:33:49 2024] kthread+0x115/0x140
[Tue Jul 16 12:33:49 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:33:49 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:33:49 2024] </TASK>
[Tue Jul 16 12:34:52 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:34:52 2024] rcu: 1-....: (84002 ticks this GP) idle=a9f/1/0x4000000000000000 softirq=30799492/30799492 fqs=19019
[Tue Jul 16 12:34:52 2024] (t=84248 jiffies g=194959137 q=8392)
[Tue Jul 16 12:34:52 2024] Task dump for CPU 1:
[Tue Jul 16 12:34:52 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:34:52 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:34:52 2024] Call Trace:
[Tue Jul 16 12:34:52 2024] <IRQ>
[Tue Jul 16 12:34:52 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:34:53 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:34:53 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:34:53 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:34:53 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:34:53 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:34:53 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:34:53 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:34:53 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:34:53 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:34:53 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:34:53 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:34:53 2024] </IRQ>
[Tue Jul 16 12:34:53 2024] <TASK>
[Tue Jul 16 12:34:53 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:34:53 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:34:53 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:34:53 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:34:53 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000049e00001
[Tue Jul 16 12:34:53 2024] RDX: 00000001420e6aad RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:34:53 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:34:53 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:34:53 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:34:53 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:34:53 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:34:53 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:34:53 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:34:53 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:34:53 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:34:53 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:34:53 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:34:53 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:34:53 2024] kthread+0x115/0x140
[Tue Jul 16 12:34:53 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:34:53 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:34:53 2024] </TASK>
[Tue Jul 16 12:35:56 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:35:56 2024] rcu: 1-....: (147006 ticks this GP) idle=a9f/1/0x4000000000000000 softirq=30799492/30799492 fqs=34774
[Tue Jul 16 12:35:56 2024] (t=147493 jiffies g=194959137 q=10065)
[Tue Jul 16 12:35:56 2024] Task dump for CPU 1:
[Tue Jul 16 12:35:56 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:35:56 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:35:56 2024] Call Trace:
[Tue Jul 16 12:35:56 2024] <IRQ>
[Tue Jul 16 12:35:56 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:35:56 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:35:56 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:35:56 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:35:56 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:35:56 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:35:56 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:35:56 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:35:56 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:35:56 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:35:56 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:35:56 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:35:56 2024] </IRQ>
[Tue Jul 16 12:35:56 2024] <TASK>
[Tue Jul 16 12:35:56 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:35:56 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 12:35:56 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 12:35:56 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
[Tue Jul 16 12:35:56 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:35:56 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:35:56 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:35:56 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:35:56 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:35:56 2024] __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 12:35:56 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:35:56 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:35:56 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:35:56 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:35:56 2024] kthread+0x115/0x140
[Tue Jul 16 12:35:56 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:35:56 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:35:56 2024] </TASK>
[Tue Jul 16 12:36:59 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:36:59 2024] rcu: 1-....: (210007 ticks this GP) idle=a9f/1/0x4000000000000000 softirq=30799492/30799492 fqs=50329
[Tue Jul 16 12:36:59 2024] (t=210712 jiffies g=194959137 q=13344)
[Tue Jul 16 12:36:59 2024] Task dump for CPU 1:
[Tue Jul 16 12:36:59 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:36:59 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:36:59 2024] Call Trace:
[Tue Jul 16 12:36:59 2024] <IRQ>
[Tue Jul 16 12:36:59 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:36:59 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:36:59 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:36:59 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:36:59 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:36:59 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:36:59 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:36:59 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:36:59 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:36:59 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:36:59 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:36:59 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:36:59 2024] </IRQ>
[Tue Jul 16 12:36:59 2024] <TASK>
[Tue Jul 16 12:36:59 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:36:59 2024] RIP: 0010:rpc_wake_up_queued_task+0x1f/0x50 [sunrpc]
[Tue Jul 16 12:36:59 2024] Code: 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 48 8b 46 30 a8 02 75 05 c3 cc cc cc cc 55 48 89 fd 53 48 89 f3 e8 a1 65 a7 e1 <48> 8b 43 30 48 8b 3d 36 41 05 00 a8 02 74 11 48 3b 6b 38 75 0b 48
[Tue Jul 16 12:36:59 2024] RSP: 0018:ffffc900087cfe20 EFLAGS: 00000246
[Tue Jul 16 12:36:59 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000017
[Tue Jul 16 12:36:59 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:36:59 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:36:59 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:36:59 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:36:59 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:36:59 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:36:59 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:36:59 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:36:59 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:36:59 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:36:59 2024] kthread+0x115/0x140
[Tue Jul 16 12:36:59 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:36:59 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:36:59 2024] </TASK>
[Tue Jul 16 12:38:02 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:38:02 2024] rcu: 1-....: (273011 ticks this GP) idle=a9f/1/0x4000000000000000 softirq=30799492/30799492 fqs=65766
[Tue Jul 16 12:38:02 2024] (t=273940 jiffies g=194959137 q=16090)
[Tue Jul 16 12:38:02 2024] Task dump for CPU 1:
[Tue Jul 16 12:38:02 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:38:02 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:38:02 2024] Call Trace:
[Tue Jul 16 12:38:02 2024] <IRQ>
[Tue Jul 16 12:38:02 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:38:02 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:38:02 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:38:02 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:38:02 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:38:02 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:38:02 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:38:02 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:38:02 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:38:02 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:38:02 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:38:02 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:38:02 2024] </IRQ>
[Tue Jul 16 12:38:02 2024] <TASK>
[Tue Jul 16 12:38:02 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:38:02 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:38:02 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:38:02 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:38:02 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000058e00001
[Tue Jul 16 12:38:02 2024] RDX: 0000000142114fa6 RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:38:02 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:38:02 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:38:02 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:38:02 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:38:02 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:38:02 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:38:02 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:38:02 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:38:02 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:38:02 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:38:02 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:38:02 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:38:02 2024] kthread+0x115/0x140
[Tue Jul 16 12:38:02 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:38:02 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:38:02 2024] </TASK>
[Tue Jul 16 12:38:28 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:38:28 2024] rcu: 7-....: (20996 ticks this GP) idle=6d1/1/0x4000000000000000 softirq=29984735/29984735 fqs=4883
[Tue Jul 16 12:38:28 2024] (t=21017 jiffies g=194959141 q=15603)
[Tue Jul 16 12:38:28 2024] Task dump for CPU 7:
[Tue Jul 16 12:38:28 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:38:28 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:38:28 2024] Call Trace:
[Tue Jul 16 12:38:28 2024] <IRQ>
[Tue Jul 16 12:38:28 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:38:28 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:38:28 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:38:28 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:38:28 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:38:28 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:38:28 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:38:28 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:38:28 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:38:28 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:38:28 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:38:28 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:38:28 2024] </IRQ>
[Tue Jul 16 12:38:28 2024] <TASK>
[Tue Jul 16 12:38:28 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:38:28 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xae/0x180 [sunrpc]
[Tue Jul 16 12:38:28 2024] Code: 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 4c 89 63 28 <49> 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78 42 49 8b 45
[Tue Jul 16 12:38:28 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 12:38:28 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 12:38:28 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:38:28 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:38:28 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 000000014211b3a8
[Tue Jul 16 12:38:28 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 12:38:28 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:38:28 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:38:28 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:38:28 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:38:28 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:38:28 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:38:28 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:38:28 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:38:28 2024] kthread+0x115/0x140
[Tue Jul 16 12:38:28 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:38:28 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:38:28 2024] </TASK>
[Tue Jul 16 12:39:32 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:39:32 2024] rcu: 7-....: (20999 ticks this GP) idle=eb7/1/0x4000000000000000 softirq=29984742/29984745 fqs=5251
[Tue Jul 16 12:39:32 2024] (t=21013 jiffies g=194959149 q=3477)
[Tue Jul 16 12:39:32 2024] Task dump for CPU 7:
[Tue Jul 16 12:39:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:39:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:39:32 2024] Call Trace:
[Tue Jul 16 12:39:32 2024] <IRQ>
[Tue Jul 16 12:39:32 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:39:32 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:39:32 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:39:32 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:39:32 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:39:32 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:39:32 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:39:32 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:39:32 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:39:32 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:39:32 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:39:32 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:39:32 2024] </IRQ>
[Tue Jul 16 12:39:32 2024] <TASK>
[Tue Jul 16 12:39:32 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:39:32 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 12:39:32 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 12:39:32 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 12:39:32 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:39:32 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 12:39:32 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:39:32 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:39:32 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:39:32 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:39:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:39:32 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:39:32 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:39:32 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:39:32 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:39:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:39:32 2024] kthread+0x115/0x140
[Tue Jul 16 12:39:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:39:32 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:39:32 2024] </TASK>
[Tue Jul 16 12:40:35 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:40:35 2024] rcu: 7-....: (84003 ticks this GP) idle=eb7/1/0x4000000000000000 softirq=29984742/29984745 fqs=20502
[Tue Jul 16 12:40:35 2024] (t=84247 jiffies g=194959149 q=7954)
[Tue Jul 16 12:40:35 2024] Task dump for CPU 7:
[Tue Jul 16 12:40:35 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:40:35 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:40:36 2024] Call Trace:
[Tue Jul 16 12:40:36 2024] <IRQ>
[Tue Jul 16 12:40:36 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:40:36 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:40:36 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:40:36 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:40:36 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:40:36 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:40:36 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:40:36 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:40:36 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:40:36 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:40:36 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:40:36 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:40:36 2024] </IRQ>
[Tue Jul 16 12:40:36 2024] <TASK>
[Tue Jul 16 12:40:36 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:40:36 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:40:36 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:40:36 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:40:36 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004ee00007
[Tue Jul 16 12:40:36 2024] RDX: 000000014213a69a RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:40:36 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:40:36 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:40:36 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:40:36 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:40:36 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:40:36 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:40:36 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:40:36 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:40:36 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:40:36 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:40:36 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:40:36 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:40:36 2024] kthread+0x115/0x140
[Tue Jul 16 12:40:36 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:40:36 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:40:36 2024] </TASK>
[Tue Jul 16 12:41:58 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:41:58 2024] rcu: 1-....: (20999 ticks this GP) idle=4a9/1/0x4000000000000000 softirq=30799513/30799514 fqs=5214
[Tue Jul 16 12:41:58 2024] (t=21016 jiffies g=194959161 q=1159)
[Tue Jul 16 12:41:58 2024] Task dump for CPU 1:
[Tue Jul 16 12:41:58 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:41:58 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:41:58 2024] Call Trace:
[Tue Jul 16 12:41:58 2024] <IRQ>
[Tue Jul 16 12:41:58 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:41:58 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:41:58 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:41:58 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:41:58 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:41:58 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:41:58 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:41:58 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:41:58 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:41:58 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:41:58 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:41:58 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:41:58 2024] </IRQ>
[Tue Jul 16 12:41:58 2024] <TASK>
[Tue Jul 16 12:41:58 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:41:58 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:41:58 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:41:58 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:41:58 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003d200001
[Tue Jul 16 12:41:58 2024] RDX: 000000014214e79a RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:41:58 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:41:58 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:41:58 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:41:58 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:41:58 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:41:58 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:41:58 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:41:58 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:41:58 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:41:58 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:41:58 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:41:58 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:41:58 2024] kthread+0x115/0x140
[Tue Jul 16 12:41:58 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:41:58 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:41:58 2024] </TASK>
[Tue Jul 16 12:43:14 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:43:14 2024] rcu: 3-....: (21000 ticks this GP) idle=55b/1/0x4000000000000000 softirq=31469426/31469426 fqs=5251
[Tue Jul 16 12:43:14 2024] (t=21013 jiffies g=194959165 q=2640)
[Tue Jul 16 12:43:14 2024] Task dump for CPU 3:
[Tue Jul 16 12:43:14 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:43:14 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:43:14 2024] Call Trace:
[Tue Jul 16 12:43:14 2024] <IRQ>
[Tue Jul 16 12:43:14 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:43:14 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:43:14 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:43:14 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:43:15 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:43:15 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:43:15 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:43:15 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:43:15 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:43:15 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:43:15 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:43:15 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:43:15 2024] </IRQ>
[Tue Jul 16 12:43:15 2024] <TASK>
[Tue Jul 16 12:43:15 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:43:15 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:43:15 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:43:15 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:43:15 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000032a00003
[Tue Jul 16 12:43:15 2024] RDX: 000000014216139a RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 12:43:15 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:43:15 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:43:15 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:43:15 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:43:15 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:43:15 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:43:15 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:43:15 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:43:15 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:43:15 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:43:15 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:43:15 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:43:15 2024] kthread+0x115/0x140
[Tue Jul 16 12:43:15 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:43:15 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:43:15 2024] </TASK>
[Tue Jul 16 12:44:22 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:44:22 2024] rcu: 11-....: (20998 ticks this GP) idle=097/1/0x4000000000000000 softirq=35298357/35298357 fqs=5033
[Tue Jul 16 12:44:22 2024] (t=21013 jiffies g=194959169 q=3625)
[Tue Jul 16 12:44:22 2024] Task dump for CPU 11:
[Tue Jul 16 12:44:22 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:44:22 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:44:22 2024] Call Trace:
[Tue Jul 16 12:44:22 2024] <IRQ>
[Tue Jul 16 12:44:22 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:44:22 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:44:22 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:44:22 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:44:22 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:44:22 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:44:22 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:44:22 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:44:22 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:44:22 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:44:22 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:44:22 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:44:22 2024] </IRQ>
[Tue Jul 16 12:44:22 2024] <TASK>
[Tue Jul 16 12:44:22 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:44:22 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:44:22 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:44:22 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:44:22 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000033a0000b
[Tue Jul 16 12:44:22 2024] RDX: 0000000142171ba9 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 12:44:22 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:44:22 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:44:22 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:44:22 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:44:22 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:44:22 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:44:22 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:44:22 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:44:22 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:44:22 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:44:22 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:44:22 2024] kthread+0x115/0x140
[Tue Jul 16 12:44:22 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:44:22 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:44:22 2024] </TASK>
[Tue Jul 16 12:44:22 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 21238 jiffies s: 21993 root: 0x800/.
[Tue Jul 16 12:44:22 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:44:22 2024] Task dump for CPU 11:
[Tue Jul 16 12:44:22 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:44:22 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:44:22 2024] Call Trace:
[Tue Jul 16 12:44:22 2024] <TASK>
[Tue Jul 16 12:44:22 2024] ? common_interrupt+0xf/0xa0
[Tue Jul 16 12:44:22 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:44:22 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:44:22 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:44:22 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:44:22 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:44:22 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:44:22 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:44:22 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:44:22 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:44:22 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:44:22 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:44:22 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:44:22 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:44:22 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:44:22 2024] </TASK>
[Tue Jul 16 12:45:25 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:45:25 2024] rcu: 11-....: (84003 ticks this GP) idle=097/1/0x4000000000000000 softirq=35298357/35298357 fqs=20284
[Tue Jul 16 12:45:25 2024] (t=84258 jiffies g=194959169 q=5244)
[Tue Jul 16 12:45:25 2024] Task dump for CPU 11:
[Tue Jul 16 12:45:25 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:45:25 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:45:25 2024] Call Trace:
[Tue Jul 16 12:45:25 2024] <IRQ>
[Tue Jul 16 12:45:25 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:45:25 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:45:25 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:45:25 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:45:25 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:45:25 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:45:25 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:45:25 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:45:25 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:45:25 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:45:25 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:45:25 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:45:25 2024] </IRQ>
[Tue Jul 16 12:45:25 2024] <TASK>
[Tue Jul 16 12:45:25 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:45:25 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:45:25 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:45:25 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:45:25 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000040a0000b
[Tue Jul 16 12:45:25 2024] RDX: 00000001421812b4 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 12:45:25 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:45:25 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:45:25 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:45:25 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:45:25 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:45:25 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:45:25 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:45:25 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:45:25 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:45:25 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:45:25 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:45:25 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:45:26 2024] kthread+0x115/0x140
[Tue Jul 16 12:45:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:45:26 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:45:26 2024] </TASK>
[Tue Jul 16 12:45:27 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 85549 jiffies s: 21993 root: 0x800/.
[Tue Jul 16 12:45:27 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:45:27 2024] Task dump for CPU 11:
[Tue Jul 16 12:45:27 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:45:27 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:45:27 2024] Call Trace:
[Tue Jul 16 12:45:27 2024] <TASK>
[Tue Jul 16 12:45:27 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:45:27 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:45:27 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:45:27 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:45:27 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 12:45:27 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:45:27 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:45:27 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:45:27 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:45:27 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:45:27 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:45:27 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:45:27 2024] </TASK>
[Tue Jul 16 12:46:13 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:46:13 2024] rcu: 11-....: (20997 ticks this GP) idle=07d/1/0x4000000000000000 softirq=35298364/35298364 fqs=5058
[Tue Jul 16 12:46:13 2024] (t=21014 jiffies g=194959177 q=4423)
[Tue Jul 16 12:46:13 2024] Task dump for CPU 11:
[Tue Jul 16 12:46:13 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:46:13 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:46:13 2024] Call Trace:
[Tue Jul 16 12:46:13 2024] <IRQ>
[Tue Jul 16 12:46:13 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:46:13 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:46:13 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:46:13 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:46:13 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:46:13 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:46:13 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:46:13 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:46:13 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:46:13 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:46:13 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:46:13 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:46:13 2024] </IRQ>
[Tue Jul 16 12:46:13 2024] <TASK>
[Tue Jul 16 12:46:13 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:46:13 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:46:13 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:46:13 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:46:13 2024] RAX: 000000005fbb76fc RBX: 000000003fbb1bf8 RCX: 000000000000100b
[Tue Jul 16 12:46:13 2024] RDX: 0000000000454427 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:46:13 2024] RBP: 0003f0d46d92dfa3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:46:13 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
[Tue Jul 16 12:46:13 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:46:13 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:46:13 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:46:13 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:46:13 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:46:13 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:46:13 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:46:13 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:46:13 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:46:13 2024] kthread+0x115/0x140
[Tue Jul 16 12:46:13 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:46:13 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:46:13 2024] </TASK>
[Tue Jul 16 12:46:13 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 21229 jiffies s: 21997 root: 0x800/.
[Tue Jul 16 12:46:13 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:46:13 2024] Task dump for CPU 11:
[Tue Jul 16 12:46:13 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:46:13 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:46:13 2024] Call Trace:
[Tue Jul 16 12:46:13 2024] <TASK>
[Tue Jul 16 12:46:13 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:46:13 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:46:13 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:46:13 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:46:13 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:46:13 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 12:46:13 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:46:13 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:46:13 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:46:13 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:46:13 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:46:13 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:46:13 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:46:13 2024] </TASK>
[Tue Jul 16 12:47:16 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:47:16 2024] rcu: 11-....: (84002 ticks this GP) idle=07d/1/0x4000000000000000 softirq=35298364/35298364 fqs=20381
[Tue Jul 16 12:47:16 2024] (t=84253 jiffies g=194959177 q=6361)
[Tue Jul 16 12:47:16 2024] Task dump for CPU 11:
[Tue Jul 16 12:47:16 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:47:16 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:47:16 2024] Call Trace:
[Tue Jul 16 12:47:16 2024] <IRQ>
[Tue Jul 16 12:47:16 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:47:16 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:47:16 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:47:16 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:47:16 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:47:16 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:47:16 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:47:16 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:47:16 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:47:16 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:47:16 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:47:16 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:47:16 2024] </IRQ>
[Tue Jul 16 12:47:16 2024] <TASK>
[Tue Jul 16 12:47:16 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:47:16 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:47:16 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:47:16 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:47:16 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004760000b
[Tue Jul 16 12:47:16 2024] RDX: 000000014219c29d RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 12:47:16 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:47:16 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:47:16 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:47:16 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:47:16 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:47:16 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:47:16 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:47:16 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:47:16 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:47:16 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:47:16 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:47:16 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:47:16 2024] kthread+0x115/0x140
[Tue Jul 16 12:47:16 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:47:16 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:47:16 2024] </TASK>
[Tue Jul 16 12:47:17 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 85564 jiffies s: 21997 root: 0x800/.
[Tue Jul 16 12:47:17 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:47:17 2024] Task dump for CPU 11:
[Tue Jul 16 12:47:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:47:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:47:17 2024] Call Trace:
[Tue Jul 16 12:47:17 2024] <TASK>
[Tue Jul 16 12:47:17 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:47:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:47:17 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:47:17 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:47:17 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 12:47:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:47:17 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:47:17 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:47:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:47:17 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:47:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:47:17 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:47:17 2024] </TASK>
[Tue Jul 16 12:48:17 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:48:17 2024] rcu: 5-....: (21000 ticks this GP) idle=49b/1/0x4000000000000000 softirq=30126214/30126214 fqs=5251
[Tue Jul 16 12:48:17 2024] (t=21013 jiffies g=194959185 q=5699)
[Tue Jul 16 12:48:17 2024] Task dump for CPU 5:
[Tue Jul 16 12:48:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:48:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:48:17 2024] Call Trace:
[Tue Jul 16 12:48:17 2024] <IRQ>
[Tue Jul 16 12:48:17 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:48:17 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:48:17 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:48:17 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:48:17 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:48:17 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:48:17 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:48:17 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:48:17 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:48:17 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:48:17 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:48:17 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:48:17 2024] </IRQ>
[Tue Jul 16 12:48:17 2024] <TASK>
[Tue Jul 16 12:48:17 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:48:17 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:48:17 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:48:17 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:48:17 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000036200005
[Tue Jul 16 12:48:17 2024] RDX: 00000001421aaf99 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 12:48:17 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:48:17 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:48:17 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:48:17 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:48:17 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:48:17 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:48:17 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:48:17 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:48:17 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:48:17 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:48:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:48:17 2024] kthread+0x115/0x140
[Tue Jul 16 12:48:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:48:17 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:48:17 2024] </TASK>
[Tue Jul 16 12:48:17 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 21508 jiffies s: 22005 root: 0x20/.
[Tue Jul 16 12:48:17 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:48:17 2024] Task dump for CPU 5:
[Tue Jul 16 12:48:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:48:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:48:17 2024] Call Trace:
[Tue Jul 16 12:48:17 2024] <TASK>
[Tue Jul 16 12:48:17 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:48:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:48:17 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:48:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:48:17 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 12:48:17 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:48:17 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:48:17 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:48:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:48:17 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:48:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:48:17 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:48:17 2024] </TASK>
[Tue Jul 16 12:49:20 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:49:20 2024] rcu: 5-....: (84005 ticks this GP) idle=49b/1/0x4000000000000000 softirq=30126214/30126214 fqs=20802
[Tue Jul 16 12:49:20 2024] (t=84257 jiffies g=194959185 q=7326)
[Tue Jul 16 12:49:20 2024] Task dump for CPU 5:
[Tue Jul 16 12:49:20 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:49:20 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:49:20 2024] Call Trace:
[Tue Jul 16 12:49:20 2024] <IRQ>
[Tue Jul 16 12:49:20 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:49:20 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:49:20 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:49:20 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:49:20 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:49:20 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:49:20 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:49:20 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:49:20 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:49:20 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:49:20 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:49:20 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:49:20 2024] </IRQ>
[Tue Jul 16 12:49:20 2024] <TASK>
[Tue Jul 16 12:49:20 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:49:20 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:49:20 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:49:20 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:49:20 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:49:20 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 12:49:20 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:49:20 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:49:20 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:49:20 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:49:20 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:49:20 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:49:20 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:49:20 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:49:20 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:49:20 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:49:20 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:49:20 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:49:20 2024] kthread+0x115/0x140
[Tue Jul 16 12:49:20 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:49:20 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:49:20 2024] </TASK>
[Tue Jul 16 12:49:24 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 88580 jiffies s: 22005 root: 0x20/.
[Tue Jul 16 12:49:24 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:49:24 2024] Task dump for CPU 5:
[Tue Jul 16 12:49:24 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:49:24 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:49:24 2024] Call Trace:
[Tue Jul 16 12:49:24 2024] <TASK>
[Tue Jul 16 12:49:24 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:49:24 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:49:24 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:49:24 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:49:24 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:49:24 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:49:24 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:49:24 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:49:24 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:49:24 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:49:24 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:49:24 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:49:24 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:49:24 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:49:24 2024] </TASK>
[Tue Jul 16 12:50:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:50:19 2024] rcu: 5-....: (20999 ticks this GP) idle=cbb/1/0x4000000000000000 softirq=30126224/30126224 fqs=4854
[Tue Jul 16 12:50:19 2024] (t=21017 jiffies g=194959201 q=4311)
[Tue Jul 16 12:50:19 2024] Task dump for CPU 5:
[Tue Jul 16 12:50:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:50:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:50:19 2024] Call Trace:
[Tue Jul 16 12:50:19 2024] <IRQ>
[Tue Jul 16 12:50:19 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:50:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:50:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:50:19 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:50:19 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:50:19 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:50:19 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:50:19 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:50:19 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:50:19 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:50:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:50:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:50:19 2024] </IRQ>
[Tue Jul 16 12:50:19 2024] <TASK>
[Tue Jul 16 12:50:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:50:19 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:50:20 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:50:20 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:50:20 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:50:20 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 12:50:20 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:50:20 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:50:20 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:50:20 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:50:20 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:50:20 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:50:20 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:50:20 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:50:20 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:50:20 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:50:20 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:50:20 2024] kthread+0x115/0x140
[Tue Jul 16 12:50:20 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:50:20 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:50:20 2024] </TASK>
[Tue Jul 16 12:50:20 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 21239 jiffies s: 22013 root: 0x20/.
[Tue Jul 16 12:50:20 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:50:20 2024] Task dump for CPU 5:
[Tue Jul 16 12:50:20 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:50:20 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:50:20 2024] Call Trace:
[Tue Jul 16 12:50:20 2024] <TASK>
[Tue Jul 16 12:50:20 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:50:20 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:50:20 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:50:20 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:50:20 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:50:20 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 12:50:20 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:50:20 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:50:20 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:50:20 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:50:20 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:50:20 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:50:20 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:50:20 2024] </TASK>
[Tue Jul 16 12:51:23 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:51:23 2024] rcu: 5-....: (84004 ticks this GP) idle=cbb/1/0x4000000000000000 softirq=30126224/30126224 fqs=18700
[Tue Jul 16 12:51:23 2024] (t=84260 jiffies g=194959201 q=5758)
[Tue Jul 16 12:51:23 2024] Task dump for CPU 5:
[Tue Jul 16 12:51:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:51:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:51:23 2024] Call Trace:
[Tue Jul 16 12:51:23 2024] <IRQ>
[Tue Jul 16 12:51:23 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:51:23 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:51:23 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:51:23 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:51:23 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:51:23 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:51:23 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:51:23 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:51:23 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:51:23 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:51:23 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:51:23 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:51:23 2024] </IRQ>
[Tue Jul 16 12:51:23 2024] <TASK>
[Tue Jul 16 12:51:23 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:51:23 2024] RIP: 0010:rpc_exit_task+0x52/0x100 [sunrpc]
[Tue Jul 16 12:51:23 2024] Code: 00 00 48 8b 50 10 48 85 d2 74 74 48 8b b3 98 00 00 00 48 89 df ff d2 0f 1f 00 48 8b 83 a0 00 00 00 48 8b 40 08 48 85 c0 74 4f <48> 8b b3 98 00 00 00 48 89 df ff d0 0f 1f 00 48 83 7b 20 00 74 39
[Tue Jul 16 12:51:23 2024] RSP: 0018:ffffc900087cfe28 EFLAGS: 00000282
[Tue Jul 16 12:51:23 2024] RAX: ffffffffa01ce520 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 12:51:23 2024] RDX: ffff8895b6f35e00 RSI: ffff889c27aa1c80 RDI: ffff88997131a500
[Tue Jul 16 12:51:23 2024] RBP: ffffffffa00d3230 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:51:23 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:51:23 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:51:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:51:23 2024] ? setup_callback_client+0x3f0/0x3f0 [nfsd]
[Tue Jul 16 12:51:23 2024] ? rpc_exit_task+0xbf/0x100 [sunrpc]
[Tue Jul 16 12:51:23 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:51:23 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:51:23 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:51:23 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:51:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:51:23 2024] kthread+0x115/0x140
[Tue Jul 16 12:51:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:51:23 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:51:23 2024] </TASK>
[Tue Jul 16 12:51:23 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 84543 jiffies s: 22013 root: 0x20/.
[Tue Jul 16 12:51:23 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:51:23 2024] Task dump for CPU 5:
[Tue Jul 16 12:51:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:51:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:51:23 2024] Call Trace:
[Tue Jul 16 12:51:23 2024] <TASK>
[Tue Jul 16 12:51:23 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:51:23 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:51:23 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:51:23 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 12:51:23 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:51:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:51:23 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:51:23 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:51:23 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:51:23 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:51:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:51:23 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:51:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:51:23 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:51:23 2024] </TASK>
[Tue Jul 16 12:51:50 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:51:50 2024] rcu: 1-....: (21000 ticks this GP) idle=e43/1/0x4000000000000000 softirq=30799551/30799551 fqs=4673
[Tue Jul 16 12:51:50 2024] (t=21017 jiffies g=194959205 q=3086)
[Tue Jul 16 12:51:50 2024] Task dump for CPU 1:
[Tue Jul 16 12:51:50 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:51:50 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:51:50 2024] Call Trace:
[Tue Jul 16 12:51:50 2024] <IRQ>
[Tue Jul 16 12:51:50 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:51:50 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:51:50 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:51:50 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:51:50 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:51:50 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:51:50 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:51:50 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:51:50 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:51:50 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:51:50 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:51:50 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:51:50 2024] </IRQ>
[Tue Jul 16 12:51:50 2024] <TASK>
[Tue Jul 16 12:51:50 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:51:50 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:51:50 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:51:50 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:51:50 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:51:50 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc1b880
[Tue Jul 16 12:51:50 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:51:50 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:51:50 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:51:50 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:51:50 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:51:50 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:51:50 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:51:50 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:51:50 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:51:50 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:51:50 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:51:50 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:51:50 2024] kthread+0x115/0x140
[Tue Jul 16 12:51:50 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:51:50 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:51:50 2024] </TASK>
[Tue Jul 16 12:52:53 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:52:53 2024] rcu: 1-....: (84005 ticks this GP) idle=e43/1/0x4000000000000000 softirq=30799551/30799551 fqs=19214
[Tue Jul 16 12:52:53 2024] (t=84260 jiffies g=194959205 q=4653)
[Tue Jul 16 12:52:53 2024] Task dump for CPU 1:
[Tue Jul 16 12:52:53 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:52:53 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:52:53 2024] Call Trace:
[Tue Jul 16 12:52:53 2024] <IRQ>
[Tue Jul 16 12:52:53 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:52:53 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:52:53 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:52:53 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:52:53 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:52:53 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:52:53 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:52:53 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:52:53 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:52:53 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:52:53 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:52:53 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:52:53 2024] </IRQ>
[Tue Jul 16 12:52:53 2024] <TASK>
[Tue Jul 16 12:52:53 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:52:53 2024] RIP: 0010:nfsd4_cb_done+0x97/0x380 [nfsd]
[Tue Jul 16 12:52:53 2024] Code: 8b 45 04 89 c2 83 e2 f7 83 fa f3 0f 84 57 01 00 00 83 f8 92 0f 84 4e 01 00 00 5b 5d 41 5c 41 5d c3 cc cc cc cc 44 0f b6 6e 59 <45> 84 ed 0f 84 9a 00 00 00 8b 46 50 3d e8 d8 ff ff 0f 84 d9 01 00
[Tue Jul 16 12:52:53 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000202
[Tue Jul 16 12:52:53 2024] RAX: ffffffffa01ce520 RBX: ffff888e83163d80 RCX: 0000000000000002
[Tue Jul 16 12:52:53 2024] RDX: ffff888473d77a00 RSI: ffff888e83163d80 RDI: ffff88997131a500
[Tue Jul 16 12:52:53 2024] RBP: ffff88997131a500 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:52:53 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff888696daa000
[Tue Jul 16 12:52:53 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:52:53 2024] ? setup_callback_client+0x3f0/0x3f0 [nfsd]
[Tue Jul 16 12:52:53 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:52:53 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:52:53 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:52:53 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:52:53 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:52:53 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:52:53 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:52:53 2024] kthread+0x115/0x140
[Tue Jul 16 12:52:53 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:52:53 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:52:53 2024] </TASK>
[Tue Jul 16 12:53:36 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:53:36 2024] rcu: 7-....: (20998 ticks this GP) idle=759/1/0x4000000000000000 softirq=29984858/29984858 fqs=4915
[Tue Jul 16 12:53:36 2024] (t=21017 jiffies g=194959209 q=5455)
[Tue Jul 16 12:53:36 2024] Task dump for CPU 7:
[Tue Jul 16 12:53:36 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:53:36 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:53:36 2024] Call Trace:
[Tue Jul 16 12:53:36 2024] <IRQ>
[Tue Jul 16 12:53:36 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:53:36 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:53:36 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:53:36 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:53:36 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:53:36 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:53:36 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:53:36 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:53:36 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:53:36 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:53:36 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:53:36 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:53:36 2024] </IRQ>
[Tue Jul 16 12:53:36 2024] <TASK>
[Tue Jul 16 12:53:36 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:53:36 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:53:36 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:53:36 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:53:36 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002fe00007
[Tue Jul 16 12:53:36 2024] RDX: 00000001421f8f9d RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 12:53:36 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:53:36 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 12:53:36 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:53:36 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:53:36 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:53:36 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:53:36 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:53:36 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:53:36 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:53:36 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:53:36 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:53:36 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:53:36 2024] kthread+0x115/0x140
[Tue Jul 16 12:53:36 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:53:36 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:53:36 2024] </TASK>
[Tue Jul 16 12:54:39 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:54:39 2024] rcu: 7-....: (84003 ticks this GP) idle=759/1/0x4000000000000000 softirq=29984858/29984858 fqs=18961
[Tue Jul 16 12:54:39 2024] (t=84260 jiffies g=194959209 q=7027)
[Tue Jul 16 12:54:39 2024] Task dump for CPU 7:
[Tue Jul 16 12:54:39 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:54:39 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:54:39 2024] Call Trace:
[Tue Jul 16 12:54:39 2024] <IRQ>
[Tue Jul 16 12:54:39 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:54:39 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:54:39 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:54:39 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:54:39 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:54:39 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:54:39 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:54:39 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:54:39 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:54:39 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:54:39 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:54:39 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:54:39 2024] </IRQ>
[Tue Jul 16 12:54:39 2024] <TASK>
[Tue Jul 16 12:54:39 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:54:39 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:54:39 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:54:39 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:54:39 2024] RAX: 0000000090bbf3f0 RBX: 000000003fca8d12 RCX: 0000000000001007
[Tue Jul 16 12:54:39 2024] RDX: 00000000004545e6 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:54:39 2024] RBP: 0003f14a3d8623a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:54:39 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 12:54:39 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:54:39 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:54:39 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:54:39 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:54:39 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:54:39 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:54:39 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:54:39 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:54:39 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:54:39 2024] kthread+0x115/0x140
[Tue Jul 16 12:54:39 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:54:39 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:54:39 2024] </TASK>
[Tue Jul 16 12:55:27 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:55:27 2024] rcu: 15-....: (21000 ticks this GP) idle=197/1/0x4000000000000000 softirq=29720541/29720545 fqs=4484
[Tue Jul 16 12:55:27 2024] (t=21017 jiffies g=194959213 q=5630)
[Tue Jul 16 12:55:27 2024] Task dump for CPU 15:
[Tue Jul 16 12:55:27 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:55:27 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:55:27 2024] Call Trace:
[Tue Jul 16 12:55:27 2024] <IRQ>
[Tue Jul 16 12:55:27 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:55:27 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:55:27 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:55:27 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:55:27 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:55:27 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:55:27 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:55:27 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:55:27 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:55:27 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:55:27 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:55:27 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:55:27 2024] </IRQ>
[Tue Jul 16 12:55:27 2024] <TASK>
[Tue Jul 16 12:55:27 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:55:27 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 12:55:27 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 12:55:27 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 12:55:27 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002fe0000f
[Tue Jul 16 12:55:27 2024] RDX: 0000000142213fa9 RSI: 0000000000000046 RDI: ffff88a07fddb880
[Tue Jul 16 12:55:27 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:55:27 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 12:55:27 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 12:55:27 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:55:27 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:55:27 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:55:27 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:55:27 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:55:27 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:55:27 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:55:27 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:55:27 2024] kthread+0x115/0x140
[Tue Jul 16 12:55:27 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:55:27 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:55:27 2024] </TASK>
[Tue Jul 16 12:55:27 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 21239 jiffies s: 22017 root: 0x8000/.
[Tue Jul 16 12:55:27 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:55:27 2024] Task dump for CPU 15:
[Tue Jul 16 12:55:27 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:55:27 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:55:27 2024] Call Trace:
[Tue Jul 16 12:55:27 2024] <TASK>
[Tue Jul 16 12:55:27 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:55:27 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:55:27 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:55:27 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:55:27 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 12:55:27 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:55:27 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:55:27 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:55:27 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:55:27 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:55:27 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:55:27 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:55:27 2024] </TASK>
[Tue Jul 16 12:55:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:55:57 2024] rcu: 1-....: (20997 ticks this GP) idle=c75/1/0x4000000000000000 softirq=30799571/30799572 fqs=4387
[Tue Jul 16 12:55:57 2024] (t=21017 jiffies g=194959221 q=1332)
[Tue Jul 16 12:55:57 2024] Task dump for CPU 1:
[Tue Jul 16 12:55:57 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:55:57 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:55:57 2024] Call Trace:
[Tue Jul 16 12:55:57 2024] <IRQ>
[Tue Jul 16 12:55:57 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:55:57 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:55:57 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:55:57 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:55:57 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:55:57 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:55:57 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:55:57 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:55:57 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:55:57 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:55:57 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:55:57 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:55:57 2024] </IRQ>
[Tue Jul 16 12:55:57 2024] <TASK>
[Tue Jul 16 12:55:57 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:55:57 2024] RIP: 0010:ktime_get+0x13/0xa0
[Tue Jul 16 12:55:57 2024] Code: ff ff e9 f4 fe ff ff e8 4b 03 a3 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 3d 13 65 01 41 54 55 53 85 c0 75 7a <45> 31 e4 eb 02 f3 90 8b 1d 60 e2 da 01 f6 c3 01 75 f3 48 8b 3d 5c
[Tue Jul 16 12:55:57 2024] RSP: 0018:ffffc900087cfe08 EFLAGS: 00000246
[Tue Jul 16 12:55:57 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000037200001
[Tue Jul 16 12:55:57 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88997131a500
[Tue Jul 16 12:55:57 2024] RBP: ffffffffa00d3230 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:55:57 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 12:55:57 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:55:57 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:55:57 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:55:57 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:55:57 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:55:57 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:55:58 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:55:58 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:55:58 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:55:58 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:55:58 2024] kthread+0x115/0x140
[Tue Jul 16 12:55:58 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:55:58 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:55:58 2024] </TASK>
[Tue Jul 16 12:56:41 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:56:41 2024] rcu: 5-....: (21000 ticks this GP) idle=437/1/0x4000000000000000 softirq=30126276/30126276 fqs=4463
[Tue Jul 16 12:56:41 2024] (t=21007 jiffies g=194959225 q=3177)
[Tue Jul 16 12:56:41 2024] Task dump for CPU 5:
[Tue Jul 16 12:56:41 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:56:41 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:56:41 2024] Call Trace:
[Tue Jul 16 12:56:41 2024] <IRQ>
[Tue Jul 16 12:56:41 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:56:41 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:56:41 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:56:41 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:56:41 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:56:41 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:56:41 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:56:41 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:56:41 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:56:41 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:56:41 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:56:41 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:56:41 2024] </IRQ>
[Tue Jul 16 12:56:41 2024] <TASK>
[Tue Jul 16 12:56:41 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:56:41 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 12:56:41 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 12:56:41 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 12:56:41 2024] RAX: 0000000059a5b5d8 RBX: 000000003fce46f2 RCX: 0000000000001005
[Tue Jul 16 12:56:41 2024] RDX: 0000000000454652 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 12:56:42 2024] RBP: 0003f166a54a67a3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:56:42 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000000
[Tue Jul 16 12:56:42 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:56:42 2024] ktime_get+0x38/0xa0
[Tue Jul 16 12:56:42 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:56:42 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 12:56:42 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:56:42 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:56:42 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:56:42 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:56:42 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:56:42 2024] kthread+0x115/0x140
[Tue Jul 16 12:56:42 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:56:42 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:56:42 2024] </TASK>
[Tue Jul 16 12:56:42 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 21229 jiffies s: 22021 root: 0x20/.
[Tue Jul 16 12:56:42 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:56:42 2024] Task dump for CPU 5:
[Tue Jul 16 12:56:42 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:56:42 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:56:42 2024] Call Trace:
[Tue Jul 16 12:56:42 2024] <TASK>
[Tue Jul 16 12:56:42 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:56:42 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:56:42 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:56:42 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 12:56:42 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:56:42 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:56:42 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:56:42 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:56:42 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:56:42 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:56:42 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:56:42 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:56:42 2024] </TASK>
[Tue Jul 16 12:57:45 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:57:45 2024] rcu: 5-....: (84005 ticks this GP) idle=437/1/0x4000000000000000 softirq=30126276/30126276 fqs=19130
[Tue Jul 16 12:57:45 2024] (t=84250 jiffies g=194959225 q=4596)
[Tue Jul 16 12:57:45 2024] Task dump for CPU 5:
[Tue Jul 16 12:57:45 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:57:45 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:57:45 2024] Call Trace:
[Tue Jul 16 12:57:45 2024] <IRQ>
[Tue Jul 16 12:57:45 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:57:45 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:57:45 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:57:45 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:57:45 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:57:45 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:57:45 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:57:45 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:57:45 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:57:45 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:57:45 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:57:45 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:57:45 2024] </IRQ>
[Tue Jul 16 12:57:45 2024] <TASK>
[Tue Jul 16 12:57:45 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:57:45 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xaa/0x180 [sunrpc]
[Tue Jul 16 12:57:45 2024] Code: 67 49 8b 45 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 <4c> 89 63 28 49 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78
[Tue Jul 16 12:57:45 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 12:57:45 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 12:57:45 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:57:45 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:57:45 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000142235aac
[Tue Jul 16 12:57:45 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 12:57:45 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:57:45 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:57:45 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:57:45 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:57:45 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:57:45 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:57:45 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:57:45 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:57:45 2024] kthread+0x115/0x140
[Tue Jul 16 12:57:45 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:57:45 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:57:45 2024] </TASK>
[Tue Jul 16 12:57:48 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 87596 jiffies s: 22021 root: 0x20/.
[Tue Jul 16 12:57:48 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:57:48 2024] Task dump for CPU 5:
[Tue Jul 16 12:57:48 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:57:48 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:57:48 2024] Call Trace:
[Tue Jul 16 12:57:48 2024] <TASK>
[Tue Jul 16 12:57:48 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:57:48 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:57:48 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 12:57:48 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:57:48 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 12:57:48 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:57:48 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:57:48 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:57:48 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:57:48 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:57:48 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:57:48 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:57:48 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:57:48 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:57:48 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:57:48 2024] </TASK>
[Tue Jul 16 12:58:48 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:58:48 2024] rcu: 5-....: (147009 ticks this GP) idle=437/1/0x4000000000000000 softirq=30126276/30126276 fqs=33949
[Tue Jul 16 12:58:48 2024] (t=147491 jiffies g=194959225 q=7074)
[Tue Jul 16 12:58:48 2024] Task dump for CPU 5:
[Tue Jul 16 12:58:48 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:58:48 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:58:48 2024] Call Trace:
[Tue Jul 16 12:58:48 2024] <IRQ>
[Tue Jul 16 12:58:48 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:58:48 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:58:48 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:58:48 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:58:48 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:58:48 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:58:48 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:58:48 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:58:48 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:58:48 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:58:48 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:58:48 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:58:48 2024] </IRQ>
[Tue Jul 16 12:58:48 2024] <TASK>
[Tue Jul 16 12:58:48 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:58:48 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 12:58:48 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 12:58:48 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 12:58:48 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 12:58:48 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 12:58:48 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:58:48 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 12:58:48 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 12:58:48 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:58:48 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:58:48 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:58:48 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:58:48 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:58:48 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:58:48 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:58:48 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:58:48 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:58:48 2024] kthread+0x115/0x140
[Tue Jul 16 12:58:48 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:58:48 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:58:48 2024] </TASK>
[Tue Jul 16 12:58:53 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 153133 jiffies s: 22021 root: 0x20/.
[Tue Jul 16 12:58:53 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:58:53 2024] Task dump for CPU 5:
[Tue Jul 16 12:58:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:58:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:58:54 2024] Call Trace:
[Tue Jul 16 12:58:54 2024] <TASK>
[Tue Jul 16 12:58:54 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:58:54 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:58:54 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:58:54 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:58:54 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 12:58:54 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 12:58:54 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 12:58:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:58:54 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:58:54 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:58:54 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:58:54 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:58:54 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:58:54 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:58:54 2024] </TASK>
[Tue Jul 16 12:59:51 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 12:59:51 2024] rcu: 5-....: (210014 ticks this GP) idle=437/1/0x4000000000000000 softirq=30126276/30126276 fqs=47653
[Tue Jul 16 12:59:51 2024] (t=210734 jiffies g=194959225 q=11791)
[Tue Jul 16 12:59:51 2024] Task dump for CPU 5:
[Tue Jul 16 12:59:51 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:59:51 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:59:51 2024] Call Trace:
[Tue Jul 16 12:59:51 2024] <IRQ>
[Tue Jul 16 12:59:51 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 12:59:51 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 12:59:51 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 12:59:51 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 12:59:51 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 12:59:51 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 12:59:51 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 12:59:51 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 12:59:51 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 12:59:51 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 12:59:51 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 12:59:51 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 12:59:51 2024] </IRQ>
[Tue Jul 16 12:59:51 2024] <TASK>
[Tue Jul 16 12:59:51 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:59:51 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xb6/0x180 [sunrpc]
[Tue Jul 16 12:59:51 2024] Code: 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 4c 89 63 28 49 8b 45 50 4d 8d 7d 50 <49> 39 c7 74 48 4d 3b 65 60 78 42 49 8b 45 50 4c 89 70 08 48 89 43
[Tue Jul 16 12:59:51 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 12:59:51 2024] RAX: ffffffffa012c750 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 12:59:51 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 12:59:51 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 12:59:51 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 00000001422548bd
[Tue Jul 16 12:59:51 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffffffffa012c750
[Tue Jul 16 12:59:51 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 12:59:51 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:59:51 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 12:59:51 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 12:59:51 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:59:51 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:59:51 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 12:59:51 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:59:51 2024] kthread+0x115/0x140
[Tue Jul 16 12:59:51 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:59:51 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 12:59:51 2024] </TASK>
[Tue Jul 16 12:59:59 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 218669 jiffies s: 22021 root: 0x20/.
[Tue Jul 16 12:59:59 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 12:59:59 2024] Task dump for CPU 5:
[Tue Jul 16 12:59:59 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 12:59:59 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 12:59:59 2024] Call Trace:
[Tue Jul 16 12:59:59 2024] <TASK>
[Tue Jul 16 12:59:59 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 12:59:59 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 12:59:59 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 12:59:59 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 12:59:59 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 12:59:59 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 12:59:59 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 12:59:59 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 12:59:59 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 12:59:59 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 12:59:59 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 12:59:59 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 12:59:59 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 12:59:59 2024] ? kthread+0x115/0x140
[Tue Jul 16 12:59:59 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 12:59:59 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 12:59:59 2024] </TASK>
[Tue Jul 16 13:00:54 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:00:54 2024] rcu: 5-....: (273019 ticks this GP) idle=437/1/0x4000000000000000 softirq=30126276/30126276 fqs=61273
[Tue Jul 16 13:00:54 2024] (t=273973 jiffies g=194959225 q=13587)
[Tue Jul 16 13:00:54 2024] Task dump for CPU 5:
[Tue Jul 16 13:00:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:00:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:00:54 2024] Call Trace:
[Tue Jul 16 13:00:54 2024] <IRQ>
[Tue Jul 16 13:00:54 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:00:54 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:00:54 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:00:54 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:00:54 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:00:54 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:00:54 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:00:54 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:00:54 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:00:54 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:00:54 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:00:54 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:00:54 2024] </IRQ>
[Tue Jul 16 13:00:54 2024] <TASK>
[Tue Jul 16 13:00:54 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:00:54 2024] RIP: 0010:rpc_exit_task+0x89/0x100 [sunrpc]
[Tue Jul 16 13:00:54 2024] Code: 1f 00 48 83 7b 20 00 74 39 48 89 df e8 60 11 ff ff 31 c0 66 81 a3 dc 00 00 00 df f7 66 89 83 de 00 00 00 0f b6 83 e2 00 00 00 <83> e0 c3 83 c8 28 88 83 e2 00 00 00 e8 76 7a 03 e1 48 89 83 d0 00
[Tue Jul 16 13:00:54 2024] RSP: 0018:ffffc900087cfe28 EFLAGS: 00000206
[Tue Jul 16 13:00:54 2024] RAX: 0000000000000029 RBX: ffff88997131a500 RCX: 0000000049200005
[Tue Jul 16 13:00:54 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88997131a500
[Tue Jul 16 13:00:54 2024] RBP: ffffffffa00d3230 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:00:54 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:00:54 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:00:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:00:54 2024] ? rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 13:00:54 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:00:54 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:00:54 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:00:55 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:00:55 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:00:55 2024] kthread+0x115/0x140
[Tue Jul 16 13:00:55 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:00:55 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:00:55 2024] </TASK>
[Tue Jul 16 13:01:05 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 284205 jiffies s: 22021 root: 0x20/.
[Tue Jul 16 13:01:05 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:01:05 2024] Task dump for CPU 5:
[Tue Jul 16 13:01:05 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:01:05 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:01:05 2024] Call Trace:
[Tue Jul 16 13:01:05 2024] <TASK>
[Tue Jul 16 13:01:05 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:01:05 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:01:05 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:01:05 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:01:05 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 13:01:05 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:01:05 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:01:05 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:01:05 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:01:05 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:01:05 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:01:05 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:01:05 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:01:05 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:01:05 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:01:05 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:01:05 2024] </TASK>
[Tue Jul 16 13:01:36 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:01:36 2024] rcu: 13-....: (20998 ticks this GP) idle=1b9/1/0x4000000000000000 softirq=31905290/31905290 fqs=4433
[Tue Jul 16 13:01:36 2024] (t=21017 jiffies g=194959229 q=14541)
[Tue Jul 16 13:01:36 2024] Task dump for CPU 13:
[Tue Jul 16 13:01:36 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:01:36 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:01:36 2024] Call Trace:
[Tue Jul 16 13:01:36 2024] <IRQ>
[Tue Jul 16 13:01:36 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:01:36 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:01:36 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:01:36 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:01:36 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:01:36 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:01:36 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:01:36 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:01:36 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:01:36 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:01:36 2024] </IRQ>
[Tue Jul 16 13:01:36 2024] <TASK>
[Tue Jul 16 13:01:36 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:01:36 2024] RIP: 0010:xprt_release+0x149/0x190 [sunrpc]
[Tue Jul 16 13:01:36 2024] Code: 24 68 04 00 00 e8 67 33 04 e1 e9 4d ff ff ff 48 83 bf a8 00 00 00 00 74 10 48 8b 9f b0 00 00 00 48 3b bb c8 04 00 00 74 09 5b <5d> 41 5c c3 cc cc cc cc 48 8d bb b8 04 00 00 e8 93 57 a8 e1 48 8b
[Tue Jul 16 13:01:36 2024] RSP: 0018:ffffc900087cfe10 EFLAGS: 00000286
[Tue Jul 16 13:01:36 2024] RAX: ffffffffa012c750 RBX: ffff88997131a500 RCX: 000000002420000d
[Tue Jul 16 13:01:36 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88997131a500
[Tue Jul 16 13:01:36 2024] RBP: ffff88997131a500 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:01:36 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:01:36 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:01:36 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:01:36 2024] rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 13:01:36 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:01:37 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:01:37 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:01:37 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:01:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:01:37 2024] kthread+0x115/0x140
[Tue Jul 16 13:01:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:01:37 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:01:37 2024] </TASK>
[Tue Jul 16 13:02:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:02:40 2024] rcu: 13-....: (84003 ticks this GP) idle=1b9/1/0x4000000000000000 softirq=31905290/31905290 fqs=18411
[Tue Jul 16 13:02:40 2024] (t=84241 jiffies g=194959229 q=18520)
[Tue Jul 16 13:02:40 2024] Task dump for CPU 13:
[Tue Jul 16 13:02:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:02:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:02:40 2024] Call Trace:
[Tue Jul 16 13:02:40 2024] <IRQ>
[Tue Jul 16 13:02:40 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:02:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:02:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:02:40 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:02:40 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:02:40 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:02:40 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:02:40 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:02:40 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:02:40 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:02:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:02:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:02:40 2024] </IRQ>
[Tue Jul 16 13:02:40 2024] <TASK>
[Tue Jul 16 13:02:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:02:40 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:02:40 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:02:40 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:02:40 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004fa0000d
[Tue Jul 16 13:02:40 2024] RDX: 000000014227dae1 RSI: 0000000000000046 RDI: ffff88a07fd9b880
[Tue Jul 16 13:02:40 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:02:40 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 13:02:40 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:02:40 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:02:40 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:02:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:02:40 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:02:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:02:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:02:40 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:02:40 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:02:40 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:02:40 2024] kthread+0x115/0x140
[Tue Jul 16 13:02:40 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:02:40 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:02:40 2024] </TASK>
[Tue Jul 16 13:03:55 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:03:55 2024] rcu: 5-....: (21001 ticks this GP) idle=cc1/1/0x4000000000000000 softirq=30126281/30126282 fqs=4992
[Tue Jul 16 13:03:55 2024] (t=21017 jiffies g=194959233 q=18837)
[Tue Jul 16 13:03:55 2024] Task dump for CPU 5:
[Tue Jul 16 13:03:55 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:03:55 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:03:55 2024] Call Trace:
[Tue Jul 16 13:03:55 2024] <IRQ>
[Tue Jul 16 13:03:55 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:03:55 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:03:55 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:03:55 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:03:55 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:03:55 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:03:55 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:03:55 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:03:55 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:03:55 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:03:55 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:03:55 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:03:55 2024] </IRQ>
[Tue Jul 16 13:03:55 2024] <TASK>
[Tue Jul 16 13:03:55 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:03:55 2024] RIP: 0010:__rpc_sleep_on_priority_timeout+0xaa/0x180 [sunrpc]
[Tue Jul 16 13:03:55 2024] Code: 67 49 8b 45 10 48 8d 53 40 49 8d 4d 08 49 89 55 10 48 89 4b 40 48 89 43 48 48 89 10 4c 89 6b 38 41 83 45 4c 01 f0 80 4b 30 02 <4c> 89 63 28 49 8b 45 50 4d 8d 7d 50 49 39 c7 74 48 4d 3b 65 60 78
[Tue Jul 16 13:03:55 2024] RSP: 0018:ffffc900087cfdc0 EFLAGS: 00000206
[Tue Jul 16 13:03:55 2024] RAX: ffffffffa012c708 RBX: ffff88997131a500 RCX: ffffffffa012c708
[Tue Jul 16 13:03:55 2024] RDX: ffff88997131a540 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 13:03:55 2024] RBP: 0000000000000000 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:03:55 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 000000014228ffdd
[Tue Jul 16 13:03:55 2024] R13: ffffffffa012c700 R14: ffff88997131a560 R15: ffff88909311c005
[Tue Jul 16 13:03:55 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:03:55 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:03:55 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:03:55 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:03:55 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:03:55 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:03:55 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:03:55 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:03:55 2024] kthread+0x115/0x140
[Tue Jul 16 13:03:55 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:03:55 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:03:55 2024] </TASK>
[Tue Jul 16 13:04:37 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:04:37 2024] rcu: 13-....: (20996 ticks this GP) idle=9cd/1/0x4000000000000000 softirq=31905300/31905309 fqs=5017
[Tue Jul 16 13:04:37 2024] (t=21013 jiffies g=194959237 q=6928)
[Tue Jul 16 13:04:37 2024] Task dump for CPU 13:
[Tue Jul 16 13:04:37 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:04:37 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:04:37 2024] Call Trace:
[Tue Jul 16 13:04:37 2024] <IRQ>
[Tue Jul 16 13:04:37 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:04:37 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:04:37 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:04:37 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:04:37 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:04:37 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:04:37 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:04:37 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:04:37 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:04:37 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:04:37 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:04:37 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:04:37 2024] </IRQ>
[Tue Jul 16 13:04:37 2024] <TASK>
[Tue Jul 16 13:04:37 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:04:37 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:04:37 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:04:37 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:04:37 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:04:37 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd9b880
[Tue Jul 16 13:04:37 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:04:37 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:04:37 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:04:37 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:04:37 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:04:37 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:04:37 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:04:37 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:04:37 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:04:37 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:04:37 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:04:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:04:37 2024] kthread+0x115/0x140
[Tue Jul 16 13:04:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:04:37 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:04:37 2024] </TASK>
[Tue Jul 16 13:05:02 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:05:02 2024] rcu: 5-....: (21000 ticks this GP) idle=4c1/1/0x4000000000000000 softirq=30126288/30126289 fqs=4714
[Tue Jul 16 13:05:02 2024] (t=21013 jiffies g=194959241 q=3209)
[Tue Jul 16 13:05:02 2024] Task dump for CPU 5:
[Tue Jul 16 13:05:02 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:05:02 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:05:02 2024] Call Trace:
[Tue Jul 16 13:05:02 2024] <IRQ>
[Tue Jul 16 13:05:02 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:05:02 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:05:02 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:05:02 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:05:02 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:05:02 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:05:02 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:05:02 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:05:02 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:05:02 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:05:02 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:05:02 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:05:02 2024] </IRQ>
[Tue Jul 16 13:05:02 2024] <TASK>
[Tue Jul 16 13:05:02 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:05:02 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 13:05:02 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 13:05:02 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 13:05:02 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 13:05:02 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 13:05:02 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:05:02 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 00000001422a0009
[Tue Jul 16 13:05:02 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:05:02 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 13:05:02 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:05:02 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:05:02 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:05:02 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:05:02 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:05:02 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:05:02 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:05:02 2024] kthread+0x115/0x140
[Tue Jul 16 13:05:02 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:05:02 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:05:02 2024] </TASK>
[Tue Jul 16 13:05:03 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 21498 jiffies s: 22025 root: 0x20/.
[Tue Jul 16 13:05:03 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:05:03 2024] Task dump for CPU 5:
[Tue Jul 16 13:05:03 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:05:03 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:05:03 2024] Call Trace:
[Tue Jul 16 13:05:03 2024] <TASK>
[Tue Jul 16 13:05:03 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:05:03 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:05:03 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:05:03 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:05:03 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:05:03 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:05:03 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:05:03 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:05:03 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:05:03 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:05:03 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:05:03 2024] </TASK>
[Tue Jul 16 13:05:55 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:05:55 2024] rcu: 7-....: (21000 ticks this GP) idle=f7d/1/0x4000000000000000 softirq=29984931/29984931 fqs=4653
[Tue Jul 16 13:05:55 2024] (t=21016 jiffies g=194959245 q=4176)
[Tue Jul 16 13:05:55 2024] Task dump for CPU 7:
[Tue Jul 16 13:05:55 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:05:55 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:05:55 2024] Call Trace:
[Tue Jul 16 13:05:55 2024] <IRQ>
[Tue Jul 16 13:05:55 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:05:55 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:05:55 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:05:55 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:05:55 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:05:55 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:05:55 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:05:55 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:05:55 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:05:55 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:05:55 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:05:55 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:05:55 2024] </IRQ>
[Tue Jul 16 13:05:55 2024] <TASK>
[Tue Jul 16 13:05:55 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:05:55 2024] RIP: 0010:try_to_grab_pending+0x14/0x150
[Tue Jul 16 13:05:55 2024] Code: 0f 0b e9 76 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 d5 53 48 89 fb 48 83 ec 08 9c <58> fa 48 89 02 40 84 f6 0f 85 9d 00 00 00 f0 48 0f ba 2b 00 72 0f
[Tue Jul 16 13:05:56 2024] RSP: 0018:ffffc900087cfd50 EFLAGS: 00000292
[Tue Jul 16 13:05:56 2024] RAX: 0000000000000000 RBX: ffffffffa012c768 RCX: 00000000000007d0
[Tue Jul 16 13:05:56 2024] RDX: ffffc900087cfd80 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 13:05:56 2024] RBP: ffffc900087cfd80 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:05:56 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:05:56 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:05:56 2024] mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 13:05:56 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:05:56 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:05:56 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:05:56 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:05:56 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:05:56 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:05:56 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:05:56 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:05:56 2024] kthread+0x115/0x140
[Tue Jul 16 13:05:56 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:05:56 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:05:56 2024] </TASK>
[Tue Jul 16 13:05:56 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 21501 jiffies s: 22029 root: 0x80/.
[Tue Jul 16 13:05:56 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:05:56 2024] Task dump for CPU 7:
[Tue Jul 16 13:05:56 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:05:56 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:05:56 2024] Call Trace:
[Tue Jul 16 13:05:56 2024] <TASK>
[Tue Jul 16 13:05:56 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:05:56 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:05:56 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:05:56 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:05:56 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:05:56 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:05:56 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:05:56 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:05:56 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:05:56 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:05:56 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:05:56 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:05:56 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:05:56 2024] </TASK>
[Tue Jul 16 13:06:59 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:06:59 2024] rcu: 7-....: (84005 ticks this GP) idle=f7d/1/0x4000000000000000 softirq=29984931/29984931 fqs=18835
[Tue Jul 16 13:06:59 2024] (t=84265 jiffies g=194959245 q=6153)
[Tue Jul 16 13:06:59 2024] Task dump for CPU 7:
[Tue Jul 16 13:06:59 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:06:59 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:06:59 2024] Call Trace:
[Tue Jul 16 13:06:59 2024] <IRQ>
[Tue Jul 16 13:06:59 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:06:59 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:06:59 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:06:59 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:06:59 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:06:59 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:06:59 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:06:59 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:06:59 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:06:59 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:06:59 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:06:59 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:06:59 2024] </IRQ>
[Tue Jul 16 13:06:59 2024] <TASK>
[Tue Jul 16 13:06:59 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:06:59 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:06:59 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:06:59 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:06:59 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004f600007
[Tue Jul 16 13:06:59 2024] RDX: 00000001422bceee RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 13:06:59 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:06:59 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:06:59 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:06:59 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:06:59 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:06:59 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:06:59 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:06:59 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:06:59 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:06:59 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:06:59 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:06:59 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:06:59 2024] kthread+0x115/0x140
[Tue Jul 16 13:06:59 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:06:59 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:06:59 2024] </TASK>
[Tue Jul 16 13:07:01 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 7-... } 86525 jiffies s: 22029 root: 0x80/.
[Tue Jul 16 13:07:01 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:07:01 2024] Task dump for CPU 7:
[Tue Jul 16 13:07:01 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:07:01 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:07:01 2024] Call Trace:
[Tue Jul 16 13:07:01 2024] <TASK>
[Tue Jul 16 13:07:01 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:07:01 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:07:01 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:07:01 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 13:07:01 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:07:01 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:07:01 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:07:01 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:07:01 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:07:01 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:07:01 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:07:01 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:07:01 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:07:01 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:07:01 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:07:01 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:07:01 2024] </TASK>
[Tue Jul 16 13:08:03 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:08:03 2024] rcu: 15-....: (20999 ticks this GP) idle=1c3/1/0x4000000000000000 softirq=29720581/29720581 fqs=5200
[Tue Jul 16 13:08:03 2024] (t=21015 jiffies g=194959249 q=9412)
[Tue Jul 16 13:08:03 2024] Task dump for CPU 15:
[Tue Jul 16 13:08:03 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:08:03 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:08:03 2024] Call Trace:
[Tue Jul 16 13:08:03 2024] <IRQ>
[Tue Jul 16 13:08:03 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:08:03 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:08:03 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:08:03 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:08:03 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:08:03 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:08:03 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:08:03 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:08:03 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:08:03 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:08:03 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:08:03 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:08:03 2024] </IRQ>
[Tue Jul 16 13:08:03 2024] <TASK>
[Tue Jul 16 13:08:03 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:08:03 2024] RIP: 0010:rpc_make_runnable+0x18/0x70 [sunrpc]
[Tue Jul 16 13:08:03 2024] Code: 00 00 e9 6b ed 0b e1 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8d 56 30 f0 48 0f ba 6e 30 00 0f 92 c0 f0 80 66 30 fd <84> c0 75 49 f6 86 dc 00 00 00 01 74 33 48 b8 e0 ff ff ff 0f 00 00
[Tue Jul 16 13:08:04 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000202
[Tue Jul 16 13:08:04 2024] RAX: dead000000000101 RBX: ffff88997131a500 RCX: 0000000000000001
[Tue Jul 16 13:08:04 2024] RDX: ffff88997131a530 RSI: ffff88997131a500 RDI: ffff888107352000
[Tue Jul 16 13:08:04 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:08:04 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:08:04 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:08:04 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:08:04 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:08:04 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:08:04 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:08:04 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:08:04 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:08:04 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:08:04 2024] kthread+0x115/0x140
[Tue Jul 16 13:08:04 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:08:04 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:08:04 2024] </TASK>
[Tue Jul 16 13:08:04 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 15-... } 21504 jiffies s: 22033 root: 0x8000/.
[Tue Jul 16 13:08:04 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:08:04 2024] Task dump for CPU 15:
[Tue Jul 16 13:08:04 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:08:04 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:08:04 2024] Call Trace:
[Tue Jul 16 13:08:04 2024] <TASK>
[Tue Jul 16 13:08:04 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:08:04 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:08:04 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:08:04 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 13:08:04 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:08:04 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:08:04 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 13:08:04 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:08:04 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:08:04 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:08:04 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:08:04 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:08:04 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:08:04 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:08:04 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:08:04 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:08:04 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:08:04 2024] </TASK>
[Tue Jul 16 13:08:32 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:08:32 2024] rcu: 3-....: (20999 ticks this GP) idle=ba9/1/0x4000000000000000 softirq=31469503/31469506 fqs=5251
[Tue Jul 16 13:08:32 2024] (t=21013 jiffies g=194959257 q=1866)
[Tue Jul 16 13:08:32 2024] Task dump for CPU 3:
[Tue Jul 16 13:08:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:08:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:08:32 2024] Call Trace:
[Tue Jul 16 13:08:32 2024] <IRQ>
[Tue Jul 16 13:08:32 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:08:32 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:08:32 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:08:32 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:08:32 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:08:32 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:08:32 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:08:32 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:08:32 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:08:32 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:08:32 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:08:32 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:08:32 2024] </IRQ>
[Tue Jul 16 13:08:32 2024] <TASK>
[Tue Jul 16 13:08:32 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:08:32 2024] RIP: 0010:mod_delayed_work_on+0x40/0xa0
[Tue Jul 16 13:08:32 2024] Code: 89 d5 53 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 48 c7 04 24 00 00 00 00 48 89 e2 be 01 00 00 00 48 89 ef <e8> 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea
[Tue Jul 16 13:08:32 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000246
[Tue Jul 16 13:08:32 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 00000000000007d0
[Tue Jul 16 13:08:32 2024] RDX: ffffc900087cfd80 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 13:08:32 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:08:32 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:08:32 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:08:32 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:08:32 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:08:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:08:32 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:08:32 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:08:32 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:08:32 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:08:32 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:08:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:08:32 2024] kthread+0x115/0x140
[Tue Jul 16 13:08:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:08:32 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:08:32 2024] </TASK>
[Tue Jul 16 13:09:52 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:09:52 2024] rcu: 11-....: (21000 ticks this GP) idle=d5b/1/0x4000000000000000 softirq=35298547/35298547 fqs=5225
[Tue Jul 16 13:09:52 2024] (t=21016 jiffies g=194959261 q=6148)
[Tue Jul 16 13:09:52 2024] Task dump for CPU 11:
[Tue Jul 16 13:09:52 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:09:52 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:09:52 2024] Call Trace:
[Tue Jul 16 13:09:52 2024] <IRQ>
[Tue Jul 16 13:09:52 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:09:52 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:09:52 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:09:52 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:09:52 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:09:52 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:09:52 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:09:52 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:09:52 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:09:52 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:09:52 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:09:52 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:09:52 2024] </IRQ>
[Tue Jul 16 13:09:52 2024] <TASK>
[Tue Jul 16 13:09:52 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:09:52 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:09:52 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:09:52 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:09:52 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:09:52 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:09:52 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:09:52 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:09:52 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:09:52 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:09:52 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:09:52 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:09:52 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:09:52 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:09:52 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:09:52 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:09:52 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:09:52 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:09:52 2024] kthread+0x115/0x140
[Tue Jul 16 13:09:52 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:09:52 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:09:52 2024] </TASK>
[Tue Jul 16 13:11:13 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:11:13 2024] rcu: 3-....: (21001 ticks this GP) idle=40d/1/0x4000000000000000 softirq=31469509/31469509 fqs=5126
[Tue Jul 16 13:11:13 2024] (t=21013 jiffies g=194959265 q=8095)
[Tue Jul 16 13:11:13 2024] Task dump for CPU 3:
[Tue Jul 16 13:11:13 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:11:13 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:11:13 2024] Call Trace:
[Tue Jul 16 13:11:13 2024] <IRQ>
[Tue Jul 16 13:11:13 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:11:13 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:11:13 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:11:13 2024] ? fq_ring_free+0xb0/0xb0
[Tue Jul 16 13:11:13 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:11:13 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:11:13 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:11:13 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:11:13 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:11:13 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:11:13 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:11:13 2024] </IRQ>
[Tue Jul 16 13:11:13 2024] <TASK>
[Tue Jul 16 13:11:13 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:11:13 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:11:13 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:11:13 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:11:13 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000020200003
[Tue Jul 16 13:11:13 2024] RDX: 00000001422fafdd RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:11:13 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:11:13 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:11:13 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:11:13 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:11:13 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:11:13 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:11:13 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:11:13 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:11:13 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:11:13 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:11:13 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:11:13 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:11:13 2024] kthread+0x115/0x140
[Tue Jul 16 13:11:13 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:11:13 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:11:13 2024] </TASK>
[Tue Jul 16 13:11:51 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:11:51 2024] rcu: 11-....: (21000 ticks this GP) idle=587/1/0x4000000000000000 softirq=35298550/35298550 fqs=5191
[Tue Jul 16 13:11:51 2024] (t=21017 jiffies g=194959269 q=7548)
[Tue Jul 16 13:11:51 2024] Task dump for CPU 11:
[Tue Jul 16 13:11:51 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:11:51 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:11:51 2024] Call Trace:
[Tue Jul 16 13:11:51 2024] <IRQ>
[Tue Jul 16 13:11:51 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:11:51 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:11:51 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:11:51 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:11:51 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:11:51 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:11:51 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:11:51 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:11:51 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:11:51 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:11:51 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:11:51 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:11:51 2024] </IRQ>
[Tue Jul 16 13:11:51 2024] <TASK>
[Tue Jul 16 13:11:51 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:11:51 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:11:51 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:11:51 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:11:51 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:11:51 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:11:51 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:11:51 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:11:51 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:11:51 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:11:51 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:11:51 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:11:51 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:11:51 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:11:51 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:11:51 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:11:51 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:11:51 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:11:51 2024] kthread+0x115/0x140
[Tue Jul 16 13:11:51 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:11:51 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:11:51 2024] </TASK>
[Tue Jul 16 13:12:30 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:12:30 2024] rcu: 3-....: (20998 ticks this GP) idle=44d/1/0x4000000000000000 softirq=31469516/31469518 fqs=5243
[Tue Jul 16 13:12:30 2024] (t=21014 jiffies g=194959273 q=3518)
[Tue Jul 16 13:12:30 2024] Task dump for CPU 3:
[Tue Jul 16 13:12:30 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:12:30 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:12:30 2024] Call Trace:
[Tue Jul 16 13:12:30 2024] <IRQ>
[Tue Jul 16 13:12:30 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:12:30 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:12:30 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:12:30 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:12:30 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:12:30 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:12:30 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:12:30 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:12:30 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:12:30 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:12:30 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:12:30 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:12:30 2024] </IRQ>
[Tue Jul 16 13:12:30 2024] <TASK>
[Tue Jul 16 13:12:30 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:12:30 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:12:30 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:12:30 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:12:30 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:12:30 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:12:30 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:12:30 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:12:30 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:12:30 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:12:30 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:12:30 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:12:30 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:12:30 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:12:30 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:12:30 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:12:30 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:12:30 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:12:30 2024] kthread+0x115/0x140
[Tue Jul 16 13:12:30 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:12:30 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:12:30 2024] </TASK>
[Tue Jul 16 13:13:33 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:13:33 2024] rcu: 3-....: (83999 ticks this GP) idle=44d/1/0x4000000000000000 softirq=31469516/31469518 fqs=20692
[Tue Jul 16 13:13:33 2024] (t=84253 jiffies g=194959273 q=5483)
[Tue Jul 16 13:13:33 2024] Task dump for CPU 3:
[Tue Jul 16 13:13:33 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:13:33 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:13:33 2024] Call Trace:
[Tue Jul 16 13:13:33 2024] <IRQ>
[Tue Jul 16 13:13:33 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:13:33 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:13:33 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:13:33 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:13:33 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:13:33 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:13:33 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:13:33 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:13:33 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:13:33 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:13:33 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:13:33 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:13:33 2024] </IRQ>
[Tue Jul 16 13:13:33 2024] <TASK>
[Tue Jul 16 13:13:33 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:13:33 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:13:33 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:13:33 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:13:33 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:13:33 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:13:33 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:13:33 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:13:33 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:13:33 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:13:33 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:13:33 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:13:33 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:13:33 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:13:33 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:13:33 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:13:33 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:13:33 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:13:33 2024] kthread+0x115/0x140
[Tue Jul 16 13:13:33 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:13:33 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:13:33 2024] </TASK>
[Tue Jul 16 13:14:06 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:14:06 2024] rcu: 3-....: (20998 ticks this GP) idle=445/1/0x4000000000000000 softirq=31469524/31469526 fqs=5183
[Tue Jul 16 13:14:06 2024] (t=21017 jiffies g=194959281 q=3330)
[Tue Jul 16 13:14:06 2024] Task dump for CPU 3:
[Tue Jul 16 13:14:06 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:14:06 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:14:06 2024] Call Trace:
[Tue Jul 16 13:14:06 2024] <IRQ>
[Tue Jul 16 13:14:06 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:14:06 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:14:06 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:14:06 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:14:06 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:14:06 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:14:06 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:14:06 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:14:06 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:14:06 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:14:06 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:14:06 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:14:06 2024] </IRQ>
[Tue Jul 16 13:14:06 2024] <TASK>
[Tue Jul 16 13:14:06 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:14:06 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:14:06 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:14:06 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:14:06 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:14:06 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:14:06 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:14:06 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:14:06 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:14:06 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:14:06 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:14:06 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:14:06 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:14:06 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:14:06 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:14:06 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:14:06 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:14:06 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:14:06 2024] kthread+0x115/0x140
[Tue Jul 16 13:14:06 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:14:06 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:14:06 2024] </TASK>
[Tue Jul 16 13:15:09 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:15:09 2024] rcu: 3-....: (84003 ticks this GP) idle=445/1/0x4000000000000000 softirq=31469524/31469526 fqs=20277
[Tue Jul 16 13:15:09 2024] (t=84256 jiffies g=194959281 q=5454)
[Tue Jul 16 13:15:09 2024] Task dump for CPU 3:
[Tue Jul 16 13:15:09 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:15:09 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:15:09 2024] Call Trace:
[Tue Jul 16 13:15:09 2024] <IRQ>
[Tue Jul 16 13:15:09 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:15:09 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:15:09 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:15:09 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:15:09 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:15:09 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:15:09 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:15:09 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:15:09 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:15:09 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:15:09 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:15:09 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:15:09 2024] </IRQ>
[Tue Jul 16 13:15:09 2024] <TASK>
[Tue Jul 16 13:15:09 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:15:09 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:15:09 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:15:09 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:15:09 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004d600003
[Tue Jul 16 13:15:09 2024] RDX: 0000000142334ae5 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:15:09 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:15:09 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 13:15:09 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:15:09 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:15:09 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:15:09 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:15:09 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:15:09 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:15:09 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:15:09 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:15:09 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:15:09 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:15:09 2024] kthread+0x115/0x140
[Tue Jul 16 13:15:09 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:15:09 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:15:09 2024] </TASK>
[Tue Jul 16 13:16:25 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:16:25 2024] rcu: 11-....: (21001 ticks this GP) idle=729/1/0x4000000000000000 softirq=35298560/35298560 fqs=5130
[Tue Jul 16 13:16:25 2024] (t=21014 jiffies g=194959285 q=5501)
[Tue Jul 16 13:16:25 2024] Task dump for CPU 11:
[Tue Jul 16 13:16:25 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:16:25 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:16:25 2024] Call Trace:
[Tue Jul 16 13:16:25 2024] <IRQ>
[Tue Jul 16 13:16:25 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:16:25 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:16:25 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:16:25 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:16:25 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:16:25 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:16:25 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:16:25 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:16:25 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:16:25 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:16:25 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:16:25 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:16:25 2024] </IRQ>
[Tue Jul 16 13:16:25 2024] <TASK>
[Tue Jul 16 13:16:25 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:16:25 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:16:25 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:16:25 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:16:25 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:16:25 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:16:25 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:16:25 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:16:25 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:16:25 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:16:25 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:16:25 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:16:25 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:16:25 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:16:25 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:16:25 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:16:25 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:16:25 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:16:25 2024] kthread+0x115/0x140
[Tue Jul 16 13:16:25 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:16:25 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:16:25 2024] </TASK>
[Tue Jul 16 13:16:26 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 21494 jiffies s: 22037 root: 0x800/.
[Tue Jul 16 13:16:26 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:16:26 2024] Task dump for CPU 11:
[Tue Jul 16 13:16:26 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:16:26 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:16:26 2024] Call Trace:
[Tue Jul 16 13:16:26 2024] <TASK>
[Tue Jul 16 13:16:26 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:16:26 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:16:26 2024] ? mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:16:26 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:16:26 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:16:26 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:16:26 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:16:26 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:16:26 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:16:26 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:16:26 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:16:26 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:16:26 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:16:26 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:16:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:16:26 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:16:26 2024] </TASK>
[Tue Jul 16 13:17:28 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:17:28 2024] rcu: 11-....: (84006 ticks this GP) idle=729/1/0x4000000000000000 softirq=35298560/35298560 fqs=20497
[Tue Jul 16 13:17:28 2024] (t=84261 jiffies g=194959285 q=7351)
[Tue Jul 16 13:17:28 2024] Task dump for CPU 11:
[Tue Jul 16 13:17:28 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:17:28 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:17:28 2024] Call Trace:
[Tue Jul 16 13:17:28 2024] <IRQ>
[Tue Jul 16 13:17:28 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:17:28 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:17:28 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:17:28 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:17:28 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:17:28 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:17:28 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:17:28 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:17:28 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:17:28 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:17:28 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:17:28 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:17:28 2024] </IRQ>
[Tue Jul 16 13:17:28 2024] <TASK>
[Tue Jul 16 13:17:28 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:17:28 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:17:28 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:17:28 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:17:29 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000045e0000b
[Tue Jul 16 13:17:29 2024] RDX: 0000000142356ae8 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:17:29 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:17:29 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000000000200
[Tue Jul 16 13:17:29 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:17:29 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:17:29 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:17:29 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:17:29 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:17:29 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:17:29 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:17:29 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:17:29 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:17:29 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:17:29 2024] kthread+0x115/0x140
[Tue Jul 16 13:17:29 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:17:29 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:17:29 2024] </TASK>
[Tue Jul 16 13:17:32 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 87541 jiffies s: 22037 root: 0x800/.
[Tue Jul 16 13:17:32 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:17:32 2024] Task dump for CPU 11:
[Tue Jul 16 13:17:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:17:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:17:32 2024] Call Trace:
[Tue Jul 16 13:17:32 2024] <TASK>
[Tue Jul 16 13:17:32 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:17:32 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:17:32 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:17:32 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:17:32 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:17:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:17:32 2024] ? __rpc_execute+0x95/0x410 [sunrpc]
[Tue Jul 16 13:17:32 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:17:32 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:17:32 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:17:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:17:32 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:17:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:17:32 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:17:32 2024] </TASK>
[Tue Jul 16 13:18:32 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:18:32 2024] rcu: 11-....: (147011 ticks this GP) idle=729/1/0x4000000000000000 softirq=35298560/35298560 fqs=36230
[Tue Jul 16 13:18:32 2024] (t=147501 jiffies g=194959285 q=10431)
[Tue Jul 16 13:18:32 2024] Task dump for CPU 11:
[Tue Jul 16 13:18:32 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:18:32 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:18:32 2024] Call Trace:
[Tue Jul 16 13:18:32 2024] <IRQ>
[Tue Jul 16 13:18:32 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:18:32 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:18:32 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:18:32 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:18:32 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:18:32 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:18:32 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:18:32 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:18:32 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:18:32 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:18:32 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:18:32 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:18:32 2024] </IRQ>
[Tue Jul 16 13:18:32 2024] <TASK>
[Tue Jul 16 13:18:32 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:18:32 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 13:18:32 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 13:18:32 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 13:18:32 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000002
[Tue Jul 16 13:18:32 2024] RDX: 0000000000000001 RSI: 00000000000007d0 RDI: ffffffffa012c700
[Tue Jul 16 13:18:32 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:18:32 2024] R10: 0000000000000003 R11: 0000000000000287 R12: 0000000142365a24
[Tue Jul 16 13:18:32 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:18:32 2024] rpc_delay+0x39/0x90 [sunrpc]
[Tue Jul 16 13:18:32 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:18:32 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:18:32 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:18:32 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:18:32 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:18:32 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:18:32 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:18:32 2024] kthread+0x115/0x140
[Tue Jul 16 13:18:32 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:18:32 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:18:32 2024] </TASK>
[Tue Jul 16 13:18:37 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 153078 jiffies s: 22037 root: 0x800/.
[Tue Jul 16 13:18:37 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:18:37 2024] Task dump for CPU 11:
[Tue Jul 16 13:18:37 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:18:37 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:18:37 2024] Call Trace:
[Tue Jul 16 13:18:37 2024] <TASK>
[Tue Jul 16 13:18:37 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:18:37 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:18:37 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:18:37 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:18:37 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:18:37 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 13:18:37 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:18:37 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:18:37 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:18:37 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:18:37 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:18:37 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:18:37 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:18:37 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:18:37 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:18:37 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:18:37 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:18:37 2024] </TASK>
[Tue Jul 16 13:19:35 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:19:35 2024] rcu: 11-....: (210013 ticks this GP) idle=729/1/0x4000000000000000 softirq=35298560/35298560 fqs=51764
[Tue Jul 16 13:19:35 2024] (t=210738 jiffies g=194959285 q=14306)
[Tue Jul 16 13:19:35 2024] Task dump for CPU 11:
[Tue Jul 16 13:19:35 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:19:35 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:19:35 2024] Call Trace:
[Tue Jul 16 13:19:35 2024] <IRQ>
[Tue Jul 16 13:19:35 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:19:35 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:19:35 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:19:35 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:19:35 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:19:35 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:19:35 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:19:35 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:19:35 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:19:35 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:19:35 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:19:35 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:19:35 2024] </IRQ>
[Tue Jul 16 13:19:35 2024] <TASK>
[Tue Jul 16 13:19:35 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:19:35 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:19:35 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:19:35 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:19:35 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:19:35 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:19:35 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:19:35 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:19:35 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:19:35 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:19:35 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:19:35 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:19:35 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:19:35 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:19:35 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:19:35 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:19:35 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:19:35 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:19:35 2024] kthread+0x115/0x140
[Tue Jul 16 13:19:35 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:19:35 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:19:35 2024] </TASK>
[Tue Jul 16 13:19:43 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 218614 jiffies s: 22037 root: 0x800/.
[Tue Jul 16 13:19:43 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:19:43 2024] Task dump for CPU 11:
[Tue Jul 16 13:19:43 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:19:43 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:19:43 2024] Call Trace:
[Tue Jul 16 13:19:43 2024] <TASK>
[Tue Jul 16 13:19:43 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:19:43 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:19:43 2024] ? del_timer+0x4e/0x80
[Tue Jul 16 13:19:43 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:19:43 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:19:43 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:19:43 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:19:43 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:19:43 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:19:43 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:19:43 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:19:43 2024] </TASK>
[Tue Jul 16 13:20:38 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:20:38 2024] rcu: 11-....: (273015 ticks this GP) idle=729/1/0x4000000000000000 softirq=35298560/35298560 fqs=66542
[Tue Jul 16 13:20:38 2024] (t=273979 jiffies g=194959285 q=19141)
[Tue Jul 16 13:20:38 2024] Task dump for CPU 11:
[Tue Jul 16 13:20:38 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:20:38 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:20:38 2024] Call Trace:
[Tue Jul 16 13:20:38 2024] <IRQ>
[Tue Jul 16 13:20:38 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:20:38 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:20:38 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:20:38 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:20:38 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:20:38 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:20:38 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:20:38 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:20:38 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:20:38 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:20:38 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:20:38 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:20:38 2024] </IRQ>
[Tue Jul 16 13:20:38 2024] <TASK>
[Tue Jul 16 13:20:38 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:20:38 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 13:20:38 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 13:20:38 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000246
[Tue Jul 16 13:20:38 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000017
[Tue Jul 16 13:20:38 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 13:20:38 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:20:38 2024] R10: 0000000000000003 R11: 0000000000000287 R12: ffff88997131a530
[Tue Jul 16 13:20:38 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:20:38 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:20:38 2024] rpc_wake_up_queued_task+0x1f/0x50 [sunrpc]
[Tue Jul 16 13:20:38 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:20:38 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:20:38 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:20:38 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:20:38 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:20:38 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:20:38 2024] kthread+0x115/0x140
[Tue Jul 16 13:20:38 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:20:38 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:20:38 2024] </TASK>
[Tue Jul 16 13:20:48 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 284149 jiffies s: 22037 root: 0x800/.
[Tue Jul 16 13:20:48 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:20:48 2024] Task dump for CPU 11:
[Tue Jul 16 13:20:48 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:20:48 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:20:48 2024] Call Trace:
[Tue Jul 16 13:20:48 2024] <TASK>
[Tue Jul 16 13:20:48 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:20:48 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:20:48 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:20:48 2024] ? mod_delayed_work_on+0x45/0xa0
[Tue Jul 16 13:20:48 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 13:20:48 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:20:48 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:20:48 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:20:48 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:20:48 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:20:48 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:20:48 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:20:48 2024] </TASK>
[Tue Jul 16 13:21:25 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:21:25 2024] rcu: 3-....: (20999 ticks this GP) idle=eaf/1/0x4000000000000000 softirq=31469530/31469530 fqs=5214
[Tue Jul 16 13:21:25 2024] (t=21013 jiffies g=194959289 q=22052)
[Tue Jul 16 13:21:25 2024] Task dump for CPU 3:
[Tue Jul 16 13:21:25 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:21:25 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:21:25 2024] Call Trace:
[Tue Jul 16 13:21:25 2024] <IRQ>
[Tue Jul 16 13:21:25 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:21:25 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:21:25 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:21:25 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:21:25 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:21:25 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:21:25 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:21:25 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:21:25 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:21:25 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:21:25 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:21:25 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:21:25 2024] </IRQ>
[Tue Jul 16 13:21:25 2024] <TASK>
[Tue Jul 16 13:21:25 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:21:25 2024] RIP: 0010:read_tsc+0x0/0x20
[Tue Jul 16 13:21:25 2024] Code: cc cc cc cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 <0f> 01 f9 66 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f
[Tue Jul 16 13:21:25 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 13:21:25 2024] RAX: ffffffff81030b50 RBX: 000000003ffb7344 RCX: 0000000028200003
[Tue Jul 16 13:21:25 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 13:21:25 2024] RBP: 0003f2c02a955fa3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:21:25 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
[Tue Jul 16 13:21:25 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:21:25 2024] ? recalibrate_cpu_khz+0x10/0x10
[Tue Jul 16 13:21:25 2024] ktime_get+0x38/0xa0
[Tue Jul 16 13:21:25 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:21:25 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 13:21:25 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:21:25 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:21:25 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:21:25 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:21:25 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:21:25 2024] kthread+0x115/0x140
[Tue Jul 16 13:21:25 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:21:25 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:21:25 2024] </TASK>
[Tue Jul 16 13:21:26 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 3-... } 21437 jiffies s: 22041 root: 0x8/.
[Tue Jul 16 13:21:26 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:21:26 2024] Task dump for CPU 3:
[Tue Jul 16 13:21:26 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:21:26 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:21:26 2024] Call Trace:
[Tue Jul 16 13:21:26 2024] <TASK>
[Tue Jul 16 13:21:26 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:21:26 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:21:26 2024] ? __mod_timer+0x327/0x3b0
[Tue Jul 16 13:21:26 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:21:26 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:21:26 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:21:26 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:21:26 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:21:26 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:21:26 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:21:26 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:21:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:21:26 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:21:26 2024] </TASK>
[Tue Jul 16 13:21:59 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:21:59 2024] rcu: 3-....: (20998 ticks this GP) idle=69d/1/0x4000000000000000 softirq=31469537/31469543 fqs=5251
[Tue Jul 16 13:21:59 2024] (t=21013 jiffies g=194959297 q=9028)
[Tue Jul 16 13:21:59 2024] Task dump for CPU 3:
[Tue Jul 16 13:21:59 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:21:59 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:21:59 2024] Call Trace:
[Tue Jul 16 13:21:59 2024] <IRQ>
[Tue Jul 16 13:21:59 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:21:59 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:21:59 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:21:59 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:21:59 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:21:59 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:21:59 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:21:59 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:21:59 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:21:59 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:21:59 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:21:59 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:21:59 2024] </IRQ>
[Tue Jul 16 13:21:59 2024] <TASK>
[Tue Jul 16 13:21:59 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:21:59 2024] RIP: 0010:xprt_release+0xb/0x190 [sunrpc]
[Tue Jul 16 13:21:59 2024] Code: c0 00 00 00 00 74 d9 48 89 df 5b e9 4f e7 ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 0f 1f 44 00 00 41 54 55 48 89 fd <53> 48 8b 9f c0 00 00 00 48 85 db 0f 84 12 01 00 00 4c 8b 23 e8 fc
[Tue Jul 16 13:21:59 2024] RSP: 0018:ffffc900087cfe10 EFLAGS: 00000286
[Tue Jul 16 13:21:59 2024] RAX: ffffffffa012c750 RBX: ffff88997131a500 RCX: 000000002c200003
[Tue Jul 16 13:21:59 2024] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffff88997131a500
[Tue Jul 16 13:21:59 2024] RBP: ffff88997131a500 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:21:59 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:21:59 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:21:59 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:21:59 2024] rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 13:21:59 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:21:59 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:21:59 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:21:59 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:21:59 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:21:59 2024] kthread+0x115/0x140
[Tue Jul 16 13:21:59 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:21:59 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:21:59 2024] </TASK>
[Tue Jul 16 13:22:54 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:22:54 2024] rcu: 7-....: (20998 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=5058
[Tue Jul 16 13:22:54 2024] (t=21017 jiffies g=194959301 q=10632)
[Tue Jul 16 13:22:54 2024] Task dump for CPU 7:
[Tue Jul 16 13:22:54 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:22:54 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:22:54 2024] Call Trace:
[Tue Jul 16 13:22:54 2024] <IRQ>
[Tue Jul 16 13:22:54 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:22:54 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:22:54 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:22:54 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:22:54 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:22:54 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:22:54 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:22:54 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:22:54 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:22:54 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:22:54 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:22:54 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:22:54 2024] </IRQ>
[Tue Jul 16 13:22:54 2024] <TASK>
[Tue Jul 16 13:22:54 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:22:54 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:22:54 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:22:54 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:22:54 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:22:54 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 13:22:54 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:22:54 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:22:54 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:22:54 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:22:54 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:22:54 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:22:54 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:22:54 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:22:54 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:22:54 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:22:54 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:22:54 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:22:54 2024] kthread+0x115/0x140
[Tue Jul 16 13:22:54 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:22:54 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:22:54 2024] </TASK>
[Tue Jul 16 13:23:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:23:57 2024] rcu: 7-....: (84003 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=19885
[Tue Jul 16 13:23:57 2024] (t=84259 jiffies g=194959301 q=17575)
[Tue Jul 16 13:23:57 2024] Task dump for CPU 7:
[Tue Jul 16 13:23:57 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:23:58 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:23:58 2024] Call Trace:
[Tue Jul 16 13:23:58 2024] <IRQ>
[Tue Jul 16 13:23:58 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:23:58 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:23:58 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:23:58 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:23:58 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:23:58 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:23:58 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:23:58 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:23:58 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:23:58 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:23:58 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:23:58 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:23:58 2024] </IRQ>
[Tue Jul 16 13:23:58 2024] <TASK>
[Tue Jul 16 13:23:58 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:23:58 2024] RIP: 0010:mod_delayed_work_on+0x14/0xa0
[Tue Jul 16 13:23:58 2024] Code: c3 cc cc cc cc e8 4c ff ff ff b8 01 00 00 00 eb e8 0f 1f 44 00 00 0f 1f 44 00 00 41 56 49 89 ce 41 55 49 89 f5 41 54 41 89 fc <55> 48 89 d5 53 48 83 ec 10 65 48 8b 04 25 28 00 00 00 48 89 44 24
[Tue Jul 16 13:23:58 2024] RSP: 0018:ffffc900087cfda0 EFLAGS: 00000287
[Tue Jul 16 13:23:58 2024] RAX: 00000001423b5315 RBX: ffff88997131a500 RCX: 00000000000007d0
[Tue Jul 16 13:23:58 2024] RDX: ffffffffa012c768 RSI: ffff888107352000 RDI: 0000000000000200
[Tue Jul 16 13:23:58 2024] RBP: 00000000000007d0 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:23:58 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:23:58 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:23:58 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:23:58 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:23:58 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:23:58 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:23:58 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:23:58 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:23:58 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:23:58 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:23:58 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:23:58 2024] kthread+0x115/0x140
[Tue Jul 16 13:23:58 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:23:58 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:23:58 2024] </TASK>
[Tue Jul 16 13:25:01 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:25:01 2024] rcu: 7-....: (147008 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=35640
[Tue Jul 16 13:25:01 2024] (t=147500 jiffies g=194959301 q=19524)
[Tue Jul 16 13:25:01 2024] Task dump for CPU 7:
[Tue Jul 16 13:25:01 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:25:01 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:25:01 2024] Call Trace:
[Tue Jul 16 13:25:01 2024] <IRQ>
[Tue Jul 16 13:25:01 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:25:01 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:25:01 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:25:01 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:25:01 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:25:01 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:25:01 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:25:01 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:25:01 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:25:01 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:25:01 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:25:01 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:25:01 2024] </IRQ>
[Tue Jul 16 13:25:01 2024] <TASK>
[Tue Jul 16 13:25:01 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:25:01 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:25:01 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:25:01 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:25:01 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:25:01 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 13:25:01 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:25:01 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:25:01 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:25:01 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:25:01 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:25:01 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:25:01 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:25:01 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:25:01 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:25:01 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:25:01 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:25:01 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:25:01 2024] kthread+0x115/0x140
[Tue Jul 16 13:25:01 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:25:01 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:25:01 2024] </TASK>
[Tue Jul 16 13:26:04 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:26:04 2024] rcu: 7-....: (210013 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=51357
[Tue Jul 16 13:26:04 2024] (t=210744 jiffies g=194959301 q=24405)
[Tue Jul 16 13:26:04 2024] Task dump for CPU 7:
[Tue Jul 16 13:26:04 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:26:04 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:26:04 2024] Call Trace:
[Tue Jul 16 13:26:04 2024] <IRQ>
[Tue Jul 16 13:26:04 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:26:04 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:26:04 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:26:04 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:26:04 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:26:04 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:26:04 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:26:04 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:26:04 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:26:04 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:26:04 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:26:04 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:26:04 2024] </IRQ>
[Tue Jul 16 13:26:04 2024] <TASK>
[Tue Jul 16 13:26:04 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:26:04 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
[Tue Jul 16 13:26:04 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
[Tue Jul 16 13:26:04 2024] RSP: 0018:ffffc900087cfe18 EFLAGS: 00000246
[Tue Jul 16 13:26:04 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000017
[Tue Jul 16 13:26:04 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
[Tue Jul 16 13:26:04 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:26:04 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:26:04 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:26:04 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:26:04 2024] rpc_wake_up_queued_task+0x1f/0x50 [sunrpc]
[Tue Jul 16 13:26:04 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:26:04 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:26:04 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:26:04 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:26:04 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:26:04 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:26:04 2024] kthread+0x115/0x140
[Tue Jul 16 13:26:04 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:26:04 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:26:04 2024] </TASK>
[Tue Jul 16 13:27:07 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:27:07 2024] rcu: 7-....: (273017 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=66386
[Tue Jul 16 13:27:07 2024] (t=273984 jiffies g=194959301 q=28427)
[Tue Jul 16 13:27:07 2024] Task dump for CPU 7:
[Tue Jul 16 13:27:07 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:27:07 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:27:07 2024] Call Trace:
[Tue Jul 16 13:27:07 2024] <IRQ>
[Tue Jul 16 13:27:07 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:27:07 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:27:07 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:27:07 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:27:07 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:27:07 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:27:07 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:27:07 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:27:07 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:27:07 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:27:07 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:27:07 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:27:07 2024] </IRQ>
[Tue Jul 16 13:27:07 2024] <TASK>
[Tue Jul 16 13:27:07 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:27:07 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:27:07 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:27:07 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:27:07 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:27:07 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 13:27:07 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:27:07 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:27:07 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:27:07 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:27:07 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:27:07 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:27:07 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:27:07 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:27:07 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:27:07 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:27:07 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:27:07 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:27:07 2024] kthread+0x115/0x140
[Tue Jul 16 13:27:07 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:27:07 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:27:07 2024] </TASK>
[Tue Jul 16 13:28:10 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:28:10 2024] rcu: 7-....: (336022 ticks this GP) idle=8bf/1/0x4000000000000000 softirq=29985027/29985036 fqs=81772
[Tue Jul 16 13:28:10 2024] (t=337227 jiffies g=194959301 q=30567)
[Tue Jul 16 13:28:10 2024] Task dump for CPU 7:
[Tue Jul 16 13:28:10 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:28:10 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:28:10 2024] Call Trace:
[Tue Jul 16 13:28:10 2024] <IRQ>
[Tue Jul 16 13:28:10 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:28:10 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:28:10 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:28:10 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:28:10 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:28:11 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:28:11 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:28:11 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:28:11 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:28:11 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:28:11 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:28:11 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:28:11 2024] </IRQ>
[Tue Jul 16 13:28:11 2024] <TASK>
[Tue Jul 16 13:28:11 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:28:11 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:28:11 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:28:11 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:28:11 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000005fe00007
[Tue Jul 16 13:28:11 2024] RDX: 00000001423f370c RSI: 0000000000000046 RDI: ffff88a07fcdb880
[Tue Jul 16 13:28:11 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:28:11 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:28:11 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:28:11 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:28:11 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:28:11 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:28:11 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:28:11 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:28:11 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:28:11 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:28:11 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:28:11 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:28:11 2024] kthread+0x115/0x140
[Tue Jul 16 13:28:11 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:28:11 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:28:11 2024] </TASK>
[Tue Jul 16 13:30:07 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:30:07 2024] rcu: 11-....: (20998 ticks this GP) idle=39f/1/0x4000000000000000 softirq=35298585/35298585 fqs=5219
[Tue Jul 16 13:30:07 2024] (t=21017 jiffies g=194959317 q=4767)
[Tue Jul 16 13:30:07 2024] Task dump for CPU 11:
[Tue Jul 16 13:30:07 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:30:07 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:30:07 2024] Call Trace:
[Tue Jul 16 13:30:07 2024] <IRQ>
[Tue Jul 16 13:30:07 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:30:07 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:30:07 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:30:07 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:30:07 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:30:07 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:30:07 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:30:07 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:30:07 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:30:07 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:30:07 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:30:07 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:30:07 2024] </IRQ>
[Tue Jul 16 13:30:07 2024] <TASK>
[Tue Jul 16 13:30:07 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:30:07 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:30:07 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:30:08 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:30:08 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000002020000b
[Tue Jul 16 13:30:08 2024] RDX: 000000014240ffda RSI: 0000000000000046 RDI: ffff88a07fd5b880
[Tue Jul 16 13:30:08 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:30:08 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:30:08 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:30:08 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:30:08 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:30:08 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:30:08 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:30:08 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:30:08 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:30:08 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:30:08 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:30:08 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:30:08 2024] kthread+0x115/0x140
[Tue Jul 16 13:30:08 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:30:08 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:30:08 2024] </TASK>
[Tue Jul 16 13:30:08 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 11-... } 21499 jiffies s: 22049 root: 0x800/.
[Tue Jul 16 13:30:08 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:30:08 2024] Task dump for CPU 11:
[Tue Jul 16 13:30:08 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:30:08 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:30:08 2024] Call Trace:
[Tue Jul 16 13:30:08 2024] <TASK>
[Tue Jul 16 13:30:08 2024] ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:30:08 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:30:08 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:30:08 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:30:08 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:30:08 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:30:08 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:30:08 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:30:08 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:30:08 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:30:08 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:30:08 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:30:08 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:30:08 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:30:08 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:30:08 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:30:08 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:30:08 2024] </TASK>
[Tue Jul 16 13:30:45 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:30:45 2024] rcu: 3-....: (21000 ticks this GP) idle=203/1/0x4000000000000000 softirq=31469565/31469565 fqs=5182
[Tue Jul 16 13:30:45 2024] (t=21017 jiffies g=194959321 q=5559)
[Tue Jul 16 13:30:45 2024] Task dump for CPU 3:
[Tue Jul 16 13:30:45 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:30:45 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:30:45 2024] Call Trace:
[Tue Jul 16 13:30:45 2024] <IRQ>
[Tue Jul 16 13:30:45 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:30:45 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:30:45 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:30:45 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:30:45 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:30:45 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:30:45 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:30:45 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:30:45 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:30:45 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:30:45 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:30:45 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:30:45 2024] </IRQ>
[Tue Jul 16 13:30:45 2024] <TASK>
[Tue Jul 16 13:30:45 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:30:45 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:30:45 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:30:45 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:30:45 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:30:45 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:30:45 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:30:45 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:30:45 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:30:45 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:30:45 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:30:45 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:30:45 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:30:45 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:30:45 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:30:45 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:30:45 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:30:45 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:30:45 2024] kthread+0x115/0x140
[Tue Jul 16 13:30:45 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:30:45 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:30:45 2024] </TASK>
[Tue Jul 16 13:31:48 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:31:49 2024] rcu: 3-....: (84005 ticks this GP) idle=203/1/0x4000000000000000 softirq=31469565/31469565 fqs=20563
[Tue Jul 16 13:31:49 2024] (t=84256 jiffies g=194959321 q=7181)
[Tue Jul 16 13:31:49 2024] Task dump for CPU 3:
[Tue Jul 16 13:31:49 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:31:49 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:31:49 2024] Call Trace:
[Tue Jul 16 13:31:49 2024] <IRQ>
[Tue Jul 16 13:31:49 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:31:49 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:31:49 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:31:49 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:31:49 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:31:49 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:31:49 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:31:49 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:31:49 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:31:49 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:31:49 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:31:49 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:31:49 2024] </IRQ>
[Tue Jul 16 13:31:49 2024] <TASK>
[Tue Jul 16 13:31:49 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:31:49 2024] RIP: 0010:__cancel_work+0x76/0xb0
[Tue Jul 16 13:31:49 2024] Code: 04 74 08 30 c9 48 8b 11 8b 52 0c 48 8b 0b 83 e1 01 74 3e 48 63 d2 48 c1 e2 05 48 89 13 f0 83 44 24 fc 00 f6 44 24 01 02 75 20 <85> c0 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 16 48
[Tue Jul 16 13:31:49 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000202
[Tue Jul 16 13:31:49 2024] RAX: 0000000000000001 RBX: ffffffffa012c768 RCX: 0000000000000001
[Tue Jul 16 13:31:49 2024] RDX: 0000000fffffffe0 RSI: 0000000000000046 RDI: ffff88a07fc5b880
[Tue Jul 16 13:31:49 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:31:49 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:31:49 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:31:49 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:31:49 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:31:49 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:31:49 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:31:49 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:31:49 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:31:49 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:31:49 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:31:49 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:31:49 2024] kthread+0x115/0x140
[Tue Jul 16 13:31:49 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:31:49 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:31:49 2024] </TASK>
[Tue Jul 16 13:32:16 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:32:16 2024] rcu: 5-....: (21000 ticks this GP) idle=88b/1/0x4000000000000000 softirq=30126362/30126362 fqs=5180
[Tue Jul 16 13:32:16 2024] (t=21013 jiffies g=194959325 q=6618)
[Tue Jul 16 13:32:16 2024] Task dump for CPU 5:
[Tue Jul 16 13:32:16 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:32:16 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:32:16 2024] Call Trace:
[Tue Jul 16 13:32:16 2024] <IRQ>
[Tue Jul 16 13:32:16 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:32:16 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:32:16 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:32:16 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:32:16 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:32:16 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:32:16 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:32:16 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:32:16 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:32:16 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:32:16 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:32:16 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:32:16 2024] </IRQ>
[Tue Jul 16 13:32:16 2024] <TASK>
[Tue Jul 16 13:32:16 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:32:17 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:32:17 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:32:17 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:32:17 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000028200005
[Tue Jul 16 13:32:17 2024] RDX: 000000014242f7d9 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 13:32:17 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:32:17 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:32:17 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:32:17 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:32:17 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:32:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:32:17 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:32:17 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:32:17 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:32:17 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:32:17 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:32:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:32:17 2024] kthread+0x115/0x140
[Tue Jul 16 13:32:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:32:17 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:32:17 2024] </TASK>
[Tue Jul 16 13:32:17 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 21502 jiffies s: 22053 root: 0x20/.
[Tue Jul 16 13:32:17 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:32:17 2024] Task dump for CPU 5:
[Tue Jul 16 13:32:17 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:32:17 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:32:17 2024] Call Trace:
[Tue Jul 16 13:32:17 2024] <TASK>
[Tue Jul 16 13:32:17 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:32:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:32:17 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:32:17 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:32:17 2024] ? del_timer+0x5c/0x80
[Tue Jul 16 13:32:17 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:32:17 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:32:17 2024] ? nfsd4_cb_done+0x28c/0x380 [nfsd]
[Tue Jul 16 13:32:17 2024] ? rpc_exit_task+0x70/0x100 [sunrpc]
[Tue Jul 16 13:32:17 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:32:17 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:32:17 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:32:17 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:32:17 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:32:17 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:32:17 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:32:17 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:32:17 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:32:17 2024] </TASK>
[Tue Jul 16 13:33:20 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:33:20 2024] rcu: 5-....: (84005 ticks this GP) idle=88b/1/0x4000000000000000 softirq=30126362/30126362 fqs=20771
[Tue Jul 16 13:33:20 2024] (t=84258 jiffies g=194959325 q=12842)
[Tue Jul 16 13:33:20 2024] Task dump for CPU 5:
[Tue Jul 16 13:33:20 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:33:20 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:33:20 2024] Call Trace:
[Tue Jul 16 13:33:20 2024] <IRQ>
[Tue Jul 16 13:33:20 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:33:20 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:33:20 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:33:20 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:33:20 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:33:20 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:33:20 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:33:20 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:33:20 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:33:20 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:33:20 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:33:20 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:33:20 2024] </IRQ>
[Tue Jul 16 13:33:20 2024] <TASK>
[Tue Jul 16 13:33:20 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:33:20 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:33:20 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:33:20 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:33:20 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000004fe00005
[Tue Jul 16 13:33:20 2024] RDX: 000000014243eee4 RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 13:33:20 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:33:20 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:33:20 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:33:20 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:33:20 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:33:20 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:33:20 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:33:20 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:33:20 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:33:20 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:33:20 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:33:20 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:33:20 2024] kthread+0x115/0x140
[Tue Jul 16 13:33:20 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:33:20 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:33:20 2024] </TASK>
[Tue Jul 16 13:33:22 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 86526 jiffies s: 22053 root: 0x20/.
[Tue Jul 16 13:33:22 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:33:22 2024] Task dump for CPU 5:
[Tue Jul 16 13:33:22 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:33:22 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:33:22 2024] Call Trace:
[Tue Jul 16 13:33:22 2024] <TASK>
[Tue Jul 16 13:33:22 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:33:22 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:33:22 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 13:33:22 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:33:22 2024] ? mod_delayed_work_on+0x61/0xa0
[Tue Jul 16 13:33:22 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:33:22 2024] ? ktime_get+0x38/0xa0
[Tue Jul 16 13:33:22 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:33:22 2024] ? rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 13:33:22 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:33:22 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:33:22 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:33:22 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:33:22 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:33:22 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:33:22 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:33:22 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:33:22 2024] </TASK>
[Tue Jul 16 13:34:23 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:34:23 2024] rcu: 5-....: (147010 ticks this GP) idle=88b/1/0x4000000000000000 softirq=30126362/30126362 fqs=35902
[Tue Jul 16 13:34:23 2024] (t=147501 jiffies g=194959325 q=15728)
[Tue Jul 16 13:34:23 2024] Task dump for CPU 5:
[Tue Jul 16 13:34:23 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:34:23 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:34:23 2024] Call Trace:
[Tue Jul 16 13:34:23 2024] <IRQ>
[Tue Jul 16 13:34:23 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:34:23 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:34:23 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:34:23 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:34:23 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:34:23 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:34:23 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:34:23 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:34:23 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:34:23 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:34:23 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:34:23 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:34:23 2024] </IRQ>
[Tue Jul 16 13:34:23 2024] <TASK>
[Tue Jul 16 13:34:23 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:34:23 2024] RIP: 0010:mod_delayed_work_on+0x68/0xa0
[Tue Jul 16 13:34:23 2024] Code: 89 ef e8 4b f7 ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 18 4c 89 f1 48 89 ea 4c 89 ee 44 89 e7 e8 df fe ff ff f6 44 24 01 02 75 26 <85> db 0f 95 c0 48 8b 54 24 08 65 48 2b 14 25 28 00 00 00 75 14 48
[Tue Jul 16 13:34:23 2024] RSP: 0018:ffffc900087cfd80 EFLAGS: 00000202
[Tue Jul 16 13:34:23 2024] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000043e00005
[Tue Jul 16 13:34:23 2024] RDX: 000000014244e5ef RSI: 0000000000000046 RDI: ffff88a07fc9b880
[Tue Jul 16 13:34:23 2024] RBP: ffffffffa012c768 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:34:23 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000200
[Tue Jul 16 13:34:23 2024] R13: ffff888107352000 R14: 00000000000007d0 R15: ffffffffa012c750
[Tue Jul 16 13:34:23 2024] __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:34:23 2024] rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:34:23 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:34:23 2024] rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:34:23 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:34:23 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:34:23 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:34:23 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:34:23 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:34:23 2024] kthread+0x115/0x140
[Tue Jul 16 13:34:23 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:34:23 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:34:23 2024] </TASK>
[Tue Jul 16 13:34:27 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 152063 jiffies s: 22053 root: 0x20/.
[Tue Jul 16 13:34:27 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:34:27 2024] Task dump for CPU 5:
[Tue Jul 16 13:34:27 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:34:27 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:34:27 2024] Call Trace:
[Tue Jul 16 13:34:27 2024] <TASK>
[Tue Jul 16 13:34:27 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:34:27 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:34:28 2024] ? __mod_timer+0x125/0x3b0
[Tue Jul 16 13:34:28 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 13:34:28 2024] ? __rpc_sleep_on_priority_timeout+0x133/0x180 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:34:28 2024] ? __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? rpc_delay+0x56/0x90 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:34:28 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:34:28 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:34:28 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:34:28 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:34:28 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:34:28 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:34:28 2024] </TASK>
[Tue Jul 16 13:35:26 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:35:26 2024] rcu: 5-....: (210011 ticks this GP) idle=88b/1/0x4000000000000000 softirq=30126362/30126362 fqs=50487
[Tue Jul 16 13:35:26 2024] (t=210741 jiffies g=194959325 q=18619)
[Tue Jul 16 13:35:26 2024] Task dump for CPU 5:
[Tue Jul 16 13:35:26 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:35:26 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:35:26 2024] Call Trace:
[Tue Jul 16 13:35:26 2024] <IRQ>
[Tue Jul 16 13:35:26 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:35:26 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:35:26 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:35:26 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:35:26 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:35:26 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:35:26 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:35:26 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:35:26 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:35:26 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:35:26 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:35:26 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:35:26 2024] </IRQ>
[Tue Jul 16 13:35:26 2024] <TASK>
[Tue Jul 16 13:35:26 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:35:26 2024] RIP: 0010:__cancel_work+0x12/0xb0
[Tue Jul 16 13:35:26 2024] Code: 89 df 48 89 04 24 e8 bd ec ff ff 48 8b 04 24 eb a4 0f 0b eb d7 0f 1f 00 0f 1f 44 00 00 55 40 0f b6 ee 53 48 89 fb 48 83 ec 10 <65> 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 48 c7 04 24 00 00
[Tue Jul 16 13:35:26 2024] RSP: 0018:ffffc900087cfdd8 EFLAGS: 00000286
[Tue Jul 16 13:35:26 2024] RAX: ffffffffa012c750 RBX: ffffffffa012c768 RCX: 0000000000000017
[Tue Jul 16 13:35:26 2024] RDX: ffffffffa012c750 RSI: 0000000000000001 RDI: ffffffffa012c768
[Tue Jul 16 13:35:26 2024] RBP: 0000000000000001 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:35:26 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
[Tue Jul 16 13:35:26 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:35:26 2024] __rpc_do_wake_up_task_on_wq+0x126/0x1a0 [sunrpc]
[Tue Jul 16 13:35:26 2024] rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:35:26 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:35:26 2024] __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:35:26 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:35:26 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:35:26 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:35:26 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:35:26 2024] kthread+0x115/0x140
[Tue Jul 16 13:35:26 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:35:26 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:35:26 2024] </TASK>
[Tue Jul 16 13:35:33 2024] rcu: INFO: rcu_sched detected expedited stalls on CPUs/tasks: { 5-... } 217599 jiffies s: 22053 root: 0x20/.
[Tue Jul 16 13:35:33 2024] rcu: blocking rcu_node structures (internal RCU debug):
[Tue Jul 16 13:35:33 2024] Task dump for CPU 5:
[Tue Jul 16 13:35:33 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:35:33 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:35:33 2024] Call Trace:
[Tue Jul 16 13:35:33 2024] <TASK>
[Tue Jul 16 13:35:33 2024] ? sysvec_apic_timer_interrupt+0xa/0x90
[Tue Jul 16 13:35:33 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:35:33 2024] ? __mod_timer+0x287/0x3b0
[Tue Jul 16 13:35:33 2024] ? lock_timer_base+0x61/0x80
[Tue Jul 16 13:35:33 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:35:33 2024] ? del_timer+0x3b/0x80
[Tue Jul 16 13:35:33 2024] ? try_to_grab_pending+0xc8/0x150
[Tue Jul 16 13:35:33 2024] ? __cancel_work+0x37/0xb0
[Tue Jul 16 13:35:33 2024] ? rpc_wake_up_queued_task+0x3f/0x50 [sunrpc]
[Tue Jul 16 13:35:33 2024] ? rpc_exit_task+0x5e/0x100 [sunrpc]
[Tue Jul 16 13:35:33 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:35:33 2024] ? __rpc_execute+0x1ba/0x410 [sunrpc]
[Tue Jul 16 13:35:33 2024] ? rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:35:33 2024] ? process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:35:33 2024] ? worker_thread+0x4a/0x3c0
[Tue Jul 16 13:35:33 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:35:33 2024] ? kthread+0x115/0x140
[Tue Jul 16 13:35:33 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:35:33 2024] ? ret_from_fork+0x1f/0x30
[Tue Jul 16 13:35:33 2024] </TASK>
[Tue Jul 16 13:36:29 2024] rcu: INFO: rcu_sched self-detected stall on CPU
[Tue Jul 16 13:36:29 2024] rcu: 5-....: (273012 ticks this GP) idle=88b/1/0x4000000000000000 softirq=30126362/30126362 fqs=65675
[Tue Jul 16 13:36:29 2024] (t=273978 jiffies g=194959325 q=20413)
[Tue Jul 16 13:36:29 2024] Task dump for CPU 5:
[Tue Jul 16 13:36:29 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
[Tue Jul 16 13:36:29 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
[Tue Jul 16 13:36:29 2024] Call Trace:
[Tue Jul 16 13:36:29 2024] <IRQ>
[Tue Jul 16 13:36:29 2024] sched_show_task.cold+0xc2/0xda
[Tue Jul 16 13:36:29 2024] rcu_dump_cpu_stacks+0xa1/0xd3
[Tue Jul 16 13:36:29 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
[Tue Jul 16 13:36:29 2024] ? trigger_load_balance+0x6d/0x300
[Tue Jul 16 13:36:29 2024] ? scheduler_tick+0xda/0x260
[Tue Jul 16 13:36:29 2024] update_process_times+0xa1/0xe0
[Tue Jul 16 13:36:29 2024] tick_sched_timer+0x8e/0xa0
[Tue Jul 16 13:36:29 2024] ? tick_sched_do_timer+0x90/0x90
[Tue Jul 16 13:36:29 2024] __hrtimer_run_queues+0x139/0x2a0
[Tue Jul 16 13:36:29 2024] hrtimer_interrupt+0xf4/0x210
[Tue Jul 16 13:36:29 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
[Tue Jul 16 13:36:29 2024] sysvec_apic_timer_interrupt+0x69/0x90
[Tue Jul 16 13:36:29 2024] </IRQ>
[Tue Jul 16 13:36:29 2024] <TASK>
[Tue Jul 16 13:36:29 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
[Tue Jul 16 13:36:29 2024] RIP: 0010:read_tsc+0x3/0x20
[Tue Jul 16 13:36:29 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
[Tue Jul 16 13:36:29 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
[Tue Jul 16 13:36:29 2024] RAX: 0000000032157ba0 RBX: 000000004016f8c6 RCX: 0000000000001005
[Tue Jul 16 13:36:29 2024] RDX: 0000000000454e8e RSI: 0000000000000046 RDI: ffffffff82435600
[Tue Jul 16 13:36:30 2024] RBP: 0003f392a52eafa3 R08: ffffffffa012c770 R09: ffffffffa012c788
[Tue Jul 16 13:36:30 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
[Tue Jul 16 13:36:30 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
[Tue Jul 16 13:36:30 2024] ktime_get+0x38/0xa0
[Tue Jul 16 13:36:30 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
[Tue Jul 16 13:36:30 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
[Tue Jul 16 13:36:30 2024] __rpc_execute+0x6e/0x410 [sunrpc]
[Tue Jul 16 13:36:30 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
[Tue Jul 16 13:36:30 2024] process_one_work+0x1d7/0x3a0
[Tue Jul 16 13:36:30 2024] worker_thread+0x4a/0x3c0
[Tue Jul 16 13:36:30 2024] ? process_one_work+0x3a0/0x3a0
[Tue Jul 16 13:36:30 2024] kthread+0x115/0x140
[Tue Jul 16 13:36:30 2024] ? set_kthread_struct+0x50/0x50
[Tue Jul 16 13:36:30 2024] ret_from_fork+0x1f/0x30
[Tue Jul 16 13:36:30 2024] </TASK>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-07-17 5:33 `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod` Paul Menzel
@ 2024-07-27 21:15 ` Salvatore Bonaccorso
2024-07-27 21:19 ` Chuck Lever III
0 siblings, 1 reply; 16+ messages in thread
From: Salvatore Bonaccorso @ 2024-07-27 21:15 UTC (permalink / raw)
To: Paul Menzel; +Cc: Chuck Lever, Jeff Layton, linux-nfs, it+linux-nfs
Hi,
On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> Dear Linux folks,
>
>
> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> 04/22/2021, a mount from another server hung. Linux logs:
>
> ```
> $ dmesg -T
> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476
> (root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils)
> 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
> crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd
> audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> […]
> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2
> 04/22/2021
> […]
> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000aeeb49cf xid b6f12d96
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000056d1aff1 xid 6ad5584a
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000008075849 xid 406ed865
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000028481e8f xid 7f81b676
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000155c8644 xid 26099b1f
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000c384ff38 xid 7ed4dbf5
> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000001bba6d7e xid a930d2bf
> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000155c8644 xid 5b099b1f
> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000c384ff38 xid b3d4dbf5
> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000001bba6d7e xid de30d2bf
> [Tue Jul 16 11:20:21 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000001bba6d7e xid 4431d2bf
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000007ce5d717 xid 2c364663
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000001bba6d7e xid df31d2bf
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000000be8f11f xid acdab0f5
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000d6d182c4 xid 3d172cb9
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000976cd55a xid a6cb0a18
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000e11f40dd xid 35f006fd
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000042906e77 xid d9415db0
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000bc03be29 xid eed92785
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000056d1aff1 xid a1d6584a
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000008075849 xid 776fd865
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000aeeb49cf xid edf22d96
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 000000009327f72c xid 12b9ab32
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000b55d160f xid 0e3dd152
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000976cd55a xid a7cb0a18
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000042906e77 xid da415db0
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000bc03be29 xid efd92785
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000008075849 xid 786fd865
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000aeeb49cf xid eef22d96
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000ee580afa xid 9f91a3d2
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000060d5bb55 xid 3aea57c8
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000d4d84570 xid 73a5017a
> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000155c8644 xid 5d0a9b1f
> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
> 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP)
> idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task
> stack: 0 pid:30413 ppid: 2 flags:0x00004008
> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> [Tue Jul 16 11:36:40 2024] Call Trace:
> [Tue Jul 16 11:36:40 2024] <IRQ>
> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> [Tue Jul 16 11:36:40 2024] </IRQ>
> [Tue Jul 16 11:36:40 2024] <TASK>
> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3
> cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00
> 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX:
> 000000000000100d
> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI:
> ffffffff82435600
> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09:
> ffffffffa012c788
> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
> 0000000000000000
> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
> ffff88909311c005
> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> [Tue Jul 16 11:36:40 2024] </TASK>
> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP)
> idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task
> stack: 0 pid:30413 ppid: 2 flags:0x00004008
> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> [Tue Jul 16 11:37:19 2024] Call Trace:
> [Tue Jul 16 11:37:19 2024] <IRQ>
> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> [Tue Jul 16 11:37:19 2024] </IRQ>
> [Tue Jul 16 11:37:19 2024] <TASK>
> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00
> 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0
> 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX:
> 0000000000000001
> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI:
> ffffffffa012c700
> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09:
> ffffffffa012c788
> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
> ffff88997131a530
> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
> ffff88909311c005
> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> [Tue Jul 16 11:37:19 2024] </TASK>
> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> […]
> ```
FWIW, on one NFS server occurence we are seeing something very close
to the above but in the 5.10.y case for the Debian kernel after
updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
big NFS related stack backported.
One backtrace we were able to catch was
[...]
Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
[...]
Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
Jul 27 15:25:15 nfsserver kernel: Call Trace:
Jul 27 15:25:15 nfsserver kernel: <IRQ>
Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
Jul 27 15:25:15 nfsserver kernel: </IRQ>
Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
[...]
Is there anything which could help debug this issue?
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-07-27 21:15 ` Salvatore Bonaccorso
@ 2024-07-27 21:19 ` Chuck Lever III
2024-07-30 12:19 ` Salvatore Bonaccorso
0 siblings, 1 reply; 16+ messages in thread
From: Chuck Lever III @ 2024-07-27 21:19 UTC (permalink / raw)
To: Salvatore Bonaccorso
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>
> Hi,
>
> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>> Dear Linux folks,
>>
>>
>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>> 04/22/2021, a mount from another server hung. Linux logs:
>>
>> ```
>> $ dmesg -T
>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476
>> (root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils)
>> 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
>> crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd
>> audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>> […]
>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2
>> 04/22/2021
>> […]
>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000aeeb49cf xid b6f12d96
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000056d1aff1 xid 6ad5584a
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000008075849 xid 406ed865
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000028481e8f xid 7f81b676
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000155c8644 xid 26099b1f
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000c384ff38 xid 7ed4dbf5
>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000001bba6d7e xid a930d2bf
>> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000155c8644 xid 5b099b1f
>> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000c384ff38 xid b3d4dbf5
>> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000001bba6d7e xid de30d2bf
>> [Tue Jul 16 11:20:21 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000001bba6d7e xid 4431d2bf
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000007ce5d717 xid 2c364663
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000001bba6d7e xid df31d2bf
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000000be8f11f xid acdab0f5
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000d6d182c4 xid 3d172cb9
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000976cd55a xid a6cb0a18
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000e11f40dd xid 35f006fd
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000042906e77 xid d9415db0
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000bc03be29 xid eed92785
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000056d1aff1 xid a1d6584a
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000008075849 xid 776fd865
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000aeeb49cf xid edf22d96
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 000000009327f72c xid 12b9ab32
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000b55d160f xid 0e3dd152
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000976cd55a xid a7cb0a18
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000042906e77 xid da415db0
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000bc03be29 xid efd92785
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000008075849 xid 786fd865
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000aeeb49cf xid eef22d96
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000ee580afa xid 9f91a3d2
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000060d5bb55 xid 3aea57c8
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000d4d84570 xid 73a5017a
>> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000155c8644 xid 5d0a9b1f
>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
>> 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP)
>> idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task
>> stack: 0 pid:30413 ppid: 2 flags:0x00004008
>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>> [Tue Jul 16 11:36:40 2024] Call Trace:
>> [Tue Jul 16 11:36:40 2024] <IRQ>
>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>> [Tue Jul 16 11:36:40 2024] </IRQ>
>> [Tue Jul 16 11:36:40 2024] <TASK>
>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3
>> cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00
>> 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX:
>> 000000000000100d
>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI:
>> ffffffff82435600
>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09:
>> ffffffffa012c788
>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
>> 0000000000000000
>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
>> ffff88909311c005
>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>> [Tue Jul 16 11:36:40 2024] </TASK>
>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP)
>> idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task
>> stack: 0 pid:30413 ppid: 2 flags:0x00004008
>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>> [Tue Jul 16 11:37:19 2024] Call Trace:
>> [Tue Jul 16 11:37:19 2024] <IRQ>
>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>> [Tue Jul 16 11:37:19 2024] </IRQ>
>> [Tue Jul 16 11:37:19 2024] <TASK>
>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00
>> 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0
>> 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX:
>> 0000000000000001
>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI:
>> ffffffffa012c700
>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09:
>> ffffffffa012c788
>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
>> ffff88997131a530
>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
>> ffff88909311c005
>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>> [Tue Jul 16 11:37:19 2024] </TASK>
>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>> […]
>> ```
>
> FWIW, on one NFS server occurence we are seeing something very close
> to the above but in the 5.10.y case for the Debian kernel after
> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> big NFS related stack backported.
>
> One backtrace we were able to catch was
>
> [...]
> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> [...]
> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: Call Trace:
> Jul 27 15:25:15 nfsserver kernel: <IRQ>
> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> Jul 27 15:25:15 nfsserver kernel: </IRQ>
> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> [...]
>
> Is there anything which could help debug this issue?
The backtrace suggests an issue in the RPC client code; the
server's NFSv4.1 backchannel would use that to send callbacks.
Since 5.10.218 and 5.10.221 are only about a thousand commits
apart, a bisect should be quick and narrow down the issue to
one or two commits.
--
Chuck Lever
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-07-27 21:19 ` Chuck Lever III
@ 2024-07-30 12:19 ` Salvatore Bonaccorso
2024-07-30 12:52 ` Paul Menzel
0 siblings, 1 reply; 16+ messages in thread
From: Salvatore Bonaccorso @ 2024-07-30 12:19 UTC (permalink / raw)
To: Chuck Lever III
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
Hi Chuck,
On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>
>
> > On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >
> > Hi,
> >
> > On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> >> Dear Linux folks,
> >>
> >>
> >> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> >> 04/22/2021, a mount from another server hung. Linux logs:
> >>
> >> ```
> >> $ dmesg -T
> >> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476
> >> (root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils)
> >> 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> >> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
> >> crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd
> >> audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> >> […]
> >> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2
> >> 04/22/2021
> >> […]
> >> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000aeeb49cf xid b6f12d96
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000056d1aff1 xid 6ad5584a
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000008075849 xid 406ed865
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000028481e8f xid 7f81b676
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000155c8644 xid 26099b1f
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000c384ff38 xid 7ed4dbf5
> >> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000001bba6d7e xid a930d2bf
> >> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000155c8644 xid 5b099b1f
> >> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000c384ff38 xid b3d4dbf5
> >> [Tue Jul 16 11:11:04 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000001bba6d7e xid de30d2bf
> >> [Tue Jul 16 11:20:21 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000001bba6d7e xid 4431d2bf
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000007ce5d717 xid 2c364663
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000001bba6d7e xid df31d2bf
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000000be8f11f xid acdab0f5
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000d6d182c4 xid 3d172cb9
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000976cd55a xid a6cb0a18
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000e11f40dd xid 35f006fd
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000042906e77 xid d9415db0
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000bc03be29 xid eed92785
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000056d1aff1 xid a1d6584a
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000008075849 xid 776fd865
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000aeeb49cf xid edf22d96
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 000000009327f72c xid 12b9ab32
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000b55d160f xid 0e3dd152
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000976cd55a xid a7cb0a18
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000042906e77 xid da415db0
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000bc03be29 xid efd92785
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000008075849 xid 786fd865
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000aeeb49cf xid eef22d96
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000ee580afa xid 9f91a3d2
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000060d5bb55 xid 3aea57c8
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000d4d84570 xid 73a5017a
> >> [Tue Jul 16 11:35:58 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000155c8644 xid 5d0a9b1f
> >> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> >> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir
> >> 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> >> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP)
> >> idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> >> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> >> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> >> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task
> >> stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >> [Tue Jul 16 11:36:40 2024] Call Trace:
> >> [Tue Jul 16 11:36:40 2024] <IRQ>
> >> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> >> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> >> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> >> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> >> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> >> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> >> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> >> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> >> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >> [Tue Jul 16 11:36:40 2024] </IRQ>
> >> [Tue Jul 16 11:36:40 2024] <TASK>
> >> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> >> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3
> >> cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00
> >> 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> >> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> >> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX:
> >> 000000000000100d
> >> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI:
> >> ffffffff82435600
> >> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09:
> >> ffffffffa012c788
> >> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
> >> 0000000000000000
> >> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
> >> ffff88909311c005
> >> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> >> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> >> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> >> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> >> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> >> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> >> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> >> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> >> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> >> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> >> [Tue Jul 16 11:36:40 2024] </TASK>
> >> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP)
> >> idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> >> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> >> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> >> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task
> >> stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >> [Tue Jul 16 11:37:19 2024] Call Trace:
> >> [Tue Jul 16 11:37:19 2024] <IRQ>
> >> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> >> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> >> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> >> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> >> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> >> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> >> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> >> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> >> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >> [Tue Jul 16 11:37:19 2024] </IRQ>
> >> [Tue Jul 16 11:37:19 2024] <TASK>
> >> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> >> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00
> >> 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0
> >> 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> >> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> >> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX:
> >> 0000000000000001
> >> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI:
> >> ffffffffa012c700
> >> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09:
> >> ffffffffa012c788
> >> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12:
> >> ffff88997131a530
> >> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15:
> >> ffff88909311c005
> >> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> >> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> >> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> >> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> >> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> >> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> >> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> >> [Tue Jul 16 11:37:19 2024] </TASK>
> >> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >> […]
> >> ```
> >
> > FWIW, on one NFS server occurence we are seeing something very close
> > to the above but in the 5.10.y case for the Debian kernel after
> > updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> > big NFS related stack backported.
> >
> > One backtrace we were able to catch was
> >
> > [...]
> > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> > [...]
> > Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> > Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> > Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> > Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> > Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> > Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: Call Trace:
> > Jul 27 15:25:15 nfsserver kernel: <IRQ>
> > Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> > Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> > Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> > Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> > Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> > Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> > Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> > Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> > Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> > Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> > Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> > Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> > Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> > Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> > Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> > Jul 27 15:25:15 nfsserver kernel: </IRQ>
> > Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> > Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> > Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> > Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> > Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> > Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> > Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> > Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> > Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> > Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> > Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> > Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> > Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> > Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> > Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> > [...]
> >
> > Is there anything which could help debug this issue?
>
> The backtrace suggests an issue in the RPC client code; the
> server's NFSv4.1 backchannel would use that to send callbacks.
>
> Since 5.10.218 and 5.10.221 are only about a thousand commits
> apart, a bisect should be quick and narrow down the issue to
> one or two commits.
Yes indeed. Unfortunately was yet unable to reproduce the issue in
more syntentic way on test environment, and the affected server in
particular is a production system.
Paul, is your case in some way reproducible in a testing environment
so that a bisection might be give enough hints on the problem?
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-07-30 12:19 ` Salvatore Bonaccorso
@ 2024-07-30 12:52 ` Paul Menzel
2024-08-17 8:39 ` Salvatore Bonaccorso
0 siblings, 1 reply; 16+ messages in thread
From: Paul Menzel @ 2024-07-30 12:52 UTC (permalink / raw)
To: Salvatore Bonaccorso, Chuck Lever III
Cc: Jeff Layton, linux-nfs, it+linux-nfs
Dear Salvatore, dear Chuck,
Thank you for your messages.
Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>
>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>
>>>> ```
>>>> $ dmesg -T
>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>>>> […]
>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
>>>> […]
>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
[…]
>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>> […]
>>>> ```
>>>
>>> FWIW, on one NFS server occurence we are seeing something very close
>>> to the above but in the 5.10.y case for the Debian kernel after
>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
>>> big NFS related stack backported.
>>>
>>> One backtrace we were able to catch was
>>>
>>> [...]
>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
>>> [...]
>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>> [...]
>>>
>>> Is there anything which could help debug this issue?
>>
>> The backtrace suggests an issue in the RPC client code; the
>> server's NFSv4.1 backchannel would use that to send callbacks.
>>
>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>> apart, a bisect should be quick and narrow down the issue to
>> one or two commits.
>
> Yes indeed. Unfortunately was yet unable to reproduce the issue in
> more syntentic way on test environment, and the affected server in
> particular is a production system.
>
> Paul, is your case in some way reproducible in a testing environment
> so that a bisection might be give enough hints on the problem?
We hit this issue once more on the same server with Linux 5.15.160, and
had to hard reboot it.
Unfortunately we did not have time yet to set up a test system to find a
reproducer. In our cases a lot of compute servers seem to have accessed
the NFS server. A lot of the many processes were `zstd` on a first glance.
Kind regards,
Paul
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-07-30 12:52 ` Paul Menzel
@ 2024-08-17 8:39 ` Salvatore Bonaccorso
2024-08-17 14:52 ` Chuck Lever III
0 siblings, 1 reply; 16+ messages in thread
From: Salvatore Bonaccorso @ 2024-08-17 8:39 UTC (permalink / raw)
To: Paul Menzel; +Cc: Chuck Lever III, Jeff Layton, linux-nfs, it+linux-nfs
Hi,
On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
> Dear Salvatore, dear Chuck,
>
>
> Thank you for your messages.
>
>
> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>
> > On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
> > >
> > > > On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>
> > > > On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>
> > > > > Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> > > > > 04/22/2021, a mount from another server hung. Linux logs:
> > > > >
> > > > > ```
> > > > > $ dmesg -T
> > > > > [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> > > > > [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> > > > > […]
> > > > > [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
> > > > > […]
> > > > > [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> > > > > [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> > > > > [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>
> […]
>
> > > > > [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> > > > > [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> > > > > [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> > > > > [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> > > > > [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> > > > > [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > > > > [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > > [Tue Jul 16 11:36:40 2024] Call Trace:
> > > > > [Tue Jul 16 11:36:40 2024] <IRQ>
> > > > > [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> > > > > [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > > > > [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > > > > [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> > > > > [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> > > > > [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> > > > > [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> > > > > [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> > > > > [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> > > > > [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> > > > > [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > > > > [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > > > > [Tue Jul 16 11:36:40 2024] </IRQ>
> > > > > [Tue Jul 16 11:36:40 2024] <TASK>
> > > > > [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > > > > [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> > > > > [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> > > > > [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> > > > > [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
> > > > > [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
> > > > > [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
> > > > > [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
> > > > > [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > > > > [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> > > > > [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> > > > > [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> > > > > [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> > > > > [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > > [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> > > > > [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> > > > > [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> > > > > [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> > > > > [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> > > > > [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> > > > > [Tue Jul 16 11:36:40 2024] </TASK>
> > > > > [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> > > > > [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> > > > > [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> > > > > [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > > > > [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > > [Tue Jul 16 11:37:19 2024] Call Trace:
> > > > > [Tue Jul 16 11:37:19 2024] <IRQ>
> > > > > [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> > > > > [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > > > > [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > > > > [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> > > > > [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> > > > > [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> > > > > [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> > > > > [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> > > > > [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> > > > > [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> > > > > [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > > > > [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > > > > [Tue Jul 16 11:37:19 2024] </IRQ>
> > > > > [Tue Jul 16 11:37:19 2024] <TASK>
> > > > > [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > > > > [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> > > > > [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> > > > > [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> > > > > [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
> > > > > [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
> > > > > [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
> > > > > [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
> > > > > [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > > > > [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> > > > > [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > > [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> > > > > [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> > > > > [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> > > > > [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> > > > > [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> > > > > [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> > > > > [Tue Jul 16 11:37:19 2024] </TASK>
> > > > > [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > […]
> > > > > ```
> > > >
> > > > FWIW, on one NFS server occurence we are seeing something very close
> > > > to the above but in the 5.10.y case for the Debian kernel after
> > > > updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> > > > big NFS related stack backported.
> > > >
> > > > One backtrace we were able to catch was
> > > >
> > > > [...]
> > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> > > > [...]
> > > > Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> > > > Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> > > > Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> > > > Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> > > > Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> > > > Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > > > Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: Call Trace:
> > > > Jul 27 15:25:15 nfsserver kernel: <IRQ>
> > > > Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> > > > Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> > > > Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> > > > Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> > > > Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> > > > Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> > > > Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> > > > Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> > > > Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> > > > Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> > > > Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> > > > Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> > > > Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> > > > Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> > > > Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> > > > Jul 27 15:25:15 nfsserver kernel: </IRQ>
> > > > Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> > > > Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> > > > Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> > > > Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> > > > Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> > > > Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> > > > Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> > > > Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> > > > Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> > > > Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> > > > Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> > > > Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> > > > Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> > > > Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> > > > Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > > > Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> > > > [...]
> > > >
> > > > Is there anything which could help debug this issue?
> > >
> > > The backtrace suggests an issue in the RPC client code; the
> > > server's NFSv4.1 backchannel would use that to send callbacks.
> > >
> > > Since 5.10.218 and 5.10.221 are only about a thousand commits
> > > apart, a bisect should be quick and narrow down the issue to
> > > one or two commits.
> >
> > Yes indeed. Unfortunately was yet unable to reproduce the issue in
> > more syntentic way on test environment, and the affected server in
> > particular is a production system.
> >
> > Paul, is your case in some way reproducible in a testing environment
> > so that a bisection might be give enough hints on the problem?
> We hit this issue once more on the same server with Linux 5.15.160, and had
> to hard reboot it.
>
> Unfortunately we did not have time yet to set up a test system to find a
> reproducer. In our cases a lot of compute servers seem to have accessed the
> NFS server. A lot of the many processes were `zstd` on a first glance.
So we neither, due to the nature of the server (production system) and
unability to reproduce the issue under some more controlled way and on
test environment.
In our case users seems to cause workloads involving use of wandb.
What we tried is to boot the recent kernel from 5.10.y series avaiable
(5.10.223-1). Then the issue showed up still. Since we disabled
fs.leases-enable the situation seems to be more stable). While this
is/might not be the solution, does that gives some additional hits?
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-08-17 8:39 ` Salvatore Bonaccorso
@ 2024-08-17 14:52 ` Chuck Lever III
2024-10-29 21:07 ` Salvatore Bonaccorso
2025-01-31 16:17 ` Salvatore Bonaccorso
0 siblings, 2 replies; 16+ messages in thread
From: Chuck Lever III @ 2024-08-17 14:52 UTC (permalink / raw)
To: Salvatore Bonaccorso
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>
> Hi,
>
> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>> Dear Salvatore, dear Chuck,
>>
>>
>> Thank you for your messages.
>>
>>
>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>
>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>
>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>
>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>
>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>
>>>>>> ```
>>>>>> $ dmesg -T
>>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>>>>>> […]
>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>> […]
>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>>
>> […]
>>
>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>> […]
>>>>>> ```
>>>>>
>>>>> FWIW, on one NFS server occurence we are seeing something very close
>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
>>>>> big NFS related stack backported.
>>>>>
>>>>> One backtrace we were able to catch was
>>>>>
>>>>> [...]
>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
>>>>> [...]
>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
>>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>> [...]
>>>>>
>>>>> Is there anything which could help debug this issue?
>>>>
>>>> The backtrace suggests an issue in the RPC client code; the
>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>
>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>> apart, a bisect should be quick and narrow down the issue to
>>>> one or two commits.
>>>
>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>> more syntentic way on test environment, and the affected server in
>>> particular is a production system.
>>>
>>> Paul, is your case in some way reproducible in a testing environment
>>> so that a bisection might be give enough hints on the problem?
>> We hit this issue once more on the same server with Linux 5.15.160, and had
>> to hard reboot it.
>>
>> Unfortunately we did not have time yet to set up a test system to find a
>> reproducer. In our cases a lot of compute servers seem to have accessed the
>> NFS server. A lot of the many processes were `zstd` on a first glance.
>
> So we neither, due to the nature of the server (production system) and
> unability to reproduce the issue under some more controlled way and on
> test environment.
>
> In our case users seems to cause workloads involving use of wandb.
>
> What we tried is to boot the recent kernel from 5.10.y series avaiable
> (5.10.223-1). Then the issue showed up still. Since we disabled
> fs.leases-enable the situation seems to be more stable). While this
> is/might not be the solution, does that gives some additional hits?
The problem is backchannel-related, and disabling delegation
will reduce the number of backchannel operations. Your finding
comports with our current theory, but I can't think of how it
narrows the field of suspects.
Is the server running short on memory, perhaps? One backchannel
operation that was added in v5.10.220 is CB_RECALL_ANY, which
is triggered on memory exhaustion. But that should be a fairly
harmless addition unless there is a bug in there somewhere.
If your NFS server does not have any NFS mounts, then we could
provide instructions for enabling client-side tracing to watch
the details of callback traffic.
--
Chuck Lever
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-08-17 14:52 ` Chuck Lever III
@ 2024-10-29 21:07 ` Salvatore Bonaccorso
2025-01-31 16:17 ` Salvatore Bonaccorso
1 sibling, 0 replies; 16+ messages in thread
From: Salvatore Bonaccorso @ 2024-10-29 21:07 UTC (permalink / raw)
To: Chuck Lever III
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
Hi,
On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>
>
> > On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >
> > Hi,
> >
> > On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
> >> Dear Salvatore, dear Chuck,
> >>
> >>
> >> Thank you for your messages.
> >>
> >>
> >> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
> >>
> >>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
> >>>>
> >>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >>
> >>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> >>
> >>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> >>>>>> 04/22/2021, a mount from another server hung. Linux logs:
> >>>>>>
> >>>>>> ```
> >>>>>> $ dmesg -T
> >>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> >>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> >>>>>> […]
> >>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
> >>>>>> […]
> >>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> >>
> >> […]
> >>
> >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> >>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> >>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> >>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
> >>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
> >>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> >>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> >>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> >>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> >>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> >>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> >>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
> >>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
> >>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> >>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> >>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> >>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
> >>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
> >>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
> >>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
> >>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> >>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> >>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> >>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> >>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> >>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> >>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> >>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> >>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> >>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
> >>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
> >>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> >>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> >>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> >>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> >>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> >>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> >>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> >>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
> >>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
> >>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> >>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> >>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> >>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
> >>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
> >>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
> >>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
> >>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> >>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> >>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> >>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> >>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> >>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> >>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> >>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
> >>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> […]
> >>>>>> ```
> >>>>>
> >>>>> FWIW, on one NFS server occurence we are seeing something very close
> >>>>> to the above but in the 5.10.y case for the Debian kernel after
> >>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> >>>>> big NFS related stack backported.
> >>>>>
> >>>>> One backtrace we were able to catch was
> >>>>>
> >>>>> [...]
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> >>>>> [...]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> >>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> >>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> >>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> >>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> >>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
> >>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
> >>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> >>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> >>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> >>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> >>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> >>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> >>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
> >>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> >>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> >>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> >>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> >>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> >>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> >>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> >>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> >>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> >>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> >>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> >>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> >>>>> [...]
> >>>>>
> >>>>> Is there anything which could help debug this issue?
> >>>>
> >>>> The backtrace suggests an issue in the RPC client code; the
> >>>> server's NFSv4.1 backchannel would use that to send callbacks.
> >>>>
> >>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
> >>>> apart, a bisect should be quick and narrow down the issue to
> >>>> one or two commits.
> >>>
> >>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
> >>> more syntentic way on test environment, and the affected server in
> >>> particular is a production system.
> >>>
> >>> Paul, is your case in some way reproducible in a testing environment
> >>> so that a bisection might be give enough hints on the problem?
> >> We hit this issue once more on the same server with Linux 5.15.160, and had
> >> to hard reboot it.
> >>
> >> Unfortunately we did not have time yet to set up a test system to find a
> >> reproducer. In our cases a lot of compute servers seem to have accessed the
> >> NFS server. A lot of the many processes were `zstd` on a first glance.
> >
> > So we neither, due to the nature of the server (production system) and
> > unability to reproduce the issue under some more controlled way and on
> > test environment.
> >
> > In our case users seems to cause workloads involving use of wandb.
> >
> > What we tried is to boot the recent kernel from 5.10.y series avaiable
> > (5.10.223-1). Then the issue showed up still. Since we disabled
> > fs.leases-enable the situation seems to be more stable). While this
> > is/might not be the solution, does that gives some additional hits?
>
> The problem is backchannel-related, and disabling delegation
> will reduce the number of backchannel operations. Your finding
> comports with our current theory, but I can't think of how it
> narrows the field of suspects.
>
> Is the server running short on memory, perhaps? One backchannel
> operation that was added in v5.10.220 is CB_RECALL_ANY, which
> is triggered on memory exhaustion. But that should be a fairly
> harmless addition unless there is a bug in there somewhere.
>
> If your NFS server does not have any NFS mounts, then we could
> provide instructions for enabling client-side tracing to watch
> the details of callback traffic.
At least in our case, we still failed to get some more information on
those issues and failed to reproduce it under more syntentic and
controllable circumstances. Guess we have to give up for now, unless
Paul Menzel got further here.
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2024-08-17 14:52 ` Chuck Lever III
2024-10-29 21:07 ` Salvatore Bonaccorso
@ 2025-01-31 16:17 ` Salvatore Bonaccorso
2025-02-02 13:35 ` Salvatore Bonaccorso
1 sibling, 1 reply; 16+ messages in thread
From: Salvatore Bonaccorso @ 2025-01-31 16:17 UTC (permalink / raw)
To: Chuck Lever III
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
Hi Chuck,
On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>
>
> > On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >
> > Hi,
> >
> > On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
> >> Dear Salvatore, dear Chuck,
> >>
> >>
> >> Thank you for your messages.
> >>
> >>
> >> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
> >>
> >>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
> >>>>
> >>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> >>
> >>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> >>
> >>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> >>>>>> 04/22/2021, a mount from another server hung. Linux logs:
> >>>>>>
> >>>>>> ```
> >>>>>> $ dmesg -T
> >>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> >>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> >>>>>> […]
> >>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
> >>>>>> […]
> >>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> >>
> >> […]
> >>
> >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> >>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> >>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> >>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
> >>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
> >>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> >>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> >>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> >>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> >>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> >>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> >>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
> >>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
> >>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> >>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> >>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> >>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
> >>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
> >>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
> >>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
> >>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> >>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> >>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> >>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> >>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> >>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> >>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> >>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> >>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> >>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> >>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> >>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
> >>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
> >>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> >>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> >>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> >>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> >>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> >>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> >>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> >>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> >>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> >>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> >>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> >>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
> >>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
> >>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> >>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> >>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> >>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> >>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
> >>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
> >>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
> >>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
> >>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> >>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> >>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> >>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> >>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> >>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> >>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> >>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
> >>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>>> […]
> >>>>>> ```
> >>>>>
> >>>>> FWIW, on one NFS server occurence we are seeing something very close
> >>>>> to the above but in the 5.10.y case for the Debian kernel after
> >>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> >>>>> big NFS related stack backported.
> >>>>>
> >>>>> One backtrace we were able to catch was
> >>>>>
> >>>>> [...]
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> >>>>> [...]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> >>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> >>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> >>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> >>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> >>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
> >>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
> >>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> >>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> >>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> >>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> >>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> >>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> >>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
> >>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> >>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> >>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> >>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> >>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> >>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> >>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> >>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> >>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> >>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> >>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> >>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> >>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> >>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> >>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> >>>>> [...]
> >>>>>
> >>>>> Is there anything which could help debug this issue?
> >>>>
> >>>> The backtrace suggests an issue in the RPC client code; the
> >>>> server's NFSv4.1 backchannel would use that to send callbacks.
> >>>>
> >>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
> >>>> apart, a bisect should be quick and narrow down the issue to
> >>>> one or two commits.
> >>>
> >>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
> >>> more syntentic way on test environment, and the affected server in
> >>> particular is a production system.
> >>>
> >>> Paul, is your case in some way reproducible in a testing environment
> >>> so that a bisection might be give enough hints on the problem?
> >> We hit this issue once more on the same server with Linux 5.15.160, and had
> >> to hard reboot it.
> >>
> >> Unfortunately we did not have time yet to set up a test system to find a
> >> reproducer. In our cases a lot of compute servers seem to have accessed the
> >> NFS server. A lot of the many processes were `zstd` on a first glance.
> >
> > So we neither, due to the nature of the server (production system) and
> > unability to reproduce the issue under some more controlled way and on
> > test environment.
> >
> > In our case users seems to cause workloads involving use of wandb.
> >
> > What we tried is to boot the recent kernel from 5.10.y series avaiable
> > (5.10.223-1). Then the issue showed up still. Since we disabled
> > fs.leases-enable the situation seems to be more stable). While this
> > is/might not be the solution, does that gives some additional hits?
>
> The problem is backchannel-related, and disabling delegation
> will reduce the number of backchannel operations. Your finding
> comports with our current theory, but I can't think of how it
> narrows the field of suspects.
>
> Is the server running short on memory, perhaps? One backchannel
> operation that was added in v5.10.220 is CB_RECALL_ANY, which
> is triggered on memory exhaustion. But that should be a fairly
> harmless addition unless there is a bug in there somewhere.
>
> If your NFS server does not have any NFS mounts, then we could
> provide instructions for enabling client-side tracing to watch
> the details of callback traffic.
The NFS server acts as well as NFS client, so tracing more
back-channel related will I guess just load the tracing more.
But we got "lucky" and we were able to trigger the issue twice in last
days, once NFSv4 delegations were enabled again and some users started
to cause more load on the specific server as well.
I did issue
rpcdebug -m rpc -c
before rebooting/resetting the server which is
Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
and the first RPC related soft lookup slapt in the log/journal I was
able to gather is:
Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
Jan 29 22:34:05 nfsserver kernel: Call Trace:
Jan 29 22:34:05 nfsserver kernel: <IRQ>
Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
Jan 29 22:34:05 nfsserver kernel: </IRQ>
Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
I can try to pick on top of the kernel the change Chuck mentioned to
me offlist, which is the posting of
https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/,
and in fact this could be interesting. If the users keep doing the
same kind of load, this might help understanding more the issue.
As we suspect that the issue is more frequently triggered after the
switch of 5.10.118 -> 5.10.221, this enforces more the above, which
says it fixes 66af25799940 ("NFSD: add courteous server support for
thread with only delegation"), which is in 5.19-rc1, but got
backported to 5.15.154 and 5.10.220 as well.
Thanks for all your help,
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-01-31 16:17 ` Salvatore Bonaccorso
@ 2025-02-02 13:35 ` Salvatore Bonaccorso
2025-02-02 16:18 ` Chuck Lever
0 siblings, 1 reply; 16+ messages in thread
From: Salvatore Bonaccorso @ 2025-02-02 13:35 UTC (permalink / raw)
To: Chuck Lever III
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
Hi Chuck,
On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
> Hi Chuck,
>
> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
> >
> >
> > > On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> > >
> > > Hi,
> > >
> > > On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
> > >> Dear Salvatore, dear Chuck,
> > >>
> > >>
> > >> Thank you for your messages.
> > >>
> > >>
> > >> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
> > >>
> > >>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
> > >>>>
> > >>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> > >>
> > >>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> > >>
> > >>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> > >>>>>> 04/22/2021, a mount from another server hung. Linux logs:
> > >>>>>>
> > >>>>>> ```
> > >>>>>> $ dmesg -T
> > >>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> > >>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> > >>>>>> […]
> > >>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
> > >>>>>> […]
> > >>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> > >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> > >>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> > >>
> > >> […]
> > >>
> > >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> > >>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> > >>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > >>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> > >>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> > >>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> > >>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > >>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > >>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
> > >>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
> > >>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> > >>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > >>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> > >>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> > >>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> > >>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> > >>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> > >>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > >>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > >>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
> > >>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
> > >>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > >>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> > >>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> > >>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> > >>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
> > >>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
> > >>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
> > >>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
> > >>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > >>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> > >>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> > >>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> > >>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > >>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> > >>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> > >>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> > >>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> > >>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> > >>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
> > >>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > >>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> > >>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> > >>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> > >>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > >>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > >>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
> > >>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
> > >>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> > >>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > >>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > >>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> > >>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> > >>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> > >>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> > >>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> > >>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> > >>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> > >>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > >>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > >>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
> > >>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
> > >>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > >>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> > >>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> > >>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> > >>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
> > >>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
> > >>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
> > >>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
> > >>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > >>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> > >>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > >>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> > >>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> > >>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> > >>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> > >>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> > >>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> > >>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
> > >>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > >>>>>> […]
> > >>>>>> ```
> > >>>>>
> > >>>>> FWIW, on one NFS server occurence we are seeing something very close
> > >>>>> to the above but in the 5.10.y case for the Debian kernel after
> > >>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> > >>>>> big NFS related stack backported.
> > >>>>>
> > >>>>> One backtrace we were able to catch was
> > >>>>>
> > >>>>> [...]
> > >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> > >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> > >>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> > >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> > >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> > >>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> > >>>>> [...]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> > >>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> > >>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> > >>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> > >>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > >>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
> > >>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
> > >>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> > >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> > >>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> > >>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> > >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> > >>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> > >>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> > >>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> > >>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> > >>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> > >>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
> > >>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> > >>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> > >>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> > >>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> > >>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> > >>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> > >>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> > >>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> > >>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> > >>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> > >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > >>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> > >>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> > >>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > >>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> > >>>>> [...]
> > >>>>>
> > >>>>> Is there anything which could help debug this issue?
> > >>>>
> > >>>> The backtrace suggests an issue in the RPC client code; the
> > >>>> server's NFSv4.1 backchannel would use that to send callbacks.
> > >>>>
> > >>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
> > >>>> apart, a bisect should be quick and narrow down the issue to
> > >>>> one or two commits.
> > >>>
> > >>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
> > >>> more syntentic way on test environment, and the affected server in
> > >>> particular is a production system.
> > >>>
> > >>> Paul, is your case in some way reproducible in a testing environment
> > >>> so that a bisection might be give enough hints on the problem?
> > >> We hit this issue once more on the same server with Linux 5.15.160, and had
> > >> to hard reboot it.
> > >>
> > >> Unfortunately we did not have time yet to set up a test system to find a
> > >> reproducer. In our cases a lot of compute servers seem to have accessed the
> > >> NFS server. A lot of the many processes were `zstd` on a first glance.
> > >
> > > So we neither, due to the nature of the server (production system) and
> > > unability to reproduce the issue under some more controlled way and on
> > > test environment.
> > >
> > > In our case users seems to cause workloads involving use of wandb.
> > >
> > > What we tried is to boot the recent kernel from 5.10.y series avaiable
> > > (5.10.223-1). Then the issue showed up still. Since we disabled
> > > fs.leases-enable the situation seems to be more stable). While this
> > > is/might not be the solution, does that gives some additional hits?
> >
> > The problem is backchannel-related, and disabling delegation
> > will reduce the number of backchannel operations. Your finding
> > comports with our current theory, but I can't think of how it
> > narrows the field of suspects.
> >
> > Is the server running short on memory, perhaps? One backchannel
> > operation that was added in v5.10.220 is CB_RECALL_ANY, which
> > is triggered on memory exhaustion. But that should be a fairly
> > harmless addition unless there is a bug in there somewhere.
> >
> > If your NFS server does not have any NFS mounts, then we could
> > provide instructions for enabling client-side tracing to watch
> > the details of callback traffic.
>
> The NFS server acts as well as NFS client, so tracing more
> back-channel related will I guess just load the tracing more.
>
> But we got "lucky" and we were able to trigger the issue twice in last
> days, once NFSv4 delegations were enabled again and some users started
> to cause more load on the specific server as well.
>
> I did issue
>
> rpcdebug -m rpc -c
>
> before rebooting/resetting the server which is
>
> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
>
> and the first RPC related soft lookup slapt in the log/journal I was
> able to gather is:
>
> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
> Jan 29 22:34:05 nfsserver kernel: Call Trace:
> Jan 29 22:34:05 nfsserver kernel: <IRQ>
> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
> Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
> Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
> Jan 29 22:34:05 nfsserver kernel: </IRQ>
> Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
> Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
> Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>
> I can try to pick on top of the kernel the change Chuck mentioned to
> me offlist, which is the posting of
> https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/,
> and in fact this could be interesting. If the users keep doing the
> same kind of load, this might help understanding more the issue.
>
> As we suspect that the issue is more frequently triggered after the
> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
> says it fixes 66af25799940 ("NFSD: add courteous server support for
> thread with only delegation"), which is in 5.19-rc1, but got
> backported to 5.15.154 and 5.10.220 as well.
Unfortunately not. The system ran slightly more stable with that patch on, and
there was a nfsd hang inbeween here, within a series of
[...]
Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
[...]
Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for more than 120 seconds.
Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1590 ppid: 2 flags:0x00004000
Feb 02 05:34:46 nfsserver kernel: Call Trace:
Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for more than 120 seconds.
Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1599 ppid: 2 flags:0x00004000
Feb 02 05:34:46 nfsserver kernel: Call Trace:
Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for more than 121 seconds.
Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1601 ppid: 2 flags:0x00004000
Feb 02 05:34:46 nfsserver kernel: Call Trace:
Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for more than 121 seconds.
Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1604 ppid: 2 flags:0x00004000
Feb 02 05:34:47 nfsserver kernel: Call Trace:
Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for more than 121 seconds.
Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1610 ppid: 2 flags:0x00004000
Feb 02 05:34:47 nfsserver kernel: Call Trace:
Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
This happend a couple of times again and "recovered", but got finally stuck
again with:
Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625 q=145785)
Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
Feb 02 10:55:50 nfsserver kernel: Call Trace:
Feb 02 10:55:50 nfsserver kernel: <IRQ>
Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
Feb 02 10:55:50 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdb/0xf0
Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
Feb 02 10:55:50 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
Feb 02 10:55:50 nfsserver kernel: </IRQ>
Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
Feb 02 10:55:50 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
Feb 02 10:55:50 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c 24 08 65 48 2b 0c 25 28 00
Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS: 00000246
Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003e60000e
Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI: 0000000000000046 RDI: 0000000000000246
Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc08f6430 R09: ffffffffc08f6448
Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc08f6428
Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14: 00000000000001f4 R15: 0000000000000000
Feb 02 10:55:50 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
Before rebooting the system, rpcdebug -m rpc -c was issued again, with the
following logged entry:
Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp- -timeout ---ops--
Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
The system is now again back booted with fs.leases-enable=0 to keep it more
"stable".
Regards,
Salvatore
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-02-02 13:35 ` Salvatore Bonaccorso
@ 2025-02-02 16:18 ` Chuck Lever
2025-02-02 16:51 ` Jeff Layton
2025-02-03 1:06 ` Dai Ngo
0 siblings, 2 replies; 16+ messages in thread
From: Chuck Lever @ 2025-02-02 16:18 UTC (permalink / raw)
To: Salvatore Bonaccorso
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
> Hi Chuck,
>
> On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
>> Hi Chuck,
>>
>> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>>>
>>>
>>>> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>>>>> Dear Salvatore, dear Chuck,
>>>>>
>>>>>
>>>>> Thank you for your messages.
>>>>>
>>>>>
>>>>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>>>>
>>>>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>>>>
>>>>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>>
>>>>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>>>
>>>>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>>>>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>>>>
>>>>>>>>> ```
>>>>>>>>> $ dmesg -T
>>>>>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>>>>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>>>>>>>>> […]
>>>>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>>>>> […]
>>>>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>>>>>
>>>>> […]
>>>>>
>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>>>>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>>>>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
>>>>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
>>>>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
>>>>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>>>>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>>>>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
>>>>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
>>>>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
>>>>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>> […]
>>>>>>>>> ```
>>>>>>>>
>>>>>>>> FWIW, on one NFS server occurence we are seeing something very close
>>>>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
>>>>>>>> big NFS related stack backported.
>>>>>>>>
>>>>>>>> One backtrace we were able to catch was
>>>>>>>>
>>>>>>>> [...]
>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
>>>>>>>> [...]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>>>> [...]
>>>>>>>>
>>>>>>>> Is there anything which could help debug this issue?
>>>>>>>
>>>>>>> The backtrace suggests an issue in the RPC client code; the
>>>>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>>>>
>>>>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>>>>> apart, a bisect should be quick and narrow down the issue to
>>>>>>> one or two commits.
>>>>>>
>>>>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>>>>> more syntentic way on test environment, and the affected server in
>>>>>> particular is a production system.
>>>>>>
>>>>>> Paul, is your case in some way reproducible in a testing environment
>>>>>> so that a bisection might be give enough hints on the problem?
>>>>> We hit this issue once more on the same server with Linux 5.15.160, and had
>>>>> to hard reboot it.
>>>>>
>>>>> Unfortunately we did not have time yet to set up a test system to find a
>>>>> reproducer. In our cases a lot of compute servers seem to have accessed the
>>>>> NFS server. A lot of the many processes were `zstd` on a first glance.
>>>>
>>>> So we neither, due to the nature of the server (production system) and
>>>> unability to reproduce the issue under some more controlled way and on
>>>> test environment.
>>>>
>>>> In our case users seems to cause workloads involving use of wandb.
>>>>
>>>> What we tried is to boot the recent kernel from 5.10.y series avaiable
>>>> (5.10.223-1). Then the issue showed up still. Since we disabled
>>>> fs.leases-enable the situation seems to be more stable). While this
>>>> is/might not be the solution, does that gives some additional hits?
>>>
>>> The problem is backchannel-related, and disabling delegation
>>> will reduce the number of backchannel operations. Your finding
>>> comports with our current theory, but I can't think of how it
>>> narrows the field of suspects.
>>>
>>> Is the server running short on memory, perhaps? One backchannel
>>> operation that was added in v5.10.220 is CB_RECALL_ANY, which
>>> is triggered on memory exhaustion. But that should be a fairly
>>> harmless addition unless there is a bug in there somewhere.
>>>
>>> If your NFS server does not have any NFS mounts, then we could
>>> provide instructions for enabling client-side tracing to watch
>>> the details of callback traffic.
>>
>> The NFS server acts as well as NFS client, so tracing more
>> back-channel related will I guess just load the tracing more.
>>
>> But we got "lucky" and we were able to trigger the issue twice in last
>> days, once NFSv4 delegations were enabled again and some users started
>> to cause more load on the specific server as well.
>>
>> I did issue
>>
>> rpcdebug -m rpc -c
>>
>> before rebooting/resetting the server which is
>>
>> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
>>
>> and the first RPC related soft lookup slapt in the log/journal I was
>> able to gather is:
>>
>> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
>> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
>> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
>> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
>> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
>> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
>> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
>> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
>> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
>> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
>> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
>> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
>> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
>> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
>> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
>> Jan 29 22:34:05 nfsserver kernel: Call Trace:
>> Jan 29 22:34:05 nfsserver kernel: <IRQ>
>> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
>> Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
>> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
>> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
>> Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
>> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
>> Jan 29 22:34:05 nfsserver kernel: </IRQ>
>> Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
>> Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
>> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
>> Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
>> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
>> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
>> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
>> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>>
>> I can try to pick on top of the kernel the change Chuck mentioned to
>> me offlist, which is the posting of
>> https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/,
>> and in fact this could be interesting. If the users keep doing the
>> same kind of load, this might help understanding more the issue.
>>
>> As we suspect that the issue is more frequently triggered after the
>> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
>> says it fixes 66af25799940 ("NFSD: add courteous server support for
>> thread with only delegation"), which is in 5.19-rc1, but got
>> backported to 5.15.154 and 5.10.220 as well.
>
> Unfortunately not. The system ran slightly more stable with that patch on, and
> there was a nfsd hang inbeween here, within a series of
>
> [...]
> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
> Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
> Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
> Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
> Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
> Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
> Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
> Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
> Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
> Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
> Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
> Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
> Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
> Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
> Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
> Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
> Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
> [...]
> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for more than 120 seconds.
> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1590 ppid: 2 flags:0x00004000
> Feb 02 05:34:46 nfsserver kernel: Call Trace:
> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for more than 120 seconds.
> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1599 ppid: 2 flags:0x00004000
> Feb 02 05:34:46 nfsserver kernel: Call Trace:
> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for more than 121 seconds.
> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1601 ppid: 2 flags:0x00004000
> Feb 02 05:34:46 nfsserver kernel: Call Trace:
> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for more than 121 seconds.
> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1604 ppid: 2 flags:0x00004000
> Feb 02 05:34:47 nfsserver kernel: Call Trace:
> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
> Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for more than 121 seconds.
> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1610 ppid: 2 flags:0x00004000
> Feb 02 05:34:47 nfsserver kernel: Call Trace:
> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
This is a totally different failure mode: it's hanging in the
ext4 write path. One of your nfsd threads is stuck in D state
waiting to get a rw semaphor.
Question is, who is holding that rw_sem and why?
> This happend a couple of times again and "recovered", but got finally stuck
> again with:
>
> Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
> Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
> Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
> Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
> Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
> Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625 q=145785)
> Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
> Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: Call Trace:
> Feb 02 10:55:50 nfsserver kernel: <IRQ>
> Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
> Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> Feb 02 10:55:50 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdb/0xf0
> Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
> Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
> Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
> Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
> Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> Feb 02 10:55:50 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> Feb 02 10:55:50 nfsserver kernel: </IRQ>
> Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> Feb 02 10:55:50 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> Feb 02 10:55:50 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c 24 08 65 48 2b 0c 25 28 00
> Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS: 00000246
> Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003e60000e
> Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI: 0000000000000046 RDI: 0000000000000246
> Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc08f6430 R09: ffffffffc08f6448
> Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc08f6428
> Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14: 00000000000001f4 R15: 0000000000000000
> Feb 02 10:55:50 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
> Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
> Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
> Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
> Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
>
> Before rebooting the system, rpcdebug -m rpc -c was issued again, with the
> following logged entry:
>
> Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp- -timeout ---ops--
> Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
This is also different: the CB_RECALL_ANY is waiting to start, it's not
retransmitting.
> The system is now again back booted with fs.leases-enable=0 to keep it more
> "stable".
Understood, but I don't yet see how this new scenario is related to
NFSv4 delegation. We can speculate, but here's nothing standing out in
the collected data.
--
Chuck Lever
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-02-02 16:18 ` Chuck Lever
@ 2025-02-02 16:51 ` Jeff Layton
2025-05-26 16:31 ` Paul Menzel
2025-02-03 1:06 ` Dai Ngo
1 sibling, 1 reply; 16+ messages in thread
From: Jeff Layton @ 2025-02-02 16:51 UTC (permalink / raw)
To: Chuck Lever, Salvatore Bonaccorso
Cc: Paul Menzel, Linux NFS Mailing List, it+linux-nfs@molgen.mpg.de
On Sun, 2025-02-02 at 11:18 -0500, Chuck Lever wrote:
> On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
> > Hi Chuck,
> >
> > On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
> > > Hi Chuck,
> > >
> > > On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
> > > >
> > > >
> > > > > On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
> > > > > > Dear Salvatore, dear Chuck,
> > > > > >
> > > > > >
> > > > > > Thank you for your messages.
> > > > > >
> > > > > >
> > > > > > Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
> > > > > >
> > > > > > > On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
> > > > > > > >
> > > > > > > > > On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
> > > > > >
> > > > > > > > > On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
> > > > > >
> > > > > > > > > > Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
> > > > > > > > > > 04/22/2021, a mount from another server hung. Linux logs:
> > > > > > > > > >
> > > > > > > > > > ```
> > > > > > > > > > $ dmesg -T
> > > > > > > > > > [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
> > > > > > > > > > [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
> > > > > > > > > > […]
> > > > > > > > > > [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
> > > > > > > > > > […]
> > > > > > > > > > [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
> > > > > > > > > > [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
> > > > > > > > > > [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
> > > > > >
> > > > > > […]
> > > > > >
> > > > > > > > > > [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
> > > > > > > > > > [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] Call Trace:
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] <IRQ>
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] </IRQ>
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] <TASK>
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
> > > > > > > > > > [Tue Jul 16 11:36:40 2024] </TASK>
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] Call Trace:
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] <IRQ>
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] </IRQ>
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] <TASK>
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
> > > > > > > > > > [Tue Jul 16 11:37:19 2024] </TASK>
> > > > > > > > > > [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > > > > > […]
> > > > > > > > > > ```
> > > > > > > > >
> > > > > > > > > FWIW, on one NFS server occurence we are seeing something very close
> > > > > > > > > to the above but in the 5.10.y case for the Debian kernel after
> > > > > > > > > updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
> > > > > > > > > big NFS related stack backported.
> > > > > > > > >
> > > > > > > > > One backtrace we were able to catch was
> > > > > > > > >
> > > > > > > > > [...]
> > > > > > > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
> > > > > > > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
> > > > > > > > > Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
> > > > > > > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
> > > > > > > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
> > > > > > > > > Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
> > > > > > > > > [...]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: Call Trace:
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: <IRQ>
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: </IRQ>
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > > > > > > > > Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
> > > > > > > > > [...]
> > > > > > > > >
> > > > > > > > > Is there anything which could help debug this issue?
> > > > > > > >
> > > > > > > > The backtrace suggests an issue in the RPC client code; the
> > > > > > > > server's NFSv4.1 backchannel would use that to send callbacks.
> > > > > > > >
> > > > > > > > Since 5.10.218 and 5.10.221 are only about a thousand commits
> > > > > > > > apart, a bisect should be quick and narrow down the issue to
> > > > > > > > one or two commits.
> > > > > > >
> > > > > > > Yes indeed. Unfortunately was yet unable to reproduce the issue in
> > > > > > > more syntentic way on test environment, and the affected server in
> > > > > > > particular is a production system.
> > > > > > >
> > > > > > > Paul, is your case in some way reproducible in a testing environment
> > > > > > > so that a bisection might be give enough hints on the problem?
> > > > > > We hit this issue once more on the same server with Linux 5.15.160, and had
> > > > > > to hard reboot it.
> > > > > >
> > > > > > Unfortunately we did not have time yet to set up a test system to find a
> > > > > > reproducer. In our cases a lot of compute servers seem to have accessed the
> > > > > > NFS server. A lot of the many processes were `zstd` on a first glance.
> > > > >
> > > > > So we neither, due to the nature of the server (production system) and
> > > > > unability to reproduce the issue under some more controlled way and on
> > > > > test environment.
> > > > >
> > > > > In our case users seems to cause workloads involving use of wandb.
> > > > >
> > > > > What we tried is to boot the recent kernel from 5.10.y series avaiable
> > > > > (5.10.223-1). Then the issue showed up still. Since we disabled
> > > > > fs.leases-enable the situation seems to be more stable). While this
> > > > > is/might not be the solution, does that gives some additional hits?
> > > >
> > > > The problem is backchannel-related, and disabling delegation
> > > > will reduce the number of backchannel operations. Your finding
> > > > comports with our current theory, but I can't think of how it
> > > > narrows the field of suspects.
> > > >
> > > > Is the server running short on memory, perhaps? One backchannel
> > > > operation that was added in v5.10.220 is CB_RECALL_ANY, which
> > > > is triggered on memory exhaustion. But that should be a fairly
> > > > harmless addition unless there is a bug in there somewhere.
> > > >
> > > > If your NFS server does not have any NFS mounts, then we could
> > > > provide instructions for enabling client-side tracing to watch
> > > > the details of callback traffic.
> > >
> > > The NFS server acts as well as NFS client, so tracing more
> > > back-channel related will I guess just load the tracing more.
> > >
> > > But we got "lucky" and we were able to trigger the issue twice in last
> > > days, once NFSv4 delegations were enabled again and some users started
> > > to cause more load on the specific server as well.
> > >
> > > I did issue
> > >
> > > rpcdebug -m rpc -c
> > >
> > > before rebooting/resetting the server which is
> > >
> > > Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
> > >
> > > and the first RPC related soft lookup slapt in the log/journal I was
> > > able to gather is:
> > >
> > > Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
> > > Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
> > > Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
> > > Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
> > > Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > > Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > > Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
> > > Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
> > > Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
> > > Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
> > > Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
> > > Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
> > > Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
> > > Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
> > > Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
> > > Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
> > > Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> > > Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
> > > Jan 29 22:34:05 nfsserver kernel: Call Trace:
> > > Jan 29 22:34:05 nfsserver kernel: <IRQ>
> > > Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
> > > Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
> > > Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
> > > Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
> > > Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
> > > Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
> > > Jan 29 22:34:05 nfsserver kernel: </IRQ>
> > > Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
> > > Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
> > > Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
> > > Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
> > > Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > > Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > > Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > > Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
> > > Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
> > > Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
> > > Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
> > > Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > > Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
> > >
> > > I can try to pick on top of the kernel the change Chuck mentioned to
> > > me offlist, which is the posting of
> > > https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/,
> > > and in fact this could be interesting. If the users keep doing the
> > > same kind of load, this might help understanding more the issue.
> > >
> > > As we suspect that the issue is more frequently triggered after the
> > > switch of 5.10.118 -> 5.10.221, this enforces more the above, which
> > > says it fixes 66af25799940 ("NFSD: add courteous server support for
> > > thread with only delegation"), which is in 5.19-rc1, but got
> > > backported to 5.15.154 and 5.10.220 as well.
> >
> > Unfortunately not. The system ran slightly more stable with that patch on, and
> > there was a nfsd hang inbeween here, within a series of
> >
> > [...]
> > Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
> > Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
> > Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
> > Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
> > Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
> > Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
> > Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
> > Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
> > Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
> > Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
> > Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
> > Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
> > Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
> > Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
> > Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
> > Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
> > Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
> > Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
> > Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
> > Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
> > Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
> > Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
> > Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
> > Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
> > Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
> > Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
> > Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
> > Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
> > Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
> > Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
> > Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
> > Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
> > Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
> > Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
> > [...]
> > Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for more than 120 seconds.
> > Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1590 ppid: 2 flags:0x00004000
> > Feb 02 05:34:46 nfsserver kernel: Call Trace:
> > Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> > Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> > Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> > Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> > Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> > Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> > Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
> > Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
> > Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> > Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for more than 120 seconds.
> > Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1599 ppid: 2 flags:0x00004000
> > Feb 02 05:34:46 nfsserver kernel: Call Trace:
> > Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> > Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> > Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> > Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> > Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> > Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> > Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
> > Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> > Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> > Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
> > Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> > Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for more than 121 seconds.
> > Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1601 ppid: 2 flags:0x00004000
> > Feb 02 05:34:46 nfsserver kernel: Call Trace:
> > Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
> > Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> > Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
> > Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> > Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> > Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> > Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> > Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> > Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> > Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for more than 121 seconds.
> > Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1604 ppid: 2 flags:0x00004000
> > Feb 02 05:34:47 nfsserver kernel: Call Trace:
> > Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
> > Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
> > Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
> > Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> > Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> > Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> > Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> > Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> > Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> > Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for more than 121 seconds.
> > Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1610 ppid: 2 flags:0x00004000
> > Feb 02 05:34:47 nfsserver kernel: Call Trace:
> > Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
> > Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
> > Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
> > Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
> > Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
> > Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
> > Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
> > Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
> > Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
> > Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>
> This is a totally different failure mode: it's hanging in the
> ext4 write path. One of your nfsd threads is stuck in D state
> waiting to get a rw semaphor.
>
> Question is, who is holding that rw_sem and why?
>
>
It looks like ext4_buffered_write_iter() takes the inode_lock, so it's
probably the inode->i_rwsem that it's waiting on. Unfortunately all
sorts of things take that lock so it's hard to speculate about the
cause of it being stuck. Consider triggering a sysrq-w if this occurs
again, which would tell us something about the contended locks.
> > This happend a couple of times again and "recovered", but got finally stuck
> > again with:
> >
> > Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
> > Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
> > Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
> > Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
> > Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
> > Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
> > Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625 q=145785)
> > Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
> > Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
> > Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
> > Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: Call Trace:
> > Feb 02 10:55:50 nfsserver kernel: <IRQ>
> > Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
> > Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
> > Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
> > Feb 02 10:55:50 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdb/0xf0
> > Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
> > Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
> > Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
> > Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
> > Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
> > Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
> > Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
> > Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
> > Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
> > Feb 02 10:55:50 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
> > Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
> > Feb 02 10:55:50 nfsserver kernel: </IRQ>
> > Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
> > Feb 02 10:55:50 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
> > Feb 02 10:55:50 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
mod_delayed_work_on() disables IRQs and then calls down into the
workqueue code to modify a wq job. If that took too long then you'd see
an rcu_sched warning like this.
> > Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c 24 08 65 48 2b 0c 25 28 00
> > Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS: 00000246
> > Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003e60000e
> > Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI: 0000000000000046 RDI: 0000000000000246
> > Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc08f6430 R09: ffffffffc08f6448
> > Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc08f6428
> > Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14: 00000000000001f4 R15: 0000000000000000
> > Feb 02 10:55:50 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
> > Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
> > Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
> > Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
> > Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
> > Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
> > Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
> >
> > Before rebooting the system, rpcdebug -m rpc -c was issued again, with the
> > following logged entry:
> >
> > Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp- -timeout ---ops--
> > Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
>
> This is also different: the CB_RECALL_ANY is waiting to start, it's not
> retransmitting.
>
>
>
> > The system is now again back booted with fs.leases-enable=0 to keep it more
> > "stable".
>
> Understood, but I don't yet see how this new scenario is related to
> NFSv4 delegation. We can speculate, but here's nothing standing out in
> the collected data.
>
>
Agreed. It looks like there are bigger issues than just nfsd here.
--
Jeff Layton <jlayton@kernel.org>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-02-02 16:18 ` Chuck Lever
2025-02-02 16:51 ` Jeff Layton
@ 2025-02-03 1:06 ` Dai Ngo
2025-02-03 14:22 ` Chuck Lever
1 sibling, 1 reply; 16+ messages in thread
From: Dai Ngo @ 2025-02-03 1:06 UTC (permalink / raw)
To: Chuck Lever, Salvatore Bonaccorso
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
On 2/2/25 8:18 AM, Chuck Lever wrote:
> On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
>> Hi Chuck,
>>
>> On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
>>> Hi Chuck,
>>>
>>> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>>>>
>>>>> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>>>>>> Dear Salvatore, dear Chuck,
>>>>>>
>>>>>>
>>>>>> Thank you for your messages.
>>>>>>
>>>>>>
>>>>>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>>>>>
>>>>>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>>>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>>>>>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> $ dmesg -T
>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>>>>>>>>>> […]
>>>>>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>>>>>> […]
>>>>>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>>>>>> […]
>>>>>>
>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>> […]
>>>>>>>>>> ```
>>>>>>>>> FWIW, on one NFS server occurence we are seeing something very close
>>>>>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
>>>>>>>>> big NFS related stack backported.
>>>>>>>>>
>>>>>>>>> One backtrace we were able to catch was
>>>>>>>>>
>>>>>>>>> [...]
>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
>>>>>>>>> [...]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>> Is there anything which could help debug this issue?
>>>>>>>> The backtrace suggests an issue in the RPC client code; the
>>>>>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>>>>>
>>>>>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>>>>>> apart, a bisect should be quick and narrow down the issue to
>>>>>>>> one or two commits.
>>>>>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>>>>>> more syntentic way on test environment, and the affected server in
>>>>>>> particular is a production system.
>>>>>>>
>>>>>>> Paul, is your case in some way reproducible in a testing environment
>>>>>>> so that a bisection might be give enough hints on the problem?
>>>>>> We hit this issue once more on the same server with Linux 5.15.160, and had
>>>>>> to hard reboot it.
>>>>>>
>>>>>> Unfortunately we did not have time yet to set up a test system to find a
>>>>>> reproducer. In our cases a lot of compute servers seem to have accessed the
>>>>>> NFS server. A lot of the many processes were `zstd` on a first glance.
>>>>> So we neither, due to the nature of the server (production system) and
>>>>> unability to reproduce the issue under some more controlled way and on
>>>>> test environment.
>>>>>
>>>>> In our case users seems to cause workloads involving use of wandb.
>>>>>
>>>>> What we tried is to boot the recent kernel from 5.10.y series avaiable
>>>>> (5.10.223-1). Then the issue showed up still. Since we disabled
>>>>> fs.leases-enable the situation seems to be more stable). While this
>>>>> is/might not be the solution, does that gives some additional hits?
>>>> The problem is backchannel-related, and disabling delegation
>>>> will reduce the number of backchannel operations. Your finding
>>>> comports with our current theory, but I can't think of how it
>>>> narrows the field of suspects.
>>>>
>>>> Is the server running short on memory, perhaps? One backchannel
>>>> operation that was added in v5.10.220 is CB_RECALL_ANY, which
>>>> is triggered on memory exhaustion. But that should be a fairly
>>>> harmless addition unless there is a bug in there somewhere.
>>>>
>>>> If your NFS server does not have any NFS mounts, then we could
>>>> provide instructions for enabling client-side tracing to watch
>>>> the details of callback traffic.
>>> The NFS server acts as well as NFS client, so tracing more
>>> back-channel related will I guess just load the tracing more.
>>>
>>> But we got "lucky" and we were able to trigger the issue twice in last
>>> days, once NFSv4 delegations were enabled again and some users started
>>> to cause more load on the specific server as well.
>>>
>>> I did issue
>>>
>>> rpcdebug -m rpc -c
>>>
>>> before rebooting/resetting the server which is
>>>
>>> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
>>>
>>> and the first RPC related soft lookup slapt in the log/journal I was
>>> able to gather is:
>>>
>>> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
>>> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
>>> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
>>> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
>>> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
>>> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
>>> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
>>> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
>>> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
>>> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
>>> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
>>> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
>>> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
>>> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
>>> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
>>> Jan 29 22:34:05 nfsserver kernel: Call Trace:
>>> Jan 29 22:34:05 nfsserver kernel: <IRQ>
>>> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
>>> Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
>>> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
>>> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
>>> Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
>>> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
>>> Jan 29 22:34:05 nfsserver kernel: </IRQ>
>>> Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
>>> Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>>> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
>>> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
>>> Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
>>> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
>>> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
>>> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
>>> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>
>>> I can try to pick on top of the kernel the change Chuck mentioned to
>>> me offlist, which is the posting of
>>> https://urldefense.com/v3/__https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/__;!!ACWV5N9M2RV99hQ!ILxf31lSoNImIDh3FjDiD-qFRBH8gPEQhUW31gF2NOYGPFzPscgj7S23PoaBR1MFs6VLMprfKi9g6WdEkyY$ ,
>>> and in fact this could be interesting. If the users keep doing the
>>> same kind of load, this might help understanding more the issue.
>>>
>>> As we suspect that the issue is more frequently triggered after the
>>> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
>>> says it fixes 66af25799940 ("NFSD: add courteous server support for
>>> thread with only delegation"), which is in 5.19-rc1, but got
>>> backported to 5.15.154 and 5.10.220 as well.
>> Unfortunately not. The system ran slightly more stable with that patch on, and
>> there was a nfsd hang inbeween here, within a series of
>>
>> [...]
>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
>> Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
>> Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
>> Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
>> Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
>> Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
>> Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
>> Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
>> Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
>> Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
>> Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
>> Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
>> Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
>> Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
>> Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
>> Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
>> Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
>> [...]
>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for more than 120 seconds.
>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1590 ppid: 2 flags:0x00004000
>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for more than 120 seconds.
>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1599 ppid: 2 flags:0x00004000
>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for more than 121 seconds.
>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1601 ppid: 2 flags:0x00004000
>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for more than 121 seconds.
>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1604 ppid: 2 flags:0x00004000
>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>> Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for more than 121 seconds.
>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1610 ppid: 2 flags:0x00004000
>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
> This is a totally different failure mode: it's hanging in the
> ext4 write path. One of your nfsd threads is stuck in D state
> waiting to get a rw semaphor.
>
> Question is, who is holding that rw_sem and why?
>
>
>> This happend a couple of times again and "recovered", but got finally stuck
>> again with:
>>
>> Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
>> Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
>> Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
>> Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
>> Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>> Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
>> Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625 q=145785)
>> Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
>> Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>> Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>> Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: Call Trace:
>> Feb 02 10:55:50 nfsserver kernel: <IRQ>
>> Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
>> Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>> Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>> Feb 02 10:55:50 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdb/0xf0
>> Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>> Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>> Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
>> Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
>> Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
>> Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
>> Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>> Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>> Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>> Feb 02 10:55:50 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>> Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>> Feb 02 10:55:50 nfsserver kernel: </IRQ>
>> Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>> Feb 02 10:55:50 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>> Feb 02 10:55:50 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>> Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c 24 08 65 48 2b 0c 25 28 00
>> Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS: 00000246
>> Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003e60000e
>> Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI: 0000000000000046 RDI: 0000000000000246
>> Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc08f6430 R09: ffffffffc08f6448
>> Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc08f6428
>> Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14: 00000000000001f4 R15: 0000000000000000
>> Feb 02 10:55:50 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>> Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
>> Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
>> Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
>> Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
>> Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>> Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
>>
>> Before rebooting the system, rpcdebug -m rpc -c was issued again, with the
>> following logged entry:
>>
>> Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp- -timeout ---ops--
>> Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
> This is also different: the CB_RECALL_ANY is waiting to start, it's not
> retransmitting.
When CB_RECALL_ANY is returned with cb_seq_status == 1, it is restarted
by nfsd4_cb_sequence_done. Restarting means the callback is re-queued in
nfsd4_cb_release which schedules a new work to re-send the callback. So
the 'call_start' status could indicate that the CB_RECALL_ANY is being
resending in a loop.
-Dai
>
>
>> The system is now again back booted with fs.leases-enable=0 to keep it more
>> "stable".
> Understood, but I don't yet see how this new scenario is related to
> NFSv4 delegation. We can speculate, but here's nothing standing out in
> the collected data.
>
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-02-03 1:06 ` Dai Ngo
@ 2025-02-03 14:22 ` Chuck Lever
0 siblings, 0 replies; 16+ messages in thread
From: Chuck Lever @ 2025-02-03 14:22 UTC (permalink / raw)
To: Dai Ngo, Salvatore Bonaccorso
Cc: Paul Menzel, Jeff Layton, Linux NFS Mailing List,
it+linux-nfs@molgen.mpg.de
On 2/2/25 8:06 PM, Dai Ngo wrote:
>
> On 2/2/25 8:18 AM, Chuck Lever wrote:
>> On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
>>> Hi Chuck,
>>>
>>> On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
>>>> Hi Chuck,
>>>>
>>>> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>>>>>
>>>>>> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso
>>>>>> <carnil@debian.org> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>>>>>>> Dear Salvatore, dear Chuck,
>>>>>>>
>>>>>>>
>>>>>>> Thank you for your messages.
>>>>>>>
>>>>>>>
>>>>>>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>>>>>>
>>>>>>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>>>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso
>>>>>>>>>> <carnil@debian.org> wrote:
>>>>>>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>>>>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS
>>>>>>>>>>> 2.11.2
>>>>>>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>>>>>>
>>>>>>>>>>> ```
>>>>>>>>>>> $ dmesg -T
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Linux version
>>>>>>>>>>> 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC)
>>>>>>>>>>> 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15
>>>>>>>>>>> CEST 2024
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
>>>>>>>>>>> crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0
>>>>>>>>>>> init=/bin/systemd audit=0 random.trust_cpu=on
>>>>>>>>>>> systemd.unified_cgroup_hierarchy
>>>>>>>>>>> […]
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge
>>>>>>>>>>> T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>>>>>>> […]
>>>>>>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized
>>>>>>>>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized
>>>>>>>>>>> reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>>>>>>> […]
>>>>>>>
>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized
>>>>>>>>>>> reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized
>>>>>>>>>>> reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected
>>>>>>>>>>> stall on CPU
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this
>>>>>>>>>>> GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928
>>>>>>>>>>> fqs=4433
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R
>>>>>>>>>>> running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod
>>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>> sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05
>>>>>>>>>>> 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66
>>>>>>>>>>> 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48
>>>>>>>>>>> 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS:
>>>>>>>>>>> 00000246
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX:
>>>>>>>>>>> 000000003f3c079e RCX: 000000000000100d
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI:
>>>>>>>>>>> 0000000000000046 RDI: ffffffff82435600
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08:
>>>>>>>>>>> ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11:
>>>>>>>>>>> 0000000000000283 R12: 0000000000000000
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14:
>>>>>>>>>>> ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected
>>>>>>>>>>> stall on CPU
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this
>>>>>>>>>>> GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492
>>>>>>>>>>> fqs=5159
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R
>>>>>>>>>>> running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod
>>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>> sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07
>>>>>>>>>>> a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44
>>>>>>>>>>> 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc
>>>>>>>>>>> 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS:
>>>>>>>>>>> 00000246
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>>> ffff88997131a500 RCX: 0000000000000001
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI:
>>>>>>>>>>> ffff88997131a500 RDI: ffffffffa012c700
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08:
>>>>>>>>>>> ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11:
>>>>>>>>>>> 0000000000000283 R12: ffff88997131a530
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14:
>>>>>>>>>>> ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected
>>>>>>>>>>> stall on CPU
>>>>>>>>>>> […]
>>>>>>>>>>> ```
>>>>>>>>>> FWIW, on one NFS server occurence we are seeing something very
>>>>>>>>>> close
>>>>>>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>>>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which
>>>>>>>>>> had the
>>>>>>>>>> big NFS related stack backported.
>>>>>>>>>>
>>>>>>>>>> One backtrace we were able to catch was
>>>>>>>>>>
>>>>>>>>>> [...]
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec
>>>>>>>>>> xid b172e1c6
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a
>>>>>>>>>> xid a90d7751
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521
>>>>>>>>>> xid 8e5e58bd
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319
>>>>>>>>>> xid c2da3c73
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21
>>>>>>>>>> xid a01bfec6
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca
>>>>>>>>>> xid c2eeeaa6
>>>>>>>>>> [...]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-
>>>>>>>>>> detected stall on CPU
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250
>>>>>>>>>> ticks this GP) idle=74e/1/0x4000000000000000
>>>>>>>>>> softirq=3160997/3161006 fqs=2233
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies
>>>>>>>>>> g=8381377 q=106333)
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm:
>>>>>>>>>> kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG
>>>>>>>>>> S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559
>>>>>>>>>> 03/19/2019
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod
>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> nmi_cpu_backtrace.cold+0x32/0x69
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>> lapic_can_unplug_cpu+0x80/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>> trigger_load_balance+0x5a/0x220
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>> tick_sched_do_timer+0x90/0x90
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> __hrtimer_run_queues+0x127/0x280
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> sysvec_apic_timer_interrupt+0x72/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RIP:
>>>>>>>>>> 0010:mod_delayed_work_on+0x5d/0x90
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe
>>>>>>>>>> ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2
>>>>>>>>>> 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90
>>>>>>>>>> EFLAGS: 00000246
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX:
>>>>>>>>>> 0000000000000000 RCX: 000000003820000f
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI:
>>>>>>>>>> 0000000000000046 RDI: 0000000000000246
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08:
>>>>>>>>>> ffffffffc0884430 R09: ffffffffc0884448
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11:
>>>>>>>>>> 0000000000000003 R12: ffffffffc0884428
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14:
>>>>>>>>>> 00000000000001f4 R15: 0000000000000000
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140
>>>>>>>>>> [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410
>>>>>>>>>> [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>> rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>> __kthread_bind_mask+0x60/0x60
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>>>>>> [...]
>>>>>>>>>>
>>>>>>>>>> Is there anything which could help debug this issue?
>>>>>>>>> The backtrace suggests an issue in the RPC client code; the
>>>>>>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>>>>>>
>>>>>>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>>>>>>> apart, a bisect should be quick and narrow down the issue to
>>>>>>>>> one or two commits.
>>>>>>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>>>>>>> more syntentic way on test environment, and the affected server in
>>>>>>>> particular is a production system.
>>>>>>>>
>>>>>>>> Paul, is your case in some way reproducible in a testing
>>>>>>>> environment
>>>>>>>> so that a bisection might be give enough hints on the problem?
>>>>>>> We hit this issue once more on the same server with Linux
>>>>>>> 5.15.160, and had
>>>>>>> to hard reboot it.
>>>>>>>
>>>>>>> Unfortunately we did not have time yet to set up a test system to
>>>>>>> find a
>>>>>>> reproducer. In our cases a lot of compute servers seem to have
>>>>>>> accessed the
>>>>>>> NFS server. A lot of the many processes were `zstd` on a first
>>>>>>> glance.
>>>>>> So we neither, due to the nature of the server (production system)
>>>>>> and
>>>>>> unability to reproduce the issue under some more controlled way
>>>>>> and on
>>>>>> test environment.
>>>>>>
>>>>>> In our case users seems to cause workloads involving use of wandb.
>>>>>>
>>>>>> What we tried is to boot the recent kernel from 5.10.y series
>>>>>> avaiable
>>>>>> (5.10.223-1). Then the issue showed up still. Since we disabled
>>>>>> fs.leases-enable the situation seems to be more stable). While this
>>>>>> is/might not be the solution, does that gives some additional hits?
>>>>> The problem is backchannel-related, and disabling delegation
>>>>> will reduce the number of backchannel operations. Your finding
>>>>> comports with our current theory, but I can't think of how it
>>>>> narrows the field of suspects.
>>>>>
>>>>> Is the server running short on memory, perhaps? One backchannel
>>>>> operation that was added in v5.10.220 is CB_RECALL_ANY, which
>>>>> is triggered on memory exhaustion. But that should be a fairly
>>>>> harmless addition unless there is a bug in there somewhere.
>>>>>
>>>>> If your NFS server does not have any NFS mounts, then we could
>>>>> provide instructions for enabling client-side tracing to watch
>>>>> the details of callback traffic.
>>>> The NFS server acts as well as NFS client, so tracing more
>>>> back-channel related will I guess just load the tracing more.
>>>>
>>>> But we got "lucky" and we were able to trigger the issue twice in last
>>>> days, once NFSv4 delegations were enabled again and some users started
>>>> to cause more load on the specific server as well.
>>>>
>>>> I did issue
>>>>
>>>> rpcdebug -m rpc -c
>>>>
>>>> before rebooting/resetting the server which is
>>>>
>>>> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92
>>>> 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc]
>>>> q:delayq
>>>>
>>>> and the first RPC related soft lookup slapt in the log/journal I was
>>>> able to gather is:
>>>>
>>>> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup -
>>>> CPU#11 stuck for 23s! [kworker/u42:3:705574]
>>>> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc
>>>> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2
>>>> quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac
>>>> skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp
>>>> coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel
>>>> libaes crypto_simd cryptd ast glue_helper drm_vram_helper
>>>> drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si
>>>> intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf
>>>> cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit
>>>> watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button
>>>> fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc
>>>> ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456
>>>> async_raid6_recov async_memcpy async_pq async_xor async_tx xor
>>>> raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear
>>>> md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif
>>>> crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
>>>> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr
>>>> crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod
>>>> pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
>>>> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/
>>>> u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
>>>> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/
>>>> S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod
>>>> rpc_async_schedule [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
>>>> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48
>>>> 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5
>>>> 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41
>>>> 5c c3 cc cc cc cc f3 90 eb 84
>>>> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS:
>>>> 00000202
>>>> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX:
>>>> 000005fce82a4372 RCX: 0000000000000018
>>>> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI:
>>>> 000000000031f238 RDI: 00385c1353c92824
>>>> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08:
>>>> ffffffffc081f410 R09: ffffffffc081f410
>>>> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11:
>>>> 0000000000000003 R12: 000000000031f238
>>>> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14:
>>>> 0000000000000001 R15: 0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000)
>>>> GS:ffff8ee94f880000(0000) knlGS:0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>> 0000000080050033
>>>> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3:
>>>> 00000017c420a002 CR4: 00000000007706e0
>>>> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1:
>>>> 0000000000000000 DR2: 0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6:
>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
>>>> Jan 29 22:34:05 nfsserver kernel: Call Trace:
>>>> Jan 29 22:34:05 nfsserver kernel: <IRQ>
>>>> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>> lockup_detector_update_enable+0x50/0x50
>>>> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
>>>> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
>>>> Jan 29 22:34:05 nfsserver kernel: </IRQ>
>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>> sysvec_apic_timer_interrupt+0x72/0x80
>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
>>>> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40
>>>> [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
>>>> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
>>>> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
>>>> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
>>>> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>
>>>> I can try to pick on top of the kernel the change Chuck mentioned to
>>>> me offlist, which is the posting of
>>>> https://urldefense.com/v3/__https://lore.kernel.org/linux-
>>>> nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/__;!!
>>>> ACWV5N9M2RV99hQ!ILxf31lSoNImIDh3FjDiD-
>>>> qFRBH8gPEQhUW31gF2NOYGPFzPscgj7S23PoaBR1MFs6VLMprfKi9g6WdEkyY$ ,
>>>> and in fact this could be interesting. If the users keep doing the
>>>> same kind of load, this might help understanding more the issue.
>>>>
>>>> As we suspect that the issue is more frequently triggered after the
>>>> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
>>>> says it fixes 66af25799940 ("NFSD: add courteous server support for
>>>> thread with only delegation"), which is in 5.19-rc1, but got
>>>> backported to 5.15.154 and 5.10.220 as well.
>>> Unfortunately not. The system ran slightly more stable with that
>>> patch on, and
>>> there was a nfsd hang inbeween here, within a series of
>>>
>>> [...]
>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
>>> Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
>>> Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
>>> Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
>>> Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
>>> Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
>>> Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
>>> Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
>>> Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
>>> Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
>>> Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
>>> Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
>>> Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
>>> Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
>>> Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
>>> Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
>>> Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
>>> [...]
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for
>>> more than 120 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>> hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>> stack: 0 pid: 1590 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel:
>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>> [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for
>>> more than 120 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>> hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>> stack: 0 pid: 1599 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel:
>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>> [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for
>>> more than 121 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>> hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>> stack: 0 pid: 1601 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel:
>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>> [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for
>>> more than 121 seconds.
>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E
>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>> hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D
>>> stack: 0 pid: 1604 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel:
>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>> [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for
>>> more than 121 seconds.
>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E
>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>> hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D
>>> stack: 0 pid: 1610 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel:
>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>> [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>> This is a totally different failure mode: it's hanging in the
>> ext4 write path. One of your nfsd threads is stuck in D state
>> waiting to get a rw semaphor.
>>
>> Question is, who is holding that rw_sem and why?
>>
>>
>>> This happend a couple of times again and "recovered", but got finally
>>> stuck
>>> again with:
>>>
>>> Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
>>> Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
>>> Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
>>> Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized
>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
>>> Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected
>>> stall on CPU
>>> Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks
>>> this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
>>> Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625
>>> q=145785)
>>> Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
>>> Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/
>>> u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian
>>> 5.10.228-1~test1
>>> Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/
>>> S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>> Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod
>>> rpc_async_schedule [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: Call Trace:
>>> Feb 02 10:55:50 nfsserver kernel: <IRQ>
>>> Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
>>> Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>> Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>> Feb 02 10:55:50 nfsserver kernel:
>>> nmi_trigger_cpumask_backtrace+0xdb/0xf0
>>> Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>> Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>> Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
>>> Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
>>> Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>> Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>> Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>> Feb 02 10:55:50 nfsserver kernel:
>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>> Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>> Feb 02 10:55:50 nfsserver kernel: </IRQ>
>>> Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>> Feb 02 10:55:50 nfsserver kernel:
>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>> Feb 02 10:55:50 nfsserver kernel: RIP:
>>> 0010:mod_delayed_work_on+0x5d/0x90
>>> Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89
>>> c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9
>>> fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c
>>> 24 08 65 48 2b 0c 25 28 00
>>> Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS:
>>> 00000246
>>> Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX:
>>> 0000000000000000 RCX: 000000003e60000e
>>> Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI:
>>> 0000000000000046 RDI: 0000000000000246
>>> Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08:
>>> ffffffffc08f6430 R09: ffffffffc08f6448
>>> Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11:
>>> 0000000000000003 R12: ffffffffc08f6428
>>> Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14:
>>> 00000000000001f4 R15: 0000000000000000
>>> Feb 02 10:55:50 nfsserver kernel:
>>> __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: ?
>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
>>> Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
>>> Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
>>> Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
>>> Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>
>>> Before rebooting the system, rpcdebug -m rpc -c was issued again,
>>> with the
>>> following logged entry:
>>>
>>> Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp-
>>> -timeout ---ops--
>>> Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038
>>> 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
>> This is also different: the CB_RECALL_ANY is waiting to start, it's not
>> retransmitting.
>
> When CB_RECALL_ANY is returned with cb_seq_status == 1, it is restarted
> by nfsd4_cb_sequence_done. Restarting means the callback is re-queued in
> nfsd4_cb_release which schedules a new work to re-send the callback. So
> the 'call_start' status could indicate that the CB_RECALL_ANY is being
> resending in a loop.
True, but I was looking at the "q:none". If CB_RECALL_ANY were
retransmitting due to NFS4ERR_DELAY, which I've seen in past
rpc_show_tasks output, that would be "q:delayq".
>>> The system is now again back booted with fs.leases-enable=0 to keep
>>> it more
>>> "stable".
>> Understood, but I don't yet see how this new scenario is related to
>> NFSv4 delegation. We can speculate, but here's nothing standing out in
>> the collected data.
>>
>>
--
Chuck Lever
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-02-02 16:51 ` Jeff Layton
@ 2025-05-26 16:31 ` Paul Menzel
2025-05-30 13:43 ` Chuck Lever
0 siblings, 1 reply; 16+ messages in thread
From: Paul Menzel @ 2025-05-26 16:31 UTC (permalink / raw)
To: Jeff Layton, Chuck Lever, Salvatore Bonaccorso; +Cc: linux-nfs, it+linux-nfs
Dear Linux folks,
Sorry for being unresponsive for so long.
Am 02.02.25 um 17:51 schrieb Jeff Layton:
> On Sun, 2025-02-02 at 11:18 -0500, Chuck Lever wrote:
>> On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
>>> On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
>>>> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>>>>>
>>>>>> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>>>
>>>>>>
>>>>>> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>>>>>>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>>>>>>
>>>>>>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>>>>>>
>>>>>>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso <carnil@debian.org> wrote:
>>>>>>>
>>>>>>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>>>>>
>>>>>>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS 2.11.2
>>>>>>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>>>>>>
>>>>>>>>>>> ```
>>>>>>>>>>> $ dmesg -T
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Linux version 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC) 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5 12:24:15 CEST 2024
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0 init=/bin/systemd audit=0 random.trust_cpu=on systemd.unified_cgroup_hierarchy
>>>>>>>>>>> […]
>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>>>>>>> […]
>>>>>>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa xid 6890a3d2
>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570 xid 3ca4017a
>>>>>>>
>>>>>>> […]
>>>>>>>
>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f xid b682b676
>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38 xid b5d5dbf5
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928 fqs=4433
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993 q=5715)
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00 EFLAGS: 00000246
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX: 000000003f3c079e RCX: 000000000000100d
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI: 0000000000000046 RDI: ffffffff82435600
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11: 0000000000000283 R12: 0000000000000000
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492 fqs=5159
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001 q=2008)
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07 a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30 EFLAGS: 00000246
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX: ffff88997131a500 RCX: 0000000000000001
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI: ffff88997131a500 RDI: ffffffffa012c700
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08: ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11: 0000000000000283 R12: ffff88997131a530
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14: ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>>> […]
>>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> FWIW, on one NFS server occurence we are seeing something very close
>>>>>>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>>>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which had the
>>>>>>>>>> big NFS related stack backported.
>>>>>>>>>>
>>>>>>>>>> One backtrace we were able to catch was
>>>>>>>>>>
>>>>>>>>>> [...]
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec xid b172e1c6
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a xid a90d7751
>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521 xid 8e5e58bd
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319 xid c2da3c73
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21 xid a01bfec6
>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca xid c2eeeaa6
>>>>>>>>>> [...]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250 ticks this GP) idle=74e/1/0x4000000000000000 softirq=3160997/3161006 fqs=2233
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies g=8381377 q=106333)
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm: kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? trigger_load_balance+0x5a/0x220
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: update_process_times+0x8c/0xc0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90 EFLAGS: 00000246
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003820000f
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI: 0000000000000046 RDI: 0000000000000246
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc0884430 R09: ffffffffc0884448
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc0884428
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14: 00000000000001f4 R15: 0000000000000000
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>>>>>> [...]
>>>>>>>>>>
>>>>>>>>>> Is there anything which could help debug this issue?
>>>>>>>>>
>>>>>>>>> The backtrace suggests an issue in the RPC client code; the
>>>>>>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>>>>>>
>>>>>>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>>>>>>> apart, a bisect should be quick and narrow down the issue to
>>>>>>>>> one or two commits.
>>>>>>>>
>>>>>>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>>>>>>> more syntentic way on test environment, and the affected server in
>>>>>>>> particular is a production system.
>>>>>>>>
>>>>>>>> Paul, is your case in some way reproducible in a testing environment
>>>>>>>> so that a bisection might be give enough hints on the problem?
>>>>>>> We hit this issue once more on the same server with Linux 5.15.160, and had
>>>>>>> to hard reboot it.
>>>>>>>
>>>>>>> Unfortunately we did not have time yet to set up a test system to find a
>>>>>>> reproducer. In our cases a lot of compute servers seem to have accessed the
>>>>>>> NFS server. A lot of the many processes were `zstd` on a first glance.
>>>>>>
>>>>>> So we neither, due to the nature of the server (production system) and
>>>>>> unability to reproduce the issue under some more controlled way and on
>>>>>> test environment.
>>>>>>
>>>>>> In our case users seems to cause workloads involving use of wandb.
>>>>>>
>>>>>> What we tried is to boot the recent kernel from 5.10.y series avaiable
>>>>>> (5.10.223-1). Then the issue showed up still. Since we disabled
>>>>>> fs.leases-enable the situation seems to be more stable). While this
>>>>>> is/might not be the solution, does that gives some additional hits?
>>>>>
>>>>> The problem is backchannel-related, and disabling delegation
>>>>> will reduce the number of backchannel operations. Your finding
>>>>> comports with our current theory, but I can't think of how it
>>>>> narrows the field of suspects.
>>>>>
>>>>> Is the server running short on memory, perhaps? One backchannel
>>>>> operation that was added in v5.10.220 is CB_RECALL_ANY, which
>>>>> is triggered on memory exhaustion. But that should be a fairly
>>>>> harmless addition unless there is a bug in there somewhere.
>>>>>
>>>>> If your NFS server does not have any NFS mounts, then we could
>>>>> provide instructions for enabling client-side tracing to watch
>>>>> the details of callback traffic.
>>>>
>>>> The NFS server acts as well as NFS client, so tracing more
>>>> back-channel related will I guess just load the tracing more.
>>>>
>>>> But we got "lucky" and we were able to trigger the issue twice in last
>>>> days, once NFSv4 delegations were enabled again and some users started
>>>> to cause more load on the specific server as well.
>>>>
>>>> I did issue
>>>>
>>>> rpcdebug -m rpc -c
>>>>
>>>> before rebooting/resetting the server which is
>>>>
>>>> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task [sunrpc] q:delayq
>>>>
>>>> and the first RPC related soft lookup slapt in the log/journal I was
>>>> able to gather is:
>>>>
>>>> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u42:3:705574]
>>>> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2 quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac skx_edac_common nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1 raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci i40e libata
>>>> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
>>>> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm: kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
>>>> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
>>>> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d 41 5c c3 cc cc cc cc f3 90 eb 84
>>>> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00 EFLAGS: 00000202
>>>> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX: 000005fce82a4372 RCX: 0000000000000018
>>>> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI: 000000000031f238 RDI: 00385c1353c92824
>>>> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08: ffffffffc081f410 R09: ffffffffc081f410
>>>> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: 000000000031f238
>>>> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14: 0000000000000001 R15: 0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000) GS:ffff8ee94f880000(0000) knlGS:0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>>>> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3: 00000017c420a002 CR4: 00000000007706e0
>>>> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
>>>> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
>>>> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
>>>> Jan 29 22:34:05 nfsserver kernel: Call Trace:
>>>> Jan 29 22:34:05 nfsserver kernel: <IRQ>
>>>> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
>>>> Jan 29 22:34:05 nfsserver kernel: ? lockup_detector_update_enable+0x50/0x50
>>>> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
>>>> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
>>>> Jan 29 22:34:05 nfsserver kernel: ? __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
>>>> Jan 29 22:34:05 nfsserver kernel: </IRQ>
>>>> Jan 29 22:34:05 nfsserver kernel: ? sysvec_apic_timer_interrupt+0x72/0x80
>>>> Jan 29 22:34:05 nfsserver kernel: ? asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
>>>> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>>> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
>>>> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
>>>> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
>>>> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
>>>> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>
>>>> I can try to pick on top of the kernel the change Chuck mentioned to
>>>> me offlist, which is the posting of
>>>> https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-dai.ngo@oracle.com/,
>>>> and in fact this could be interesting. If the users keep doing the
>>>> same kind of load, this might help understanding more the issue.
>>>>
>>>> As we suspect that the issue is more frequently triggered after the
>>>> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
>>>> says it fixes 66af25799940 ("NFSD: add courteous server support for
>>>> thread with only delegation"), which is in 5.19-rc1, but got
>>>> backported to 5.15.154 and 5.10.220 as well.
>>>
>>> Unfortunately not. The system ran slightly more stable with that patch on, and
>>> there was a nfsd hang inbeween here, within a series of
>>>
>>> [...]
>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
>>> Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
>>> Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
>>> Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
>>> Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
>>> Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
>>> Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
>>> Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
>>> Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
>>> Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
>>> Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
>>> Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
>>> Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
>>> Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
>>> Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
>>> Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
>>> Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
>>> [...]
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for more than 120 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1590 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for more than 120 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1599 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:46 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for more than 121 seconds.
>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1601 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for more than 121 seconds.
>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1604 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for more than 121 seconds.
>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D stack: 0 pid: 1610 ppid: 2 flags:0x00004000
>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>> Feb 02 05:34:47 nfsserver kernel: rwsem_down_write_slowpath+0x257/0x4d0
>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ext4_buffered_write_iter+0x33/0x160 [ext4]
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>
>> This is a totally different failure mode: it's hanging in the
>> ext4 write path. One of your nfsd threads is stuck in D state
>> waiting to get a rw semaphor.
>>
>> Question is, who is holding that rw_sem and why?
>
> It looks like ext4_buffered_write_iter() takes the inode_lock, so it's
> probably the inode->i_rwsem that it's waiting on. Unfortunately all
> sorts of things take that lock so it's hard to speculate about the
> cause of it being stuck. Consider triggering a sysrq-w if this occurs
> again, which would tell us something about the contended locks.
>
>
>>> This happend a couple of times again and "recovered", but got finally stuck
>>> again with:
>>>
>>> Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
>>> Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
>>> Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
>>> Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
>>> Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected stall on CPU
>>> Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
>>> Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625 q=145785)
>>> Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
>>> Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>> Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>> Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod rpc_async_schedule [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: Call Trace:
>>> Feb 02 10:55:50 nfsserver kernel: <IRQ>
>>> Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
>>> Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>> Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>> Feb 02 10:55:50 nfsserver kernel: nmi_trigger_cpumask_backtrace+0xdb/0xf0
>>> Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>> Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>> Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
>>> Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
>>> Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>> Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>> Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>> Feb 02 10:55:50 nfsserver kernel: __sysvec_apic_timer_interrupt+0x5c/0xe0
>>> Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>> Feb 02 10:55:50 nfsserver kernel: </IRQ>
>>> Feb 02 10:55:50 nfsserver kernel: sysvec_apic_timer_interrupt+0x72/0x80
>>> Feb 02 10:55:50 nfsserver kernel: asm_sysvec_apic_timer_interrupt+0x12/0x20
>>> Feb 02 10:55:50 nfsserver kernel: RIP: 0010:mod_delayed_work_on+0x5d/0x90
>
> mod_delayed_work_on() disables IRQs and then calls down into the
> workqueue code to modify a wq job. If that took too long then you'd see
> an rcu_sched warning like this.
>
>>> Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b 4c 24 08 65 48 2b 0c 25 28 00
>>> Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS: 00000246
>>> Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000003e60000e
>>> Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI: 0000000000000046 RDI: 0000000000000246
>>> Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08: ffffffffc08f6430 R09: ffffffffc08f6448
>>> Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11: 0000000000000003 R12: ffffffffc08f6428
>>> Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14: 00000000000001f4 R15: 0000000000000000
>>> Feb 02 10:55:50 nfsserver kernel: __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: ? __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40 [sunrpc]
>>> Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
>>> Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
>>> Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
>>> Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
>>> Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>> Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>
>>> Before rebooting the system, rpcdebug -m rpc -c was issued again, with the
>>> following logged entry:
>>>
>>> Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --rqstp- -timeout ---ops--
>>> Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc] q:none
>>
>> This is also different: the CB_RECALL_ANY is waiting to start, it's not
>> retransmitting.
>>
>>> The system is now again back booted with fs.leases-enable=0 to keep it more
>>> "stable".
>>
>> Understood, but I don't yet see how this new scenario is related to
>> NFSv4 delegation. We can speculate, but here's nothing standing out in
>> the collected data.
>
> Agreed. It looks like there are bigger issues than just nfsd here.
We were not brave enough to test any recent Linux kernels on our file
servers, and stayed with the unaffected 5.15.131.
Were you able to pinpoint the issue? I understand, there are patches
available. Savatore writes something about `fs.leases-enable=0`. We
could give 6.12.29 another try, but would like to integrate possible
patches beforehand, so any hints are appreciated.
If the problem still exists, we’d be willing to get quotes for contract
work to fix this.
Kind regards,
Paul
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod`
2025-05-26 16:31 ` Paul Menzel
@ 2025-05-30 13:43 ` Chuck Lever
0 siblings, 0 replies; 16+ messages in thread
From: Chuck Lever @ 2025-05-30 13:43 UTC (permalink / raw)
To: Paul Menzel; +Cc: linux-nfs, it+linux-nfs, Jeff Layton, Salvatore Bonaccorso
On 5/26/25 12:31 PM, Paul Menzel wrote:
> Dear Linux folks,
>
>
> Sorry for being unresponsive for so long.
>
>
> Am 02.02.25 um 17:51 schrieb Jeff Layton:
>> On Sun, 2025-02-02 at 11:18 -0500, Chuck Lever wrote:
>>> On 2/2/25 8:35 AM, Salvatore Bonaccorso wrote:
>
>>>> On Fri, Jan 31, 2025 at 05:17:08PM +0100, Salvatore Bonaccorso wrote:
>
>>>>> On Sat, Aug 17, 2024 at 02:52:38PM +0000, Chuck Lever III wrote:
>>>>>>
>>>>>>> On Aug 17, 2024, at 4:39 AM, Salvatore Bonaccorso
>>>>>>> <carnil@debian.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Jul 30, 2024 at 02:52:47PM +0200, Paul Menzel wrote:
>
>>>>>>>> Am 30.07.24 um 14:19 schrieb Salvatore Bonaccorso:
>>>>>>>>
>>>>>>>>> On Sat, Jul 27, 2024 at 09:19:24PM +0000, Chuck Lever III wrote:
>>>>>>>>>>
>>>>>>>>>>> On Jul 27, 2024, at 5:15 PM, Salvatore Bonaccorso
>>>>>>>>>>> <carnil@debian.org> wrote:
>>>>>>>>
>>>>>>>>>>> On Wed, Jul 17, 2024 at 07:33:24AM +0200, Paul Menzel wrote:
>>>>>>>>
>>>>>>>>>>>> Using Linux 5.15.160 on a Dell PowerEdge T440/021KCD, BIOS
>>>>>>>>>>>> 2.11.2
>>>>>>>>>>>> 04/22/2021, a mount from another server hung. Linux logs:
>>>>>>>>>>>>
>>>>>>>>>>>> ```
>>>>>>>>>>>> $ dmesg -T
>>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Linux version
>>>>>>>>>>>> 5.15.160.mx64.476(root@theinternet.molgen.mpg.de) (gcc (GCC)
>>>>>>>>>>>> 12.3.0, GNU ld (GNU Binutils) 2.41) #1 SMP Wed Jun 5
>>>>>>>>>>>> 12:24:15 CEST 2024
>>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] Command line: root=LABEL=root ro
>>>>>>>>>>>> crashkernel=64G-:256M console=ttyS0,115200n8 console=tty0
>>>>>>>>>>>> init=/bin/systemd audit=0 random.trust_cpu=on
>>>>>>>>>>>> systemd.unified_cgroup_hierarchy
>>>>>>>>>>>> […]
>>>>>>>>>>>> [Wed Jul 3 16:39:34 2024] DMI: Dell Inc. PowerEdge
>>>>>>>>>>>> T440/021KCD, BIOS 2.11.2 04/22/2021
>>>>>>>>>>>> […]
>>>>>>>>>>>> [Tue Jul 16 06:00:10 2024] md: md3: data-check interrupted.
>>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got
>>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ee580afa
>>>>>>>>>>>> xid 6890a3d2
>>>>>>>>>>>> [Tue Jul 16 11:06:01 2024] receive_cb_reply: Got
>>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d4d84570
>>>>>>>>>>>> xid 3ca4017a
>>>>>>>>
>>>>>>>> […]
>>>>>>>>
>>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got
>>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000028481e8f
>>>>>>>>>>>> xid b682b676
>>>>>>>>>>>> [Tue Jul 16 11:35:59 2024] receive_cb_reply: Got
>>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c384ff38
>>>>>>>>>>>> xid b5d5dbf5
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: INFO: rcu_sched self-
>>>>>>>>>>>> detected stall on CPU
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu: 13-....: (20997 ticks this
>>>>>>>>>>>> GP) idle=54f/1/0x4000000000000000 softirq=31904928/31904928
>>>>>>>>>>>> fqs=4433
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] (t=21017 jiffies g=194958993
>>>>>>>>>>>> q=5715)
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Task dump for CPU 13:
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] task:kworker/u34:2 state:R
>>>>>>>>>>>> running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Workqueue: rpciod
>>>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Call Trace:
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <IRQ>
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>>> sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </IRQ>
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] <TASK>
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024]
>>>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RIP: 0010:read_tsc+0x3/0x20
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] Code: cc cc cc cc cc cc cc 8b 05
>>>>>>>>>>>> 56 ab 72 01 c3 cc cc cc cc 0f 1f 44 00 00 c3 cc cc cc cc 66
>>>>>>>>>>>> 66 2e 0f 1f 84 00 00 00 00 00 0f 01 f9 <66> 90 48 c1 e2 20
>>>>>>>>>>>> 48 09 d0 c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RSP: 0018:ffffc900087cfe00
>>>>>>>>>>>> EFLAGS: 00000246
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RAX: 00000000226dc8b8 RBX:
>>>>>>>>>>>> 000000003f3c079e RCX: 000000000000100d
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RDX: 00000000004535c4 RSI:
>>>>>>>>>>>> 0000000000000046 RDI: ffffffff82435600
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] RBP: 0003ed08d3641da3 R08:
>>>>>>>>>>>> ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R10: 0000000000000003 R11:
>>>>>>>>>>>> 0000000000000283 R12: 0000000000000000
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] R13: 0000000000000001 R14:
>>>>>>>>>>>> ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ktime_get+0x38/0xa0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ?
>>>>>>>>>>>> rpc_sleep_on_priority+0x70/0x70 [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_exit_task+0x9a/0x100 [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] __rpc_execute+0x6e/0x410 [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] rpc_async_schedule+0x29/0x40
>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] kthread+0x115/0x140
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>>> [Tue Jul 16 11:36:40 2024] </TASK>
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: INFO: rcu_sched self-
>>>>>>>>>>>> detected stall on CPU
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu: 7-....: (21000 ticks this
>>>>>>>>>>>> GP) idle=5b1/1/0x4000000000000000 softirq=29984492/29984492
>>>>>>>>>>>> fqs=5159
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] (t=21017 jiffies g=194959001
>>>>>>>>>>>> q=2008)
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Task dump for CPU 7:
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] task:kworker/u34:2 state:R
>>>>>>>>>>>> running task stack: 0 pid:30413 ppid: 2 flags:0x00004008
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Workqueue: rpciod
>>>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Call Trace:
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <IRQ>
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] sched_show_task.cold+0xc2/0xda
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_dump_cpu_stacks+0xa1/0xd3
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rcu_sched_clock_irq.cold+0xc7/0x1e7
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? trigger_load_balance+0x6d/0x300
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? scheduler_tick+0xda/0x260
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] update_process_times+0xa1/0xe0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] tick_sched_timer+0x8e/0xa0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? tick_sched_do_timer+0x90/0x90
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __hrtimer_run_queues+0x139/0x2a0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] hrtimer_interrupt+0xf4/0x210
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5f/0xe0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>>> sysvec_apic_timer_interrupt+0x69/0x90
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </IRQ>
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] <TASK>
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024]
>>>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x16/0x20
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RIP: 0010:_raw_spin_lock+0x10/0x20
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] Code: b8 00 02 00 00 f0 0f c1 07
>>>>>>>>>>>> a9 ff 01 00 00 75 05 c3 cc cc cc cc e9 f0 05 59 ff 0f 1f 44
>>>>>>>>>>>> 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <75> 05 c3 cc cc cc
>>>>>>>>>>>> cc 89 c6 e9 62 02 59 ff 66 90 0f 1f 44 00 00 fa
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RSP: 0018:ffffc900087cfe30
>>>>>>>>>>>> EFLAGS: 00000246
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RAX: 0000000000000000 RBX:
>>>>>>>>>>>> ffff88997131a500 RCX: 0000000000000001
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RDX: 0000000000000001 RSI:
>>>>>>>>>>>> ffff88997131a500 RDI: ffffffffa012c700
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] RBP: ffffffffa012c700 R08:
>>>>>>>>>>>> ffffffffa012c770 R09: ffffffffa012c788
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R10: 0000000000000003 R11:
>>>>>>>>>>>> 0000000000000283 R12: ffff88997131a530
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] R13: 0000000000000001 R14:
>>>>>>>>>>>> ffff88909311c000 R15: ffff88909311c005
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] __rpc_execute+0x95/0x410 [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] rpc_async_schedule+0x29/0x40
>>>>>>>>>>>> [sunrpc]
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] process_one_work+0x1d7/0x3a0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] worker_thread+0x4a/0x3c0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? process_one_work+0x3a0/0x3a0
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] kthread+0x115/0x140
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ? set_kthread_struct+0x50/0x50
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] ret_from_fork+0x1f/0x30
>>>>>>>>>>>> [Tue Jul 16 11:37:19 2024] </TASK>
>>>>>>>>>>>> [Tue Jul 16 11:37:57 2024] rcu: INFO: rcu_sched self-
>>>>>>>>>>>> detected stall on CPU
>>>>>>>>>>>> […]
>>>>>>>>>>>> ```
>>>>>>>>>>>
>>>>>>>>>>> FWIW, on one NFS server occurence we are seeing something
>>>>>>>>>>> very close
>>>>>>>>>>> to the above but in the 5.10.y case for the Debian kernel after
>>>>>>>>>>> updating to 5.10.218-1 to 5.10.221-1, so kernel after which
>>>>>>>>>>> had the
>>>>>>>>>>> big NFS related stack backported.
>>>>>>>>>>>
>>>>>>>>>>> One backtrace we were able to catch was
>>>>>>>>>>>
>>>>>>>>>>> [...]
>>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000003d26f7ec
>>>>>>>>>>> xid b172e1c6
>>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000017f1552a
>>>>>>>>>>> xid a90d7751
>>>>>>>>>>> Jul 27 15:24:52 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006337c521
>>>>>>>>>>> xid 8e5e58bd
>>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000cbf89319
>>>>>>>>>>> xid c2da3c73
>>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e2588a21
>>>>>>>>>>> xid a01bfec6
>>>>>>>>>>> Jul 27 15:24:54 nfsserver kernel: receive_cb_reply: Got
>>>>>>>>>>> unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005fda63ca
>>>>>>>>>>> xid c2eeeaa6
>>>>>>>>>>> [...]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: INFO: rcu_sched self-
>>>>>>>>>>> detected stall on CPU
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu: 15-....: (5250
>>>>>>>>>>> ticks this GP) idle=74e/1/0x4000000000000000
>>>>>>>>>>> softirq=3160997/3161006 fqs=2233
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: (t=5255 jiffies
>>>>>>>>>>> g=8381377 q=106333)
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: NMI backtrace for cpu 15
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: CPU: 15 PID: 3725556 Comm:
>>>>>>>>>>> kworker/u42:4 Not tainted 5.10.0-31-amd64 #1 Debian 5.10.221-1
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Hardware name: DALCO AG
>>>>>>>>>>> S2600WFT/S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559
>>>>>>>>>>> 03/19/2019
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Workqueue: rpciod
>>>>>>>>>>> rpc_async_schedule [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Call Trace:
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: <IRQ>
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: dump_stack+0x6b/0x83
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> nmi_cpu_backtrace.cold+0x32/0x69
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> lapic_can_unplug_cpu+0x80/0x80
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> nmi_trigger_cpumask_backtrace+0xdf/0xf0
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> rcu_sched_clock_irq.cold+0x202/0x3d9
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> trigger_load_balance+0x5a/0x220
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> update_process_times+0x8c/0xc0
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> tick_sched_do_timer+0x90/0x90
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> __hrtimer_run_queues+0x127/0x280
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> asm_call_irq_on_stack+0xf/0x20
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: </IRQ>
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> sysvec_apic_timer_interrupt+0x72/0x80
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RIP:
>>>>>>>>>>> 0010:mod_delayed_work_on+0x5d/0x90
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe
>>>>>>>>>>> ff ff 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89
>>>>>>>>>>> e2 4c 89 ee e8 f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 >
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RSP: 0018:ffffb5efe356fd90
>>>>>>>>>>> EFLAGS: 00000246
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RAX: 0000000000000000 RBX:
>>>>>>>>>>> 0000000000000000 RCX: 000000003820000f
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RDX: 0000000038000000 RSI:
>>>>>>>>>>> 0000000000000046 RDI: 0000000000000246
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: RBP: 0000000000002000 R08:
>>>>>>>>>>> ffffffffc0884430 R09: ffffffffc0884448
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R10: 0000000000000003 R11:
>>>>>>>>>>> 0000000000000003 R12: ffffffffc0884428
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: R13: ffff8c89d0f6b800 R14:
>>>>>>>>>>> 00000000000001f4 R15: 0000000000000000
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: rpc_exit_task+0x5a/0x140
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: __rpc_execute+0x6d/0x410
>>>>>>>>>>> [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel:
>>>>>>>>>>> rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> process_one_work+0x350/0x350
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: kthread+0x118/0x140
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ?
>>>>>>>>>>> __kthread_bind_mask+0x60/0x60
>>>>>>>>>>> Jul 27 15:25:15 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>>>>>>> [...]
>>>>>>>>>>>
>>>>>>>>>>> Is there anything which could help debug this issue?
>>>>>>>>>>
>>>>>>>>>> The backtrace suggests an issue in the RPC client code; the
>>>>>>>>>> server's NFSv4.1 backchannel would use that to send callbacks.
>>>>>>>>>>
>>>>>>>>>> Since 5.10.218 and 5.10.221 are only about a thousand commits
>>>>>>>>>> apart, a bisect should be quick and narrow down the issue to
>>>>>>>>>> one or two commits.
>>>>>>>>>
>>>>>>>>> Yes indeed. Unfortunately was yet unable to reproduce the issue in
>>>>>>>>> more syntentic way on test environment, and the affected server in
>>>>>>>>> particular is a production system.
>>>>>>>>>
>>>>>>>>> Paul, is your case in some way reproducible in a testing
>>>>>>>>> environment
>>>>>>>>> so that a bisection might be give enough hints on the problem?
>>>>>>>> We hit this issue once more on the same server with Linux
>>>>>>>> 5.15.160, and had
>>>>>>>> to hard reboot it.
>>>>>>>>
>>>>>>>> Unfortunately we did not have time yet to set up a test system
>>>>>>>> to find a
>>>>>>>> reproducer. In our cases a lot of compute servers seem to have
>>>>>>>> accessed the
>>>>>>>> NFS server. A lot of the many processes were `zstd` on a first
>>>>>>>> glance.
>>>>>>>
>>>>>>> So we neither, due to the nature of the server (production
>>>>>>> system) and
>>>>>>> unability to reproduce the issue under some more controlled way
>>>>>>> and on
>>>>>>> test environment.
>>>>>>>
>>>>>>> In our case users seems to cause workloads involving use of wandb.
>>>>>>>
>>>>>>> What we tried is to boot the recent kernel from 5.10.y series
>>>>>>> avaiable
>>>>>>> (5.10.223-1). Then the issue showed up still. Since we disabled
>>>>>>> fs.leases-enable the situation seems to be more stable). While this
>>>>>>> is/might not be the solution, does that gives some additional hits?
>>>>>>
>>>>>> The problem is backchannel-related, and disabling delegation
>>>>>> will reduce the number of backchannel operations. Your finding
>>>>>> comports with our current theory, but I can't think of how it
>>>>>> narrows the field of suspects.
>>>>>>
>>>>>> Is the server running short on memory, perhaps? One backchannel
>>>>>> operation that was added in v5.10.220 is CB_RECALL_ANY, which
>>>>>> is triggered on memory exhaustion. But that should be a fairly
>>>>>> harmless addition unless there is a bug in there somewhere.
>>>>>>
>>>>>> If your NFS server does not have any NFS mounts, then we could
>>>>>> provide instructions for enabling client-side tracing to watch
>>>>>> the details of callback traffic.
>>>>>
>>>>> The NFS server acts as well as NFS client, so tracing more
>>>>> back-channel related will I guess just load the tracing more.
>>>>>
>>>>> But we got "lucky" and we were able to trigger the issue twice in last
>>>>> days, once NFSv4 delegations were enabled again and some users started
>>>>> to cause more load on the specific server as well.
>>>>>
>>>>> I did issue
>>>>>
>>>>> rpcdebug -m rpc -c
>>>>>
>>>>> before rebooting/resetting the server which is
>>>>>
>>>>> Jan 30 05:27:05 nfsserver kernel: 26407 2281 -512 3d1fdb92
>>>>> 0 0 79bc1aa5 nfs4_cbv1 CB_RECALL_ANY a:rpc_exit_task
>>>>> [sunrpc] q:delayq
>>>>>
>>>>> and the first RPC related soft lookup slapt in the log/journal I was
>>>>> able to gather is:
>>>>>
>>>>> Jan 29 22:34:05 nfsserver kernel: watchdog: BUG: soft lockup -
>>>>> CPU#11 stuck for 23s! [kworker/u42:3:705574]
>>>>> Jan 29 22:34:05 nfsserver kernel: Modules linked in: binfmt_misc
>>>>> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bonding quota_v2
>>>>> quota_tree ipmi_ssif intel_rapl_msr intel_rapl_common skx_edac
>>>>> skx_edac_common nfit libnvdimm x86_pkg_temp_thermal
>>>>> intel_powerclamp coretemp kvm_intel kvm irqbypass
>>>>> ghash_clmulni_intel aesni_intel libaes crypto_simd cryptd ast
>>>>> glue_helper drm_vram_helper drm_ttm_helper rapl acpi_ipmi ttm
>>>>> iTCO_wdt intel_cstate ipmi_si intel_pmc_bxt drm_kms_helper mei_me
>>>>> iTCO_vendor_support ipmi_devintf cec ioatdma intel_uncore pcspkr
>>>>> evdev joydev sg i2c_algo_bit watchdog mei dca ipmi_msghandler
>>>>> acpi_power_meter acpi_pad button fuse drm configfs nfsd auth_rpcgss
>>>>> nfs_acl lockd grace sunrpc ip_tables x_tables autofs4 ext4 crc16
>>>>> mbcache jbd2 raid10 raid456 async_raid6_recov async_memcpy async_pq
>>>>> async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid1
>>>>> raid0 multipath linear md_mod dm_mod hid_generic usbhid hid sd_mod
>>>>> t10_pi crc_t10dif crct10dif_generic xhci_pci ahci xhci_hcd libahci
>>>>> i40e libata
>>>>> Jan 29 22:34:05 nfsserver kernel: crct10dif_pclmul arcmsr
>>>>> crct10dif_common ptp crc32_pclmul usbcore crc32c_intel scsi_mod
>>>>> pps_core i2c_i801 lpc_ich i2c_smbus wmi usb_common
>>>>> Jan 29 22:34:05 nfsserver kernel: CPU: 11 PID: 705574 Comm:
>>>>> kworker/u42:3 Not tainted 5.10.0-33-amd64 #1 Debian 5.10.226-1
>>>>> Jan 29 22:34:05 nfsserver kernel: Hardware name: DALCO AG S2600WFT/
>>>>> S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>>> Jan 29 22:34:05 nfsserver kernel: Workqueue: rpciod
>>>>> rpc_async_schedule [sunrpc]
>>>>> Jan 29 22:34:05 nfsserver kernel: RIP: 0010:ktime_get+0x7b/0xa0
>>>>> Jan 29 22:34:05 nfsserver kernel: Code: d1 e9 48 f7 d1 48 89 c2 48
>>>>> 85 c1 8b 05 ae 2c a5 02 8b 0d ac 2c a5 02 48 0f 45 d5 8b 35 7e 2c
>>>>> a5 02 41 39 f4 75 9e 48 0f af c2 <48> 01 f8 48 d3 e8 48 01 d8 5b 5d
>>>>> 41 5c c3 cc cc cc cc f3 90 eb 84
>>>>> Jan 29 22:34:05 nfsserver kernel: RSP: 0018:ffffa1aca9227e00
>>>>> EFLAGS: 00000202
>>>>> Jan 29 22:34:05 nfsserver kernel: RAX: 0000371a545e1910 RBX:
>>>>> 000005fce82a4372 RCX: 0000000000000018
>>>>> Jan 29 22:34:05 nfsserver kernel: RDX: 000000000078efbe RSI:
>>>>> 000000000031f238 RDI: 00385c1353c92824
>>>>> Jan 29 22:34:05 nfsserver kernel: RBP: 0000000000000000 R08:
>>>>> ffffffffc081f410 R09: ffffffffc081f410
>>>>> Jan 29 22:34:05 nfsserver kernel: R10: 0000000000000003 R11:
>>>>> 0000000000000003 R12: 000000000031f238
>>>>> Jan 29 22:34:05 nfsserver kernel: R13: ffff8ed42bf34830 R14:
>>>>> 0000000000000001 R15: 0000000000000000
>>>>> Jan 29 22:34:05 nfsserver kernel: FS: 0000000000000000(0000)
>>>>> GS:ffff8ee94f880000(0000) knlGS:0000000000000000
>>>>> Jan 29 22:34:05 nfsserver kernel: CS: 0010 DS: 0000 ES: 0000 CR0:
>>>>> 0000000080050033
>>>>> Jan 29 22:34:05 nfsserver kernel: CR2: 00007ffddf306080 CR3:
>>>>> 00000017c420a002 CR4: 00000000007706e0
>>>>> Jan 29 22:34:05 nfsserver kernel: DR0: 0000000000000000 DR1:
>>>>> 0000000000000000 DR2: 0000000000000000
>>>>> Jan 29 22:34:05 nfsserver kernel: DR3: 0000000000000000 DR6:
>>>>> 00000000fffe0ff0 DR7: 0000000000000400
>>>>> Jan 29 22:34:05 nfsserver kernel: PKRU: 55555554
>>>>> Jan 29 22:34:05 nfsserver kernel: Call Trace:
>>>>> Jan 29 22:34:05 nfsserver kernel: <IRQ>
>>>>> Jan 29 22:34:05 nfsserver kernel: ? watchdog_timer_fn+0x1bb/0x210
>>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>>> lockup_detector_update_enable+0x50/0x50
>>>>> Jan 29 22:34:05 nfsserver kernel: ? __hrtimer_run_queues+0x127/0x280
>>>>> Jan 29 22:34:05 nfsserver kernel: ? hrtimer_interrupt+0x110/0x2c0
>>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>>> Jan 29 22:34:05 nfsserver kernel: ? asm_call_irq_on_stack+0xf/0x20
>>>>> Jan 29 22:34:05 nfsserver kernel: </IRQ>
>>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>>> sysvec_apic_timer_interrupt+0x72/0x80
>>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>>> Jan 29 22:34:05 nfsserver kernel: ? ktime_get+0x7b/0xa0
>>>>> Jan 29 22:34:05 nfsserver kernel: rpc_exit_task+0x96/0x140 [sunrpc]
>>>>> Jan 29 22:34:05 nfsserver kernel: ?
>>>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>>> Jan 29 22:34:05 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>>> Jan 29 22:34:05 nfsserver kernel: rpc_async_schedule+0x29/0x40
>>>>> [sunrpc]
>>>>> Jan 29 22:34:05 nfsserver kernel: process_one_work+0x1b3/0x350
>>>>> Jan 29 22:34:05 nfsserver kernel: worker_thread+0x53/0x3e0
>>>>> Jan 29 22:34:05 nfsserver kernel: ? process_one_work+0x350/0x350
>>>>> Jan 29 22:34:05 nfsserver kernel: kthread+0x118/0x140
>>>>> Jan 29 22:34:05 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>>> Jan 29 22:34:05 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>>
>>>>> I can try to pick on top of the kernel the change Chuck mentioned to
>>>>> me offlist, which is the posting of
>>>>> https://lore.kernel.org/linux-nfs/1738271066-6727-1-git-send-email-
>>>>> dai.ngo@oracle.com/,
>>>>> and in fact this could be interesting. If the users keep doing the
>>>>> same kind of load, this might help understanding more the issue.
>>>>>
>>>>> As we suspect that the issue is more frequently triggered after the
>>>>> switch of 5.10.118 -> 5.10.221, this enforces more the above, which
>>>>> says it fixes 66af25799940 ("NFSD: add courteous server support for
>>>>> thread with only delegation"), which is in 5.19-rc1, but got
>>>>> backported to 5.15.154 and 5.10.220 as well.
>>>>
>>>> Unfortunately not. The system ran slightly more stable with that
>>>> patch on, and
>>>> there was a nfsd hang inbeween here, within a series of
>>>>
>>>> [...]
>>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5d31fb84
>>>> Feb 02 03:22:40 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9ec25b24
>>>> Feb 02 03:23:09 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 9fc25b24
>>>> Feb 02 03:23:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5e31fb84
>>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a0c25b24
>>>> Feb 02 03:23:24 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 5f31fb84
>>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 756103e9
>>>> Feb 02 03:23:31 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid ef4f583e
>>>> Feb 02 03:23:33 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 1ec77a2e
>>>> Feb 02 03:23:35 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid d0b95b44
>>>> Feb 02 03:27:43 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 7d31fb84
>>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bec25b24
>>>> Feb 02 03:27:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e0be7eef
>>>> Feb 02 03:28:07 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid bfc25b24
>>>> Feb 02 03:28:09 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid e1be7eef
>>>> Feb 02 03:31:41 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid f96ccce2
>>>> Feb 02 03:31:44 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 06ba5b44
>>>> Feb 02 03:31:49 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 9531fb84
>>>> Feb 02 03:31:51 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid f7be7eef
>>>> Feb 02 03:31:52 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid 2550583e
>>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d5c25b24
>>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid ab6103e9
>>>> Feb 02 03:31:53 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 9da4f045
>>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid d8c25b24
>>>> Feb 02 03:32:32 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid fabe7eef
>>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid a1c35b24
>>>> Feb 02 04:18:12 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000009715512e xid 29a849e3
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000fe9013df xid 786203e9
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000471650a0 xid f150583e
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid c66dcce2
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000008f30d648 xid 21c87a2e
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 0000000053af79cb xid 49da29a2
>>>> Feb 02 04:18:13 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 6132fb84
>>>> Feb 02 04:49:18 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 2ebb5b44
>>>> Feb 02 04:49:21 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 226ecce2
>>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid fdc35b24
>>>> Feb 02 04:49:22 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000f83dcedd xid 1fc07eef
>>>> Feb 02 05:01:25 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 25c45b24
>>>> Feb 02 05:09:27 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000ca92d20a xid 51c45b24
>>>> [...]
>>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1590 blocked for
>>>> more than 120 seconds.
>>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>>> hung_task_timeout_secs" disables this message.
>>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>>> stack: 0 pid: 1590 ppid: 2 flags:0x00004000
>>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>>> Feb 02 05:34:46 nfsserver kernel:
>>>> rwsem_down_write_slowpath+0x257/0x4d0
>>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel:
>>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>>> [sunrpc]
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1599 blocked for
>>>> more than 120 seconds.
>>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>>> hung_task_timeout_secs" disables this message.
>>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>>> stack: 0 pid: 1599 ppid: 2 flags:0x00004000
>>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>>> Feb 02 05:34:46 nfsserver kernel:
>>>> rwsem_down_write_slowpath+0x257/0x4d0
>>>> Feb 02 05:34:46 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel:
>>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>>> Feb 02 05:34:46 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>>> Feb 02 05:34:46 nfsserver kernel: do_iter_write+0x80/0x1c0
>>>> Feb 02 05:34:46 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>>> [sunrpc]
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>>> Feb 02 05:34:46 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>>> Feb 02 05:34:46 nfsserver kernel: ? kthread+0x118/0x140
>>>> Feb 02 05:34:46 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 05:34:46 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>>> Feb 02 05:34:46 nfsserver kernel: INFO: task nfsd:1601 blocked for
>>>> more than 121 seconds.
>>>> Feb 02 05:34:46 nfsserver kernel: Tainted: G E
>>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>>> Feb 02 05:34:46 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>>> hung_task_timeout_secs" disables this message.
>>>> Feb 02 05:34:46 nfsserver kernel: task:nfsd state:D
>>>> stack: 0 pid: 1601 ppid: 2 flags:0x00004000
>>>> Feb 02 05:34:46 nfsserver kernel: Call Trace:
>>>> Feb 02 05:34:46 nfsserver kernel: __schedule+0x282/0x870
>>>> Feb 02 05:34:46 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>>> Feb 02 05:34:46 nfsserver kernel: schedule+0x46/0xb0
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> rwsem_down_write_slowpath+0x257/0x4d0
>>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>>> [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1604 blocked for
>>>> more than 121 seconds.
>>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E
>>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>>> hung_task_timeout_secs" disables this message.
>>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D
>>>> stack: 0 pid: 1604 ppid: 2 flags:0x00004000
>>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>>> Feb 02 05:34:47 nfsserver kernel: ? rwsem_spin_on_owner+0x74/0xd0
>>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> rwsem_down_write_slowpath+0x257/0x4d0
>>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>>> [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>>> Feb 02 05:34:47 nfsserver kernel: INFO: task nfsd:1610 blocked for
>>>> more than 121 seconds.
>>>> Feb 02 05:34:47 nfsserver kernel: Tainted: G E
>>>> 5.10.0-34-amd64 #1 Debian 5.10.228-1~test1
>>>> Feb 02 05:34:47 nfsserver kernel: "echo 0 > /proc/sys/kernel/
>>>> hung_task_timeout_secs" disables this message.
>>>> Feb 02 05:34:47 nfsserver kernel: task:nfsd state:D
>>>> stack: 0 pid: 1610 ppid: 2 flags:0x00004000
>>>> Feb 02 05:34:47 nfsserver kernel: Call Trace:
>>>> Feb 02 05:34:47 nfsserver kernel: __schedule+0x282/0x870
>>>> Feb 02 05:34:47 nfsserver kernel: schedule+0x46/0xb0
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> rwsem_down_write_slowpath+0x257/0x4d0
>>>> Feb 02 05:34:47 nfsserver kernel: ? trace_call_bpf+0x76/0xe0
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd4_write+0x1/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel:
>>>> ext4_buffered_write_iter+0x33/0x160 [ext4]
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_readv_writev+0x14f/0x1b0
>>>> Feb 02 05:34:47 nfsserver kernel: do_iter_write+0x80/0x1c0
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd_vfs_write+0x17f/0x680 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: nfsd4_write+0xd0/0x1a0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: elfcorehdr_read+0x40/0x40
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_dispatch+0x15b/0x250 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process_common+0x3e1/0x6e0
>>>> [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd_svc+0x390/0x390 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? svc_process+0xb7/0xf0 [sunrpc]
>>>> Feb 02 05:34:47 nfsserver kernel: ? nfsd+0x91/0xb0 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? get_order+0x20/0x20 [nfsd]
>>>> Feb 02 05:34:47 nfsserver kernel: ? kthread+0x118/0x140
>>>> Feb 02 05:34:47 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 05:34:47 nfsserver kernel: ? ret_from_fork+0x1f/0x30
>>>
>>> This is a totally different failure mode: it's hanging in the
>>> ext4 write path. One of your nfsd threads is stuck in D state
>>> waiting to get a rw semaphor.
>>>
>>> Question is, who is holding that rw_sem and why?
>>
>> It looks like ext4_buffered_write_iter() takes the inode_lock, so it's
>> probably the inode->i_rwsem that it's waiting on. Unfortunately all
>> sorts of things take that lock so it's hard to speculate about the
>> cause of it being stuck. Consider triggering a sysrq-w if this occurs
>> again, which would tell us something about the contended locks.
>>
>>
>>>> This happend a couple of times again and "recovered", but got
>>>> finally stuck
>>>> again with:
>>>>
>>>> Feb 02 10:55:25 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 00000000b31acdd9 xid 1639fb84
>>>> Feb 02 10:55:26 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004111342b xid 24acf045
>>>> Feb 02 10:55:27 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 0000000035c718f5 xid 89c15b44
>>>> Feb 02 10:55:28 nfsserver kernel: receive_cb_reply: Got unrecognized
>>>> reply: calldir 0x1 xpt_bc_xprt 000000004563c9e7 xid 8c74cce2
>>>> Feb 02 10:55:50 nfsserver kernel: rcu: INFO: rcu_sched self-detected
>>>> stall on CPU
>>>> Feb 02 10:55:50 nfsserver kernel: rcu: 14-....: (5249 ticks
>>>> this GP) idle=c4e/1/0x4000000000000000 softirq=3120573/3120573 fqs=2624
>>>> Feb 02 10:55:50 nfsserver kernel: (t=5250 jiffies g=4585625
>>>> q=145785)
>>>> Feb 02 10:55:50 nfsserver kernel: NMI backtrace for cpu 14
>>>> Feb 02 10:55:50 nfsserver kernel: CPU: 14 PID: 614435 Comm: kworker/
>>>> u42:2 Tainted: G E 5.10.0-34-amd64 #1 Debian
>>>> 5.10.228-1~test1
>>>> Feb 02 10:55:50 nfsserver kernel: Hardware name: DALCO AG S2600WFT/
>>>> S2600WFT, BIOS SE5C620.86B.02.01.0008.031920191559 03/19/2019
>>>> Feb 02 10:55:50 nfsserver kernel: Workqueue: rpciod
>>>> rpc_async_schedule [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: Call Trace:
>>>> Feb 02 10:55:50 nfsserver kernel: <IRQ>
>>>> Feb 02 10:55:50 nfsserver kernel: dump_stack+0x6b/0x83
>>>> Feb 02 10:55:50 nfsserver kernel: nmi_cpu_backtrace.cold+0x32/0x69
>>>> Feb 02 10:55:50 nfsserver kernel: ? lapic_can_unplug_cpu+0x80/0x80
>>>> Feb 02 10:55:50 nfsserver kernel:
>>>> nmi_trigger_cpumask_backtrace+0xdb/0xf0
>>>> Feb 02 10:55:50 nfsserver kernel: rcu_dump_cpu_stacks+0xa5/0xd7
>>>> Feb 02 10:55:50 nfsserver kernel: rcu_sched_clock_irq.cold+0x202/0x3d9
>>>> Feb 02 10:55:50 nfsserver kernel: ? timekeeping_advance+0x370/0x5c0
>>>> Feb 02 10:55:50 nfsserver kernel: update_process_times+0x8c/0xc0
>>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_handle+0x22/0x60
>>>> Feb 02 10:55:50 nfsserver kernel: tick_sched_timer+0x65/0x80
>>>> Feb 02 10:55:50 nfsserver kernel: ? tick_sched_do_timer+0x90/0x90
>>>> Feb 02 10:55:50 nfsserver kernel: __hrtimer_run_queues+0x127/0x280
>>>> Feb 02 10:55:50 nfsserver kernel: hrtimer_interrupt+0x110/0x2c0
>>>> Feb 02 10:55:50 nfsserver kernel:
>>>> __sysvec_apic_timer_interrupt+0x5c/0xe0
>>>> Feb 02 10:55:50 nfsserver kernel: asm_call_irq_on_stack+0xf/0x20
>>>> Feb 02 10:55:50 nfsserver kernel: </IRQ>
>>>> Feb 02 10:55:50 nfsserver kernel:
>>>> sysvec_apic_timer_interrupt+0x72/0x80
>>>> Feb 02 10:55:50 nfsserver kernel:
>>>> asm_sysvec_apic_timer_interrupt+0x12/0x20
>>>> Feb 02 10:55:50 nfsserver kernel: RIP:
>>>> 0010:mod_delayed_work_on+0x5d/0x90
>>
>> mod_delayed_work_on() disables IRQs and then calls down into the
>> workqueue code to modify a wq job. If that took too long then you'd see
>> an rcu_sched warning like this.
>>
>>>> Feb 02 10:55:50 nfsserver kernel: Code: 00 4c 89 e7 e8 34 fe ff ff
>>>> 89 c3 83 f8 f5 74 e9 85 c0 78 1b 89 ef 4c 89 f1 4c 89 e2 4c 89 ee e8
>>>> f9 fc ff ff 48 8b 3c 24 57 9d <0f> 1f 44 00 00 85 db 0f 95 c0 48 8b
>>>> 4c 24 08 65 48 2b 0c 25 28 00
>>>> Feb 02 10:55:50 nfsserver kernel: RSP: 0018:ffffaaff25d57d90 EFLAGS:
>>>> 00000246
>>>> Feb 02 10:55:50 nfsserver kernel: RAX: 0000000000000000 RBX:
>>>> 0000000000000000 RCX: 000000003e60000e
>>>> Feb 02 10:55:50 nfsserver kernel: RDX: 000000003e400000 RSI:
>>>> 0000000000000046 RDI: 0000000000000246
>>>> Feb 02 10:55:50 nfsserver kernel: RBP: 0000000000002000 R08:
>>>> ffffffffc08f6430 R09: ffffffffc08f6448
>>>> Feb 02 10:55:50 nfsserver kernel: R10: 0000000000000003 R11:
>>>> 0000000000000003 R12: ffffffffc08f6428
>>>> Feb 02 10:55:50 nfsserver kernel: R13: ffff8e4083a4b400 R14:
>>>> 00000000000001f4 R15: 0000000000000000
>>>> Feb 02 10:55:50 nfsserver kernel:
>>>> __rpc_sleep_on_priority_timeout+0x111/0x120 [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: rpc_delay+0x56/0x90 [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: rpc_exit_task+0x5a/0x140 [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: ?
>>>> __rpc_do_wake_up_task_on_wq+0x1e0/0x1e0 [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: __rpc_execute+0x6d/0x410 [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: rpc_async_schedule+0x29/0x40
>>>> [sunrpc]
>>>> Feb 02 10:55:50 nfsserver kernel: process_one_work+0x1b3/0x350
>>>> Feb 02 10:55:50 nfsserver kernel: worker_thread+0x53/0x3e0
>>>> Feb 02 10:55:50 nfsserver kernel: ? process_one_work+0x350/0x350
>>>> Feb 02 10:55:50 nfsserver kernel: kthread+0x118/0x140
>>>> Feb 02 10:55:50 nfsserver kernel: ? __kthread_bind_mask+0x60/0x60
>>>> Feb 02 10:55:50 nfsserver kernel: ret_from_fork+0x1f/0x30
>>>>
>>>> Before rebooting the system, rpcdebug -m rpc -c was issued again,
>>>> with the
>>>> following logged entry:
>>>>
>>>> Feb 02 11:01:52 nfsserver kernel: -pid- flgs status -client- --
>>>> rqstp- -timeout ---ops--
>>>> Feb 02 11:01:52 nfsserver kernel: 42135 2281 0 8ff8d038
>>>> 0 500 1a6bcc0 nfs4_cbv1 CB_RECALL_ANY a:call_start [sunrpc]
>>>> q:none
>>>
>>> This is also different: the CB_RECALL_ANY is waiting to start, it's not
>>> retransmitting.
>>>
>>>> The system is now again back booted with fs.leases-enable=0 to keep
>>>> it more
>>>> "stable".
>>>
>>> Understood, but I don't yet see how this new scenario is related to
>>> NFSv4 delegation. We can speculate, but here's nothing standing out in
>>> the collected data.
>>
>> Agreed. It looks like there are bigger issues than just nfsd here.
>
> We were not brave enough to test any recent Linux kernels on our file
> servers, and stayed with the unaffected 5.15.131.
>
> Were you able to pinpoint the issue? I understand, there are patches
> available. Savatore writes something about `fs.leases-enable=0`. We
> could give 6.12.29 another try, but would like to integrate possible
> patches beforehand, so any hints are appreciated.
>
> If the problem still exists, we’d be willing to get quotes for contract
> work to fix this.
We believe the issue was addressed by backporting 036ac2778f7b ("NFSD:
fix hang in nfsd4_shutdown_callback"). That fix is available from
v6.12.16 onward.
--
Chuck Lever
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2025-05-30 13:44 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-07-17 5:33 `rcu: INFO: rcu_sched self-detected stall on CPU` and spinning kworker `rpciod` Paul Menzel
2024-07-27 21:15 ` Salvatore Bonaccorso
2024-07-27 21:19 ` Chuck Lever III
2024-07-30 12:19 ` Salvatore Bonaccorso
2024-07-30 12:52 ` Paul Menzel
2024-08-17 8:39 ` Salvatore Bonaccorso
2024-08-17 14:52 ` Chuck Lever III
2024-10-29 21:07 ` Salvatore Bonaccorso
2025-01-31 16:17 ` Salvatore Bonaccorso
2025-02-02 13:35 ` Salvatore Bonaccorso
2025-02-02 16:18 ` Chuck Lever
2025-02-02 16:51 ` Jeff Layton
2025-05-26 16:31 ` Paul Menzel
2025-05-30 13:43 ` Chuck Lever
2025-02-03 1:06 ` Dai Ngo
2025-02-03 14:22 ` Chuck Lever
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox