Re: nfsd becomes a zombie

Chuck Lever III
2024-07-02T18:17:18Z

> On Jul 2, 2024, at 1:25 PM, Harald Dunkel <harald.dunkel@aixigo.com> wrote:
>
> Hi folks,
>
> my NAS ran into this problem again. NFS got stuck somehow, and
> the nfsd couldn't be killed :-(.
>
> dmesg:
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000749c823f xid 5bf8d3d0
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ce307050 xid 3b4fbd9f
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f7f9161e xid 0a26635c
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000007c978512 xid 384cbf0c
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000dc3c09f6 xid 53cc0e3e
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d1675728 xid 129006af
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000047159b90 xid 0c06b6e0
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000008b3b3ac xid 641bb0da
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009eb832dc xid 005fcc99
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000042dcce88 xid b3cf5de4
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b66bbd6f xid d4f06b56
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b5e5e5a3 xid c032dbba
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e123efc9 xid 99fa75d9
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ca43f6f0 xid e38d5b74
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ad683927 xid 277cde8c
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000e8e01f09 xid 641df4a4
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000006223d195 xid 3dba2d2a
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b73943aa xid a688e47f
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004cd80e49 xid 64e688ca
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ef92587f xid 70bf2e44
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000a5ff94a6 xid c0f7a668
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000fd9a0890 xid 0df7d2c7
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c42ddaac xid 800e710e
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f43275cf xid 8b05e704
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009a1d5dcf xid 3c2ba924
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000007cad732d xid e73a0429
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000008e7d297f xid 075a98e5
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ed964446 xid 8bb8e568
> [Tue Jul 2 17:20:19 2024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b14782f0 xid 4c4ae7c5
> [Tue Jul 2 17:23:28 2024] INFO: task nfsd:3037 blocked for more than 120 seconds.
> [Tue Jul 2 17:23:28 2024] Not tainted 6.1.0-21-amd64 #1 Debian 6.1.90-1
> [Tue Jul 2 17:23:28 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [Tue Jul 2 17:23:28 2024] task:nfsd state:D stack:0 pid:3037 ppid:2 flags:0x00004000
> [Tue Jul 2 17:23:28 2024] Call Trace:
> [Tue Jul 2 17:23:28 2024] <TASK>
> [Tue Jul 2 17:23:28 2024] __schedule+0x34d/0x9e0
> [Tue Jul 2 17:23:28 2024] schedule+0x5a/0xd0
> [Tue Jul 2 17:23:28 2024] schedule_timeout+0x118/0x150
> [Tue Jul 2 17:23:28 2024] wait_for_completion+0x86/0x160
> [Tue Jul 2 17:23:28 2024] __flush_workqueue+0x152/0x420
> [Tue Jul 2 17:23:28 2024] nfsd4_destroy_session+0x1b6/0x250 [nfsd]
> [Tue Jul 2 17:23:28 2024] nfsd4_proc_compound+0x355/0x660 [nfsd]
> [Tue Jul 2 17:23:28 2024] nfsd_dispatch+0x1a1/0x2b0 [nfsd]
> [Tue Jul 2 17:23:28 2024] svc_process_common+0x289/0x5e0 [sunrpc]
> [Tue Jul 2 17:23:28 2024] ? svc_recv+0x4e5/0x890 [sunrpc]
> [Tue Jul 2 17:23:28 2024] ? nfsd_svc+0x360/0x360 [nfsd]
> [Tue Jul 2 17:23:28 2024] ? nfsd_shutdown_threads+0x90/0x90 [nfsd]
> [Tue Jul 2 17:23:28 2024] svc_process+0xad/0x100 [sunrpc]
> [Tue Jul 2 17:23:28 2024] nfsd+0xd5/0x190 [nfsd]
> [Tue Jul 2 17:23:28 2024] kthread+0xda/0x100
> [Tue Jul 2 17:23:28 2024] ? kthread_complete_and_exit+0x20/0x20
> [Tue Jul 2 17:23:28 2024] ret_from_fork+0x22/0x30
> [Tue Jul 2 17:23:28 2024] </TASK>
> [Tue Jul 2 17:23:28 2024] INFO: task nfsd:3038 blocked for more than 120 seconds.
> [Tue Jul 2 17:23:28 2024] Not tainted 6.1.0-21-amd64 #1 Debian 6.1.90-1
> [Tue Jul 2 17:23:28 2024] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>
>
> /var/log/kern.log:
> 2024-06-28T10:40:40.273493+02:00 nasl006b kernel: [959982.169372] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d1675728 xid 372e06af
> 2024-06-28T10:40:40.273507+02:00 nasl006b kernel: [959982.169374] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000506887ca xid 5be3c4d4
> 2024-06-28T10:40:40.273508+02:00 nasl006b kernel: [959982.169379] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b5e5e5a3 xid e5d0daba
> 2024-06-28T10:40:40.273509+02:00 nasl006b kernel: [959982.169423] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b66bbd6f xid 69696b56
> 2024-06-28T10:40:40.273509+02:00 nasl006b kernel: [959982.169498] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000008b3b3ac xid 89b9afda
> 2024-06-28T10:40:40.273510+02:00 nasl006b kernel: [959982.169504] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000a5ff94a6 xid e595a668
> 2024-06-28T10:40:40.273512+02:00 nasl006b kernel: [959982.169529] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000f7f9161e xid 2fc4625c
> 2024-06-28T10:40:40.273513+02:00 nasl006b kernel: [959982.169659] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b73943aa xid cb26e47f
> 2024-06-28T10:40:40.273514+02:00 nasl006b kernel: [959982.169691] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000009a1d5dcf xid 61c9a824
> 2024-06-28T10:40:40.273514+02:00 nasl006b kernel: [959982.169697] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000749c823f xid 8096d3d0
> 2024-06-28T10:40:40.944609+02:00 nasl006b kernel: [959983.506736] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000190b801c xid bdd1dcd0
> 2024-06-28T10:40:40.948612+02:00 nasl006b kernel: [959983.512235] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000042dcce88 xid d76d5de4
> 2024-06-28T10:40:40.952617+02:00 nasl006b kernel: [959983.514349] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000007cad732d xid 0bd90329
> 2024-06-28T10:40:40.952623+02:00 nasl006b kernel: [959983.514564] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000004cd80e49 xid 898488ca
> 2024-06-28T10:40:40.952624+02:00 nasl006b kernel: [959983.514951] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000abccd646 xid d0d28401
> 2024-06-28T10:40:40.952624+02:00 nasl006b kernel: [959983.515009] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ef92587f xid 955d2e44
> 2024-06-28T10:40:40.952625+02:00 nasl006b kernel: [959983.515060] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000ed964446 xid b056e568
> 2024-07-02T17:20:23.113792+02:00 nasl006b kernel: [1329564.790305] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b14782f0 xid 4c4ae7c5
> 2024-07-02T17:23:32.268700+02:00 nasl006b kernel: [1329753.944957] nfsd_dispatch+0x1a1/0x2b0 [nfsd]
> 2024-07-02T17:23:32.300740+02:00 nasl006b kernel: [1329753.969482] svc_process_common+0x289/0x5e0 [sunrpc]
> 2024-07-02T17:23:32.300757+02:00 nasl006b kernel: [1329753.969919] nfsd+0xd5/0x190 [nfsd]
> 2024-07-02T17:23:32.364636+02:00 nasl006b kernel: [1329754.041100] ? nfsd_svc+0x360/0x360 [nfsd]
> 2024-07-02T17:23:32.419012+02:00 nasl006b kernel: [1329754.088290] svc_process+0xad/0x100 [sunrpc]
> 2024-07-02T17:23:32.419020+02:00 nasl006b kernel: [1329754.088337] ret_from_fork+0x22/0x30
> 2024-07-02T17:23:32.443744+02:00 nasl006b kernel: [1329754.111842] svc_process+0xad/0x100 [sunrpc]
> 2024-07-02T17:23:32.443749+02:00 nasl006b kernel: [1329754.111882] ? kthread_complete_and_exit+0x20/0x20
> 2024-07-02T17:23:32.488628+02:00 nasl006b kernel: [1329754.161331] ? kthread_complete_and_exit+0x20/0x20

Harald, none of this is any more probative than the first
report you sent. We can't tell what's going on unless you
can help us debug the problem. We're just not set up as a
help desk. Have you contacted your Linux vendor and asked
for help?

--
Chuck Lever