leaked pNFS DS nfs_client references

leaked pNFS DS nfs_client references

[Jeff Layton]

Hi Trond/Anna:

We (at Meta) have been hunting a number of problems surrounding leaked
network namespaces with containerized workloads. We recently deployed a
v6.9 based kernel on the clients that has all the known containerized
NFS fixes from upstream.

Usually, when we've found problems with leaked netns's it has been
because there were still outstanding RPCs associated with the rpc_clnt.
Today, we found a host that seems to have some leaked nfs_client
structures, but there is no associated RPC activity.

In this case, we had 2 leaked net namespaces. We discovered them by
looking under /sys/kernel/debugfs/rpc_xprt for xprts associated with
netns's that no longer have any userland tasks attached.

Some drgn (pardon my terrible Python):

>>> for net in for_each_net():
...     if (net.ns.inum == 4026558887 or net.ns.inum == 4026558805):                                   
...         print("netns:", net.ns.inum)                                                               
...         nfs_net = cast("struct nfs_net *", net.gen.ptr[prog["nfs_net_id"]])
...         print("Volume list empty:", list_empty(nfs_net.nfs_volume_list.address_of_()))
...         for clnt in list_for_each_entry("struct nfs_client", nfs_net.nfs_client_list.address_of_(), "cl_share_link"):
...             rpcclnt = clnt.cl_rpcclient
...             print(clnt.cl_count.refs.counter, clnt.cl_hostname, rpcclnt.cl_vers, "tasks: ", list_count_nodes(rpcclnt.cl_tasks.address_of_()))
... 
netns: (unsigned int)4026558805
Volume list empty: True
(int)1 (char *)0xffff8a12e988a500 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
(int)1 (char *)0xffff8881a0f694c0 = "f00::bfaa:cec2:8ee2:295" (u32)3 tasks:  0
(int)1 (char *)0xffff889e81a74e40 = "f00::8f23:f52d:9d79:a7b0" (u32)3 tasks:  0
(int)1 (char *)0xffff8a027d8e0780 = "f00::d209:97ba:1c6:3282" (u32)3 tasks:  0
netns: (unsigned int)4026558887
Volume list empty: True
(int)1 (char *)0xffff8a14d5b0e2c0 = "f00::3f52:fea6:4ccb:96dd" (u32)3 tasks:  0
(int)1 (char *)0xffff8881e6626cc0 = "f00::705:c924:ddc1:51e4" (u32)3 tasks:  0
(int)1 (char *)0xffff8a149cdb6680 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
(int)1 (char *)0xffff8896ada2f800 = "f00::d56c:cd93:1f0c:99c7" (u32)3 tasks:  0
(int)1 (char *)0xffff8a159251f240 = "f00::614d:87c1:a73f:1f09" (u32)3 tasks:  0
(int)1 (char *)0xffff888e699f4940 = "f00::1285:b785:f114:d38b" (u32)3 tasks:  0
(int)1 (char *)0xffff88812ae41500 = "f00::fb1c:bc4a:3d9a:c2a6" (u32)3 tasks:  0
(int)1 (char *)0xffff8a137dbc4e00 = "f00::bd2f:5851:b552:5bce" (u32)3 tasks:  0

There are 12 leaked nfs_clients in 2 netns's. There are no longer any
struct nfs_servers associated with either netns. Each leaked client has
a single outstanding reference. They're all connections to different
DS's (except for one between the two netns's, but I suspect that's just
coincidence). They're all NFSv3, which indicates that they are pNFS DS
clients. None of them have any running RPCs.

I took a look at the nfs_client refcount handling in the pNFS code but
didn't see any obvious bugs.

One thing we could consider is adding a refcount tracker for these
objects. That would tell us pretty quickly what took the leftover
references in the first place, assuming this is reproducible.

This kernel is based on v6.9, so it's possible we missed a fix that we
need. I didn't see anything obvious in recent git fixes though.

Any thoughts?
-- 
Jeff Layton <jlayton@kernel.org>

Re: leaked pNFS DS nfs_client references

[Jeff Layton]

On Mon, 2025-04-21 at 15:46 -0400, Jeff Layton wrote:
> Hi Trond/Anna:
> 
> We (at Meta) have been hunting a number of problems surrounding leaked
> network namespaces with containerized workloads. We recently deployed a
> v6.9 based kernel on the clients that has all the known containerized
> NFS fixes from upstream.
> 
> Usually, when we've found problems with leaked netns's it has been
> because there were still outstanding RPCs associated with the rpc_clnt.
> Today, we found a host that seems to have some leaked nfs_client
> structures, but there is no associated RPC activity.
> 
> In this case, we had 2 leaked net namespaces. We discovered them by
> looking under /sys/kernel/debugfs/rpc_xprt for xprts associated with
> netns's that no longer have any userland tasks attached.
> 
> Some drgn (pardon my terrible Python):
> 
> > > > for net in for_each_net():
> ...     if (net.ns.inum == 4026558887 or net.ns.inum == 4026558805):                                   
> ...         print("netns:", net.ns.inum)                                                               
> ...         nfs_net = cast("struct nfs_net *", net.gen.ptr[prog["nfs_net_id"]])
> ...         print("Volume list empty:", list_empty(nfs_net.nfs_volume_list.address_of_()))
> ...         for clnt in list_for_each_entry("struct nfs_client", nfs_net.nfs_client_list.address_of_(), "cl_share_link"):
> ...             rpcclnt = clnt.cl_rpcclient
> ...             print(clnt.cl_count.refs.counter, clnt.cl_hostname, rpcclnt.cl_vers, "tasks: ", list_count_nodes(rpcclnt.cl_tasks.address_of_()))
> ... 
> netns: (unsigned int)4026558805
> Volume list empty: True
> (int)1 (char *)0xffff8a12e988a500 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
> (int)1 (char *)0xffff8881a0f694c0 = "f00::bfaa:cec2:8ee2:295" (u32)3 tasks:  0
> (int)1 (char *)0xffff889e81a74e40 = "f00::8f23:f52d:9d79:a7b0" (u32)3 tasks:  0
> (int)1 (char *)0xffff8a027d8e0780 = "f00::d209:97ba:1c6:3282" (u32)3 tasks:  0
> netns: (unsigned int)4026558887
> Volume list empty: True
> (int)1 (char *)0xffff8a14d5b0e2c0 = "f00::3f52:fea6:4ccb:96dd" (u32)3 tasks:  0
> (int)1 (char *)0xffff8881e6626cc0 = "f00::705:c924:ddc1:51e4" (u32)3 tasks:  0
> (int)1 (char *)0xffff8a149cdb6680 = "f00::3117:a4f1:a940:94af" (u32)3 tasks:  0
> (int)1 (char *)0xffff8896ada2f800 = "f00::d56c:cd93:1f0c:99c7" (u32)3 tasks:  0
> (int)1 (char *)0xffff8a159251f240 = "f00::614d:87c1:a73f:1f09" (u32)3 tasks:  0
> (int)1 (char *)0xffff888e699f4940 = "f00::1285:b785:f114:d38b" (u32)3 tasks:  0
> (int)1 (char *)0xffff88812ae41500 = "f00::fb1c:bc4a:3d9a:c2a6" (u32)3 tasks:  0
> (int)1 (char *)0xffff8a137dbc4e00 = "f00::bd2f:5851:b552:5bce" (u32)3 tasks:  0
> 
> There are 12 leaked nfs_clients in 2 netns's. There are no longer any
> struct nfs_servers associated with either netns. Each leaked client has
> a single outstanding reference. They're all connections to different
> DS's (except for one between the two netns's, but I suspect that's just
> coincidence). They're all NFSv3, which indicates that they are pNFS DS
> clients. None of them have any running RPCs.
> 
> I took a look at the nfs_client refcount handling in the pNFS code but
> didn't see any obvious bugs.
> 
> One thing we could consider is adding a refcount tracker for these
> objects. That would tell us pretty quickly what took the leftover
> references in the first place, assuming this is reproducible.
> 
> This kernel is based on v6.9, so it's possible we missed a fix that we
> need. I didn't see anything obvious in recent git fixes though.
> 
> Any thoughts?

An update:

Omar and I worked together yesterday, and confirmed that this is the
same problem that he reported a week or two ago. This kernel has the
two patches you sent on April 6th:

    [PATCH v2 1/2] NFSv4: Handle fatal ENETDOWN and ENETUNREACH errors
    [PATCH v2 2/2] NFSv4/pnfs: Layoutreturn on close must handle fatal networking errors

...so that's evidently not enough to fix it.

Note that this is a potential memory corruptor. The leaked layout segs
were all sitting on the layouts plh_return_segs list. If they get freed
later, then that will likely scribble over the list_head in the freed
layout.

Looking over the code, it appears that when the inode is evicted, the
plh_return_segs list should get cleaned out via:

nfs4_evict_inode
    pnfs_destroy_layout_final
        __pnfs_destroy_layout
            pnfs_mark_layout_stateid_invalid
                pnfs_free_returned_lsegs

pnfs_free_returned_lsegs() should put them all on the tmp_list and then
__pnfs_destroy_layout() should free the contents of the list via
pnfs_free_lseg_list().

It's not clear to me why that didn't happen here.
-- 
Jeff Layton <jlayton@kernel.org>