Re: Concerns about SO_REUSEPORT usage in NFS client

[NeilBrown <neilb@suse.com>]

[-- Attachment #1: Type: text/plain, Size: 2239 bytes --]


Hi Trond (or anyone),
 did you get a chance to look at this?
 It really seems that the SO_REUSEPORT solution has problems.

Thanks,
NeilBrown

On Thu, Dec 17 2015, NeilBrown wrote:

> Hi Trond et al.
>
> I have concerns about the new use of SO_REUSEPORT in the NFS client.
> Partly this is a theoretical concern.  The documentation in socket(7)
> talks about using this flag on UDP sockets and on TCP sockets in LISTEN
> mode, but not about using it with connected TCP sockets.  So the NFS
> usage isn't covered by the documentation ... maybe fixing the
> documentation would relieve that concern.
>
> But there is also a practical concern: it seems to sometime cause
> failures.
> This is reported here:
>     https://bugzilla.suse.com/show_bug.cgi?id=959216
>
> I cannot reproduce exactly the same symptoms as described there but I
> can get close. I:
>   - establish an NFSv3 mount to a server
>   - determine the port number used on the client side
>   - write numbers to /proc/sys/sunrpc/{min,max}_resvport which bracket
>     that port number in a range of 10 or so
>   - try to establish NFSv4 mounts in a loop (unmounting each time)
>
> Then the mount will sometimes hang.
> While it is hanging mount.nfs might be in permanently runnable and
> "cat /proc/`pidof mount.nfs`/stack" can show:
>
> [<ffffffff81001012>] ___preempt_schedule+0x12/0x14
> [<ffffffffffffffff>] 0xffffffffffffffff
>
>
> I've also sometime seen the stack trace mentioned in the bugzilla
>
> [<ffffffffa030b469>] xprt_connect+0x119/0x170 [sunrpc]
> [<ffffffffa0308c06>] call_connect+0x56/0xb0 [sunrpc]
> [<ffffffffa0312212>] __rpc_execute+0x82/0x450 [sunrpc]
> [<ffffffffa0314fda>] rpc_execute+0x5a/0xb0 [sunrpc]
> ....
>
> I typically see a 3 minute timeout before the mount fails with
>   mount.nfs: Connection timed out
>
> My guess is that SO_REUSEPORT can allow the NFSv4 mount to use the same
> connection that the NFSv3 mount is using, though over a different socket.
> NFSv4 sends a request, the reply is received by the NFSv3 client's socket
> which rejects it and the NFSv4 client keeps waiting.
>
> I think that we can only continue to use SO_REUSEPORT if we find a way
> to ensure that we don't re-use a currently active connection.
>
> NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]