Re: [PATCH v5 00/19] nfs/nfsd: add support for localio

[Mike Snitzer <snitzer@kernel.org>]

On Thu, Jun 20, 2024 at 07:09:23AM +1000, NeilBrown wrote:
> On Wed, 19 Jun 2024, Jeff Layton wrote:
> > On Wed, 2024-06-19 at 17:10 +1000, NeilBrown wrote:
> > > On Wed, 19 Jun 2024, Christoph Hellwig wrote:
> > > > What happened to the requirement that all protocol extensions added
> > > > to Linux need to be standardized in IETF RFCs?
> > > > 
> > > > 
> > > 
> > > Is that requirement documented somewhere?  Not that I doubt it, but it
> > > would be nice to know where it is explicit.  I couldn't quickly find
> > > anything in Documentation/
> > > 
> > > Can we get by without the LOCALIO protocol?
> > > 
> > > For NFSv4.1 we could use the server_owner4 returned by EXCHANGE_ID.  It
> > > is explicitly documented as being usable to determine if two servers are
> > > the same.
> > > 
> > > For NFSv4.0 ... I don't think we should encourage that to be used.
> > > 
> > > For NFSv3 it is harder.  I'm not as ready to deprecate it as I am for
> > > 4.0.  There is nothing in NFSv3 or MOUNT or NLM that is comparable to
> > > server_owner4.  If krb5 was used there would probably be a server
> > > identity in there that could be used.
> > > I think the server could theoretically return an AUTH_SYS verifier in
> > > each RPC reply and that could be used to identify the server.  I'm not
> > > sure that is a good idea though.
> > > 
> > 
> > My idea for v3 was that the localio client could do an O_TMPFILE create
> > on the exported fs and write some random junk to it (a uuid or
> > something). Construct the filehandle for that and then the client could
> > try to issue a READ for that filehandle via the NFS server. If it finds
> > that filehandle and the contents are correct then you're on the same
> > host. Then you just close the file and it should clean itself up.
> 
> I can't see how this would work, but maybe I don't have a good enough
> imagination.
> 
> The high-level view of the proposed protocol is:
>   - client asks remote server to identify itself.
>   - server returns an identity
>   - client uses local-sideband to ask each local server if it has the
>     given identity.
> 
> I don't see where an O_TMPFILE could fit into this, or how a different
> high-level approach would be any better.
> 
> For NFSv3 the client could ask with a new Program or Version or
> Procedure, or all three.  Or it could ask with a new file-handle or path
> name.  I imagine using a webnfs (rfc2054) multi-component lookup on the
> public filehandle for "/linux/config/server-id" and getting back a
> filehandle which encodes the server ID somehow.  All these seem credible
> options and it is not clear than any one is better than any other.
> 
> For NFSv4.1 I think that LOCALIO looks a lot like trunking and so using
> exactly the same mechanism to determine if two servers are the same is a
> good idea.
> But then LOCALIO also looks a lot like a new pNFS/DS protocol so maybe
> we should specify that protocol and use GETDEVICELIST or GETDEVICEINFO
> to find the identity of the server.

Easy enough to switch the RPC call used.  If either GETDEVICELIST or
GETDEVICEINFO can convey a UUID it sounds fine to me.  But for v4
EXCHANGE_ID already exists.

> > This is a little less straightforward and efficient than the localio
> > protocol that Mike is proposing, but requires no protocol extensions.
> 
> I think that if we use anything other than the server-id in the
> EXCHANGE_ID response, then we are defining a new protocol as it is a new
> request which we expect existing servers to ignore or fail, even though
> they have never been tested to ignore/fail that particular request.
> 
> Of all the options I would guess that a new version for an existing
> protocol would be safest as that is the most likely to have been tested.
> A new RPC program is probably conceptually simplest.  A little hack in
> LOOKUPv3 to detect the public filehandle etc is probably the easiest to
> code, and a new pnfs/ds protocol is probably the cleanest overall
> except that it doesn't support NFSv3.

NFSv3 support is pretty important. So when faced with no options for
v3, I decided to implement LOCALIO (with Trond's encouragement) and
just have both NFS versions use it.

I _can_ frame the v4 support in terms of EXCHANGE_ID (and have already
implemented it before writing LOCALIO, patches aren't on the internet
but I can unearth that work if needed).  But I'd still maintain the
nfsd_uuids list, and have nfs_localio's nfsd_uuid_is_local() lookup
the UUID that was embedded in the v4 EXCHANGE_ID payload...

But yes, we'd still need LOCALIO's GETUUID rpc for v3.  So EXCHANGE_ID
really doesn't buy much (because we'd still need an IANA registered
rpc program number).

> My purpose in all this is not to replace Mike's LOCALIO protocol, but to
> explore the solution space to ensure there is nothing that is obviously
> better.  As yet, I don't think there is.

Thanks, I really appreciate your professionalism and attention to
detail.  Pleasure working with you again Neil!

Mike