Re: [PATCH v11 00/20] nfs/nfsd: add support for localio

Chuck Lever III
2024-07-06T17:15:39Z

> On Jul 6, 2024, at 2:42 AM, Christoph Hellwig <hch@infradead.org> wrote:
>
> On Sat, Jul 06, 2024 at 04:37:22PM +1000, NeilBrown wrote:
>>> a different scheme for bypassing the server for I/O. Maybe there is
>>> a really good killer argument for doing that, but it needs to be clearly
>>> stated and defended instead of assumed.
>>
>> Could you provide a reference to the text book - or RFC - that describes
>> a pNFS DS protocol that completely bypasses the network, allowing the
>> client and MDS to determine if they are the same host and to potentially
>> do zero-copy IO.
>
> I did not say that we have the exact same functionality available and
> there is no work to do at all, just that it is the standard way to bypass
> the server.
>
> RFC 5662, RFC 5663 and RFC 8154 specify layouts that completely bypass
> the network and require the client and server to find out that they talk
> to the same storage devuce, and directly perform zero copy I/O.
> They do not require to be on the same host, though.
>
>> If not, I will find it hard to understand your claim that it is "the
>> text book example".
>
> pNFS is all about handing out grants to bypass the server for I/O.
> That is exactly what localio is doing.

In particular, Neil, a pNFS block/SCSI layout provides the
client with a set of device IDs. If the client is on the
same storage fabric as those devices, it can then access
those devices directly using SCSI commands rather than
going on the network [RFC8154].

This is equivalent to a loopback acceleration mechanism. If
the client and server are on the same host, then there are
natural ways to expose the devices to both peers, and the
existing pNFS protocol and SCSI Persistent Reservation
provide strong access authorization.

Both the Linux NFS client and server implement RFC 8154
well enough that this could be an alternative or even a
better solution than LOCALIO. The server stores an XFS
file system on the devices, and hands out layouts with
the device ID and LBAs of the extents where file content
is located.

The fly in this ointment is the need for NFSv3 support.

In an earlier email Mike mentioned that Hammerspace isn't
interested in providing a centrally managed directory of
block devices that could be utilized by the MDS to simply
inform the client of local devices. I don't think that's
the only possible solution for discovering the locality of
storage devices.

--
Chuck Lever