[RFC PATCH 0/2] some progress on rpcrdma bug [was: Re: [PATCH v8 5/7] NFSD: issue READs using O_DIRECT even if IO is misaligned]

[Mike Snitzer <snitzer@kernel.org>]

Hi Chuck,

[just including context from thread in cover-letter for these 2 RFC patches]

On Thu, Aug 28, 2025 at 02:52:34PM -0400, Mike Snitzer wrote:
> On Thu, Aug 28, 2025 at 10:53:49AM -0400, Chuck Lever wrote:
> > On 8/28/25 4:09 AM, Mike Snitzer wrote:
> > > On Wed, Aug 27, 2025 at 09:57:39PM -0400, Chuck Lever wrote:
> > >>>
<snip>
> > >>>> - When the I/O is complete, adjust the offset in the first bvec entry
> > >>>>   forward by setting a non-zero page offset, and adjust the returned
> > >>>>   count downward to match the requested byte count from the client
> > >>>
> > >>> Tried it long ago, such bvec manipulation only works when not using
> > >>> RDMA.  When the memory is remote, twiddling a local bvec isn't going
> > >>> to ensure the correct pages have the correct data upon return to the
> > >>> client.
> > >>>
> > >>> RDMA is why the pages must be used in-place, and RDMA is also why
> > >>> the extra page needed by this patch (for use as throwaway front-pad
> > >>> for expanded misaligned DIO READ) must either be allocated _or_
> > >>> hopefully it can be from rq_pages (after the end of the client
> > >>> requested READ payload).
<snip>

> > >> There's nothing I can think of in the RDMA or RPC/RDMA protocols that
> > >> mandates that the first page offset must always be zero. Moving data
> > >> at one address on the server to an entirely different address and
> > >> alignment on the client is exactly what RDMA is supposed to do.
> > >>
> > >> It sounds like an implementation omission because the server's upper
> > >> layers have never needed it before now. If TCP already handles it, I'm
> > >> guessing it's going to be straightforward to fix.
> > > 
> > > I never said that first page offset must be zero.  I said that I
> > > already did what you suggested and it didn't work with RDMA.  This is
> > > recall of too many months ago now, but: the client will see the
> > > correct READ payload _except_ IIRC it is offset by whatever front-pad
> > > was added to expand the misaligned DIO; no matter whether
> > > rqstp->rq_bvec updated when IO completes.
> > > 
> > > But I'll revisit it again.
> > 
> > For the record, this email thread is the very first time I've heard that
> > you tried the simple approach and that it worked with TCP and not with
> > RDMA. I wish I had known that a while ago.
> 
> Likewise, but the story is all in the patch header and the code tells
> the story too. Hence your finding it with closer review (thanks for
> that BTW!). I agree something is off so I'm happy to work it further.
> 
> I have iterated on quite a few aspects to this patch 5. Christoph had
> suggestion for using memmove in nfsd_complete_misaligned_read_dio.
> You had the feedback that required ensuring the lightest touch
> relative to branching so that buffered IO mode remain as fast as
> possible.
> 
> Looking forward to tackling this RDMA-specific weirdness now.

Hopeful these 2 patches more clearly demonstrate what I'm finding
needed when using RDMA with my NFSD misaligned DIO READ patch.

These patches build ontop of my v8 patchset. I've included quite a lot
of context for the data mismatch seen by the NFS client, etc in the
patch headers.

If I'm understanding you correctly, next step is to look closer at the
rpcrdma code that would skip the throwaway front-pad page from being
mapped to the start of the RDMA READ payload returned to the NFS
client?

Such important adjustment code would need to know that the rq_bvec[]
that reflects the READ payload doesn't include a bvec that points to
the first page of rqstp->rq_pages (pointed to by rqstp->rq_next_page
on entry to nfsd_iter_read) -- so it must skip past that memory in
the READ payload's RDMA memory returned to NFS client?

Thanks,
Mike

Mike Snitzer (2):
  NFSD: fix misaligned DIO READ to not use a start_extra_page, exposes rpcrdma bug?
  NFSD: use /end/ of rq_pages for front_pad page, simpler workaround for rpcrdma bug

 fs/nfsd/vfs.c | 27 ++++++++-------------------
 1 file changed, 8 insertions(+), 19 deletions(-)

-- 
2.44.0