Re: [PATCH v15 00/26] nfs/nfsd: add support for LOCALIO

[Mike Snitzer <snitzer@kernel.org>]

On Tue, Sep 10, 2024 at 06:11:50PM -0400, Mike Snitzer wrote:
> On Tue, Sep 10, 2024 at 04:31:23PM -0400, Anna Schumaker wrote:
> > Hi Mike,
> > 
> > On 9/10/24 3:14 PM, Mike Snitzer wrote:
> > > 
> > > In case it helps, I did just rebase LOCALIO (v16 + 1 fix) ontop of
> > > cel/nfsd-next (v6.11-rc6 based), and I've pushed the result here:
> > > https://git.kernel.org/pub/scm/linux/kernel/git/snitzer/linux.git/log/?h=nfs-localio-for-next
> > 
> > I'm seeing the same hang on generic/525 with your latest branch.
> > 
> > Anna
> 
> Interesting, I just looked at ktest and it shows the regression point
> to be this commit:
>    nfs: implement client support for NFS_LOCALIO_PROGRAM
> 
> See:
> https://evilpiepirate.org/~testdashboard/ci?user=snitzer&branch=snitm-nfs-next&test=^fs.nfs.fstests.generic.525$
> 
> I think 525 has been like this for a while, really not sure why I
> ignored it... will dig deeper!

I haven't found/fixed this yet but when LOCALIO is used
xfs_file_buffered_read calls filemap_get_pages and filemap_get_pages
livelocks trying to complete.

Here is ftrace of LOCALIO's xfs_file_buffered_read immediately before
filemap_get_pages never returns (note that for debugging I reverted
the workqueue patch.. so the IO is issued in xfs_io context):

=> entry_SYSCALL_64_after_hwframe
          xfs_io-5609    [007] .....   211.831093: xfs_file_buffered_read: dev 8:16 ino 0x84 disize 0x7fffffffffffffff pos 0x7ffffffffffff000 bytecount 0x1000
          xfs_io-5609    [007] ...1.   211.831098: <stack trace>
 => trace_event_raw_event_xfs_file_class
 => xfs_file_buffered_read
 => xfs_file_read_iter
 => nfs_local_doio
 => nfs_initiate_pgio
 => nfs_generic_pg_pgios
 => nfs_pageio_doio
 => nfs_pageio_complete
 => nfs_pageio_complete_read
 => nfs_readahead
 => read_pages
 => page_cache_ra_unbounded
 => page_cache_sync_ra
 => filemap_get_pages
 => filemap_read
 => generic_file_read_iter
 => nfs_file_read
 => vfs_read
 => __x64_sys_pread64
 => x64_sys_call
 => do_syscall_64

Here is the same when testing only against XFS:

 => entry_SYSCALL_64_after_hwframe
          xfs_io-3451    [015] .....  1034.767416: xfs_file_buffered_read: dev 8:16 ino 0x84 disize 0x7fffffffffffffff pos 0x7ffffffffffffffe bytecount 0x1
          xfs_io-3451    [015] ...1.  1034.767418: <stack trace>
 => trace_event_raw_event_xfs_file_class
 => xfs_file_buffered_read
 => xfs_file_read_iter
 => vfs_read
 => __x64_sys_pread64
 => x64_sys_call
 => do_syscall_64

Will keep after this with urgency, just wanted to let you know what I
have found so far...

Thanks,
Mike