mount NFS with localio

mount NFS with localio

[Anton Gavriliuk]

Hi

How to mount NFS with localio on Fedora Server 42 (6.15.8 kernel) ?

Localio enabled in kernel

[root@23-127-77-5 ~]# cat /boot/config-6.15.8-200.fc42.x86_64 | grep -i localio
CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m
CONFIG_NFS_LOCALIO=y

[root@23-127-77-5 ~]# lsmod | grep -i localio
nfs_localio            36864  2 nfsd,nfs
sunrpc                925696  30
nfs_localio,nfsd,rpcrdma,nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl,nfs

[root@23-127-77-5 ~]# mount -t nfs 127.0.0.1:/mnt /mnt1
[root@23-127-77-5 ~]#
[root@23-127-77-5 ~]# mount | grep -i mnt1
127.0.0.1:/mnt on /mnt1 type nfs4
(rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

So /mnt1 is mounted with localio ?

Anton

Re: mount NFS with localio

[Mike Snitzer]

On Wed, Jul 30, 2025 at 03:43:00PM +0300, Anton Gavriliuk wrote:
> Hi
> 
> How to mount NFS with localio on Fedora Server 42 (6.15.8 kernel) ?
> 
> Localio enabled in kernel
> 
> [root@23-127-77-5 ~]# cat /boot/config-6.15.8-200.fc42.x86_64 | grep -i localio
> CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m
> CONFIG_NFS_LOCALIO=y
> 
> [root@23-127-77-5 ~]# lsmod | grep -i localio
> nfs_localio            36864  2 nfsd,nfs
> sunrpc                925696  30
> nfs_localio,nfsd,rpcrdma,nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl,nfs
> 
> [root@23-127-77-5 ~]# mount -t nfs 127.0.0.1:/mnt /mnt1
> [root@23-127-77-5 ~]#
> [root@23-127-77-5 ~]# mount | grep -i mnt1
> 127.0.0.1:/mnt on /mnt1 type nfs4
> (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)
> 
> So /mnt1 is mounted with localio ?

Should be.

One way to tell, albeit less ironclad, is that performance improves
(reduced use of network/sunrpc is also a tell-tale).

But see commit 62d2cde203def ("NFS: add localio to sysfs")
It adds semi-useful exposure of whether LOCALIO is active for a given
rpc client.  I say "semi-useful" because I'm left wanting on the
reliability of this interface (particularly when pnfs flexfiles is
used, the separate NFS client that is established to connect to the
local NFSD over v3 isn't getting added to sysfs for some reason.. not
put time to fixing that yet).  But for simple loopback NFS mount that
doesn't use pnfs flexfiles it _should_ work, e.g.:

  # cat /sys/fs/nfs/0\:46/localio
  1

I've found that the best way to _know_ LOCALIO enabled is to use trace
points, but yeah that is kind of obscure and certainly not common in
production.

These tracepoints really showcase LOCALIO is being used:

 echo 1 > /sys/kernel/tracing/events/sunrpc/svc_process/enable
 echo 1 > /sys/kernel/tracing/events/nfs_localio/nfs_localio_enable_client/enable
 echo 1 > /sys/kernel/tracing/events/nfs_localio/nfs_localio_disable_client/enable
 echo 1 > /sys/kernel/tracing/events/nfs/nfs_local_open_fh/enable

(NOTE: it is only with this patch applied that the nfs_local_open_fh
tracepoint is made much more useful, otherwise it'd only show if the
function failed to open the fh.. unfortunately not yet picked up for
upstream:
https://lore.kernel.org/linux-nfs/20250724193102.65111-9-snitzer@kernel.org/
)

 echo nop > /sys/kernel/debug/tracing/current_tracer
 echo 1 > /sys/kernel/debug/tracing/tracing_on

With these enabled, mounting NFS via loopback and doing a simple dd
shows the following in: cat /sys/kernel/debug/tracing/trace

            nfsd-10448   [024] .....  4316.520916: svc_process: addr=192.168.1.106 xid=0xcf2cdbdf service=nfslocalio vers=1 proc=UUID_IS_LOCAL
  kworker/u194:0-9772    [042] .....  4316.520951: nfs_localio_enable_client: server=192.168.1.106 NFSv3
  kworker/u194:0-9772    [042] .....  4316.647334: nfs_local_open_fh: fhandle=0x4d34e6c1 mode=READ|WRITE result=0

Also, enabling various nfsd tracepoints and seeing the absence of them
is telling.  Similarly, enabling tracepoints for NFSD's underlying
filesystem (e.g. xfs) and seeing the process that is triggering the
trace isn't nfsd showcases LOCALIO being used, e.g.:

with LOCALIO:

  kworker/u194:3-9540    [027] .....  5155.011380: xfs_file_direct_write: dev 259:16 ino 0x3e00008f disize 0xb7a0 pos 0xc000 bytecount 0xa000

without LOCALIO:

            nfsd-10448   [034] .....  5730.314274: xfs_file_direct_write: dev 259:16 ino 0x3e00008f disize 0xb7a0 pos 0xc000 bytecount 0xa000

Hope this helps,
Mike

Re: mount NFS with localio

[Anton Gavriliuk]

> But for simple loopback NFS mount that
> doesn't use pnfs flexfiles it _should_ work, e.g.:
>
>  # cat /sys/fs/nfs/0\:46/localio
>  1

Yes, it works, thank you.  For NFS localio single thread sequential
read with enabled localio_O_DIRECT_semantics, performance improved 3.5
times!

I'm also looking for non-localio NFS performance improvements,
interested in end-to-end O_DIRECT.

How to enable non-localio end-to-end O_DIRECT ?

Anton

чт, 31 июл. 2025 г. в 20:58, Mike Snitzer <snitzer@kernel.org>:
>
> On Wed, Jul 30, 2025 at 03:43:00PM +0300, Anton Gavriliuk wrote:
> > Hi
> >
> > How to mount NFS with localio on Fedora Server 42 (6.15.8 kernel) ?
> >
> > Localio enabled in kernel
> >
> > [root@23-127-77-5 ~]# cat /boot/config-6.15.8-200.fc42.x86_64 | grep -i localio
> > CONFIG_NFS_COMMON_LOCALIO_SUPPORT=m
> > CONFIG_NFS_LOCALIO=y
> >
> > [root@23-127-77-5 ~]# lsmod | grep -i localio
> > nfs_localio            36864  2 nfsd,nfs
> > sunrpc                925696  30
> > nfs_localio,nfsd,rpcrdma,nfsv4,auth_rpcgss,lockd,rpcsec_gss_krb5,nfs_acl,nfs
> >
> > [root@23-127-77-5 ~]# mount -t nfs 127.0.0.1:/mnt /mnt1
> > [root@23-127-77-5 ~]#
> > [root@23-127-77-5 ~]# mount | grep -i mnt1
> > 127.0.0.1:/mnt on /mnt1 type nfs4
> > (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,fatal_neterrors=none,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)
> >
> > So /mnt1 is mounted with localio ?
>
> Should be.
>
> One way to tell, albeit less ironclad, is that performance improves
> (reduced use of network/sunrpc is also a tell-tale).
>
> But see commit 62d2cde203def ("NFS: add localio to sysfs")
> It adds semi-useful exposure of whether LOCALIO is active for a given
> rpc client.  I say "semi-useful" because I'm left wanting on the
> reliability of this interface (particularly when pnfs flexfiles is
> used, the separate NFS client that is established to connect to the
> local NFSD over v3 isn't getting added to sysfs for some reason.. not
> put time to fixing that yet).  But for simple loopback NFS mount that
> doesn't use pnfs flexfiles it _should_ work, e.g.:
>
>   # cat /sys/fs/nfs/0\:46/localio
>   1
>
> I've found that the best way to _know_ LOCALIO enabled is to use trace
> points, but yeah that is kind of obscure and certainly not common in
> production.
>
> These tracepoints really showcase LOCALIO is being used:
>
>  echo 1 > /sys/kernel/tracing/events/sunrpc/svc_process/enable
>  echo 1 > /sys/kernel/tracing/events/nfs_localio/nfs_localio_enable_client/enable
>  echo 1 > /sys/kernel/tracing/events/nfs_localio/nfs_localio_disable_client/enable
>  echo 1 > /sys/kernel/tracing/events/nfs/nfs_local_open_fh/enable
>
> (NOTE: it is only with this patch applied that the nfs_local_open_fh
> tracepoint is made much more useful, otherwise it'd only show if the
> function failed to open the fh.. unfortunately not yet picked up for
> upstream:
> https://lore.kernel.org/linux-nfs/20250724193102.65111-9-snitzer@kernel.org/
> )
>
>  echo nop > /sys/kernel/debug/tracing/current_tracer
>  echo 1 > /sys/kernel/debug/tracing/tracing_on
>
> With these enabled, mounting NFS via loopback and doing a simple dd
> shows the following in: cat /sys/kernel/debug/tracing/trace
>
>             nfsd-10448   [024] .....  4316.520916: svc_process: addr=192.168.1.106 xid=0xcf2cdbdf service=nfslocalio vers=1 proc=UUID_IS_LOCAL
>   kworker/u194:0-9772    [042] .....  4316.520951: nfs_localio_enable_client: server=192.168.1.106 NFSv3
>   kworker/u194:0-9772    [042] .....  4316.647334: nfs_local_open_fh: fhandle=0x4d34e6c1 mode=READ|WRITE result=0
>
> Also, enabling various nfsd tracepoints and seeing the absence of them
> is telling.  Similarly, enabling tracepoints for NFSD's underlying
> filesystem (e.g. xfs) and seeing the process that is triggering the
> trace isn't nfsd showcases LOCALIO being used, e.g.:
>
> with LOCALIO:
>
>   kworker/u194:3-9540    [027] .....  5155.011380: xfs_file_direct_write: dev 259:16 ino 0x3e00008f disize 0xb7a0 pos 0xc000 bytecount 0xa000
>
> without LOCALIO:
>
>             nfsd-10448   [034] .....  5730.314274: xfs_file_direct_write: dev 259:16 ino 0x3e00008f disize 0xb7a0 pos 0xc000 bytecount 0xa000
>
> Hope this helps,
> Mike