[PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[Antonio Alvarez Feijoo]

An NFS root mounted over NFSv4 against a Linux NFS server hangs during
boot, either wedging permanently in the switch-root exec or, on a loaded
machine, spinning until the softlockup watchdog fires.

The flexfiles layout driver reaches nfs4_pnfs_ds_connect() on the first
I/O through a layout segment.  For an NFSv3 data server that calls
symbol_request(nfs3_set_ds_client), which becomes
request_module("symbol:nfs3_set_ds_client") when nfsv3 is not already
resident.  request_module() execs /sbin/modprobe via the usermode
helper, and on an NFS root that binary -- and libkmod, and the libraries
behind it -- must be read from the very filesystem whose I/O is blocked
waiting on this DS connection to be established.  The helper cannot make
progress, so neither can the mount.

On a system whose init is on NFS the first such I/O is typically the
execve() of init itself, immediately after pivot_root(), at which point
the initramfs -- and the only reachable copy of modprobe -- is gone:

  WARNING: fs/nfs/pnfs_nfs.c:792 at nfs4_pnfs_ds_connect+0x47e/0x490 [nfsv4], CPU#0: systemd/1
  Modules linked in: nfs_layout_flexfiles rpcsec_gss_krb5 krb5 auth_rpcgss
                     nfsv4 dns_resolver nfs lockd grace netfs af_packet [...]
  Call Trace:
   nfs4_pnfs_ds_connect+0x47e/0x490 [nfsv4]
   nfs4_ff_layout_prepare_ds+0x...  [nfs_layout_flexfiles]
   ff_layout_choose_ds_for_read+0x...
   ff_layout_pg_init_read+0x...
   nfs_pageio_add_request+0x...
   nfs_readahead+0x...
   filemap_read+0x...
   __kernel_read+0x...
   bprm_execve+0x...
   do_execveat_common.isra.0+0x...
   __x64_sys_execve+0x...

Note the absence of nfsv3 from the module list.

This has been latent since the flexfiles driver was added, because a
fatal DS connect error used to be absorbed: ff_layout_read_pagelist()
returned PNFS_NOT_ATTEMPTED and pnfs_do_read() reissued the I/O through
pnfs_read_through_mds().  The read completed over plain NFSv4 against
the metadata server and nobody noticed that the layout had been
abandoned.

Commit 7a375cafc14e ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS on
fatal DS connect errors") removes that fallback for layouts carrying
FF_FLAGS_NO_IO_THRU_MDS, correctly, since silently contradicting the
flag was a bug.  But nfsd sets that flag on every layout it hands out
(fs/nfsd/flexfilelayout.c: FF_FLAGS_NO_LAYOUTCOMMIT |
FF_FLAGS_NO_IO_THRU_MDS | FF_FLAGS_NO_READ_IO, unconditionally), so
against a Linux server the no-fallback path is not the appliance corner
case the commit describes -- it is every layout.  With no fallback left,
a DS connect that cannot complete is now fatal to the boot.

FF_FLAGS_NO_READ_IO does not save the read: the client only diverts
reads off IOMODE_RW segments, and a read-only NFS root is served
IOMODE_READ segments, where the flag does not apply.

Declare the dependency via MODULE_SOFTDEP("pre: nfsv3") instead of
discovering it at I/O time.  The layout driver is itself loaded by
request_module("nfs-layouttype4-4") from find_pnfs_driver() during
mount(), which on an initramfs boot happens while the initramfs and a
working /sbin/modprobe are still in place, so a softdep is resolved at a
point where the usermode helper can actually run.

This pulls in nfsv3 whenever the flexfiles driver is loaded, including
for layouts whose data servers all speak NFSv4.1 and which would never
have needed it.  That seemed the better trade against an unbootable
system.

Reported on openSUSE Tumbleweed with kernel 7.2.2 under QEMU,
reproducible with dracut's TEST-60-NFS suite against any knfsd built with
CONFIG_NFSD_FLEXFILELAYOUT=y.  Booting with rd.driver.pre=nfsv3, which
loads the module from the initramfs before the pivot, is a complete
workaround and confirms the diagnosis.

Fixes: d67ae825a59d ("pnfs/flexfiles: Add the FlexFile Layout Driver")
Cc: stable@vger.kernel.org
Assisted-by: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Antonio Alvarez Feijoo <antonio.feijoo@suse.com>
---
 fs/nfs/flexfilelayout/flexfilelayout.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index 7fe8b91fa47c..12a4a7908fe3 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -3079,7 +3079,7 @@ static void __exit nfs4flexfilelayout_exit(void)
 }
 
 MODULE_ALIAS("nfs-layouttype4-4");
-
+MODULE_SOFTDEP("pre: nfsv3");
 MODULE_LICENSE("GPL");
 MODULE_DESCRIPTION("The NFSv4 flexfile layout driver");
 
-- 
2.51.0

Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[Roberto Bergantinos Corpas]

There is another problematic case for this problem : if an
unprivileged container attempts pNFS I/O, SELinux may block the
request_module() call, leading to a writeback hang instead of a boot hang.

Having the module pre-loaded at mount time avoids that scenario too.

On Wed, Sep 2, 2026 at 4:04 PM Antonio Alvarez Feijoo
<antonio.feijoo@suse.com> wrote:
>
> An NFS root mounted over NFSv4 against a Linux NFS server hangs during
> boot, either wedging permanently in the switch-root exec or, on a loaded
> machine, spinning until the softlockup watchdog fires.
>
> The flexfiles layout driver reaches nfs4_pnfs_ds_connect() on the first
> I/O through a layout segment.  For an NFSv3 data server that calls
> symbol_request(nfs3_set_ds_client), which becomes
> request_module("symbol:nfs3_set_ds_client") when nfsv3 is not already
> resident.  request_module() execs /sbin/modprobe via the usermode
> helper, and on an NFS root that binary -- and libkmod, and the libraries
> behind it -- must be read from the very filesystem whose I/O is blocked
> waiting on this DS connection to be established.  The helper cannot make
> progress, so neither can the mount.
>
> On a system whose init is on NFS the first such I/O is typically the
> execve() of init itself, immediately after pivot_root(), at which point
> the initramfs -- and the only reachable copy of modprobe -- is gone:
>
>   WARNING: fs/nfs/pnfs_nfs.c:792 at nfs4_pnfs_ds_connect+0x47e/0x490 [nfsv4], CPU#0: systemd/1
>   Modules linked in: nfs_layout_flexfiles rpcsec_gss_krb5 krb5 auth_rpcgss
>                      nfsv4 dns_resolver nfs lockd grace netfs af_packet [...]
>   Call Trace:
>    nfs4_pnfs_ds_connect+0x47e/0x490 [nfsv4]
>    nfs4_ff_layout_prepare_ds+0x...  [nfs_layout_flexfiles]
>    ff_layout_choose_ds_for_read+0x...
>    ff_layout_pg_init_read+0x...
>    nfs_pageio_add_request+0x...
>    nfs_readahead+0x...
>    filemap_read+0x...
>    __kernel_read+0x...
>    bprm_execve+0x...
>    do_execveat_common.isra.0+0x...
>    __x64_sys_execve+0x...
>
> Note the absence of nfsv3 from the module list.
>
> This has been latent since the flexfiles driver was added, because a
> fatal DS connect error used to be absorbed: ff_layout_read_pagelist()
> returned PNFS_NOT_ATTEMPTED and pnfs_do_read() reissued the I/O through
> pnfs_read_through_mds().  The read completed over plain NFSv4 against
> the metadata server and nobody noticed that the layout had been
> abandoned.
>
> Commit 7a375cafc14e ("NFSv4/flexfiles: honor FF_FLAGS_NO_IO_THRU_MDS on
> fatal DS connect errors") removes that fallback for layouts carrying
> FF_FLAGS_NO_IO_THRU_MDS, correctly, since silently contradicting the
> flag was a bug.  But nfsd sets that flag on every layout it hands out
> (fs/nfsd/flexfilelayout.c: FF_FLAGS_NO_LAYOUTCOMMIT |
> FF_FLAGS_NO_IO_THRU_MDS | FF_FLAGS_NO_READ_IO, unconditionally), so
> against a Linux server the no-fallback path is not the appliance corner
> case the commit describes -- it is every layout.  With no fallback left,
> a DS connect that cannot complete is now fatal to the boot.
>
> FF_FLAGS_NO_READ_IO does not save the read: the client only diverts
> reads off IOMODE_RW segments, and a read-only NFS root is served
> IOMODE_READ segments, where the flag does not apply.
>
> Declare the dependency via MODULE_SOFTDEP("pre: nfsv3") instead of
> discovering it at I/O time.  The layout driver is itself loaded by
> request_module("nfs-layouttype4-4") from find_pnfs_driver() during
> mount(), which on an initramfs boot happens while the initramfs and a
> working /sbin/modprobe are still in place, so a softdep is resolved at a
> point where the usermode helper can actually run.
>
> This pulls in nfsv3 whenever the flexfiles driver is loaded, including
> for layouts whose data servers all speak NFSv4.1 and which would never
> have needed it.  That seemed the better trade against an unbootable
> system.
>
> Reported on openSUSE Tumbleweed with kernel 7.2.2 under QEMU,
> reproducible with dracut's TEST-60-NFS suite against any knfsd built with
> CONFIG_NFSD_FLEXFILELAYOUT=y.  Booting with rd.driver.pre=nfsv3, which
> loads the module from the initramfs before the pivot, is a complete
> workaround and confirms the diagnosis.
>
> Fixes: d67ae825a59d ("pnfs/flexfiles: Add the FlexFile Layout Driver")
> Cc: stable@vger.kernel.org
> Assisted-by: Claude Opus 5 <noreply@anthropic.com>
> Signed-off-by: Antonio Alvarez Feijoo <antonio.feijoo@suse.com>
> ---
>  fs/nfs/flexfilelayout/flexfilelayout.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
> index 7fe8b91fa47c..12a4a7908fe3 100644
> --- a/fs/nfs/flexfilelayout/flexfilelayout.c
> +++ b/fs/nfs/flexfilelayout/flexfilelayout.c
> @@ -3079,7 +3079,7 @@ static void __exit nfs4flexfilelayout_exit(void)
>  }
>
>  MODULE_ALIAS("nfs-layouttype4-4");
> -
> +MODULE_SOFTDEP("pre: nfsv3");
>  MODULE_LICENSE("GPL");
>  MODULE_DESCRIPTION("The NFSv4 flexfile layout driver");
>
> --
> 2.51.0
>
>

Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[Benjamin Coddington]

On 23 Oct 2026, at 8:30, Roberto Bergantinos Corpas wrote:

> There is another problematic case for this problem : if an
> unprivileged container attempts pNFS I/O, SELinux may block the
> request_module() call, leading to a writeback hang instead of a boot hang.
>
> Having the module pre-loaded at mount time avoids that scenario too.

I think this is a stronger case than the nfs-root, which already requires a
sysadmin to have knowledge and desire to configure the client a certain way,
and can certainly apply the work-around.

Roberto's case would look functional to the admin, and the failure would
happen later and be difficult to diagnose.

I think Fixes: 7a375cafc14e (7.2.y) is the accurate tag, though - because
before that the -EPROTONOSUPPORT would fall back to the MDS - it gives the
WARN, but doesn't hang the system.

I also think 7a375cafc14e also needs a follow-on fix (Cc: Mike) - because it
now causes permanent failure cases to retry forever- we should probably just
fail the IO.

The last thing here is that the MODULE_SOFTDEP is just informative, it only
advises modprobe and initramfs builders.  It not going to fix
CONFIG_NFS_V4=y with CONFIG_NFS_V3=n or m.  So - I think we should just make
flexfiles publish a hard dependency on a v3 symbol, which fixes the module
dependency directly, and also fixup Kconfig so that there aren't unexpected
build failures as a result.

My 2 cents,

Ben

Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[Mkrtchyan, Tigran]

[-- Attachment #1: Type: text/plain, Size: 2342 bytes --]


Frankly, I think this is my change that has introduced the trouble:

a68c6fbb638a ("nfs41: drop dependency between flexfiles layout driver and NFSv3 modules")

The intent was to be able to build a kernel with flexfiles support even if NFSv3 is disabled.
No objections if you want to revert it.

Best regards,
   Tigran.


----- Original Message -----
> From: "Benjamin Coddington" <ben.coddington@hammerspace.com>
> To: "Roberto Bergantinos Corpas" <rbergant@redhat.com>, "Mike Snitzer" <snitzer@kernel.org>
> Cc: "Antonio Alvarez Feijoo" <antonio.feijoo@suse.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Mike Snitzer"
> <snitzer@kernel.org>, "Anna Schumaker" <anna.schumaker@hammerspace.com>, "Trond Myklebust"
> <trond.myklebust@hammerspace.com>
> Sent: Friday, 23 October, 2026 16:16:32
> Subject: Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

> On 23 Oct 2026, at 8:30, Roberto Bergantinos Corpas wrote:
> 
>> There is another problematic case for this problem : if an
>> unprivileged container attempts pNFS I/O, SELinux may block the
>> request_module() call, leading to a writeback hang instead of a boot hang.
>>
>> Having the module pre-loaded at mount time avoids that scenario too.
> 
> I think this is a stronger case than the nfs-root, which already requires a
> sysadmin to have knowledge and desire to configure the client a certain way,
> and can certainly apply the work-around.
> 
> Roberto's case would look functional to the admin, and the failure would
> happen later and be difficult to diagnose.
> 
> I think Fixes: 7a375cafc14e (7.2.y) is the accurate tag, though - because
> before that the -EPROTONOSUPPORT would fall back to the MDS - it gives the
> WARN, but doesn't hang the system.
> 
> I also think 7a375cafc14e also needs a follow-on fix (Cc: Mike) - because it
> now causes permanent failure cases to retry forever- we should probably just
> fail the IO.
> 
> The last thing here is that the MODULE_SOFTDEP is just informative, it only
> advises modprobe and initramfs builders.  It not going to fix
> CONFIG_NFS_V4=y with CONFIG_NFS_V3=n or m.  So - I think we should just make
> flexfiles publish a hard dependency on a v3 symbol, which fixes the module
> dependency directly, and also fixup Kconfig so that there aren't unexpected
> build failures as a result.
> 
> My 2 cents,
> 
> Ben

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2309 bytes --]

Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3

[Roberto Bergantinos Corpas]

On Fri, Oct 23, 2026 at 4:40 PM Mkrtchyan, Tigran
<tigran.mkrtchyan@desy.de> wrote:
>
>
> Frankly, I think this is my change that has introduced the trouble:
>
> a68c6fbb638a ("nfs41: drop dependency between flexfiles layout driver and NFSv3 modules")

Im not sure of this, that enables flexfiles when NFS_V3=n at compile
time, but i think it does not prevent
the actual hangs at runtime. If i understand this correctly the hang
requires nfsv3 not loaded at runtime
(which can happen with NFS_V3=m) + no MDS fallback (7a375cafc14e) +
i.e. unprivileged container.


>
>
> ----- Original Message -----
> > From: "Benjamin Coddington" <ben.coddington@hammerspace.com>
> > To: "Roberto Bergantinos Corpas" <rbergant@redhat.com>, "Mike Snitzer" <snitzer@kernel.org>
> > Cc: "Antonio Alvarez Feijoo" <antonio.feijoo@suse.com>, "linux-nfs" <linux-nfs@vger.kernel.org>, "Mike Snitzer"
> > <snitzer@kernel.org>, "Anna Schumaker" <anna.schumaker@hammerspace.com>, "Trond Myklebust"
> > <trond.myklebust@hammerspace.com>
> > Sent: Friday, 23 October, 2026 16:16:32
> > Subject: Re: [PATCH] NFSv4/flexfiles: add a soft dependency on nfsv3
>
> > On 23 Oct 2026, at 8:30, Roberto Bergantinos Corpas wrote:
> >
> >> There is another problematic case for this problem : if an
> >> unprivileged container attempts pNFS I/O, SELinux may block the
> >> request_module() call, leading to a writeback hang instead of a boot hang.
> >>
> >> Having the module pre-loaded at mount time avoids that scenario too.
> >
> > I think this is a stronger case than the nfs-root, which already requires a
> > sysadmin to have knowledge and desire to configure the client a certain way,
> > and can certainly apply the work-around.
> >
> > Roberto's case would look functional to the admin, and the failure would
> > happen later and be difficult to diagnose.
> >
> > I think Fixes: 7a375cafc14e (7.2.y) is the accurate tag, though - because
> > before that the -EPROTONOSUPPORT would fall back to the MDS - it gives the
> > WARN, but doesn't hang the system.
> >
> > I also think 7a375cafc14e also needs a follow-on fix (Cc: Mike) - because it
> > now causes permanent failure cases to retry forever- we should probably just
> > fail the IO.
> >
> > The last thing here is that the MODULE_SOFTDEP is just informative, it only
> > advises modprobe and initramfs builders.  It not going to fix
> > CONFIG_NFS_V4=y with CONFIG_NFS_V3=n or m.  So - I think we should just make
> > flexfiles publish a hard dependency on a v3 symbol, which fixes the module
> > dependency directly, and also fixup Kconfig so that there aren't unexpected
> > build failures as a result.
> >
> > My 2 cents,
> >
> > Ben