Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[Benjamin Maynard]

Hi,

I was recently experimenting with NFS re-exporting using the new patch
set that is in the Linux 5.11 kernel
(https://patchwork.kernel.org/project/linux-nfs/list/?series=393561).

After applying these patches, I consistently faced an error when
trying to perform a previously working NFS re-export: "exportfs:
/files does not support NFS export".

I (with help from some other interested parties) began troubleshooting
and after stepping through each patch individually we identified that
the error only occurred when the following patch was applied:
https://patchwork.kernel.org/project/linux-nfs/patch/20201130220319.501064-3-trond.myklebust@hammerspace.com/.

This patch prevents re-exporting if subtree checking is enabled on the
originating NFS server. The strange thing was that no_subtree_check
export option was already set on the export from the originating NFS
Filer, but the error message persisted.

After lots of troubleshooting, eventually we tried updating NFS Utils
from 1.3.4 to 2.5.2 and we were able to successfully perform
re-export. It appears that the old version of the nfs-utils package
was the cause of the issue.

I appreciate that 1.3.4 is a very old version of nfs-utils, but it is
the default version that ships with Ubuntu and Debian and the error
message does not immediately point to the outdated version being the
cause of the problem.

I was wondering if it was possible to detail the requirement for a
more recent version of nfs-utils in the NFS Re-exporting section of
the Wiki (http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export) to
help others who may encounter this problem in the future?


Kind Regards
Benjamin Maynard

Re: Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[J. Bruce Fields]

On Mon, Jan 18, 2021 at 05:57:54PM +0000, Benjamin Maynard wrote:
> Hi,
> 
> I was recently experimenting with NFS re-exporting using the new patch
> set that is in the Linux 5.11 kernel
> (https://patchwork.kernel.org/project/linux-nfs/list/?series=393561).
> 
> After applying these patches, I consistently faced an error when
> trying to perform a previously working NFS re-export: "exportfs:
> /files does not support NFS export".
> 
> I (with help from some other interested parties) began troubleshooting
> and after stepping through each patch individually we identified that
> the error only occurred when the following patch was applied:
> https://patchwork.kernel.org/project/linux-nfs/patch/20201130220319.501064-3-trond.myklebust@hammerspace.com/.

That link isn't working for me for some reason.

Looks like we're talking about ba5e8187c555 "nfsd: allow filesystems to
opt out of subtree checking".

> This patch prevents re-exporting if subtree checking is enabled on the
> originating NFS server.

That's not correct.

I'm assuming there are two servers: a reexporting server, which mounts
the originating NFS server, which is mounting ext4 or xfs or some other
local filesystem.

It's hard for the reexporting server to even tell if the originating
server is using subtree checking, so I'm surprised that would make a
difference in behavior.

> The strange thing was that no_subtree_check
> export option was already set on the export from the originating NFS
> Filer, but the error message persisted.

So, this patch is only checks whether you've got no_subtree_check set on
the reexporting server.

> After lots of troubleshooting, eventually we tried updating NFS Utils
> from 1.3.4 to 2.5.2 and we were able to successfully perform
> re-export. It appears that the old version of the nfs-utils package
> was the cause of the issue.

I'm a little confused about what happened here.  Which server were you
applying the above patches on?  Which server did you upgrade NFS utils
on?

Could be that you're actually running into some filehandle size limit or
something.  (Subtree checking can make that problem worse.)  Hard to
tell.

--b.

> I appreciate that 1.3.4 is a very old version of nfs-utils, but it is
> the default version that ships with Ubuntu and Debian and the error
> message does not immediately point to the outdated version being the
> cause of the problem.
> 
> I was wondering if it was possible to detail the requirement for a
> more recent version of nfs-utils in the NFS Re-exporting section of
> the Wiki (http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export) to
> help others who may encounter this problem in the future?

Re: Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[Benjamin Maynard]

Thanks for the response. Comments inline below.

On Tue, 19 Jan 2021 at 18:02, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Mon, Jan 18, 2021 at 05:57:54PM +0000, Benjamin Maynard wrote:
> > Hi,
> >
> > I was recently experimenting with NFS re-exporting using the new patch
> > set that is in the Linux 5.11 kernel
> > (https://patchwork.kernel.org/project/linux-nfs/list/?series=393561).
> >
> > After applying these patches, I consistently faced an error when
> > trying to perform a previously working NFS re-export: "exportfs:
> > /files does not support NFS export".
> >
> > I (with help from some other interested parties) began troubleshooting
> > and after stepping through each patch individually we identified that
> > the error only occurred when the following patch was applied:
> > https://patchwork.kernel.org/project/linux-nfs/patch/20201130220319.501064-3-trond.myklebust@hammerspace.com/.
>
> That link isn't working for me for some reason.
>
> Looks like we're talking about ba5e8187c555 "nfsd: allow filesystems to
> opt out of subtree checking".
>

Correct, this is the patch I am referencing.

> > This patch prevents re-exporting if subtree checking is enabled on the
> > originating NFS server.
>
> That's not correct.
>
> I'm assuming there are two servers: a reexporting server, which mounts
> the originating NFS server, which is mounting ext4 or xfs or some other
> local filesystem.
>
> It's hard for the reexporting server to even tell if the originating
> server is using subtree checking, so I'm surprised that would make a
> difference in behavior.
>

That is correct, there is an originating NFS Server (Ubuntu 20.04 -
5.4.0-1034-gcp) that is exporting a directory from the local ext4
filesystem. This is exported with the following options:

/files 10.0.0.0/8(rw,no_subtree_check,fsid=10)

This is then mounted from the re-exporting server (export from /proc/mounts):

10.70.1.2:/files /files nfs
rw,sync,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.70.1.2,mountvers=3,mountport=20048,mountproto=udp,fsc,local_lock=none,addr=10.70.1.2
0 0

We then attempt to re-export the mounted directory from the
re-exporting server with the following entry in /etc/exports:

/files   10.67.0.0/16(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash)

If you perform this set of steps with the 5.10 kernel with nfs-utils
1.3.4 (Ubuntu & Debian default version), the re-export will work. If
you perform the same set of steps with the ba5e8187c555 patch applied
(still on nfs-utils 1.3.4) then the re-export will fail with the error
message "exportfs: /files does not support NFS export". dmesg further
reveals the cause "check_export: nfs does not support subtree
checking!".

This message appears even though we have no_subtree_check set on both
the exports of the originating NFS server, and the re-export server.

If you then upgrade nfs-utils to 2.5.2 on the re-export server, the
re-export works as expected.


> > The strange thing was that no_subtree_check
> > export option was already set on the export from the originating NFS
> > Filer, but the error message persisted.
>
> So, this patch is only checks whether you've got no_subtree_check set on
> the reexporting server.
>

Thanks. I misunderstood and thought it checked the exports on the
originating server. Regardless as detailed above no_subtree_check is
also set on the re-export server and the error persists when using an
older nfs-utils version.

> > After lots of troubleshooting, eventually we tried updating NFS Utils
> > from 1.3.4 to 2.5.2 and we were able to successfully perform
> > re-export. It appears that the old version of the nfs-utils package
> > was the cause of the issue.
>
> I'm a little confused about what happened here.  Which server were you
> applying the above patches on?  Which server did you upgrade NFS utils
> on?
>

The re-exporting server in both cases. The originating server is using
an older, unmodified kernel and version of nfs-utils. See above for a
more detailed overview.


> Could be that you're actually running into some filehandle size limit or
> something.  (Subtree checking can make that problem worse.)  Hard to
> tell.
>
> --b.
>
> > I appreciate that 1.3.4 is a very old version of nfs-utils, but it is
> > the default version that ships with Ubuntu and Debian and the error
> > message does not immediately point to the outdated version being the
> > cause of the problem.
> >
> > I was wondering if it was possible to detail the requirement for a
> > more recent version of nfs-utils in the NFS Re-exporting section of
> > the Wiki (http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export) to
> > help others who may encounter this problem in the future?

Re: Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[J. Bruce Fields]

On Thu, Jan 21, 2021 at 11:21:56AM +0000, Benjamin Maynard wrote:
> That is correct, there is an originating NFS Server (Ubuntu 20.04 -
> 5.4.0-1034-gcp) that is exporting a directory from the local ext4
> filesystem. This is exported with the following options:
> 
> /files 10.0.0.0/8(rw,no_subtree_check,fsid=10)
> 
> This is then mounted from the re-exporting server (export from /proc/mounts):
> 
> 10.70.1.2:/files /files nfs
> rw,sync,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.70.1.2,mountvers=3,mountport=20048,mountproto=udp,fsc,local_lock=none,addr=10.70.1.2
> 0 0
> 
> We then attempt to re-export the mounted directory from the
> re-exporting server with the following entry in /etc/exports:
> 
> /files   10.67.0.0/16(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash)
> 
> If you perform this set of steps with the 5.10 kernel with nfs-utils
> 1.3.4 (Ubuntu & Debian default version), the re-export will work. If
> you perform the same set of steps with the ba5e8187c555 patch applied
> (still on nfs-utils 1.3.4) then the re-export will fail with the error
> message "exportfs: /files does not support NFS export". dmesg further
> reveals the cause "check_export: nfs does not support subtree
> checking!".
> 
> This message appears even though we have no_subtree_check set on both
> the exports of the originating NFS server, and the re-export server.
> 
> If you then upgrade nfs-utils to 2.5.2 on the re-export server, the
> re-export works as expected.

Oh, got it, looks like the bug fixed by nfs-utils commit 63f520e8f6f5
"exportfs: Make sure pass all valid export flags to nfsd".

Rough explanation: export information isn't normally passed down to the
kernel when exportfs is called.  Instead the kernel waits till it needs
to know about some new client and/or filesystem and calls up to mountd
to ask for the relevant export entry.

Anyway, that's fine but it means the user doesn't find about errors
right away.

So, trying to be helpful, exportfs actually does pass down a dummy
export to the kernel at exportfs time, just to check for errors like a
typo'd export path or an unexportable filesystem.

Before that fix, it passed down that dumy export without the
"no_subtree_check" flag, even when you set that flag.

So, for nfs reexport, you need an nfs-utils new enough to include that
patch.

We're normally pretty strict about kernel regressions: if something
stopped working on kernel upgrade, that's a bug.  But I think we really
do need to fail attempts to re-export NFS with subtree checking, so
we've got to make an exception here.  Re-export is still a bit of an
experimental feature, so there may be hiccups like this.

--b.

Re: Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[Benjamin Maynard]

That makes sense, and thanks for explaining.

Seeing as the error message does not immediately point to an outdated
nfs-utils version would we be able to add a note to the Wiki
(http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export) to help
others that may come across this issue?

It looks like the Wiki is locked down to registered collaborators
otherwise I would add it myself.

Kind regards
Benjamin Maynard


On Thu, 21 Jan 2021 at 15:37, J. Bruce Fields <bfields@fieldses.org> wrote:
>
> On Thu, Jan 21, 2021 at 11:21:56AM +0000, Benjamin Maynard wrote:
> > That is correct, there is an originating NFS Server (Ubuntu 20.04 -
> > 5.4.0-1034-gcp) that is exporting a directory from the local ext4
> > filesystem. This is exported with the following options:
> >
> > /files 10.0.0.0/8(rw,no_subtree_check,fsid=10)
> >
> > This is then mounted from the re-exporting server (export from /proc/mounts):
> >
> > 10.70.1.2:/files /files nfs
> > rw,sync,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.70.1.2,mountvers=3,mountport=20048,mountproto=udp,fsc,local_lock=none,addr=10.70.1.2
> > 0 0
> >
> > We then attempt to re-export the mounted directory from the
> > re-exporting server with the following entry in /etc/exports:
> >
> > /files   10.67.0.0/16(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash)
> >
> > If you perform this set of steps with the 5.10 kernel with nfs-utils
> > 1.3.4 (Ubuntu & Debian default version), the re-export will work. If
> > you perform the same set of steps with the ba5e8187c555 patch applied
> > (still on nfs-utils 1.3.4) then the re-export will fail with the error
> > message "exportfs: /files does not support NFS export". dmesg further
> > reveals the cause "check_export: nfs does not support subtree
> > checking!".
> >
> > This message appears even though we have no_subtree_check set on both
> > the exports of the originating NFS server, and the re-export server.
> >
> > If you then upgrade nfs-utils to 2.5.2 on the re-export server, the
> > re-export works as expected.
>
> Oh, got it, looks like the bug fixed by nfs-utils commit 63f520e8f6f5
> "exportfs: Make sure pass all valid export flags to nfsd".
>
> Rough explanation: export information isn't normally passed down to the
> kernel when exportfs is called.  Instead the kernel waits till it needs
> to know about some new client and/or filesystem and calls up to mountd
> to ask for the relevant export entry.
>
> Anyway, that's fine but it means the user doesn't find about errors
> right away.
>
> So, trying to be helpful, exportfs actually does pass down a dummy
> export to the kernel at exportfs time, just to check for errors like a
> typo'd export path or an unexportable filesystem.
>
> Before that fix, it passed down that dumy export without the
> "no_subtree_check" flag, even when you set that flag.
>
> So, for nfs reexport, you need an nfs-utils new enough to include that
> patch.
>
> We're normally pretty strict about kernel regressions: if something
> stopped working on kernel upgrade, that's a bug.  But I think we really
> do need to fail attempts to re-export NFS with subtree checking, so
> we've got to make an exception here.  Re-export is still a bit of an
> experimental feature, so there may be hiccups like this.
>
> --b.

Re: Linux 5.11 Kernel: NFS re-export errors with older nfs-utils package versions

[J. Bruce Fields]

On Thu, Jan 21, 2021 at 05:46:31PM +0000, Benjamin Maynard wrote:
> That makes sense, and thanks for explaining.
> 
> Seeing as the error message does not immediately point to an outdated
> nfs-utils version would we be able to add a note to the Wiki
> (http://wiki.linux-nfs.org/wiki/index.php/NFS_re-export) to help
> others that may come across this issue?

Yep, done.

> It looks like the Wiki is locked down to registered collaborators
> otherwise I would add it myself.

The log in page has a link to the form to request an account.

I think those requests are routinely granted, it's just there to deal
with spam.

--b.

> 
> Kind regards
> Benjamin Maynard
> 
> 
> On Thu, 21 Jan 2021 at 15:37, J. Bruce Fields <bfields@fieldses.org> wrote:
> >
> > On Thu, Jan 21, 2021 at 11:21:56AM +0000, Benjamin Maynard wrote:
> > > That is correct, there is an originating NFS Server (Ubuntu 20.04 -
> > > 5.4.0-1034-gcp) that is exporting a directory from the local ext4
> > > filesystem. This is exported with the following options:
> > >
> > > /files 10.0.0.0/8(rw,no_subtree_check,fsid=10)
> > >
> > > This is then mounted from the re-exporting server (export from /proc/mounts):
> > >
> > > 10.70.1.2:/files /files nfs
> > > rw,sync,noatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,proto=tcp,nconnect=16,timeo=600,retrans=2,sec=sys,mountaddr=10.70.1.2,mountvers=3,mountport=20048,mountproto=udp,fsc,local_lock=none,addr=10.70.1.2
> > > 0 0
> > >
> > > We then attempt to re-export the mounted directory from the
> > > re-exporting server with the following entry in /etc/exports:
> > >
> > > /files   10.67.0.0/16(rw,wdelay,no_root_squash,no_subtree_check,fsid=10,sec=sys,rw,secure,no_root_squash,no_all_squash)
> > >
> > > If you perform this set of steps with the 5.10 kernel with nfs-utils
> > > 1.3.4 (Ubuntu & Debian default version), the re-export will work. If
> > > you perform the same set of steps with the ba5e8187c555 patch applied
> > > (still on nfs-utils 1.3.4) then the re-export will fail with the error
> > > message "exportfs: /files does not support NFS export". dmesg further
> > > reveals the cause "check_export: nfs does not support subtree
> > > checking!".
> > >
> > > This message appears even though we have no_subtree_check set on both
> > > the exports of the originating NFS server, and the re-export server.
> > >
> > > If you then upgrade nfs-utils to 2.5.2 on the re-export server, the
> > > re-export works as expected.
> >
> > Oh, got it, looks like the bug fixed by nfs-utils commit 63f520e8f6f5
> > "exportfs: Make sure pass all valid export flags to nfsd".
> >
> > Rough explanation: export information isn't normally passed down to the
> > kernel when exportfs is called.  Instead the kernel waits till it needs
> > to know about some new client and/or filesystem and calls up to mountd
> > to ask for the relevant export entry.
> >
> > Anyway, that's fine but it means the user doesn't find about errors
> > right away.
> >
> > So, trying to be helpful, exportfs actually does pass down a dummy
> > export to the kernel at exportfs time, just to check for errors like a
> > typo'd export path or an unexportable filesystem.
> >
> > Before that fix, it passed down that dumy export without the
> > "no_subtree_check" flag, even when you set that flag.
> >
> > So, for nfs reexport, you need an nfs-utils new enough to include that
> > patch.
> >
> > We're normally pretty strict about kernel regressions: if something
> > stopped working on kernel upgrade, that's a bug.  But I think we really
> > do need to fail attempts to re-export NFS with subtree checking, so
> > we've got to make an exception here.  Re-export is still a bit of an
> > experimental feature, so there may be hiccups like this.
> >
> > --b.