Why do very few filesystems have umount helpers

Why do very few filesystems have umount helpers

[Steve French]

I noticed that nfs has a umount helper (/sbin/umount.nfs) as does hfs
(as does /sbin/umount.udisks2).  Any ideas why those are the only
three filesystems have them but other fs don't?

Since umount does not notify the filesystem on unmount until
references are closed (unless you do "umount --force") and therefore
the filesystem is only notified at kill_sb time, an easier approach to
fixing some of the problems where resources are kept around too long
(e.g. cached handles or directory entries etc. or references on the
mount are held) may be to add a mount helper which notifies the fs
(e.g. via fs specific ioctl) when umount has begun.   That may be an
easier solution that adding a VFS call to notify the fs when umount
begins.   As you can see from fs/namespace.c there is no mount
notification normally (only on "force" unmounts)

        /*
         * If we may have to abort operations to get out of this
         * mount, and they will themselves hold resources we must
         * allow the fs to do things. In the Unix tradition of
         * 'Gee thats tricky lets do it in userspace' the umount_begin
         * might fail to complete on the first run through as other tasks
         * must return, and the like. Thats for the mount program to worry
         * about for the moment.
         */

        if (flags & MNT_FORCE && sb->s_op->umount_begin) {
                sb->s_op->umount_begin(sb);
        }


Any thoughts on why those three fs are the only cases where there are
umount helpers? And why they added them?

I do notice umount failures (which can cause the subsequent mount to
fail) on some of our functional test runs e.g. generic/043 and
generic/044 often fail to Samba with

     QA output created by 043
    +umount: /mnt-local-xfstest/scratch: target is busy.
    +mount error(16): Device or resource busy

Ideas?


-- 
Thanks,

Steve

Re: Why do very few filesystems have umount helpers

[Al Viro]

On Sun, Jul 28, 2024 at 02:09:14PM -0500, Steve French wrote:

> Since umount does not notify the filesystem on unmount until
> references are closed (unless you do "umount --force") and therefore
> the filesystem is only notified at kill_sb time, an easier approach to
> fixing some of the problems where resources are kept around too long
> (e.g. cached handles or directory entries etc. or references on the
> mount are held) may be to add a mount helper which notifies the fs
> (e.g. via fs specific ioctl) when umount has begun.   That may be an
> easier solution that adding a VFS call to notify the fs when umount
> begins.

Huh?

"references on the mount being held" is not something any userland
helpers have a chance to help with.

What exactly gets leaked in your tests?  And what would that userland
helper do when umount happens due to the last process in given namespace
getting killed, for example?  Any unexpected "busy" at umount(2) time
would translate into filesystem instances stuck around (already detached
from any mount trees) for unspecified time; not a good thing, obviously,
and not something a userland helper had a chance to help with...

Details, please.

Re: Why do very few filesystems have umount helpers

[Steve French]

On Sun, Jul 28, 2024 at 7:01 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Sun, Jul 28, 2024 at 02:09:14PM -0500, Steve French wrote:
>
> > Since umount does not notify the filesystem on unmount until
> > references are closed (unless you do "umount --force") and therefore
> > the filesystem is only notified at kill_sb time, an easier approach to
> > fixing some of the problems where resources are kept around too long
> > (e.g. cached handles or directory entries etc. or references on the
> > mount are held) may be to add a mount helper which notifies the fs
> > (e.g. via fs specific ioctl) when umount has begun.   That may be an
> > easier solution that adding a VFS call to notify the fs when umount
> > begins.
>
> "references on the mount being held" is not something any userland
> helpers have a chance to help with.

I don't know the exact reasons why at least three filesystems have umount
helpers but presumably they can issue private ioctls (or equivalent)
to release resources, but I am very curious if their reasons would
overlap any common SMB3.1.1 network mount use cases.

> What exactly gets leaked in your tests?  And what would that userland
> helper do when umount happens due to the last process in given namespace
> getting killed, for example?  Any unexpected "busy" at umount(2) time
> would translate into filesystem instances stuck around (already detached
> from any mount trees) for unspecified time; not a good thing, obviously,
> and not something a userland helper had a chance to help with...
>
> Details, please.

There are three things in particular that got me thinking about how
other filesystems handle umount (and whether the umount helper
concept is a bad idea or a good idea for network fs)

1) Better resource usage: network filesystems often have cached
information due to
leases (or 'delegations' in NFS terminology) on files or directory
entries.  Waiting until kill_superblock (rather than when umount began) can
waste resources.  This cached information is not automatically released
when the file or directory is closed (note that "deferred close" of files can
be a huge performance win for network filesystems which support safe
caching via leases/delegations) ... but these caches consume
resources that ideally would be freed when umount starts, but often have to
wait longer until kill_sb is invoked to be freed.  If "umount_begin"
were called always e.g. then (assuming this were not multiple mounts
from the same client that server share) cifs.ko could
  a) close all deferred network file handles (freeing up some resources)
  b) stop waiting for any pending network i/o requests
  c) mark the tree connection (connection to the server share) as "EXITING"
   so we don't have races sending new i/o operations on that share

2) fixing races between umount and mount:
There are some common test scenarios where we can run a series of
xfstests that will eventually fail (e.g. by the time xfstest runs gets
to 043 or 044
(to Samba server on localhost e.g.) they sometimes hit races which
cause this message:

     QA output created by 043
    +umount: /mnt-local-xfstest/scratch: target is busy.
    +mount error(16): Device or resource busy

but it works fine if delay is inserted between these tests.  I will
try some experiments to
see if changing xfstests to call force unmount which calls "umount_begin" (or
adding a umount wrapper to do the same) also avoids the problem. It
could be that
references may be being held by cifs.ko briefly that are causing the VFS to
think that files are open and not calling into cifs.ko to
kill_superblock. This needs
more investigation but "umount --force" (or equivalent) may help.

3) races in cleaning up directory cache information.  There was a
patch introduced for
periodically cleaning up the directory cache (this is only an issue to
servers like
Windows or NetApp etc. that support directory leases so you don't see
it to Samba
and various other common servers that don't enable directory leases)
that can cause
crashes in unmount (use after free).  I want to try to narrow it down
soon, but it was a little
tricky (and the assumption was that force unmount would avoid the
problem - ie the call to
"umount_begin").  Looks like this patch causes the intermittent umount
crash to Windows
servers:

commit d14de8067e3f9653cdef5a094176d00f3260ab20
Author: Ronnie Sahlberg <lsahlber@redhat.com>
Date:   Thu Jul 6 12:32:24 2023 +1000

    cifs: Add a laundromat thread for cached directories

    and drop cached directories after 30 seconds



Steve

Re: Why do very few filesystems have umount helpers

[Steve French]

And here is a recent bugzilla related to umount issues (although this
could be distinct from the three other umount issues I mentioned
above)

https://bugzilla.kernel.org/show_bug.cgi?id=219097

On Sun, Jul 28, 2024 at 11:16 PM Steve French <smfrench@gmail.com> wrote:
>
> On Sun, Jul 28, 2024 at 7:01 PM Al Viro <viro@zeniv.linux.org.uk> wrote:
> >
> > On Sun, Jul 28, 2024 at 02:09:14PM -0500, Steve French wrote:
> >
> > > Since umount does not notify the filesystem on unmount until
> > > references are closed (unless you do "umount --force") and therefore
> > > the filesystem is only notified at kill_sb time, an easier approach to
> > > fixing some of the problems where resources are kept around too long
> > > (e.g. cached handles or directory entries etc. or references on the
> > > mount are held) may be to add a mount helper which notifies the fs
> > > (e.g. via fs specific ioctl) when umount has begun.   That may be an
> > > easier solution that adding a VFS call to notify the fs when umount
> > > begins.
> >
> > "references on the mount being held" is not something any userland
> > helpers have a chance to help with.
>
> I don't know the exact reasons why at least three filesystems have umount
> helpers but presumably they can issue private ioctls (or equivalent)
> to release resources, but I am very curious if their reasons would
> overlap any common SMB3.1.1 network mount use cases.
>
> > What exactly gets leaked in your tests?  And what would that userland
> > helper do when umount happens due to the last process in given namespace
> > getting killed, for example?  Any unexpected "busy" at umount(2) time
> > would translate into filesystem instances stuck around (already detached
> > from any mount trees) for unspecified time; not a good thing, obviously,
> > and not something a userland helper had a chance to help with...
> >
> > Details, please.
>
> There are three things in particular that got me thinking about how
> other filesystems handle umount (and whether the umount helper
> concept is a bad idea or a good idea for network fs)
>
> 1) Better resource usage: network filesystems often have cached
> information due to
> leases (or 'delegations' in NFS terminology) on files or directory
> entries.  Waiting until kill_superblock (rather than when umount began) can
> waste resources.  This cached information is not automatically released
> when the file or directory is closed (note that "deferred close" of files can
> be a huge performance win for network filesystems which support safe
> caching via leases/delegations) ... but these caches consume
> resources that ideally would be freed when umount starts, but often have to
> wait longer until kill_sb is invoked to be freed.  If "umount_begin"
> were called always e.g. then (assuming this were not multiple mounts
> from the same client that server share) cifs.ko could
>   a) close all deferred network file handles (freeing up some resources)
>   b) stop waiting for any pending network i/o requests
>   c) mark the tree connection (connection to the server share) as "EXITING"
>    so we don't have races sending new i/o operations on that share
>
> 2) fixing races between umount and mount:
> There are some common test scenarios where we can run a series of
> xfstests that will eventually fail (e.g. by the time xfstest runs gets
> to 043 or 044
> (to Samba server on localhost e.g.) they sometimes hit races which
> cause this message:
>
>      QA output created by 043
>     +umount: /mnt-local-xfstest/scratch: target is busy.
>     +mount error(16): Device or resource busy
>
> but it works fine if delay is inserted between these tests.  I will
> try some experiments to
> see if changing xfstests to call force unmount which calls "umount_begin" (or
> adding a umount wrapper to do the same) also avoids the problem. It
> could be that
> references may be being held by cifs.ko briefly that are causing the VFS to
> think that files are open and not calling into cifs.ko to
> kill_superblock. This needs
> more investigation but "umount --force" (or equivalent) may help.
>
> 3) races in cleaning up directory cache information.  There was a
> patch introduced for
> periodically cleaning up the directory cache (this is only an issue to
> servers like
> Windows or NetApp etc. that support directory leases so you don't see
> it to Samba
> and various other common servers that don't enable directory leases)
> that can cause
> crashes in unmount (use after free).  I want to try to narrow it down
> soon, but it was a little
> tricky (and the assumption was that force unmount would avoid the
> problem - ie the call to
> "umount_begin").  Looks like this patch causes the intermittent umount
> crash to Windows
> servers:
>
> commit d14de8067e3f9653cdef5a094176d00f3260ab20
> Author: Ronnie Sahlberg <lsahlber@redhat.com>
> Date:   Thu Jul 6 12:32:24 2023 +1000
>
>     cifs: Add a laundromat thread for cached directories
>
>     and drop cached directories after 30 seconds
>
>
>
> Steve



-- 
Thanks,

Steve

Re: Why do very few filesystems have umount helpers

[Christian Brauner]

On Sun, Jul 28, 2024 at 02:09:14PM GMT, Steve French wrote:
> I noticed that nfs has a umount helper (/sbin/umount.nfs) as does hfs
> (as does /sbin/umount.udisks2).  Any ideas why those are the only
> three filesystems have them but other fs don't?

Helpers such as mount.* or umount.* are used by util-linux. They're not
supposed to be directly used (usually).

For example, umount.udisks talks to the udisks daemon which keeps state
on the block devices it manages and it also cleans up things that were
created (directories etc.) at mount time. Such mounts are usually marked
e.g., via helper=udisks to instruct util-linux to call umount.udisks

Similar things probably apply to the others.

> Since umount does not notify the filesystem on unmount until
> references are closed (unless you do "umount --force") and therefore
> the filesystem is only notified at kill_sb time, an easier approach to
> fixing some of the problems where resources are kept around too long
> (e.g. cached handles or directory entries etc. or references on the
> mount are held) may be to add a mount helper which notifies the fs
> (e.g. via fs specific ioctl) when umount has begun.   That may be an
> easier solution that adding a VFS call to notify the fs when umount
> begins.   As you can see from fs/namespace.c there is no mount
> notification normally (only on "force" unmounts)

The first step should be to identify what exactly keeps your mount busy
in generic/044 and generic/043. If you don't know what the cause of this
is no notification from VFS will help you. My guess is that this ends up
being fixable in cifs.

Re: Why do very few filesystems have umount helpers

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 2963 bytes --]

On Jul 28, 2024, at 1:09 PM, Steve French <smfrench@gmail.com> wrote:
> 
> I noticed that nfs has a umount helper (/sbin/umount.nfs) as does hfs
> (as does /sbin/umount.udisks2).  Any ideas why those are the only
> three filesystems have them but other fs don't?

I think one of the reasons for this is that *unmount* helpers have been
available only for a relatively short time compared to *mount* helpers,
so not nearly as many filesystems have created them (though I'd wanted
this functionality on occasion over the years).

> Since umount does not notify the filesystem on unmount until
> references are closed (unless you do "umount --force") and therefore
> the filesystem is only notified at kill_sb time, an easier approach to
> fixing some of the problems where resources are kept around too long
> (e.g. cached handles or directory entries etc. or references on the
> mount are held) may be to add a mount helper which notifies the fs
> (e.g. via fs specific ioctl) when umount has begun.   That may be an
> easier solution that adding a VFS call to notify the fs when umount
> begins.

I don't think that would be easier in the end, since you still need to
change the kernel code to handle the new ioctl, and coordinate the umount
helper to call this ioctl in userspace, rather than just have the kernel
notify that an unmount is being called.

One potential issue is with namespaces and virtualization, which may
"unmount" the filesystem pretty frequently, even though the filesystem
on the host is kept mounted the whole time.  If the host filesystem is
flushing its cache "in anticipation" of being fully unmounted, but is
actually servicing dozens of guests, then it could be a significant hit
to system performance each time a guest/container starts and stops.

Cheers, Andreas

> As you can see from fs/namespace.c there is no mount
> notification normally (only on "force" unmounts)
> 
>        /*
>         * If we may have to abort operations to get out of this
>         * mount, and they will themselves hold resources we must
>         * allow the fs to do things. In the Unix tradition of
>         * 'Gee thats tricky lets do it in userspace' the umount_begin
>         * might fail to complete on the first run through as other tasks
>         * must return, and the like. Thats for the mount program to worry
>         * about for the moment.
>         */
> 
>        if (flags & MNT_FORCE && sb->s_op->umount_begin) {
>                sb->s_op->umount_begin(sb);
>        }
> 
> 
> Any thoughts on why those three fs are the only cases where there are
> umount helpers? And why they added them?
> 
> I do notice umount failures (which can cause the subsequent mount to
> fail) on some of our functional test runs e.g. generic/043 and
> generic/044 often fail to Samba with
> 
>     QA output created by 043
>    +umount: /mnt-local-xfstest/scratch: target is busy.
>    +mount error(16): Device or resource busy


Cheers, Andreas






[-- Attachment #2: Message signed with OpenPGP --]
[-- Type: application/pgp-signature, Size: 873 bytes --]

Re: Why do very few filesystems have umount helpers

[Steve French]

> umount.udisks talks to the udisks daemon which keeps state
> on the block devices it manages and it also cleans up things that were
> created (directories etc.) at mount time

That does sound similar to the problem that some network fs face.
How to cleanup resources (e.g. cached metadata) better at umount time
(since kill_sb can take a while to be invoked)

> The first step should be to identify what exactly keeps your mount busy
> in generic/044 and generic/043.

That is a little tricky to debug (AFAIK no easy way to tell exactly which
reference is preventing the VFS from proceeding with the umount and
calling kill_sb).  My best guess is something related to deferred close
(cached network file handles) that had a brief refcount on
something being checked by umount, but when I experimented with
deferred close settings that did not seem to affect the problem so
looking for other possible causes.

I just did a quick experiment by adding a 1 second wait inside umount
and confirmed that that does fix it for those two tests when mounted to Samba,
but not clear why the slight delay in umount helps as there is no pending
network traffic at that point.

On Mon, Jul 29, 2024 at 4:50 AM Christian Brauner <brauner@kernel.org> wrote:
>
> On Sun, Jul 28, 2024 at 02:09:14PM GMT, Steve French wrote:
> > I noticed that nfs has a umount helper (/sbin/umount.nfs) as does hfs
> > (as does /sbin/umount.udisks2).  Any ideas why those are the only
> > three filesystems have them but other fs don't?
>
> Helpers such as mount.* or umount.* are used by util-linux. They're not
> supposed to be directly used (usually).
>
> For example, umount.udisks talks to the udisks daemon which keeps state
> on the block devices it manages and it also cleans up things that were
> created (directories etc.) at mount time. Such mounts are usually marked
> e.g., via helper=udisks to instruct util-linux to call umount.udisks
>
> Similar things probably apply to the others.
>
> > Since umount does not notify the filesystem on unmount until
> > references are closed (unless you do "umount --force") and therefore
> > the filesystem is only notified at kill_sb time, an easier approach to
> > fixing some of the problems where resources are kept around too long
> > (e.g. cached handles or directory entries etc. or references on the
> > mount are held) may be to add a mount helper which notifies the fs
> > (e.g. via fs specific ioctl) when umount has begun.   That may be an
> > easier solution that adding a VFS call to notify the fs when umount
> > begins.   As you can see from fs/namespace.c there is no mount
> > notification normally (only on "force" unmounts)
>
> The first step should be to identify what exactly keeps your mount busy
> in generic/044 and generic/043. If you don't know what the cause of this
> is no notification from VFS will help you. My guess is that this ends up
> being fixable in cifs.



-- 
Thanks,

Steve

Re: Why do very few filesystems have umount helpers

[Steve French]

>  even though the filesystem on the host is kept mounted the whole time.  If the host filesystem
> is flushing its cache "in anticipation" of being fully unmounted, but is actually servicing dozens
> of guests, then it could be a significant hit to system performance

The good news (at least with cifs.ko) is that when multiple
superblocks get mounted on the same share
("tree connection" in cifs.ko) then the cached files (deferred close
for network handles with file leases) and
directory entries (with directory leases) won't get freed until the
last superblock mounted to that //server/sharename
is unmounted.


On Mon, Jul 29, 2024 at 12:31 PM Andreas Dilger <adilger@dilger.ca> wrote:
>
> On Jul 28, 2024, at 1:09 PM, Steve French <smfrench@gmail.com> wrote:
> >
> > I noticed that nfs has a umount helper (/sbin/umount.nfs) as does hfs
> > (as does /sbin/umount.udisks2).  Any ideas why those are the only
> > three filesystems have them but other fs don't?
>
> I think one of the reasons for this is that *unmount* helpers have been
> available only for a relatively short time compared to *mount* helpers,
> so not nearly as many filesystems have created them (though I'd wanted
> this functionality on occasion over the years).
>
> > Since umount does not notify the filesystem on unmount until
> > references are closed (unless you do "umount --force") and therefore
> > the filesystem is only notified at kill_sb time, an easier approach to
> > fixing some of the problems where resources are kept around too long
> > (e.g. cached handles or directory entries etc. or references on the
> > mount are held) may be to add a mount helper which notifies the fs
> > (e.g. via fs specific ioctl) when umount has begun.   That may be an
> > easier solution that adding a VFS call to notify the fs when umount
> > begins.
>
> I don't think that would be easier in the end, since you still need to
> change the kernel code to handle the new ioctl, and coordinate the umount
> helper to call this ioctl in userspace, rather than just have the kernel
> notify that an unmount is being called.
>
> One potential issue is with namespaces and virtualization, which may
> "unmount" the filesystem pretty frequently, even though the filesystem
> on the host is kept mounted the whole time.  If the host filesystem is
> flushing its cache "in anticipation" of being fully unmounted, but is
> actually servicing dozens of guests, then it could be a significant hit
> to system performance each time a guest/container starts and stops.
>
> Cheers, Andreas
>
> > As you can see from fs/namespace.c there is no mount
> > notification normally (only on "force" unmounts)
> >
> >        /*
> >         * If we may have to abort operations to get out of this
> >         * mount, and they will themselves hold resources we must
> >         * allow the fs to do things. In the Unix tradition of
> >         * 'Gee thats tricky lets do it in userspace' the umount_begin
> >         * might fail to complete on the first run through as other tasks
> >         * must return, and the like. Thats for the mount program to worry
> >         * about for the moment.
> >         */
> >
> >        if (flags & MNT_FORCE && sb->s_op->umount_begin) {
> >                sb->s_op->umount_begin(sb);
> >        }
> >
> >
> > Any thoughts on why those three fs are the only cases where there are
> > umount helpers? And why they added them?
> >
> > I do notice umount failures (which can cause the subsequent mount to
> > fail) on some of our functional test runs e.g. generic/043 and
> > generic/044 often fail to Samba with
> >
> >     QA output created by 043
> >    +umount: /mnt-local-xfstest/scratch: target is busy.
> >    +mount error(16): Device or resource busy
>
>
> Cheers, Andreas
>
>
>
>
>


-- 
Thanks,

Steve

Re: Why do very few filesystems have umount helpers

[Steve French]

On Mon, Jul 29, 2024 at 4:50 AM Christian Brauner <brauner@kernel.org> wrote:
On Mon, Jul 29, 2024 at 12:33 PM Steve French <smfrench@gmail.com> wrote:
> > The first step should be to identify what exactly keeps your mount busy
> > in generic/044 and generic/043.
>
> That is a little tricky to debug (AFAIK no easy way to tell exactly which
> reference is preventing the VFS from proceeding with the umount and
> calling kill_sb).  My best guess is something related to deferred close
> (cached network file handles) that had a brief refcount on
> something being checked by umount, but when I experimented with
> deferred close settings that did not seem to affect the problem so
> looking for other possible causes.
>
> I just did a quick experiment by adding a 1 second wait inside umount
> and confirmed that that does fix it for those two tests when mounted to Samba,
> but not clear why the slight delay in umount helps as there is no pending
> network traffic at that point.

I did some more experimentation and it looks like the umount problem
with those two xfstests to Samba is related to IOC_SHUTDOWN.
If I return EOPNOTSUPP on IOC_SHUTDOWN
then the 1 second delay in umount is not necessary - so something that
happens after IOC_SHUTDOWN races with umount (thus the 1 second delay
that I tried as a quick experiment fixes it indirectly) in this
testcase (although
apparently this race between IOC_SHUTDOWN and umount is not an issue
to some other servers but is reproducible to Samba and ksmbd (at least
in some easy to setup configurations)



-- 
Thanks,

Steve

Re: Why do very few filesystems have umount helpers

[Dave Chinner]

On Mon, Jul 29, 2024 at 04:50:27PM -0500, Steve French wrote:
> On Mon, Jul 29, 2024 at 4:50 AM Christian Brauner <brauner@kernel.org> wrote:
> On Mon, Jul 29, 2024 at 12:33 PM Steve French <smfrench@gmail.com> wrote:
> > > The first step should be to identify what exactly keeps your mount busy
> > > in generic/044 and generic/043.
> >
> > That is a little tricky to debug (AFAIK no easy way to tell exactly which
> > reference is preventing the VFS from proceeding with the umount and
> > calling kill_sb).  My best guess is something related to deferred close
> > (cached network file handles) that had a brief refcount on
> > something being checked by umount, but when I experimented with
> > deferred close settings that did not seem to affect the problem so
> > looking for other possible causes.
> >
> > I just did a quick experiment by adding a 1 second wait inside umount
> > and confirmed that that does fix it for those two tests when mounted to Samba,
> > but not clear why the slight delay in umount helps as there is no pending
> > network traffic at that point.
> 
> I did some more experimentation and it looks like the umount problem
> with those two xfstests to Samba is related to IOC_SHUTDOWN.
> If I return EOPNOTSUPP on IOC_SHUTDOWN
> then the 1 second delay in umount is not necessary - so something that
> happens after IOC_SHUTDOWN races with umount (thus the 1 second delay
> that I tried as a quick experiment fixes it indirectly) in this
> testcase (although
> apparently this race between IOC_SHUTDOWN and umount is not an issue
> to some other servers but is reproducible to Samba and ksmbd (at least
> in some easy to setup configurations)

So you've likely got a race condition where something takes longer
when the shutdown flag is set then when the filesystem is operating
normally. There's not a lot in the CIFS code that pays attention to
the shutdown flag - almost all of them are aborting front end
(syscall) operations before they are started.

The only back end check appears to be in cifs_issue_write(). Perhaps
that is failing to wake the request queue when it is being failed
with -EIO on a shutdown, and so it takes some time for something
else to wake it up and empty it and complete the pending writes
before the fs can be unmounted...

-Dave.
-- 
Dave Chinner
david@fromorbit.com