Using findmnt in the presence of a stale NFS mount

Using findmnt in the presence of a stale NFS mount

[Eric Rannaud]

Hi,

On Arch Linux, various package install scripts use findmnt to check
the status of /boot or /usr. However, if there exists a stale NFS
mount on the system (for instance on /mnt), findmnt hangs when used in
commands such as:

findmnt -snero fstype --tab-file '/etc/fstab' -T '/usr'
findmnt -s /boot
findmnt -s /usr

(note that on my system, /usr is not a mount point of any kind, while
/boot is ext4. /mnt is the stale NFS mount)

This older Arch bug https://bugs.archlinux.org/task/36629 refers to
the problem, but as far as I can tell the problem still exists in the
same form (at the time, the package maintainer added the -T option to
the command "findmnt -snero fstype --tab-file '/etc/fstab' -T '/usr'",
but I don't believe it changed anything).

In any case, from having a look at the code on the latest git (I can
reproduce the hang with 710ed55dcd), I believe the problem comes from
the following sequence.

When used with the -s|--fstab flag, findmnt goes through the list of
mount points in the tab file, and compares them with the target
specified, say '/usr'.

The comparison is done using mnt_fs_match_target() in
findmnt.c:match_func(). mnt_fs_match_target() uses mnt_resolve_path()
on both target ('/usr') and
fs->target, which uses canonicalize_path(fs->target), which uses
realpath(fs->target). As the fs->target is the stale NFS mount point,
this last call hangs in a lstat().

While it is clear that a system with a stale NFS mount point is a
minefield, would it be possible to avoid the call to
"realpath(fs->target)" (maybe controlled by a command line option)?

As it is, it is impossible to upgrade an Arch Linux install when there
is a stale NFS mount anywhere on the system (and listed in fstab).

Thanks,
Eric

Re: Using findmnt in the presence of a stale NFS mount

[Karel Zak]

On Tue, Jun 10, 2014 at 08:03:35AM -0700, Eric Rannaud wrote:
> findmnt -snero fstype --tab-file '/etc/fstab' -T '/usr'
> findmnt -s /boot
> findmnt -s /usr
> 
> (note that on my system, /usr is not a mount point of any kind, while
> /boot is ext4. /mnt is the stale NFS mount)
> 
> This older Arch bug https://bugs.archlinux.org/task/36629 refers to
> the problem, but as far as I can tell the problem still exists in the
> same form (at the time, the package maintainer added the -T option to
> the command "findmnt -snero fstype --tab-file '/etc/fstab' -T '/usr'",
> but I don't believe it changed anything).
> 
> In any case, from having a look at the code on the latest git (I can
> reproduce the hang with 710ed55dcd), I believe the problem comes from
> the following sequence.
> 
> When used with the -s|--fstab flag, findmnt goes through the list of
> mount points in the tab file, and compares them with the target
> specified, say '/usr'.
> 
> The comparison is done using mnt_fs_match_target() in
> findmnt.c:match_func(). mnt_fs_match_target() uses mnt_resolve_path()
> on both target ('/usr') and
> fs->target, which uses canonicalize_path(fs->target), which uses
> realpath(fs->target). As the fs->target is the stale NFS mount point,
> this last call hangs in a lstat().
> 
> While it is clear that a system with a stale NFS mount point is a
> minefield, would it be possible to avoid the call to
> "realpath(fs->target)" (maybe controlled by a command line option)?
> 
> As it is, it is impossible to upgrade an Arch Linux install when there
> is a stale NFS mount anywhere on the system (and listed in fstab).

 You can try to  exclude NFS at all by --types nonfs, but maybe we can
 add option --nocanonicalize. I'll try to play with that tomorrow. 

    Karel


-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Using findmnt in the presence of a stale NFS mount

[Karel Zak]

On Mon, Jun 16, 2014 at 03:28:16PM +0200, Karel Zak wrote:
> > As it is, it is impossible to upgrade an Arch Linux install when there
> > is a stale NFS mount anywhere on the system (and listed in fstab).
> 
>  You can try to  exclude NFS at all by --types nonfs, but maybe we can
>  add option --nocanonicalize. I'll try to play with that tomorrow. 

OK, tomorrow is now ;-) (the option added)

The problem is that this is not commonly usable solution, because it
very significantly affects the way how libmount search in fstab,
mountinfo etc. The option disallow to reliable compare symlinks, tags etc.

It would be better to have NFS specific solution, or in the
update/install script use --types nonfs and when only if not found
required directories try fallback solution with --nocanonicalize or
so.

    Karel
-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Using findmnt in the presence of a stale NFS mount

[Eric Rannaud]

Hi Karel,

Thanks for having a look.

I tried --types nonfs, and indeed it works. But that does mean
open-coding the don't-step-on-a-stale-nfs-mount logic a bit
everywhere.

Another solution would be to change canonicalize_path(): before
calling realpath(fs->target), it would first check if fs->target is
otherwise listed in mtab (using the comparison
strcmp(from_mtab_fs->target, fs->target), not recursively using
mnt_fs_target_match()). If found in mtab, canonicalize_path() knows
that fs->target is not a symlink and it does not need to call
realpath(fs->target), and it won't blow up.

There is no need for a command line option: canonicalize_path() can do
this all the time, it will not change its semantics. However, findmnt
would have to systematically read /proc/mounts or mtab in an
additional secondary structure, even if --fstab is specified. That
requires changing findmnt's logic quite a bit, however, as currently
it doesn't read mtab if --fstab is specified.

If you wish, I can have a crack at writing such a patch.

Thanks,
Eric


On Mon, Jun 16, 2014 at 7:00 AM, Karel Zak <kzak@redhat.com> wrote:
> On Mon, Jun 16, 2014 at 03:28:16PM +0200, Karel Zak wrote:
>> > As it is, it is impossible to upgrade an Arch Linux install when there
>> > is a stale NFS mount anywhere on the system (and listed in fstab).
>>
>>  You can try to  exclude NFS at all by --types nonfs, but maybe we can
>>  add option --nocanonicalize. I'll try to play with that tomorrow.
>
> OK, tomorrow is now ;-) (the option added)
>
> The problem is that this is not commonly usable solution, because it
> very significantly affects the way how libmount search in fstab,
> mountinfo etc. The option disallow to reliable compare symlinks, tags etc.
>
> It would be better to have NFS specific solution, or in the
> update/install script use --types nonfs and when only if not found
> required directories try fallback solution with --nocanonicalize or
> so.
>
>     Karel
> --
>  Karel Zak  <kzak@redhat.com>
>  http://karelzak.blogspot.com

Re: Using findmnt in the presence of a stale NFS mount

[Karel Zak]

On Mon, Jun 16, 2014 at 09:02:14AM -0700, Eric Rannaud wrote:
> Another solution would be to change canonicalize_path(): before
> calling realpath(fs->target), it would first check if fs->target is
> otherwise listed in mtab (using the comparison
> strcmp(from_mtab_fs->target, fs->target), not recursively using
> mnt_fs_target_match()). If found in mtab, canonicalize_path() knows
> that fs->target is not a symlink and it does not need to call
> realpath(fs->target), and it won't blow up.
> 
> There is no need for a command line option: canonicalize_path() can do
> this all the time, it will not change its semantics. However, findmnt

 I like the idea that we can rely on /proc/self/mountinfo and assume
 that all the mountpoints are already canonicalized so we can skip the
 extra canonicalize_path().

 But it would be probably better to extend struct libmnt_cache than
 change too low-level canonicalize_path().

 We can add two functions:

    mnt_cache_set_targets(struct libmnt_cache *cache, struct libmnt_table *mtab);
    mnt_resolve_target(char *path, struct libmnt_cache *cache);

 and check for @path in @mtab before canonicalize_path().

 Then we can use mnt_resolve_target() in mnt_fs_match_target() and
 mnt_table_find_target().
    
 And in findmnt.c all you need is to parse /proc/self/mountinfo and call 
 mnt_cache_set_targets() for the global cache when --target is specified
 (or always?).

> would have to systematically read /proc/mounts or mtab in an
> additional secondary structure, even if --fstab is specified. That
> requires changing findmnt's logic quite a bit, however, as currently
> it doesn't read mtab if --fstab is specified.

 We have to be very careful with /proc/mounts (and mountinfo), unfortunately 
 it has horrible performance impact for kernel on huge servers to generate 
 the /proc files.

> If you wish, I can have a crack at writing such a patch.

 Go ahead.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com