autofs map updates not working properly for multi-mounts

autofs map updates not working properly for multi-mounts

[Jeff Moyer]

Hi, Ian, list,

I've been doing some testing of the map update code, and found that the
following configuration does not work as it should:

auto.master contains:

/testing    /etc/auto.testing

/etc/auto.testing contains:

foo     -ro,hard,intr \
        /dir/bar        server:/export/a \
        /dir/baz        server:/export/b

Given that we are running a current version of the automounter, and we are
up to date on kernel patches, we should be able to update the "foo" entry
to list another subdir.  After this update occurs, any attempt to access
the newly created directory should succeed.  However, it does not because
lookup_file.c:lookup_mount doesn't find a cache hit for
foo/dir/<new_entry> (in this case it is foo/dir/aaa).

Here is the output from a debug log:

kernel: pid 2492: autofs4_root_lookup: name = aaa
kernel: pid 2492: autofs4_root_lookup: pid = 2492, pgrp = 2492, catatonic =
0, oz_mode = 0
kernel: pid 2492: autofs4_revalidate: dentry=c1b2d700 aaa flags=0
kernel: pid 2492: try_to_fill_entry: dentry=c1b2d700 aaa ino=00000000
kernel: pid 2492: try_to_fill_entry: waiting for mount name=aaa
kernel: pid 2492: autofs4_wait: new wait id = 0x00000002, name =
foo/dir/aaa, nfy=1
kernel: pid 2492: autofs4_notify: wait id = 0x00000002, name = foo/dir/aaa,
type=0
automount[2095]: handle_packet: type = 0
automount[2095]: handle_packet_missing: token 2, name foo/dir/aaa
automount[2095]: attempting to mount entry /testing/foo/dir/aaa
automount[2493]: ret = 8
kernel: pid 2095: autofs4_revalidate: dentry=c1daef00 foo flags=4
automount[2095]: sig 1 switching from 1 to 4
kernel: pid 2095: autofs4_revalidate: dentry=c1daee80 dir flags=4
automount[2493]: failed to mount /testing/foo/dir/aaa

Ian, I'll continue to look into this, but figured I'd report the issue here
in case you had any suggestions.

Note that I have reproduced this with 4.1.4_beta1.  Also note that I've
only tested this with file maps.  Other lookup modules may or may not
exhibit the same behaviour.

Thanks!

Jeff

Re: autofs map updates not working properly for multi-mounts

[Ian Kent]

On Tue, 8 Feb 2005, Jeff Moyer wrote:

> Hi, Ian, list,
> 
> I've been doing some testing of the map update code, and found that the
> following configuration does not work as it should:
> 
> auto.master contains:
> 
> /testing    /etc/auto.testing
> 
> /etc/auto.testing contains:
> 
> foo     -ro,hard,intr \
>         /dir/bar        server:/export/a \
>         /dir/baz        server:/export/b
> 
> Given that we are running a current version of the automounter, and we are
> up to date on kernel patches, we should be able to update the "foo" entry
> to list another subdir.  After this update occurs, any attempt to access
> the newly created directory should succeed.  However, it does not because
> lookup_file.c:lookup_mount doesn't find a cache hit for
> foo/dir/<new_entry> (in this case it is foo/dir/aaa).
> 
> Here is the output from a debug log:
> 
> kernel: pid 2492: autofs4_root_lookup: name = aaa
> kernel: pid 2492: autofs4_root_lookup: pid = 2492, pgrp = 2492, catatonic =
> 0, oz_mode = 0
> kernel: pid 2492: autofs4_revalidate: dentry=c1b2d700 aaa flags=0
> kernel: pid 2492: try_to_fill_entry: dentry=c1b2d700 aaa ino=00000000
> kernel: pid 2492: try_to_fill_entry: waiting for mount name=aaa
> kernel: pid 2492: autofs4_wait: new wait id = 0x00000002, name =
> foo/dir/aaa, nfy=1
> kernel: pid 2492: autofs4_notify: wait id = 0x00000002, name = foo/dir/aaa,
> type=0
> automount[2095]: handle_packet: type = 0
> automount[2095]: handle_packet_missing: token 2, name foo/dir/aaa
> automount[2095]: attempting to mount entry /testing/foo/dir/aaa
> automount[2493]: ret = 8
> kernel: pid 2095: autofs4_revalidate: dentry=c1daef00 foo flags=4
> automount[2095]: sig 1 switching from 1 to 4
> kernel: pid 2095: autofs4_revalidate: dentry=c1daee80 dir flags=4
> automount[2493]: failed to mount /testing/foo/dir/aaa
> 
> Ian, I'll continue to look into this, but figured I'd report the issue here
> in case you had any suggestions.

I'll try and get to the bottom of this as well.

This shouldn't happen. The mount should occur as soon as the path walk 
hits foo and all mounts from the multi-mount should be mounted then?

> 
> Note that I have reproduced this with 4.1.4_beta1.  Also note that I've
> only tested this with file maps.  Other lookup modules may or may not
> exhibit the same behaviour.

It's probably like it in all.

Ian

Re: autofs map updates not working properly for multi-mounts

[raven]

Sorry about this rant but after thinking about it you've hit on a sore 
point.

On Tue, 8 Feb 2005, Jeff Moyer wrote:

> Hi, Ian, list,
>
> I've been doing some testing of the map update code, and found that the
> following configuration does not work as it should:
>
> auto.master contains:
>
> /testing    /etc/auto.testing
>
> /etc/auto.testing contains:
>
> foo     -ro,hard,intr \
>        /dir/bar        server:/export/a \
>        /dir/baz        server:/export/b
>
> Given that we are running a current version of the automounter, and we are
> up to date on kernel patches, we should be able to update the "foo" entry
> to list another subdir.  After this update occurs, any attempt to access
> the newly created directory should succeed.  However, it does not because
> lookup_file.c:lookup_mount doesn't find a cache hit for
> foo/dir/<new_entry> (in this case it is foo/dir/aaa).

I've seen this ocassionally in testing but could never work out exatly 
what was causing it.

Thanks to your excelent problem description I now understand what is 
happening. The description has given me more valueable information on 
a significant outstanding problem with autofs. Thanks Jeff.

So, to the description of the way autofs handles multi-mount entries.

Multi-mount map entries must be treated as a single unit.

This is so because nested mounts are allowed within them.

Consider

foo	-rw \
 	/one/a	server:/one/a \
 	/one/b	server:/one/b \
 	/one/b/c server:/one/b/c

If this isn't handled as a single unit then it's possible for /one/b/c to 
be mounted before /one/b. This could happen due to a map update, for 
example.

So, just as with individual mount entries, it can't be updated until it's 
umounted and mounted again.

The handling of lazy mounting of multi-mounts is the second biggest 
problem that I want to solve after direct mounts and a problem that I have 
struggled with for a long time. And the one problem I have the least ideas 
on how to solve.

So what is happening.

The format of a multi-mount is

key [-options] [[/] mount-entry] [path [-options] mount-entry] and so on

where the brackets signify optional bits.

When the mount "key" is triggered all of the entries are mounted.
When all the entries have been idle at least the timeout period they are 
all umounted.

No surprises here.

In the example you are working with there is no "/" entry so lookups 
in non-mountpoint directories trigger VFS callbacks to the autofs4 kernel 
module.

It doesn't know that it's a multi-mount (it just gets called by the VFS 
to carry out VFS operations it's asked to be called for) and so uses it's 
rules to decide when to trigger a mount.

A mount is triggered if it finds an empty directory that is not 
already a mounted and then sends a mount request to the daemon.

So the key in the example below that is sent to the daemon is 
"foo/dir/aaa" when in fact the key for the map entry is "foo".

Consequently it's not found in the map.

But it gets worse. Having passed over foo in the path lookup it's already 
too late to take action. If we want things to work we must notify the 
daemon when we walk over foo not afterwards.

So this is not a simple problem but I hope to stand corrected.

The question then is can we improve the situation?

Prossibly yes, perhaps by flaging multi-mount entries so that the correct 
key is sent to the daemon at the right time would offer some improvement.

Our inability to adequately handle the nested mounts has always been a 
show stopper in this case.

Once again I have some ideas and a tiny bit of code from a false start but 
I still don't have a plan for what I consider proper solution.

So the bottom line at the moment is "multi-mount entries must be handled a 
a single mount entry". So they can't be dynamically updated until the 
next time they are mounted following the update.

>
> Here is the output from a debug log:
>
> kernel: pid 2492: autofs4_root_lookup: name = aaa
> kernel: pid 2492: autofs4_root_lookup: pid = 2492, pgrp = 2492, catatonic =
> 0, oz_mode = 0
> kernel: pid 2492: autofs4_revalidate: dentry=c1b2d700 aaa flags=0
> kernel: pid 2492: try_to_fill_entry: dentry=c1b2d700 aaa ino=00000000
> kernel: pid 2492: try_to_fill_entry: waiting for mount name=aaa
> kernel: pid 2492: autofs4_wait: new wait id = 0x00000002, name =
> foo/dir/aaa, nfy=1
> kernel: pid 2492: autofs4_notify: wait id = 0x00000002, name = foo/dir/aaa,
> type=0
> automount[2095]: handle_packet: type = 0
> automount[2095]: handle_packet_missing: token 2, name foo/dir/aaa
> automount[2095]: attempting to mount entry /testing/foo/dir/aaa
> automount[2493]: ret = 8
> kernel: pid 2095: autofs4_revalidate: dentry=c1daef00 foo flags=4
> automount[2095]: sig 1 switching from 1 to 4
> kernel: pid 2095: autofs4_revalidate: dentry=c1daee80 dir flags=4
> automount[2493]: failed to mount /testing/foo/dir/aaa
>
> Ian, I'll continue to look into this, but figured I'd report the issue here
> in case you had any suggestions.
>
> Note that I have reproduced this with 4.1.4_beta1.  Also note that I've
> only tested this with file maps.  Other lookup modules may or may not
> exhibit the same behaviour.
>
> Thanks!
>
> Jeff
>

Re: Re: autofs map updates not working properly for multi-mounts

[Mike Waychison]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

raven@themaw.net wrote:

> So the bottom line at the moment is "multi-mount entries must be handled
> a a single mount entry". So they can't be dynamically updated until the
> next time they are mounted following the update.
> 

This is a perfectly fair assumption and is handled the same way on other
platforms.

- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE:  The opinions expressed in this email are held by me,
and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFCEQuvdQs4kOxk3/MRAoWNAJ9crYNXPjvM90EdaNxLkHPCaoVU1wCdEm0A
K+voKmkaB9DWhcCjaC/Ub50=
=IV4t
-----END PGP SIGNATURE-----

Re: Re: autofs map updates not working properly for multi-mounts

[Jim Carter]

Maybe I shouldn't be doing the "talk without patching" thing, but I'm not 
exactly at guru level with the autofs code, so...

On Wed, 9 Feb 2005 raven@themaw.net wrote:

> Multi-mount map entries must be treated as a single unit.
> This is so because nested mounts are allowed within them.
> Consider
> 
> foo	-rw \
> /one/a	server:/one/a \
> /one/b	server:/one/b \
> /one/b/c server:/one/b/c
> 
> If this isn't handled as a single unit then it's possible for /one/b/c to be
> mounted before /one/b. This could happen due to a map update, for example.

I've always wondered about that.  To mount on /net/foo/one/b/c, don't we 
have to readdir /net/foo/one/b first, find the inode of mount point c, and 
stat it?  That would (should :-) trigger an attempt by the module to have 
/net/foo/one/b mounted.  

Or is it a single thread issue, so the daemon can't begin mounting 
/net/foo/one/b until it's finished mounting /net/foo/one/b/c (which it 
can't do until after /net/foo/one/b is mounted)?

> So, just as with individual mount entries, it can't be updated until it's
> umounted and mounted again.

I think you're saying that if the new map says to mount /net/foo/one/b/c on 
a different mount point (or to obtain it from a different place), the 
daemon will remember the new map but (obviously) can't make it happen until 
the old mount times out.  If ever.  Using the "multiple uni-mounts" model 
described below, actually there should be no problem obeying new map 
content for directories that aren't mounted; it's just not possible to get 
rid of old content immediately.

> So the key in the example below that is sent to the daemon is "foo/dir/aaa"
> when in fact the key for the map entry is "foo".
> 
> Consequently it's not found in the map.

Hmm.  It sounds like the map lookup algorithm needs to have foo/dir/aaa 
available as a key and not just the first word of the map row, "foo".  In
other words, multiple keys point to one map row.  

Alternatively, you might think of transforming the multi-mount into a set 
of uni-mounts.  The example above would turn into:

foo		mkdir but don't mount anything
foo/one		mkdir but don't mount anything
foo/one/a	-rw	server:/one/a
foo/one/b	-rw	server:/one/b
foo/one/b/c	-rw	server:/one/b/c

Not that you would accept this syntax from a map file, but the term on the
left is just what the module would send up, and the term on the right is
what the daemon needs to do when it sees that key.

Does any of this make any sense?


James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: Re: autofs map updates not working properly for multi-mounts

[Ian Kent]

On Thu, 17 Feb 2005, Jim Carter wrote:

> Maybe I shouldn't be doing the "talk without patching" thing, but I'm not 
> exactly at guru level with the autofs code, so...
> 
> On Wed, 9 Feb 2005 raven@themaw.net wrote:
> 
> > Multi-mount map entries must be treated as a single unit.
> > This is so because nested mounts are allowed within them.
> > Consider
> > 
> > foo	-rw \
> > /one/a	server:/one/a \
> > /one/b	server:/one/b \
> > /one/b/c server:/one/b/c
> > 
> > If this isn't handled as a single unit then it's possible for /one/b/c to be
> > mounted before /one/b. This could happen due to a map update, for example.
> 
> I've always wondered about that.  To mount on /net/foo/one/b/c, don't we 
> have to readdir /net/foo/one/b first, find the inode of mount point c, and 
> stat it?  That would (should :-) trigger an attempt by the module to have 
> /net/foo/one/b mounted.  

Yes. But if lazy mounting is used that would trigger a mount of 
/net/foo/one/b and autofs would never see the mount request for 
/net/foo/one/b/c as it is now in a different filesystem.

> 
> Or is it a single thread issue, so the daemon can't begin mounting 
> /net/foo/one/b until it's finished mounting /net/foo/one/b/c (which it 
> can't do until after /net/foo/one/b is mounted)?

No. Just the reverse. Nesting order has to be preserved to maintain any 
sanity at all.

> 
> > So, just as with individual mount entries, it can't be updated until it's
> > umounted and mounted again.
> 
> I think you're saying that if the new map says to mount /net/foo/one/b/c on 
> a different mount point (or to obtain it from a different place), the 
> daemon will remember the new map but (obviously) can't make it happen until 
> the old mount times out.  If ever.  Using the "multiple uni-mounts" model 
> described below, actually there should be no problem obeying new map 
> content for directories that aren't mounted; it's just not possible to get 
> rid of old content immediately.
> 
> > So the key in the example below that is sent to the daemon is "foo/dir/aaa"
> > when in fact the key for the map entry is "foo".
> > 
> > Consequently it's not found in the map.
> 
> Hmm.  It sounds like the map lookup algorithm needs to have foo/dir/aaa 
> available as a key and not just the first word of the map row, "foo".  In
> other words, multiple keys point to one map row.  
> 
> Alternatively, you might think of transforming the multi-mount into a set 
> of uni-mounts.  The example above would turn into:
> 
> foo		mkdir but don't mount anything
> foo/one		mkdir but don't mount anything
> foo/one/a	-rw	server:/one/a
> foo/one/b	-rw	server:/one/b
> foo/one/b/c	-rw	server:/one/b/c
> 
> Not that you would accept this syntax from a map file, but the term on the
> left is just what the module would send up, and the term on the right is
> what the daemon needs to do when it sees that key.

Been there, done that.
Never completed the implementation.

The approach does have merit but not without some difficulty.
I'll get back to it sooner or later.

What if foo/one/b is not initially present and is added later and 
foo/one/b/c is already mounted!
 
> 
> Does any of this make any sense?

Yep. It's pretty much the only way I think I can get it to work but there 
are some big thorns.

Ian