statfs() and autofs

statfs() and autofs

[Chris Walker]

autofs 5.0.3
linux 2.6.18

When statfs() is called on an automounted directory, it returns before
the automount occurs. This results in output like this initially:

% df /home/XXX
Filesystem           1K-blocks      Used Available Use% Mounted on
-                            0         0         0   -  /home

If df is run a second later,

% df /home/XXX
Filesystem           1K-blocks      Used Available Use% Mounted on
XXX:/home/XXX
                      80634688  77952736   2681952  97% /home/XXX

Perhaps there are options of which I'm not aware to mitigate this
behavior.

Apologies if the answer to my query lies elsewhere in the kernel.

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Ian Kent]

On Tue, 2008-07-15 at 13:48 -0700, Chris Walker wrote:
> autofs 5.0.3
> linux 2.6.18
> 
> When statfs() is called on an automounted directory, it returns before
> the automount occurs. This results in output like this initially:

That may be because there are a group of system calls that don't trigger
a mount because we have to avoid mount storms for map entries that use
the browse option. In particular stat(2) and calls that use the same
kernel path lookup flags, such as statfs(2).

Basically, if the mount point directory exists in the autofs file system
but doesn't currently have a mount for some reason (perhaps it wasn't
removed at expire or the entry uses the browse option), these calls will
return the attributes of the autofs directory without performing a
mount.

> 
> % df /home/XXX
> Filesystem           1K-blocks      Used Available Use% Mounted on
> -                            0         0         0   -  /home
> 
> If df is run a second later,
> 
> % df /home/XXX
> Filesystem           1K-blocks      Used Available Use% Mounted on
> XXX:/home/XXX
>                       80634688  77952736   2681952  97% /home/XXX
> 
> Perhaps there are options of which I'm not aware to mitigate this
> behavior.
> 
> Apologies if the answer to my query lies elsewhere in the kernel.
> 

Re: statfs() and autofs

[Jonathan Loran]

[-- Attachment #1.1: Type: text/plain, Size: 1885 bytes --]



Ian Kent wrote:
> On Tue, 2008-07-15 at 13:48 -0700, Chris Walker wrote:
>   
>> autofs 5.0.3
>> linux 2.6.18
>>
>> When statfs() is called on an automounted directory, it returns before
>> the automount occurs. This results in output like this initially:
>>     
>
> That may be because there are a group of system calls that don't trigger
> a mount because we have to avoid mount storms for map entries that use
> the browse option. In particular stat(2) and calls that use the same
> kernel path lookup flags, such as statfs(2).
>
> Basically, if the mount point directory exists in the autofs file system
> but doesn't currently have a mount for some reason (perhaps it wasn't
> removed at expire or the entry uses the browse option), these calls will
> return the attributes of the autofs directory without performing a
> mount.
>
>   
Hi Ian,

I need to say that this basically has to be declared a bug.  I have been 
silent on this until now because I am so over committed with other 
tasks, and haven't had time to test it so cleanly as Chris has.

At my lab we find that on machines that do large batch processing, 
automounted directories often aren't there at the time that they are 
stat-ed.  The workaround has been to put in some nanosleeps to give the 
mount point a chance to mount, but that is very much a hack, since now 
the code authors need to know the point where an automount is joined, or 
they at least have to find it out empirically.  The code they write has 
become unportable and for all we know, incompatible on other systems if 
the autofs mount points are different. 

To throw this in the mix, somehow, our Solaris systems, which share 
identical automount maps, simply never have this issue.  Is there an 
architectural reason for this?  And if so, can we learn how this is so, 
and apply similar logic to the Linux autofs/mount path?

Thanks,

Jon

[-- Attachment #1.2: Type: text/html, Size: 2337 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: statfs() and autofs

[Chris Walker]

Thanks for your response.

In this case, it appears that statsfs triggers the mount, but returns
before the mount completes. stat waits for the automount to complete
before returning.

I would like statfs() to wait--is there a way to make it?

Thanks for all your hard work on this critical piece of software.

From raven [07/15 20:47]:
> 
> On Tue, 2008-07-15 at 13:48 -0700, Chris Walker wrote:
> > autofs 5.0.3
> > linux 2.6.18
> > 
> > When statfs() is called on an automounted directory, it returns before
> > the automount occurs. This results in output like this initially:
> 
> That may be because there are a group of system calls that don't trigger
> a mount because we have to avoid mount storms for map entries that use
> the browse option. In particular stat(2) and calls that use the same
> kernel path lookup flags, such as statfs(2).
> 
> Basically, if the mount point directory exists in the autofs file system
> but doesn't currently have a mount for some reason (perhaps it wasn't
> removed at expire or the entry uses the browse option), these calls will
> return the attributes of the autofs directory without performing a
> mount.
> 
> > 
> > % df /home/XXX
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > -                            0         0         0   -  /home
> > 
> > If df is run a second later,
> > 
> > % df /home/XXX
> > Filesystem           1K-blocks      Used Available Use% Mounted on
> > XXX:/home/XXX
> >                       80634688  77952736   2681952  97% /home/XXX
> > 
> > Perhaps there are options of which I'm not aware to mitigate this
> > behavior.
> > 
> > Apologies if the answer to my query lies elsewhere in the kernel.
> > 
> 

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Ian Kent]

On Tue, 2008-07-15 at 22:30 -0700, Chris Walker wrote:
> Thanks for your response.
> 
> In this case, it appears that statsfs triggers the mount, but returns
> before the mount completes. stat waits for the automount to complete
> before returning.

If statfs really is triggering the mount (not something else you haven't
noticed, maybe another command or program system call) then it should
wait. If your map doesn't use the "browse" option or the --ghost option
then stat and statfs will trigger mounts and wait for them to complete.
That's not to say there aren't any races relating to multiple concurrent
processes accessing the mount points. I'm working on that right now. But
that isn't the situation your describing by the sound of it.

So something else is going on that we're not yet aware of.

> 
> I would like statfs() to wait--is there a way to make it?
> 
> Thanks for all your hard work on this critical piece of software.
> 
> From raven [07/15 20:47]:
> > 
> > On Tue, 2008-07-15 at 13:48 -0700, Chris Walker wrote:
> > > autofs 5.0.3
> > > linux 2.6.18

What patches have you applied to this kernel.
I have a bunch of maintenance and bug fixes that aren't in this kernel.

> > > 
> > > When statfs() is called on an automounted directory, it returns before
> > > the automount occurs. This results in output like this initially:
> > 
> > That may be because there are a group of system calls that don't trigger
> > a mount because we have to avoid mount storms for map entries that use
> > the browse option. In particular stat(2) and calls that use the same
> > kernel path lookup flags, such as statfs(2).
> > 
> > Basically, if the mount point directory exists in the autofs file system
> > but doesn't currently have a mount for some reason (perhaps it wasn't
> > removed at expire or the entry uses the browse option), these calls will
> > return the attributes of the autofs directory without performing a
> > mount.
> > 
> > > 
> > > % df /home/XXX
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > -                            0         0         0   -  /home
> > > 
> > > If df is run a second later,
> > > 
> > > % df /home/XXX
> > > Filesystem           1K-blocks      Used Available Use% Mounted on
> > > XXX:/home/XXX
> > >                       80634688  77952736   2681952  97% /home/XXX
> > > 
> > > Perhaps there are options of which I'm not aware to mitigate this
> > > behavior.
> > > 
> > > Apologies if the answer to my query lies elsewhere in the kernel.
> > > 
> > 
> 

Re: statfs() and autofs

[Jim Carter]

On Tue, 15 Jul 2008, Jonathan Loran wrote:
> Ian Kent wrote:
> > On Tue, 2008-07-15 at 13:48 -0700, Chris Walker wrote:
> > >
> > > When statfs() is called on an automounted directory, it returns before
> > > the automount occurs. This results in output like this initially:

> > That may be because there are a group of system calls that don't trigger
> > a mount because we have to avoid mount storms for map entries that use
> > the browse option. In particular stat(2) and calls that use the same
> > kernel path lookup flags, such as statfs(2).
> 
> I need to say that this basically has to be declared a bug.  

I see this two ways.  First, for the folks who have giant NAS boxes with a 
thousand exported filesysems, if ghost mounts (visibility on the client of 
non-mounted filesystems) are turned on and someone does "ls -F /net/nasbox" 
the resulting mount storm, when it stats every exported directory, is a 
serious pain in the butt, and when ghost mounts (browsing) were introduced 
there was a lot of howling until the mount storms were tamed.

On the other hand, the issue is Not Autofs' Problem, and if stat() or 
statfs() is directed at a nonlocal directory, information about its 
referent should be returned, not about the soon-to-be-underlying mount 
point.  "ls", Nautilus, etc. are supposed to use mental telepathy to avoid 
causing mount storms; it is not autofs' place to interpret why the 
information is wanted and whether major resources should be expended to 
make it available.

Actually, only the sysop knows which is more important: doing the mount 
storm to respond to stat() or statfs(), or avoiding the storm.

Could we have two kinds of browsing, with or without mount storms, i.e. 
with or without a response to stat() or statfs()?  This would be 
configurable the same way browsing is, individually per map row.  The 
default should be to do the mount, because most sites export a reasonable 
number of filesystems from each server and so the mount storm is not a big 
deal, but a sysop who exports lots of directories from one NAS box and also 
turns on browsing in the client is advised to pick the non-mounting browse 
variant.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 520 Portola Plaza; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: statfs() and autofs

[Chris Walker]

From jimc [07/20 15:31]:
> Actually, only the sysop knows which is more important: doing the mount 
> storm to respond to stat() or statfs(), or avoiding the storm.

Agreed. In our case, we have 'heavily discouraged' use of /net on our
big servers and we would prefer that statfs()/stat() waited for the
automount to complete.

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Ian Kent]

On Sun, 2008-07-20 at 20:41 -0700, Chris Walker wrote:
> From jimc [07/20 15:31]:
> > Actually, only the sysop knows which is more important: doing the mount 
> > storm to respond to stat() or statfs(), or avoiding the storm.
> 
> Agreed. In our case, we have 'heavily discouraged' use of /net on our
> big servers and we would prefer that statfs()/stat() waited for the
> automount to complete.

Just to be absolutely clear.

The "browse" or "--ghost option isn't being used, in oder to avoid mount
storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
mount to complete.

Is that right?
Is anyone in a position to test a kernel patch?

Ian

Re: statfs() and autofs

[Ian Kent]

On Mon, 2008-07-21 at 12:39 +0800, Ian Kent wrote:
> On Sun, 2008-07-20 at 20:41 -0700, Chris Walker wrote:
> > From jimc [07/20 15:31]:
> > > Actually, only the sysop knows which is more important: doing the mount 
> > > storm to respond to stat() or statfs(), or avoiding the storm.
> > 
> > Agreed. In our case, we have 'heavily discouraged' use of /net on our
> > big servers and we would prefer that statfs()/stat() waited for the
> > automount to complete.
> 
> Just to be absolutely clear.
> 
> The "browse" or "--ghost option isn't being used, in oder to avoid mount
> storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
> mount to complete.
> 
> Is that right?
> Is anyone in a position to test a kernel patch?

We could add a third case to the description of the recently posted
"[autofs] [PATCH 3/7] autofs4 - fix pending checks" patch but it's
already been posted.

This can also happen via ->lookup() when only the LOOKUP_FOLLOW flag is
present, as is the case for stat(2) and statfs(2) and the like.

Ian

Re: statfs() and autofs

[Chris Walker]

From raven [07/20 21:43]:
> Just to be absolutely clear.
> 
> The "browse" or "--ghost option isn't being used, in oder to avoid mount
> storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
> mount to complete.
> 
> Is that right?
> Is anyone in a position to test a kernel patch?

This is what I'm seeing and I'm only running automount with the --debug
option. If I don't pass --ghost to automount, is there any other way
browsing is enabled?

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Jonathan Loran]

[-- Attachment #1.1: Type: text/plain, Size: 1121 bytes --]



Ian Kent wrote:
> On Sun, 2008-07-20 at 20:41 -0700, Chris Walker wrote:
>   
>> From jimc [07/20 15:31]:
>>     
>>> Actually, only the sysop knows which is more important: doing the mount 
>>> storm to respond to stat() or statfs(), or avoiding the storm.
>>>       
>> Agreed. In our case, we have 'heavily discouraged' use of /net on our
>> big servers and we would prefer that statfs()/stat() waited for the
>> automount to complete.
>>     
>
> Just to be absolutely clear.
>
> The "browse" or "--ghost option isn't being used, in oder to avoid mount
> storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
> mount to complete.
>
> Is that right?
> Is anyone in a position to test a kernel patch?
>
>   
I may be willing to try the patch, but first I need to see if I can 
recreate the problem reliably.  If not, there isn't much use.  I just 
know that on our "faster" servers, the programmers I work with have 
demonstrated the issue.  I will need their help.  I'll let know if I 
make head way.  Thanks,

Jon

P.S. Chris, can you reliably demonstrate the not waiting for the mount 
problem?


[-- Attachment #1.2: Type: text/html, Size: 1657 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: statfs() and autofs

[Ian Kent]

On Sun, 2008-07-20 at 22:00 -0700, Chris Walker wrote:
> From raven [07/20 21:43]:
> > Just to be absolutely clear.
> > 
> > The "browse" or "--ghost option isn't being used, in oder to avoid mount
> > storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
> > mount to complete.
> > 
> > Is that right?
> > Is anyone in a position to test a kernel patch?
> 
> This is what I'm seeing and I'm only running automount with the --debug
> option. If I don't pass --ghost to automount, is there any other way
> browsing is enabled?

Depends on version.

Version 5 will honor the "browse" option in shared maps, essentially
causing indirect mounts to have "--ghost" option.

Ian

Re: statfs() and autofs

[Chris Walker]

From jonathan [07/20 22:08]:
> P.S. Chris, can you reliably demonstrate the not waiting for the mount 
> problem?

I can, but I'm a few revs back. I'll see if I can replicate on current
revs of linux and autofs.

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Ian Kent]

On Sun, 2008-07-20 at 22:50 -0700, Chris Walker wrote:
> From jonathan [07/20 22:08]:
> > P.S. Chris, can you reliably demonstrate the not waiting for the mount 
> > problem?
> 
> I can, but I'm a few revs back. I'll see if I can replicate on current
> revs of linux and autofs.

It should be reproducible on any version of autofs and any kernel, if I
understand correctly what is causing it. But, the real gotcha is stat(2)
or statfs(2) are often accompanied by other system calls that will wait
for the mount to complete. The mount can complete sometimes before but
also sometimes after, and possibly independent of system call order (due
to scheduling) so the problem should be intermittent and appear
unpredictable.

Ian

Re: statfs() and autofs

[Steve Thompson]

On Mon, 21 Jul 2008, Ian Kent wrote:

> Just to be absolutely clear.
>
> The "browse" or "--ghost option isn't being used, in oder to avoid mount
> storms, but we still see stat(2) and statfs(2) (et. al.) not wait for a
> mount to complete.

I see this problem quite often. I have a CentOS 5.2 x86_64 system with 
autofs-5.0.1-0.rc2.88, kernel 2.6.18-92.1.6.el5xen. Automount maps are 
distributed by NIS, and I have explicitly set BROWSE_MODE=no. I see that:

* after an inactive mount times out, an umount is performed, but the mount
   point that was created is never removed, leading to:

* if the NIS map is changed to point the same key to a different location,
   a "df /foo/bar" _always_ returns /foo, and then /foo/bar (correctly) a
   few moments later.

Steve

Re: statfs() and autofs

[Chris Walker]

From raven [07/21 00:39]:
> 
> On Sun, 2008-07-20 at 22:50 -0700, Chris Walker wrote:
> > From jonathan [07/20 22:08]:
> > > P.S. Chris, can you reliably demonstrate the not waiting for the mount 
> > > problem?
> > 
> > I can, but I'm a few revs back. I'll see if I can replicate on current
> > revs of linux and autofs.
> 
> It should be reproducible on any version of autofs and any kernel, if I
> understand correctly what is causing it. But, the real gotcha is stat(2)
> or statfs(2) are often accompanied by other system calls that will wait
> for the mount to complete. The mount can complete sometimes before but
> also sometimes after, and possibly independent of system call order (due
> to scheduling) so the problem should be intermittent and appear
> unpredictable.

I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3).
Maybe I'll do some git bisecting to see where it got fixed.

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Chris Walker]

From cwalker [07/22 10:20]:
> From raven [07/21 00:39]:
> > 
> > On Sun, 2008-07-20 at 22:50 -0700, Chris Walker wrote:
> > > From jonathan [07/20 22:08]:
> > > > P.S. Chris, can you reliably demonstrate the not waiting for the mount 
> > > > problem?
> > > 
> > > I can, but I'm a few revs back. I'll see if I can replicate on current
> > > revs of linux and autofs.
> > 
> > It should be reproducible on any version of autofs and any kernel, if I
> > understand correctly what is causing it. But, the real gotcha is stat(2)
> > or statfs(2) are often accompanied by other system calls that will wait
> > for the mount to complete. The mount can complete sometimes before but
> > also sometimes after, and possibly independent of system call order (due
> > to scheduling) so the problem should be intermittent and appear
> > unpredictable.
> 
> I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3).
> Maybe I'll do some git bisecting to see where it got fixed.

Looks like git bisect only helps to find introduced bugs, but not where
something got fixed. Oh well. I think we were going to upgrade to 2.6.25
anyway... :)

-- 
Chris Walker -- cwalker@pixar.com

Re: statfs() and autofs

[Chris Walker]

From cwalker [07/22 10:20]:
> From raven [07/21 00:39]:
> > 
> > On Sun, 2008-07-20 at 22:50 -0700, Chris Walker wrote:
> > > From jonathan [07/20 22:08]:
> > > > P.S. Chris, can you reliably demonstrate the not waiting for the mount 
> > > > problem?
> > > 
> > > I can, but I'm a few revs back. I'll see if I can replicate on current
> > > revs of linux and autofs.
> > 
> > It should be reproducible on any version of autofs and any kernel, if I
> > understand correctly what is causing it. But, the real gotcha is stat(2)
> > or statfs(2) are often accompanied by other system calls that will wait
> > for the mount to complete. The mount can complete sometimes before but
> > also sometimes after, and possibly independent of system call order (due
> > to scheduling) so the problem should be intermittent and appear
> > unpredictable.
> 
> I cannot reproduce this on fedora 9 (kernel: 2.6.25.6, autofs: autofs-5.0.3).
> Maybe I'll do some git bisecting to see where it got fixed.

Just closing the loop on this one--it turns out browsing was on.
Disabling browsing (-nobrowse for each map in auto.master) corrected
this behavior.

-- 
Chris Walker -- cwalker@pixar.com