"Busy inodes after unmount. Self destruct in 5 seconds..."

"Busy inodes after unmount. Self destruct in 5 seconds..."

[Raymond Prach]

[-- Attachment #1.1: Type: text/plain, Size: 1244 bytes --]

Dear Folks,

I am administrating a set of Linux servers that use automount and amd in 
order to access a variety of NFS volumes. The servers are, for the most 
part, running kernel 2.4.25 (but see below). We are using the kernel 
automounter version 3 and the userspace automount binary v3.1.7. 

We have a script that accesses resources all over the network. To this 
end, it mounts, via autofs, a large number of remote filesystems. Whenever 
this script is run, the following alarming message appears in the system 
log:

kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have 
a nice day...

I searched the mailing list of this archive and found some threads from 
September of last year which seem to detail the same problem, however, no 
solution or workaround was proposed at that time.

Looking at kernel.org, I see that kernel 2.4.26-pre3 contains a patch that 
is supposed to help fix this problem. This patch was duly applied, but the 
message persists. 

Questions: What is causing this message? Is there anything I can do about 
it? Is there anything I should be doing about it, or is it an alarming yet 
irrelevant message? 

Any insight you could shed would be greatly appreciated.

Yours Truly,

Raymond Prach

[-- Attachment #1.2: Type: text/html, Size: 1693 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Chris Walker]

We have this problem as well. The problem occurs during unmount, so we
reduced the number of unmounts by adding a high --timeout (86400) to
automount. This worked in our case because we have a (relatively) small
number of filesystems that are accessed frequently.

This may not be an option in your case as you are accessing many
filesystems.

From rprach:
> Dear Folks,
> 
> I am administrating a set of Linux servers that use automount and amd in 
> order to access a variety of NFS volumes. The servers are, for the most 
> part, running kernel 2.4.25 (but see below). We are using the kernel 
> automounter version 3 and the userspace automount binary v3.1.7. 
> 
> We have a script that accesses resources all over the network. To this 
> end, it mounts, via autofs, a large number of remote filesystems. Whenever 
> this script is run, the following alarming message appears in the system 
> log:
> 
> kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have 
> a nice day...
> 
> I searched the mailing list of this archive and found some threads from 
> September of last year which seem to detail the same problem, however, no 
> solution or workaround was proposed at that time.
> 
> Looking at kernel.org, I see that kernel 2.4.26-pre3 contains a patch that 
> is supposed to help fix this problem. This patch was duly applied, but the 
> message persists. 
> 
> Questions: What is causing this message? Is there anything I can do about 
> it? Is there anything I should be doing about it, or is it an alarming yet 
> irrelevant message? 
> 
> Any insight you could shed would be greatly appreciated.
> 
> Yours Truly,
> 
> Raymond Prach
> _______________________________________________
> autofs mailing list
> autofs@linux.kernel.org
> http://linux.kernel.org/mailman/listinfo/autofs


-- 
Chris Walker
Render Pipeline Group
Pixar Animation Studios
510/752.3736
cwalker@pixar.com

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Wed, 7 Apr 2004, Raymond Prach wrote:

> Dear Folks,
> 
> I am administrating a set of Linux servers that use automount and amd in 
> order to access a variety of NFS volumes. The servers are, for the most 
> part, running kernel 2.4.25 (but see below). We are using the kernel 
> automounter version 3 and the userspace automount binary v3.1.7. 

My main interest is with version 4 and autofs4 but I believe the 
issues are the same. It comes up in both versions.

> 
> We have a script that accesses resources all over the network. To this 
> end, it mounts, via autofs, a large number of remote filesystems. Whenever 
> this script is run, the following alarming message appears in the system 
> log:
> 
> kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have 
> a nice day...

It would be good if I could duplicate this reliably. Perhaps then I could 
work out what is happening. Can you help with a description of the 
scenario or a simple script?

I can't really do much unless you are willing to duplicate this 
against, at least autofs-4.1.1 (4.1.2 is latest), together with the kernel 
patches for autofs4. I haven't tested 2.4.25 but the 2.4.22 patch applies 
OK against 2.4.24.

> 
> I searched the mailing list of this archive and found some threads from 
> September of last year which seem to detail the same problem, however, no 
> solution or workaround was proposed at that time.
> 
> Looking at kernel.org, I see that kernel 2.4.26-pre3 contains a patch that 
> is supposed to help fix this problem. This patch was duly applied, but the 
> message persists. 

What patch was this? I must have missed it.

> 
> Questions: What is causing this message? Is there anything I can do about 
> it? Is there anything I should be doing about it, or is it an alarming yet 
> irrelevant message? 

That's the big question.

There is more than one way in which it may be caused.

One common cause is automount exiting without cleaning up directories in 
the autofs filesystem.

Just this last day or so I have been thinking that another cause could be 
the communications file descriptor not being closed before exit.

It's possible that this could be a the result of a bug in another 
filesystem, such as NFS. I saw a thread on the NFS list that discussed 
this and there was a patch posted. I haven't as yet done any testing. 

One idea that I had was to add a umount_begin method to the autofs4 module 
so that I could try and cleanup any remaining references even if the 
daemon was already gone but unless I can reliably duplicate the problem 
it's academic.

Is the message bad?

Well I'm not sure. On the face of it no, but I find that if I am trying to 
fix something which results in messages of this type I get the impression 
that my system becomes less stable and ultimately I need to reboot. Of 
course it may not actually be the result of that message as I've usually 
got other more sinister things happening as well.

Finally, one of my production systems that has ghosting (browsable 
automount tree) turned of and I occassionally get them on that system and 
it seems to keep working OK for weeks. So I can't be sure on that either.

Not much help here aye! Sorry.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Raymond Prach]

[-- Attachment #1.1: Type: text/plain, Size: 2391 bytes --]

It would be good if I could duplicate this reliably. Perhaps then I could 
work out what is happening. Can you help with a description of the 
scenario or a simple script?

Unfortunately the script is not simple, and neither is our environment. ;) 
However, what seems to happen is this:

1. The script reads from the home directories of a large number of users
2. Users' home directories, by and large, are located on their 
workstations, which are AIX 5.1
3. Thus, via autofs, the server mounts a whole bunch of home directories 
using NFS (v2, over UDP, if that matters) from a whole bunch of AIX 
workstations -- say 20 at a time
4. Eventually, the script attempts to mount a home directory from a 
workstation that is down or has been removed from the network, so it waits 
around for a long time for the mount attempt to time out
5. While it is waiting for this mount attempt to fail, a bunch of mounts 
that autofs previously mounted time out due to inactivity, so autofs 
unmounts them
6. At this point, the "self destruct" message appears in the syslog
7. Lather, rinse, repeat.

I hope that is somewhat helpful, or at least informative.

I can't really do much unless you are willing to duplicate this 
against, at least autofs-4.1.1 (4.1.2 is latest), together with the kernel 

patches for autofs4. I haven't tested 2.4.25 but the 2.4.22 patch applies 
OK against 2.4.24.

Ehhh. Unfortunately these are at least nominally production systems, and 
I'm loath to start applying bleeding-edge patches to them when it doesn't 
look like it will make the problem go away. Are there other advantages to 
switching to autofs4? (Also, off-topic, but my 2.4.25 kernel has an 
autofs4 module. What is the difference between that autofs4 and the 
autofs4 you are talking about?)

What patch was this? I must have missed it.

From kernel.org: o A patch by Greg Banks to help fix the "VFS: Busy inodes 
after unmount." problem that occurs if autofs expires the mountpoint while 
an NFS sillydelete is still pending. Now didn't have high hopes for this 
patch, as I don't think that what we're seeing has anything to do with 
sillydeletes (or indeed that any are being performed), but I was running 
out of ideas here.

Not much help here aye! Sorry.

No, this is as informative as anything I've encountered so far. Thanks 
very much for taking the time to answer my questions. 

Raymond Prach

[-- Attachment #1.2: Type: text/html, Size: 3107 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Jim Carter]

On Wed, 7 Apr 2004, Raymond Prach wrote:
> We have a script that accesses resources all over the network. To this 
> end, it mounts, via autofs, a large number of remote filesystems. Whenever 
> this script is run, the following alarming message appears in the system 
> log:
> 
> kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds.  Have 
> a nice day...

No big sweat, any I.T. professional knows to just blow off all error 
messages :-)

This happens with autofs4-4.0.0pre10 from SuSE 8.2 and 9.0.  That's with 
module version... um, well, whatever SuSE is giving us.  expire.c says
 *  Copyright 1997-1998 Transmeta Corporation -- All Rights Reserved
 *  Copyright 1999-2000 Jeremy Fitzhardinge <jeremy@goop.org>
This is definitely not the version of December 2003; I didn't install it.

When I installed the autofs-4.1.2 daemon, the message stopped appearing. 
This is with the same module, so the finger of blame points to userspace.
On 4.0.0pre10, the message appears whenever you shut down autofs, as well 
as on some (but not all?) automatic dismounts, which were not happening 
reliably (but dismounting does happen with 4.1.2).  I'm pretty sure that 
4.1.0 also avoided the "self destruct" messages entirely, and I know it 
fixed the dismount problems.

v4.1.2 seems happy with the back-version module.  However, being unable to
do "ghost mounts", it (seems to) mount every exported filesystem from the
target host (and they expire normally if unused).  That could be a problem
if many filesystems are exported.  4.1.0 would complain and bypass the
ghost mounts.  My preference would be for it to not complain and to mount
only the requested filesystem.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Thu, 8 Apr 2004, Raymond Prach wrote:

> 
> Unfortunately the script is not simple, and neither is our environment. ;) 

Expected that. Oh well.

> However, what seems to happen is this:
> 
> 1. The script reads from the home directories of a large number of users
> 2. Users' home directories, by and large, are located on their 
> workstations, which are AIX 5.1

Can I assume that you're sure you're not reaching the anonymous mounts 
limit that exists in the 2.4 kernel?

> 3. Thus, via autofs, the server mounts a whole bunch of home directories 
> using NFS (v2, over UDP, if that matters) from a whole bunch of AIX 
> workstations -- say 20 at a time
> 4. Eventually, the script attempts to mount a home directory from a 
> workstation that is down or has been removed from the network, so it waits 
> around for a long time for the mount attempt to time out
> 5. While it is waiting for this mount attempt to fail, a bunch of mounts 
> that autofs previously mounted time out due to inactivity, so autofs 
> unmounts them
> 6. At this point, the "self destruct" message appears in the syslog
> 7. Lather, rinse, repeat.

OK so I should be able to duplicate this with three machines.
The picture sound fairly clear. I'll have a go at duplicating this.

Certainly, the mount behaviour in the presence of an unavailable server 
is unacceptable. I have some ideas for this but am still (and have been 
for some time) working through bug fix and minor updates. This type of 
change may amount to something a little more than minor.

> 
> I hope that is somewhat helpful, or at least informative.

Yep. very helpfull.

> 
> I can't really do much unless you are willing to duplicate this 
> against, at least autofs-4.1.1 (4.1.2 is latest), together with the kernel 
> 
> patches for autofs4. I haven't tested 2.4.25 but the 2.4.22 patch applies 
> OK against 2.4.24.
> 
> Ehhh. Unfortunately these are at least nominally production systems, and 
> I'm loath to start applying bleeding-edge patches to them when it doesn't 
> look like it will make the problem go away. Are there other advantages to 
> switching to autofs4? (Also, off-topic, but my 2.4.25 kernel has an 
> autofs4 module. What is the difference between that autofs4 and the 
> autofs4 you are talking about?)

Yes. I have trouble testing also, as my work systems are in production use 
as well.

I've been eorking on autofs v4 for quite a while now and a number of 
people have taken advantage of this. My focus is on improving stability 
and adding features that people in enterprise ebvironments need.

I've made some progress but have to carry some annoying limitations due 
largely to limitations in the kernel.

Essentailly what I have added is the ability for autofs to understand 
direct mount maps for file, NIS and LDAP maps. I'm about to start on NIS+. 
There remains a significant limitation so see README.direct if you are 
going to use this.

The kernel module update allows the daemon to provide a browsable autofs 
mount tree (aka ghosting in autofs v4).

You could call them bleeding edge, I suppose. I have been making 
corrections and enhancements for the couple of years that myself and 
others have been using.

> 
> What patch was this? I must have missed it.

Yep. Above.

They are in the 2.6.5-mm2 tree at the moment and I hope they will make it 
through to a 2.6 release soonish (fingers crossed).

> 
> >From kernel.org: o A patch by Greg Banks to help fix the "VFS: Busy inodes 
> after unmount." problem that occurs if autofs expires the mountpoint while 
> an NFS sillydelete is still pending. Now didn't have high hopes for this 
> patch, as I don't think that what we're seeing has anything to do with 
> sillydeletes (or indeed that any are being performed), but I was running 
> out of ideas here.

Yes, that is the patch I refered to but haven't tested. So it sounds like 
you have saved me some time.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Thu, 8 Apr 2004, Raymond Prach wrote:

> 4. Eventually, the script attempts to mount a home directory from a 
> workstation that is down or has been removed from the network, so it waits 
> around for a long time for the mount attempt to time out

How stupid of me.

I may already have already added someting that will help with this in 
4.1.1. I gad to deal with this when I merged a patch to deal with 
replicated server mounts. Like when the mount refers to multiple 
weighted servers and needs to choose the best one. I added an rcp_ping 
function which appears to timeout much more quickly than an attempted 
mount (order of 10 seconds).

This may help you somewhat but the problem you report still needs to be 
fixed as the situation you describe could easily still happen.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Thu, 8 Apr 2004, Raymond Prach wrote:

> 4. Eventually, the script attempts to mount a home directory from a 
> workstation that is down or has been removed from the network, so it waits 
> around for a long time for the mount attempt to time out

Sorry to be a pest but looking at the changelog I see that I fixed a 
problem in the replicated server code that dealt with this situation in 
4.1.2 (latest release). So I guess I'm recommending 4.1.2 if you have the 
chance to try anything.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Thu, 8 Apr 2004, Jim Carter wrote:

> 
> v4.1.2 seems happy with the back-version module.  However, being unable to
> do "ghost mounts", it (seems to) mount every exported filesystem from the
> target host (and they expire normally if unused).  That could be a problem
> if many filesystems are exported.  4.1.0 would complain and bypass the
> ghost mounts.  My preference would be for it to not complain and to mount
> only the requested filesystem.

More info please Jim?

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Thu, 8 Apr 2004, Raymond Prach wrote:

> 
> 1. The script reads from the home directories of a large number of users
> 2. Users' home directories, by and large, are located on their 
> workstations, which are AIX 5.1

I'm curious. Does the AIX 5 automounter still polute the mount table with 
heaps of entries if you have a large direct mount table (as it did in AIX 
4)?

> 3. Thus, via autofs, the server mounts a whole bunch of home directories 
> using NFS (v2, over UDP, if that matters) from a whole bunch of AIX 
> workstations -- say 20 at a time
> 4. Eventually, the script attempts to mount a home directory from a 
> workstation that is down or has been removed from the network, so it waits 
> around for a long time for the mount attempt to time out
> 5. While it is waiting for this mount attempt to fail, a bunch of mounts 
> that autofs previously mounted time out due to inactivity, so autofs 
> unmounts them
> 6. At this point, the "self destruct" message appears in the syslog
> 7. Lather, rinse, repeat.
> 
> I hope that is somewhat helpful, or at least informative.

I've tried to duplicate this just now and am having trouble.

First, the version of mount that I'm using doesn't seem to hang the way it 
used to. What version of util-linux (or mount) are you using? I have 
2.11y.

Second thing I notice is that autofs-4.1.2 discovers the bogus mount very 
quickly (<1 sec) and so even with a stop watch I can't get the mount to 
be active at the same time as the expire.

Hopefully some of this info will help.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Fri, 9 Apr 2004, Ian Kent wrote:

> 
> I've tried to duplicate this just now and am having trouble.
> 
> First, the version of mount that I'm using doesn't seem to hang the way it 
> used to. What version of util-linux (or mount) are you using? I have 
> 2.11y.
> 
> Second thing I notice is that autofs-4.1.2 discovers the bogus mount very 
> quickly (<1 sec) and so even with a stop watch I can't get the mount to 
> be active at the same time as the expire.
> 
> Hopefully some of this info will help.

Forgot kernel version info.

This is with a vanilla 2.4.24 kernel.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Raymond Prach]

[-- Attachment #1.1: Type: text/plain, Size: 1525 bytes --]

Thanks so much for your ongoing help. I will try to answer your questions 
as best as I'm able...

I'm curious. Does the AIX 5 automounter still polute the mount table with 
heaps of entries if you have a large direct mount table (as it did in AIX 
4)?

Not sure. The AIX systems in question are using the am-utils automounter, 
which leaves the mount table tidy but, at least for us, has a habit of 
blowing up every so often, taking the machine down with it....

First, the version of mount that I'm using doesn't seem to hang the way it 

used to. What version of util-linux (or mount) are you using? I have 
2.11y.

Second thing I notice is that autofs-4.1.2 discovers the bogus mount very 
quickly (<1 sec) and so even with a stop watch I can't get the mount to 
be active at the same time as the expire.

Mount is 2.11y as well. I will upgrade the userspace automounter to 4.1.2 
and see if it helps. But I don't understand how autofs can possibly decide 
that the remote host is down or unavailable and quit attempting the mount 
in that short (<1 sec) time period. The remote hosts in question have 
valid DNS names, they're just powered off or otherwise unavailable. Hence 
the mount attempt taking some time is perfectly reasonable... right? Or am 
I being an idiot? I feel like I've been unclear in describing the 
situation...

Oh yes: the kernel version is 2.4.26-pre3, though 2.4.25 exhibits the same 
behavior.

I will update the userspace automounter. Do I need to load autofs4 instead 
of autofs as well?

r.

[-- Attachment #1.2: Type: text/html, Size: 1897 bytes --]

[-- Attachment #2: Type: text/plain, Size: 140 bytes --]

_______________________________________________
autofs mailing list
autofs@linux.kernel.org
http://linux.kernel.org/mailman/listinfo/autofs

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Jim Carter]

On Fri, 9 Apr 2004, Ian Kent wrote:

> On Thu, 8 Apr 2004, Jim Carter wrote:
> > v4.1.2 seems happy with the back-version module.  However, being unable to
> > do "ghost mounts", it (seems to) mount every exported filesystem from the
> > target host (and they expire normally if unused).  That could be a problem
> > if many filesystems are exported.  4.1.0 would complain and bypass the
> > ghost mounts.  My preference would be for it to not complain and to mount
> > only the requested filesystem.
> 
> More info please Jim?

Oops, I misinterpreted a test.  With the back-version module that can't do
ghost mounts, 4.1.2 in fact silently mounts only the requested filesystem,
and there is no trace of potentially but not actually requested
filesystems, with both Linux and Solaris servers.  (Unless one has
previously mounted the filesystem and forgotten that one has done it.)

Sorry about that.

James F. Carter          Voice 310 825 2897    FAX 310 206 6673
UCLA-Mathnet;  6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA  90095-1555
Email: jimc@math.ucla.edu    http://www.math.ucla.edu/~jimc (q.v. for PGP key)

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Fri, 9 Apr 2004, Raymond Prach wrote:

> Thanks so much for your ongoing help. I will try to answer your questions 
> as best as I'm able...
> 
> I'm curious. Does the AIX 5 automounter still polute the mount table with 
> heaps of entries if you have a large direct mount table (as it did in AIX 
> 4)?
> 
> Not sure. The AIX systems in question are using the am-utils automounter, 
> which leaves the mount table tidy but, at least for us, has a habit of 
> blowing up every so often, taking the machine down with it....

OK.

> 
> First, the version of mount that I'm using doesn't seem to hang the way it 
> 
> used to. What version of util-linux (or mount) are you using? I have 
> 2.11y.
> 
> Second thing I notice is that autofs-4.1.2 discovers the bogus mount very 
> quickly (<1 sec) and so even with a stop watch I can't get the mount to 
> be active at the same time as the expire.
> 
> Mount is 2.11y as well. I will upgrade the userspace automounter to 4.1.2 
> and see if it helps. But I don't understand how autofs can possibly decide 
> that the remote host is down or unavailable and quit attempting the mount 
> in that short (<1 sec) time period. The remote hosts in question have 
> valid DNS names, they're just powered off or otherwise unavailable. Hence 
> the mount attempt taking some time is perfectly reasonable... right? Or am 
> I being an idiot? I feel like I've been unclear in describing the 
> situation...

Good. Same version of mount.

Good question.

I merged a patch in 4.1.1 that provided replicated server type behaviour.
The point bieng that it needs to deal with host specs like 
host1(10),host2(5),host3 and so on. The point of it is to select the best 
available host to mount from. To do this we have an rpc_ping function. It 
is set to time out fairly quickly. All the mount requests pass through 
this code and if none (one in this case) is not available it return a fail 
and doesn't attempt the mount.

This situation is quite interesting as your script could show up a bug in 
that, even though the mount may be significantly less the problem may 
still arise from time to time.

Lets see how it goes first.

> 
> Oh yes: the kernel version is 2.4.26-pre3, though 2.4.25 exhibits the same 
> behavior.
> 
> I will update the userspace automounter. Do I need to load autofs4 instead 
> of autofs as well?

The init script should do this for you.

In the past it you needed to add a line like

alias autofs autofs4

to your modules.conf.

Ian

Re: "Busy inodes after unmount. Self destruct in 5 seconds..."

[Ian Kent]

On Sat, 10 Apr 2004, Ian Kent wrote:

> 
> Good question.
> 
> I merged a patch in 4.1.1 that provided replicated server type behaviour.
> The point bieng that it needs to deal with host specs like 
> host1(10),host2(5),host3 and so on. The point of it is to select the best 
> available host to mount from. To do this we have an rpc_ping function. It 
> is set to time out fairly quickly. All the mount requests pass through 
> this code and if none (one in this case) is not available it return a fail 
> and doesn't attempt the mount.
> 

Indeed, to my knowledge (anyone got more info), this has only been tested 
against Linux servers so it's entirely possible that the rpc_ping needs 
more work to function against other OS servers.

Ian