* VFS: Busy inodes after unmount on 2 way SMP
@ 2003-09-12 0:11 Arun Sharma
2003-09-12 1:05 ` H. Peter Anvin
` (2 more replies)
0 siblings, 3 replies; 53+ messages in thread
From: Arun Sharma @ 2003-09-12 0:11 UTC (permalink / raw)
To: autofs; +Cc: nfs
We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case:
- Due to network load issues, the NFS server becomes unreachable for some time
- The automounter tries to expire the mount
- The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. have inode->i_count == 1. But they don't have any waiters on inode->i_wait. Further, the inodes that are busy are
/mnt/foo -> autofs mount point
/mnt/foo/bar -> bar is a symbolic link
It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are:
fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino));
fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino));
-Arun
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
^ permalink raw reply [flat|nested] 53+ messages in thread* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 0:11 VFS: Busy inodes after unmount on 2 way SMP Arun Sharma @ 2003-09-12 1:05 ` H. Peter Anvin 2003-09-12 14:23 ` Ryan Go 2003-09-12 17:57 ` Arun Sharma 2003-09-12 23:33 ` Ion Badulescu 2003-09-17 0:28 ` Ian Kent 2 siblings, 2 replies; 53+ messages in thread From: H. Peter Anvin @ 2003-09-12 1:05 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs Arun Sharma wrote: > > We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 > seconds. Have a nice day.." messages on a dual processor NFS client. > Here's the use case: > > - Due to network load issues, the NFS server becomes unreachable for > some time > - The automounter tries to expire the mount > - The unmount finds a couple of busy inodes. Putting some debug printks > shows that typically two inodes are busy i.e. have inode->i_count == 1. > But they don't have any waiters on inode->i_wait. Further, the inodes > that are busy are > /mnt/foo -> autofs mount point > /mnt/foo/bar -> bar is a symbolic link > > It's not clear if this is a NFS issue or a autofs issue, but it's seen > often with autofs. Are there any known race conditions that have been > fixed after 2.4.20 ? The two calls I'm worried about are: > > fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); > fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); > Which version of autofs are you running? What version of mount(8)? It's possible NFS lets you umount a mount which has busy inodes under certain conditions. autofs stresses mounting/umounting NFS in ways that sometimes exposes bugs that otherwise wouldn't be seen. -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 1:05 ` H. Peter Anvin @ 2003-09-12 14:23 ` Ryan Go 2003-09-12 17:57 ` Arun Sharma 1 sibling, 0 replies; 53+ messages in thread From: Ryan Go @ 2003-09-12 14:23 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Arun Sharma, autofs, nfs Hello, We have seen the same message on a box with vanilla 2.4.18 kernel with rmap12h patch. Mount version is 2.11b and autofs4-4.0.0pre10-10 (SuSe package). If it matters, we are using only nisplus to get our maps. I know the above setup is kinda old. We are still having some other issues moving to a newer release. << ryan On Thu, 11 Sep 2003, H. Peter Anvin wrote: > Arun Sharma wrote: > > > > We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 > > seconds. Have a nice day.." messages on a dual processor NFS client. > > Here's the use case: > > > > - Due to network load issues, the NFS server becomes unreachable for > > some time > > - The automounter tries to expire the mount > > - The unmount finds a couple of busy inodes. Putting some debug printks > > shows that typically two inodes are busy i.e. have inode->i_count == 1. > > But they don't have any waiters on inode->i_wait. Further, the inodes > > that are busy are > > /mnt/foo -> autofs mount point > > /mnt/foo/bar -> bar is a symbolic link > > > > It's not clear if this is a NFS issue or a autofs issue, but it's seen > > often with autofs. Are there any known race conditions that have been > > fixed after 2.4.20 ? The two calls I'm worried about are: > > > > fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); > > fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); > > > > Which version of autofs are you running? What version of mount(8)? > > It's possible NFS lets you umount a mount which has busy inodes under > certain conditions. autofs stresses mounting/umounting NFS in ways that > sometimes exposes bugs that otherwise wouldn't be seen. > > -hpa > > _______________________________________________ > autofs mailing list > autofs@linux.kernel.org > http://linux.kernel.org/mailman/listinfo/autofs > ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 1:05 ` H. Peter Anvin 2003-09-12 14:23 ` Ryan Go @ 2003-09-12 17:57 ` Arun Sharma 2003-09-12 20:51 ` H. Peter Anvin 1 sibling, 1 reply; 53+ messages in thread From: Arun Sharma @ 2003-09-12 17:57 UTC (permalink / raw) To: H. Peter Anvin; +Cc: autofs, nfs H. Peter Anvin wrote: > > Which version of autofs are you running? What version of mount(8)? > autofs-3.1.7 and mount-2.11g > It's possible NFS lets you umount a mount which has busy inodes under > certain conditions. Are you thinking of the MNT_FORCE flag ? I have reasons to believe that it won't help. Mainly because if it was a RPC task that marked the inode busy, it'd have put itself on the waitq of the inode. However I found that waitqueue_active(&inode->i_wait) == 0. -Arun ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 17:57 ` Arun Sharma @ 2003-09-12 20:51 ` H. Peter Anvin 0 siblings, 0 replies; 53+ messages in thread From: H. Peter Anvin @ 2003-09-12 20:51 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs Arun Sharma wrote: > H. Peter Anvin wrote: > >> >> Which version of autofs are you running? What version of mount(8)? >> > > autofs-3.1.7 and mount-2.11g > >> It's possible NFS lets you umount a mount which has busy inodes under >> certain conditions. > > > Are you thinking of the MNT_FORCE flag ? I have reasons to believe that > it won't help. Mainly because if it was a RPC task that marked the inode > busy, it'd have put itself on the waitq of the inode. However I found > that waitqueue_active(&inode->i_wait) == 0. > No, I'm thinking about the possibility that there is a bug. autofs depends critically on a filesystem *not* allowing itself to be unmounted when there are busy inodes; however, this is something that things other than autofs doesn't really stress a whole lot. -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 0:11 VFS: Busy inodes after unmount on 2 way SMP Arun Sharma 2003-09-12 1:05 ` H. Peter Anvin @ 2003-09-12 23:33 ` Ion Badulescu 2003-09-13 0:08 ` Arun Sharma 2003-09-17 0:28 ` Ian Kent 2 siblings, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-12 23:33 UTC (permalink / raw) To: Arun Sharma; +Cc: nfs, autofs On Thu, 11 Sep 2003 17:11:40 -0700, Arun Sharma <arun.sharma@intel.com> wrote: > We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case: > > - Due to network load issues, the NFS server becomes unreachable for some time > - The automounter tries to expire the mount > - The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. have inode->i_count == 1. But they don't have any waiters on inode->i_wait. Further, the inodes that are busy are > /mnt/foo -> autofs mount point > /mnt/foo/bar -> bar is a symbolic link > > It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are: > > fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); > fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); I don't know if it's the same issue, but I've seen it many many times while testing amd's autofs functionality. It occurs when a non-empty autofs filesystem is unmounted, e.g. if it has some leftover subdirectories from previous mounts. Therefore I'd say it's quite clearly an autofs issue, at least in my case. Do you see it when shutting down the automounter, or in other cases as well? Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 23:33 ` Ion Badulescu @ 2003-09-13 0:08 ` Arun Sharma 2003-09-13 0:16 ` Ion Badulescu 0 siblings, 1 reply; 53+ messages in thread From: Arun Sharma @ 2003-09-13 0:08 UTC (permalink / raw) To: Ion Badulescu; +Cc: nfs, autofs Ion Badulescu wrote: > > I don't know if it's the same issue, but I've seen it many many times > while testing amd's autofs functionality. It occurs when a non-empty > autofs filesystem is unmounted, e.g. if it has some leftover > subdirectories from previous mounts. Therefore I'd say it's quite > clearly an autofs issue, at least in my case. Do you have a case that can deterministically reproduce the issue ? > Do you see it when shutting down the automounter, or in other cases as > well? Everytime I've seen it, it's during an unmount by /usr/sbin/automount after the mount has expired. It's usually coupled with a temporarily unreachable NFS server. -Arun ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-13 0:08 ` Arun Sharma @ 2003-09-13 0:16 ` Ion Badulescu 0 siblings, 0 replies; 53+ messages in thread From: Ion Badulescu @ 2003-09-13 0:16 UTC (permalink / raw) To: Arun Sharma; +Cc: nfs, autofs On Fri, 12 Sep 2003, Arun Sharma wrote: > Ion Badulescu wrote: > > > > I don't know if it's the same issue, but I've seen it many many times > > while testing amd's autofs functionality. It occurs when a non-empty > > autofs filesystem is unmounted, e.g. if it has some leftover > > subdirectories from previous mounts. Therefore I'd say it's quite > > clearly an autofs issue, at least in my case. > > Do you have a case that can deterministically reproduce the issue ? Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message. > Everytime I've seen it, it's during an unmount by /usr/sbin/automount > after the mount has expired. It's usually coupled with a temporarily > unreachable NFS server. So maybe this is a different issue than the one I am describing... Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP @ 2003-09-13 0:16 ` Ion Badulescu 0 siblings, 0 replies; 53+ messages in thread From: Ion Badulescu @ 2003-09-13 0:16 UTC (permalink / raw) To: Arun Sharma; +Cc: nfs, autofs On Fri, 12 Sep 2003, Arun Sharma wrote: > Ion Badulescu wrote: > > > > I don't know if it's the same issue, but I've seen it many many times > > while testing amd's autofs functionality. It occurs when a non-empty > > autofs filesystem is unmounted, e.g. if it has some leftover > > subdirectories from previous mounts. Therefore I'd say it's quite > > clearly an autofs issue, at least in my case. > > Do you have a case that can deterministically reproduce the issue ? Oh yeah, easily. Kill the automounter while some things are mounted, unmount everything by hand but do not rmdir the mountpoints. When you unmount the autofs filesystem itself, you will get the self destruct message. > Everytime I've seen it, it's during an unmount by /usr/sbin/automount > after the mount has expired. It's usually coupled with a temporarily > unreachable NFS server. So maybe this is a different issue than the one I am describing... Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-13 0:16 ` Ion Badulescu (?) @ 2003-09-15 21:42 ` Arun Sharma 2003-09-15 22:25 ` H. Peter Anvin -1 siblings, 1 reply; 53+ messages in thread From: Arun Sharma @ 2003-09-15 21:42 UTC (permalink / raw) To: Ion Badulescu; +Cc: autofs, nfs Ion Badulescu wrote: > On Fri, 12 Sep 2003, Arun Sharma wrote: > > >>Ion Badulescu wrote: >> >>>I don't know if it's the same issue, but I've seen it many many times >>>while testing amd's autofs functionality. It occurs when a non-empty >>>autofs filesystem is unmounted, e.g. if it has some leftover >>>subdirectories from previous mounts. Therefore I'd say it's quite >>>clearly an autofs issue, at least in my case. >> >>Do you have a case that can deterministically reproduce the issue ? > > > Oh yeah, easily. Kill the automounter while some things are mounted, > unmount everything by hand but do not rmdir the mountpoints. When you > unmount the autofs filesystem itself, you will get the self destruct > message. > Thanks. This reproduces the problem I'm chasing very well. I think this is clearly an unacceptable kernel bug. -Arun Sep 15 13:58:57 arun-desktop kernel: VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day... <snip> Sep 15 14:00:00 arun-desktop kernel: Unable to handle kernel NULL pointer dereference at virtual address 00000004 Sep 15 14:00:00 arun-desktop kernel: printing eip: Sep 15 14:00:00 arun-desktop kernel: c015c738 Sep 15 14:00:00 arun-desktop kernel: *pde = 00000000 Sep 15 14:00:00 arun-desktop kernel: Oops: 0000 Sep 15 14:00:00 arun-desktop kernel: ide-cd cdrom emu10k1 ac97_codec sound soundcore nfs lockd sunrpc parport_pc lp parport autofs iptable_filter ip_tables e100 keybdev mousedev hid input usb-uhc Sep 15 14:00:00 arun-desktop kernel: CPU: 0 Sep 15 14:00:00 arun-desktop kernel: EIP: 0060:[<c015c738>] Not tainted Sep 15 14:00:00 arun-desktop kernel: EFLAGS: 00010246 Sep 15 14:00:00 arun-desktop kernel: Sep 15 14:00:00 arun-desktop kernel: EIP is at destroy_inode [kernel] 0x28 (2.4.20-8) Sep 15 14:00:00 arun-desktop kernel: eax: 00000000 ebx: cee44c80 ecx: 00000000 edx: cee44c80 Sep 15 14:00:00 arun-desktop kernel: esi: cb32ee00 edi: cee44c80 ebp: 000003c6 esp: cff87f84 Sep 15 14:00:01 arun-desktop kernel: ds: 0068 es: 0068 ss: 0068 Sep 15 14:00:01 arun-desktop kernel: Process kswapd (pid: 5, stackpage=cff87000) Sep 15 14:00:01 arun-desktop kernel: Stack: cee44c80 cb32ee00 cb32ee18 c015b563 cee44c80 cb0ff180 cff86000 00000000 Sep 15 14:00:01 arun-desktop kernel: 000001d0 00000000 c015b904 0000057c c013cd5b 00000006 000001d0 cff86000 Sep 15 14:00:01 arun-desktop kernel: cff86000 00000000 cff86305 00000000 c013d023 000001d0 00000002 000001d0 Sep 15 14:00:01 arun-desktop kernel: Call Trace: [<c015b563>] prune_dcache [kernel] 0xc3 (0xcff87f90)) Sep 15 14:00:01 arun-desktop kernel: [<c015b904>] shrink_dcache_memory [kernel] 0x24 (0xcff87fac)) Sep 15 14:00:01 arun-desktop kernel: [<c013cd5b>] do_try_to_free_pages_kswapd [kernel] 0x1b (0xcff87fb4)) Sep 15 14:00:01 arun-desktop kernel: [<c013d023>] kswapd [kernel] 0x83 (0xcff87fd4)) Sep 15 14:00:01 arun-desktop kernel: [<c013cfa0>] kswapd [kernel] 0x0 (0xcff87fe4)) Sep 15 14:00:01 arun-desktop kernel: [<c010742d>] kernel_thread_helper [kernel] 0x5 (0xcff87ff0)) ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-15 21:42 ` [NFS] " Arun Sharma @ 2003-09-15 22:25 ` H. Peter Anvin 2003-09-15 22:34 ` Arun Sharma ` (2 more replies) 0 siblings, 3 replies; 53+ messages in thread From: H. Peter Anvin @ 2003-09-15 22:25 UTC (permalink / raw) To: Arun Sharma; +Cc: Ion Badulescu, autofs, nfs Arun Sharma wrote: >> >> Oh yeah, easily. Kill the automounter while some things are mounted, >> unmount everything by hand but do not rmdir the mountpoints. When you >> unmount the autofs filesystem itself, you will get the self destruct >> message. > > Thanks. This reproduces the problem I'm chasing very well. I think this > is clearly an unacceptable kernel bug. > Indeed. This is on autofs 3? -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-15 22:25 ` H. Peter Anvin @ 2003-09-15 22:34 ` Arun Sharma 2003-09-15 23:42 ` Ion Badulescu 2003-09-16 20:37 ` Re: [NFS] " Arun Sharma 2 siblings, 0 replies; 53+ messages in thread From: Arun Sharma @ 2003-09-15 22:34 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Ion Badulescu, autofs, nfs H. Peter Anvin wrote: > > Indeed. This is on autofs 3? Yes. -Arun ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-15 22:25 ` H. Peter Anvin 2003-09-15 22:34 ` Arun Sharma @ 2003-09-15 23:42 ` Ion Badulescu 2003-09-16 1:34 ` Matt C 2003-09-16 20:37 ` Re: [NFS] " Arun Sharma 2 siblings, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-15 23:42 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Arun Sharma, autofs, nfs On Mon, 15 Sep 2003, H. Peter Anvin wrote: > Arun Sharma wrote: > >> > >> Oh yeah, easily. Kill the automounter while some things are mounted, > >> unmount everything by hand but do not rmdir the mountpoints. When you > >> unmount the autofs filesystem itself, you will get the self destruct > >> message. > > > > Thanks. This reproduces the problem I'm chasing very well. I think this > > is clearly an unacceptable kernel bug. > > > Indeed. This is on autofs 3? Both 3 and 4, at least as far as the "self-destruct" message goes. Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [autofs] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-15 23:42 ` Ion Badulescu @ 2003-09-16 1:34 ` Matt C 0 siblings, 0 replies; 53+ messages in thread From: Matt C @ 2003-09-16 1:34 UTC (permalink / raw) To: Ion Badulescu; +Cc: H. Peter Anvin, Arun Sharma, autofs, nfs Don't know if it helps, but we also run a few hundred autofs4 clients, with a mix of 2.4.21 and 2.4.18. The 2.4.21 machines are seeing quite a few of these messages, but not on the 2.4.18 machines. We haven't seen any of the oopses described below, though. -matt On Mon, 15 Sep 2003, Ion Badulescu wrote: > On Mon, 15 Sep 2003, H. Peter Anvin wrote: > > > Arun Sharma wrote: > > >> > > >> Oh yeah, easily. Kill the automounter while some things are mounted, > > >> unmount everything by hand but do not rmdir the mountpoints. When you > > >> unmount the autofs filesystem itself, you will get the self destruct > > >> message. > > > > > > Thanks. This reproduces the problem I'm chasing very well. I think this > > > is clearly an unacceptable kernel bug. > > > > > Indeed. This is on autofs 3? > > Both 3 and 4, at least as far as the "self-destruct" message goes. > > Ion > > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [autofs] Re: VFS: Busy inodes after unmount on 2 way SMP @ 2003-09-16 1:34 ` Matt C 0 siblings, 0 replies; 53+ messages in thread From: Matt C @ 2003-09-16 1:34 UTC (permalink / raw) To: Ion Badulescu; +Cc: H. Peter Anvin, Arun Sharma, autofs, nfs Don't know if it helps, but we also run a few hundred autofs4 clients, with a mix of 2.4.21 and 2.4.18. The 2.4.21 machines are seeing quite a few of these messages, but not on the 2.4.18 machines. We haven't seen any of the oopses described below, though. -matt On Mon, 15 Sep 2003, Ion Badulescu wrote: > On Mon, 15 Sep 2003, H. Peter Anvin wrote: > > > Arun Sharma wrote: > > >> > > >> Oh yeah, easily. Kill the automounter while some things are mounted, > > >> unmount everything by hand but do not rmdir the mountpoints. When you > > >> unmount the autofs filesystem itself, you will get the self destruct > > >> message. > > > > > > Thanks. This reproduces the problem I'm chasing very well. I think this > > > is clearly an unacceptable kernel bug. > > > > > Indeed. This is on autofs 3? > > Both 3 and 4, at least as far as the "self-destruct" message goes. > > Ion > > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [autofs] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-16 1:34 ` Matt C (?) @ 2003-09-19 2:57 ` Frank Cusack -1 siblings, 0 replies; 53+ messages in thread From: Frank Cusack @ 2003-09-19 2:57 UTC (permalink / raw) To: H. Peter Anvin, autofs, nfs On Mon, Sep 15, 2003 at 06:34:33PM -0700, Matt C wrote: > On Mon, 15 Sep 2003, Ion Badulescu wrote: > > On Mon, 15 Sep 2003, H. Peter Anvin wrote: > > > Arun Sharma wrote: > > > >> > > > >> Oh yeah, easily. Kill the automounter while some things are mounted, > > > >> unmount everything by hand but do not rmdir the mountpoints. When you > > > >> unmount the autofs filesystem itself, you will get the self destruct > > > >> message. > > > > > > > > Thanks. This reproduces the problem I'm chasing very well. I think this > > > > is clearly an unacceptable kernel bug. > > > > > > > Indeed. This is on autofs 3? > > > > Both 3 and 4, at least as far as the "self-destruct" message goes. > > Don't know if it helps, but we also run a few hundred autofs4 clients, > with a mix of 2.4.21 and 2.4.18. The 2.4.21 machines are seeing quite a > few of these messages, but not on the 2.4.18 machines. We haven't seen any > of the oopses described below, though. Same here, 2.4.22, autofs 3 and 4. I'll be playing with 2.6.0 next week. I haven't tried to reproduce it with your method, but I do see the bug; no oops though. /fc ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-15 22:25 ` H. Peter Anvin 2003-09-15 22:34 ` Arun Sharma 2003-09-15 23:42 ` Ion Badulescu @ 2003-09-16 20:37 ` Arun Sharma 2003-09-16 20:54 ` H. Peter Anvin 2 siblings, 1 reply; 53+ messages in thread From: Arun Sharma @ 2003-09-16 20:37 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Ion Badulescu, autofs H. Peter Anvin wrote: > > Indeed. This is on autofs 3? > Correction: the problem was primarily seen on autofs4. However the oops that I reported yesterday was autofs3. So the problem affects both. -Arun ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [NFS] VFS: Busy inodes after unmount on 2 way SMP 2003-09-16 20:37 ` Re: [NFS] " Arun Sharma @ 2003-09-16 20:54 ` H. Peter Anvin 0 siblings, 0 replies; 53+ messages in thread From: H. Peter Anvin @ 2003-09-16 20:54 UTC (permalink / raw) To: Arun Sharma; +Cc: Ion Badulescu, autofs Arun Sharma wrote: > H. Peter Anvin wrote: > >> >> Indeed. This is on autofs 3? >> > > Correction: the problem was primarily seen on autofs4. However the oops > that I reported yesterday was autofs3. So the problem affects both. > > -Arun Thanks. That's useful information. -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-12 0:11 VFS: Busy inodes after unmount on 2 way SMP Arun Sharma 2003-09-12 1:05 ` H. Peter Anvin 2003-09-12 23:33 ` Ion Badulescu @ 2003-09-17 0:28 ` Ian Kent 2003-09-17 2:05 ` Arun Sharma 2003-09-17 13:14 ` Ryan Go 2 siblings, 2 replies; 53+ messages in thread From: Ian Kent @ 2003-09-17 0:28 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs On Thu, 11 Sep 2003, Arun Sharma wrote: > > We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case: > > - Due to network load issues, the NFS server becomes unreachable for some time > - The automounter tries to expire the mount > - The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. have inode->i_count == 1. But they don't have any waiters on inode->i_wait. Further, the inodes that are busy are > /mnt/foo -> autofs mount point > /mnt/foo/bar -> bar is a symbolic link > > It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are: > > fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); > fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); > This is in the autofs (aka v3) kernel module not autofs4. Please specify which version of the userspace daemon and kernel module? -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 0:28 ` Ian Kent @ 2003-09-17 2:05 ` Arun Sharma 2003-09-17 2:28 ` H. Peter Anvin 2003-09-17 4:52 ` Ian Kent 2003-09-17 13:14 ` Ryan Go 1 sibling, 2 replies; 53+ messages in thread From: Arun Sharma @ 2003-09-17 2:05 UTC (permalink / raw) To: Ian Kent; +Cc: autofs, nfs Ian Kent wrote: >>It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are: >> >>fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); >>fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); >> > > > This is in the autofs (aka v3) kernel module not autofs4. > Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon. -Arun ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 2:05 ` Arun Sharma @ 2003-09-17 2:28 ` H. Peter Anvin 2003-09-17 4:56 ` Ian Kent 2003-09-17 4:52 ` Ian Kent 1 sibling, 1 reply; 53+ messages in thread From: H. Peter Anvin @ 2003-09-17 2:28 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs, Ian Kent Arun Sharma wrote: > Ian Kent wrote: > >>> It's not clear if this is a NFS issue or a autofs issue, but it's >>> seen often with autofs. Are there any known race conditions that have >>> been fixed after 2.4.20 ? The two calls I'm worried about are: >>> >>> fs/autofs/root.c:305: d_instantiate(dentry, >>> iget(dir->i_sb,ent->ino)); >>> fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); >>> >> This is in the autofs (aka v3) kernel module not autofs4. >> > Please ignore the above lines. I corrected myself in an earlier message. > The problem was seen on the autofs4 kernel module in 2.4.20 + > autofs-3.1.7 user daemon. > Please don't mix the v3 and v4 stuff (this was a misguided change on Linus' part which I still very much disagree with.) -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 2:28 ` H. Peter Anvin @ 2003-09-17 4:56 ` Ian Kent 0 siblings, 0 replies; 53+ messages in thread From: Ian Kent @ 2003-09-17 4:56 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs On Tue, 16 Sep 2003, H. Peter Anvin wrote: > > Please don't mix the v3 and v4 stuff (this was a misguided change on > Linus' part which I still very much disagree with.) > Yes. It should work but if it doesn't it becomes very difficult to support. -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 2:05 ` Arun Sharma 2003-09-17 2:28 ` H. Peter Anvin @ 2003-09-17 4:52 ` Ian Kent 2003-09-17 17:47 ` Arun Sharma 1 sibling, 1 reply; 53+ messages in thread From: Ian Kent @ 2003-09-17 4:52 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs On Tue, 16 Sep 2003, Arun Sharma wrote: > > Please ignore the above lines. I corrected myself in an earlier message. The problem was seen on the autofs4 kernel module in 2.4.20 + autofs-3.1.7 user daemon. > Vanilla 2.4.20? -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 4:52 ` Ian Kent @ 2003-09-17 17:47 ` Arun Sharma 2003-09-17 20:41 ` H. Peter Anvin 0 siblings, 1 reply; 53+ messages in thread From: Arun Sharma @ 2003-09-17 17:47 UTC (permalink / raw) To: Ian Kent; +Cc: autofs, nfs Ian Kent wrote: > On Tue, 16 Sep 2003, Arun Sharma wrote: > > >> Please ignore the above lines. I corrected myself in an earlier >> message. The problem was seen on the autofs4 kernel module in >> 2.4.20 + autofs-3.1.7 user daemon. > > > Vanilla 2.4.20? > Yes, vanilla 2.4.20. -Arun ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 17:47 ` Arun Sharma @ 2003-09-17 20:41 ` H. Peter Anvin 2003-09-17 21:00 ` Olaf Hering 2003-09-18 2:26 ` [NFS] " Ion Badulescu 0 siblings, 2 replies; 53+ messages in thread From: H. Peter Anvin @ 2003-09-17 20:41 UTC (permalink / raw) To: Arun Sharma; +Cc: autofs, nfs, Ian Kent Okay... let me ask this so I get it straight... Has anyone seen this (busy inodes after stopping the automounter) using autofs v3 kernel module and daemon? -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 20:41 ` H. Peter Anvin @ 2003-09-17 21:00 ` Olaf Hering 2003-09-18 0:31 ` Ian Kent 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust 2003-09-18 2:26 ` [NFS] " Ion Badulescu 1 sibling, 2 replies; 53+ messages in thread From: Olaf Hering @ 2003-09-17 21:00 UTC (permalink / raw) To: H. Peter Anvin; +Cc: Arun Sharma, autofs, nfs, Ian Kent On Wed, Sep 17, H. Peter Anvin wrote: > Okay... let me ask this so I get it straight... > > Has anyone seen this (busy inodes after stopping the automounter) using > autofs v3 kernel module and daemon? I'm not sure, but everyone who has seen this should fiddle this patch into the kernel and see how it goes. The whole thing started very recently (post 2.4.21) for us. This patch is untested, any feedback appreciated. This patch will hopefully fix the "VFS: Busy inodes after unmount. Self-destruct in 5 seconds" oopses generated by autofs+nfs. diff -u -r linux-2.4.21/fs/nfs/inode.c smp/fs/nfs/inode.c --- linux-2.4.21/fs/nfs/inode.c 2003-09-17 17:51:53.000000000 +0200 +++ smp/fs/nfs/inode.c 2003-09-17 21:24:57.000000000 +0200 @@ -976,9 +976,19 @@ int nfs_release(struct inode *inode, struct file *filp) { + struct dentry *dentry; struct rpc_cred *cred; lock_kernel(); + + /* Do the silly unlink here as long as we still have a reference + * on the vfsmount. Otherwise the unlink cache may hold a reference + * to the parent dentry across umount() calls. */ + if ((dentry = filp->f_dentry) && (dentry->d_flags & DCACHE_NFSFS_RENAMED)) { + /* Just trigger the NFS call */ + nfs_complete_unlink(dentry); + } + cred = nfs_file_cred(filp); if (cred) put_rpccred(cred); -- USB is for mice, FireWire is for men! sUse lINUX ag, n√úRNBERG ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 21:00 ` Olaf Hering @ 2003-09-18 0:31 ` Ian Kent 2003-09-18 5:04 ` Olaf Hering 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust 1 sibling, 1 reply; 53+ messages in thread From: Ian Kent @ 2003-09-18 0:31 UTC (permalink / raw) To: Olaf Hering; +Cc: autofs, nfs On Wed, 17 Sep 2003, Olaf Hering wrote: > > This patch is untested, any feedback appreciated. > > This patch will hopefully fix the "VFS: Busy inodes after unmount. > Self-destruct in 5 seconds" oopses generated by autofs+nfs. > Could you elaborate on reasoning behind the patch a bit please. -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 0:31 ` Ian Kent @ 2003-09-18 5:04 ` Olaf Hering 0 siblings, 0 replies; 53+ messages in thread From: Olaf Hering @ 2003-09-18 5:04 UTC (permalink / raw) To: Ian Kent; +Cc: autofs, nfs On Thu, Sep 18, Ian Kent wrote: > On Wed, 17 Sep 2003, Olaf Hering wrote: > > > > > This patch is untested, any feedback appreciated. > > > > This patch will hopefully fix the "VFS: Busy inodes after unmount. > > Self-destruct in 5 seconds" oopses generated by autofs+nfs. > > > > Could you elaborate on reasoning behind the patch a bit please. probably that: ... Additional Comment #10 From Olaf Kirch 2003-09-16 21:13 The way I understood it the code is supposed to work like umount(8) calls umount(2) with MNT_FORCE umount(2) calls NFS to signal all RPC tasks later notices there are still buys inodes and returns to user space (-EBUSY) umount(8) retries umount(2) call If that assumption is incorrect, we indeed to wait for the rpc tasks to complete. Nasty :-( ... -- USB is for mice, FireWire is for men! sUse lINUX ag, n√úRNBERG ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 21:00 ` Olaf Hering 2003-09-18 0:31 ` Ian Kent @ 2003-09-18 5:52 ` Trond Myklebust 2003-09-18 8:26 ` Olaf Kirch ` (2 more replies) 1 sibling, 3 replies; 53+ messages in thread From: Trond Myklebust @ 2003-09-18 5:52 UTC (permalink / raw) To: Olaf Hering; +Cc: H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent >>>>> " " == Olaf Hering <olh@suse.de> writes: > On Wed, Sep 17, H. Peter Anvin wrote: >> Okay... let me ask this so I get it straight... >> >> Has anyone seen this (busy inodes after stopping the >> automounter) using autofs v3 kernel module and daemon? > I'm not sure, but everyone who has seen this should fiddle this > patch into the kernel and see how it goes. The whole thing > started very recently (post 2.4.21) for us. > This patch is untested, any feedback appreciated. ...and is indeed wrong... It does the exact opposite of what sillydelete should do. Instead of causing the last application that closes the file to perform the sillydelete, you are asking the *first* application that closes it to do so. Sillydelete *has* to be tied to dentries. Not files, and not inodes. It is purely a namespace operation... So exactly what are you trying to do, and why? Cheers, Trond ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust @ 2003-09-18 8:26 ` Olaf Kirch 2003-09-19 23:36 ` [NFS] " Olaf Hering 2003-09-25 23:17 ` Matt C 2 siblings, 0 replies; 53+ messages in thread From: Olaf Kirch @ 2003-09-18 8:26 UTC (permalink / raw) To: Trond Myklebust Cc: Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent On Thu, Sep 18, 2003 at 01:52:05AM -0400, Trond Myklebust wrote: > ...and is indeed wrong... It does the exact opposite of what > sillydelete should do. Instead of causing the last application that > closes the file to perform the sillydelete, you are asking the *first* > application that closes it to do so. I know. I was not claiming it's perfectly right, what I'm interested in whether it cures the oopses we're seeing. If that were true, one could still look for The Right Solution... > Sillydelete *has* to be tied to dentries. Not files, and not > inodes. It is purely a namespace operation... > > So exactly what are you trying to do, and why? We have seen several reports of autofs+nfs leading to oopses, preceded by "Busy inodes on umount, self-destruct in 5 seconds". It's somewhat hard to reproduce, so I'm currently trying to come up with possible patches by looking at the source. You cannot umount if the vfsmount mnt_count is != 2. So either something hosed the mnt_count completely (by doing too many mntput() calls for instance), or something holds dentry references without bumping mnt_count along with it. Do you agree? The only place in the nfs client code that actually does a dget is the sillydelete stuff in unlink.c. So my idea in generating this patch was that the nfs_complete_unlink was happening too late, when the file is already closed and the vfsmount reference is gone. It's a long shot admittedly... Have a nice day, Olaf -- Olaf Kirch | Anyone who has had to work with X.509 has probably okir@suse.de | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust 2003-09-18 8:26 ` Olaf Kirch @ 2003-09-19 23:36 ` Olaf Hering 2003-09-25 23:17 ` Matt C 2 siblings, 0 replies; 53+ messages in thread From: Olaf Hering @ 2003-09-19 23:36 UTC (permalink / raw) To: Trond Myklebust Cc: Arun Sharma, autofs, nfs, H. Peter Anvin, Olaf Kirch, Ian Kent On Thu, Sep 18, Trond Myklebust wrote: > >>>>> " " == Olaf Hering <olh@suse.de> writes: > > > On Wed, Sep 17, H. Peter Anvin wrote: > >> Okay... let me ask this so I get it straight... > >> > >> Has anyone seen this (busy inodes after stopping the > >> automounter) using autofs v3 kernel module and daemon? > > > I'm not sure, but everyone who has seen this should fiddle this > > patch into the kernel and see how it goes. The whole thing > > started very recently (post 2.4.21) for us. After thinking more about the conditions, I got a crash with 2.4.19 client. the codebase was unchanged for a long time. But the server side was updated to 2.4.21 with lots of nfs patches. Possible that the newer server side causes trouble for the client. -- USB is for mice, FireWire is for men! sUse lINUX ag, n√úRNBERG ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust @ 2003-09-25 23:17 ` Matt C 2003-09-19 23:36 ` [NFS] " Olaf Hering 2003-09-25 23:17 ` Matt C 2 siblings, 0 replies; 53+ messages in thread From: Matt C @ 2003-09-25 23:17 UTC (permalink / raw) To: Trond Myklebust Cc: Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent Hi All- We've been having the 'VFS: Busy inodes after unmount' problems like crazy on our desktop and server linux boxes running autofs under 2.4.21 (vanilla). We're using the autofs-3.1.7 userspace tools, but with the autofs4 kernel module. I applied the patch that was attached earlier in this thread to our kernel and have been running it for the past few days on my desktop. These errors have completely disappeared, and autofs is behaving as expected so far. So, while it might be the wrong thing to do, it does fix our problems. -matt On 18 Sep 2003, Trond Myklebust wrote: > >>>>> " " == Olaf Hering <olh@suse.de> writes: > > > On Wed, Sep 17, H. Peter Anvin wrote: > >> Okay... let me ask this so I get it straight... > >> > >> Has anyone seen this (busy inodes after stopping the > >> automounter) using autofs v3 kernel module and daemon? > > > I'm not sure, but everyone who has seen this should fiddle this > > patch into the kernel and see how it goes. The whole thing > > started very recently (post 2.4.21) for us. > > > This patch is untested, any feedback appreciated. > > ...and is indeed wrong... It does the exact opposite of what > sillydelete should do. Instead of causing the last application that > closes the file to perform the sillydelete, you are asking the *first* > application that closes it to do so. > Sillydelete *has* to be tied to dentries. Not files, and not > inodes. It is purely a namespace operation... > > So exactly what are you trying to do, and why? > > Cheers, > Trond > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > NFS maillist - NFS@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP @ 2003-09-25 23:17 ` Matt C 0 siblings, 0 replies; 53+ messages in thread From: Matt C @ 2003-09-25 23:17 UTC (permalink / raw) To: Trond Myklebust Cc: Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent Hi All- We've been having the 'VFS: Busy inodes after unmount' problems like crazy on our desktop and server linux boxes running autofs under 2.4.21 (vanilla). We're using the autofs-3.1.7 userspace tools, but with the autofs4 kernel module. I applied the patch that was attached earlier in this thread to our kernel and have been running it for the past few days on my desktop. These errors have completely disappeared, and autofs is behaving as expected so far. So, while it might be the wrong thing to do, it does fix our problems. -matt On 18 Sep 2003, Trond Myklebust wrote: > >>>>> " " == Olaf Hering <olh@suse.de> writes: > > > On Wed, Sep 17, H. Peter Anvin wrote: > >> Okay... let me ask this so I get it straight... > >> > >> Has anyone seen this (busy inodes after stopping the > >> automounter) using autofs v3 kernel module and daemon? > > > I'm not sure, but everyone who has seen this should fiddle this > > patch into the kernel and see how it goes. The whole thing > > started very recently (post 2.4.21) for us. > > > This patch is untested, any feedback appreciated. > > ...and is indeed wrong... It does the exact opposite of what > sillydelete should do. Instead of causing the last application that > closes the file to perform the sillydelete, you are asking the *first* > application that closes it to do so. > Sillydelete *has* to be tied to dentries. Not files, and not > inodes. It is purely a namespace operation... > > So exactly what are you trying to do, and why? > > Cheers, > Trond > > > ------------------------------------------------------- > This sf.net email is sponsored by:ThinkGeek > Welcome to geek heaven. > http://thinkgeek.com/sf > _______________________________________________ > NFS maillist - NFS@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs > ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-25 23:17 ` Matt C @ 2003-09-26 0:24 ` Trond Myklebust -1 siblings, 0 replies; 53+ messages in thread From: Trond Myklebust @ 2003-09-26 0:24 UTC (permalink / raw) To: Matt C Cc: Trond Myklebust, Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent >>>>> " " == Matt C <Matt> writes: > Hi All- We've been having the 'VFS: Busy inodes after unmount' > problems like crazy on our desktop and server linux boxes > running autofs under 2.4.21 (vanilla). We're using the > autofs-3.1.7 userspace tools, but with the autofs4 kernel > module. > I applied the patch that was attached earlier in this thread to > our kernel and have been running it for the past few days on my > desktop. These errors have completely disappeared, and autofs > is behaving as expected so far. > So, while it might be the wrong thing to do, it does fix our > problems. That changes nothing. It is still not a "fix" that I want to see go into the main kernel. The problem here is that sillyrename needs to hold onto the parent directory's inode until it is done, and yet it cannot know which vfsmount struct to hold onto because that information is not passed down by the VFS. I would rather then have something like the appended patch go in to which force the last process to close the file to actually wait on completion (unless someone signals it with a 'kill -9' - in which case they are knowingly breaking things). Cheers, Trond diff -u --recursive --new-file linux-2.4.22-03-soft2/fs/nfs/dir.c linux-2.4.22-04-fix_unlink/fs/nfs/dir.c --- linux-2.4.22-03-soft2/fs/nfs/dir.c 2002-10-15 06:59:27.000000000 +0200 +++ linux-2.4.22-04-fix_unlink/fs/nfs/dir.c 2003-07-03 12:29:28.000000000 +0200 @@ -994,7 +994,7 @@ struct inode *old_inode = old_dentry->d_inode; struct inode *new_inode = new_dentry->d_inode; struct dentry *dentry = NULL, *rehash = NULL; - int error = -EBUSY; + int error; /* * To prevent any new references to the target during the rename, @@ -1020,6 +1020,12 @@ */ if (!new_inode) goto go_ahead; + /* If target is a hard link to the source, then noop */ + error = 0; + if (NFS_FILEID(new_inode) == NFS_FILEID(old_inode)) + goto out; + + error = -EBUSY; if (S_ISDIR(new_inode->i_mode)) goto out; else if (atomic_read(&new_dentry->d_count) > 1) { diff -u --recursive --new-file linux-2.4.22-03-soft2/fs/nfs/unlink.c linux-2.4.22-04-fix_unlink/fs/nfs/unlink.c --- linux-2.4.22-03-soft2/fs/nfs/unlink.c 2002-08-11 13:34:02.000000000 +0200 +++ linux-2.4.22-04-fix_unlink/fs/nfs/unlink.c 2003-07-03 12:29:28.000000000 +0200 @@ -12,6 +12,7 @@ #include <linux/sunrpc/sched.h> #include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> +#include <linux/wait.h> struct nfs_unlinkdata { @@ -21,6 +22,9 @@ struct rpc_task task; struct rpc_cred *cred; unsigned int count; + + wait_queue_head_t waitq; + int completed; }; static struct nfs_unlinkdata *nfs_deletes; @@ -133,6 +137,8 @@ put_rpccred(data->cred); data->cred = NULL; dput(dir); + data->completed = 1; + wake_up(&data->waitq); } /** @@ -175,6 +181,8 @@ nfs_deletes = data; data->count = 1; + init_waitqueue_head(&data->waitq); + task = &data->task; rpc_init_task(task, clnt, nfs_async_unlink_done , RPC_TASK_ASYNC); task->tk_calldata = data; @@ -212,7 +220,10 @@ data->count++; nfs_copy_dname(dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED; - if (data->task.tk_rpcwait == &nfs_delete_queue) + if (data->task.tk_rpcwait == &nfs_delete_queue) { + struct rpc_clnt *clnt = data->task.tk_client; rpc_wake_up_task(&data->task); + nfs_wait_event(clnt, data->waitq, data->completed == 1); + } nfs_put_unlinkdata(data); } ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP @ 2003-09-26 0:24 ` Trond Myklebust 0 siblings, 0 replies; 53+ messages in thread From: Trond Myklebust @ 2003-09-26 0:24 UTC (permalink / raw) To: Matt C Cc: Trond Myklebust, Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent >>>>> " " == Matt C <Matt> writes: > Hi All- We've been having the 'VFS: Busy inodes after unmount' > problems like crazy on our desktop and server linux boxes > running autofs under 2.4.21 (vanilla). We're using the > autofs-3.1.7 userspace tools, but with the autofs4 kernel > module. > I applied the patch that was attached earlier in this thread to > our kernel and have been running it for the past few days on my > desktop. These errors have completely disappeared, and autofs > is behaving as expected so far. > So, while it might be the wrong thing to do, it does fix our > problems. That changes nothing. It is still not a "fix" that I want to see go into the main kernel. The problem here is that sillyrename needs to hold onto the parent directory's inode until it is done, and yet it cannot know which vfsmount struct to hold onto because that information is not passed down by the VFS. I would rather then have something like the appended patch go in to which force the last process to close the file to actually wait on completion (unless someone signals it with a 'kill -9' - in which case they are knowingly breaking things). Cheers, Trond diff -u --recursive --new-file linux-2.4.22-03-soft2/fs/nfs/dir.c linux-2.4.22-04-fix_unlink/fs/nfs/dir.c --- linux-2.4.22-03-soft2/fs/nfs/dir.c 2002-10-15 06:59:27.000000000 +0200 +++ linux-2.4.22-04-fix_unlink/fs/nfs/dir.c 2003-07-03 12:29:28.000000000 +0200 @@ -994,7 +994,7 @@ struct inode *old_inode = old_dentry->d_inode; struct inode *new_inode = new_dentry->d_inode; struct dentry *dentry = NULL, *rehash = NULL; - int error = -EBUSY; + int error; /* * To prevent any new references to the target during the rename, @@ -1020,6 +1020,12 @@ */ if (!new_inode) goto go_ahead; + /* If target is a hard link to the source, then noop */ + error = 0; + if (NFS_FILEID(new_inode) == NFS_FILEID(old_inode)) + goto out; + + error = -EBUSY; if (S_ISDIR(new_inode->i_mode)) goto out; else if (atomic_read(&new_dentry->d_count) > 1) { diff -u --recursive --new-file linux-2.4.22-03-soft2/fs/nfs/unlink.c linux-2.4.22-04-fix_unlink/fs/nfs/unlink.c --- linux-2.4.22-03-soft2/fs/nfs/unlink.c 2002-08-11 13:34:02.000000000 +0200 +++ linux-2.4.22-04-fix_unlink/fs/nfs/unlink.c 2003-07-03 12:29:28.000000000 +0200 @@ -12,6 +12,7 @@ #include <linux/sunrpc/sched.h> #include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> +#include <linux/wait.h> struct nfs_unlinkdata { @@ -21,6 +22,9 @@ struct rpc_task task; struct rpc_cred *cred; unsigned int count; + + wait_queue_head_t waitq; + int completed; }; static struct nfs_unlinkdata *nfs_deletes; @@ -133,6 +137,8 @@ put_rpccred(data->cred); data->cred = NULL; dput(dir); + data->completed = 1; + wake_up(&data->waitq); } /** @@ -175,6 +181,8 @@ nfs_deletes = data; data->count = 1; + init_waitqueue_head(&data->waitq); + task = &data->task; rpc_init_task(task, clnt, nfs_async_unlink_done , RPC_TASK_ASYNC); task->tk_calldata = data; @@ -212,7 +220,10 @@ data->count++; nfs_copy_dname(dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED; - if (data->task.tk_rpcwait == &nfs_delete_queue) + if (data->task.tk_rpcwait == &nfs_delete_queue) { + struct rpc_clnt *clnt = data->task.tk_client; rpc_wake_up_task(&data->task); + nfs_wait_event(clnt, data->waitq, data->completed == 1); + } nfs_put_unlinkdata(data); } ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-26 0:24 ` Trond Myklebust (?) @ 2003-09-26 18:31 ` Ion Badulescu 2003-09-26 22:29 ` Re: [autofs] " Trond Myklebust -1 siblings, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-26 18:31 UTC (permalink / raw) To: trond.myklebust; +Cc: autofs, nfs On Thu, 25 Sep 2003 17:24:49 -0700, Trond Myklebust <trond.myklebust@fys.uio.no> wrote: > I would rather then have something like the appended patch go in to > which force the last process to close the file to actually wait on > completion (unless someone signals it with a 'kill -9' - in which case > they are knowingly breaking things). Ugh, that's ugly -- and potentially a DoS issue as well. Since you don't need any privileges to issue the 'kill -9' for your own processes, it becomes fairly easy to write some code that exploits this and causes kernel crashes across the board... Not that the current situation is any better, of course. Perhaps making that wait an uninterruptable wait would be enough to take care of the problem? Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-26 18:31 ` [NFS] " Ion Badulescu @ 2003-09-26 22:29 ` Trond Myklebust 2003-09-27 16:55 ` Olaf Kirch 2003-09-29 12:07 ` Ion Badulescu 0 siblings, 2 replies; 53+ messages in thread From: Trond Myklebust @ 2003-09-26 22:29 UTC (permalink / raw) To: Ion Badulescu; +Cc: nfs >>>>> " " == Ion Badulescu <lists@news.cs.columbia.edu> writes: > Ugh, that's ugly -- and potentially a DoS issue as well. Since > you don't need any privileges to issue the 'kill -9' for your > own processes, it becomes fairly easy to write some code that > exploits this and causes kernel crashes across the board... Then don't set the "intr" flag. > Not that the current situation is any better, of course. > Perhaps making that wait an uninterruptable wait would be > enough to take care of the problem? People expect the "intr" flag to work if they set it. Cheers, Trond ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-26 22:29 ` Re: [autofs] " Trond Myklebust @ 2003-09-27 16:55 ` Olaf Kirch 2003-09-28 23:16 ` Steve Fosdick 2003-09-29 12:07 ` Ion Badulescu 1 sibling, 1 reply; 53+ messages in thread From: Olaf Kirch @ 2003-09-27 16:55 UTC (permalink / raw) To: Trond Myklebust; +Cc: Ion Badulescu, nfs On Fri, Sep 26, 2003 at 03:29:40PM -0700, Trond Myklebust wrote: > > Not that the current situation is any better, of course. > > > Perhaps making that wait an uninterruptable wait would be > > enough to take care of the problem? > > People expect the "intr" flag to work if they set it. In this case, not removing the sillyrenamed file instead of oopsing may be the preferred action. Olaf -- Olaf Kirch | Anyone who has had to work with X.509 has probably okir@suse.de | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-27 16:55 ` Olaf Kirch @ 2003-09-28 23:16 ` Steve Fosdick 0 siblings, 0 replies; 53+ messages in thread From: Steve Fosdick @ 2003-09-28 23:16 UTC (permalink / raw) To: nfs On 2003.09.27 17:55, Olaf Kirch wrote: > In this case, not removing the sillyrenamed file instead > of oopsing may be the preferred action. Would it be possibly to combine both ideas, i.e. have the last process wait, but if the user attempt to kill that process and it will be allowed because intr is set the sillyrenamed file is not removed. Surely it is reasonable for people who kill -9 things to expect to do some tidying up, but not to expect a kernel panic. Steve. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-26 22:29 ` Re: [autofs] " Trond Myklebust 2003-09-27 16:55 ` Olaf Kirch @ 2003-09-29 12:07 ` Ion Badulescu 2003-09-29 17:22 ` Trond Myklebust 1 sibling, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-29 12:07 UTC (permalink / raw) To: Trond Myklebust; +Cc: nfs On Fri, 26 Sep 2003, Trond Myklebust wrote: > > Perhaps making that wait an uninterruptable wait would be > > enough to take care of the problem? > > People expect the "intr" flag to work if they set it. People also expect the supported "intr" flag not to cause security problems and would gladly take an unkillable process over a kernel oops... Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-29 12:07 ` Ion Badulescu @ 2003-09-29 17:22 ` Trond Myklebust 2003-09-30 12:50 ` Olaf Kirch 0 siblings, 1 reply; 53+ messages in thread From: Trond Myklebust @ 2003-09-29 17:22 UTC (permalink / raw) To: Ion Badulescu; +Cc: nfs >>>>> " " == Ion Badulescu <lists@news.cs.columbia.edu> writes: >> People expect the "intr" flag to work if they set it. > People also expect the supported "intr" flag not to cause > security problems and would gladly take an unkillable process > over a kernel oops... You'd have to be extremely unlucky to kill the process and hit the window for the Oops. I defy you to come up with an exploit for it. That said, I agree that a full fix would be preferable. I'm working on other projects right now, that's why I'm being slow about this issue (plus the fact that it's not exactly easy to reproduce). I'll get onto it soon... Cheers, Trond ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-29 17:22 ` Trond Myklebust @ 2003-09-30 12:50 ` Olaf Kirch 2003-09-30 13:31 ` Trond Myklebust 0 siblings, 1 reply; 53+ messages in thread From: Olaf Kirch @ 2003-09-30 12:50 UTC (permalink / raw) To: Trond Myklebust; +Cc: Ion Badulescu, nfs [-- Attachment #1: Type: text/plain, Size: 1617 bytes --] On Mon, Sep 29, 2003 at 10:22:40AM -0700, Trond Myklebust wrote: > You'd have to be extremely unlucky to kill the process and hit the > window for the Oops. I defy you to come up with an exploit for it. > > That said, I agree that a full fix would be preferable. I'm working on > other projects right now, that's why I'm being slow about this issue > (plus the fact that it's not exactly easy to reproduce). I'll get onto > it soon... I've had no luck reproducing it either in a controlled environment. However the bug is common enough to crash a few of our build machines over a weekend... I took your patch, Trond, and modified it a little to not crash if the user ctrl-c's the unlink. Still not pretty - maybe it should zap the cached attributes: @@ -212,7 +222,15 @@ data->count++; nfs_copy_dname(dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED; - if (data->task.tk_rpcwait == &nfs_delete_queue) + if (data->task.tk_rpcwait == &nfs_delete_queue) { + struct rpc_clnt *clnt = data->task.tk_client; rpc_wake_up_task(&data->task); + nfs_wait_event(clnt, data->waitq, data->completed == 1); + /* This is safe as we hold the BKL */ + if (!data->completed) { + dput(data->dir); + data->dir = NULL; + } + } nfs_put_unlinkdata(data); } Olaf -- Olaf Kirch | Anyone who has had to work with X.509 has probably okir@suse.de | experienced what can best be described as ---------------+ ISO water torture. -- Peter Gutmann [-- Attachment #2: nfs-autofs-umount-crash --] [-- Type: text/plain, Size: 2362 bytes --] diff -ur linux-2.4.21/fs/nfs/dir.c nfs/fs/nfs/dir.c --- linux-2.4.21/fs/nfs/dir.c 2003-09-29 10:33:41.000000000 +0200 +++ nfs/fs/nfs/dir.c 2003-09-29 12:34:36.000000000 +0200 @@ -1144,7 +1144,7 @@ struct inode *old_inode = old_dentry->d_inode; struct inode *new_inode = new_dentry->d_inode; struct dentry *dentry = NULL, *rehash = NULL; - int error = -EBUSY; + int error; /* * To prevent any new references to the target during the rename, @@ -1170,6 +1170,12 @@ */ if (!new_inode) goto go_ahead; + /* If target is a hard link to the source, then noop */ + error = 0; + if (NFS_FILEID(new_inode) == NFS_FILEID(old_inode)) + goto out; + + error = -EBUSY; if (S_ISDIR(new_inode->i_mode)) goto out; else if (atomic_read(&new_dentry->d_count) > 1) { diff -ur linux-2.4.21/fs/nfs/unlink.c nfs/fs/nfs/unlink.c --- linux-2.4.21/fs/nfs/unlink.c 2002-11-29 00:53:15.000000000 +0100 +++ nfs/fs/nfs/unlink.c 2003-09-29 13:38:49.000000000 +0200 @@ -12,6 +12,7 @@ #include <linux/sunrpc/sched.h> #include <linux/sunrpc/clnt.h> #include <linux/nfs_fs.h> +#include <linux/wait.h> struct nfs_unlinkdata { @@ -21,6 +22,9 @@ struct rpc_task task; struct rpc_cred *cred; unsigned int count; + + wait_queue_head_t waitq; + int completed; }; static struct nfs_unlinkdata *nfs_deletes; @@ -54,6 +58,8 @@ nfs_detach_unlinkdata(data); if (data->name.name != NULL) kfree(data->name.name); + if (data->cred) + put_rpccred(data->cred); kfree(data); } } @@ -133,6 +139,8 @@ put_rpccred(data->cred); data->cred = NULL; dput(dir); + data->completed = 1; + wake_up(&data->waitq); } /** @@ -175,6 +183,8 @@ nfs_deletes = data; data->count = 1; + init_waitqueue_head(&data->waitq); + task = &data->task; rpc_init_task(task, clnt, nfs_async_unlink_done , RPC_TASK_ASYNC); task->tk_calldata = data; @@ -212,7 +222,15 @@ data->count++; nfs_copy_dname(dentry, data); dentry->d_flags &= ~DCACHE_NFSFS_RENAMED; - if (data->task.tk_rpcwait == &nfs_delete_queue) + if (data->task.tk_rpcwait == &nfs_delete_queue) { + struct rpc_clnt *clnt = data->task.tk_client; rpc_wake_up_task(&data->task); + nfs_wait_event(clnt, data->waitq, data->completed == 1); + /* This is safe as we hold the BKL */ + if (!data->completed) { + dput(data->dir); + data->dir = NULL; + } + } nfs_put_unlinkdata(data); } ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-30 12:50 ` Olaf Kirch @ 2003-09-30 13:31 ` Trond Myklebust 0 siblings, 0 replies; 53+ messages in thread From: Trond Myklebust @ 2003-09-30 13:31 UTC (permalink / raw) To: Olaf Kirch; +Cc: Ion Badulescu, nfs >>>>> " " == Olaf Kirch <okir@suse.de> writes: > On Mon, Sep 29, 2003 at 10:22:40AM -0700, Trond Myklebust >> That said, I agree that a full fix would be preferable. I'm >> working on other projects right now, that's why I'm being slow >> about this issue (plus the fact that it's not exactly easy to >> reproduce). I'll get onto it soon... > I've had no luck reproducing it either in a controlled > environment. However the bug is common enough to crash a few > of our build machines over a weekend... > I took your patch, Trond, and modified it a little to not crash > if the user ctrl-c's the unlink. Still not pretty - maybe it > should zap the cached attributes: Alternatively, we should perhaps make fsync_super() sync out all dirty inodes and readaheads, then wait on all pending RPC ops. That would be cleaner, I think... Cheers, Trond ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: Re: [autofs] VFS: Busy inodes after unmount on 2 way SMP 2003-09-25 23:17 ` Matt C (?) (?) @ 2003-09-29 3:27 ` Frank Cusack -1 siblings, 0 replies; 53+ messages in thread From: Frank Cusack @ 2003-09-29 3:27 UTC (permalink / raw) To: Matt C Cc: Trond Myklebust, Olaf Hering, H. Peter Anvin, Arun Sharma, autofs, nfs, Ian Kent Does this patch help? --- linux-2.4.22/fs/namei.c 2003-08-25 04:44:43.000000000 -0700 +++ linux-2.4.22-gg4/fs/namei.c 2003-09-26 00:03:16.000000000 -0700 @@ -893,6 +893,8 @@ static inline int check_sticky(struct in * 7. If we were asked to remove a directory and victim isn't one - ENOTDIR. * 8. If we were asked to remove a non-directory and victim isn't one - EISDIR. * 9. We can't remove a root or mountpoint. + * 10. We don't allow removal of NFS sillyrenamed files; it's handled by + * nfs_async_unlink(). */ static inline int may_delete(struct inode *dir,struct dentry *victim, int isdir) { @@ -916,6 +918,8 @@ static inline int may_delete(struct inod return -EISDIR; if (IS_DEADDIR(dir)) return -ENOENT; + if (victim->d_flags & DCACHE_NFSFS_RENAMED) + return -EBUSY; return 0; } @@ -1484,13 +1488,14 @@ int vfs_unlink(struct inode *dir, struct lock_kernel(); error = dir->i_op->unlink(dir, dentry); unlock_kernel(); - if (!error) + if (!error && + !(dentry->d_flags & DCACHE_NFSFS_RENAMED)) d_delete(dentry); } } } up(&dir->i_zombie); - if (!error) + if (!error && !(dentry->d_flags & DCACHE_NFSFS_RENAMED)) inode_dir_notify(dir, DN_DELETE); return error; } ------------------------------------------------------- This sf.net email is sponsored by:ThinkGeek Welcome to geek heaven. http://thinkgeek.com/sf _______________________________________________ NFS maillist - NFS@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 20:41 ` H. Peter Anvin 2003-09-17 21:00 ` Olaf Hering @ 2003-09-18 2:26 ` Ion Badulescu 2003-09-18 2:54 ` H. Peter Anvin 1 sibling, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-18 2:26 UTC (permalink / raw) To: H. Peter Anvin; +Cc: autofs, Arun Sharma, nfs, Ian Kent On Wed, 17 Sep 2003 13:41:15 -0700, H. Peter Anvin <hpa@zytor.com> wrote: > Okay... let me ask this so I get it straight... > > Has anyone seen this (busy inodes after stopping the automounter) using > autofs v3 kernel module and daemon? Why do you bring the daemon into the picture, it is irrelevant here. Sure it probably does things differently which might avoid the error, but the kernel issue exists nonetheless. To answer your question again: yes, the issue is present in both v3 and v4 kernel modules. You can always replicate it yourself if you don't believe me, using the little recipe I posted earlier: # lsmod | grep autofs autofs 11236 4 (autoclean) # cat /proc/mounts [snip] boxter:(pid29327) /bar autofs rw 0 0 boxter:(pid29327) /net autofs rw 0 0 boxter:(pid29327) /src autofs rw 0 0 boxter:(pid29327) /misc autofs rw 0 0 buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 # umount /misc/homes # umount /misc # dmesg |tail VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day... Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 2:26 ` [NFS] " Ion Badulescu @ 2003-09-18 2:54 ` H. Peter Anvin 2003-09-29 12:19 ` Ion Badulescu 0 siblings, 1 reply; 53+ messages in thread From: H. Peter Anvin @ 2003-09-18 2:54 UTC (permalink / raw) To: Ion Badulescu; +Cc: autofs, Arun Sharma, nfs, Ian Kent Ion Badulescu wrote: > On Wed, 17 Sep 2003 13:41:15 -0700, H. Peter Anvin <hpa@zytor.com> wrote: > >>Okay... let me ask this so I get it straight... >> >>Has anyone seen this (busy inodes after stopping the automounter) using >>autofs v3 kernel module and daemon? > > Why do you bring the daemon into the picture, it is irrelevant here. Because it makes it a lot harder to debug the problem. > Sure it probably does things differently which might avoid the error, > but the kernel issue exists nonetheless. > > To answer your question again: yes, the issue is present in both v3 and > v4 kernel modules. You can always replicate it yourself if you don't > believe me, using the little recipe I posted earlier: > > # lsmod | grep autofs > autofs 11236 4 (autoclean) > # cat /proc/mounts > [snip] > boxter:(pid29327) /bar autofs rw 0 0 > boxter:(pid29327) /net autofs rw 0 0 > boxter:(pid29327) /src autofs rw 0 0 > boxter:(pid29327) /misc autofs rw 0 0 > buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 > # umount /misc/homes > # umount /misc > # dmesg |tail > VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day... OK. Did you try Olaf's NFS patch? -hpa ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-18 2:54 ` H. Peter Anvin @ 2003-09-29 12:19 ` Ion Badulescu 2003-09-30 12:24 ` Ian Kent 0 siblings, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-29 12:19 UTC (permalink / raw) To: H. Peter Anvin; +Cc: autofs, Arun Sharma, nfs, Ian Kent On Wed, 17 Sep 2003, H. Peter Anvin wrote: > > Why do you bring the daemon into the picture, it is irrelevant here. > > Because it makes it a lot harder to debug the problem. Ok, fair enough. The daemon version doesn't matter, any daemon will do. All you need is to kill -9 automountd while something is mounted, and then unmount everything by hand without removing the mountpoint directories. > > # cat /proc/mounts > > [snip] > > boxter:(pid29327) /bar autofs rw 0 0 > > boxter:(pid29327) /net autofs rw 0 0 > > boxter:(pid29327) /src autofs rw 0 0 > > boxter:(pid29327) /misc autofs rw 0 0 > > buggy:/n/buggy/misc/homes /misc/homes nfs rw,v3,rsize=8192,wsize=8192,hard,intr,tcp,lock,addr=buggy 0 0 > > # umount /misc/homes > > # umount /misc > > # dmesg |tail > > VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day... > > OK. Did you try Olaf's NFS patch? This is not an NFS problem. The message occurs regardless of what is mounted on /misc/homes (NFS, local ext2, bind mount, whatever). Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-29 12:19 ` Ion Badulescu @ 2003-09-30 12:24 ` Ian Kent 2003-09-30 12:51 ` Mike Waychison 2003-09-30 13:12 ` Ion Badulescu 0 siblings, 2 replies; 53+ messages in thread From: Ian Kent @ 2003-09-30 12:24 UTC (permalink / raw) To: Ion Badulescu; +Cc: autofs, Arun Sharma, nfs, H. Peter Anvin On Mon, 29 Sep 2003, Ion Badulescu wrote: > On Wed, 17 Sep 2003, H. Peter Anvin wrote: > > > > Why do you bring the daemon into the picture, it is irrelevant here. > > > > Because it makes it a lot harder to debug the problem. > > Ok, fair enough. > > The daemon version doesn't matter, any daemon will do. All you need is to > kill -9 automountd while something is mounted, and then unmount everything > by hand without removing the mountpoint directories. > Yes, your'e exactly right. That's expected behaviour as the directory dentries still exist and there is no daemon to remove them. Hence the message. Seeing this message has always been a signal to me that the daemon has a bug. I have been thinking that implementing a umount_begin routine in the super_operations for autofs4 would remedy the problem. I believe that, correct me if I'm wrong, the umount must be done with a force option in this case, so the umount_begin would be called. snip ... > > > > OK. Did you try Olaf's NFS patch? > > This is not an NFS problem. The message occurs regardless of what is > mounted on /misc/homes (NFS, local ext2, bind mount, whatever). I must admit, reading through the thread, I was wondering how it could be an NFS problem. -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-30 12:24 ` Ian Kent @ 2003-09-30 12:51 ` Mike Waychison 2003-10-01 12:56 ` Ian Kent 2003-09-30 13:12 ` Ion Badulescu 1 sibling, 1 reply; 53+ messages in thread From: Mike Waychison @ 2003-09-30 12:51 UTC (permalink / raw) To: Ian Kent; +Cc: autofs, Arun Sharma, Ion Badulescu, nfs, H. Peter Anvin Ian Kent wrote: >>>OK. Did you try Olaf's NFS patch? >>> >>> >>This is not an NFS problem. The message occurs regardless of what is >>mounted on /misc/homes (NFS, local ext2, bind mount, whatever). >> >> > >I must admit, reading through the thread, I was wondering how it could >be an NFS problem. > > > Inodes are only initialized once, when they are first allocated from the slab. Any reused inodes will continue having a bad refcount, causing similar behaviour on different filesystems. Can this be reproduced on a filesystem other than NFS for which the system has never made an NFS mount? Mike Waychison ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-30 12:51 ` Mike Waychison @ 2003-10-01 12:56 ` Ian Kent 0 siblings, 0 replies; 53+ messages in thread From: Ian Kent @ 2003-10-01 12:56 UTC (permalink / raw) To: Mike Waychison; +Cc: autofs, Arun Sharma, Ion Badulescu, nfs, H. Peter Anvin On Tue, 30 Sep 2003, Mike Waychison wrote: > Ian Kent wrote: > > > >I must admit, reading through the thread, I was wondering how it could > >be an NFS problem. > > > > > > > Inodes are only initialized once, when they are first allocated from the > slab. Any reused inodes will continue having a bad refcount, causing > similar behaviour on different filesystems. > > Can this be reproduced on a filesystem other than NFS for which the > system has never made an NFS mount? > All my usage is NFS so I can't say. -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-30 12:24 ` Ian Kent 2003-09-30 12:51 ` Mike Waychison @ 2003-09-30 13:12 ` Ion Badulescu 2003-09-30 16:44 ` Ian Kent 1 sibling, 1 reply; 53+ messages in thread From: Ion Badulescu @ 2003-09-30 13:12 UTC (permalink / raw) To: Ian Kent; +Cc: autofs, Arun Sharma, H. Peter Anvin Hi Ian, On Tue, 30 Sep 2003, Ian Kent wrote: > On Mon, 29 Sep 2003, Ion Badulescu wrote: > > The daemon version doesn't matter, any daemon will do. All you need is to > > kill -9 automountd while something is mounted, and then unmount everything > > by hand without removing the mountpoint directories. > > > > Yes, your'e exactly right. That's expected behaviour as the directory > dentries still exist and there is no daemon to remove them. Hmm. Expected behaviour, you say? Maybe, maybe not, it's hard to tell what's to be expected here, since the autofs umount semantics are so different from a regular filesystems. Normally a filesystem is allowed to be unmounted if it is not busy, right? But with autofs it's different, it should be unmounted only if it is completely empty, with no entries at all -- and I'm not convinced the VFS layer is up to the task of dealing with this requirement. Alternatively the autofs code should be prepared to clean up after itself, but that might not be possible in all circumstances. > Hence the message. Seeing this message has always been a signal to me > that the daemon has a bug. Oh, you're absolutely right here (although we're talking about different daemons, I'm using--and expanding, and debugging--amd's autofs support). :-) Nonetheless, it's a bug in the kernel code that should be remedied. > I have been thinking that implementing a umount_begin routine in the > super_operations for autofs4 would remedy the problem. I believe that, > correct me if I'm wrong, the umount must be done with a force option in > this case, so the umount_begin would be called. No, you're right, you need MNT_FORCE for umount_begin to be called. That, however, would be a reasonable solution if the filesystem also prevented its own unmounting when it is not empty (by having each entry hold a reference to the vfsmount in its private data, perhaps, I'm not sure). Then, inside umount_begin, and only if the filesystem is catatonic, it would garbage-collect any entries not mounted upon and not busy. That should work, I think... Thanks, Ion -- It is better to keep your mouth shut and be thought a fool, than to open it and remove all doubt. ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: [NFS] Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-30 13:12 ` Ion Badulescu @ 2003-09-30 16:44 ` Ian Kent 0 siblings, 0 replies; 53+ messages in thread From: Ian Kent @ 2003-09-30 16:44 UTC (permalink / raw) To: Ion Badulescu; +Cc: autofs, Arun Sharma, H. Peter Anvin On Tue, 30 Sep 2003, Ion Badulescu wrote: > Hi Ian, > > On Tue, 30 Sep 2003, Ian Kent wrote: > > > On Mon, 29 Sep 2003, Ion Badulescu wrote: > > > The daemon version doesn't matter, any daemon will do. All you need is to > > > kill -9 automountd while something is mounted, and then unmount everything > > > by hand without removing the mountpoint directories. > > > > > > > Yes, your'e exactly right. That's expected behaviour as the directory > > dentries still exist and there is no daemon to remove them. > > Hmm. Expected behaviour, you say? Maybe, maybe not, it's hard to tell > what's to be expected here, since the autofs umount semantics are so > different from a regular filesystems. For autofs v4 anyway possibly v3 also. Cause of the nature of it. > > Normally a filesystem is allowed to be unmounted if it is not busy, right? > But with autofs it's different, it should be unmounted only if it is > completely empty, with no entries at all -- and I'm not convinced the VFS > layer is up to the task of dealing with this requirement. Alternatively > the autofs code should be prepared to clean up after itself, but that > might not be possible in all circumstances. > > > Hence the message. Seeing this message has always been a signal to me > > that the daemon has a bug. > > Oh, you're absolutely right here (although we're talking about different > daemons, I'm using--and expanding, and debugging--amd's autofs support). > :-) Nonetheless, it's a bug in the kernel code that should be remedied. > > > I have been thinking that implementing a umount_begin routine in the > > super_operations for autofs4 would remedy the problem. I believe that, > > correct me if I'm wrong, the umount must be done with a force option in > > this case, so the umount_begin would be called. > > No, you're right, you need MNT_FORCE for umount_begin to be called. > > That, however, would be a reasonable solution if the filesystem also > prevented its own unmounting when it is not empty (by having each entry > hold a reference to the vfsmount in its private data, perhaps, I'm not > sure). Then, inside umount_begin, and only if the filesystem is catatonic, > it would garbage-collect any entries not mounted upon and not busy. That > should work, I think... As you point out I am talking about autofs not amd. I think it's a lot simpler than it sounds above. Granted if the tree is busy with mounts you can probably only give up but much of the time it's not. The logic to traverse the filesystem would be similar to what I have used in my rewrite of the expire module in autofs4. Check it out at ftp://ftp.kernel.org/pub/linux/daemons/autofs/v4 and let me know if you think the idea has merit. -- ,-._|\ Ian Kent / \ Perth, Western Australia *_.--._/ E-mail: raven@themaw.net v Web: http://themaw.net/ ^ permalink raw reply [flat|nested] 53+ messages in thread
* Re: VFS: Busy inodes after unmount on 2 way SMP 2003-09-17 0:28 ` Ian Kent 2003-09-17 2:05 ` Arun Sharma @ 2003-09-17 13:14 ` Ryan Go 1 sibling, 0 replies; 53+ messages in thread From: Ryan Go @ 2003-09-17 13:14 UTC (permalink / raw) To: Ian Kent; +Cc: Arun Sharma, autofs Hello Ian, We have also seen the self-destruct message on boxes with autofs4 in kernel 2.4.18 (w/ rmap12h patch) and autofs4-4.0.0pre10-10 userspace daemon. Thanks, << ryan On Wed, 17 Sep 2003, Ian Kent wrote: > On Thu, 11 Sep 2003, Arun Sharma wrote: > > > > > We've seen a few "VFS: Busy inodes after unmount. Self-destruct in 5 seconds. Have a nice day.." messages on a dual processor NFS client. Here's the use case: > > > > - Due to network load issues, the NFS server becomes unreachable for some time > > - The automounter tries to expire the mount > > - The unmount finds a couple of busy inodes. Putting some debug printks shows that typically two inodes are busy i.e. have inode->i_count == 1. But they don't have any waiters on inode->i_wait. Further, the inodes that are busy are > > /mnt/foo -> autofs mount point > > /mnt/foo/bar -> bar is a symbolic link > > > > It's not clear if this is a NFS issue or a autofs issue, but it's seen often with autofs. Are there any known race conditions that have been fixed after 2.4.20 ? The two calls I'm worried about are: > > > > fs/autofs/root.c:305: d_instantiate(dentry, iget(dir->i_sb,ent->ino)); > > fs/autofs/root.c:416: d_instantiate(dentry, iget(dir->i_sb,ino)); > > > > This is in the autofs (aka v3) kernel module not autofs4. > > Please specify which version of the userspace daemon and kernel module? > > -- > > ,-._|\ Ian Kent > / \ Perth, Western Australia > *_.--._/ E-mail: raven@themaw.net > v Web: http://themaw.net/ > > _______________________________________________ > autofs mailing list > autofs@linux.kernel.org > http://linux.kernel.org/mailman/listinfo/autofs > ^ permalink raw reply [flat|nested] 53+ messages in thread
end of thread, other threads:[~2003-10-01 12:56 UTC | newest] Thread overview: 53+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2003-09-12 0:11 VFS: Busy inodes after unmount on 2 way SMP Arun Sharma 2003-09-12 1:05 ` H. Peter Anvin 2003-09-12 14:23 ` Ryan Go 2003-09-12 17:57 ` Arun Sharma 2003-09-12 20:51 ` H. Peter Anvin 2003-09-12 23:33 ` Ion Badulescu 2003-09-13 0:08 ` Arun Sharma 2003-09-13 0:16 ` Ion Badulescu 2003-09-13 0:16 ` Ion Badulescu 2003-09-15 21:42 ` [NFS] " Arun Sharma 2003-09-15 22:25 ` H. Peter Anvin 2003-09-15 22:34 ` Arun Sharma 2003-09-15 23:42 ` Ion Badulescu 2003-09-16 1:34 ` [autofs] " Matt C 2003-09-16 1:34 ` Matt C 2003-09-19 2:57 ` Frank Cusack 2003-09-16 20:37 ` Re: [NFS] " Arun Sharma 2003-09-16 20:54 ` H. Peter Anvin 2003-09-17 0:28 ` Ian Kent 2003-09-17 2:05 ` Arun Sharma 2003-09-17 2:28 ` H. Peter Anvin 2003-09-17 4:56 ` Ian Kent 2003-09-17 4:52 ` Ian Kent 2003-09-17 17:47 ` Arun Sharma 2003-09-17 20:41 ` H. Peter Anvin 2003-09-17 21:00 ` Olaf Hering 2003-09-18 0:31 ` Ian Kent 2003-09-18 5:04 ` Olaf Hering 2003-09-18 5:52 ` Re: [autofs] " Trond Myklebust 2003-09-18 8:26 ` Olaf Kirch 2003-09-19 23:36 ` [NFS] " Olaf Hering 2003-09-25 23:17 ` Re: [autofs] " Matt C 2003-09-25 23:17 ` Matt C 2003-09-26 0:24 ` Trond Myklebust 2003-09-26 0:24 ` Trond Myklebust 2003-09-26 18:31 ` [NFS] " Ion Badulescu 2003-09-26 22:29 ` Re: [autofs] " Trond Myklebust 2003-09-27 16:55 ` Olaf Kirch 2003-09-28 23:16 ` Steve Fosdick 2003-09-29 12:07 ` Ion Badulescu 2003-09-29 17:22 ` Trond Myklebust 2003-09-30 12:50 ` Olaf Kirch 2003-09-30 13:31 ` Trond Myklebust 2003-09-29 3:27 ` Frank Cusack 2003-09-18 2:26 ` [NFS] " Ion Badulescu 2003-09-18 2:54 ` H. Peter Anvin 2003-09-29 12:19 ` Ion Badulescu 2003-09-30 12:24 ` Ian Kent 2003-09-30 12:51 ` Mike Waychison 2003-10-01 12:56 ` Ian Kent 2003-09-30 13:12 ` Ion Badulescu 2003-09-30 16:44 ` Ian Kent 2003-09-17 13:14 ` Ryan Go
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.