RE: [NFS] Re: broken umount -f

RE: [NFS] Re: broken umount -f

[Lever, Charles]

"umount -f" doesn't end pending RPCs.  if there are processes
with pending RPCs, then they are stuck and you will have to
reboot.  "intr" may allow some of these processes to be killed
before trying the "umount."

however, if there are no outstanding RPCs on the client, but
the server is not available, umount -f works as advertised.

> -----Original Message-----
> From: Peter =C5strand [mailto:peter@cendio.se]=20
> Sent: Monday, January 13, 2003 4:45 AM
> To: Trond Myklebust
> Cc: nfs@lists.sourceforge.net; linux-kernel@vger.kernel.org
> Subject: [NFS] Re: broken umount -f
>=20
>=20
> >>For as long as I remember, umount -f has been broken. I got=20
> a reminder=20
> >>of this fact today when we took an older NFS server out of=20
> use. I had=20
> >>to reboot almost all machines that had mounts from this server. Not=
=20
> >>nice.
>=20
> ...
>=20
> > AFAICS It works for me.
> >=20
> > Are you using the 'intr' mount option,
>=20
> Yes, as often I can. But IMHO, it should be possible to=20
> unmount an unreachable NFS fs even if it wasn't mounted with=20
> "intr". Otherwise we have a quite silly "sysadmin trap".
>=20
> >and are you remembering to kill
> > those processes that are actually using the mount point first?
>=20
> One some machines, I killed more or less everything. It=20
> didn't help. One some other machines, I couldn't kill so=20
> blindly. Remember, both "lsof" and "fuser" hangs.
>=20
> Also, as far as I understand, Solaris 8 does not require that=20
> you kill all processes before unmounting, if you use the "-f"=20
> flag (processes will get EIO). Would it be possible to=20
> implement this feature in Linux? That would be really nice.
>=20
> Regards, Peter
>=20
>=20
> >>For as long as I remember, umount -f has been broken. I got=20
> a reminder=20
> >>of this fact today when we took an older NFS server out of=20
> use. I had=20
> >>to reboot almost all machines that had mounts from this server. Not=
=20
> >>nice.
> >>
> >>Anyone knows why -f does not work? When I try, I get:
> >>
> >># umount -f /import/applix Cannot MOUNTPROG RPC: RPC: Port mapper=20
> >>failure - RPC: Unable to receive umount2: Device or resource busy
> >>umount: /import/applix: device is busy
> >>
> >>lsof and fuser hangs, as do "df" and "du". Really frustrating. It's=
=20
> >>not even possible to cleanly reboot the system, since=20
> RedHats shutdown=20
> >>scripts wants to unmount NFS fs's.
> >>
> >>I'm not exactly sure I understand what -f is supposed to do. Is it=20
> >>correct that it is supposed to unmount without contacting the NFS=20
> >>server? I assume that I still have to make sure no=20
> processes are using=20
> >>the FS? Would it be possible to add a "-9" flag (or something like=20
> >>that) that kills off all processes that uses the NFS fs=20
> automatically?
> >>
> >>(I'm using all kinds of RedHat Linux versions, from 5.0 up to 7.3.=20
> >>From what I can tell, this problems exists in all versions.)
> >>
>=20
>=20
>=20
>=20
>=20
> -------------------------------------------------------
> This SF.NET email is sponsored by: FREE  SSL Guide from=20
> Thawte are you planning your Web Server Security? Click here=20
> to get a FREE Thawte SSL guide and find the answers to all=20
> your  SSL security issues.=20
> http://ads.sourceforge.net/cgi-> bin/redirect.pl?thaw0026en
>=20
>=20
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net=20
> https://lists.sourceforge.net/lists/listinfo/n> fs
>=20

RE: [NFS] Re: broken umount -f

[Lever, Charles]

"umount -f" doesn't end pending RPCs.  if there are processes
with pending RPCs, then they are stuck and you will have to
reboot.  "intr" may allow some of these processes to be killed
before trying the "umount."

however, if there are no outstanding RPCs on the client, but
the server is not available, umount -f works as advertised.

> -----Original Message-----
> From: Peter Åstrand [mailto:peter@cendio.se] 
> Sent: Monday, January 13, 2003 4:45 AM
> To: Trond Myklebust
> Cc: nfs@lists.sourceforge.net; linux-kernel@vger.kernel.org
> Subject: [NFS] Re: broken umount -f
> 
> 
> >>For as long as I remember, umount -f has been broken. I got 
> a reminder 
> >>of this fact today when we took an older NFS server out of 
> use. I had 
> >>to reboot almost all machines that had mounts from this server. Not 
> >>nice.
> 
> ...
> 
> > AFAICS It works for me.
> > 
> > Are you using the 'intr' mount option,
> 
> Yes, as often I can. But IMHO, it should be possible to 
> unmount an unreachable NFS fs even if it wasn't mounted with 
> "intr". Otherwise we have a quite silly "sysadmin trap".
> 
> >and are you remembering to kill
> > those processes that are actually using the mount point first?
> 
> One some machines, I killed more or less everything. It 
> didn't help. One some other machines, I couldn't kill so 
> blindly. Remember, both "lsof" and "fuser" hangs.
> 
> Also, as far as I understand, Solaris 8 does not require that 
> you kill all processes before unmounting, if you use the "-f" 
> flag (processes will get EIO). Would it be possible to 
> implement this feature in Linux? That would be really nice.
> 
> Regards, Peter
> 
> 
> >>For as long as I remember, umount -f has been broken. I got 
> a reminder 
> >>of this fact today when we took an older NFS server out of 
> use. I had 
> >>to reboot almost all machines that had mounts from this server. Not 
> >>nice.
> >>
> >>Anyone knows why -f does not work? When I try, I get:
> >>
> >># umount -f /import/applix Cannot MOUNTPROG RPC: RPC: Port mapper 
> >>failure - RPC: Unable to receive umount2: Device or resource busy
> >>umount: /import/applix: device is busy
> >>
> >>lsof and fuser hangs, as do "df" and "du". Really frustrating. It's 
> >>not even possible to cleanly reboot the system, since 
> RedHats shutdown 
> >>scripts wants to unmount NFS fs's.
> >>
> >>I'm not exactly sure I understand what -f is supposed to do. Is it 
> >>correct that it is supposed to unmount without contacting the NFS 
> >>server? I assume that I still have to make sure no 
> processes are using 
> >>the FS? Would it be possible to add a "-9" flag (or something like 
> >>that) that kills off all processes that uses the NFS fs 
> automatically?
> >>
> >>(I'm using all kinds of RedHat Linux versions, from 5.0 up to 7.3. 
> >>From what I can tell, this problems exists in all versions.)
> >>
> 
> 
> 
> 
> 
> -------------------------------------------------------
> This SF.NET email is sponsored by: FREE  SSL Guide from 
> Thawte are you planning your Web Server Security? Click here 
> to get a FREE Thawte SSL guide and find the answers to all 
> your  SSL security issues. 
> http://ads.sourceforge.net/cgi-> bin/redirect.pl?thaw0026en
> 
> 
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net 
> https://lists.sourceforge.net/lists/listinfo/n> fs
> 

Re: Re: broken umount -f

[Scott Mcdermott]

Lever, Charles on Tue 14/01 07:56 -0800:
> however, if there are no outstanding RPCs on the client, but
> the server is not available, umount -f works as advertised.

I think his point was that Solaris will allow root to umount -f in any
case.

Right now I have to reboot my Linux NFS clients remotely with a "sync;
reboot -f" if they are stuck on a stale mount.  `init 6' will time out
forever trying to unmount NFS filesystems (and yes, the init scripts use
`-f'), and by that time my sshd is killed so I have to drive out to the
colo site.


-------------------------------------------------------
This SF.NET email is sponsored by: FREE  SSL Guide from Thawte
are you planning your Web Server Security? Click here to get a FREE
Thawte SSL guide and find the answers to all your  SSL security issues.
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0026en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Trond Myklebust]

>>>>> " " == Scott Mcdermott <smcdermott@questra.com> writes:

     > Lever, Charles on Tue 14/01 07:56 -0800:
    >> however, if there are no outstanding RPCs on the client, but
    >> the server is not available, umount -f works as advertised.

     > I think his point was that Solaris will allow root to umount -f
     > in any case.

     > Right now I have to reboot my Linux NFS clients remotely with a
     > "sync; reboot -f" if they are stuck on a stale mount.  `init 6'
     > will time out forever trying to unmount NFS filesystems (and
     > yes, the init scripts use `-f'), and by that time my sshd is
     > killed so I have to drive out to the colo site.

Linux will not allow you to unmount without killing those processes,
and I'd be opposed to any patch that tries to kill active processes
from within the filesystem.
This is something that needs to be resolved in userland.

Cheers,
  Trond


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Trond Myklebust on Tue 14/01 20:06 +0100:
> Linux will not allow you to unmount without killing those processes,
> and I'd be opposed to any patch that tries to kill active processes
> from within the filesystem.  This is something that needs to be
> resolved in userland.

Last I checked, the programs wouldn't die even with -KILL when they were
stuck in device-wait state.  The only way to reboot a machine with such
processes is to reboot -f, which is wrong.  The filesystems should be
able to have forced umount at sysadmin's discretion.


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Brian Tinsley]

Scott Mcdermott wrote:

>Trond Myklebust on Tue 14/01 20:06 +0100:
>  
>
>>Linux will not allow you to unmount without killing those processes,
>>and I'd be opposed to any patch that tries to kill active processes
>>from within the filesystem.  This is something that needs to be
>>resolved in userland.
>>    
>>
>
>Last I checked, the programs wouldn't die even with -KILL when they were
>stuck in device-wait state.  The only way to reboot a machine with such
>processes is to reboot -f, which is wrong.  The filesystems should be
>able to have forced umount at sysadmin's discretion.
>  
>
I've had luck with:

kill -9 <pids>
umount -f <nfs_dirs>
kill -9 <pids>
umount -f <nfs_dirs>

The last umount always works for me.


-- 

Brian Tinsley
Chief Systems Engineer
Emageon






-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Trond Myklebust]

>>>>> " " == Scott Mcdermott <smcdermott@questra.com> writes:

     > Last I checked, the programs wouldn't die even with -KILL when
     > they were stuck in device-wait state.

They will if you mount with 'intr', and make sure that you kill *all*
programs that are using that mountpoint.

If somebody wants to work on 'umount -f' then by all means do...

Cheers,
  Trond


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Trond Myklebust on Tue 14/01 20:35 +0100:
>      > Last I checked, the programs wouldn't die even with -KILL when
>      > they were stuck in device-wait state.
> 
> They will if you mount with 'intr', and make sure that you kill *all*
> programs that are using that mountpoint.

I don't want users to be able to corrupt their own data by using `intr'
when I am just bouncing the server for some reason.  But, I want root to
be able to force it if necessary, as in the case of a client reboot when
the server is gone (eg when switching servers and they need new handles
for their automounts, but still have the old ones and the server is
never coming back up).  Usually this means the admin screwed up, but
unfortunately that does happen.

"umount -f" seems like the right place for this.


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Steven N. Hirsch]

On Tue, 14 Jan 2003, Scott Mcdermott wrote:

> Trond Myklebust on Tue 14/01 20:35 +0100:
> >      > Last I checked, the programs wouldn't die even with -KILL when
> >      > they were stuck in device-wait state.
> > 
> > They will if you mount with 'intr', and make sure that you kill *all*
> > programs that are using that mountpoint.
> 
> I don't want users to be able to corrupt their own data by using `intr'
> when I am just bouncing the server for some reason.  But, I want root to
> be able to force it if necessary, as in the case of a client reboot when
> the server is gone (eg when switching servers and they need new handles
> for their automounts, but still have the old ones and the server is
> never coming back up).  Usually this means the admin screwed up, but
> unfortunately that does happen.
> 
> "umount -f" seems like the right place for this.

You have my vote.  For sanity's sake, there needs to be a big, red lever 
under the locked access hatch - labled "I said umount -f, dammit!".




-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Steven N. Hirsch]

On Tue, 14 Jan 2003, Trond Myklebust wrote:

> >>>>> " " == Scott Mcdermott <smcdermott@questra.com> writes:
> 
>      > Last I checked, the programs wouldn't die even with -KILL when
>      > they were stuck in device-wait state.
> 
> They will if you mount with 'intr', and make sure that you kill *all*
> programs that are using that mountpoint.

I can honestly say that I end up rebooting more than 80% of the time when 
the server goes away.  Believe me, I've tried everything including taking 
the system down to single-user mode and killing everything not bolted to 
the kernel <g>.  

Steve




-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Benjamin LaHaise]

On Tue, Jan 14, 2003 at 08:06:09PM +0100, Trond Myklebust wrote:
> Linux will not allow you to unmount without killing those processes,
> and I'd be opposed to any patch that tries to kill active processes
> from within the filesystem.
> This is something that needs to be resolved in userland.

It's up to NFS to make the outstanding IOs against the filesystem 
return -EIO.

		-ben


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Trond Myklebust]

>>>>> " " == Benjamin LaHaise <bcrl@redhat.com> writes:

     > It's up to NFS to make the outstanding IOs against the
     > filesystem return -EIO.

It does that.

void
nfs_umount_begin(struct super_block *sb)
{
        struct nfs_server *server = NFS_SB(sb);
        struct rpc_clnt *rpc;

        /* -EIO all pending I/O */
        if ((rpc = server->client) != NULL)
                rpc_killall_tasks(rpc);
}

What it does not do is hunt down every task that holds an open file,
or has  been started from an NFS directory.

Cheers,
  Trond


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Benjamin LaHaise]

On Tue, Jan 14, 2003 at 08:52:58PM +0100, Trond Myklebust wrote:
> What it does not do is hunt down every task that holds an open file,
> or has  been started from an NFS directory.

Leaving them with stale directories that EIO on everything should work 
fine.  The unmount is actually a distinct operation from the freeing 
of all the data structures, which thanks to Al Viro's changes will 
now be properly garbage collected as the processes die off.

		-ben
-- 
"Do you seek knowledge in time travel?"


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Cole, Timothy D.]

> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> Sent: Tuesday, January 14, 2003 14:06
> To: Scott Mcdermott
> Cc: nfs@lists.sourceforge.net
> Subject: Re: [NFS] Re: broken umount -f

> Linux will not allow you to unmount without killing those processes,
> and I'd be opposed to any patch that tries to kill active processes
> from within the filesystem.

> This is something that needs to be resolved in userland.

The few times I've needed to use umount -f, the processes in question
weren't killable from userland.  Is there an architectural reason for this?

(i.e. instead of killing them, why can't their pending system calls return
with -EIO if a umount is forced, as I've seen some other unices do in
similar situations [pending RPCs + a dead server pinning a mount]?)


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Lever, Charles]

> -----Original Message-----
> From: Scott Mcdermott [mailto:smcdermott@questra.com]
> Sent: Tuesday, January 14, 2003 2:20 PM
> To: Trond Myklebust
> Cc: nfs@lists.sourceforge.net
> Subject: Re: [NFS] Re: broken umount -f
> 
> Last I checked, the programs wouldn't die even with -KILL 
> when they were
> stuck in device-wait state.  The only way to reboot a machine 
> with such
> processes is to reboot -f, which is wrong.  The filesystems should be
> able to have forced umount at sysadmin's discretion.

do you remember which kernel this was?

trond fixed a long-standing "processes stuck in 'D' state" bug in
2.4.20.  this bug may be the reason these processes didn't die
when you killed them.


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Lever, Charles on Tue 14/01 11:36 -0800:
> > Last I checked, the programs wouldn't die even with -KILL when they
> > were stuck in device-wait state.  The only way to reboot a machine
> > with such processes is to reboot -f, which is wrong.  The
> > filesystems should be able to have forced umount at sysadmin's
> > discretion.
> 
> do you remember which kernel this was?
> 
> trond fixed a long-standing "processes stuck in 'D' state" bug in
> 2.4.20.  this bug may be the reason these processes didn't die when
> you killed them.

<on client>

# uname -r
2.4.21-pre3-NFS_ALL

# showmount --exports nfsserver
Export list for nfsserver
/tmp           10.0.0.5

# mount nfsserver:/tmp /mnt/tmp

# cd /mnt/tmp

# ssh nfsserver /etc/init.d/nfs stop
Shutting down NFS mountd: [  OK  ]
Shutting down NFS daemon: [  OK  ]
Shutting down NFS services:  [  OK  ]
Shutting down NFS quotas: [  OK  ]

# /bin/pwd
nfs: server nfsserver not responding, still trying

<hangs>

<now on other tty with pwd=/>

# ps -eo state,wchan,pid,command | grep ^D
D end     1472 /bin/pwd

# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472
# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472

# ps -eo state,wchan,pid,command | grep ^D
D end     1472 /bin/pwd


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

To nfs@lists.sourceforge.net on Wed 15/01 00:19 -0500:
> Lever, Charles on Tue 14/01 11:36 -0800:
> <on client>

forgot to include

# grep nfsserver /proc/mounts
nfsserver:/tmp /mnt/tmp nfs rw,v3,rsize=8192,wsize=8192,hard,udp,lock,addr=nfsserver 0 0


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Cole, Timothy D.]

> -----Original Message-----
> From: Trond Myklebust [mailto:trond.myklebust@fys.uio.no]
> Sent: Tuesday, January 14, 2003 14:36
> To: Scott Mcdermott
> Cc: nfs@lists.sourceforge.net
> Subject: Re: [NFS] Re: broken umount -f
> 
> They will if you mount with 'intr', and make sure that you kill *all*
> programs that are using that mountpoint.

That doesn't always appear to be the case in practice (maybe a long-standing
bug/strange interaction?).  Can there be users not reported by fuser?



-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Lever, Charles]

> To nfs@lists.sourceforge.net on Wed 15/01 00:19 -0500:
> > Lever, Charles on Tue 14/01 11:36 -0800:
> > <on client>
> 
> forgot to include
> 
> # grep nfsserver /proc/mounts
> nfsserver:/tmp /mnt/tmp nfs 
> rw,v3,rsize=8192,wsize=8192,hard,udp,lock,addr=nfsserver 0 0

thanks!

what if you try it again with the intr mount option?

if that doesn't help, enable rpc level debugging and send me
the kernel log contents.

echo  3 > /proc/sys/sunrpc/rpc_debug   #  to enable debugging
echo  0 > /proc/sys/sunrpc/rpc_debug   #  to turn it off again


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Lever, Charles on Wed 15/01 06:45 -0800:
> what if you try it again with the intr mount option?

I'm sure it *will* work with the `intr' mount option.  But I don't want
my users to be able to corrupt their own data just because I decided to
bounce to server for whatever reason.  Their IO to that filesystem
should hang, uninterruptibly, as is the conventional wisdom (that hard,
nointr is the Right Way), and I agree with.

"pick one or the other" doesn't work because there are situtations where
the filesystem IO will never complete and a umount *has* to be forced or
there is never any recovery option.

> if that doesn't help, enable rpc level debugging and send me the
> kernel log contents.

if you'd like I can still do this, but it probably works fine with intr,
that's not what the problem is.  The problem is having a system that one
cannot even reboot without using "reboot -f" just because the server is
down and the client mounts with hard,intr.


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Murata, Dennis W (SAIC)]

If you cd out of /mnt/tmp can you then umount -f?  This looks similar to
what happens on Solaris 8 if the directory happens to be the home directory
of a user logged into a system.  The login session has to be killed before
the umount -f works.  I would not think in your case that the user would
have to be forced off.

Wayne

-----Original Message-----
From: Scott Mcdermott [mailto:smcdermott@questra.com]
Sent: Tuesday, January 14, 2003 11:19 PM
To: nfs@lists.sourceforge.net
Subject: Re: [NFS] Re: broken umount -f


Lever, Charles on Tue 14/01 11:36 -0800:
> > Last I checked, the programs wouldn't die even with -KILL when they
> > were stuck in device-wait state.  The only way to reboot a machine
> > with such processes is to reboot -f, which is wrong.  The
> > filesystems should be able to have forced umount at sysadmin's
> > discretion.
> 
> do you remember which kernel this was?
> 
> trond fixed a long-standing "processes stuck in 'D' state" bug in
> 2.4.20.  this bug may be the reason these processes didn't die when
> you killed them.

<on client>

# uname -r
2.4.21-pre3-NFS_ALL

# showmount --exports nfsserver
Export list for nfsserver
/tmp           10.0.0.5

# mount nfsserver:/tmp /mnt/tmp

# cd /mnt/tmp

# ssh nfsserver /etc/init.d/nfs stop
Shutting down NFS mountd: [  OK  ]
Shutting down NFS daemon: [  OK  ]
Shutting down NFS services:  [  OK  ]
Shutting down NFS quotas: [  OK  ]

# /bin/pwd
nfs: server nfsserver not responding, still trying

<hangs>

<now on other tty with pwd=/>

# ps -eo state,wchan,pid,command | grep ^D
D end     1472 /bin/pwd

# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472
# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472
# kill -KILL 1472

# umount -f /mnt/tmp
Cannot MOUNTPROG RPC: RPC: Program not registered
umount2: Device or resource busy
umount: /mnt/tmp: device is busy

# kill -KILL 1472

# ps -eo state,wchan,pid,command | grep ^D
D end     1472 /bin/pwd


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Lever, Charles]

> I'm sure it *will* work with the `intr' mount option.  But I 
> don't want
> my users to be able to corrupt their own data just because I 
> decided to
> bounce to server for whatever reason.  Their IO to that filesystem
> should hang, uninterruptibly, as is the conventional wisdom 
> (that hard,
> nointr is the Right Way), and I agree with.

do you know what the risk of data corruption is when using "intr"?
seems pretty low to me.


-------------------------------------------------------
This SF.NET email is sponsored by: Take your first step towards giving 
your online business a competitive advantage. Test-drive a Thawte SSL 
certificate - our easy online guide will show you how. Click here to get 
started: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0027en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Lever, Charles on Wed 15/01 09:04 -0800:
> > I'm sure it *will* work with the `intr' mount option.  But I don't
> > want my users to be able to corrupt their own data just because I
> > decided to bounce to server for whatever reason.
> 
> do you know what the risk of data corruption is when using "intr"?
> seems pretty low to me.

User saving his mail spool, sees "nfs server not responding, still
trying" and decides to try killing his MUA.  Too bad it works and now
his spool is a steaming pile of ASCII.

any number of other possible problems with intr

`soft' and `intr' are evil and should be banned.


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Bill Rugolsky Jr.]

[Sorry, for the resend; I broke my return address on the first try.]

On Wed, Jan 15, 2003 at 09:04:58AM -0800, Lever, Charles wrote:
> do you know what the risk of data corruption is when using "intr"?
> seems pretty low to me.

Any reason why we can't add a mount option that makes cl_intr only
effective for SIGKILL in sunrpc/clnt.c?  (A general sigmask= option
doesn't seem immediately useful, unless there is some use for [T]STOP.)
  
Regards,
 
    Bill Rugolsky


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Scott Mcdermott]

Bill Rugolsky Jr. on Wed 15/01 13:08 -0500:
> > do you know what the risk of data corruption is when using "intr"?
> > seems pretty low to me.
> 
> Any reason why we can't add a mount option that makes cl_intr only
> effective for SIGKILL in sunrpc/clnt.c?

and how about also only if euid == 0


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: broken umount -f

[Ion Badulescu]

On Wed, 15 Jan 2003 09:04:58 -0800, Lever, Charles <Charles.Lever@netapp.com> wrote:
> 
> do you know what the risk of data corruption is when using "intr"?
> seems pretty low to me.

What is the risk, anyway? That the user will press Ctrl-C (SIGINT) and 
kill the process? How is that different from doing the same thing when
NFS is not hung? Sure, you can envision a case where the process traps
SIGINT so it is not fatal, yet the NFS request ends up being canceled, 
but it's hardly the end of the world.

I also think we have a misunderstanding here. SIGKILL should _always_ be able
to kill a process hanging on NFS. Unconditionally. It may not do it right
away, it may take a few seconds until the client exists the noninterruptible
sequence, but it should succeed eventually. The role of 'umount -f' then
becomes mostly to speed up the effects of the SIGKILL.

SIGINT should be able to do the same thing if the mount is done with 'intr'.
Nothing more and nothing less.

>From the Solaris mount_nfs(1M) man page:

           intr | nointr
                 Allow (do not allow) keyboard interrupts to kill
                 a process that  is  hung  while  waiting  for  a
                 response  on  a  hard-mounted  file system.  The
                 default is intr, which  makes  it  possible  for
                 clients  to  interrupt  applications that may be
                 waiting for a remote mount.

Linux does the above, mostly. The biggest problem is that sometimes
the hanging NFS access will be done by rpciod, not by the process
itself, and so it's rpciod that needs the SIGKILL (or SIGINT) in
order to abort the access. Unfortunately, rpciod is owned by root,
so a regular user can't send it any signals. For the sysadmin,
killall -KILL rpciod combined with umount -f does the trick most of
the time.

The other problem I've seen occasionally is that umount -f hangs
(interruptibly) instead of aborting all outstanding I/O's. I haven't
been able to find a pattern as to when it happens, yet.

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.


-------------------------------------------------------
This SF.NET email is sponsored by: Thawte.com
Understand how to protect your customers personal information by implementing
SSL on your Apache Web Server. Click here to get our FREE Thawte Apache 
Guide: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0029en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: broken umount -f

[Cole, Timothy D.]

> -----Original Message-----
> From: Scott Mcdermott [mailto:smcdermott@questra.com]
> Sent: Wednesday, January 15, 2003 12:24
> To: nfs@lists.sourceforge.net
> Subject: Re: [NFS] Re: broken umount -f
> 
> User saving his mail spool, sees "nfs server not responding, still
> trying" and decides to try killing his MUA.  Too bad it works and now
> his spool is a steaming pile of ASCII.

That's possible with non-NFS filesystems too -- just normally a smaller
window of opportunity.  It doesn't require a filesystem hang in any case --
most mailbox operations are not a single atomic write().  Imagine someone
killing the MUA in the middle of deleting a large mail from a ~40MB mail
spool on any filesystem, local or remote.

Also consider the nointr case -- process hangs, user can't kill it, user
naively closes the terminal window.  Server comes back up.  SIGHUP is
finally handled when the write() returns.  Process dies.  ASCII soup again.

This is of course assuming that the NFS server _can_ come back up.  If not,
totally unkillable processes are a pain-in-the-ssh.

> `soft' and `intr' are evil and should be banned.

Agreed WRT soft's evil-ness, anyway.  But hard,intr seems to be a pretty
good combination, as far as safety from data corruption, and from a
standpoint of not having to reboot-and-kill-week-long-simulations just
because a few unrelated (but important) processes got wedged by a
recalcitrant NFS server.


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Broken umount -f

[Heflin, Roger A.]

> Message: 1
> From: "Cole, Timothy D." <tdcole@northropgrumman.com>
> To: 'Scott Mcdermott' <smcdermott@questra.com>, =
nfs@lists.sourceforge.net
> Subject: RE: [NFS] Re: broken umount -f
> Date: Wed, 15 Jan 2003 10:46:42 -0800
>=20
> > -----Original Message-----
> > From: Scott Mcdermott [mailto:smcdermott@questra.com]
> > Sent: Wednesday, January 15, 2003 12:24
> > To: nfs@lists.sourceforge.net
> > Subject: Re: [NFS] Re: broken umount -f
> >=20
> > User saving his mail spool, sees "nfs server not responding, still
> > trying" and decides to try killing his MUA.  Too bad it works and =
now
> > his spool is a steaming pile of ASCII.
>=20
> That's possible with non-NFS filesystems too -- just normally a =
smaller
> window of opportunity.  It doesn't require a filesystem hang in any =
case --
> most mailbox operations are not a single atomic write().  Imagine =
someone
> killing the MUA in the middle of deleting a large mail from a ~40MB =
mail
> spool on any filesystem, local or remote.
>=20
> Also consider the nointr case -- process hangs, user can't kill it, =
user
> naively closes the terminal window.  Server comes back up.  SIGHUP is
> finally handled when the write() returns.  Process dies.  ASCII soup =
again.
>=20
> This is of course assuming that the NFS server _can_ come back up.  If =
not,
> totally unkillable processes are a pain-in-the-ssh.
>=20
> > `soft' and `intr' are evil and should be banned.
>=20
> Agreed WRT soft's evil-ness, anyway.  But hard,intr seems to be a =
pretty
> good combination, as far as safety from data corruption, and from a
> standpoint of not having to reboot-and-kill-week-long-simulations just
> because a few unrelated (but important) processes got wedged by a
> recalcitrant NFS server.
>=20
	Thinking about it, having non-intr won't save you in any way shape for
	form.

	If the nfs server pauses for some reason and you have non-intr and the=20
	user hits ctrl-c or something similir, it will just wait for the server =
to come
	back and then take the ctrl-c and abruptly kill the program, if the=20
	operation(s) takes any amount of time you will still have corrupted =
files.

					Roger


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: Broken umount -f

[Scott Mcdermott]

Heflin, Roger A. on Wed 15/01 13:59 -0600:
> Thinking about it, having non-intr won't save you in any way shape for
> form.

although I tried, I can't think of a case where this is wrong.  I'll
just remount my filesystems with `intr'.  But, I still think that with
`intr', umount -f should work.

perhaps `nointr' should be the default mount option.

btw the NFS HOWTO recommends hard,nointr


-------------------------------------------------------
This SF.NET email is sponsored by: A Thawte Code Signing Certificate 
is essential in establishing user confidence by providing assurance of 
authenticity and code integrity. Download our Free Code Signing guide:
http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0028en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: Broken umount -f

[Cole, Timothy D.]

> -----Original Message-----
> From: Scott Mcdermott [mailto:smcdermott@questra.com]
> Sent: Wednesday, January 15, 2003 22:38
> To: nfs@lists.sourceforge.net
> Subject: Re: [NFS] Re: Broken umount -f

> But, I still think that with `intr', umount -f should work.

Agreed on that count -- I didn't really mean to suggest otherwise.
Unmounting is orthagonal to signal delivery, so really intr/nointr shouldn't
influence umount -f's behavior.

> btw the NFS HOWTO recommends hard,nointr

I think there was a typo in an some earlier versions -- the current version
of the NFS HOWTO (http://nfs.sourceforge.net/nfs-howto/client.html)
recommends hard,intr.


-------------------------------------------------------
This SF.NET email is sponsored by: Thawte.com
Understand how to protect your customers personal information by implementing
SSL on your Apache Web Server. Click here to get our FREE Thawte Apache 
Guide: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0029en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: Broken umount -f

[Scott Mcdermott]

Cole, Timothy D. on Thu 16/01 11:45 -0800:
> > But, I still think that with `intr', umount -f should work.
> 
> Agreed on that count -- I didn't really mean to suggest otherwise.

I meant, that with `nointr' it should work.  I made a couple of typos in
that email :)

> Unmounting is orthagonal to signal delivery, so really intr/nointr
> shouldn't influence umount -f's behavior.

as it is now, with `nointr' the signals aren't delivered when there is
pending IO, apparently (don't know if that's what really happens in the
kernel)

> > btw the NFS HOWTO recommends hard,nointr
> 
> I think there was a typo in an some earlier versions -- the current
> version of the NFS HOWTO
> (http://nfs.sourceforge.net/nfs-howto/client.html) recommends
> hard,intr.

I think I read it wrong.  I seem to have dyslexia or something...

anyways thanks to all for this useful discussion, I have remounted my
systems with intr and that solution is good enough for now.


-------------------------------------------------------
This SF.NET email is sponsored by: Thawte.com
Understand how to protect your customers personal information by implementing
SSL on your Apache Web Server. Click here to get our FREE Thawte Apache 
Guide: http://ads.sourceforge.net/cgi-bin/redirect.pl?thaw0029en
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs