Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

[devzero]

>Indeed: This workaround seems to work!

Unfortunately not in every situation.

We have differnt XEN servers (Citrix XS7) at remote location which have hung/stale NFS mount problems at regular intervals (.ISO storage repo is mounted via WAN) and i always need to reboot, which really really(!) sucks.

At least a fake NFS server as described below releases the stuck mount, i.e. df -h and other processes touching do not hang anymore, so at least this workaround helps to some degree... 

BUT:

# umount /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e
umount.nfs: /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle

# mount|grep xen-sr-iso
172.16.28.10:/mnt/S2V2/xen-sr-iso on /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2147483647,sec=sys,mountaddr=172.16.28.10,mountvers=3,mountport=680,mountproto=tcp,local_lock=none,addr=172.16.28.10)

# umount -l -f /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e
umount.nfs: /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle

# mount|grep xen-sr-iso
172.16.28.10:/mnt/S2V2/xen-sr-iso on /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2147483647,sec=sys,mountaddr=172.16.28.10,mountvers=3,mountport=680,mountproto=tcp,local_lock=none,addr=172.16.28.10)

# ls -la
ls: cannot access 2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle
total 0
drwx------  3 root root   60 Feb 18 17:43 .
drwxr-xr-x 36 root root 1660 Jun  7 18:45 ..
d?????????  ? ?    ?       ?            ? 2b5c5c60-3744-c860-28a7-eb106d3a339e

Any hint on how to circumvent rebooting to remount the nfs share or proactively avoid stale NFS mounts would be very appreciated. (disabling NFS by module unload/load is no option, as our XEN servers do have other NFS mounts for shared storage)

regards
Roland

ps:
I`m not sure if linux-nfs ML will allow anonymous posts (probably not), so maybe someone subscribed be so kind to reply with list cc´ed. I`d like to avoid subscribing to a list because of a single post...




>List:       linux-nfs
>Subject:    Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?
>From:       Peter Funk <pf () artcom-gmbh ! de>
>Date:       2013-07-26 12:08:49
>Message-ID: 20130726120849.GA12584 () pfmaster
>[Download message RAW]

>Dick Streefland wrote 24.07.2013 13:03:
>> Peter Funk <pf@artcom-gmbh.de> wrote:
>> | We've researched this question for quite a while now and nobody here
>> | found a solution to the following problem:
>> | 
>> |  1: A Linux computer is NFS client of some other Linux NFS server
>> |     and has some active mounts and some processes working with files 
>> |     on that NFS server.  
>> | 
>> |  2: Now the NFS server becomes unavailable and a system administrator 
>> |     wants to clean up the situation on the NFS client computer without 
>> |     having to reboot this client computer.
>> | 
>> | Is this possible?  And if how exactly?
>> 
>> What you could try is temporarily add the IP number of the dead NFS
>> server to another NFS server. The other NFS server should reject any
>> request for the dead mount, and the client can continue with an error.
>
>Indeed: This workaround seems to work!
>
>Assume example: The NFS-server has IP 192.168.123.45 and the client
>has also the nfs-kernel-server package installed and it is running.
>Then this sequence on the client did the trick::
>
>    ifconfig eth0:fakesrv 192.168.123.45 up
>    umount -f -l ....
>    umount -f -l ....
>    ....
>    ifconfig eth0:fakesrv down
>
>Best Regards and many thanks for your suggestion,
>Peter Funk

Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

[J. Bruce Fields]

On Thu, Jun 08, 2017 at 10:49:50AM +0200, devzero@web.de wrote:
> Any hint on how to circumvent rebooting to remount the nfs share or proactively avoid stale NFS mounts would be very appreciated. (disabling NFS by module unload/load is no option, as our XEN servers do have other NFS mounts for shared storage)

Neil had some recent posts that might be relevant:

	http://marc.info/?l=util-linux-ng&m=149662998931157&w=2

Ripping out the storage from underneath your applications is a pretty
drastic step and may lose data, so NFS hasn't traditionally tried very
hard to make it easy.  But it may be possible at this point if you kill
-9 all the users and umount carefully.

> I`m not sure if linux-nfs ML will allow anonymous posts (probably
> not), so maybe someone subscribed be so kind to reply with list cc´ed.
> I`d like to avoid subscribing to a list because of a single post...

That's fine.

--b.

Aw: Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

[devzero]

> Ripping out the storage from underneath your applications is a pretty
> drastic step and may lose data,

We did not rip out our storage, it`s still there and reachable without problems. Our hung nfs is NEVER being caused by nfs server reboots or poweroff, it seems to happen on network disruption or whatever.

Maybe the problem is the WAN connection or some firewall session invalidation - or the nfs server itself (freenas, btw) not responding in a way the client expects.

For my curiousity, after switching off the fake ip and do a showmount -e to the original server where we had the hung nfs mount (to show you it`s still there and reachable), the mount suddenly unmounts cleanly, even without -f. 

And i can remount it without any problem.

It looks as if the interim fake nfs server (which did not export any share) made the client work correctly again...

regards
Roland

> Gesendet: Donnerstag, 08. Juni 2017 um 15:34 Uhr
> Von: "J. Bruce Fields" <bfields@fieldses.org>
> An: devzero@web.de
> Cc: linux-nfs@vger.kernel.org, pf@artcom-gmbh.de, rnews@altium.nl, jlayton@redhat.com
> Betreff: Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?
>
> On Thu, Jun 08, 2017 at 10:49:50AM +0200, devzero@web.de wrote:
> > Any hint on how to circumvent rebooting to remount the nfs share or proactively avoid stale NFS mounts would be very appreciated. (disabling NFS by module unload/load is no option, as our XEN servers do have other NFS mounts for shared storage)
> 
> Neil had some recent posts that might be relevant:
> 
> 	http://marc.info/?l=util-linux-ng&m=149662998931157&w=2
> 
> Ripping out the storage from underneath your applications is a pretty
> drastic step and may lose data, so NFS hasn't traditionally tried very
> hard to make it easy.  But it may be possible at this point if you kill
> -9 all the users and umount carefully.
> 
> > I`m not sure if linux-nfs ML will allow anonymous posts (probably
> > not), so maybe someone subscribed be so kind to reply with list cc´ed.
> > I`d like to avoid subscribing to a list because of a single post...
> 
> That's fine.
> 
> --b.
>

Re: Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

[J. Bruce Fields]

On Thu, Jun 08, 2017 at 04:06:32PM +0200, devzero@web.de wrote:
> > Ripping out the storage from underneath your applications is a pretty
> > drastic step and may lose data,
> 
> We did not rip out our storage, it`s still there and reachable without problems. Our hung nfs is NEVER being caused by nfs server reboots or poweroff, it seems to happen on network disruption or whatever.

OK, sounds like there's some other bug causing the original hang then, I
don't know what that is.

--b.

Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 4742 bytes --]

On Thu, Jun 08 2017, devzero@web.de wrote:

>>Indeed: This workaround seems to work!
>
> Unfortunately not in every situation.
>
> We have differnt XEN servers (Citrix XS7) at remote location which have hung/stale NFS mount problems at regular intervals (.ISO storage repo is mounted via WAN) and i always need to reboot, which really really(!) sucks.
>
> At least a fake NFS server as described below releases the stuck mount, i.e. df -h and other processes touching do not hang anymore, so at least this workaround helps to some degree... 
>
> BUT:
>
> # umount /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e
> umount.nfs: /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle
>
> # mount|grep xen-sr-iso
> 172.16.28.10:/mnt/S2V2/xen-sr-iso on /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2147483647,sec=sys,mountaddr=172.16.28.10,mountvers=3,mountport=680,mountproto=tcp,local_lock=none,addr=172.16.28.10)
>
> # umount -l -f /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e
> umount.nfs: /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle

This really shouldn't happen.
Since Linux 3.12 (commit 8033426e6bdb) it has been possible to
umount an NFS filesystem, no matter what state it is in.

Since util-linux 2.23 (commit 6d5d2b5fd342308), umount -l or umount -f
should have avoided statting the filesystem.

If you have a new util-linux (mount -V) and new kernel (uname -a),
then I'd be interested to see
  strace -o /tmp/strace -f umount -f /run/......



>
> # mount|grep xen-sr-iso
> 172.16.28.10:/mnt/S2V2/xen-sr-iso on /run/sr-mount/2b5c5c60-3744-c860-28a7-eb106d3a339e type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2147483647,sec=sys,mountaddr=172.16.28.10,mountvers=3,mountport=680,mountproto=tcp,local_lock=none,addr=172.16.28.10)
>
> # ls -la
> ls: cannot access 2b5c5c60-3744-c860-28a7-eb106d3a339e: Stale file handle
> total 0
> drwx------  3 root root   60 Feb 18 17:43 .
> drwxr-xr-x 36 root root 1660 Jun  7 18:45 ..
> d?????????  ? ?    ?       ?            ? 2b5c5c60-3744-c860-28a7-eb106d3a339e
>
> Any hint on how to circumvent rebooting to remount the nfs share or proactively avoid stale NFS mounts would be very appreciated. (disabling NFS by module unload/load is no option, as our XEN servers do have other NFS mounts for shared storage)
>
> regards
> Roland
>
> ps:
> I`m not sure if linux-nfs ML will allow anonymous posts (probably not), so maybe someone subscribed be so kind to reply with list cc´ed. I`d like to avoid subscribing to a list because of a single post...
>

We aren't that closed-minded ;-)  Your message went to the list.

NeilBrown

>
>
>
>>List:       linux-nfs
>>Subject:    Re: How to avoid rebooting Linux NFS-client when NFS-server is not available?
>>From:       Peter Funk <pf () artcom-gmbh ! de>
>>Date:       2013-07-26 12:08:49
>>Message-ID: 20130726120849.GA12584 () pfmaster
>>[Download message RAW]
>
>>Dick Streefland wrote 24.07.2013 13:03:
>>> Peter Funk <pf@artcom-gmbh.de> wrote:
>>> | We've researched this question for quite a while now and nobody here
>>> | found a solution to the following problem:
>>> | 
>>> |  1: A Linux computer is NFS client of some other Linux NFS server
>>> |     and has some active mounts and some processes working with files 
>>> |     on that NFS server.  
>>> | 
>>> |  2: Now the NFS server becomes unavailable and a system administrator 
>>> |     wants to clean up the situation on the NFS client computer without 
>>> |     having to reboot this client computer.
>>> | 
>>> | Is this possible?  And if how exactly?
>>> 
>>> What you could try is temporarily add the IP number of the dead NFS
>>> server to another NFS server. The other NFS server should reject any
>>> request for the dead mount, and the client can continue with an error.
>>
>>Indeed: This workaround seems to work!
>>
>>Assume example: The NFS-server has IP 192.168.123.45 and the client
>>has also the nfs-kernel-server package installed and it is running.
>>Then this sequence on the client did the trick::
>>
>>    ifconfig eth0:fakesrv 192.168.123.45 up
>>    umount -f -l ....
>>    umount -f -l ....
>>    ....
>>    ifconfig eth0:fakesrv down
>>
>>Best Regards and many thanks for your suggestion,
>>Peter Funk
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]