Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Benny Halevy]

On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com> wrote:
> Anyone ?
> 
> 2009/7/29 Carlos André <candrecn@gmail.com>:
>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with Kerberos
>> and AutoFS, but i got a problem: If NFS server goes down i get a LOOOOOOONG
>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>
>> Since i need mount some (3 to 6) dirs at user logon process, if mount hangs,
>> user logon hangs. Then i want configure it to timeout (if server down) after
>> 10-15 secs (MAX) on each mount attempt.
>>
>> I already make a lab and tried a LOT of combinations, there my findings
>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using basic command
>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>
>> - Once i try access mount point using AutoFS (proto=tcp OR proto=udp) it
>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error (mount: mount to
>> NFS server '172.16.0.10' failed: timed out (giving up))

Sounds like you're hitting the server's grace period.
You can try hacking it in /etc/init.d/nfs
by adding
	echo 15 > /proc/fs/nfsd/nfsv4leasetime

before
        daemon rpc.nfsd $RPCNFSDARGS $RPCNFSDCOUNT

Then, look at your /var/log/messages file for
NFSD: starting 15-second grace period

That said, I'm not sure why it can't be passed as a command line
option to the nfsd daemon and controlled via RPCNFSDARGS
in /etc/sysconfig/nfs

Benny

>>
>> Mounting manually using NFSv4 i got same timeouts of AutoFS.
>>
>> The only way to get a lower timeout value is using only proto=udp,retry=0
>> (not using sec=krb5) any another combination i get 3m9s (sec=krb5,proto=tcp
>>
>>
>> I tried change another NFS mount options putting a lower value (timeo,
>> retrans, retry), and they only change something then i use NFSv3 with
>> proto=udp. But i want NFSv4/TCP (coz Kerberos) and a timeout lower then 15
>> secs... :(
>>
>> I'm using these packages (server and client side):
>> autofs-5.0.1-0.rc2.102.el5_3.1
>> nfs-utils-1.0.9-40.el5
>> kernel-2.6.18-128.1.16.el5
>>
>> The only way to resolve this behavior is changing the source code? There's
>> no way to lower timeout with NFSv4/TCP in this case ?
>>
>> Thanks.
>>
>>
> _______________________________________________
> NFSv4 mailing list
> NFSv4@linux-nfs.org
> http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[J. Bruce Fields]

T24gRnJpLCBBdWcgMDcsIDIwMDkgYXQgMDk6NDI6MThBTSArMDMwMCwgQmVubnkgSGFsZXZ5IHdy
b3RlOgo+IE9uIEF1Zy4gMDcsIDIwMDksIDM6MTggKzAzMDAsIENhcmxvcyBBbmRyw6kgPGNhbmRy
ZWNuQGdtYWlsLmNvbT4gd3JvdGU6Cj4gPiBBbnlvbmUgPwo+ID4gCj4gPiAyMDA5LzcvMjkgQ2Fy
bG9zIEFuZHLDqSA8Y2FuZHJlY25AZ21haWwuY29tPjoKPiA+PiBQUEwsIEkgbmVlZCBwdXQgYSBD
ZW50T1MgNS4zICh1cGRhdGVkKSBORlN2NCBzZXJ2ZXIgdG8gd29yayB3aXRoIEtlcmJlcm9zCj4g
Pj4gYW5kIEF1dG9GUywgYnV0IGkgZ290IGEgcHJvYmxlbTogSWYgTkZTIHNlcnZlciBnb2VzIGRv
d24gaSBnZXQgYSBMT09PT09PT05HCj4gPj4gbW91bnQgdGltZW91dCBvbiBDZW50T1MgNS4zICh1
cGRhdGVkKSBORlN2NCBjbGllbnQuLi4KPiA+Pgo+ID4+IFNpbmNlIGkgbmVlZCBtb3VudCBzb21l
ICgzIHRvIDYpIGRpcnMgYXQgdXNlciBsb2dvbiBwcm9jZXNzLCBpZiBtb3VudCBoYW5ncywKPiA+
PiB1c2VyIGxvZ29uIGhhbmdzLiBUaGVuIGkgd2FudCBjb25maWd1cmUgaXQgdG8gdGltZW91dCAo
aWYgc2VydmVyIGRvd24pIGFmdGVyCj4gPj4gMTAtMTUgc2VjcyAoTUFYKSBvbiBlYWNoIG1vdW50
IGF0dGVtcHQuCj4gPj4KPiA+PiBJIGFscmVhZHkgbWFrZSBhIGxhYiBhbmQgdHJpZWQgYSBMT1Qg
b2YgY29tYmluYXRpb25zLCB0aGVyZSBteSBmaW5kaW5ncwo+ID4+IChzZXJ2ZXIgRE9XTiBJUDog
MTcyLjE2LjAuMTAgLyBjbGllbnQgSVA6IDE3Mi4xNi4xLjEwKSB1c2luZyBiYXNpYyBjb21tYW5k
Cj4gPj4gKHRpbWUgbW91bnQgMTcyLjE2LjAuMTA6L3JlbW90ZWRpciAvbG9jYWxkaXIvIC10IG5m
czQgLW8KPiA+PiBzZWM9a3JiNSxwcm90bz08dGNwL3VkcD4pIGZyb20gTkZTIGNsaWVudDoKPiA+
Pgo+ID4+IC0gT25jZSBpIHRyeSBhY2Nlc3MgbW91bnQgcG9pbnQgdXNpbmcgQXV0b0ZTIChwcm90
bz10Y3AgT1IgcHJvdG89dWRwKSBpdAo+ID4+IGhhbmdzIGZvciAxODkgc2VjcyAoM205czogcmVh
bCAgM205LjAwMXMpICB1bnRpbCBzaG93IGVycm9yIChtb3VudDogbW91bnQgdG8KPiA+PiBORlMg
c2VydmVyICcxNzIuMTYuMC4xMCcgZmFpbGVkOiB0aW1lZCBvdXQgKGdpdmluZyB1cCkpCj4gCj4g
U291bmRzIGxpa2UgeW91J3JlIGhpdHRpbmcgdGhlIHNlcnZlcidzIGdyYWNlIHBlcmlvZC4KCkkg
dGhvdWdodCBoZSB3YXMgZGVzY3JpYmluZyBhIHNpdHVhdGlvbiB3aGVyZSB0aGUgc2VydmVyIHRo
ZSBzZXJ2ZXIKaXMgY29tcGxldGVseSBnb25lIGFuZCBpc24ndCBjb21pbmcgYmFjaywgYW5kIHdv
bmRlcmluZyBob3cgdG8gbWFrZSB0aGUKbW91bnQgZmFpbCBmYXN0ZXIuICBCdXQgSSBtYXkgYmUg
bWlzdW5kZXJzdGFuZGluZy4KCi0tYi4KX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX18KTkZTdjQgbWFpbGluZyBsaXN0Ck5GU3Y0QGxpbnV4LW5mcy5vcmcKaHR0
cDovL2xpbnV4LW5mcy5vcmcvY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL25mc3Y0

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Bruce, no... you're right.  I'm describing a situation where my server
died... i need mount fail faster (10 or 15 secs max) than 3 minutes
and 9 seconds...

2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com> wr=
ote:
>> > Anyone ?
>> >
>> > 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>> >> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with =
Kerberos
>> >> and AutoFS, but i got a problem: If NFS server goes down i get a =
LOOOOOOONG
>> >> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>> >>
>> >> Since i need mount some (3 to 6) dirs at user logon process, if m=
ount hangs,
>> >> user logon hangs. Then i want configure it to timeout (if server =
down) after
>> >> 10-15 secs (MAX) on each mount attempt.
>> >>
>> >> I already make a lab and tried a LOT of combinations, there my fi=
ndings
>> >> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using basi=
c command
>> >> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>> >> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>> >>
>> >> - Once i try access mount point using AutoFS (proto=3Dtcp OR prot=
o=3Dudp) it
>> >> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show error (=
mount: mount to
>> >> NFS server '172.16.0.10' failed: timed out (giving up))
>>
>> Sounds like you're hitting the server's grace period.
>
> I thought he was describing a situation where the server the server
> is completely gone and isn't coming back, and wondering how to make t=
he
> mount fail faster. =A0But I may be misunderstanding.
>
> --b.
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
> Bruce, no... you're right.  I'm describing a situation where my server
> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
> and 9 seconds...

The 189 second timeout is likely how long it takes the kernel to give  
up trying to connect a TCP socket to the server (6 SYN attempts with  
exponential retries, or something like that).  For stock CentOS 5.3, I  
think user space does only a DNS lookup for normal NFSv4 mounts -- the  
kernel just tries to connect a TCP socket to port 2049, with no  
preceding rpcbind request.

Carlos, let us know if you have replaced any NFS-related CentOS  
components (kernel, nfs-utils) with something you've built yourself.

> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com>  
>>> wrote:
>>>> Anyone ?
>>>>
>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with  
>>>>> Kerberos
>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a  
>>>>> LOOOOOOONG
>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>
>>>>> Since i need mount some (3 to 6) dirs at user logon process, if  
>>>>> mount hangs,
>>>>> user logon hangs. Then i want configure it to timeout (if server  
>>>>> down) after
>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>
>>>>> I already make a lab and tried a LOT of combinations, there my  
>>>>> findings
>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using  
>>>>> basic command
>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>
>>>>> - Once i try access mount point using AutoFS (proto=tcp OR  
>>>>> proto=udp) it
>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error  
>>>>> (mount: mount to
>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>
>>> Sounds like you're hitting the server's grace period.
>>
>> I thought he was describing a situation where the server the server
>> is completely gone and isn't coming back, and wondering how to make  
>> the
>> mount fail faster.  But I may be misunderstanding.
>>
>> --b.
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs"  
> in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

No, i'm just using packages from CentOS repo...

And u're right about expo retries... with tcpdump i've monitored
traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
2049...
I tried use "retry=3D1" option on mount without any change... I dont
want change source or tcp timers... just NFSv4 client.

2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>
>> Bruce, no... you're right. =A0I'm describing a situation where my server
>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>> and 9 seconds...
>
> The 189 second timeout is likely how long it takes the kernel to give up
> trying to connect a TCP socket to the server (6 SYN attempts with
> exponential retries, or something like that). =A0For stock CentOS 5.3, I =
think
> user space does only a DNS lookup for normal NFSv4 mounts -- the kernel j=
ust
> tries to connect a TCP socket to port 2049, with no preceding rpcbind
> request.
>
> Carlos, let us know if you have replaced any NFS-related CentOS components
> (kernel, nfs-utils) with something you've built yourself.
>
>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>
>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>
>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com> wrot=
e:
>>>>>
>>>>> Anyone ?
>>>>>
>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>
>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>> Kerberos
>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>> LOOOOOOONG
>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>
>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if mount
>>>>>> hangs,
>>>>>> user logon hangs. Then i want configure it to timeout (if server dow=
n)
>>>>>> after
>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>
>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>> findings
>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using basic
>>>>>> command
>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>
>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR proto=
=3Dudp)
>>>>>> it
>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show error (mou=
nt:
>>>>>> mount to
>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>
>>>> Sounds like you're hitting the server's grace period.
>>>
>>> I thought he was describing a situation where the server the server
>>> is completely gone and isn't coming back, and wondering how to make the
>>> mount fail faster. =A0But I may be misunderstanding.
>>>
>>> --b.
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Something funny: Using default tcp_syn_retries (5) i got
"3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
1 i got "3,6,3,6,3,6..." secs interval...

[root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
sec=3Dkrb5p,proto=3Dtcp
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    3m9.000s
user    0m0.000s
sys     0m0.002s

[root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
[root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    2m6.004s
user    0m0.000s
sys     0m0.004s

(3,6,3,6... secs interval)




2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
> No, i'm just using packages from CentOS repo...
>
> And u're right about expo retries... with tcpdump i've monitored
> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
> 2049...
> I tried use "retry=3D1" option on mount without any change... I dont
> want change source or tcp timers... just NFSv4 client.
>
> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>
>>> Bruce, no... you're right. =A0I'm describing a situation where my server
>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>> and 9 seconds...
>>
>> The 189 second timeout is likely how long it takes the kernel to give up
>> trying to connect a TCP socket to the server (6 SYN attempts with
>> exponential retries, or something like that). =A0For stock CentOS 5.3, I=
 think
>> user space does only a DNS lookup for normal NFSv4 mounts -- the kernel =
just
>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>> request.
>>
>> Carlos, let us know if you have replaced any NFS-related CentOS componen=
ts
>> (kernel, nfs-utils) with something you've built yourself.
>>
>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>
>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>
>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com> wro=
te:
>>>>>>
>>>>>> Anyone ?
>>>>>>
>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>
>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>> Kerberos
>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>> LOOOOOOONG
>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>
>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if mou=
nt
>>>>>>> hangs,
>>>>>>> user logon hangs. Then i want configure it to timeout (if server do=
wn)
>>>>>>> after
>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>
>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>> findings
>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using basic
>>>>>>> command
>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>
>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR proto=
=3Dudp)
>>>>>>> it
>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show error (mo=
unt:
>>>>>>> mount to
>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>
>>>>> Sounds like you're hitting the server's grace period.
>>>>
>>>> I thought he was describing a situation where the server the server
>>>> is completely gone and isn't coming back, and wondering how to make the
>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>
>>>> --b.
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>>
>> --
>> Chuck Lever
>> chuck[dot]lever[at]oracle[dot]com
>>
>>
>>
>>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
> Something funny: Using default tcp_syn_retries (5) i got
> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
> 1 i got "3,6,3,6,3,6..." secs interval...

Right.  Normally the RPC client calls the kernel's socket connect =20
function, which does 6 SYN retries.  That one call usually takes =20
longer than the RPC client's connect timeout, so it only makes one =20
connect call, and then fails.

Reducing the number of SYN retries per connect attempt causes the RPC =20
client to retry the connect call until its connect timeout expires.  =20
Each connect call resets the SYN timeout to 3 seconds.

> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    3m9.000s
> user    0m0.000s
> sys     0m0.002s
>
> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    2m6.004s
> user    0m0.000s
> sys     0m0.004s
>
> (3,6,3,6... secs interval)
>
>
>
>
> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>> No, i'm just using packages from CentOS repo...
>>
>> And u're right about expo retries... with tcpdump i've monitored
>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>> 2049...
>> I tried use "retry=3D1" option on mount without any change... I dont
>> want change source or tcp timers... just NFSv4 client.
>>
>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>
>>>> Bruce, no... you're right.  I'm describing a situation where my =20
>>>> server
>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>> and 9 seconds...
>>>
>>> The 189 second timeout is likely how long it takes the kernel to =20
>>> give up
>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>> exponential retries, or something like that).  For stock CentOS =20
>>> 5.3, I think
>>> user space does only a DNS lookup for normal NFSv4 mounts -- the =20
>>> kernel just
>>> tries to connect a TCP socket to port 2049, with no preceding =20
>>> rpcbind
>>> request.
>>>
>>> Carlos, let us know if you have replaced any NFS-related CentOS =20
>>> components
>>> (kernel, nfs-utils) with something you've built yourself.
>>>
>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>
>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>
>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com> =20
>>>>>> wrote:
>>>>>>>
>>>>>>> Anyone ?
>>>>>>>
>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>
>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work =20
>>>>>>>> with
>>>>>>>> Kerberos
>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i =20
>>>>>>>> get a
>>>>>>>> LOOOOOOONG
>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>
>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, =20
>>>>>>>> if mount
>>>>>>>> hangs,
>>>>>>>> user logon hangs. Then i want configure it to timeout (if =20
>>>>>>>> server down)
>>>>>>>> after
>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>
>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>> findings
>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using =20
>>>>>>>> basic
>>>>>>>> command
>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>
>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR =20
>>>>>>>> proto=3Dudp)
>>>>>>>> it
>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error =20
>>>>>>>> (mount:
>>>>>>>> mount to
>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>
>>>>>> Sounds like you're hitting the server's grace period.
>>>>>
>>>>> I thought he was describing a situation where the server the =20
>>>>> server
>>>>> is completely gone and isn't coming back, and wondering how to =20
>>>>> make the
>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>
>>>>> --b.
>>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-=20
>>>> nfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>> --
>>> Chuck Lever
>>> chuck[dot]lever[at]oracle[dot]com
>>>
>>>
>>>
>>>
>>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

This long timeout is good if workstation need mount a critical
directory using /etc/fstab on boot (for example)..
But in my case, using this loooong timeout doesnt make any sense,
since autofs retry mount directory on-access. This in fact gives me
alot of headaches, coz user login 'll just hangs if one server goes
down for any reason, and will again hangs if user try access directory
pointing to a NFS down server...


Am losing something or there have was something weirdo...!?
------------------------------------------------
[root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEFAULT]
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o proto=tcp,retry=1
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    3m9.000s
user    0m0.002s
sys     0m0.001s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
sec=krb5p,proto=tcp,retry=1
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    3m9.000s
user    0m0.000s
sys     0m0.002s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o proto=tcp,retry=0
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    3m9.001s
user    0m0.000s
sys     0m0.003s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
sec=krb5p,proto=tcp,retry=0
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    3m9.001s
user    0m0.002s
sys     0m0.001s

[root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]

[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o proto=tcp,retry=1
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    1m3.002s
user    0m0.000s
sys     0m0.002s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
sec=krb5p,proto=tcp,retry=1
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    2m6.000s
user    0m0.000s
sys     0m0.002s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o proto=tcp,retry=0
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    0m9.003s
user    0m0.001s
sys     0m0.002s
[root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
sec=krb5p,proto=tcp,retry=0
mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).

real    2m6.001s
user    0m0.001s
sys     0m0.002s
[root@KSTATION ~]#
------------------------------------------------
max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
using retry=0 without kerberos I got only 9s...

*sigh*



2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>
>> Something funny: Using default tcp_syn_retries (5) i got
>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>> 1 i got "3,6,3,6,3,6..." secs interval...
>
> Right.  Normally the RPC client calls the kernel's socket connect function,
> which does 6 SYN retries.  That one call usually takes longer than the RPC
> client's connect timeout, so it only makes one connect call, and then fails.
>
> Reducing the number of SYN retries per connect attempt causes the RPC client
> to retry the connect call until its connect timeout expires.  Each connect
> call resets the SYN timeout to 3 seconds.
>
>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    3m9.000s
>> user    0m0.000s
>> sys     0m0.002s
>>
>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp  ("retry=1" = no change)
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    2m6.004s
>> user    0m0.000s
>> sys     0m0.004s
>>
>> (3,6,3,6... secs interval)
>>
>>
>>
>>
>> 2009/8/10 Carlos André <candrecn@gmail.com>:
>>>
>>> No, i'm just using packages from CentOS repo...
>>>
>>> And u're right about expo retries... with tcpdump i've monitored
>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>> 2049...
>>> I tried use "retry=1" option on mount without any change... I dont
>>> want change source or tcp timers... just NFSv4 client.
>>>
>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>
>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>
>>>>> Bruce, no... you're right.  I'm describing a situation where my server
>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>> and 9 seconds...
>>>>
>>>> The 189 second timeout is likely how long it takes the kernel to give up
>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>> exponential retries, or something like that).  For stock CentOS 5.3, I
>>>> think
>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the kernel
>>>> just
>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>> request.
>>>>
>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>> components
>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>
>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>
>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>
>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Anyone ?
>>>>>>>>
>>>>>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>>>>>>
>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>> Kerberos
>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>> LOOOOOOONG
>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>
>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>> mount
>>>>>>>>> hangs,
>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>> down)
>>>>>>>>> after
>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>
>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>> findings
>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using basic
>>>>>>>>> command
>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>
>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>> proto=udp)
>>>>>>>>> it
>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error (mount:
>>>>>>>>> mount to
>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>
>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>
>>>>>> I thought he was describing a situation where the server the server
>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>> the
>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>
>>>>>> --b.
>>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>> --
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>>>>
>>>
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
> This long timeout is good if workstation need mount a critical
> directory using /etc/fstab on boot (for example)..
> But in my case, using this loooong timeout doesnt make any sense,
> since autofs retry mount directory on-access. This in fact gives me
> alot of headaches, coz user login 'll just hangs if one server goes
> down for any reason, and will again hangs if user try access directory
> pointing to a NFS down server...

"retry=3D0" means the mount command will fail as soon as the first =20
mount(2) system call fails.  When you set SYN retries to 1, this means =20
after 9 seconds, the connect fails, and that causes the mount(2) =20
system call to fail.

Recent conversations with Ian suggested that a long timeout was =20
desired for automounter as well as other cases.  Ian, is there =20
something else we need to consider to determine the correct retry =20
timeout for NFS/TCP mount points handled via automounter?  How should =20
mount.nfs wait so we don't make other use cases worse?  (Looks like =20
most of the history is intact below).

How long do you think is appropriate for the automounter to wait if =20
the server is down, in your case, Carlos?

> Am losing something or there have was something weirdo...!?
> ------------------------------------------------
> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  =20
> [DEFAULT]
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o =20
> proto=3Dtcp,retry=3D1
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    3m9.000s
> user    0m0.002s
> sys     0m0.001s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    3m9.000s
> user    0m0.000s
> sys     0m0.002s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o =20
> proto=3Dtcp,retry=3D0
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    3m9.001s
> user    0m0.000s
> sys     0m0.003s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    3m9.001s
> user    0m0.002s
> sys     0m0.001s
>
> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 =20
> to 1 ]
>
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o =20
> proto=3Dtcp,retry=3D1
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =20
> [x 6]
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    1m3.002s
> user    0m0.000s
> sys     0m0.002s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =20
> [x 13]
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    2m6.000s
> user    0m0.000s
> sys     0m0.002s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o =20
> proto=3Dtcp,retry=3D0
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    0m9.003s
> user    0m0.001s
> sys     0m0.002s
> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =20
> [x 13]
> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>
> real    2m6.001s
> user    0m0.001s
> sys     0m0.002s
> [root@KSTATION ~]#
> ------------------------------------------------
> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
> using retry=3D0 without kerberos I got only 9s...
>
> *sigh*
>
>
>
> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>
>>> Something funny: Using default tcp_syn_retries (5) i got
>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries =20
>>> to
>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>
>> Right.  Normally the RPC client calls the kernel's socket connect =20
>> function,
>> which does 6 SYN retries.  That one call usually takes longer than =20
>> the RPC
>> client's connect timeout, so it only makes one connect call, and =20
>> then fails.
>>
>> Reducing the number of SYN retries per connect attempt causes the =20
>> RPC client
>> to retry the connect call until its connect timeout expires.  Each =20
>> connect
>> call resets the SYN timeout to 3 seconds.
>>
>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    3m9.000s
>>> user    0m0.000s
>>> sys     0m0.002s
>>>
>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    2m6.004s
>>> user    0m0.000s
>>> sys     0m0.004s
>>>
>>> (3,6,3,6... secs interval)
>>>
>>>
>>>
>>>
>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>
>>>> No, i'm just using packages from CentOS repo...
>>>>
>>>> And u're right about expo retries... with tcpdump i've monitored
>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>> 2049...
>>>> I tried use "retry=3D1" option on mount without any change... I dont
>>>> want change source or tcp timers... just NFSv4 client.
>>>>
>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>
>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>
>>>>>> Bruce, no... you're right.  I'm describing a situation where my =20
>>>>>> server
>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 =20
>>>>>> minutes
>>>>>> and 9 seconds...
>>>>>
>>>>> The 189 second timeout is likely how long it takes the kernel to =20
>>>>> give up
>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>> exponential retries, or something like that).  For stock CentOS =20
>>>>> 5.3, I
>>>>> think
>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the =20
>>>>> kernel
>>>>> just
>>>>> tries to connect a TCP socket to port 2049, with no preceding =20
>>>>> rpcbind
>>>>> request.
>>>>>
>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>> components
>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>
>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>
>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>
>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Anyone ?
>>>>>>>>>
>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>
>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work =20
>>>>>>>>>> with
>>>>>>>>>> Kerberos
>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i =20
>>>>>>>>>> get a
>>>>>>>>>> LOOOOOOONG
>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>
>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon =20
>>>>>>>>>> process, if
>>>>>>>>>> mount
>>>>>>>>>> hangs,
>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if =20
>>>>>>>>>> server
>>>>>>>>>> down)
>>>>>>>>>> after
>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>
>>>>>>>>>> I already make a lab and tried a LOT of combinations, there =20
>>>>>>>>>> my
>>>>>>>>>> findings
>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) =20
>>>>>>>>>> using basic
>>>>>>>>>> command
>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>
>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR
>>>>>>>>>> proto=3Dudp)
>>>>>>>>>> it
>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error =20
>>>>>>>>>> (mount:
>>>>>>>>>> mount to
>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>
>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>
>>>>>>> I thought he was describing a situation where the server the =20
>>>>>>> server
>>>>>>> is completely gone and isn't coming back, and wondering how to =20
>>>>>>> make
>>>>>>> the
>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>
>>>>>>> --b.
>>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-=20
>>>>>> nfs" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-=20
>>>>>> info.html
>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>
>> --
>> Chuck Lever
>> chuck[dot]lever[at]oracle[dot]com
>>
>>
>>
>>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Chuck,
Since we'll use some NFS servers, and (some) they are not critical to
user work (if they're down, user can just sit and wait for them
working on something else like  a webapp, printing, etc...) and if we
lose a server or a router (worst case) acumulative timeouts on
workstation login-boot process will make users try kill us (lol). A
acceptable timeout(first mount fail timeout) per automount try is
something around 10-15 (max) seconds. For us the best is a option to
permit us to do adjustments (1 to X secs), and putting this like a new
feature/non-default option will not mess with another users.... I dont
wanna make a mess on sources by myself just to modify automounter for
our needs, I just want the "right" solution in our case...

Thanks for help.

2009/8/11 Chuck Lever <chuck.lever@oracle.com>:
> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>
>> This long timeout is good if workstation need mount a critical
>> directory using /etc/fstab on boot (for example)..
>> But in my case, using this loooong timeout doesnt make any sense,
>> since autofs retry mount directory on-access. This in fact gives me
>> alot of headaches, coz user login 'll just hangs if one server goes
>> down for any reason, and will again hangs if user try access directory
>> pointing to a NFS down server...
>
> "retry=3D0" means the mount command will fail as soon as the first mount(=
2)
> system call fails. =A0When you set SYN retries to 1, this means after 9
> seconds, the connect fails, and that causes the mount(2) system call to
> fail.
>
> Recent conversations with Ian suggested that a long timeout was desired f=
or
> automounter as well as other cases. =A0Ian, is there something else we ne=
ed to
> consider to determine the correct retry timeout for NFS/TCP mount points
> handled via automounter? =A0How should mount.nfs wait so we don't make ot=
her
> use cases worse? =A0(Looks like most of the history is intact below).
>
> How long do you think is appropriate for the automounter to wait if the
> server is down, in your case, Carlos?
>
>> Am losing something or there have was something weirdo...!?
>> ------------------------------------------------
>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0[DEFAU=
LT]
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=3Dtcp,retry=3D1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A03m9.000s
>> user =A0 =A00m0.002s
>> sys =A0 =A0 0m0.001s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A03m9.000s
>> user =A0 =A00m0.000s
>> sys =A0 =A0 0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=3Dtcp,retry=3D0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A03m9.001s
>> user =A0 =A00m0.000s
>> sys =A0 =A0 0m0.003s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A03m9.001s
>> user =A0 =A00m0.002s
>> sys =A0 =A0 0m0.001s
>>
>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 =
]
>>
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=3Dtcp,retry=3D1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A01m3.002s
>> user =A0 =A00m0.000s
>> sys =A0 =A0 0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13=
]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A02m6.000s
>> user =A0 =A00m0.000s
>> sys =A0 =A0 0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=3Dtcp,retry=3D0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A00m9.003s
>> user =A0 =A00m0.001s
>> sys =A0 =A0 0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13=
]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real =A0 =A02m6.001s
>> user =A0 =A00m0.001s
>> sys =A0 =A0 0m0.002s
>> [root@KSTATION ~]#
>> ------------------------------------------------
>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>> using retry=3D0 without kerberos I got only 9s...
>>
>> *sigh*
>>
>>
>>
>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>
>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>
>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>
>>> Right. =A0Normally the RPC client calls the kernel's socket connect
>>> function,
>>> which does 6 SYN retries. =A0That one call usually takes longer than th=
e
>>> RPC
>>> client's connect timeout, so it only makes one connect call, and then
>>> fails.
>>>
>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>> client
>>> to retry the connect call until its connect timeout expires. =A0Each
>>> connect
>>> call resets the SYN timeout to 3 seconds.
>>>
>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real =A0 =A03m9.000s
>>>> user =A0 =A00m0.000s
>>>> sys =A0 =A0 0m0.002s
>>>>
>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real =A0 =A02m6.004s
>>>> user =A0 =A00m0.000s
>>>> sys =A0 =A0 0m0.004s
>>>>
>>>> (3,6,3,6... secs interval)
>>>>
>>>>
>>>>
>>>>
>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>
>>>>> No, i'm just using packages from CentOS repo...
>>>>>
>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>> 2049...
>>>>> I tried use "retry=3D1" option on mount without any change... I dont
>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>
>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>
>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>
>>>>>>> Bruce, no... you're right. =A0I'm describing a situation where my
>>>>>>> server
>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>> and 9 seconds...
>>>>>>
>>>>>> The 189 second timeout is likely how long it takes the kernel to giv=
e
>>>>>> up
>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>> exponential retries, or something like that). =A0For stock CentOS 5.=
3, I
>>>>>> think
>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>> kernel
>>>>>> just
>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbin=
d
>>>>>> request.
>>>>>>
>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>> components
>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>
>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>
>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>
>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Anyone ?
>>>>>>>>>>
>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work wit=
h
>>>>>>>>>>> Kerberos
>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get =
a
>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>
>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>> mount
>>>>>>>>>>> hangs,
>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if serve=
r
>>>>>>>>>>> down)
>>>>>>>>>>> after
>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>
>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>> findings
>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>> basic
>>>>>>>>>>> command
>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>
>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR
>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>> it
>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show error
>>>>>>>>>>> (mount:
>>>>>>>>>>> mount to
>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>
>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>
>>>>>>>> I thought he was describing a situation where the server the serve=
r
>>>>>>>> is completely gone and isn't coming back, and wondering how to mak=
e
>>>>>>>> the
>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>
>>>>>>>> --b.
>>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs=
"
>>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.htm=
l
>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>> --
>>> Chuck Lever
>>> chuck[dot]lever[at]oracle[dot]com
>>>
>>>
>>>
>>>
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
>
_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Carlos André wrote:
> Chuck,
> Since we'll use some NFS servers, and (some) they are not critical to
> user work (if they're down, user can just sit and wait for them
> working on something else like  a webapp, printing, etc...) and if we
> lose a server or a router (worst case) acumulative timeouts on
> workstation login-boot process will make users try kill us (lol). A
> acceptable timeout(first mount fail timeout) per automount try is
> something around 10-15 (max) seconds. For us the best is a option to
> permit us to do adjustments (1 to X secs), and putting this like a new
> feature/non-default option will not mess with another users.... I dont
> wanna make a mess on sources by myself just to modify automounter for
> our needs, I just want the "right" solution in our case...

But that also begs the question.
What version of autofs is in use?

The latest versions have the ability to specify a time to wait before
killing the mount process but I think that itself is an option that
hasn't had the bugs worked out yet because the RPC layer seems to want
to refuse to give up even when we kill the process.

Perhaps we can work on this further, depending on your version
constraints, of course.

> 
> Thanks for help.
> 
> 2009/8/11 Chuck Lever <chuck.lever@oracle.com>:
>> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>>> This long timeout is good if workstation need mount a critical
>>> directory using /etc/fstab on boot (for example)..
>>> But in my case, using this loooong timeout doesnt make any sense,
>>> since autofs retry mount directory on-access. This in fact gives me
>>> alot of headaches, coz user login 'll just hangs if one server goes
>>> down for any reason, and will again hangs if user try access directory
>>> pointing to a NFS down server...
>> "retry=0" means the mount command will fail as soon as the first mount(2)
>> system call fails.  When you set SYN retries to 1, this means after 9
>> seconds, the connect fails, and that causes the mount(2) system call to
>> fail.
>>
>> Recent conversations with Ian suggested that a long timeout was desired for
>> automounter as well as other cases.  Ian, is there something else we need to
>> consider to determine the correct retry timeout for NFS/TCP mount points
>> handled via automounter?  How should mount.nfs wait so we don't make other
>> use cases worse?  (Looks like most of the history is intact below).
>>
>> How long do you think is appropriate for the automounter to wait if the
>> server is down, in your case, Carlos?
>>
>>> Am losing something or there have was something weirdo...!?
>>> ------------------------------------------------
>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEFAULT]
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=tcp,retry=1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    3m9.000s
>>> user    0m0.002s
>>> sys     0m0.001s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=krb5p,proto=tcp,retry=1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    3m9.000s
>>> user    0m0.000s
>>> sys     0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=tcp,retry=0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    3m9.001s
>>> user    0m0.000s
>>> sys     0m0.003s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=krb5p,proto=tcp,retry=0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    3m9.001s
>>> user    0m0.002s
>>> sys     0m0.001s
>>>
>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]
>>>
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=tcp,retry=1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    1m3.002s
>>> user    0m0.000s
>>> sys     0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=krb5p,proto=tcp,retry=1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    2m6.000s
>>> user    0m0.000s
>>> sys     0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=tcp,retry=0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    0m9.003s
>>> user    0m0.001s
>>> sys     0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=krb5p,proto=tcp,retry=0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real    2m6.001s
>>> user    0m0.001s
>>> sys     0m0.002s
>>> [root@KSTATION ~]#
>>> ------------------------------------------------
>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>> using retry=0 without kerberos I got only 9s...
>>>
>>> *sigh*
>>>
>>>
>>>
>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>> Right.  Normally the RPC client calls the kernel's socket connect
>>>> function,
>>>> which does 6 SYN retries.  That one call usually takes longer than the
>>>> RPC
>>>> client's connect timeout, so it only makes one connect call, and then
>>>> fails.
>>>>
>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>> client
>>>> to retry the connect call until its connect timeout expires.  Each
>>>> connect
>>>> call resets the SYN timeout to 3 seconds.
>>>>
>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real    3m9.000s
>>>>> user    0m0.000s
>>>>> sys     0m0.002s
>>>>>
>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>> sec=krb5p,proto=tcp  ("retry=1" = no change)
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real    2m6.004s
>>>>> user    0m0.000s
>>>>> sys     0m0.004s
>>>>>
>>>>> (3,6,3,6... secs interval)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2009/8/10 Carlos André <candrecn@gmail.com>:
>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>
>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>> 2049...
>>>>>> I tried use "retry=1" option on mount without any change... I dont
>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>
>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>> Bruce, no... you're right.  I'm describing a situation where my
>>>>>>>> server
>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>>> and 9 seconds...
>>>>>>> The 189 second timeout is likely how long it takes the kernel to give
>>>>>>> up
>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>> exponential retries, or something like that).  For stock CentOS 5.3, I
>>>>>>> think
>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>> kernel
>>>>>>> just
>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>>>>> request.
>>>>>>>
>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>> components
>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>
>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> Anyone ?
>>>>>>>>>>>
>>>>>>>>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>>>>> Kerberos
>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>
>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>>> mount
>>>>>>>>>>>> hangs,
>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>>>>> down)
>>>>>>>>>>>> after
>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>
>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>>> findings
>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>> basic
>>>>>>>>>>>> command
>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>>
>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>>>>> proto=udp)
>>>>>>>>>>>> it
>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error
>>>>>>>>>>>> (mount:
>>>>>>>>>>>> mount to
>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>> I thought he was describing a situation where the server the server
>>>>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>>>>> the
>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>
>>>>>>>>> --b.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-nfs"
>>>>>>>> in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>> --
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>>>>
>> --
>> Chuck Lever
>> chuck[dot]lever[at]oracle[dot]com
>>
>>
>>
>>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Chuck Lever wrote:
> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>> This long timeout is good if workstation need mount a critical
>> directory using /etc/fstab on boot (for example)..
>> But in my case, using this loooong timeout doesnt make any sense,
>> since autofs retry mount directory on-access. This in fact gives me
>> alot of headaches, coz user login 'll just hangs if one server goes
>> down for any reason, and will again hangs if user try access directory
>> pointing to a NFS down server...
> 
> "retry=0" means the mount command will fail as soon as the first
> mount(2) system call fails.  When you set SYN retries to 1, this means
> after 9 seconds, the connect fails, and that causes the mount(2) system
> call to fail.
> 
> Recent conversations with Ian suggested that a long timeout was desired
> for automounter as well as other cases.  Ian, is there something else we
> need to consider to determine the correct retry timeout for NFS/TCP
> mount points handled via automounter?  How should mount.nfs wait so we
> don't make other use cases worse?  (Looks like most of the history is
> intact below).

Of course we know that autofs is entirely at the mercy of mount(8) (and
mount.nfs in particular). This has always been a difficult situation for
the automounter because interactive mount invocations should wait. But I
believe automount mounts should always time out quickly, but that leads
to its own set of problems, especially when home directories are concerned.

I think adding "retry=0" is the right thing to do myself but I'm not
certain that will work as we expect. I'll have to do some experimentation.

> 
> How long do you think is appropriate for the automounter to wait if the
> server is down, in your case, Carlos?
> 
>> Am losing something or there have was something weirdo...!?
>> ------------------------------------------------
>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEFAULT]
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=tcp,retry=1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    3m9.000s
>> user    0m0.002s
>> sys     0m0.001s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp,retry=1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    3m9.000s
>> user    0m0.000s
>> sys     0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=tcp,retry=0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    3m9.001s
>> user    0m0.000s
>> sys     0m0.003s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp,retry=0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    3m9.001s
>> user    0m0.002s
>> sys     0m0.001s
>>
>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to 1 ]
>>
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=tcp,retry=1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 6]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    1m3.002s
>> user    0m0.000s
>> sys     0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp,retry=1
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    2m6.000s
>> user    0m0.000s
>> sys     0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> proto=tcp,retry=0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    0m9.003s
>> user    0m0.001s
>> sys     0m0.002s
>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>> sec=krb5p,proto=tcp,retry=0
>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x 13]
>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>
>> real    2m6.001s
>> user    0m0.001s
>> sys     0m0.002s
>> [root@KSTATION ~]#
>> ------------------------------------------------
>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>> using retry=0 without kerberos I got only 9s...
>>
>> *sigh*
>>
>>
>>
>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>
>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>
>>> Right.  Normally the RPC client calls the kernel's socket connect
>>> function,
>>> which does 6 SYN retries.  That one call usually takes longer than
>>> the RPC
>>> client's connect timeout, so it only makes one connect call, and then
>>> fails.
>>>
>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>> client
>>> to retry the connect call until its connect timeout expires.  Each
>>> connect
>>> call resets the SYN timeout to 3 seconds.
>>>
>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>> sec=krb5p,proto=tcp
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    3m9.000s
>>>> user    0m0.000s
>>>> sys     0m0.002s
>>>>
>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>> sec=krb5p,proto=tcp  ("retry=1" = no change)
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    2m6.004s
>>>> user    0m0.000s
>>>> sys     0m0.004s
>>>>
>>>> (3,6,3,6... secs interval)
>>>>
>>>>
>>>>
>>>>
>>>> 2009/8/10 Carlos André <candrecn@gmail.com>:
>>>>>
>>>>> No, i'm just using packages from CentOS repo...
>>>>>
>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>> 2049...
>>>>> I tried use "retry=1" option on mount without any change... I dont
>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>
>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>
>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>
>>>>>>> Bruce, no... you're right.  I'm describing a situation where my
>>>>>>> server
>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>>>>>> and 9 seconds...
>>>>>>
>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>> give up
>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>> exponential retries, or something like that).  For stock CentOS
>>>>>> 5.3, I
>>>>>> think
>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>> kernel
>>>>>> just
>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>>>>>> request.
>>>>>>
>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>> components
>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>
>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>
>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>
>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Anyone ?
>>>>>>>>>>
>>>>>>>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>>>>>> Kerberos
>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>
>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, if
>>>>>>>>>>> mount
>>>>>>>>>>> hangs,
>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if server
>>>>>>>>>>> down)
>>>>>>>>>>> after
>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>
>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>> findings
>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>> basic
>>>>>>>>>>> command
>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>
>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR
>>>>>>>>>>> proto=udp)
>>>>>>>>>>> it
>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error
>>>>>>>>>>> (mount:
>>>>>>>>>>> mount to
>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>
>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>
>>>>>>>> I thought he was describing a situation where the server the server
>>>>>>>> is completely gone and isn't coming back, and wondering how to make
>>>>>>>> the
>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>
>>>>>>>> --b.
>>>>>>>>
>>>>>>> -- 
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>> linux-nfs" in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>> -- 
>>>>>> Chuck Lever
>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>> -- 
>>> Chuck Lever
>>> chuck[dot]lever[at]oracle[dot]com
>>>
>>>
>>>
>>>
> 
> -- 
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
> 
> 
> 

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Hi Ian,
I'm getting crazy trying put "retry=3D" to work on mount... this option
just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/krb=
5p)
like you can see on my previous emails...

I appreciate any help.

Carlos.


2009/8/12 Ian Kent <ikent@redhat.com>:
> Chuck Lever wrote:
>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>> This long timeout is good if workstation need mount a critical
>>> directory using /etc/fstab on boot (for example)..
>>> But in my case, using this loooong timeout doesnt make any sense,
>>> since autofs retry mount directory on-access. This in fact gives me
>>> alot of headaches, coz user login 'll just hangs if one server goes
>>> down for any reason, and will again hangs if user try access direct=
ory
>>> pointing to a NFS down server...
>>
>> "retry=3D0" means the mount command will fail as soon as the first
>> mount(2) system call fails. =A0When you set SYN retries to 1, this m=
eans
>> after 9 seconds, the connect fails, and that causes the mount(2) sys=
tem
>> call to fail.
>>
>> Recent conversations with Ian suggested that a long timeout was desi=
red
>> for automounter as well as other cases. =A0Ian, is there something e=
lse we
>> need to consider to determine the correct retry timeout for NFS/TCP
>> mount points handled via automounter? =A0How should mount.nfs wait s=
o we
>> don't make other use cases worse? =A0(Looks like most of the history=
 is
>> intact below).
>
> Of course we know that autofs is entirely at the mercy of mount(8) (a=
nd
> mount.nfs in particular). This has always been a difficult situation =
for
> the automounter because interactive mount invocations should wait. Bu=
t I
> believe automount mounts should always time out quickly, but that lea=
ds
> to its own set of problems, especially when home directories are conc=
erned.
>
> I think adding "retry=3D0" is the right thing to do myself but I'm no=
t
> certain that will work as we expect. I'll have to do some experimenta=
tion.
>
>>
>> How long do you think is appropriate for the automounter to wait if =
the
>> server is down, in your case, Carlos?
>>
>>> Am losing something or there have was something weirdo...!?
>>> ------------------------------------------------
>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0[=
DEFAULT]
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=3Dtcp,retry=3D1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A03m9.000s
>>> user =A0 =A00m0.002s
>>> sys =A0 =A0 0m0.001s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A03m9.000s
>>> user =A0 =A00m0.000s
>>> sys =A0 =A0 0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=3Dtcp,retry=3D0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A03m9.001s
>>> user =A0 =A00m0.000s
>>> sys =A0 =A0 0m0.003s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A03m9.001s
>>> user =A0 =A00m0.002s
>>> sys =A0 =A0 0m0.001s
>>>
>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 =
to 1 ]
>>>
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=3Dtcp,retry=3D1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 6]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A01m3.002s
>>> user =A0 =A00m0.000s
>>> sys =A0 =A0 0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 13]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A02m6.000s
>>> user =A0 =A00m0.000s
>>> sys =A0 =A0 0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> proto=3Dtcp,retry=3D0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A00m9.003s
>>> user =A0 =A00m0.001s
>>> sys =A0 =A0 0m0.002s
>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 13]
>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>
>>> real =A0 =A02m6.001s
>>> user =A0 =A00m0.001s
>>> sys =A0 =A0 0m0.002s
>>> [root@KSTATION ~]#
>>> ------------------------------------------------
>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... an=
d
>>> using retry=3D0 without kerberos I got only 9s...
>>>
>>> *sigh*
>>>
>>>
>>>
>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>
>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retrie=
s to
>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>
>>>> Right. =A0Normally the RPC client calls the kernel's socket connec=
t
>>>> function,
>>>> which does 6 SYN retries. =A0That one call usually takes longer th=
an
>>>> the RPC
>>>> client's connect timeout, so it only makes one connect call, and t=
hen
>>>> fails.
>>>>
>>>> Reducing the number of SYN retries per connect attempt causes the =
RPC
>>>> client
>>>> to retry the connect call until its connect timeout expires. =A0Ea=
ch
>>>> connect
>>>> call resets the SYN timeout to 3 seconds.
>>>>
>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up=
).
>>>>>
>>>>> real =A0 =A03m9.000s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.002s
>>>>>
>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying)=
=2E
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up=
).
>>>>>
>>>>> real =A0 =A02m6.004s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.004s
>>>>>
>>>>> (3,6,3,6... secs interval)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>
>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>
>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on po=
rt
>>>>>> 2049...
>>>>>> I tried use "retry=3D1" option on mount without any change... I =
dont
>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>
>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>
>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>
>>>>>>>> Bruce, no... you're right. =A0I'm describing a situation where=
 my
>>>>>>>> server
>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 mi=
nutes
>>>>>>>> and 9 seconds...
>>>>>>>
>>>>>>> The 189 second timeout is likely how long it takes the kernel t=
o
>>>>>>> give up
>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts wi=
th
>>>>>>> exponential retries, or something like that). =A0For stock Cent=
OS
>>>>>>> 5.3, I
>>>>>>> think
>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- th=
e
>>>>>>> kernel
>>>>>>> just
>>>>>>> tries to connect a TCP socket to port 2049, with no preceding r=
pcbind
>>>>>>> request.
>>>>>>>
>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>> components
>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>
>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>
>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>
>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail=
=2Ecom>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Anyone ?
>>>>>>>>>>>
>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to wor=
k with
>>>>>>>>>>>> Kerberos
>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i=
 get a
>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>
>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon proces=
s, if
>>>>>>>>>>>> mount
>>>>>>>>>>>> hangs,
>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if =
server
>>>>>>>>>>>> down)
>>>>>>>>>>>> after
>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>
>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, ther=
e my
>>>>>>>>>>>> findings
>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) usi=
ng
>>>>>>>>>>>> basic
>>>>>>>>>>>> command
>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>
>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp =
OR
>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>> it
>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show =
error
>>>>>>>>>>>> (mount:
>>>>>>>>>>>> mount to
>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>
>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>
>>>>>>>>> I thought he was describing a situation where the server the =
server
>>>>>>>>> is completely gone and isn't coming back, and wondering how t=
o make
>>>>>>>>> the
>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>
>>>>>>>>> --b.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-nfs" in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-inf=
o.html
>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>>>>
>>
>> --
>> Chuck Lever
>> chuck[dot]lever[at]oracle[dot]com
>>
>>
>>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Carlos Andr=E9 wrote:
> Hi Ian,
> I'm getting crazy trying put "retry=3D" to work on mount... this option
> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/krb5p)
> like you can see on my previous emails...

Right, my mistake for not looking closely enough at post.

Maybe this is related to the same sort of problem we had with mount in
the past, before the options parsing went into the kernel, where other
services, like portmapper (or rpcbind), were being done with different
timeout parameters before the RPC calls for mounting. That's just an
example as NFSv4 shouldn't be sensitive to portmapper anyway.

But what version of autofs and kernel did you say you were using?

> =

> I appreciate any help.
> =

> Carlos.
> =

> =

> 2009/8/12 Ian Kent <ikent@redhat.com>:
>> Chuck Lever wrote:
>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>> This long timeout is good if workstation need mount a critical
>>>> directory using /etc/fstab on boot (for example)..
>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>> since autofs retry mount directory on-access. This in fact gives me
>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>> down for any reason, and will again hangs if user try access directory
>>>> pointing to a NFS down server...
>>> "retry=3D0" means the mount command will fail as soon as the first
>>> mount(2) system call fails.  When you set SYN retries to 1, this means
>>> after 9 seconds, the connect fails, and that causes the mount(2) system
>>> call to fail.
>>>
>>> Recent conversations with Ian suggested that a long timeout was desired
>>> for automounter as well as other cases.  Ian, is there something else we
>>> need to consider to determine the correct retry timeout for NFS/TCP
>>> mount points handled via automounter?  How should mount.nfs wait so we
>>> don't make other use cases worse?  (Looks like most of the history is
>>> intact below).
>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>> mount.nfs in particular). This has always been a difficult situation for
>> the automounter because interactive mount invocations should wait. But I
>> believe automount mounts should always time out quickly, but that leads
>> to its own set of problems, especially when home directories are concern=
ed.
>>
>> I think adding "retry=3D0" is the right thing to do myself but I'm not
>> certain that will work as we expect. I'll have to do some experimentatio=
n.
>>
>>> How long do you think is appropriate for the automounter to wait if the
>>> server is down, in your case, Carlos?
>>>
>>>> Am losing something or there have was something weirdo...!?
>>>> ------------------------------------------------
>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEFAU=
LT]
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> proto=3Dtcp,retry=3D1
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    3m9.000s
>>>> user    0m0.002s
>>>> sys     0m0.001s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    3m9.000s
>>>> user    0m0.000s
>>>> sys     0m0.002s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> proto=3Dtcp,retry=3D0
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    3m9.001s
>>>> user    0m0.000s
>>>> sys     0m0.003s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    3m9.001s
>>>> user    0m0.002s
>>>> sys     0m0.001s
>>>>
>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to =
1 ]
>>>>
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> proto=3Dtcp,retry=3D1
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x =
6]
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    1m3.002s
>>>> user    0m0.000s
>>>> sys     0m0.002s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x =
13]
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    2m6.000s
>>>> user    0m0.000s
>>>> sys     0m0.002s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> proto=3Dtcp,retry=3D0
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    0m9.003s
>>>> user    0m0.001s
>>>> sys     0m0.002s
>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x =
13]
>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>
>>>> real    2m6.001s
>>>> user    0m0.001s
>>>> sys     0m0.002s
>>>> [root@KSTATION ~]#
>>>> ------------------------------------------------
>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>> using retry=3D0 without kerberos I got only 9s...
>>>>
>>>> *sigh*
>>>>
>>>>
>>>>
>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries to
>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>> Right.  Normally the RPC client calls the kernel's socket connect
>>>>> function,
>>>>> which does 6 SYN retries.  That one call usually takes longer than
>>>>> the RPC
>>>>> client's connect timeout, so it only makes one connect call, and then
>>>>> fails.
>>>>>
>>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>>> client
>>>>> to retry the connect call until its connect timeout expires.  Each
>>>>> connect
>>>>> call resets the SYN timeout to 3 seconds.
>>>>>
>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    3m9.000s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>>
>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    2m6.004s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.004s
>>>>>>
>>>>>> (3,6,3,6... secs interval)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>
>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>> 2049...
>>>>>>> I tried use "retry=3D1" option on mount without any change... I dont
>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>
>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>> Bruce, no... you're right.  I'm describing a situation where my
>>>>>>>>> server
>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minut=
es
>>>>>>>>> and 9 seconds...
>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>> give up
>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>> exponential retries, or something like that).  For stock CentOS
>>>>>>>> 5.3, I
>>>>>>>> think
>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>> kernel
>>>>>>>> just
>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpcb=
ind
>>>>>>>> request.
>>>>>>>>
>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>> components
>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>
>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.co=
m>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>
>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work w=
ith
>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i ge=
t a
>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>
>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process, =
if
>>>>>>>>>>>>> mount
>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if ser=
ver
>>>>>>>>>>>>> down)
>>>>>>>>>>>>> after
>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>>>>>>>> findings
>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>> basic
>>>>>>>>>>>>> command
>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR
>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>> it
>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error
>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>> mount to
>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>> I thought he was describing a situation where the server the ser=
ver
>>>>>>>>>> is completely gone and isn't coming back, and wondering how to m=
ake
>>>>>>>>>> the
>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>
>>>>>>>>>> --b.
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>> linux-nfs" in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>> --
>>>>>>>> Chuck Lever
>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>
>>>>>
>>>>>
>>>>>
>>> --
>>> Chuck Lever
>>> chuck[dot]lever[at]oracle[dot]com
>>>
>>>
>>>
>>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Today (2009-08-12) I'm using:
kernel-2.6.18-128.2.1.el5
autofs-5.0.1-0.rc2.102.el5_3.1


Look my last test:
--------------------------------------------------------------
[root@KSTATION areas]# time ls testdown
ls: testdown: No such file or directory

real    3m9.025s
user    0m0.000s
sys     0m0.002s




Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
mounting root /misc/areas, mountpoint testdown, what
1.2.3.4:/areas/testdown, fstype nfs4, options
acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D0, ro=3D0
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
calling mkdir_path /misc/areas/testdown
Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
1.2.3.4:/areas/testdown /misc/areas/testdown
Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =3D
3078093712 path /misc
Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
submounts remaining in /misc
Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
3078093712 path /misc stat 3
Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
exp 3078093712 finished, switching from 2 to 1
Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
=3D 2 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =3D
3078093712 path /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
submounts remaining in /misc
Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
3078093712 path /misc stat 3
Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
exp 3078093712 finished, switching from 2 to 1
Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
=3D 2 path /misc
Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
server '1.2.3.4' failed: timed out (giving up).
Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/test=
down
Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
--------------------------------------------------------------

2009/8/12 Ian Kent <ikent@redhat.com>:
> Carlos Andr=E9 wrote:
>> Hi Ian,
>> I'm getting crazy trying put "retry=3D" to work on mount... this option
>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/krb5=
p)
>> like you can see on my previous emails...
>
> Right, my mistake for not looking closely enough at post.
>
> Maybe this is related to the same sort of problem we had with mount in
> the past, before the options parsing went into the kernel, where other
> services, like portmapper (or rpcbind), were being done with different
> timeout parameters before the RPC calls for mounting. That's just an
> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>
> But what version of autofs and kernel did you say you were using?
>
>>
>> I appreciate any help.
>>
>> Carlos.
>>
>>
>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>> Chuck Lever wrote:
>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>> This long timeout is good if workstation need mount a critical
>>>>> directory using /etc/fstab on boot (for example)..
>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>> down for any reason, and will again hangs if user try access directory
>>>>> pointing to a NFS down server...
>>>> "retry=3D0" means the mount command will fail as soon as the first
>>>> mount(2) system call fails. =A0When you set SYN retries to 1, this mea=
ns
>>>> after 9 seconds, the connect fails, and that causes the mount(2) system
>>>> call to fail.
>>>>
>>>> Recent conversations with Ian suggested that a long timeout was desired
>>>> for automounter as well as other cases. =A0Ian, is there something els=
e we
>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>> mount points handled via automounter? =A0How should mount.nfs wait so =
we
>>>> don't make other use cases worse? =A0(Looks like most of the history is
>>>> intact below).
>>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>>> mount.nfs in particular). This has always been a difficult situation for
>>> the automounter because interactive mount invocations should wait. But I
>>> believe automount mounts should always time out quickly, but that leads
>>> to its own set of problems, especially when home directories are concer=
ned.
>>>
>>> I think adding "retry=3D0" is the right thing to do myself but I'm not
>>> certain that will work as we expect. I'll have to do some experimentati=
on.
>>>
>>>> How long do you think is appropriate for the automounter to wait if the
>>>> server is down, in your case, Carlos?
>>>>
>>>>> Am losing something or there have was something weirdo...!?
>>>>> ------------------------------------------------
>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0[DE=
FAULT]
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=3Dtcp,retry=3D1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A03m9.000s
>>>>> user =A0 =A00m0.002s
>>>>> sys =A0 =A0 0m0.001s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A03m9.000s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=3Dtcp,retry=3D0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A03m9.001s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.003s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A03m9.001s
>>>>> user =A0 =A00m0.002s
>>>>> sys =A0 =A0 0m0.001s
>>>>>
>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 to=
 1 ]
>>>>>
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=3Dtcp,retry=3D1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x=
 6]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A01m3.002s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x=
 13]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A02m6.000s
>>>>> user =A0 =A00m0.000s
>>>>> sys =A0 =A0 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> proto=3Dtcp,retry=3D0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A00m9.003s
>>>>> user =A0 =A00m0.001s
>>>>> sys =A0 =A0 0m0.002s
>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [x=
 13]
>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>
>>>>> real =A0 =A02m6.001s
>>>>> user =A0 =A00m0.001s
>>>>> sys =A0 =A0 0m0.002s
>>>>> [root@KSTATION ~]#
>>>>> ------------------------------------------------
>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>
>>>>> *sigh*
>>>>>
>>>>>
>>>>>
>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries =
to
>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>> Right. =A0Normally the RPC client calls the kernel's socket connect
>>>>>> function,
>>>>>> which does 6 SYN retries. =A0That one call usually takes longer than
>>>>>> the RPC
>>>>>> client's connect timeout, so it only makes one connect call, and then
>>>>>> fails.
>>>>>>
>>>>>> Reducing the number of SYN retries per connect attempt causes the RPC
>>>>>> client
>>>>>> to retry the connect call until its connect timeout expires. =A0Each
>>>>>> connect
>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>
>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A03m9.000s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>
>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A02m6.004s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.004s
>>>>>>>
>>>>>>> (3,6,3,6... secs interval)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>
>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>>> 2049...
>>>>>>>> I tried use "retry=3D1" option on mount without any change... I do=
nt
>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>
>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situation where my
>>>>>>>>>> server
>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 minu=
tes
>>>>>>>>>> and 9 seconds...
>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>> give up
>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>>> exponential retries, or something like that). =A0For stock CentOS
>>>>>>>>> 5.3, I
>>>>>>>>> think
>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>> kernel
>>>>>>>>> just
>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rpc=
bind
>>>>>>>>> request.
>>>>>>>>>
>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>> components
>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>
>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.c=
om>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work =
with
>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i g=
et a
>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process,=
 if
>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if se=
rver
>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>> after
>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there =
my
>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>> command
>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR
>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show er=
ror
>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>> I thought he was describing a situation where the server the se=
rver
>>>>>>>>>>> is completely gone and isn't coming back, and wondering how to =
make
>>>>>>>>>>> the
>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>>>
>>>>>>>>>>> --b.
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>> linux-nfs" in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-info.=
html
>>>>>>>>> --
>>>>>>>>> Chuck Lever
>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>> --
>>>> Chuck Lever
>>>> chuck[dot]lever[at]oracle[dot]com
>>>>
>>>>
>>>>
>>>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Carlos Andr=E9 wrote:
> Today (2009-08-12) I'm using:
> kernel-2.6.18-128.2.1.el5
> autofs-5.0.1-0.rc2.102.el5_3.1

Thanks,

My mistake, the wait time I was referring to is used for umounts during
expires and is present in rev rc2.102.

It shouldn't be hard to add this for mount as well.
Would you like me to put something together?

Probably would be good to test something out to see if we can make a
difference with the killing mount after some configured timeout but, if
we make progress, probably the best way to deal with it is for you to
log a bug against rhel-5 so I can get it committed to the rhel package.
The possible issue is that I'm not sure if the RPC subsystem in the
above rhel kernel will respond well to process death with potential
outstanding requests. But we'll see.

> =

> =

> Look my last test:
> --------------------------------------------------------------
> [root@KSTATION areas]# time ls testdown
> ls: testdown: No such file or directory
> =

> real    3m9.025s
> user    0m0.000s
> sys     0m0.002s
> =

> =

> =

> =

> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
> mounting root /misc/areas, mountpoint testdown, what
> 1.2.3.4:/areas/testdown, fstype nfs4, options
> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D0, ro=
=3D0
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> calling mkdir_path /misc/areas/testdown
> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
> 1.2.3.4:/areas/testdown /misc/areas/testdown
> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =3D
> 3078093712 path /misc
> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
> submounts remaining in /misc
> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
> 3078093712 path /misc stat 3
> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
> exp 3078093712 finished, switching from 2 to 1
> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
> =3D 2 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =3D
> 3078093712 path /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
> submounts remaining in /misc
> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
> 3078093712 path /misc stat 3
> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
> exp 3078093712 finished, switching from 2 to 1
> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
> =3D 2 path /misc
> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
> server '1.2.3.4' failed: timed out (giving up).
> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/te=
stdown
> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
> --------------------------------------------------------------
> =

> 2009/8/12 Ian Kent <ikent@redhat.com>:
>> Carlos Andr=E9 wrote:
>>> Hi Ian,
>>> I'm getting crazy trying put "retry=3D" to work on mount... this option
>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/krb=
5p)
>>> like you can see on my previous emails...
>> Right, my mistake for not looking closely enough at post.
>>
>> Maybe this is related to the same sort of problem we had with mount in
>> the past, before the options parsing went into the kernel, where other
>> services, like portmapper (or rpcbind), were being done with different
>> timeout parameters before the RPC calls for mounting. That's just an
>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>
>> But what version of autofs and kernel did you say you were using?
>>
>>> I appreciate any help.
>>>
>>> Carlos.
>>>
>>>
>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>> Chuck Lever wrote:
>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>> This long timeout is good if workstation need mount a critical
>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>>> down for any reason, and will again hangs if user try access directo=
ry
>>>>>> pointing to a NFS down server...
>>>>> "retry=3D0" means the mount command will fail as soon as the first
>>>>> mount(2) system call fails.  When you set SYN retries to 1, this means
>>>>> after 9 seconds, the connect fails, and that causes the mount(2) syst=
em
>>>>> call to fail.
>>>>>
>>>>> Recent conversations with Ian suggested that a long timeout was desir=
ed
>>>>> for automounter as well as other cases.  Ian, is there something else=
 we
>>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>>> mount points handled via automounter?  How should mount.nfs wait so we
>>>>> don't make other use cases worse?  (Looks like most of the history is
>>>>> intact below).
>>>> Of course we know that autofs is entirely at the mercy of mount(8) (and
>>>> mount.nfs in particular). This has always been a difficult situation f=
or
>>>> the automounter because interactive mount invocations should wait. But=
 I
>>>> believe automount mounts should always time out quickly, but that leads
>>>> to its own set of problems, especially when home directories are conce=
rned.
>>>>
>>>> I think adding "retry=3D0" is the right thing to do myself but I'm not
>>>> certain that will work as we expect. I'll have to do some experimentat=
ion.
>>>>
>>>>> How long do you think is appropriate for the automounter to wait if t=
he
>>>>> server is down, in your case, Carlos?
>>>>>
>>>>>> Am losing something or there have was something weirdo...!?
>>>>>> ------------------------------------------------
>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries  [DEF=
AULT]
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=3Dtcp,retry=3D1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    3m9.000s
>>>>>> user    0m0.002s
>>>>>> sys     0m0.001s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    3m9.000s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=3Dtcp,retry=3D0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    3m9.001s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.003s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    3m9.001s
>>>>>> user    0m0.002s
>>>>>> sys     0m0.001s
>>>>>>
>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 t=
o 1 ]
>>>>>>
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=3Dtcp,retry=3D1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [=
x 6]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    1m3.002s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [=
x 13]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    2m6.000s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> proto=3Dtcp,retry=3D0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    0m9.003s
>>>>>> user    0m0.001s
>>>>>> sys     0m0.002s
>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). [=
x 13]
>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>
>>>>>> real    2m6.001s
>>>>>> user    0m0.001s
>>>>>> sys     0m0.002s
>>>>>> [root@KSTATION ~]#
>>>>>> ------------------------------------------------
>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>
>>>>>> *sigh*
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retries=
 to
>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>> Right.  Normally the RPC client calls the kernel's socket connect
>>>>>>> function,
>>>>>>> which does 6 SYN retries.  That one call usually takes longer than
>>>>>>> the RPC
>>>>>>> client's connect timeout, so it only makes one connect call, and th=
en
>>>>>>> fails.
>>>>>>>
>>>>>>> Reducing the number of SYN retries per connect attempt causes the R=
PC
>>>>>>> client
>>>>>>> to retry the connect call until its connect timeout expires.  Each
>>>>>>> connect
>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>
>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>
>>>>>>>> real    3m9.000s
>>>>>>>> user    0m0.000s
>>>>>>>> sys     0m0.002s
>>>>>>>>
>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>>
>>>>>>>> real    2m6.004s
>>>>>>>> user    0m0.000s
>>>>>>>> sys     0m0.004s
>>>>>>>>
>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>
>>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
>>>>>>>>> 2049...
>>>>>>>>> I tried use "retry=3D1" option on mount without any change... I d=
ont
>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>
>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation where my
>>>>>>>>>>> server
>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 min=
utes
>>>>>>>>>>> and 9 seconds...
>>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>>> give up
>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts with
>>>>>>>>>> exponential retries, or something like that).  For stock CentOS
>>>>>>>>>> 5.3, I
>>>>>>>>>> think
>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>>> kernel
>>>>>>>>>> just
>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding rp=
cbind
>>>>>>>>>> request.
>>>>>>>>>>
>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>>> components
>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>
>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail.=
com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work=
 with
>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i =
get a
>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon process=
, if
>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if s=
erver
>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, there=
 my
>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using
>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp OR
>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error
>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>> I thought he was describing a situation where the server the s=
erver
>>>>>>>>>>>> is completely gone and isn't coming back, and wondering how to=
 make
>>>>>>>>>>>> the
>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>
>>>>>>>>>>>> --b.
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.h=
tml
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>> --
>>>>> Chuck Lever
>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>
>>>>>
>>>>>
>>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

2009/8/13 Ian Kent <ikent@redhat.com>:
> Carlos Andr=E9 wrote:
>> Today (2009-08-12) I'm using:
>> kernel-2.6.18-128.2.1.el5
>> autofs-5.0.1-0.rc2.102.el5_3.1
>
> Thanks,
>
> My mistake, the wait time I was referring to is used for umounts during
> expires and is present in rev rc2.102.
>
> It shouldn't be hard to add this for mount as well.
> Would you like me to put something together?

Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)

>
> Probably would be good to test something out to see if we can make a
> difference with the killing mount after some configured timeout but, if
> we make progress, probably the best way to deal with it is for you to
> log a bug against rhel-5 so I can get it committed to the rhel package.
> The possible issue is that I'm not sure if the RPC subsystem in the
> above rhel kernel will respond well to process death with potential
> outstanding requests. But we'll see.

Ok, on my way :)

Thanks a lot!

>
>>
>>
>> Look my last test:
>> --------------------------------------------------------------
>> [root@KSTATION areas]# time ls testdown
>> ls: testdown: No such file or directory
>>
>> real =A0 =A03m9.025s
>> user =A0 =A00m0.000s
>> sys =A0 =A0 0m0.002s
>>
>>
>>
>>
>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>> mounting root /misc/areas, mountpoint testdown, what
>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D0, ro=
=3D0
>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>> calling mkdir_path /misc/areas/testdown
>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /misc
>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =3D
>> 3078093712 path /misc
>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>> submounts remaining in /misc
>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>> 3078093712 path /misc stat 3
>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>> exp 3078093712 finished, switching from 2 to 1
>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
>> =3D 2 path /misc
>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /misc
>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =3D
>> 3078093712 path /misc
>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>> submounts remaining in /misc
>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>> 3078093712 path /misc stat 3
>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>> exp 3078093712 finished, switching from 2 to 1
>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
>> =3D 2 path /misc
>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>> server '1.2.3.4' failed: timed out (giving up).
>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/t=
estdown
>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /misc
>> --------------------------------------------------------------
>>
>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>> Carlos Andr=E9 wrote:
>>>> Hi Ian,
>>>> I'm getting crazy trying put "retry=3D" to work on mount... this option
>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/kr=
b5p)
>>>> like you can see on my previous emails...
>>> Right, my mistake for not looking closely enough at post.
>>>
>>> Maybe this is related to the same sort of problem we had with mount in
>>> the past, before the options parsing went into the kernel, where other
>>> services, like portmapper (or rpcbind), were being done with different
>>> timeout parameters before the RPC calls for mounting. That's just an
>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>
>>> But what version of autofs and kernel did you say you were using?
>>>
>>>> I appreciate any help.
>>>>
>>>> Carlos.
>>>>
>>>>
>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>> Chuck Lever wrote:
>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>>>> since autofs retry mount directory on-access. This in fact gives me
>>>>>>> alot of headaches, coz user login 'll just hangs if one server goes
>>>>>>> down for any reason, and will again hangs if user try access direct=
ory
>>>>>>> pointing to a NFS down server...
>>>>>> "retry=3D0" means the mount command will fail as soon as the first
>>>>>> mount(2) system call fails. =A0When you set SYN retries to 1, this m=
eans
>>>>>> after 9 seconds, the connect fails, and that causes the mount(2) sys=
tem
>>>>>> call to fail.
>>>>>>
>>>>>> Recent conversations with Ian suggested that a long timeout was desi=
red
>>>>>> for automounter as well as other cases. =A0Ian, is there something e=
lse we
>>>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>>>> mount points handled via automounter? =A0How should mount.nfs wait s=
o we
>>>>>> don't make other use cases worse? =A0(Looks like most of the history=
 is
>>>>>> intact below).
>>>>> Of course we know that autofs is entirely at the mercy of mount(8) (a=
nd
>>>>> mount.nfs in particular). This has always been a difficult situation =
for
>>>>> the automounter because interactive mount invocations should wait. Bu=
t I
>>>>> believe automount mounts should always time out quickly, but that lea=
ds
>>>>> to its own set of problems, especially when home directories are conc=
erned.
>>>>>
>>>>> I think adding "retry=3D0" is the right thing to do myself but I'm not
>>>>> certain that will work as we expect. I'll have to do some experimenta=
tion.
>>>>>
>>>>>> How long do you think is appropriate for the automounter to wait if =
the
>>>>>> server is down, in your case, Carlos?
>>>>>>
>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>> ------------------------------------------------
>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0[=
DEFAULT]
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A03m9.000s
>>>>>>> user =A0 =A00m0.002s
>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A03m9.000s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A03m9.001s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.003s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A03m9.001s
>>>>>>> user =A0 =A00m0.002s
>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>
>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5 =
to 1 ]
>>>>>>>
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 6]
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A01m3.002s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 13]
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A02m6.000s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A00m9.003s
>>>>>>> user =A0 =A00m0.001s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying). =
[x 13]
>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up).
>>>>>>>
>>>>>>> real =A0 =A02m6.001s
>>>>>>> user =A0 =A00m0.001s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>> [root@KSTATION ~]#
>>>>>>> ------------------------------------------------
>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... and
>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>
>>>>>>> *sigh*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retrie=
s to
>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>> Right. =A0Normally the RPC client calls the kernel's socket connect
>>>>>>>> function,
>>>>>>>> which does 6 SYN retries. =A0That one call usually takes longer th=
an
>>>>>>>> the RPC
>>>>>>>> client's connect timeout, so it only makes one connect call, and t=
hen
>>>>>>>> fails.
>>>>>>>>
>>>>>>>> Reducing the number of SYN retries per connect attempt causes the =
RPC
>>>>>>>> client
>>>>>>>> to retry the connect call until its connect timeout expires. =A0Ea=
ch
>>>>>>>> connect
>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>
>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up=
).
>>>>>>>>>
>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>
>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).
>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up=
).
>>>>>>>>>
>>>>>>>>> real =A0 =A02m6.004s
>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>> sys =A0 =A0 0m0.004s
>>>>>>>>>
>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>
>>>>>>>>>> And u're right about expo retries... with tcpdump i've monitored
>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on po=
rt
>>>>>>>>>> 2049...
>>>>>>>>>> I tried use "retry=3D1" option on mount without any change... I =
dont
>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>
>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situation where=
 my
>>>>>>>>>>>> server
>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 mi=
nutes
>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>> The 189 second timeout is likely how long it takes the kernel to
>>>>>>>>>>> give up
>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts wi=
th
>>>>>>>>>>> exponential retries, or something like that). =A0For stock Cent=
OS
>>>>>>>>>>> 5.3, I
>>>>>>>>>>> think
>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- the
>>>>>>>>>>> kernel
>>>>>>>>>>> just
>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding r=
pcbind
>>>>>>>>>>> request.
>>>>>>>>>>>
>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentOS
>>>>>>>>>>> components
>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>
>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmail=
.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to wor=
k with
>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i=
 get a
>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon proces=
s, if
>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if =
server
>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, ther=
e my
>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) usi=
ng
>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp =
OR
>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show =
error
>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>> I thought he was describing a situation where the server the =
server
>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering how t=
o make
>>>>>>>>>>>>> the
>>>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-inf=
o.html
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Lever
>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>> --
>>>>>>>> Chuck Lever
>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>> --
>>>>>> Chuck Lever
>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>
>>>>>>
>>>>>>
>>>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Filled bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=3D517349

Thanks!

2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
> 2009/8/13 Ian Kent <ikent@redhat.com>:
>> Carlos Andr=E9 wrote:
>>> Today (2009-08-12) I'm using:
>>> kernel-2.6.18-128.2.1.el5
>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>
>> Thanks,
>>
>> My mistake, the wait time I was referring to is used for umounts during
>> expires and is present in rev rc2.102.
>>
>> It shouldn't be hard to add this for mount as well.
>> Would you like me to put something together?
>
> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>
>>
>> Probably would be good to test something out to see if we can make a
>> difference with the killing mount after some configured timeout but, if
>> we make progress, probably the best way to deal with it is for you to
>> log a bug against rhel-5 so I can get it committed to the rhel package.
>> The possible issue is that I'm not sure if the RPC subsystem in the
>> above rhel kernel will respond well to process death with potential
>> outstanding requests. But we'll see.
>
> Ok, on my way :)
>
> Thanks a lot!
>
>>
>>>
>>>
>>> Look my last test:
>>> --------------------------------------------------------------
>>> [root@KSTATION areas]# time ls testdown
>>> ls: testdown: No such file or directory
>>>
>>> real =A0 =A03m9.025s
>>> user =A0 =A00m0.000s
>>> sys =A0 =A0 0m0.002s
>>>
>>>
>>>
>>>
>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>> mounting root /misc/areas, mountpoint testdown, what
>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D0, r=
o=3D0
>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>> calling mkdir_path /misc/areas/testdown
>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /mis=
c
>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =3D
>>> 3078093712 path /misc
>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>> submounts remaining in /misc
>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>>> 3078093712 path /misc stat 3
>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>>> exp 3078093712 finished, switching from 2 to 1
>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): state
>>> =3D 2 path /misc
>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /mis=
c
>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =3D
>>> 3078093712 path /misc
>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>> submounts remaining in /misc
>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>>> 3078093712 path /misc stat 3
>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>>> exp 3078093712 finished, switching from 2 to 1
>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): state
>>> =3D 2 path /misc
>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>> server '1.2.3.4' failed: timed out (giving up).
>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/areas/=
testdown
>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /mis=
c
>>> --------------------------------------------------------------
>>>
>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>> Carlos Andr=E9 wrote:
>>>>> Hi Ian,
>>>>> I'm getting crazy trying put "retry=3D" to work on mount... this opti=
on
>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5i/k=
rb5p)
>>>>> like you can see on my previous emails...
>>>> Right, my mistake for not looking closely enough at post.
>>>>
>>>> Maybe this is related to the same sort of problem we had with mount in
>>>> the past, before the options parsing went into the kernel, where other
>>>> services, like portmapper (or rpcbind), were being done with different
>>>> timeout parameters before the RPC calls for mounting. That's just an
>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>
>>>> But what version of autofs and kernel did you say you were using?
>>>>
>>>>> I appreciate any help.
>>>>>
>>>>> Carlos.
>>>>>
>>>>>
>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>> Chuck Lever wrote:
>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>> But in my case, using this loooong timeout doesnt make any sense,
>>>>>>>> since autofs retry mount directory on-access. This in fact gives m=
e
>>>>>>>> alot of headaches, coz user login 'll just hangs if one server goe=
s
>>>>>>>> down for any reason, and will again hangs if user try access direc=
tory
>>>>>>>> pointing to a NFS down server...
>>>>>>> "retry=3D0" means the mount command will fail as soon as the first
>>>>>>> mount(2) system call fails. =A0When you set SYN retries to 1, this =
means
>>>>>>> after 9 seconds, the connect fails, and that causes the mount(2) sy=
stem
>>>>>>> call to fail.
>>>>>>>
>>>>>>> Recent conversations with Ian suggested that a long timeout was des=
ired
>>>>>>> for automounter as well as other cases. =A0Ian, is there something =
else we
>>>>>>> need to consider to determine the correct retry timeout for NFS/TCP
>>>>>>> mount points handled via automounter? =A0How should mount.nfs wait =
so we
>>>>>>> don't make other use cases worse? =A0(Looks like most of the histor=
y is
>>>>>>> intact below).
>>>>>> Of course we know that autofs is entirely at the mercy of mount(8) (=
and
>>>>>> mount.nfs in particular). This has always been a difficult situation=
 for
>>>>>> the automounter because interactive mount invocations should wait. B=
ut I
>>>>>> believe automount mounts should always time out quickly, but that le=
ads
>>>>>> to its own set of problems, especially when home directories are con=
cerned.
>>>>>>
>>>>>> I think adding "retry=3D0" is the right thing to do myself but I'm n=
ot
>>>>>> certain that will work as we expect. I'll have to do some experiment=
ation.
>>>>>>
>>>>>>> How long do you think is appropriate for the automounter to wait if=
 the
>>>>>>> server is down, in your case, Carlos?
>>>>>>>
>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>> ------------------------------------------------
>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =A0=
[DEFAULT]
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A03m9.000s
>>>>>>>> user =A0 =A00m0.002s
>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A03m9.000s
>>>>>>>> user =A0 =A00m0.000s
>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A03m9.001s
>>>>>>>> user =A0 =A00m0.000s
>>>>>>>> sys =A0 =A0 0m0.003s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A03m9.001s
>>>>>>>> user =A0 =A00m0.002s
>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>>
>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries [ 5=
 to 1 ]
>>>>>>>>
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).=
 [x 6]
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A01m3.002s
>>>>>>>> user =A0 =A00m0.000s
>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).=
 [x 13]
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A02m6.000s
>>>>>>>> user =A0 =A00m0.000s
>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A00m9.003s
>>>>>>>> user =A0 =A00m0.001s
>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying).=
 [x 13]
>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving up)=
.
>>>>>>>>
>>>>>>>> real =A0 =A02m6.001s
>>>>>>>> user =A0 =A00m0.001s
>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>> [root@KSTATION ~]#
>>>>>>>> ------------------------------------------------
>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1... a=
nd
>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>
>>>>>>>> *sigh*
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_retri=
es to
>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>> Right. =A0Normally the RPC client calls the kernel's socket conne=
ct
>>>>>>>>> function,
>>>>>>>>> which does 6 SYN retries. =A0That one call usually takes longer t=
han
>>>>>>>>> the RPC
>>>>>>>>> client's connect timeout, so it only makes one connect call, and =
then
>>>>>>>>> fails.
>>>>>>>>>
>>>>>>>>> Reducing the number of SYN retries per connect attempt causes the=
 RPC
>>>>>>>>> client
>>>>>>>>> to retry the connect call until its connect timeout expires. =A0E=
ach
>>>>>>>>> connect
>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>
>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving u=
p).
>>>>>>>>>>
>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>
>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retrying=
).
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving u=
p).
>>>>>>>>>>
>>>>>>>>>> real =A0 =A02m6.004s
>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>> sys =A0 =A0 0m0.004s
>>>>>>>>>>
>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>
>>>>>>>>>>> And u're right about expo retries... with tcpdump i've monitore=
d
>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on p=
ort
>>>>>>>>>>> 2049...
>>>>>>>>>>> I tried use "retry=3D1" option on mount without any change... I=
 dont
>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>
>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situation wher=
e my
>>>>>>>>>>>>> server
>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than 3 m=
inutes
>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>> The 189 second timeout is likely how long it takes the kernel =
to
>>>>>>>>>>>> give up
>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempts w=
ith
>>>>>>>>>>>> exponential retries, or something like that). =A0For stock Cen=
tOS
>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>> think
>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -- t=
he
>>>>>>>>>>>> kernel
>>>>>>>>>>>> just
>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preceding =
rpcbind
>>>>>>>>>>>> request.
>>>>>>>>>>>>
>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related CentO=
S
>>>>>>>>>>>> components
>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote=
:
>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@gmai=
l.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to wo=
rk with
>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes down =
i get a
>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon proce=
ss, if
>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout (if=
 server
>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, the=
re my
>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) us=
ing
>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3Dtcp=
 OR
>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until show=
 error
>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>> I thought he was describing a situation where the server the=
 server
>>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering how =
to make
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordomo-in=
fo.html
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Chuck Lever
>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>> --
>>>>>>> Chuck Lever
>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>
>>
>
_______________________________________________
NFSv4 mailing list
NFSv4@linux-nfs.org
http://linux-nfs.org/cgi-bin/mailman/listinfo/nfsv4

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

T24gVGh1LCAyMDA5LTA4LTEzIGF0IDEyOjE4IC0wMzAwLCBDYXJsb3MgQW5kcsOpIHdyb3RlOgo+
IEZpbGxlZCBidWcgcmVwb3J0Ogo+IGh0dHBzOi8vYnVnemlsbGEucmVkaGF0LmNvbS9zaG93X2J1
Zy5jZ2k/aWQ9NTE3MzQ5CgpIaSBDYXJsb3MsCgpJIGhhdmUgYSBwYXRjaGVkIHNvdXJjZSBycG0g
dG8gYWRkIGEgbW91bnQgd2FpdCBwYXJhbWV0ZXIgdG8gYXV0b2ZzCmxvY2F0ZWQgYXQ6Cmh0dHA6
Ly9wZW9wbGUucmVkaGF0LmNvbS9+aWtlbnQvYXV0b2ZzLTUuMC4xLTAucmMyLjEzMS5iejUxNzM0
OS4xCgpDb3VsZCB5b3UgYnVpbGQgaXQgYW5kIHNlZSBpZiBpdCB3b3Jrcy4KSSBoYXZlbid0IHRl
c3RlZCBpdCBhdCBhbGwgYnV0IGl0IGlzIGZhaXJseSBzdHJhaWdodCBmb3J3YXJkLgpJdCBpcyBz
dGlsbCB1bmNsZWFyIGlmIHRoaXMgaXMgdGhlIHJpZ2h0IHdheSB0byBkbyB0aGlzIGFuZCB3aGF0
IHRoZQpjb25zZXF1ZW5jZXMgYXJlIGluIHNlbmRpbmcgYSB0ZXJtIHNpZ25hbCB0byBtb3VudC4g
VGhpcyBtb3VudCByZXF1ZXN0CndpbGwgbGlrZWx5IGJlIGZvbGxvd2VkIGJ5IG90aGVyIHJlcXVl
c3RzIGZvciB0aGUgc2FtZSBtb3VudCBjYXVzaW5nIGFuCmFjY3VtdWxhdGlvbiBvZiBtb3VudCg4
KSBwcm9jZXNzZXMgd2FpdGluZyBmb3IgUlBDIHRpbWVvdXRzIGJlZm9yZSB0aGV5CmNhbiBhbnN3
ZXIgdGhlIFRFUk0gc2lnbmFsLgoKQW55d2F5LCBmb3IgaW5mb3JtYXRpb24gdGhlIHBhdGNoIGlu
Y2x1ZGVkIGluIHRoZSBzb3VyY2UgcnBtIGFib3ZlIGlzOgoKYXV0b2ZzLTUuMC40IC0gYWRkIG1v
dW50IHdhaXQgcGFyYW1ldGVyCgpGcm9tOiBJYW4gS2VudCA8cmF2ZW5AdGhlbWF3Lm5ldD4KCk9m
dGVuIGRlbGF5cyB3aGVuIHRyeWluZyB0byBtb3VudCBmcm9tIGEgc2VydmVyIHRoYXQgaXMgbm90
IHJlcG9uZGluZwpmb3Igc29tZSByZWFzb24gYXJlIHVuZGVzaXJhYmxlLiBUbyB0cnkgYW5kIHBy
ZXZlbnQgdGhlc2UgZGVsYXlzIHdlCnByb3ZpZGUgYSBjb25maWd1cmF0aW9uIHNldHRpbmcgdG8g
bGltaXQgdGhlIHRpbWUgdGhhdCB3ZSB3YWl0IGZvcgpvdXIgc3Bhd25lZCBtb3VudCg4KSBwcm9j
ZXNzIHRvIGNvbXBsZXRlIGJlZm9yZSBzZW5kaW5nIGl0IGEgU0lHVEVSTQpzaWduYWwuIFRoaXMg
cGF0Y2ggYWRkcyBhIGNvbmZpZ3VyYXRpb24gcGFyYW1ldGVyIHRvIGFsbG93IHVzIHRvCnJlcXVl
c3Qgd2UgbGltaXQgdGhlIHRpbWUgd2Ugd2FpdCBmb3IgbW91bnQoOCkgdG8gY29tcGxldGUgYmVm
b3JlCnNlbmQgaXQgYSBURVJNIHNpZ25hbC4KLS0tCgogZGFlbW9uL3NwYXduLmMgICAgICAgICAg
ICAgICAgIHwgICAgMyArKy0KIGluY2x1ZGUvZGVmYXVsdHMuaCAgICAgICAgICAgICB8ICAgIDIg
KysKIGxpYi9kZWZhdWx0cy5jICAgICAgICAgICAgICAgICB8ICAgMTMgKysrKysrKysrKysrKwog
bWFuL2F1dG8ubWFzdGVyLjUuaW4gICAgICAgICAgIHwgICAgNyArKysrKysrCiByZWRoYXQvYXV0
b2ZzLnN5c2NvbmZpZy5pbiAgICAgfCAgICA5ICsrKysrKysrKwogc2FtcGxlcy9hdXRvZnMuY29u
Zi5kZWZhdWx0LmluIHwgICAgOSArKysrKysrKysKIDYgZmlsZXMgY2hhbmdlZCwgNDIgaW5zZXJ0
aW9ucygrKSwgMSBkZWxldGlvbigtKQoKCi0tLSBhdXRvZnMtNS4wLjEub3JpZy9kYWVtb24vc3Bh
d24uYworKysgYXV0b2ZzLTUuMC4xL2RhZW1vbi9zcGF3bi5jCkBAIC0zMTIsNiArMzEyLDcgQEAg
aW50IHNwYXduX21vdW50KHVuc2lnbmVkIGxvZ29wdCwgLi4uKQogCXVuc2lnbmVkIGludCBvcHRp
b25zOwogCXVuc2lnbmVkIGludCByZXRyaWVzID0gTVRBQl9MT0NLX1JFVFJJRVM7CiAJaW50IHVw
ZGF0ZV9tdGFiID0gMSwgcmV0LCBwcmludGVkID0gMDsKKwl1bnNpZ25lZCBpbnQgd2FpdCA9IGRl
ZmF1bHRzX2dldF9tb3VudF93YWl0KCk7CiAJY2hhciBidWZbUEFUSF9NQVhdOwogCiAJLyogSWYg
d2UgdXNlIG1vdW50IGxvY2tpbmcgd2UgY2FuJ3QgdmFsaWRhdGUgdGhlIGxvY2F0aW9uICovCkBA
IC0zNTMsNyArMzU0LDcgQEAgaW50IHNwYXduX21vdW50KHVuc2lnbmVkIGxvZ29wdCwgLi4uKQog
CXZhX2VuZChhcmcpOwogCiAJd2hpbGUgKHJldHJpZXMtLSkgewotCQlyZXQgPSBkb19zcGF3bihs
b2dvcHQsIC0xLCBvcHRpb25zLCBwcm9nLCAoY29uc3QgY2hhciAqKikgYXJndik7CisJCXJldCA9
IGRvX3NwYXduKGxvZ29wdCwgd2FpdCwgb3B0aW9ucywgcHJvZywgKGNvbnN0IGNoYXIgKiopIGFy
Z3YpOwogCQlpZiAocmV0ICYgTVRBQl9OT1RVUERBVEVEKSB7CiAJCQlzdHJ1Y3QgdGltZXNwZWMg
dG0gPSB7MywgMH07CiAKLS0tIGF1dG9mcy01LjAuMS5vcmlnL2luY2x1ZGUvZGVmYXVsdHMuaAor
KysgYXV0b2ZzLTUuMC4xL2luY2x1ZGUvZGVmYXVsdHMuaApAQCAtMjQsNiArMjQsNyBAQAogCiAj
ZGVmaW5lIERFRkFVTFRfVElNRU9VVAkJCTYwMAogI2RlZmluZSBERUZBVUxUX05FR0FUSVZFX1RJ
TUVPVVQJNjAKKyNkZWZpbmUgREVGQVVMVF9NT1VOVF9XQUlUCQktMQogI2RlZmluZSBERUZBVUxU
X1VNT1VOVF9XQUlUCQkxMgogI2RlZmluZSBERUZBVUxUX0JST1dTRV9NT0RFCQkxCiAjZGVmaW5l
IERFRkFVTFRfTE9HR0lORwkJCTAKQEAgLTYyLDYgKzYzLDcgQEAgc3RydWN0IGxkYXBfc2NoZW1h
ICpkZWZhdWx0c19nZXRfc2NoZW1hKAogc3RydWN0IGxkYXBfc2VhcmNoZG4gKmRlZmF1bHRzX2dl
dF9zZWFyY2hkbnModm9pZCk7CiB2b2lkIGRlZmF1bHRzX2ZyZWVfc2VhcmNoZG5zKHN0cnVjdCBs
ZGFwX3NlYXJjaGRuICopOwogdW5zaWduZWQgaW50IGRlZmF1bHRzX2dldF9hcHBlbmRfb3B0aW9u
cyh2b2lkKTsKK3Vuc2lnbmVkIGludCBkZWZhdWx0c19nZXRfbW91bnRfd2FpdCh2b2lkKTsKIHVu
c2lnbmVkIGludCBkZWZhdWx0c19nZXRfdW1vdW50X3dhaXQodm9pZCk7CiBjb25zdCBjaGFyICpk
ZWZhdWx0c19nZXRfYXV0aF9jb25mX2ZpbGUodm9pZCk7CiB1bnNpZ25lZCBpbnQgZGVmYXVsdHNf
Z2V0X21hcF9oYXNoX3RhYmxlX3NpemUodm9pZCk7Ci0tLSBhdXRvZnMtNS4wLjEub3JpZy9saWIv
ZGVmYXVsdHMuYworKysgYXV0b2ZzLTUuMC4xL2xpYi9kZWZhdWx0cy5jCkBAIC00NSw2ICs0NSw3
IEBACiAjZGVmaW5lIEVOVl9OQU1FX1ZBTFVFX0FUVFIJCSJWQUxVRV9BVFRSSUJVVEUiCiAKICNk
ZWZpbmUgRU5WX0FQUEVORF9PUFRJT05TCQkiQVBQRU5EX09QVElPTlMiCisjZGVmaW5lIEVOVl9N
T1VOVF9XQUlUCQkJIk1PVU5UX1dBSVQiCiAjZGVmaW5lIEVOVl9VTU9VTlRfV0FJVAkJCSJVTU9V
TlRfV0FJVCIKICNkZWZpbmUgRU5WX0FVVEhfQ09ORl9GSUxFCQkiQVVUSF9DT05GX0ZJTEUiCiAK
QEAgLTMyMyw2ICszMjQsNyBAQCB1bnNpZ25lZCBpbnQgZGVmYXVsdHNfcmVhZF9jb25maWcodW5z
aWduCiAJCSAgICBjaGVja19zZXRfY29uZmlnX3ZhbHVlKGtleSwgRU5WX05BTUVfRU5UUllfQVRU
UiwgdmFsdWUsIHRvX3N5c2xvZykgfHwKIAkJICAgIGNoZWNrX3NldF9jb25maWdfdmFsdWUoa2V5
LCBFTlZfTkFNRV9WQUxVRV9BVFRSLCB2YWx1ZSwgdG9fc3lzbG9nKSB8fAogCQkgICAgY2hlY2tf
c2V0X2NvbmZpZ192YWx1ZShrZXksIEVOVl9BUFBFTkRfT1BUSU9OUywgdmFsdWUsIHRvX3N5c2xv
ZykgfHwKKwkJICAgIGNoZWNrX3NldF9jb25maWdfdmFsdWUoa2V5LCBFTlZfTU9VTlRfV0FJVCwg
dmFsdWUsIHRvX3N5c2xvZykgfHwKIAkJICAgIGNoZWNrX3NldF9jb25maWdfdmFsdWUoa2V5LCBF
TlZfVU1PVU5UX1dBSVQsIHZhbHVlLCB0b19zeXNsb2cpIHx8CiAJCSAgICBjaGVja19zZXRfY29u
ZmlnX3ZhbHVlKGtleSwgRU5WX0FVVEhfQ09ORl9GSUxFLCB2YWx1ZSwgdG9fc3lzbG9nKSB8fAog
CQkgICAgY2hlY2tfc2V0X2NvbmZpZ192YWx1ZShrZXksIEVOVl9NQVBfSEFTSF9UQUJMRV9TSVpF
LCB2YWx1ZSwgdG9fc3lzbG9nKSkKQEAgLTY1Miw2ICs2NTQsMTcgQEAgdW5zaWduZWQgaW50IGRl
ZmF1bHRzX2dldF9hcHBlbmRfb3B0aW9ucwogCXJldHVybiByZXM7CiB9CiAKK3Vuc2lnbmVkIGlu
dCBkZWZhdWx0c19nZXRfbW91bnRfd2FpdCh2b2lkKQoreworCWxvbmcgd2FpdDsKKworCXdhaXQg
PSBnZXRfZW52X251bWJlcihFTlZfTU9VTlRfV0FJVCk7CisJaWYgKHdhaXQgPCAwKQorCQl3YWl0
ID0gREVGQVVMVF9NT1VOVF9XQUlUOworCisJcmV0dXJuICh1bnNpZ25lZCBpbnQpIHdhaXQ7Cit9
CisKIHVuc2lnbmVkIGludCBkZWZhdWx0c19nZXRfdW1vdW50X3dhaXQodm9pZCkKIHsKIAlsb25n
IHdhaXQ7Ci0tLSBhdXRvZnMtNS4wLjEub3JpZy9tYW4vYXV0by5tYXN0ZXIuNS5pbgorKysgYXV0
b2ZzLTUuMC4xL21hbi9hdXRvLm1hc3Rlci41LmluCkBAIC0xNzUsNiArMTc1LDEzIEBAIFNldCB0
aGUgZGVmYXVsdCB0aW1lb3V0IGZvciBjYWNoaW5nIGZhaWwKIDYwKS4gSWYgdGhlIGVxdWl2YWxl
bnQgY29tbWFuZCBsaW5lIG9wdGlvbiBpcyBnaXZlbiBpdCB3aWxsIG92ZXJyaWRlIHRoaXMKIHNl
dHRpbmcuCiAuVFAKKy5CIE1PVU5UX1dBSVQKK1NldCB0aGUgZGVmYXVsdCB0aW1lIHRvIHdhaXQg
Zm9yIGEgcmVzcG9uc2UgZnJvbSBhIHNwYXduZWQgbW91bnQoOCkKK2JlZm9yZSBzZW5kaW5nIGl0
IGEgU0lHVEVSTS4gTm90ZSB0aGF0IHdlIHN0aWxsIG5lZWQgdG8gd2FpdCBmb3IgdGhlCitSUEMg
bGF5ZXIgdG8gdGltZW91dCBiZWZvcmUgdGhlIHN1Yi1wcm9jZXNzIGV4aXRzIHNvIHRoaXMgaXNu
J3QgaWRlYWwKK2J1dCBpdCBpcyB0aGUgYmVzdCB3ZSBjYW4gZG8uIFRoZSBkZWZhdWx0IGlzIHRv
IHdhaXQgdW50aWwgbW91bnQoOCkKK3JldHVybnMgd2l0aG91dCBpbnRlcnZlbnRpb24uCisuVFAK
IC5CIFVNT1VOVF9XQUlUCiBTZXQgdGhlIGRlZmF1bHQgdGltZSB0byB3YWl0IGZvciBhIHJlc3Bv
bnNlIGZyb20gYSBzcGF3bmVkIHVtb3VudCg4KQogYmVmb3JlIHNlbmRpbmcgaXQgYSBTSUdURVJN
LiBOb3RlIHRoYXQgd2Ugc3RpbGwgbmVlZCB0byB3YWl0IGZvciB0aGUKLS0tIGF1dG9mcy01LjAu
MS5vcmlnL3JlZGhhdC9hdXRvZnMuc3lzY29uZmlnLmluCisrKyBhdXRvZnMtNS4wLjEvcmVkaGF0
L2F1dG9mcy5zeXNjb25maWcuaW4KQEAgLTE0LDYgKzE0LDE1IEBAIFRJTUVPVVQ9MzAwCiAjCiAj
TkVHQVRJVkVfVElNRU9VVD02MAogIworIyBNT1VOVF9XQUlUIC0gdGltZSB0byB3YWl0IGZvciBh
IHJlc3BvbnNlIGZyb20gdW1vdW50KDgpLgorIyAJICAgICAgIFNldHRpbmcgdGhpcyB0aW1lb3V0
IGNhbiBjYXVzZSBwcm9ibGVtcyB3aGVuCisjIAkgICAgICAgbW91bnQgd291bGQgb3RoZXJ3aXNl
IHdhaXQgZm9yIGEgc2VydmVyIHRoYXQKKyMgCSAgICAgICBpcyB0ZW1wb3JhcmlseSB1bmF2YWls
YWJsZSwgc3VjaCBhcyB3aGVuIGl0J3MKKyMgCSAgICAgICByZXN0YXJ0aW5nLiBUaGUgZGVmYWls
dCBvZiB3YWl0aW5nIGZvciBtb3VudCg4KQorIyAJICAgICAgIHVzdWFsbHkgcmVzdWx0cyBpbiBh
IHdhaXQgb2YgYXJvdW5kIDMgbWludXRlcy4KKyMKKyNNT1VOVF9XQUlUPS0xCisjCiAjIFVNT1VO
VF9XQUlUIC0gdGltZSB0byB3YWl0IGZvciBhIHJlc3BvbnNlIGZyb20gdW1vdW50KDgpLgogIwog
I1VNT1VOVF9XQUlUPTEyCi0tLSBhdXRvZnMtNS4wLjEub3JpZy9zYW1wbGVzL2F1dG9mcy5jb25m
LmRlZmF1bHQuaW4KKysrIGF1dG9mcy01LjAuMS9zYW1wbGVzL2F1dG9mcy5jb25mLmRlZmF1bHQu
aW4KQEAgLTE0LDYgKzE0LDE1IEBAIFRJTUVPVVQ9MzAwCiAjCiAjTkVHQVRJVkVfVElNRU9VVD02
MAogIworIyBNT1VOVF9XQUlUIC0gdGltZSB0byB3YWl0IGZvciBhIHJlc3BvbnNlIGZyb20gdW1v
dW50KDgpLgorIyAJICAgICAgIFNldHRpbmcgdGhpcyB0aW1lb3V0IGNhbiBjYXVzZSBwcm9ibGVt
cyB3aGVuCisjIAkgICAgICAgbW91bnQgd291bGQgb3RoZXJ3aXNlIHdhaXQgZm9yIGEgc2VydmVy
IHRoYXQKKyMgCSAgICAgICBpcyB0ZW1wb3JhcmlseSB1bmF2YWlsYWJsZSwgc3VjaCBhcyB3aGVu
IGl0J3MKKyMgCSAgICAgICByZXN0YXJ0aW5nLiBUaGUgZGVmYWlsdCBvZiB3YWl0aW5nIGZvciBt
b3VudCg4KQorIyAJICAgICAgIHVzdWFsbHkgcmVzdWx0cyBpbiBhIHdhaXQgb2YgYXJvdW5kIDMg
bWludXRlcy4KKyMKKyNNT1VOVF9XQUlUPS0xCisjCiAjIFVNT1VOVF9XQUlUIC0gdGltZSB0byB3
YWl0IGZvciBhIHJlc3BvbnNlIGZyb20gdW1vdW50KDgpLgogIwogI1VNT1VOVF9XQUlUPTEyCgoK
PiAKPiBUaGFua3MhCj4gCj4gMjAwOS84LzEzIENhcmxvcyBBbmRyw6kgPGNhbmRyZWNuQGdtYWls
LmNvbT46Cj4gPiAyMDA5LzgvMTMgSWFuIEtlbnQgPGlrZW50QHJlZGhhdC5jb20+Ogo+ID4+IENh
cmxvcyBBbmRyw6kgd3JvdGU6Cj4gPj4+IFRvZGF5ICgyMDA5LTA4LTEyKSBJJ20gdXNpbmc6Cj4g
Pj4+IGtlcm5lbC0yLjYuMTgtMTI4LjIuMS5lbDUKPiA+Pj4gYXV0b2ZzLTUuMC4xLTAucmMyLjEw
Mi5lbDVfMy4xCj4gPj4KPiA+PiBUaGFua3MsCj4gPj4KPiA+PiBNeSBtaXN0YWtlLCB0aGUgd2Fp
dCB0aW1lIEkgd2FzIHJlZmVycmluZyB0byBpcyB1c2VkIGZvciB1bW91bnRzIGR1cmluZwo+ID4+
IGV4cGlyZXMgYW5kIGlzIHByZXNlbnQgaW4gcmV2IHJjMi4xMDIuCj4gPj4KPiA+PiBJdCBzaG91
bGRuJ3QgYmUgaGFyZCB0byBhZGQgdGhpcyBmb3IgbW91bnQgYXMgd2VsbC4KPiA+PiBXb3VsZCB5
b3UgbGlrZSBtZSB0byBwdXQgc29tZXRoaW5nIHRvZ2V0aGVyPwo+ID4KPiA+IFN1cmUhIHRoYXQg
J2xsIGhlbHAgbWUgYSBsb3QgKGFuZCBmb3Igc3VyZSBhbm90aGVyIHBwbCkgOikgVGhhbmtzIDop
Cj4gPgo+ID4+Cj4gPj4gUHJvYmFibHkgd291bGQgYmUgZ29vZCB0byB0ZXN0IHNvbWV0aGluZyBv
dXQgdG8gc2VlIGlmIHdlIGNhbiBtYWtlIGEKPiA+PiBkaWZmZXJlbmNlIHdpdGggdGhlIGtpbGxp
bmcgbW91bnQgYWZ0ZXIgc29tZSBjb25maWd1cmVkIHRpbWVvdXQgYnV0LCBpZgo+ID4+IHdlIG1h
a2UgcHJvZ3Jlc3MsIHByb2JhYmx5IHRoZSBiZXN0IHdheSB0byBkZWFsIHdpdGggaXQgaXMgZm9y
IHlvdSB0bwo+ID4+IGxvZyBhIGJ1ZyBhZ2FpbnN0IHJoZWwtNSBzbyBJIGNhbiBnZXQgaXQgY29t
bWl0dGVkIHRvIHRoZSByaGVsIHBhY2thZ2UuCj4gPj4gVGhlIHBvc3NpYmxlIGlzc3VlIGlzIHRo
YXQgSSdtIG5vdCBzdXJlIGlmIHRoZSBSUEMgc3Vic3lzdGVtIGluIHRoZQo+ID4+IGFib3ZlIHJo
ZWwga2VybmVsIHdpbGwgcmVzcG9uZCB3ZWxsIHRvIHByb2Nlc3MgZGVhdGggd2l0aCBwb3RlbnRp
YWwKPiA+PiBvdXRzdGFuZGluZyByZXF1ZXN0cy4gQnV0IHdlJ2xsIHNlZS4KPiA+Cj4gPiBPaywg
b24gbXkgd2F5IDopCj4gPgo+ID4gVGhhbmtzIGEgbG90IQo+ID4KPiA+Pgo+ID4+Pgo+ID4+Pgo+
ID4+PiBMb29rIG15IGxhc3QgdGVzdDoKPiA+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KPiA+Pj4gW3Jvb3RAS1NUQVRJT04g
YXJlYXNdIyB0aW1lIGxzIHRlc3Rkb3duCj4gPj4+IGxzOiB0ZXN0ZG93bjogTm8gc3VjaCBmaWxl
IG9yIGRpcmVjdG9yeQo+ID4+Pgo+ID4+PiByZWFsICAgIDNtOS4wMjVzCj4gPj4+IHVzZXIgICAg
MG0wLjAwMHMKPiA+Pj4gc3lzICAgICAwbTAuMDAycwo+ID4+Pgo+ID4+Pgo+ID4+Pgo+ID4+Pgo+
ID4+PiBBdWcgMTIgMTI6NTc6MDcgS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTogc3VuX21vdW50
OiBwYXJzZShzdW4pOgo+ID4+PiBtb3VudGluZyByb290IC9taXNjL2FyZWFzLCBtb3VudHBvaW50
IHRlc3Rkb3duLCB3aGF0Cj4gPj4+IDEuMi4zLjQ6L2FyZWFzL3Rlc3Rkb3duLCBmc3R5cGUgbmZz
NCwgb3B0aW9ucwo+ID4+PiBhY2wsc2VjPWtyYjVwLHByb3RvPXRjcCxyZXRyeT0wCj4gPj4+IEF1
ZyAxMiAxMjo1NzowNyBLU1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBkb19tb3VudDoKPiA+Pj4g
MS4yLjMuNDovYXJlYXMvdGVzdGRvd24gL21pc2MvYXJlYXMvdGVzdGRvd24gdHlwZSBuZnM0IG9w
dGlvbnMKPiA+Pj4gYWNsLHNlYz1rcmI1cCxwcm90bz10Y3AscmV0cnk9MCB1c2luZyBtb2R1bGUg
bmZzNAo+ID4+PiBBdWcgMTIgMTI6NTc6MDcgS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTogbW91
bnRfbW91bnQ6IG1vdW50KG5mcyk6Cj4gPj4+IHJvb3Q9L21pc2MvYXJlYXMgbmFtZT10ZXN0ZG93
biB3aGF0PTEuMi4zLjQ6L2FyZWFzL3Rlc3Rkb3duLAo+ID4+PiBmc3R5cGU9bmZzNCwgb3B0aW9u
cz1hY2wsc2VjPWtyYjVwLHByb3RvPXRjcCxyZXRyeT0wCj4gPj4+IEF1ZyAxMiAxMjo1NzowNyBL
U1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBtb3VudF9tb3VudDogbW91bnQobmZzKToKPiA+Pj4g
bmZzIG9wdGlvbnM9ImFjbCxzZWM9a3JiNXAscHJvdG89dGNwLHJldHJ5PTAiLCBub3N5bWxpbms9
MCwgcm89MAo+ID4+PiBBdWcgMTIgMTI6NTc6MDcgS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTog
bW91bnRfbW91bnQ6IG1vdW50KG5mcyk6Cj4gPj4+IGNhbGxpbmcgbWtkaXJfcGF0aCAvbWlzYy9h
cmVhcy90ZXN0ZG93bgo+ID4+PiBBdWcgMTIgMTI6NTc6MDcgS1NUQVRJT04gYXV0b21vdW50WzE1
NDcxXTogbW91bnRfbW91bnQ6IG1vdW50KG5mcyk6Cj4gPj4+IGNhbGxpbmcgbW91bnQgLXQgbmZz
NCAtcyAtbyBhY2wsc2VjPWtyYjVwLHByb3RvPXRjcCxyZXRyeT0wCj4gPj4+IDEuMi4zLjQ6L2Fy
ZWFzL3Rlc3Rkb3duIC9taXNjL2FyZWFzL3Rlc3Rkb3duCj4gPj4+IEF1ZyAxMiAxMjo1ODoxMiBL
U1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBzdF9leHBpcmU6IHN0YXRlIDEgcGF0aCAvbWlzYwo+
ID4+PiBBdWcgMTIgMTI6NTg6MTIgS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTogZXhwaXJlX3By
b2M6IGV4cF9wcm9jID0KPiA+Pj4gMzA3ODA5MzcxMiBwYXRoIC9taXNjCj4gPj4+IEF1ZyAxMiAx
Mjo1ODoxMyBLU1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBleHBpcmVfcHJvY19pbmRpcmVjdDog
Mgo+ID4+PiBzdWJtb3VudHMgcmVtYWluaW5nIGluIC9taXNjCj4gPj4+IEF1ZyAxMiAxMjo1ODox
MyBLU1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBleHBpcmVfY2xlYW51cDogZ290IHRoaWQKPiA+
Pj4gMzA3ODA5MzcxMiBwYXRoIC9taXNjIHN0YXQgMwo+ID4+PiBBdWcgMTIgMTI6NTg6MTMgS1NU
QVRJT04gYXV0b21vdW50WzE1NDcxXTogZXhwaXJlX2NsZWFudXA6IHNpZ2NobGQ6Cj4gPj4+IGV4
cCAzMDc4MDkzNzEyIGZpbmlzaGVkLCBzd2l0Y2hpbmcgZnJvbSAyIHRvIDEKPiA+Pj4gQXVnIDEy
IDEyOjU4OjEzIEtTVEFUSU9OIGF1dG9tb3VudFsxNTQ3MV06IHN0X3JlYWR5OiBzdF9yZWFkeSgp
OiBzdGF0ZQo+ID4+PiA9IDIgcGF0aCAvbWlzYwo+ID4+PiBBdWcgMTIgMTI6NTk6MjggS1NUQVRJ
T04gYXV0b21vdW50WzE1NDcxXTogc3RfZXhwaXJlOiBzdGF0ZSAxIHBhdGggL21pc2MKPiA+Pj4g
QXVnIDEyIDEyOjU5OjI4IEtTVEFUSU9OIGF1dG9tb3VudFsxNTQ3MV06IGV4cGlyZV9wcm9jOiBl
eHBfcHJvYyA9Cj4gPj4+IDMwNzgwOTM3MTIgcGF0aCAvbWlzYwo+ID4+PiBBdWcgMTIgMTI6NTk6
MjggS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTogZXhwaXJlX3Byb2NfaW5kaXJlY3Q6IDIKPiA+
Pj4gc3VibW91bnRzIHJlbWFpbmluZyBpbiAvbWlzYwo+ID4+PiBBdWcgMTIgMTI6NTk6MjggS1NU
QVRJT04gYXV0b21vdW50WzE1NDcxXTogZXhwaXJlX2NsZWFudXA6IGdvdCB0aGlkCj4gPj4+IDMw
NzgwOTM3MTIgcGF0aCAvbWlzYyBzdGF0IDMKPiA+Pj4gQXVnIDEyIDEyOjU5OjI4IEtTVEFUSU9O
IGF1dG9tb3VudFsxNTQ3MV06IGV4cGlyZV9jbGVhbnVwOiBzaWdjaGxkOgo+ID4+PiBleHAgMzA3
ODA5MzcxMiBmaW5pc2hlZCwgc3dpdGNoaW5nIGZyb20gMiB0byAxCj4gPj4+IEF1ZyAxMiAxMjo1
OToyOCBLU1RBVElPTiBhdXRvbW91bnRbMTU0NzFdOiBzdF9yZWFkeTogc3RfcmVhZHkoKTogc3Rh
dGUKPiA+Pj4gPSAyIHBhdGggL21pc2MKPiA+Pj4gQXVnIDEyIDEzOjAwOjE2IEtTVEFUSU9OIGF1
dG9tb3VudFsxNTQ3MV06ID4+IG1vdW50OiBtb3VudCB0byBORlMKPiA+Pj4gc2VydmVyICcxLjIu
My40JyBmYWlsZWQ6IHRpbWVkIG91dCAoZ2l2aW5nIHVwKS4KPiA+Pj4gQXVnIDEyIDEzOjAwOjE2
IEtTVEFUSU9OIGF1dG9tb3VudFsxNTQ3MV06IG1vdW50KG5mcyk6IG5mczogbW91bnQKPiA+Pj4g
ZmFpbHVyZSAxLjIuMy40Oi9hcmVhcy90ZXN0ZG93biBvbiAvbWlzYy9hcmVhcy90ZXN0ZG93bgo+
ID4+PiBBdWcgMTIgMTM6MDA6MTYgS1NUQVRJT04gYXV0b21vdW50WzE1NDcxXTogc2VuZF9mYWls
OiB0b2tlbiA9IDE3Cj4gPj4+IEF1ZyAxMiAxMzowMDoxNiBLU1RBVElPTiBhdXRvbW91bnRbMTU0
NzFdOiBmYWlsZWQgdG8gbW91bnQgL21pc2MvYXJlYXMvdGVzdGRvd24KPiA+Pj4gQXVnIDEyIDEz
OjAwOjQzIEtTVEFUSU9OIGF1dG9tb3VudFsxNTQ3MV06IHN0X2V4cGlyZTogc3RhdGUgMSBwYXRo
IC9taXNjCj4gPj4+IC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tCj4gPj4+Cj4gPj4+IDIwMDkvOC8xMiBJYW4gS2VudCA8aWtlbnRA
cmVkaGF0LmNvbT46Cj4gPj4+PiBDYXJsb3MgQW5kcsOpIHdyb3RlOgo+ID4+Pj4+IEhpIElhbiwK
PiA+Pj4+PiBJJ20gZ2V0dGluZyBjcmF6eSB0cnlpbmcgcHV0ICJyZXRyeT0iIHRvIHdvcmsgb24g
bW91bnQuLi4gdGhpcyBvcHRpb24KPiA+Pj4+PiBqdXN0IERPTlQgV09SSyBpZiB1c2UgcHJvdG89
dGNwIGFuZC9PUiBrZXJiZXJvcyAoc2VjPWtyYjUva3JiNWkva3JiNXApCj4gPj4+Pj4gbGlrZSB5
b3UgY2FuIHNlZSBvbiBteSBwcmV2aW91cyBlbWFpbHMuLi4KPiA+Pj4+IFJpZ2h0LCBteSBtaXN0
YWtlIGZvciBub3QgbG9va2luZyBjbG9zZWx5IGVub3VnaCBhdCBwb3N0Lgo+ID4+Pj4KPiA+Pj4+
IE1heWJlIHRoaXMgaXMgcmVsYXRlZCB0byB0aGUgc2FtZSBzb3J0IG9mIHByb2JsZW0gd2UgaGFk
IHdpdGggbW91bnQgaW4KPiA+Pj4+IHRoZSBwYXN0LCBiZWZvcmUgdGhlIG9wdGlvbnMgcGFyc2lu
ZyB3ZW50IGludG8gdGhlIGtlcm5lbCwgd2hlcmUgb3RoZXIKPiA+Pj4+IHNlcnZpY2VzLCBsaWtl
IHBvcnRtYXBwZXIgKG9yIHJwY2JpbmQpLCB3ZXJlIGJlaW5nIGRvbmUgd2l0aCBkaWZmZXJlbnQK
PiA+Pj4+IHRpbWVvdXQgcGFyYW1ldGVycyBiZWZvcmUgdGhlIFJQQyBjYWxscyBmb3IgbW91bnRp
bmcuIFRoYXQncyBqdXN0IGFuCj4gPj4+PiBleGFtcGxlIGFzIE5GU3Y0IHNob3VsZG4ndCBiZSBz
ZW5zaXRpdmUgdG8gcG9ydG1hcHBlciBhbnl3YXkuCj4gPj4+Pgo+ID4+Pj4gQnV0IHdoYXQgdmVy
c2lvbiBvZiBhdXRvZnMgYW5kIGtlcm5lbCBkaWQgeW91IHNheSB5b3Ugd2VyZSB1c2luZz8KPiA+
Pj4+Cj4gPj4+Pj4gSSBhcHByZWNpYXRlIGFueSBoZWxwLgo+ID4+Pj4+Cj4gPj4+Pj4gQ2FybG9z
Lgo+ID4+Pj4+Cj4gPj4+Pj4KPiA+Pj4+PiAyMDA5LzgvMTIgSWFuIEtlbnQgPGlrZW50QHJlZGhh
dC5jb20+Ogo+ID4+Pj4+PiBDaHVjayBMZXZlciB3cm90ZToKPiA+Pj4+Pj4+IE9uIEF1ZyAxMSwg
MjAwOSwgYXQgODo0MSBBTSwgQ2FybG9zIEFuZHLDqSB3cm90ZToKPiA+Pj4+Pj4+PiBUaGlzIGxv
bmcgdGltZW91dCBpcyBnb29kIGlmIHdvcmtzdGF0aW9uIG5lZWQgbW91bnQgYSBjcml0aWNhbAo+
ID4+Pj4+Pj4+IGRpcmVjdG9yeSB1c2luZyAvZXRjL2ZzdGFiIG9uIGJvb3QgKGZvciBleGFtcGxl
KS4uCj4gPj4+Pj4+Pj4gQnV0IGluIG15IGNhc2UsIHVzaW5nIHRoaXMgbG9vb29uZyB0aW1lb3V0
IGRvZXNudCBtYWtlIGFueSBzZW5zZSwKPiA+Pj4+Pj4+PiBzaW5jZSBhdXRvZnMgcmV0cnkgbW91
bnQgZGlyZWN0b3J5IG9uLWFjY2Vzcy4gVGhpcyBpbiBmYWN0IGdpdmVzIG1lCj4gPj4+Pj4+Pj4g
YWxvdCBvZiBoZWFkYWNoZXMsIGNveiB1c2VyIGxvZ2luICdsbCBqdXN0IGhhbmdzIGlmIG9uZSBz
ZXJ2ZXIgZ29lcwo+ID4+Pj4+Pj4+IGRvd24gZm9yIGFueSByZWFzb24sIGFuZCB3aWxsIGFnYWlu
IGhhbmdzIGlmIHVzZXIgdHJ5IGFjY2VzcyBkaXJlY3RvcnkKPiA+Pj4+Pj4+PiBwb2ludGluZyB0
byBhIE5GUyBkb3duIHNlcnZlci4uLgo+ID4+Pj4+Pj4gInJldHJ5PTAiIG1lYW5zIHRoZSBtb3Vu
dCBjb21tYW5kIHdpbGwgZmFpbCBhcyBzb29uIGFzIHRoZSBmaXJzdAo+ID4+Pj4+Pj4gbW91bnQo
Mikgc3lzdGVtIGNhbGwgZmFpbHMuICBXaGVuIHlvdSBzZXQgU1lOIHJldHJpZXMgdG8gMSwgdGhp
cyBtZWFucwo+ID4+Pj4+Pj4gYWZ0ZXIgOSBzZWNvbmRzLCB0aGUgY29ubmVjdCBmYWlscywgYW5k
IHRoYXQgY2F1c2VzIHRoZSBtb3VudCgyKSBzeXN0ZW0KPiA+Pj4+Pj4+IGNhbGwgdG8gZmFpbC4K
PiA+Pj4+Pj4+Cj4gPj4+Pj4+PiBSZWNlbnQgY29udmVyc2F0aW9ucyB3aXRoIElhbiBzdWdnZXN0
ZWQgdGhhdCBhIGxvbmcgdGltZW91dCB3YXMgZGVzaXJlZAo+ID4+Pj4+Pj4gZm9yIGF1dG9tb3Vu
dGVyIGFzIHdlbGwgYXMgb3RoZXIgY2FzZXMuICBJYW4sIGlzIHRoZXJlIHNvbWV0aGluZyBlbHNl
IHdlCj4gPj4+Pj4+PiBuZWVkIHRvIGNvbnNpZGVyIHRvIGRldGVybWluZSB0aGUgY29ycmVjdCBy
ZXRyeSB0aW1lb3V0IGZvciBORlMvVENQCj4gPj4+Pj4+PiBtb3VudCBwb2ludHMgaGFuZGxlZCB2
aWEgYXV0b21vdW50ZXI/ICBIb3cgc2hvdWxkIG1vdW50Lm5mcyB3YWl0IHNvIHdlCj4gPj4+Pj4+
PiBkb24ndCBtYWtlIG90aGVyIHVzZSBjYXNlcyB3b3JzZT8gIChMb29rcyBsaWtlIG1vc3Qgb2Yg
dGhlIGhpc3RvcnkgaXMKPiA+Pj4+Pj4+IGludGFjdCBiZWxvdykuCj4gPj4+Pj4+IE9mIGNvdXJz
ZSB3ZSBrbm93IHRoYXQgYXV0b2ZzIGlzIGVudGlyZWx5IGF0IHRoZSBtZXJjeSBvZiBtb3VudCg4
KSAoYW5kCj4gPj4+Pj4+IG1vdW50Lm5mcyBpbiBwYXJ0aWN1bGFyKS4gVGhpcyBoYXMgYWx3YXlz
IGJlZW4gYSBkaWZmaWN1bHQgc2l0dWF0aW9uIGZvcgo+ID4+Pj4+PiB0aGUgYXV0b21vdW50ZXIg
YmVjYXVzZSBpbnRlcmFjdGl2ZSBtb3VudCBpbnZvY2F0aW9ucyBzaG91bGQgd2FpdC4gQnV0IEkK
PiA+Pj4+Pj4gYmVsaWV2ZSBhdXRvbW91bnQgbW91bnRzIHNob3VsZCBhbHdheXMgdGltZSBvdXQg
cXVpY2tseSwgYnV0IHRoYXQgbGVhZHMKPiA+Pj4+Pj4gdG8gaXRzIG93biBzZXQgb2YgcHJvYmxl
bXMsIGVzcGVjaWFsbHkgd2hlbiBob21lIGRpcmVjdG9yaWVzIGFyZSBjb25jZXJuZWQuCj4gPj4+
Pj4+Cj4gPj4+Pj4+IEkgdGhpbmsgYWRkaW5nICJyZXRyeT0wIiBpcyB0aGUgcmlnaHQgdGhpbmcg
dG8gZG8gbXlzZWxmIGJ1dCBJJ20gbm90Cj4gPj4+Pj4+IGNlcnRhaW4gdGhhdCB3aWxsIHdvcmsg
YXMgd2UgZXhwZWN0LiBJJ2xsIGhhdmUgdG8gZG8gc29tZSBleHBlcmltZW50YXRpb24uCj4gPj4+
Pj4+Cj4gPj4+Pj4+PiBIb3cgbG9uZyBkbyB5b3UgdGhpbmsgaXMgYXBwcm9wcmlhdGUgZm9yIHRo
ZSBhdXRvbW91bnRlciB0byB3YWl0IGlmIHRoZQo+ID4+Pj4+Pj4gc2VydmVyIGlzIGRvd24sIGlu
IHlvdXIgY2FzZSwgQ2FybG9zPwo+ID4+Pj4+Pj4KPiA+Pj4+Pj4+PiBBbSBsb3Npbmcgc29tZXRo
aW5nIG9yIHRoZXJlIGhhdmUgd2FzIHNvbWV0aGluZyB3ZWlyZG8uLi4hPwo+ID4+Pj4+Pj4+IC0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo+ID4+Pj4+Pj4+
IFtyb290QEtTVEFUSU9OIH5dIyBlY2hvIDUgPiAvcHJvYy9zeXMvbmV0L2lwdjQvdGNwX3N5bl9y
ZXRyaWVzICBbREVGQVVMVF0KPiA+Pj4+Pj4+PiBbcm9vdEBLU1RBVElPTiB+XSMgdGltZSBtb3Vu
dCAxLjIuMy40Oi9ibGFibGEgL3RtcC8gLXQgbmZzNCAtbwo+ID4+Pj4+Pj4+IHByb3RvPXRjcCxy
ZXRyeT0xCj4gPj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZh
aWxlZDogdGltZWQgb3V0IChnaXZpbmcgdXApLgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gcmVhbCAg
ICAzbTkuMDAwcwo+ID4+Pj4+Pj4+IHVzZXIgICAgMG0wLjAwMnMKPiA+Pj4+Pj4+PiBzeXMgICAg
IDBtMC4wMDFzCj4gPj4+Pj4+Pj4gW3Jvb3RAS1NUQVRJT04gfl0jIHRpbWUgbW91bnQgMS4yLjMu
NDovYmxhYmxhIC90bXAvIC10IG5mczQgLW8KPiA+Pj4+Pj4+PiBzZWM9a3JiNXAscHJvdG89dGNw
LHJldHJ5PTEKPiA+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcg
ZmFpbGVkOiB0aW1lZCBvdXQgKGdpdmluZyB1cCkuCj4gPj4+Pj4+Pj4KPiA+Pj4+Pj4+PiByZWFs
ICAgIDNtOS4wMDBzCj4gPj4+Pj4+Pj4gdXNlciAgICAwbTAuMDAwcwo+ID4+Pj4+Pj4+IHN5cyAg
ICAgMG0wLjAwMnMKPiA+Pj4+Pj4+PiBbcm9vdEBLU1RBVElPTiB+XSMgdGltZSBtb3VudCAxLjIu
My40Oi9ibGFibGEgL3RtcC8gLXQgbmZzNCAtbwo+ID4+Pj4+Pj4+IHByb3RvPXRjcCxyZXRyeT0w
Cj4gPj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZhaWxlZDog
dGltZWQgb3V0IChnaXZpbmcgdXApLgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gcmVhbCAgICAzbTku
MDAxcwo+ID4+Pj4+Pj4+IHVzZXIgICAgMG0wLjAwMHMKPiA+Pj4+Pj4+PiBzeXMgICAgIDBtMC4w
MDNzCj4gPj4+Pj4+Pj4gW3Jvb3RAS1NUQVRJT04gfl0jIHRpbWUgbW91bnQgMS4yLjMuNDovYmxh
YmxhIC90bXAvIC10IG5mczQgLW8KPiA+Pj4+Pj4+PiBzZWM9a3JiNXAscHJvdG89dGNwLHJldHJ5
PTAKPiA+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVk
OiB0aW1lZCBvdXQgKGdpdmluZyB1cCkuCj4gPj4+Pj4+Pj4KPiA+Pj4+Pj4+PiByZWFsICAgIDNt
OS4wMDFzCj4gPj4+Pj4+Pj4gdXNlciAgICAwbTAuMDAycwo+ID4+Pj4+Pj4+IHN5cyAgICAgMG0w
LjAwMXMKPiA+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+IFtyb290QEtTVEFUSU9OIH5dIyBlY2hvIDEgPiAv
cHJvYy9zeXMvbmV0L2lwdjQvdGNwX3N5bl9yZXRyaWVzIFsgNSB0byAxIF0KPiA+Pj4+Pj4+Pgo+
ID4+Pj4+Pj4+IFtyb290QEtTVEFUSU9OIH5dIyB0aW1lIG1vdW50IDEuMi4zLjQ6L2JsYWJsYSAv
dG1wLyAtdCBuZnM0IC1vCj4gPj4+Pj4+Pj4gcHJvdG89dGNwLHJldHJ5PTEKPiA+Pj4+Pj4+PiBt
b3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKHJl
dHJ5aW5nKS4gW3ggNl0KPiA+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4y
LjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKGdpdmluZyB1cCkuCj4gPj4+Pj4+Pj4KPiA+Pj4+Pj4+
PiByZWFsICAgIDFtMy4wMDJzCj4gPj4+Pj4+Pj4gdXNlciAgICAwbTAuMDAwcwo+ID4+Pj4+Pj4+
IHN5cyAgICAgMG0wLjAwMnMKPiA+Pj4+Pj4+PiBbcm9vdEBLU1RBVElPTiB+XSMgdGltZSBtb3Vu
dCAxLjIuMy40Oi9ibGFibGEgL3RtcC8gLXQgbmZzNCAtbwo+ID4+Pj4+Pj4+IHNlYz1rcmI1cCxw
cm90bz10Y3AscmV0cnk9MQo+ID4+Pj4+Pj4+IG1vdW50OiBtb3VudCB0byBORlMgc2VydmVyICcx
LjIuMy40JyBmYWlsZWQ6IHRpbWVkIG91dCAocmV0cnlpbmcpLiBbeCAxM10KPiA+Pj4+Pj4+PiBt
b3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKGdp
dmluZyB1cCkuCj4gPj4+Pj4+Pj4KPiA+Pj4+Pj4+PiByZWFsICAgIDJtNi4wMDBzCj4gPj4+Pj4+
Pj4gdXNlciAgICAwbTAuMDAwcwo+ID4+Pj4+Pj4+IHN5cyAgICAgMG0wLjAwMnMKPiA+Pj4+Pj4+
PiBbcm9vdEBLU1RBVElPTiB+XSMgdGltZSBtb3VudCAxLjIuMy40Oi9ibGFibGEgL3RtcC8gLXQg
bmZzNCAtbwo+ID4+Pj4+Pj4+IHByb3RvPXRjcCxyZXRyeT0wCj4gPj4+Pj4+Pj4gbW91bnQ6IG1v
dW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZhaWxlZDogdGltZWQgb3V0IChnaXZpbmcgdXAp
Lgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gcmVhbCAgICAwbTkuMDAzcwo+ID4+Pj4+Pj4+IHVzZXIg
ICAgMG0wLjAwMXMKPiA+Pj4+Pj4+PiBzeXMgICAgIDBtMC4wMDJzCj4gPj4+Pj4+Pj4gW3Jvb3RA
S1NUQVRJT04gfl0jIHRpbWUgbW91bnQgMS4yLjMuNDovYmxhYmxhIC90bXAvIC10IG5mczQgLW8K
PiA+Pj4+Pj4+PiBzZWM9a3JiNXAscHJvdG89dGNwLHJldHJ5PTAKPiA+Pj4+Pj4+PiBtb3VudDog
bW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKHJldHJ5aW5n
KS4gW3ggMTNdCj4gPj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQn
IGZhaWxlZDogdGltZWQgb3V0IChnaXZpbmcgdXApLgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gcmVh
bCAgICAybTYuMDAxcwo+ID4+Pj4+Pj4+IHVzZXIgICAgMG0wLjAwMXMKPiA+Pj4+Pj4+PiBzeXMg
ICAgIDBtMC4wMDJzCj4gPj4+Pj4+Pj4gW3Jvb3RAS1NUQVRJT04gfl0jCj4gPj4+Pj4+Pj4gLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCj4gPj4+Pj4+Pj4g
bWF4IHRpbWVvdXQgZ29lcyB0byAybTZzIGNoYW5naW5nIHRjcF9zeW5fcmV0cmllcyBmcm9tIDUg
dG8gMS4uLiBhbmQKPiA+Pj4+Pj4+PiB1c2luZyByZXRyeT0wIHdpdGhvdXQga2VyYmVyb3MgSSBn
b3Qgb25seSA5cy4uLgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gKnNpZ2gqCj4gPj4+Pj4+Pj4KPiA+
Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4gMjAwOS84LzEwIENodWNrIExldmVyIDxjaHVj
ay5sZXZlckBvcmFjbGUuY29tPjoKPiA+Pj4+Pj4+Pj4gT24gQXVnIDEwLCAyMDA5LCBhdCA0OjA1
IFBNLCBDYXJsb3MgQW5kcsOpIHdyb3RlOgo+ID4+Pj4+Pj4+Pj4gU29tZXRoaW5nIGZ1bm55OiBV
c2luZyBkZWZhdWx0IHRjcF9zeW5fcmV0cmllcyAoNSkgaSBnb3QKPiA+Pj4+Pj4+Pj4+ICIzLDYs
MTIsMjQsNDgsOTYiIHNlY3MgaW50ZXJ2YWwuLi4gYnV0IGlmIGkgY2hhbmdlIHRjcF9zeW5fcmV0
cmllcyB0bwo+ID4+Pj4+Pj4+Pj4gMSBpIGdvdCAiMyw2LDMsNiwzLDYuLi4iIHNlY3MgaW50ZXJ2
YWwuLi4KPiA+Pj4+Pj4+Pj4gUmlnaHQuICBOb3JtYWxseSB0aGUgUlBDIGNsaWVudCBjYWxscyB0
aGUga2VybmVsJ3Mgc29ja2V0IGNvbm5lY3QKPiA+Pj4+Pj4+Pj4gZnVuY3Rpb24sCj4gPj4+Pj4+
Pj4+IHdoaWNoIGRvZXMgNiBTWU4gcmV0cmllcy4gIFRoYXQgb25lIGNhbGwgdXN1YWxseSB0YWtl
cyBsb25nZXIgdGhhbgo+ID4+Pj4+Pj4+PiB0aGUgUlBDCj4gPj4+Pj4+Pj4+IGNsaWVudCdzIGNv
bm5lY3QgdGltZW91dCwgc28gaXQgb25seSBtYWtlcyBvbmUgY29ubmVjdCBjYWxsLCBhbmQgdGhl
bgo+ID4+Pj4+Pj4+PiBmYWlscy4KPiA+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4gUmVkdWNpbmcgdGhl
IG51bWJlciBvZiBTWU4gcmV0cmllcyBwZXIgY29ubmVjdCBhdHRlbXB0IGNhdXNlcyB0aGUgUlBD
Cj4gPj4+Pj4+Pj4+IGNsaWVudAo+ID4+Pj4+Pj4+PiB0byByZXRyeSB0aGUgY29ubmVjdCBjYWxs
IHVudGlsIGl0cyBjb25uZWN0IHRpbWVvdXQgZXhwaXJlcy4gIEVhY2gKPiA+Pj4+Pj4+Pj4gY29u
bmVjdAo+ID4+Pj4+Pj4+PiBjYWxsIHJlc2V0cyB0aGUgU1lOIHRpbWVvdXQgdG8gMyBzZWNvbmRz
Lgo+ID4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pj4gW3Jvb3RAS1NFUlZFUiBtbnRdIyB0aW1lIG1vdW50
IDEuMi4zLjQ6L2JsYWJsYSB0bXAvIC10IG5mczQgLW8KPiA+Pj4+Pj4+Pj4+IHNlYz1rcmI1cCxw
cm90bz10Y3AKPiA+Pj4+Pj4+Pj4+IG1vdW50OiBtb3VudCB0byBORlMgc2VydmVyICcxLjIuMy40
JyBmYWlsZWQ6IHRpbWVkIG91dCAoZ2l2aW5nIHVwKS4KPiA+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+
PiByZWFsICAgIDNtOS4wMDBzCj4gPj4+Pj4+Pj4+PiB1c2VyICAgIDBtMC4wMDBzCj4gPj4+Pj4+
Pj4+PiBzeXMgICAgIDBtMC4wMDJzCj4gPj4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pj4gW3Jvb3RAS1NF
UlZFUiAvXSMgZWNobyAxID4gL3Byb2Mvc3lzL25ldC9pcHY0L3RjcF9zeW5fcmV0cmllcwo+ID4+
Pj4+Pj4+Pj4gW3Jvb3RAS1NFUlZFUiBtbnRdIyB0aW1lIG1vdW50IDEuMi4zLjQ6L2JsYWJsYSB0
bXAvIC10IG5mczQgLW8KPiA+Pj4+Pj4+Pj4+IHNlYz1rcmI1cCxwcm90bz10Y3AgICgicmV0cnk9
MSIgPSBubyBjaGFuZ2UpCj4gPj4+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAn
MS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKHJldHJ5aW5nKS4KPiA+Pj4+Pj4+Pj4+IG1vdW50
OiBtb3VudCB0byBORlMgc2VydmVyICcxLjIuMy40JyBmYWlsZWQ6IHRpbWVkIG91dCAocmV0cnlp
bmcpLgo+ID4+Pj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZh
aWxlZDogdGltZWQgb3V0IChyZXRyeWluZykuCj4gPj4+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8g
TkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKHJldHJ5aW5nKS4KPiA+Pj4+
Pj4+Pj4+IG1vdW50OiBtb3VudCB0byBORlMgc2VydmVyICcxLjIuMy40JyBmYWlsZWQ6IHRpbWVk
IG91dCAocmV0cnlpbmcpLgo+ID4+Pj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIg
JzEuMi4zLjQnIGZhaWxlZDogdGltZWQgb3V0IChyZXRyeWluZykuCj4gPj4+Pj4+Pj4+PiBtb3Vu
dDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1lZCBvdXQgKHJldHJ5
aW5nKS4KPiA+Pj4+Pj4+Pj4+IG1vdW50OiBtb3VudCB0byBORlMgc2VydmVyICcxLjIuMy40JyBm
YWlsZWQ6IHRpbWVkIG91dCAocmV0cnlpbmcpLgo+ID4+Pj4+Pj4+Pj4gbW91bnQ6IG1vdW50IHRv
IE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZhaWxlZDogdGltZWQgb3V0IChyZXRyeWluZykuCj4gPj4+
Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcgZmFpbGVkOiB0aW1l
ZCBvdXQgKHJldHJ5aW5nKS4KPiA+Pj4+Pj4+Pj4+IG1vdW50OiBtb3VudCB0byBORlMgc2VydmVy
ICcxLjIuMy40JyBmYWlsZWQ6IHRpbWVkIG91dCAocmV0cnlpbmcpLgo+ID4+Pj4+Pj4+Pj4gbW91
bnQ6IG1vdW50IHRvIE5GUyBzZXJ2ZXIgJzEuMi4zLjQnIGZhaWxlZDogdGltZWQgb3V0IChyZXRy
eWluZykuCj4gPj4+Pj4+Pj4+PiBtb3VudDogbW91bnQgdG8gTkZTIHNlcnZlciAnMS4yLjMuNCcg
ZmFpbGVkOiB0aW1lZCBvdXQgKHJldHJ5aW5nKS4KPiA+Pj4+Pj4+Pj4+IG1vdW50OiBtb3VudCB0
byBORlMgc2VydmVyICcxLjIuMy40JyBmYWlsZWQ6IHRpbWVkIG91dCAoZ2l2aW5nIHVwKS4KPiA+
Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+PiByZWFsICAgIDJtNi4wMDRzCj4gPj4+Pj4+Pj4+PiB1c2Vy
ICAgIDBtMC4wMDBzCj4gPj4+Pj4+Pj4+PiBzeXMgICAgIDBtMC4wMDRzCj4gPj4+Pj4+Pj4+Pgo+
ID4+Pj4+Pj4+Pj4gKDMsNiwzLDYuLi4gc2VjcyBpbnRlcnZhbCkKPiA+Pj4+Pj4+Pj4+Cj4gPj4+
Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+PiAyMDA5LzgvMTAg
Q2FybG9zIEFuZHLDqSA8Y2FuZHJlY25AZ21haWwuY29tPjoKPiA+Pj4+Pj4+Pj4+PiBObywgaSdt
IGp1c3QgdXNpbmcgcGFja2FnZXMgZnJvbSBDZW50T1MgcmVwby4uLgo+ID4+Pj4+Pj4+Pj4+Cj4g
Pj4+Pj4+Pj4+Pj4gQW5kIHUncmUgcmlnaHQgYWJvdXQgZXhwbyByZXRyaWVzLi4uIHdpdGggdGNw
ZHVtcCBpJ3ZlIG1vbml0b3JlZAo+ID4+Pj4+Pj4+Pj4+IHRyYWZmaWMgYW5kIGkgZ290IFNZTiBy
ZXRyaWVzIGluIDMsIDYsIDEyLCAyNCwgNDgsIDk2IHNlY3Mgb24gcG9ydAo+ID4+Pj4+Pj4+Pj4+
IDIwNDkuLi4KPiA+Pj4+Pj4+Pj4+PiBJIHRyaWVkIHVzZSAicmV0cnk9MSIgb3B0aW9uIG9uIG1v
dW50IHdpdGhvdXQgYW55IGNoYW5nZS4uLiBJIGRvbnQKPiA+Pj4+Pj4+Pj4+PiB3YW50IGNoYW5n
ZSBzb3VyY2Ugb3IgdGNwIHRpbWVycy4uLiBqdXN0IE5GU3Y0IGNsaWVudC4KPiA+Pj4+Pj4+Pj4+
Pgo+ID4+Pj4+Pj4+Pj4+IDIwMDkvOC8xMCBDaHVjayBMZXZlciA8Y2h1Y2subGV2ZXJAb3JhY2xl
LmNvbT46Cj4gPj4+Pj4+Pj4+Pj4+IE9uIEF1ZyAxMCwgMjAwOSwgYXQgMjoyOSBQTSwgQ2FybG9z
IEFuZHLDqSB3cm90ZToKPiA+Pj4+Pj4+Pj4+Pj4+IEJydWNlLCBuby4uLiB5b3UncmUgcmlnaHQu
ICBJJ20gZGVzY3JpYmluZyBhIHNpdHVhdGlvbiB3aGVyZSBteQo+ID4+Pj4+Pj4+Pj4+Pj4gc2Vy
dmVyCj4gPj4+Pj4+Pj4+Pj4+PiBkaWVkLi4uIGkgbmVlZCBtb3VudCBmYWlsIGZhc3RlciAoMTAg
b3IgMTUgc2VjcyBtYXgpIHRoYW4gMyBtaW51dGVzCj4gPj4+Pj4+Pj4+Pj4+PiBhbmQgOSBzZWNv
bmRzLi4uCj4gPj4+Pj4+Pj4+Pj4+IFRoZSAxODkgc2Vjb25kIHRpbWVvdXQgaXMgbGlrZWx5IGhv
dyBsb25nIGl0IHRha2VzIHRoZSBrZXJuZWwgdG8KPiA+Pj4+Pj4+Pj4+Pj4gZ2l2ZSB1cAo+ID4+
Pj4+Pj4+Pj4+PiB0cnlpbmcgdG8gY29ubmVjdCBhIFRDUCBzb2NrZXQgdG8gdGhlIHNlcnZlciAo
NiBTWU4gYXR0ZW1wdHMgd2l0aAo+ID4+Pj4+Pj4+Pj4+PiBleHBvbmVudGlhbCByZXRyaWVzLCBv
ciBzb21ldGhpbmcgbGlrZSB0aGF0KS4gIEZvciBzdG9jayBDZW50T1MKPiA+Pj4+Pj4+Pj4+Pj4g
NS4zLCBJCj4gPj4+Pj4+Pj4+Pj4+IHRoaW5rCj4gPj4+Pj4+Pj4+Pj4+IHVzZXIgc3BhY2UgZG9l
cyBvbmx5IGEgRE5TIGxvb2t1cCBmb3Igbm9ybWFsIE5GU3Y0IG1vdW50cyAtLSB0aGUKPiA+Pj4+
Pj4+Pj4+Pj4ga2VybmVsCj4gPj4+Pj4+Pj4+Pj4+IGp1c3QKPiA+Pj4+Pj4+Pj4+Pj4gdHJpZXMg
dG8gY29ubmVjdCBhIFRDUCBzb2NrZXQgdG8gcG9ydCAyMDQ5LCB3aXRoIG5vIHByZWNlZGluZyBy
cGNiaW5kCj4gPj4+Pj4+Pj4+Pj4+IHJlcXVlc3QuCj4gPj4+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+
Pj4+IENhcmxvcywgbGV0IHVzIGtub3cgaWYgeW91IGhhdmUgcmVwbGFjZWQgYW55IE5GUy1yZWxh
dGVkIENlbnRPUwo+ID4+Pj4+Pj4+Pj4+PiBjb21wb25lbnRzCj4gPj4+Pj4+Pj4+Pj4+IChrZXJu
ZWwsIG5mcy11dGlscykgd2l0aCBzb21ldGhpbmcgeW91J3ZlIGJ1aWx0IHlvdXJzZWxmLgo+ID4+
Pj4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pj4+Pj4gMjAwOS84LzcgSi4gQnJ1Y2UgRmllbGRzIDxiZmll
bGRzQGZpZWxkc2VzLm9yZz46Cj4gPj4+Pj4+Pj4+Pj4+Pj4gT24gRnJpLCBBdWcgMDcsIDIwMDkg
YXQgMDk6NDI6MThBTSArMDMwMCwgQmVubnkgSGFsZXZ5IHdyb3RlOgo+ID4+Pj4+Pj4+Pj4+Pj4+
PiBPbiBBdWcuIDA3LCAyMDA5LCAzOjE4ICswMzAwLCBDYXJsb3MgQW5kcsOpIDxjYW5kcmVjbkBn
bWFpbC5jb20+Cj4gPj4+Pj4+Pj4+Pj4+Pj4+IHdyb3RlOgo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4gQW55
b25lID8KPiA+Pj4+Pj4+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+Pj4+Pj4+PiAyMDA5LzcvMjkgQ2Fy
bG9zIEFuZHLDqSA8Y2FuZHJlY25AZ21haWwuY29tPjoKPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBQUEws
IEkgbmVlZCBwdXQgYSBDZW50T1MgNS4zICh1cGRhdGVkKSBORlN2NCBzZXJ2ZXIgdG8gd29yayB3
aXRoCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4gS2VyYmVyb3MKPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBhbmQg
QXV0b0ZTLCBidXQgaSBnb3QgYSBwcm9ibGVtOiBJZiBORlMgc2VydmVyIGdvZXMgZG93biBpIGdl
dCBhCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4gTE9PT09PT09ORwo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IG1v
dW50IHRpbWVvdXQgb24gQ2VudE9TIDUuMyAodXBkYXRlZCkgTkZTdjQgY2xpZW50Li4uCj4gPj4+
Pj4+Pj4+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBTaW5jZSBpIG5lZWQgbW91bnQgc29t
ZSAoMyB0byA2KSBkaXJzIGF0IHVzZXIgbG9nb24gcHJvY2VzcywgaWYKPiA+Pj4+Pj4+Pj4+Pj4+
Pj4+PiBtb3VudAo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IGhhbmdzLAo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+
IHVzZXIgbG9nb24gaGFuZ3MuIFRoZW4gaSB3YW50IGNvbmZpZ3VyZSBpdCB0byB0aW1lb3V0IChp
ZiBzZXJ2ZXIKPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBkb3duKQo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IGFm
dGVyCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4gMTAtMTUgc2VjcyAoTUFYKSBvbiBlYWNoIG1vdW50IGF0
dGVtcHQuCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBJIGFscmVhZHkg
bWFrZSBhIGxhYiBhbmQgdHJpZWQgYSBMT1Qgb2YgY29tYmluYXRpb25zLCB0aGVyZSBteQo+ID4+
Pj4+Pj4+Pj4+Pj4+Pj4+IGZpbmRpbmdzCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4gKHNlcnZlciBET1dO
IElQOiAxNzIuMTYuMC4xMCAvIGNsaWVudCBJUDogMTcyLjE2LjEuMTApIHVzaW5nCj4gPj4+Pj4+
Pj4+Pj4+Pj4+Pj4gYmFzaWMKPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBjb21tYW5kCj4gPj4+Pj4+Pj4+
Pj4+Pj4+Pj4gKHRpbWUgbW91bnQgMTcyLjE2LjAuMTA6L3JlbW90ZWRpciAvbG9jYWxkaXIvIC10
IG5mczQgLW8KPiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBzZWM9a3JiNSxwcm90bz08dGNwL3VkcD4pIGZy
b20gTkZTIGNsaWVudDoKPiA+Pj4+Pj4+Pj4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IC0g
T25jZSBpIHRyeSBhY2Nlc3MgbW91bnQgcG9pbnQgdXNpbmcgQXV0b0ZTIChwcm90bz10Y3AgT1IK
PiA+Pj4+Pj4+Pj4+Pj4+Pj4+PiBwcm90bz11ZHApCj4gPj4+Pj4+Pj4+Pj4+Pj4+Pj4gaXQKPiA+
Pj4+Pj4+Pj4+Pj4+Pj4+PiBoYW5ncyBmb3IgMTg5IHNlY3MgKDNtOXM6IHJlYWwgIDNtOS4wMDFz
KSAgdW50aWwgc2hvdyBlcnJvcgo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IChtb3VudDoKPiA+Pj4+Pj4+
Pj4+Pj4+Pj4+PiBtb3VudCB0bwo+ID4+Pj4+Pj4+Pj4+Pj4+Pj4+IE5GUyBzZXJ2ZXIgJzE3Mi4x
Ni4wLjEwJyBmYWlsZWQ6IHRpbWVkIG91dCAoZ2l2aW5nIHVwKSkKPiA+Pj4+Pj4+Pj4+Pj4+Pj4g
U291bmRzIGxpa2UgeW91J3JlIGhpdHRpbmcgdGhlIHNlcnZlcidzIGdyYWNlIHBlcmlvZC4KPiA+
Pj4+Pj4+Pj4+Pj4+PiBJIHRob3VnaHQgaGUgd2FzIGRlc2NyaWJpbmcgYSBzaXR1YXRpb24gd2hl
cmUgdGhlIHNlcnZlciB0aGUgc2VydmVyCj4gPj4+Pj4+Pj4+Pj4+Pj4gaXMgY29tcGxldGVseSBn
b25lIGFuZCBpc24ndCBjb21pbmcgYmFjaywgYW5kIHdvbmRlcmluZyBob3cgdG8gbWFrZQo+ID4+
Pj4+Pj4+Pj4+Pj4+IHRoZQo+ID4+Pj4+Pj4+Pj4+Pj4+IG1vdW50IGZhaWwgZmFzdGVyLiAgQnV0
IEkgbWF5IGJlIG1pc3VuZGVyc3RhbmRpbmcuCj4gPj4+Pj4+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4+
Pj4+PiAtLWIuCj4gPj4+Pj4+Pj4+Pj4+Pj4KPiA+Pj4+Pj4+Pj4+Pj4+IC0tCj4gPj4+Pj4+Pj4+
Pj4+PiBUbyB1bnN1YnNjcmliZSBmcm9tIHRoaXMgbGlzdDogc2VuZCB0aGUgbGluZSAidW5zdWJz
Y3JpYmUKPiA+Pj4+Pj4+Pj4+Pj4+IGxpbnV4LW5mcyIgaW4KPiA+Pj4+Pj4+Pj4+Pj4+IHRoZSBi
b2R5IG9mIGEgbWVzc2FnZSB0byBtYWpvcmRvbW9Admdlci5rZXJuZWwub3JnCj4gPj4+Pj4+Pj4+
Pj4+PiBNb3JlIG1ham9yZG9tbyBpbmZvIGF0ICBodHRwOi8vdmdlci5rZXJuZWwub3JnL21ham9y
ZG9tby1pbmZvLmh0bWwKPiA+Pj4+Pj4+Pj4+Pj4gLS0KPiA+Pj4+Pj4+Pj4+Pj4gQ2h1Y2sgTGV2
ZXIKPiA+Pj4+Pj4+Pj4+Pj4gY2h1Y2tbZG90XWxldmVyW2F0XW9yYWNsZVtkb3RdY29tCj4gPj4+
Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+Pj4+Cj4gPj4+Pj4+Pj4+Pj4+Cj4g
Pj4+Pj4+Pj4+IC0tCj4gPj4+Pj4+Pj4+IENodWNrIExldmVyCj4gPj4+Pj4+Pj4+IGNodWNrW2Rv
dF1sZXZlclthdF1vcmFjbGVbZG90XWNvbQo+ID4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4+Pgo+ID4+Pj4+
Pj4+Pgo+ID4+Pj4+Pj4+Pgo+ID4+Pj4+Pj4gLS0KPiA+Pj4+Pj4+IENodWNrIExldmVyCj4gPj4+
Pj4+PiBjaHVja1tkb3RdbGV2ZXJbYXRdb3JhY2xlW2RvdF1jb20KPiA+Pj4+Pj4+Cj4gPj4+Pj4+
Pgo+ID4+Pj4+Pj4KPiA+Pj4+Cj4gPj4KPiA+Pgo+ID4KCl9fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fCk5GU3Y0IG1haWxpbmcgbGlzdApORlN2NEBsaW51eC1u
ZnMub3JnCmh0dHA6Ly9saW51eC1uZnMub3JnL2NnaS1iaW4vbWFpbG1hbi9saXN0aW5mby9uZnN2
NA==

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 17, 2009, at 8:30 PM, Ian Kent wrote:
> On Thu, 2009-08-13 at 12:18 -0300, Carlos André wrote:
>> Filled bug report:
>> https://bugzilla.redhat.com/show_bug.cgi?id=517349
>
> Hi Carlos,
>
> I have a patched source rpm to add a mount wait parameter to autofs
> located at:
> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>
> Could you build it and see if it works.
> I haven't tested it at all but it is fairly straight forward.

For NFSv2/v3, we have an rpcbind query followed by a MNT request, on  
separate transports that the kernel can set up with fast timeouts  
(when the underlying transport implementations eventually support this).

It could be more challenging for NFSv4 because the initial connection  
is done on the same NFS transport that will be used for normal  
operation.  The connection timeout parameters can't easily be changed  
from giving up quickly to normal, at least with the current RPC  
infrastructure.  Perhaps the first few NFSv4 operations could be done  
on a separate transport.

> It is still unclear if this is the right way to do this and what the
> consequences are in sending a term signal to mount. This mount request
> will likely be followed by other requests for the same mount causing  
> an
> accumulation of mount(8) processes waiting for RPC timeouts before  
> they
> can answer the TERM signal.

One concern is that this would leave a number of outgoing privileged  
ports in use (until the mount attempts time out, then after another  
two minutes the port leaves TIME_WAIT).  Clearly we need an in-kernel  
solution to canceling mount requests without leaving a bunch of  
resources tied up.  Probably the best approach, in general terms, is  
to tell mount(2) how long to wait, then let the kernel abort it as  
needed.

> Anyway, for information the patch included in the source rpm above is:
>
> autofs-5.0.4 - add mount wait parameter
>
> From: Ian Kent <raven@themaw.net>
>
> Often delays when trying to mount from a server that is not reponding
> for some reason are undesirable. To try and prevent these delays we
> provide a configuration setting to limit the time that we wait for
> our spawned mount(8) process to complete before sending it a SIGTERM
> signal. This patch adds a configuration parameter to allow us to
> request we limit the time we wait for mount(8) to complete before
> send it a TERM signal.
> ---
>
> daemon/spawn.c                 |    3 ++-
> include/defaults.h             |    2 ++
> lib/defaults.c                 |   13 +++++++++++++
> man/auto.master.5.in           |    7 +++++++
> redhat/autofs.sysconfig.in     |    9 +++++++++
> samples/autofs.conf.default.in |    9 +++++++++
> 6 files changed, 42 insertions(+), 1 deletion(-)
>
>
> --- autofs-5.0.1.orig/daemon/spawn.c
> +++ autofs-5.0.1/daemon/spawn.c
> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
> 	unsigned int options;
> 	unsigned int retries = MTAB_LOCK_RETRIES;
> 	int update_mtab = 1, ret, printed = 0;
> +	unsigned int wait = defaults_get_mount_wait();
> 	char buf[PATH_MAX];
>
> 	/* If we use mount locking we can't validate the location */
> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
> 	va_end(arg);
>
> 	while (retries--) {
> -		ret = do_spawn(logopt, -1, options, prog, (const char **) argv);
> +		ret = do_spawn(logopt, wait, options, prog, (const char **) argv);
> 		if (ret & MTAB_NOTUPDATED) {
> 			struct timespec tm = {3, 0};
>
> --- autofs-5.0.1.orig/include/defaults.h
> +++ autofs-5.0.1/include/defaults.h
> @@ -24,6 +24,7 @@
>
> #define DEFAULT_TIMEOUT			600
> #define DEFAULT_NEGATIVE_TIMEOUT	60
> +#define DEFAULT_MOUNT_WAIT		-1
> #define DEFAULT_UMOUNT_WAIT		12
> #define DEFAULT_BROWSE_MODE		1
> #define DEFAULT_LOGGING			0
> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
> struct ldap_searchdn *defaults_get_searchdns(void);
> void defaults_free_searchdns(struct ldap_searchdn *);
> unsigned int defaults_get_append_options(void);
> +unsigned int defaults_get_mount_wait(void);
> unsigned int defaults_get_umount_wait(void);
> const char *defaults_get_auth_conf_file(void);
> unsigned int defaults_get_map_hash_table_size(void);
> --- autofs-5.0.1.orig/lib/defaults.c
> +++ autofs-5.0.1/lib/defaults.c
> @@ -45,6 +45,7 @@
> #define ENV_NAME_VALUE_ATTR		"VALUE_ATTRIBUTE"
>
> #define ENV_APPEND_OPTIONS		"APPEND_OPTIONS"
> +#define ENV_MOUNT_WAIT			"MOUNT_WAIT"
> #define ENV_UMOUNT_WAIT			"UMOUNT_WAIT"
> #define ENV_AUTH_CONF_FILE		"AUTH_CONF_FILE"
>
> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
> 		    check_set_config_value(key, ENV_NAME_ENTRY_ATTR, value,  
> to_syslog) ||
> 		    check_set_config_value(key, ENV_NAME_VALUE_ATTR, value,  
> to_syslog) ||
> 		    check_set_config_value(key, ENV_APPEND_OPTIONS, value,  
> to_syslog) ||
> +		    check_set_config_value(key, ENV_MOUNT_WAIT, value, to_syslog)  
> ||
> 		    check_set_config_value(key, ENV_UMOUNT_WAIT, value, to_syslog)  
> ||
> 		    check_set_config_value(key, ENV_AUTH_CONF_FILE, value,  
> to_syslog) ||
> 		    check_set_config_value(key, ENV_MAP_HASH_TABLE_SIZE, value,  
> to_syslog))
> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
> 	return res;
> }
>
> +unsigned int defaults_get_mount_wait(void)
> +{
> +	long wait;
> +
> +	wait = get_env_number(ENV_MOUNT_WAIT);
> +	if (wait < 0)
> +		wait = DEFAULT_MOUNT_WAIT;
> +
> +	return (unsigned int) wait;
> +}
> +
> unsigned int defaults_get_umount_wait(void)
> {
> 	long wait;
> --- autofs-5.0.1.orig/man/auto.master.5.in
> +++ autofs-5.0.1/man/auto.master.5.in
> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
> 60). If the equivalent command line option is given it will override  
> this
> setting.
> .TP
> +.B MOUNT_WAIT
> +Set the default time to wait for a response from a spawned mount(8)
> +before sending it a SIGTERM. Note that we still need to wait for the
> +RPC layer to timeout before the sub-process exits so this isn't ideal
> +but it is the best we can do. The default is to wait until mount(8)
> +returns without intervention.
> +.TP
> .B UMOUNT_WAIT
> Set the default time to wait for a response from a spawned umount(8)
> before sending it a SIGTERM. Note that we still need to wait for the
> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
> @@ -14,6 +14,15 @@ TIMEOUT=300
> #
> #NEGATIVE_TIMEOUT=60
> #
> +# MOUNT_WAIT - time to wait for a response from umount(8).
> +# 	       Setting this timeout can cause problems when
> +# 	       mount would otherwise wait for a server that
> +# 	       is temporarily unavailable, such as when it's
> +# 	       restarting. The defailt of waiting for mount(8)
> +# 	       usually results in a wait of around 3 minutes.
> +#
> +#MOUNT_WAIT=-1
> +#
> # UMOUNT_WAIT - time to wait for a response from umount(8).
> #
> #UMOUNT_WAIT=12
> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
> +++ autofs-5.0.1/samples/autofs.conf.default.in
> @@ -14,6 +14,15 @@ TIMEOUT=300
> #
> #NEGATIVE_TIMEOUT=60
> #
> +# MOUNT_WAIT - time to wait for a response from umount(8).
> +# 	       Setting this timeout can cause problems when
> +# 	       mount would otherwise wait for a server that
> +# 	       is temporarily unavailable, such as when it's
> +# 	       restarting. The defailt of waiting for mount(8)
> +# 	       usually results in a wait of around 3 minutes.
> +#
> +#MOUNT_WAIT=-1
> +#
> # UMOUNT_WAIT - time to wait for a response from umount(8).
> #
> #UMOUNT_WAIT=12
>
>
>>
>> Thanks!
>>
>> 2009/8/13 Carlos André <candrecn@gmail.com>:
>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>> Carlos André wrote:
>>>>> Today (2009-08-12) I'm using:
>>>>> kernel-2.6.18-128.2.1.el5
>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>
>>>> Thanks,
>>>>
>>>> My mistake, the wait time I was referring to is used for umounts  
>>>> during
>>>> expires and is present in rev rc2.102.
>>>>
>>>> It shouldn't be hard to add this for mount as well.
>>>> Would you like me to put something together?
>>>
>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>>>
>>>>
>>>> Probably would be good to test something out to see if we can  
>>>> make a
>>>> difference with the killing mount after some configured timeout  
>>>> but, if
>>>> we make progress, probably the best way to deal with it is for  
>>>> you to
>>>> log a bug against rhel-5 so I can get it committed to the rhel  
>>>> package.
>>>> The possible issue is that I'm not sure if the RPC subsystem in the
>>>> above rhel kernel will respond well to process death with potential
>>>> outstanding requests. But we'll see.
>>>
>>> Ok, on my way :)
>>>
>>> Thanks a lot!
>>>
>>>>
>>>>>
>>>>>
>>>>> Look my last test:
>>>>> --------------------------------------------------------------
>>>>> [root@KSTATION areas]# time ls testdown
>>>>> ls: testdown: No such file or directory
>>>>>
>>>>> real    3m9.025s
>>>>> user    0m0.000s
>>>>> sys     0m0.002s
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>> acl,sec=krb5p,proto=tcp,retry=0
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>> acl,sec=krb5p,proto=tcp,retry=0 using module nfs4
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  
>>>>> mount(nfs):
>>>>> root=/misc/areas name=testdown what=1.2.3.4:/areas/testdown,
>>>>> fstype=nfs4, options=acl,sec=krb5p,proto=tcp,retry=0
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  
>>>>> mount(nfs):
>>>>> nfs options="acl,sec=krb5p,proto=tcp,retry=0", nosymlink=0, ro=0
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  
>>>>> mount(nfs):
>>>>> calling mkdir_path /misc/areas/testdown
>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  
>>>>> mount(nfs):
>>>>> calling mount -t nfs4 -s -o acl,sec=krb5p,proto=tcp,retry=0
>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1  
>>>>> path /misc
>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
>>>>> 3078093712 path /misc
>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>> submounts remaining in /misc
>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got  
>>>>> thid
>>>>> 3078093712 path /misc stat 3
>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup:  
>>>>> sigchld:
>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready():  
>>>>> state
>>>>> = 2 path /misc
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1  
>>>>> path /misc
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
>>>>> 3078093712 path /misc
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>> submounts remaining in /misc
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got  
>>>>> thid
>>>>> 3078093712 path /misc stat 3
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup:  
>>>>> sigchld:
>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready():  
>>>>> state
>>>>> = 2 path /misc
>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token = 17
>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/ 
>>>>> areas/testdown
>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1  
>>>>> path /misc
>>>>> --------------------------------------------------------------
>>>>>
>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>> Carlos André wrote:
>>>>>>> Hi Ian,
>>>>>>> I'm getting crazy trying put "retry=" to work on mount... this  
>>>>>>> option
>>>>>>> just DONT WORK if use proto=tcp and/OR kerberos (sec=krb5/ 
>>>>>>> krb5i/krb5p)
>>>>>>> like you can see on my previous emails...
>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>
>>>>>> Maybe this is related to the same sort of problem we had with  
>>>>>> mount in
>>>>>> the past, before the options parsing went into the kernel,  
>>>>>> where other
>>>>>> services, like portmapper (or rpcbind), were being done with  
>>>>>> different
>>>>>> timeout parameters before the RPC calls for mounting. That's  
>>>>>> just an
>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>
>>>>>> But what version of autofs and kernel did you say you were using?
>>>>>>
>>>>>>> I appreciate any help.
>>>>>>>
>>>>>>> Carlos.
>>>>>>>
>>>>>>>
>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>> Chuck Lever wrote:
>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos André wrote:
>>>>>>>>>> This long timeout is good if workstation need mount a  
>>>>>>>>>> critical
>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>> But in my case, using this loooong timeout doesnt make any  
>>>>>>>>>> sense,
>>>>>>>>>> since autofs retry mount directory on-access. This in fact  
>>>>>>>>>> gives me
>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one  
>>>>>>>>>> server goes
>>>>>>>>>> down for any reason, and will again hangs if user try  
>>>>>>>>>> access directory
>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>> "retry=0" means the mount command will fail as soon as the  
>>>>>>>>> first
>>>>>>>>> mount(2) system call fails.  When you set SYN retries to 1,  
>>>>>>>>> this means
>>>>>>>>> after 9 seconds, the connect fails, and that causes the  
>>>>>>>>> mount(2) system
>>>>>>>>> call to fail.
>>>>>>>>>
>>>>>>>>> Recent conversations with Ian suggested that a long timeout  
>>>>>>>>> was desired
>>>>>>>>> for automounter as well as other cases.  Ian, is there  
>>>>>>>>> something else we
>>>>>>>>> need to consider to determine the correct retry timeout for  
>>>>>>>>> NFS/TCP
>>>>>>>>> mount points handled via automounter?  How should mount.nfs  
>>>>>>>>> wait so we
>>>>>>>>> don't make other use cases worse?  (Looks like most of the  
>>>>>>>>> history is
>>>>>>>>> intact below).
>>>>>>>> Of course we know that autofs is entirely at the mercy of  
>>>>>>>> mount(8) (and
>>>>>>>> mount.nfs in particular). This has always been a difficult  
>>>>>>>> situation for
>>>>>>>> the automounter because interactive mount invocations should  
>>>>>>>> wait. But I
>>>>>>>> believe automount mounts should always time out quickly, but  
>>>>>>>> that leads
>>>>>>>> to its own set of problems, especially when home directories  
>>>>>>>> are concerned.
>>>>>>>>
>>>>>>>> I think adding "retry=0" is the right thing to do myself but  
>>>>>>>> I'm not
>>>>>>>> certain that will work as we expect. I'll have to do some  
>>>>>>>> experimentation.
>>>>>>>>
>>>>>>>>> How long do you think is appropriate for the automounter to  
>>>>>>>>> wait if the
>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>
>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>> ------------------------------------------------
>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/ 
>>>>>>>>>> tcp_syn_retries  [DEFAULT]
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> proto=tcp,retry=1
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    3m9.000s
>>>>>>>>>> user    0m0.002s
>>>>>>>>>> sys     0m0.001s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    3m9.000s
>>>>>>>>>> user    0m0.000s
>>>>>>>>>> sys     0m0.002s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> proto=tcp,retry=0
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    3m9.001s
>>>>>>>>>> user    0m0.000s
>>>>>>>>>> sys     0m0.003s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    3m9.001s
>>>>>>>>>> user    0m0.002s
>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>
>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/ 
>>>>>>>>>> tcp_syn_retries [ 5 to 1 ]
>>>>>>>>>>
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> proto=tcp,retry=1
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (retrying). [x 6]
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    1m3.002s
>>>>>>>>>> user    0m0.000s
>>>>>>>>>> sys     0m0.002s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> sec=krb5p,proto=tcp,retry=1
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (retrying). [x 13]
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    2m6.000s
>>>>>>>>>> user    0m0.000s
>>>>>>>>>> sys     0m0.002s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> proto=tcp,retry=0
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    0m9.003s
>>>>>>>>>> user    0m0.001s
>>>>>>>>>> sys     0m0.002s
>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4  
>>>>>>>>>> -o
>>>>>>>>>> sec=krb5p,proto=tcp,retry=0
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (retrying). [x 13]
>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>> (giving up).
>>>>>>>>>>
>>>>>>>>>> real    2m6.001s
>>>>>>>>>> user    0m0.001s
>>>>>>>>>> sys     0m0.002s
>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>> ------------------------------------------------
>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to  
>>>>>>>>>> 1... and
>>>>>>>>>> using retry=0 without kerberos I got only 9s...
>>>>>>>>>>
>>>>>>>>>> *sigh*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos André wrote:
>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change  
>>>>>>>>>>>> tcp_syn_retries to
>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's socket  
>>>>>>>>>>> connect
>>>>>>>>>>> function,
>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes  
>>>>>>>>>>> longer than
>>>>>>>>>>> the RPC
>>>>>>>>>>> client's connect timeout, so it only makes one connect  
>>>>>>>>>>> call, and then
>>>>>>>>>>> fails.
>>>>>>>>>>>
>>>>>>>>>>> Reducing the number of SYN retries per connect attempt  
>>>>>>>>>>> causes the RPC
>>>>>>>>>>> client
>>>>>>>>>>> to retry the connect call until its connect timeout  
>>>>>>>>>>> expires.  Each
>>>>>>>>>>> connect
>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>
>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t  
>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>> sec=krb5p,proto=tcp
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/ 
>>>>>>>>>>>> tcp_syn_retries
>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t  
>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>> sec=krb5p,proto=tcp  ("retry=1" = no change)
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (retrying).
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  
>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>
>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2009/8/10 Carlos André <candrecn@gmail.com>:
>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>
>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've  
>>>>>>>>>>>>> monitored
>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96  
>>>>>>>>>>>>> secs on port
>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>> I tried use "retry=1" option on mount without any  
>>>>>>>>>>>>> change... I dont
>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation  
>>>>>>>>>>>>>>> where my
>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max)  
>>>>>>>>>>>>>>> than 3 minutes
>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the  
>>>>>>>>>>>>>> kernel to
>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN  
>>>>>>>>>>>>>> attempts with
>>>>>>>>>>>>>> exponential retries, or something like that).  For  
>>>>>>>>>>>>>> stock CentOS
>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>> think
>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4  
>>>>>>>>>>>>>> mounts -- the
>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>> just
>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no  
>>>>>>>>>>>>>> preceding rpcbind
>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS- 
>>>>>>>>>>>>>> related CentOS
>>>>>>>>>>>>>> components
>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny  
>>>>>>>>>>>>>>>> Halevy wrote:
>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com 
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4  
>>>>>>>>>>>>>>>>>>> server to work with
>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server  
>>>>>>>>>>>>>>>>>>> goes down i get a
>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4  
>>>>>>>>>>>>>>>>>>> client...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user  
>>>>>>>>>>>>>>>>>>> logon process, if
>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to  
>>>>>>>>>>>>>>>>>>> timeout (if server
>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of  
>>>>>>>>>>>>>>>>>>> combinations, there my
>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP:  
>>>>>>>>>>>>>>>>>>> 172.16.1.10) using
>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t  
>>>>>>>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS  
>>>>>>>>>>>>>>>>>>> (proto=tcp OR
>>>>>>>>>>>>>>>>>>> proto=udp)
>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until  
>>>>>>>>>>>>>>>>>>> show error
>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving  
>>>>>>>>>>>>>>>>>>> up))
>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>>>> I thought he was describing a situation where the  
>>>>>>>>>>>>>>>> server the server
>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and  
>>>>>>>>>>>>>>>> wondering how to make
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> To unsubscribe from this list: send the line  
>>>>>>>>>>>>>>> "unsubscribe
>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Lever
>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Chuck Lever
>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>>>
>>>
>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Hi Ian,

Thanks for patch and sorry for delay (i'm expecting receive u reply on
bug track, not here) :)

But, this patch doesnt worked to me like expected...  :(


=46irstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
and later changed "10" to "2" with same results...
(always restarting service, of course :)

Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
same results again.

Or i'm doing something wrong?


[root@KSTATION areas]# automount -V

Linux automount version 5.0.1-0.rc2.131.bz517349.1
[...]

[root@KSTATION areas]# time ls -la testdown
ls: testedown: No such file or directory

real    3m9.006s
user    0m0.002s
sys     0m0.000s


LOGGING:
-----------------------------------------
Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdown
/misc/areas/testdown
Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D 9=
1
Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/areas/=
testdown
-----------------------------------------





2009/8/17 Ian Kent <ikent@redhat.com>:
> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>> Filled bug report:
>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>
> Hi Carlos,
>
> I have a patched source rpm to add a mount wait parameter to autofs
> located at:
> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>
> Could you build it and see if it works.
> I haven't tested it at all but it is fairly straight forward.
> It is still unclear if this is the right way to do this and what the
> consequences are in sending a term signal to mount. This mount reques=
t
> will likely be followed by other requests for the same mount causing =
an
> accumulation of mount(8) processes waiting for RPC timeouts before th=
ey
> can answer the TERM signal.
>
> Anyway, for information the patch included in the source rpm above is=
:
>
> autofs-5.0.4 - add mount wait parameter
>
> From: Ian Kent <raven@themaw.net>
>
> Often delays when trying to mount from a server that is not reponding
> for some reason are undesirable. To try and prevent these delays we
> provide a configuration setting to limit the time that we wait for
> our spawned mount(8) process to complete before sending it a SIGTERM
> signal. This patch adds a configuration parameter to allow us to
> request we limit the time we wait for mount(8) to complete before
> send it a TERM signal.
> ---
>
> =A0daemon/spawn.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A03 ++-
> =A0include/defaults.h =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A02 ++
> =A0lib/defaults.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 13 ++++++++++=
+++
> =A0man/auto.master.5.in =A0 =A0 =A0 =A0 =A0 | =A0 =A07 +++++++
> =A0redhat/autofs.sysconfig.in =A0 =A0 | =A0 =A09 +++++++++
> =A0samples/autofs.conf.default.in | =A0 =A09 +++++++++
> =A06 files changed, 42 insertions(+), 1 deletion(-)
>
>
> --- autofs-5.0.1.orig/daemon/spawn.c
> +++ autofs-5.0.1/daemon/spawn.c
> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
> =A0 =A0 =A0 =A0unsigned int options;
> =A0 =A0 =A0 =A0unsigned int retries =3D MTAB_LOCK_RETRIES;
> =A0 =A0 =A0 =A0int update_mtab =3D 1, ret, printed =3D 0;
> + =A0 =A0 =A0 unsigned int wait =3D defaults_get_mount_wait();
> =A0 =A0 =A0 =A0char buf[PATH_MAX];
>
> =A0 =A0 =A0 =A0/* If we use mount locking we can't validate the locat=
ion */
> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
> =A0 =A0 =A0 =A0va_end(arg);
>
> =A0 =A0 =A0 =A0while (retries--) {
> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, -1, options, p=
rog, (const char **) argv);
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, wait, options,=
 prog, (const char **) argv);
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (ret & MTAB_NOTUPDATED) {
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct timespec tm =3D=
 {3, 0};
>
> --- autofs-5.0.1.orig/include/defaults.h
> +++ autofs-5.0.1/include/defaults.h
> @@ -24,6 +24,7 @@
>
> =A0#define DEFAULT_TIMEOUT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0600
> =A0#define DEFAULT_NEGATIVE_TIMEOUT =A0 =A0 =A0 60
> +#define DEFAULT_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 -1
> =A0#define DEFAULT_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A012
> =A0#define DEFAULT_BROWSE_MODE =A0 =A0 =A0 =A0 =A0 =A01
> =A0#define DEFAULT_LOGGING =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A00
> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
> =A0struct ldap_searchdn *defaults_get_searchdns(void);
> =A0void defaults_free_searchdns(struct ldap_searchdn *);
> =A0unsigned int defaults_get_append_options(void);
> +unsigned int defaults_get_mount_wait(void);
> =A0unsigned int defaults_get_umount_wait(void);
> =A0const char *defaults_get_auth_conf_file(void);
> =A0unsigned int defaults_get_map_hash_table_size(void);
> --- autofs-5.0.1.orig/lib/defaults.c
> +++ autofs-5.0.1/lib/defaults.c
> @@ -45,6 +45,7 @@
> =A0#define ENV_NAME_VALUE_ATTR =A0 =A0 =A0 =A0 =A0 =A0"VALUE_ATTRIBUT=
E"
>
> =A0#define ENV_APPEND_OPTIONS =A0 =A0 =A0 =A0 =A0 =A0 "APPEND_OPTIONS=
"
> +#define ENV_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 "MOUNT_WAIT"
> =A0#define ENV_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0=
 =A0"UMOUNT_WAIT"
> =A0#define ENV_AUTH_CONF_FILE =A0 =A0 =A0 =A0 =A0 =A0 "AUTH_CONF_FILE=
"
>
> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_NAME_ENTRY_ATTR, value, to_syslog) ||
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_NAME_VALUE_ATTR, value, to_syslog) ||
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_APPEND_OPTIONS, value, to_syslog) ||
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 check_set_config_value(key, ENV=
_MOUNT_WAIT, value, to_syslog) ||
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_UMOUNT_WAIT, value, to_syslog) ||
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_AUTH_CONF_FILE, value, to_syslog) ||
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, EN=
V_MAP_HASH_TABLE_SIZE, value, to_syslog))
> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
> =A0 =A0 =A0 =A0return res;
> =A0}
>
> +unsigned int defaults_get_mount_wait(void)
> +{
> + =A0 =A0 =A0 long wait;
> +
> + =A0 =A0 =A0 wait =3D get_env_number(ENV_MOUNT_WAIT);
> + =A0 =A0 =A0 if (wait < 0)
> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 wait =3D DEFAULT_MOUNT_WAIT;
> +
> + =A0 =A0 =A0 return (unsigned int) wait;
> +}
> +
> =A0unsigned int defaults_get_umount_wait(void)
> =A0{
> =A0 =A0 =A0 =A0long wait;
> --- autofs-5.0.1.orig/man/auto.master.5.in
> +++ autofs-5.0.1/man/auto.master.5.in
> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
> =A060). If the equivalent command line option is given it will overri=
de this
> =A0setting.
> =A0.TP
> +.B MOUNT_WAIT
> +Set the default time to wait for a response from a spawned mount(8)
> +before sending it a SIGTERM. Note that we still need to wait for the
> +RPC layer to timeout before the sub-process exits so this isn't idea=
l
> +but it is the best we can do. The default is to wait until mount(8)
> +returns without intervention.
> +.TP
> =A0.B UMOUNT_WAIT
> =A0Set the default time to wait for a response from a spawned umount(=
8)
> =A0before sending it a SIGTERM. Note that we still need to wait for t=
he
> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
> @@ -14,6 +14,15 @@ TIMEOUT=3D300
> =A0#
> =A0#NEGATIVE_TIMEOUT=3D60
> =A0#
> +# MOUNT_WAIT - time to wait for a response from umount(8).
> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems wh=
en
> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server th=
at
> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when i=
t's
> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for mou=
nt(8)
> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 minu=
tes.
> +#
> +#MOUNT_WAIT=3D-1
> +#
> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
> =A0#
> =A0#UMOUNT_WAIT=3D12
> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
> +++ autofs-5.0.1/samples/autofs.conf.default.in
> @@ -14,6 +14,15 @@ TIMEOUT=3D300
> =A0#
> =A0#NEGATIVE_TIMEOUT=3D60
> =A0#
> +# MOUNT_WAIT - time to wait for a response from umount(8).
> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems wh=
en
> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server th=
at
> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when i=
t's
> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for mou=
nt(8)
> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 minu=
tes.
> +#
> +#MOUNT_WAIT=3D-1
> +#
> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
> =A0#
> =A0#UMOUNT_WAIT=3D12
>
>
>>
>> Thanks!
>>
>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>> > 2009/8/13 Ian Kent <ikent@redhat.com>:
>> >> Carlos Andr=E9 wrote:
>> >>> Today (2009-08-12) I'm using:
>> >>> kernel-2.6.18-128.2.1.el5
>> >>> autofs-5.0.1-0.rc2.102.el5_3.1
>> >>
>> >> Thanks,
>> >>
>> >> My mistake, the wait time I was referring to is used for umounts =
during
>> >> expires and is present in rev rc2.102.
>> >>
>> >> It shouldn't be hard to add this for mount as well.
>> >> Would you like me to put something together?
>> >
>> > Sure! that 'll help me a lot (and for sure another ppl) :) Thanks =
:)
>> >
>> >>
>> >> Probably would be good to test something out to see if we can mak=
e a
>> >> difference with the killing mount after some configured timeout b=
ut, if
>> >> we make progress, probably the best way to deal with it is for yo=
u to
>> >> log a bug against rhel-5 so I can get it committed to the rhel pa=
ckage.
>> >> The possible issue is that I'm not sure if the RPC subsystem in t=
he
>> >> above rhel kernel will respond well to process death with potenti=
al
>> >> outstanding requests. But we'll see.
>> >
>> > Ok, on my way :)
>> >
>> > Thanks a lot!
>> >
>> >>
>> >>>
>> >>>
>> >>> Look my last test:
>> >>> --------------------------------------------------------------
>> >>> [root@KSTATION areas]# time ls testdown
>> >>> ls: testdown: No such file or directory
>> >>>
>> >>> real =A0 =A03m9.025s
>> >>> user =A0 =A00m0.000s
>> >>> sys =A0 =A0 0m0.002s
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun)=
:
>> >>> mounting root /misc/areas, mountpoint testdown, what
>> >>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>> >>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>> >>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>> >>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>> >>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdow=
n,
>> >>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>> >>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=
=3D0, ro=3D0
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>> >>> calling mkdir_path /misc/areas/testdown
>> >>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>> >>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D=
0
>> >>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>> >>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>> >>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc=
 =3D
>> >>> 3078093712 path /misc
>> >>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect:=
 2
>> >>> submounts remaining in /misc
>> >>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got t=
hid
>> >>> 3078093712 path /misc stat 3
>> >>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigch=
ld:
>> >>> exp 3078093712 finished, switching from 2 to 1
>> >>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready():=
 state
>> >>> =3D 2 path /misc
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc=
 =3D
>> >>> 3078093712 path /misc
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect:=
 2
>> >>> submounts remaining in /misc
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got t=
hid
>> >>> 3078093712 path /misc stat 3
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigch=
ld:
>> >>> exp 3078093712 finished, switching from 2 to 1
>> >>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready():=
 state
>> >>> =3D 2 path /misc
>> >>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NF=
S
>> >>> server '1.2.3.4' failed: timed out (giving up).
>> >>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: moun=
t
>> >>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>> >>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D =
17
>> >>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc=
/areas/testdown
>> >>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>> >>> --------------------------------------------------------------
>> >>>
>> >>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>> >>>> Carlos Andr=E9 wrote:
>> >>>>> Hi Ian,
>> >>>>> I'm getting crazy trying put "retry=3D" to work on mount... th=
is option
>> >>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/=
krb5i/krb5p)
>> >>>>> like you can see on my previous emails...
>> >>>> Right, my mistake for not looking closely enough at post.
>> >>>>
>> >>>> Maybe this is related to the same sort of problem we had with m=
ount in
>> >>>> the past, before the options parsing went into the kernel, wher=
e other
>> >>>> services, like portmapper (or rpcbind), were being done with di=
fferent
>> >>>> timeout parameters before the RPC calls for mounting. That's ju=
st an
>> >>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>> >>>>
>> >>>> But what version of autofs and kernel did you say you were usin=
g?
>> >>>>
>> >>>>> I appreciate any help.
>> >>>>>
>> >>>>> Carlos.
>> >>>>>
>> >>>>>
>> >>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>> >>>>>> Chuck Lever wrote:
>> >>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>> >>>>>>>> This long timeout is good if workstation need mount a criti=
cal
>> >>>>>>>> directory using /etc/fstab on boot (for example)..
>> >>>>>>>> But in my case, using this loooong timeout doesnt make any =
sense,
>> >>>>>>>> since autofs retry mount directory on-access. This in fact =
gives me
>> >>>>>>>> alot of headaches, coz user login 'll just hangs if one ser=
ver goes
>> >>>>>>>> down for any reason, and will again hangs if user try acces=
s directory
>> >>>>>>>> pointing to a NFS down server...
>> >>>>>>> "retry=3D0" means the mount command will fail as soon as the=
 first
>> >>>>>>> mount(2) system call fails. =A0When you set SYN retries to 1=
, this means
>> >>>>>>> after 9 seconds, the connect fails, and that causes the moun=
t(2) system
>> >>>>>>> call to fail.
>> >>>>>>>
>> >>>>>>> Recent conversations with Ian suggested that a long timeout =
was desired
>> >>>>>>> for automounter as well as other cases. =A0Ian, is there som=
ething else we
>> >>>>>>> need to consider to determine the correct retry timeout for =
NFS/TCP
>> >>>>>>> mount points handled via automounter? =A0How should mount.nf=
s wait so we
>> >>>>>>> don't make other use cases worse? =A0(Looks like most of the=
 history is
>> >>>>>>> intact below).
>> >>>>>> Of course we know that autofs is entirely at the mercy of mou=
nt(8) (and
>> >>>>>> mount.nfs in particular). This has always been a difficult si=
tuation for
>> >>>>>> the automounter because interactive mount invocations should =
wait. But I
>> >>>>>> believe automount mounts should always time out quickly, but =
that leads
>> >>>>>> to its own set of problems, especially when home directories =
are concerned.
>> >>>>>>
>> >>>>>> I think adding "retry=3D0" is the right thing to do myself bu=
t I'm not
>> >>>>>> certain that will work as we expect. I'll have to do some exp=
erimentation.
>> >>>>>>
>> >>>>>>> How long do you think is appropriate for the automounter to =
wait if the
>> >>>>>>> server is down, in your case, Carlos?
>> >>>>>>>
>> >>>>>>>> Am losing something or there have was something weirdo...!?
>> >>>>>>>> ------------------------------------------------
>> >>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retr=
ies =A0[DEFAULT]
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> proto=3Dtcp,retry=3D1
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A03m9.000s
>> >>>>>>>> user =A0 =A00m0.002s
>> >>>>>>>> sys =A0 =A0 0m0.001s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A03m9.000s
>> >>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> proto=3Dtcp,retry=3D0
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A03m9.001s
>> >>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>> sys =A0 =A0 0m0.003s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A03m9.001s
>> >>>>>>>> user =A0 =A00m0.002s
>> >>>>>>>> sys =A0 =A0 0m0.001s
>> >>>>>>>>
>> >>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retr=
ies [ 5 to 1 ]
>> >>>>>>>>
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> proto=3Dtcp,retry=3D1
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 6]
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A01m3.002s
>> >>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 13]
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A02m6.000s
>> >>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> proto=3Dtcp,retry=3D0
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A00m9.003s
>> >>>>>>>> user =A0 =A00m0.001s
>> >>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>> >>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 13]
>> >>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>> >>>>>>>>
>> >>>>>>>> real =A0 =A02m6.001s
>> >>>>>>>> user =A0 =A00m0.001s
>> >>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>> [root@KSTATION ~]#
>> >>>>>>>> ------------------------------------------------
>> >>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to=
 1... and
>> >>>>>>>> using retry=3D0 without kerberos I got only 9s...
>> >>>>>>>>
>> >>>>>>>> *sigh*
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>> >>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>> >>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>> >>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_sy=
n_retries to
>> >>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>> >>>>>>>>> Right. =A0Normally the RPC client calls the kernel's socke=
t connect
>> >>>>>>>>> function,
>> >>>>>>>>> which does 6 SYN retries. =A0That one call usually takes l=
onger than
>> >>>>>>>>> the RPC
>> >>>>>>>>> client's connect timeout, so it only makes one connect cal=
l, and then
>> >>>>>>>>> fails.
>> >>>>>>>>>
>> >>>>>>>>> Reducing the number of SYN retries per connect attempt cau=
ses the RPC
>> >>>>>>>>> client
>> >>>>>>>>> to retry the connect call until its connect timeout expire=
s. =A0Each
>> >>>>>>>>> connect
>> >>>>>>>>> call resets the SYN timeout to 3 seconds.
>> >>>>>>>>>
>> >>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nf=
s4 -o
>> >>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (g=
iving up).
>> >>>>>>>>>>
>> >>>>>>>>>> real =A0 =A03m9.000s
>> >>>>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>>>> sys =A0 =A0 0m0.002s
>> >>>>>>>>>>
>> >>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_ret=
ries
>> >>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nf=
s4 -o
>> >>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>> >>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (g=
iving up).
>> >>>>>>>>>>
>> >>>>>>>>>> real =A0 =A02m6.004s
>> >>>>>>>>>> user =A0 =A00m0.000s
>> >>>>>>>>>> sys =A0 =A0 0m0.004s
>> >>>>>>>>>>
>> >>>>>>>>>> (3,6,3,6... secs interval)
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>> >>>>>>>>>>> No, i'm just using packages from CentOS repo...
>> >>>>>>>>>>>
>> >>>>>>>>>>> And u're right about expo retries... with tcpdump i've m=
onitored
>> >>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 se=
cs on port
>> >>>>>>>>>>> 2049...
>> >>>>>>>>>>> I tried use "retry=3D1" option on mount without any chan=
ge... I dont
>> >>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>> >>>>>>>>>>>
>> >>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>> >>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>> >>>>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situati=
on where my
>> >>>>>>>>>>>>> server
>> >>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) t=
han 3 minutes
>> >>>>>>>>>>>>> and 9 seconds...
>> >>>>>>>>>>>> The 189 second timeout is likely how long it takes the =
kernel to
>> >>>>>>>>>>>> give up
>> >>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN att=
empts with
>> >>>>>>>>>>>> exponential retries, or something like that). =A0For st=
ock CentOS
>> >>>>>>>>>>>> 5.3, I
>> >>>>>>>>>>>> think
>> >>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 moun=
ts -- the
>> >>>>>>>>>>>> kernel
>> >>>>>>>>>>>> just
>> >>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no pre=
ceding rpcbind
>> >>>>>>>>>>>> request.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-relate=
d CentOS
>> >>>>>>>>>>>> components
>> >>>>>>>>>>>> (kernel, nfs-utils) with something you've built yoursel=
f.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>> >>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halev=
y wrote:
>> >>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candre=
cn@gmail.com>
>> >>>>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>>>>> Anyone ?
>> >>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>> >>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 serve=
r to work with
>> >>>>>>>>>>>>>>>>> Kerberos
>> >>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goe=
s down i get a
>> >>>>>>>>>>>>>>>>> LOOOOOOONG
>> >>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client=
=2E..
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logo=
n process, if
>> >>>>>>>>>>>>>>>>> mount
>> >>>>>>>>>>>>>>>>> hangs,
>> >>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to time=
out (if server
>> >>>>>>>>>>>>>>>>> down)
>> >>>>>>>>>>>>>>>>> after
>> >>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinatio=
ns, there my
>> >>>>>>>>>>>>>>>>> findings
>> >>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1=
=2E10) using
>> >>>>>>>>>>>>>>>>> basic
>> >>>>>>>>>>>>>>>>> command
>> >>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t n=
fs4 -o
>> >>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>> >>>>>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (prot=
o=3Dtcp OR
>> >>>>>>>>>>>>>>>>> proto=3Dudp)
>> >>>>>>>>>>>>>>>>> it
>> >>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0unt=
il show error
>> >>>>>>>>>>>>>>>>> (mount:
>> >>>>>>>>>>>>>>>>> mount to
>> >>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving=
 up))
>> >>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period=
=2E
>> >>>>>>>>>>>>>> I thought he was describing a situation where the ser=
ver the server
>> >>>>>>>>>>>>>> is completely gone and isn't coming back, and wonderi=
ng how to make
>> >>>>>>>>>>>>>> the
>> >>>>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> --b.
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>> --
>> >>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscr=
ibe
>> >>>>>>>>>>>>> linux-nfs" in
>> >>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>> >>>>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/major=
domo-info.html
>> >>>>>>>>>>>> --
>> >>>>>>>>>>>> Chuck Lever
>> >>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> Chuck Lever
>> >>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>> --
>> >>>>>>> Chuck Lever
>> >>>>>>> chuck[dot]lever[at]oracle[dot]com
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>
>> >>
>> >>
>> >
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Carlos Andr=E9 wrote:
> Hi Ian,
>=20
> Thanks for patch and sorry for delay (i'm expecting receive u reply o=
n
> bug track, not here) :)
>=20
> But, this patch doesnt worked to me like expected...  :(
>=20
>=20
> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
> and later changed "10" to "2" with same results...
> (always restarting service, of course :)
>=20
> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
> same results again.
>=20
> Or i'm doing something wrong?
>=20
>=20
> [root@KSTATION areas]# automount -V
>=20
> Linux automount version 5.0.1-0.rc2.131.bz517349.1
> [...]
>=20
> [root@KSTATION areas]# time ls -la testdown
> ls: testedown: No such file or directory
>=20
> real    3m9.006s
> user    0m0.002s
> sys     0m0.000s

OK, that isn't behaving the way I expect, I'll have a look.

>=20
>=20
> LOGGING:
> -----------------------------------------
> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdow=
n
> /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D=
 91
> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/area=
s/testdown
> -----------------------------------------
>=20
>=20
>=20
>=20
>=20
> 2009/8/17 Ian Kent <ikent@redhat.com>:
>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>> Filled bug report:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>> Hi Carlos,
>>
>> I have a patched source rpm to add a mount wait parameter to autofs
>> located at:
>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>
>> Could you build it and see if it works.
>> I haven't tested it at all but it is fairly straight forward.
>> It is still unclear if this is the right way to do this and what the
>> consequences are in sending a term signal to mount. This mount reque=
st
>> will likely be followed by other requests for the same mount causing=
 an
>> accumulation of mount(8) processes waiting for RPC timeouts before t=
hey
>> can answer the TERM signal.
>>
>> Anyway, for information the patch included in the source rpm above i=
s:
>>
>> autofs-5.0.4 - add mount wait parameter
>>
>> From: Ian Kent <raven@themaw.net>
>>
>> Often delays when trying to mount from a server that is not repondin=
g
>> for some reason are undesirable. To try and prevent these delays we
>> provide a configuration setting to limit the time that we wait for
>> our spawned mount(8) process to complete before sending it a SIGTERM
>> signal. This patch adds a configuration parameter to allow us to
>> request we limit the time we wait for mount(8) to complete before
>> send it a TERM signal.
>> ---
>>
>>  daemon/spawn.c                 |    3 ++-
>>  include/defaults.h             |    2 ++
>>  lib/defaults.c                 |   13 +++++++++++++
>>  man/auto.master.5.in           |    7 +++++++
>>  redhat/autofs.sysconfig.in     |    9 +++++++++
>>  samples/autofs.conf.default.in |    9 +++++++++
>>  6 files changed, 42 insertions(+), 1 deletion(-)
>>
>>
>> --- autofs-5.0.1.orig/daemon/spawn.c
>> +++ autofs-5.0.1/daemon/spawn.c
>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>        unsigned int options;
>>        unsigned int retries =3D MTAB_LOCK_RETRIES;
>>        int update_mtab =3D 1, ret, printed =3D 0;
>> +       unsigned int wait =3D defaults_get_mount_wait();
>>        char buf[PATH_MAX];
>>
>>        /* If we use mount locking we can't validate the location */
>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>        va_end(arg);
>>
>>        while (retries--) {
>> -               ret =3D do_spawn(logopt, -1, options, prog, (const c=
har **) argv);
>> +               ret =3D do_spawn(logopt, wait, options, prog, (const=
 char **) argv);
>>                if (ret & MTAB_NOTUPDATED) {
>>                        struct timespec tm =3D {3, 0};
>>
>> --- autofs-5.0.1.orig/include/defaults.h
>> +++ autofs-5.0.1/include/defaults.h
>> @@ -24,6 +24,7 @@
>>
>>  #define DEFAULT_TIMEOUT                        600
>>  #define DEFAULT_NEGATIVE_TIMEOUT       60
>> +#define DEFAULT_MOUNT_WAIT             -1
>>  #define DEFAULT_UMOUNT_WAIT            12
>>  #define DEFAULT_BROWSE_MODE            1
>>  #define DEFAULT_LOGGING                        0
>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>  struct ldap_searchdn *defaults_get_searchdns(void);
>>  void defaults_free_searchdns(struct ldap_searchdn *);
>>  unsigned int defaults_get_append_options(void);
>> +unsigned int defaults_get_mount_wait(void);
>>  unsigned int defaults_get_umount_wait(void);
>>  const char *defaults_get_auth_conf_file(void);
>>  unsigned int defaults_get_map_hash_table_size(void);
>> --- autofs-5.0.1.orig/lib/defaults.c
>> +++ autofs-5.0.1/lib/defaults.c
>> @@ -45,6 +45,7 @@
>>  #define ENV_NAME_VALUE_ATTR            "VALUE_ATTRIBUTE"
>>
>>  #define ENV_APPEND_OPTIONS             "APPEND_OPTIONS"
>> +#define ENV_MOUNT_WAIT                 "MOUNT_WAIT"
>>  #define ENV_UMOUNT_WAIT                        "UMOUNT_WAIT"
>>  #define ENV_AUTH_CONF_FILE             "AUTH_CONF_FILE"
>>
>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>                    check_set_config_value(key, ENV_NAME_ENTRY_ATTR, =
value, to_syslog) ||
>>                    check_set_config_value(key, ENV_NAME_VALUE_ATTR, =
value, to_syslog) ||
>>                    check_set_config_value(key, ENV_APPEND_OPTIONS, v=
alue, to_syslog) ||
>> +                   check_set_config_value(key, ENV_MOUNT_WAIT, valu=
e, to_syslog) ||
>>                    check_set_config_value(key, ENV_UMOUNT_WAIT, valu=
e, to_syslog) ||
>>                    check_set_config_value(key, ENV_AUTH_CONF_FILE, v=
alue, to_syslog) ||
>>                    check_set_config_value(key, ENV_MAP_HASH_TABLE_SI=
ZE, value, to_syslog))
>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>        return res;
>>  }
>>
>> +unsigned int defaults_get_mount_wait(void)
>> +{
>> +       long wait;
>> +
>> +       wait =3D get_env_number(ENV_MOUNT_WAIT);
>> +       if (wait < 0)
>> +               wait =3D DEFAULT_MOUNT_WAIT;
>> +
>> +       return (unsigned int) wait;
>> +}
>> +
>>  unsigned int defaults_get_umount_wait(void)
>>  {
>>        long wait;
>> --- autofs-5.0.1.orig/man/auto.master.5.in
>> +++ autofs-5.0.1/man/auto.master.5.in
>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>  60). If the equivalent command line option is given it will overrid=
e this
>>  setting.
>>  .TP
>> +.B MOUNT_WAIT
>> +Set the default time to wait for a response from a spawned mount(8)
>> +before sending it a SIGTERM. Note that we still need to wait for th=
e
>> +RPC layer to timeout before the sub-process exits so this isn't ide=
al
>> +but it is the best we can do. The default is to wait until mount(8)
>> +returns without intervention.
>> +.TP
>>  .B UMOUNT_WAIT
>>  Set the default time to wait for a response from a spawned umount(8=
)
>>  before sending it a SIGTERM. Note that we still need to wait for th=
e
>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>  #
>>  #NEGATIVE_TIMEOUT=3D60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=3D-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=3D12
>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>  #
>>  #NEGATIVE_TIMEOUT=3D60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=3D-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=3D12
>>
>>
>>> Thanks!
>>>
>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>> Carlos Andr=E9 wrote:
>>>>>> Today (2009-08-12) I'm using:
>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>> Thanks,
>>>>>
>>>>> My mistake, the wait time I was referring to is used for umounts =
during
>>>>> expires and is present in rev rc2.102.
>>>>>
>>>>> It shouldn't be hard to add this for mount as well.
>>>>> Would you like me to put something together?
>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks =
:)
>>>>
>>>>> Probably would be good to test something out to see if we can mak=
e a
>>>>> difference with the killing mount after some configured timeout b=
ut, if
>>>>> we make progress, probably the best way to deal with it is for yo=
u to
>>>>> log a bug against rhel-5 so I can get it committed to the rhel pa=
ckage.
>>>>> The possible issue is that I'm not sure if the RPC subsystem in t=
he
>>>>> above rhel kernel will respond well to process death with potenti=
al
>>>>> outstanding requests. But we'll see.
>>>> Ok, on my way :)
>>>>
>>>> Thanks a lot!
>>>>
>>>>>>
>>>>>> Look my last test:
>>>>>> --------------------------------------------------------------
>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>> ls: testdown: No such file or directory
>>>>>>
>>>>>> real    3m9.025s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun)=
:
>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdow=
n,
>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=
=3D0, ro=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nf=
s):
>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D=
0
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc=
 =3D
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect:=
 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got t=
hid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigch=
ld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready():=
 state
>>>>>> =3D 2 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc=
 =3D
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect:=
 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got t=
hid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigch=
ld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready():=
 state
>>>>>> =3D 2 path /misc
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NF=
S
>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: moun=
t
>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D =
17
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc=
/areas/testdown
>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 pa=
th /misc
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>> Hi Ian,
>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount... th=
is option
>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/=
krb5i/krb5p)
>>>>>>>> like you can see on my previous emails...
>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>
>>>>>>> Maybe this is related to the same sort of problem we had with m=
ount in
>>>>>>> the past, before the options parsing went into the kernel, wher=
e other
>>>>>>> services, like portmapper (or rpcbind), were being done with di=
fferent
>>>>>>> timeout parameters before the RPC calls for mounting. That's ju=
st an
>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>
>>>>>>> But what version of autofs and kernel did you say you were usin=
g?
>>>>>>>
>>>>>>>> I appreciate any help.
>>>>>>>>
>>>>>>>> Carlos.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>> This long timeout is good if workstation need mount a criti=
cal
>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any =
sense,
>>>>>>>>>>> since autofs retry mount directory on-access. This in fact =
gives me
>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one ser=
ver goes
>>>>>>>>>>> down for any reason, and will again hangs if user try acces=
s directory
>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as the=
 first
>>>>>>>>>> mount(2) system call fails.  When you set SYN retries to 1, =
this means
>>>>>>>>>> after 9 seconds, the connect fails, and that causes the moun=
t(2) system
>>>>>>>>>> call to fail.
>>>>>>>>>>
>>>>>>>>>> Recent conversations with Ian suggested that a long timeout =
was desired
>>>>>>>>>> for automounter as well as other cases.  Ian, is there somet=
hing else we
>>>>>>>>>> need to consider to determine the correct retry timeout for =
NFS/TCP
>>>>>>>>>> mount points handled via automounter?  How should mount.nfs =
wait so we
>>>>>>>>>> don't make other use cases worse?  (Looks like most of the h=
istory is
>>>>>>>>>> intact below).
>>>>>>>>> Of course we know that autofs is entirely at the mercy of mou=
nt(8) (and
>>>>>>>>> mount.nfs in particular). This has always been a difficult si=
tuation for
>>>>>>>>> the automounter because interactive mount invocations should =
wait. But I
>>>>>>>>> believe automount mounts should always time out quickly, but =
that leads
>>>>>>>>> to its own set of problems, especially when home directories =
are concerned.
>>>>>>>>>
>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself bu=
t I'm not
>>>>>>>>> certain that will work as we expect. I'll have to do some exp=
erimentation.
>>>>>>>>>
>>>>>>>>>> How long do you think is appropriate for the automounter to =
wait if the
>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>
>>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retr=
ies  [DEFAULT]
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.003s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retr=
ies [ 5 to 1 ]
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 6]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    1m3.002s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    0m9.003s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4=
 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (ret=
rying). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giv=
ing up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.001s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to=
 1... and
>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>
>>>>>>>>>>> *sigh*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_sy=
n_retries to
>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's socket =
connect
>>>>>>>>>>>> function,
>>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes lon=
ger than
>>>>>>>>>>>> the RPC
>>>>>>>>>>>> client's connect timeout, so it only makes one connect cal=
l, and then
>>>>>>>>>>>> fails.
>>>>>>>>>>>>
>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt cau=
ses the RPC
>>>>>>>>>>>> client
>>>>>>>>>>>> to retry the connect call until its connect timeout expire=
s.  Each
>>>>>>>>>>>> connect
>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nf=
s4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (g=
iving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_ret=
ries
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nf=
s4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (r=
etrying).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (g=
iving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>>
>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've m=
onitored
>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 se=
cs on port
>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any chan=
ge... I dont
>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation=
 where my
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) t=
han 3 minutes
>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the =
kernel to
>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN att=
empts with
>>>>>>>>>>>>>>> exponential retries, or something like that).  For stoc=
k CentOS
>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 moun=
ts -- the
>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no pre=
ceding rpcbind
>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-relate=
d CentOS
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yoursel=
f.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halev=
y wrote:
>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candre=
cn@gmail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 serve=
r to work with
>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goe=
s down i get a
>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client=
=2E..
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logo=
n process, if
>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to time=
out (if server
>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinatio=
ns, there my
>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1=
=2E10) using
>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t n=
fs4 -o
>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (prot=
o=3Dtcp OR
>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until s=
how error
>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving=
 up))
>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period=
=2E
>>>>>>>>>>>>>>>>> I thought he was describing a situation where the ser=
ver the server
>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wonderi=
ng how to make
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscr=
ibe
>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordo=
mo-info.html
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Ian,
Thanks for Support/Help :)


2009/8/24 Ian Kent <ikent@redhat.com>:
> Carlos Andr=E9 wrote:
>> Hi Ian,
>>
>> Thanks for patch and sorry for delay (i'm expecting receive u reply on
>> bug track, not here) :)
>>
>> But, this patch doesnt worked to me like expected... =A0:(
>>
>>
>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>> and later changed "10" to "2" with same results...
>> (always restarting service, of course :)
>>
>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
>> same results again.
>>
>> Or i'm doing something wrong?
>>
>>
>> [root@KSTATION areas]# automount -V
>>
>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>> [...]
>>
>> [root@KSTATION areas]# time ls -la testdown
>> ls: testedown: No such file or directory
>>
>> real =A0 =A03m9.006s
>> user =A0 =A00m0.002s
>> sys =A0 =A0 0m0.000s
>
> OK, that isn't behaving the way I expect, I'll have a look.
>
>>
>>
>> LOGGING:
>> -----------------------------------------
>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdown
>> /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D 91
>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/areas/t=
estdown
>> -----------------------------------------
>>
>>
>>
>>
>>
>> 2009/8/17 Ian Kent <ikent@redhat.com>:
>>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>>> Filled bug report:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>>> Hi Carlos,
>>>
>>> I have a patched source rpm to add a mount wait parameter to autofs
>>> located at:
>>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>>
>>> Could you build it and see if it works.
>>> I haven't tested it at all but it is fairly straight forward.
>>> It is still unclear if this is the right way to do this and what the
>>> consequences are in sending a term signal to mount. This mount request
>>> will likely be followed by other requests for the same mount causing an
>>> accumulation of mount(8) processes waiting for RPC timeouts before they
>>> can answer the TERM signal.
>>>
>>> Anyway, for information the patch included in the source rpm above is:
>>>
>>> autofs-5.0.4 - add mount wait parameter
>>>
>>> From: Ian Kent <raven@themaw.net>
>>>
>>> Often delays when trying to mount from a server that is not reponding
>>> for some reason are undesirable. To try and prevent these delays we
>>> provide a configuration setting to limit the time that we wait for
>>> our spawned mount(8) process to complete before sending it a SIGTERM
>>> signal. This patch adds a configuration parameter to allow us to
>>> request we limit the time we wait for mount(8) to complete before
>>> send it a TERM signal.
>>> ---
>>>
>>> =A0daemon/spawn.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A03 ++-
>>> =A0include/defaults.h =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A02 ++
>>> =A0lib/defaults.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 13 +++++++++++++
>>> =A0man/auto.master.5.in =A0 =A0 =A0 =A0 =A0 | =A0 =A07 +++++++
>>> =A0redhat/autofs.sysconfig.in =A0 =A0 | =A0 =A09 +++++++++
>>> =A0samples/autofs.conf.default.in | =A0 =A09 +++++++++
>>> =A06 files changed, 42 insertions(+), 1 deletion(-)
>>>
>>>
>>> --- autofs-5.0.1.orig/daemon/spawn.c
>>> +++ autofs-5.0.1/daemon/spawn.c
>>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>> =A0 =A0 =A0 =A0unsigned int options;
>>> =A0 =A0 =A0 =A0unsigned int retries =3D MTAB_LOCK_RETRIES;
>>> =A0 =A0 =A0 =A0int update_mtab =3D 1, ret, printed =3D 0;
>>> + =A0 =A0 =A0 unsigned int wait =3D defaults_get_mount_wait();
>>> =A0 =A0 =A0 =A0char buf[PATH_MAX];
>>>
>>> =A0 =A0 =A0 =A0/* If we use mount locking we can't validate the locatio=
n */
>>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>> =A0 =A0 =A0 =A0va_end(arg);
>>>
>>> =A0 =A0 =A0 =A0while (retries--) {
>>> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, -1, options, pro=
g, (const char **) argv);
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, wait, options, p=
rog, (const char **) argv);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (ret & MTAB_NOTUPDATED) {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct timespec tm =3D {=
3, 0};
>>>
>>> --- autofs-5.0.1.orig/include/defaults.h
>>> +++ autofs-5.0.1/include/defaults.h
>>> @@ -24,6 +24,7 @@
>>>
>>> =A0#define DEFAULT_TIMEOUT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0600
>>> =A0#define DEFAULT_NEGATIVE_TIMEOUT =A0 =A0 =A0 60
>>> +#define DEFAULT_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 -1
>>> =A0#define DEFAULT_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A012
>>> =A0#define DEFAULT_BROWSE_MODE =A0 =A0 =A0 =A0 =A0 =A01
>>> =A0#define DEFAULT_LOGGING =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A00
>>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>> =A0struct ldap_searchdn *defaults_get_searchdns(void);
>>> =A0void defaults_free_searchdns(struct ldap_searchdn *);
>>> =A0unsigned int defaults_get_append_options(void);
>>> +unsigned int defaults_get_mount_wait(void);
>>> =A0unsigned int defaults_get_umount_wait(void);
>>> =A0const char *defaults_get_auth_conf_file(void);
>>> =A0unsigned int defaults_get_map_hash_table_size(void);
>>> --- autofs-5.0.1.orig/lib/defaults.c
>>> +++ autofs-5.0.1/lib/defaults.c
>>> @@ -45,6 +45,7 @@
>>> =A0#define ENV_NAME_VALUE_ATTR =A0 =A0 =A0 =A0 =A0 =A0"VALUE_ATTRIBUTE"
>>>
>>> =A0#define ENV_APPEND_OPTIONS =A0 =A0 =A0 =A0 =A0 =A0 "APPEND_OPTIONS"
>>> +#define ENV_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 "MOUNT_WAIT"
>>> =A0#define ENV_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0"UMOUNT_WAIT"
>>> =A0#define ENV_AUTH_CONF_FILE =A0 =A0 =A0 =A0 =A0 =A0 "AUTH_CONF_FILE"
>>>
>>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
NAME_ENTRY_ATTR, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
NAME_VALUE_ATTR, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
APPEND_OPTIONS, value, to_syslog) ||
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 check_set_config_value(key, ENV_M=
OUNT_WAIT, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
UMOUNT_WAIT, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
AUTH_CONF_FILE, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, ENV_=
MAP_HASH_TABLE_SIZE, value, to_syslog))
>>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>> =A0 =A0 =A0 =A0return res;
>>> =A0}
>>>
>>> +unsigned int defaults_get_mount_wait(void)
>>> +{
>>> + =A0 =A0 =A0 long wait;
>>> +
>>> + =A0 =A0 =A0 wait =3D get_env_number(ENV_MOUNT_WAIT);
>>> + =A0 =A0 =A0 if (wait < 0)
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 wait =3D DEFAULT_MOUNT_WAIT;
>>> +
>>> + =A0 =A0 =A0 return (unsigned int) wait;
>>> +}
>>> +
>>> =A0unsigned int defaults_get_umount_wait(void)
>>> =A0{
>>> =A0 =A0 =A0 =A0long wait;
>>> --- autofs-5.0.1.orig/man/auto.master.5.in
>>> +++ autofs-5.0.1/man/auto.master.5.in
>>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>> =A060). If the equivalent command line option is given it will override=
 this
>>> =A0setting.
>>> =A0.TP
>>> +.B MOUNT_WAIT
>>> +Set the default time to wait for a response from a spawned mount(8)
>>> +before sending it a SIGTERM. Note that we still need to wait for the
>>> +RPC layer to timeout before the sub-process exits so this isn't ideal
>>> +but it is the best we can do. The default is to wait until mount(8)
>>> +returns without intervention.
>>> +.TP
>>> =A0.B UMOUNT_WAIT
>>> =A0Set the default time to wait for a response from a spawned umount(8)
>>> =A0before sending it a SIGTERM. Note that we still need to wait for the
>>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>> =A0#
>>> =A0#NEGATIVE_TIMEOUT=3D60
>>> =A0#
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems when
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server that
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when it's
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for mount=
(8)
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 minute=
s.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
>>> =A0#
>>> =A0#UMOUNT_WAIT=3D12
>>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>> =A0#
>>> =A0#NEGATIVE_TIMEOUT=3D60
>>> =A0#
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems when
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server that
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when it's
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for mount=
(8)
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 minute=
s.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
>>> =A0#
>>> =A0#UMOUNT_WAIT=3D12
>>>
>>>
>>>> Thanks!
>>>>
>>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>>> Carlos Andr=E9 wrote:
>>>>>>> Today (2009-08-12) I'm using:
>>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>>> Thanks,
>>>>>>
>>>>>> My mistake, the wait time I was referring to is used for umounts dur=
ing
>>>>>> expires and is present in rev rc2.102.
>>>>>>
>>>>>> It shouldn't be hard to add this for mount as well.
>>>>>> Would you like me to put something together?
>>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>>>>>
>>>>>> Probably would be good to test something out to see if we can make a
>>>>>> difference with the killing mount after some configured timeout but,=
 if
>>>>>> we make progress, probably the best way to deal with it is for you to
>>>>>> log a bug against rhel-5 so I can get it committed to the rhel packa=
ge.
>>>>>> The possible issue is that I'm not sure if the RPC subsystem in the
>>>>>> above rhel kernel will respond well to process death with potential
>>>>>> outstanding requests. But we'll see.
>>>>> Ok, on my way :)
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>>>>
>>>>>>> Look my last test:
>>>>>>> --------------------------------------------------------------
>>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>>> ls: testdown: No such file or directory
>>>>>>>
>>>>>>> real =A0 =A03m9.025s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D=
0, ro=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
=3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): st=
ate
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
=3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): st=
ate
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/ar=
eas/testdown
>>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> --------------------------------------------------------------
>>>>>>>
>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>>> Hi Ian,
>>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount... this =
option
>>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb=
5i/krb5p)
>>>>>>>>> like you can see on my previous emails...
>>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>>
>>>>>>>> Maybe this is related to the same sort of problem we had with moun=
t in
>>>>>>>> the past, before the options parsing went into the kernel, where o=
ther
>>>>>>>> services, like portmapper (or rpcbind), were being done with diffe=
rent
>>>>>>>> timeout parameters before the RPC calls for mounting. That's just =
an
>>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>>
>>>>>>>> But what version of autofs and kernel did you say you were using?
>>>>>>>>
>>>>>>>>> I appreciate any help.
>>>>>>>>>
>>>>>>>>> Carlos.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any sen=
se,
>>>>>>>>>>>> since autofs retry mount directory on-access. This in fact giv=
es me
>>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one server=
 goes
>>>>>>>>>>>> down for any reason, and will again hangs if user try access d=
irectory
>>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as the fi=
rst
>>>>>>>>>>> mount(2) system call fails. =A0When you set SYN retries to 1, t=
his means
>>>>>>>>>>> after 9 seconds, the connect fails, and that causes the mount(2=
) system
>>>>>>>>>>> call to fail.
>>>>>>>>>>>
>>>>>>>>>>> Recent conversations with Ian suggested that a long timeout was=
 desired
>>>>>>>>>>> for automounter as well as other cases. =A0Ian, is there someth=
ing else we
>>>>>>>>>>> need to consider to determine the correct retry timeout for NFS=
/TCP
>>>>>>>>>>> mount points handled via automounter? =A0How should mount.nfs w=
ait so we
>>>>>>>>>>> don't make other use cases worse? =A0(Looks like most of the hi=
story is
>>>>>>>>>>> intact below).
>>>>>>>>>> Of course we know that autofs is entirely at the mercy of mount(=
8) (and
>>>>>>>>>> mount.nfs in particular). This has always been a difficult situa=
tion for
>>>>>>>>>> the automounter because interactive mount invocations should wai=
t. But I
>>>>>>>>>> believe automount mounts should always time out quickly, but tha=
t leads
>>>>>>>>>> to its own set of problems, especially when home directories are=
 concerned.
>>>>>>>>>>
>>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself but I=
'm not
>>>>>>>>>> certain that will work as we expect. I'll have to do some experi=
mentation.
>>>>>>>>>>
>>>>>>>>>>> How long do you think is appropriate for the automounter to wai=
t if the
>>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>>
>>>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries=
 =A0[DEFAULT]
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>> user =A0 =A00m0.002s
>>>>>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.001s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.003s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.001s
>>>>>>>>>>>> user =A0 =A00m0.002s
>>>>>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries=
 [ 5 to 1 ]
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 6]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A01m3.002s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A02m6.000s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A00m9.003s
>>>>>>>>>>>> user =A0 =A00m0.001s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A02m6.001s
>>>>>>>>>>>> user =A0 =A00m0.001s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1.=
.. and
>>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>>
>>>>>>>>>>>> *sigh*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_r=
etries to
>>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>>> Right. =A0Normally the RPC client calls the kernel's socket c=
onnect
>>>>>>>>>>>>> function,
>>>>>>>>>>>>> which does 6 SYN retries. =A0That one call usually takes long=
er than
>>>>>>>>>>>>> the RPC
>>>>>>>>>>>>> client's connect timeout, so it only makes one connect call, =
and then
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt causes=
 the RPC
>>>>>>>>>>>>> client
>>>>>>>>>>>>> to retry the connect call until its connect timeout expires. =
=A0Each
>>>>>>>>>>>>> connect
>>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 =
-o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givi=
ng up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 =
-o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givi=
ng up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real =A0 =A02m6.004s
>>>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>>>> sys =A0 =A0 0m0.004s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've moni=
tored
>>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs =
on port
>>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any change.=
.. I dont
>>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situation =
where my
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than=
 3 minutes
>>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the ker=
nel to
>>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attemp=
ts with
>>>>>>>>>>>>>>>> exponential retries, or something like that). =A0For stock=
 CentOS
>>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts =
-- the
>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preced=
ing rpcbind
>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related C=
entOS
>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy w=
rote:
>>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@=
gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server t=
o work with
>>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes d=
own i get a
>>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon p=
rocess, if
>>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout=
 (if server
>>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations,=
 there my
>>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10=
) using
>>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4=
 -o
>>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=
=3Dtcp OR
>>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0until =
show error
>>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up=
))
>>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>>>>>> I thought he was describing a situation where the server=
 the server
>>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering =
how to make
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majordom=
o-info.html
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Lever
>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Ian Kent wrote:
> Carlos Andr=E9 wrote:
>> Hi Ian,
>>
>> Thanks for patch and sorry for delay (i'm expecting receive u reply on
>> bug track, not here) :)
>>
>> But, this patch doesnt worked to me like expected...  :(
>>
>>
>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>> and later changed "10" to "2" with same results...
>> (always restarting service, of course :)
>>
>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
>> same results again.
>>
>> Or i'm doing something wrong?
>>
>>
>> [root@KSTATION areas]# automount -V
>>
>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>> [...]
>>
>> [root@KSTATION areas]# time ls -la testdown
>> ls: testedown: No such file or directory
>>
>> real    3m9.006s
>> user    0m0.002s
>> sys     0m0.000s
> =

> OK, that isn't behaving the way I expect, I'll have a look.
> =

>>
>> LOGGING:
>> -----------------------------------------
>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdown
>> /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D 91
>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/areas/t=
estdown
>> -----------------------------------------

Having a look at this I suspect the reason it doesn't work as expected
is the waitpid(2) we do after sending the TERM signal to the mount
process (which we have to do) is not returning. This is likely because
the mount process isn't giving up in a shorter time as it used to. We
could send a KILL signal to the mount process but that does seem to
cause problems later on since there are still outstanding RPC requests.

I suspect that the early termination of blocked umount request will also
now be broken now.

Not sure what to do next here.
Anyone want to volunteer some indepth detail on kernel RPC request
termination on the issuing process receiving a TERM signal?

>>
>>
>>
>>
>>
>> 2009/8/17 Ian Kent <ikent@redhat.com>:
>>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>>> Filled bug report:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>>> Hi Carlos,
>>>
>>> I have a patched source rpm to add a mount wait parameter to autofs
>>> located at:
>>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>>
>>> Could you build it and see if it works.
>>> I haven't tested it at all but it is fairly straight forward.
>>> It is still unclear if this is the right way to do this and what the
>>> consequences are in sending a term signal to mount. This mount request
>>> will likely be followed by other requests for the same mount causing an
>>> accumulation of mount(8) processes waiting for RPC timeouts before they
>>> can answer the TERM signal.
>>>
>>> Anyway, for information the patch included in the source rpm above is:
>>>
>>> autofs-5.0.4 - add mount wait parameter
>>>
>>> From: Ian Kent <raven@themaw.net>
>>>
>>> Often delays when trying to mount from a server that is not reponding
>>> for some reason are undesirable. To try and prevent these delays we
>>> provide a configuration setting to limit the time that we wait for
>>> our spawned mount(8) process to complete before sending it a SIGTERM
>>> signal. This patch adds a configuration parameter to allow us to
>>> request we limit the time we wait for mount(8) to complete before
>>> send it a TERM signal.
>>> ---
>>>
>>>  daemon/spawn.c                 |    3 ++-
>>>  include/defaults.h             |    2 ++
>>>  lib/defaults.c                 |   13 +++++++++++++
>>>  man/auto.master.5.in           |    7 +++++++
>>>  redhat/autofs.sysconfig.in     |    9 +++++++++
>>>  samples/autofs.conf.default.in |    9 +++++++++
>>>  6 files changed, 42 insertions(+), 1 deletion(-)
>>>
>>>
>>> --- autofs-5.0.1.orig/daemon/spawn.c
>>> +++ autofs-5.0.1/daemon/spawn.c
>>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>>        unsigned int options;
>>>        unsigned int retries =3D MTAB_LOCK_RETRIES;
>>>        int update_mtab =3D 1, ret, printed =3D 0;
>>> +       unsigned int wait =3D defaults_get_mount_wait();
>>>        char buf[PATH_MAX];
>>>
>>>        /* If we use mount locking we can't validate the location */
>>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>>        va_end(arg);
>>>
>>>        while (retries--) {
>>> -               ret =3D do_spawn(logopt, -1, options, prog, (const char=
 **) argv);
>>> +               ret =3D do_spawn(logopt, wait, options, prog, (const ch=
ar **) argv);
>>>                if (ret & MTAB_NOTUPDATED) {
>>>                        struct timespec tm =3D {3, 0};
>>>
>>> --- autofs-5.0.1.orig/include/defaults.h
>>> +++ autofs-5.0.1/include/defaults.h
>>> @@ -24,6 +24,7 @@
>>>
>>>  #define DEFAULT_TIMEOUT                        600
>>>  #define DEFAULT_NEGATIVE_TIMEOUT       60
>>> +#define DEFAULT_MOUNT_WAIT             -1
>>>  #define DEFAULT_UMOUNT_WAIT            12
>>>  #define DEFAULT_BROWSE_MODE            1
>>>  #define DEFAULT_LOGGING                        0
>>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>>  struct ldap_searchdn *defaults_get_searchdns(void);
>>>  void defaults_free_searchdns(struct ldap_searchdn *);
>>>  unsigned int defaults_get_append_options(void);
>>> +unsigned int defaults_get_mount_wait(void);
>>>  unsigned int defaults_get_umount_wait(void);
>>>  const char *defaults_get_auth_conf_file(void);
>>>  unsigned int defaults_get_map_hash_table_size(void);
>>> --- autofs-5.0.1.orig/lib/defaults.c
>>> +++ autofs-5.0.1/lib/defaults.c
>>> @@ -45,6 +45,7 @@
>>>  #define ENV_NAME_VALUE_ATTR            "VALUE_ATTRIBUTE"
>>>
>>>  #define ENV_APPEND_OPTIONS             "APPEND_OPTIONS"
>>> +#define ENV_MOUNT_WAIT                 "MOUNT_WAIT"
>>>  #define ENV_UMOUNT_WAIT                        "UMOUNT_WAIT"
>>>  #define ENV_AUTH_CONF_FILE             "AUTH_CONF_FILE"
>>>
>>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>>                    check_set_config_value(key, ENV_NAME_ENTRY_ATTR, val=
ue, to_syslog) ||
>>>                    check_set_config_value(key, ENV_NAME_VALUE_ATTR, val=
ue, to_syslog) ||
>>>                    check_set_config_value(key, ENV_APPEND_OPTIONS, valu=
e, to_syslog) ||
>>> +                   check_set_config_value(key, ENV_MOUNT_WAIT, value, =
to_syslog) ||
>>>                    check_set_config_value(key, ENV_UMOUNT_WAIT, value, =
to_syslog) ||
>>>                    check_set_config_value(key, ENV_AUTH_CONF_FILE, valu=
e, to_syslog) ||
>>>                    check_set_config_value(key, ENV_MAP_HASH_TABLE_SIZE,=
 value, to_syslog))
>>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>>        return res;
>>>  }
>>>
>>> +unsigned int defaults_get_mount_wait(void)
>>> +{
>>> +       long wait;
>>> +
>>> +       wait =3D get_env_number(ENV_MOUNT_WAIT);
>>> +       if (wait < 0)
>>> +               wait =3D DEFAULT_MOUNT_WAIT;
>>> +
>>> +       return (unsigned int) wait;
>>> +}
>>> +
>>>  unsigned int defaults_get_umount_wait(void)
>>>  {
>>>        long wait;
>>> --- autofs-5.0.1.orig/man/auto.master.5.in
>>> +++ autofs-5.0.1/man/auto.master.5.in
>>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>>  60). If the equivalent command line option is given it will override t=
his
>>>  setting.
>>>  .TP
>>> +.B MOUNT_WAIT
>>> +Set the default time to wait for a response from a spawned mount(8)
>>> +before sending it a SIGTERM. Note that we still need to wait for the
>>> +RPC layer to timeout before the sub-process exits so this isn't ideal
>>> +but it is the best we can do. The default is to wait until mount(8)
>>> +returns without intervention.
>>> +.TP
>>>  .B UMOUNT_WAIT
>>>  Set the default time to wait for a response from a spawned umount(8)
>>>  before sending it a SIGTERM. Note that we still need to wait for the
>>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>>  #
>>>  #NEGATIVE_TIMEOUT=3D60
>>>  #
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +#             Setting this timeout can cause problems when
>>> +#             mount would otherwise wait for a server that
>>> +#             is temporarily unavailable, such as when it's
>>> +#             restarting. The defailt of waiting for mount(8)
>>> +#             usually results in a wait of around 3 minutes.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>>  #
>>>  #UMOUNT_WAIT=3D12
>>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>>  #
>>>  #NEGATIVE_TIMEOUT=3D60
>>>  #
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +#             Setting this timeout can cause problems when
>>> +#             mount would otherwise wait for a server that
>>> +#             is temporarily unavailable, such as when it's
>>> +#             restarting. The defailt of waiting for mount(8)
>>> +#             usually results in a wait of around 3 minutes.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>>  #
>>>  #UMOUNT_WAIT=3D12
>>>
>>>
>>>> Thanks!
>>>>
>>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>>> Carlos Andr=E9 wrote:
>>>>>>> Today (2009-08-12) I'm using:
>>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>>> Thanks,
>>>>>>
>>>>>> My mistake, the wait time I was referring to is used for umounts dur=
ing
>>>>>> expires and is present in rev rc2.102.
>>>>>>
>>>>>> It shouldn't be hard to add this for mount as well.
>>>>>> Would you like me to put something together?
>>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>>>>>
>>>>>> Probably would be good to test something out to see if we can make a
>>>>>> difference with the killing mount after some configured timeout but,=
 if
>>>>>> we make progress, probably the best way to deal with it is for you to
>>>>>> log a bug against rhel-5 so I can get it committed to the rhel packa=
ge.
>>>>>> The possible issue is that I'm not sure if the RPC subsystem in the
>>>>>> above rhel kernel will respond well to process death with potential
>>>>>> outstanding requests. But we'll see.
>>>>> Ok, on my way :)
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>>>> Look my last test:
>>>>>>> --------------------------------------------------------------
>>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>>> ls: testdown: No such file or directory
>>>>>>>
>>>>>>> real    3m9.025s
>>>>>>> user    0m0.000s
>>>>>>> sys     0m0.002s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D=
0, ro=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =
=3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): st=
ate
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =
=3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): st=
ate
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/ar=
eas/testdown
>>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path =
/misc
>>>>>>> --------------------------------------------------------------
>>>>>>>
>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>>> Hi Ian,
>>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount... this =
option
>>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb=
5i/krb5p)
>>>>>>>>> like you can see on my previous emails...
>>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>>
>>>>>>>> Maybe this is related to the same sort of problem we had with moun=
t in
>>>>>>>> the past, before the options parsing went into the kernel, where o=
ther
>>>>>>>> services, like portmapper (or rpcbind), were being done with diffe=
rent
>>>>>>>> timeout parameters before the RPC calls for mounting. That's just =
an
>>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>>
>>>>>>>> But what version of autofs and kernel did you say you were using?
>>>>>>>>
>>>>>>>>> I appreciate any help.
>>>>>>>>>
>>>>>>>>> Carlos.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any sen=
se,
>>>>>>>>>>>> since autofs retry mount directory on-access. This in fact giv=
es me
>>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one server=
 goes
>>>>>>>>>>>> down for any reason, and will again hangs if user try access d=
irectory
>>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as the fi=
rst
>>>>>>>>>>> mount(2) system call fails.  When you set SYN retries to 1, thi=
s means
>>>>>>>>>>> after 9 seconds, the connect fails, and that causes the mount(2=
) system
>>>>>>>>>>> call to fail.
>>>>>>>>>>>
>>>>>>>>>>> Recent conversations with Ian suggested that a long timeout was=
 desired
>>>>>>>>>>> for automounter as well as other cases.  Ian, is there somethin=
g else we
>>>>>>>>>>> need to consider to determine the correct retry timeout for NFS=
/TCP
>>>>>>>>>>> mount points handled via automounter?  How should mount.nfs wai=
t so we
>>>>>>>>>>> don't make other use cases worse?  (Looks like most of the hist=
ory is
>>>>>>>>>>> intact below).
>>>>>>>>>> Of course we know that autofs is entirely at the mercy of mount(=
8) (and
>>>>>>>>>> mount.nfs in particular). This has always been a difficult situa=
tion for
>>>>>>>>>> the automounter because interactive mount invocations should wai=
t. But I
>>>>>>>>>> believe automount mounts should always time out quickly, but tha=
t leads
>>>>>>>>>> to its own set of problems, especially when home directories are=
 concerned.
>>>>>>>>>>
>>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself but I=
'm not
>>>>>>>>>> certain that will work as we expect. I'll have to do some experi=
mentation.
>>>>>>>>>>
>>>>>>>>>>> How long do you think is appropriate for the automounter to wai=
t if the
>>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>>
>>>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries=
  [DEFAULT]
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.003s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries=
 [ 5 to 1 ]
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 6]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    1m3.002s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    2m6.000s
>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    0m9.003s
>>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryi=
ng). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving=
 up).
>>>>>>>>>>>>
>>>>>>>>>>>> real    2m6.001s
>>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1.=
.. and
>>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>>
>>>>>>>>>>>> *sigh*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_r=
etries to
>>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's socket con=
nect
>>>>>>>>>>>>> function,
>>>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes longer=
 than
>>>>>>>>>>>>> the RPC
>>>>>>>>>>>>> client's connect timeout, so it only makes one connect call, =
and then
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt causes=
 the RPC
>>>>>>>>>>>>> client
>>>>>>>>>>>>> to retry the connect call until its connect timeout expires. =
 Each
>>>>>>>>>>>>> connect
>>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 =
-o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givi=
ng up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 =
-o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retr=
ying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givi=
ng up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've moni=
tored
>>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs =
on port
>>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any change.=
.. I dont
>>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation wh=
ere my
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than=
 3 minutes
>>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the ker=
nel to
>>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attemp=
ts with
>>>>>>>>>>>>>>>> exponential retries, or something like that).  For stock C=
entOS
>>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts =
-- the
>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no preced=
ing rpcbind
>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related C=
entOS
>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy w=
rote:
>>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@=
gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server t=
o work with
>>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes d=
own i get a
>>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon p=
rocess, if
>>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout=
 (if server
>>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations,=
 there my
>>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10=
) using
>>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4=
 -o
>>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=
=3Dtcp OR
>>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show=
 error
>>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up=
))
>>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>>>>>> I thought he was describing a situation where the server=
 the server
>>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering =
how to make
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-=
info.html
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Lever
>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
> =

> =

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 27, 2009, at 4:54 AM, Ian Kent wrote:
> Ian Kent wrote:
>> Carlos Andr=E9 wrote:
>>> Hi Ian,
>>>
>>> Thanks for patch and sorry for delay (i'm expecting receive u  =

>>> reply on
>>> bug track, not here) :)
>>>
>>> But, this patch doesnt worked to me like expected...  :(
>>>
>>>
>>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>>> and later changed "10" to "2" with same results...
>>> (always restarting service, of course :)
>>>
>>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
>>> same results again.
>>>
>>> Or i'm doing something wrong?
>>>
>>>
>>> [root@KSTATION areas]# automount -V
>>>
>>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>>> [...]
>>>
>>> [root@KSTATION areas]# time ls -la testdown
>>> ls: testedown: No such file or directory
>>>
>>> real    3m9.006s
>>> user    0m0.002s
>>> sys     0m0.000s
>>
>> OK, that isn't behaving the way I expect, I'll have a look.
>>
>>>
>>> LOGGING:
>>> -----------------------------------------
>>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
>>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdown
>>> /misc/areas/testdown
>>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token  =

>>> =3D 91
>>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/ =

>>> areas/testdown
>>> -----------------------------------------
>
> Having a look at this I suspect the reason it doesn't work as expected
> is the waitpid(2) we do after sending the TERM signal to the mount
> process (which we have to do) is not returning. This is likely because
> the mount process isn't giving up in a shorter time as it used to.

You're thinking maybe mount(2) should be as interruptible as the  =

socket calls that the mount command used to do?  That might be  =

reasonable, and I can take a look at that.

In the kernel, if the rpcbind for the MNT request is async, that would  =

be done by rpciod.  That's a different process, so the signal wouldn't  =

have any effect on the mount.  I have a patch that converts the MNT  =

client to use rpcb_getport_sync() which might help in this case.

> We could send a KILL signal to the mount process but that does seem to
> cause problems later on since there are still outstanding RPC  =

> requests.
>
> I suspect that the early termination of blocked umount request will  =

> also
> now be broken now.

The network part of umount.nfs is still done in user space, just like  =

it used to be.  Worth checking, but I can't see that being a problem.

> Not sure what to do next here.
> Anyone want to volunteer some indepth detail on kernel RPC request
> termination on the issuing process receiving a TERM signal?
>
>>> 2009/8/17 Ian Kent <ikent@redhat.com>:
>>>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>>>> Filled bug report:
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>>>> Hi Carlos,
>>>>
>>>> I have a patched source rpm to add a mount wait parameter to autofs
>>>> located at:
>>>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>>>
>>>> Could you build it and see if it works.
>>>> I haven't tested it at all but it is fairly straight forward.
>>>> It is still unclear if this is the right way to do this and what  =

>>>> the
>>>> consequences are in sending a term signal to mount. This mount  =

>>>> request
>>>> will likely be followed by other requests for the same mount  =

>>>> causing an
>>>> accumulation of mount(8) processes waiting for RPC timeouts  =

>>>> before they
>>>> can answer the TERM signal.
>>>>
>>>> Anyway, for information the patch included in the source rpm  =

>>>> above is:
>>>>
>>>> autofs-5.0.4 - add mount wait parameter
>>>>
>>>> From: Ian Kent <raven@themaw.net>
>>>>
>>>> Often delays when trying to mount from a server that is not  =

>>>> reponding
>>>> for some reason are undesirable. To try and prevent these delays we
>>>> provide a configuration setting to limit the time that we wait for
>>>> our spawned mount(8) process to complete before sending it a  =

>>>> SIGTERM
>>>> signal. This patch adds a configuration parameter to allow us to
>>>> request we limit the time we wait for mount(8) to complete before
>>>> send it a TERM signal.
>>>> ---
>>>>
>>>> daemon/spawn.c                 |    3 ++-
>>>> include/defaults.h             |    2 ++
>>>> lib/defaults.c                 |   13 +++++++++++++
>>>> man/auto.master.5.in           |    7 +++++++
>>>> redhat/autofs.sysconfig.in     |    9 +++++++++
>>>> samples/autofs.conf.default.in |    9 +++++++++
>>>> 6 files changed, 42 insertions(+), 1 deletion(-)
>>>>
>>>>
>>>> --- autofs-5.0.1.orig/daemon/spawn.c
>>>> +++ autofs-5.0.1/daemon/spawn.c
>>>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>>>       unsigned int options;
>>>>       unsigned int retries =3D MTAB_LOCK_RETRIES;
>>>>       int update_mtab =3D 1, ret, printed =3D 0;
>>>> +       unsigned int wait =3D defaults_get_mount_wait();
>>>>       char buf[PATH_MAX];
>>>>
>>>>       /* If we use mount locking we can't validate the location */
>>>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>>>       va_end(arg);
>>>>
>>>>       while (retries--) {
>>>> -               ret =3D do_spawn(logopt, -1, options, prog, (const  =

>>>> char **) argv);
>>>> +               ret =3D do_spawn(logopt, wait, options, prog,  =

>>>> (const char **) argv);
>>>>               if (ret & MTAB_NOTUPDATED) {
>>>>                       struct timespec tm =3D {3, 0};
>>>>
>>>> --- autofs-5.0.1.orig/include/defaults.h
>>>> +++ autofs-5.0.1/include/defaults.h
>>>> @@ -24,6 +24,7 @@
>>>>
>>>> #define DEFAULT_TIMEOUT                        600
>>>> #define DEFAULT_NEGATIVE_TIMEOUT       60
>>>> +#define DEFAULT_MOUNT_WAIT             -1
>>>> #define DEFAULT_UMOUNT_WAIT            12
>>>> #define DEFAULT_BROWSE_MODE            1
>>>> #define DEFAULT_LOGGING                        0
>>>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>>> struct ldap_searchdn *defaults_get_searchdns(void);
>>>> void defaults_free_searchdns(struct ldap_searchdn *);
>>>> unsigned int defaults_get_append_options(void);
>>>> +unsigned int defaults_get_mount_wait(void);
>>>> unsigned int defaults_get_umount_wait(void);
>>>> const char *defaults_get_auth_conf_file(void);
>>>> unsigned int defaults_get_map_hash_table_size(void);
>>>> --- autofs-5.0.1.orig/lib/defaults.c
>>>> +++ autofs-5.0.1/lib/defaults.c
>>>> @@ -45,6 +45,7 @@
>>>> #define ENV_NAME_VALUE_ATTR            "VALUE_ATTRIBUTE"
>>>>
>>>> #define ENV_APPEND_OPTIONS             "APPEND_OPTIONS"
>>>> +#define ENV_MOUNT_WAIT                 "MOUNT_WAIT"
>>>> #define ENV_UMOUNT_WAIT                        "UMOUNT_WAIT"
>>>> #define ENV_AUTH_CONF_FILE             "AUTH_CONF_FILE"
>>>>
>>>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>>>                   check_set_config_value(key,  =

>>>> ENV_NAME_ENTRY_ATTR, value, to_syslog) ||
>>>>                   check_set_config_value(key,  =

>>>> ENV_NAME_VALUE_ATTR, value, to_syslog) ||
>>>>                   check_set_config_value(key, ENV_APPEND_OPTIONS,  =

>>>> value, to_syslog) ||
>>>> +                   check_set_config_value(key, ENV_MOUNT_WAIT,  =

>>>> value, to_syslog) ||
>>>>                   check_set_config_value(key, ENV_UMOUNT_WAIT,  =

>>>> value, to_syslog) ||
>>>>                   check_set_config_value(key, ENV_AUTH_CONF_FILE,  =

>>>> value, to_syslog) ||
>>>>                   check_set_config_value(key,  =

>>>> ENV_MAP_HASH_TABLE_SIZE, value, to_syslog))
>>>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>>>       return res;
>>>> }
>>>>
>>>> +unsigned int defaults_get_mount_wait(void)
>>>> +{
>>>> +       long wait;
>>>> +
>>>> +       wait =3D get_env_number(ENV_MOUNT_WAIT);
>>>> +       if (wait < 0)
>>>> +               wait =3D DEFAULT_MOUNT_WAIT;
>>>> +
>>>> +       return (unsigned int) wait;
>>>> +}
>>>> +
>>>> unsigned int defaults_get_umount_wait(void)
>>>> {
>>>>       long wait;
>>>> --- autofs-5.0.1.orig/man/auto.master.5.in
>>>> +++ autofs-5.0.1/man/auto.master.5.in
>>>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>>> 60). If the equivalent command line option is given it will  =

>>>> override this
>>>> setting.
>>>> .TP
>>>> +.B MOUNT_WAIT
>>>> +Set the default time to wait for a response from a spawned  =

>>>> mount(8)
>>>> +before sending it a SIGTERM. Note that we still need to wait for  =

>>>> the
>>>> +RPC layer to timeout before the sub-process exits so this isn't  =

>>>> ideal
>>>> +but it is the best we can do. The default is to wait until  =

>>>> mount(8)
>>>> +returns without intervention.
>>>> +.TP
>>>> .B UMOUNT_WAIT
>>>> Set the default time to wait for a response from a spawned  =

>>>> umount(8)
>>>> before sending it a SIGTERM. Note that we still need to wait for  =

>>>> the
>>>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>>>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>>> #
>>>> #NEGATIVE_TIMEOUT=3D60
>>>> #
>>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>>> +#             Setting this timeout can cause problems when
>>>> +#             mount would otherwise wait for a server that
>>>> +#             is temporarily unavailable, such as when it's
>>>> +#             restarting. The defailt of waiting for mount(8)
>>>> +#             usually results in a wait of around 3 minutes.
>>>> +#
>>>> +#MOUNT_WAIT=3D-1
>>>> +#
>>>> # UMOUNT_WAIT - time to wait for a response from umount(8).
>>>> #
>>>> #UMOUNT_WAIT=3D12
>>>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>>>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>>> #
>>>> #NEGATIVE_TIMEOUT=3D60
>>>> #
>>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>>> +#             Setting this timeout can cause problems when
>>>> +#             mount would otherwise wait for a server that
>>>> +#             is temporarily unavailable, such as when it's
>>>> +#             restarting. The defailt of waiting for mount(8)
>>>> +#             usually results in a wait of around 3 minutes.
>>>> +#
>>>> +#MOUNT_WAIT=3D-1
>>>> +#
>>>> # UMOUNT_WAIT - time to wait for a response from umount(8).
>>>> #
>>>> #UMOUNT_WAIT=3D12
>>>>
>>>>
>>>>> Thanks!
>>>>>
>>>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>> Today (2009-08-12) I'm using:
>>>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>>>> Thanks,
>>>>>>>
>>>>>>> My mistake, the wait time I was referring to is used for  =

>>>>>>> umounts during
>>>>>>> expires and is present in rev rc2.102.
>>>>>>>
>>>>>>> It shouldn't be hard to add this for mount as well.
>>>>>>> Would you like me to put something together?
>>>>>> Sure! that 'll help me a lot (and for sure another ppl) :)  =

>>>>>> Thanks :)
>>>>>>
>>>>>>> Probably would be good to test something out to see if we can  =

>>>>>>> make a
>>>>>>> difference with the killing mount after some configured  =

>>>>>>> timeout but, if
>>>>>>> we make progress, probably the best way to deal with it is for  =

>>>>>>> you to
>>>>>>> log a bug against rhel-5 so I can get it committed to the rhel  =

>>>>>>> package.
>>>>>>> The possible issue is that I'm not sure if the RPC subsystem  =

>>>>>>> in the
>>>>>>> above rhel kernel will respond well to process death with  =

>>>>>>> potential
>>>>>>> outstanding requests. But we'll see.
>>>>>> Ok, on my way :)
>>>>>>
>>>>>> Thanks a lot!
>>>>>>
>>>>>>>> Look my last test:
>>>>>>>> --------------------------------------------------------------
>>>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>>>> ls: testdown: No such file or directory
>>>>>>>>
>>>>>>>> real    3m9.025s
>>>>>>>> user    0m0.000s
>>>>>>>> sys     0m0.002s
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount:  =

>>>>>>>> parse(sun):
>>>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  =

>>>>>>>> mount(nfs):
>>>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>>>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  =

>>>>>>>> mount(nfs):
>>>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=
=3D0,  =

>>>>>>>> ro=3D0
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  =

>>>>>>>> mount(nfs):
>>>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount:  =

>>>>>>>> mount(nfs):
>>>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1  =

>>>>>>>> path /misc
>>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc:  =

>>>>>>>> exp_proc =3D
>>>>>>>> 3078093712 path /misc
>>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]:  =

>>>>>>>> expire_proc_indirect: 2
>>>>>>>> submounts remaining in /misc
>>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup:  =

>>>>>>>> got thid
>>>>>>>> 3078093712 path /misc stat 3
>>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup:  =

>>>>>>>> sigchld:
>>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready:  =

>>>>>>>> st_ready(): state
>>>>>>>> =3D 2 path /misc
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1  =

>>>>>>>> path /misc
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc:  =

>>>>>>>> exp_proc =3D
>>>>>>>> 3078093712 path /misc
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]:  =

>>>>>>>> expire_proc_indirect: 2
>>>>>>>> submounts remaining in /misc
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup:  =

>>>>>>>> got thid
>>>>>>>> 3078093712 path /misc stat 3
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup:  =

>>>>>>>> sigchld:
>>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready:  =

>>>>>>>> st_ready(): state
>>>>>>>> =3D 2 path /misc
>>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to  =

>>>>>>>> NFS
>>>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs:  =

>>>>>>>> mount
>>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D  =

>>>>>>>> 17
>>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount / =

>>>>>>>> misc/areas/testdown
>>>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1  =

>>>>>>>> path /misc
>>>>>>>> --------------------------------------------------------------
>>>>>>>>
>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>>>> Hi Ian,
>>>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount...  =

>>>>>>>>>> this option
>>>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/ =

>>>>>>>>>> krb5i/krb5p)
>>>>>>>>>> like you can see on my previous emails...
>>>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>>>
>>>>>>>>> Maybe this is related to the same sort of problem we had  =

>>>>>>>>> with mount in
>>>>>>>>> the past, before the options parsing went into the kernel,  =

>>>>>>>>> where other
>>>>>>>>> services, like portmapper (or rpcbind), were being done with  =

>>>>>>>>> different
>>>>>>>>> timeout parameters before the RPC calls for mounting. That's  =

>>>>>>>>> just an
>>>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>>>
>>>>>>>>> But what version of autofs and kernel did you say you were  =

>>>>>>>>> using?
>>>>>>>>>
>>>>>>>>>> I appreciate any help.
>>>>>>>>>>
>>>>>>>>>> Carlos.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>> This long timeout is good if workstation need mount a  =

>>>>>>>>>>>>> critical
>>>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>>>> But in my case, using this loooong timeout doesnt make  =

>>>>>>>>>>>>> any sense,
>>>>>>>>>>>>> since autofs retry mount directory on-access. This in  =

>>>>>>>>>>>>> fact gives me
>>>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one  =

>>>>>>>>>>>>> server goes
>>>>>>>>>>>>> down for any reason, and will again hangs if user try  =

>>>>>>>>>>>>> access directory
>>>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as  =

>>>>>>>>>>>> the first
>>>>>>>>>>>> mount(2) system call fails.  When you set SYN retries to  =

>>>>>>>>>>>> 1, this means
>>>>>>>>>>>> after 9 seconds, the connect fails, and that causes the  =

>>>>>>>>>>>> mount(2) system
>>>>>>>>>>>> call to fail.
>>>>>>>>>>>>
>>>>>>>>>>>> Recent conversations with Ian suggested that a long  =

>>>>>>>>>>>> timeout was desired
>>>>>>>>>>>> for automounter as well as other cases.  Ian, is there  =

>>>>>>>>>>>> something else we
>>>>>>>>>>>> need to consider to determine the correct retry timeout  =

>>>>>>>>>>>> for NFS/TCP
>>>>>>>>>>>> mount points handled via automounter?  How should  =

>>>>>>>>>>>> mount.nfs wait so we
>>>>>>>>>>>> don't make other use cases worse?  (Looks like most of  =

>>>>>>>>>>>> the history is
>>>>>>>>>>>> intact below).
>>>>>>>>>>> Of course we know that autofs is entirely at the mercy of  =

>>>>>>>>>>> mount(8) (and
>>>>>>>>>>> mount.nfs in particular). This has always been a difficult  =

>>>>>>>>>>> situation for
>>>>>>>>>>> the automounter because interactive mount invocations  =

>>>>>>>>>>> should wait. But I
>>>>>>>>>>> believe automount mounts should always time out quickly,  =

>>>>>>>>>>> but that leads
>>>>>>>>>>> to its own set of problems, especially when home  =

>>>>>>>>>>> directories are concerned.
>>>>>>>>>>>
>>>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself  =

>>>>>>>>>>> but I'm not
>>>>>>>>>>> certain that will work as we expect. I'll have to do some  =

>>>>>>>>>>> experimentation.
>>>>>>>>>>>
>>>>>>>>>>>> How long do you think is appropriate for the automounter  =

>>>>>>>>>>>> to wait if the
>>>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>>>
>>>>>>>>>>>>> Am losing something or there have was something  =

>>>>>>>>>>>>> weirdo...!?
>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/ =

>>>>>>>>>>>>> tcp_syn_retries  [DEFAULT]
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.003s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/ =

>>>>>>>>>>>>> tcp_syn_retries [ 5 to 1 ]
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (retrying). [x 6]
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    1m3.002s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (retrying). [x 13]
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    2m6.000s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    0m9.003s
>>>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t  =

>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (retrying). [x 13]
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    2m6.001s
>>>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5  =

>>>>>>>>>>>>> to 1... and
>>>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>>>
>>>>>>>>>>>>> *sigh*
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change  =

>>>>>>>>>>>>>>> tcp_syn_retries to
>>>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's  =

>>>>>>>>>>>>>> socket connect
>>>>>>>>>>>>>> function,
>>>>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes  =

>>>>>>>>>>>>>> longer than
>>>>>>>>>>>>>> the RPC
>>>>>>>>>>>>>> client's connect timeout, so it only makes one connect  =

>>>>>>>>>>>>>> call, and then
>>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt  =

>>>>>>>>>>>>>> causes the RPC
>>>>>>>>>>>>>> client
>>>>>>>>>>>>>> to retry the connect call until its connect timeout  =

>>>>>>>>>>>>>> expires.  Each
>>>>>>>>>>>>>> connect
>>>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t  =

>>>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/ =

>>>>>>>>>>>>>>> tcp_syn_retries
>>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t  =

>>>>>>>>>>>>>>> nfs4 -o
>>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (retrying).
>>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out  =

>>>>>>>>>>>>>>> (giving up).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump  =

>>>>>>>>>>>>>>>> i've monitored
>>>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96  =

>>>>>>>>>>>>>>>> secs on port
>>>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any  =

>>>>>>>>>>>>>>>> change... I dont
>>>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a  =

>>>>>>>>>>>>>>>>>> situation where my
>>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs  =

>>>>>>>>>>>>>>>>>> max) than 3 minutes
>>>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes  =

>>>>>>>>>>>>>>>>> the kernel to
>>>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN  =

>>>>>>>>>>>>>>>>> attempts with
>>>>>>>>>>>>>>>>> exponential retries, or something like that).  For  =

>>>>>>>>>>>>>>>>> stock CentOS
>>>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4  =

>>>>>>>>>>>>>>>>> mounts -- the
>>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no  =

>>>>>>>>>>>>>>>>> preceding rpcbind
>>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS- =

>>>>>>>>>>>>>>>>> related CentOS
>>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built  =

>>>>>>>>>>>>>>>>> yourself.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny  =

>>>>>>>>>>>>>>>>>>> Halevy wrote:
>>>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn=
@gmail.com =

>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4  =

>>>>>>>>>>>>>>>>>>>>>> server to work with
>>>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server  =

>>>>>>>>>>>>>>>>>>>>>> goes down i get a
>>>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4  =

>>>>>>>>>>>>>>>>>>>>>> client...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user  =

>>>>>>>>>>>>>>>>>>>>>> logon process, if
>>>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to  =

>>>>>>>>>>>>>>>>>>>>>> timeout (if server
>>>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of  =

>>>>>>>>>>>>>>>>>>>>>> combinations, there my
>>>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP:  =

>>>>>>>>>>>>>>>>>>>>>> 172.16.1.10) using
>>>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ - =

>>>>>>>>>>>>>>>>>>>>>> t nfs4 -o
>>>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS  =

>>>>>>>>>>>>>>>>>>>>>> (proto=3Dtcp OR
>>>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)   =

>>>>>>>>>>>>>>>>>>>>>> until show error
>>>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out  =

>>>>>>>>>>>>>>>>>>>>>> (giving up))
>>>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace  =

>>>>>>>>>>>>>>>>>>>> period.
>>>>>>>>>>>>>>>>>>> I thought he was describing a situation where the  =

>>>>>>>>>>>>>>>>>>> server the server
>>>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and  =

>>>>>>>>>>>>>>>>>>> wondering how to make
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line  =

>>>>>>>>>>>>>>>>>> "unsubscribe
>>>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo=
-info.html
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>
>>
>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Trond Myklebust]

T24gVGh1LCAyMDA5LTA4LTI3IGF0IDEwOjM4IC0wNDAwLCBDaHVjayBMZXZlciB3cm90ZToKPiBP
biBBdWcgMjcsIDIwMDksIGF0IDQ6NTQgQU0sIElhbiBLZW50IHdyb3RlOgo+ID4gSWFuIEtlbnQg
d3JvdGU6Cj4gPj4gQ2FybG9zIEFuZHLDqSB3cm90ZToKPiA+Pj4gSGkgSWFuLAo+ID4+Pgo+ID4+
PiBUaGFua3MgZm9yIHBhdGNoIGFuZCBzb3JyeSBmb3IgZGVsYXkgKGknbSBleHBlY3RpbmcgcmVj
ZWl2ZSB1ICAKPiA+Pj4gcmVwbHkgb24KPiA+Pj4gYnVnIHRyYWNrLCBub3QgaGVyZSkgOikKPiA+
Pj4KPiA+Pj4gQnV0LCB0aGlzIHBhdGNoIGRvZXNudCB3b3JrZWQgdG8gbWUgbGlrZSBleHBlY3Rl
ZC4uLiAgOigKPiA+Pj4KPiA+Pj4KPiA+Pj4gRmlyc3RseSBJJ3ZlIGNoYW5nZWQgIiNNT1VOVF9X
QUlUPS0xIiB0byAiTU9VTlRfV0FJVD0xMCIKPiA+Pj4gYW5kIGxhdGVyIGNoYW5nZWQgIjEwIiB0
byAiMiIgd2l0aCBzYW1lIHJlc3VsdHMuLi4KPiA+Pj4gKGFsd2F5cyByZXN0YXJ0aW5nIHNlcnZp
Y2UsIG9mIGNvdXJzZSA6KQo+ID4+Pgo+ID4+PiBUaGVuLCB0cmllZCByZW1vdmUgInNlYz1rcmI1
cCIsIGFuZCBsYXRlciByZW1vdmVkICJuZnM0IiBidXQgaSBnb3QKPiA+Pj4gc2FtZSByZXN1bHRz
IGFnYWluLgo+ID4+Pgo+ID4+PiBPciBpJ20gZG9pbmcgc29tZXRoaW5nIHdyb25nPwo+ID4+Pgo+
ID4+Pgo+ID4+PiBbcm9vdEBLU1RBVElPTiBhcmVhc10jIGF1dG9tb3VudCAtVgo+ID4+Pgo+ID4+
PiBMaW51eCBhdXRvbW91bnQgdmVyc2lvbiA1LjAuMS0wLnJjMi4xMzEuYno1MTczNDkuMQo+ID4+
PiBbLi4uXQo+ID4+Pgo+ID4+PiBbcm9vdEBLU1RBVElPTiBhcmVhc10jIHRpbWUgbHMgLWxhIHRl
c3Rkb3duCj4gPj4+IGxzOiB0ZXN0ZWRvd246IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnkKPiA+
Pj4KPiA+Pj4gcmVhbCAgICAzbTkuMDA2cwo+ID4+PiB1c2VyICAgIDBtMC4wMDJzCj4gPj4+IHN5
cyAgICAgMG0wLjAwMHMKPiA+Pgo+ID4+IE9LLCB0aGF0IGlzbid0IGJlaGF2aW5nIHRoZSB3YXkg
SSBleHBlY3QsIEknbGwgaGF2ZSBhIGxvb2suCj4gPj4KPiA+Pj4KPiA+Pj4gTE9HR0lORzoKPiA+
Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KPiA+Pj4gQXVnIDI0
IDA5OjIzOjUxIEtTVEFUSU9OIGF1dG9tb3VudFsyMDgwM106IG1vdW50X21vdW50OiBtb3VudChu
ZnMpOgo+ID4+PiBjYWxsaW5nIG1vdW50IC10IG5mczQgLXMgLW8gcncsYWNsLHNlYz1rcmI1cCAx
LjIuMy40Oi9hcmVhcy90ZXN0ZG93bgo+ID4+PiAvbWlzYy9hcmVhcy90ZXN0ZG93bgo+ID4+PiBB
dWcgMjQgMDk6Mjc6MDAgS1NUQVRJT04gYXV0b21vdW50WzIwODAzXTogbW91bnQobmZzKTogbmZz
OiBtb3VudAo+ID4+PiBmYWlsdXJlIDEuMi4zLjQ6L2FyZWFzL3Rlc3Rkb3duIG9uIC9taXNjL2Fy
ZWFzL3Rlc3Rkb3duCj4gPj4+IEF1ZyAyNCAwOToyNzowMCBLU1RBVElPTiBhdXRvbW91bnRbMjA4
MDNdOiBpb2N0bF9zZW5kX2ZhaWw6IHRva2VuICAKPiA+Pj4gPSA5MQo+ID4+PiBBdWcgMjQgMDk6
Mjc6MDAgS1NUQVRJT04gYXV0b21vdW50WzIwODAzXTogZmFpbGVkIHRvIG1vdW50IC9taXNjLyAK
PiA+Pj4gYXJlYXMvdGVzdGRvd24KPiA+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0KPiA+Cj4gPiBIYXZpbmcgYSBsb29rIGF0IHRoaXMgSSBzdXNwZWN0IHRoZSBy
ZWFzb24gaXQgZG9lc24ndCB3b3JrIGFzIGV4cGVjdGVkCj4gPiBpcyB0aGUgd2FpdHBpZCgyKSB3
ZSBkbyBhZnRlciBzZW5kaW5nIHRoZSBURVJNIHNpZ25hbCB0byB0aGUgbW91bnQKPiA+IHByb2Nl
c3MgKHdoaWNoIHdlIGhhdmUgdG8gZG8pIGlzIG5vdCByZXR1cm5pbmcuIFRoaXMgaXMgbGlrZWx5
IGJlY2F1c2UKPiA+IHRoZSBtb3VudCBwcm9jZXNzIGlzbid0IGdpdmluZyB1cCBpbiBhIHNob3J0
ZXIgdGltZSBhcyBpdCB1c2VkIHRvLgo+IAo+IFlvdSdyZSB0aGlua2luZyBtYXliZSBtb3VudCgy
KSBzaG91bGQgYmUgYXMgaW50ZXJydXB0aWJsZSBhcyB0aGUgIAo+IHNvY2tldCBjYWxscyB0aGF0
IHRoZSBtb3VudCBjb21tYW5kIHVzZWQgdG8gZG8/ICBUaGF0IG1pZ2h0IGJlICAKPiByZWFzb25h
YmxlLCBhbmQgSSBjYW4gdGFrZSBhIGxvb2sgYXQgdGhhdC4KCkluIHJlY2VudCBrZXJuZWxzLCBh
bGwgdGhvc2UgUlBDIGNhbGxzIHNob3VsZCBiZSB1c2luZyBUQVNLX0tJTExBQkxFCnNsZWVwIHN0
YXRlcy4gU0lHVEVSTSBzaG91bGQgY2F1c2UgdGhlbSB0byBhYm9ydCwgcHJvdmlkZWQgdGhhdCBz
b21lCnByb2Nlc3MgaXNuJ3QgYmxvY2tpbmcgaXQuCgpQZXJoYXBzIFRBU0tfS0lMTEFCTEUgY291
bGQgYmUgYmFja3BvcnRlZCB0byBSSEVMLTU/Cgo+IEluIHRoZSBrZXJuZWwsIGlmIHRoZSBycGNi
aW5kIGZvciB0aGUgTU5UIHJlcXVlc3QgaXMgYXN5bmMsIHRoYXQgd291bGQgIAo+IGJlIGRvbmUg
YnkgcnBjaW9kLiAgVGhhdCdzIGEgZGlmZmVyZW50IHByb2Nlc3MsIHNvIHRoZSBzaWduYWwgd291
bGRuJ3QgIAo+IGhhdmUgYW55IGVmZmVjdCBvbiB0aGUgbW91bnQuICBJIGhhdmUgYSBwYXRjaCB0
aGF0IGNvbnZlcnRzIHRoZSBNTlQgIAo+IGNsaWVudCB0byB1c2UgcnBjYl9nZXRwb3J0X3N5bmMo
KSB3aGljaCBtaWdodCBoZWxwIGluIHRoaXMgY2FzZS4KClRoZSBjbGllbnQgc2hvdWxkbid0IGJl
IHVzaW5nIHJwY2JpbmQgYXQgYWxsIHdoZW4gZG9pbmcgYSBORlN2NCBtb3VudC4KCkNoZWVycwog
IFRyb25kCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpO
RlN2NCBtYWlsaW5nIGxpc3QKTkZTdjRAbGludXgtbmZzLm9yZwpodHRwOi8vbGludXgtbmZzLm9y
Zy9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vbmZzdjQ=

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 27, 2009, at 10:52 AM, Trond Myklebust wrote:
> On Thu, 2009-08-27 at 10:38 -0400, Chuck Lever wrote:
>> On Aug 27, 2009, at 4:54 AM, Ian Kent wrote:
>>> Ian Kent wrote:
>>>> Carlos Andr=E9 wrote:
>>>>> Hi Ian,
>>>>>
>>>>> Thanks for patch and sorry for delay (i'm expecting receive u
>>>>> reply on
>>>>> bug track, not here) :)
>>>>>
>>>>> But, this patch doesnt worked to me like expected...  :(
>>>>>
>>>>>
>>>>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>>>>> and later changed "10" to "2" with same results...
>>>>> (always restarting service, of course :)
>>>>>
>>>>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
>>>>> same results again.
>>>>>
>>>>> Or i'm doing something wrong?
>>>>>
>>>>>
>>>>> [root@KSTATION areas]# automount -V
>>>>>
>>>>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>>>>> [...]
>>>>>
>>>>> [root@KSTATION areas]# time ls -la testdown
>>>>> ls: testedown: No such file or directory
>>>>>
>>>>> real    3m9.006s
>>>>> user    0m0.002s
>>>>> sys     0m0.000s
>>>>
>>>> OK, that isn't behaving the way I expect, I'll have a look.
>>>>
>>>>>
>>>>> LOGGING:
>>>>> -----------------------------------------
>>>>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount:  =

>>>>> mount(nfs):
>>>>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/ =

>>>>> testdown
>>>>> /misc/areas/testdown
>>>>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token
>>>>> =3D 91
>>>>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/
>>>>> areas/testdown
>>>>> -----------------------------------------
>>>
>>> Having a look at this I suspect the reason it doesn't work as  =

>>> expected
>>> is the waitpid(2) we do after sending the TERM signal to the mount
>>> process (which we have to do) is not returning. This is likely  =

>>> because
>>> the mount process isn't giving up in a shorter time as it used to.
>>
>> You're thinking maybe mount(2) should be as interruptible as the
>> socket calls that the mount command used to do?  That might be
>> reasonable, and I can take a look at that.
>
> In recent kernels, all those RPC calls should be using TASK_KILLABLE
> sleep states. SIGTERM should cause them to abort, provided that some
> process isn't blocking it.
>
> Perhaps TASK_KILLABLE could be backported to RHEL-5?

That's pretty extensive, with hooks in the page cache.  I doubt RH  =

would go for that.

>> In the kernel, if the rpcbind for the MNT request is async, that  =

>> would
>> be done by rpciod.  That's a different process, so the signal  =

>> wouldn't
>> have any effect on the mount.  I have a patch that converts the MNT
>> client to use rpcb_getport_sync() which might help in this case.
>
> The client shouldn't be using rpcbind at all when doing a NFSv4 mount.

Yep, forgot this was NFSv4.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Trond Myklebust]

On Thu, 2009-08-27 at 10:54 -0400, Chuck Lever wrote:
> On Aug 27, 2009, at 10:52 AM, Trond Myklebust wrote:
> > On Thu, 2009-08-27 at 10:38 -0400, Chuck Lever wrote:
> >> On Aug 27, 2009, at 4:54 AM, Ian Kent wrote:
> >>> Ian Kent wrote:
> >>>> Carlos Andr=C3=A9 wrote:
> >>>>> Hi Ian,
> >>>>>
> >>>>> Thanks for patch and sorry for delay (i'm expecting receive u
> >>>>> reply on
> >>>>> bug track, not here) :)
> >>>>>
> >>>>> But, this patch doesnt worked to me like expected...  :(
> >>>>>
> >>>>>
> >>>>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
> >>>>> and later changed "10" to "2" with same results...
> >>>>> (always restarting service, of course :)
> >>>>>
> >>>>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but =
i got
> >>>>> same results again.
> >>>>>
> >>>>> Or i'm doing something wrong?
> >>>>>
> >>>>>
> >>>>> [root@KSTATION areas]# automount -V
> >>>>>
> >>>>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
> >>>>> [...]
> >>>>>
> >>>>> [root@KSTATION areas]# time ls -la testdown
> >>>>> ls: testedown: No such file or directory
> >>>>>
> >>>>> real    3m9.006s
> >>>>> user    0m0.002s
> >>>>> sys     0m0.000s
> >>>>
> >>>> OK, that isn't behaving the way I expect, I'll have a look.
> >>>>
> >>>>>
> >>>>> LOGGING:
> >>>>> -----------------------------------------
> >>>>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: =20
> >>>>> mount(nfs):
> >>>>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/=20
> >>>>> testdown
> >>>>> /misc/areas/testdown
> >>>>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mou=
nt
> >>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> >>>>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: tok=
en
> >>>>> =3D 91
> >>>>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /mis=
c/
> >>>>> areas/testdown
> >>>>> -----------------------------------------
> >>>
> >>> Having a look at this I suspect the reason it doesn't work as =20
> >>> expected
> >>> is the waitpid(2) we do after sending the TERM signal to the moun=
t
> >>> process (which we have to do) is not returning. This is likely =20
> >>> because
> >>> the mount process isn't giving up in a shorter time as it used to=
=2E
> >>
> >> You're thinking maybe mount(2) should be as interruptible as the
> >> socket calls that the mount command used to do?  That might be
> >> reasonable, and I can take a look at that.
> >
> > In recent kernels, all those RPC calls should be using TASK_KILLABL=
E
> > sleep states. SIGTERM should cause them to abort, provided that som=
e
> > process isn't blocking it.
> >
> > Perhaps TASK_KILLABLE could be backported to RHEL-5?
>=20
> That's pretty extensive, with hooks in the page cache.  I doubt RH =20
> would go for that.

You don't have to add the hooks in the page cache in order to make moun=
t
interruptible. You just need to replace the sigmask-manipulation in
net/sunrpc and fs/nfs (a.k.a. rpc_clnt_sigmask()/rpc_clnt_sigunmask())
with TASK_KILLABLE.

Alternatively, it might suffice to just turn on the 'intr' flag
temporarily while doing the mount path walk, and then switch it to
whatever default the user actually specified afterwards.

Trond

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 27, 2009, at 11:00 AM, Trond Myklebust wrote:
> On Thu, 2009-08-27 at 10:54 -0400, Chuck Lever wrote:
>> On Aug 27, 2009, at 10:52 AM, Trond Myklebust wrote:
>>> On Thu, 2009-08-27 at 10:38 -0400, Chuck Lever wrote:
>>>> On Aug 27, 2009, at 4:54 AM, Ian Kent wrote:
>>>>> Ian Kent wrote:
>>>>>> Carlos Andr=E9 wrote:
>>>>>>> Hi Ian,
>>>>>>>
>>>>>>> Thanks for patch and sorry for delay (i'm expecting receive u
>>>>>>> reply on
>>>>>>> bug track, not here) :)
>>>>>>>
>>>>>>> But, this patch doesnt worked to me like expected...  :(
>>>>>>>
>>>>>>>
>>>>>>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>>>>>>> and later changed "10" to "2" with same results...
>>>>>>> (always restarting service, of course :)
>>>>>>>
>>>>>>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i  =

>>>>>>> got
>>>>>>> same results again.
>>>>>>>
>>>>>>> Or i'm doing something wrong?
>>>>>>>
>>>>>>>
>>>>>>> [root@KSTATION areas]# automount -V
>>>>>>>
>>>>>>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>>>>>>> [...]
>>>>>>>
>>>>>>> [root@KSTATION areas]# time ls -la testdown
>>>>>>> ls: testedown: No such file or directory
>>>>>>>
>>>>>>> real    3m9.006s
>>>>>>> user    0m0.002s
>>>>>>> sys     0m0.000s
>>>>>>
>>>>>> OK, that isn't behaving the way I expect, I'll have a look.
>>>>>>
>>>>>>>
>>>>>>> LOGGING:
>>>>>>> -----------------------------------------
>>>>>>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount:
>>>>>>> mount(nfs):
>>>>>>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/
>>>>>>> testdown
>>>>>>> /misc/areas/testdown
>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs:  =

>>>>>>> mount
>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail:  =

>>>>>>> token
>>>>>>> =3D 91
>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount / =

>>>>>>> misc/
>>>>>>> areas/testdown
>>>>>>> -----------------------------------------
>>>>>
>>>>> Having a look at this I suspect the reason it doesn't work as
>>>>> expected
>>>>> is the waitpid(2) we do after sending the TERM signal to the mount
>>>>> process (which we have to do) is not returning. This is likely
>>>>> because
>>>>> the mount process isn't giving up in a shorter time as it used to.
>>>>
>>>> You're thinking maybe mount(2) should be as interruptible as the
>>>> socket calls that the mount command used to do?  That might be
>>>> reasonable, and I can take a look at that.
>>>
>>> In recent kernels, all those RPC calls should be using TASK_KILLABLE
>>> sleep states. SIGTERM should cause them to abort, provided that some
>>> process isn't blocking it.
>>>
>>> Perhaps TASK_KILLABLE could be backported to RHEL-5?
>>
>> That's pretty extensive, with hooks in the page cache.  I doubt RH
>> would go for that.
>
> You don't have to add the hooks in the page cache in order to make  =

> mount
> interruptible. You just need to replace the sigmask-manipulation in
> net/sunrpc and fs/nfs (a.k.a. rpc_clnt_sigmask()/rpc_clnt_sigunmask())
> with TASK_KILLABLE.

That sounds like a schlep.

> Alternatively, it might suffice to just turn on the 'intr' flag
> temporarily while doing the mount path walk, and then switch it to
> whatever default the user actually specified afterwards.

That sounds easy, especially for an EL5 kernel.  Maybe "soft" too for  =

the first few requests?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Hi ppl,

any news about this problem? :)

Thanks.

2009/8/27 Chuck Lever <chuck.lever@oracle.com>:
> On Aug 27, 2009, at 11:00 AM, Trond Myklebust wrote:
>>
>> On Thu, 2009-08-27 at 10:54 -0400, Chuck Lever wrote:
>>>
>>> On Aug 27, 2009, at 10:52 AM, Trond Myklebust wrote:
>>>>
>>>> On Thu, 2009-08-27 at 10:38 -0400, Chuck Lever wrote:
>>>>>
>>>>> On Aug 27, 2009, at 4:54 AM, Ian Kent wrote:
>>>>>>
>>>>>> Ian Kent wrote:
>>>>>>>
>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>>
>>>>>>>> Hi Ian,
>>>>>>>>
>>>>>>>> Thanks for patch and sorry for delay (i'm expecting receive u
>>>>>>>> reply on
>>>>>>>> bug track, not here) :)
>>>>>>>>
>>>>>>>> But, this patch doesnt worked to me like expected... =A0:(
>>>>>>>>
>>>>>>>>
>>>>>>>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>>>>>>>> and later changed "10" to "2" with same results...
>>>>>>>> (always restarting service, of course :)
>>>>>>>>
>>>>>>>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i g=
ot
>>>>>>>> same results again.
>>>>>>>>
>>>>>>>> Or i'm doing something wrong?
>>>>>>>>
>>>>>>>>
>>>>>>>> [root@KSTATION areas]# automount -V
>>>>>>>>
>>>>>>>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>>>>>>>> [...]
>>>>>>>>
>>>>>>>> [root@KSTATION areas]# time ls -la testdown
>>>>>>>> ls: testedown: No such file or directory
>>>>>>>>
>>>>>>>> real =A0 =A03m9.006s
>>>>>>>> user =A0 =A00m0.002s
>>>>>>>> sys =A0 =A0 0m0.000s
>>>>>>>
>>>>>>> OK, that isn't behaving the way I expect, I'll have a look.
>>>>>>>
>>>>>>>>
>>>>>>>> LOGGING:
>>>>>>>> -----------------------------------------
>>>>>>>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount:
>>>>>>>> mount(nfs):
>>>>>>>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/
>>>>>>>> testdown
>>>>>>>> /misc/areas/testdown
>>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token
>>>>>>>> =3D 91
>>>>>>>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/
>>>>>>>> areas/testdown
>>>>>>>> -----------------------------------------
>>>>>>
>>>>>> Having a look at this I suspect the reason it doesn't work as
>>>>>> expected
>>>>>> is the waitpid(2) we do after sending the TERM signal to the mount
>>>>>> process (which we have to do) is not returning. This is likely
>>>>>> because
>>>>>> the mount process isn't giving up in a shorter time as it used to.
>>>>>
>>>>> You're thinking maybe mount(2) should be as interruptible as the
>>>>> socket calls that the mount command used to do? =A0That might be
>>>>> reasonable, and I can take a look at that.
>>>>
>>>> In recent kernels, all those RPC calls should be using TASK_KILLABLE
>>>> sleep states. SIGTERM should cause them to abort, provided that some
>>>> process isn't blocking it.
>>>>
>>>> Perhaps TASK_KILLABLE could be backported to RHEL-5?
>>>
>>> That's pretty extensive, with hooks in the page cache. =A0I doubt RH
>>> would go for that.
>>
>> You don't have to add the hooks in the page cache in order to make mount
>> interruptible. You just need to replace the sigmask-manipulation in
>> net/sunrpc and fs/nfs (a.k.a. rpc_clnt_sigmask()/rpc_clnt_sigunmask())
>> with TASK_KILLABLE.
>
> That sounds like a schlep.
>
>> Alternatively, it might suffice to just turn on the 'intr' flag
>> temporarily while doing the mount path walk, and then switch it to
>> whatever default the user actually specified afterwards.
>
> That sounds easy, especially for an EL5 kernel. =A0Maybe "soft" too for t=
he
> first few requests?
>
> --
> Chuck Lever
> chuck[dot]lever[at]oracle[dot]com
>
>
>
>



-- =

Atenciosamente,
Carlos Andr=E9
LPIC-1 / LPIC-2 / CCNA / CCNP

candrecn.at.gmail.dot.com

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ondrej Valousek]

aHR0cHM6Ly9idWd6aWxsYS5yZWRoYXQuY29tL3Nob3dfYnVnLmNnaT9pZD01MTczNDkKbm8gbmV3
cy4uLi4uCgpDYXJsb3MgQW5kcsOpIHdyb3RlOgo+IEhpIHBwbCwKPgo+IGFueSBuZXdzIGFib3V0
IHRoaXMgcHJvYmxlbT8gOikKPgo+IFRoYW5rcy4KPgo+IDIwMDkvOC8yNyBDaHVjayBMZXZlciA8
Y2h1Y2subGV2ZXJAb3JhY2xlLmNvbT46Cj4gICAKPj4gT24gQXVnIDI3LCAyMDA5LCBhdCAxMTow
MCBBTSwgVHJvbmQgTXlrbGVidXN0IHdyb3RlOgo+PiAgICAgCj4+PiBPbiBUaHUsIDIwMDktMDgt
MjcgYXQgMTA6NTQgLTA0MDAsIENodWNrIExldmVyIHdyb3RlOgo+Pj4gICAgICAgCj4+Pj4gT24g
QXVnIDI3LCAyMDA5LCBhdCAxMDo1MiBBTSwgVHJvbmQgTXlrbGVidXN0IHdyb3RlOgo+Pj4+ICAg
ICAgICAgCj4+Pj4+IE9uIFRodSwgMjAwOS0wOC0yNyBhdCAxMDozOCAtMDQwMCwgQ2h1Y2sgTGV2
ZXIgd3JvdGU6Cj4+Pj4+ICAgICAgICAgICAKPj4+Pj4+IE9uIEF1ZyAyNywgMjAwOSwgYXQgNDo1
NCBBTSwgSWFuIEtlbnQgd3JvdGU6Cj4+Pj4+PiAgICAgICAgICAgICAKPj4+Pj4+PiBJYW4gS2Vu
dCB3cm90ZToKPj4+Pj4+PiAgICAgICAgICAgICAgIAo+Pj4+Pj4+PiBDYXJsb3MgQW5kcsOpIHdy
b3RlOgo+Pj4+Pj4+PiAgICAgICAgICAgICAgICAgCj4+Pj4+Pj4+PiBIaSBJYW4sCj4+Pj4+Pj4+
Pgo+Pj4+Pj4+Pj4gVGhhbmtzIGZvciBwYXRjaCBhbmQgc29ycnkgZm9yIGRlbGF5IChpJ20gZXhw
ZWN0aW5nIHJlY2VpdmUgdQo+Pj4+Pj4+Pj4gcmVwbHkgb24KPj4+Pj4+Pj4+IGJ1ZyB0cmFjaywg
bm90IGhlcmUpIDopCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gQnV0LCB0aGlzIHBhdGNoIGRvZXNudCB3
b3JrZWQgdG8gbWUgbGlrZSBleHBlY3RlZC4uLiAgOigKPj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pgo+Pj4+
Pj4+Pj4gRmlyc3RseSBJJ3ZlIGNoYW5nZWQgIiNNT1VOVF9XQUlUPS0xIiB0byAiTU9VTlRfV0FJ
VD0xMCIKPj4+Pj4+Pj4+IGFuZCBsYXRlciBjaGFuZ2VkICIxMCIgdG8gIjIiIHdpdGggc2FtZSBy
ZXN1bHRzLi4uCj4+Pj4+Pj4+PiAoYWx3YXlzIHJlc3RhcnRpbmcgc2VydmljZSwgb2YgY291cnNl
IDopCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gVGhlbiwgdHJpZWQgcmVtb3ZlICJzZWM9a3JiNXAiLCBh
bmQgbGF0ZXIgcmVtb3ZlZCAibmZzNCIgYnV0IGkgZ290Cj4+Pj4+Pj4+PiBzYW1lIHJlc3VsdHMg
YWdhaW4uCj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gT3IgaSdtIGRvaW5nIHNvbWV0aGluZyB3cm9uZz8K
Pj4+Pj4+Pj4+Cj4+Pj4+Pj4+Pgo+Pj4+Pj4+Pj4gW3Jvb3RAS1NUQVRJT04gYXJlYXNdIyBhdXRv
bW91bnQgLVYKPj4+Pj4+Pj4+Cj4+Pj4+Pj4+PiBMaW51eCBhdXRvbW91bnQgdmVyc2lvbiA1LjAu
MS0wLnJjMi4xMzEuYno1MTczNDkuMQo+Pj4+Pj4+Pj4gWy4uLl0KPj4+Pj4+Pj4+Cj4+Pj4+Pj4+
PiBbcm9vdEBLU1RBVElPTiBhcmVhc10jIHRpbWUgbHMgLWxhIHRlc3Rkb3duCj4+Pj4+Pj4+PiBs
czogdGVzdGVkb3duOiBObyBzdWNoIGZpbGUgb3IgZGlyZWN0b3J5Cj4+Pj4+Pj4+Pgo+Pj4+Pj4+
Pj4gcmVhbCAgICAzbTkuMDA2cwo+Pj4+Pj4+Pj4gdXNlciAgICAwbTAuMDAycwo+Pj4+Pj4+Pj4g
c3lzICAgICAwbTAuMDAwcwo+Pj4+Pj4+Pj4gICAgICAgICAgICAgICAgICAgCj4+Pj4+Pj4+IE9L
LCB0aGF0IGlzbid0IGJlaGF2aW5nIHRoZSB3YXkgSSBleHBlY3QsIEknbGwgaGF2ZSBhIGxvb2su
Cj4+Pj4+Pj4+Cj4+Pj4+Pj4+ICAgICAgICAgICAgICAgICAKPj4+Pj4+Pj4+IExPR0dJTkc6Cj4+
Pj4+Pj4+PiAtLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLQo+Pj4+Pj4+
Pj4gQXVnIDI0IDA5OjIzOjUxIEtTVEFUSU9OIGF1dG9tb3VudFsyMDgwM106IG1vdW50X21vdW50
Ogo+Pj4+Pj4+Pj4gbW91bnQobmZzKToKPj4+Pj4+Pj4+IGNhbGxpbmcgbW91bnQgLXQgbmZzNCAt
cyAtbyBydyxhY2wsc2VjPWtyYjVwIDEuMi4zLjQ6L2FyZWFzLwo+Pj4+Pj4+Pj4gdGVzdGRvd24K
Pj4+Pj4+Pj4+IC9taXNjL2FyZWFzL3Rlc3Rkb3duCj4+Pj4+Pj4+PiBBdWcgMjQgMDk6Mjc6MDAg
S1NUQVRJT04gYXV0b21vdW50WzIwODAzXTogbW91bnQobmZzKTogbmZzOiBtb3VudAo+Pj4+Pj4+
Pj4gZmFpbHVyZSAxLjIuMy40Oi9hcmVhcy90ZXN0ZG93biBvbiAvbWlzYy9hcmVhcy90ZXN0ZG93
bgo+Pj4+Pj4+Pj4gQXVnIDI0IDA5OjI3OjAwIEtTVEFUSU9OIGF1dG9tb3VudFsyMDgwM106IGlv
Y3RsX3NlbmRfZmFpbDogdG9rZW4KPj4+Pj4+Pj4+ID0gOTEKPj4+Pj4+Pj4+IEF1ZyAyNCAwOToy
NzowMCBLU1RBVElPTiBhdXRvbW91bnRbMjA4MDNdOiBmYWlsZWQgdG8gbW91bnQgL21pc2MvCj4+
Pj4+Pj4+PiBhcmVhcy90ZXN0ZG93bgo+Pj4+Pj4+Pj4gLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0KPj4+Pj4+Pj4+ICAgICAgICAgICAgICAgICAgIAo+Pj4+Pj4+IEhh
dmluZyBhIGxvb2sgYXQgdGhpcyBJIHN1c3BlY3QgdGhlIHJlYXNvbiBpdCBkb2Vzbid0IHdvcmsg
YXMKPj4+Pj4+PiBleHBlY3RlZAo+Pj4+Pj4+IGlzIHRoZSB3YWl0cGlkKDIpIHdlIGRvIGFmdGVy
IHNlbmRpbmcgdGhlIFRFUk0gc2lnbmFsIHRvIHRoZSBtb3VudAo+Pj4+Pj4+IHByb2Nlc3MgKHdo
aWNoIHdlIGhhdmUgdG8gZG8pIGlzIG5vdCByZXR1cm5pbmcuIFRoaXMgaXMgbGlrZWx5Cj4+Pj4+
Pj4gYmVjYXVzZQo+Pj4+Pj4+IHRoZSBtb3VudCBwcm9jZXNzIGlzbid0IGdpdmluZyB1cCBpbiBh
IHNob3J0ZXIgdGltZSBhcyBpdCB1c2VkIHRvLgo+Pj4+Pj4+ICAgICAgICAgICAgICAgCj4+Pj4+
PiBZb3UncmUgdGhpbmtpbmcgbWF5YmUgbW91bnQoMikgc2hvdWxkIGJlIGFzIGludGVycnVwdGli
bGUgYXMgdGhlCj4+Pj4+PiBzb2NrZXQgY2FsbHMgdGhhdCB0aGUgbW91bnQgY29tbWFuZCB1c2Vk
IHRvIGRvPyAgVGhhdCBtaWdodCBiZQo+Pj4+Pj4gcmVhc29uYWJsZSwgYW5kIEkgY2FuIHRha2Ug
YSBsb29rIGF0IHRoYXQuCj4+Pj4+PiAgICAgICAgICAgICAKPj4+Pj4gSW4gcmVjZW50IGtlcm5l
bHMsIGFsbCB0aG9zZSBSUEMgY2FsbHMgc2hvdWxkIGJlIHVzaW5nIFRBU0tfS0lMTEFCTEUKPj4+
Pj4gc2xlZXAgc3RhdGVzLiBTSUdURVJNIHNob3VsZCBjYXVzZSB0aGVtIHRvIGFib3J0LCBwcm92
aWRlZCB0aGF0IHNvbWUKPj4+Pj4gcHJvY2VzcyBpc24ndCBibG9ja2luZyBpdC4KPj4+Pj4KPj4+
Pj4gUGVyaGFwcyBUQVNLX0tJTExBQkxFIGNvdWxkIGJlIGJhY2twb3J0ZWQgdG8gUkhFTC01Pwo+
Pj4+PiAgICAgICAgICAgCj4+Pj4gVGhhdCdzIHByZXR0eSBleHRlbnNpdmUsIHdpdGggaG9va3Mg
aW4gdGhlIHBhZ2UgY2FjaGUuICBJIGRvdWJ0IFJICj4+Pj4gd291bGQgZ28gZm9yIHRoYXQuCj4+
Pj4gICAgICAgICAKPj4+IFlvdSBkb24ndCBoYXZlIHRvIGFkZCB0aGUgaG9va3MgaW4gdGhlIHBh
Z2UgY2FjaGUgaW4gb3JkZXIgdG8gbWFrZSBtb3VudAo+Pj4gaW50ZXJydXB0aWJsZS4gWW91IGp1
c3QgbmVlZCB0byByZXBsYWNlIHRoZSBzaWdtYXNrLW1hbmlwdWxhdGlvbiBpbgo+Pj4gbmV0L3N1
bnJwYyBhbmQgZnMvbmZzIChhLmsuYS4gcnBjX2NsbnRfc2lnbWFzaygpL3JwY19jbG50X3NpZ3Vu
bWFzaygpKQo+Pj4gd2l0aCBUQVNLX0tJTExBQkxFLgo+Pj4gICAgICAgCj4+IFRoYXQgc291bmRz
IGxpa2UgYSBzY2hsZXAuCj4+Cj4+ICAgICAKPj4+IEFsdGVybmF0aXZlbHksIGl0IG1pZ2h0IHN1
ZmZpY2UgdG8ganVzdCB0dXJuIG9uIHRoZSAnaW50cicgZmxhZwo+Pj4gdGVtcG9yYXJpbHkgd2hp
bGUgZG9pbmcgdGhlIG1vdW50IHBhdGggd2FsaywgYW5kIHRoZW4gc3dpdGNoIGl0IHRvCj4+PiB3
aGF0ZXZlciBkZWZhdWx0IHRoZSB1c2VyIGFjdHVhbGx5IHNwZWNpZmllZCBhZnRlcndhcmRzLgo+
Pj4gICAgICAgCj4+IFRoYXQgc291bmRzIGVhc3ksIGVzcGVjaWFsbHkgZm9yIGFuIEVMNSBrZXJu
ZWwuICBNYXliZSAic29mdCIgdG9vIGZvciB0aGUKPj4gZmlyc3QgZmV3IHJlcXVlc3RzPwo+Pgo+
PiAtLQo+PiBDaHVjayBMZXZlcgo+PiBjaHVja1tkb3RdbGV2ZXJbYXRdb3JhY2xlW2RvdF1jb20K
Pj4KPj4KPj4KPj4KPj4gICAgIAo+Cj4KPgo+ICAgCgpfX19fX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fXwpORlN2NCBtYWlsaW5nIGxpc3QKTkZTdjRAbGludXgtbmZz
Lm9yZwpodHRwOi8vbGludXgtbmZzLm9yZy9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vbmZzdjQ=

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Ian Kent]

Carlos Andr=E9 wrote:
> Hi Ian,
> =

> Thanks for patch and sorry for delay (i'm expecting receive u reply on
> bug track, not here) :)
> =

> But, this patch doesnt worked to me like expected...  :(

OK, I've been off on a wild goose chase, thinking this was related to
the moving of the mount option handling and initial file handle open
into the kernel, but that isn't even included in the kernel you are
using. Suffice it to say this behaviour exists at least back to RHEL-4
and NFS v3 and v2 mount take around 1 minute to time out and v4 about 3
minutes. Not only that, mount attempts from the command line appear to
respond to an TERM signal, including using a relatively recent kernel,
but I might not have that quite right.

Anyway, now that I'm back on track, we might make some progress.

> =

> =

> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
> and later changed "10" to "2" with same results...
> (always restarting service, of course :)
> =

> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
> same results again.
> =

> Or i'm doing something wrong?

Maybe.

I've tested this out now with some interesting results.
I can't easily setup Kerberos for NFS so lets work on plain mounts to
begin with.

Using the patch I posted with plain mounts autofs did indeed return
after the configured timeout. After sending the TERM signal to the mount
the mount process went away but the mount.nfs child process remained
waiting for to timeout. User space received the usual ENOENT error after
the configured timeout. The same occurred with nfs4. This is much the
same as the timed umount behaviour so it's expected.

So, there must be something wrong with the patching of autofs.
I'll put together a patched RHEL package and we will continue this in
the RedHat bug you've logged.

> =

> =

> [root@KSTATION areas]# automount -V
> =

> Linux automount version 5.0.1-0.rc2.131.bz517349.1
> [...]
> =

> [root@KSTATION areas]# time ls -la testdown
> ls: testedown: No such file or directory
> =

> real    3m9.006s
> user    0m0.002s
> sys     0m0.000s
> =

> =

> LOGGING:
> -----------------------------------------
> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdown
> /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D 91
> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/areas/te=
stdown
> -----------------------------------------
> =

> =

> =

> =

> =

> 2009/8/17 Ian Kent <ikent@redhat.com>:
>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>> Filled bug report:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>> Hi Carlos,
>>
>> I have a patched source rpm to add a mount wait parameter to autofs
>> located at:
>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>
>> Could you build it and see if it works.
>> I haven't tested it at all but it is fairly straight forward.
>> It is still unclear if this is the right way to do this and what the
>> consequences are in sending a term signal to mount. This mount request
>> will likely be followed by other requests for the same mount causing an
>> accumulation of mount(8) processes waiting for RPC timeouts before they
>> can answer the TERM signal.
>>
>> Anyway, for information the patch included in the source rpm above is:
>>
>> autofs-5.0.4 - add mount wait parameter
>>
>> From: Ian Kent <raven@themaw.net>
>>
>> Often delays when trying to mount from a server that is not reponding
>> for some reason are undesirable. To try and prevent these delays we
>> provide a configuration setting to limit the time that we wait for
>> our spawned mount(8) process to complete before sending it a SIGTERM
>> signal. This patch adds a configuration parameter to allow us to
>> request we limit the time we wait for mount(8) to complete before
>> send it a TERM signal.
>> ---
>>
>>  daemon/spawn.c                 |    3 ++-
>>  include/defaults.h             |    2 ++
>>  lib/defaults.c                 |   13 +++++++++++++
>>  man/auto.master.5.in           |    7 +++++++
>>  redhat/autofs.sysconfig.in     |    9 +++++++++
>>  samples/autofs.conf.default.in |    9 +++++++++
>>  6 files changed, 42 insertions(+), 1 deletion(-)
>>
>>
>> --- autofs-5.0.1.orig/daemon/spawn.c
>> +++ autofs-5.0.1/daemon/spawn.c
>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>        unsigned int options;
>>        unsigned int retries =3D MTAB_LOCK_RETRIES;
>>        int update_mtab =3D 1, ret, printed =3D 0;
>> +       unsigned int wait =3D defaults_get_mount_wait();
>>        char buf[PATH_MAX];
>>
>>        /* If we use mount locking we can't validate the location */
>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>        va_end(arg);
>>
>>        while (retries--) {
>> -               ret =3D do_spawn(logopt, -1, options, prog, (const char =
**) argv);
>> +               ret =3D do_spawn(logopt, wait, options, prog, (const cha=
r **) argv);
>>                if (ret & MTAB_NOTUPDATED) {
>>                        struct timespec tm =3D {3, 0};
>>
>> --- autofs-5.0.1.orig/include/defaults.h
>> +++ autofs-5.0.1/include/defaults.h
>> @@ -24,6 +24,7 @@
>>
>>  #define DEFAULT_TIMEOUT                        600
>>  #define DEFAULT_NEGATIVE_TIMEOUT       60
>> +#define DEFAULT_MOUNT_WAIT             -1
>>  #define DEFAULT_UMOUNT_WAIT            12
>>  #define DEFAULT_BROWSE_MODE            1
>>  #define DEFAULT_LOGGING                        0
>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>  struct ldap_searchdn *defaults_get_searchdns(void);
>>  void defaults_free_searchdns(struct ldap_searchdn *);
>>  unsigned int defaults_get_append_options(void);
>> +unsigned int defaults_get_mount_wait(void);
>>  unsigned int defaults_get_umount_wait(void);
>>  const char *defaults_get_auth_conf_file(void);
>>  unsigned int defaults_get_map_hash_table_size(void);
>> --- autofs-5.0.1.orig/lib/defaults.c
>> +++ autofs-5.0.1/lib/defaults.c
>> @@ -45,6 +45,7 @@
>>  #define ENV_NAME_VALUE_ATTR            "VALUE_ATTRIBUTE"
>>
>>  #define ENV_APPEND_OPTIONS             "APPEND_OPTIONS"
>> +#define ENV_MOUNT_WAIT                 "MOUNT_WAIT"
>>  #define ENV_UMOUNT_WAIT                        "UMOUNT_WAIT"
>>  #define ENV_AUTH_CONF_FILE             "AUTH_CONF_FILE"
>>
>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>                    check_set_config_value(key, ENV_NAME_ENTRY_ATTR, valu=
e, to_syslog) ||
>>                    check_set_config_value(key, ENV_NAME_VALUE_ATTR, valu=
e, to_syslog) ||
>>                    check_set_config_value(key, ENV_APPEND_OPTIONS, value=
, to_syslog) ||
>> +                   check_set_config_value(key, ENV_MOUNT_WAIT, value, t=
o_syslog) ||
>>                    check_set_config_value(key, ENV_UMOUNT_WAIT, value, t=
o_syslog) ||
>>                    check_set_config_value(key, ENV_AUTH_CONF_FILE, value=
, to_syslog) ||
>>                    check_set_config_value(key, ENV_MAP_HASH_TABLE_SIZE, =
value, to_syslog))
>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>        return res;
>>  }
>>
>> +unsigned int defaults_get_mount_wait(void)
>> +{
>> +       long wait;
>> +
>> +       wait =3D get_env_number(ENV_MOUNT_WAIT);
>> +       if (wait < 0)
>> +               wait =3D DEFAULT_MOUNT_WAIT;
>> +
>> +       return (unsigned int) wait;
>> +}
>> +
>>  unsigned int defaults_get_umount_wait(void)
>>  {
>>        long wait;
>> --- autofs-5.0.1.orig/man/auto.master.5.in
>> +++ autofs-5.0.1/man/auto.master.5.in
>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>  60). If the equivalent command line option is given it will override th=
is
>>  setting.
>>  .TP
>> +.B MOUNT_WAIT
>> +Set the default time to wait for a response from a spawned mount(8)
>> +before sending it a SIGTERM. Note that we still need to wait for the
>> +RPC layer to timeout before the sub-process exits so this isn't ideal
>> +but it is the best we can do. The default is to wait until mount(8)
>> +returns without intervention.
>> +.TP
>>  .B UMOUNT_WAIT
>>  Set the default time to wait for a response from a spawned umount(8)
>>  before sending it a SIGTERM. Note that we still need to wait for the
>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>  #
>>  #NEGATIVE_TIMEOUT=3D60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=3D-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=3D12
>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>  #
>>  #NEGATIVE_TIMEOUT=3D60
>>  #
>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>> +#             Setting this timeout can cause problems when
>> +#             mount would otherwise wait for a server that
>> +#             is temporarily unavailable, such as when it's
>> +#             restarting. The defailt of waiting for mount(8)
>> +#             usually results in a wait of around 3 minutes.
>> +#
>> +#MOUNT_WAIT=3D-1
>> +#
>>  # UMOUNT_WAIT - time to wait for a response from umount(8).
>>  #
>>  #UMOUNT_WAIT=3D12
>>
>>
>>> Thanks!
>>>
>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>> Carlos Andr=E9 wrote:
>>>>>> Today (2009-08-12) I'm using:
>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>> Thanks,
>>>>>
>>>>> My mistake, the wait time I was referring to is used for umounts duri=
ng
>>>>> expires and is present in rev rc2.102.
>>>>>
>>>>> It shouldn't be hard to add this for mount as well.
>>>>> Would you like me to put something together?
>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks :)
>>>>
>>>>> Probably would be good to test something out to see if we can make a
>>>>> difference with the killing mount after some configured timeout but, =
if
>>>>> we make progress, probably the best way to deal with it is for you to
>>>>> log a bug against rhel-5 so I can get it committed to the rhel packag=
e.
>>>>> The possible issue is that I'm not sure if the RPC subsystem in the
>>>>> above rhel kernel will respond well to process death with potential
>>>>> outstanding requests. But we'll see.
>>>> Ok, on my way :)
>>>>
>>>> Thanks a lot!
>>>>
>>>>>>
>>>>>> Look my last test:
>>>>>> --------------------------------------------------------------
>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>> ls: testdown: No such file or directory
>>>>>>
>>>>>> real    3m9.025s
>>>>>> user    0m0.000s
>>>>>> sys     0m0.002s
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun):
>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdown,
>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlink=3D0=
, ro=3D0
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(nfs):
>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 path /=
misc
>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_proc =3D
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready(): sta=
te
>>>>>> =3D 2 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 path /=
misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_proc =3D
>>>>>> 3078093712 path /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect: 2
>>>>>> submounts remaining in /misc
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got thid
>>>>>> 3078093712 path /misc stat 3
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigchld:
>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready(): sta=
te
>>>>>> =3D 2 path /misc
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to NFS
>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mount
>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D 17
>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /misc/are=
as/testdown
>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 path /=
misc
>>>>>> --------------------------------------------------------------
>>>>>>
>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>> Hi Ian,
>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount... this o=
ption
>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5/krb5=
i/krb5p)
>>>>>>>> like you can see on my previous emails...
>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>
>>>>>>> Maybe this is related to the same sort of problem we had with mount=
 in
>>>>>>> the past, before the options parsing went into the kernel, where ot=
her
>>>>>>> services, like portmapper (or rpcbind), were being done with differ=
ent
>>>>>>> timeout parameters before the RPC calls for mounting. That's just an
>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>
>>>>>>> But what version of autofs and kernel did you say you were using?
>>>>>>>
>>>>>>>> I appreciate any help.
>>>>>>>>
>>>>>>>> Carlos.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>> This long timeout is good if workstation need mount a critical
>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any sens=
e,
>>>>>>>>>>> since autofs retry mount directory on-access. This in fact give=
s me
>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one server =
goes
>>>>>>>>>>> down for any reason, and will again hangs if user try access di=
rectory
>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as the fir=
st
>>>>>>>>>> mount(2) system call fails.  When you set SYN retries to 1, this=
 means
>>>>>>>>>> after 9 seconds, the connect fails, and that causes the mount(2)=
 system
>>>>>>>>>> call to fail.
>>>>>>>>>>
>>>>>>>>>> Recent conversations with Ian suggested that a long timeout was =
desired
>>>>>>>>>> for automounter as well as other cases.  Ian, is there something=
 else we
>>>>>>>>>> need to consider to determine the correct retry timeout for NFS/=
TCP
>>>>>>>>>> mount points handled via automounter?  How should mount.nfs wait=
 so we
>>>>>>>>>> don't make other use cases worse?  (Looks like most of the histo=
ry is
>>>>>>>>>> intact below).
>>>>>>>>> Of course we know that autofs is entirely at the mercy of mount(8=
) (and
>>>>>>>>> mount.nfs in particular). This has always been a difficult situat=
ion for
>>>>>>>>> the automounter because interactive mount invocations should wait=
. But I
>>>>>>>>> believe automount mounts should always time out quickly, but that=
 leads
>>>>>>>>> to its own set of problems, especially when home directories are =
concerned.
>>>>>>>>>
>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself but I'=
m not
>>>>>>>>> certain that will work as we expect. I'll have to do some experim=
entation.
>>>>>>>>>
>>>>>>>>>> How long do you think is appropriate for the automounter to wait=
 if the
>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>
>>>>>>>>>>> Am losing something or there have was something weirdo...!?
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_retries =
 [DEFAULT]
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.003s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    3m9.001s
>>>>>>>>>>> user    0m0.002s
>>>>>>>>>>> sys     0m0.001s
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries =
[ 5 to 1 ]
>>>>>>>>>>>
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryin=
g). [x 6]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    1m3.002s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryin=
g). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.000s
>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    0m9.003s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs4 -o
>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retryin=
g). [x 13]
>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (giving =
up).
>>>>>>>>>>>
>>>>>>>>>>> real    2m6.001s
>>>>>>>>>>> user    0m0.001s
>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 to 1..=
. and
>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>
>>>>>>>>>>> *sigh*
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_syn_re=
tries to
>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>> Right.  Normally the RPC client calls the kernel's socket conn=
ect
>>>>>>>>>>>> function,
>>>>>>>>>>>> which does 6 SYN retries.  That one call usually takes longer =
than
>>>>>>>>>>>> the RPC
>>>>>>>>>>>> client's connect timeout, so it only makes one connect call, a=
nd then
>>>>>>>>>>>> fails.
>>>>>>>>>>>>
>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt causes =
the RPC
>>>>>>>>>>>> client
>>>>>>>>>>>> to retry the connect call until its connect timeout expires.  =
Each
>>>>>>>>>>>> connect
>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givin=
g up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    3m9.000s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.002s
>>>>>>>>>>>>>
>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_retries
>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t nfs4 -o
>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp  ("retry=3D1" =3D no change)
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (retry=
ing).
>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (givin=
g up).
>>>>>>>>>>>>>
>>>>>>>>>>>>> real    2m6.004s
>>>>>>>>>>>>> user    0m0.000s
>>>>>>>>>>>>> sys     0m0.004s
>>>>>>>>>>>>>
>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've monit=
ored
>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs o=
n port
>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any change..=
. I dont
>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>> Bruce, no... you're right.  I'm describing a situation whe=
re my
>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) than =
3 minutes
>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the kern=
el to
>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN attempt=
s with
>>>>>>>>>>>>>>> exponential retries, or something like that).  For stock Ce=
ntOS
>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mounts -=
- the
>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no precedi=
ng rpcbind
>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-related Ce=
ntOS
>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourself.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wr=
ote:
>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candrecn@g=
mail.com>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to=
 work with
>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server goes do=
wn i get a
>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user logon pr=
ocess, if
>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to timeout =
(if server
>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinations, =
there my
>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10)=
 using
>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 =
-o
>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (proto=3D=
tcp OR
>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show =
error
>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace period.
>>>>>>>>>>>>>>>>> I thought he was describing a situation where the server =
the server
>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wondering h=
ow to make
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>> mount fail faster.  But I may be misunderstanding.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-i=
nfo.html
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Chuck Lever
>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Carlos André]

Ok then, i'll be waiting for patch :)

Thanks a lot.

2009/9/22 Ian Kent <ikent@redhat.com>:
> Carlos Andr=E9 wrote:
>> Hi Ian,
>>
>> Thanks for patch and sorry for delay (i'm expecting receive u reply =
on
>> bug track, not here) :)
>>
>> But, this patch doesnt worked to me like expected... =A0:(
>
> OK, I've been off on a wild goose chase, thinking this was related to
> the moving of the mount option handling and initial file handle open
> into the kernel, but that isn't even included in the kernel you are
> using. Suffice it to say this behaviour exists at least back to RHEL-=
4
> and NFS v3 and v2 mount take around 1 minute to time out and v4 about=
 3
> minutes. Not only that, mount attempts from the command line appear t=
o
> respond to an TERM signal, including using a relatively recent kernel=
,
> but I might not have that quite right.
>
> Anyway, now that I'm back on track, we might make some progress.
>
>>
>>
>> Firstly I've changed "#MOUNT_WAIT=3D-1" to "MOUNT_WAIT=3D10"
>> and later changed "10" to "2" with same results...
>> (always restarting service, of course :)
>>
>> Then, tried remove "sec=3Dkrb5p", and later removed "nfs4" but i got
>> same results again.
>>
>> Or i'm doing something wrong?
>
> Maybe.
>
> I've tested this out now with some interesting results.
> I can't easily setup Kerberos for NFS so lets work on plain mounts to
> begin with.
>
> Using the patch I posted with plain mounts autofs did indeed return
> after the configured timeout. After sending the TERM signal to the mo=
unt
> the mount process went away but the mount.nfs child process remained
> waiting for to timeout. User space received the usual ENOENT error af=
ter
> the configured timeout. The same occurred with nfs4. This is much the
> same as the timed umount behaviour so it's expected.
>
> So, there must be something wrong with the patching of autofs.
> I'll put together a patched RHEL package and we will continue this in
> the RedHat bug you've logged.
>
>>
>>
>> [root@KSTATION areas]# automount -V
>>
>> Linux automount version 5.0.1-0.rc2.131.bz517349.1
>> [...]
>>
>> [root@KSTATION areas]# time ls -la testdown
>> ls: testedown: No such file or directory
>>
>> real =A0 =A03m9.006s
>> user =A0 =A00m0.002s
>> sys =A0 =A0 0m0.000s
>>
>>
>> LOGGING:
>> -----------------------------------------
>> Aug 24 09:23:51 KSTATION automount[20803]: mount_mount: mount(nfs):
>> calling mount -t nfs4 -s -o rw,acl,sec=3Dkrb5p 1.2.3.4:/areas/testdo=
wn
>> /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: mount(nfs): nfs: mount
>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>> Aug 24 09:27:00 KSTATION automount[20803]: ioctl_send_fail: token =3D=
 91
>> Aug 24 09:27:00 KSTATION automount[20803]: failed to mount /misc/are=
as/testdown
>> -----------------------------------------
>>
>>
>>
>>
>>
>> 2009/8/17 Ian Kent <ikent@redhat.com>:
>>> On Thu, 2009-08-13 at 12:18 -0300, Carlos Andr=E9 wrote:
>>>> Filled bug report:
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=3D517349
>>> Hi Carlos,
>>>
>>> I have a patched source rpm to add a mount wait parameter to autofs
>>> located at:
>>> http://people.redhat.com/~ikent/autofs-5.0.1-0.rc2.131.bz517349.1
>>>
>>> Could you build it and see if it works.
>>> I haven't tested it at all but it is fairly straight forward.
>>> It is still unclear if this is the right way to do this and what th=
e
>>> consequences are in sending a term signal to mount. This mount requ=
est
>>> will likely be followed by other requests for the same mount causin=
g an
>>> accumulation of mount(8) processes waiting for RPC timeouts before =
they
>>> can answer the TERM signal.
>>>
>>> Anyway, for information the patch included in the source rpm above =
is:
>>>
>>> autofs-5.0.4 - add mount wait parameter
>>>
>>> From: Ian Kent <raven@themaw.net>
>>>
>>> Often delays when trying to mount from a server that is not repondi=
ng
>>> for some reason are undesirable. To try and prevent these delays we
>>> provide a configuration setting to limit the time that we wait for
>>> our spawned mount(8) process to complete before sending it a SIGTER=
M
>>> signal. This patch adds a configuration parameter to allow us to
>>> request we limit the time we wait for mount(8) to complete before
>>> send it a TERM signal.
>>> ---
>>>
>>> =A0daemon/spawn.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A03 ++-
>>> =A0include/defaults.h =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A02 ++
>>> =A0lib/defaults.c =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 13 ++++++++=
+++++
>>> =A0man/auto.master.5.in =A0 =A0 =A0 =A0 =A0 | =A0 =A07 +++++++
>>> =A0redhat/autofs.sysconfig.in =A0 =A0 | =A0 =A09 +++++++++
>>> =A0samples/autofs.conf.default.in | =A0 =A09 +++++++++
>>> =A06 files changed, 42 insertions(+), 1 deletion(-)
>>>
>>>
>>> --- autofs-5.0.1.orig/daemon/spawn.c
>>> +++ autofs-5.0.1/daemon/spawn.c
>>> @@ -312,6 +312,7 @@ int spawn_mount(unsigned logopt, ...)
>>> =A0 =A0 =A0 =A0unsigned int options;
>>> =A0 =A0 =A0 =A0unsigned int retries =3D MTAB_LOCK_RETRIES;
>>> =A0 =A0 =A0 =A0int update_mtab =3D 1, ret, printed =3D 0;
>>> + =A0 =A0 =A0 unsigned int wait =3D defaults_get_mount_wait();
>>> =A0 =A0 =A0 =A0char buf[PATH_MAX];
>>>
>>> =A0 =A0 =A0 =A0/* If we use mount locking we can't validate the loc=
ation */
>>> @@ -353,7 +354,7 @@ int spawn_mount(unsigned logopt, ...)
>>> =A0 =A0 =A0 =A0va_end(arg);
>>>
>>> =A0 =A0 =A0 =A0while (retries--) {
>>> - =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, -1, options,=
 prog, (const char **) argv);
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 ret =3D do_spawn(logopt, wait, option=
s, prog, (const char **) argv);
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0if (ret & MTAB_NOTUPDATED) {
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0struct timespec tm =3D=
 {3, 0};
>>>
>>> --- autofs-5.0.1.orig/include/defaults.h
>>> +++ autofs-5.0.1/include/defaults.h
>>> @@ -24,6 +24,7 @@
>>>
>>> =A0#define DEFAULT_TIMEOUT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0600
>>> =A0#define DEFAULT_NEGATIVE_TIMEOUT =A0 =A0 =A0 60
>>> +#define DEFAULT_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 -1
>>> =A0#define DEFAULT_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A012
>>> =A0#define DEFAULT_BROWSE_MODE =A0 =A0 =A0 =A0 =A0 =A01
>>> =A0#define DEFAULT_LOGGING =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A00
>>> @@ -62,6 +63,7 @@ struct ldap_schema *defaults_get_schema(
>>> =A0struct ldap_searchdn *defaults_get_searchdns(void);
>>> =A0void defaults_free_searchdns(struct ldap_searchdn *);
>>> =A0unsigned int defaults_get_append_options(void);
>>> +unsigned int defaults_get_mount_wait(void);
>>> =A0unsigned int defaults_get_umount_wait(void);
>>> =A0const char *defaults_get_auth_conf_file(void);
>>> =A0unsigned int defaults_get_map_hash_table_size(void);
>>> --- autofs-5.0.1.orig/lib/defaults.c
>>> +++ autofs-5.0.1/lib/defaults.c
>>> @@ -45,6 +45,7 @@
>>> =A0#define ENV_NAME_VALUE_ATTR =A0 =A0 =A0 =A0 =A0 =A0"VALUE_ATTRIB=
UTE"
>>>
>>> =A0#define ENV_APPEND_OPTIONS =A0 =A0 =A0 =A0 =A0 =A0 "APPEND_OPTIO=
NS"
>>> +#define ENV_MOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 "MOUNT_WAIT=
"
>>> =A0#define ENV_UMOUNT_WAIT =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =
=A0 =A0"UMOUNT_WAIT"
>>> =A0#define ENV_AUTH_CONF_FILE =A0 =A0 =A0 =A0 =A0 =A0 "AUTH_CONF_FI=
LE"
>>>
>>> @@ -323,6 +324,7 @@ unsigned int defaults_read_config(unsign
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_NAME_ENTRY_ATTR, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_NAME_VALUE_ATTR, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_APPEND_OPTIONS, value, to_syslog) ||
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 check_set_config_value(key, E=
NV_MOUNT_WAIT, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_UMOUNT_WAIT, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_AUTH_CONF_FILE, value, to_syslog) ||
>>> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0check_set_config_value(key, =
ENV_MAP_HASH_TABLE_SIZE, value, to_syslog))
>>> @@ -652,6 +654,17 @@ unsigned int defaults_get_append_options
>>> =A0 =A0 =A0 =A0return res;
>>> =A0}
>>>
>>> +unsigned int defaults_get_mount_wait(void)
>>> +{
>>> + =A0 =A0 =A0 long wait;
>>> +
>>> + =A0 =A0 =A0 wait =3D get_env_number(ENV_MOUNT_WAIT);
>>> + =A0 =A0 =A0 if (wait < 0)
>>> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 wait =3D DEFAULT_MOUNT_WAIT;
>>> +
>>> + =A0 =A0 =A0 return (unsigned int) wait;
>>> +}
>>> +
>>> =A0unsigned int defaults_get_umount_wait(void)
>>> =A0{
>>> =A0 =A0 =A0 =A0long wait;
>>> --- autofs-5.0.1.orig/man/auto.master.5.in
>>> +++ autofs-5.0.1/man/auto.master.5.in
>>> @@ -175,6 +175,13 @@ Set the default timeout for caching fail
>>> =A060). If the equivalent command line option is given it will over=
ride this
>>> =A0setting.
>>> =A0.TP
>>> +.B MOUNT_WAIT
>>> +Set the default time to wait for a response from a spawned mount(8=
)
>>> +before sending it a SIGTERM. Note that we still need to wait for t=
he
>>> +RPC layer to timeout before the sub-process exits so this isn't id=
eal
>>> +but it is the best we can do. The default is to wait until mount(8=
)
>>> +returns without intervention.
>>> +.TP
>>> =A0.B UMOUNT_WAIT
>>> =A0Set the default time to wait for a response from a spawned umoun=
t(8)
>>> =A0before sending it a SIGTERM. Note that we still need to wait for=
 the
>>> --- autofs-5.0.1.orig/redhat/autofs.sysconfig.in
>>> +++ autofs-5.0.1/redhat/autofs.sysconfig.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>> =A0#
>>> =A0#NEGATIVE_TIMEOUT=3D60
>>> =A0#
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems =
when
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server =
that
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when=
 it's
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for m=
ount(8)
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 mi=
nutes.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
>>> =A0#
>>> =A0#UMOUNT_WAIT=3D12
>>> --- autofs-5.0.1.orig/samples/autofs.conf.default.in
>>> +++ autofs-5.0.1/samples/autofs.conf.default.in
>>> @@ -14,6 +14,15 @@ TIMEOUT=3D300
>>> =A0#
>>> =A0#NEGATIVE_TIMEOUT=3D60
>>> =A0#
>>> +# MOUNT_WAIT - time to wait for a response from umount(8).
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 Setting this timeout can cause problems =
when
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 mount would otherwise wait for a server =
that
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 is temporarily unavailable, such as when=
 it's
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 restarting. The defailt of waiting for m=
ount(8)
>>> +# =A0 =A0 =A0 =A0 =A0 =A0 usually results in a wait of around 3 mi=
nutes.
>>> +#
>>> +#MOUNT_WAIT=3D-1
>>> +#
>>> =A0# UMOUNT_WAIT - time to wait for a response from umount(8).
>>> =A0#
>>> =A0#UMOUNT_WAIT=3D12
>>>
>>>
>>>> Thanks!
>>>>
>>>> 2009/8/13 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>> 2009/8/13 Ian Kent <ikent@redhat.com>:
>>>>>> Carlos Andr=E9 wrote:
>>>>>>> Today (2009-08-12) I'm using:
>>>>>>> kernel-2.6.18-128.2.1.el5
>>>>>>> autofs-5.0.1-0.rc2.102.el5_3.1
>>>>>> Thanks,
>>>>>>
>>>>>> My mistake, the wait time I was referring to is used for umounts=
 during
>>>>>> expires and is present in rev rc2.102.
>>>>>>
>>>>>> It shouldn't be hard to add this for mount as well.
>>>>>> Would you like me to put something together?
>>>>> Sure! that 'll help me a lot (and for sure another ppl) :) Thanks=
 :)
>>>>>
>>>>>> Probably would be good to test something out to see if we can ma=
ke a
>>>>>> difference with the killing mount after some configured timeout =
but, if
>>>>>> we make progress, probably the best way to deal with it is for y=
ou to
>>>>>> log a bug against rhel-5 so I can get it committed to the rhel p=
ackage.
>>>>>> The possible issue is that I'm not sure if the RPC subsystem in =
the
>>>>>> above rhel kernel will respond well to process death with potent=
ial
>>>>>> outstanding requests. But we'll see.
>>>>> Ok, on my way :)
>>>>>
>>>>> Thanks a lot!
>>>>>
>>>>>>>
>>>>>>> Look my last test:
>>>>>>> --------------------------------------------------------------
>>>>>>> [root@KSTATION areas]# time ls testdown
>>>>>>> ls: testdown: No such file or directory
>>>>>>>
>>>>>>> real =A0 =A03m9.025s
>>>>>>> user =A0 =A00m0.000s
>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: sun_mount: parse(sun=
):
>>>>>>> mounting root /misc/areas, mountpoint testdown, what
>>>>>>> 1.2.3.4:/areas/testdown, fstype nfs4, options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: do_mount:
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown type nfs4 options
>>>>>>> acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0 using module nfs4
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(n=
fs):
>>>>>>> root=3D/misc/areas name=3Dtestdown what=3D1.2.3.4:/areas/testdo=
wn,
>>>>>>> fstype=3Dnfs4, options=3Dacl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(n=
fs):
>>>>>>> nfs options=3D"acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D0", nosymlin=
k=3D0, ro=3D0
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(n=
fs):
>>>>>>> calling mkdir_path /misc/areas/testdown
>>>>>>> Aug 12 12:57:07 KSTATION automount[15471]: mount_mount: mount(n=
fs):
>>>>>>> calling mount -t nfs4 -s -o acl,sec=3Dkrb5p,proto=3Dtcp,retry=3D=
0
>>>>>>> 1.2.3.4:/areas/testdown /misc/areas/testdown
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: st_expire: state 1 p=
ath /misc
>>>>>>> Aug 12 12:58:12 KSTATION automount[15471]: expire_proc: exp_pro=
c =3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_proc_indirect=
: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: got =
thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: expire_cleanup: sigc=
hld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:58:13 KSTATION automount[15471]: st_ready: st_ready()=
: state
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_expire: state 1 p=
ath /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc: exp_pro=
c =3D
>>>>>>> 3078093712 path /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_proc_indirect=
: 2
>>>>>>> submounts remaining in /misc
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: got =
thid
>>>>>>> 3078093712 path /misc stat 3
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: expire_cleanup: sigc=
hld:
>>>>>>> exp 3078093712 finished, switching from 2 to 1
>>>>>>> Aug 12 12:59:28 KSTATION automount[15471]: st_ready: st_ready()=
: state
>>>>>>> =3D 2 path /misc
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: >> mount: mount to N=
=46S
>>>>>>> server '1.2.3.4' failed: timed out (giving up).
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: mount(nfs): nfs: mou=
nt
>>>>>>> failure 1.2.3.4:/areas/testdown on /misc/areas/testdown
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: send_fail: token =3D=
 17
>>>>>>> Aug 12 13:00:16 KSTATION automount[15471]: failed to mount /mis=
c/areas/testdown
>>>>>>> Aug 12 13:00:43 KSTATION automount[15471]: st_expire: state 1 p=
ath /misc
>>>>>>> --------------------------------------------------------------
>>>>>>>
>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>> Carlos Andr=E9 wrote:
>>>>>>>>> Hi Ian,
>>>>>>>>> I'm getting crazy trying put "retry=3D" to work on mount... t=
his option
>>>>>>>>> just DONT WORK if use proto=3Dtcp and/OR kerberos (sec=3Dkrb5=
/krb5i/krb5p)
>>>>>>>>> like you can see on my previous emails...
>>>>>>>> Right, my mistake for not looking closely enough at post.
>>>>>>>>
>>>>>>>> Maybe this is related to the same sort of problem we had with =
mount in
>>>>>>>> the past, before the options parsing went into the kernel, whe=
re other
>>>>>>>> services, like portmapper (or rpcbind), were being done with d=
ifferent
>>>>>>>> timeout parameters before the RPC calls for mounting. That's j=
ust an
>>>>>>>> example as NFSv4 shouldn't be sensitive to portmapper anyway.
>>>>>>>>
>>>>>>>> But what version of autofs and kernel did you say you were usi=
ng?
>>>>>>>>
>>>>>>>>> I appreciate any help.
>>>>>>>>>
>>>>>>>>> Carlos.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2009/8/12 Ian Kent <ikent@redhat.com>:
>>>>>>>>>> Chuck Lever wrote:
>>>>>>>>>>> On Aug 11, 2009, at 8:41 AM, Carlos Andr=E9 wrote:
>>>>>>>>>>>> This long timeout is good if workstation need mount a crit=
ical
>>>>>>>>>>>> directory using /etc/fstab on boot (for example)..
>>>>>>>>>>>> But in my case, using this loooong timeout doesnt make any=
 sense,
>>>>>>>>>>>> since autofs retry mount directory on-access. This in fact=
 gives me
>>>>>>>>>>>> alot of headaches, coz user login 'll just hangs if one se=
rver goes
>>>>>>>>>>>> down for any reason, and will again hangs if user try acce=
ss directory
>>>>>>>>>>>> pointing to a NFS down server...
>>>>>>>>>>> "retry=3D0" means the mount command will fail as soon as th=
e first
>>>>>>>>>>> mount(2) system call fails. =A0When you set SYN retries to =
1, this means
>>>>>>>>>>> after 9 seconds, the connect fails, and that causes the mou=
nt(2) system
>>>>>>>>>>> call to fail.
>>>>>>>>>>>
>>>>>>>>>>> Recent conversations with Ian suggested that a long timeout=
 was desired
>>>>>>>>>>> for automounter as well as other cases. =A0Ian, is there so=
mething else we
>>>>>>>>>>> need to consider to determine the correct retry timeout for=
 NFS/TCP
>>>>>>>>>>> mount points handled via automounter? =A0How should mount.n=
fs wait so we
>>>>>>>>>>> don't make other use cases worse? =A0(Looks like most of th=
e history is
>>>>>>>>>>> intact below).
>>>>>>>>>> Of course we know that autofs is entirely at the mercy of mo=
unt(8) (and
>>>>>>>>>> mount.nfs in particular). This has always been a difficult s=
ituation for
>>>>>>>>>> the automounter because interactive mount invocations should=
 wait. But I
>>>>>>>>>> believe automount mounts should always time out quickly, but=
 that leads
>>>>>>>>>> to its own set of problems, especially when home directories=
 are concerned.
>>>>>>>>>>
>>>>>>>>>> I think adding "retry=3D0" is the right thing to do myself b=
ut I'm not
>>>>>>>>>> certain that will work as we expect. I'll have to do some ex=
perimentation.
>>>>>>>>>>
>>>>>>>>>>> How long do you think is appropriate for the automounter to=
 wait if the
>>>>>>>>>>> server is down, in your case, Carlos?
>>>>>>>>>>>
>>>>>>>>>>>> Am losing something or there have was something weirdo...!=
?
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> [root@KSTATION ~]# echo 5 > /proc/sys/net/ipv4/tcp_syn_ret=
ries =A0[DEFAULT]
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>> user =A0 =A00m0.002s
>>>>>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.001s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.003s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A03m9.001s
>>>>>>>>>>>> user =A0 =A00m0.002s
>>>>>>>>>>>> sys =A0 =A0 0m0.001s
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# echo 1 > /proc/sys/net/ipv4/tcp_syn_ret=
ries [ 5 to 1 ]
>>>>>>>>>>>>
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (re=
trying). [x 6]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A01m3.002s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D1
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (re=
trying). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A02m6.000s
>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A00m9.003s
>>>>>>>>>>>> user =A0 =A00m0.001s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]# time mount 1.2.3.4:/blabla /tmp/ -t nfs=
4 -o
>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp,retry=3D0
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (re=
trying). [x 13]
>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (gi=
ving up).
>>>>>>>>>>>>
>>>>>>>>>>>> real =A0 =A02m6.001s
>>>>>>>>>>>> user =A0 =A00m0.001s
>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>> [root@KSTATION ~]#
>>>>>>>>>>>> ------------------------------------------------
>>>>>>>>>>>> max timeout goes to 2m6s changing tcp_syn_retries from 5 t=
o 1... and
>>>>>>>>>>>> using retry=3D0 without kerberos I got only 9s...
>>>>>>>>>>>>
>>>>>>>>>>>> *sigh*
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>> On Aug 10, 2009, at 4:05 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>> Something funny: Using default tcp_syn_retries (5) i got
>>>>>>>>>>>>>> "3,6,12,24,48,96" secs interval... but if i change tcp_s=
yn_retries to
>>>>>>>>>>>>>> 1 i got "3,6,3,6,3,6..." secs interval...
>>>>>>>>>>>>> Right. =A0Normally the RPC client calls the kernel's sock=
et connect
>>>>>>>>>>>>> function,
>>>>>>>>>>>>> which does 6 SYN retries. =A0That one call usually takes =
longer than
>>>>>>>>>>>>> the RPC
>>>>>>>>>>>>> client's connect timeout, so it only makes one connect ca=
ll, and then
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Reducing the number of SYN retries per connect attempt ca=
uses the RPC
>>>>>>>>>>>>> client
>>>>>>>>>>>>> to retry the connect call until its connect timeout expir=
es. =A0Each
>>>>>>>>>>>>> connect
>>>>>>>>>>>>> call resets the SYN timeout to 3 seconds.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t n=
fs4 -o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
giving up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real =A0 =A03m9.000s
>>>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>>>> sys =A0 =A0 0m0.002s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [root@KSERVER /]# echo 1 > /proc/sys/net/ipv4/tcp_syn_re=
tries
>>>>>>>>>>>>>> [root@KSERVER mnt]# time mount 1.2.3.4:/blabla tmp/ -t n=
fs4 -o
>>>>>>>>>>>>>> sec=3Dkrb5p,proto=3Dtcp =A0("retry=3D1" =3D no change)
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
retrying).
>>>>>>>>>>>>>> mount: mount to NFS server '1.2.3.4' failed: timed out (=
giving up).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> real =A0 =A02m6.004s
>>>>>>>>>>>>>> user =A0 =A00m0.000s
>>>>>>>>>>>>>> sys =A0 =A0 0m0.004s
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (3,6,3,6... secs interval)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2009/8/10 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>> No, i'm just using packages from CentOS repo...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And u're right about expo retries... with tcpdump i've =
monitored
>>>>>>>>>>>>>>> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 s=
ecs on port
>>>>>>>>>>>>>>> 2049...
>>>>>>>>>>>>>>> I tried use "retry=3D1" option on mount without any cha=
nge... I dont
>>>>>>>>>>>>>>> want change source or tcp timers... just NFSv4 client.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>>>>>>>>>>>>>>>> On Aug 10, 2009, at 2:29 PM, Carlos Andr=E9 wrote:
>>>>>>>>>>>>>>>>> Bruce, no... you're right. =A0I'm describing a situat=
ion where my
>>>>>>>>>>>>>>>>> server
>>>>>>>>>>>>>>>>> died... i need mount fail faster (10 or 15 secs max) =
than 3 minutes
>>>>>>>>>>>>>>>>> and 9 seconds...
>>>>>>>>>>>>>>>> The 189 second timeout is likely how long it takes the=
 kernel to
>>>>>>>>>>>>>>>> give up
>>>>>>>>>>>>>>>> trying to connect a TCP socket to the server (6 SYN at=
tempts with
>>>>>>>>>>>>>>>> exponential retries, or something like that). =A0For s=
tock CentOS
>>>>>>>>>>>>>>>> 5.3, I
>>>>>>>>>>>>>>>> think
>>>>>>>>>>>>>>>> user space does only a DNS lookup for normal NFSv4 mou=
nts -- the
>>>>>>>>>>>>>>>> kernel
>>>>>>>>>>>>>>>> just
>>>>>>>>>>>>>>>> tries to connect a TCP socket to port 2049, with no pr=
eceding rpcbind
>>>>>>>>>>>>>>>> request.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Carlos, let us know if you have replaced any NFS-relat=
ed CentOS
>>>>>>>>>>>>>>>> components
>>>>>>>>>>>>>>>> (kernel, nfs-utils) with something you've built yourse=
lf.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>>>>>>>>>>>>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Hale=
vy wrote:
>>>>>>>>>>>>>>>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos Andr=E9 <candr=
ecn@gmail.com>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>> Anyone ?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2009/7/29 Carlos Andr=E9 <candrecn@gmail.com>:
>>>>>>>>>>>>>>>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 serv=
er to work with
>>>>>>>>>>>>>>>>>>>>> Kerberos
>>>>>>>>>>>>>>>>>>>>> and AutoFS, but i got a problem: If NFS server go=
es down i get a
>>>>>>>>>>>>>>>>>>>>> LOOOOOOONG
>>>>>>>>>>>>>>>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 clien=
t...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Since i need mount some (3 to 6) dirs at user log=
on process, if
>>>>>>>>>>>>>>>>>>>>> mount
>>>>>>>>>>>>>>>>>>>>> hangs,
>>>>>>>>>>>>>>>>>>>>> user logon hangs. Then i want configure it to tim=
eout (if server
>>>>>>>>>>>>>>>>>>>>> down)
>>>>>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I already make a lab and tried a LOT of combinati=
ons, there my
>>>>>>>>>>>>>>>>>>>>> findings
>>>>>>>>>>>>>>>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.=
1.10) using
>>>>>>>>>>>>>>>>>>>>> basic
>>>>>>>>>>>>>>>>>>>>> command
>>>>>>>>>>>>>>>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t =
nfs4 -o
>>>>>>>>>>>>>>>>>>>>> sec=3Dkrb5,proto=3D<tcp/udp>) from NFS client:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> - Once i try access mount point using AutoFS (pro=
to=3Dtcp OR
>>>>>>>>>>>>>>>>>>>>> proto=3Dudp)
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>>>>>>> hangs for 189 secs (3m9s: real =A03m9.001s) =A0un=
til show error
>>>>>>>>>>>>>>>>>>>>> (mount:
>>>>>>>>>>>>>>>>>>>>> mount to
>>>>>>>>>>>>>>>>>>>>> NFS server '172.16.0.10' failed: timed out (givin=
g up))
>>>>>>>>>>>>>>>>>>> Sounds like you're hitting the server's grace perio=
d.
>>>>>>>>>>>>>>>>>> I thought he was describing a situation where the se=
rver the server
>>>>>>>>>>>>>>>>>> is completely gone and isn't coming back, and wonder=
ing how to make
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>> mount fail faster. =A0But I may be misunderstanding.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --b.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubsc=
ribe
>>>>>>>>>>>>>>>>> linux-nfs" in
>>>>>>>>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>>>>>>>>> More majordomo info at =A0http://vger.kernel.org/majo=
rdomo-info.html
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Chuck Lever
>>>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Chuck Lever
>>>>>>>>>>> chuck[dot]lever[at]oracle[dot]com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>
>>>
>
>

Re: AutoFS+NFSv4 server down = LOOOOONG timeout.

[Chuck Lever]

On Aug 10, 2009, at 3:43 PM, Carlos André wrote:
> No, i'm just using packages from CentOS repo...
>
> And u're right about expo retries... with tcpdump i've monitored
> traffic and i got SYN retries in 3, 6, 12, 24, 48, 96 secs on port
> 2049...
> I tried use "retry=1" option on mount without any change...

That won't have any effect on the kernel's TCP connect behavior.  It  
is simply used by the mount command to know when to stop redriving  
mount(2) system calls.  The current mount command doesn't actually  
interrupt the mount(2) system call if it's taking longer than the  
specified "retry=" setting.

> I don't want change source or tcp timers... just NFSv4 client.

I don't know of any way to effect a change in the kernel's TCP connect  
behavior short of a code change, and that would affect all RPC/TCP  
programs.

Basically the server is down.  I suppose the client's kernel can  
detect this is the case as soon as the ARP request for the server's  
MAC address times out, but normally we retry TCP connects for a while  
(even in this case) because we assume the server is coming back up as  
quickly as it can, and want to catch it as quickly as possible.

But we can't shorten this timeout in the general case, I don't think.   
It could take quite a while on a busy network or if a long round trip  
is involved for a TCP connect to complete.

> 2009/8/10 Chuck Lever <chuck.lever@oracle.com>:
>> On Aug 10, 2009, at 2:29 PM, Carlos André wrote:
>>>
>>> Bruce, no... you're right.  I'm describing a situation where my  
>>> server
>>> died... i need mount fail faster (10 or 15 secs max) than 3 minutes
>>> and 9 seconds...
>>
>> The 189 second timeout is likely how long it takes the kernel to  
>> give up
>> trying to connect a TCP socket to the server (6 SYN attempts with
>> exponential retries, or something like that).  For stock CentOS  
>> 5.3, I think
>> user space does only a DNS lookup for normal NFSv4 mounts -- the  
>> kernel just
>> tries to connect a TCP socket to port 2049, with no preceding rpcbind
>> request.
>>
>> Carlos, let us know if you have replaced any NFS-related CentOS  
>> components
>> (kernel, nfs-utils) with something you've built yourself.
>>
>>> 2009/8/7 J. Bruce Fields <bfields@fieldses.org>:
>>>>
>>>> On Fri, Aug 07, 2009 at 09:42:18AM +0300, Benny Halevy wrote:
>>>>>
>>>>> On Aug. 07, 2009, 3:18 +0300, Carlos André <candrecn@gmail.com>  
>>>>> wrote:
>>>>>>
>>>>>> Anyone ?
>>>>>>
>>>>>> 2009/7/29 Carlos André <candrecn@gmail.com>:
>>>>>>>
>>>>>>> PPL, I need put a CentOS 5.3 (updated) NFSv4 server to work with
>>>>>>> Kerberos
>>>>>>> and AutoFS, but i got a problem: If NFS server goes down i get a
>>>>>>> LOOOOOOONG
>>>>>>> mount timeout on CentOS 5.3 (updated) NFSv4 client...
>>>>>>>
>>>>>>> Since i need mount some (3 to 6) dirs at user logon process,  
>>>>>>> if mount
>>>>>>> hangs,
>>>>>>> user logon hangs. Then i want configure it to timeout (if  
>>>>>>> server down)
>>>>>>> after
>>>>>>> 10-15 secs (MAX) on each mount attempt.
>>>>>>>
>>>>>>> I already make a lab and tried a LOT of combinations, there my
>>>>>>> findings
>>>>>>> (server DOWN IP: 172.16.0.10 / client IP: 172.16.1.10) using  
>>>>>>> basic
>>>>>>> command
>>>>>>> (time mount 172.16.0.10:/remotedir /localdir/ -t nfs4 -o
>>>>>>> sec=krb5,proto=<tcp/udp>) from NFS client:
>>>>>>>
>>>>>>> - Once i try access mount point using AutoFS (proto=tcp OR  
>>>>>>> proto=udp)
>>>>>>> it
>>>>>>> hangs for 189 secs (3m9s: real  3m9.001s)  until show error  
>>>>>>> (mount:
>>>>>>> mount to
>>>>>>> NFS server '172.16.0.10' failed: timed out (giving up))
>>>>>
>>>>> Sounds like you're hitting the server's grace period.
>>>>
>>>> I thought he was describing a situation where the server the server
>>>> is completely gone and isn't coming back, and wondering how to  
>>>> make the
>>>> mount fail faster.  But I may be misunderstanding.
>>>>
>>>> --b.
>>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux- 
>>> nfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> --
>> Chuck Lever
>> chuck[dot]lever[at]oracle[dot]com
>>
>>
>>
>>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com