[PATCH] Timeouts gone wild on ia64

[PATCH] Timeouts gone wild on ia64

[Steve Dickson]

[-- Attachment #1: Type: text/plain, Size: 704 bytes --]

Here is a patch that greatly reduces that number of
timeout (and EIO errors with soft mounts) that
occur when a fast client is talking to a slow server.
 
We were noticing a large number of EIO errors when
a ia64 client was talking to a x86 server with
soft mounts (ala autofs)....

True, EIO errors should be expect with soft mounts but
it turns out that thousands of timeouts were occurring on
a ia64 client compared to 50 to 60 timeouts with
a x86 client when talking to the same slow server and
generating the same traffic.

What this patch does is make the minimal Round Trip
time value relative to HZ. So When HZ is greater (as
in the case of ia64) the minimal value goes up.
 
Comments?

SteveD.


[-- Attachment #2: linux-2.4.20-nfs-ia64-EIO.patch --]
[-- Type: text/plain, Size: 333 bytes --]

--- linux-2.4.20/net/sunrpc/timer.c.orig	2003-05-08 15:10:24.000000000 -0400
+++ linux-2.4.20/net/sunrpc/timer.c	2003-05-08 15:40:01.000000000 -0400
@@ -8,7 +8,7 @@
 
 #define RPC_RTO_MAX (60*HZ)
 #define RPC_RTO_INIT (HZ/5)
-#define RPC_RTO_MIN (2)
+#define RPC_RTO_MIN (HZ/30)
 
 void
 rpc_init_rtt(struct rpc_rtt *rt, long timeo)

Re: [PATCH] Timeouts gone wild on ia64

[Trond Myklebust]

>>>>> " " == Steve Dickson <SteveD@RedHat.com> writes:

     > What this patch does is make the minimal Round Trip time value
     > relative to HZ. So When HZ is greater (as in the case of ia64)
     > the minimal value goes up.
     >  Comments?

1/30 of a second is a long time. Why that particular choice?

Note: "It works" won't cut ice as that is completely a function of the
choice of hardware.

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

Hi Trond,

Trond Myklebust wrote:

>>>>>>" " == Steve Dickson <SteveD@RedHat.com> writes:
>>>>>>            
>>>>>>
>
>     > What this patch does is make the minimal Round Trip time value
>     > relative to HZ. So When HZ is greater (as in the case of ia64)
>     > the minimal value goes up.
>     >  Comments?
>
>1/30 of a second is a long time. Why that particular choice?
>
>Note: "It works" won't cut ice as that is completely a function of the
>choice of hardware.
>
I realize this but... I figured waiting a few extra ticks of time before 
timing
out was probably a good thing with respect to soft mounts to slow 
servers...
Especially  since EIO errors can be pretty disruptive...

And as I see, its a no-op with a fast (or slow) client  talking to a fast
server because the timeout will should never happen... (i.e. the waiting
thread will get the response before the time out occurs)....

SteveD.





-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [PATCH] Timeouts gone wild on ia64

[Lever, Charles]

[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]

steve-
 
can you explain why there are more timeouts for ia64?  do you
have a network trace you can share?

-----Original Message----- 
From: Steve Dickson [mailto:SteveD@RedHat.com] 
Sent: Fri 5/9/2003 8:41 AM 
To: nfs@lists.sourceforge.net 
Cc: 
Subject: [NFS] [PATCH] Timeouts gone wild on ia64



Here is a patch that greatly reduces that number of 
timeout (and EIO errors with soft mounts) that 
occur when a fast client is talking to a slow server. 
  
We were noticing a large number of EIO errors when 
a ia64 client was talking to a x86 server with 
soft mounts (ala autofs).... 

True, EIO errors should be expect with soft mounts but 
it turns out that thousands of timeouts were occurring on 
a ia64 client compared to 50 to 60 timeouts with 
a x86 client when talking to the same slow server and 
generating the same traffic. 

What this patch does is make the minimal Round Trip 
time value relative to HZ. So When HZ is greater (as 
in the case of ia64) the minimal value goes up. 
  
Comments? 

SteveD. 


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 4482 bytes --]

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

It has to do with the value of HZ.... On a ia64, HZ is
at 1024 and on an x86 machine its 100. Not taking this
difference in account when figure the the minimal
timeout values was causing timeouts to occur every 4ms
instead of 40ms.

The network trace didn't show anything substantial,
but here is the debugging trail.

By logging (and counting) the number of times call_status() was called
with a -ETIMEDOUT status, it became very apparent that ia64 machine
were timing out thousands of times more often than an x86 machine.
The actual numbers was something like 1400 to 50 when I generated
traffic by doing  md5sum /nfs/mounted/*.rpm > /dev/null.

Next I took a look at what task->tk_timeout was being set to in
do_xprt_transmit(). On an x86 it was being set to ~40ms. On
an ia64 machine it was being set to ~4ms.

That lead me to how rpc_calc_rto() was figuring out
the RTOs... I noticed that RPC_RTO_MIN was the only
constant that was not relative to HZ. So I did some
experiments and found out by making it relative to HZ
the timeout decreased substantially...

SteveD.

Lever, Charles wrote:

>steve-
> 
>can you explain why there are more timeouts for ia64?  do you
>have a network trace you can share?
>
>-----Original Message----- 
>From: Steve Dickson [mailto:SteveD@RedHat.com] 
>Sent: Fri 5/9/2003 8:41 AM 
>To: nfs@lists.sourceforge.net 
>Cc: 
>Subject: [NFS] [PATCH] Timeouts gone wild on ia64
>
>
>
>Here is a patch that greatly reduces that number of 
>timeout (and EIO errors with soft mounts) that 
>occur when a fast client is talking to a slow server. 
>  
>We were noticing a large number of EIO errors when 
>a ia64 client was talking to a x86 server with 
>soft mounts (ala autofs).... 
>
>True, EIO errors should be expect with soft mounts but 
>it turns out that thousands of timeouts were occurring on 
>a ia64 client compared to 50 to 60 timeouts with 
>a x86 client when talking to the same slow server and 
>generating the same traffic. 
>
>What this patch does is make the minimal Round Trip 
>time value relative to HZ. So When HZ is greater (as 
>in the case of ia64) the minimal value goes up. 
>  
>Comments? 
>
>SteveD. 
>
>  
>




-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [PATCH] Timeouts gone wild on ia64

[Lever, Charles]

b2ssIGkgd2FzIGNvbmZ1c2VkLiAgeW91ciBpbml0aWFsIGV4cGxhbmF0aW9uDQpzdWdnZXN0ZWQg
eW91IHdlcmUgY2hhbmdpbmcgdGhlIGJlaGF2aW9yIG9mDQp0aGUgUlRPIGVzdGltYXRvciwgd2hl
cmVhcyBpbiBmYWN0LCB5b3UgYXJlDQpjb3JyZWN0aW5nIGl0LiAgb24gaWE2NCwgdGhlIG1pbmlt
dW0gdGltZW91dA0KdmFsdWUgaXMgaW5jb3JyZWN0IGJlY2F1c2UgSFogaXMgYW4gb3JkZXIgb2YN
Cm1hZ25pdHVkZSBsYXJnZXIuDQoNCmhvd2V2ZXIsIGVxdWl2YWxlbnQgYmVoYXZpb3Igd291bGQg
YmU6DQoNCiAgI2RlZmluZSBSUENfUlRPX01JTiAoSFovNTApDQoNCm5vdCBIWi8zMC4gIHdoeSBk
aWQgeW91IGNob29zZSAzMD8NCg0KPiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9t
OiBTdGV2ZSBEaWNrc29uIFttYWlsdG86U3RldmVEQHJlZGhhdC5jb21dDQo+IFNlbnQ6IEZyaWRh
eSwgTWF5IDA5LCAyMDAzIDEwOjEyIEFNDQo+IFRvOiBMZXZlciwgQ2hhcmxlcw0KPiBDYzogbmZz
QGxpc3RzLnNvdXJjZWZvcmdlLm5ldA0KPiBTdWJqZWN0OiBSZTogW05GU10gW1BBVENIXSBUaW1l
b3V0cyBnb25lIHdpbGQgb24gaWE2NA0KPiANCj4gDQo+IEl0IGhhcyB0byBkbyB3aXRoIHRoZSB2
YWx1ZSBvZiBIWi4uLi4gT24gYSBpYTY0LCBIWiBpcw0KPiBhdCAxMDI0IGFuZCBvbiBhbiB4ODYg
bWFjaGluZSBpdHMgMTAwLiBOb3QgdGFraW5nIHRoaXMNCj4gZGlmZmVyZW5jZSBpbiBhY2NvdW50
IHdoZW4gZmlndXJlIHRoZSB0aGUgbWluaW1hbA0KPiB0aW1lb3V0IHZhbHVlcyB3YXMgY2F1c2lu
ZyB0aW1lb3V0cyB0byBvY2N1ciBldmVyeSA0bXMNCj4gaW5zdGVhZCBvZiA0MG1zLg0KPiANCj4g
VGhlIG5ldHdvcmsgdHJhY2UgZGlkbid0IHNob3cgYW55dGhpbmcgc3Vic3RhbnRpYWwsDQo+IGJ1
dCBoZXJlIGlzIHRoZSBkZWJ1Z2dpbmcgdHJhaWwuDQo+IA0KPiBCeSBsb2dnaW5nIChhbmQgY291
bnRpbmcpIHRoZSBudW1iZXIgb2YgdGltZXMgY2FsbF9zdGF0dXMoKSB3YXMgY2FsbGVkDQo+IHdp
dGggYSAtRVRJTUVET1VUIHN0YXR1cywgaXQgYmVjYW1lIHZlcnkgYXBwYXJlbnQgdGhhdCBpYTY0
IG1hY2hpbmUNCj4gd2VyZSB0aW1pbmcgb3V0IHRob3VzYW5kcyBvZiB0aW1lcyBtb3JlIG9mdGVu
IHRoYW4gYW4geDg2IG1hY2hpbmUuDQo+IFRoZSBhY3R1YWwgbnVtYmVycyB3YXMgc29tZXRoaW5n
IGxpa2UgMTQwMCB0byA1MCB3aGVuIEkgZ2VuZXJhdGVkDQo+IHRyYWZmaWMgYnkgZG9pbmcgIG1k
NXN1bSAvbmZzL21vdW50ZWQvKi5ycG0gPiAvZGV2L251bGwuDQo+IA0KPiBOZXh0IEkgdG9vayBh
IGxvb2sgYXQgd2hhdCB0YXNrLT50a190aW1lb3V0IHdhcyBiZWluZyBzZXQgdG8gaW4NCj4gZG9f
eHBydF90cmFuc21pdCgpLiBPbiBhbiB4ODYgaXQgd2FzIGJlaW5nIHNldCB0byB+NDBtcy4gT24N
Cj4gYW4gaWE2NCBtYWNoaW5lIGl0IHdhcyBiZWluZyBzZXQgdG8gfjRtcy4NCj4gDQo+IFRoYXQg
bGVhZCBtZSB0byBob3cgcnBjX2NhbGNfcnRvKCkgd2FzIGZpZ3VyaW5nIG91dA0KPiB0aGUgUlRP
cy4uLiBJIG5vdGljZWQgdGhhdCBSUENfUlRPX01JTiB3YXMgdGhlIG9ubHkNCj4gY29uc3RhbnQg
dGhhdCB3YXMgbm90IHJlbGF0aXZlIHRvIEhaLiBTbyBJIGRpZCBzb21lDQo+IGV4cGVyaW1lbnRz
IGFuZCBmb3VuZCBvdXQgYnkgbWFraW5nIGl0IHJlbGF0aXZlIHRvIEhaDQo+IHRoZSB0aW1lb3V0
IGRlY3JlYXNlZCBzdWJzdGFudGlhbGx5Li4uDQo+IA0KPiBTdGV2ZUQuDQo+IA0KPiBMZXZlciwg
Q2hhcmxlcyB3cm90ZToNCj4gDQo+ID5zdGV2ZS0NCj4gPiANCj4gPmNhbiB5b3UgZXhwbGFpbiB3
aHkgdGhlcmUgYXJlIG1vcmUgdGltZW91dHMgZm9yIGlhNjQ/ICBkbyB5b3UNCj4gPmhhdmUgYSBu
ZXR3b3JrIHRyYWNlIHlvdSBjYW4gc2hhcmU/DQo+ID4NCj4gPi0tLS0tT3JpZ2luYWwgTWVzc2Fn
ZS0tLS0tIA0KPiA+RnJvbTogU3RldmUgRGlja3NvbiBbbWFpbHRvOlN0ZXZlREBSZWRIYXQuY29t
XSANCj4gPlNlbnQ6IEZyaSA1LzkvMjAwMyA4OjQxIEFNIA0KPiA+VG86IG5mc0BsaXN0cy5zb3Vy
Y2Vmb3JnZS5uZXQgDQo+ID5DYzogDQo+ID5TdWJqZWN0OiBbTkZTXSBbUEFUQ0hdIFRpbWVvdXRz
IGdvbmUgd2lsZCBvbiBpYTY0DQo+ID4NCj4gPg0KPiA+DQo+ID5IZXJlIGlzIGEgcGF0Y2ggdGhh
dCBncmVhdGx5IHJlZHVjZXMgdGhhdCBudW1iZXIgb2YgDQo+ID50aW1lb3V0IChhbmQgRUlPIGVy
cm9ycyB3aXRoIHNvZnQgbW91bnRzKSB0aGF0IA0KPiA+b2NjdXIgd2hlbiBhIGZhc3QgY2xpZW50
IGlzIHRhbGtpbmcgdG8gYSBzbG93IHNlcnZlci4gDQo+ID4gIA0KPiA+V2Ugd2VyZSBub3RpY2lu
ZyBhIGxhcmdlIG51bWJlciBvZiBFSU8gZXJyb3JzIHdoZW4gDQo+ID5hIGlhNjQgY2xpZW50IHdh
cyB0YWxraW5nIHRvIGEgeDg2IHNlcnZlciB3aXRoIA0KPiA+c29mdCBtb3VudHMgKGFsYSBhdXRv
ZnMpLi4uLiANCj4gPg0KPiA+VHJ1ZSwgRUlPIGVycm9ycyBzaG91bGQgYmUgZXhwZWN0IHdpdGgg
c29mdCBtb3VudHMgYnV0IA0KPiA+aXQgdHVybnMgb3V0IHRoYXQgdGhvdXNhbmRzIG9mIHRpbWVv
dXRzIHdlcmUgb2NjdXJyaW5nIG9uIA0KPiA+YSBpYTY0IGNsaWVudCBjb21wYXJlZCB0byA1MCB0
byA2MCB0aW1lb3V0cyB3aXRoIA0KPiA+YSB4ODYgY2xpZW50IHdoZW4gdGFsa2luZyB0byB0aGUg
c2FtZSBzbG93IHNlcnZlciBhbmQgDQo+ID5nZW5lcmF0aW5nIHRoZSBzYW1lIHRyYWZmaWMuIA0K
PiA+DQo+ID5XaGF0IHRoaXMgcGF0Y2ggZG9lcyBpcyBtYWtlIHRoZSBtaW5pbWFsIFJvdW5kIFRy
aXAgDQo+ID50aW1lIHZhbHVlIHJlbGF0aXZlIHRvIEhaLiBTbyBXaGVuIEhaIGlzIGdyZWF0ZXIg
KGFzIA0KPiA+aW4gdGhlIGNhc2Ugb2YgaWE2NCkgdGhlIG1pbmltYWwgdmFsdWUgZ29lcyB1cC4g
DQo+ID4gIA0KPiA+Q29tbWVudHM/IA0KPiA+DQo+ID5TdGV2ZUQuIA0KPiA+DQo+ID4gIA0KPiA+
DQo+IA0KPiANCj4gDQo=


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

Lever, Charles wrote:

>however, equivalent behavior would be:
>
>  #define RPC_RTO_MIN (HZ/50)
>
>not HZ/30.  why did you choose 30?
>
I realize this but... I figured waiting a
a few extra ticks of time before timing
out was probably a good thing with respect
to soft mounts on slow servers... Especially
since EIO errors can be pretty disruptive... ;-)\

SteveD.



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [PATCH] Timeouts gone wild on ia64

[Lever, Charles]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

steve-
 
i think there must be an underlying problem here.  soft mounts
to slow servers should work correctly without raising the RTO
minimum.  
 
the RTO estimator should automatically raise the RTO values to
avoid extra timeouts.  if not, there is a problem with the RTO
estimator that needs to be addressed.
 

-----Original Message----- 
From: Steve Dickson [mailto:SteveD@RedHat.com] 
Sent: Wed 5/14/2003 8:34 PM 
To: Trond Myklebust; nfs@lists.sourceforge.net 
Cc: 
Subject: Re: [NFS] [PATCH] Timeouts gone wild on ia64



Hi Trond, 

Trond Myklebust wrote: 

>>>>>>" " == Steve Dickson <SteveD@RedHat.com> writes: 
>>>>>>            
>>>>>> 
> 
>     > What this patch does is make the minimal Round Trip time value 
>     > relative to HZ. So When HZ is greater (as in the case of ia64) 
>     > the minimal value goes up. 
>     >  Comments? 
> 
>1/30 of a second is a long time. Why that particular choice? 
> 
>Note: "It works" won't cut ice as that is completely a function of the 
>choice of hardware. 
> 
I realize this but... I figured waiting a few extra ticks of time before 
timing 
out was probably a good thing with respect to soft mounts to slow 
servers... 
Especially  since EIO errors can be pretty disruptive... 

And as I see, its a no-op with a fast (or slow) client  talking to a fast 
server because the timeout will should never happen... (i.e. the waiting 
thread will get the response before the time out occurs).... 

SteveD. 





------------------------------------------------------- 
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara 
The only event dedicated to issues related to Linux enterprise solutions 
www.enterpriselinuxforum.com 

_______________________________________________ 
NFS maillist  -  NFS@lists.sourceforge.net 
https://lists.sourceforge.net/lists/listinfo/nfs 


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 6178 bytes --]

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

It appears the RTO code does seem to be working but when the
minimums start so low (like at 4m instead 40ms) it takes some
time for the timeout value to build up and with soft mounts
there is no time...

Maybe I'm missing something... increasing the timeout value should
not have any affect on performance since in a well tuned
client and server these timeout will never occur since the
responses from the server will be returned before the
timeout expires... right? Also decreasing the number of timeouts
will decrease the number of retransmits which is another good
thing... True?

SteveD.

Lever, Charles wrote:

>steve-
> 
>i think there must be an underlying problem here.  soft mounts
>to slow servers should work correctly without raising the RTO
>minimum.  
> 
>the RTO estimator should automatically raise the RTO values to
>avoid extra timeouts.  if not, there is a problem with the RTO
>estimator that needs to be addressed.
> 
>
>-----Original Message----- 
>From: Steve Dickson [mailto:SteveD@RedHat.com] 
>Sent: Wed 5/14/2003 8:34 PM 
>To: Trond Myklebust; nfs@lists.sourceforge.net 
>Cc: 
>Subject: Re: [NFS] [PATCH] Timeouts gone wild on ia64
>
>
>
>Hi Trond, 
>
>Trond Myklebust wrote: 
>
>  
>
>>>>>>>" " == Steve Dickson <SteveD@RedHat.com> writes: 
>>>>>>>           
>>>>>>>
>>>>>>>              
>>>>>>>
>>    > What this patch does is make the minimal Round Trip time value 
>>    > relative to HZ. So When HZ is greater (as in the case of ia64) 
>>    > the minimal value goes up. 
>>    >  Comments? 
>>
>>1/30 of a second is a long time. Why that particular choice? 
>>
>>Note: "It works" won't cut ice as that is completely a function of the 
>>choice of hardware. 
>>
>>    
>>
>I realize this but... I figured waiting a few extra ticks of time before 
>timing 
>out was probably a good thing with respect to soft mounts to slow 
>servers... 
>Especially  since EIO errors can be pretty disruptive... 
>
>And as I see, its a no-op with a fast (or slow) client  talking to a fast 
>server because the timeout will should never happen... (i.e. the waiting 
>thread will get the response before the time out occurs).... 
>
>SteveD. 
>
>
>
>
>
>------------------------------------------------------- 
>Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara 
>The only event dedicated to issues related to Linux enterprise solutions 
>www.enterpriselinuxforum.com 
>
>_______________________________________________ 
>NFS maillist  -  NFS@lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/nfs 
>
>  
>



-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Trond Myklebust]

>>>>> " " == Steve Dickson <SteveD@redhat.com> writes:

     > It appears the RTO code does seem to be working but when the
     > minimums start so low (like at 4m instead 40ms) it takes some
     > time for the timeout value to build up and with soft mounts
     > there is no time...

     > Maybe I'm missing something... increasing the timeout value
     > should not have any affect on performance since in a well tuned
     > client and server these timeout will never occur since the
     > responses from the server will be returned before the timeout
     > expires... right? Also decreasing the number of timeouts will
     > decrease the number of retransmits which is another good
     > thing... True?

With a properly implemented algorithm, the number of retransmits
should be small anyway as it is supposed to take into account the
variance on the estimated RTO. We don't want any extra artificial
limits if we can avoid it.

You may well be right in asserting that we're setting the initial RTO
estimate too low, but then the answer should be to increase the value
of the 'timeo' mount parameter as that is what defines the initial
estimate.
The default value of 'retrans' should also be looked at. I'm not at
all comfortable with a default retrans value of '3' when doing soft
mounts.

At the moment I believe that the default values for these 2 parameters
differ in the kernel from those in the 'mount' program. IMHO, the
mount program is overriding the kernel with too low values. It would
be better if 'mount' did not set timeo/retrans (unless the user
overrides) and left that to the kernel.

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

Trond Myklebust wrote:

>With a properly implemented algorithm, the number of retransmits
>should be small anyway as it is supposed to take into account the
>variance on the estimated RTO. We don't want any extra artificial
>limits if we can avoid it.
>
I totally greed... But this change, IMHO, will give a better estimated RTO
since all of the constants are based on the machine's HZ... At
least that's how I saw it...

>You may well be right in asserting that we're setting the initial RTO
>estimate too low, but then the answer should be to increase the value
>of the 'timeo' mount parameter as that is what defines the initial
>estimate.
>The default value of 'retrans' should also be looked at. I'm not at
>all comfortable with a default retrans value of '3' when doing soft
>mounts.
>
This was the direction I was headed until I saw the minimal RTO was
not HZ based... When I changed that which seem to take care of the
problem... I just stopped there... :(

>At the moment I believe that the default values for these 2 parameters
>differ in the kernel from those in the 'mount' program. IMHO, the
>mount program is overriding the kernel with too low values. It would
>be better if 'mount' did not set timeo/retrans (unless the user
>overrides) and left that to the kernel.
>
This definitely seems wrong....

SteveD.




-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Yusuf Goolamabbas]

> >>>>> " " == Steve Dickson <SteveD@redhat.com> writes:
> 
>      > It appears the RTO code does seem to be working but when the
>      > minimums start so low (like at 4m instead 40ms) it takes some
>      > time for the timeout value to build up and with soft mounts
>      > there is no time...
> 
>      > Maybe I'm missing something... increasing the timeout value
>      > should not have any affect on performance since in a well tuned
>      > client and server these timeout will never occur since the
>      > responses from the server will be returned before the timeout
>      > expires... right? Also decreasing the number of timeouts will
>      > decrease the number of retransmits which is another good
>      > thing... True?
> 
> With a properly implemented algorithm, the number of retransmits
> should be small anyway as it is supposed to take into account the
> variance on the estimated RTO. We don't want any extra artificial
> limits if we can avoid it.
> 
> You may well be right in asserting that we're setting the initial RTO
> estimate too low, but then the answer should be to increase the value
> of the 'timeo' mount parameter as that is what defines the initial
> estimate.
> The default value of 'retrans' should also be looked at. I'm not at
> all comfortable with a default retrans value of '3' when doing soft
> mounts.
> 
> At the moment I believe that the default values for these 2 parameters
> differ in the kernel from those in the 'mount' program. IMHO, the
> mount program is overriding the kernel with too low values. It would
> be better if 'mount' did not set timeo/retrans (unless the user
> overrides) and left that to the kernel.

Trond, In your opinion. What should be the default retrans value given
your recent work in having adaptive timeouts in the NFS client ?

Regards, Yusuf



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Trond Myklebust]

>>>>> " " == Yusuf Goolamabbas <yusufg@outblaze.com> writes:

     > Trond, In your opinion. What should be the default retrans
     > value given your recent work in having adaptive timeouts in the
     > NFS client ?

I'd like to cap the minimum value for the estimated error on the RTO
at 1/10sec as per the 2.6 kernel (currently it is set at 2/HZ sec ~
2/100sec on most architectures, but 2/1000sec on ia64).

Then I'd like to increase the default value that "mount" uses for
"retrans" to the kernel value (which is currently set at 5).

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Yusuf Goolamabbas]

>      > Trond, In your opinion. What should be the default retrans
>      > value given your recent work in having adaptive timeouts in the
>      > NFS client ?
> 
> I'd like to cap the minimum value for the estimated error on the RTO
> at 1/10sec as per the 2.6 kernel (currently it is set at 2/HZ sec ~
> 2/100sec on most architectures, but 2/1000sec on ia64).
> 
> Then I'd like to increase the default value that "mount" uses for
> "retrans" to the kernel value (which is currently set at 5).

What value are you looking at ?
Also man 5 nfs seems (in RH 9) that default retrans value is 3




-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Trond Myklebust]

>>>>> " " == Yusuf Goolamabbas <yusufg@outblaze.com> writes:

    >> Then I'd like to increase the default value that "mount" uses
    >> for "retrans" to the kernel value (which is currently set at
    >> 5).

     > What value are you looking at ?

The point is that we need statistics from those people who are seeing
problems. On my own setups, "5" is quite sufficient to eliminate all
the false positives. With the cap of timeo at 1/10sec, that
corresponds to a minimum "soft" timeout value after 6.3 seconds.

I've already asked once on these lists whether or not that suffices
for other people too, and have received no reply.

Without adequate statistics, I see no reason to change the existing
default...

     > Also man 5 nfs seems (in RH 9) that default retrans value is 3

The manpage needs to be changed too but not before 'mount' is.

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [PATCH] Timeouts gone wild on ia64

[Lever, Charles]

[-- Attachment #1: Type: text/plain, Size: 3603 bytes --]

you want to keep the retransmit timeout as short as possible,
just before things start timing out.  this means you get the fastest
possible recovery when the server drops a request.  a 4ms
timeout is probably reasonable for a fast server on a fast
network for small requests like GETATTR and LOOKUP.
 
but what i'm hearing is the starting RTO is probably not
optimal for slow servers.  right now the initial value is:
 
  #define RPC_RTO_INIT (HZ/5)
 
(200ms) which is perhaps too small.  a better value for
general use might be HZ/2 (half a second).  then the
estimator can adjust downward for faster servers while
behaving practically for slow ones.

-----Original Message----- 
From: Steve Dickson [mailto:SteveD@redhat.com] 
Sent: Thu 5/15/2003 10:10 AM 
To: Lever, Charles 
Cc: Trond Myklebust; nfs@lists.sourceforge.net 
Subject: Re: [NFS] [PATCH] Timeouts gone wild on ia64




It appears the RTO code does seem to be working but when the 
minimums start so low (like at 4m instead 40ms) it takes some 
time for the timeout value to build up and with soft mounts 
there is no time... 

Maybe I'm missing something... increasing the timeout value should 
not have any affect on performance since in a well tuned 
client and server these timeout will never occur since the 
responses from the server will be returned before the 
timeout expires... right? Also decreasing the number of timeouts 
will decrease the number of retransmits which is another good 
thing... True? 

SteveD. 

Lever, Charles wrote: 

>steve- 
> 
>i think there must be an underlying problem here.  soft mounts 
>to slow servers should work correctly without raising the RTO 
>minimum.  
> 
>the RTO estimator should automatically raise the RTO values to 
>avoid extra timeouts.  if not, there is a problem with the RTO 
>estimator that needs to be addressed. 
> 
> 
>-----Original Message----- 
>From: Steve Dickson [mailto:SteveD@RedHat.com] 
>Sent: Wed 5/14/2003 8:34 PM 
>To: Trond Myklebust; nfs@lists.sourceforge.net 
>Cc: 
>Subject: Re: [NFS] [PATCH] Timeouts gone wild on ia64 
> 
> 
> 
>Hi Trond, 
> 
>Trond Myklebust wrote: 
> 
>  
> 
>>>>>>>" " == Steve Dickson <SteveD@RedHat.com> writes: 
>>>>>>>           
>>>>>>> 
>>>>>>>              
>>>>>>> 
>>    > What this patch does is make the minimal Round Trip time value 
>>    > relative to HZ. So When HZ is greater (as in the case of ia64) 
>>    > the minimal value goes up. 
>>    >  Comments? 
>> 
>>1/30 of a second is a long time. Why that particular choice? 
>> 
>>Note: "It works" won't cut ice as that is completely a function of the 
>>choice of hardware. 
>> 
>>    
>> 
>I realize this but... I figured waiting a few extra ticks of time before 
>timing 
>out was probably a good thing with respect to soft mounts to slow 
>servers... 
>Especially  since EIO errors can be pretty disruptive... 
> 
>And as I see, its a no-op with a fast (or slow) client  talking to a fast 
>server because the timeout will should never happen... (i.e. the waiting 
>thread will get the response before the time out occurs).... 
> 
>SteveD. 
> 
> 
> 
> 
> 
>------------------------------------------------------- 
>Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara 
>The only event dedicated to issues related to Linux enterprise solutions 
>www.enterpriselinuxforum.com 
> 
>_______________________________________________ 
>NFS maillist  -  NFS@lists.sourceforge.net 
>https://lists.sourceforge.net/lists/listinfo/nfs 
> 
>  
> 


[-- Attachment #2: winmail.dat --]
[-- Type: application/ms-tnef, Size: 9202 bytes --]

RE: [PATCH] Timeouts gone wild on ia64

[Trond Myklebust]

Nope. Look again

The actual timer value is initialized from xprt->timeout.to_initval

RPC_RTO_INIT is just used to provide an initial estimate of the variance.

Cheers,
  Trond


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [PATCH] Timeouts gone wild on ia64

[Steve Dickson]

Lever, Charles wrote:

>you want to keep the retransmit timeout as short as possible,
>just before things start timing out.  this means you get the fastest
>possible recovery when the server drops a request.  
>
That's assuming server drops the request... now if the server is
simply buzy because its severing hundreds of clients and it
takes 6ms to respond, you now have hundreds of clients retransmitting
very 4ms (for basically for no reason) which is just adding to the 
problem...
I'm sure the RTO code would eventually increase the timeout which
would smooth everything out but before that happens you would be
blasting the network with a ton of unnecessary retransmits... True?

>but what i'm hearing is the starting RTO is probably not
>optimal for slow servers.  right now the initial value is:
> 
>  #define RPC_RTO_INIT (HZ/5)
> 
>(200ms) which is perhaps too small.  a better value for
>general use might be HZ/2 (half a second).  then the
>estimator can adjust downward for faster servers while
>behaving practically for slow ones.
>
By increasing the initial timeout, ISTM, that the client
is assuming a slower server verses a fast one... which will
probably work as well... Its just that I thought making
all of the RTO constants value relative to HZ was a good idea...

SteveD.




-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [PATCH] Timeouts gone wild on ia64

[Lever, Charles]

> >you want to keep the retransmit timeout as short as possible,
> >just before things start timing out.  this means you get the fastest
> >possible recovery when the server drops a request. =20
> >
> That's assuming server drops the request... now if the server is
> simply buzy because its severing hundreds of clients and it
> takes 6ms to respond, you now have hundreds of clients retransmitting
> very 4ms (for basically for no reason) which is just adding to the=20
> problem...
> I'm sure the RTO code would eventually increase the timeout which
> would smooth everything out but before that happens you would be
> blasting the network with a ton of unnecessary retransmits... True?

we're agreeing vehemently.  the RTO estimator should
*start* at a larger timeout value to prevent this.

> >but what i'm hearing is the starting RTO is probably not
> >optimal for slow servers.  right now the initial value is:
> >=20
> >  #define RPC_RTO_INIT (HZ/5)
> >=20
> >(200ms) which is perhaps too small.  a better value for
> >general use might be HZ/2 (half a second).  then the
> >estimator can adjust downward for faster servers while
> >behaving practically for slow ones.

i agree with trond that fixing mount is a good idea...
however, the mount command's initial RTO value is up
in the hundreds of msec.  so why does the estimator
allow the RTO values to drop for slow servers?

the default retransmit count is too low for UDP.  but
i think we all agree on that.

> By increasing the initial timeout, ISTM, that the client
> is assuming a slower server verses a fast one... which will
> probably work as well... Its just that I thought making
> all of the RTO constants value relative to HZ was a good idea...

yes, making the RTO constants relative to HZ is a good
idea.  i think the objection is to raising the minimum
RTO at the same time.


-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs