Re: can't get request slot, write timeout

Re: can't get request slot, write timeout

[Bruce Janson]

    ...
    From: Bogdan Costescu <bogdan.costescu@iwr.uni-heidelberg.de>
    To: Kenneth Howlett <aw464@osfn.org>
    cc: nfs@lists.sourceforge.net
    ...
    Date: Mon, 12 Aug 2002 14:15:37 +0200 (CEST)
    ...
    On Sun, 11 Aug 2002, Kenneth Howlett wrote:
    ...
    > I do not think this problem is caused by network congestion or an
    > overloaded server because there is no other network activity.
    
    No other network activity doesn't mean that there is no congestion. 
    Congestion can also be created with 2 computers when one sends faster than 
    the other can receive. Check /proc/net/dev file on both computers for 
    errors.
    
    > I have searched through the list archives and found many similar
    > problems. ... Each of these problem reports is slightly different, but I
    > think most are the same problem. But in the list archives, it appears
    > that the developers do not recognize this as the same problem, because
    > everyone reports it differently.
    
    I posted some messages in the past about this. As the error message says,
    the communication between the client and the server is broken - that is
    the problem and it's the same in all cases; however, there can be 1001 
    causes for it and each one may have a different solution.
    
    > If I do 'ping -f -s nnnn' with various numbers for nnnn; the
    > higher nnnn is, the more packets are lost.
    
    That points clearly toward a network problem. Any UDP based service will 
    have problems on a network that looses packets. That's why it's called UDP 
    = Unreliable.
    ...

No, User.
    
    > I have fixed my problem by using mount options of rsize=2048,wsize=2048.
    > rsize=1024,wsize=1024 also works, but is a little slower.
    
    That confirms the network problem.
    ...

Eh?
    
    > Most of the people who have reported similar problems also
    > reported using 2.4.x clients, and some reported that the problem
    > did not occur before they upgraded to 2.4.x. This suggests that
    > there might be a bug in 2.4.x clients.
    ...

The surprising thing about this error condition (which has been reported
on this list for a number of years now) is that under such conditions
the Linux NFS client code fails so spectacularly.  Rather than performance
degrading gracefully as one might expect from a congested network (or a
flaky NIC or a marginal cable or a slow receiver or a client kernel RAM
shortage or ...) the kernel instead emits one or more

  can't get a request slot

messages and the affected transactions freeze for extended periods (hours!).


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: can't get request slot, write timeout

[Bogdan Costescu]

On Tue, 13 Aug 2002, Bruce Janson wrote:

>     have problems on a network that looses packets. That's why it's
>     called UDP = Unreliable.
>     ...
> No, User.

Sorry, there were supposed to be some quotes around Unreliable. I was just 
playing with LOCALE settings when I wrote the message... But 'man udp' 
says that in the first few lines on my machine.

>     > I have fixed my problem by using mount options of rsize=2048,wsize=2048.
>     > rsize=1024,wsize=1024 also works, but is a little slower.
>     That confirms the network problem.
> Eh?

With rsize/wsize=1024, the whole NFS data chunk fits into one Ethernet 
packet, with rsize/wsize=2048 it fits into 2 packets. Network problems 
generally manifest themselves (to the upper levels) as packet losses, 
which means a small rsize/wsize has much better chances to arrive 
completely at the other end. Also in case of packets missing, it's much 
easier to retransmit a small data chunk than a large one.

> The surprising thing about this error condition (which has been reported
> on this list for a number of years now) is that under such conditions
> the Linux NFS client code fails so spectacularly. 

The problem is known and work has be done to improve the situation: NFS 
client over TCP exists for some time and Trond worked on improving the 
behaviour of NFS client over UDP.

> Rather than performance degrading gracefully as one might expect from a
> congested network (or a flaky NIC or a marginal cable or a slow receiver
> or a client kernel RAM shortage or ...) ...

I beg to disagree. If you want performance degrading (more) gracefully
switch to NFS over TCP. And "one might expect" that only for slow
receiver, all other causes are much likely to produce a sudden
interruption. 
F.e. in a free memory shortage lots of consecutive packets can be dropped.  
A congested network, flaky NIC and marginal cable could lead f.e. to the
dreaded "transmit timed out" error (due to colisions, media problems,
etc.) which often does not "repair itself", as it takes some time until
the reset comes; the Tx ring is full, meaning that even if the reset
clears the error state, there is a chance that the flurry of packets to be
sent will trigger it again; in such a case (at either client or server
side), the packets are much delayed or even lost. That's why a healthy
network is a must for good UDP-based services.

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: can't get request slot, write timeout

[Trond Myklebust]

>>>>> " " == Bruce Janson <bruce@it.usyd.edu.au> writes:

     > The surprising thing about this error condition (which has been
     > reported on this list for a number of years now) is that under
     > such conditions the Linux NFS client code fails so
     > spectacularly.  Rather than performance degrading gracefully as
     > one might expect from a congested network (or a flaky NIC or a
     > marginal cable or a slow receiver or a client kernel RAM
     > shortage or ...) the kernel instead emits one or more

     >   can't get a request slot

     > messages and the affected transactions freeze for extended
     > periods (hours!).

If you know of a cure-all solution, feel free to send me patches...

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

can't get request slot, write timeout

[Kenneth Howlett]

I have two computers, namelessp2, a PII which I usually use; and
junk486, an old 486. Namelessp2 has a davicom dm9102f ethernet
chip built into the motherboard. Junk486 has a 3com 3c503
ethernet card. I have a surecom 505st hub. Both computers run
redhat 7.2 with a 2.4.18 kernel. This is a low cost 10mbps
network made with cheap components because I do not need a high
performance network.

I want to backup namelessp2 to the hard drive of junk486 using
nfs. However, when I write data to the nfs drive, the program
which is writing data appears to hang, and I get many messages
like

   kernel: nfs: server server.domain.name not responding, still
trying
   kernel: nfs: task 10754 can't get a request slot
   kernel: nfs: server server.domain.name OK

If I wait many hours, the program which is trying to write data
to the nfs drive will eventually succeed, so it is not really
hung, but it is so slow that it appears to be hung.

I do not think this problem is caused by network congestion or an
overloaded server because there is no other network activity. I
do not think this problem is caused by buggy network cards or
drivers because I changed network cards and drivers, and the
problem was unchanged.

I have searched through the list archives and found many similar
problems. Some people have reported a hung server because the
client reports that the server is not responding. Other people
have reported a client problem because the problem only occurs
with certain clients. Some people say it is hung and other people
say performance is low. Some people reported the request slot
error message and some did not.

Each of these problem reports is slightly different, but I think
most are the same problem. But in the list archives, it appears
that the developers do not recognize this as the same problem,
because everyone reports it differently.

In my search of the archive, all the suggested solutions I found
were too vague to be useful, and I did not find any messages
which confirmed that anyone had ever solved this problem.

The problem only occurred on writes. I have no problem with
reads. The problem may occur when either computer writes to the
other, but is more likely to occur when namelessp2 writes to
junk486. The problem is more likely to occur when writing a large
file than when writing a small file. I tried changing the block
size used by the program which writes the file; the problem
occurred with any block size but was more likely to occur with a
large block size. The problem did not occur if I used a small
block size and opened and closed the file for each block. Opening
the file with O_SYNC had no effect on the problem. Exporting with
sync or async had no effect on the problem. Mounting with
rsize=8192,wsize=8192 had no effect on the problem. The problem
does not occur when namelessp2 nfs mounts itself.

If I do 'ping -f -s nnnn' with various numbers for nnnn; the
higher nnnn is, the more packets are lost. If nnnn is a small
number, no packets are lost. If nnnn is a large number, all
packets are lost. More packets are lost when namelessp2 pings
junk486 than when junk486 pings namelessp2. If namelessp2 pings
junk486, 1% packet loss occurs at a packet size of 1000 to 4000.
If junk486 pings namelessp2, 1% packet loss occurs at a packet
size of 5000 to 9000.

I have fixed my problem by using mount options of
rsize=2048,wsize=2048. rsize=1024,wsize=1024 also works, but is a
little slower.

Most of the people who have reported similar problems also
reported using 2.4.x clients, and some reported that the problem
did not occur before they upgraded to 2.4.x. This suggests that
there might be a bug in 2.4.x clients. However, I think the
problem is not the 2.4.x  clients, but the utils used with 2.4.x
clients; the newer utils have a higher default rsize/wsize than
the old utils. Apparently the new default rsize/wsize does not
work with some hardware.

I have already solved my problem, so I do not need any help. I am
posting this message for the benefit of anyone who is searching
the archives for solutions to this problem.

Next time someone asks what to do about request slot error
messages, or a hung server, or very slow performance; that person
should be told to try a smaller rsize/wsize. This should be added
to the FAQ and to the HOWTO and to the nfs manpage. Since so many
people have had this problem, this should be added to the
documentation.

In the FAQ it says that tcp is faster than udp on congested
networks.  I think the same effect could be achieved by using udp
with a small rsize/wsize.

When I was researching this, I spent a while looking for the FAQ
before I realized the FAQ was right in front of me at
http://nfs.sourceforge.net. I think that page should say 'this is
the FAQ'.

When I was searching the archive, it would have been easier if I
could have downloaded one large digest of all messages from the
last year or two.
\x1a


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: can't get request slot, write timeout

[Bogdan Costescu]

On Sun, 11 Aug 2002, Kenneth Howlett wrote:

> Namelessp2 has a davicom dm9102f ethernet chip built into the
> motherboard. Junk486 has a 3com 3c503 ethernet card. I have a surecom
> 505st hub.  ... This is a low cost 10mbps network made with cheap
> components because I do not need a high performance network.

Then why are you asking NFS over UDP to work well on it ? Especially a 
computer called "junk" :-)

> I want to backup namelessp2 to the hard drive of junk486 using
> nfs.

Maybe you should use something TCP based: either NFS over TCP or something 
else like tar and netcat, rsync, etc.

> I do not think this problem is caused by network congestion or an
> overloaded server because there is no other network activity.

No other network activity doesn't mean that there is no congestion. 
Congestion can also be created with 2 computers when one sends faster than 
the other can receive. Check /proc/net/dev file on both computers for 
errors.

> I have searched through the list archives and found many similar
> problems. ... Each of these problem reports is slightly different, but I
> think most are the same problem. But in the list archives, it appears
> that the developers do not recognize this as the same problem, because
> everyone reports it differently.

I posted some messages in the past about this. As the error message says,
the communication between the client and the server is broken - that is
the problem and it's the same in all cases; however, there can be 1001 
causes for it and each one may have a different solution.

> If I do 'ping -f -s nnnn' with various numbers for nnnn; the
> higher nnnn is, the more packets are lost.

That points clearly toward a network problem. Any UDP based service will 
have problems on a network that looses packets. That's why it's called UDP 
= Unreliable.

> I have fixed my problem by using mount options of rsize=2048,wsize=2048.
> rsize=1024,wsize=1024 also works, but is a little slower.

That confirms the network problem.

> Most of the people who have reported similar problems also
> reported using 2.4.x clients, and some reported that the problem
> did not occur before they upgraded to 2.4.x. This suggests that
> there might be a bug in 2.4.x clients.

There are a lot of differences between 2.2 and 2.4, including differences 
in network drivers which might impact your network.

> I have already solved my problem, so I do not need any help.

You've done well. But you've cured the symptoms, not the real cause.

> Next time someone asks what to do about request slot error
> messages, or a hung server, or very slow performance; that person
> should be told to try a smaller rsize/wsize. This should be added
> to the FAQ and to the HOWTO and to the nfs manpage.

Why ? So that afterwards this person complains that the performance of
Linux NFS is poor ? No, this person should read the page at
nfs.sourceforge.net and look at point 11 which is supposed to be a
starting point for investigation, not a panacea. And only after being sure
that there are no problems with the whole setup (hardware + software),
he/she should complain.

> In the FAQ it says that tcp is faster than udp on congested
> networks.  I think the same effect could be achieved by using udp
> with a small rsize/wsize.

That's debatable. TCP is supposed to handle these cases well 'cause that's
what it was designed for. The NFS client over UDP just recently got better
behaviour through Trond's efforts, but having it perform just as over TCP 
would probably mean re-implementing TCP within NFS. Eh, now Trond can 
shoot me :-)

> When I was searching the archive, it would have been easier if I
> could have downloaded one large digest of all messages from the
> last year or two.

All lists hosted at SourceForge behave the same way. Complain to them, if 
you want ;-)

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De



-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: can't get request slot, write timeout

[Trond Myklebust]

>>>>> " " == Bogdan Costescu <bogdan.costescu@iwr.uni-heidelberg.de> writes:

    >> In the FAQ it says that tcp is faster than udp on congested
    >> networks.  I think the same effect could be achieved by using
    >> udp with a small rsize/wsize.

     > That's debatable. TCP is supposed to handle these cases well
     > 'cause that's what it was designed for. The NFS client over UDP
     > just recently got better behaviour through Trond's efforts, but
     > having it perform just as over TCP would probably mean
     > re-implementing TCP within NFS. Eh, now Trond can shoot me :-)

No, I agree 100% with what Bogdan said above.

I wasn't aware that you'd seen better behaviour with the last set of
patches. Feedback is always welcome - particularly now when I'm trying
to work out exactly what we should aim to get into kernel 2.4.20! 

The most likely explanation for the improved behaviour would be that
those patches implement most of the known TCP performance tricks for
congested networks. The only TCP trick we cannot implement is that of
sending messages of length > 1.5k without using fragmentation. That
again means that if 1 fragment is lost, we have to resend 32k (or 8k
or whatever the r/wsize is set to) rather than only the 1.5k that was
actually missing...

The above is why the NFSv4 protocol explicitly specifies that NFS over
TCP *must* be supported, whereas NFS over UDP will only be
optional (In practice most implementations are likely to support both).

Cheers,
  Trond


-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: can't get request slot, write timeout

[Bogdan Costescu]

On 12 Aug 2002, Trond Myklebust wrote:

> I wasn't aware that you'd seen better behaviour with the last set of
> patches. Feedback is always welcome - particularly now when I'm trying
> to work out exactly what we should aim to get into kernel 2.4.20! 

I actually haven't reached a conclusion yet as a result of lack of time.  
My plan was to test it in a controlled "bad" environment, with big speed
(CPU and disk) difference between client and server (and changing roles
afterwards) and with simulated network problems (duplex mismatch, ping
flood, forced Tx timeout). I _think_ that I've already seen an 
improvement, but testing was interrupted by other duties and I'll probably 
have to restart it as I don't have the same hardware anymore :-(

> The only TCP trick we cannot implement is that of sending messages of
> length > 1.5k without using fragmentation. That again means that if 1
> fragment is lost, we have to resend 32k (or 8k or whatever the r/wsize
> is set to) rather than only the 1.5k that was actually missing...

Would it be possible that if the client detects too many fragments to be 
retransmitted _and_ the fragment size is high (like 32k) to scale back and 
only send smaller fragments ? Would that break any NFS rule ?
And I don't know how the actual size would be communicated back to 
user-space...

-- 
Bogdan Costescu

IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: Bogdan.Costescu@IWR.Uni-Heidelberg.De



-------------------------------------------------------
This sf.net email is sponsored by: Dice - The leading online job board
for high-tech professionals. Search and apply for tech jobs today!
http://seeker.dice.com/seeker.epl?rel_code=31
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs