Re: unexpected NFS timeouts, related to sync/async soft mounts over TCP

[Andrew Cooper <andrew.cooper3@citrix.com>]

[-- Attachment #1: Type: text/plain, Size: 2950 bytes --]

On 11/11/11 22:38, Trond Myklebust wrote:
> On Fri, 2011-11-11 at 10:31 +0000, Andrew Cooper wrote: 
>> On 10/11/11 20:43, Trond Myklebust wrote:
>>> On Thu, 2011-11-10 at 15:52 +0000, Andrew Cooper wrote: 
>>>> On 10/11/11 15:29, Chuck Lever wrote:
>>>>> On Nov 10, 2011, at 6:15 AM, Andrew Cooper wrote:
>>>>>
>>>>>> On 09/11/11 22:36, Chuck Lever wrote:
>>>>>>> On Nov 9, 2011, at 1:38 PM, Andrew Cooper wrote:
>>>> Sorry.  I am not sure I was clear.  An EIO does not present itself with
>>>> a hard mount, but a TCP FIN is still injected into the stream by the
>>>> client, causing 15 seconds of deadlock, eventually fixed by sending a
>>>> RST and restarting with a new TCP stream.  At this point, softmounts
>>>> throw an EIO while hardmounts restart and continue successfully.
>>>>
>>>> My problem is not the EIO on softmount or lack of EIO for hardmout, but
>>>> the fact that the client sees fit to try and close the TCP stream while
>>>> an apparently otherwise healthy NFS session is ongoing.
>>> The client will attempt to close the TCP connection on any RPC level
>>> error. That can happen, e.g., if the server sends a faulty RPC/TCP
>>> record fragment header or some other garbage data.
>>>
>>> I'm assuming that you've checked that the TCP parameters are set to sane
>>> values for a 10GigE connection (i.e. tcp_timestamps is on) so that there
>>> is no corruption happening at that level?
>>>
>>> Cheers
>>>   Trond
>> I have a TCPdump/wireshark analysis of the entire packet stream (4GiB). 
>> I cant see any RPC level errors (rpc.replystat != 0 yields no matches). 
>> What specifically would I be looking for?  Wireshark seems not to have
>> any problem decoding any of the RPC packets, so I hope that indicates no
>> RPC level corruption.
>>
>> There is one case where the server sends a double write reply for the
>> same write, with different length fields.  However, this is a good 20
>> seconds before the FIN is sent, so I was hoping that it was unrelated.
>> Might it not be?
> Can you send us just that portion of the trace so that we can have a look?

Attached is a small extract from the stream.  It starts with the final
NFS write, and continues through the FIN, RST and until the TCP stream
gets reopened.  Is this what you want?

>> As for TCP timestamps; I have a Timestamp option in each TCP packet. 
>> Nothing appears corrupted.  What would I be looking for with corrupted
>> timestamps?
> I just meant that you should check that you've enabled tcp window
> scaling and timestamps in order to avoid problems with wrapped sequence
> numbers (See  http://tools.ietf.org/html/rfc1323 for details).
>
> On Linux this means that you need to check
>
> 	sysctl net.ipv4.tcp_timestamps
> and
> 	sysctl net.ipv4.tcp_window_scaling
>
> They should both be set to the value '1'.

They both are.

> Cheers
>   Trond
>
-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com


[-- Attachment #2: nfs-fin.capture.gz --]
[-- Type: application/x-gzip, Size: 1621 bytes --]