Re: 2.6.38.6 - state manager constantly respawns

[Harry Edmon <harry@uw.edu>]

On 05/16/11 13:33, Trond Myklebust wrote:
> On Mon, 2011-05-16 at 16:21 -0400, Chuck Lever wrote:
>    
>> On May 16, 2011, at 3:43 PM, Trond Myklebust wrote:
>>
>>      
>>> On Mon, 2011-05-16 at 12:36 -0700, Harry Edmon wrote:
>>>        
>>>> On 05/16/11 12:22, Chuck Lever wrote:
>>>>          
>>>>> On May 16, 2011, at 3:12 PM, Harry Edmon wrote:
>>>>>
>>>>>
>>>>>            
>>>>>> Attached is 1000 lines of output from tshark when the problem is occurring.   The client and server are connected by a private ethernet.
>>>>>>
>>>>>>              
>>>>> Disappointing: tshark is not telling us the return codes.  However, I see "PUTFH;READ" then "RENEW" in a loop, which indicates the state manager thread is being kicked off because of ongoing difficulties with state recovery.  Is there a stuck application on that client?
>>>>>
>>>>> Try again with "tshark -V".
>>>>>
>>>>>            
>>>> Here is the output from tshark -V (first 50,000 lines).   Nothing
>>>> appears to be stuck, and as I said when I reboot the client into 2.6.32
>>>> the problem goes away, only to reappear when I reboot it back into 2.6.38.6.
>>>>
>>>>          
>>> Possibly, but it definitely indicates a server bug. What kind of server
>>> are you using?
>>>
>>> Basically, the client is getting confused because when it sends a READ,
>>> the server is telling it that the lease has expired, then when it sends
>>> a RENEW, the same server replies that the lease is OK...
>>>        
>> I've seen this during migration recovery testing.  The client may be testing the wrong client ID.
>>
>> But I wonder if it's really worth doing that separate RENEW.  I've seen the client send a RENEW after it gets STALE_STATEID.  Would RENEW really tell the client anything in that case?
>>      
> It is needed.
>
> Without the RENEW, we have no idea whether or not we need to do a full
> state recovery. Running a full recovery when we don't have to is _bad_,
> and will usually cause us to lose delegations and may possibly even
> cause us to lose locks.
>
>    
By the way, this is not the only client/server running 2.6.38 that I 
have this problem on.   It is occurring on other random ones I 
maintain.  This example is happens to be the cleanest one I have, this 
NFS server is only talking to this specific NFS client over a private 
network.
-- 

  Dr. Harry Edmon			E-MAIL: harry@uw.edu
  206-543-0547 FAX: 206-543-0308			harry-qmPYOCrcNLLyFCzt5hm0YvZ8FUJU4vz8@public.gmane.org
  Director of IT, College of the Environment and
  Director of Computing, Dept of Atmospheric Sciences
  University of Washington, Box 351640, Seattle, WA 98195-1640