nfsd4_stateowners eating memory like candy... sometimes....

nfsd4_stateowners eating memory like candy... sometimes....

[Todd Freeman]

Good day all!

I have a nfs server handling the load for a shared file system for 5 web 
servers...  some relevant info:
   libnfsidmap2                                                         
0.23-2
   nfs-common                                                           
1:1.2.2-1ubuntu1.1
   nfs-kernel-server                                                    
1:1.2.2-1ubuntu1.1

   Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37 UTC 
2011 x86_64 GNU/Linux

On this server everything runs great for a couple weeks to a month and 
then we start getting sluggish performance... and within a couple days 
it seizes up (at least all nfs services stop... console is still accessible)

In trying to debug this we have been taking a snap shot every 5 minutes 
of the slabinfo...   we got a totally clean capture this time and I see 
nfsd4_stateowners running away with memory. When we start the server and 
for the first several days the most memory it uses is 200MB or so...  
over time though there come points were it suddenly starts munching 
more...  sometimes slowly... other times instantly. It finally kills the 
machine when it reaches the 1.7-1.8 GB level (just under the memory size 
of the machine). oom-killer is killing everything left and right at the 
end and we end up with a machine that is comatose NFS wise till we do a 
full reboot.

You can see a graph of this usage pattern at: http://imgur.com/ecLPh

I see mentions of a problem along this line back in the 2.6.16-18 types 
days... but supposedly it was fixed.

Does anyone have any ideas?

-- 
Todd Freeman  Ext 6103                   .^.    Don't fear the penguins!
Programming Department                   /V\
Andrews University                      // \\    http://www.linux.org/
http://www.andrews.edu/~freeman/       /(   )\   http://www.debian.org/
                                         ^^ ^^

Re: nfsd4_stateowners eating memory like candy... sometimes....

[J. Bruce Fields]

On Wed, Feb 08, 2012 at 05:28:54PM -0500, Todd Freeman wrote:
> Good day all!
> 
> I have a nfs server handling the load for a shared file system for 5
> web servers...  some relevant info:
>   libnfsidmap2
> 0.23-2
>   nfs-common
> 1:1.2.2-1ubuntu1.1
>   nfs-kernel-server
> 1:1.2.2-1ubuntu1.1
> 
>   Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37
> UTC 2011 x86_64 GNU/Linux
> 
> On this server everything runs great for a couple weeks to a month
> and then we start getting sluggish performance... and within a
> couple days it seizes up (at least all nfs services stop... console
> is still accessible)
> 
> In trying to debug this we have been taking a snap shot every 5
> minutes of the slabinfo...   we got a totally clean capture this
> time and I see nfsd4_stateowners running away with memory. When we
> start the server and for the first several days the most memory it
> uses is 200MB or so...  over time though there come points were it
> suddenly starts munching more...  sometimes slowly... other times
> instantly. It finally kills the machine when it reaches the 1.7-1.8
> GB level (just under the memory size of the machine). oom-killer is
> killing everything left and right at the end and we end up with a
> machine that is comatose NFS wise till we do a full reboot.
> 
> You can see a graph of this usage pattern at: http://imgur.com/ecLPh
> 
> I see mentions of a problem along this line back in the 2.6.16-18
> types days... but supposedly it was fixed.

There have been a number of stateowner leaks fixed since 2.6.35.  I
think all the ones I know of were fixes as of 3.1 or so.

--b.

Re: nfsd4_stateowners eating memory like candy... sometimes....

[Todd Freeman]

Thanks!  I'll try a kernel update and see what sticks :)



On 02/09/2012 01:05 PM, J. Bruce Fields wrote:
> On Wed, Feb 08, 2012 at 05:28:54PM -0500, Todd Freeman wrote:
>> Good day all!
>>
>> I have a nfs server handling the load for a shared file system for 5
>> web servers...  some relevant info:
>>    libnfsidmap2
>> 0.23-2
>>    nfs-common
>> 1:1.2.2-1ubuntu1.1
>>    nfs-kernel-server
>> 1:1.2.2-1ubuntu1.1
>>
>>    Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37
>> UTC 2011 x86_64 GNU/Linux
>>
>> On this server everything runs great for a couple weeks to a month
>> and then we start getting sluggish performance... and within a
>> couple days it seizes up (at least all nfs services stop... console
>> is still accessible)
>>
>> In trying to debug this we have been taking a snap shot every 5
>> minutes of the slabinfo...   we got a totally clean capture this
>> time and I see nfsd4_stateowners running away with memory. When we
>> start the server and for the first several days the most memory it
>> uses is 200MB or so...  over time though there come points were it
>> suddenly starts munching more...  sometimes slowly... other times
>> instantly. It finally kills the machine when it reaches the 1.7-1.8
>> GB level (just under the memory size of the machine). oom-killer is
>> killing everything left and right at the end and we end up with a
>> machine that is comatose NFS wise till we do a full reboot.
>>
>> You can see a graph of this usage pattern at: http://imgur.com/ecLPh
>>
>> I see mentions of a problem along this line back in the 2.6.16-18
>> types days... but supposedly it was fixed.
> There have been a number of stateowner leaks fixed since 2.6.35.  I
> think all the ones I know of were fixes as of 3.1 or so.
>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Todd Freeman  Ext 6103                   .^.    Don't fear the penguins!
Programming Department                   /V\
Andrews University                      // \\    http://www.linux.org/
http://www.andrews.edu/~freeman/       /(   )\   http://www.debian.org/
                                         ^^ ^^