From: Todd Freeman <freeman@andrews.edu>
To: "J. Bruce Fields" <bfields@fieldses.org>
Cc: linux-nfs@vger.kernel.org
Subject: Re: nfsd4_stateowners eating memory like candy... sometimes....
Date: Thu, 09 Feb 2012 13:18:01 -0500 [thread overview]
Message-ID: <4F340DD9.7070402@andrews.edu> (raw)
In-Reply-To: <20120209180531.GA22168@fieldses.org>
Thanks! I'll try a kernel update and see what sticks :)
On 02/09/2012 01:05 PM, J. Bruce Fields wrote:
> On Wed, Feb 08, 2012 at 05:28:54PM -0500, Todd Freeman wrote:
>> Good day all!
>>
>> I have a nfs server handling the load for a shared file system for 5
>> web servers... some relevant info:
>> libnfsidmap2
>> 0.23-2
>> nfs-common
>> 1:1.2.2-1ubuntu1.1
>> nfs-kernel-server
>> 1:1.2.2-1ubuntu1.1
>>
>> Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37
>> UTC 2011 x86_64 GNU/Linux
>>
>> On this server everything runs great for a couple weeks to a month
>> and then we start getting sluggish performance... and within a
>> couple days it seizes up (at least all nfs services stop... console
>> is still accessible)
>>
>> In trying to debug this we have been taking a snap shot every 5
>> minutes of the slabinfo... we got a totally clean capture this
>> time and I see nfsd4_stateowners running away with memory. When we
>> start the server and for the first several days the most memory it
>> uses is 200MB or so... over time though there come points were it
>> suddenly starts munching more... sometimes slowly... other times
>> instantly. It finally kills the machine when it reaches the 1.7-1.8
>> GB level (just under the memory size of the machine). oom-killer is
>> killing everything left and right at the end and we end up with a
>> machine that is comatose NFS wise till we do a full reboot.
>>
>> You can see a graph of this usage pattern at: http://imgur.com/ecLPh
>>
>> I see mentions of a problem along this line back in the 2.6.16-18
>> types days... but supposedly it was fixed.
> There have been a number of stateowner leaks fixed since 2.6.35. I
> think all the ones I know of were fixes as of 3.1 or so.
>
> --b.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Todd Freeman Ext 6103 .^. Don't fear the penguins!
Programming Department /V\
Andrews University // \\ http://www.linux.org/
http://www.andrews.edu/~freeman/ /( )\ http://www.debian.org/
^^ ^^
prev parent reply other threads:[~2012-02-09 18:18 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-08 22:28 nfsd4_stateowners eating memory like candy... sometimes Todd Freeman
2012-02-09 18:05 ` J. Bruce Fields
2012-02-09 18:18 ` Todd Freeman [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4F340DD9.7070402@andrews.edu \
--to=freeman@andrews.edu \
--cc=bfields@fieldses.org \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).