From: "J. Bruce Fields" <bfields@fieldses.org>
To: Todd Freeman <freeman@andrews.edu>
Cc: linux-nfs@vger.kernel.org
Subject: Re: nfsd4_stateowners eating memory like candy... sometimes....
Date: Thu, 9 Feb 2012 13:05:32 -0500 [thread overview]
Message-ID: <20120209180531.GA22168@fieldses.org> (raw)
In-Reply-To: <4F32F726.4060704@andrews.edu>
On Wed, Feb 08, 2012 at 05:28:54PM -0500, Todd Freeman wrote:
> Good day all!
>
> I have a nfs server handling the load for a shared file system for 5
> web servers... some relevant info:
> libnfsidmap2
> 0.23-2
> nfs-common
> 1:1.2.2-1ubuntu1.1
> nfs-kernel-server
> 1:1.2.2-1ubuntu1.1
>
> Linux webnfs 2.6.35-31-server #63-Ubuntu SMP Mon Nov 28 21:03:37
> UTC 2011 x86_64 GNU/Linux
>
> On this server everything runs great for a couple weeks to a month
> and then we start getting sluggish performance... and within a
> couple days it seizes up (at least all nfs services stop... console
> is still accessible)
>
> In trying to debug this we have been taking a snap shot every 5
> minutes of the slabinfo... we got a totally clean capture this
> time and I see nfsd4_stateowners running away with memory. When we
> start the server and for the first several days the most memory it
> uses is 200MB or so... over time though there come points were it
> suddenly starts munching more... sometimes slowly... other times
> instantly. It finally kills the machine when it reaches the 1.7-1.8
> GB level (just under the memory size of the machine). oom-killer is
> killing everything left and right at the end and we end up with a
> machine that is comatose NFS wise till we do a full reboot.
>
> You can see a graph of this usage pattern at: http://imgur.com/ecLPh
>
> I see mentions of a problem along this line back in the 2.6.16-18
> types days... but supposedly it was fixed.
There have been a number of stateowner leaks fixed since 2.6.35. I
think all the ones I know of were fixes as of 3.1 or so.
--b.
next prev parent reply other threads:[~2012-02-09 18:05 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-02-08 22:28 nfsd4_stateowners eating memory like candy... sometimes Todd Freeman
2012-02-09 18:05 ` J. Bruce Fields [this message]
2012-02-09 18:18 ` Todd Freeman
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120209180531.GA22168@fieldses.org \
--to=bfields@fieldses.org \
--cc=freeman@andrews.edu \
--cc=linux-nfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).