From: Rik van Riel <riel@redhat.com>
To: "Karol Šebesta" <sebesta.karol@gmail.com>
Cc: linux-kernel@vger.kernel.org, linux-admin@vger.kernel.org
Subject: Re: extreme system load [kswapd]
Date: Wed, 21 Mar 2012 10:58:16 -0400 [thread overview]
Message-ID: <4F69EC88.2070006@redhat.com> (raw)
In-Reply-To: <CA+pWcCZ28ByXuOu72FAAAQAPny3O0WDvd+vv1TryXw+9DMKwiA@mail.gmail.com>
On 03/20/2012 05:08 AM, Karol Šebesta wrote:
> Hello @All,
>
> We have a problem on our production machine with high CPU utilization
> caused by kswapd3 daemon. Server is 128GB of physical memory and 81GB
> of SWAP.
> # free -m
> total used free shared buffers cached
> Mem: 128989 75577 53412 0 416 57131
> -/+ buffers/cache: 18029 110960
> Swap: 81919 31310 50609
> #
Looks like a combination of NUMA and the workload thrown
at the system. You did not post any vmstat output, or
info on the size of your Oracle SGA, so I will take some
wild guesses here :)
Not only are you 31GB in swap, you also have 53GB of
memory free. Additionally, only kswapd3 is very busy,
while kswapd on the other NUMA nodes do not even show
up in top.
I would guess that the value of /proc/sys/vm/zone_reclaim_mode
is 1, causing the system to reclaim memory from the NUMA node
where things are running, instead of overflowing memory
allocations into other NUMA nodes.
Setting zone_reclaim_mode to 0 could resolve some of your
issues.
Another, more fundamental, issue is that on older kernels
we mix page cache and process pages on the same LRU lists.
This causes the pageout code to scan over many pages that
we do not want to evict, increasing CPU use by kswapd and
other processes invoking the pageout code.
That issue got fixed in newer kernels, including the
kernel in RHEL 6.
prev parent reply other threads:[~2012-03-21 14:58 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-03-20 9:08 extreme system load [kswapd] Karol Šebesta
2012-03-20 9:47 ` Nicolas Maupu
2012-03-20 18:59 ` Peter Zijlstra
2012-03-21 14:58 ` Rik van Riel [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4F69EC88.2070006@redhat.com \
--to=riel@redhat.com \
--cc=linux-admin@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=sebesta.karol@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).