From: "Robert Mueller" <robm@fastmail.fm>
To: Christoph Lameter <cl@linux.com>,
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Mel Gorman <mel@csn.ul.ie>,
linux-kernel@vger.kernel.org, Bron Gondwana <brong@fastmail.fm>,
linux-mm <linux-mm@kvack.org>
Subject: Re: Default zone_reclaim_mode = 1 on NUMA kernel is bad forfile/email/web servers
Date: Tue, 28 Sep 2010 09:17:00 +1000 [thread overview]
Message-ID: <1285629420.10278.1397188599@webmail.messagingengine.com> (raw)
In-Reply-To: <alpine.DEB.2.00.1009270828510.7000@router.home>
> You can switch off zone reclaim of course which means that the
> applications will not be getting memory thats optimal for them to access.
That's true, but also remember that going to disk is going to be way
more expensive than memory on another node. What we found was that data
that should have been cached because it was being accessed a lot, wasn't
being cached, so it had to keep going back to disk to get it. That's
even worse.
> 1. Fix the ACPI information to indicate lower memory access
> differences (was that info actually acurate?) so that zone reclaim
> defaults to off.
>
> 2. Change the RECLAIM_DISTANCE setting for the arch so that the ACPI
> information does not trigger zone reclaim to be enabled.
How would the ACPI information actually be changed?
I ran numactl -H to get the hardware information, and that seems to
include distances. As mentioned previously, this is a very standard
Intel server motherboard.
http://www.intel.com/Products/Server/Motherboards/S5520UR/S5520UR-specifications.htm
Intel 5520 chipset with Intel I/O Controller Hub ICH10R
$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10 12 14
node 0 size: 24517 MB
node 0 free: 1523 MB
node 1 cpus: 1 3 5 7 9 11 13 15
node 1 size: 24576 MB
node 1 free: 39 MB
node distances:
node 0 1
0: 10 21
1: 21 10
Since I'm not sure what the "distance" values mean, I have no idea if
those values large or not?
> 3. Run the application with numactl settings for interleaving of
> memory accesses (or corresponding cpuset settings).
>
> 4. Fix the application to be conscious of the effect of memory
> allocations on a NUMA systems. Use the numa memory allocations API
> to allocate anonymous memory locally for optimal access and set
> interleave for the file backed pages.
The problem we saw was purely with file caching. The application wasn't
actually allocating much memory itself, but it was reading lots of files
from disk (via mmap'ed memory mostly), and as most people would, we
expected that data would be cached in memory to reduce future reads from
disk. That was not happening.
Rob
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-09-27 23:17 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <1284349152.15254.1394658481@webmail.messagingengine.com>
2010-09-16 10:01 ` Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers KOSAKI Motohiro
2010-09-16 17:06 ` Christoph Lameter
2010-09-17 0:50 ` Robert Mueller
2010-09-17 6:01 ` Shaohua Li
2010-09-17 7:32 ` Robert Mueller
2010-09-17 13:56 ` Christoph Lameter
2010-09-17 14:09 ` Bron Gondwana
2010-09-17 14:22 ` Christoph Lameter
2010-09-17 23:01 ` Bron Gondwana
2010-09-20 9:34 ` Mel Gorman
2010-09-20 23:41 ` Default zone_reclaim_mode = 1 on NUMA kernel is bad forfile/email/web servers Rob Mueller
2010-09-21 9:04 ` Mel Gorman
2010-09-21 14:14 ` Christoph Lameter
2010-09-22 3:44 ` Rob Mueller
2010-09-27 2:01 ` KOSAKI Motohiro
2010-09-27 13:53 ` Christoph Lameter
2010-09-27 23:17 ` Robert Mueller [this message]
2010-09-28 12:35 ` Christoph Lameter
2010-09-28 12:42 ` Bron Gondwana
2010-09-28 12:49 ` Christoph Lameter
2010-09-30 7:05 ` Andi Kleen
2010-10-04 12:45 ` KOSAKI Motohiro
2010-10-04 13:07 ` Christoph Lameter
2010-10-05 5:32 ` KOSAKI Motohiro
2010-10-04 19:43 ` David Rientjes
2010-09-21 1:05 ` Default zone_reclaim_mode = 1 on NUMA kernel is bad for file/email/web servers KAMEZAWA Hiroyuki
2010-09-27 2:04 ` KOSAKI Motohiro
2010-09-27 2:06 ` KAMEZAWA Hiroyuki
2010-09-23 11:44 ` Balbir Singh
2010-09-30 8:38 ` Bron Gondwana
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1285629420.10278.1397188599@webmail.messagingengine.com \
--to=robm@fastmail.fm \
--cc=brong@fastmail.fm \
--cc=cl@linux.com \
--cc=kosaki.motohiro@jp.fujitsu.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mel@csn.ul.ie \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).