From: Nishanth Aravamudan <nacc@us.ibm.com>
To: Christoph Lameter <clameter@sgi.com>
Cc: lee.schermerhorn@hp.com, wli@holomorphy.com, melgor@ie.ibm.com,
akpm@linux-foundation.org, linux-mm@kvack.org, agl@us.ibm.com,
pj@sgi.com
Subject: Re: [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing
Date: Tue, 7 Aug 2007 18:50:42 -0700 [thread overview]
Message-ID: <20070808015042.GF15714@us.ibm.com> (raw)
In-Reply-To: <Pine.LNX.4.64.0708061101470.24256@schroedinger.engr.sgi.com>
On 06.08.2007 [11:04:48 -0700], Christoph Lameter wrote:
> On Mon, 6 Aug 2007, Nishanth Aravamudan wrote:
>
> > hugetlb: fix cpuset-constrained pool resizing
> >
> > With the previous 3 patches in this series applied, if a process is in a
> > constrained cpuset, and tries to grow the hugetlb pool, hugepages may be
> > allocated on nodes outside of the process' cpuset. More concretely,
> > growing the pool via
> >
> > echo some_value > /proc/sys/vm/nr_hugepages
> >
> > interleaves across all nodes with memory such that hugepage allocations
> > occur on nodes outside the cpuset. Similarly, this process is able to
> > change the values in values in
> > /sys/devices/system/node/nodeX/nr_hugepages, even when X is not in the
> > cpuset. This directly violates the isolation that cpusets is supposed to
> > guarantee.
>
> No it does not. Cpusets do not affect the administrative rights of users.
For reference here (as I just ran my simple script against
2.6.23-rc1-mm2, 2.6.23-rc1-mm2 + your patches, 2.6.23-rc1-mm2 + your
patches + each of my patches in turn), this is completely untrue with
-mm2 and your patches. I was actually trying to restore this behavior
with this patch. I realize I didn't mention this earlier... On a 4-node
x86_64:
2.6.23-rc1-mm2:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 100
Node 0 HugePages_Free: 0
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
2.6.23-rc1-mm2 + your patches:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 100
Node 0 HugePages_Free: 0
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
After my patch 1/2 (try harder) from this morning:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 75
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 0
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 100
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 25
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 100
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 25
Done. 150 free
After patch 2/2 (memoryless nodes) from this morning (the results are
actually the same as the above, just that the values are shifted around
the nodes a bit):
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 75
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 75
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 100
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 100
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 25
Done. 175 free
Finally, after my hugetlb interleave dequeue patch is applied:
/cpuset ~
Trying to resize the pool to 200 from the top cpuset
Node 3 HugePages_Free: 50
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 50
Done. 200 free
Trying to resize the pool back to 100 from the top cpuset
Node 3 HugePages_Free: 25
Node 2 HugePages_Free: 25
Node 1 HugePages_Free: 25
Node 0 HugePages_Free: 25
Done. 100 free
/cpuset/set1 /cpuset ~
Trying to resize the pool to 200 from a cpuset restricted to node 1
Node 3 HugePages_Free: 50
Node 2 HugePages_Free: 50
Node 1 HugePages_Free: 50
Node 0 HugePages_Free: 50
Done. 200 free
Trying to shrink the pool down to 0 from a cpuset restricted to node 1
Node 3 HugePages_Free: 0
Node 2 HugePages_Free: 0
Node 1 HugePages_Free: 0
Node 0 HugePages_Free: 0
Done. 0 free
So, it would appear that, in your opinion, this set of patches
constitutes a pseudo-bug-fix? Without the last patch, it seems, cpusets
are able to constrain what nodes a root process can remove hugepages
from.
Thanks,
Nish
--
Nishanth Aravamudan <nacc@us.ibm.com>
IBM Linux Technology Center
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2007-08-08 1:50 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-08-06 16:32 [RFC][PATCH 0/5] hugetlb NUMA improvements Nishanth Aravamudan
2007-08-06 16:37 ` [RFC][PATCH 1/5] Fix hugetlb pool allocation with empty nodes V9 Nishanth Aravamudan
2007-08-06 16:38 ` [RFC][PATCH 2/5] hugetlb: numafy several functions Nishanth Aravamudan
2007-08-06 16:40 ` [RFC][PATCH 3/5] hugetlb: add per-node nr_hugepages sysfs attribute Nishanth Aravamudan
2007-08-06 16:44 ` [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing Nishanth Aravamudan
2007-08-06 16:45 ` Nishanth Aravamudan
2007-08-06 16:48 ` [RFC][PATCH 5/5] hugetlb: interleave dequeueing of huge pages Nishanth Aravamudan
2007-08-06 18:04 ` [RFC][PATCH 4/5] hugetlb: fix cpuset-constrained pool resizing Christoph Lameter
2007-08-06 18:26 ` Nishanth Aravamudan
2007-08-06 18:41 ` Christoph Lameter
2007-08-07 0:03 ` Nishanth Aravamudan
2007-08-06 19:37 ` Lee Schermerhorn
2007-08-08 1:50 ` Nishanth Aravamudan [this message]
2007-08-08 13:26 ` Lee Schermerhorn
2007-08-06 17:59 ` [RFC][PATCH 2/5] hugetlb: numafy several functions Christoph Lameter
2007-08-06 18:15 ` Nishanth Aravamudan
2007-08-07 0:34 ` Nishanth Aravamudan
2007-08-06 18:00 ` [RFC][PATCH 1/5] Fix hugetlb pool allocation with empty nodes V9 Christoph Lameter
2007-08-06 18:19 ` Nishanth Aravamudan
2007-08-06 18:37 ` Christoph Lameter
2007-08-06 19:52 ` Lee Schermerhorn
2007-08-06 20:15 ` Christoph Lameter
2007-08-07 0:04 ` Nishanth Aravamudan
2007-08-06 16:39 ` [RFC][PATCH 0/5] hugetlb NUMA improvements Nishanth Aravamudan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20070808015042.GF15714@us.ibm.com \
--to=nacc@us.ibm.com \
--cc=agl@us.ibm.com \
--cc=akpm@linux-foundation.org \
--cc=clameter@sgi.com \
--cc=lee.schermerhorn@hp.com \
--cc=linux-mm@kvack.org \
--cc=melgor@ie.ibm.com \
--cc=pj@sgi.com \
--cc=wli@holomorphy.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.