Re: [PATCH 5/5] [hugetlb] Try to grow pool for MAP_SHARED mappings

Ken Chen
2007-07-13T21:24:47Z

On 7/13/07, Adam Litke <agl@us.ibm.com> wrote:
> To be honest, I just don't think a global hugetlb pool and cpusets are
> compatible, period.

Agreed. It's a mess.

> > But the cpuset behaviour of this hugetlb stuff looks suspicious to me:
> > 1) The code in alloc_fresh_huge_page() seems to round robin over
> > the entire system, spreading the hugetlb pages uniformly on all nodes.
> > If one a task in one small cpuset starts aggressively allocating hugetlb
> > pages, do you think this will work, Adam -- looks to me like we will end
> > up calling alloc_fresh_huge_page() many times, most of which will fail to
> > alloc_pages_node() anything because the 'static nid' clock hand will be
> > pointing at a node outside of the current tasks cpuset (not in that tasks
> > mems_allowed). Inefficient, but I guess ok.
>
> Very good point. I guess we call alloc_fresh_huge_page in two scenarios
> now... 1) By echoing a number into /proc/sys/vm/nr_hugepages, and 2) by
> trying to dynamically increase the pool size for a particular process.
> Case 1 is not in the context of any process (per se) and so
> node_online_map makes sense. For case 2 we could teach the
> __alloc_fresh_huge_page() to take a nodemask. That could get nasty
> though since we'd have to move away from a static variable to get proper
> interleaving.

alloc_fresh_huge_page
alloc_pages_node
get_page_from_freelist {
...
if ((alloc_flags & ALLOC_CPUSET) &&
!cpuset_zone_allowed_softwall(zone, gfp_mask))
goto try_next_zone;
...

It looks to me that cpuset rule is buried deep down in the buddy
allocator. So the cpuset mem_allowed rule is enforced in both pool
reservation time (in get_page_from_freelist) and hugetlb page fault
time in dequeue_huge_page().