Re: alloc_pages_node(... GFP_KERNEL | GFP_THISNODE ...) fails

[Max Laier <max@laiers.net>]

On Friday 29 May 2009 02:46:57 Christoph Lameter wrote:
> On Thu, 28 May 2009, Christoph Lameter wrote:
> > I'm having a bit of trouble with the NUMA allocator in the kernel. 
> > This is in a numa=fake test-setup (though this shouldn't matter, I
> > guess).
>
> Not sure how fake numa works. This could affect the result. page_to_nid

Simply by giving "numa=fake=256,256*" as a boot option I get several nodes 
of 256M each - seemed the easiest way to go for testing.  It's setup from 
arch/x86/mm/numa_64.c::numa_emulation and I didn't spot any obvious 
difference in the setup compared to the hardware detection.  I do have a 
box with k8 NUMA, but unfortunately no KVM support.  Again, remember that 
my test module works flawlessly.

My first reaction was that there must be something in the context that 
affects the allocation (waiting not allowed or some such), but I couldn't 
find anything like this in the call path.

> shows a different number than the node from which the page actually
> came? Sounds broken.
>
> > node 7 - "numactl --hardware" doesn't show any allocations from node
> > 7. In fact it seems that the memory is allocated from the first node
> > with free pages until these run out.  Only then pages from the
> > selected (last) node are given out.  Once the selected node is full,
> > alloc_pages_node(... GFP_THISNODE ...) returns NULL - as it should -
> > and I fall back to a normal allocation that then also reports a
> > different node ID from page_to_nid (c.f. the attached diff).
> >
> > The strange thing is, that a simple test module (attached as well)
> > works as expected.  The allocation succeeds, reports the selected
> > node in page_to_nid *and* the free memory reported from "numactl
> > --hardware" in the selected node decreases.
> >
> > Any insight as to why the KVM allocation might be special are very
> > appreciated.  I tried to follow the call path, but didn't find any
> > red flags that would indicate the difference.
>
> Please verify your numbers using /proc/zoneinfo.

Same result.  "numa_hit" in node 7 increases, while "nr_free_pages" stays 
the same.  Anything else you'd want me to watch out for?

-- 
/"\  Best regards,                      | mlaier@freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier@EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News