Re: Nodes with no memory

[Milton Miller <miltonm@bga.com>]

On Sat Nov 22 at 12:17:22 EST in 2008 Dave Hansen wrote:
> On Fri, 2008-11-21 at 18:49 -0600, Nathan Lynch wrote:
>> Dave Hansen wrote:
>>> I was handed off a bug report about a blade not booting with a, um
>>> "newer" kernel.
>>
>> If you're unable to provide basic information such as the kernel
>> version then perhaps this isn't the best forum for discussing this. =20=

>> :)
>
> Let's just say a derivative of 2.6.27.5.  I will, of course be trying=20=

> to reproduce on mainline.  I'm just going with the kernel closest to=20=

> the bug report as I can get for now.

This reminds me.  I was asked to look at a system that had all cpus and=20=

memory on node 1.  I recently switched to 2.6.27.0, and had a similar=20
failure when I tried my latest development kernel.  However, I realized=20=

that the user was wanting to run my previously supported 2.6.24 kernel,=20=

and that did not have this issue, so I never got back to debugging this=20=

problem.  (Both kernels had similar patches applied, but very little to=20=

mm or numa selection).  I was able to fix the problem they were having=20=

and returned the machine to them without debugging the issue, but I=20
suspect the problem was introduced to mainline between 2.6.24 and=20
2.6.27.

>>> I'm thinking that we need to at least fix careful_allocation() to=20
>>> oops
>>> and not return NULL, or check to make sure all it callers check its
>>> return code.
>>
>> Well, careful_allocation() in current mainline tries pretty hard to
>> panic if it can't satisfy the request.  Why isn't that happening?
>
> I added some random debugging to careful_alloc() to find out.
>
> careful_allocation(1, 7680, 80, 0)
> careful_allocation() ret1: 00000001dffe4100
> careful_allocation() ret2: 00000001dffe4100
> careful_allocation() ret3: 00000001dffe4100
> careful_allocation() ret4: c000000000000000
> careful_allocation() ret5: 0000000000000000
>
> It looks to me like it is hitting 'the memory came from a previously
> =A0allocated node' check.  So, the __lmb_alloc_base() appears to get
> something worthwhile, but that gets overwritten later.
>
> I'm still not quite sure what this comment means.  Are we just trying=20=

> to
> get node locality from the allocation?

My memory (and a quick look) is that careful alloc is used while we are=20=

in the process of creating the memory maps for the node.  We want them=20=

to be allocated from memory on the node, but will accept memory from=20
any node to handle the case that memory is not available in the desired=20=

node.  Linux requires the maps exist for every online node.

Because we are in the process transferring the memory between=20
allocators, the check for new_nid < nid is meant to say "if the memory=20=

did not come from the preferred node, but instead came from one we=20
already transfered, then we need to obtain that memory from the new=20
allocator".  If it came from the preferred node or a later node, the=20
allocation we did is valid, and will be marked in-use when we transfer=20=

that node's memory.

> I also need to go look at how __alloc_bootmem_node() ends up returning
> c000000000000000.  It should be returning NULL, and panic'ing, in
> careful_alloc().  This probably has to do with the fact that=20
> NODE_DATA()
> isn't set up, yet, but I'll double check.

We setup NODE_DATA with the result of this alloc in nid order.  If=20
early_pfs_to_nid returns the wrong value then we would obviously be in=20=

trouble here.

>         /*
>          * If the memory came from a previously allocated node, we =
must
>          * retry with the bootmem allocator.
>          */
>         new_nid =3D early_pfn_to_nid(ret >> PAGE_SHIFT);
>         if (new_nid < nid) {
>                 ret =3D (unsigned=20
> long)__alloc_bootmem_node(NODE_DATA(new_nid),
>                                 size, align, 0);
>                 dbg("careful_allocation() ret4: %016lx\n", ret);
>
>                 if (!ret)
>                         panic("numa.c: cannot allocate %lu bytes on=20
> node %d",
>                               size, new_nid);
>
>                 ret =3D __pa(ret);
>                 dbg("careful_allocation() ret5: %016lx\n", ret);
>
>                 dbg("alloc_bootmem %lx %lx\n", ret, size);
>         }

Perhaps someone can recreate this with the fake numa stuff that was=20
added since 2.6.24?   Or edit a device tree to fake the numa=20
assignments for memory and kexec using the modified tree.

milton