Re: [RFC PATCH 2/3] topology: support node_numa_mem() for determining the fallback node

Nishanth Aravamudan
2014-02-13T06:52:20Z

Hi Joonsoo,

On 11.02.2014 [16:42:00 +0900], Joonsoo Kim wrote:
> On Mon, Feb 10, 2014 at 11:13:21AM -0800, Nishanth Aravamudan wrote:
> > Hi Christoph,
> >
> > On 07.02.2014 [12:51:07 -0600], Christoph Lameter wrote:
> > > Here is a draft of a patch to make this work with memoryless nodes.
> > >
> > > The first thing is that we modify node_match to also match if we hit an
> > > empty node. In that case we simply take the current slab if its there.
> > >
> > > If there is no current slab then a regular allocation occurs with the
> > > memoryless node. The page allocator will fallback to a possible node and
> > > that will become the current slab. Next alloc from a memoryless node
> > > will then use that slab.
> > >
> > > For that we also add some tracking of allocations on nodes that were not
> > > satisfied using the empty_node[] array. A successful alloc on a node
> > > clears that flag.
> > >
> > > I would rather avoid the empty_node[] array since its global and there may
> > > be thread specific allocation restrictions but it would be expensive to do
> > > an allocation attempt via the page allocator to make sure that there is
> > > really no page available from the page allocator.
> >
> > With this patch on our test system (I pulled out the numa_mem_id()
> > change, since you Acked Joonsoo's already), on top of 3.13.0 + my
> > kthread locality change + CONFIG_HAVE_MEMORYLESS_NODES + Joonsoo's RFC
> > patch 1):
> >
> > MemTotal: 8264704 kB
> > MemFree: 5924608 kB
> > ...
> > Slab: 1402496 kB
> > SReclaimable: 102848 kB
> > SUnreclaim: 1299648 kB
> >
> > And Anton's slabusage reports:
> >
> > slab mem objs slabs
> > used active active
> > ------------------------------------------------------------
> > kmalloc-16384 207 MB 98.60% 100.00%
> > task_struct 134 MB 97.82% 100.00%
> > kmalloc-8192 117 MB 100.00% 100.00%
> > pgtable-2^12 111 MB 100.00% 100.00%
> > pgtable-2^10 104 MB 100.00% 100.00%
> >
> > For comparison, Anton's patch applied at the same point in the series:
> >
> > meminfo:
> >
> > MemTotal: 8264704 kB
> > MemFree: 4150464 kB
> > ...
> > Slab: 1590336 kB
> > SReclaimable: 208768 kB
> > SUnreclaim: 1381568 kB
> >
> > slabusage:
> >
> > slab mem objs slabs
> > used active active
> > ------------------------------------------------------------
> > kmalloc-16384 227 MB 98.63% 100.00%
> > kmalloc-8192 130 MB 100.00% 100.00%
> > task_struct 129 MB 97.73% 100.00%
> > pgtable-2^12 112 MB 100.00% 100.00%
> > pgtable-2^10 106 MB 100.00% 100.00%
> >
> >
> > Consider this patch:
> >
> > Acked-by: Nishanth Aravamudan <nacc@linux.vnet.ibm.com>
> > Tested-by: Nishanth Aravamudan <nacc@linux.vnet.ibm.com>
>
> Hello,
>
> I still think that there is another problem.
> Your report about CONFIG_SLAB said that SLAB uses just 200MB.
> Below is your previous report.
>
> Ok, with your patches applied and CONFIG_SLAB enabled:
>
> MemTotal: 8264640 kB
> MemFree: 7119680 kB
> Slab: 207232 kB
> SReclaimable: 32896 kB
> SUnreclaim: 174336 kB
>
> The number on CONFIG_SLUB with these patches tell us that SLUB uses 1.4GB.
> There is large difference on slab usage.

Agreed. But, at least for now, this gets us to not OOM all the time :) I
think that's significant progress. I will continue to look at this
issue for where the other gaps are, but would like to see Christoph's
latest patch get merged (pending my re-testing).

> And, I should note that number of active objects on slabinfo can be
> wrong on some situation, since it doesn't consider cpu slab (and cpu
> partial slab).

Well, I grabbed everything from /sys/kernel/slab for you in the
tarballs, I believe.

> I recommend to confirm page_to_nid() and other things as I mentioned
> earlier.

I believe these all work once CONFIG_HAVE_MEMORYLESS_NODES was set for
ppc64, but will test it again when I have access to the test system.

Also, given that only ia64 and (hopefuly soon) ppc64 can set
CONFIG_HAVE_MEMORYLESS_NODES, does that mean x86_64 can't have
memoryless nodes present? Even with fakenuma? Just curious.

-Nish