Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes

[Catalin Marinas <catalin.marinas@arm.com>]

On Thu, Oct 29, 2020 at 12:48:50PM -0700, Vanshidhar Konda wrote:
> On Thu, Oct 29, 2020 at 01:37:10PM +0000, Catalin Marinas wrote:
> > On Wed, Oct 21, 2020 at 11:29:41PM +0100, Valentin Schneider wrote:
> > > On 21/10/20 12:02, Jonathan Cameron wrote:
> > > > On Wed, 21 Oct 2020 09:43:21 +0530
> > > > Anshuman Khandual <anshuman.khandual@arm.com> wrote:
> > > >> Agreed. Do we really need to match X86 right now ? Do we really have
> > > >> systems that has 64 nodes ? We should not increase the default node
> > > >> value and then try to solve some new problems, when there might not
> > > >> be any system which could even use that. I would suggest increase
> > > >> NODES_SHIFT value upto as required by a real and available system.
> > > >
> > > > I'm not going to give precise numbers on near future systems but it is public
> > > > that we ship 8 NUMA node ARM64 systems today.  Things will get more
> > > > interesting as CXL and CCIX enter the market on ARM systems,
> > > > given chances are every CXL device will look like another NUMA
> > > > node (CXL spec says they should be presented as such) and you
> > > > may be able to rack up lots of them.
> > > >
> > > > So I'd argue minimum that makes sense today is 16 nodes, but looking forward
> > > > even a little and 64 is not a great stretch.
> > > > I'd make the jump to 64 so we can forget about this again for a year or two.
> > > > People will want to run today's distros on these new machines and we'd
> > > > rather not have to go around all the distros asking them to carry a patch
> > > > increasing this count (I assume they are already carrying such a patch
> > > > due to those 8 node systems)
> > > 
> > > I agree that 4 nodes is somewhat anemic; I've had to bump that just to
> > > run some scheduler tests under QEMU. However I still believe we should
> > > exercise caution before cranking it too high, especially when seeing things
> > > like:
> > > 
> > >   ee38d94a0ad8 ("page flags: prioritize kasan bits over last-cpuid")
> > > 
> > > To give some numbers, a defconfig build gives me:
> > > 
> > >   SECTIONS_WIDTH=0 ZONES_WIDTH=2 NODES_SHIFT=2 LAST_CPUPID_SHIFT=(8+8) KASAN_TAG_WIDTH=0
> > >   BITS_PER_LONG=64 NR_PAGEFLAGS=24
> > > 
> > > IOW, we need 18 + NODES_SHIFT <= 40 -> NODES_SHIFT <= 22. That looks to be
> > > plenty, however this can get cramped fairly easily with any combination of:
> > > 
> > >   CONFIG_SPARSEMEM_VMEMMAP=n (-18)
> > >   CONFIG_IDLE_PAGE_TRACKING=y (-2)
> > >   CONFIG_KASAN=y + CONFIG_KASAN_SW_TAGS (-8)
> > > 
> > > Taking Arnd's above example, a randconfig build picking !VMEMMAP already
> > > limits the NODES_SHIFT to 4 *if* we want to keep the CPUPID thing within
> > > the flags (it gets a dedicated field at the tail of struct page
> > > otherwise). If that is something we don't care too much about, then
> > > consider my concerns taken care of.
> > 
> > I don't think there's any value in allowing SPARSEMEM_VMEMMAP to be
> > disabled but the option is in the core mm/Kconfig file. We could make
> > NODES_SHIFT depend on SPARSEMEM_VMEMMAP (there's DISCONTIGMEM as well
> > but hopefully that's going away soon).
> > 
> > > One more thing though: NR_CPUS can be cranked up to 4096 but we've only set
> > > it to 256 IIRC to support the TX2. From that PoV, I'm agreeing with
> > > Anshuman in that we should set it to match the max encountered on platforms
> > > that are in use right now.
> > 
> > I agree. Let's bump NODES_SHIFT to 4 now to cover existing platforms. If
> > distros have a 10-year view, they can always ship a kernel configured to
> > 64 nodes, no need to change Kconfig (distros never ship with defconfig).
> > 
> > It may have an impact on more memory constrained platforms but that's
> > not what defconfig is about. It should allow existing hardware to run
> > Linux but not necessarily run it in the most efficient way possible.
> > 
> 
> From the discussion it looks like 4 is an acceptable number to support
> current hardware. I'll send a patch with NODES_SHIFT set to 4. Is it still
> possible to add this change to the 5.10 kernel?

I think we can but I'll leave the decision to Will (and don't forget to
cc the arm64 maintainers on your next post).

-- 
Catalin