Re: [RFC 3/3] ACPI: NUMA: Apply SRAT proximity domain to entire CFMWS window

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Thu, 11 May 2023 17:18:27 -0700
Dan Williams <dan.j.williams@intel.com> wrote:

> Alison Schofield wrote:
> > On Thu, May 11, 2023 at 04:16:53PM -0700, Dan Williams wrote:  
> > > alison.schofield@ wrote:  
> > 
> > snip...
> >   
> > > > +	/*
> > > > +	 * The SRAT may have already described the NUMA details for
> > > > +	 * this CFMWS HPA range, yet it may not have created memblks
> > > > +	 * for the entire range. Look for a node with a memblk covering
> > > > +	 * any part of the HPA range. Don't bother figuring out if it
> > > > +	 * is partially or wholly described. Replace any memblks in the
> > > > +	 * range with one single memblk that covers the entire range.
> > > > +	 *
> > > > +	 * This preserves the SRAT defined node and Proximity Domain.
> > > > +	 */
> > > > +
> > > > +	found_node = numa_find_node(start, end);
> > > > +	if (found_node != NUMA_NO_NODE) {
> > > > +		numa_remove_memblks(found_node, start, end);
> > > > +		if (numa_add_memblk(found_node, start, end) < 0) {  
> > > 
> > > I worry that this blows away information in the case where the BIOS for
> > > some reason decides to describe multiple proximity domains per CXL
> > > window.
> > > 
> > > I think this wants to be an algorithm like
> > > 
> > > numa_fill_memblks(start, end)
> > > 
> > > ...where that walks the range and for any gaps in that range extend the
> > > last memblk to the end of the gap.
> > > 
> > > For example if the CFWMS to SRAT mapping is this:
> > > 
> > >  ┌──────────────────────────────────────────────────┐
> > >  │                  WINDOW0                         │
> > >  ├──────────┬────────────┬────────────┬─────────────┤
> > >  │PXM0      │ GAP        │ PXM1       │ GAP         │
> > >  └──────────┴────────────┴────────────┴─────────────┘
> > > 
> > > 
> > >  ┌──────────────────────────────────────────────────┐
> > >  │                  WINDOW0                         │
> > >  ├───────────────────────┬──────────────────────────┤
> > >  │PXM0                   │ PXM1                     │
> > >  └───────────────────────┴──────────────────────────┘
> > > 
> > > ...and if that finds nothing to fill, *then* create the next proximity
> > > domain for the window.  
> > 
> > Thanks for the feedback Dan.
> > 
> > Is there any guarantee that the first PXM begins at cfmws.start?
> > ie...that gaps always 'follow'
> > 
> > This one's easy:	gap-pxm0 	becomes all pxm0
> > This one's easy too:	gap-pxm0-gap 	becomes all pxm0
> > 
> > This one's not so obvious: gap-pxm0-gap-pxm1   
> 
> Good point.
> 
> I would say that pxm0 consuming both gaps is ok. pxm1 grabbing the
> second gap is also ok, whatever makes the code simpler.

Agreed - it's best effort on a heuristic.  My guess is we'll end up
revisiting this when hotplug means we see similar patterns to those
that led a bios to do the above, but this is fine for now.

> 
> > Imagine those are quarters. Is the end result 50/50 or 75/25?  
> 
> The window is all one QoS class anyway as far as CXL is concerned. So
> the end distribution is not much of a concern. I.e. I would not worry if
> you end up with 75/25 even if 50/50 was possible.

I don't like tying QoS class to a narrow range of access characteristics
but that's a problem for another day.

> 
> > I'm wondering if there is any policy in existence or this is new,
> > and we are setting the policy now?  
> 
> We are trailblazing on our own. The reason why I think we have the
> freedom to make an arbitrary decision here is that it is unlikely that the
> BIOS describes multiple proximity domains per window, but on the off
> chance they do lets not throw away information.

Agree with not throwing it away, fairly sure you'll see this :)
If nothing else it's handy for a system that doesn't do interleaving
across devices for some reason where the OS is then going to use NUMA Interleave.

So it can be useful to have multiple proximity domains even for identical
memory if the access paths are different.

Ha! I see an excuse to reuse old work:
https://github.com/hisilicon/acpi-numa-whitepaper/releases/tag/v0.93
See 5.1

I should bring that back one day when things are quieter (so likely never).
Was targeted as an ACPI WP and out for review when something happened (check
the date and who I work for if curious ;)

Jonathan