Re: [RFC 3/3] ACPI: NUMA: Apply SRAT proximity domain to entire CFMWS window

[Dan Williams <dan.j.williams@intel.com>]

Alison Schofield wrote:
> On Thu, May 11, 2023 at 04:16:53PM -0700, Dan Williams wrote:
> > alison.schofield@ wrote:
> 
> snip...
> 
> > > +	/*
> > > +	 * The SRAT may have already described the NUMA details for
> > > +	 * this CFMWS HPA range, yet it may not have created memblks
> > > +	 * for the entire range. Look for a node with a memblk covering
> > > +	 * any part of the HPA range. Don't bother figuring out if it
> > > +	 * is partially or wholly described. Replace any memblks in the
> > > +	 * range with one single memblk that covers the entire range.
> > > +	 *
> > > +	 * This preserves the SRAT defined node and Proximity Domain.
> > > +	 */
> > > +
> > > +	found_node = numa_find_node(start, end);
> > > +	if (found_node != NUMA_NO_NODE) {
> > > +		numa_remove_memblks(found_node, start, end);
> > > +		if (numa_add_memblk(found_node, start, end) < 0) {
> > 
> > I worry that this blows away information in the case where the BIOS for
> > some reason decides to describe multiple proximity domains per CXL
> > window.
> > 
> > I think this wants to be an algorithm like
> > 
> > numa_fill_memblks(start, end)
> > 
> > ...where that walks the range and for any gaps in that range extend the
> > last memblk to the end of the gap.
> > 
> > For example if the CFWMS to SRAT mapping is this:
> > 
> >  ┌──────────────────────────────────────────────────┐
> >  │                  WINDOW0                         │
> >  ├──────────┬────────────┬────────────┬─────────────┤
> >  │PXM0      │ GAP        │ PXM1       │ GAP         │
> >  └──────────┴────────────┴────────────┴─────────────┘
> > 
> > 
> >  ┌──────────────────────────────────────────────────┐
> >  │                  WINDOW0                         │
> >  ├───────────────────────┬──────────────────────────┤
> >  │PXM0                   │ PXM1                     │
> >  └───────────────────────┴──────────────────────────┘
> > 
> > ...and if that finds nothing to fill, *then* create the next proximity
> > domain for the window.
> 
> Thanks for the feedback Dan.
> 
> Is there any guarantee that the first PXM begins at cfmws.start?
> ie...that gaps always 'follow'
> 
> This one's easy:	gap-pxm0 	becomes all pxm0
> This one's easy too:	gap-pxm0-gap 	becomes all pxm0
> 
> This one's not so obvious: gap-pxm0-gap-pxm1 

Good point.

I would say that pxm0 consuming both gaps is ok. pxm1 grabbing the
second gap is also ok, whatever makes the code simpler.

> Imagine those are quarters. Is the end result 50/50 or 75/25?

The window is all one QoS class anyway as far as CXL is concerned. So
the end distribution is not much of a concern. I.e. I would not worry if
you end up with 75/25 even if 50/50 was possible.

> I'm wondering if there is any policy in existence or this is new,
> and we are setting the policy now?

We are trailblazing on our own. The reason why I think we have the
freedom to make an arbitrary decision here is that it is unlikely that the
BIOS describes multiple proximity domains per window, but on the off
chance they do lets not throw away information.