Re: [PATCH] ACPI: NUMA: Add a node and memblk for each CFMWS not in SRAT

Dan Williams
2021-10-15T18:58:49Z

On Fri, Oct 15, 2021 at 10:00 AM Jonathan Cameron
<Jonathan.Cameron@huawei.com> wrote:
>
> On Fri, 8 Oct 2021 18:53:39 -0700
> <alison.schofield@intel.com> wrote:
>
> > From: Alison Schofield <alison.schofield@intel.com>
> >
> > During NUMA init, CXL memory defined in the SRAT Memory Affinity
> > subtable may be assigned to a NUMA node. Since there is no
> > requirement that the SRAT be comprehensive for CXL memory another
> > mechanism is needed to assign NUMA nodes to CXL memory not identified
> > in the SRAT.
> >
> > Use the CXL Fixed Memory Window Structure's (CFMWS) of the ACPI CXL
> > Early Discovery Table (CEDT) to find all CXL memory ranges. Create a
> > NUMA node for each range that is not already assigned to a NUMA node.
> > Add a memblk attaching its host physical address range to the node.
> >
> > Note that these ranges may not actually map any memory at boot time.
> > They may describe persistent capacity or may be present to enable
> > hot-plug.
> >
> > Consumers can use phys_to_target_node() to discover the NUMA node.
> >
> > Signed-off-by: Alison Schofield <alison.schofield@intel.com>
> Hi Alison,
>
> I'm not sure that a CFMWS entry should map to a single NUMA node...
>
> Each entry corresponds to a contiguous HPA range into which CXL devices
> below a set of ports (if interleaved) or one port should be mapped.
>
> That could be multiple devices, each with it's own performance characteristics,
> or potentially a mix of persistent and volatile memory on a system with limited
> qtg groups.
>
> Maybe it's the best we can do though given information available
> before any devices are present.
>

Regardless of the performance of the individual devices they can only
map to one of the available CFMWS entries. So the maximum number of
degrees of freedom is one node per CFMWS. Now if you have only one
entry to pick from, but have interleave sets with widely different
performance characteristics to online it becomes a policy decision
about whether to force map those interleave sets into the same node,
and that policy can be maintained outside the kernel.

The alternative is to rework NUMA nodes to be something that can be
declared dynamically as currently there are assumptions throughout the
kernel that num_possible_nodes() is statically determined early in
boot. I am not seeing strong evidence that complexity needs to be
tackled in the near term, and "NUMA-node per CFMWS" should (famous
last words) serve CXL needs for the foreseeable future.