Re: [PATCH 0/3] cxl/core: Enable Region creation on x86 with Low Mem Hole

["Fabio M. De Francesco" <fabio.m.de.francesco@linux.intel.com>]

On Monday, November 25, 2024 11:00:41 PM GMT+1 Alison Schofield wrote:
> On Fri, Nov 22, 2024 at 04:51:51PM +0100, Fabio M. De Francesco wrote:
> > The CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> > Physical Address (HPA) windows that are associated with each CXL Host
> > Bridge. Each window represents a contiguous HPA that may be interleaved
> > with one or more targets (CXL v3.1 - 9.18.1.3).
> > 
> > The Low Memory Hole (LMH) of x86 is a range of addresses of physical low
> > memory to which systems cannot send transactions. On those systems, BIOS
> > publishes CFMWS which communicate the active System Physical Address (SPA)
> > ranges that map to a subset of the Host Physical Address (HPA) ranges. The
> > SPA range trims out the hole, and capacity in the endpoint is lost with no
> > SPA to map to CXL HPA in that hole.
> > 
> > In the early stages of CXL Regions construction and attach on platforms
> > with Low Memory Holes, the driver fails and returns an error because it
> > expects that the CXL Endpoint Decoder range is a subset of the Root
> > Decoder's.
> > 
> > Then detect SPA/HPA misalignment and allow CXL Regions construction and 
> > attach if and only if the misalignment is due to x86 Low Memory Holes.
> > 
> 
> Hi Fabio,
> 
> I took this for a test drive in cxl-test - thanks for that patch!
> 
> Here's a couple of observations on what users will see. Just stirring
> the pot here, not knowing if there is, or even needs to be an explanation
> to userspace about LMH.
> 
> 1) Users will see that the endpoint decoders intend to map more than the
> root decoder. Users may question their trimmed region size.
> 
> 2) At least in this example, I don't think users can re-create this
> region in place, ie via hotplug.  Once this region is destroyed, we
> default to creating a smaller, aligned region, in its place.
> 
Hi Alison,

Thank you for your valuable comments and suggestions on my series. I will 
take them into account for the next version (v2).

I also appreciate your observations regarding the consistency between the two 
methods of creating regions. I agree that we should aim for more uniformity in
this area and we will consider that in the future but we will only create spec 
standard aligned  user created regions now.

Thanks again for your insights.

Fabio
>
> cxl-cli output is appended showing the auto created region, it's decoders,
> and then the creation of a user requested region, not exactly in its
> place.
> 
> 
> Upon load of cxl-test:
> 
> # cxl list -r region0 --decoders -u
> [
>   {
>     "root decoders":[
>       {
>         "decoder":"decoder0.0",
>         "resource":"0xf010000000",
>         "size":"768.00 MiB (805.31 MB)",
>         "interleave_ways":1,
>         "max_available_extent":0,
>         "volatile_capable":true,
>         "qos_class":42,
>         "nr_targets":1
>       }
>     ]
>   },
>   {
>     "port decoders":[
>       {
>         "decoder":"decoder1.0",
>         "resource":"0xf010000000",
>         "size":"1024.00 MiB (1073.74 MB)",
>         "interleave_ways":1,
>         "region":"region0",
>         "nr_targets":1
>       },
>       {
>         "decoder":"decoder6.0",
>         "resource":"0xf010000000",
>         "size":"1024.00 MiB (1073.74 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":4096,
>         "region":"region0",
>         "nr_targets":2
>       }
>     ]
>   },
>   {
>     "endpoint decoders":[
>       {
>         "decoder":"decoder10.0",
>         "resource":"0xf010000000",
>         "size":"1024.00 MiB (1073.74 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":4096,
>         "region":"region0",
>         "dpa_resource":"0",
>         "dpa_size":"512.00 MiB (536.87 MB)",
>         "mode":"ram"
>       },
>       {
>         "decoder":"decoder13.0",
>         "resource":"0xf010000000",
>         "size":"1024.00 MiB (1073.74 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":4096,
>         "region":"region0",
>         "dpa_resource":"0",
>         "dpa_size":"512.00 MiB (536.87 MB)",
>         "mode":"ram"
>       }
>     ]
>   }
> ]
> 
> After destroying the auto region, root decoder show the 768MiB available:
> 
> # cxl list -d decoder0.0 -u
> {
>   "decoder":"decoder0.0",
>   "resource":"0xf010000000",
>   "size":"768.00 MiB (805.31 MB)",
>   "interleave_ways":1,
>   "max_available_extent":"768.00 MiB (805.31 MB)",
>   "volatile_capable":true,
>   "qos_class":42,
>   "nr_targets":1
> }
> 
> 
> # cxl create-region -d decoder0.0 -m mem5 mem4
> {
>   "region":"region0",
>   "resource":"0xf010000000",
>   "size":"512.00 MiB (536.87 MB)",
>   "type":"ram",
>   "interleave_ways":2,
>   "interleave_granularity":256,
>   "decode_state":"commit",
> 
> snip
> 
> # cxl list -r region0 --decoders -u
> [
>   {
>     "root decoders":[
>       {
>         "decoder":"decoder0.0",
>         "resource":"0xf010000000",
>         "size":"768.00 MiB (805.31 MB)",
>         "interleave_ways":1,
>         "max_available_extent":"256.00 MiB (268.44 MB)",
>         "volatile_capable":true,
>         "qos_class":42,
>         "nr_targets":1
>       }
>     ]
>   },
>   {
>     "port decoders":[
>       {
>         "decoder":"decoder1.0",
>         "resource":"0xf010000000",
>         "size":"512.00 MiB (536.87 MB)",
>         "interleave_ways":1,
>         "region":"region0",
>         "nr_targets":1
>       },
>       {
>         "decoder":"decoder6.0",
>         "resource":"0xf010000000",
>         "size":"512.00 MiB (536.87 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":256,
>         "region":"region0",
>         "nr_targets":2
>       }
>     ]
>   },
>   {
>     "endpoint decoders":[
>       {
>         "decoder":"decoder10.0",
>         "resource":"0xf010000000",
>         "size":"512.00 MiB (536.87 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":256,
>         "region":"region0",
>         "dpa_resource":"0",
>         "dpa_size":"256.00 MiB (268.44 MB)",
>         "mode":"ram"
>       },
>       {
>         "decoder":"decoder13.0",
>         "resource":"0xf010000000",
>         "size":"512.00 MiB (536.87 MB)",
>         "interleave_ways":2,
>         "interleave_granularity":256,
>         "region":"region0",
>         "dpa_resource":"0",
>         "dpa_size":"256.00 MiB (268.44 MB)",
>         "mode":"ram"
>       }
>     ]
>   }
> ]
> 
>