Re: [PATCH 0/3] cxl/core: Enable Region creation on x86 with Low Mem Hole

[Alison Schofield <alison.schofield@intel.com>]

On Fri, Nov 22, 2024 at 04:51:51PM +0100, Fabio M. De Francesco wrote:
> The CXL Fixed Memory Window Structure (CFMWS) describes zero or more Host
> Physical Address (HPA) windows that are associated with each CXL Host
> Bridge. Each window represents a contiguous HPA that may be interleaved
> with one or more targets (CXL v3.1 - 9.18.1.3).
> 
> The Low Memory Hole (LMH) of x86 is a range of addresses of physical low
> memory to which systems cannot send transactions. On those systems, BIOS
> publishes CFMWS which communicate the active System Physical Address (SPA)
> ranges that map to a subset of the Host Physical Address (HPA) ranges. The
> SPA range trims out the hole, and capacity in the endpoint is lost with no
> SPA to map to CXL HPA in that hole.
> 
> In the early stages of CXL Regions construction and attach on platforms
> with Low Memory Holes, the driver fails and returns an error because it
> expects that the CXL Endpoint Decoder range is a subset of the Root
> Decoder's.
> 
> Then detect SPA/HPA misalignment and allow CXL Regions construction and 
> attach if and only if the misalignment is due to x86 Low Memory Holes.
> 

Hi Fabio,

I took this for a test drive in cxl-test - thanks for that patch!

Here's a couple of observations on what users will see. Just stirring
the pot here, not knowing if there is, or even needs to be an explanation
to userspace about LMH.

1) Users will see that the endpoint decoders intend to map more than the
root decoder. Users may question their trimmed region size.

2) At least in this example, I don't think users can re-create this
region in place, ie via hotplug.  Once this region is destroyed, we
default to creating a smaller, aligned region, in its place.

cxl-cli output is appended showing the auto created region, it's decoders,
and then the creation of a user requested region, not exactly in its
place.


Upon load of cxl-test:

# cxl list -r region0 --decoders -u
[
  {
    "root decoders":[
      {
        "decoder":"decoder0.0",
        "resource":"0xf010000000",
        "size":"768.00 MiB (805.31 MB)",
        "interleave_ways":1,
        "max_available_extent":0,
        "volatile_capable":true,
        "qos_class":42,
        "nr_targets":1
      }
    ]
  },
  {
    "port decoders":[
      {
        "decoder":"decoder1.0",
        "resource":"0xf010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":1,
        "region":"region0",
        "nr_targets":1
      },
      {
        "decoder":"decoder6.0",
        "resource":"0xf010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region0",
        "nr_targets":2
      }
    ]
  },
  {
    "endpoint decoders":[
      {
        "decoder":"decoder10.0",
        "resource":"0xf010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region0",
        "dpa_resource":"0",
        "dpa_size":"512.00 MiB (536.87 MB)",
        "mode":"ram"
      },
      {
        "decoder":"decoder13.0",
        "resource":"0xf010000000",
        "size":"1024.00 MiB (1073.74 MB)",
        "interleave_ways":2,
        "interleave_granularity":4096,
        "region":"region0",
        "dpa_resource":"0",
        "dpa_size":"512.00 MiB (536.87 MB)",
        "mode":"ram"
      }
    ]
  }
]

After destroying the auto region, root decoder show the 768MiB available:

# cxl list -d decoder0.0 -u
{
  "decoder":"decoder0.0",
  "resource":"0xf010000000",
  "size":"768.00 MiB (805.31 MB)",
  "interleave_ways":1,
  "max_available_extent":"768.00 MiB (805.31 MB)",
  "volatile_capable":true,
  "qos_class":42,
  "nr_targets":1
}


# cxl create-region -d decoder0.0 -m mem5 mem4
{
  "region":"region0",
  "resource":"0xf010000000",
  "size":"512.00 MiB (536.87 MB)",
  "type":"ram",
  "interleave_ways":2,
  "interleave_granularity":256,
  "decode_state":"commit",

snip

# cxl list -r region0 --decoders -u
[
  {
    "root decoders":[
      {
        "decoder":"decoder0.0",
        "resource":"0xf010000000",
        "size":"768.00 MiB (805.31 MB)",
        "interleave_ways":1,
        "max_available_extent":"256.00 MiB (268.44 MB)",
        "volatile_capable":true,
        "qos_class":42,
        "nr_targets":1
      }
    ]
  },
  {
    "port decoders":[
      {
        "decoder":"decoder1.0",
        "resource":"0xf010000000",
        "size":"512.00 MiB (536.87 MB)",
        "interleave_ways":1,
        "region":"region0",
        "nr_targets":1
      },
      {
        "decoder":"decoder6.0",
        "resource":"0xf010000000",
        "size":"512.00 MiB (536.87 MB)",
        "interleave_ways":2,
        "interleave_granularity":256,
        "region":"region0",
        "nr_targets":2
      }
    ]
  },
  {
    "endpoint decoders":[
      {
        "decoder":"decoder10.0",
        "resource":"0xf010000000",
        "size":"512.00 MiB (536.87 MB)",
        "interleave_ways":2,
        "interleave_granularity":256,
        "region":"region0",
        "dpa_resource":"0",
        "dpa_size":"256.00 MiB (268.44 MB)",
        "mode":"ram"
      },
      {
        "decoder":"decoder13.0",
        "resource":"0xf010000000",
        "size":"512.00 MiB (536.87 MB)",
        "interleave_ways":2,
        "interleave_granularity":256,
        "region":"region0",
        "dpa_resource":"0",
        "dpa_size":"256.00 MiB (268.44 MB)",
        "mode":"ram"
      }
    ]
  }
]