RE: Assistance Needed with CXL Memory Region Creation

[Dan Williams <dan.j.williams@intel.com>]

Yasunori Gotou (Fujitsu) wrote:
> Ping....
> Is there anyone who can provide information to help resolve this issue?

I missed this earlier... holidays + merge window opening == inbox
overflow.

> I met this issue at kernel 6.13-rc4. Are there any fix for this issue? 
> Otherwise, are there any kernel version which can create region under the special purpose memory?
> 
> I'll add some more information below...

Thanks for that!
 
> I think cxl create-region should succeed in the special purpose memory area, but I cannot
> understand why the above code results in an ERANGE error...
[..]
> > Below is the relevant section from /proc/iomem:
> > ====
> > 89d500000-89fffffff : Reserved
> > 8a0000000-189fffffff : CXL Window 0
> >   8a0000000-189fffffff : Soft Reserved  ------!!!!
> > fd00000000-fd03ffffff : Reserved
> > =====

This memory map is saying that BIOS created a CXL Window "CXL Window 0"
AND it populated that window with a CXL region marked "Soft Reserved".
The window is fully consumed by that existing BIOS created region. The
expectation is that the CXL subsystem parses the BIOS configuration and
creates a "cxl_region" object as a child of that Soft Reserved range.

However, for whatever reason it looks like the driver failed to parse
the CXL configuration. If that had worked the flow to create a new
region in that space would require first deleting the BIOS created
region.

So, the behavior you are seeing is expected. You can not create a
cxl_region in a space that already has a BIOS created cxl_region.

The work in this patchset [1] is aimed at making sure that even if the
kernel does not understand the BIOS CXL configuration it will still at
least transfer control of that address range to the device-dax
subsystem.

[1]: http://lore.kernel.org/cover.1737046620.git.nathan.fontenot@amd.com

...but that's just a fallback crutch. I would be interested to see more
details on why the kernel failed to assemble the region that the BIOS
created in this case. 

The overall flow is:

- BIOS creates ACPI CFMWS
- BIOS optionally creates regions within one more CFMWS ranges
- BIOS builds EFI memory map with the CXL regions marked EFI_MEMORY_SP
- Linux boots and sees EFI_MEMORY_SP + CXL overlap and waits for the CXL
  subsystem to assemble the region
- Region creation is only allowed with free capacity

The known bugs are:
- Corner case CXL configurations that trip up the driver (memory side
  caching and CXL interleaved with DDR are current examples being
  worked)
  - Reliable fallback to "CXL unaware" behavior when region assembly
    fails, should be address by [1]. A temporary workaround is to
    disable the cxl_acpi driver so that hmem_register_device() skips CXL
    range deferral.
- Inability to delete regions that were created by the BIOS. Should also
  be addressed by [1].