Question on deferring dax registration to cxl module for CXL_REGION

Question on deferring dax registration to cxl module for CXL_REGION

[Hongjian Fan]

Hi CXL experts,


I have observed the following behavior on iomem_resourece when CONFIG_CXL_REGION is enabled in the kernel.

CXL windows are inserted into iomem_resourece based on CEDT CFMWS. If there is only one CXL device attached to the host, the CXL window matches the soft reserved memory range, and the CXL window is inserted as the child of the PCI mem and the parent of the soft reserve. But if there are multiple CXL windows, each of the CXL window is part of the soft reserved memory range, the CXL window is inserted as the child of the soft reserved memory.

Function dax_hmem_platform_probe defers the dax region registration for the CXL window to cxl module.
However, two issues seem to occur:
        1) If the CXL window is not the direct child of the iomem_resourece, dax_hmem_platform_probe will not be able to detect and defer it. This means that if CFMWS contains multiple CXL windows, no deferral would happen.
        2) If a CXL1.1 device is behind the CXL window, and the dax region registration is deferred. The dax region will not be created because CXL1.1 device doesn't have the HDM decoder and other features needed by the CXL module to create the dax region.

DAX ( and hmem ) module is not visible to the CXL device's features behind a CXL window, so it is impossible to defer only the CXL window for CXL2.0 devices.

If I want to make dax region show up when a single CXL1.1 device is attached, I can see two potential approaches:
        1) Do not defer the CXL window in dax_hmem_platform_probe.
                Can we simply not defer? Current code will not defer if multiple CXL windows presents. Is any issue observed when multiple CXL devices are attached?
        2) Defer all CXL windows, and let cxl module create the dax region for CXL1.1 device.
                But where should this creation be? It would be a long path to handle all the unvailable features from function cxl_pci_probe to reach function devm_cxl_add_dax_region.

Please provide your comments.



Below is the /proc/iomem output from my hardware:

1) When there is a single CXL2.0 device on the host, the CXL window is inserted in PCI mem and the soft reserved region is a child of the CXL window:

        6080000000-707fffffff : CXL Window 0
          6080000000-707fffffff : region0
                6080000000-707fffffff : Soft Reserved
                  6080000000-707fffffff : dax0.0
                        6080000000-707fffffff : System RAM (kmem)

        A cxl region is inserted under the CXL window by function discover_region and the dax region is registered by cxl_dax_region_probe

2) When there is a single CXL1.1 device on the host, it is similar but neither cxl region nor dax is created:

        6080000000-707fffffff : CXL Window 0
          6080000000-707fffffff : Soft Reserved

        HDM decoder and other CXL2.0 features are missing from the CXL1.1 device so the CXL driver will not create related CXL structures. Because of the absence of the dax region, there is no numa node created for the cxl memory and the cxl memory is not usable in user space.

3) When there are multiple CXL devices, regardless CXL1.1 or 2.0, the CXL window is created under the soft reserved region:

        6080000000-807fffffff : Soft Reserved
          6080000000-707fffffff : CXL Window 0
                6080000000-707fffffff : region0
                  6080000000-707fffffff : dax2.0
                        6080000000-707fffffff : System RAM (kmem)
          7080000000-807fffffff : CXL Window 1
                7080000000-807fffffff : dax3.0
                  7080000000-807fffffff : System RAM (kmem)

        Both dax regions are registered by dax_hmem_platform_probe. The cxl region is created under CXL Window for the CXL2.0 devices.



Thanks,
Hongjian Fan

Seagate Internal

Re: Question on deferring dax registration to cxl module for CXL_REGION

[Alison Schofield]

On Thu, Jan 11, 2024 at 09:03:48PM +0000, Hongjian Fan wrote:
> Hi CXL experts,
> 
> 
> I have observed the following behavior on iomem_resourece when CONFIG_CXL_REGION is enabled in the kernel.
> 
> CXL windows are inserted into iomem_resourece based on CEDT CFMWS. If there is only one CXL device attached to the host, the CXL window matches the soft reserved memory range, and the CXL window is inserted as the child of the PCI mem and the parent of the soft reserve. But if there are multiple CXL windows, each of the CXL window is part of the soft reserved memory range, the CXL window is inserted as the child of the soft reserved memory.
> 
> Function dax_hmem_platform_probe defers the dax region registration for the CXL window to cxl module.
> However, two issues seem to occur:
>         1) If the CXL window is not the direct child of the iomem_resourece, dax_hmem_platform_probe will not be able to detect and defer it. This means that if CFMWS contains multiple CXL windows, no deferral would happen.
>         2) If a CXL1.1 device is behind the CXL window, and the dax region registration is deferred. The dax region will not be created because CXL1.1 device doesn't have the HDM decoder and other features needed by the CXL module to create the dax region.
> 
> DAX ( and hmem ) module is not visible to the CXL device's features behind a CXL window, so it is impossible to defer only the CXL window for CXL2.0 devices.
> 
> If I want to make dax region show up when a single CXL1.1 device is attached, I can see two potential approaches:
>         1) Do not defer the CXL window in dax_hmem_platform_probe.
>                 Can we simply not defer? Current code will not defer if multiple CXL windows presents. Is any issue observed when multiple CXL devices are attached?
>         2) Defer all CXL windows, and let cxl module create the dax region for CXL1.1 device.
>                 But where should this creation be? It would be a long path to handle all the unvailable features from function cxl_pci_probe to reach function devm_cxl_add_dax_region.
> 
> Please provide your comments.
> 
> 

Hi Hondjian Fan,

This is familiar. In Aug '23 I stopped work on a patchset [1] aimed at
improving the soft reserved resource handling. From that cover letter:

1) Soft reserved resources were observed as sometimes being the parent
and sometimes being the child of a region resource. Patch 1 clears up
that inconsistency.

2) Soft reserved resources were also observed as stranded after region 
teardown, making the address space the region released unavailable for
reallocation. Patch 2 implements soft reserved resource removal.

By v3 of the set, we were rethinking the approach as Patch 2's juggling
of soft reserved spaces seemed silly and error prone. Also, the folks who
were hitting the soft reserved issue during hotplug were able to use CFMWS
address space not in the Soft Reserved range as a work-around.

Dan offered a couple of new approaches since then:
(I hope I'm not misquoting)

1) Insert cxl intersecting soft reserved resources into a separate
(non iomem_resource) resource tree, when / if any CXL region assembly
fails walk that side tree and move them all over to iomem_resource.

2) Given that it is already the case that the device-dax core waits for
cxl_acpi to mark ranges as IORES_DESC_CXL, and that we do not expect that
to fail. It means that cxl_acpi can then turn around and ask the device-dax
core to cache and delete the soft reserve address ranges. Then if CXL notices
a region assembly failure it can signal device-dax to release that cached
range as a new CXL disconnected DAX region.

3) CXL acpi walks the resource range knowing that at the beginning of time
Soft Reserved ranges are unparented making them easier to delete and
register them as "just in case" recovery ranges to device-dax.

Can you comment on whether any/all of these suggestions seems to address
what you are seeing?

Others thoughts on the approach this might take next.

Thanks,
Alison


[1] https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/






> 
> Below is the /proc/iomem output from my hardware:
> 
> 1) When there is a single CXL2.0 device on the host, the CXL window is inserted in PCI mem and the soft reserved region is a child of the CXL window:
> 
>         6080000000-707fffffff : CXL Window 0
>           6080000000-707fffffff : region0
>                 6080000000-707fffffff : Soft Reserved
>                   6080000000-707fffffff : dax0.0
>                         6080000000-707fffffff : System RAM (kmem)
> 
>         A cxl region is inserted under the CXL window by function discover_region and the dax region is registered by cxl_dax_region_probe
> 
> 2) When there is a single CXL1.1 device on the host, it is similar but neither cxl region nor dax is created:
> 
>         6080000000-707fffffff : CXL Window 0
>           6080000000-707fffffff : Soft Reserved
> 
>         HDM decoder and other CXL2.0 features are missing from the CXL1.1 device so the CXL driver will not create related CXL structures. Because of the absence of the dax region, there is no numa node created for the cxl memory and the cxl memory is not usable in user space.
> 
> 3) When there are multiple CXL devices, regardless CXL1.1 or 2.0, the CXL window is created under the soft reserved region:
> 
>         6080000000-807fffffff : Soft Reserved
>           6080000000-707fffffff : CXL Window 0
>                 6080000000-707fffffff : region0
>                   6080000000-707fffffff : dax2.0
>                         6080000000-707fffffff : System RAM (kmem)
>           7080000000-807fffffff : CXL Window 1
>                 7080000000-807fffffff : dax3.0
>                   7080000000-807fffffff : System RAM (kmem)
> 
>         Both dax regions are registered by dax_hmem_platform_probe. The cxl region is created under CXL Window for the CXL2.0 devices.
> 
> 
> 
> Thanks,
> Hongjian Fan
> 
> Seagate Internal
> 

Re: Question on deferring dax registration to cxl module for CXL_REGION

[Hongjian Fan]

Hi Alison,

Thanks for providing the information and different approaches!

> 
> On Thu, Jan 11, 2024 at 09:03:48PM +0000, Hongjian Fan wrote:
> > Hi CXL experts,
> >
> >
> > I have observed the following behavior on iomem_resourece when CONFIG_CXL_REGION is enabled in the kernel.
> >
> > CXL windows are inserted into iomem_resourece based on CEDT CFMWS. If there is only one CXL device attached to the host, the CXL window matches the soft reserved memory range, and the CXL window is inserted as the child of the PCI mem and the parent of the soft reserve. But if there are multiple CXL windows, each of the CXL window is part of the soft reserved memory range, the CXL window is inserted as the child of the soft reserved memory.
> >
> > Function dax_hmem_platform_probe defers the dax region registration for the CXL window to cxl module.
> > However, two issues seem to occur:
> >         1) If the CXL window is not the direct child of the iomem_resourece, dax_hmem_platform_probe will not be able to detect and defer it. This means that if CFMWS contains multiple CXL windows, no deferral would happen.
> >         2) If a CXL1.1 device is behind the CXL window, and the dax region registration is deferred. The dax region will not be created because CXL1.1 device doesn't have the HDM decoder and other features needed by the CXL module to create the dax region.
> >
> > DAX ( and hmem ) module is not visible to the CXL device's features behind a CXL window, so it is impossible to defer only the CXL window for CXL2.0 devices.
> >
> > If I want to make dax region show up when a single CXL1.1 device is attached, I can see two potential approaches:
> >         1) Do not defer the CXL window in dax_hmem_platform_probe.
> >                 Can we simply not defer? Current code will not defer if multiple CXL windows presents. Is any issue observed when multiple CXL devices are attached?
> >         2) Defer all CXL windows, and let cxl module create the dax region for CXL1.1 device.
> >                 But where should this creation be? It would be a long path to handle all the unvailable features from function cxl_pci_probe to reach function devm_cxl_add_dax_region.
> >
> > Please provide your comments.
> >
> >
> 
> Hi Hondjian Fan,
> 
> This is familiar. In Aug '23 I stopped work on a patchset [1] aimed at
> improving the soft reserved resource handling. From that cover letter:
> 
> 1) Soft reserved resources were observed as sometimes being the parent
> and sometimes being the child of a region resource. Patch 1 clears up
> that inconsistency.
> 
> 2) Soft reserved resources were also observed as stranded after region
> teardown, making the address space the region released unavailable for
> reallocation. Patch 2 implements soft reserved resource removal.
> 
> By v3 of the set, we were rethinking the approach as Patch 2's juggling
> of soft reserved spaces seemed silly and error prone. Also, the folks who
> were hitting the soft reserved issue during hotplug were able to use CFMWS
> address space not in the Soft Reserved range as a work-around.
> 
> Dan offered a couple of new approaches since then:
> (I hope I'm not misquoting)
> 
> 1) Insert cxl intersecting soft reserved resources into a separate
> (non iomem_resource) resource tree, when / if any CXL region assembly
> fails walk that side tree and move them all over to iomem_resource.
> 
> 2) Given that it is already the case that the device-dax core waits for
> cxl_acpi to mark ranges as IORES_DESC_CXL, and that we do not expect that
> to fail. It means that cxl_acpi can then turn around and ask the device-dax
> core to cache and delete the soft reserve address ranges. Then if CXL notices
> a region assembly failure it can signal device-dax to release that cached
> range as a new CXL disconnected DAX region.
> 
> 3) CXL acpi walks the resource range knowing that at the beginning of time
> Soft Reserved ranges are unparented making them easier to delete and
> register them as "just in case" recovery ranges to device-dax.
> 
> Can you comment on whether any/all of these suggestions seems to address
> what you are seeing?

So the goal is to have the flow of  ( defer -> create region -> recovery ). Let cxl module to signal the recover might be tricky because there are so many places the module could exit before having the chance to create a region. How about combing approach 1 and 3? Creating a separate resource tree for cxl intersecting soft reserved resources and use it for recovery. 

> 
> Others thoughts on the approach this might take next.
> 
> Thanks,
> Alison
> 
> 
> [1]  https://secure-web.cisco.com/1_-vlaqCzY403VxKY3uh275hNxVtkmKw-ivst_y1c2018jAb-ohPZQ9l9BjAdn2Gja8wR012IpQsvtfpqB5pBZf3Ni4Bq5QKvZ9eRhq3selFEDjOZLCLZ4KoX1AvPkGx6JKWiXd05MYRRrlE4INdiE4Ynwa-wrk7DgZCJFtfP2Q1GYlib57E_26VnA-Bc_wUtb7YHaz9tVexsNatKD-Hm5DofMPUO0K8aPf_1VkWIOO5DSBrDMoO2EyZVa-P35U1h_3DqxwnyHtnrJbU2jsexgvPbax750fTP1cifhb1-WfKvtjabUpo6tk659S0PeihX7_ikum30-TaJscvo9yZpJTr8-xAVNRG1hhfhJRRGG8b5LjCm4KCkwiO5kw6wNXrkqmTnskhGd5k9MD95D-3Z7jGBJ1NjJRQz1OVdLXWUgZU/https%3A%2F%2Flore.kernel.org%2Flinux-cxl%2Fcover.1692638817.git.alison.schofield%40intel.com%2F


Thanks,
Hongjian Fan

Re: Question on deferring dax registration to cxl module for CXL_REGION

[Nathan Fontenot]

On 1/11/24 18:26, Alison Schofield wrote:
> On Thu, Jan 11, 2024 at 09:03:48PM +0000, Hongjian Fan wrote:
>> Hi CXL experts,
>>
>>
>> I have observed the following behavior on iomem_resourece when CONFIG_CXL_REGION is enabled in the kernel.
>>
>> CXL windows are inserted into iomem_resourece based on CEDT CFMWS. If there is only one CXL device attached to the host, the CXL window matches the soft reserved memory range, and the CXL window is inserted as the child of the PCI mem and the parent of the soft reserve. But if there are multiple CXL windows, each of the CXL window is part of the soft reserved memory range, the CXL window is inserted as the child of the soft reserved memory.
>>
>> Function dax_hmem_platform_probe defers the dax region registration for the CXL window to cxl module.
>> However, two issues seem to occur:
>>         1) If the CXL window is not the direct child of the iomem_resourece, dax_hmem_platform_probe will not be able to detect and defer it. This means that if CFMWS contains multiple CXL windows, no deferral would happen.
>>         2) If a CXL1.1 device is behind the CXL window, and the dax region registration is deferred. The dax region will not be created because CXL1.1 device doesn't have the HDM decoder and other features needed by the CXL module to create the dax region.
>>
>> DAX ( and hmem ) module is not visible to the CXL device's features behind a CXL window, so it is impossible to defer only the CXL window for CXL2.0 devices.
>>
>> If I want to make dax region show up when a single CXL1.1 device is attached, I can see two potential approaches:
>>         1) Do not defer the CXL window in dax_hmem_platform_probe.
>>                 Can we simply not defer? Current code will not defer if multiple CXL windows presents. Is any issue observed when multiple CXL devices are attached?
>>         2) Defer all CXL windows, and let cxl module create the dax region for CXL1.1 device.
>>                 But where should this creation be? It would be a long path to handle all the unvailable features from function cxl_pci_probe to reach function devm_cxl_add_dax_region.
>>
>> Please provide your comments.
>>
>>
> 
> Hi Hondjian Fan,
> 
> This is familiar. In Aug '23 I stopped work on a patchset [1] aimed at
> improving the soft reserved resource handling. From that cover letter:
> 
> 1) Soft reserved resources were observed as sometimes being the parent
> and sometimes being the child of a region resource. Patch 1 clears up
> that inconsistency.
> 
> 2) Soft reserved resources were also observed as stranded after region 
> teardown, making the address space the region released unavailable for
> reallocation. Patch 2 implements soft reserved resource removal.
> 
> By v3 of the set, we were rethinking the approach as Patch 2's juggling
> of soft reserved spaces seemed silly and error prone. Also, the folks who
> were hitting the soft reserved issue during hotplug were able to use CFMWS
> address space not in the Soft Reserved range as a work-around.
> 
> Dan offered a couple of new approaches since then:
> (I hope I'm not misquoting)
> 
> 1) Insert cxl intersecting soft reserved resources into a separate
> (non iomem_resource) resource tree, when / if any CXL region assembly
> fails walk that side tree and move them all over to iomem_resource.
> 
> 2) Given that it is already the case that the device-dax core waits for
> cxl_acpi to mark ranges as IORES_DESC_CXL, and that we do not expect that
> to fail. It means that cxl_acpi can then turn around and ask the device-dax
> core to cache and delete the soft reserve address ranges. Then if CXL notices
> a region assembly failure it can signal device-dax to release that cached
> range as a new CXL disconnected DAX region.
> 
> 3) CXL acpi walks the resource range knowing that at the beginning of time
> Soft Reserved ranges are unparented making them easier to delete and
> register them as "just in case" recovery ranges to device-dax.
> 
> Can you comment on whether any/all of these suggestions seems to address
> what you are seeing?
> 
> Others thoughts on the approach this might take next.

Alison, and others,

Can you provide some additional details on this new approach. I'm trying to wrap
my head around management of the the separate cxl resource tree and what resources
would be put in it.

I've also wondered if you were looking to use this to manage cxl resources outside
of the iomem resource tree or is it just for management of 'soft reserve' resources
under the CFMWS.

thanks,
-Nathan


> 
> Thanks,
> Alison
> 
> 
> [1] https://lore.kernel.org/linux-cxl/cover.1692638817.git.alison.schofield@intel.com/
> 
> 
> 
> 
> 
> 
>>
>> Below is the /proc/iomem output from my hardware:
>>
>> 1) When there is a single CXL2.0 device on the host, the CXL window is inserted in PCI mem and the soft reserved region is a child of the CXL window:
>>
>>         6080000000-707fffffff : CXL Window 0
>>           6080000000-707fffffff : region0
>>                 6080000000-707fffffff : Soft Reserved
>>                   6080000000-707fffffff : dax0.0
>>                         6080000000-707fffffff : System RAM (kmem)
>>
>>         A cxl region is inserted under the CXL window by function discover_region and the dax region is registered by cxl_dax_region_probe
>>
>> 2) When there is a single CXL1.1 device on the host, it is similar but neither cxl region nor dax is created:
>>
>>         6080000000-707fffffff : CXL Window 0
>>           6080000000-707fffffff : Soft Reserved
>>
>>         HDM decoder and other CXL2.0 features are missing from the CXL1.1 device so the CXL driver will not create related CXL structures. Because of the absence of the dax region, there is no numa node created for the cxl memory and the cxl memory is not usable in user space.
>>
>> 3) When there are multiple CXL devices, regardless CXL1.1 or 2.0, the CXL window is created under the soft reserved region:
>>
>>         6080000000-807fffffff : Soft Reserved
>>           6080000000-707fffffff : CXL Window 0
>>                 6080000000-707fffffff : region0
>>                   6080000000-707fffffff : dax2.0
>>                         6080000000-707fffffff : System RAM (kmem)
>>           7080000000-807fffffff : CXL Window 1
>>                 7080000000-807fffffff : dax3.0
>>                   7080000000-807fffffff : System RAM (kmem)
>>
>>         Both dax regions are registered by dax_hmem_platform_probe. The cxl region is created under CXL Window for the CXL2.0 devices.
>>
>>
>>
>> Thanks,
>> Hongjian Fan
>>
>> Seagate Internal
>>

Re: Question on deferring dax registration to cxl module for CXL_REGION

[Dan Williams]

Nathan Fontenot wrote:
[..] 
> Alison, and others,
> 
> Can you provide some additional details on this new approach. I'm trying to wrap
> my head around management of the the separate cxl resource tree and what resources
> would be put in it.
> 
> I've also wondered if you were looking to use this to manage cxl resources outside
> of the iomem resource tree or is it just for management of 'soft reserve' resources
> under the CFMWS.

Hey Nathan, here is my current thinking. The tl;dr is quite a bit of
spelunking in early init code is needed to get this fixed up properly.
If you have cycles to take this on, here is a roadmap:

The BIOS is responsible for building the CFMWS and optionally deciding
what memory should not be included in the OS general pool by default
(EFI_MEMORY_SP attribute). That attribute allows the OS the *option* to
use it in a dedicated fashion. That could either be because the memory
is too slow and needs to be set aside for applications that can tolerate
the latency, or memory that is too fast, i.e. so precious that only a
specific application should use it.

However, when the OS has no policy for that dedicated memory it should
find its way back into the general pool. So EFI_MEMORY_SP is a mechanism
that prevents immovable allocations landing in dedicated memory early in
boot, but with the understanding that most system-owners likely want to
proceed with all memory in the general pool unless and until it becomes
abundantly clear that a policy to do something different needs to be
deployed.

So, how does this affect CXL? Given there is only one EFI_MEMORY_SP bit
that is not CXL specific, it may be applied across one large address
range that includes CXL, HBM, PMEM, and/or even some DDR. For HBM and
DDR there is no driver subsystem like CXL that can decorate that address
range with a device like a cxl_region device. Those ranges need to go
straight to device-dax so that the dedicated memory-policy can be
deployed.

The problem is what do with the remaining Soft Reserve ranges that
intersect CFMWS that *might* later have a cxl_region established. Right
now a "Soft Reserve" entry in iomem_resource can collide with "CXL
Window" entries and impede the CXL subsystem's ability to manage the
resource tree.

The proposal is to delay the decision about installing a "Soft Reserve"
entry until after the CXL intersections have been determined.

NUMA init should run before e820__reserve_resources_late() which means
that e820__reserve_resources_late() should be able to discover where
a "Soft Reserve" entry intersects a CXL range, and trim that "Soft
Reserve" entry to only reference non-CXL ranges.

Sometime later the cxl_acpi driver loads, re-parses CFMWS and inserts
"CXL Window" resources into iomem resource. You might wonder, "could not
NUMA init do this resource registration?". The problem is that NUMA init
does not know about any of the resources deferred by
e820__reserve_resources() to e820__reserve_resources_late(), so it does
not know how to do the fixups that add_cxl_resource() (in cxl_acpi)
does.

After cxl_acpi is loaded Linux knows where all the CXL windows are, but
it does not know if there are cxl_region instances populated into those
windows. That process involves waiting for PCI probe to complete and the
CXL autoassembly process to run its course.

That could be achieved with something like cxl_acpi kicks off a work
item to do:

	wait_for_device_probe()
	/* trim soft reserve where cxl took over */
	for_each_deferred_cxl_soft_reserve_range()
		for_each_cxl_region()
			if (does_region_intersect_softreserve_range())
				trim_soft_reserve_range()

	/* create standalone Soft Reserve entries for the rest */
	for_each_deferred_cxl_soft_reserve_range()
		insert_soft_reserve_resource()

	/*
	 * CXL soft reserve with no cxl_region likely means something
	 * like Type-2 memory that may be invisible to cxl_pci
	 */
	notify_device_dax_of_late_discover_soft_reserve()

Hope this helps clarify what I think is needed to fix this up, patches
are welcome.