Re: [PATCH v3 0/5] dax/hmem, cxl: Coordinate Soft Reserved handling with CXL

["Koralahalli Channabasappa, Smita" <skoralah@amd.com>]

Hi Tomasz,

On 11/3/2025 3:18 AM, Tomasz Wolski wrote:
> Hi Alison and Smita,
> 
> I’ve been following your patch proposal and testing it on a few QEMU setups
> 
>> Will it work to search directly for the region above by using params
>> IORESOURCE_MEM, IORES_DESC_NONE. This way we only get region conflicts,
>> no empty windows to examine. I think that might replace cxl_region_exists()
>> work below.
> 
> I see expected 'dropping CXL range' message (case when region covers full CXL window)
> 
> [   31.783945] hmem_platform hmem_platform.0: deferring range to CXL: [mem 0xa90000000-0xb8fffffff flags 0x80000200]
> [   31.784609] deferring range to CXL: [mem 0xa90000000-0xb8fffffff flags 0x80000200]
> [   31.790588] hmem_platform hmem_platform.0: dropping CXL range: [mem 0xa90000000-0xb8fffffff flags 0x80000200]
> [   31.791102] dropping CXL range: [mem 0xa90000000-0xb8fffffff flags 0x80000200]
> 
> a90000000-b8fffffff : CXL Window 0
>    a90000000-b8fffffff : region0
>      a90000000-b8fffffff : dax0.0
>        a90000000-b8fffffff : System RAM (kmem)
> 
> [   31.384899] hmem_platform hmem_platform.0: deferring range to CXL: [mem 0xa90000000-0xc8fffffff flags 0x80000200]
> [   31.385586] deferring range to CXL: [mem 0xa90000000-0xc8fffffff flags 0x80000200]
> [   31.391107] hmem_platform hmem_platform.0: dropping CXL range: [mem 0xa90000000-0xc8fffffff flags 0x80000200]
> [   31.391676] dropping CXL range: [mem 0xa90000000-0xc8fffffff flags 0x80000200]
> 
> a90000000-c8fffffff : CXL Window 0
>    a90000000-b8fffffff : region0
>      a90000000-b8fffffff : dax0.0
>        a90000000-b8fffffff : System RAM (kmem)
>    b90000000-c8fffffff : region1
>      b90000000-c8fffffff : dax1.0
>        b90000000-c8fffffff : System RAM (kmem)
> 	
> a90000000-b8fffffff : CXL Window 0
>    a90000000-b8fffffff : region0
>      a90000000-b8fffffff : dax0.0
>        a90000000-b8fffffff : System RAM (kmem)
> b90000000-c8fffffff : CXL Window 1
>    b90000000-c8fffffff : region1
>      b90000000-c8fffffff : dax1.0
>        b90000000-c8fffffff : System RAM (kmem)
> 
> However, when testing version with cxl_region_exists() I didn't see expected 'registering CXL range' message
> when the CXL region does not fully occupy CXL window - please see below.
> I should mention that I’m still getting familiar with CXL internals, so maybe I might be missing some context :)
> 
> a90000000-bcfffffff : CXL Window 0
>    a90000000-b8fffffff : region0
>      a90000000-b8fffffff : dax0.0
>        a90000000-b8fffffff : System RAM (kmem)
> 
> [   30.434385] hmem_platform hmem_platform.0: deferring range to CXL: [mem 0xa90000000-0xbcfffffff flags 0x80000200]
> [   30.435116] deferring range to CXL: [mem 0xa90000000-0xbcfffffff flags 0x80000200]
> [   30.436530] hmem_platform hmem_platform.0: dropping CXL range: [mem 0xa90000000-0xbcfffffff flags 0x80000200]
> [   30.437070] hmem_platform hmem_platform.0: dropping CXL range: [mem 0xa90000000-0xbcfffffff flags 0x80000200]
> [   30.437599] dropping CXL range: [mem 0xa90000000-0xbcfffffff flags 0x80000200]

Thanks for testing and sharing the logs.

After off-list discussion with Alison and Dan (please jump in if I’m 
misrepresenting anything)

Ownership is determined by CXL regions, not window sizing. A CXL Window 
may be larger or smaller than the Soft Reserved (SR) span and that 
should not affect the decision.

Key thing to check is: Do the CXL regions fully and contiguously cover 
the entire Soft Reserved range?

Yes - CXL owns SR (“dropping CXL range”).

No - CXL must give up SR (“registering CXL range”). More on giving up SR 
below.

The previous child->start <= start && child->end <= end check needs to 
be replaced with a full coverage test:

1. Decide ownership based on region coverage: We check whether all CXL 
regions together fully and contiguously cover the "given" SR range.
If fully covered - CXL owns it.
If not fully covered - CXL must give up and the SR is owned by HMEM.

2. If CXL must give up - Remove the CXL regions that overlap SR before 
registering the SR via hmem_register_device().

3. Ensure dax_kmem never onlines memory until after this decision. 
dax_kmem must always probe after dax_hmem decides ownership.

Some of the valid configs (CXL owns: drop CXL range)

1.3ff0d0000000-3ff10fffffff : SR
     3ff0d0000000-3ff10fffffff : Window 1
         3ff0d0000000-3ff0dfffffff : region1
         3ff0e0000000-3ff0efffffff : region2
          3ff0f0000000-3ff0ffffffff : region3
          3ff100000000-3ff10fffffff : region4

2. 3ff0d0000000-3ff10fffffff : Window 1
      3ff0d0000000-3ff0dfffffff : SR
         3ff0d0000000-3ff0dfffffff : region1
      3ff0e0000000-3ff0efffffff : SR
         3ff0e0000000-3ff0efffffff : region2
      3ff0f0000000-3ff0ffffffff : SR
          3ff0f0000000-3ff0ffffffff : region3
      3ff100000000-3ff10fffffff : SR
          3ff100000000-3ff10fffffff : region4

3. 3ff0d0000000-3ff20fffffff : Window 1
       3ff0d0000000-3ff10fffffff : SR
         3ff0d0000000-3ff0dfffffff : region1
         3ff0e0000000-3ff0efffffff : region2
          3ff0f0000000-3ff0ffffffff : region3
          3ff100000000-3ff10fffffff : region4

4. 3ff0d0000000-3ff10fffffff : SR
     3ff0d0000000-3ff10fffffff : Window 1
         3ff0d0000000-3ff10fffffff : region1

Invalid configs (HMEM owns: registering CXL range)

1. 3ff0d0000000-3ff20fffffff : SR
     3ff0d0000000-3ff20fffffff : Window 1
         3ff0d0000000-3ff10fffffff : region1

2. 3ff0d0000000-3ff20fffffff : SR
     3ff0d0000000-3ff10fffffff : Window 1
         3ff0d0000000-3ff0dfffffff : region1
         3ff0e0000000-3ff0efffffff : region2
          3ff0f0000000-3ff0ffffffff : region3
          3ff100000000-3ff10fffffff : region4

3. region2 assembly failed or incorrect BIOS config
3ff0d0000000-3ff10fffffff : SR
     3ff0d0000000-3ff10fffffff : Window 1
         3ff0d0000000-3ff0dfffffff : region1
          3ff0f0000000-3ff0ffffffff : region3
          3ff100000000-3ff10fffffff : region4

I will work on incorporating the 3 steps mentioned above.

Thanks
Smita

> 
> Thanks,
> Tomasz