Re: [PATCH v5 3/7] cxl/acpi: Add background worker to coordinate with cxl_mem probe completion

["Zhijian Li (Fujitsu)" <lizhijian@fujitsu.com>]

All,


I have confirmed that in the !CXL_REGION configuration, the same environment may fail to fall back to hmem.(Your new patch cannot resolve this issue)

In my environment:
- There are two CXL memory devices corresponding to:
   ```
   5d0000000-6cffffff : CXL Window 0
   6d0000000-7cffffff : CXL Window 1
   ```
- E820 table contains a 'soft reserved' entry:
   ```
   [    0.000000] BIOS-e820: [mem 0x00000005d0000000-0x00000007cfffffff] soft reserved
   ```

However, since my ACPI SRAT doesn't describe the CXL memory devices (the point), `acpi/hmat.c` won't allocate memory targets for them. This prevents the call chain:
```c
hmat_register_target_devices() // for each SRAT-described target
   -> hmem_register_resource()
     -> insert entry into "HMEM devices" resource
```

Therefore, for successful fallback to hmem in this environment: `dax_hmem.ko` and `kmem.ko` must request resources BEFORE `cxl_acpi.ko` inserts 'CXL Window X'

However the kernel cannot guarantee this initialization order.

When cxl_acpi runs before dax_kmem/kmem:
```
(built-in)                 CXL_REGION=n
driver/dax/hmem/device.c  cxl_acpi.ko      dax_hmem.ko               kmem.ko

(1) Add entry '15d0000000-7cfffffff'
                                          (2) Traverse "HMEM devices"
                                              Insert to iomem:
                                              5d0000000-7cffffff : Soft Reserved

                      (3) Insert CXL Window 0/1
                          /proc/iomem shows:
                          5d0000000-7cffffff : Soft Reserved
                            5d0000000-6cffffff : CXL Window 0
                            6d0000000-7cffffff : CXL Window 1

                                         (4) Create dax device
                                                                 (5) request_mem_region() fails
                                                                   for 5d0000000-7cffffff
                                                                   Reason: Children of 'Soft Reserved'
                                                                   (CXL Windows 0/1) don't cover full range
```

---------------------
In my another environment where ACPI SRAT has separate entries per CXL device:
1. `acpi/hmat.c` inserts two entries into "HMEM devices":
    - 5d0000000-6cffffff
    - 6d0000000-7cffffff

2. Regardless of module order, dax/kmem requests per-device resources, resulting in:
    ```
    5d0000000-7cffffff : Soft Reserved
        5d0000000-6cffffff : CXL Window 0
            5d0000000-6cffffff : dax0.0
                5d0000000-6cffffff : System RAM (kmem)
        6d0000000-7cffffff : CXL Window 1
            6d0000000-7cffffff : dax1.0
                6d0000000-7cffffff : System RAM (kmem)
    ```

Thanks,
Zhijian


On 25/08/2025 15:50, Li Zhijian wrote:
> 
> 
> On 22/08/2025 11:56, Koralahalli Channabasappa, Smita wrote:
>>>
>>>>
>>>>>       ```
>>>>>
>>>>> 3. When CXL_REGION is disabled, there is a failure to fallback to dax_hmem, in which case only CXL Window X is visible.
>>>>
>>>> Haven't tested !CXL_REGION yet.
>>
>> When CXL_REGION is disabled, DEV_DAX_CXL will also be disabled. So dax_hmem should handle it. 
> 
> Yes, falling back to dax_hmem/kmem is the result we expect.
> I haven't figured out the root cause of the issue yet, but I can tell you that in my QEMU environment,
> there is currently a certain probability that it cannot fall back to dax_hmem/kmem.
> 
> Upon its failure, I observed the following warnings and errors (with my local fixup kernel).
> [   12.203254] kmem dax0.0: mapping0: 0x5d0000000-0x7cfffffff could not reserve region
> [   12.203437] kmem dax0.0: probe with driver kmem failed with error -16
> 
> 
> 
>> I was able to fallback to dax_hmem. But let me know if I'm missing something.
>>
>> config DEV_DAX_CXL
>>          tristate "CXL DAX: direct access to CXL RAM regions"
>>          depends on CXL_BUS && CXL_REGION && DEV_DAX
>> ..
>>
>>>>
>>>>>       On failure:
>>>>>       ```
>>>>>       100000000-27ffffff : System RAM
>>>>>       5c0001128-5c00011b7 : port1
>>>>>       5c0011128-5c00111b7 : port2
>>>>>       5d0000000-6cffffff : CXL Window 0
>>>>>       6d0000000-7cffffff : CXL Window 1
>>>>>       7000000000-700000ffff : PCI Bus 0000:0c
>>>>>         7000000000-700000ffff : 0000:0c:00.0
>>>>>           7000001080-70000010d7 : mem1
>>>>>       ```
>>>>>
>>>>>       On success:
>>>>>       ```
>>>>>       5d0000000-7cffffff : dax0.0
>>>>>         5d0000000-7cffffff : System RAM (kmem)
>>>>>           5d0000000-6cffffff : CXL Window 0
>>>>>           6d0000000-7cffffff : CXL Window 1
>>>>>       ```