Re: [PATCH v21 15/23] sfc: get endpoint decoder

[Alejandro Lucero Palau <alucerop@amd.com>]

On 11/26/25 01:27, PJ Waskiewicz wrote:
> Hi Alejandro,
>
> On Wed, 2025-11-19 at 19:22 +0000, alejandro.lucero-palau@amd.com
> wrote:
>> From: Alejandro Lucero <alucerop@amd.com>
>>
>> Use cxl api for getting DPA (Device Physical Address) to use through
>> an
>> endpoint decoder.
>>
>> Signed-off-by: Alejandro Lucero <alucerop@amd.com>
>> Reviewed-by: Martin Habets <habetsm.xilinx@gmail.com>
>> Acked-by: Edward Cree <ecree.xilinx@gmail.com>
>> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Reviewed-by: Ben Cheatham <benjamin.cheatham@amd.com>
>> Reviewed-by: Dave Jiang <dave.jiang@intel.com>
>> ---
>>   drivers/net/ethernet/sfc/efx_cxl.c | 12 +++++++++++-
>>   1 file changed, 11 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/net/ethernet/sfc/efx_cxl.c
>> b/drivers/net/ethernet/sfc/efx_cxl.c
>> index d7c34c978434..1a50bb2c0913 100644
>> --- a/drivers/net/ethernet/sfc/efx_cxl.c
>> +++ b/drivers/net/ethernet/sfc/efx_cxl.c
>> @@ -108,6 +108,14 @@ int efx_cxl_init(struct efx_probe_data
>> *probe_data)
>>   		return -ENOSPC;
>>   	}
>>   
>> +	cxl->cxled = cxl_request_dpa(cxl->cxlmd, CXL_PARTMODE_RAM,
>> +				     EFX_CTPIO_BUFFER_SIZE);
> I've been really struggling to get this flow working in my environment.
> The above function call has a call-chain like this:
>
> - cxl_request_dpa()
>    => cxl_dpa_alloc()
>       => __cxl_dpa_alloc()
>          => __cxl_dpa_reserve()
>             => __request_region()
>
> That last call to __request_region() is not handling a Type2 device
> that has its mem region defined as EFI Special Purpose memory.
> Basically if the underlying hardware has the memory region marked that
> way, it's still getting mapped into the host's physical address map,
> but it's explicitly telling the OS to bugger off and not try to map it
> as system RAM, which is what we want. Since this is being used as an
> acceleration path, we don't want the OS to muck about with it.
>
> The issue here is now that I have to build CXL into the kernel itself
> to get around the circular dependency issue with depmod, I see this
> when my kernel boots and the device trains, but *before* my driver
> loads:
>
> # cat /proc/iomem
> [...snip...]
> c050000000-c08fffffff : CXL Window 0
>    c050000000-c08fffffff : Soft Reserved
>
> That right there is my device.  And it's being presented correctly that
> it's reserved so the OS doesn't mess with it.  However, that call to
> __request_region() fails with -EBUSY since it can't take ownership of
> that region since it's already owned by the core.
>
> I can't just skip over this flow for DPA init, so I'm at a bit of a
> loss how to proceed.  How is your device presenting the .mem region to
> the host?


Hi PJ,


My work is based on the device not using EFI_CONVENTIONAL_MEMORY + 
EFI_MEMORY_SP but EFI_RESERVED_TYPE. In the first case the kernel can 
try to use that memory and the BIOS goes through default initialization, 
the latter will avoid BIOS or kernel to mess with such a memory. Because 
there is no BIOS yet supporting this I had to remove DAX support from 
the kernel and deal (for testing) with some BIOS initialization we will 
not have in production.


For your case I thought this work 
https://lore.kernel.org/linux-cxl/20251120031925.87762-1-Smita.KoralahalliChannabasappa@amd.com/T/#me2bc0d25a2129993e68df444aae073addf886751 
was solving your problem but after looking at it now, I think that will 
only be useful for Type3 and the hotplug case. Maybe it is time to add 
Type2 handling there. I'll study that patchset with more detail and 
comment for solving your case.


FWIW, last year in Vienna I raised the concern of the kernel doing 
exactly what you are witnessing, and I proposed having a way for taking 
the device/memory from DAX but I was told unanimously that was not 
necessary and if the BIOS did the wrong thing, not fixing that in the 
kernel. In hindsight I would say this conflict was not well understood 
then (me included) with all the details, so maybe it is time to have 
this capacity, maybe from user space or maybe specific kernel param 
triggering the device passing from DAX.


>
> Cheers,
> -PJ