Re: [PATCH v18 04/20] cxl: allow Type2 drivers to map cxl component regs

[PJ Waskiewicz <ppwaskie@kernel.org>]

On Thu, 2025-10-02 at 10:36 +0100, Alejandro Lucero Palau wrote:
> 
> 
> > > +	if (!cxl->cxlds.reg_map.component_map.ras.valid)
> > > +		return dev_err_probe(&pci_dev->dev, -ENODEV,
> > > +				     "Expected RAS component
> > > register not found\n");
> > > +
> > > +	rc = cxl_map_component_regs(&cxl->cxlds.reg_map,
> > > +				    &cxl->cxlds.regs.component,
> > > +				    BIT(CXL_CM_CAP_CAP_ID_RAS));
> > > +	if (rc) {
> > > +		dev_err(&pci_dev->dev, "Failed to map RAS
> > > capability.\n");
> > > +		return rc;
> > > +	}
> > I've finally made some serious headway integrating v17 into my
> > environment to better comment on this flow.
> > 
> > I'm running into what I'm seeing as a fundamental issue of resource
> > ownership between a device driver, and the CXL driver core.  I'm
> > having
> > a hard time trying to resolve this.
> > 
> > If I do the above and call cxl_map_component_regs() with a valid
> > CAP_ID
> > (RAS, HDM, etc.), that eventually calls devm_cxl_iomap_block() from
> > inside the CXL core drivers.  That calls devm_request_mem_region(),
> > and
> > this is where things get interesting.
> > 
> > If my device happens to land the CXL component registers inside of
> > a
> > BAR that has other items needed by my Type 2 device's driver, then
> > we
> > have a conflict.  My driver and the CXL core drivers cannot hold
> > the
> > same regions mapped.  i.e. I can't call pci_request_region() on my
> > BAR,
> > and then call the above.  One loses, and then we all lose.
> > 
> > Curious if you have any ideas how we can improve this?
> 
> 
> I ran into this issue early in my development when working with 
> emulation, and I had to share the mapping somehow.
> 
> 
> It went away with newer more real emulation so I had to no worry
> about 
> it anymore. But yes, the problem does exist. I can not find my old
> code 
> but I guess the solution is to pass a pointer to the already mapped 
> region implying the mapping is not required. It requires changes to
> the 
> cxl core but it should not be too much work.

This is a real issue in the API that has implications on hardware
design for people building endpoint devices.

Previously, the CXL core owned the whole thing.  Here, a "normal"
driver that would call something like pci_request_region() on the BAR
now can't do that.  Or, like you said, the API to the kernel would need
to have some way of taking an existing mapping.  But I don't think
that's what we want to do here.

The flow I'm seeing is the driver will start, create the memdev state,
and begin hooking everything up.  Once the memdev is "added," then the
endpoint will start its probe() on the new device.  This goes into the
outstanding issue of the EPROBE_DEFER problems with the endpoint device
not ready when the type 2 driver may try poking at it.

In order for that probe() to succeed for the endpoint, it needs to map
and own that memory region that the CXL regs belong to.  I don't see
any clean path that doesn't involve separating these memory regions
apart.

So I see either:

- An endpoint device's hardware design will need to put the CXL regs
onto a BAR that the endpoint's driver won't need to try and map/own
(big ask of HW vendors)

OR:

- An endpoint device lands the CXL regs on either side of an existing
BAR (front or back) and the endpoint's driver manually maps the portion
of the BAR minus the CXL regs.  And then that should "just work" if you
manually handle what pci_request_region() would do, just at a more
granular level.

If there's a third option I'm not seeing, I'm all ears!

-PJ