Re: [PATCH v4 3/3] cxl/pci: Add sysfs attribute for CXL 1.1 device link status

[Lukas Wunner <lukas@wunner.de>]

On Tue, Apr 09, 2024 at 10:05:40AM -0500, Bjorn Helgaas wrote:
> On Tue, Apr 09, 2024 at 04:35:28PM +0900, Kobayashi,Daisuke wrote:
> > Add sysfs attribute for CXL 1.1 device link status to the cxl pci device.
> > 
> > In CXL1.1, the link status of the device is included in the RCRB mapped to
> > the memory mapped register area. Critically, that arrangement makes the link
> > status and control registers invisible to existing PCI user tooling.
> 
> Idle thought: PCIe does define RCRB, even pre-CXL.  Maybe the PCI core
> should be enhanced to comprehend RCRB directly?

The way CXL 1.1 (ab)uses the RCRB differs from what the PCIe Base Spec
envisions:

Per PCIe r6.2 sec 7.2.3, the RCRB contains additional Extended Capabilities
of a Root Port -- in addition to those in the Root Port's Config Space.
What we could do in the PCI core to support this is to amend our helpers
which search for Extended Capabilities to also search for them in the RCRB.

In fact, two years ago I cooked up a patch which does exactly that:

https://github.com/l1k/linux/commit/3eb94f042527

And I cooked up another patch to fetch the RCRB's address from the CXL
Early Discovery ACPI table:

https://github.com/l1k/linux/commit/d9d3cf45cf8c

The reason I never submitted the patches?  I realized after the fact that
CXL 1.1 uses the RCRB in a completely different way:

Per CXL r3.0 sec 8.2.1, RCH Downstream and RCD Upstream Ports do not
actually possess a Config Space.  Instead, they possess *only* an RCRB.
And that RCRB contains a Type 1 Configuration Space Header.

But because the PCIe Base Spec prescribes that there has to be an
Extended Capability at offset 0 of the RCRB, the CXL spec puts a
Null Extended Capability at offset 0 so that the Type 1 Config Space
Header is skipped.

However this means that the first dword of the Type 1 Config Space
Header does not contain a Vendor ID and Device ID.

So what we could do is create a fake pci_dev for each RCH Downstream and
RCD Upstream Port plus a specially crafted struct pci_ops whose ->read()
and write() callbacks access the RCRB.  But how do we know which Vendor
and Device ID to return from a ->read()?  There is none in the RCRB!

The CXL Consortium seems to have realized the mess they made with
CXL 1.1 and from CXL 2.0 onwards everything is now a proper PCI device.
I talked to Dan about my findings and his decision was basically to
not enable any of that legacy CXL 1.1 RCRB functionality in the kernel.

Thanks,

Lukas