[PATCH v14 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[PATCH v14 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[Kobayashi,Daisuke]

CXL devices are extensions of PCIe. Therefore, from CXL2.0 onwards,
the link status can be output in the same way as traditional PCIe.
However, unlike devices from CXL2.0 onwards, CXL1.1 requires a
different method to obtain the link status from traditional PCIe.
This is because the link status of the CXL1.1 device is not mapped
in the configuration space (as per cxl3.0 specification 8.1).
Instead, the configuration space containing the link status is mapped
to the memory mapped register region (as per cxl3.0 specification 8.2,
Table 8-18). Therefore, the current lspci has a problem where it does
not display the link status of the CXL1.1 device. 
Solve these issues with sysfs attributes to export the status
registers hidden in the RCRB.

The procedure is as follows:
First, obtain the RCRB address within the cxl driver, then access 
the configuration space. Next, output the link status information from
the configuration space to sysfs. Ultimately, the expectation is that
this sysfs file will be consumed by PCI user tools to utilize link status
information.


Changes
v1[1] -> v2:
- Modified to perform rcrb access within the CXL driver.
- Added new attributes to the sysfs of the PCI device.
- Output the link status information to the sysfs of the PCI device.
- Retrieve information from sysfs as the source when displaying information in lspci.

v2[2] -> v3:
- Fix unnecessary initialization and wrong types (Bjohn).
- Create a helper function for getting a PCIe capability offset (Bjohn).
- Move platform-specific implementation to the lib directory in pciutils (Martin).

v3[3] -> v4:
- RCRB register values are read once and cached.
- Added a new attribute to the sysfs of the PCI device.
- Separate lspci implementation from this patch.

v4[4] -> v5:
- Use macros for bitwise operations
- Fix RCRB access to use cxl_memdev

v5[5] -> v6:
- Add and use masks for RCRB register values

v6[6] -> v7:
- Fix comments on white space inline

v7[7] -> v8:
- Change the cache value to offset
- Access memory map area in rcd_*_show() functions

v8[8] -> v9:
- Map the pcie cap in for all the time the driver is bound to the device.
- Add mapping the pcie cap in cxl_rcd_component_reg_phys().

v9[9] -> v10:
- Change a utility function for getting PCIe capability.
- Fix tab alignment issue, error handling, and apply suggestions from Jonathan.

v10[10] -> v11:
- Add functions to have one function do only one thing.
- Add a size parameter to utility function arguments and consolidated them into one.

v11[11] -> v12:
- Fix the error handling in cxl_pci_setup_regs().
- Fix and clean up some details.

v12[12] -> v13:
- Fix and clean up some details.

v13[13] -> v14:
- Fix and clean up some details.

[1]
https://lore.kernel.org/linux-cxl/20231220050738.178481-1-kobayashi.da-06@fujitsu.com/
[2]
https://lore.kernel.org/linux-cxl/20240227083313.87699-1-kobayashi.da-06@fujitsu.com/
[3]
https://lore.kernel.org/linux-cxl/20240312080559.14904-1-kobayashi.da-06@fujitsu.com/
[4]
https://lore.kernel.org/linux-cxl/20240409073528.13214-1-kobayashi.da-06@fujitsu.com/
[5]
https://lore.kernel.org/linux-cxl/20240412070715.16160-1-kobayashi.da-06@fujitsu.com/
[6]
https://lore.kernel.org/linux-cxl/20240424050102.26788-1-kobayashi.da-06@fujitsu.com/
[7]
https://lore.kernel.org/linux-cxl/20240510073710.98953-1-kobayashi.da-06@fujitsu.com/
[8]
https://lore.kernel.org/linux-cxl/20240606074814.5633-1-kobayashi.da-06@fujitsu.com/
[9]
https://lore.kernel.org/linux-cxl/20240610082222.22772-1-kobayashi.da-06@fujitsu.com/
[10]
https://lore.kernel.org/linux-cxl/20240611055254.61203-1-kobayashi.da-06@fujitsu.com/
[11]
https://lore.kernel.org/linux-cxl/20240612075940.92500-1-kobayashi.da-06@fujitsu.com/
[12]
https://lore.kernel.org/linux-cxl/20240614045611.58658-1-kobayashi.da-06@fujitsu.com/
[13]
https://lore.kernel.org/linux-cxl/20240617043702.62028-1-kobayashi.da-06@fujitsu.com/

Kobayashi,Daisuke (2):
  cxl/core/regs: Add rcd_pcie_cap initialization
  cxl/pci: Add sysfs attribute for CXL 1.1 device link status

 drivers/cxl/core/core.h |  6 +++
 drivers/cxl/core/regs.c | 61 ++++++++++++++++++++++++++++
 drivers/cxl/cxl.h       |  9 +++++
 drivers/cxl/pci.c       | 89 ++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 163 insertions(+), 2 deletions(-)

-- 
2.44.0

[PATCH v14 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Kobayashi,Daisuke]

Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1 
device link status information. By caching it, avoid the walking 
memory map area to find the offset when output the register value.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
---
 drivers/cxl/core/core.h |  6 ++++
 drivers/cxl/core/regs.c | 61 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h       |  9 ++++++
 drivers/cxl/pci.c       |  8 ++++--
 4 files changed, 82 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 3b64fb1b9ed0..69006bde7ad5 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device *dev,
 				    struct cxl_rcrb_info *ri,
 				    enum cxl_rcrb which);
 u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
+resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport);
+
+#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
+#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
+#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
+#define PCI_CAP_EXP_SIZEOF		0x3c
 
 extern struct rw_semaphore cxl_dpa_rwsem;
 extern struct rw_semaphore cxl_region_rwsem;
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 372786f80955..8a7cef1cf87d 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -505,6 +505,67 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)
 	return offset;
 }
 
+resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport)
+{
+	resource_size_t rcrb = dport->rcrb.base;
+	void __iomem *addr;
+	u32 cap_hdr;
+	u16 offset;
+
+	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
+		return CXL_RESOURCE_NONE;
+
+	addr = ioremap(rcrb, SZ_4K);
+	if (!addr) {
+		dev_err(dev, "Failed to map region %pr\n", addr);
+		release_mem_region(rcrb, SZ_4K);
+		return CXL_RESOURCE_NONE;
+	}
+
+	offset = FIELD_GET(PCI_RCRB_CAP_LIST_ID_MASK, readw(addr + PCI_CAPABILITY_LIST));
+	cap_hdr = readl(addr + offset);
+	while ((FIELD_GET(PCI_RCRB_CAP_HDR_ID_MASK, cap_hdr)) != PCI_CAP_ID_EXP) {
+		offset = FIELD_GET(PCI_RCRB_CAP_HDR_NEXT_MASK, cap_hdr);
+		if (offset == 0 || offset > SZ_4K) {
+			offset = 0;
+			break;
+		}
+		cap_hdr = readl(addr + offset);
+	}
+
+	iounmap(addr);
+	release_mem_region(rcrb, SZ_4K);
+	if (!offset)
+		return CXL_RESOURCE_NONE;
+
+	return offset;
+}
+
+int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev)
+{
+	void __iomem *dport_pcie_cap = NULL;
+	resource_size_t rcd_pcie_offset;
+	struct cxl_rcrb_info *ri;
+	struct cxl_dport *dport;
+	struct cxl_port *port;
+
+	port = cxl_pci_find_port(pdev, &dport);
+	if (!port)
+		return -EPROBE_DEFER;
+
+	ri = &dport->rcrb;
+	rcd_pcie_offset = cxl_rcrb_to_linkcap(&pdev->dev, dport);
+	if (rcd_pcie_offset > 0)
+		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev,
+						      ri->base + rcd_pcie_offset,
+						      PCI_CAP_EXP_SIZEOF);
+
+	dport->regs.rcd_pcie_cap = dport_pcie_cap;
+
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_dport_map_rcd_linkcap, CXL);
+
 resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
 				    enum cxl_rcrb which)
 {
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 003feebab79b..7e37ad6d84d6 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -230,6 +230,14 @@ struct cxl_regs {
 	struct_group_tagged(cxl_rch_regs, rch_regs,
 		void __iomem *dport_aer;
 	);
+
+	/*
+	 * RCD upstream port specific PCIe cap register
+	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
+	 */
+	struct_group_tagged(cxl_rcd_regs, rcd_regs,
+		void __iomem *rcd_pcie_cap;
+	);
 };
 
 struct cxl_reg_map {
@@ -299,6 +307,7 @@ int cxl_setup_regs(struct cxl_register_map *map);
 struct cxl_dport;
 resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
 					   struct cxl_dport *dport);
+int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev);
 
 #define CXL_RESOURCE_NONE ((resource_size_t) -1)
 #define CXL_TARGET_STRLEN 20
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 2ff361e756d6..bbc55732d6c1 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -512,11 +512,15 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
 	 * is an RCH and try to extract the Component Registers from
 	 * an RCRB.
 	 */
-	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
+	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
 		rc = cxl_rcrb_get_comp_regs(pdev, map);
+		if (rc)
+			return rc;
 
-	if (rc)
+		cxl_dport_map_rcd_linkcap(pdev);
+	} else if (rc) {
 		return rc;
+	}
 
 	return cxl_setup_regs(map);
 }
-- 
2.44.0

Re: [PATCH v14 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Dan Williams]

Kobayashi,Daisuke wrote:
> Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1 
> device link status information. By caching it, avoid the walking 
> memory map area to find the offset when output the register value.
> 
> Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
> ---
>  drivers/cxl/core/core.h |  6 ++++
>  drivers/cxl/core/regs.c | 61 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h       |  9 ++++++
>  drivers/cxl/pci.c       |  8 ++++--
>  4 files changed, 82 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 3b64fb1b9ed0..69006bde7ad5 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device *dev,
>  				    struct cxl_rcrb_info *ri,
>  				    enum cxl_rcrb which);
>  u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
> +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport);

Why is this function public? If it only has a caller within regs.c it
can be marked static.

> +#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
> +#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
> +#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
> +#define PCI_CAP_EXP_SIZEOF		0x3c
>  
>  extern struct rw_semaphore cxl_dpa_rwsem;
>  extern struct rw_semaphore cxl_region_rwsem;
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 372786f80955..8a7cef1cf87d 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -505,6 +505,67 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)
>  	return offset;
>  }
>  
> +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport)
> +{
> +	resource_size_t rcrb = dport->rcrb.base;
> +	void __iomem *addr;
> +	u32 cap_hdr;
> +	u16 offset;
> +
> +	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
> +		return CXL_RESOURCE_NONE;
> +
> +	addr = ioremap(rcrb, SZ_4K);
> +	if (!addr) {
> +		dev_err(dev, "Failed to map region %pr\n", addr);
> +		release_mem_region(rcrb, SZ_4K);
> +		return CXL_RESOURCE_NONE;
> +	}
> +
> +	offset = FIELD_GET(PCI_RCRB_CAP_LIST_ID_MASK, readw(addr + PCI_CAPABILITY_LIST));
> +	cap_hdr = readl(addr + offset);
> +	while ((FIELD_GET(PCI_RCRB_CAP_HDR_ID_MASK, cap_hdr)) != PCI_CAP_ID_EXP) {
> +		offset = FIELD_GET(PCI_RCRB_CAP_HDR_NEXT_MASK, cap_hdr);
> +		if (offset == 0 || offset > SZ_4K) {
> +			offset = 0;
> +			break;
> +		}
> +		cap_hdr = readl(addr + offset);

I would feel better if this duplicated __pci_find_next_cap_ttl() as
__rcrb_find_next_cap_ttl() and built an rcrb_find_capability() helper on
top of that.

> +	}
> +
> +	iounmap(addr);
> +	release_mem_region(rcrb, SZ_4K);
> +	if (!offset)
> +		return CXL_RESOURCE_NONE;
> +
> +	return offset;
> +}
> +
> +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev)
> +{
> +	void __iomem *dport_pcie_cap = NULL;
> +	resource_size_t rcd_pcie_offset;
> +	struct cxl_rcrb_info *ri;
> +	struct cxl_dport *dport;
> +	struct cxl_port *port;
> +
> +	port = cxl_pci_find_port(pdev, &dport);
> +	if (!port)
> +		return -EPROBE_DEFER;

There is a missing put_device() for this find. It also seems silly to
find the port again after already finding it for
cxl_rcrb_get_comp_regs().

> +
> +	ri = &dport->rcrb;
> +	rcd_pcie_offset = cxl_rcrb_to_linkcap(&pdev->dev, dport);
> +	if (rcd_pcie_offset > 0)

This looks broken if cxl_rcrb_to_linkcap() returns CXL_RESOURCE_NONE.

Also, lets s/rcd_pcie_offset/pos/ since @pos is already the common
variable name for tracking PCI config register offsets.

> +		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev,
> +						      ri->base + rcd_pcie_offset,
> +						      PCI_CAP_EXP_SIZEOF);
> +
> +	dport->regs.rcd_pcie_cap = dport_pcie_cap;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_dport_map_rcd_linkcap, CXL);
> +
>  resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
>  				    enum cxl_rcrb which)
>  {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 003feebab79b..7e37ad6d84d6 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -230,6 +230,14 @@ struct cxl_regs {
>  	struct_group_tagged(cxl_rch_regs, rch_regs,
>  		void __iomem *dport_aer;
>  	);
> +
> +	/*
> +	 * RCD upstream port specific PCIe cap register
> +	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
> +	 */
> +	struct_group_tagged(cxl_rcd_regs, rcd_regs,
> +		void __iomem *rcd_pcie_cap;
> +	);
>  };
>  
>  struct cxl_reg_map {
> @@ -299,6 +307,7 @@ int cxl_setup_regs(struct cxl_register_map *map);
>  struct cxl_dport;
>  resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
>  					   struct cxl_dport *dport);
> +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev);
>  
>  #define CXL_RESOURCE_NONE ((resource_size_t) -1)
>  #define CXL_TARGET_STRLEN 20
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 2ff361e756d6..bbc55732d6c1 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -512,11 +512,15 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
>  	 * is an RCH and try to extract the Component Registers from
>  	 * an RCRB.
>  	 */
> -	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
> +	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
>  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> +		if (rc)
> +			return rc;
>  
> -	if (rc)
> +		cxl_dport_map_rcd_linkcap(pdev);

There is a chance that the port gets removed between finding it for
cxl_rcrb_get_comp_regs() and finding it again for
cxl_dport_map_rcd_linkcap(), so this misses the need to return
EPROBE_DEFER if that latter race is lost. However, is also a chance that
race can be lost *internal* to cxl_rcrb_get_comp_regs().

The only guarantee of cxl_pci_find_port() is that it returns a 'struct
cxl_port' instance that will not be freed until put_device(&port->dev),
but it makes no guarantee that the dport information is stable over that
lifetime. I think this needs a leading fix patch that finds and locks
the port and then passes that port to cxl_rcrb_get_comp_regs() and
cxl_dport_map_rcd_linkcap().

Otherwise I think you could crash the kernel with a test that repeatedly
reloads the cxl_acpi module while also reloading cxl_pci.

Ugh, I was going to say copy what cxl_mem_probe() does around locking
endpoint_parent before attaching further ports, but that also appears to
not handle the same race. I.e. I think cxl_mem_probe() needs a fix to do
this as well. I will copy you on a proposed patch for that.

Re: [PATCH v14 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Dan Williams]

Dan Williams wrote:
> Kobayashi,Daisuke wrote:
> > Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1 
> > device link status information. By caching it, avoid the walking 
> > memory map area to find the offset when output the register value.
> > 
> > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
> > ---
> >  drivers/cxl/core/core.h |  6 ++++
> >  drivers/cxl/core/regs.c | 61 +++++++++++++++++++++++++++++++++++++++++
> >  drivers/cxl/cxl.h       |  9 ++++++
> >  drivers/cxl/pci.c       |  8 ++++--
> >  4 files changed, 82 insertions(+), 2 deletions(-)
> > 
[..]
> > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > index 2ff361e756d6..bbc55732d6c1 100644
> > --- a/drivers/cxl/pci.c
> > +++ b/drivers/cxl/pci.c
> > @@ -512,11 +512,15 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
> >  	 * is an RCH and try to extract the Component Registers from
> >  	 * an RCRB.
> >  	 */
> > -	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
> > +	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
> >  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> > +		if (rc)
> > +			return rc;
> >  
> > -	if (rc)
> > +		cxl_dport_map_rcd_linkcap(pdev);
> 
[..]
> Ugh, I was going to say copy what cxl_mem_probe() does around locking
> endpoint_parent before attaching further ports, but that also appears to
> not handle the same race. I.e. I think cxl_mem_probe() needs a fix to do
> this as well. I will copy you on a proposed patch for that.

I attempted to add the proper locking to keep cxl_dport live, but that
runs into lockdep issues.

So I think a better fix is rework dport lifetime to stay alive until the
final put_device() of the port. In other words dport instances get added
dynamically to the cxl_port, but only get destroyed after all port
references are dropped. Then the @dport result from find_cxl_port() is
not ephemeral.

Given this is a latent bug that affects all current
cxl_{mem,pci}_find_port() users, the planned fix is to just make dport
lifetime longer, and that I will not have time to do that rework before
v6.11 merge window, then I am ok for this lnkcap code to introduce
another instance of the same bug.

So, just make cxl_rcrb_get_comp_regs() and cxl_dport_map_rcd_linkcap()
share the same port reference from one cxl_pci_find_port() call.

RE: [PATCH v14 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Daisuke Kobayashi (Fujitsu)]

Dan Williams wrote:
> Dan Williams wrote:
> > Kobayashi,Daisuke wrote:
> > > Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1
> > > device link status information. By caching it, avoid the walking
> > > memory map area to find the offset when output the register value.
> > >
> > > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
> > > ---
> > >  drivers/cxl/core/core.h |  6 ++++
> > >  drivers/cxl/core/regs.c | 61
> +++++++++++++++++++++++++++++++++++++++++
> > >  drivers/cxl/cxl.h       |  9 ++++++
> > >  drivers/cxl/pci.c       |  8 ++++--
> > >  4 files changed, 82 insertions(+), 2 deletions(-)
> > >
> [..]
> > > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > > index 2ff361e756d6..bbc55732d6c1 100644
> > > --- a/drivers/cxl/pci.c
> > > +++ b/drivers/cxl/pci.c
> > > @@ -512,11 +512,15 @@ static int cxl_pci_setup_regs(struct pci_dev
> *pdev, enum cxl_regloc_type type,
> > >  	 * is an RCH and try to extract the Component Registers from
> > >  	 * an RCRB.
> > >  	 */
> > > -	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> is_cxl_restricted(pdev))
> > > +	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> is_cxl_restricted(pdev)) {
> > >  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> > > +		if (rc)
> > > +			return rc;
> > >
> > > -	if (rc)
> > > +		cxl_dport_map_rcd_linkcap(pdev);
> >
> [..]
> > Ugh, I was going to say copy what cxl_mem_probe() does around locking
> > endpoint_parent before attaching further ports, but that also appears to
> > not handle the same race. I.e. I think cxl_mem_probe() needs a fix to do
> > this as well. I will copy you on a proposed patch for that.
> 
> I attempted to add the proper locking to keep cxl_dport live, but that
> runs into lockdep issues.
> 
> So I think a better fix is rework dport lifetime to stay alive until the
> final put_device() of the port. In other words dport instances get added
> dynamically to the cxl_port, but only get destroyed after all port
> references are dropped. Then the @dport result from find_cxl_port() is
> not ephemeral.
> 
> Given this is a latent bug that affects all current
> cxl_{mem,pci}_find_port() users, the planned fix is to just make dport
> lifetime longer, and that I will not have time to do that rework before
> v6.11 merge window, then I am ok for this lnkcap code to introduce
> another instance of the same bug.
> 
> So, just make cxl_rcrb_get_comp_regs() and cxl_dport_map_rcd_linkcap()
> share the same port reference from one cxl_pci_find_port() call.

Thanks for checking.

I'd like to confirm my understanding of the comment. Are you suggesting that,
due to time constraints with the current patch, cxl_rcrb_get_comp_regs() and
cxl_dport_map_rcd_linkcap() should share the same dport reference as a temporary
workaround for the bug regarding the dport lifetime?

If that's what you mean, I think I can solve this problem by adding
"struct cxl_dport *dport" to the arguments of the two functions to share the reference.
In this implementation, I'm planning to run cxl_pci_find_port() in cxl_rcrb_get_comp_regs()
and share the dport obtained there. You said that find requires a corresponding put_device(),
but where is the correct place to run it in this case? Or is it better not to run it in this patch?

RE: [PATCH v14 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Dan Williams]

Daisuke Kobayashi (Fujitsu) wrote:
> Dan Williams wrote:
> > Dan Williams wrote:
> > > Kobayashi,Daisuke wrote:
> > > > Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1
> > > > device link status information. By caching it, avoid the walking
> > > > memory map area to find the offset when output the register value.
> > > >
> > > > Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > > Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
> > > > ---
> > > >  drivers/cxl/core/core.h |  6 ++++
> > > >  drivers/cxl/core/regs.c | 61
> > +++++++++++++++++++++++++++++++++++++++++
> > > >  drivers/cxl/cxl.h       |  9 ++++++
> > > >  drivers/cxl/pci.c       |  8 ++++--
> > > >  4 files changed, 82 insertions(+), 2 deletions(-)
> > > >
> > [..]
> > > > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > > > index 2ff361e756d6..bbc55732d6c1 100644
> > > > --- a/drivers/cxl/pci.c
> > > > +++ b/drivers/cxl/pci.c
> > > > @@ -512,11 +512,15 @@ static int cxl_pci_setup_regs(struct pci_dev
> > *pdev, enum cxl_regloc_type type,
> > > >  	 * is an RCH and try to extract the Component Registers from
> > > >  	 * an RCRB.
> > > >  	 */
> > > > -	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> > is_cxl_restricted(pdev))
> > > > +	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> > is_cxl_restricted(pdev)) {
> > > >  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> > > > +		if (rc)
> > > > +			return rc;
> > > >
> > > > -	if (rc)
> > > > +		cxl_dport_map_rcd_linkcap(pdev);
> > >
> > [..]
> > > Ugh, I was going to say copy what cxl_mem_probe() does around locking
> > > endpoint_parent before attaching further ports, but that also appears to
> > > not handle the same race. I.e. I think cxl_mem_probe() needs a fix to do
> > > this as well. I will copy you on a proposed patch for that.
> > 
> > I attempted to add the proper locking to keep cxl_dport live, but that
> > runs into lockdep issues.
> > 
> > So I think a better fix is rework dport lifetime to stay alive until the
> > final put_device() of the port. In other words dport instances get added
> > dynamically to the cxl_port, but only get destroyed after all port
> > references are dropped. Then the @dport result from find_cxl_port() is
> > not ephemeral.
> > 
> > Given this is a latent bug that affects all current
> > cxl_{mem,pci}_find_port() users, the planned fix is to just make dport
> > lifetime longer, and that I will not have time to do that rework before
> > v6.11 merge window, then I am ok for this lnkcap code to introduce
> > another instance of the same bug.
> > 
> > So, just make cxl_rcrb_get_comp_regs() and cxl_dport_map_rcd_linkcap()
> > share the same port reference from one cxl_pci_find_port() call.
> 
> Thanks for checking.
> 
> I'd like to confirm my understanding of the comment. Are you suggesting that,
> due to time constraints with the current patch, cxl_rcrb_get_comp_regs() and
> cxl_dport_map_rcd_linkcap() should share the same dport reference as a temporary
> workaround for the bug regarding the dport lifetime?

What I am saying is forget the bug for now, just trust that the @dport
result from cxl_pci_find_port() is valid until the put_device() on the
port.

> If that's what you mean, I think I can solve this problem by adding
> "struct cxl_dport *dport" to the arguments of the two functions to share the reference.

Yes, that's what I want for this patch, but to be clear this does not
fix the bug with cxl_pci_find_port(). That bug needs deeper work that
you can ignore for now. Adding another cxl_pci_find_port() user just
increases the urgency to get that bug fixed.

To be clear it is definitely a use after-free issue, but it needs root to be bringing
ports up and down during the "cxl_pci_find_port() ->
put_device(@port->dev)" window.

I expect you could trigger a crash by a "modprobe -r cxl_acpi; modprobe
cxl_acpi" loop while accessing these sysfs files.

> In this implementation, I'm planning to run cxl_pci_find_port() in
> cxl_rcrb_get_comp_regs() and share the dport obtained there. You said
> that find requires a corresponding put_device(), but where is the
> correct place to run it in this case? Or is it better not to run it in
> this patch?

So the ordering I expect is something like:

	port = cxl_pci_find_port(...)
	if (!port)
		return -EPROBE_DEFER;
	rc = cxl_rcrb_get_comp_regs(dport, ...)
	if (rc)
		goto put;
	rc = cxl_dport_map_rcd_linkcap(dport, ...)
	if (rc)
		goto put;
put:
	put_device(...)
	return rc;

[PATCH v14 2/2] cxl/pci: Add sysfs attribute for CXL 1.1 device link status

[Kobayashi,Daisuke]

Add sysfs attribute for CXL 1.1 device link status to the cxl pci device.

In CXL1.1, the link status of the device is included in the RCRB mapped to
the memory mapped register area. Critically, that arrangement makes the
link status and control registers invisible to existing PCI user tooling.

Export those registers via sysfs with the expectation that PCI user
tooling will alternatively look for these sysfs files when attempting to
access to these CXL 1.1 endpoints registers.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
---
 drivers/cxl/pci.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index bbc55732d6c1..6fa8a1a031bb 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -790,6 +790,86 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	return 0;
 }
 
+static ssize_t rcd_pcie_cap_emit(struct device *dev, u16 offset, char *buf, size_t width)
+{
+	struct cxl_dev_state *cxlds = dev_get_drvdata(dev);
+	struct cxl_memdev *cxlmd = cxlds->cxlmd;
+	struct device *endpoint_parent;
+	struct cxl_dport *dport;
+	struct cxl_port *port;
+
+	port = cxl_mem_find_port(cxlmd, &dport);
+	if (!port)
+		return -EINVAL;
+
+	endpoint_parent = port->uport_dev;
+	if (!endpoint_parent)
+		return -ENXIO;
+
+	guard(device)(endpoint_parent);
+	if (!endpoint_parent->driver)
+		return -ENXIO;
+
+	if (dport->regs.rcd_pcie_cap == NULL)
+		return -EINVAL;
+
+	switch (width) {
+	case sizeof(u16):
+		return sysfs_emit(buf, "%x\n",
+				  readw(dport->regs.rcd_pcie_cap + offset));
+	case sizeof(u32):
+		return sysfs_emit(buf, "%x\n",
+				  readl(dport->regs.rcd_pcie_cap + offset));
+	default:
+		return -EINVAL;
+	}
+}
+
+static ssize_t rcd_link_cap_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKCAP, buf, sizeof(u32));
+}
+static DEVICE_ATTR_RO(rcd_link_cap);
+
+static ssize_t rcd_link_ctrl_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKCTL, buf, sizeof(u16));
+}
+static DEVICE_ATTR_RO(rcd_link_ctrl);
+
+static ssize_t rcd_link_status_show(struct device *dev,
+				    struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKSTA, buf, sizeof(u16));
+}
+static DEVICE_ATTR_RO(rcd_link_status);
+
+static struct attribute *cxl_rcd_attrs[] = {
+	&dev_attr_rcd_link_cap.attr,
+	&dev_attr_rcd_link_ctrl.attr,
+	&dev_attr_rcd_link_status.attr,
+	NULL
+};
+
+static umode_t cxl_rcd_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	if (is_cxl_restricted(pdev))
+		return a->mode;
+
+	return 0;
+}
+
+static struct attribute_group cxl_rcd_group = {
+	.attrs = cxl_rcd_attrs,
+	.is_visible = cxl_rcd_visible,
+};
+__ATTRIBUTE_GROUPS(cxl_rcd);
+
 static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus);
@@ -973,6 +1053,7 @@ static struct pci_driver cxl_pci_driver = {
 	.id_table		= cxl_mem_pci_tbl,
 	.probe			= cxl_pci_probe,
 	.err_handler		= &cxl_error_handlers,
+	.dev_groups		= cxl_rcd_groups,
 	.driver	= {
 		.probe_type	= PROBE_PREFER_ASYNCHRONOUS,
 	},
-- 
2.44.0

RE: [PATCH v14 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[Daisuke Kobayashi (Fujitsu)]

Kobayashi Daisuke wrote:
> CXL devices are extensions of PCIe. Therefore, from CXL2.0 onwards, the link
> status can be output in the same way as traditional PCIe.
> However, unlike devices from CXL2.0 onwards, CXL1.1 requires a different
> method to obtain the link status from traditional PCIe.
> This is because the link status of the CXL1.1 device is not mapped in the
> configuration space (as per cxl3.0 specification 8.1).
> Instead, the configuration space containing the link status is mapped to the
> memory mapped register region (as per cxl3.0 specification 8.2, Table 8-18).
> Therefore, the current lspci has a problem where it does not display the link
> status of the CXL1.1 device.
> Solve these issues with sysfs attributes to export the status registers hidden in
> the RCRB.
> 
> The procedure is as follows:
> First, obtain the RCRB address within the cxl driver, then access the
> configuration space. Next, output the link status information from the
> configuration space to sysfs. Ultimately, the expectation is that this sysfs file
> will be consumed by PCI user tools to utilize link status information.
> 
> 
> Changes
> v1[1] -> v2:
> - Modified to perform rcrb access within the CXL driver.
> - Added new attributes to the sysfs of the PCI device.
> - Output the link status information to the sysfs of the PCI device.
> - Retrieve information from sysfs as the source when displaying information in
> lspci.
> 
> v2[2] -> v3:
> - Fix unnecessary initialization and wrong types (Bjohn).
> - Create a helper function for getting a PCIe capability offset (Bjohn).
> - Move platform-specific implementation to the lib directory in pciutils
> (Martin).
> 
> v3[3] -> v4:
> - RCRB register values are read once and cached.
> - Added a new attribute to the sysfs of the PCI device.
> - Separate lspci implementation from this patch.
> 
> v4[4] -> v5:
> - Use macros for bitwise operations
> - Fix RCRB access to use cxl_memdev
> 
> v5[5] -> v6:
> - Add and use masks for RCRB register values
> 
> v6[6] -> v7:
> - Fix comments on white space inline
> 
> v7[7] -> v8:
> - Change the cache value to offset
> - Access memory map area in rcd_*_show() functions
> 
> v8[8] -> v9:
> - Map the pcie cap in for all the time the driver is bound to the device.
> - Add mapping the pcie cap in cxl_rcd_component_reg_phys().
> 
> v9[9] -> v10:
> - Change a utility function for getting PCIe capability.
> - Fix tab alignment issue, error handling, and apply suggestions from Jonathan.
> 
> v10[10] -> v11:
> - Add functions to have one function do only one thing.
> - Add a size parameter to utility function arguments and consolidated them into
> one.
> 
> v11[11] -> v12:
> - Fix the error handling in cxl_pci_setup_regs().
> - Fix and clean up some details.
> 
> v12[12] -> v13:
> - Fix and clean up some details.
> 
> v13[13] -> v14:
> - Fix and clean up some details.
> 
> [1]
> https://lore.kernel.org/linux-cxl/20231220050738.178481-1-kobayashi.da-06
> @fujitsu.com/
> [2]
> https://lore.kernel.org/linux-cxl/20240227083313.87699-1-kobayashi.da-06@
> fujitsu.com/
> [3]
> https://lore.kernel.org/linux-cxl/20240312080559.14904-1-kobayashi.da-06@
> fujitsu.com/
> [4]
> https://lore.kernel.org/linux-cxl/20240409073528.13214-1-kobayashi.da-06@
> fujitsu.com/
> [5]
> https://lore.kernel.org/linux-cxl/20240412070715.16160-1-kobayashi.da-06@
> fujitsu.com/
> [6]
> https://lore.kernel.org/linux-cxl/20240424050102.26788-1-kobayashi.da-06@
> fujitsu.com/
> [7]
> https://lore.kernel.org/linux-cxl/20240510073710.98953-1-kobayashi.da-06@
> fujitsu.com/
> [8]
> https://lore.kernel.org/linux-cxl/20240606074814.5633-1-kobayashi.da-06@f
> ujitsu.com/
> [9]
> https://lore.kernel.org/linux-cxl/20240610082222.22772-1-kobayashi.da-06@
> fujitsu.com/
> [10]
> https://lore.kernel.org/linux-cxl/20240611055254.61203-1-kobayashi.da-06@
> fujitsu.com/
> [11]
> https://lore.kernel.org/linux-cxl/20240612075940.92500-1-kobayashi.da-06@
> fujitsu.com/
> [12]
> https://lore.kernel.org/linux-cxl/20240614045611.58658-1-kobayashi.da-06@
> fujitsu.com/
> [13]
> https://lore.kernel.org/linux-cxl/20240617043702.62028-1-kobayashi.da-06@
> fujitsu.com/
> 
> Kobayashi,Daisuke (2):
>   cxl/core/regs: Add rcd_pcie_cap initialization
>   cxl/pci: Add sysfs attribute for CXL 1.1 device link status
> 
>  drivers/cxl/core/core.h |  6 +++
>  drivers/cxl/core/regs.c | 61 ++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h       |  9 +++++
>  drivers/cxl/pci.c       | 89
> ++++++++++++++++++++++++++++++++++++++++-
>  4 files changed, 163 insertions(+), 2 deletions(-)
> 
> --
> 2.44.0

Can you merge this patch into the next kernel?
It has been reviewed and I believe it is ready to be merged.
If it is already in process, please let me know as it is my fault for not checking thoroughly.

Re: [PATCH v14 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[Dave Jiang]

On 7/7/24 8:05 PM, Daisuke Kobayashi (Fujitsu) wrote:
> 
> Kobayashi Daisuke wrote:
>> CXL devices are extensions of PCIe. Therefore, from CXL2.0 onwards, the link
>> status can be output in the same way as traditional PCIe.
>> However, unlike devices from CXL2.0 onwards, CXL1.1 requires a different
>> method to obtain the link status from traditional PCIe.
>> This is because the link status of the CXL1.1 device is not mapped in the
>> configuration space (as per cxl3.0 specification 8.1).
>> Instead, the configuration space containing the link status is mapped to the
>> memory mapped register region (as per cxl3.0 specification 8.2, Table 8-18).
>> Therefore, the current lspci has a problem where it does not display the link
>> status of the CXL1.1 device.
>> Solve these issues with sysfs attributes to export the status registers hidden in
>> the RCRB.
>>
>> The procedure is as follows:
>> First, obtain the RCRB address within the cxl driver, then access the
>> configuration space. Next, output the link status information from the
>> configuration space to sysfs. Ultimately, the expectation is that this sysfs file
>> will be consumed by PCI user tools to utilize link status information.
>>
>>
>> Changes
>> v1[1] -> v2:
>> - Modified to perform rcrb access within the CXL driver.
>> - Added new attributes to the sysfs of the PCI device.
>> - Output the link status information to the sysfs of the PCI device.
>> - Retrieve information from sysfs as the source when displaying information in
>> lspci.
>>
>> v2[2] -> v3:
>> - Fix unnecessary initialization and wrong types (Bjohn).
>> - Create a helper function for getting a PCIe capability offset (Bjohn).
>> - Move platform-specific implementation to the lib directory in pciutils
>> (Martin).
>>
>> v3[3] -> v4:
>> - RCRB register values are read once and cached.
>> - Added a new attribute to the sysfs of the PCI device.
>> - Separate lspci implementation from this patch.
>>
>> v4[4] -> v5:
>> - Use macros for bitwise operations
>> - Fix RCRB access to use cxl_memdev
>>
>> v5[5] -> v6:
>> - Add and use masks for RCRB register values
>>
>> v6[6] -> v7:
>> - Fix comments on white space inline
>>
>> v7[7] -> v8:
>> - Change the cache value to offset
>> - Access memory map area in rcd_*_show() functions
>>
>> v8[8] -> v9:
>> - Map the pcie cap in for all the time the driver is bound to the device.
>> - Add mapping the pcie cap in cxl_rcd_component_reg_phys().
>>
>> v9[9] -> v10:
>> - Change a utility function for getting PCIe capability.
>> - Fix tab alignment issue, error handling, and apply suggestions from Jonathan.
>>
>> v10[10] -> v11:
>> - Add functions to have one function do only one thing.
>> - Add a size parameter to utility function arguments and consolidated them into
>> one.
>>
>> v11[11] -> v12:
>> - Fix the error handling in cxl_pci_setup_regs().
>> - Fix and clean up some details.
>>
>> v12[12] -> v13:
>> - Fix and clean up some details.
>>
>> v13[13] -> v14:
>> - Fix and clean up some details.
>>
>> [1]
>> https://lore.kernel.org/linux-cxl/20231220050738.178481-1-kobayashi.da-06
>> @fujitsu.com/
>> [2]
>> https://lore.kernel.org/linux-cxl/20240227083313.87699-1-kobayashi.da-06@
>> fujitsu.com/
>> [3]
>> https://lore.kernel.org/linux-cxl/20240312080559.14904-1-kobayashi.da-06@
>> fujitsu.com/
>> [4]
>> https://lore.kernel.org/linux-cxl/20240409073528.13214-1-kobayashi.da-06@
>> fujitsu.com/
>> [5]
>> https://lore.kernel.org/linux-cxl/20240412070715.16160-1-kobayashi.da-06@
>> fujitsu.com/
>> [6]
>> https://lore.kernel.org/linux-cxl/20240424050102.26788-1-kobayashi.da-06@
>> fujitsu.com/
>> [7]
>> https://lore.kernel.org/linux-cxl/20240510073710.98953-1-kobayashi.da-06@
>> fujitsu.com/
>> [8]
>> https://lore.kernel.org/linux-cxl/20240606074814.5633-1-kobayashi.da-06@f
>> ujitsu.com/
>> [9]
>> https://lore.kernel.org/linux-cxl/20240610082222.22772-1-kobayashi.da-06@
>> fujitsu.com/
>> [10]
>> https://lore.kernel.org/linux-cxl/20240611055254.61203-1-kobayashi.da-06@
>> fujitsu.com/
>> [11]
>> https://lore.kernel.org/linux-cxl/20240612075940.92500-1-kobayashi.da-06@
>> fujitsu.com/
>> [12]
>> https://lore.kernel.org/linux-cxl/20240614045611.58658-1-kobayashi.da-06@
>> fujitsu.com/
>> [13]
>> https://lore.kernel.org/linux-cxl/20240617043702.62028-1-kobayashi.da-06@
>> fujitsu.com/
>>
>> Kobayashi,Daisuke (2):
>>   cxl/core/regs: Add rcd_pcie_cap initialization
>>   cxl/pci: Add sysfs attribute for CXL 1.1 device link status
>>
>>  drivers/cxl/core/core.h |  6 +++
>>  drivers/cxl/core/regs.c | 61 ++++++++++++++++++++++++++++
>>  drivers/cxl/cxl.h       |  9 +++++
>>  drivers/cxl/pci.c       | 89
>> ++++++++++++++++++++++++++++++++++++++++-
>>  4 files changed, 163 insertions(+), 2 deletions(-)
>>
>> --
>> 2.44.0
> 
> Can you merge this patch into the next kernel?
> It has been reviewed and I believe it is ready to be merged.
> If it is already in process, please let me know as it is my fault for not checking thoroughly.

I'm waiting on review tags from Dan for this series. As I recall he had some previous comments. 
> 

RE: [PATCH v14 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[Daisuke Kobayashi (Fujitsu)]

Dave Jiang wrote:
> On 7/7/24 8:05 PM, Daisuke Kobayashi (Fujitsu) wrote:
> >
> > Kobayashi Daisuke wrote:
> >> CXL devices are extensions of PCIe. Therefore, from CXL2.0 onwards,
> >> the link status can be output in the same way as traditional PCIe.
> >> However, unlike devices from CXL2.0 onwards, CXL1.1 requires a
> >> different method to obtain the link status from traditional PCIe.
> >> This is because the link status of the CXL1.1 device is not mapped in
> >> the configuration space (as per cxl3.0 specification 8.1).
> >> Instead, the configuration space containing the link status is mapped
> >> to the memory mapped register region (as per cxl3.0 specification 8.2, Table
> 8-18).
> >> Therefore, the current lspci has a problem where it does not display
> >> the link status of the CXL1.1 device.
> >> Solve these issues with sysfs attributes to export the status
> >> registers hidden in the RCRB.
> >>
> >> The procedure is as follows:
> >> First, obtain the RCRB address within the cxl driver, then access the
> >> configuration space. Next, output the link status information from
> >> the configuration space to sysfs. Ultimately, the expectation is that
> >> this sysfs file will be consumed by PCI user tools to utilize link status
> information.
> >>
> >>
> >> Changes
> >> v1[1] -> v2:
> >> - Modified to perform rcrb access within the CXL driver.
> >> - Added new attributes to the sysfs of the PCI device.
> >> - Output the link status information to the sysfs of the PCI device.
> >> - Retrieve information from sysfs as the source when displaying
> >> information in lspci.
> >>
> >> v2[2] -> v3:
> >> - Fix unnecessary initialization and wrong types (Bjohn).
> >> - Create a helper function for getting a PCIe capability offset (Bjohn).
> >> - Move platform-specific implementation to the lib directory in
> >> pciutils (Martin).
> >>
> >> v3[3] -> v4:
> >> - RCRB register values are read once and cached.
> >> - Added a new attribute to the sysfs of the PCI device.
> >> - Separate lspci implementation from this patch.
> >>
> >> v4[4] -> v5:
> >> - Use macros for bitwise operations
> >> - Fix RCRB access to use cxl_memdev
> >>
> >> v5[5] -> v6:
> >> - Add and use masks for RCRB register values
> >>
> >> v6[6] -> v7:
> >> - Fix comments on white space inline
> >>
> >> v7[7] -> v8:
> >> - Change the cache value to offset
> >> - Access memory map area in rcd_*_show() functions
> >>
> >> v8[8] -> v9:
> >> - Map the pcie cap in for all the time the driver is bound to the device.
> >> - Add mapping the pcie cap in cxl_rcd_component_reg_phys().
> >>
> >> v9[9] -> v10:
> >> - Change a utility function for getting PCIe capability.
> >> - Fix tab alignment issue, error handling, and apply suggestions from
> Jonathan.
> >>
> >> v10[10] -> v11:
> >> - Add functions to have one function do only one thing.
> >> - Add a size parameter to utility function arguments and consolidated
> >> them into one.
> >>
> >> v11[11] -> v12:
> >> - Fix the error handling in cxl_pci_setup_regs().
> >> - Fix and clean up some details.
> >>
> >> v12[12] -> v13:
> >> - Fix and clean up some details.
> >>
> >> v13[13] -> v14:
> >> - Fix and clean up some details.
> >>
> >> [1]
> >> https://lore.kernel.org/linux-cxl/20231220050738.178481-1-kobayashi.d
> >> a-06
> >> @fujitsu.com/
> >> [2]
> >> https://lore.kernel.org/linux-cxl/20240227083313.87699-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [3]
> >> https://lore.kernel.org/linux-cxl/20240312080559.14904-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [4]
> >> https://lore.kernel.org/linux-cxl/20240409073528.13214-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [5]
> >> https://lore.kernel.org/linux-cxl/20240412070715.16160-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [6]
> >> https://lore.kernel.org/linux-cxl/20240424050102.26788-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [7]
> >> https://lore.kernel.org/linux-cxl/20240510073710.98953-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [8]
> >> https://lore.kernel.org/linux-cxl/20240606074814.5633-1-kobayashi.da-
> >> 06@f
> >> ujitsu.com/
> >> [9]
> >> https://lore.kernel.org/linux-cxl/20240610082222.22772-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [10]
> >> https://lore.kernel.org/linux-cxl/20240611055254.61203-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [11]
> >> https://lore.kernel.org/linux-cxl/20240612075940.92500-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [12]
> >> https://lore.kernel.org/linux-cxl/20240614045611.58658-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >> [13]
> >> https://lore.kernel.org/linux-cxl/20240617043702.62028-1-kobayashi.da
> >> -06@
> >> fujitsu.com/
> >>
> >> Kobayashi,Daisuke (2):
> >>   cxl/core/regs: Add rcd_pcie_cap initialization
> >>   cxl/pci: Add sysfs attribute for CXL 1.1 device link status
> >>
> >>  drivers/cxl/core/core.h |  6 +++
> >>  drivers/cxl/core/regs.c | 61 ++++++++++++++++++++++++++++
> >>  drivers/cxl/cxl.h       |  9 +++++
> >>  drivers/cxl/pci.c       | 89
> >> ++++++++++++++++++++++++++++++++++++++++-
> >>  4 files changed, 163 insertions(+), 2 deletions(-)
> >>
> >> --
> >> 2.44.0
> >
> > Can you merge this patch into the next kernel?
> > It has been reviewed and I believe it is ready to be merged.
> > If it is already in process, please let me know as it is my fault for not checking
> thoroughly.
> 
> I'm waiting on review tags from Dan for this series. As I recall he had some
> previous comments.
> >

Thank you for explaining the patch situation. Dan, would you mind reviewing this patch?