[PATCH v12 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[PATCH v12 0/2] Export cxl1.1 device link status register value to pci device sysfs.

[Kobayashi,Daisuke]

CXL devices are extensions of PCIe. Therefore, from CXL2.0 onwards,
the link status can be output in the same way as traditional PCIe.
However, unlike devices from CXL2.0 onwards, CXL1.1 requires a
different method to obtain the link status from traditional PCIe.
This is because the link status of the CXL1.1 device is not mapped
in the configuration space (as per cxl3.0 specification 8.1).
Instead, the configuration space containing the link status is mapped
to the memory mapped register region (as per cxl3.0 specification 8.2,
Table 8-18). Therefore, the current lspci has a problem where it does
not display the link status of the CXL1.1 device. 
Solve these issues with sysfs attributes to export the status
registers hidden in the RCRB.

The procedure is as follows:
First, obtain the RCRB address within the cxl driver, then access 
the configuration space. Next, output the link status information from
the configuration space to sysfs. Ultimately, the expectation is that
this sysfs file will be consumed by PCI user tools to utilize link status
information.


Changes
v1[1] -> v2:
- Modified to perform rcrb access within the CXL driver.
- Added new attributes to the sysfs of the PCI device.
- Output the link status information to the sysfs of the PCI device.
- Retrieve information from sysfs as the source when displaying information in lspci.

v2[2] -> v3:
- Fix unnecessary initialization and wrong types (Bjohn).
- Create a helper function for getting a PCIe capability offset (Bjohn).
- Move platform-specific implementation to the lib directory in pciutils (Martin).

v3[3] -> v4:
- RCRB register values are read once and cached.
- Added a new attribute to the sysfs of the PCI device.
- Separate lspci implementation from this patch.

v4[4] -> v5:
- Use macros for bitwise operations
- Fix RCRB access to use cxl_memdev

v5[5] -> v6:
- Add and use masks for RCRB register values

v6[6] -> v7:
- Fix comments on white space inline

v7[7] -> v8:
- Change the cache value to offset
- Access memory map area in rcd_*_show() functions

v8[8] -> v9:
- Map the pcie cap in for all the time the driver is bound to the device.
- Add mapping the pcie cap in cxl_rcd_component_reg_phys().

v9[9] -> v10:
- Change a utility function for getting PCIe capability.
- Fix tab alignment issue, error handling, and apply suggestions from Jonathan.

v10[10] -> v11:
- Add functions to have one function do only one thing.
- Add a size parameter to utility function arguments and consolidated them into one.

v11[11] -> v12:
- Fix the error handling in cxl_pci_setup_regs().
- Fix and clean up some details.

[1]
https://lore.kernel.org/linux-cxl/20231220050738.178481-1-kobayashi.da-06@fujitsu.com/
[2]
https://lore.kernel.org/linux-cxl/20240227083313.87699-1-kobayashi.da-06@fujitsu.com/
[3]
https://lore.kernel.org/linux-cxl/20240312080559.14904-1-kobayashi.da-06@fujitsu.com/
[4]
https://lore.kernel.org/linux-cxl/20240409073528.13214-1-kobayashi.da-06@fujitsu.com/
[5]
https://lore.kernel.org/linux-cxl/20240412070715.16160-1-kobayashi.da-06@fujitsu.com/
[6]
https://lore.kernel.org/linux-cxl/20240424050102.26788-1-kobayashi.da-06@fujitsu.com/
[7]
https://lore.kernel.org/linux-cxl/20240510073710.98953-1-kobayashi.da-06@fujitsu.com/
[8]
https://lore.kernel.org/linux-cxl/20240606074814.5633-1-kobayashi.da-06@fujitsu.com/
[9]
https://lore.kernel.org/linux-cxl/20240610082222.22772-1-kobayashi.da-06@fujitsu.com/
[10]
https://lore.kernel.org/linux-cxl/20240611055254.61203-1-kobayashi.da-06@fujitsu.com/
[11]
https://lore.kernel.org/linux-cxl/20240612075940.92500-1-kobayashi.da-06@fujitsu.com/

Kobayashi,Daisuke (2):
  cxl/core/regs: Add rcd_pcie_cap initialization
  cxl/pci: Add sysfs attribute for CXL 1.1 device link status

 drivers/cxl/core/core.h |  6 +++
 drivers/cxl/core/regs.c | 62 +++++++++++++++++++++++++++++
 drivers/cxl/cxl.h       | 10 +++++
 drivers/cxl/pci.c       | 88 ++++++++++++++++++++++++++++++++++++++++-
 4 files changed, 164 insertions(+), 2 deletions(-)

-- 
2.44.0

[PATCH v12 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Kobayashi,Daisuke]

Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1 
device link status information. By caching it, avoid the walking 
memory map area to find the offset when output the register value.

Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
---
 drivers/cxl/core/core.h |  6 ++++
 drivers/cxl/core/regs.c | 62 +++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/cxl.h       | 10 +++++++
 drivers/cxl/pci.c       |  7 +++--
 4 files changed, 83 insertions(+), 2 deletions(-)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 3b64fb1b9ed0..69006bde7ad5 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device *dev,
 				    struct cxl_rcrb_info *ri,
 				    enum cxl_rcrb which);
 u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
+resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport);
+
+#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
+#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
+#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
+#define PCI_CAP_EXP_SIZEOF		0x3c
 
 extern struct rw_semaphore cxl_dpa_rwsem;
 extern struct rw_semaphore cxl_region_rwsem;
diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
index 372786f80955..96c2a289cfb7 100644
--- a/drivers/cxl/core/regs.c
+++ b/drivers/cxl/core/regs.c
@@ -505,6 +505,68 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)
 	return offset;
 }
 
+resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport)
+{
+	resource_size_t rcrb = dport->rcrb.base;
+	void __iomem *addr;
+	u32 cap_hdr;
+	u16 offset;
+
+	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
+		return CXL_RESOURCE_NONE;
+
+	addr = ioremap(rcrb, SZ_4K);
+	if (!addr) {
+		dev_err(dev, "Failed to map region %pr\n", addr);
+		release_mem_region(rcrb, SZ_4K);
+		return CXL_RESOURCE_NONE;
+	}
+
+	offset = FIELD_GET(PCI_RCRB_CAP_LIST_ID_MASK, readw(addr + PCI_CAPABILITY_LIST));
+	cap_hdr = readl(addr + offset);
+	while ((FIELD_GET(PCI_RCRB_CAP_HDR_ID_MASK, cap_hdr)) != PCI_CAP_ID_EXP) {
+		offset = FIELD_GET(PCI_RCRB_CAP_HDR_NEXT_MASK, cap_hdr);
+		if (offset == 0 || offset > SZ_4K) {
+			offset = 0;
+			break;
+		}
+		cap_hdr = readl(addr + offset);
+	}
+	if (offset)
+		dport->rcrb.rcd_pcie_cap = offset;
+
+	iounmap(addr);
+	release_mem_region(rcrb, SZ_4K);
+
+	return offset;
+}
+
+int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev)
+{
+	void __iomem *dport_pcie_cap = NULL;
+	resource_size_t rcd_pcie_offset;
+	struct cxl_rcrb_info *ri;
+	struct cxl_dport *dport;
+	struct cxl_port *port;
+
+	port = cxl_pci_find_port(pdev, &dport);
+	if (!port)
+		return -EPROBE_DEFER;
+
+	cxl_rcrb_to_linkcap(&pdev->dev, dport);
+
+	ri = &dport->rcrb;
+	if (dport->rcrb.rcd_pcie_cap) {
+		rcd_pcie_offset = ri->base + ri->rcd_pcie_cap;
+		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev, rcd_pcie_offset,
+						      PCI_CAP_EXP_SIZEOF);
+	}
+
+	dport->regs.rcd_pcie_cap = dport_pcie_cap;
+	return 0;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_dport_map_rcd_linkcap, CXL);
+
 resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
 				    enum cxl_rcrb which)
 {
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 003feebab79b..b1fca98ddf8c 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -230,6 +230,14 @@ struct cxl_regs {
 	struct_group_tagged(cxl_rch_regs, rch_regs,
 		void __iomem *dport_aer;
 	);
+
+	/*
+	 * RCD upstream port specific PCIe cap register
+	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
+	 */
+	struct_group_tagged(cxl_rcd_regs, rcd_regs,
+		void __iomem *rcd_pcie_cap;
+	);
 };
 
 struct cxl_reg_map {
@@ -299,6 +307,7 @@ int cxl_setup_regs(struct cxl_register_map *map);
 struct cxl_dport;
 resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
 					   struct cxl_dport *dport);
+int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev);
 
 #define CXL_RESOURCE_NONE ((resource_size_t) -1)
 #define CXL_TARGET_STRLEN 20
@@ -646,6 +655,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
 
 struct cxl_rcrb_info {
 	resource_size_t base;
+	u16 rcd_pcie_cap;
 	u16 aer_cap;
 };
 
diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 2ff361e756d6..62343a5a39ff 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -512,10 +512,13 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
 	 * is an RCH and try to extract the Component Registers from
 	 * an RCRB.
 	 */
-	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
+	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
 		rc = cxl_rcrb_get_comp_regs(pdev, map);
+		if (rc)
+			return rc;
 
-	if (rc)
+		cxl_dport_map_rcd_linkcap(pdev);
+	} else if (rc)
 		return rc;
 
 	return cxl_setup_regs(map);
-- 
2.44.0

[PATCH v12 2/2] cxl/pci: Add sysfs attribute for CXL 1.1 device link status

[Kobayashi,Daisuke]

Add sysfs attribute for CXL 1.1 device link status to the cxl pci device.

In CXL1.1, the link status of the device is included in the RCRB mapped to
the memory mapped register area. Critically, that arrangement makes the
link status and control registers invisible to existing PCI user tooling.

Export those registers via sysfs with the expectation that PCI user
tooling will alternatively look for these sysfs files when attempting to
access to these CXL 1.1 endpoints registers.

Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
---
 drivers/cxl/pci.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 81 insertions(+)

diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
index 62343a5a39ff..3743e9d265c3 100644
--- a/drivers/cxl/pci.c
+++ b/drivers/cxl/pci.c
@@ -789,6 +789,86 @@ static int cxl_event_config(struct pci_host_bridge *host_bridge,
 	return 0;
 }
 
+static ssize_t rcd_pcie_cap_emit(struct device *dev, u16 offset, char *buf, size_t width)
+{
+	struct cxl_dev_state *cxlds = dev_get_drvdata(dev);
+	struct cxl_memdev *cxlmd = cxlds->cxlmd;
+	struct device *endpoint_parent;
+	struct cxl_dport *dport;
+	struct cxl_port *port;
+
+	port = cxl_mem_find_port(cxlmd, &dport);
+	if (!port)
+		return -EINVAL;
+
+	endpoint_parent = port->uport_dev;
+	if (!endpoint_parent)
+		return -ENXIO;
+
+	guard(device)(endpoint_parent);
+	if (!endpoint_parent->driver)
+		return -ENXIO;
+
+	if (dport->regs.rcd_pcie_cap == NULL)
+		return -EINVAL;
+
+	switch (width) {
+	case sizeof(u16):
+		return sysfs_emit(buf, "%x\n",
+				  readw(dport->regs.rcd_pcie_cap + offset));
+	case sizeof(u32):
+		return sysfs_emit(buf, "%x\n",
+				  readl(dport->regs.rcd_pcie_cap + offset));
+	default:
+		return -EINVAL;
+	}
+}
+
+static ssize_t rcd_link_cap_show(struct device *dev,
+				 struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKCAP, buf, sizeof(u32));
+}
+static DEVICE_ATTR_RO(rcd_link_cap);
+
+static ssize_t rcd_link_ctrl_show(struct device *dev,
+				  struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKCTL, buf, sizeof(u16));
+}
+static DEVICE_ATTR_RO(rcd_link_ctrl);
+
+static ssize_t rcd_link_status_show(struct device *dev,
+				    struct device_attribute *attr, char *buf)
+{
+	return rcd_pcie_cap_emit(dev, PCI_EXP_LNKSTA, buf, sizeof(u16));
+}
+static DEVICE_ATTR_RO(rcd_link_status);
+
+static struct attribute *cxl_rcd_attrs[] = {
+	&dev_attr_rcd_link_cap.attr,
+	&dev_attr_rcd_link_ctrl.attr,
+	&dev_attr_rcd_link_status.attr,
+	NULL
+};
+
+static umode_t cxl_rcd_visible(struct kobject *kobj, struct attribute *a, int n)
+{
+	struct device *dev = kobj_to_dev(kobj);
+	struct pci_dev *pdev = to_pci_dev(dev);
+
+	if (is_cxl_restricted(pdev))
+		return a->mode;
+
+	return 0;
+}
+
+static struct attribute_group cxl_rcd_group = {
+	.attrs = cxl_rcd_attrs,
+	.is_visible = cxl_rcd_visible,
+};
+__ATTRIBUTE_GROUPS(cxl_rcd);
+
 static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
 	struct pci_host_bridge *host_bridge = pci_find_host_bridge(pdev->bus);
@@ -972,6 +1052,7 @@ static struct pci_driver cxl_pci_driver = {
 	.id_table		= cxl_mem_pci_tbl,
 	.probe			= cxl_pci_probe,
 	.err_handler		= &cxl_error_handlers,
+	.dev_groups		= cxl_rcd_groups,
 	.driver	= {
 		.probe_type	= PROBE_PREFER_ASYNCHRONOUS,
 	},
-- 
2.44.0

Re: [PATCH v12 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Jonathan Cameron]

On Fri, 14 Jun 2024 13:56:10 +0900
"Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com> wrote:

> Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1 
> device link status information. By caching it, avoid the walking 
> memory map area to find the offset when output the register value.
> 
> Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
Hi.

A couple more change inline.  Problem with evolving code is that as a reviewer
I sometime forget to check my expectations.  This now stores a variable
in a core structure just to use it once a few lines later. No point in keeping it.
Looking at that made me notice the error returns were wrong and unhandled.

> ---
>  drivers/cxl/core/core.h |  6 ++++
>  drivers/cxl/core/regs.c | 62 +++++++++++++++++++++++++++++++++++++++++
>  drivers/cxl/cxl.h       | 10 +++++++
>  drivers/cxl/pci.c       |  7 +++--
>  4 files changed, 83 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 3b64fb1b9ed0..69006bde7ad5 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device *dev,
>  				    struct cxl_rcrb_info *ri,
>  				    enum cxl_rcrb which);
>  u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
> +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport);
> +
> +#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
> +#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
> +#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
> +#define PCI_CAP_EXP_SIZEOF		0x3c

I'd like a follow up patch making these defines apply for all usage
of these fields in the PCI core code.  Right now there are pointless
hard coded values.

>  
>  extern struct rw_semaphore cxl_dpa_rwsem;
>  extern struct rw_semaphore cxl_region_rwsem;
> diff --git a/drivers/cxl/core/regs.c b/drivers/cxl/core/regs.c
> index 372786f80955..96c2a289cfb7 100644
> --- a/drivers/cxl/core/regs.c
> +++ b/drivers/cxl/core/regs.c
> @@ -505,6 +505,68 @@ u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb)
>  	return offset;
>  }
>  
> +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct cxl_dport *dport)
> +{
> +	resource_size_t rcrb = dport->rcrb.base;
> +	void __iomem *addr;
> +	u32 cap_hdr;
> +	u16 offset;
> +
> +	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
> +		return CXL_RESOURCE_NONE;
> +
> +	addr = ioremap(rcrb, SZ_4K);
> +	if (!addr) {
> +		dev_err(dev, "Failed to map region %pr\n", addr);
> +		release_mem_region(rcrb, SZ_4K);
> +		return CXL_RESOURCE_NONE;

If you hit this then the value returned will be all fs.
We are treating and offset of 0 as the error return below.

> +	}
> +
> +	offset = FIELD_GET(PCI_RCRB_CAP_LIST_ID_MASK, readw(addr + PCI_CAPABILITY_LIST));
> +	cap_hdr = readl(addr + offset);
> +	while ((FIELD_GET(PCI_RCRB_CAP_HDR_ID_MASK, cap_hdr)) != PCI_CAP_ID_EXP) {
> +		offset = FIELD_GET(PCI_RCRB_CAP_HDR_NEXT_MASK, cap_hdr);
> +		if (offset == 0 || offset > SZ_4K) {
> +			offset = 0;

			offset = CXL_RESOURCE_NONE; 

> +			break;
> +		}
> +		cap_hdr = readl(addr + offset);
> +	}
> +	if (offset)
> +		dport->rcrb.rcd_pcie_cap = offset;

As below - this storage in the rcrb structure no longer has
any purpose. We can use the returned value from this function
to get to it in the one place it is used.



> +
> +	iounmap(addr);
> +	release_mem_region(rcrb, SZ_4K);
> +
> +	return offset;
> +}
> +
> +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev)
> +{
> +	void __iomem *dport_pcie_cap = NULL;
> +	resource_size_t rcd_pcie_offset;
> +	struct cxl_rcrb_info *ri;
> +	struct cxl_dport *dport;
> +	struct cxl_port *port;
> +
> +	port = cxl_pci_find_port(pdev, &dport);
> +	if (!port)
> +		return -EPROBE_DEFER;
> +
> +	cxl_rcrb_to_linkcap(&pdev->dev, dport);

Sorry - with all the changes I'd missed this previously.
There is no longer any point in storing the rcd_pcie_cap
in ri.  Just use the return value from the above to fill
a local variable and map that here.

	ri = &dport->rcrb;
	rcd_pcie_offset = cxl_rcrb_to_link_cap(&pdev->dev, dpor);
	if (rcd_pcie_offset != CXL_RESOURCE_NONE)
		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev,
						      ri->base + rcd_pcie_offset,
						      PCI_CAP_EXP_SIZEOF);

	dport->regs.rcd_pcie_cap = dport_pcie_cap;

	return 0;


> +
> +	ri = &dport->rcrb;
> +	if (dport->rcrb.rcd_pcie_cap) {
> +		rcd_pcie_offset = ri->base + ri->rcd_pcie_cap;
> +		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev, rcd_pcie_offset,
> +						      PCI_CAP_EXP_SIZEOF);
> +	}
> +
> +	dport->regs.rcd_pcie_cap = dport_pcie_cap;
> +	return 0;
> +}
> +EXPORT_SYMBOL_NS_GPL(cxl_dport_map_rcd_linkcap, CXL);
> +
>  resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
>  				    enum cxl_rcrb which)
>  {
> diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
> index 003feebab79b..b1fca98ddf8c 100644
> --- a/drivers/cxl/cxl.h
> +++ b/drivers/cxl/cxl.h
> @@ -230,6 +230,14 @@ struct cxl_regs {
>  	struct_group_tagged(cxl_rch_regs, rch_regs,
>  		void __iomem *dport_aer;
>  	);
> +
> +	/*
> +	 * RCD upstream port specific PCIe cap register
> +	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
> +	 */
> +	struct_group_tagged(cxl_rcd_regs, rcd_regs,
> +		void __iomem *rcd_pcie_cap;
> +	);
>  };
>  
>  struct cxl_reg_map {
> @@ -299,6 +307,7 @@ int cxl_setup_regs(struct cxl_register_map *map);
>  struct cxl_dport;
>  resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
>  					   struct cxl_dport *dport);
> +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev);
>  
>  #define CXL_RESOURCE_NONE ((resource_size_t) -1)
>  #define CXL_TARGET_STRLEN 20
> @@ -646,6 +655,7 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
>  
>  struct cxl_rcrb_info {
>  	resource_size_t base;
> +	u16 rcd_pcie_cap;

This is no longer needed I think after the suggested change above.

>  	u16 aer_cap;
>  };
>  
> diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> index 2ff361e756d6..62343a5a39ff 100644
> --- a/drivers/cxl/pci.c
> +++ b/drivers/cxl/pci.c
> @@ -512,10 +512,13 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev, enum cxl_regloc_type type,
>  	 * is an RCH and try to extract the Component Registers from
>  	 * an RCRB.
>  	 */
> -	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev))
> +	if (rc && type == CXL_REGLOC_RBI_COMPONENT && is_cxl_restricted(pdev)) {
>  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> +		if (rc)
> +			return rc;
>  
> -	if (rc)
> +		cxl_dport_map_rcd_linkcap(pdev);
> +	} else if (rc)
>  		return rc;
Style wise, I'd put {} round this second leg as well.  It's common to do
this when we have a mixture of single line and multiline legs.
There are several examples of this in this file already so we should
match local style.

>  
>  	return cxl_setup_regs(map);

RE: [PATCH v12 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Daisuke Kobayashi (Fujitsu)]

Jonathan Cameron wrote:
> On Fri, 14 Jun 2024 13:56:10 +0900
> "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com> wrote:
> 
> > Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1
> > device link status information. By caching it, avoid the walking
> > memory map area to find the offset when output the register value.
> >
> > Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>
> Hi.
> 
> A couple more change inline.  Problem with evolving code is that as a reviewer
> I sometime forget to check my expectations.  This now stores a variable in a
> core structure just to use it once a few lines later. No point in keeping it.
> Looking at that made me notice the error returns were wrong and unhandled.
> 
Thank you for your kind review. I can't post the next version this week, 
but let me check if my understanding is correct.

> > ---
> >  drivers/cxl/core/core.h |  6 ++++
> >  drivers/cxl/core/regs.c | 62
> +++++++++++++++++++++++++++++++++++++++++
> >  drivers/cxl/cxl.h       | 10 +++++++
> >  drivers/cxl/pci.c       |  7 +++--
> >  4 files changed, 83 insertions(+), 2 deletions(-)
> >
> > diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index
> > 3b64fb1b9ed0..69006bde7ad5 100644
> > --- a/drivers/cxl/core/core.h
> > +++ b/drivers/cxl/core/core.h
> > @@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device
> *dev,
> >  				    struct cxl_rcrb_info *ri,
> >  				    enum cxl_rcrb which);
> >  u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
> > +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct
> > +cxl_dport *dport);
> > +
> > +#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
> > +#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
> > +#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
> > +#define PCI_CAP_EXP_SIZEOF		0x3c
> 
> I'd like a follow up patch making these defines apply for all usage of these fields
> in the PCI core code.  Right now there are pointless hard coded values.
> 
Does this mean that you want me to post another patch later that 
defines PCI_CAP_EXP_SIZEOF to replace the hardcoded value in the PCI core code?
If so, I'll create a follow-up patch in the future.

> >
> >  extern struct rw_semaphore cxl_dpa_rwsem;  extern struct rw_semaphore
> > cxl_region_rwsem; diff --git a/drivers/cxl/core/regs.c
> > b/drivers/cxl/core/regs.c index 372786f80955..96c2a289cfb7 100644
> > --- a/drivers/cxl/core/regs.c
> > +++ b/drivers/cxl/core/regs.c
> > @@ -505,6 +505,68 @@ u16 cxl_rcrb_to_aer(struct device *dev,
> resource_size_t rcrb)
> >  	return offset;
> >  }
> >
> > +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct
> > +cxl_dport *dport) {
> > +	resource_size_t rcrb = dport->rcrb.base;
> > +	void __iomem *addr;
> > +	u32 cap_hdr;
> > +	u16 offset;
> > +
> > +	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
> > +		return CXL_RESOURCE_NONE;
> > +
> > +	addr = ioremap(rcrb, SZ_4K);
> > +	if (!addr) {
> > +		dev_err(dev, "Failed to map region %pr\n", addr);
> > +		release_mem_region(rcrb, SZ_4K);
> > +		return CXL_RESOURCE_NONE;
> 
> If you hit this then the value returned will be all fs.
> We are treating and offset of 0 as the error return below.
> 
I will change the return type to int and return 0 on error. Does this match your intent?

> > +	}
> > +
> > +	offset = FIELD_GET(PCI_RCRB_CAP_LIST_ID_MASK, readw(addr +
> PCI_CAPABILITY_LIST));
> > +	cap_hdr = readl(addr + offset);
> > +	while ((FIELD_GET(PCI_RCRB_CAP_HDR_ID_MASK, cap_hdr)) !=
> PCI_CAP_ID_EXP) {
> > +		offset = FIELD_GET(PCI_RCRB_CAP_HDR_NEXT_MASK,
> cap_hdr);
> > +		if (offset == 0 || offset > SZ_4K) {
> > +			offset = 0;
> 
> 			offset = CXL_RESOURCE_NONE;
> 
> > +			break;
> > +		}
> > +		cap_hdr = readl(addr + offset);
> > +	}
> > +	if (offset)
> > +		dport->rcrb.rcd_pcie_cap = offset;
> 
> As below - this storage in the rcrb structure no longer has any purpose. We can
> use the returned value from this function to get to it in the one place it is used.
> 
> 
> 
> > +
> > +	iounmap(addr);
> > +	release_mem_region(rcrb, SZ_4K);
> > +
> > +	return offset;
> > +}
> > +
> > +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev) {
> > +	void __iomem *dport_pcie_cap = NULL;
> > +	resource_size_t rcd_pcie_offset;
> > +	struct cxl_rcrb_info *ri;
> > +	struct cxl_dport *dport;
> > +	struct cxl_port *port;
> > +
> > +	port = cxl_pci_find_port(pdev, &dport);
> > +	if (!port)
> > +		return -EPROBE_DEFER;
> > +
> > +	cxl_rcrb_to_linkcap(&pdev->dev, dport);
> 
> Sorry - with all the changes I'd missed this previously.
> There is no longer any point in storing the rcd_pcie_cap in ri.  Just use the
> return value from the above to fill a local variable and map that here.
> 
I was too focused on fixing the issue. I'll remove the variable.

> 	ri = &dport->rcrb;
> 	rcd_pcie_offset = cxl_rcrb_to_link_cap(&pdev->dev, dpor);
> 	if (rcd_pcie_offset != CXL_RESOURCE_NONE)
> 		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev,
> 						      ri->base +
> rcd_pcie_offset,
> 
> PCI_CAP_EXP_SIZEOF);
> 
> 	dport->regs.rcd_pcie_cap = dport_pcie_cap;
> 
> 	return 0;
> 
> 
> > +
> > +	ri = &dport->rcrb;
> > +	if (dport->rcrb.rcd_pcie_cap) {
> > +		rcd_pcie_offset = ri->base + ri->rcd_pcie_cap;
> > +		dport_pcie_cap = devm_cxl_iomap_block(&pdev->dev,
> rcd_pcie_offset,
> > +
> PCI_CAP_EXP_SIZEOF);
> > +	}
> > +
> > +	dport->regs.rcd_pcie_cap = dport_pcie_cap;
> > +	return 0;
> > +}
> > +EXPORT_SYMBOL_NS_GPL(cxl_dport_map_rcd_linkcap, CXL);
> > +
> >  resource_size_t __rcrb_to_component(struct device *dev, struct
> cxl_rcrb_info *ri,
> >  				    enum cxl_rcrb which)
> >  {
> > diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index
> > 003feebab79b..b1fca98ddf8c 100644
> > --- a/drivers/cxl/cxl.h
> > +++ b/drivers/cxl/cxl.h
> > @@ -230,6 +230,14 @@ struct cxl_regs {
> >  	struct_group_tagged(cxl_rch_regs, rch_regs,
> >  		void __iomem *dport_aer;
> >  	);
> > +
> > +	/*
> > +	 * RCD upstream port specific PCIe cap register
> > +	 * @pcie_cap: CXL 3.0 8.2.1.2 RCD Upstream Port RCRB
> > +	 */
> > +	struct_group_tagged(cxl_rcd_regs, rcd_regs,
> > +		void __iomem *rcd_pcie_cap;
> > +	);
> >  };
> >
> >  struct cxl_reg_map {
> > @@ -299,6 +307,7 @@ int cxl_setup_regs(struct cxl_register_map *map);
> > struct cxl_dport;  resource_size_t cxl_rcd_component_reg_phys(struct
> > device *dev,
> >  					   struct cxl_dport *dport);
> > +int cxl_dport_map_rcd_linkcap(struct pci_dev *pdev);
> >
> >  #define CXL_RESOURCE_NONE ((resource_size_t) -1)  #define
> > CXL_TARGET_STRLEN 20 @@ -646,6 +655,7 @@
> cxl_find_dport_by_dev(struct
> > cxl_port *port, const struct device *dport_dev)
> >
> >  struct cxl_rcrb_info {
> >  	resource_size_t base;
> > +	u16 rcd_pcie_cap;
> 
> This is no longer needed I think after the suggested change above.
> 
> >  	u16 aer_cap;
> >  };
> >
> > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c index
> > 2ff361e756d6..62343a5a39ff 100644
> > --- a/drivers/cxl/pci.c
> > +++ b/drivers/cxl/pci.c
> > @@ -512,10 +512,13 @@ static int cxl_pci_setup_regs(struct pci_dev *pdev,
> enum cxl_regloc_type type,
> >  	 * is an RCH and try to extract the Component Registers from
> >  	 * an RCRB.
> >  	 */
> > -	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> is_cxl_restricted(pdev))
> > +	if (rc && type == CXL_REGLOC_RBI_COMPONENT &&
> > +is_cxl_restricted(pdev)) {
> >  		rc = cxl_rcrb_get_comp_regs(pdev, map);
> > +		if (rc)
> > +			return rc;
> >
> > -	if (rc)
> > +		cxl_dport_map_rcd_linkcap(pdev);
> > +	} else if (rc)
> >  		return rc;
> Style wise, I'd put {} round this second leg as well.  It's common to do this
> when we have a mixture of single line and multiline legs.
> There are several examples of this in this file already so we should match local
> style.
> 
Thank you. I'll fix it

> >
> >  	return cxl_setup_regs(map);

Re: [PATCH v12 1/2] cxl/core/regs: Add rcd_pcie_cap initialization

[Jonathan Cameron]

On Fri, 14 Jun 2024 09:51:55 +0000
"Daisuke Kobayashi (Fujitsu)" <kobayashi.da-06@fujitsu.com> wrote:

> Jonathan Cameron wrote:
> > On Fri, 14 Jun 2024 13:56:10 +0900
> > "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com> wrote:
> >   
> > > Add rcd_pcie_cap and its initialization to cache the offset of cxl1.1
> > > device link status information. By caching it, avoid the walking
> > > memory map area to find the offset when output the register value.
> > >
> > > Signed-off-by: "Kobayashi,Daisuke" <kobayashi.da-06@fujitsu.com>  
> > Hi.
> > 
> > A couple more change inline.  Problem with evolving code is that as a reviewer
> > I sometime forget to check my expectations.  This now stores a variable in a
> > core structure just to use it once a few lines later. No point in keeping it.
> > Looking at that made me notice the error returns were wrong and unhandled.
> >   
> Thank you for your kind review. I can't post the next version this week, 
> but let me check if my understanding is correct.

> 
> > > ---
> > >  drivers/cxl/core/core.h |  6 ++++
> > >  drivers/cxl/core/regs.c | 62  
> > +++++++++++++++++++++++++++++++++++++++++  
> > >  drivers/cxl/cxl.h       | 10 +++++++
> > >  drivers/cxl/pci.c       |  7 +++--
> > >  4 files changed, 83 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index
> > > 3b64fb1b9ed0..69006bde7ad5 100644
> > > --- a/drivers/cxl/core/core.h
> > > +++ b/drivers/cxl/core/core.h
> > > @@ -74,6 +74,12 @@ resource_size_t __rcrb_to_component(struct device  
> > *dev,  
> > >  				    struct cxl_rcrb_info *ri,
> > >  				    enum cxl_rcrb which);
> > >  u16 cxl_rcrb_to_aer(struct device *dev, resource_size_t rcrb);
> > > +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct
> > > +cxl_dport *dport);
> > > +
> > > +#define PCI_RCRB_CAP_LIST_ID_MASK	GENMASK(7, 0)
> > > +#define PCI_RCRB_CAP_HDR_ID_MASK	GENMASK(7, 0)
> > > +#define PCI_RCRB_CAP_HDR_NEXT_MASK	GENMASK(15, 8)
> > > +#define PCI_CAP_EXP_SIZEOF		0x3c  
> > 
> > I'd like a follow up patch making these defines apply for all usage of these fields
> > in the PCI core code.  Right now there are pointless hard coded values.
> >   
> Does this mean that you want me to post another patch later that 
> defines PCI_CAP_EXP_SIZEOF to replace the hardcoded value in the PCI core code?
> If so, I'll create a follow-up patch in the future.

Yes but also include the masks for the fields of a PCI capability.
So definitely the last 3 defines. For the first one is the PCIExpress
Capabilities pointer mask, but it is carefully constructed
to line up with the pointers in the PCI capabilities themselves
so you can probably share a define for the first 2.



> 
> > >
> > >  extern struct rw_semaphore cxl_dpa_rwsem;  extern struct rw_semaphore
> > > cxl_region_rwsem; diff --git a/drivers/cxl/core/regs.c
> > > b/drivers/cxl/core/regs.c index 372786f80955..96c2a289cfb7 100644
> > > --- a/drivers/cxl/core/regs.c
> > > +++ b/drivers/cxl/core/regs.c
> > > @@ -505,6 +505,68 @@ u16 cxl_rcrb_to_aer(struct device *dev,  
> > resource_size_t rcrb)  
> > >  	return offset;
> > >  }
> > >
> > > +resource_size_t cxl_rcrb_to_linkcap(struct device *dev, struct
> > > +cxl_dport *dport) {
> > > +	resource_size_t rcrb = dport->rcrb.base;
> > > +	void __iomem *addr;
> > > +	u32 cap_hdr;
> > > +	u16 offset;
> > > +
> > > +	if (!request_mem_region(rcrb, SZ_4K, "CXL RCRB"))
> > > +		return CXL_RESOURCE_NONE;
> > > +
> > > +	addr = ioremap(rcrb, SZ_4K);
> > > +	if (!addr) {
> > > +		dev_err(dev, "Failed to map region %pr\n", addr);
> > > +		release_mem_region(rcrb, SZ_4K);
> > > +		return CXL_RESOURCE_NONE;  
> > 
> > If you hit this then the value returned will be all fs.
> > We are treating and offset of 0 as the error return below.
> >   
> I will change the return type to int and return 0 on error. Does this match your intent?

I was suggesting returning CXL_RESOURCE_NONE on error rather than 0
0 is normally success in Linux, you could have used an int and returned < 0
for the error paths -EINVAL or similar.  I don't really mind which
you chose between that and returning CXL_RESOURCE_NONE in all the failure
paths.


Jonathan