[PATCH 0/3] PCI: dwc: Add ECAM support with iATU configuration

[PATCH 0/3] PCI: dwc: Add ECAM support with iATU configuration

[Krishna chaitanya chundru]

The current implementation requires iATU for every configuration
space access which increases latency & cpu utilization.

Configuring iATU in config shift mode enables ECAM feature to access the
config space, which avoids iATU configuration for every config access.

Add cfg_shft_mode into struct dw_pcie_ob_atu_cfg to enable config shift mode.

As DBI comes under config space, this avoids remapping of DBI space
separately. Instead, it uses the mapped config space address returned from
ECAM initialization. Change the order of dw_pcie_get_resources() execution
to acheive this.

Introduce new ecam_init() function op for the clients to use configure
after ecam init has done.

Enable the ECAM feature if the config space size is equal to size required
to represent number of buses in the bus range property.

The ELBI and iATU registers also fall after the DBI space, so use the cfg
win returned from the ecam init to map these regions instead of doing the
ioremap again. iATU starts after 4KB of dbi address and ELBI starts at
offset 0xf20 from dbi.

On bus 0, we have only the root complex. Any access other than that should
not go out of the link and should return all F's. Since the IATU is
configured for bus 1 onwards, block the transactions for bus 0:0:1 to
0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
link through ecam blocker through parf registers.

Increase the configuration size to 256MB as required by the ECAM feature
and also move config space, dbi, iatu to upper space and use lower space
entirely for BAR region.

Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
---
---
Krishna chaitanya chundru (3):
      arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature
      PCI: dwc: Add ECAM support with iATU configuration
      PCI: qcom: Enable ECAM feature based on config size

 arch/arm64/boot/dts/qcom/sc7280.dtsi              |  12 +--
 drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
 drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
 drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
 drivers/pci/controller/dwc/pcie-qcom.c            | 104 +++++++++++++++++++-
 5 files changed, 208 insertions(+), 30 deletions(-)
---
base-commit: 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
change-id: 20241016-ecam-b1f5febcfc3c

Best regards,
-- 
Krishna chaitanya chundru <quic_krichai@quicinc.com>

[PATCH 1/3] arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature

[Krishna chaitanya chundru]

Increase the configuration size to 256MB as required by the ECAM feature.
And also move config space, DBI, ELBI, IATU to upper PCIe region and use
lower PCIe region entierly for BAR region.

Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
---
 arch/arm64/boot/dts/qcom/sc7280.dtsi | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi
index 3d8410683402..a7e3d3e9d034 100644
--- a/arch/arm64/boot/dts/qcom/sc7280.dtsi
+++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi
@@ -2196,10 +2196,10 @@ wifi: wifi@17a10040 {
 		pcie1: pcie@1c08000 {
 			compatible = "qcom,pcie-sc7280";
 			reg = <0 0x01c08000 0 0x3000>,
-			      <0 0x40000000 0 0xf1d>,
-			      <0 0x40000f20 0 0xa8>,
-			      <0 0x40001000 0 0x1000>,
-			      <0 0x40100000 0 0x100000>;
+			      <4 0x00000000 0 0xf1d>,
+			      <4 0x00000f20 0 0xa8>,
+			      <4 0x10000000 0 0x1000>,
+			      <4 0x00000000 0 0x10000000>;
 
 			reg-names = "parf", "dbi", "elbi", "atu", "config";
 			device_type = "pci";
@@ -2210,8 +2210,8 @@ pcie1: pcie@1c08000 {
 			#address-cells = <3>;
 			#size-cells = <2>;
 
-			ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>,
-				 <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>;
+			ranges = <0x01000000 0x0 0x00000000 0x0 0x40000000 0x0 0x100000>,
+				 <0x02000000 0x0 0x40100000 0x0 0x40100000 0x0 0x1ff00000>;
 
 			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>,
 				     <GIC_SPI 308 IRQ_TYPE_LEVEL_HIGH>,

-- 
2.34.1

Re: [PATCH 1/3] arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature

[Manivannan Sadhasivam]

On Sun, Nov 17, 2024 at 03:30:18AM +0530, Krishna chaitanya chundru wrote:
> Increase the configuration size to 256MB as required by the ECAM feature.
> And also move config space, DBI, ELBI, IATU to upper PCIe region and use
> lower PCIe region entierly for BAR region.
> 

Is this change compatible with old kernels before commit '10ba0854c5e6 ("PCI:
qcom: Disable mirroring of DBI and iATU register space in BAR region")'?

- Mani

> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> ---
>  arch/arm64/boot/dts/qcom/sc7280.dtsi | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> index 3d8410683402..a7e3d3e9d034 100644
> --- a/arch/arm64/boot/dts/qcom/sc7280.dtsi
> +++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> @@ -2196,10 +2196,10 @@ wifi: wifi@17a10040 {
>  		pcie1: pcie@1c08000 {
>  			compatible = "qcom,pcie-sc7280";
>  			reg = <0 0x01c08000 0 0x3000>,
> -			      <0 0x40000000 0 0xf1d>,
> -			      <0 0x40000f20 0 0xa8>,
> -			      <0 0x40001000 0 0x1000>,
> -			      <0 0x40100000 0 0x100000>;
> +			      <4 0x00000000 0 0xf1d>,
> +			      <4 0x00000f20 0 0xa8>,
> +			      <4 0x10000000 0 0x1000>,
> +			      <4 0x00000000 0 0x10000000>;
>  
>  			reg-names = "parf", "dbi", "elbi", "atu", "config";
>  			device_type = "pci";
> @@ -2210,8 +2210,8 @@ pcie1: pcie@1c08000 {
>  			#address-cells = <3>;
>  			#size-cells = <2>;
>  
> -			ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>,
> -				 <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>;
> +			ranges = <0x01000000 0x0 0x00000000 0x0 0x40000000 0x0 0x100000>,
> +				 <0x02000000 0x0 0x40100000 0x0 0x40100000 0x0 0x1ff00000>;
>  
>  			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>,
>  				     <GIC_SPI 308 IRQ_TYPE_LEVEL_HIGH>,
> 
> -- 
> 2.34.1
> 

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH 1/3] arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature

[Krishna Chaitanya Chundru]

On 12/2/2024 8:36 PM, Manivannan Sadhasivam wrote:
> On Sun, Nov 17, 2024 at 03:30:18AM +0530, Krishna chaitanya chundru wrote:
>> Increase the configuration size to 256MB as required by the ECAM feature.
>> And also move config space, DBI, ELBI, IATU to upper PCIe region and use
>> lower PCIe region entierly for BAR region.
>>
> 
> Is this change compatible with old kernels before commit '10ba0854c5e6 ("PCI:
> qcom: Disable mirroring of DBI and iATU register space in BAR region")'?
> 
> - Mani
No mani, we need this commit '10ba0854c5e6 ("PCI:
 > qcom: Disable mirroring of DBI and iATU register space in BAR region")
for this.

- Krishna Chaitanya.
> 
>> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
>> ---
>>   arch/arm64/boot/dts/qcom/sc7280.dtsi | 12 ++++++------
>>   1 file changed, 6 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi
>> index 3d8410683402..a7e3d3e9d034 100644
>> --- a/arch/arm64/boot/dts/qcom/sc7280.dtsi
>> +++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi
>> @@ -2196,10 +2196,10 @@ wifi: wifi@17a10040 {
>>   		pcie1: pcie@1c08000 {
>>   			compatible = "qcom,pcie-sc7280";
>>   			reg = <0 0x01c08000 0 0x3000>,
>> -			      <0 0x40000000 0 0xf1d>,
>> -			      <0 0x40000f20 0 0xa8>,
>> -			      <0 0x40001000 0 0x1000>,
>> -			      <0 0x40100000 0 0x100000>;
>> +			      <4 0x00000000 0 0xf1d>,
>> +			      <4 0x00000f20 0 0xa8>,
>> +			      <4 0x10000000 0 0x1000>,
>> +			      <4 0x00000000 0 0x10000000>;
>>   
>>   			reg-names = "parf", "dbi", "elbi", "atu", "config";
>>   			device_type = "pci";
>> @@ -2210,8 +2210,8 @@ pcie1: pcie@1c08000 {
>>   			#address-cells = <3>;
>>   			#size-cells = <2>;
>>   
>> -			ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>,
>> -				 <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>;
>> +			ranges = <0x01000000 0x0 0x00000000 0x0 0x40000000 0x0 0x100000>,
>> +				 <0x02000000 0x0 0x40100000 0x0 0x40100000 0x0 0x1ff00000>;
>>   
>>   			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>,
>>   				     <GIC_SPI 308 IRQ_TYPE_LEVEL_HIGH>,
>>
>> -- 
>> 2.34.1
>>
> 

Re: [PATCH 1/3] arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature

[Konrad Dybcio]

On 16.11.2024 11:00 PM, Krishna chaitanya chundru wrote:
> Increase the configuration size to 256MB as required by the ECAM feature.
> And also move config space, DBI, ELBI, IATU to upper PCIe region and use
> lower PCIe region entierly for BAR region.
> 
> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> ---
>  arch/arm64/boot/dts/qcom/sc7280.dtsi | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> index 3d8410683402..a7e3d3e9d034 100644
> --- a/arch/arm64/boot/dts/qcom/sc7280.dtsi
> +++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> @@ -2196,10 +2196,10 @@ wifi: wifi@17a10040 {
>  		pcie1: pcie@1c08000 {
>  			compatible = "qcom,pcie-sc7280";
>  			reg = <0 0x01c08000 0 0x3000>,
> -			      <0 0x40000000 0 0xf1d>,
> -			      <0 0x40000f20 0 0xa8>,
> -			      <0 0x40001000 0 0x1000>,
> -			      <0 0x40100000 0 0x100000>;
> +			      <4 0x00000000 0 0xf1d>,
> +			      <4 0x00000f20 0 0xa8>,
> +			      <4 0x10000000 0 0x1000>,
> +			      <4 0x00000000 0 0x10000000>;

So this region is far bigger, any reason to use 256MiB specifically?

Konrad
>  
>  			reg-names = "parf", "dbi", "elbi", "atu", "config";
>  			device_type = "pci";
> @@ -2210,8 +2210,8 @@ pcie1: pcie@1c08000 {
>  			#address-cells = <3>;
>  			#size-cells = <2>;
>  
> -			ranges = <0x01000000 0x0 0x00000000 0x0 0x40200000 0x0 0x100000>,
> -				 <0x02000000 0x0 0x40300000 0x0 0x40300000 0x0 0x1fd00000>;
> +			ranges = <0x01000000 0x0 0x00000000 0x0 0x40000000 0x0 0x100000>,
> +				 <0x02000000 0x0 0x40100000 0x0 0x40100000 0x0 0x1ff00000>;
>  
>  			interrupts = <GIC_SPI 307 IRQ_TYPE_LEVEL_HIGH>,
>  				     <GIC_SPI 308 IRQ_TYPE_LEVEL_HIGH>,
> 

Re: [PATCH 1/3] arm64: dts: qcom: sc7280: Increase config size to 256MB for ECAM feature

[Bjorn Helgaas]

On Thu, Dec 05, 2024 at 05:11:25PM +0100, Konrad Dybcio wrote:
> On 16.11.2024 11:00 PM, Krishna chaitanya chundru wrote:
> > Increase the configuration size to 256MB as required by the ECAM feature.
> > And also move config space, DBI, ELBI, IATU to upper PCIe region and use
> > lower PCIe region entierly for BAR region.
> > 
> > Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> > ---
> >  arch/arm64/boot/dts/qcom/sc7280.dtsi | 12 ++++++------
> >  1 file changed, 6 insertions(+), 6 deletions(-)
> > 
> > diff --git a/arch/arm64/boot/dts/qcom/sc7280.dtsi b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> > index 3d8410683402..a7e3d3e9d034 100644
> > --- a/arch/arm64/boot/dts/qcom/sc7280.dtsi
> > +++ b/arch/arm64/boot/dts/qcom/sc7280.dtsi
> > @@ -2196,10 +2196,10 @@ wifi: wifi@17a10040 {
> >  		pcie1: pcie@1c08000 {
> >  			compatible = "qcom,pcie-sc7280";
> >  			reg = <0 0x01c08000 0 0x3000>,
> > -			      <0 0x40000000 0 0xf1d>,
> > -			      <0 0x40000f20 0 0xa8>,
> > -			      <0 0x40001000 0 0x1000>,
> > -			      <0 0x40100000 0 0x100000>;
> > +			      <4 0x00000000 0 0xf1d>,
> > +			      <4 0x00000f20 0 0xa8>,
> > +			      <4 0x10000000 0 0x1000>,
> > +			      <4 0x00000000 0 0x10000000>;
> 
> So this region is far bigger, any reason to use 256MiB specifically?

I assume this is because ECAM takes 1MB/bus, and pcie1 has
bus-range = <0x00 0xff>.  If one wanted a smaller ECAM area,
I assume one would limit bus-range to match.

Bjorn

[PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Krishna chaitanya chundru]

The current implementation requires iATU for every configuration
space access which increases latency & cpu utilization.

Configuring iATU in config shift mode enables ECAM feature to access the
config space, which avoids iATU configuration for every config access.

Add "ctrl2" into struct dw_pcie_ob_atu_cfg  to enable config shift mode.

As DBI comes under config space, this avoids remapping of DBI space
separately. Instead, it uses the mapped config space address returned from
ECAM initialization. Change the order of dw_pcie_get_resources() execution
to acheive this.

Introduce new ecam_init() function op for the clients to configure after
ecam window creation has been done.

Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
---
 drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
 drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
 drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
 3 files changed, 102 insertions(+), 20 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
index 3e41865c7290..e98cc841a2a9 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -418,6 +418,62 @@ static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
 	}
 }
 
+static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct dw_pcie_ob_atu_cfg atu = {0};
+	struct resource_entry *bus;
+	int ret, bus_range_max;
+
+	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
+
+	/*
+	 * Bus 1 config space needs type 0 atu configuration
+	 * Remaining buses need type 1 atu configuration
+	 */
+	atu.index = 0;
+	atu.type = PCIE_ATU_TYPE_CFG0;
+	atu.cpu_addr = pp->cfg0_base + SZ_1M;
+	atu.size = SZ_1M;
+	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
+	ret = dw_pcie_prog_outbound_atu(pci, &atu);
+	if (ret)
+		return ret;
+
+	bus_range_max = bus->res->end - bus->res->start + 1;
+
+	/* Configure for bus 2 - bus_range_max in type 1 */
+	atu.index = 1;
+	atu.type = PCIE_ATU_TYPE_CFG1;
+	atu.cpu_addr = pp->cfg0_base + SZ_2M;
+	atu.size = (SZ_1M * (bus_range_max - 2));
+	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
+	return dw_pcie_prog_outbound_atu(pci, &atu);
+}
+
+static int dw_pcie_create_ecam_window(struct dw_pcie_rp *pp, struct resource *res)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct device *dev = pci->dev;
+	struct resource_entry *bus;
+
+	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
+	if (!bus)
+		return -ENODEV;
+
+	pp->cfg = pci_ecam_create(dev, res, bus->res, &pci_generic_ecam_ops);
+	if (IS_ERR(pp->cfg))
+		return PTR_ERR(pp->cfg);
+
+	pci->dbi_base = pp->cfg->win;
+	pci->dbi_phys_addr = res->start;
+
+	if (pp->ops->ecam_init)
+		pp->ops->ecam_init(pci, pp->cfg);
+
+	return 0;
+}
+
 int dw_pcie_host_init(struct dw_pcie_rp *pp)
 {
 	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
@@ -431,19 +487,8 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 
 	raw_spin_lock_init(&pp->lock);
 
-	ret = dw_pcie_get_resources(pci);
-	if (ret)
-		return ret;
-
 	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
-	if (res) {
-		pp->cfg0_size = resource_size(res);
-		pp->cfg0_base = res->start;
-
-		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
-		if (IS_ERR(pp->va_cfg0_base))
-			return PTR_ERR(pp->va_cfg0_base);
-	} else {
+	if (!res) {
 		dev_err(dev, "Missing *config* reg space\n");
 		return -ENODEV;
 	}
@@ -454,6 +499,30 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 
 	pp->bridge = bridge;
 
+	pp->cfg0_size = resource_size(res);
+	pp->cfg0_base = res->start;
+
+	if (!pp->enable_ecam) {
+		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
+		if (IS_ERR(pp->va_cfg0_base))
+			return PTR_ERR(pp->va_cfg0_base);
+
+		/* Set default bus ops */
+		bridge->ops = &dw_pcie_ops;
+		bridge->child_ops = &dw_child_pcie_ops;
+		bridge->sysdata = pp;
+	} else {
+		ret = dw_pcie_create_ecam_window(pp, res);
+		if (ret)
+			return ret;
+		bridge->ops = (struct pci_ops *)&pci_generic_ecam_ops.pci_ops;
+		pp->bridge->sysdata = pp->cfg;
+	}
+
+	ret = dw_pcie_get_resources(pci);
+	if (ret)
+		goto err_free_ecam;
+
 	/* Get the I/O range from DT */
 	win = resource_list_first_type(&bridge->windows, IORESOURCE_IO);
 	if (win) {
@@ -462,14 +531,10 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 		pp->io_base = pci_pio_to_address(win->res->start);
 	}
 
-	/* Set default bus ops */
-	bridge->ops = &dw_pcie_ops;
-	bridge->child_ops = &dw_child_pcie_ops;
-
 	if (pp->ops->init) {
 		ret = pp->ops->init(pp);
 		if (ret)
-			return ret;
+			goto err_free_ecam;
 	}
 
 	if (pci_msi_enabled()) {
@@ -504,6 +569,12 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 
 	dw_pcie_iatu_detect(pci);
 
+	if (pp->enable_ecam) {
+		ret = dw_pcie_config_ecam_iatu(pp);
+		if (ret)
+			goto err_free_msi;
+	}
+
 	/*
 	 * Allocate the resource for MSG TLP before programming the iATU
 	 * outbound window in dw_pcie_setup_rc(). Since the allocation depends
@@ -533,8 +604,6 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 	/* Ignore errors, the link may come up later */
 	dw_pcie_wait_for_link(pci);
 
-	bridge->sysdata = pp;
-
 	ret = pci_host_probe(bridge);
 	if (ret)
 		goto err_stop_link;
@@ -558,6 +627,10 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
 	if (pp->ops->deinit)
 		pp->ops->deinit(pp);
 
+err_free_ecam:
+	if (pp->cfg)
+		pci_ecam_free(pp->cfg);
+
 	return ret;
 }
 EXPORT_SYMBOL_GPL(dw_pcie_host_init);
@@ -578,6 +651,9 @@ void dw_pcie_host_deinit(struct dw_pcie_rp *pp)
 
 	if (pp->ops->deinit)
 		pp->ops->deinit(pp);
+
+	if (pp->cfg)
+		pci_ecam_free(pp->cfg);
 }
 EXPORT_SYMBOL_GPL(dw_pcie_host_deinit);
 
diff --git a/drivers/pci/controller/dwc/pcie-designware.c b/drivers/pci/controller/dwc/pcie-designware.c
index 6d6cbc8b5b2c..63d36676f858 100644
--- a/drivers/pci/controller/dwc/pcie-designware.c
+++ b/drivers/pci/controller/dwc/pcie-designware.c
@@ -509,7 +509,7 @@ int dw_pcie_prog_outbound_atu(struct dw_pcie *pci,
 		val = dw_pcie_enable_ecrc(val);
 	dw_pcie_writel_atu_ob(pci, atu->index, PCIE_ATU_REGION_CTRL1, val);
 
-	val = PCIE_ATU_ENABLE;
+	val = PCIE_ATU_ENABLE | atu->ctrl2;
 	if (atu->type == PCIE_ATU_TYPE_MSG) {
 		/* The data-less messages only for now */
 		val |= PCIE_ATU_INHIBIT_PAYLOAD | atu->code;
diff --git a/drivers/pci/controller/dwc/pcie-designware.h b/drivers/pci/controller/dwc/pcie-designware.h
index 347ab74ac35a..33afa91b402c 100644
--- a/drivers/pci/controller/dwc/pcie-designware.h
+++ b/drivers/pci/controller/dwc/pcie-designware.h
@@ -20,6 +20,7 @@
 #include <linux/irq.h>
 #include <linux/msi.h>
 #include <linux/pci.h>
+#include <linux/pci-ecam.h>
 #include <linux/reset.h>
 
 #include <linux/pci-epc.h>
@@ -171,6 +172,7 @@
 #define PCIE_ATU_REGION_CTRL2		0x004
 #define PCIE_ATU_ENABLE			BIT(31)
 #define PCIE_ATU_BAR_MODE_ENABLE	BIT(30)
+#define PCIE_ATU_CFG_SHIFT_MODE_ENABLE	BIT(28)
 #define PCIE_ATU_INHIBIT_PAYLOAD	BIT(22)
 #define PCIE_ATU_FUNC_NUM_MATCH_EN      BIT(19)
 #define PCIE_ATU_LOWER_BASE		0x008
@@ -342,6 +344,7 @@ struct dw_pcie_ob_atu_cfg {
 	u8 func_no;
 	u8 code;
 	u8 routing;
+	u32 ctrl2;
 	u64 cpu_addr;
 	u64 pci_addr;
 	u64 size;
@@ -353,6 +356,7 @@ struct dw_pcie_host_ops {
 	void (*post_init)(struct dw_pcie_rp *pp);
 	int (*msi_init)(struct dw_pcie_rp *pp);
 	void (*pme_turn_off)(struct dw_pcie_rp *pp);
+	int (*ecam_init)(struct dw_pcie *pcie, struct pci_config_window *cfg);
 };
 
 struct dw_pcie_rp {
@@ -379,6 +383,8 @@ struct dw_pcie_rp {
 	bool			use_atu_msg;
 	int			msg_atu_index;
 	struct resource		*msg_res;
+	bool			enable_ecam;
+	struct pci_config_window *cfg;
 };
 
 struct dw_pcie_ep_ops {

-- 
2.34.1

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[kernel test robot]

Hi Krishna,

kernel test robot noticed the following build errors:

[auto build test ERROR on 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2]

url:    https://github.com/intel-lab-lkp/linux/commits/Krishna-chaitanya-chundru/arm64-dts-qcom-sc7280-Increase-config-size-to-256MB-for-ECAM-feature/20241121-095614
base:   2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
patch link:    https://lore.kernel.org/r/20241117-ecam-v1-2-6059faf38d07%40quicinc.com
patch subject: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration
config: i386-buildonly-randconfig-005-20241121 (https://download.01.org/0day-ci/archive/20241121/202411212032.CbThzjSv-lkp@intel.com/config)
compiler: clang version 19.1.3 (https://github.com/llvm/llvm-project ab51eccf88f5321e7c60591c5546b254b6afab99)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241121/202411212032.CbThzjSv-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202411212032.CbThzjSv-lkp@intel.com/

All errors (new ones prefixed by >>):

>> ld.lld: error: undefined symbol: pci_generic_ecam_ops
   >>> referenced by pcie-designware-host.c
   >>>               drivers/pci/controller/dwc/pcie-designware-host.o:(dw_pcie_host_init) in archive vmlinux.a
   >>> referenced by pcie-designware-host.c
   >>>               drivers/pci/controller/dwc/pcie-designware-host.o:(dw_pcie_host_init) in archive vmlinux.a
--
>> ld.lld: error: undefined symbol: pci_ecam_create
   >>> referenced by pcie-designware-host.c
   >>>               drivers/pci/controller/dwc/pcie-designware-host.o:(dw_pcie_host_init) in archive vmlinux.a
--
>> ld.lld: error: undefined symbol: pci_ecam_free
   >>> referenced by pcie-designware-host.c
   >>>               drivers/pci/controller/dwc/pcie-designware-host.o:(dw_pcie_host_init) in archive vmlinux.a
   >>> referenced by pcie-designware-host.c
   >>>               drivers/pci/controller/dwc/pcie-designware-host.o:(dw_pcie_host_deinit) in archive vmlinux.a

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[kernel test robot]

Hi Krishna,

kernel test robot noticed the following build errors:

[auto build test ERROR on 2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2]

url:    https://github.com/intel-lab-lkp/linux/commits/Krishna-chaitanya-chundru/arm64-dts-qcom-sc7280-Increase-config-size-to-256MB-for-ECAM-feature/20241121-095614
base:   2f87d0916ce0d2925cedbc9e8f5d6291ba2ac7b2
patch link:    https://lore.kernel.org/r/20241117-ecam-v1-2-6059faf38d07%40quicinc.com
patch subject: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration
config: alpha-randconfig-r064-20241121 (https://download.01.org/0day-ci/archive/20241122/202411220541.dAciinyb-lkp@intel.com/config)
compiler: alpha-linux-gcc (GCC) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241122/202411220541.dAciinyb-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202411220541.dAciinyb-lkp@intel.com/

All errors (new ones prefixed by >>):

   alpha-linux-ld: drivers/pci/controller/dwc/pcie-designware-host.o: in function `dw_pcie_host_deinit':
>> (.text+0x17a4): undefined reference to `pci_ecam_free'
>> alpha-linux-ld: (.text+0x17a8): undefined reference to `pci_ecam_free'
   alpha-linux-ld: drivers/pci/controller/dwc/pcie-designware-host.o: in function `dw_pcie_host_init':
>> (.text+0x21e4): undefined reference to `pci_generic_ecam_ops'
>> alpha-linux-ld: (.text+0x21e8): undefined reference to `pci_ecam_create'
   alpha-linux-ld: (.text+0x21f0): undefined reference to `pci_ecam_create'
>> alpha-linux-ld: (.text+0x2240): undefined reference to `pci_generic_ecam_ops'
   alpha-linux-ld: (.text+0x2834): undefined reference to `pci_ecam_free'
   alpha-linux-ld: (.text+0x2838): undefined reference to `pci_ecam_free'

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Manivannan Sadhasivam]

On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
> The current implementation requires iATU for every configuration
> space access which increases latency & cpu utilization.
> 
> Configuring iATU in config shift mode enables ECAM feature to access the

Can you please elaborate 'config shift mode'? Quote relevant section in DWC
databook for reference.

> config space, which avoids iATU configuration for every config access.
> 
> Add "ctrl2" into struct dw_pcie_ob_atu_cfg  to enable config shift mode.
> 
> As DBI comes under config space, this avoids remapping of DBI space
> separately. Instead, it uses the mapped config space address returned from
> ECAM initialization. Change the order of dw_pcie_get_resources() execution
> to acheive this.
> 
> Introduce new ecam_init() function op for the clients to configure after

We use 'DWC glue drivers' to refer the 'clients' of this driver.

> ecam window creation has been done.
> 

Use 'ECAM' everywhere.

> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
>  drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
>  drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
>  3 files changed, 102 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 3e41865c7290..e98cc841a2a9 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -418,6 +418,62 @@ static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
>  	}
>  }
>  
> +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = {0};
> +	struct resource_entry *bus;
> +	int ret, bus_range_max;
> +
> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> +
> +	/*
> +	 * Bus 1 config space needs type 0 atu configuration
> +	 * Remaining buses need type 1 atu configuration
> +	 */
> +	atu.index = 0;
> +	atu.type = PCIE_ATU_TYPE_CFG0;
> +	atu.cpu_addr = pp->cfg0_base + SZ_1M;

You didn't mention what occupies the first 1MB.

> +	atu.size = SZ_1M;
> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
> +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
> +	if (ret)
> +		return ret;
> +
> +	bus_range_max = bus->res->end - bus->res->start + 1;

resource_size(bus->res)

> +
> +	/* Configure for bus 2 - bus_range_max in type 1 */
> +	atu.index = 1;
> +	atu.type = PCIE_ATU_TYPE_CFG1;
> +	atu.cpu_addr = pp->cfg0_base + SZ_2M;
> +	atu.size = (SZ_1M * (bus_range_max - 2));
> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
> +	return dw_pcie_prog_outbound_atu(pci, &atu);
> +}
> +
> +static int dw_pcie_create_ecam_window(struct dw_pcie_rp *pp, struct resource *res)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct device *dev = pci->dev;
> +	struct resource_entry *bus;
> +
> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> +	if (!bus)
> +		return -ENODEV;
> +
> +	pp->cfg = pci_ecam_create(dev, res, bus->res, &pci_generic_ecam_ops);
> +	if (IS_ERR(pp->cfg))
> +		return PTR_ERR(pp->cfg);
> +
> +	pci->dbi_base = pp->cfg->win;
> +	pci->dbi_phys_addr = res->start;
> +
> +	if (pp->ops->ecam_init)
> +		pp->ops->ecam_init(pci, pp->cfg);
> +
> +	return 0;
> +}
> +
>  int dw_pcie_host_init(struct dw_pcie_rp *pp)
>  {
>  	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> @@ -431,19 +487,8 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>  
>  	raw_spin_lock_init(&pp->lock);
>  
> -	ret = dw_pcie_get_resources(pci);
> -	if (ret)
> -		return ret;
> -
>  	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
> -	if (res) {
> -		pp->cfg0_size = resource_size(res);
> -		pp->cfg0_base = res->start;
> -
> -		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
> -		if (IS_ERR(pp->va_cfg0_base))
> -			return PTR_ERR(pp->va_cfg0_base);
> -	} else {
> +	if (!res) {
>  		dev_err(dev, "Missing *config* reg space\n");
>  		return -ENODEV;
>  	}
> @@ -454,6 +499,30 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>  
>  	pp->bridge = bridge;
>  
> +	pp->cfg0_size = resource_size(res);
> +	pp->cfg0_base = res->start;
> +
> +	if (!pp->enable_ecam) {

Why can't you just use the ECAM mode when there is enough memory defined in DT?
Using this flag slightly defeats the purpose of the ECAM mode.

> +		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
> +		if (IS_ERR(pp->va_cfg0_base))
> +			return PTR_ERR(pp->va_cfg0_base);
> +
> +		/* Set default bus ops */
> +		bridge->ops = &dw_pcie_ops;
> +		bridge->child_ops = &dw_child_pcie_ops;
> +		bridge->sysdata = pp;
> +	} else {
> +		ret = dw_pcie_create_ecam_window(pp, res);
> +		if (ret)
> +			return ret;
> +		bridge->ops = (struct pci_ops *)&pci_generic_ecam_ops.pci_ops;
> +		pp->bridge->sysdata = pp->cfg;

'bridge->sysdata = pp->cfg'?

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Krishna Chaitanya Chundru]

On 12/2/2024 10:12 PM, Manivannan Sadhasivam wrote:
> On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
>> The current implementation requires iATU for every configuration
>> space access which increases latency & cpu utilization.
>>
>> Configuring iATU in config shift mode enables ECAM feature to access the
> 
> Can you please elaborate 'config shift mode'? Quote relevant section in DWC
> databook for reference.
> 
>> config space, which avoids iATU configuration for every config access.
>>
>> Add "ctrl2" into struct dw_pcie_ob_atu_cfg  to enable config shift mode.
>>
>> As DBI comes under config space, this avoids remapping of DBI space
>> separately. Instead, it uses the mapped config space address returned from
>> ECAM initialization. Change the order of dw_pcie_get_resources() execution
>> to acheive this.
>>
>> Introduce new ecam_init() function op for the clients to configure after
> 
> We use 'DWC glue drivers' to refer the 'clients' of this driver.
> 
>> ecam window creation has been done.
>>
> 
> Use 'ECAM' everywhere.
> 
>> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
>> ---
>>   drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
>>   drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
>>   drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
>>   3 files changed, 102 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
>> index 3e41865c7290..e98cc841a2a9 100644
>> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
>> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
>> @@ -418,6 +418,62 @@ static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
>>   	}
>>   }
>>   
>> +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct dw_pcie_ob_atu_cfg atu = {0};
>> +	struct resource_entry *bus;
>> +	int ret, bus_range_max;
>> +
>> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
>> +
>> +	/*
>> +	 * Bus 1 config space needs type 0 atu configuration
>> +	 * Remaining buses need type 1 atu configuration
>> +	 */
>> +	atu.index = 0;
>> +	atu.type = PCIE_ATU_TYPE_CFG0;
>> +	atu.cpu_addr = pp->cfg0_base + SZ_1M;
> 
> You didn't mention what occupies the first 1MB.
> 
>> +	atu.size = SZ_1M;
>> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
>> +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
>> +	if (ret)
>> +		return ret;
>> +
>> +	bus_range_max = bus->res->end - bus->res->start + 1;
> 
> resource_size(bus->res)
> 
>> +
>> +	/* Configure for bus 2 - bus_range_max in type 1 */
>> +	atu.index = 1;
>> +	atu.type = PCIE_ATU_TYPE_CFG1;
>> +	atu.cpu_addr = pp->cfg0_base + SZ_2M;
>> +	atu.size = (SZ_1M * (bus_range_max - 2));
>> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
>> +	return dw_pcie_prog_outbound_atu(pci, &atu);
>> +}
>> +
>> +static int dw_pcie_create_ecam_window(struct dw_pcie_rp *pp, struct resource *res)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct device *dev = pci->dev;
>> +	struct resource_entry *bus;
>> +
>> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
>> +	if (!bus)
>> +		return -ENODEV;
>> +
>> +	pp->cfg = pci_ecam_create(dev, res, bus->res, &pci_generic_ecam_ops);
>> +	if (IS_ERR(pp->cfg))
>> +		return PTR_ERR(pp->cfg);
>> +
>> +	pci->dbi_base = pp->cfg->win;
>> +	pci->dbi_phys_addr = res->start;
>> +
>> +	if (pp->ops->ecam_init)
>> +		pp->ops->ecam_init(pci, pp->cfg);
>> +
>> +	return 0;
>> +}
>> +
>>   int dw_pcie_host_init(struct dw_pcie_rp *pp)
>>   {
>>   	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> @@ -431,19 +487,8 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>>   
>>   	raw_spin_lock_init(&pp->lock);
>>   
>> -	ret = dw_pcie_get_resources(pci);
>> -	if (ret)
>> -		return ret;
>> -
>>   	res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
>> -	if (res) {
>> -		pp->cfg0_size = resource_size(res);
>> -		pp->cfg0_base = res->start;
>> -
>> -		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
>> -		if (IS_ERR(pp->va_cfg0_base))
>> -			return PTR_ERR(pp->va_cfg0_base);
>> -	} else {
>> +	if (!res) {
>>   		dev_err(dev, "Missing *config* reg space\n");
>>   		return -ENODEV;
>>   	}
>> @@ -454,6 +499,30 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>>   
>>   	pp->bridge = bridge;
>>   
>> +	pp->cfg0_size = resource_size(res);
>> +	pp->cfg0_base = res->start;
>> +
>> +	if (!pp->enable_ecam) {
> 
> Why can't you just use the ECAM mode when there is enough memory defined in DT?
> Using this flag slightly defeats the purpose of the ECAM mode.
> 
>> +		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
>> +		if (IS_ERR(pp->va_cfg0_base))
>> +			return PTR_ERR(pp->va_cfg0_base);
>> +
>> +		/* Set default bus ops */
>> +		bridge->ops = &dw_pcie_ops;
>> +		bridge->child_ops = &dw_child_pcie_ops;
>> +		bridge->sysdata = pp;
>> +	} else {
>> +		ret = dw_pcie_create_ecam_window(pp, res);
>> +		if (ret)
>> +			return ret;
>> +		bridge->ops = (struct pci_ops *)&pci_generic_ecam_ops.pci_ops;
>> +		pp->bridge->sysdata = pp->cfg;
> 
> 'bridge->sysdata = pp->cfg'?
>
as we are using pci_generic_ecam_ops.pci_ops for config reads & writes
it expects cfg space as sysdata[1].

[1] https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/ecam.c#L170

- Krishna Chaitanya.

> - Mani
> 

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Bjorn Helgaas]

On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
> The current implementation requires iATU for every configuration
> space access which increases latency & cpu utilization.
> 
> Configuring iATU in config shift mode enables ECAM feature to access the
> config space, which avoids iATU configuration for every config access.
> 
> Add "ctrl2" into struct dw_pcie_ob_atu_cfg  to enable config shift mode.
> 
> As DBI comes under config space, this avoids remapping of DBI space
> separately. Instead, it uses the mapped config space address returned from
> ECAM initialization. Change the order of dw_pcie_get_resources() execution
> to acheive this.

s/acheive/achieve/

> Introduce new ecam_init() function op for the clients to configure after
> ecam window creation has been done.
> 
> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> ---
>  drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
>  drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
>  drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
>  3 files changed, 102 insertions(+), 20 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 3e41865c7290..e98cc841a2a9 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -418,6 +418,62 @@ static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
>  	}
>  }
>  
> +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct dw_pcie_ob_atu_cfg atu = {0};
> +	struct resource_entry *bus;
> +	int ret, bus_range_max;
> +
> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> +
> +	/*
> +	 * Bus 1 config space needs type 0 atu configuration
> +	 * Remaining buses need type 1 atu configuration

s/atu/ATU/ (initialism, looks like "iATU" might be appropriate here?)

I'm confused about the bus numbering; you refer to "bus 1" and "bus
2".  Is bus 1 the root bus, i.e., the primary bus of a Root Port?

The root bus number would typically be 0, not 1, and is sometimes
programmable.  I don't know how the DesignWare core works, but since
you have "bus" here, referring to "bus 1" and "bus 2" here seems
overly specific.

> +	 */
> +	atu.index = 0;
> +	atu.type = PCIE_ATU_TYPE_CFG0;
> +	atu.cpu_addr = pp->cfg0_base + SZ_1M;
> +	atu.size = SZ_1M;
> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
> +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
> +	if (ret)
> +		return ret;
> +
> +	bus_range_max = bus->res->end - bus->res->start + 1;
> +
> +	/* Configure for bus 2 - bus_range_max in type 1 */
> +	atu.index = 1;
> +	atu.type = PCIE_ATU_TYPE_CFG1;
> +	atu.cpu_addr = pp->cfg0_base + SZ_2M;
> +	atu.size = (SZ_1M * (bus_range_max - 2));
> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
> +	return dw_pcie_prog_outbound_atu(pci, &atu);
> +}
> +
> +static int dw_pcie_create_ecam_window(struct dw_pcie_rp *pp, struct resource *res)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct device *dev = pci->dev;
> +	struct resource_entry *bus;
> +
> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> +	if (!bus)
> +		return -ENODEV;
> +
> +	pp->cfg = pci_ecam_create(dev, res, bus->res, &pci_generic_ecam_ops);
> +	if (IS_ERR(pp->cfg))
> +		return PTR_ERR(pp->cfg);
> +
> +	pci->dbi_base = pp->cfg->win;
> +	pci->dbi_phys_addr = res->start;
> +
> +	if (pp->ops->ecam_init)
> +		pp->ops->ecam_init(pci, pp->cfg);

.ecam_init() is defined to return int, but you ignore the return value.
If it's practical, I think it would be nicer if you could manage to:

  - Drop .enable_ecam.

  - Have .ecam_init() return failure if there's not enough ECAM space
    or whatever, i.e., move the qcom_pcie_check_ecam_support() code
    there.

  - Handle .ecam_init() failure here by doing whatever we did before.

If there's no useful return value from .ecam_init(), make it void.

> +	return 0;
> +}

> @@ -454,6 +499,30 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>  
>  	pp->bridge = bridge;
>  
> +	pp->cfg0_size = resource_size(res);
> +	pp->cfg0_base = res->start;
> +
> +	if (!pp->enable_ecam) {

If you can't get rid of .enable_ecam, reverse order so this uses
positive logic:

  if (pp->enable_ecam) {
    ...
  } else {
    ...
  }

> +		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
> +		if (IS_ERR(pp->va_cfg0_base))
> +			return PTR_ERR(pp->va_cfg0_base);
> +
> +		/* Set default bus ops */
> +		bridge->ops = &dw_pcie_ops;
> +		bridge->child_ops = &dw_child_pcie_ops;
> +		bridge->sysdata = pp;
> +	} else {
> +		ret = dw_pcie_create_ecam_window(pp, res);
> +		if (ret)
> +			return ret;
> +		bridge->ops = (struct pci_ops *)&pci_generic_ecam_ops.pci_ops;
> +		pp->bridge->sysdata = pp->cfg;
> +	}

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Krishna Chaitanya Chundru]

On 12/4/2024 12:25 AM, Bjorn Helgaas wrote:
> On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
>> The current implementation requires iATU for every configuration
>> space access which increases latency & cpu utilization.
>>
>> Configuring iATU in config shift mode enables ECAM feature to access the
>> config space, which avoids iATU configuration for every config access.
>>
>> Add "ctrl2" into struct dw_pcie_ob_atu_cfg  to enable config shift mode.
>>
>> As DBI comes under config space, this avoids remapping of DBI space
>> separately. Instead, it uses the mapped config space address returned from
>> ECAM initialization. Change the order of dw_pcie_get_resources() execution
>> to acheive this.
> 
> s/acheive/achieve/
> 
>> Introduce new ecam_init() function op for the clients to configure after
>> ecam window creation has been done.
>>
>> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
>> ---
>>   drivers/pci/controller/dwc/pcie-designware-host.c | 114 ++++++++++++++++++----
>>   drivers/pci/controller/dwc/pcie-designware.c      |   2 +-
>>   drivers/pci/controller/dwc/pcie-designware.h      |   6 ++
>>   3 files changed, 102 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c b/drivers/pci/controller/dwc/pcie-designware-host.c
>> index 3e41865c7290..e98cc841a2a9 100644
>> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
>> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
>> @@ -418,6 +418,62 @@ static void dw_pcie_host_request_msg_tlp_res(struct dw_pcie_rp *pp)
>>   	}
>>   }
>>   
>> +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct dw_pcie_ob_atu_cfg atu = {0};
>> +	struct resource_entry *bus;
>> +	int ret, bus_range_max;
>> +
>> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
>> +
>> +	/*
>> +	 * Bus 1 config space needs type 0 atu configuration
>> +	 * Remaining buses need type 1 atu configuration
> 
> s/atu/ATU/ (initialism, looks like "iATU" might be appropriate here?)
> 
> I'm confused about the bus numbering; you refer to "bus 1" and "bus
> 2".  Is bus 1 the root bus, i.e., the primary bus of a Root Port?
> 
> The root bus number would typically be 0, not 1, and is sometimes
> programmable.  I don't know how the DesignWare core works, but since
> you have "bus" here, referring to "bus 1" and "bus 2" here seems
> overly specific.
> 
root bus is bus 0 and we don't need any iATU configuration for it as
its config space is accessible from the system memory, for usp port of
the switch or the direct the endpoint i.e bus 1 we need to send
Configuration Type 0 requests and for other buses we need to send
Configuration Type 1 requests this is as per PCIe spec, I will try to
include PCIe spec details in next patch.
>> +	 */
>> +	atu.index = 0;
>> +	atu.type = PCIE_ATU_TYPE_CFG0;
>> +	atu.cpu_addr = pp->cfg0_base + SZ_1M;
>> +	atu.size = SZ_1M;
>> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
>> +	ret = dw_pcie_prog_outbound_atu(pci, &atu);
>> +	if (ret)
>> +		return ret;
>> +
>> +	bus_range_max = bus->res->end - bus->res->start + 1;
>> +
>> +	/* Configure for bus 2 - bus_range_max in type 1 */
>> +	atu.index = 1;
>> +	atu.type = PCIE_ATU_TYPE_CFG1;
>> +	atu.cpu_addr = pp->cfg0_base + SZ_2M;
>> +	atu.size = (SZ_1M * (bus_range_max - 2));
>> +	atu.ctrl2 = PCIE_ATU_CFG_SHIFT_MODE_ENABLE;
>> +	return dw_pcie_prog_outbound_atu(pci, &atu);
>> +}
>> +
>> +static int dw_pcie_create_ecam_window(struct dw_pcie_rp *pp, struct resource *res)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct device *dev = pci->dev;
>> +	struct resource_entry *bus;
>> +
>> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
>> +	if (!bus)
>> +		return -ENODEV;
>> +
>> +	pp->cfg = pci_ecam_create(dev, res, bus->res, &pci_generic_ecam_ops);
>> +	if (IS_ERR(pp->cfg))
>> +		return PTR_ERR(pp->cfg);
>> +
>> +	pci->dbi_base = pp->cfg->win;
>> +	pci->dbi_phys_addr = res->start;
>> +
>> +	if (pp->ops->ecam_init)
>> +		pp->ops->ecam_init(pci, pp->cfg);
> 
> .ecam_init() is defined to return int, but you ignore the return value.
> If it's practical, I think it would be nicer if you could manage to:
> 
>    - Drop .enable_ecam.
> 
>    - Have .ecam_init() return failure if there's not enough ECAM space
>      or whatever, i.e., move the qcom_pcie_check_ecam_support() code
>      there.
> 
>    - Handle .ecam_init() failure here by doing whatever we did before.
> 
> If there's no useful return value from .ecam_init(), make it void.
> 
In controller driver we need to skip few thing if we want to enable
this feature before calling dw_pcie_host_init, better to have this way
>> +	return 0;
>> +}
> 
>> @@ -454,6 +499,30 @@ int dw_pcie_host_init(struct dw_pcie_rp *pp)
>>   
>>   	pp->bridge = bridge;
>>   
>> +	pp->cfg0_size = resource_size(res);
>> +	pp->cfg0_base = res->start;
>> +
>> +	if (!pp->enable_ecam) {
> 
> If you can't get rid of .enable_ecam, reverse order so this uses
> positive logic:
> 
>    if (pp->enable_ecam) {
>      ...
>    } else {
>      ...
>    }
> 
ack.

- Krishna Chaitanya.
>> +		pp->va_cfg0_base = devm_pci_remap_cfg_resource(dev, res);
>> +		if (IS_ERR(pp->va_cfg0_base))
>> +			return PTR_ERR(pp->va_cfg0_base);
>> +
>> +		/* Set default bus ops */
>> +		bridge->ops = &dw_pcie_ops;
>> +		bridge->child_ops = &dw_child_pcie_ops;
>> +		bridge->sysdata = pp;
>> +	} else {
>> +		ret = dw_pcie_create_ecam_window(pp, res);
>> +		if (ret)
>> +			return ret;
>> +		bridge->ops = (struct pci_ops *)&pci_generic_ecam_ops.pci_ops;
>> +		pp->bridge->sysdata = pp->cfg;
>> +	}
> 

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Bjorn Helgaas]

On Wed, Dec 04, 2024 at 07:45:29AM +0530, Krishna Chaitanya Chundru wrote:
> On 12/4/2024 12:25 AM, Bjorn Helgaas wrote:
> > On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
> > > The current implementation requires iATU for every configuration
> > > space access which increases latency & cpu utilization.
> > > 
> > > Configuring iATU in config shift mode enables ECAM feature to access the
> > > config space, which avoids iATU configuration for every config access.

> > > +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
> > > +{
> > > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > > +	struct dw_pcie_ob_atu_cfg atu = {0};
> > > +	struct resource_entry *bus;
> > > +	int ret, bus_range_max;
> > > +
> > > +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> > > +
> > > +	/*
> > > +	 * Bus 1 config space needs type 0 atu configuration
> > > +	 * Remaining buses need type 1 atu configuration
> > 
> > I'm confused about the bus numbering; you refer to "bus 1" and "bus
> > 2".  Is bus 1 the root bus, i.e., the primary bus of a Root Port?
> > 
> > The root bus number would typically be 0, not 1, and is sometimes
> > programmable.  I don't know how the DesignWare core works, but since
> > you have "bus" here, referring to "bus 1" and "bus 2" here seems
> > overly specific.
> > 
> root bus is bus 0 and we don't need any iATU configuration for it as
> its config space is accessible from the system memory, for usp port of
> the switch or the direct the endpoint i.e bus 1 we need to send
> Configuration Type 0 requests and for other buses we need to send
> Configuration Type 1 requests this is as per PCIe spec, I will try to
> include PCIe spec details in next patch.

I understand the Type 0/Type 1 differences.  The question is whether
the root bus number is hard-wired to 0.

I don't think specifying "bus 1" really adds anything.  The point is
that we need Type 0 accesses for anything directly below a Root Port
(regardless of what the RP's secondary bus number is), and Type 1 for
things deeper.

When DWC supports multiple Root Ports in a Root Complex, they will not
all have a secondary bus number of 1.

Bjorn

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Krishna Chaitanya Chundru]

On 12/5/2024 3:47 AM, Bjorn Helgaas wrote:
> On Wed, Dec 04, 2024 at 07:45:29AM +0530, Krishna Chaitanya Chundru wrote:
>> On 12/4/2024 12:25 AM, Bjorn Helgaas wrote:
>>> On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
>>>> The current implementation requires iATU for every configuration
>>>> space access which increases latency & cpu utilization.
>>>>
>>>> Configuring iATU in config shift mode enables ECAM feature to access the
>>>> config space, which avoids iATU configuration for every config access.
> 
>>>> +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
>>>> +{
>>>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>>>> +	struct dw_pcie_ob_atu_cfg atu = {0};
>>>> +	struct resource_entry *bus;
>>>> +	int ret, bus_range_max;
>>>> +
>>>> +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
>>>> +
>>>> +	/*
>>>> +	 * Bus 1 config space needs type 0 atu configuration
>>>> +	 * Remaining buses need type 1 atu configuration
>>>
>>> I'm confused about the bus numbering; you refer to "bus 1" and "bus
>>> 2".  Is bus 1 the root bus, i.e., the primary bus of a Root Port?
>>>
>>> The root bus number would typically be 0, not 1, and is sometimes
>>> programmable.  I don't know how the DesignWare core works, but since
>>> you have "bus" here, referring to "bus 1" and "bus 2" here seems
>>> overly specific.
>>>
>> root bus is bus 0 and we don't need any iATU configuration for it as
>> its config space is accessible from the system memory, for usp port of
>> the switch or the direct the endpoint i.e bus 1 we need to send
>> Configuration Type 0 requests and for other buses we need to send
>> Configuration Type 1 requests this is as per PCIe spec, I will try to
>> include PCIe spec details in next patch.
> 
> I understand the Type 0/Type 1 differences.  The question is whether
> the root bus number is hard-wired to 0.
> 
It is not hard-wired to 0, we can configure it though bus-range property
> I don't think specifying "bus 1" really adds anything.  The point is
> that we need Type 0 accesses for anything directly below a Root Port
> (regardless of what the RP's secondary bus number is), and Type 1 for
> things deeper.
> 
I will update the comment without mentioning the buses as suggested.
> When DWC supports multiple Root Ports in a Root Complex, they will not
> all have a secondary bus number of 1.
mostly they should be in bus number 0 with different device numbers, but
it mostly depends upon the design, currently we don't have any multiple
root ports.

- Krishna Chaitanya.
> 
> Bjorn

Re: [PATCH 2/3] PCI: dwc: Add ECAM support with iATU configuration

[Bjorn Helgaas]

On Mon, Dec 09, 2024 at 10:00:06AM +0530, Krishna Chaitanya Chundru wrote:
> On 12/5/2024 3:47 AM, Bjorn Helgaas wrote:
> > On Wed, Dec 04, 2024 at 07:45:29AM +0530, Krishna Chaitanya Chundru wrote:
> > > On 12/4/2024 12:25 AM, Bjorn Helgaas wrote:
> > > > On Sun, Nov 17, 2024 at 03:30:19AM +0530, Krishna chaitanya chundru wrote:
> > > > > The current implementation requires iATU for every configuration
> > > > > space access which increases latency & cpu utilization.
> > > > > 
> > > > > Configuring iATU in config shift mode enables ECAM feature to access the
> > > > > config space, which avoids iATU configuration for every config access.
> > 
> > > > > +static int dw_pcie_config_ecam_iatu(struct dw_pcie_rp *pp)
> > > > > +{
> > > > > +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> > > > > +	struct dw_pcie_ob_atu_cfg atu = {0};
> > > > > +	struct resource_entry *bus;
> > > > > +	int ret, bus_range_max;
> > > > > +
> > > > > +	bus = resource_list_first_type(&pp->bridge->windows, IORESOURCE_BUS);
> > > > > +
> > > > > +	/*
> > > > > +	 * Bus 1 config space needs type 0 atu configuration
> > > > > +	 * Remaining buses need type 1 atu configuration
> > > > 
> > > > I'm confused about the bus numbering; you refer to "bus 1" and "bus
> > > > 2".  Is bus 1 the root bus, i.e., the primary bus of a Root Port?
> > > > 
> > > > The root bus number would typically be 0, not 1, and is sometimes
> > > > programmable.  I don't know how the DesignWare core works, but since
> > > > you have "bus" here, referring to "bus 1" and "bus 2" here seems
> > > > overly specific.
> > > > 
> > > root bus is bus 0 and we don't need any iATU configuration for it as
> > > its config space is accessible from the system memory, for usp port of
> > > the switch or the direct the endpoint i.e bus 1 we need to send
> > > Configuration Type 0 requests and for other buses we need to send
> > > Configuration Type 1 requests this is as per PCIe spec, I will try to
> > > include PCIe spec details in next patch.
> > 
> > I understand the Type 0/Type 1 differences.  The question is whether
> > the root bus number is hard-wired to 0.
> > 
> It is not hard-wired to 0, we can configure it though bus-range property
>
> > I don't think specifying "bus 1" really adds anything.  The point is
> > that we need Type 0 accesses for anything directly below a Root Port
> > (regardless of what the RP's secondary bus number is), and Type 1 for
> > things deeper.
>
> I will update the comment without mentioning the buses as suggested.
>
> > When DWC supports multiple Root Ports in a Root Complex, they will not
> > all have a secondary bus number of 1.
>
> mostly they should be in bus number 0 with different device numbers, but
> it mostly depends upon the design, currently we don't have any multiple
> root ports.

Say "root bus" instead of "bus 0", since you said above that the root
bus number is configurable.

Root Ports should all have a *primary* bus number of the root bus, but
if there are multiple Root Ports, they will all have different
secondary bus numbers.

[PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Krishna chaitanya chundru]

Enable the ECAM feature if the config space size is equal to size required
to represent number of buses in the bus range property.

The ELBI registers falls after the DBI space, so use the cfg win returned
from the ecam init to map these regions instead of doing the ioremap again.
ELBI starts at offset 0xf20 from dbi.

On bus 0, we have only the root complex. Any access other than that should
not go out of the link and should return all F's. Since the IATU is
configured for bus 1 onwards, block the transactions for bus 0:0:1 to
0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
link through ecam blocker through parf registers.

Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
---
 drivers/pci/controller/dwc/pcie-qcom.c | 104 +++++++++++++++++++++++++++++++--
 1 file changed, 100 insertions(+), 4 deletions(-)

diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
index ef44a82be058..266de2aa3a71 100644
--- a/drivers/pci/controller/dwc/pcie-qcom.c
+++ b/drivers/pci/controller/dwc/pcie-qcom.c
@@ -61,6 +61,17 @@
 #define PARF_DBI_BASE_ADDR_V2_HI		0x354
 #define PARF_SLV_ADDR_SPACE_SIZE_V2		0x358
 #define PARF_SLV_ADDR_SPACE_SIZE_V2_HI		0x35c
+#define PARF_BLOCK_SLV_AXI_WR_BASE		0x360
+#define PARF_BLOCK_SLV_AXI_WR_BASE_HI		0x364
+#define PARF_BLOCK_SLV_AXI_WR_LIMIT		0x368
+#define PARF_BLOCK_SLV_AXI_WR_LIMIT_HI		0x36c
+#define PARF_BLOCK_SLV_AXI_RD_BASE		0x370
+#define PARF_BLOCK_SLV_AXI_RD_BASE_HI		0x374
+#define PARF_BLOCK_SLV_AXI_RD_LIMIT		0x378
+#define PARF_BLOCK_SLV_AXI_RD_LIMIT_HI		0x37c
+#define PARF_ECAM_BASE				0x380
+#define PARF_ECAM_BASE_HI			0x384
+
 #define PARF_NO_SNOOP_OVERIDE			0x3d4
 #define PARF_ATU_BASE_ADDR			0x634
 #define PARF_ATU_BASE_ADDR_HI			0x638
@@ -68,6 +79,8 @@
 #define PARF_BDF_TO_SID_TABLE_N			0x2000
 #define PARF_BDF_TO_SID_CFG			0x2c00
 
+#define ELBI_OFFSET				0xf20
+
 /* ELBI registers */
 #define ELBI_SYS_CTRL				0x04
 
@@ -84,6 +97,7 @@
 
 /* PARF_SYS_CTRL register fields */
 #define MAC_PHY_POWERDOWN_IN_P2_D_MUX_EN	BIT(29)
+#define PCIE_ECAM_BLOCKER_EN			BIT(26)
 #define MST_WAKEUP_EN				BIT(13)
 #define SLV_WAKEUP_EN				BIT(12)
 #define MSTR_ACLK_CGC_DIS			BIT(10)
@@ -293,15 +307,68 @@ static void qcom_ep_reset_deassert(struct qcom_pcie *pcie)
 	usleep_range(PERST_DELAY_US, PERST_DELAY_US + 500);
 }
 
+static int qcom_pci_config_ecam_blocker(struct dw_pcie_rp *pp)
+{
+	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
+	struct qcom_pcie *pcie = to_qcom_pcie(pci);
+	u64 addr, addr_end;
+	u32 val;
+
+	/* Set the ECAM base */
+	writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE);
+	writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE_HI);
+
+	/*
+	 * On bus 0, we have only the root complex. Any access other than that
+	 * should not go out of the link and should return all F's. Since the
+	 * IATU is configured for bus 1 onwards, block the transactions for
+	 * bus 0:0:1 to 0:31:7 (i.e from dbi_base + 4kb to dbi_base + 1MB) from
+	 * going outside the link.
+	 */
+	addr = pci->dbi_phys_addr + SZ_4K;
+	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE);
+	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE_HI);
+
+	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE);
+	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE_HI);
+
+	addr_end = pci->dbi_phys_addr + SZ_1M - 1;
+
+	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT);
+	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT_HI);
+
+	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT);
+	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT_HI);
+
+	val = readl(pcie->parf + PARF_SYS_CTRL);
+	val |= PCIE_ECAM_BLOCKER_EN;
+	writel(val, pcie->parf + PARF_SYS_CTRL);
+	return 0;
+}
+
+static int qcom_pcie_ecam_init(struct dw_pcie *pci, struct pci_config_window *cfg)
+{
+	struct qcom_pcie *pcie = to_qcom_pcie(pci);
+
+	pcie->elbi = pci->dbi_base + ELBI_OFFSET;
+	return 0;
+}
+
 static int qcom_pcie_start_link(struct dw_pcie *pci)
 {
 	struct qcom_pcie *pcie = to_qcom_pcie(pci);
+	int ret;
 
 	if (pcie_link_speed[pci->max_link_speed] == PCIE_SPEED_16_0GT) {
 		qcom_pcie_common_set_16gt_equalization(pci);
 		qcom_pcie_common_set_16gt_lane_margining(pci);
 	}
 
+	if (pci->pp.enable_ecam) {
+		ret = qcom_pci_config_ecam_blocker(&pci->pp);
+		if (ret)
+			return ret;
+	}
 	/* Enable Link Training state machine */
 	if (pcie->cfg->ops->ltssm_enable)
 		pcie->cfg->ops->ltssm_enable(pcie);
@@ -1297,6 +1364,7 @@ static const struct dw_pcie_host_ops qcom_pcie_dw_ops = {
 	.init		= qcom_pcie_host_init,
 	.deinit		= qcom_pcie_host_deinit,
 	.post_init	= qcom_pcie_host_post_init,
+	.ecam_init	= qcom_pcie_ecam_init,
 };
 
 /* Qcom IP rev.: 2.1.0	Synopsys IP rev.: 4.01a */
@@ -1566,6 +1634,31 @@ static irqreturn_t qcom_pcie_global_irq_thread(int irq, void *data)
 	return IRQ_HANDLED;
 }
 
+static bool qcom_pcie_check_ecam_support(struct device *dev)
+{
+	struct platform_device *pdev = to_platform_device(dev);
+	struct resource bus_range, *config_res;
+	u64 bus_config_space_count;
+	int ret;
+
+	/* If bus range is not present, keep the bus range as maximum value */
+	ret = of_pci_parse_bus_range(dev->of_node, &bus_range);
+	if (ret) {
+		bus_range.start = 0x0;
+		bus_range.end = 0xff;
+	}
+
+	config_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
+	if (!config_res)
+		return false;
+
+	bus_config_space_count = resource_size(config_res) >> PCIE_ECAM_BUS_SHIFT;
+	if (resource_size(&bus_range) > bus_config_space_count)
+		return false;
+
+	return true;
+}
+
 static int qcom_pcie_probe(struct platform_device *pdev)
 {
 	const struct qcom_pcie_cfg *pcie_cfg;
@@ -1600,6 +1693,7 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 
 	pci->dev = dev;
 	pci->ops = &dw_pcie_ops;
+	pci->pp.enable_ecam = qcom_pcie_check_ecam_support(dev);
 	pp = &pci->pp;
 
 	pcie->pci = pci;
@@ -1618,10 +1712,12 @@ static int qcom_pcie_probe(struct platform_device *pdev)
 		goto err_pm_runtime_put;
 	}
 
-	pcie->elbi = devm_platform_ioremap_resource_byname(pdev, "elbi");
-	if (IS_ERR(pcie->elbi)) {
-		ret = PTR_ERR(pcie->elbi);
-		goto err_pm_runtime_put;
+	if (!pp->enable_ecam) {
+		pcie->elbi = devm_platform_ioremap_resource_byname(pdev, "elbi");
+		if (IS_ERR(pcie->elbi)) {
+			ret = PTR_ERR(pcie->elbi);
+			goto err_pm_runtime_put;
+		}
 	}
 
 	/* MHI region is optional */

-- 
2.34.1

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Manivannan Sadhasivam]

On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
> Enable the ECAM feature if the config space size is equal to size required
> to represent number of buses in the bus range property.
> 

Please move this change to DWC core.

> The ELBI registers falls after the DBI space, so use the cfg win returned
> from the ecam init to map these regions instead of doing the ioremap again.
> ELBI starts at offset 0xf20 from dbi.
> 
> On bus 0, we have only the root complex. Any access other than that should
> not go out of the link and should return all F's. Since the IATU is
> configured for bus 1 onwards, block the transactions for bus 0:0:1 to
> 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
> link through ecam blocker through parf registers.
> 
> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
> ---
>  drivers/pci/controller/dwc/pcie-qcom.c | 104 +++++++++++++++++++++++++++++++--
>  1 file changed, 100 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
> index ef44a82be058..266de2aa3a71 100644
> --- a/drivers/pci/controller/dwc/pcie-qcom.c
> +++ b/drivers/pci/controller/dwc/pcie-qcom.c
> @@ -61,6 +61,17 @@
>  #define PARF_DBI_BASE_ADDR_V2_HI		0x354
>  #define PARF_SLV_ADDR_SPACE_SIZE_V2		0x358
>  #define PARF_SLV_ADDR_SPACE_SIZE_V2_HI		0x35c
> +#define PARF_BLOCK_SLV_AXI_WR_BASE		0x360
> +#define PARF_BLOCK_SLV_AXI_WR_BASE_HI		0x364
> +#define PARF_BLOCK_SLV_AXI_WR_LIMIT		0x368
> +#define PARF_BLOCK_SLV_AXI_WR_LIMIT_HI		0x36c
> +#define PARF_BLOCK_SLV_AXI_RD_BASE		0x370
> +#define PARF_BLOCK_SLV_AXI_RD_BASE_HI		0x374
> +#define PARF_BLOCK_SLV_AXI_RD_LIMIT		0x378
> +#define PARF_BLOCK_SLV_AXI_RD_LIMIT_HI		0x37c
> +#define PARF_ECAM_BASE				0x380
> +#define PARF_ECAM_BASE_HI			0x384
> +
>  #define PARF_NO_SNOOP_OVERIDE			0x3d4
>  #define PARF_ATU_BASE_ADDR			0x634
>  #define PARF_ATU_BASE_ADDR_HI			0x638
> @@ -68,6 +79,8 @@
>  #define PARF_BDF_TO_SID_TABLE_N			0x2000
>  #define PARF_BDF_TO_SID_CFG			0x2c00
>  
> +#define ELBI_OFFSET				0xf20
> +
>  /* ELBI registers */
>  #define ELBI_SYS_CTRL				0x04
>  
> @@ -84,6 +97,7 @@
>  
>  /* PARF_SYS_CTRL register fields */
>  #define MAC_PHY_POWERDOWN_IN_P2_D_MUX_EN	BIT(29)
> +#define PCIE_ECAM_BLOCKER_EN			BIT(26)
>  #define MST_WAKEUP_EN				BIT(13)
>  #define SLV_WAKEUP_EN				BIT(12)
>  #define MSTR_ACLK_CGC_DIS			BIT(10)
> @@ -293,15 +307,68 @@ static void qcom_ep_reset_deassert(struct qcom_pcie *pcie)
>  	usleep_range(PERST_DELAY_US, PERST_DELAY_US + 500);
>  }
>  
> +static int qcom_pci_config_ecam_blocker(struct dw_pcie_rp *pp)

'config_ecam_blocker' is one of the use of this function, not the only one. So
use something like, 'qcom_pci_config_ecam()'.

> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
> +	u64 addr, addr_end;
> +	u32 val;
> +
> +	/* Set the ECAM base */
> +	writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE);
> +	writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE_HI);
> +
> +	/*
> +	 * On bus 0, we have only the root complex. Any access other than that
> +	 * should not go out of the link and should return all F's. Since the
> +	 * IATU is configured for bus 1 onwards, block the transactions for
> +	 * bus 0:0:1 to 0:31:7 (i.e from dbi_base + 4kb to dbi_base + 1MB) from

s/"for bus 0:0:1 to 0:31:7"/"starting from 0:0.1 to 0:31:7"

> +	 * going outside the link.
> +	 */
> +	addr = pci->dbi_phys_addr + SZ_4K;
> +	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE);
> +	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE_HI);
> +
> +	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE);
> +	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE_HI);
> +
> +	addr_end = pci->dbi_phys_addr + SZ_1M - 1;
> +
> +	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT);
> +	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT_HI);
> +
> +	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT);
> +	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT_HI);
> +
> +	val = readl(pcie->parf + PARF_SYS_CTRL);
> +	val |= PCIE_ECAM_BLOCKER_EN;
> +	writel(val, pcie->parf + PARF_SYS_CTRL);
> +	return 0;
> +}
> +
> +static int qcom_pcie_ecam_init(struct dw_pcie *pci, struct pci_config_window *cfg)
> +{
> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
> +
> +	pcie->elbi = pci->dbi_base + ELBI_OFFSET;

Can't you derive this offset from DT?

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Krishna Chaitanya Chundru]

On 12/2/2024 10:23 PM, Manivannan Sadhasivam wrote:
> On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
>> Enable the ECAM feature if the config space size is equal to size required
>> to represent number of buses in the bus range property.
>>
> 
> Please move this change to DWC core.
> 
>> The ELBI registers falls after the DBI space, so use the cfg win returned
>> from the ecam init to map these regions instead of doing the ioremap again.
>> ELBI starts at offset 0xf20 from dbi.
>>
>> On bus 0, we have only the root complex. Any access other than that should
>> not go out of the link and should return all F's. Since the IATU is
>> configured for bus 1 onwards, block the transactions for bus 0:0:1 to
>> 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
>> link through ecam blocker through parf registers.
>>
>> Signed-off-by: Krishna chaitanya chundru <quic_krichai@quicinc.com>
>> ---
>>   drivers/pci/controller/dwc/pcie-qcom.c | 104 +++++++++++++++++++++++++++++++--
>>   1 file changed, 100 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
>> index ef44a82be058..266de2aa3a71 100644
>> --- a/drivers/pci/controller/dwc/pcie-qcom.c
>> +++ b/drivers/pci/controller/dwc/pcie-qcom.c
>> @@ -61,6 +61,17 @@
>>   #define PARF_DBI_BASE_ADDR_V2_HI		0x354
>>   #define PARF_SLV_ADDR_SPACE_SIZE_V2		0x358
>>   #define PARF_SLV_ADDR_SPACE_SIZE_V2_HI		0x35c
>> +#define PARF_BLOCK_SLV_AXI_WR_BASE		0x360
>> +#define PARF_BLOCK_SLV_AXI_WR_BASE_HI		0x364
>> +#define PARF_BLOCK_SLV_AXI_WR_LIMIT		0x368
>> +#define PARF_BLOCK_SLV_AXI_WR_LIMIT_HI		0x36c
>> +#define PARF_BLOCK_SLV_AXI_RD_BASE		0x370
>> +#define PARF_BLOCK_SLV_AXI_RD_BASE_HI		0x374
>> +#define PARF_BLOCK_SLV_AXI_RD_LIMIT		0x378
>> +#define PARF_BLOCK_SLV_AXI_RD_LIMIT_HI		0x37c
>> +#define PARF_ECAM_BASE				0x380
>> +#define PARF_ECAM_BASE_HI			0x384
>> +
>>   #define PARF_NO_SNOOP_OVERIDE			0x3d4
>>   #define PARF_ATU_BASE_ADDR			0x634
>>   #define PARF_ATU_BASE_ADDR_HI			0x638
>> @@ -68,6 +79,8 @@
>>   #define PARF_BDF_TO_SID_TABLE_N			0x2000
>>   #define PARF_BDF_TO_SID_CFG			0x2c00
>>   
>> +#define ELBI_OFFSET				0xf20
>> +
>>   /* ELBI registers */
>>   #define ELBI_SYS_CTRL				0x04
>>   
>> @@ -84,6 +97,7 @@
>>   
>>   /* PARF_SYS_CTRL register fields */
>>   #define MAC_PHY_POWERDOWN_IN_P2_D_MUX_EN	BIT(29)
>> +#define PCIE_ECAM_BLOCKER_EN			BIT(26)
>>   #define MST_WAKEUP_EN				BIT(13)
>>   #define SLV_WAKEUP_EN				BIT(12)
>>   #define MSTR_ACLK_CGC_DIS			BIT(10)
>> @@ -293,15 +307,68 @@ static void qcom_ep_reset_deassert(struct qcom_pcie *pcie)
>>   	usleep_range(PERST_DELAY_US, PERST_DELAY_US + 500);
>>   }
>>   
>> +static int qcom_pci_config_ecam_blocker(struct dw_pcie_rp *pp)
> 
> 'config_ecam_blocker' is one of the use of this function, not the only one. So
> use something like, 'qcom_pci_config_ecam()'.
> 
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
>> +	u64 addr, addr_end;
>> +	u32 val;
>> +
>> +	/* Set the ECAM base */
>> +	writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE);
>> +	writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE_HI);
>> +
>> +	/*
>> +	 * On bus 0, we have only the root complex. Any access other than that
>> +	 * should not go out of the link and should return all F's. Since the
>> +	 * IATU is configured for bus 1 onwards, block the transactions for
>> +	 * bus 0:0:1 to 0:31:7 (i.e from dbi_base + 4kb to dbi_base + 1MB) from
> 
> s/"for bus 0:0:1 to 0:31:7"/"starting from 0:0.1 to 0:31:7"
> 
>> +	 * going outside the link.
>> +	 */
>> +	addr = pci->dbi_phys_addr + SZ_4K;
>> +	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE);
>> +	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_WR_BASE_HI);
>> +
>> +	writel(lower_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE);
>> +	writel(upper_32_bits(addr), pcie->parf + PARF_BLOCK_SLV_AXI_RD_BASE_HI);
>> +
>> +	addr_end = pci->dbi_phys_addr + SZ_1M - 1;
>> +
>> +	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT);
>> +	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_WR_LIMIT_HI);
>> +
>> +	writel(lower_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT);
>> +	writel(upper_32_bits(addr_end), pcie->parf + PARF_BLOCK_SLV_AXI_RD_LIMIT_HI);
>> +
>> +	val = readl(pcie->parf + PARF_SYS_CTRL);
>> +	val |= PCIE_ECAM_BLOCKER_EN;
>> +	writel(val, pcie->parf + PARF_SYS_CTRL);
>> +	return 0;
>> +}
>> +
>> +static int qcom_pcie_ecam_init(struct dw_pcie *pci, struct pci_config_window *cfg)
>> +{
>> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
>> +
>> +	pcie->elbi = pci->dbi_base + ELBI_OFFSET;
> 
> Can't you derive this offset from DT?
> 
> - Mani
instread of macro & dt In the next patch we will read a parf register
which will give the offset value from dbi, PARF_SLV_DBI_ELBI (0x1C001B4)

- Krishna Chaitanya.
> 

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Bjorn Helgaas]

On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
> Enable the ECAM feature if the config space size is equal to size required
> to represent number of buses in the bus range property.
> 
> The ELBI registers falls after the DBI space, so use the cfg win returned
> from the ecam init to map these regions instead of doing the ioremap again.
> ELBI starts at offset 0xf20 from dbi.
> 
> On bus 0, we have only the root complex. Any access other than that should
> not go out of the link and should return all F's. Since the IATU is
> configured for bus 1 onwards, block the transactions for bus 0:0:1 to
> 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
> link through ecam blocker through parf registers.

s/ecam/ECAM/
s/dbi/DBI/
s/IATU/iATU/ (Seems to be the convention?  Also below)
s/parf/PARF/ (I assume an initialism?)

Use conventional format for PCI bus addresses ... I assume "0:0:1"
means bus 0, device 0, function 1, which would normally be formatted
as "00:00.1" (also below).

> +static int qcom_pci_config_ecam_blocker(struct dw_pcie_rp *pp)
> +{
> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
> +	u64 addr, addr_end;
> +	u32 val;
> +
> +	/* Set the ECAM base */
> +	writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE);
> +	writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE_HI);
> +
> +	/*
> +	 * On bus 0, we have only the root complex. Any access other than that
> +	 * should not go out of the link and should return all F's. Since the
> +	 * IATU is configured for bus 1 onwards, block the transactions for
> +	 * bus 0:0:1 to 0:31:7 (i.e from dbi_base + 4kb to dbi_base + 1MB) from
> +	 * going outside the link.

s/IATU/iATU/ to match other usage.

Use conventional formatting of PCI bus/device/function addresses.

Unless the root bus number is hard-wired to be zero, maybe this should
say "root bus" instead of "bus 0"?

There is no architected presence of a PCIe Root Complex as a PCI
device.  Maybe this should say "the only device on bus 0 is the *Root
Port*"?

Or maybe there's a PCI device with some sort of device-specific
interface to Root Complex registers?  But if that were the *only*
device on bus 0, there would be no Root Port to reach other devices,
so this doesn't seem likely.

> +static bool qcom_pcie_check_ecam_support(struct device *dev)

Rename to be an assertion that can be either true or false, e.g.,
"ecam_supported".  "Check" doesn't hint about what true/false mean.

> +{
> +	struct platform_device *pdev = to_platform_device(dev);
> +	struct resource bus_range, *config_res;
> +	u64 bus_config_space_count;
> +	int ret;
> +
> +	/* If bus range is not present, keep the bus range as maximum value */
> +	ret = of_pci_parse_bus_range(dev->of_node, &bus_range);
> +	if (ret) {
> +		bus_range.start = 0x0;
> +		bus_range.end = 0xff;
> +	}

I would have thought the generic OF parsing would already default to
[bus 00-ff]?

> +	config_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
> +	if (!config_res)
> +		return false;

Move of_pci_parse_bus_range() (if it's needed) down here so it's
together with the use of the results.  No point in calling it before
looking for "config".

> +	bus_config_space_count = resource_size(config_res) >> PCIE_ECAM_BUS_SHIFT;
> +	if (resource_size(&bus_range) > bus_config_space_count)
> +		return false;
> +
> +	return true;
> +}

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Krishna Chaitanya Chundru]

On 12/4/2024 12:29 AM, Bjorn Helgaas wrote:
> On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
>> Enable the ECAM feature if the config space size is equal to size required
>> to represent number of buses in the bus range property.
>>
>> The ELBI registers falls after the DBI space, so use the cfg win returned
>> from the ecam init to map these regions instead of doing the ioremap again.
>> ELBI starts at offset 0xf20 from dbi.
>>
>> On bus 0, we have only the root complex. Any access other than that should
>> not go out of the link and should return all F's. Since the IATU is
>> configured for bus 1 onwards, block the transactions for bus 0:0:1 to
>> 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
>> link through ecam blocker through parf registers.
> 
> s/ecam/ECAM/
> s/dbi/DBI/
> s/IATU/iATU/ (Seems to be the convention?  Also below)
> s/parf/PARF/ (I assume an initialism?)
> 
> Use conventional format for PCI bus addresses ... I assume "0:0:1"
> means bus 0, device 0, function 1, which would normally be formatted
> as "00:00.1" (also below).
> 
>> +static int qcom_pci_config_ecam_blocker(struct dw_pcie_rp *pp)
>> +{
>> +	struct dw_pcie *pci = to_dw_pcie_from_pp(pp);
>> +	struct qcom_pcie *pcie = to_qcom_pcie(pci);
>> +	u64 addr, addr_end;
>> +	u32 val;
>> +
>> +	/* Set the ECAM base */
>> +	writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE);
>> +	writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf + PARF_ECAM_BASE_HI);
>> +
>> +	/*
>> +	 * On bus 0, we have only the root complex. Any access other than that
>> +	 * should not go out of the link and should return all F's. Since the
>> +	 * IATU is configured for bus 1 onwards, block the transactions for
>> +	 * bus 0:0:1 to 0:31:7 (i.e from dbi_base + 4kb to dbi_base + 1MB) from
>> +	 * going outside the link.
> 
> s/IATU/iATU/ to match other usage.
> 
> Use conventional formatting of PCI bus/device/function addresses.
> 
> Unless the root bus number is hard-wired to be zero, maybe this should
> say "root bus" instead of "bus 0"?
> 
> There is no architected presence of a PCIe Root Complex as a PCI
> device.  Maybe this should say "the only device on bus 0 is the *Root
> Port*"?
> 
> Or maybe there's a PCI device with some sort of device-specific
> interface to Root Complex registers?  But if that were the *only*
> device on bus 0, there would be no Root Port to reach other devices,
> so this doesn't seem likely.
> 
>> +static bool qcom_pcie_check_ecam_support(struct device *dev)
> 
> Rename to be an assertion that can be either true or false, e.g.,
> "ecam_supported".  "Check" doesn't hint about what true/false mean.
> 
>> +{
>> +	struct platform_device *pdev = to_platform_device(dev);
>> +	struct resource bus_range, *config_res;
>> +	u64 bus_config_space_count;
>> +	int ret;
>> +
>> +	/* If bus range is not present, keep the bus range as maximum value */
>> +	ret = of_pci_parse_bus_range(dev->of_node, &bus_range);
>> +	if (ret) {
>> +		bus_range.start = 0x0;
>> +		bus_range.end = 0xff;
>> +	}
> 
> I would have thought the generic OF parsing would already default to
> [bus 00-ff]?
> 
if there is no bus-range of_pci_parse_bus_range is not updating it[1],
the bus ranges is being updated to default value in
devm_of_pci_get_host_bridge_resources()[2]

[1]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L193
[2]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L347

- Krishna Chaitanya.
>> +	config_res = platform_get_resource_byname(pdev, IORESOURCE_MEM, "config");
>> +	if (!config_res)
>> +		return false;
> 
> Move of_pci_parse_bus_range() (if it's needed) down here so it's
> together with the use of the results.  No point in calling it before
> looking for "config".
> 
>> +	bus_config_space_count = resource_size(config_res) >> PCIE_ECAM_BUS_SHIFT;
>> +	if (resource_size(&bus_range) > bus_config_space_count)
>> +		return false;
>> +
>> +	return true;
>> +}

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Bjorn Helgaas]

On Wed, Dec 04, 2024 at 07:56:54AM +0530, Krishna Chaitanya Chundru wrote:
> On 12/4/2024 12:29 AM, Bjorn Helgaas wrote:
> > On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
> > > Enable the ECAM feature if the config space size is equal to size required
> > > to represent number of buses in the bus range property.
> > > 
> > > The ELBI registers falls after the DBI space, so use the cfg win returned
> > > from the ecam init to map these regions instead of doing the ioremap again.
> > > ELBI starts at offset 0xf20 from dbi.
> > > 
> > > On bus 0, we have only the root complex. Any access other than that should
> > > not go out of the link and should return all F's. Since the IATU is
> > > configured for bus 1 onwards, block the transactions for bus 0:0:1 to
> > > 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
> > > link through ecam blocker through parf registers.

> > > +static bool qcom_pcie_check_ecam_support(struct device *dev)
> > > +{
> > > +	struct platform_device *pdev = to_platform_device(dev);
> > > +	struct resource bus_range, *config_res;
> > > +	u64 bus_config_space_count;
> > > +	int ret;
> > > +
> > > +	/* If bus range is not present, keep the bus range as maximum value */
> > > +	ret = of_pci_parse_bus_range(dev->of_node, &bus_range);
> > > +	if (ret) {
> > > +		bus_range.start = 0x0;
> > > +		bus_range.end = 0xff;
> > > +	}
> > 
> > I would have thought the generic OF parsing would already default to
> > [bus 00-ff]?
> > 
> if there is no bus-range of_pci_parse_bus_range is not updating it[1],
> the bus ranges is being updated to default value in
> devm_of_pci_get_host_bridge_resources()[2]

Understood.  But qcom uses dw_pcie_host_init(), which calls
devm_pci_alloc_host_bridge(), which ultimately calls
of_pci_parse_bus_range() and defaults to [bus 00-ff] if there's no
bus-range in DT:

  qcom_pcie_probe
    dw_pcie_host_init
      devm_pci_alloc_host_bridge
        devm_of_pci_bridge_init
          pci_parse_request_of_pci_ranges
            devm_of_pci_get_host_bridge_resources(0, 0xff)
              of_pci_parse_bus_range

So the question is why you need to do that again here.

I see that qcom_pcie_probe() calls qcom_pcie_check_ecam_support()
*before* it calls dw_pcie_host_init(), so I guess that's the immediate
answer.

But this is another reason why I think qcom_pcie_check_ecam_support()
is kind of a sub-optimal solution here.

I wonder if we should factor the devm_pci_alloc_host_bridge() call out
of dw_pcie_host_init() so drivers can take advantage of the DT parsing
it does.  It looks like mobiveil does it that way:

  ls_g4_pcie_probe
    devm_pci_alloc_host_bridge
    mobiveil_pcie_host_probe

  mobiveil_pcie_probe
    devm_pci_alloc_host_bridge
    mobiveil_pcie_host_probe

> [1]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L193
> [2]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L347

Re: [PATCH 3/3] PCI: qcom: Enable ECAM feature based on config size

[Krishna Chaitanya Chundru]

On 12/5/2024 4:10 AM, Bjorn Helgaas wrote:
> On Wed, Dec 04, 2024 at 07:56:54AM +0530, Krishna Chaitanya Chundru wrote:
>> On 12/4/2024 12:29 AM, Bjorn Helgaas wrote:
>>> On Sun, Nov 17, 2024 at 03:30:20AM +0530, Krishna chaitanya chundru wrote:
>>>> Enable the ECAM feature if the config space size is equal to size required
>>>> to represent number of buses in the bus range property.
>>>>
>>>> The ELBI registers falls after the DBI space, so use the cfg win returned
>>>> from the ecam init to map these regions instead of doing the ioremap again.
>>>> ELBI starts at offset 0xf20 from dbi.
>>>>
>>>> On bus 0, we have only the root complex. Any access other than that should
>>>> not go out of the link and should return all F's. Since the IATU is
>>>> configured for bus 1 onwards, block the transactions for bus 0:0:1 to
>>>> 0:31:7 (i.e., from dbi_base + 4KB to dbi_base + 1MB) from going outside the
>>>> link through ecam blocker through parf registers.
> 
>>>> +static bool qcom_pcie_check_ecam_support(struct device *dev)
>>>> +{
>>>> +	struct platform_device *pdev = to_platform_device(dev);
>>>> +	struct resource bus_range, *config_res;
>>>> +	u64 bus_config_space_count;
>>>> +	int ret;
>>>> +
>>>> +	/* If bus range is not present, keep the bus range as maximum value */
>>>> +	ret = of_pci_parse_bus_range(dev->of_node, &bus_range);
>>>> +	if (ret) {
>>>> +		bus_range.start = 0x0;
>>>> +		bus_range.end = 0xff;
>>>> +	}
>>>
>>> I would have thought the generic OF parsing would already default to
>>> [bus 00-ff]?
>>>
>> if there is no bus-range of_pci_parse_bus_range is not updating it[1],
>> the bus ranges is being updated to default value in
>> devm_of_pci_get_host_bridge_resources()[2]
> 
> Understood.  But qcom uses dw_pcie_host_init(), which calls
> devm_pci_alloc_host_bridge(), which ultimately calls
> of_pci_parse_bus_range() and defaults to [bus 00-ff] if there's no
> bus-range in DT:
> 
>    qcom_pcie_probe
>      dw_pcie_host_init
>        devm_pci_alloc_host_bridge
>          devm_of_pci_bridge_init
>            pci_parse_request_of_pci_ranges
>              devm_of_pci_get_host_bridge_resources(0, 0xff)
>                of_pci_parse_bus_range
> 
> So the question is why you need to do that again here.
> 
> I see that qcom_pcie_probe() calls qcom_pcie_check_ecam_support()
> *before* it calls dw_pcie_host_init(), so I guess that's the immediate
> answer.
> 
> But this is another reason why I think qcom_pcie_check_ecam_support()
> is kind of a sub-optimal solution here.
> 
> I wonder if we should factor the devm_pci_alloc_host_bridge() call out
> of dw_pcie_host_init() so drivers can take advantage of the DT parsing
> it does.  It looks like mobiveil does it that way:
It makes sense to use this way in the next patch in the qcom driver will
call devm_pci_alloc_host_bridge() before calling dw_pcie_host_init() and
in dw_pcie_host_init() if the bridge is allocated dwc driver will skip
allocating the bridge so that other drivers will not be affected.

- Krishna Chaitanya.

> 
>    ls_g4_pcie_probe
>      devm_pci_alloc_host_bridge
>      mobiveil_pcie_host_probe
> 
>    mobiveil_pcie_probe
>      devm_pci_alloc_host_bridge
>      mobiveil_pcie_host_probe
> 
>> [1]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L193
>> [2]https://elixir.bootlin.com/linux/v6.12.1/source/drivers/pci/of.c#L347