Re: [PATCH v3 2/2] PCI: qcom: Avoid DBI and ATU register space mirror to BAR/MMIO region

[Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com>]

Hi Bjorn,

On 7/29/2024 3:51 PM, Bjorn Helgaas wrote:
> On Thu, Jul 25, 2024 at 04:03:56PM -0700, Prudhvi Yarlagadda wrote:
>> Hi Bjorn,
>>
>> Thanks for the review comments.
>>
>> On 7/24/2024 11:43 AM, Bjorn Helgaas wrote:
>>> On Tue, Jul 23, 2024 at 07:27:19PM -0700, Prudhvi Yarlagadda wrote:
>>>> PARF hardware block which is a wrapper on top of DWC PCIe controller
>>>> mirrors the DBI and ATU register space. It uses PARF_SLV_ADDR_SPACE_SIZE
>>>> register to get the size of the memory block to be mirrored and uses
>>>> PARF_DBI_BASE_ADDR, PARF_ATU_BASE_ADDR registers to determine the base
>>>> address of DBI and ATU space inside the memory block that is being
>>>> mirrored.
>>>>
>>>> When a memory region which is located above the SLV_ADDR_SPACE_SIZE
>>>> boundary is used for BAR region then there could be an overlap of DBI and
>>>> ATU address space that is getting mirrored and the BAR region. This
>>>> results in DBI and ATU address space contents getting updated when a PCIe
>>>> function driver tries updating the BAR/MMIO memory region. Reference
>>>> memory map of the PCIe memory region with DBI and ATU address space
>>>> overlapping BAR region is as below.
>>>>
>>>>                         |---------------|
>>>>                         |               |
>>>>                         |               |
>>>>         ------- --------|---------------|
>>>>            |       |    |---------------|
>>>>            |       |    |       DBI     |
>>>>            |       |    |---------------|---->DBI_BASE_ADDR
>>>>            |       |    |               |
>>>>            |       |    |               |
>>>>            |    PCIe    |               |---->2*SLV_ADDR_SPACE_SIZE
>>>>            |    BAR/MMIO|---------------|
>>>>            |    Region  |       ATU     |
>>>>            |       |    |---------------|---->ATU_BASE_ADDR
>>>>            |       |    |               |
>>>>         PCIe       |    |---------------|
>>>>         Memory     |    |       DBI     |
>>>>         Region     |    |---------------|---->DBI_BASE_ADDR
>>>>            |       |    |               |
>>>>            |    --------|               |
>>>>            |            |               |---->SLV_ADDR_SPACE_SIZE
>>>>            |            |---------------|
>>>>            |            |       ATU     |
>>>>            |            |---------------|---->ATU_BASE_ADDR
>>>>            |            |               |
>>>>            |            |---------------|
>>>>            |            |       DBI     |
>>>>            |            |---------------|---->DBI_BASE_ADDR
>>>>            |            |               |
>>>>            |            |               |
>>>>         ----------------|---------------|
>>>>                         |               |
>>>>                         |               |
>>>>                         |               |
>>>>                         |---------------|
>>>>
>>>> Currently memory region beyond the SLV_ADDR_SPACE_SIZE boundary is not
>>>> used for BAR region which is why the above mentioned issue is not
>>>> encountered. This issue is discovered as part of internal testing when we
>>>> tried moving the BAR region beyond the SLV_ADDR_SPACE_SIZE boundary. Hence
>>>> we are trying to fix this.
>>>>
>>>> As PARF hardware block mirrors DBI and ATU register space after every
>>>> PARF_SLV_ADDR_SPACE_SIZE (default 0x1000000) boundary multiple, write
>>>> U32_MAX (all 0xFF's) to PARF_SLV_ADDR_SPACE_SIZE register to avoid
>>>> mirroring DBI and ATU to BAR/MMIO region. Write the physical base address
>>>> of DBI and ATU register blocks to PARF_DBI_BASE_ADDR (default 0x0) and
>>>> PARF_ATU_BASE_ADDR (default 0x1000) respectively to make sure DBI and ATU
>>>> blocks are at expected memory locations.
>>>>
>>>> The register offsets PARF_DBI_BASE_ADDR_V2, PARF_SLV_ADDR_SPACE_SIZE_V2
>>>> and PARF_ATU_BASE_ADDR are applicable for platforms that use PARF
>>>> Qcom IP rev 1.9.0, 2.7.0 and 2.9.0. PARF_DBI_BASE_ADDR_V2 and
>>>> PARF_SLV_ADDR_SPACE_SIZE_V2 are applicable for PARF Qcom IP rev 2.3.3.
>>>> PARF_DBI_BASE_ADDR and PARF_SLV_ADDR_SPACE_SIZE are applicable for PARF
>>>> Qcom IP rev 1.0.0, 2.3.2 and 2.4.0. Updating the init()/post_init()
>>>> functions of the respective PARF versions to program applicable
>>>> PARF_DBI_BASE_ADDR, PARF_SLV_ADDR_SPACE_SIZE and PARF_ATU_BASE_ADDR
>>>> register offsets. And remove the unused SLV_ADDR_SPACE_SZ macro.
>>>>
>>>> Signed-off-by: Prudhvi Yarlagadda <quic_pyarlaga@quicinc.com>
>>>> Reviewed-by: Mayank Rana <quic_mrana@quicinc.com>
>>>> ---
>>>>  drivers/pci/controller/dwc/pcie-qcom.c | 62 +++++++++++++++++++-------
>>>>  1 file changed, 45 insertions(+), 17 deletions(-)
>>>>
>>>> diff --git a/drivers/pci/controller/dwc/pcie-qcom.c b/drivers/pci/controller/dwc/pcie-qcom.c
>>>> index 0180edf3310e..6976efb8e2f0 100644
>>>> --- a/drivers/pci/controller/dwc/pcie-qcom.c
>>>> +++ b/drivers/pci/controller/dwc/pcie-qcom.c
>>>> @@ -45,6 +45,7 @@
>>>>  #define PARF_PHY_REFCLK				0x4c
>>>>  #define PARF_CONFIG_BITS			0x50
>>>>  #define PARF_DBI_BASE_ADDR			0x168
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE		0x16C
>>>>  #define PARF_MHI_CLOCK_RESET_CTRL		0x174
>>>>  #define PARF_AXI_MSTR_WR_ADDR_HALT		0x178
>>>>  #define PARF_AXI_MSTR_WR_ADDR_HALT_V2		0x1a8
>>>> @@ -52,8 +53,13 @@
>>>>  #define PARF_LTSSM				0x1b0
>>>>  #define PARF_SID_OFFSET				0x234
>>>>  #define PARF_BDF_TRANSLATE_CFG			0x24c
>>>> -#define PARF_SLV_ADDR_SPACE_SIZE		0x358
>>>> +#define PARF_DBI_BASE_ADDR_V2			0x350
>>>> +#define PARF_DBI_BASE_ADDR_V2_HI		0x354
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2		0x358
>>>> +#define PARF_SLV_ADDR_SPACE_SIZE_V2_HI		0x35C
>>>>  #define PARF_NO_SNOOP_OVERIDE			0x3d4
>>>> +#define PARF_ATU_BASE_ADDR			0x634
>>>> +#define PARF_ATU_BASE_ADDR_HI			0x638
>>>>  #define PARF_DEVICE_TYPE			0x1000
>>>>  #define PARF_BDF_TO_SID_TABLE_N			0x2000
>>>>  #define PARF_BDF_TO_SID_CFG			0x2c00
>>>> @@ -107,9 +113,6 @@
>>>>  /* PARF_CONFIG_BITS register fields */
>>>>  #define PHY_RX0_EQ(x)				FIELD_PREP(GENMASK(26, 24), x)
>>>>  
>>>> -/* PARF_SLV_ADDR_SPACE_SIZE register value */
>>>> -#define SLV_ADDR_SPACE_SZ			0x10000000
>>>> -
>>>>  /* PARF_MHI_CLOCK_RESET_CTRL register fields */
>>>>  #define AHB_CLK_EN				BIT(0)
>>>>  #define MSTR_AXI_CLK_EN				BIT(1)
>>>> @@ -324,6 +327,39 @@ static void qcom_pcie_clear_hpc(struct dw_pcie *pci)
>>>>  	dw_pcie_dbi_ro_wr_dis(pci);
>>>>  }
>>>>  
>>>> +static void qcom_pcie_configure_dbi_base(struct qcom_pcie *pcie)
>>>> +{
>>>> +	struct dw_pcie *pci = pcie->pci;
>>>> +
>>>> +	if (pci->dbi_phys_addr) {
>>>> +		writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> +							PARF_DBI_BASE_ADDR);
>>>> +		writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE);
>>>> +	}
>>>> +}
>>>> +
>>>> +static void qcom_pcie_configure_dbi_atu_base(struct qcom_pcie *pcie)
>>>> +{
>>>> +	struct dw_pcie *pci = pcie->pci;
>>>> +
>>>> +	if (pci->dbi_phys_addr) {
>>>> +		writel(lower_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> +							PARF_DBI_BASE_ADDR_V2);
>>>> +		writel(upper_32_bits(pci->dbi_phys_addr), pcie->parf +
>>>> +						PARF_DBI_BASE_ADDR_V2_HI);
>>>> +
>>>> +		if (pci->atu_phys_addr) {
>>>> +			writel(lower_32_bits(pci->atu_phys_addr), pcie->parf +
>>>> +							PARF_ATU_BASE_ADDR);
>>>> +			writel(upper_32_bits(pci->atu_phys_addr), pcie->parf +
>>>> +							PARF_ATU_BASE_ADDR_HI);
>>>> +		}
>>>> +
>>>> +		writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2);
>>>> +		writel(U32_MAX, pcie->parf + PARF_SLV_ADDR_SPACE_SIZE_V2_HI);
>>>> +	}
>>>> +}
>>>
>>> These functions write CPU physical addresses into registers.  I don't
>>> know where these registers live.  If they are on the PCI side of the
>>> world, they most likely should contain PCI bus addresses, not CPU
>>> physical addresses.
>>>
>>> In some systems, PCI bus addresses are the same as CPU physical
>>> addresses, but on many systems they are not the same, so it's better
>>> if we don't make implicit assumptions that they are the same.  
>>
>> On Qualcomm platforms, CPU physical address and PCI bus addresses
>> for DBI and ATU registers are same. PARF registers live outside the
>> PCI address space in the system memory.
>>
>> There is a mapping logic in the QCOM PARF wrapper which detects any
>> incoming read/write transactions from the NOC towards PCIe
>> controller and checks its addresses against PARF_DBI_BASE_ADDR and
>> PARF_ATU_BASE_ADDR registers so that these transactions can be
>> routed to DBI and ATU registers inside the PCIe controller.
>>
>> So, these PARF registers needs to be programmed with base CPU
>> physical addresses of DBI and ATU as the incoming DBI and ATU
>> transactions from the NOC contain CPU physical adresses.
> 
> Can you add a comment to this effect that these registers are
> effectively in the CPU address domain, not the PCI bus domain?
> Otherwise the next person who reviews this will have the same
> question, and somebody may even try to "fix" this by converting it to
> a PCI bus address.
> 
> Bjorn

ACK. I will add a comment in the next patch.

Thanks,
Prudhvi