Re: [PATCH v3] PCI: Add ACS errata for Pericom PI7C9X2G switches

[Bjorn Helgaas <helgaas@kernel.org>]

On Fri, Sep 10, 2021 at 02:58:23AM +0000, Nathan Rossi wrote:
> From: Nathan Rossi <nathan.rossi@digi.com>
> 
> The Pericom PI7C9X2G404/PI7C9X2G304/PI7C9X2G303 PCIe switches have an
> errata for ACS P2P Request Redirect behaviour when used in the
> cut-through forwarding mode. The recommended work around for this issue
> is to use the switch in store and forward mode. The errata results in
> packets being queued and not being delivered upstream, this can be
> observed as very poor downstream device performance and/or dropped
> device generated data/interrupts.
> 
> This change adds a fixup specific to this switch that when enabling or
> resuming the downstream port it checks if it has enabled ACS P2P Request
> Redirect, and if so changes the device (via the upstream port) to use
> the store and forward operating mode.
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=177471
> Signed-off-by: Nathan Rossi <nathan.rossi@digi.com>
> Tested-by: Alex Williamson <alex.williamson@redhat.com>

Applied to pci/virtualization for v5.16, thanks!

> ---
> Changes in v2:
> - Added DECLARE_PCI_FIXUP_RESUME to handle applying fixup upon resume as
>   switch operation may have been reset or ACS configuration may have
>   changed
> Changes in v3:
> - Apply fixup to PI7C9X2G303 and PI7C9X2G304 switch models, these models
>   are also covered by the errata, although have not been validated
> - Rename PI7C9X2G404 defines to more generic PI7C9X2Gxxx
> ---
>  drivers/pci/quirks.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 56 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index e5089af8ad..f7cbc4fa40 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -5790,3 +5790,59 @@ static void apex_pci_fixup_class(struct pci_dev *pdev)
>  }
>  DECLARE_PCI_FIXUP_CLASS_HEADER(0x1ac1, 0x089a,
>  			       PCI_CLASS_NOT_DEFINED, 8, apex_pci_fixup_class);
> +
> +/*
> + * Pericom PI7C9X2G404/PI7C9X2G304/PI7C9X2G303 switch errata E5 - ACS P2P Request
> + * Redirect is not functional
> + *
> + * When ACS P2P Request Redirect is enabled and bandwidth is not balanced
> + * between upstream and downstream ports, packets are queued in an internal
> + * buffer until CPLD packet. The workaround is to use the switch in store and
> + * forward mode.
> + */
> +#define PI7C9X2Gxxx_MODE_REG		0x74
> +#define PI7C9X2Gxxx_STORE_FORWARD_MODE	BIT(0)
> +static void pci_fixup_pericom_acs_store_forward(struct pci_dev *pdev)
> +{
> +	struct pci_dev *upstream;
> +	u16 val;
> +
> +	/* Downstream ports only */
> +	if (pci_pcie_type(pdev) != PCI_EXP_TYPE_DOWNSTREAM)
> +		return;
> +
> +	/* Check for ACS P2P Request Redirect use */
> +	if (!pdev->acs_cap)
> +		return;
> +	pci_read_config_word(pdev, pdev->acs_cap + PCI_ACS_CTRL, &val);
> +	if (!(val & PCI_ACS_RR))
> +		return;
> +
> +	upstream = pci_upstream_bridge(pdev);
> +	if (!upstream)
> +		return;
> +
> +	pci_read_config_word(upstream, PI7C9X2Gxxx_MODE_REG, &val);
> +	if (!(val & PI7C9X2Gxxx_STORE_FORWARD_MODE)) {
> +		pci_info(upstream, "Setting PI7C9X2Gxxx store-forward mode\n");
> +		/* Enable store-foward mode */
> +		pci_write_config_word(upstream, PI7C9X2Gxxx_MODE_REG, val |
> +				      PI7C9X2Gxxx_STORE_FORWARD_MODE);
> +	}
> +}
> +/*
> + * Apply fixup on enable and on resume, in order to apply the fix up whenever
> + * ACS configuration changes or switch mode is reset
> + */
> +DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2404,
> +			 pci_fixup_pericom_acs_store_forward);
> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2404,
> +			 pci_fixup_pericom_acs_store_forward);
> +DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2304,
> +			 pci_fixup_pericom_acs_store_forward);
> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2304,
> +			 pci_fixup_pericom_acs_store_forward);
> +DECLARE_PCI_FIXUP_ENABLE(PCI_VENDOR_ID_PERICOM, 0x2303,
> +			 pci_fixup_pericom_acs_store_forward);
> +DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_PERICOM, 0x2303,
> +			 pci_fixup_pericom_acs_store_forward);
> ---
> 2.33.0