[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[David Daney]

From: David Daney <david.daney@cavium.com>

The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
Since it is arm64, create a new quirks.c file there to contain arm64
related quirks.  Add the ThunderX bridge quirk, gated by a new config
variable, so that it can be disabled for kernels that aren't expected
to be used on ThunderX.

Signed-off-by: David Daney <david.daney@cavium.com>
---
 arch/arm64/Kconfig         | 11 +++++++++++
 arch/arm64/kernel/Makefile |  2 +-
 arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 48 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/kernel/quirks.c

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 8bd55c5..7fdf94a 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -359,6 +359,17 @@ config CAVIUM_ERRATUM_23154
 
 	  If unsure, say Y.
 
+config CAVIUM_THUNDER_PCI_QUIRKS
+	bool "Cavium PCI quirk workarounds"
+	depends on PCI
+	help
+
+	  Some ThunderX systems have PCI quirk workarounds that must
+	  be enabled to be able to use PCI devices.  This option
+	  enables the workarounds.
+
+	  If unsure, say Y.
+
 endmenu
 
 
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 22dc9bc..f80aa01 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -33,7 +33,7 @@ arm64-obj-$(CONFIG_CPU_IDLE)		+= cpuidle.o
 arm64-obj-$(CONFIG_JUMP_LABEL)		+= jump_label.o
 arm64-obj-$(CONFIG_KGDB)		+= kgdb.o
 arm64-obj-$(CONFIG_EFI)			+= efi.o efi-stub.o efi-entry.o
-arm64-obj-$(CONFIG_PCI)			+= pci.o
+arm64-obj-$(CONFIG_PCI)			+= pci.o quirks.o
 arm64-obj-$(CONFIG_ARMV8_DEPRECATED)	+= armv8_deprecated.o
 arm64-obj-$(CONFIG_ACPI)		+= acpi.o
 
diff --git a/arch/arm64/kernel/quirks.c b/arch/arm64/kernel/quirks.c
new file mode 100644
index 0000000..7352bd4
--- /dev/null
+++ b/arch/arm64/kernel/quirks.c
@@ -0,0 +1,36 @@
+/*
+ * PCIe quirks for arm64
+ *
+ * Copyright (C) 2015 Cavium, Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/pci.h>
+
+#ifdef CONFIG_CAVIUM_THUNDER_PCI_QUIRKS
+static void thunder_bridge_fixup(struct pci_dev *dev)
+{
+	/*
+	 * This bridge is broken in that it doesn't have correct
+	 * resource ranges for the buses behind it.
+	 *
+	 * The upstream bus resources are a close enough approximation
+	 * to what is needed, that they can be used instead.  Copy
+	 * upstream root bus resources so that resource claiming for
+	 * downstream devices can be done.
+	 */
+	int resno;
+	struct pci_bus *bus = dev->subordinate;
+
+	for (resno = 0; resno < PCI_BRIDGE_RESOURCE_NUM; resno++) {
+		bus->resource[resno] =
+			pci_bus_resource_n(bus->parent,
+					   PCI_BRIDGE_RESOURCE_NUM + resno);
+	}
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CAVIUM, 0xa002, thunder_bridge_fixup);
+#endif /* CONFIG_CAVIUM_THUNDER_PCI_QUIRKS */
-- 
1.9.1

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[Arnd Bergmann]

On Tuesday 22 September 2015 17:09:56 David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
> Since it is arm64, create a new quirks.c file there to contain arm64
> related quirks.  Add the ThunderX bridge quirk, gated by a new config
> variable, so that it can be disabled for kernels that aren't expected
> to be used on ThunderX.
> 
> Signed-off-by: David Daney <david.daney@cavium.com>
> ---
>  arch/arm64/Kconfig         | 11 +++++++++++
>  arch/arm64/kernel/Makefile |  2 +-
>  arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 48 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/kernel/quirks.c
> 

Looks reasonable to me. Just one question: Is the same bridge used
on MIPS machines? If so, maybe it should be moved to drivers/pci/quirks.c
instead for better reuse.

	Arnd

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[David Daney]

On 09/23/2015 12:51 AM, Arnd Bergmann wrote:
> On Tuesday 22 September 2015 17:09:56 David Daney wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
>> Since it is arm64, create a new quirks.c file there to contain arm64
>> related quirks.  Add the ThunderX bridge quirk, gated by a new config
>> variable, so that it can be disabled for kernels that aren't expected
>> to be used on ThunderX.
>>
>> Signed-off-by: David Daney <david.daney@cavium.com>
>> ---
>>   arch/arm64/Kconfig         | 11 +++++++++++
>>   arch/arm64/kernel/Makefile |  2 +-
>>   arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 48 insertions(+), 1 deletion(-)
>>   create mode 100644 arch/arm64/kernel/quirks.c
>>
>
> Looks reasonable to me. Just one question: Is the same bridge used
> on MIPS machines?

No.  The MIPS64 based OCTEON family of SoCs does not contain 
PCI-buses/config-space/bridges for on-chip hardware blocks.  The on-chip 
blocks in OCTEON are all platform devices.  So, ...

> If so, maybe it should be moved to drivers/pci/quirks.c
> instead for better reuse.

The quirk is specific to some arm64 based SoCs, thus my idea to have 
arch specific quirks.

David Daney

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[Arnd Bergmann]

On Wednesday 23 September 2015 09:00:36 David Daney wrote:
> On 09/23/2015 12:51 AM, Arnd Bergmann wrote:
> > On Tuesday 22 September 2015 17:09:56 David Daney wrote:
> >> From: David Daney <david.daney@cavium.com>
> >>
> >> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
> >> Since it is arm64, create a new quirks.c file there to contain arm64
> >> related quirks.  Add the ThunderX bridge quirk, gated by a new config
> >> variable, so that it can be disabled for kernels that aren't expected
> >> to be used on ThunderX.
> >>
> >> Signed-off-by: David Daney <david.daney@cavium.com>
> >> ---
> >>   arch/arm64/Kconfig         | 11 +++++++++++
> >>   arch/arm64/kernel/Makefile |  2 +-
> >>   arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
> >>   3 files changed, 48 insertions(+), 1 deletion(-)
> >>   create mode 100644 arch/arm64/kernel/quirks.c
> >>
> >
> > Looks reasonable to me. Just one question: Is the same bridge used
> > on MIPS machines?
> 
> No.  The MIPS64 based OCTEON family of SoCs does not contain 
> PCI-buses/config-space/bridges for on-chip hardware blocks.  The on-chip 
> blocks in OCTEON are all platform devices.  So, ...
> 
> > If so, maybe it should be moved to drivers/pci/quirks.c
> > instead for better reuse.
> 
> The quirk is specific to some arm64 based SoCs, thus my idea to have 
> arch specific quirks.

Ok.

	Arnd

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[Will Deacon]

On Wed, Sep 23, 2015 at 01:09:56AM +0100, David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
> Since it is arm64, create a new quirks.c file there to contain arm64
> related quirks.  Add the ThunderX bridge quirk, gated by a new config
> variable, so that it can be disabled for kernels that aren't expected
> to be used on ThunderX.
> 
> Signed-off-by: David Daney <david.daney@cavium.com>
> ---
>  arch/arm64/Kconfig         | 11 +++++++++++
>  arch/arm64/kernel/Makefile |  2 +-
>  arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 48 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/kernel/quirks.c

Why does this have to live in the arch/arm64/ directory? The quirks have
nothing to do with the architecture code.

Will

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[David Daney]

On 09/23/2015 11:40 AM, Will Deacon wrote:
> On Wed, Sep 23, 2015 at 01:09:56AM +0100, David Daney wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
>> Since it is arm64, create a new quirks.c file there to contain arm64
>> related quirks.  Add the ThunderX bridge quirk, gated by a new config
>> variable, so that it can be disabled for kernels that aren't expected
>> to be used on ThunderX.
>>
>> Signed-off-by: David Daney <david.daney@cavium.com>
>> ---
>>   arch/arm64/Kconfig         | 11 +++++++++++
>>   arch/arm64/kernel/Makefile |  2 +-
>>   arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 48 insertions(+), 1 deletion(-)
>>   create mode 100644 arch/arm64/kernel/quirks.c
>
> Why does this have to live in the arch/arm64/ directory? The quirks have
> nothing to do with the architecture code.
>

It doesn't have to live there.  It is something that will never be used 
for any other architecture.  So I thought we could do like 
arch/x86/kernel/quirks.c

I could move it to drivers/pci/quirks.c if people think that is a much 
better place for it.

David Daney

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[Bjorn Helgaas]

Hi David,

On Tue, Sep 22, 2015 at 05:09:56PM -0700, David Daney wrote:
> From: David Daney <david.daney@cavium.com>
> 
> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
> Since it is arm64, create a new quirks.c file there to contain arm64
> related quirks.  Add the ThunderX bridge quirk, gated by a new config
> variable, so that it can be disabled for kernels that aren't expected
> to be used on ThunderX.

Is this still needed?  I've seen some follow-up Cavium stuff, but
nothing further on this one.  If we still need it, we should figure
out who should take it.  It's all under arch/arm64, so ordinarily I
would leave it up to the arm64 guys.

> Signed-off-by: David Daney <david.daney@cavium.com>
> ---
>  arch/arm64/Kconfig         | 11 +++++++++++
>  arch/arm64/kernel/Makefile |  2 +-
>  arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>  3 files changed, 48 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/kernel/quirks.c

> +#ifdef CONFIG_CAVIUM_THUNDER_PCI_QUIRKS
> +static void thunder_bridge_fixup(struct pci_dev *dev)
> +{
> +	/*
> +	 * This bridge is broken in that it doesn't have correct
> +	 * resource ranges for the buses behind it.
> +	 *
> +	 * The upstream bus resources are a close enough approximation
> +	 * to what is needed, that they can be used instead.  Copy
> +	 * upstream root bus resources so that resource claiming for
> +	 * downstream devices can be done.
> +	 */
> +	int resno;
> +	struct pci_bus *bus = dev->subordinate;
> +
> +	for (resno = 0; resno < PCI_BRIDGE_RESOURCE_NUM; resno++) {
> +		bus->resource[resno] =
> +			pci_bus_resource_n(bus->parent,
> +					   PCI_BRIDGE_RESOURCE_NUM + resno);

I am curious about this.  Is this a standard PCI-PCI bridge, or is it
a host bridge.  If the former, I guess it must be broken in some way
that keeps the usual bridge window reading code from working?

Copying the upstream bus resources to the downstream bus might sort of
work sometimes, but seems fragile in general.  If the bridge has any
peers, it seems like we're headed for a conflict between the peer's
resources and anything downstream of the bridge.

> +	}
> +}
> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CAVIUM, 0xa002, thunder_bridge_fixup);
> +#endif /* CONFIG_CAVIUM_THUNDER_PCI_QUIRKS */
> -- 
> 1.9.1
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] arm64/pci: Add quirks for Cavium Thunder PCI bridges.

[David Daney]

On 11/24/2015 01:52 PM, Bjorn Helgaas wrote:
> Hi David,
>
> On Tue, Sep 22, 2015 at 05:09:56PM -0700, David Daney wrote:
>> From: David Daney <david.daney@cavium.com>
>>
>> The Cavium ThunderX SoC needs a PCI quirk for its on-chip bridges.
>> Since it is arm64, create a new quirks.c file there to contain arm64
>> related quirks.  Add the ThunderX bridge quirk, gated by a new config
>> variable, so that it can be disabled for kernels that aren't expected
>> to be used on ThunderX.
>
> Is this still needed?

No.

The EA patch set that you merged to 4.4 supersedes this.

> I've seen some follow-up Cavium stuff, but
> nothing further on this one.  If we still need it, we should figure
> out who should take it.  It's all under arch/arm64, so ordinarily I
> would leave it up to the arm64 guys.
>
>> Signed-off-by: David Daney <david.daney@cavium.com>
>> ---
>>   arch/arm64/Kconfig         | 11 +++++++++++
>>   arch/arm64/kernel/Makefile |  2 +-
>>   arch/arm64/kernel/quirks.c | 36 ++++++++++++++++++++++++++++++++++++
>>   3 files changed, 48 insertions(+), 1 deletion(-)
>>   create mode 100644 arch/arm64/kernel/quirks.c
>
>> +#ifdef CONFIG_CAVIUM_THUNDER_PCI_QUIRKS
>> +static void thunder_bridge_fixup(struct pci_dev *dev)
>> +{
>> +	/*
>> +	 * This bridge is broken in that it doesn't have correct
>> +	 * resource ranges for the buses behind it.
>> +	 *
>> +	 * The upstream bus resources are a close enough approximation
>> +	 * to what is needed, that they can be used instead.  Copy
>> +	 * upstream root bus resources so that resource claiming for
>> +	 * downstream devices can be done.
>> +	 */
>> +	int resno;
>> +	struct pci_bus *bus = dev->subordinate;
>> +
>> +	for (resno = 0; resno < PCI_BRIDGE_RESOURCE_NUM; resno++) {
>> +		bus->resource[resno] =
>> +			pci_bus_resource_n(bus->parent,
>> +					   PCI_BRIDGE_RESOURCE_NUM + resno);
>
> I am curious about this.  Is this a standard PCI-PCI bridge,

Yes it is PCI-PCI

> or is it
> a host bridge.  If the former, I guess it must be broken in some way
> that keeps the usual bridge window reading code from working?
>
> Copying the upstream bus resources to the downstream bus might sort of
> work sometimes, but seems fragile in general.  If the bridge has any
> peers, it seems like we're headed for a conflict between the peer's
> resources and anything downstream of the bridge.

The problem is that all the downstream devices require 64-bit 
non-prefetchable addressing, which cannot be satisfied with the bridge 
provisioning code.  EA allows this to be described.

In any event the patch is not needed now that the EA code is in place.

>
>> +	}
>> +}
>> +DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_CAVIUM, 0xa002, thunder_bridge_fixup);
>> +#endif /* CONFIG_CAVIUM_THUNDER_PCI_QUIRKS */
>> --
>> 1.9.1
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>> the body of a message to majordomo at vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html