[PATCH 0/2] Add PCI quirk for Tenstorrent Blackhole to work on systems without enough host bridge aperture space

[PATCH 0/2] Add PCI quirk for Tenstorrent Blackhole to work on systems without enough host bridge aperture space

[Anirudh Srinivasan]

This series adds a PCI quirk to let Tenstorrent Blackhole cards to
function on systems that have smaller PCI host bridge windows by
dropping the 32GiB BAR4 that is unused by software. First patch adds
the Tenstorrent vendor ID to pci_ids.h and second patch adds the quirk.
The quirk checks if the host doesn't have any windows that are atleast
32GiB, and drops BAR4 if that's the case.

I realize that this patch doesn't account for every single case where
BAR 4 prevents Blackhole from being usable. Many of the reported cases
are ones where one Blackhole would work, but multiple won't. Examples
are cases on some x86 mining systems with limited BAR space/KVM guests
with limited PA bits that don't have BAR space for 8 blackholes. To do
this correctly, one would have to try allocating every other device
first, try Blackhole last, and drop BAR 4 if that fails. This doesn't
seem to be possible to my understanding.

I'm still submitting this patch cause it helps in many cases where even
a single Blackhole wouldn't work.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
Anirudh Srinivasan (2):
      PCI: Add Tenstorrent vendor ID to linux/pci_ids.h
      PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

 drivers/pci/quirks.c    | 32 ++++++++++++++++++++++++++++++++
 include/linux/pci_ids.h |  2 ++
 2 files changed, 34 insertions(+)
---
base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
change-id: 20260824-blackhole_quirk-a31209323a62

Best regards,
--  
Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

[PATCH 1/2] PCI: Add Tenstorrent vendor ID to linux/pci_ids.h

[Anirudh Srinivasan]

Add Tenstorrent vendor ID to pci_ids.h. This will be used to apply a
pci quirk for the Tenstorrent Blackhole card.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
 include/linux/pci_ids.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 1c9d40e09107d..8ebe4c468d39d 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2630,6 +2630,8 @@
 
 #define PCI_VENDOR_ID_ALIBABA		0x1ded
 
+#define PCI_VENDOR_ID_TENSTORRENT		0x1e52
+
 #define PCI_VENDOR_ID_CXL		0x1e98
 
 #define PCI_VENDOR_ID_TEHUTI		0x1fc9

-- 
2.43.0

Re: [PATCH 1/2] PCI: Add Tenstorrent vendor ID to linux/pci_ids.h

[sashiko-bot]

> Add Tenstorrent vendor ID to pci_ids.h. This will be used to apply a
> pci quirk for the Tenstorrent Blackhole card.
> 
> Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824-blackhole_quirk-v1-0-fc004672cdbd@oss.tenstorrent.com?part=1

[PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Anirudh Srinivasan]

Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
software talking to the card. This BAR cannot be assigned on hosts whose
PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
because bridge windows are sized as the sum of all enabled child BARs.
This makes the card completely unusable on such systems.

    spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
    pci_bus 0000:00: root bus resource [bus 00-ff]
    pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
    pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
    pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]

    <snip>

    pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
    pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
    pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
    pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]

    <snip>

    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign

Add a header fixup that walks the root-bus apertures and drops BAR 4
when it is larger than every root-bus aperture. Resource assignment
skips resources with zero flags (pdev_resource_assignable()), so the
bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
tt-bh-linux are able to run on the card.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
 drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index b09f27f7846fc..df54e379c116b 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
 #endif
+
+/*
+ * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
+ * usable on systems with smaller host bridge apertures.
+ */
+static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
+{
+	struct resource *res = &pdev->resource[4];
+	struct pci_bus *bus = pdev->bus;
+	struct resource *win;
+
+	if (!(res->flags & IORESOURCE_MEM))
+		return;
+
+	/* Find the root bus; its resource list holds the host apertures */
+	while (bus->parent)
+		bus = bus->parent;
+
+	pci_bus_for_each_resource(bus, win) {
+		if (!win || !(win->flags & IORESOURCE_MEM))
+			continue;
+		if (resource_size(res) <= resource_size(win))
+			return;
+	}
+
+	pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
+		 res);
+	res->start = 0;
+	res->end = 0;
+	res->flags = 0;
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TENSTORRENT, 0xb140, quirk_tenstorrent_blackhole_bar4);

-- 
2.43.0

Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[sashiko-bot]

> Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> software talking to the card. This BAR cannot be assigned on hosts whose
> PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> because bridge windows are sized as the sum of all enabled child BARs.
> This makes the card completely unusable on such systems.
> 
>     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
>     pci_bus 0000:00: root bus resource [bus 00-ff]
>     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
>     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
>     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> 
>     <snip>
> 
> [ ... ]
> bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> tt-bh-linux are able to run on the card.
> 
> Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824-blackhole_quirk-v1-0-fc004672cdbd@oss.tenstorrent.com?part=2