[PATCH 0/2] Add PCI quirk for Tenstorrent Blackhole to work on systems without enough host bridge aperture space

[PATCH 0/2] Add PCI quirk for Tenstorrent Blackhole to work on systems without enough host bridge aperture space

[Anirudh Srinivasan]

This series adds a PCI quirk to let Tenstorrent Blackhole cards to
function on systems that have smaller PCI host bridge windows by
dropping the 32GiB BAR4 that is unused by software. First patch adds
the Tenstorrent vendor ID to pci_ids.h and second patch adds the quirk.
The quirk checks if the host doesn't have any windows that are atleast
32GiB, and drops BAR4 if that's the case.

I realize that this patch doesn't account for every single case where
BAR 4 prevents Blackhole from being usable. Many of the reported cases
are ones where one Blackhole would work, but multiple won't. Examples
are cases on some x86 mining systems with limited BAR space/KVM guests
with limited PA bits that don't have BAR space for 8 blackholes. To do
this correctly, one would have to try allocating every other device
first, try Blackhole last, and drop BAR 4 if that fails. This doesn't
seem to be possible to my understanding.

I'm still submitting this patch cause it helps in many cases where even
a single Blackhole wouldn't work.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
Anirudh Srinivasan (2):
      PCI: Add Tenstorrent vendor ID to linux/pci_ids.h
      PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

 drivers/pci/quirks.c    | 32 ++++++++++++++++++++++++++++++++
 include/linux/pci_ids.h |  2 ++
 2 files changed, 34 insertions(+)
---
base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
change-id: 20260824-blackhole_quirk-a31209323a62

Best regards,
--  
Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

[PATCH 1/2] PCI: Add Tenstorrent vendor ID to linux/pci_ids.h

[Anirudh Srinivasan]

Add Tenstorrent vendor ID to pci_ids.h. This will be used to apply a
pci quirk for the Tenstorrent Blackhole card.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
 include/linux/pci_ids.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 1c9d40e09107d..8ebe4c468d39d 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2630,6 +2630,8 @@
 
 #define PCI_VENDOR_ID_ALIBABA		0x1ded
 
+#define PCI_VENDOR_ID_TENSTORRENT		0x1e52
+
 #define PCI_VENDOR_ID_CXL		0x1e98
 
 #define PCI_VENDOR_ID_TEHUTI		0x1fc9

-- 
2.43.0

[PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Anirudh Srinivasan]

Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
software talking to the card. This BAR cannot be assigned on hosts whose
PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
because bridge windows are sized as the sum of all enabled child BARs.
This makes the card completely unusable on such systems.

    spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
    pci_bus 0000:00: root bus resource [bus 00-ff]
    pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
    pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
    pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]

    <snip>

    pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
    pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
    pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
    pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]

    <snip>

    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
    pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
    pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
    pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
    pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign

Add a header fixup that walks the root-bus apertures and drops BAR 4
when it is larger than every root-bus aperture. Resource assignment
skips resources with zero flags (pdev_resource_assignable()), so the
bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
tt-bh-linux are able to run on the card.

Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
---
 drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index b09f27f7846fc..df54e379c116b 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
 DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
 #endif
+
+/*
+ * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
+ * usable on systems with smaller host bridge apertures.
+ */
+static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
+{
+	struct resource *res = &pdev->resource[4];
+	struct pci_bus *bus = pdev->bus;
+	struct resource *win;
+
+	if (!(res->flags & IORESOURCE_MEM))
+		return;
+
+	/* Find the root bus; its resource list holds the host apertures */
+	while (bus->parent)
+		bus = bus->parent;
+
+	pci_bus_for_each_resource(bus, win) {
+		if (!win || !(win->flags & IORESOURCE_MEM))
+			continue;
+		if (resource_size(res) <= resource_size(win))
+			return;
+	}
+
+	pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
+		 res);
+	res->start = 0;
+	res->end = 0;
+	res->flags = 0;
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TENSTORRENT, 0xb140, quirk_tenstorrent_blackhole_bar4);

-- 
2.43.0

Re: [PATCH 1/2] PCI: Add Tenstorrent vendor ID to linux/pci_ids.h

[sashiko-bot]

> Add Tenstorrent vendor ID to pci_ids.h. This will be used to apply a
> pci quirk for the Tenstorrent Blackhole card.
> 
> Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824-blackhole_quirk-v1-0-fc004672cdbd@oss.tenstorrent.com?part=1

Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[sashiko-bot]

> Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> software talking to the card. This BAR cannot be assigned on hosts whose
> PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> because bridge windows are sized as the sum of all enabled child BARs.
> This makes the card completely unusable on such systems.
> 
>     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
>     pci_bus 0000:00: root bus resource [bus 00-ff]
>     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
>     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
>     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> 
>     <snip>
> 
> [ ... ]
> bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> tt-bh-linux are able to run on the card.
> 
> Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260824-blackhole_quirk-v1-0-fc004672cdbd@oss.tenstorrent.com?part=2

Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Bjorn Helgaas]

[+cc Ilpo]

On Mon, Aug 24, 2026 at 11:57:07AM -0500, Anirudh Srinivasan wrote:
> Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> software talking to the card. This BAR cannot be assigned on hosts whose
> PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> because bridge windows are sized as the sum of all enabled child BARs.
> This makes the card completely unusable on such systems.
> 
>     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
>     pci_bus 0000:00: root bus resource [bus 00-ff]
>     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
>     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
>     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> 
>     <snip>
> 
>     pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
>     pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
>     pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
>     pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]
> 
>     <snip>
> 
>     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
>     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
>     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
>     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
>     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
>     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
>     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
>     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
>     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
>     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
>     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
>     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
>     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
>     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
>     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
>     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
>     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> 
> Add a header fixup that walks the root-bus apertures and drops BAR 4
> when it is larger than every root-bus aperture. Resource assignment
> skips resources with zero flags (pdev_resource_assignable()), so the
> bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> tt-bh-linux are able to run on the card.
> 
> Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
> ---
>  drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
>  1 file changed, 32 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index b09f27f7846fc..df54e379c116b 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
>  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
>  #endif
> +
> +/*
> + * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
> + * usable on systems with smaller host bridge apertures.
> + */
> +static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
> +{
> +	struct resource *res = &pdev->resource[4];
> +	struct pci_bus *bus = pdev->bus;
> +	struct resource *win;
> +
> +	if (!(res->flags & IORESOURCE_MEM))
> +		return;
> +
> +	/* Find the root bus; its resource list holds the host apertures */
> +	while (bus->parent)
> +		bus = bus->parent;
> +
> +	pci_bus_for_each_resource(bus, win) {
> +		if (!win || !(win->flags & IORESOURCE_MEM))
> +			continue;
> +		if (resource_size(res) <= resource_size(win))
> +			return;
> +	}
> +
> +	pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
> +		 res);
> +	res->start = 0;
> +	res->end = 0;
> +	res->flags = 0;

This seems problematic because the hardware BAR still exists even if
we zero out res->flags.

This is a problem because pci_enable_device() will successfully enable
a device and turn on PCI_COMMAND_MEMORY even though the hardware BAR
may contain junk.  And if a driver ioremaps this BAR, it is
ioremapping physical address 0, which doesn't end well.

I know this from bitter experience :)

If you have some device-specific way to actually disable the BAR in
the hardware, that could work.

> +}
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TENSTORRENT, 0xb140, quirk_tenstorrent_blackhole_bar4);
> 
> -- 
> 2.43.0
> 

Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Anirudh Srinivasan]

Hello Bjorn,

On Wed, Aug 26, 2026 at 2:55 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
>
> [+cc Ilpo]
>
> On Mon, Aug 24, 2026 at 11:57:07AM -0500, Anirudh Srinivasan wrote:
> > Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> > software talking to the card. This BAR cannot be assigned on hosts whose
> > PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> > failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> > because bridge windows are sized as the sum of all enabled child BARs.
> > This makes the card completely unusable on such systems.
> >
> >     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
> >     pci_bus 0000:00: root bus resource [bus 00-ff]
> >     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
> >     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
> >     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> >
> >     <snip>
> >
> >     pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
> >     pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
> >     pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
> >     pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]
> >
> >     <snip>
> >
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> >
> > Add a header fixup that walks the root-bus apertures and drops BAR 4
> > when it is larger than every root-bus aperture. Resource assignment
> > skips resources with zero flags (pdev_resource_assignable()), so the
> > bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> > Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> > tt-bh-linux are able to run on the card.
> >
> > Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
> > ---
> >  drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
> >  1 file changed, 32 insertions(+)
> >
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index b09f27f7846fc..df54e379c116b 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
> >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
> >  #endif
> > +
> > +/*
> > + * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
> > + * usable on systems with smaller host bridge apertures.
> > + */
> > +static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
> > +{
> > +     struct resource *res = &pdev->resource[4];
> > +     struct pci_bus *bus = pdev->bus;
> > +     struct resource *win;
> > +
> > +     if (!(res->flags & IORESOURCE_MEM))
> > +             return;
> > +
> > +     /* Find the root bus; its resource list holds the host apertures */
> > +     while (bus->parent)
> > +             bus = bus->parent;
> > +
> > +     pci_bus_for_each_resource(bus, win) {
> > +             if (!win || !(win->flags & IORESOURCE_MEM))
> > +                     continue;
> > +             if (resource_size(res) <= resource_size(win))
> > +                     return;
> > +     }
> > +
> > +     pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
> > +              res);
> > +     res->start = 0;
> > +     res->end = 0;
> > +     res->flags = 0;
>
> This seems problematic because the hardware BAR still exists even if
> we zero out res->flags.
>
> This is a problem because pci_enable_device() will successfully enable
> a device and turn on PCI_COMMAND_MEMORY even though the hardware BAR
> may contain junk.  And if a driver ioremaps this BAR, it is
> ioremapping physical address 0, which doesn't end well.
>
> I know this from bitter experience :)
>
> If you have some device-specific way to actually disable the BAR in
> the hardware, that could work.

lspci for this device doesn't show this BAR region. Are you saying
that BAR4 still exists (but points to 0) ? Our driver doesn't seem to
have any issues with this, but some other driver may not play well
with this.

$ lspci -d 1e52: -vvv
0000:01:00.0 Processing accelerators: Tenstorrent Inc Blackhole
        Subsystem: Tenstorrent Inc p150a
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0
        Interrupt: pin A routed to IRQ 27
        Region 0: Memory at 1800000000 (64-bit, prefetchable) [size=512M]
        Region 2: Memory at 1820000000 (64-bit, prefetchable) [size=1M]
        Capabilities: <access denied>
        Kernel driver in use: tenstorrent

>
> > +}
> > +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TENSTORRENT, 0xb140, quirk_tenstorrent_blackhole_bar4);
> >
> > --
> > 2.43.0
> >

Re: [PATCH 2/2] PCI: quirks: Drop unassignable 32 GiB BAR 4 on Tenstorrent Blackhole

[Bjorn Helgaas]

On Wed, Aug 26, 2026 at 03:16:29PM -0500, Anirudh Srinivasan wrote:
> On Wed, Aug 26, 2026 at 2:55 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Mon, Aug 24, 2026 at 11:57:07AM -0500, Anirudh Srinivasan wrote:
> > > Tenstorrent Blackhole cards expose a 32 GiB BAR 4 that is unused by most
> > > software talking to the card. This BAR cannot be assigned on hosts whose
> > > PCI aperture is smaller than 32 GiB (e.g some ARM/RISC-V systems). This
> > > failure also results in BAR 0 (256 MiB) and 2 (2 MiB) being unassigned
> > > because bridge windows are sized as the sum of all enabled child BARs.
> > > This makes the card completely unusable on such systems.
> > >
> > >     spacemit-k1-pcie 80000000.pcie: PCI host bridge to bus 0000:00
> > >     pci_bus 0000:00: root bus resource [bus 00-ff]
> > >     pci_bus 0000:00: root bus resource [io  0x100000-0x1fffff] (bus address [0x10000-0x10ffff])
> > >     pci_bus 0000:00: root bus resource [mem 0x1100110000-0x117fffffff] (bus address [0x00110000-0x7fffffff])
> > >     pci_bus 0000:00: root bus resource [mem 0x1800000000-0x18ffffffff pref]
> > >
> > >     <snip>
> > >
> > >     pci 0000:01:00.0: [1e52:b140] type 00 class 0x120000 PCIe Endpoint
> > >     pci 0000:01:00.0: BAR 0 [mem 0x00000000-0x1fffffff 64bit pref]
> > >     pci 0000:01:00.0: BAR 2 [mem 0x1120000000-0x11200fffff 64bit pref]
> > >     pci 0000:01:00.0: BAR 4 [mem 0xfffffff800000000-0xffffffffffffffff 64bit pref]
> > >
> > >     <snip>
> > >
> > >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> > >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> > >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> > >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> > >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: releasing
> > >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: releasing
> > >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: can't assign; no space
> > >     pci 0000:00:00.0: bridge window [mem size 0x820100000 64bit pref]: failed to assign
> > >     pci 0000:00:00.0: BAR 0 [mem 0x1108000000-0x110fffffff]: assigned
> > >     pci 0000:00:00.0: BAR 1 [mem 0x1110000000-0x1117ffffff]: assigned
> > >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> > >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> > >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> > >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 4 [mem size 0x800000000 64bit pref]: failed to assign
> > >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 0 [mem size 0x20000000 64bit pref]: failed to assign
> > >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: can't assign; no space
> > >     pci 0000:01:00.0: BAR 2 [mem size 0x00100000 64bit pref]: failed to assign
> > >
> > > Add a header fixup that walks the root-bus apertures and drops BAR 4
> > > when it is larger than every root-bus aperture. Resource assignment
> > > skips resources with zero flags (pdev_resource_assignable()), so the
> > > bridge window then sizes to BARs 0/2 and succeeds. This was tested on a
> > > Spacemit-K3 RISC-V board that has only 4GiB of BAR space. tt-smi and
> > > tt-bh-linux are able to run on the card.
> > >
> > > Signed-off-by: Anirudh Srinivasan <asrinivasan@oss.tenstorrent.com>
> > > ---
> > >  drivers/pci/quirks.c | 32 ++++++++++++++++++++++++++++++++
> > >  1 file changed, 32 insertions(+)
> > >
> > > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > > index b09f27f7846fc..df54e379c116b 100644
> > > --- a/drivers/pci/quirks.c
> > > +++ b/drivers/pci/quirks.c
> > > @@ -6415,3 +6415,35 @@ static void pci_mask_replay_timer_timeout(struct pci_dev *pdev)
> > >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9750, pci_mask_replay_timer_timeout);
> > >  DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_GLI, 0x9755, pci_mask_replay_timer_timeout);
> > >  #endif
> > > +
> > > +/*
> > > + * Drop unused 32 GiB BAR 4 on Tenstorrent Blackhole cards so the card is
> > > + * usable on systems with smaller host bridge apertures.
> > > + */
> > > +static void quirk_tenstorrent_blackhole_bar4(struct pci_dev *pdev)
> > > +{
> > > +     struct resource *res = &pdev->resource[4];
> > > +     struct pci_bus *bus = pdev->bus;
> > > +     struct resource *win;
> > > +
> > > +     if (!(res->flags & IORESOURCE_MEM))
> > > +             return;
> > > +
> > > +     /* Find the root bus; its resource list holds the host apertures */
> > > +     while (bus->parent)
> > > +             bus = bus->parent;
> > > +
> > > +     pci_bus_for_each_resource(bus, win) {
> > > +             if (!win || !(win->flags & IORESOURCE_MEM))
> > > +                     continue;
> > > +             if (resource_size(res) <= resource_size(win))
> > > +                     return;
> > > +     }
> > > +
> > > +     pci_info(pdev, "BAR 4 %pR larger than every host bridge aperture, dropping it\n",
> > > +              res);
> > > +     res->start = 0;
> > > +     res->end = 0;
> > > +     res->flags = 0;
> >
> > This seems problematic because the hardware BAR still exists even if
> > we zero out res->flags.
> >
> > This is a problem because pci_enable_device() will successfully enable
> > a device and turn on PCI_COMMAND_MEMORY even though the hardware BAR
> > may contain junk.  And if a driver ioremaps this BAR, it is
> > ioremapping physical address 0, which doesn't end well.
> >
> > I know this from bitter experience :)
> >
> > If you have some device-specific way to actually disable the BAR in
> > the hardware, that could work.
> 
> lspci for this device doesn't show this BAR region. Are you saying
> that BAR4 still exists (but points to 0) ? Our driver doesn't seem to
> have any issues with this, but some other driver may not play well
> with this.
> 
> $ lspci -d 1e52: -vvv
> 0000:01:00.0 Processing accelerators: Tenstorrent Inc Blackhole
>         Subsystem: Tenstorrent Inc p150a
>         Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0
>         Interrupt: pin A routed to IRQ 27
>         Region 0: Memory at 1800000000 (64-bit, prefetchable) [size=512M]
>         Region 2: Memory at 1820000000 (64-bit, prefetchable) [size=1M]
>         Capabilities: <access denied>
>         Kernel driver in use: tenstorrent

I think by default lspci will show you what the kernel knows about the
device, i.e., what's in the dev->resource[] array.  If res->flags is
zeroed out, the kernel doesn't know anything about the BAR, so it
probably won't appear in lspci output.

But of course that doesn't change anything from the device's
perspective.  If you ioremap BAR 4 and read a word from it, I think it
might not work.  Based on your dmesg log, it doesn't look like
physical address 0 would be routed to PCI, so it probably causes some
kind of exception unless you happen to have RAM at physical address 0.