* [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
@ 2026-11-09 18:42 Vlad Poenaru
2026-11-09 18:48 ` sashiko-bot
2026-11-19 20:06 ` Vlad Poenaru
0 siblings, 2 replies; 8+ messages in thread
From: Vlad Poenaru @ 2026-11-09 18:42 UTC (permalink / raw)
To: Michael S . Tsirkin, Jason Wang
Cc: Eugenio Pérez, Xuan Zhuo, Brett Creeley, Kyle McMartin,
virtualization, linux-kernel, stable, kernel-team
The virtio-net function of the AMD Pensando DSC2-200 (Elba) DPU, as
shipped by Oracle (1af4:1041, subsystem 108e:1200), cannot DMA to
arbitrary 64-bit bus addresses, and Oracle advises limiting it to 48
bits. virtio_pci gives it a 64-bit mask, so its virtqueues stop
completing requests once the IOMMU starts handing it IOVAs above 2^48.
The card is used in OCI bare-metal shapes such as BM.Standard.E6.288.
On those AMD EPYC 9005 hosts the AMD IOMMU supports a 6-level v1 page
table (EFR[HATS] = 2), so the aperture spans all 64 bits and the IOVA
allocator, which works down from min(mask, aperture), returns addresses
just below 2^64:
- With iommu.forcedac=1, the first control virtqueue command never
completes and virtio_net hangs in probe.
- Without it, iommu_dma_alloc_iova() prefers 32-bit IOVAs until that
space runs out once, logs "Using 64-bit DMA addresses", and stops
preferring them for the device. Across several thousand of these
hosts, 21 logged that for the virtio-net function within 0.2 to 7.3
days of boot. On 14 of them, TX queues then stopped completing
descriptors within hours ("NETDEV WATCHDOG: ... transmit queue N
timed out"). No host without the fallback reported a TX timeout.
virtio gives a device no way to advertise a DMA address limit, and the
platform describes none for this device, so the driver has to apply
one. Cap the device at 48 bits through the dma_mask override that
vp_modern_probe() already provides for devices with address limits.
The device still gets 32-bit IOVAs first, and then IOVAs below 2^48
instead of 2^64. pds_vdpa uses 52 bits for the same ASIC; this uses
Oracle's narrower 48-bit limit, which still leaves 256 TiB of IOVA
space.
Only the virtio-net function is matched, since it is the one that was
tested. force_legacy does not bypass the quirk: the legacy probe
refuses device IDs above 0x103f, so the device always goes through the
modern probe.
Tested on a BM.Standard.E6.288 with 6-hour iperf3 --bidir soaks, at up
to 57 Gbit/s each way with iommu.forcedac=1 and at about 26 Gbit/s each
way without it. Neither run saw a TX timeout or an AMD-Vi
IO_PAGE_FAULT. With forcedac, every traced IOVA for the device fell
between 2^32 and 2^48; without it, every IOVA stayed below 2^32, as
before.
Cc: stable@vger.kernel.org # 6.5+
Co-developed-by: Kyle McMartin <jkkm@meta.com>
Signed-off-by: Kyle McMartin <jkkm@meta.com>
Signed-off-by: Vlad Poenaru <vlad.wing@gmail.com>
---
drivers/virtio/virtio_pci_common.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c
index 79d3395f37050..0a7c97cd80509 100644
--- a/drivers/virtio/virtio_pci_common.c
+++ b/drivers/virtio/virtio_pci_common.c
@@ -667,8 +667,18 @@ static const struct dev_pm_ops virtio_pci_pm_ops = {
#endif
+/*
+ * The virtio-net function of the AMD Pensando DSC2-200 (Elba) DPU, as
+ * shipped by Oracle (subsystem 108e:1200), cannot DMA to arbitrary 64-bit
+ * bus addresses. Oracle advises limiting it to 48 bits.
+ */
+#define VIRTIO_PCI_QUIRK_DMA_48BIT BIT(0)
+
/* Qumranet donated their vendor ID for devices 0x1000 thru 0x10FF. */
static const struct pci_device_id virtio_pci_id_table[] = {
+ { PCI_DEVICE_SUB(PCI_VENDOR_ID_REDHAT_QUMRANET, 0x1041,
+ PCI_VENDOR_ID_SUN, 0x1200),
+ .driver_data = VIRTIO_PCI_QUIRK_DMA_48BIT },
{ PCI_DEVICE(PCI_VENDOR_ID_REDHAT_QUMRANET, PCI_ANY_ID) },
{ 0 }
};
@@ -710,6 +720,9 @@ static int virtio_pci_probe(struct pci_dev *pci_dev,
if (rc)
goto err_enable_device;
+ if (id->driver_data & VIRTIO_PCI_QUIRK_DMA_48BIT)
+ vp_dev->mdev.dma_mask = DMA_BIT_MASK(48);
+
if (force_legacy) {
rc = virtio_pci_legacy_probe(vp_dev);
/* Also try modern mode if we can't map BAR0 (no IO space). */
--
2.53.0-Meta
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-09 18:42 [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA Vlad Poenaru
@ 2026-11-09 18:48 ` sashiko-bot
2026-11-19 20:06 ` Vlad Poenaru
1 sibling, 0 replies; 8+ messages in thread
From: sashiko-bot @ 2026-11-09 18:48 UTC (permalink / raw)
To: Vlad Poenaru; +Cc: virtualization, Michael S. Tsirkin, Eugenio Perez
> The virtio-net function of the AMD Pensando DSC2-200 (Elba) DPU, as
> shipped by Oracle (1af4:1041, subsystem 108e:1200), cannot DMA to
> arbitrary 64-bit bus addresses, and Oracle advises limiting it to 48
> bits. virtio_pci gives it a 64-bit mask, so its virtqueues stop
> completing requests once the IOMMU starts handing it IOVAs above 2^48.
> The card is used in OCI bare-metal shapes such as BM.Standard.E6.288.
>
> On those AMD EPYC 9005 hosts the AMD IOMMU supports a 6-level v1 page
> table (EFR[HATS] = 2), so the aperture spans all 64 bits and the IOVA
> allocator, which works down from min(mask, aperture), returns addresses
> just below 2^64:
>
> - With iommu.forcedac=1, the first control virtqueue command never
> completes and virtio_net hangs in probe.
>
> [ ... ]
>
> Cc: stable@vger.kernel.org # 6.5+
> Co-developed-by: Kyle McMartin <jkkm@meta.com>
> Signed-off-by: Kyle McMartin <jkkm@meta.com>
> Signed-off-by: Vlad Poenaru <vlad.wing@gmail.com>
Sashiko has reviewed this patch and found no issues.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261109184253.531677-1-vlad.wing@gmail.com?part=1
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-09 18:42 [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA Vlad Poenaru
2026-11-09 18:48 ` sashiko-bot
@ 2026-11-19 20:06 ` Vlad Poenaru
2026-11-19 23:16 ` Michael S. Tsirkin
1 sibling, 1 reply; 8+ messages in thread
From: Vlad Poenaru @ 2026-11-19 20:06 UTC (permalink / raw)
To: Michael S . Tsirkin, Jason Wang
Cc: Eugenio Pérez, Xuan Zhuo, Brett Creeley, Kyle McMartin,
virtualization, linux-kernel, kernel-team
Hi Michael, Jason,
Gentle ping on this one. Without it, the virtio-net function of the
DSC2-200 in OCI bare-metal hosts stops completing TX once the IOMMU
hands it IOVAs above 2^48. With iommu.forcedac=1, virtio_net hangs
in probe instead.
The patch still applies cleanly to current mainline (v7.4-rc3). If you
would rather have the quirk cover every virtio function under
subsystem 108e:1200 instead of only virtio-net, I can respin it that
way.
Thanks,
Vlad
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-19 20:06 ` Vlad Poenaru
@ 2026-11-19 23:16 ` Michael S. Tsirkin
2026-11-20 17:14 ` Jonah Palmer
0 siblings, 1 reply; 8+ messages in thread
From: Michael S. Tsirkin @ 2026-11-19 23:16 UTC (permalink / raw)
To: Vlad Poenaru
Cc: Jason Wang, Eugenio Pérez, Xuan Zhuo, Brett Creeley,
Kyle McMartin, virtualization, linux-kernel, kernel-team
On Thu, Nov 19, 2026 at 12:06:54PM -0800, Vlad Poenaru wrote:
> Hi Michael, Jason,
>
> Gentle ping on this one. Without it, the virtio-net function of the
> DSC2-200 in OCI bare-metal hosts stops completing TX once the IOMMU
> hands it IOVAs above 2^48. With iommu.forcedac=1, virtio_net hangs
> in probe instead.
>
> The patch still applies cleanly to current mainline (v7.4-rc3). If you
> would rather have the quirk cover every virtio function under
> subsystem 108e:1200 instead of only virtio-net, I can respin it that
> way.
>
> Thanks,
> Vlad
it's in my tree.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-19 23:16 ` Michael S. Tsirkin
@ 2026-11-20 17:14 ` Jonah Palmer
2026-11-20 17:35 ` Michael S. Tsirkin
0 siblings, 1 reply; 8+ messages in thread
From: Jonah Palmer @ 2026-11-20 17:14 UTC (permalink / raw)
To: mst, vlad.wing
Cc: brett.creeley, eperezma, jasowangio, jkkm, kernel-team,
linux-kernel, virtualization, xuanzhuo, joao.m.martins,
si-wei.liu
On November 19, 2026 at 6:16 PM EST, Michael S. Tsirkin wrote:
> it's in my tree.
Hi Vlad, Michael.
I've been looking into this issue extensively. The proposed 48-bit limit
is a suitable conservative workaround. My virtio-net testing for this
issue shows that a 51-bit DMA mask also avoids the mentioned issue.
Full 52-bit DMA mask support is planned. We will revisit the driver's
DMA mask once the firmware has been updated.
Thanks,
Jonah
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-20 17:14 ` Jonah Palmer
@ 2026-11-20 17:35 ` Michael S. Tsirkin
2026-11-21 4:35 ` Si-Wei Liu
0 siblings, 1 reply; 8+ messages in thread
From: Michael S. Tsirkin @ 2026-11-20 17:35 UTC (permalink / raw)
To: Jonah Palmer
Cc: vlad.wing, brett.creeley, eperezma, jasowangio, jkkm, kernel-team,
linux-kernel, virtualization, xuanzhuo, joao.m.martins,
si-wei.liu
On Fri, Nov 20, 2026 at 12:14:35PM -0500, Jonah Palmer wrote:
> On November 19, 2026 at 6:16 PM EST, Michael S. Tsirkin wrote:
> > it's in my tree.
>
> Hi Vlad, Michael.
>
> I've been looking into this issue extensively. The proposed 48-bit limit
> is a suitable conservative workaround. My virtio-net testing for this
> issue shows that a 51-bit DMA mask also avoids the mentioned issue.
Do u want 51 bit then maybe? x8 memory with no BB?
> Full 52-bit DMA mask support is planned. We will revisit the driver's
> DMA mask once the firmware has been updated.
>
> Thanks,
> Jonah
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-20 17:35 ` Michael S. Tsirkin
@ 2026-11-21 4:35 ` Si-Wei Liu
2026-11-21 11:38 ` Michael S. Tsirkin
0 siblings, 1 reply; 8+ messages in thread
From: Si-Wei Liu @ 2026-11-21 4:35 UTC (permalink / raw)
To: Michael S. Tsirkin, Jonah Palmer
Cc: vlad.wing, brett.creeley, eperezma, jasowangio, jkkm, kernel-team,
linux-kernel, virtualization, xuanzhuo, joao.m.martins
On 11/20/2026 9:35 AM, Michael S. Tsirkin wrote:
> ZjQcmQRYFpfptBannerEnd
> On Fri, Nov 20, 2026 at 12:14:35PM -0500, Jonah Palmer wrote:
> > On November 19, 2026 at 6:16 PM EST, Michael S. Tsirkin wrote:
> > > it's in my tree.
> >
> > Hi Vlad, Michael.
> >
> > I've been looking into this issue extensively. The proposed 48-bit limit
> > is a suitable conservative workaround. My virtio-net testing for this
> > issue shows that a 51-bit DMA mask also avoids the mentioned issue.
>
> Do u want 51 bit then maybe? x8 memory with no BB?
51 should be fine for now, but it's not the final expected limit. We
just wanted to inform you that this is subject to change, once we have
firmware issue fixed. Maybe you could wait until the FW fix is cooked?
It'll be a short while but we will let you know asap.
-Siwei
>
> > Full 52-bit DMA mask support is planned. We will revisit the driver's
> > DMA mask once the firmware has been updated.
> >
> > Thanks,
> > Jonah
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA
2026-11-21 4:35 ` Si-Wei Liu
@ 2026-11-21 11:38 ` Michael S. Tsirkin
0 siblings, 0 replies; 8+ messages in thread
From: Michael S. Tsirkin @ 2026-11-21 11:38 UTC (permalink / raw)
To: Si-Wei Liu
Cc: Jonah Palmer, vlad.wing, brett.creeley, eperezma, jasowangio,
jkkm, kernel-team, linux-kernel, virtualization, xuanzhuo,
joao.m.martins
On Fri, Nov 20, 2026 at 08:35:12PM -0800, Si-Wei Liu wrote:
>
>
> On 11/20/2026 9:35 AM, Michael S. Tsirkin wrote:
> > ZjQcmQRYFpfptBannerEnd
> > On Fri, Nov 20, 2026 at 12:14:35PM -0500, Jonah Palmer wrote:
> > > On November 19, 2026 at 6:16 PM EST, Michael S. Tsirkin wrote:
> > > > it's in my tree.
> > > > Hi Vlad, Michael.
> > > > I've been looking into this issue extensively. The proposed 48-bit
> > limit
> > > is a suitable conservative workaround. My virtio-net testing for this
> > > issue shows that a 51-bit DMA mask also avoids the mentioned issue.
> >
> > Do u want 51 bit then maybe? x8 memory with no BB?
>
> 51 should be fine for now, but it's not the final expected limit. We just
> wanted to inform you that this is subject to change, once we have firmware
> issue fixed. Maybe you could wait until the FW fix is cooked? It'll be a
> short while but we will let you know asap.
>
> -Siwei
Makes sense. If we can get by without kernel patches, that's best.
> >
> > > Full 52-bit DMA mask support is planned. We will revisit the driver's
> > > DMA mask once the firmware has been updated.
> > > > Thanks,
> > > Jonah
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2026-11-21 11:39 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-09 18:42 [PATCH] virtio_pci: limit Oracle DSC2-200 virtio-net to 48-bit DMA Vlad Poenaru
2026-11-09 18:48 ` sashiko-bot
2026-11-19 20:06 ` Vlad Poenaru
2026-11-19 23:16 ` Michael S. Tsirkin
2026-11-20 17:14 ` Jonah Palmer
2026-11-20 17:35 ` Michael S. Tsirkin
2026-11-21 4:35 ` Si-Wei Liu
2026-11-21 11:38 ` Michael S. Tsirkin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.