[PATCH v2 00/12] virtio: support devices that own their virtqueue memory

[Alexander Graf <graf@amazon.com>]

Virtio drivers use guest memory to back virtqueues and their buffers.
That means a VMM needs to be able to map guest memory. That is ok in the
normal virt case. It gets icky with confidential computing (where we use
swiotlb as workaround) and it defeats the purpose of isolated vhost-user
backing devices, because they end up with full RAM access to the guest.

So instead, I'm proposing an extension to virtio which allows it to give
each virtio device its own dedicated memory region to communicate with the
host, called DMB (Device Memory Buffer). A trusted hypervisor can force
DMB to be present, which then enables safer, more isolated and resilient
communication between guest and host.

With DMB, the device provides a shared memory region that both parties
agree is the full memory map both have access to. All memory offsets
that previously would have been into guest RAM, are then offsets into
this shared memory buffer region. One nice property of this is that it
is a generic mechanism in the virtio transport layer, so higher level
drivers work unmodified.

I was exploring to use swiotlb instead to create individual pools. But
that approach has multiple downsides:

1. Swiotlb is an OS primitive which is not available in all Operating
Systems. DMB however lives in the virtio transport layer, which means we
can add support for it in any OS independent of generic layers. This
helps with Windows support.

2. We munge DMA space together. DMB provides a separate DMA space per
virtio device. This means we can for example implement a device in
vhost-user and give the implementing process only visibility to the DMB
region, not all of guest memory. That reduces the exposure the
vhost-user provider has, improving security.

3. Devices can opt-in. A hypervisor can choose to use standard virtio
semantics for self-implemented devices (e.g. NSM), while requiring DMB
for devices implemented by less trustworthy providers. The
non-trustworthy devices do not get any visibility into the trustworthy
ones, even with DMB in place for both.

I also looked at virtio-iommu. To restrict DMA visibility, virtio-iommu
allows the guest to open specific windows into guest memory to the
device, but it comes with its own bag of problems, such as dynamic
allocations and complicated device <-> iommu connections that need to be
represented reliably.

== Limitations ==

  - Only PCI is wired up.
  - Feature bit 44 and the two registers at offsets 0x40 and 0x42 of the
    PCI common configuration are provisional: the OASIS technical
    committee has the specification and has allocated none of them.

  https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/

== Testing ==

I ran this against a device that offers DMB and not
VIRTIO_F_ACCESS_PLATFORM, with a 16 MiB region on each device. A kernel
without patch 10 finds the region and then hangs with nothing in the log;
with it, 512 MiB of O_DIRECT block reads and 137 MiB of loopback network
traffic go through the regions with no allocation failure.

Patches 1 to 4 do not need the rest of the series. They remove an API no
driver calls and a stale worked example in a vdpa comment. Patch 3 and
patch 4 carry Fixes:; patch 4's are older than this series, a failed
mapping arriving from a packed ring as -EIO, which fails an I/O that on a
split ring is only back-pressure.

I wrote this series with an AI coding assistant, which drafted the code,
the changelogs and this cover letter. I reviewed and reworked all of it,
and every commit carries an Assisted-by: trailer.

Alex

v1: https://lore.kernel.org/all/20260809182010.32931-1-graf@amazon.com/

v1 -> v2:

  - Remove the unused map sync API (Michael)
  - Drop the sync operations from virtio_map_ops (Michael)
  - Drop the stale mask expansion instead of correcting it (Michael)
  - Treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
  - Read the region's memory type and accept only a coherent one
  - Rewrite the region allocator over gen_pool
  - Return -ENOMEM instead of -EIO from a failed packed ring mapping
  - Drop the patch validating premapped addresses through the map
  - Drop the patch reporting a bounded pool's exhaustion as -ENOSPC
  - Drop the range withheld for a virtqueue's first descriptor chain
  - Drop the .rst and describe the region in the headers instead

Alexander Graf (12):
  virtio_ring: remove the unused map sync API
  virtio: drop the sync operations from virtio_map_ops
  vdpa: drop the VIRTIO_DEVICE_F_MASK example value
  virtio_ring: return -ENOMEM when a packed ring mapping fails
  virtio: add the VIRTIO_F_DMB feature bit
  virtio_pci: read the device memory buffer registers
  virtio_pci: create virtqueues with the device's mapping token
  virtio: add a device memory buffer region allocator
  virtio: locate the device memory buffer after feature negotiation
  virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
  virtio_pci: support VIRTIO_F_DMB
  virtio: expose device memory buffer occupancy over debugfs

 drivers/vdpa/vdpa.c                    |   7 +-
 drivers/vdpa/vdpa_user/iova_domain.c   |  20 -
 drivers/vdpa/vdpa_user/iova_domain.h   |   8 -
 drivers/vdpa/vdpa_user/vduse_dev.c     |  40 --
 drivers/virtio/Kconfig                 |  17 +
 drivers/virtio/Makefile                |   3 +-
 drivers/virtio/virtio.c                |  25 +-
 drivers/virtio/virtio_dmb.c            | 959 +++++++++++++++++++++++++
 drivers/virtio/virtio_dmb.h            |  28 +
 drivers/virtio/virtio_pci_modern.c     |  75 +-
 drivers/virtio/virtio_pci_modern_dev.c |  47 +-
 drivers/virtio/virtio_ring.c           |  93 +--
 include/linux/virtio.h                 |  11 +-
 include/linux/virtio_config.h          |  41 +-
 include/linux/virtio_pci_modern.h      |   2 +
 include/uapi/linux/virtio_config.h     |  24 +-
 include/uapi/linux/virtio_pci.h        |  19 +
 tools/virtio/linux/dma-mapping.h       |   7 -
 18 files changed, 1217 insertions(+), 209 deletions(-)
 create mode 100644 drivers/virtio/virtio_dmb.c
 create mode 100644 drivers/virtio/virtio_dmb.h


base-commit: fc02acf6ac0ccde0c805c2daa9148683cdd01ba8