[PATCH v2 00/12] virtio: support devices that own their virtqueue memory

[PATCH v2 00/12] virtio: support devices that own their virtqueue memory

[Alexander Graf]

Virtio drivers use guest memory to back virtqueues and their buffers.
That means a VMM needs to be able to map guest memory. That is ok in the
normal virt case. It gets icky with confidential computing (where we use
swiotlb as workaround) and it defeats the purpose of isolated vhost-user
backing devices, because they end up with full RAM access to the guest.

So instead, I'm proposing an extension to virtio which allows it to give
each virtio device its own dedicated memory region to communicate with the
host, called DMB (Device Memory Buffer). A trusted hypervisor can force
DMB to be present, which then enables safer, more isolated and resilient
communication between guest and host.

With DMB, the device provides a shared memory region that both parties
agree is the full memory map both have access to. All memory offsets
that previously would have been into guest RAM, are then offsets into
this shared memory buffer region. One nice property of this is that it
is a generic mechanism in the virtio transport layer, so higher level
drivers work unmodified.

I was exploring to use swiotlb instead to create individual pools. But
that approach has multiple downsides:

1. Swiotlb is an OS primitive which is not available in all Operating
Systems. DMB however lives in the virtio transport layer, which means we
can add support for it in any OS independent of generic layers. This
helps with Windows support.

2. We munge DMA space together. DMB provides a separate DMA space per
virtio device. This means we can for example implement a device in
vhost-user and give the implementing process only visibility to the DMB
region, not all of guest memory. That reduces the exposure the
vhost-user provider has, improving security.

3. Devices can opt-in. A hypervisor can choose to use standard virtio
semantics for self-implemented devices (e.g. NSM), while requiring DMB
for devices implemented by less trustworthy providers. The
non-trustworthy devices do not get any visibility into the trustworthy
ones, even with DMB in place for both.

I also looked at virtio-iommu. To restrict DMA visibility, virtio-iommu
allows the guest to open specific windows into guest memory to the
device, but it comes with its own bag of problems, such as dynamic
allocations and complicated device <-> iommu connections that need to be
represented reliably.

== Limitations ==

  - Only PCI is wired up.
  - Feature bit 44 and the two registers at offsets 0x40 and 0x42 of the
    PCI common configuration are provisional: the OASIS technical
    committee has the specification and has allocated none of them.

  https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/

== Testing ==

I ran this against a device that offers DMB and not
VIRTIO_F_ACCESS_PLATFORM, with a 16 MiB region on each device. A kernel
without patch 10 finds the region and then hangs with nothing in the log;
with it, 512 MiB of O_DIRECT block reads and 137 MiB of loopback network
traffic go through the regions with no allocation failure.

Patches 1 to 4 do not need the rest of the series. They remove an API no
driver calls and a stale worked example in a vdpa comment. Patch 3 and
patch 4 carry Fixes:; patch 4's are older than this series, a failed
mapping arriving from a packed ring as -EIO, which fails an I/O that on a
split ring is only back-pressure.

I wrote this series with an AI coding assistant, which drafted the code,
the changelogs and this cover letter. I reviewed and reworked all of it,
and every commit carries an Assisted-by: trailer.

Alex

v1: https://lore.kernel.org/all/20260809182010.32931-1-graf@amazon.com/

v1 -> v2:

  - Remove the unused map sync API (Michael)
  - Drop the sync operations from virtio_map_ops (Michael)
  - Drop the stale mask expansion instead of correcting it (Michael)
  - Treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
  - Read the region's memory type and accept only a coherent one
  - Rewrite the region allocator over gen_pool
  - Return -ENOMEM instead of -EIO from a failed packed ring mapping
  - Drop the patch validating premapped addresses through the map
  - Drop the patch reporting a bounded pool's exhaustion as -ENOSPC
  - Drop the range withheld for a virtqueue's first descriptor chain
  - Drop the .rst and describe the region in the headers instead

Alexander Graf (12):
  virtio_ring: remove the unused map sync API
  virtio: drop the sync operations from virtio_map_ops
  vdpa: drop the VIRTIO_DEVICE_F_MASK example value
  virtio_ring: return -ENOMEM when a packed ring mapping fails
  virtio: add the VIRTIO_F_DMB feature bit
  virtio_pci: read the device memory buffer registers
  virtio_pci: create virtqueues with the device's mapping token
  virtio: add a device memory buffer region allocator
  virtio: locate the device memory buffer after feature negotiation
  virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM
  virtio_pci: support VIRTIO_F_DMB
  virtio: expose device memory buffer occupancy over debugfs

 drivers/vdpa/vdpa.c                    |   7 +-
 drivers/vdpa/vdpa_user/iova_domain.c   |  20 -
 drivers/vdpa/vdpa_user/iova_domain.h   |   8 -
 drivers/vdpa/vdpa_user/vduse_dev.c     |  40 --
 drivers/virtio/Kconfig                 |  17 +
 drivers/virtio/Makefile                |   3 +-
 drivers/virtio/virtio.c                |  25 +-
 drivers/virtio/virtio_dmb.c            | 959 +++++++++++++++++++++++++
 drivers/virtio/virtio_dmb.h            |  28 +
 drivers/virtio/virtio_pci_modern.c     |  75 +-
 drivers/virtio/virtio_pci_modern_dev.c |  47 +-
 drivers/virtio/virtio_ring.c           |  93 +--
 include/linux/virtio.h                 |  11 +-
 include/linux/virtio_config.h          |  41 +-
 include/linux/virtio_pci_modern.h      |   2 +
 include/uapi/linux/virtio_config.h     |  24 +-
 include/uapi/linux/virtio_pci.h        |  19 +
 tools/virtio/linux/dma-mapping.h       |   7 -
 18 files changed, 1217 insertions(+), 209 deletions(-)
 create mode 100644 drivers/virtio/virtio_dmb.c
 create mode 100644 drivers/virtio/virtio_dmb.h


base-commit: fc02acf6ac0ccde0c805c2daa9148683cdd01ba8

[PATCH v2 01/12] virtio_ring: remove the unused map sync API

[Alexander Graf]

virtqueue_map_need_sync(), virtqueue_map_sync_single_range_for_cpu() and
virtqueue_map_sync_single_range_for_device() are exported, but no driver
in the tree calls them. They are the only path into the sync members of
struct virtio_map_ops, so whoever implements that struct next has to
decide what those members do with no caller to learn it from.

Remove the three functions, their prototypes and their exports, and the
DMA sync shims that the virtio userspace harness kept for them. With
this, virtio_ring no longer exports a way to sync a mapping.

Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Kiro:claude-opus-5 checkpatch
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio_ring.c     | 89 --------------------------------
 include/linux/virtio.h           |  8 ---
 tools/virtio/linux/dma-mapping.h |  7 ---
 3 files changed, 104 deletions(-)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index b438dc2ce1b8..ea8e774b6d8e 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -3890,94 +3890,5 @@ int virtqueue_map_mapping_error(const struct virtqueue *_vq, dma_addr_t addr)
 }
 EXPORT_SYMBOL_GPL(virtqueue_map_mapping_error);
 
-/**
- * virtqueue_map_need_sync - check a dma address needs sync
- * @_vq: the struct virtqueue we're talking about.
- * @addr: DMA address
- *
- * Check if the dma address mapped by the virtqueue_map_* APIs needs to be
- * synchronized
- *
- * return bool
- */
-bool virtqueue_map_need_sync(const struct virtqueue *_vq, dma_addr_t addr)
-{
-	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
-
-	if (!vq->use_map_api)
-		return false;
-
-	if (vdev->map)
-		return vdev->map->need_sync(vq->map, addr);
-	else
-		return dma_need_sync(vring_dma_dev(vq), addr);
-}
-EXPORT_SYMBOL_GPL(virtqueue_map_need_sync);
-
-/**
- * virtqueue_map_sync_single_range_for_cpu - map sync for cpu
- * @_vq: the struct virtqueue we're talking about.
- * @addr: DMA address
- * @offset: DMA address offset
- * @size: buf size for sync
- * @dir: DMA direction
- *
- * Before calling this function, use virtqueue_map_need_sync() to confirm that
- * the DMA address really needs to be synchronized
- *
- */
-void virtqueue_map_sync_single_range_for_cpu(const struct virtqueue *_vq,
-					     dma_addr_t addr,
-					     unsigned long offset, size_t size,
-					     enum dma_data_direction dir)
-{
-	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
-
-	if (!vq->use_map_api)
-		return;
-
-	if (vdev->map)
-		vdev->map->sync_single_for_cpu(vq->map,
-					       addr + offset, size, dir);
-	else
-		dma_sync_single_range_for_cpu(vring_dma_dev(vq),
-					      addr, offset, size, dir);
-}
-EXPORT_SYMBOL_GPL(virtqueue_map_sync_single_range_for_cpu);
-
-/**
- * virtqueue_map_sync_single_range_for_device - map sync for device
- * @_vq: the struct virtqueue we're talking about.
- * @addr: DMA address
- * @offset: DMA address offset
- * @size: buf size for sync
- * @dir: DMA direction
- *
- * Before calling this function, use virtqueue_map_need_sync() to confirm that
- * the DMA address really needs to be synchronized
- */
-void virtqueue_map_sync_single_range_for_device(const struct virtqueue *_vq,
-						dma_addr_t addr,
-						unsigned long offset, size_t size,
-						enum dma_data_direction dir)
-{
-	const struct vring_virtqueue *vq = to_vvq(_vq);
-	struct virtio_device *vdev = _vq->vdev;
-
-	if (!vq->use_map_api)
-		return;
-
-	if (vdev->map)
-		vdev->map->sync_single_for_device(vq->map,
-						  addr + offset,
-						  size, dir);
-	else
-		dma_sync_single_range_for_device(vring_dma_dev(vq), addr,
-						 offset, size, dir);
-}
-EXPORT_SYMBOL_GPL(virtqueue_map_sync_single_range_for_device);
-
 MODULE_DESCRIPTION("Virtio ring implementation");
 MODULE_LICENSE("GPL");
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 93e573c56563..3122fc52a7c3 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -312,14 +312,6 @@ void virtqueue_unmap_single_attrs(const struct virtqueue *_vq, dma_addr_t addr,
 				      unsigned long attrs);
 int virtqueue_map_mapping_error(const struct virtqueue *_vq, dma_addr_t addr);
 
-bool virtqueue_map_need_sync(const struct virtqueue *_vq, dma_addr_t addr);
-void virtqueue_map_sync_single_range_for_cpu(const struct virtqueue *_vq, dma_addr_t addr,
-					     unsigned long offset, size_t size,
-					     enum dma_data_direction dir);
-void virtqueue_map_sync_single_range_for_device(const struct virtqueue *_vq, dma_addr_t addr,
-						unsigned long offset, size_t size,
-						enum dma_data_direction dir);
-
 #ifdef CONFIG_VIRTIO_DEBUG
 void virtio_debug_device_init(struct virtio_device *dev);
 void virtio_debug_device_exit(struct virtio_device *dev);
diff --git a/tools/virtio/linux/dma-mapping.h b/tools/virtio/linux/dma-mapping.h
index b9fc5e8338e3..eae160ce7142 100644
--- a/tools/virtio/linux/dma-mapping.h
+++ b/tools/virtio/linux/dma-mapping.h
@@ -36,16 +36,9 @@ enum dma_data_direction {
 
 #define sg_dma_address(sg) (0)
 #define sg_dma_len(sg) (0)
-#define dma_need_sync(v, a) (0)
 #define dma_unmap_single_attrs(d, a, s, r, t) do { \
 	(void)(d); (void)(a); (void)(s); (void)(r); (void)(t); \
 } while (0)
-#define dma_sync_single_range_for_cpu(d, a, o, s, r) do { \
-	(void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \
-} while (0)
-#define dma_sync_single_range_for_device(d, a, o, s, r) do { \
-	(void)(d); (void)(a); (void)(o); (void)(s); (void)(r); \
-} while (0)
 #define dma_max_mapping_size(...) SIZE_MAX
 
 /*

[PATCH v2 02/12] virtio: drop the sync operations from virtio_map_ops

[Alexander Graf]

With the map sync API gone, sync_single_for_cpu, sync_single_for_device
and need_sync in struct virtio_map_ops are members the virtqueue never
calls. vduse is the only implementation in the tree and provides all
three, and the next implementer has to decide what to put in them.

Drop the three members and their kernel-doc, and remove the vduse
functions that provided them. vduse_domain_bounce() stays, because
vduse_domain_map_page() and vduse_domain_unmap_page() bounce through it
as well. That way the struct asks an implementation for six operations
instead of nine.

Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Kiro:claude-opus-5 checkpatch
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/vdpa/vdpa_user/iova_domain.c | 20 --------------
 drivers/vdpa/vdpa_user/iova_domain.h |  8 ------
 drivers/vdpa/vdpa_user/vduse_dev.c   | 40 ----------------------------
 include/linux/virtio_config.h        | 20 --------------
 4 files changed, 88 deletions(-)

diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c
index 4dc76c0d0d13..094fe1ec02c3 100644
--- a/drivers/vdpa/vdpa_user/iova_domain.c
+++ b/drivers/vdpa/vdpa_user/iova_domain.c
@@ -424,26 +424,6 @@ static void vduse_domain_free_iova(struct iova_domain *iovad,
 	free_iova_fast(iovad, iova >> shift, iova_len);
 }
 
-void vduse_domain_sync_single_for_device(struct vduse_iova_domain *domain,
-				      dma_addr_t dma_addr, size_t size,
-				      enum dma_data_direction dir)
-{
-	read_lock(&domain->bounce_lock);
-	if (dir == DMA_TO_DEVICE || dir == DMA_BIDIRECTIONAL)
-		vduse_domain_bounce(domain, dma_addr, size, DMA_TO_DEVICE);
-	read_unlock(&domain->bounce_lock);
-}
-
-void vduse_domain_sync_single_for_cpu(struct vduse_iova_domain *domain,
-				      dma_addr_t dma_addr, size_t size,
-				      enum dma_data_direction dir)
-{
-	read_lock(&domain->bounce_lock);
-	if (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL)
-		vduse_domain_bounce(domain, dma_addr, size, DMA_FROM_DEVICE);
-	read_unlock(&domain->bounce_lock);
-}
-
 dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain,
 				 struct page *page, unsigned long offset,
 				 size_t size, enum dma_data_direction dir,
diff --git a/drivers/vdpa/vdpa_user/iova_domain.h b/drivers/vdpa/vdpa_user/iova_domain.h
index e50e55d1396f..66f14764a5db 100644
--- a/drivers/vdpa/vdpa_user/iova_domain.h
+++ b/drivers/vdpa/vdpa_user/iova_domain.h
@@ -48,14 +48,6 @@ int vduse_domain_set_map(struct vduse_iova_domain *domain,
 void vduse_domain_clear_map(struct vduse_iova_domain *domain,
 			    struct vhost_iotlb *iotlb);
 
-void vduse_domain_sync_single_for_device(struct vduse_iova_domain *domain,
-				      dma_addr_t dma_addr, size_t size,
-				      enum dma_data_direction dir);
-
-void vduse_domain_sync_single_for_cpu(struct vduse_iova_domain *domain,
-				      dma_addr_t dma_addr, size_t size,
-				      enum dma_data_direction dir);
-
 dma_addr_t vduse_domain_map_page(struct vduse_iova_domain *domain,
 				 struct page *page, unsigned long offset,
 				 size_t size, enum dma_data_direction dir,
diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c
index 10dcf016bfb0..7a1a344539b8 100644
--- a/drivers/vdpa/vdpa_user/vduse_dev.c
+++ b/drivers/vdpa/vdpa_user/vduse_dev.c
@@ -932,34 +932,6 @@ static const struct vdpa_config_ops vduse_vdpa_config_ops = {
 	.free			= vduse_vdpa_free,
 };
 
-static void vduse_dev_sync_single_for_device(union virtio_map token,
-					     dma_addr_t dma_addr, size_t size,
-					     enum dma_data_direction dir)
-{
-	struct vduse_iova_domain *domain;
-
-	if (!token.group)
-		return;
-
-	guard(vq_group_as_read_lock)(token.group);
-	domain = token.group->as->domain;
-	vduse_domain_sync_single_for_device(domain, dma_addr, size, dir);
-}
-
-static void vduse_dev_sync_single_for_cpu(union virtio_map token,
-					     dma_addr_t dma_addr, size_t size,
-					     enum dma_data_direction dir)
-{
-	struct vduse_iova_domain *domain;
-
-	if (!token.group)
-		return;
-
-	guard(vq_group_as_read_lock)(token.group);
-	domain = token.group->as->domain;
-	vduse_domain_sync_single_for_cpu(domain, dma_addr, size, dir);
-}
-
 static dma_addr_t vduse_dev_map_page(union virtio_map token, struct page *page,
 				     unsigned long offset, size_t size,
 				     enum dma_data_direction dir,
@@ -1037,15 +1009,6 @@ static void vduse_dev_free_coherent(union virtio_map token, size_t size,
 	free_pages_exact(vaddr, size);
 }
 
-static bool vduse_dev_need_sync(union virtio_map token, dma_addr_t dma_addr)
-{
-	if (!token.group)
-		return false;
-
-	guard(vq_group_as_read_lock)(token.group);
-	return dma_addr < token.group->as->domain->bounce_size;
-}
-
 static int vduse_dev_mapping_error(union virtio_map token, dma_addr_t dma_addr)
 {
 	if (unlikely(dma_addr == DMA_MAPPING_ERROR))
@@ -1063,13 +1026,10 @@ static size_t vduse_dev_max_mapping_size(union virtio_map token)
 }
 
 static const struct virtio_map_ops vduse_map_ops = {
-	.sync_single_for_device = vduse_dev_sync_single_for_device,
-	.sync_single_for_cpu = vduse_dev_sync_single_for_cpu,
 	.map_page = vduse_dev_map_page,
 	.unmap_page = vduse_dev_unmap_page,
 	.alloc = vduse_dev_alloc_coherent,
 	.free = vduse_dev_free_coherent,
-	.need_sync = vduse_dev_need_sync,
 	.mapping_error = vduse_dev_mapping_error,
 	.max_mapping_size = vduse_dev_max_mapping_size,
 };
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index 69f84ea85d71..eb38c876a628 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -157,16 +157,6 @@ struct virtio_config_ops {
  *      size: the buffer size
  *      dir: mapping direction
  *      attrs: unmapping attributes
- * @sync_single_for_cpu: sync a single buffer from device to cpu
- *      map: metadata for performing mapping
- *      map_handle: the mapping address to sync
- *      size: the size of the buffer
- *      dir: synchronization direction
- * @sync_single_for_device: sync a single buffer from cpu to device
- *      map: metadata for performing mapping
- *      map_handle: the mapping address to sync
- *      size: the size of the buffer
- *      dir: synchronization direction
  * @alloc: alloc a coherent buffer mapping
  *      map: metadata for performing mapping
  *      size: the size of the buffer
@@ -179,10 +169,6 @@ struct virtio_config_ops {
  *      vaddr: virtual address of the buffer
  *      map_handle: the mapping address that needs to be freed
  *      attrs: unmapping attributes
- * @need_sync: if the buffer needs synchronization
- *      map: metadata for performing mapping
- *      map_handle: the mapped address
- *      Returns whether the buffer needs synchronization
  * @mapping_error: if the mapping address is error
  *      map: metadata for performing mapping
  *      map_handle: the mapped address
@@ -197,16 +183,10 @@ struct virtio_map_ops {
 	void (*unmap_page)(union virtio_map map, dma_addr_t map_handle,
 			   size_t size, enum dma_data_direction dir,
 			   unsigned long attrs);
-	void (*sync_single_for_cpu)(union virtio_map map, dma_addr_t map_handle,
-				    size_t size, enum dma_data_direction dir);
-	void (*sync_single_for_device)(union virtio_map map,
-				       dma_addr_t map_handle, size_t size,
-				       enum dma_data_direction dir);
 	void *(*alloc)(union virtio_map map, size_t size,
 		       dma_addr_t *map_handle, gfp_t gfp);
 	void (*free)(union virtio_map map, size_t size, void *vaddr,
 		     dma_addr_t map_handle, unsigned long attrs);
-	bool (*need_sync)(union virtio_map map, dma_addr_t map_handle);
 	int (*mapping_error)(union virtio_map map, dma_addr_t map_handle);
 	size_t (*max_mapping_size)(union virtio_map map);
 };

[PATCH v2 03/12] vdpa: drop the VIRTIO_DEVICE_F_MASK example value

[Alexander Graf]

The comment above VIRTIO_DEVICE_F_MASK spells the mask out by hand as
0xfffffc000fffffff, which was correct while VIRTIO_TRANSPORT_F_END was
41. The bound is 42 now, so the macro expands to 0xfffff8000fffffff and
whoever reads the comment to check a feature mask against it gets the
wrong boundary between transport and per-device bits.

Correcting the number would leave a comment that has to be updated by
hand every time the bound moves, and we already missed it once. Drop the
expansion instead. The macro derives its value from the bound and needs
no change.

Fixes: 838bebb4c926 ("virtio: Define feature bit for administration virtqueue")
Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/vdpa/vdpa.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c
index 47c6c3d23f5c..395ca549573a 100644
--- a/drivers/vdpa/vdpa.c
+++ b/drivers/vdpa/vdpa.c
@@ -548,10 +548,9 @@ vdpa_nl_cmd_mgmtdev_get_dumpit(struct sk_buff *msg, struct netlink_callback *cb)
 
 /*
  * Bitmask for all per-device features: feature bits VIRTIO_TRANSPORT_F_START
- * through VIRTIO_TRANSPORT_F_END are unset, i.e. 0xfffffc000fffffff for
- * all 64bit features. If the features are extended beyond 64 bits, or new
- * "holes" are reserved for other type of features than per-device, this
- * macro would have to be updated.
+ * through VIRTIO_TRANSPORT_F_END are unset. If the features are extended
+ * beyond 64 bits, or new "holes" are reserved for other type of features
+ * than per-device, this macro would have to be updated.
  */
 #define VIRTIO_DEVICE_F_MASK (~0ULL << (VIRTIO_TRANSPORT_F_END + 1) | \
 			      ((1ULL << VIRTIO_TRANSPORT_F_START) - 1))

[PATCH v2 04/12] virtio_ring: return -ENOMEM when a packed ring mapping fails

[Alexander Graf]

Commit f7728002c1c7 ("virtio_ring: fix return code on DMA mapping
fails") moved virtqueue_add_split() and virtqueue_add_indirect_packed()
to -ENOMEM, because virtio_queue_rq() maps -EIO to BLK_STS_IOERR and
the request fails. We still return -EIO from virtqueue_add_packed(),
and virtqueue_add_packed_in_order() copied that when it was added later.

Guests that bounce their I/O through swiotlb (SEV-SNP, TDX, s390 secure
execution) run the pool out with enough I/O in flight. On a split ring
virtio_queue_rq() reports BLK_STS_RESOURCE and the block layer requeues
the request. On a packed ring virtio_queue_rq() reports BLK_STS_IOERR
instead and the error reaches the filesystem.

Return -ENOMEM from the packed unmap_release paths too. Both are reached
from a single goto on a failed mapping, which is where
vring_map_one_sg() already produces -ENOMEM.

That way every ring layout reports the same errno, and the block layer
requeues the request instead of failing it.

Fixes: f7728002c1c7 ("virtio_ring: fix return code on DMA mapping fails")
Fixes: f6a15d854986 ("virtio_ring: add in order support")
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio_ring.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index ea8e774b6d8e..b7b03166a301 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -1810,7 +1810,7 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq,
 	}
 
 	END_USE(vq);
-	return -EIO;
+	return -ENOMEM;
 }
 
 static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
@@ -1966,7 +1966,7 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
 	}
 
 	END_USE(vq);
-	return -EIO;
+	return -ENOMEM;
 }
 
 static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq)

[PATCH v2 05/12] virtio: add the VIRTIO_F_DMB feature bit

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, define the feature bit. A device
that negotiates it holds its virtqueues and the buffers they reference in
a Device Memory Buffer, a shared memory region the device owns. The data
DMA of such a device is routed through a per-device transparent IOMMU
whose bus address space maps 1:1 onto that region, so every address the
driver places in a virtqueue is an address in that space and the device
can reach nothing the driver has not published there.

Define VIRTIO_F_DMB as 44 and raise VIRTIO_TRANSPORT_F_END to 45, so
virtio_dev_probe() preserves the bit into finalize_features() where a
transport can act on it. VIRTIO_DEVICE_F_MASK derives from the same bound
and moves with it. vdpa reads the bound as VIRTIO_TRANSPORT_F_END + 1
while vring_transport_features() stops below it, so the mask drops bits
43 to 45 from the device_features vdpa reports over netlink, one bit
beyond the last bit a transport can act on.

Define VIRTIO_DMB_MEM_TYPE_COHERENT too. A device reports the memory
type of its region, and this is the one type that says a write by either
side becomes visible to the other with no cache maintenance by the
driver. Every transport reports the same type, so the name belongs here
beside the feature bit and not in the header of one of them.

Link: https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---

  - Say bus address space rather than IOVA space

---
 include/uapi/linux/virtio_config.h | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/virtio_config.h b/include/uapi/linux/virtio_config.h
index 2445f365bce7..8cfd9436a09e 100644
--- a/include/uapi/linux/virtio_config.h
+++ b/include/uapi/linux/virtio_config.h
@@ -52,7 +52,7 @@
  * rest are per-device feature bits.
  */
 #define VIRTIO_TRANSPORT_F_START	28
-#define VIRTIO_TRANSPORT_F_END		42
+#define VIRTIO_TRANSPORT_F_END		45
 
 #ifndef VIRTIO_CONFIG_NO_LEGACY
 /* Do we get callbacks when the ring is completely used, even if we've
@@ -120,4 +120,26 @@
  */
 #define VIRTIO_F_ADMIN_VQ		41
 
+/*
+ * This feature indicates that the device offers a Device Memory Buffer: a
+ * shared memory region, owned by the device, that holds the virtqueues and
+ * the buffers they reference.  When this feature is negotiated, the data DMA
+ * of the device is routed through a per-device transparent IOMMU whose bus
+ * address space maps 1:1 onto that region, so every address the driver
+ * places in a virtqueue is an address in that space and the device can
+ * reach nothing the driver has not published there.  Interrupts are
+ * unaffected.
+ */
+#define VIRTIO_F_DMB			44
+
+/*
+ * The memory type of a Device Memory Buffer region, as the device reports it.
+ * VIRTIO_DMB_MEM_TYPE_COHERENT means a write by either side becomes visible to
+ * the other with no cache maintenance by the driver.  Every other value is
+ * reserved.  A device reports the type from the moment it offers
+ * VIRTIO_F_DMB, and a driver must not accept the feature unless the type is
+ * one it supports.
+ */
+#define VIRTIO_DMB_MEM_TYPE_COHERENT	0
+
 #endif /* _UAPI_LINUX_VIRTIO_CONFIG_H */

[PATCH v2 06/12] virtio_pci: read the device memory buffer registers

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, read the two registers a device
uses to describe its Device Memory Buffer. Both are read-only le16
fields at the end of the common configuration structure. dmb_shm_id
names the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that
virtio_pci_find_shm_cap() has to find to map the region, and is valid
once the feature is negotiated. dmb_mem_type describes the memory the
region is made of: VIRTIO_DMB_MEM_TYPE_COHERENT means a write by either
side becomes visible to the other with no cache maintenance by the
driver. It is valid as soon as the device offers the feature, so a
driver that does not support the value it reads can decline the offer
instead of failing the device afterwards.

Add both fields, VIRTIO_PCI_COMMON_DMB_SHM_ID and
VIRTIO_PCI_COMMON_DMB_MEM_TYPE for the offsets check_offsets() asserts
them against, and vp_modern_get_dmb_shm_id() and
vp_modern_get_dmb_mem_type() to read them. vp_modern_probe() capped the
common cfg mapping at the end of admin_queue_num, exactly where
dmb_shm_id starts, so extend it to the end of dmb_mem_type.
vp_modern_map_capability() takes that as an upper bound, so a device with
a shorter common cfg maps what it has and mdev->common_len records how
much.

Link: https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio_pci_modern_dev.c | 47 +++++++++++++++++++++++++-
 include/linux/virtio_pci_modern.h      |  2 ++
 include/uapi/linux/virtio_pci.h        | 19 +++++++++++
 3 files changed, 67 insertions(+), 1 deletion(-)

diff --git a/drivers/virtio/virtio_pci_modern_dev.c b/drivers/virtio/virtio_pci_modern_dev.c
index 413a8c353463..270c67fe58ff 100644
--- a/drivers/virtio/virtio_pci_modern_dev.c
+++ b/drivers/virtio/virtio_pci_modern_dev.c
@@ -211,6 +211,10 @@ static inline void check_offsets(void)
 		     offsetof(struct virtio_pci_modern_common_cfg, admin_queue_index));
 	BUILD_BUG_ON(VIRTIO_PCI_COMMON_ADM_Q_NUM !=
 		     offsetof(struct virtio_pci_modern_common_cfg, admin_queue_num));
+	BUILD_BUG_ON(VIRTIO_PCI_COMMON_DMB_SHM_ID !=
+		     offsetof(struct virtio_pci_modern_common_cfg, dmb_shm_id));
+	BUILD_BUG_ON(VIRTIO_PCI_COMMON_DMB_MEM_TYPE !=
+		     offsetof(struct virtio_pci_modern_common_cfg, dmb_mem_type));
 }
 
 /*
@@ -300,7 +304,7 @@ int vp_modern_probe(struct virtio_pci_modern_device *mdev)
 	mdev->common = vp_modern_map_capability(mdev, common,
 			      sizeof(struct virtio_pci_common_cfg), 4, 0,
 			      offsetofend(struct virtio_pci_modern_common_cfg,
-					  admin_queue_num),
+					  dmb_mem_type),
 			      &mdev->common_len, NULL);
 	if (!mdev->common)
 		goto err_map_common;
@@ -752,6 +756,47 @@ u16 vp_modern_avq_index(struct virtio_pci_modern_device *mdev)
 }
 EXPORT_SYMBOL_GPL(vp_modern_avq_index);
 
+/*
+ * vp_modern_get_dmb_shm_id - read the Device Memory Buffer shared memory id
+ * @mdev: the modern virtio-pci device
+ *
+ * The value identifies the VIRTIO_PCI_CAP_SHARED_MEMORY_CFG capability that
+ * describes the Device Memory Buffer region.  Only valid once VIRTIO_F_DMB
+ * has been negotiated, and the caller has to have established that the common
+ * configuration structure is long enough to hold the field.
+ *
+ * Returns the shared memory id.
+ */
+u16 vp_modern_get_dmb_shm_id(struct virtio_pci_modern_device *mdev)
+{
+	struct virtio_pci_modern_common_cfg __iomem *cfg;
+
+	cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common;
+	return vp_ioread16(&cfg->dmb_shm_id);
+}
+EXPORT_SYMBOL_GPL(vp_modern_get_dmb_shm_id);
+
+/*
+ * vp_modern_get_dmb_mem_type - read the Device Memory Buffer memory type
+ * @mdev: the modern virtio-pci device
+ *
+ * The value describes the memory the Device Memory Buffer region is made of.
+ * VIRTIO_DMB_MEM_TYPE_COHERENT means the region is cache coherent.  Valid
+ * whenever the device offers VIRTIO_F_DMB, and the caller has to have
+ * established that the common configuration structure is long enough to hold
+ * the field.
+ *
+ * Returns the memory type.
+ */
+u16 vp_modern_get_dmb_mem_type(struct virtio_pci_modern_device *mdev)
+{
+	struct virtio_pci_modern_common_cfg __iomem *cfg;
+
+	cfg = (struct virtio_pci_modern_common_cfg __iomem *)mdev->common;
+	return vp_ioread16(&cfg->dmb_mem_type);
+}
+EXPORT_SYMBOL_GPL(vp_modern_get_dmb_mem_type);
+
 MODULE_VERSION("0.1");
 MODULE_DESCRIPTION("Modern Virtio PCI Device");
 MODULE_AUTHOR("Jason Wang <jasowang@redhat.com>");
diff --git a/include/linux/virtio_pci_modern.h b/include/linux/virtio_pci_modern.h
index 9a3f2fc53bd6..80643c895495 100644
--- a/include/linux/virtio_pci_modern.h
+++ b/include/linux/virtio_pci_modern.h
@@ -162,4 +162,6 @@ int vp_modern_get_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
 void vp_modern_set_queue_reset(struct virtio_pci_modern_device *mdev, u16 index);
 u16 vp_modern_avq_num(struct virtio_pci_modern_device *mdev);
 u16 vp_modern_avq_index(struct virtio_pci_modern_device *mdev);
+u16 vp_modern_get_dmb_shm_id(struct virtio_pci_modern_device *mdev);
+u16 vp_modern_get_dmb_mem_type(struct virtio_pci_modern_device *mdev);
 #endif
diff --git a/include/uapi/linux/virtio_pci.h b/include/uapi/linux/virtio_pci.h
index e732e3456e27..1bd2bb981dc9 100644
--- a/include/uapi/linux/virtio_pci.h
+++ b/include/uapi/linux/virtio_pci.h
@@ -193,6 +193,23 @@ struct virtio_pci_modern_common_cfg {
 
 	__le16 admin_queue_index;	/* read-only */
 	__le16 admin_queue_num;		/* read-only */
+
+	/*
+	 * Reports the shmid of the Device Memory Buffer region.  Valid once
+	 * VIRTIO_F_DMB has been negotiated.
+	 */
+	__le16 dmb_shm_id;		/* read-only */
+
+	/*
+	 * Reports the memory type of the Device Memory Buffer region.
+	 * VIRTIO_DMB_MEM_TYPE_COHERENT means the region is cache coherent: a
+	 * write by either side becomes visible to the other with no cache
+	 * maintenance by the driver.  Every other value is reserved, and a
+	 * driver must not accept VIRTIO_F_DMB unless the value is one it
+	 * supports.  Valid whenever the device offers VIRTIO_F_DMB, so that a
+	 * driver can read it before it accepts.
+	 */
+	__le16 dmb_mem_type;		/* read-only */
 };
 
 /* Fields in VIRTIO_PCI_CAP_PCI_CFG: */
@@ -235,6 +252,8 @@ struct virtio_pci_cfg_cap {
 #define VIRTIO_PCI_COMMON_Q_RESET	58
 #define VIRTIO_PCI_COMMON_ADM_Q_IDX	60
 #define VIRTIO_PCI_COMMON_ADM_Q_NUM	62
+#define VIRTIO_PCI_COMMON_DMB_SHM_ID	64
+#define VIRTIO_PCI_COMMON_DMB_MEM_TYPE	66
 
 #endif /* VIRTIO_PCI_NO_MODERN */
 

[PATCH v2 07/12] virtio_pci: create virtqueues with the device's mapping token

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, create the modern PCI
transport's virtqueues with the mapping token held on the
virtio_device. Every mapping call for a virtqueue is handed a union
virtio_map token, and vring_create_virtqueue() derives that token as
{.dma_dev = vdev->dev.parent} without letting its callers choose.

Set vdev->vmap.dma_dev in virtio_pci_modern_probe() and create the
virtqueues through vring_create_virtqueue_map() instead. That covers
both the virtqueues from vp_modern_find_vqs() and the admin virtqueue,
which go through the same setup_vq(). virtio_pci_probe() assigns
vdev->dev.parent first, so the token holds the pointer
vring_create_virtqueue() would have derived.

The legacy transport keeps its own vring_create_virtqueue() call:
vp_legacy_get_features() reads 32 feature bits, so VIRTIO_F_DMB cannot
reach it.

Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio_pci_modern.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
index 6d8ae2a6a8ca..565d37b630b3 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -715,10 +715,11 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
 	info->msix_vector = msix_vec;
 
 	/* create the vring */
-	vq = vring_create_virtqueue(index, num,
-				    SMP_CACHE_BYTES, &vp_dev->vdev,
-				    true, true, ctx,
-				    notify, callback, name);
+	vq = vring_create_virtqueue_map(index, num,
+					SMP_CACHE_BYTES, &vp_dev->vdev,
+					true, true, ctx,
+					notify, callback, name,
+					vp_dev->vdev.vmap);
 	if (!vq)
 		return ERR_PTR(-ENOMEM);
 
@@ -1289,6 +1290,14 @@ int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
 	vp_dev->isr = mdev->isr;
 	vp_dev->vdev.id = mdev->id;
 
+	/*
+	 * The mapping token every virtqueue of this device is created with.
+	 * This is the same value vring_create_virtqueue() would derive from
+	 * vdev->dev.parent, kept here so that a transport feature can replace
+	 * it in one place.
+	 */
+	vp_dev->vdev.vmap.dma_dev = &pci_dev->dev;
+
 	spin_lock_init(&vp_dev->admin_vq.lock);
 	return 0;
 }

[PATCH v2 08/12] virtio: add a device memory buffer region allocator

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, create a mechanism to allocate
and map memory from the Device Memory Buffer (DMB). The DMB is a shared
memory region a device exposes and owns. A device that negotiates the
feature expects its virtqueues and all the buffers we hand it to live in
that region, and its data DMA is routed through a per-device transparent
IOMMU whose bus address space maps 1:1 onto that region.

Add virtio_dmb_init(), which locates the region by the shared memory id
the device reports and builds a gen_pool over it at page granularity,
and virtio_dmb_destroy() to tear that down. Add virtio_dmb_map_ops, a
struct virtio_map_ops implementation that hands out allocations as
addresses in that space: alloc() places a virtqueue area in the region,
map_page() copies a buffer that lives elsewhere into it and copies it
back on unmap. A new dmb member of union virtio_map carries the pool to
those operations.

gen_pool tracks which pages are busy but not what an allocation is, so we
keep one record per region page with the length and the bounced address of
the allocation that starts there. The records are a single array allocated
with the region, 16 bytes a page, which is a 256th of it.

The shared memory id is transport specific, so add a get_dmb_shm_id()
callback to struct virtio_config_ops for a transport to report it. A
transport that does not implement it must not accept VIRTIO_F_DMB. Add
CONFIG_VIRTIO_DMB to enable this support, which defaults to y.

Link: https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---

  - Say bus address space rather than IOVA space

---
 drivers/virtio/Kconfig        |  17 +
 drivers/virtio/Makefile       |   3 +-
 drivers/virtio/virtio_dmb.c   | 798 ++++++++++++++++++++++++++++++++++
 drivers/virtio/virtio_dmb.h   |  28 ++
 include/linux/virtio.h        |   3 +
 include/linux/virtio_config.h |   8 +
 6 files changed, 856 insertions(+), 1 deletion(-)
 create mode 100644 drivers/virtio/virtio_dmb.c
 create mode 100644 drivers/virtio/virtio_dmb.h

diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index ce5bc0d9ea28..64734fd83302 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -188,6 +188,23 @@ config VIRTIO_DEBUG
 
 	  If unsure, say N.
 
+config VIRTIO_DMB
+	bool "Device Memory Buffer support"
+	depends on VIRTIO
+	select GENERIC_ALLOCATOR
+	default y
+	help
+	  Support devices that place their virtqueues and buffers in a shared
+	  memory region they own, rather than in memory the driver allocates.
+
+	  Enabling this adds an allocator over that region and one record per
+	  page of the region, both of which are set up only for a device that
+	  negotiates the feature.  When disabled no transport accepts the
+	  feature, and a device offering it is driven as an ordinary one on a
+	  platform that does not restrict memory access.
+
+	  If unsure, say Y.
+
 config VIRTIO_RTC
 	tristate "Virtio RTC driver"
 	depends on VIRTIO
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index eefcfe90d6b8..ba785ff44a16 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -1,5 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
+virtio-dmb-$(CONFIG_VIRTIO_DMB) := virtio_dmb.o
+obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o $(virtio-dmb-y)
 obj-$(CONFIG_VIRTIO_ANCHOR) += virtio_anchor.o
 obj-$(CONFIG_VIRTIO_PCI_LIB) += virtio_pci_modern_dev.o
 obj-$(CONFIG_VIRTIO_PCI_LIB_LEGACY) += virtio_pci_legacy_dev.o
diff --git a/drivers/virtio/virtio_dmb.c b/drivers/virtio/virtio_dmb.c
new file mode 100644
index 000000000000..884f4017780f
--- /dev/null
+++ b/drivers/virtio/virtio_dmb.c
@@ -0,0 +1,798 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Device Memory Buffer support for virtio devices.
+ *
+ * A device that negotiates VIRTIO_F_DMB owns one shared memory region, the
+ * Device Memory Buffer, that holds its virtqueues and the buffers they
+ * reference.  The data DMA of such a device is routed through a per-device
+ * transparent IOMMU whose bus address space maps 1:1 onto that region, so
+ * every address the driver publishes to the device is an address in that
+ * space.
+ *
+ * This file carves the region up with a gen_pool and implements the
+ * virtio_map_ops that turn allocations into those addresses.  A mapping handle
+ * belonging to a DMB device is an address in that space and nothing else: no
+ * code outside these operations may treat it as a DMA address.
+ *
+ * The region is shared with the device, which may read or write any of it at
+ * any time.  Nothing this file reads back from the region is used to compute a
+ * kernel address, a length or an index.
+ */
+
+#include <linux/align.h>
+#include <linux/dma-mapping.h>
+#include <linux/export.h>
+#include <linux/genalloc.h>
+#include <linux/highmem.h>
+#include <linux/io.h>
+#include <linux/ioport.h>
+#include <linux/limits.h>
+#include <linux/math.h>
+#include <linux/minmax.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/virtio.h>
+#include <linux/virtio_config.h>
+
+#include "virtio_dmb.h"
+
+/* No source recorded: this allocation holds no bounced mapping. */
+#define DMB_SRC_NONE		((phys_addr_t)-1)
+
+/**
+ * struct virtio_dmb_alloc - what the driver remembers about one allocation
+ * @len: length in bytes, which is what bounds a copy
+ * @src: physical address the allocation bounces, DMB_SRC_NONE for an area
+ *
+ * gen_pool owns which pages are allocated; this records what an allocation is.
+ * gen_pool cannot hold either field: its only opaque value is per chunk rather
+ * than per allocation (lib/genalloc.c:493), and it derives a freed extent from
+ * the length its caller passes (lib/genalloc.c:501) rather than from anything
+ * it remembers.
+ *
+ * A zero @len is what marks a record as holding no live allocation, which is
+ * the state the records are allocated in.  No live allocation can have that
+ * length, because both callers of virtio_dmb_claim() reject a zero size before
+ * reaching it.
+ *
+ * @len is a byte count, not a page count.  A page-granular bound would admit a
+ * length ending anywhere inside the allocation's last page, up to PAGE_SIZE - 1
+ * bytes past what was mapped, and virtio_dmb_copy() would then touch the page
+ * after the source run.
+ *
+ * @src doubles as the mark that tells a bounced mapping from a virtqueue area.
+ * Without it, unmap_page() on an area would copy from (phys_addr_t)-1.
+ */
+struct virtio_dmb_alloc {
+	size_t		len;
+	phys_addr_t	src;
+};
+
+/**
+ * struct virtio_dmb - driver-side state for one Device Memory Buffer
+ * @vdev: the device that owns the region, for message context
+ * @map_va: what memremap() returned, for memunmap()
+ * @map_phys: physical base of the region, for release_mem_region()
+ * @map_len: length of the claimed and mapped part of the region
+ * @map_claimed: whether request_mem_region() succeeded for that range
+ * @prev_map: map operations the transport had installed, restored on teardown
+ * @prev_vmap: mapping token that went with @prev_map
+ * @pool: allocator over the region, addressed by kernel address
+ * @allocs: one record per pool page, indexed by the page
+ * @base_va: kernel address the pool starts at, inside the mapping
+ * @base_off: region address the pool starts at
+ * @nslots: pool size in PAGE_SIZE pages
+ * @shm_id: shared memory id the device reported for the region
+ */
+struct virtio_dmb {
+	struct virtio_device	*vdev;
+	void			*map_va;
+	phys_addr_t		 map_phys;
+	size_t			 map_len;
+	bool			 map_claimed;
+	const struct virtio_map_ops *prev_map;
+	union virtio_map	 prev_vmap;
+	struct gen_pool		*pool;
+	struct virtio_dmb_alloc *allocs;
+	void			*base_va;
+	u64			 base_off;
+	unsigned int		 nslots;
+	u16			 shm_id;
+};
+
+static unsigned int virtio_dmb_slots(size_t size)
+{
+	return DIV_ROUND_UP(size, PAGE_SIZE);
+}
+
+static size_t virtio_dmb_pool_size(const struct virtio_dmb *dmb)
+{
+	return (size_t)dmb->nslots << PAGE_SHIFT;
+}
+
+/* Handle the driver publishes for the allocation starting at pool page @slot. */
+static dma_addr_t virtio_dmb_handle(const struct virtio_dmb *dmb,
+				    unsigned int slot)
+{
+	return dmb->base_off + ((u64)slot << PAGE_SHIFT);
+}
+
+/* Pool page of @va, which every caller has already obtained from the pool. */
+static unsigned int virtio_dmb_slot_of(const struct virtio_dmb *dmb, void *va)
+{
+	return (unsigned int)(((char *)va - (char *)dmb->base_va) >> PAGE_SHIFT);
+}
+
+/*
+ * The largest buffer mapping this pool will serve: an eighth of it, and never
+ * less than one page, which is the floor a region at the four-page minimum
+ * sits on.  The eighth is a policy choice.  It bounds the capacity one mapping
+ * can deny the rest of the device, so that a device with several virtqueues
+ * still makes progress while one large mapping is outstanding.
+ *
+ * The cap applies to map_page() only.  An alloc() is a virtqueue area, which
+ * lives as long as the queue and which no back-pressure can defer, so capping
+ * it could only shrink a queue or refuse one outright.  Sizing the region for
+ * the areas as well as the buffers is the device's obligation.
+ */
+static size_t virtio_dmb_max_mapping(const struct virtio_dmb *dmb)
+{
+	size_t eighth = ALIGN_DOWN(virtio_dmb_pool_size(dmb) / 8, PAGE_SIZE);
+
+	return max_t(size_t, eighth, PAGE_SIZE);
+}
+
+/*
+ * Claim @len bytes and remember what the allocation is, or return NULL.  @slot
+ * receives the first pool page, which is what names the allocation.
+ *
+ * Exhaustion is routine, because the region's length bounds how much virtqueue
+ * data can be in flight, and it is not always back-pressure: a network receive
+ * fill has nothing to push back on and repolls instead.  Reporting it at a
+ * level a working device would print would be a log flood, so it goes to
+ * dynamic debug.
+ */
+static void *virtio_dmb_claim(struct virtio_dmb *dmb, size_t len,
+			      phys_addr_t src, unsigned int *slot,
+			      struct virtio_dmb_alloc **out)
+{
+	struct virtio_dmb_alloc *rec;
+	unsigned long va;
+
+	va = gen_pool_alloc(dmb->pool, len);
+	if (!va) {
+		dev_dbg_ratelimited(&dmb->vdev->dev,
+				    "device memory buffer has no run of %u of %u pages\n",
+				    virtio_dmb_slots(len), dmb->nslots);
+		return NULL;
+	}
+
+	*slot = virtio_dmb_slot_of(dmb, (void *)va);
+
+	/*
+	 * gen_pool has just published these pages exclusively to this caller,
+	 * so nothing else can be writing this record.
+	 */
+	rec = &dmb->allocs[*slot];
+	rec->len = len;
+	rec->src = src;
+
+	*out = rec;
+	return (void *)va;
+}
+
+/*
+ * Release the allocation @rec starting at pool page @slot.  The record goes
+ * first, so that a page reachable from the pool never carries an extent
+ * virtio_dmb_resolve() would trust.
+ */
+static void virtio_dmb_release(struct virtio_dmb *dmb, unsigned int slot,
+			       struct virtio_dmb_alloc *rec)
+{
+	size_t len = rec->len;
+
+	memset(rec, 0, sizeof(*rec));
+
+	gen_pool_free(dmb->pool,
+		      (unsigned long)dmb->base_va + ((size_t)slot << PAGE_SHIFT),
+		      len);
+}
+
+/*
+ * Resolve a handle to the allocation it names and the pool page it starts at,
+ * rejecting anything that is not a live allocation of at least @size bytes.
+ * @bounced additionally requires an allocation this file bounced rather than a
+ * virtqueue area.
+ *
+ * Every caller runs this before touching the pool, which is what keeps a stale
+ * or malformed handle away from gen_pool_free().  That function derives the
+ * extent it frees from the length it is given (lib/genalloc.c:501), so an
+ * over-long length clears a neighbouring allocation's bits with no diagnostic
+ * at all, and reaches BUG_ON(remain) (lib/genalloc.c:508) only once the range
+ * runs into free space.  gen_pool_has_addr() answers a different question,
+ * testing chunk containment alone (lib/genalloc.c:553).
+ */
+static bool virtio_dmb_resolve(struct virtio_dmb *dmb, dma_addr_t handle,
+			       size_t size, bool bounced, unsigned int *slot,
+			       struct virtio_dmb_alloc **out)
+{
+	struct virtio_dmb_alloc *rec;
+	u64 off;
+
+	if (!size || (u64)handle < dmb->base_off)
+		goto bad_handle;
+
+	off = (u64)handle - dmb->base_off;
+	if (!IS_ALIGNED(off, PAGE_SIZE) ||
+	    (off >> PAGE_SHIFT) >= dmb->nslots)
+		goto bad_handle;
+
+	rec = &dmb->allocs[off >> PAGE_SHIFT];
+	if (dev_WARN_ONCE(&dmb->vdev->dev, !rec->len,
+			  "device memory buffer handle %pad holds no allocation\n",
+			  &handle))
+		return false;
+
+	if (dev_WARN_ONCE(&dmb->vdev->dev, size > rec->len,
+			  "device memory buffer handle %pad length %zu leaves its allocation\n",
+			  &handle, size))
+		return false;
+
+	if (bounced &&
+	    dev_WARN_ONCE(&dmb->vdev->dev, rec->src == DMB_SRC_NONE,
+			  "device memory buffer handle %pad holds no mapping\n",
+			  &handle))
+		return false;
+
+	*slot = off >> PAGE_SHIFT;
+	*out = rec;
+	return true;
+
+bad_handle:
+	dev_WARN_ONCE(&dmb->vdev->dev, 1,
+		      "device memory buffer handle %pad length %zu out of range\n",
+		      &handle, size);
+	return false;
+}
+
+/*
+ * Copy @size bytes between the region and the pages the allocation at pool
+ * page @slot bounces.  Walk the source a page at a time through
+ * kmap_local_page(): it may be highmem, which is why the record is a physical
+ * address.  The region side is one contiguous mapping and needs no such split.
+ */
+static void virtio_dmb_copy(struct virtio_dmb *dmb, unsigned int slot,
+			    const struct virtio_dmb_alloc *rec, size_t size,
+			    bool to_region)
+{
+	void *region = dmb->base_va + ((size_t)slot << PAGE_SHIFT);
+	size_t done = 0;
+
+	while (done < size) {
+		phys_addr_t src = rec->src + done;
+		unsigned int in_src = offset_in_page(src);
+		size_t n = min(size - done, (size_t)PAGE_SIZE - in_src);
+		void *va;
+
+		va = kmap_local_page(pfn_to_page(PHYS_PFN(src)));
+		if (to_region)
+			memcpy(region + done, va + in_src, n);
+		else
+			memcpy(va + in_src, region + done, n);
+		kunmap_local(va);
+
+		done += n;
+	}
+}
+
+static void *virtio_dmb_op_alloc(union virtio_map map, size_t size,
+				 dma_addr_t *map_handle, gfp_t gfp)
+{
+	struct virtio_dmb *dmb = map.dmb;
+	struct virtio_dmb_alloc *rec;
+	unsigned int slot;
+	void *va;
+
+	/*
+	 * The result is zeroed because this stands in for
+	 * dma_alloc_coherent(), whose callers rely on that.
+	 */
+	if (!size || size > virtio_dmb_pool_size(dmb))
+		goto no_room;
+
+	va = virtio_dmb_claim(dmb, size, DMB_SRC_NONE, &slot, &rec);
+	if (!va)
+		goto no_room;
+
+	memset(va, 0, size);
+	*map_handle = virtio_dmb_handle(dmb, slot);
+
+	return va;
+
+no_room:
+	/*
+	 * A buffer that does not fit is back-pressure and stays quiet, but a
+	 * virtqueue area cannot be deferred, and a region sized for the buffers
+	 * but not for the areas otherwise fails queue setup with nothing to
+	 * tell it apart from every other reason find_vqs() can fail.
+	 *
+	 * Which is why __GFP_NOWARN is honoured rather than ignored.
+	 * vring_alloc_queue_split() walks the queue size down from the size the
+	 * device asked for and marks every attempt but the last with the flag,
+	 * so warning regardless would print a line per attempt for a probe that
+	 * then succeeds.  Dynamic debug still carries the message, which is
+	 * what a packed ring relies on: none of its three areas can be made
+	 * smaller and all three set the flag.
+	 */
+	if (gfp & __GFP_NOWARN)
+		dev_dbg(&dmb->vdev->dev,
+			"no room for a %zu-byte virtqueue area in %u pages\n",
+			size, dmb->nslots);
+	else
+		dev_warn(&dmb->vdev->dev,
+			 "no room for a %zu-byte virtqueue area in %u pages\n",
+			 size, dmb->nslots);
+	return NULL;
+}
+
+/*
+ * A caller reaching this with a bounced mapping has called free() on something
+ * it got from map_page(), and loses the copy-out that unmap_page() does; the
+ * bytes it loses are its own, and refusing would trade that for a leak of the
+ * pages, which is worse.  So this asks for an allocation and not for an area.
+ */
+static void virtio_dmb_op_free(union virtio_map map, size_t size, void *vaddr,
+			       dma_addr_t map_handle, unsigned long attrs)
+{
+	struct virtio_dmb *dmb = map.dmb;
+	struct virtio_dmb_alloc *rec;
+	unsigned int slot;
+
+	if (!virtio_dmb_resolve(dmb, map_handle, size, false, &slot, &rec))
+		return;
+
+	virtio_dmb_release(dmb, slot, rec);
+}
+
+static dma_addr_t virtio_dmb_op_map_page(union virtio_map map,
+					 struct page *page,
+					 unsigned long offset, size_t size,
+					 enum dma_data_direction dir,
+					 unsigned long attrs)
+{
+	struct virtio_dmb *dmb = map.dmb;
+	phys_addr_t src = page_to_phys(page) + offset;
+	struct virtio_dmb_alloc *rec;
+	unsigned int slot;
+	void *va;
+
+	if (!size || size > virtio_dmb_max_mapping(dmb))
+		return DMA_MAPPING_ERROR;
+
+	va = virtio_dmb_claim(dmb, size, src, &slot, &rec);
+	if (!va)
+		return DMA_MAPPING_ERROR;
+
+	/*
+	 * Copy in whatever the direction is, and without honouring
+	 * DMA_ATTR_SKIP_CPU_SYNC.  swiotlb_tbl_map_single() bounces
+	 * unconditionally for the same two reasons: a device that writes less
+	 * than the whole buffer must leave the rest of the caller's bytes
+	 * intact, and the mapped bytes must not reach the device as whatever
+	 * the pages held before.
+	 *
+	 * Those bytes and no others.  The rest of the allocation's last page
+	 * keeps what it held before, which the device may already have put
+	 * there itself, and the descriptor carries a length.
+	 */
+	virtio_dmb_copy(dmb, slot, rec, size, true);
+
+	return virtio_dmb_handle(dmb, slot);
+}
+
+static void virtio_dmb_op_unmap_page(union virtio_map map,
+				     dma_addr_t map_handle, size_t size,
+				     enum dma_data_direction dir,
+				     unsigned long attrs)
+{
+	struct virtio_dmb *dmb = map.dmb;
+	struct virtio_dmb_alloc *rec;
+	unsigned int slot;
+
+	if (!virtio_dmb_resolve(dmb, map_handle, size, true, &slot, &rec))
+		return;
+
+	if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC) &&
+	    (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL))
+		virtio_dmb_copy(dmb, slot, rec, size, false);
+
+	/*
+	 * The extent freed comes from the record rather than from the caller's
+	 * size, so a mismatched size cannot release a different number of pages
+	 * than were claimed.
+	 */
+	virtio_dmb_release(dmb, slot, rec);
+}
+
+static int virtio_dmb_op_mapping_error(union virtio_map map,
+				       dma_addr_t map_handle)
+{
+	/*
+	 * DMA_MAPPING_ERROR is the value virtio_ring reserves.  Address 0 is
+	 * reserved too, both by the feature and by every device that reads a
+	 * queue address of zero as a queue never programmed, so this driver may
+	 * not publish it either.  Nothing allocated here yields it, but a
+	 * premapped buffer carries an address its caller obtained and
+	 * vring_map_one_sg() asks this operation to judge that one.  Whether
+	 * such an address came from this map cannot be answered here; zero can
+	 * be.
+	 */
+	if (map_handle == DMA_MAPPING_ERROR || !map_handle)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static size_t virtio_dmb_op_max_mapping_size(union virtio_map map)
+{
+	return virtio_dmb_max_mapping(map.dmb);
+}
+
+static const struct virtio_map_ops virtio_dmb_map_ops = {
+	.map_page		= virtio_dmb_op_map_page,
+	.unmap_page		= virtio_dmb_op_unmap_page,
+	.alloc			= virtio_dmb_op_alloc,
+	.free			= virtio_dmb_op_free,
+	.mapping_error		= virtio_dmb_op_mapping_error,
+	.max_mapping_size	= virtio_dmb_op_max_mapping_size,
+};
+
+/*
+ * Whether the device still has virtqueues.  No caller here can race an adder,
+ * because every path that reaches this runs under the device lock, but the
+ * invariant is worth enforcing rather than inheriting from callers this file
+ * does not control.
+ */
+static bool virtio_dmb_vqs_live(struct virtio_device *vdev)
+{
+	bool live;
+
+	spin_lock(&vdev->vqs_list_lock);
+	live = !list_empty(&vdev->vqs);
+	spin_unlock(&vdev->vqs_list_lock);
+
+	return live;
+}
+
+/**
+ * virtio_dmb_destroy - release the Device Memory Buffer state of a device
+ * @vdev: the device
+ *
+ * Does nothing unless @vdev is currently using a Device Memory Buffer, and
+ * refuses if the device still has virtqueues.  The mapping and the region
+ * claim then stay behind until something deletes those virtqueues and calls
+ * again, which unbinding the driver does: virtio_dev_remove() calls the
+ * driver's remove() before this.
+ */
+void virtio_dmb_destroy(struct virtio_device *vdev)
+{
+	struct virtio_dmb *dmb;
+
+	/* vdev->map identifies which member of vdev->vmap is live. */
+	if (vdev->map != &virtio_dmb_map_ops)
+		return;
+
+	/*
+	 * A virtqueue keeps the mapping token it was created with, while
+	 * vdev->map is consulted afresh on every dispatch.  Clearing vdev->map
+	 * therefore does not disarm a live virtqueue, it redirects that
+	 * virtqueue's copy of the token into the DMA API, where the pointer
+	 * this frees would be used as a struct device.  Refuse instead and leak
+	 * the mapping, which is better than a use-after-free.
+	 *
+	 * Reported rather than warned about: a device that fails to report its
+	 * region on the way back from a suspend takes virtio_device_restore()
+	 * to its error path, which calls this, and a driver with no freeze
+	 * callback still has its virtqueues at that point, correctly.  A
+	 * condition a correct driver can satisfy must not taint the kernel.
+	 */
+	if (virtio_dmb_vqs_live(vdev)) {
+		dev_warn(&vdev->dev,
+			 "device memory buffer not released, virtqueues are still live\n");
+		return;
+	}
+
+	dmb = vdev->vmap.dmb;
+
+	/* Put back exactly what the transport had installed. */
+	vdev->map = dmb->prev_map;
+	vdev->vmap = dmb->prev_vmap;
+
+	/*
+	 * gen_pool_destroy() ends at a BUG_ON() for a pool with an allocation
+	 * outstanding (lib/genalloc.c:255).  Every allocation belongs to a
+	 * virtqueue, and the refusal above establishes that none is left, which
+	 * is also why the records are empty by now.
+	 */
+	gen_pool_destroy(dmb->pool);
+	kvfree(dmb->allocs);
+	memunmap(dmb->map_va);
+	if (dmb->map_claimed)
+		release_mem_region(dmb->map_phys, dmb->map_len);
+	kfree(dmb);
+}
+EXPORT_SYMBOL_GPL(virtio_dmb_destroy);
+
+/**
+ * virtio_dmb_init - make a device's Device Memory Buffer state current
+ * @vdev: the device, with feature negotiation complete
+ *
+ * Reads the shared memory id the device reports, locates the region, builds an
+ * allocator over it and routes every mapping of the device through it.  When
+ * the feature is not negotiated, releases any state a previous negotiation
+ * left behind.
+ *
+ * Reached again from resume and from reset completion.  A device that reports
+ * the region it reported last time keeps the state it already has, so handles
+ * held by a virtqueue that outlived the transition stay valid.
+ *
+ * A caller that gets an error must set the FAILED device status bit, and must
+ * not touch the device for anything else before it does.  The device has
+ * already confirmed the feature by the time this runs, so it is entitled to
+ * assume the driver will address it through the region; the bit is what tells
+ * it the driver gave up instead.
+ *
+ * Return: 0 on success, or a negative errno.
+ */
+int virtio_dmb_init(struct virtio_device *vdev)
+{
+	struct virtio_shm_region region;
+	struct virtio_dmb *dmb;
+	unsigned int nslots, skew;
+	u64 base_off, slots = 0;
+	size_t map_len;
+	u16 shm_id;
+	int err;
+
+	if (!virtio_has_feature(vdev, VIRTIO_F_DMB)) {
+		/*
+		 * The feature may have been withdrawn across re-negotiation.
+		 * virtio_dmb_destroy() refuses under live virtqueues, and
+		 * returning 0 after a refusal would leave these map operations
+		 * installed for a device that has not negotiated the feature,
+		 * so every later mapping would resolve a handle against a
+		 * region the device no longer agrees it has.
+		 */
+		virtio_dmb_destroy(vdev);
+		if (vdev->map == &virtio_dmb_map_ops)
+			return -EBUSY;
+		return 0;
+	}
+
+	/*
+	 * Without both of these the region cannot be located at all, which is
+	 * what keeps a transport that does not implement them from offering the
+	 * feature in the first place.
+	 */
+	if (!vdev->config->get_dmb_shm_id || !vdev->config->get_shm_region) {
+		dev_warn(&vdev->dev,
+			 "transport cannot locate a device memory buffer\n");
+		return -EINVAL;
+	}
+
+	err = vdev->config->get_dmb_shm_id(vdev, &shm_id);
+	if (err)
+		return err;
+
+	/* A region is looked up by a u8 id. */
+	if (shm_id > U8_MAX) {
+		dev_warn(&vdev->dev,
+			 "device memory buffer id %u out of range\n", shm_id);
+		return -EINVAL;
+	}
+
+	if (!virtio_get_shm_region(vdev, &region, shm_id)) {
+		dev_warn(&vdev->dev,
+			 "cannot locate device memory buffer region %u\n",
+			 shm_id);
+		return -ENODEV;
+	}
+
+	/*
+	 * A device reports a region whose base is aligned to at least the
+	 * largest alignment any virtqueue layout requires of an area.  The
+	 * driver holds the pool to PAGE_SIZE, which is stronger, and checks
+	 * below that memremap() preserved the base's page offset.  Start the
+	 * pool at a PAGE_SIZE-aligned address and record the skew, so that
+	 * page-granular allocation makes every absolute address aligned.
+	 *
+	 * PAGE_SIZE - skew is the distance to the first aligned address
+	 * strictly after the region start, so the pool never begins at the
+	 * region's first byte and no handle is ever 0.  Address 0 is reserved,
+	 * a device that predates the reservation reads a queue address of 0 as
+	 * a queue never programmed, and gen_pool_alloc() returns 0 for failure,
+	 * so the value is unusable three times over.
+	 */
+	skew = offset_in_page(region.addr);
+	base_off = PAGE_SIZE - skew;
+
+	if (region.len > base_off)
+		slots = (region.len - base_off) >> PAGE_SHIFT;
+
+	/*
+	 * The least a region could hold: one minimally-sized virtqueue plus one
+	 * buffer in flight against it.  A packed queue costs three allocations
+	 * and a split queue up to two when the transport aligns its areas to
+	 * PAGE_SIZE, so four pages is the floor for either layout.  It is a
+	 * floor and not a sufficiency check: how much a device needs depends on
+	 * how many virtqueues its driver creates and how deep they are, neither
+	 * of which is known before find_vqs().
+	 *
+	 * The upper bounds are what a page index, a mapping length and a
+	 * published handle can each represent.  The length bound is exclusive
+	 * because the mapping is base_off longer than the pool, and at the last
+	 * representable page count that sum would wrap on a 32-bit size_t.
+	 */
+	if (slots < 4 || slots > UINT_MAX ||
+	    slots >= (u64)(SIZE_MAX >> PAGE_SHIFT) ||
+	    base_off + (slots << PAGE_SHIFT) - 1 >
+			DMA_BIT_MASK(BITS_PER_TYPE(dma_addr_t))) {
+		dev_warn(&vdev->dev,
+			 "device memory buffer region holds %llu usable pages\n",
+			 slots);
+		return -EINVAL;
+	}
+	nslots = slots;
+
+	/* Nothing outside the pool and the bytes ahead of it is used. */
+	map_len = base_off + ((size_t)nslots << PAGE_SHIFT);
+
+	/*
+	 * The transport reports the region in 64 bits while a resource is
+	 * addressed in resource_size_t.  Refuse a region that does not fit
+	 * rather than claim and map a truncated one.
+	 */
+	if (region.addr > (u64)(resource_size_t)-1 - map_len) {
+		dev_warn(&vdev->dev,
+			 "device memory buffer region at 0x%llx is not addressable\n",
+			 region.addr);
+		return -EINVAL;
+	}
+
+	/*
+	 * Everything that identifies the region is known now and nothing has
+	 * been touched, so an unchanged region is adopted instead of being torn
+	 * down and rebuilt identically.  That is what lets a virtqueue which
+	 * outlived a suspend or a reset keep handles that are still valid.  A
+	 * region that moved can be neither adopted nor replaced under live
+	 * virtqueues, so refuse without a warning: a device that moves its
+	 * region while its driver still has virtqueues is misbehaving, which is
+	 * not evidence of a kernel bug.
+	 */
+	if (vdev->map == &virtio_dmb_map_ops) {
+		dmb = vdev->vmap.dmb;
+
+		if (dmb->shm_id == shm_id && dmb->map_phys == region.addr &&
+		    dmb->map_len == map_len)
+			return 0;
+
+		if (virtio_dmb_vqs_live(vdev))
+			return -EBUSY;
+
+		virtio_dmb_destroy(vdev);
+	}
+
+	dmb = kzalloc(sizeof(*dmb), GFP_KERNEL);
+	if (!dmb)
+		return -ENOMEM;
+
+	dmb->vdev = vdev;
+	dmb->shm_id = shm_id;
+	dmb->base_off = base_off;
+	dmb->map_phys = region.addr;
+	dmb->map_len = map_len;
+	dmb->nslots = nslots;
+
+	/*
+	 * A shared memory region need not lie in a BAR the transport already
+	 * claimed, so record a claim on the range here.  The claim is advisory:
+	 * where the region does lie in such a BAR the transport's own claim
+	 * covers it already, which is not a conflict, so it must not fail the
+	 * device.
+	 */
+	dmb->map_claimed = request_mem_region(region.addr, map_len,
+					      "virtio-dmb") != NULL;
+	if (!dmb->map_claimed)
+		dev_dbg(&vdev->dev,
+			"device memory buffer region at 0x%llx already reserved\n",
+			region.addr);
+
+	/*
+	 * MEMREMAP_WB - All transports require VIRTIO_DMB_MEM_TYPE_COHERENT
+	 * which indicates that the DMB SHM region is cache coherent.
+	 * MEMREMAP_DEC - The DMB region is shared memory to a hypervisor so
+	 * that both sides can see it.
+	 */
+	dmb->map_va = memremap(region.addr, map_len,
+			       MEMREMAP_WB | MEMREMAP_DEC);
+	if (!dmb->map_va) {
+		/* memremap() is silent, and every sibling failure names itself. */
+		dev_warn(&vdev->dev,
+			 "cannot map device memory buffer region at 0x%llx\n",
+			 region.addr);
+		err = -ENOMEM;
+		goto err_unclaim;
+	}
+
+	/*
+	 * The skew was derived from the physical base, so the pool is aligned
+	 * in the mapping only if the mapping kept that page offset.
+	 */
+	if (offset_in_page(dmb->map_va) != skew) {
+		dev_warn(&vdev->dev,
+			 "device memory buffer mapping is not page aligned\n");
+		err = -EINVAL;
+		goto err_unmap;
+	}
+	dmb->base_va = dmb->map_va + base_off;
+
+	/*
+	 * One record per pool page, allocated up front so that claiming an
+	 * allocation never has to allocate one.  kvcalloc() because the array
+	 * grows with the region and a large one has no need to be physically
+	 * contiguous.
+	 */
+	dmb->allocs = kvcalloc(dmb->nslots, sizeof(*dmb->allocs), GFP_KERNEL);
+	if (!dmb->allocs) {
+		err = -ENOMEM;
+		goto err_unmap;
+	}
+
+	/*
+	 * PAGE_SHIFT as the minimum allocation order, so that every handle is
+	 * page-aligned and one page is the granule a record accounts in.  The
+	 * pool is addressed by kernel address and carries the address within
+	 * the region as its physical base, so gen_pool_virt_to_phys() yields
+	 * the address the driver publishes to the device.
+	 */
+	dmb->pool = gen_pool_create(PAGE_SHIFT, -1);
+	if (!dmb->pool) {
+		err = -ENOMEM;
+		goto err_free_allocs;
+	}
+
+	err = gen_pool_add_virt(dmb->pool, (unsigned long)dmb->base_va,
+				base_off, virtio_dmb_pool_size(dmb), -1);
+	if (err)
+		goto err_destroy_pool;
+
+	dmb->prev_map = vdev->map;
+	dmb->prev_vmap = vdev->vmap;
+
+	vdev->vmap.dmb = dmb;
+	vdev->map = &virtio_dmb_map_ops;
+
+	dev_info(&vdev->dev,
+		 "device memory buffer %u at 0x%016llx, %u usable pages\n",
+		 shm_id, region.addr, nslots);
+
+	return 0;
+
+err_destroy_pool:
+	gen_pool_destroy(dmb->pool);
+err_free_allocs:
+	kvfree(dmb->allocs);
+err_unmap:
+	memunmap(dmb->map_va);
+err_unclaim:
+	if (dmb->map_claimed)
+		release_mem_region(region.addr, map_len);
+	kfree(dmb);
+
+	return err;
+}
+EXPORT_SYMBOL_GPL(virtio_dmb_init);
+
+MODULE_DESCRIPTION("Virtio device memory buffer allocator");
+MODULE_LICENSE("GPL");
diff --git a/drivers/virtio/virtio_dmb.h b/drivers/virtio/virtio_dmb.h
new file mode 100644
index 000000000000..69fbcbb9c2c0
--- /dev/null
+++ b/drivers/virtio/virtio_dmb.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Device Memory Buffer support for virtio devices.
+ */
+#ifndef _DRIVERS_VIRTIO_VIRTIO_DMB_H
+#define _DRIVERS_VIRTIO_VIRTIO_DMB_H
+
+struct virtio_device;
+
+#if IS_ENABLED(CONFIG_VIRTIO_DMB)
+
+int virtio_dmb_init(struct virtio_device *vdev);
+void virtio_dmb_destroy(struct virtio_device *vdev);
+
+#else
+
+static inline int virtio_dmb_init(struct virtio_device *vdev)
+{
+	return 0;
+}
+
+static inline void virtio_dmb_destroy(struct virtio_device *vdev)
+{
+}
+
+#endif /* CONFIG_VIRTIO_DMB */
+
+#endif /* _DRIVERS_VIRTIO_VIRTIO_DMB_H */
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 3122fc52a7c3..08bf47ed753e 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -44,12 +44,15 @@ struct virtqueue {
 };
 
 struct vduse_vq_group;
+struct virtio_dmb;
 
 union virtio_map {
 	/* Device that performs DMA */
 	struct device *dma_dev;
 	/* VDUSE specific virtqueue group for doing map */
 	struct vduse_vq_group *group;
+	/* Device Memory Buffer holding the virtqueues and their buffers */
+	struct virtio_dmb *dmb;
 };
 
 int virtqueue_add_outbuf(struct virtqueue *vq,
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index eb38c876a628..a10d7c27a2aa 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -96,6 +96,13 @@ struct virtqueue_info {
  * @set_vq_affinity: set the affinity for a virtqueue (optional).
  * @get_vq_affinity: get the affinity for a virtqueue (optional).
  * @get_shm_region: get a shared memory region based on the index.
+ * @get_dmb_shm_id: get the shared memory id of the Device Memory Buffer
+ *	(optional).
+ *	vdev: the device
+ *	id: where to store the shared memory id
+ *	Returns 0 on success or error status
+ *	Only valid once VIRTIO_F_DMB has been negotiated.  A transport that
+ *	does not implement this must not accept VIRTIO_F_DMB.
  * @disable_vq_and_reset: reset a queue individually (optional).
  *	vq: the virtqueue
  *	Returns 0 on success or error status
@@ -135,6 +142,7 @@ struct virtio_config_ops {
 						 int index);
 	bool (*get_shm_region)(struct virtio_device *vdev,
 			       struct virtio_shm_region *region, u8 id);
+	int (*get_dmb_shm_id)(struct virtio_device *vdev, u16 *id);
 	int (*disable_vq_and_reset)(struct virtqueue *vq);
 	int (*enable_vq_after_reset)(struct virtqueue *vq);
 };

[PATCH v2 09/12] virtio: locate the device memory buffer after feature negotiation

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, locate a device's Device Memory
Buffer once feature negotiation is complete. The shared memory id that
locates the region may only be read after the device has confirmed
FEATURES_OK, and finalize_features() runs before that.

virtio_features_ok() is the one place in the core that has just read
FEATURES_OK back, so locate the region from there, and release it from
virtio_dev_remove() and from the error paths of probe and restore.
Restore and reset completion reach it too, where virtio_dmb_init() keeps
the state of a device reporting the region it had, rebuilds it for a
device reporting a different one, and refuses a device that moves its
region while its driver has virtqueues.

Suspend and reset need no handling of their own. Neither
virtio_device_freeze() nor virtio_reset_device() deletes a virtqueue,
and a live virtqueue holds addresses inside the region, so we keep it
across both.

Link: https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 299fa83be1d5..6f112593566c 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -8,6 +8,8 @@
 #include <linux/of.h>
 #include <uapi/linux/virtio_ids.h>
 
+#include "virtio_dmb.h"
+
 /* Unique numbering for virtio devices. */
 static DEFINE_IDA(virtio_index_ida);
 
@@ -231,7 +233,14 @@ static int virtio_features_ok(struct virtio_device *dev)
 			status);
 		return -ENODEV;
 	}
-	return 0;
+
+	/*
+	 * Negotiation is complete, so a Device Memory Buffer may now be
+	 * located.  Reached from probe, from resume and from reset
+	 * completion, all of which have to end with the state matching what
+	 * the device reports now.
+	 */
+	return virtio_dmb_init(dev);
 }
 
 /**
@@ -361,6 +370,7 @@ static int virtio_dev_probe(struct device *_d)
 
 err:
 	virtio_add_status(dev, VIRTIO_CONFIG_S_FAILED);
+	virtio_dmb_destroy(dev);
 	return err;
 
 }
@@ -377,6 +387,8 @@ static void virtio_dev_remove(struct device *_d)
 	/* Driver should have reset device. */
 	WARN_ON_ONCE(dev->config->get_status(dev));
 
+	virtio_dmb_destroy(dev);
+
 	/* Acknowledge the device's existence again. */
 	virtio_add_status(dev, VIRTIO_CONFIG_S_ACKNOWLEDGE);
 
@@ -650,6 +662,7 @@ static int virtio_device_restore_priv(struct virtio_device *dev, bool restore)
 
 err:
 	virtio_add_status(dev, VIRTIO_CONFIG_S_FAILED);
+	virtio_dmb_destroy(dev);
 	return ret;
 }
 

[PATCH v2 10/12] virtio: treat VIRTIO_F_DMB as implying VIRTIO_F_ACCESS_PLATFORM

[Alexander Graf]

In preparation to support VIRTIO_F_DMB, read the platform DMA bypass
quirk from either feature. The data DMA of a device that negotiates DMB
is routed through the IOMMU in front of the region it owns, so the
driver has to map every address it publishes through virtio_map_ops.
VIRTIO_F_ACCESS_PLATFORM asks a driver for the same thing, and
virtio_has_dma_quirk() looks only at it.

virtio_features_ok() demands VIRTIO_F_ACCESS_PLATFORM of a device under
restricted memory access, so accept VIRTIO_F_DMB there too.

Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---
 drivers/virtio/virtio.c       | 10 ++++++++--
 include/linux/virtio_config.h | 13 ++++++++++++-
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 6f112593566c..3f9d4a3b5930 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -216,9 +216,15 @@ static int virtio_features_ok(struct virtio_device *dev)
 			return -ENODEV;
 		}
 
-		if (!virtio_has_feature(dev, VIRTIO_F_ACCESS_PLATFORM)) {
+		/*
+		 * VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_DMB both ensure that
+		 * the device does not access guest memory directly, bypassing
+		 * the platform DMA topology.
+		 */
+		if (!virtio_has_feature(dev, VIRTIO_F_ACCESS_PLATFORM) &&
+		    !virtio_has_feature(dev, VIRTIO_F_DMB)) {
 			dev_warn(&dev->dev,
-				 "device must provide VIRTIO_F_ACCESS_PLATFORM\n");
+				 "device must provide VIRTIO_F_ACCESS_PLATFORM or VIRTIO_F_DMB\n");
 			return -ENODEV;
 		}
 	}
diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
index a10d7c27a2aa..a30eb708046a 100644
--- a/include/linux/virtio_config.h
+++ b/include/linux/virtio_config.h
@@ -273,8 +273,19 @@ static inline bool virtio_has_dma_quirk(const struct virtio_device *vdev)
 	/*
 	 * Note the reverse polarity of the quirk feature (compared to most
 	 * other features), this is for compatibility with legacy systems.
+	 *
+	 * VIRTIO_F_ACCESS_PLATFORM says that the device is subject to the
+	 * platform DMA topology and the driver maps every address it
+	 * publishes through that topology.  The device performs no direct
+	 * guest memory access, so the quirk does not apply.
+	 *
+	 * VIRTIO_F_DMB says that the device holds the virtqueues and the
+	 * buffers they reference in a region it owns, and routes its data
+	 * DMA through the IOMMU in front of that region.  The driver maps
+	 * through that IOMMU, so the quirk does not apply either.
 	 */
-	return !virtio_has_feature(vdev, VIRTIO_F_ACCESS_PLATFORM);
+	return !(virtio_has_feature(vdev, VIRTIO_F_ACCESS_PLATFORM) ||
+		 virtio_has_feature(vdev, VIRTIO_F_DMB));
 }
 
 static inline

[PATCH v2 11/12] virtio_pci: support VIRTIO_F_DMB

[Alexander Graf]

Let a modern virtio-pci device place its virtqueues and the buffers they
reference in a Device Memory Buffer of its own: accept VIRTIO_F_DMB from
vp_transport_features(), and implement the get_dmb_shm_id config op on
top of vp_modern_get_dmb_shm_id(). get_dmb_shm_id reports the shared
memory id of the region that holds the buffer.

Take the offer only for a dmb_mem_type of VIRTIO_DMB_MEM_TYPE_COHERENT,
the memory type that says a write by either side becomes visible to the
other with no cache maintenance by the driver. The driver uses plain
loads and stores on the region, so no other type will do. The field is
valid from the moment the device offers the feature, so a type the driver
does not support leaves the device driven as an ordinary one. The shmid
is not readable that early, so a region we fail to locate after the
accept fails virtio_features_ok() and probe sets the FAILED status bit.

Refuse a device whose common configuration is too short to hold
dmb_mem_type, which would put that read outside what vp_modern_probe()
mapped. The accept path measures the structure itself, because
vp_check_common_size() runs on the features the driver has already
accepted and so cannot cover a read that precedes it.

Two more things gate the accept. CONFIG_VIRTIO_DMB, so that a device
offering the feature to a kernel built without it is driven as an ordinary
device. And VIRTIO_F_VERSION_1, because virtio_features_ok() returns early
without it, which would leave the feature negotiated and the region never
built; this transport refuses such a device anyway.

Link: https://lore.kernel.org/virtio-comment/20260818060255.6853-1-graf@amazon.com/
Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---

  - Explain in a comment how a transport feature survives negotiation
  - Say ACCESS_PLATFORM is implied rather than dismissing it

---
 drivers/virtio/virtio_pci_modern.c | 58 ++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)

diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
index 565d37b630b3..c893e31e5159 100644
--- a/drivers/virtio/virtio_pci_modern.c
+++ b/drivers/virtio/virtio_pci_modern.c
@@ -364,6 +364,29 @@ static void vp_modern_avq_cleanup(struct virtio_device *vdev)
 	}
 }
 
+/*
+ * A device reports the memory type of its Device Memory Buffer region in
+ * dmb_mem_type, and the field is valid as soon as the device offers
+ * VIRTIO_F_DMB.  VIRTIO_DMB_MEM_TYPE_COHERENT says that a write by either side
+ * becomes visible to the other with no cache maintenance by the driver, which
+ * is what plain loads and stores on the region need.  A common configuration
+ * structure too short to hold the field reports no type at all, and reading it
+ * there would fall outside what vp_modern_probe() mapped.  This is the only
+ * place that measures the structure against the field, because the read
+ * happens before vp_check_common_size() runs.
+ */
+static bool vp_dmb_mem_type_supported(struct virtio_pci_device *vp_dev)
+{
+	size_t need = offsetofend(struct virtio_pci_modern_common_cfg,
+				  dmb_mem_type);
+
+	if (vp_dev->mdev.common_len < need)
+		return false;
+
+	return vp_modern_get_dmb_mem_type(&vp_dev->mdev) ==
+	       VIRTIO_DMB_MEM_TYPE_COHERENT;
+}
+
 static void vp_transport_features(struct virtio_device *vdev, u64 features)
 {
 	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
@@ -378,6 +401,30 @@ static void vp_transport_features(struct virtio_device *vdev, u64 features)
 
 	if (features & BIT_ULL(VIRTIO_F_ADMIN_VQ))
 		__virtio_set_bit(vdev, VIRTIO_F_ADMIN_VQ);
+
+	/*
+	 * vring_transport_features() clears every bit in the transport range
+	 * that the ring itself does not consume, and vp_finalize_features()
+	 * hands the features the device offered to this function afterwards.
+	 * A transport feature therefore survives negotiation only if the
+	 * transport sets it again here, the way VIRTIO_F_SR_IOV,
+	 * VIRTIO_F_RING_RESET and VIRTIO_F_ADMIN_VQ do above, so this is where
+	 * the driver accepts a Device Memory Buffer and declining refuses it.
+	 *
+	 * VIRTIO_F_VERSION_1 is required because the core locates and releases
+	 * the region from virtio_features_ok(), which returns before it gets
+	 * that far for a device without VERSION_1, so accepting the feature
+	 * without it would leave the feature negotiated and the region never
+	 * built.  VIRTIO_F_ACCESS_PLATFORM is not required alongside it,
+	 * because virtio_has_dma_quirk() already reads this feature as
+	 * carrying what that one asserts.  Whether the device asks for the
+	 * stronger barriers is its own business and does not bear on this one.
+	 */
+	if (IS_ENABLED(CONFIG_VIRTIO_DMB) &&
+	    (features & BIT_ULL(VIRTIO_F_DMB)) &&
+	    (features & BIT_ULL(VIRTIO_F_VERSION_1)) &&
+	    vp_dmb_mem_type_supported(vp_dev))
+		__virtio_set_bit(vdev, VIRTIO_F_DMB);
 }
 
 static int __vp_check_common_size_one_feature(struct virtio_device *vdev, u32 fbit,
@@ -878,6 +925,15 @@ static bool vp_get_shm_region(struct virtio_device *vdev,
 	return true;
 }
 
+static int vp_get_dmb_shm_id(struct virtio_device *vdev, u16 *id)
+{
+	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
+
+	*id = vp_modern_get_dmb_shm_id(&vp_dev->mdev);
+
+	return 0;
+}
+
 /*
  * virtio_pci_admin_has_dev_parts - Checks whether the device parts
  * functionality is supported
@@ -1241,6 +1297,7 @@ static const struct virtio_config_ops virtio_pci_config_nodev_ops = {
 	.set_vq_affinity = vp_set_vq_affinity,
 	.get_vq_affinity = vp_get_vq_affinity,
 	.get_shm_region  = vp_get_shm_region,
+	.get_dmb_shm_id = vp_get_dmb_shm_id,
 	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
 	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
 };
@@ -1261,6 +1318,7 @@ static const struct virtio_config_ops virtio_pci_config_ops = {
 	.set_vq_affinity = vp_set_vq_affinity,
 	.get_vq_affinity = vp_get_vq_affinity,
 	.get_shm_region  = vp_get_shm_region,
+	.get_dmb_shm_id = vp_get_dmb_shm_id,
 	.disable_vq_and_reset = vp_modern_disable_vq_and_reset,
 	.enable_vq_after_reset = vp_modern_enable_vq_after_reset,
 };

[PATCH v2 12/12] virtio: expose device memory buffer occupancy over debugfs

[Alexander Graf]

A driver allocates from a pool bounded by the region a Device Memory
Buffer device reports, so running out of room in it is routine:
vring_map_errno() reports the shortage as the -ENOSPC a full queue
reports, and the driver retries and carries on. The allocator must not
log that at any level a working device prints, or an undersized region
turns into a log flood. That leaves an operator nothing but unexplained
throughput loss to go on.

swiotlb can afford dev_warn_ratelimited() because exhaustion there is a
misconfiguration, and it exports io_tlb_used and io_tlb_used_hiwater
through debugfs, which a log line cannot do for a sampled quantity.
Follow it, plus one file swiotlb has no need for:

  dmb/pages              pages the allocator can hand out
  dmb/used_pages         pages allocated now
  dmb/used_pages_hiwater the largest used_pages has been
  dmb/alloc_failed       buffer mappings the pool had no room for

used_pages comes from the allocator's own free count, so no counter is
maintained alongside it and no read can report a torn total. Writing 0 to
used_pages_hiwater restarts the measurement from the occupancy now, so a
peak never reads below the used_pages read alongside it.

used_pages_hiwater is the number to size a region against, because a
burst that fills the pool between two samples of used_pages leaves no
other trace.

Assisted-by: Kiro:claude-opus-5 checkpatch sparse
Signed-off-by: Alexander Graf <graf@amazon.com>
---

  - Rework description

---
 drivers/virtio/virtio_dmb.c | 163 +++++++++++++++++++++++++++++++++++-
 1 file changed, 162 insertions(+), 1 deletion(-)

diff --git a/drivers/virtio/virtio_dmb.c b/drivers/virtio/virtio_dmb.c
index 884f4017780f..01b7a3542828 100644
--- a/drivers/virtio/virtio_dmb.c
+++ b/drivers/virtio/virtio_dmb.c
@@ -20,6 +20,8 @@
  */
 
 #include <linux/align.h>
+#include <linux/atomic.h>
+#include <linux/debugfs.h>
 #include <linux/dma-mapping.h>
 #include <linux/export.h>
 #include <linux/genalloc.h>
@@ -83,6 +85,10 @@ struct virtio_dmb_alloc {
  * @base_va: kernel address the pool starts at, inside the mapping
  * @base_off: region address the pool starts at
  * @nslots: pool size in PAGE_SIZE pages
+ * @used_hiwater: the largest occupancy has been since the last reset through
+ *	debugfs, or since init, in pages; CONFIG_VIRTIO_DEBUG
+ * @alloc_failed: buffer mappings the pool had no room for; CONFIG_VIRTIO_DEBUG
+ * @debugfs_dir: directory holding this region's debugfs files
  * @shm_id: shared memory id the device reported for the region
  */
 struct virtio_dmb {
@@ -98,6 +104,11 @@ struct virtio_dmb {
 	void			*base_va;
 	u64			 base_off;
 	unsigned int		 nslots;
+#ifdef CONFIG_VIRTIO_DEBUG
+	atomic_long_t		 used_hiwater;
+	atomic_long_t		 alloc_failed;
+#endif
+	struct dentry		*debugfs_dir;
 	u16			 shm_id;
 };
 
@@ -111,6 +122,52 @@ static size_t virtio_dmb_pool_size(const struct virtio_dmb *dmb)
 	return (size_t)dmb->nslots << PAGE_SHIFT;
 }
 
+#ifdef CONFIG_VIRTIO_DEBUG
+
+/*
+ * Pages allocated now.  gen_pool_avail() sums its chunks' free counts under
+ * RCU, so this needs no counter of its own and no lock, and it is exact
+ * between two claims rather than approximate.
+ */
+static unsigned long virtio_dmb_used(struct virtio_dmb *dmb)
+{
+	return (gen_pool_size(dmb->pool) - gen_pool_avail(dmb->pool))
+		>> PAGE_SHIFT;
+}
+
+/*
+ * Raise the high-water mark to the occupancy a claim has just established.
+ * Two racing claims each observe a real total and the larger wins, so the
+ * figure is a occupancy the pool genuinely held rather than a sum of readings
+ * taken at different moments.
+ */
+static void virtio_dmb_note_used(struct virtio_dmb *dmb)
+{
+	long old = atomic_long_read(&dmb->used_hiwater);
+	long now = virtio_dmb_used(dmb);
+
+	while (now > old &&
+	       !atomic_long_try_cmpxchg(&dmb->used_hiwater, &old, now))
+		;
+}
+
+static void virtio_dmb_inc_alloc_failed(struct virtio_dmb *dmb)
+{
+	atomic_long_inc(&dmb->alloc_failed);
+}
+
+#else /* !CONFIG_VIRTIO_DEBUG */
+
+static void virtio_dmb_note_used(struct virtio_dmb *dmb)
+{
+}
+
+static void virtio_dmb_inc_alloc_failed(struct virtio_dmb *dmb)
+{
+}
+
+#endif /* CONFIG_VIRTIO_DEBUG */
+
 /* Handle the driver publishes for the allocation starting at pool page @slot. */
 static dma_addr_t virtio_dmb_handle(const struct virtio_dmb *dmb,
 				    unsigned int slot)
@@ -178,6 +235,8 @@ static void *virtio_dmb_claim(struct virtio_dmb *dmb, size_t len,
 	rec->len = len;
 	rec->src = src;
 
+	virtio_dmb_note_used(dmb);
+
 	*out = rec;
 	return (void *)va;
 }
@@ -371,8 +430,22 @@ static dma_addr_t virtio_dmb_op_map_page(union virtio_map map,
 		return DMA_MAPPING_ERROR;
 
 	va = virtio_dmb_claim(dmb, size, src, &slot, &rec);
-	if (!va)
+	if (!va) {
+		/*
+		 * Counted here and not in virtio_dmb_claim(), which alloc()
+		 * reaches as well.  A virtqueue area that does not fit is a
+		 * step of vring_alloc_queue_split()'s search for a size that
+		 * does, so counting it would have a correctly sized region boot
+		 * with a failure for every attempt but the last, in the one
+		 * file whose purpose is to answer whether the region is too
+		 * small for the traffic.  A request over the per-mapping cap is
+		 * not counted either: the cap is what max_mapping_size()
+		 * advertises, so exceeding it is a caller bug rather than a
+		 * property of the region.
+		 */
+		virtio_dmb_inc_alloc_failed(dmb);
 		return DMA_MAPPING_ERROR;
+	}
 
 	/*
 	 * Copy in whatever the direction is, and without honouring
@@ -448,6 +521,91 @@ static const struct virtio_map_ops virtio_dmb_map_ops = {
 	.max_mapping_size	= virtio_dmb_op_max_mapping_size,
 };
 
+#ifdef CONFIG_VIRTIO_DEBUG
+
+static int virtio_dmb_used_get(void *data, u64 *val)
+{
+	struct virtio_dmb *dmb = data;
+
+	*val = virtio_dmb_used(dmb);
+
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(virtio_dmb_used_fops, virtio_dmb_used_get, NULL,
+			 "%llu\n");
+
+static int virtio_dmb_hiwater_get(void *data, u64 *val)
+{
+	struct virtio_dmb *dmb = data;
+
+	*val = atomic_long_read(&dmb->used_hiwater);
+
+	return 0;
+}
+
+/*
+ * Restart the measurement from the occupancy now, so that a peak never reads
+ * below the used_pages read alongside it.
+ */
+static int virtio_dmb_hiwater_set(void *data, u64 val)
+{
+	struct virtio_dmb *dmb = data;
+
+	if (val)
+		return -EINVAL;
+
+	atomic_long_set(&dmb->used_hiwater, virtio_dmb_used(dmb));
+
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(virtio_dmb_hiwater_fops, virtio_dmb_hiwater_get,
+			 virtio_dmb_hiwater_set, "%llu\n");
+
+static int virtio_dmb_alloc_failed_get(void *data, u64 *val)
+{
+	struct virtio_dmb *dmb = data;
+
+	*val = atomic_long_read(&dmb->alloc_failed);
+
+	return 0;
+}
+DEFINE_DEBUGFS_ATTRIBUTE(virtio_dmb_alloc_failed_fops,
+			 virtio_dmb_alloc_failed_get, NULL, "%llu\n");
+
+static void virtio_dmb_debugfs_init(struct virtio_dmb *dmb)
+{
+	struct dentry *dir;
+
+	dir = debugfs_create_dir("dmb", dmb->vdev->debugfs_dir);
+	dmb->debugfs_dir = dir;
+
+	debugfs_create_u32("pages", 0400, dir, &dmb->nslots);
+	debugfs_create_file("used_pages", 0400, dir, dmb,
+			    &virtio_dmb_used_fops);
+	debugfs_create_file("used_pages_hiwater", 0600, dir, dmb,
+			    &virtio_dmb_hiwater_fops);
+	debugfs_create_file("alloc_failed", 0400, dir, dmb,
+			    &virtio_dmb_alloc_failed_fops);
+}
+
+static void virtio_dmb_debugfs_exit(struct virtio_dmb *dmb)
+{
+	debugfs_remove_recursive(dmb->debugfs_dir);
+	dmb->debugfs_dir = NULL;
+}
+
+#else /* !CONFIG_VIRTIO_DEBUG */
+
+static void virtio_dmb_debugfs_init(struct virtio_dmb *dmb)
+{
+}
+
+static void virtio_dmb_debugfs_exit(struct virtio_dmb *dmb)
+{
+}
+
+#endif /* CONFIG_VIRTIO_DEBUG */
+
 /*
  * Whether the device still has virtqueues.  No caller here can race an adder,
  * because every path that reaches this runs under the device lock, but the
@@ -515,6 +673,7 @@ void virtio_dmb_destroy(struct virtio_device *vdev)
 	 * virtqueue, and the refusal above establishes that none is left, which
 	 * is also why the records are empty by now.
 	 */
+	virtio_dmb_debugfs_exit(dmb);
 	gen_pool_destroy(dmb->pool);
 	kvfree(dmb->allocs);
 	memunmap(dmb->map_va);
@@ -773,6 +932,8 @@ int virtio_dmb_init(struct virtio_device *vdev)
 	vdev->vmap.dmb = dmb;
 	vdev->map = &virtio_dmb_map_ops;
 
+	virtio_dmb_debugfs_init(dmb);
+
 	dev_info(&vdev->dev,
 		 "device memory buffer %u at 0x%016llx, %u usable pages\n",
 		 shm_id, region.addr, nslots);