Re: [RFC PATCH 12/12] virtio: guarantee a virtqueue can publish its first descriptor chain

["Michael S. Tsirkin" <mst@redhat.com>]

On Sun, Aug 09, 2026 at 06:20:10PM +0000, Alexander Graf wrote:
> A device's pool is shared by every virtqueue it has and nothing
> partitions it between them, so a virtqueue that asks late can get
> nothing at all. virtnet_open() pre-fills the receive queues in index
> order and discards the result, and a queue that came up with nothing is
> one the device cannot signal a completion on.
> 
> Withhold a range of the pool from ordinary claims and let a virtqueue
> whose ring is empty draw on it for one descriptor chain.

Why bother? one chain can just go inline in the ring.

> vring_map_reserve_attr() marks those mappings with a new attrs bit,
> VIRTIO_MAP_ATTR_RESERVE, which asks a bounded map implementation to
> serve them out of the withheld range and which it may ignore.
> VIRTIO_MAP_ATTR_MASK collects the attrs bits virtio owns, which
> virtqueue_map_page_attrs() strips before any dma_map_*() call.
> 
> VIRTIO_MAP_RESERVE_PAGES bounds what one chain may draw at 32 pages,
> enough for a page-granular chain at the default CONFIG_MAX_SKB_FRAGS.
> Size the withheld range as one chain's allowance plus one page for each
> virtqueue beyond the one drawing the chain; an allowance per virtqueue
> would be a permanent shortage, since only an empty ring may draw.
> virtio_dmb_note_vqs() takes the count from vp_modern_find_vqs(), which
> counts the administration virtqueue too, and the size it arrives at
> appears in a second probe-time message and in dmb/reserved_pages.
> 
> The new VIRTIO_DMB_KUNIT_TEST option covers the range's geometry and the
> withdrawal of the feature, neither of which we reach on a real device.
> 
> That way a virtqueue that comes up against a full pool publishes its
> first chain and gets a completion to wait for.
> 
> Assisted-by: Kiro:claude-opus-5 checkpatch sparse
> Signed-off-by: Alexander Graf <graf@amazon.com>
> ---
>  .../driver-api/virtio/virtio-dmb.rst          | 178 +++++++++--
>  drivers/virtio/Kconfig                        |  13 +
>  drivers/virtio/virtio_dmb.c                   | 247 ++++++++++++++--
>  drivers/virtio/virtio_dmb.h                   |   6 +
>  drivers/virtio/virtio_dmb_test.c              | 279 ++++++++++++++++++
>  drivers/virtio/virtio_pci_modern.c            |  18 +-
>  drivers/virtio/virtio_ring.c                  |  84 +++++-
>  include/linux/virtio_config.h                 |  33 +++
>  8 files changed, 784 insertions(+), 74 deletions(-)
>  create mode 100644 drivers/virtio/virtio_dmb_test.c
> 
> diff --git a/Documentation/driver-api/virtio/virtio-dmb.rst b/Documentation/driver-api/virtio/virtio-dmb.rst
> index 610db080dee5..d84f0771f010 100644
> --- a/Documentation/driver-api/virtio/virtio-dmb.rst
> +++ b/Documentation/driver-api/virtio/virtio-dmb.rst
> @@ -302,15 +302,18 @@ So a device must offer at least::
>                  + sum over virtqueues of
>                      [ ring_pages(num) + num * slots(max_request) ]
>                      * PAGE_SIZE
> +                + reserved * PAGE_SIZE
>  
>    where the leading PAGE_SIZE covers the bytes ahead of the pool, and is
>    a whole page because a device cannot know the guest's page size and so
>    cannot know whether its region base is aligned to one,
>    slots(max_request) is the largest slots(request) value from the
>    formula above the driver can produce for one request on that queue,
> -  and ring_pages(num) is the pages one virtqueue's Descriptor, Driver and
> +  ring_pages(num) is the pages one virtqueue's Descriptor, Driver and
>    Device Areas occupy: ceil(vring_size(num, align) / PAGE_SIZE) for a
> -  split ring, and ceil(num * 16 / PAGE_SIZE) + 2 for a packed one
> +  split ring, and ceil(num * 16 / PAGE_SIZE) + 2 for a packed one,
> +  and reserved is the range withheld from ordinary claims, described
> +  under "The withheld range" below
>  
>  A device that also offers an administration virtqueue pays for it out of
>  the same region: its virtqueue areas are allocated through the same path,
> @@ -391,6 +394,13 @@ the ones it goes on to use: ``virtnet_find_vqs()`` creates
>    floor_slots = rings + Q * B_rx + tx_min + ctrl   multi-queue works: one
>                                                     receive buffer per queue
>    work_slots  = rings + rx_fill + tx_min + ctrl    receive rings full
> +  reserved    = min(32 + V - 1, pages / 2)     withheld, and zero below 256
> +                                               pages; a short last pool
> +                                               area can add to it, see
> +                                               below.  The V there counts
> +                                               an administration virtqueue
> +                                               too where a device has one
> +  region_slots = work_slots + reserved         what a device must offer
>  
>  Every term above assumes ``VIRTIO_F_INDIRECT_DESC``, which is what the
>  ``+ 3`` and the ``ctrl`` table account for.  Without it a chain occupies one
> @@ -402,25 +412,33 @@ so ``rx_fill`` and not ``floor_slots`` is the working figure.  For a split
>  ring, x86_64, 4 KiB pages, ``MAX_SKB_FRAGS`` 17, mergeable receive buffers
>  and ``C = 1``:
>  
> -=====  =====  =======  =========  ============  ==========
> -Q      num    rings    rx_fill    work_slots    work MiB
> -=====  =====  =======  =========  ============  ==========
> -1      256    6        256        287           1.1
> -4      256    18       1024       1127          4.4
> -8      256    34       2048       2247          8.8
> -8      1024   119      8192       8476          33.1
> -16     1024   231      16384      16940         66.2
> -32     1024   455      32768      33868         132.3
> -64     1024   903      65536      67724         264.5
> -=====  =====  =======  =========  ============  ==========
> -
> -Two things to take from it.  ``rx_fill`` dominates: it is 65536 slots of the
> -67724 sixty-four queue pairs come to, so sizing a region for multiple queues
> -is "how many receive buffers will be posted" to within three and a quarter
> -percent, and the virtqueue rings are the smaller part of what is left -- 903
> -slots against ``tx_min``'s 1280.  And the pool areas of the previous
> -subsection do not appear at all, because they cost guest memory rather than
> -region pages.
> +=====  =====  =======  =========  ============  ==========  ========  ========
> +Q      num    rings    rx_fill    work_slots    work MiB    reserved  region
> +=====  =====  =======  =========  ============  ==========  ========  ========
> +1      256    6        256        287           1.1         34        321
> +4      256    18       1024       1127          4.4         40        1167
> +8      256    34       2048       2247          8.8         48        2295
> +8      1024   119      8192       8476          33.1        48        8524
> +16     1024   231      16384      16940         66.2        64        17004
> +32     1024   455      32768      33868         132.3       96        33964
> +64     1024   903      65536      67724         264.5       160       67884
> +=====  =====  =======  =========  ============  ==========  ========  ========
> +
> +``region`` is ``region_slots``, the figure to offer; ``work MiB`` is
> +``work_slots`` in mebibytes and excludes ``reserved``.  The
> +sixty-four-queue-pair row is 265.2 MiB of region against the 264.5
> +``work_slots`` alone comes to.
> +
> +Three things to take from it.  ``rx_fill`` dominates: it is 65536 slots of
> +the 67884 a sixty-four-queue-pair region comes to, so sizing a region for
> +multiple queues is "how many receive buffers will be posted" to within three
> +and a half percent, and the virtqueue rings are the smaller part of what is
> +left -- 903 slots against ``tx_min``'s 1280.  ``reserved`` is small and grows
> +with the virtqueue count rather than with the region -- 160 slots at
> +sixty-four queue pairs, under a quarter of one percent -- but it is not
> +optional, because it is withheld rather than merely spent.  And the pool
> +areas of the previous subsection do not appear at all, because they cost
> +guest memory rather than region pages.
>  
>  The receive buffer mode is a twenty-fold multiplier the device cannot
>  predict.  A driver that negotiates any of ``VIRTIO_NET_F_GUEST_TSO4``,
> @@ -432,6 +450,83 @@ which is 160.8 MiB instead of 8.8.  A device offering guest segmentation
>  offload without ``MRG_RXBUF`` must size for that.  Offering ``MRG_RXBUF``
>  is the better answer.
>  
> +The withheld range
> +------------------
> +
> +Part of the pool is withheld from ordinary claims so that a virtqueue
> +whose ring is empty can publish a chain whatever the others have mapped::
> +
> +  reserved = 0                            if pages < 256
> +           = min(32 + V - 1, pages / 2)   otherwise, plus the last pool
> +                                          area's page count where that
> +                                          area is short enough to split
> +                                          the range -- see below
> +
> +One page for each of the ``V`` virtqueues, and a chain's allowance of 32
> +pages on top, less the one page already counted for whichever virtqueue
> +draws the chain.  A virtqueue may draw on the range only while its ring is
> +empty, and only for one chain of up to 32 pages, which the ring asks for by
> +setting ``VIRTIO_MAP_ATTR_RESERVE`` on the mapping.  That is the case worth
> +protecting: a virtqueue with nothing published has no completion of its own
> +to be woken by, because the device cannot signal a used buffer on a queue
> +with no buffers posted, so it depends entirely on its owner retrying.  A
> +virtqueue that has published a chain does not need the guarantee, and does
> +not get it.
> +
> +Both terms follow from that restriction.  Because a running virtqueue
> +cannot draw, the range is contended only by the virtqueues that have
> +published nothing; there are at most ``V`` of those, and what each of them
> +needs is the one page that takes its ring from empty to non-empty.  The 31
> +pages left over, plus that virtqueue's own one, are then a whole chain for
> +whichever of them is publishing more than a single page.
> +
> +So the guarantee is: every virtqueue can obtain one page that no running
> +virtqueue can take, and one virtqueue at a time can obtain a whole chain.
> +For virtio_net that is the whole of it at any queue count, because a
> +mergeable receive buffer is one slot and a linear transmit is one slot: the
> +resident draw is ``2 * Q + 5`` pages against the ``31 + V`` withheld, and
> +``2 * Q + 5 <= 2 * Q + 32`` holds for every ``Q``.  What is *not*
> +guaranteed is ``V`` simultaneous multi-page first chains.  A second
> +virtqueue whose first chain needs more than its one page competes for the
> +32 and, losing, gets ``-ENOMEM`` -- the value it got before the range
> +existed, and one every caller answers by retrying later rather than by
> +waiting for a completion it has not got.  Sizing the range for ``V`` whole
> +chains instead would withhold ``32 * V``, which on a small region is a
> +fixed fraction of the pool and is capacity the transmit path then does not
> +have; that trades a rare mapping failure for a permanent shortage, so the
> +quantity is deliberately not the product of the two worst cases.
> +
> +The range is the tail of the pool, and an allocation has to lie inside one
> +pool area, so a whole chain needs 32 of the withheld pages contiguous within
> +one area and not merely free.  Where the last pool area is shorter than 32
> +pages the range straddles the boundary below it, and where the piece above
> +that boundary and the piece below are both shorter than a chain, the range
> +is widened by the short area's length so that the piece below it is a whole
> +``31 + V`` pages.  That costs under 32 pages and only where ``pages`` is just
> +above a multiple of ``area_pages``; no row of the table above reaches it, and
> +neither does any region whose page count is a multiple of ``area_pages`` or at
> +least 32 above one.
> +
> +The guarantee holds for every virtqueue on any pool of 256 pages or more, and
> +below that the range is inert altogether, because it would withhold more than
> +it protects.  Above ninety-seven virtqueues the half-of-the-pool bound is the
> +later of the two conditions, and the pool has to be ``62 + 2 * V`` pages or
> +more instead.  Every row of the table above is far clear of both: the
> +tightest is 287 pages against the 68 three virtqueues need.  That bound is
> +not a policy, only what keeps the ordinary range non-empty; where it does
> +bind, the pool has fewer pages than the virtqueues alone want and the
> +guarantee covers as many of them as it has pages for.
> +
> +This is a floor and not a fair share.  A single busy virtqueue may still
> +use everything outside the withheld range: on a 16 MiB region with eight
> +queue pairs and depth-1024 rings that is 3928 of 4095 pages, the region
> +less the 48 withheld and the 119 the virtqueue areas hold.  The range is
> +sized from every virtqueue the transport creates, which is the driver's
> +count plus an administration virtqueue where the device offers one, and
> +which over-counts a device that offers more queue pairs than the driver
> +uses; over-counting withholds one page per unused virtqueue and never
> +denies capacity.
> +
>  An undersized multi-queue region fails in two ways, and both are
>  properties of the network driver's existing behaviour rather than of the
>  region.
> @@ -439,12 +534,16 @@ region.
>  First, cross-queue starvation.  ``virtnet_open()`` pre-fills the receive
>  queues in index order and discards the result, so queue 0 takes what it
>  needs before queue 1 asks.  A region that cannot hold every queue's fill
> -leaves the later queues with no buffers at all; the device's receive
> -steering then drops whatever it sends to them, and transmit fails on every
> -queue with ``tx_fifo_errors``, ``tx_dropped`` and a rate-limited
> -``Unexpected TXQ`` message.  Pool areas do not help: at
> -``virtnet_open()`` every fill runs on whichever CPU brought the link up, so
> -they all share one home area.  Areas give preference, never reservation.
> +leaves the later queues with far fewer buffers than the earlier ones; the
> +device's receive steering then drops most of what it sends to them, and
> +transmit fails on those queues with ``tx_fifo_errors``, ``tx_dropped`` and
> +a rate-limited ``Unexpected TXQ`` message.  Pool areas do not help: at
> +``virtnet_open()`` every fill runs on whichever CPU brought the link up,
> +so they all share one home area, and areas give preference rather than
> +reservation.  The withheld range bounds how bad this gets -- no virtqueue
> +is left unable to publish anything at all -- but one buffer per queue is a
> +floor for forward progress, not a working receive ring.  Sizing the region
> +for ``region_slots`` is what avoids it.
>  
>  Second, a receive queue that holds no buffers and cannot refill spends
>  softirq time without making progress.  ``try_fill_recv()`` reports failure
> @@ -624,6 +723,14 @@ The claimed range also appears in ``/proc/iomem`` as
>  appears nowhere else at all.  The three counts appear again in
>  ``dmb/pages``, ``dmb/areas`` and ``dmb/area_pages`` below.
>  
> +A second line follows when the driver creates its virtqueues, because the
> +count they come to is not known any earlier::
> +
> +  virtio_net virtio5: device memory buffer withholds 48 of 4095 pages for 17 virtqueues
> +
> +That is the withheld range of the previous section, and the virtqueue count
> +it was sized from.  It appears again in ``dmb/reserved_pages``.
> +
>  With ``CONFIG_VIRTIO_DEBUG`` the state of the region is also available
>  under the device's virtio debugfs directory, in ``dmb/``.  The directory
>  exists only while the device has a region, so a device that did not
> @@ -643,11 +750,20 @@ they accept may change or go away.
>    ``area_pages`` divides ``pages``.  Both are fixed when the region is
>    installed and are the same two values the probe-time message prints.
>  
> +``reserved_pages``
> +  how many pages are withheld from ordinary claims so that a virtqueue whose
> +  ring is empty can publish a chain.  Zero until the driver asks for its
> +  virtqueues, and zero for good on a pool too small for the mechanism to
> +  mean anything.  Not subtracted from ``pages``: the withheld pages are
> +  part of the pool and are counted in ``used_pages`` when a virtqueue draws
> +  on them.
> +
>  ``used_pages``
> -  how many of them are allocated.  One counter maintained across all areas
> -  rather than a sum of per-area figures read at different moments, so it
> -  never reports a torn total; it is raised just outside the area lock, so a
> -  read taken during a claim or a release can lag the bitmap by that claim.
> +  how many of the pool's pages are allocated.  One counter maintained
> +  across all areas rather than a sum of per-area figures read at different
> +  moments, so it never reports a torn total; it is raised just outside the
> +  area lock, so a read taken during a claim or a release can lag the bitmap
> +  by that claim.
>  
>  ``used_pages_hiwater``
>    the largest ``used_pages`` has been.  This, rather than a sample of
> diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
> index 67a7dc1a87df..628ce92edf81 100644
> --- a/drivers/virtio/Kconfig
> +++ b/drivers/virtio/Kconfig
> @@ -204,6 +204,19 @@ config VIRTIO_DMB
>  
>  	  If unsure, say Y.
>  
> +config VIRTIO_DMB_KUNIT_TEST
> +	bool "Device Memory Buffer allocator tests" if !KUNIT_ALL_TESTS
> +	depends on VIRTIO_DMB && KUNIT=y
> +	default KUNIT_ALL_TESTS
> +	help
> +	  Tests for the geometry of the range the allocator withholds so that
> +	  a virtqueue with an empty ring can publish its first descriptor
> +	  chain, and for what becomes of a region when a negotiation drops
> +	  the feature.  Neither case arises on a device anyone is likely to
> +	  build, so a test rather than a measurement is what keeps both true.
> +
> +	  If unsure, say N.
> +
>  config VIRTIO_RTC
>  	tristate "Virtio RTC driver"
>  	depends on VIRTIO
> diff --git a/drivers/virtio/virtio_dmb.c b/drivers/virtio/virtio_dmb.c
> index fe556d585f24..773b5bb38dc6 100644
> --- a/drivers/virtio/virtio_dmb.c
> +++ b/drivers/virtio/virtio_dmb.c
> @@ -128,6 +128,7 @@ struct virtio_dmb_area {
>   * @area_slots: slots one area covers, a power of two; the last area covers
>   *	fewer when @nslots is not a multiple of it
>   * @area_shift: ilog2(@area_slots), so slot >> @area_shift names its area
> + * @nvqs: virtqueues the transport created, which sizes the withheld range
>   * @total_used: slots allocated across every area, exact; CONFIG_VIRTIO_DEBUG
>   * @used_hiwater: the largest @total_used has been since the last reset through
>   *	debugfs, or since init; CONFIG_VIRTIO_DEBUG
> @@ -152,6 +153,7 @@ struct virtio_dmb {
>  	unsigned int		 nareas;
>  	unsigned int		 area_slots;
>  	unsigned int		 area_shift;
> +	unsigned int		 nvqs;
>  #ifdef CONFIG_VIRTIO_DEBUG
>  	/*
>  	 * Exact occupancy for the two debugfs files, kept outside the area
> @@ -286,26 +288,116 @@ static void virtio_dmb_inc_alloc_failed(struct virtio_dmb *dmb)
>  #endif /* CONFIG_VIRTIO_DEBUG */
>  
>  /*
> - * Claim @nr contiguous slots from area @i, or -ENOMEM when that one area
> - * cannot satisfy the request.  Takes and drops that area's lock and touches
> - * no other area's state, so no path ever holds two of these locks and there
> - * is no ordering between them to get right.
> + * Slots withheld from ordinary claims so that a virtqueue with an empty ring
> + * can publish a chain even when every other virtqueue has filled the rest of
> + * the pool.  A virtqueue with nothing published has no completion of its own
> + * to be woken by, because the device cannot signal a used buffer on a queue
> + * with no buffers posted, so it depends entirely on its owner retrying; one
> + * that has published a chain does not.
> + *
> + * One page for every virtqueue, and the rest of one chain's allowance on top.
> + * Both terms follow from who can draw: only a virtqueue whose ring is empty
> + * may, so the range is contended by the virtqueues that have published
> + * nothing and never by the ones that are running.  At most nvqs of those
> + * exist, each needing the one page that takes its ring from empty to
> + * non-empty, and VIRTIO_MAP_RESERVE_PAGES less that one page is then what is
> + * left for whichever of them is publishing a chain longer than a single page.
> + *
> + * Withholding a whole chain's allowance for every virtqueue instead would be
> + * sizing for a state the device cannot be in, and it costs
> + * nvqs * VIRTIO_MAP_RESERVE_PAGES -- on a small pool a fixed fraction of it,
> + * which is capacity the transmit path then does not have.  That trades a
> + * mapping failure which is rare for a shortage which is permanent, so the
> + * quantity is deliberately not a product of the two worst cases.
> + *
> + * The range is the tail of the pool, but an allocation lies inside one area,
> + * so a whole chain needs VIRTIO_MAP_RESERVE_PAGES of the range contiguous
> + * within one area rather than merely free.  Where the last area is short the
> + * range straddles the boundary below it and the two pieces are the last
> + * area's length and the remainder; when both fall short of a chain, no area
> + * holds one however much of the range is free.  Widening the range by the
> + * short area's length in that case moves its start down to the boundary,
> + * which gives the piece below a full VIRTIO_MAP_RESERVE_PAGES + nvqs - 1
> + * slots.  It costs the short area's length, which is under
> + * VIRTIO_MAP_RESERVE_PAGES because that is the case being tested for, and it
> + * is reached only where nslots is just above a multiple of area_slots, so no
> + * geometry in Documentation/driver-api/virtio/virtio-dmb.rst pays for it.
> + *
> + * Sized for every virtqueue the transport creates, which over-counts a device
> + * offering more queue pairs than the driver uses.  Over-counting withholds one
> + * page per unused virtqueue and never denies capacity.  The half-of-the-pool
> + * bound is not part of the policy: it is what keeps the ordinary range
> + * non-empty, and where it binds the pool has fewer pages than the virtqueues
> + * alone want, so the guarantee covers as many of them as it has pages for.
> + * Zero on a pool too small for the mechanism to mean anything, where it is
> + * inert rather than crippling, and zero until the count is known.
> + */
> +static unsigned int virtio_dmb_reserved(const struct virtio_dmb *dmb)
> +{
> +	unsigned int nvqs = READ_ONCE(dmb->nvqs);
> +	unsigned int last, reserved;
> +
> +	if (!nvqs || dmb->nslots < 8 * VIRTIO_MAP_RESERVE_PAGES)
> +		return 0;
> +
> +	reserved = min(VIRTIO_MAP_RESERVE_PAGES + nvqs - 1, dmb->nslots / 2);
> +
> +	/*
> +	 * reserved is at least VIRTIO_MAP_RESERVE_PAGES here, so the
> +	 * subtraction cannot wrap: the branch is taken only where last is
> +	 * below it.  Both pieces short of a chain bounds reserved below 2 *
> +	 * VIRTIO_MAP_RESERVE_PAGES, and nslots is at least 8 of them, so the
> +	 * widened range is still inside the half-of-the-pool bound and leaves
> +	 * the ordinary range a whole area less the reserve.
> +	 */
> +	last = virtio_dmb_area_len(dmb, dmb->nareas - 1);
> +	if (last < VIRTIO_MAP_RESERVE_PAGES &&
> +	    reserved - last < VIRTIO_MAP_RESERVE_PAGES)
> +		reserved += last;
> +
> +	return reserved;
> +}
> +
> +/*
> + * virtio_dmb_note_vqs() records the count that sizes the range above.  It is
> + * defined further down, next to the other entry points, because it has to test
> + * vdev->map against this file's operations.
> + */
> +
> +/*
> + * Claim @nr contiguous slots from area @i, bounded above at @end_max so that
> + * an ordinary claim cannot reach the withheld range.  Returns the first slot,
> + * or -ENOMEM when that one area cannot satisfy the request.  Takes and drops
> + * that area's lock and touches no other area's state, so no path ever holds
> + * two of these locks and there is no ordering between them to get right.
>   */
>  static long virtio_dmb_area_claim(struct virtio_dmb *dmb, unsigned int i,
> -				  unsigned int nr)
> +				  unsigned int nr, unsigned int end_max)
>  {
>  	struct virtio_dmb_area *area = &dmb->areas[i];
>  	unsigned int base = virtio_dmb_area_base(dmb, i);
> -	unsigned int end = base + virtio_dmb_area_len(dmb, i);
> +	unsigned int len = virtio_dmb_area_len(dmb, i);
> +	unsigned int end = min(base + len, end_max);
>  	unsigned long flags, slot;
>  
> +	if (end <= base)
> +		return -ENOMEM;
> +
>  	spin_lock_irqsave(&area->lock, flags);
>  
>  	/*
>  	 * Exact, and inside the lock.  Written as a subtraction from the
> -	 * area's own length rather than as len - used < nr, which underflows.
> +	 * length being searched rather than as len - used < nr, which
> +	 * underflows.
> +	 *
> +	 * @area->used counts the whole area, including any withheld slots in
> +	 * use, so for the one area that straddles @end_max this is
> +	 * conservative: it can refuse an ordinary claim that area could have
> +	 * satisfied, and the walk then tries the next one.  It cannot admit a
> +	 * claim the area could not satisfy.  That is the only approximation
> +	 * here, and it is confined to at most one area out of @nareas.
>  	 */
> -	if (nr > (end - base) - area->used)
> +	if (area->used >= end - base || nr > (end - base) - area->used)
>  		goto not_found;
>  
>  	/*
> @@ -314,9 +406,13 @@ static long virtio_dmb_area_claim(struct virtio_dmb *dmb, unsigned int i,
>  	 * index.  bitmap_find_next_zero_area() returns a value whose sum with
>  	 * @nr exceeds the size it was given when it finds nothing, so that sum
>  	 * is the test; the whole-pool "slot >= nslots" form does not transfer.
> +	 *
> +	 * @area->index is relative to the whole area, so it can point past a
> +	 * clamped @end; starting the sweep there would search nothing, hence
> +	 * the min().  The second sweep from the base then covers the range.
>  	 */
>  	slot = bitmap_find_next_zero_area(dmb->bitmap, end,
> -					  base + area->index, nr, 0);
> +					  min(base + area->index, end), nr, 0);
>  	if (slot + nr > end && area->index)
>  		slot = bitmap_find_next_zero_area(dmb->bitmap, end, base,
>  						  nr, 0);
> @@ -325,7 +421,7 @@ static long virtio_dmb_area_claim(struct virtio_dmb *dmb, unsigned int i,
>  
>  	bitmap_set(dmb->bitmap, slot, nr);
>  	area->used += nr;
> -	area->index = slot + nr < end ? slot + nr - base : 0;
> +	area->index = slot + nr < base + len ? slot + nr - base : 0;
>  
>  	spin_unlock_irqrestore(&area->lock, flags);
>  
> @@ -343,9 +439,39 @@ static long virtio_dmb_area_claim(struct virtio_dmb *dmb, unsigned int i,
>  	return -ENOMEM;
>  }
>  
> +/* One pass over every area, each bounded at @end_max. */
> +static long virtio_dmb_walk(struct virtio_dmb *dmb, unsigned int nr,
> +			    unsigned int end_max)
> +{
> +	unsigned int i, start;
> +	long ret;
> +
> +	/*
> +	 * raw_smp_processor_id() and not smp_processor_id(): the index is
> +	 * computed before any lock is taken, so preemption or migration
> +	 * between the read and the claim only changes which area is tried
> +	 * first.  kernel/dma/swiotlb.c picks its home area on the same
> +	 * reasoning.
> +	 */
> +	start = raw_smp_processor_id() % dmb->nareas;
> +	i = start;
> +	do {
> +		ret = virtio_dmb_area_claim(dmb, i, nr, end_max);
> +		if (ret >= 0)
> +			return ret;
> +
> +		if (++i >= dmb->nareas)
> +			i = 0;
> +	} while (i != start);
> +
> +	return -ENOMEM;
> +}
> +
>  /*
>   * Claim @nr contiguous slots.  Returns the first slot, or -ENOMEM when no
> - * area can satisfy the request.  Exhaustion is a routine condition: the
> + * area can satisfy the request.  @reserved permits the withheld tail of the
> + * pool, and is set only for a mapping of the first chain a virtqueue is
> + * publishing.  Exhaustion is a routine condition: the
>   * region's length bounds how much virtqueue data can be in flight.  What a
>   * caller makes of it is the caller's, and it is not always back-pressure: a
>   * network receive fill has nothing to push back on when it cannot post a
> @@ -371,28 +497,32 @@ static long virtio_dmb_area_claim(struct virtio_dmb *dmb, unsigned int i,
>   * bounds the interrupts-off window to a single area's sweep; the total work in
>   * the failing case is a whole-pool sweep either way.
>   */
> -static long virtio_dmb_claim(struct virtio_dmb *dmb, unsigned int nr)
> +static long virtio_dmb_claim(struct virtio_dmb *dmb, unsigned int nr,
> +			     bool reserved)
>  {
> -	unsigned int i, start;
>  	long ret;
>  
>  	/*
> -	 * raw_smp_processor_id() and not smp_processor_id(): the index is
> -	 * computed before any lock is taken, so preemption or migration
> -	 * between the read and the claim only changes which area is tried
> -	 * first.  kernel/dma/swiotlb.c picks its home area on the same
> -	 * reasoning.
> +	 * The bitmap is its own accounting for the withheld range: bounding
> +	 * the search is what bounds the sum, so no counter is added to a
> +	 * production build's hot path.
>  	 */
> -	start = raw_smp_processor_id() % dmb->nareas;
> -	i = start;
> -	do {
> -		ret = virtio_dmb_area_claim(dmb, i, nr);
> +	ret = virtio_dmb_walk(dmb, nr, dmb->nslots - virtio_dmb_reserved(dmb));
> +	if (ret >= 0)
> +		return ret;
> +
> +	/*
> +	 * The second walk runs only once the first has failed in every area,
> +	 * so the withheld range is a last resort rather than a second pool.
> +	 * That is not the same as "only when the pool is full": next fit can
> +	 * fail on fragmentation while capacity remains, and this reaches the
> +	 * withheld range then too.
> +	 */
> +	if (reserved) {
> +		ret = virtio_dmb_walk(dmb, nr, dmb->nslots);
>  		if (ret >= 0)
>  			return ret;
> -
> -		if (++i >= dmb->nareas)
> -			i = 0;
> -	} while (i != start);
> +	}
>  
>  	/*
>  	 * The geometry rather than a free count: there is no instant at which
> @@ -733,7 +863,13 @@ static void *virtio_dmb_op_alloc(union virtio_map map, size_t size,
>  		goto no_room;
>  
>  	nr = virtio_dmb_slots(size);
> -	ret = virtio_dmb_claim(dmb, nr);
> +	/*
> +	 * false: a virtqueue area is structural, claimed when a queue is
> +	 * created or resized and never under back-pressure, so letting it into
> +	 * the withheld range would consume the reserve for the life of the
> +	 * queue rather than for one chain.
> +	 */
> +	ret = virtio_dmb_claim(dmb, nr, false);
>  	if (ret < 0)
>  		goto no_room;
>  	slot = ret;
> @@ -811,7 +947,7 @@ static dma_addr_t virtio_dmb_op_map_page(union virtio_map map,
>  		return DMA_MAPPING_ERROR;
>  
>  	nr = virtio_dmb_slots(size);
> -	ret = virtio_dmb_claim(dmb, nr);
> +	ret = virtio_dmb_claim(dmb, nr, attrs & VIRTIO_MAP_ATTR_RESERVE);
>  	if (ret < 0) {
>  		/*
>  		 * Counted here rather than in virtio_dmb_claim(), which
> @@ -1023,6 +1159,21 @@ static int virtio_dmb_alloc_failed_get(void *data, u64 *val)
>  DEFINE_DEBUGFS_ATTRIBUTE(virtio_dmb_alloc_failed_fops,
>  			 virtio_dmb_alloc_failed_get, NULL, "%llu\n");
>  
> +/*
> + * A getter rather than debugfs_create_u32(), because the value is derived
> + * from @nvqs and the pool size rather than stored.
> + */
> +static int virtio_dmb_reserved_get(void *data, u64 *val)
> +{
> +	struct virtio_dmb *dmb = data;
> +
> +	*val = virtio_dmb_reserved(dmb);
> +
> +	return 0;
> +}
> +DEFINE_DEBUGFS_ATTRIBUTE(virtio_dmb_reserved_fops, virtio_dmb_reserved_get,
> +			 NULL, "%llu\n");
> +
>  /*
>   * The files go under the device's existing virtio debugfs directory, and exist
>   * only while the device has a region.  They describe one, so their presence is
> @@ -1049,6 +1200,8 @@ static void virtio_dmb_debugfs_init(struct virtio_dmb *dmb)
>  			    &virtio_dmb_hiwater_fops);
>  	debugfs_create_file("alloc_failed", 0400, dir, dmb,
>  			    &virtio_dmb_alloc_failed_fops);
> +	debugfs_create_file("reserved_pages", 0400, dir, dmb,
> +			    &virtio_dmb_reserved_fops);
>  }
>  
>  static void virtio_dmb_debugfs_exit(struct virtio_dmb *dmb)
> @@ -1068,6 +1221,40 @@ static void virtio_dmb_debugfs_exit(struct virtio_dmb *dmb)
>  
>  #endif /* CONFIG_VIRTIO_DEBUG */
>  
> +/**
> + * virtio_dmb_note_vqs - record how many virtqueues the transport created
> + * @vdev: the device
> + * @nvqs: virtqueues about to be created
> + *
> + * Sizes the range withheld so that a virtqueue whose ring is empty can
> + * publish a chain.  Does nothing unless @vdev is using a Device Memory
> + * Buffer.
> + *
> + * Called before the virtqueues exist, so the ring allocations that follow are
> + * ordinary claims and cannot land in the withheld range; and @nvqs therefore
> + * only ever changes while the device has no virtqueues and so no mappings.
> + * WRITE_ONCE() because the claim path reads it without any lock.
> + */
> +void virtio_dmb_note_vqs(struct virtio_device *vdev, unsigned int nvqs)
> +{
> +	struct virtio_dmb *dmb;
> +
> +	if (vdev->map != &virtio_dmb_map_ops)
> +		return;
> +
> +	dmb = vdev->vmap.dmb;
> +	WRITE_ONCE(dmb->nvqs, nvqs);
> +
> +	/*
> +	 * Reported here rather than from virtio_dmb_init(), which runs during
> +	 * feature negotiation and cannot know the count.  One line per probe.
> +	 */
> +	dev_info(&vdev->dev,
> +		 "device memory buffer withholds %u of %u pages for %u virtqueues\n",
> +		 virtio_dmb_reserved(dmb), dmb->nslots, nvqs);
> +}
> +EXPORT_SYMBOL_GPL(virtio_dmb_note_vqs);
> +
>  /*
>   * Whether the device still has virtqueues.  vqs_list_lock is what protects
>   * that list against a concurrent adder.  No caller here can race one, because
> @@ -1524,5 +1711,9 @@ int virtio_dmb_init(struct virtio_device *vdev)
>  }
>  EXPORT_SYMBOL_GPL(virtio_dmb_init);
>  
> +#if IS_ENABLED(CONFIG_VIRTIO_DMB_KUNIT_TEST)
> +#include "virtio_dmb_test.c"
> +#endif
> +
>  MODULE_DESCRIPTION("Virtio device memory buffer allocator");
>  MODULE_LICENSE("GPL");
> diff --git a/drivers/virtio/virtio_dmb.h b/drivers/virtio/virtio_dmb.h
> index 69fbcbb9c2c0..38e4c3ecc9cb 100644
> --- a/drivers/virtio/virtio_dmb.h
> +++ b/drivers/virtio/virtio_dmb.h
> @@ -11,6 +11,7 @@ struct virtio_device;
>  
>  int virtio_dmb_init(struct virtio_device *vdev);
>  void virtio_dmb_destroy(struct virtio_device *vdev);
> +void virtio_dmb_note_vqs(struct virtio_device *vdev, unsigned int nvqs);
>  
>  #else
>  
> @@ -23,6 +24,11 @@ static inline void virtio_dmb_destroy(struct virtio_device *vdev)
>  {
>  }
>  
> +static inline void virtio_dmb_note_vqs(struct virtio_device *vdev,
> +				       unsigned int nvqs)
> +{
> +}
> +
>  #endif /* CONFIG_VIRTIO_DMB */
>  
>  #endif /* _DRIVERS_VIRTIO_VIRTIO_DMB_H */
> diff --git a/drivers/virtio/virtio_dmb_test.c b/drivers/virtio/virtio_dmb_test.c
> new file mode 100644
> index 000000000000..d9dff81a8eb3
> --- /dev/null
> +++ b/drivers/virtio/virtio_dmb_test.c
> @@ -0,0 +1,279 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Tests for the geometry of the withheld range, and for what becomes of a
> + * region when a negotiation drops the feature.
> + *
> + * Included by virtio_dmb.c rather than compiled on its own, so that the tests
> + * reach its static functions and its private structure without widening
> + * either into a header for their benefit.
> + *
> + * The geometry cases test a property rather than a value: that whatever the
> + * pool size, the area size and the virtqueue count, the withheld range holds
> + * VIRTIO_MAP_RESERVE_PAGES slots that are contiguous *within one area*.  That
> + * qualifier is the whole point.  An allocation cannot span two areas, so a
> + * range that is merely large enough does not guarantee a chain, and a pool
> + * whose last area is short can withhold a range whose two pieces are each too
> + * small.  A region has to be just above a multiple of the area size to reach
> + * it, which no plausible device offers and so no test on real hardware
> + * exercises.
> + */
> +#include <kunit/test.h>
> +
> +/* Geometry only: virtio_dmb_reserved() reads no more of the structure. */
> +static void dmb_test_shape(struct kunit *test, struct virtio_dmb *dmb,
> +			   unsigned int nslots, unsigned int area_slots,
> +			   unsigned int nvqs)
> +{
> +	KUNIT_ASSERT_TRUE(test, is_power_of_2(area_slots));
> +
> +	dmb->nslots = nslots;
> +	dmb->area_slots = area_slots;
> +	dmb->area_shift = ilog2(area_slots);
> +	dmb->nareas = DIV_ROUND_UP(nslots, area_slots);
> +	dmb->nvqs = nvqs;
> +}
> +
> +static struct virtio_dmb *dmb_test_pool(struct kunit *test)
> +{
> +	struct virtio_dmb *dmb = kunit_kzalloc(test, sizeof(*dmb), GFP_KERNEL);
> +
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, dmb);
> +
> +	return dmb;
> +}
> +
> +/*
> + * The longest run of withheld slots that lies inside a single area, which is
> + * the largest chain the range can serve.
> + */
> +static unsigned int dmb_test_longest_run(const struct virtio_dmb *dmb,
> +					 unsigned int reserved)
> +{
> +	unsigned int i, best = 0;
> +	unsigned int start;
> +
> +	if (!reserved)
> +		return 0;
> +
> +	start = dmb->nslots - reserved;
> +
> +	for (i = 0; i < dmb->nareas; i++) {
> +		unsigned int base = virtio_dmb_area_base(dmb, i);
> +		unsigned int end = base + virtio_dmb_area_len(dmb, i);
> +		unsigned int lo = max(start, base);
> +
> +		if (end > lo)
> +			best = max(best, end - lo);
> +	}
> +
> +	return best;
> +}
> +
> +static void dmb_test_assert_guarantees(struct kunit *test,
> +				       const struct virtio_dmb *dmb)
> +{
> +	unsigned int reserved = virtio_dmb_reserved(dmb);
> +	unsigned int run;
> +
> +	if (!reserved)
> +		return;
> +
> +	/* One virtqueue at a time can obtain a whole chain. */
> +	run = dmb_test_longest_run(dmb, reserved);
> +	KUNIT_EXPECT_GE_MSG(test, run, VIRTIO_MAP_RESERVE_PAGES,
> +			    "nslots=%u area_slots=%u nareas=%u nvqs=%u reserved=%u",
> +			    dmb->nslots, dmb->area_slots, dmb->nareas,
> +			    dmb->nvqs, reserved);
> +
> +	/* Every virtqueue can obtain one page no running virtqueue can take. */
> +	KUNIT_EXPECT_GE_MSG(test, reserved,
> +			    min(dmb->nvqs, dmb->nslots / 2),
> +			    "nslots=%u nvqs=%u reserved=%u",
> +			    dmb->nslots, dmb->nvqs, reserved);
> +
> +	/* The range never takes more than half the pool, nor all of it. */
> +	KUNIT_EXPECT_LE_MSG(test, reserved, dmb->nslots / 2,
> +			    "nslots=%u nvqs=%u reserved=%u",
> +			    dmb->nslots, dmb->nvqs, reserved);
> +}
> +
> +/*
> + * Every area size the allocator can derive, against pool sizes that put the
> + * last area at every length a straddle needs, and virtqueue counts either
> + * side of a chain's allowance.
> + */
> +static void dmb_test_reserve_geometry(struct kunit *test)
> +{
> +	static const unsigned int area_sizes[] = { 512, 1024, 2048, 4096 };
> +	struct virtio_dmb *dmb = dmb_test_pool(test);
> +	unsigned int a, k, tail, nvqs;
> +
> +	for (a = 0; a < ARRAY_SIZE(area_sizes); a++) {
> +		unsigned int area_slots = area_sizes[a];
> +
> +		for (k = 1; k <= 3; k++) {
> +			for (tail = 0; tail <= 2 * VIRTIO_MAP_RESERVE_PAGES;
> +			     tail++) {
> +				unsigned int nslots = k * area_slots + tail;
> +
> +				for (nvqs = 1; nvqs <= 40; nvqs++) {
> +					dmb_test_shape(test, dmb, nslots,
> +						       area_slots, nvqs);
> +					dmb_test_assert_guarantees(test, dmb);
> +				}
> +			}
> +		}
> +	}
> +}
> +
> +/*
> + * The geometries that showed the guarantee was conditional.  Named so that a
> + * revision that reintroduces the pool-tail bound fails here rather than in
> + * the sweep, where the reason is harder to read off.
> + */
> +static void dmb_test_reserve_straddle(struct kunit *test)
> +{
> +	static const struct {
> +		unsigned int nslots, area_slots, nvqs;
> +	} cases[] = {
> +		{ 528, 512, 3 },	/* last area 16, pieces 18 and 16 */
> +		{ 527, 512, 1 },	/* last area 15, pieces 17 and 15 */
> +		{ 4114, 4096, 7 },	/* last area 18, pieces 20 and 18 */
> +		{ 4114, 1024, 7 },	/* same tail, more areas */
> +		{ 1048600, 512, 17 },	/* last area 24, pieces 24 and 24 */
> +		{ 4097, 4096, 17 },	/* last area 1, but 47 below it: fine */
> +		{ 2100, 512, 7 },	/* last area 52: whole range fits it */
> +	};
> +	struct virtio_dmb *dmb = dmb_test_pool(test);
> +	unsigned int i;
> +
> +	for (i = 0; i < ARRAY_SIZE(cases); i++) {
> +		dmb_test_shape(test, dmb, cases[i].nslots, cases[i].area_slots,
> +			       cases[i].nvqs);
> +		dmb_test_assert_guarantees(test, dmb);
> +	}
> +}
> +
> +/*
> + * The sizing table in Documentation/driver-api/virtio/virtio-dmb.rst states a
> + * withheld count for each of its rows, and a device implementer sizes against
> + * it.  Every row's last area is long enough to hold the range, so the area
> + * term must not change any of them.
> + */
> +static void dmb_test_reserve_documented_sizing(struct kunit *test)
> +{
> +	static const struct {
> +		unsigned int nslots, nvqs, reserved;
> +	} rows[] = {
> +		{ 321, 3, 34 },
> +		{ 1167, 9, 40 },
> +		{ 2295, 17, 48 },
> +		{ 8524, 17, 48 },
> +		{ 17004, 33, 64 },
> +		{ 33964, 65, 96 },
> +		{ 67884, 129, 160 },
> +	};
> +	static const unsigned int area_sizes[] = { 512, 1024, 2048, 4096 };
> +	struct virtio_dmb *dmb = dmb_test_pool(test);
> +	unsigned int i, a;
> +
> +	for (i = 0; i < ARRAY_SIZE(rows); i++) {
> +		for (a = 0; a < ARRAY_SIZE(area_sizes); a++) {
> +			dmb_test_shape(test, dmb, rows[i].nslots,
> +				       area_sizes[a], rows[i].nvqs);
> +
> +			KUNIT_EXPECT_EQ_MSG(test, virtio_dmb_reserved(dmb),
> +					    rows[i].reserved,
> +					    "nslots=%u nvqs=%u area_slots=%u",
> +					    rows[i].nslots, rows[i].nvqs,
> +					    area_sizes[a]);
> +			dmb_test_assert_guarantees(test, dmb);
> +		}
> +	}
> +}
> +
> +/* Inert below the threshold, and before the transport reports a count. */
> +static void dmb_test_reserve_inert(struct kunit *test)
> +{
> +	struct virtio_dmb *dmb = dmb_test_pool(test);
> +	unsigned int nslots;
> +
> +	dmb_test_shape(test, dmb, 4096, 512, 0);
> +	KUNIT_EXPECT_EQ(test, virtio_dmb_reserved(dmb), 0);
> +
> +	for (nslots = 4; nslots < 8 * VIRTIO_MAP_RESERVE_PAGES; nslots++) {
> +		dmb_test_shape(test, dmb, nslots, 512, 3);
> +		KUNIT_EXPECT_EQ_MSG(test, virtio_dmb_reserved(dmb), 0,
> +				    "nslots=%u", nslots);
> +	}
> +
> +	dmb_test_shape(test, dmb, 8 * VIRTIO_MAP_RESERVE_PAGES, 512, 1);
> +	KUNIT_EXPECT_EQ(test, virtio_dmb_reserved(dmb),
> +			VIRTIO_MAP_RESERVE_PAGES);
> +}
> +
> +/*
> + * What virtio_dmb_init() does with state an earlier negotiation left behind
> + * when this one did not accept the feature: releases it, and refuses to
> + * release it under a live virtqueue, which holds kernel addresses inside the
> + * mapping.  The refusal is the only error it returns for a device it is taking
> + * the region away from, so it is the one a caller turns into the FAILED status
> + * bit; releasing it is not an error and sets nothing.  Nothing can reach the
> + * refusal, for the reason the function itself gives, so a test is what covers
> + * it.  Its dev_warn() is expected output.
> + */
> +static void dmb_test_withdrawn_feature(struct kunit *test)
> +{
> +	struct virtio_device *vdev;
> +	const struct virtio_map_ops *prev;
> +	struct virtqueue *vq;
> +	struct virtio_dmb *dmb;
> +
> +	vdev = kunit_kzalloc(test, sizeof(*vdev), GFP_KERNEL);
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vdev);
> +	vq = kunit_kzalloc(test, sizeof(*vq), GFP_KERNEL);
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, vq);
> +	prev = kunit_kzalloc(test, sizeof(*prev), GFP_KERNEL);
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, prev);
> +
> +	spin_lock_init(&vdev->vqs_list_lock);
> +	INIT_LIST_HEAD(&vdev->vqs);
> +
> +	/* Nothing installed: nothing to release, and not an error. */
> +	KUNIT_EXPECT_EQ(test, virtio_dmb_init(vdev), 0);
> +
> +	/* Not kunit_kzalloc(): the last call below frees this. */
> +	dmb = kzalloc_obj(*dmb, GFP_KERNEL);
> +	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, dmb);
> +	dmb->prev_map = prev;
> +	vdev->map = &virtio_dmb_map_ops;
> +	vdev->vmap.dmb = dmb;
> +	list_add(&vq->list, &vdev->vqs);
> +
> +	/* Installed, under a virtqueue: refused, and left addressable. */
> +	KUNIT_EXPECT_EQ(test, virtio_dmb_init(vdev), -EBUSY);
> +	KUNIT_EXPECT_PTR_EQ(test, vdev->map, &virtio_dmb_map_ops);
> +	KUNIT_EXPECT_PTR_EQ(test, vdev->vmap.dmb, dmb);
> +
> +	/* The same withdrawal with no virtqueue left: released, and no error. */
> +	list_del(&vq->list);
> +	KUNIT_EXPECT_EQ(test, virtio_dmb_init(vdev), 0);
> +	KUNIT_EXPECT_PTR_EQ(test, vdev->map, prev);
> +}
> +
> +static struct kunit_case virtio_dmb_test_cases[] = {
> +	/* Slow: the sweep is tens of thousands of geometries. */
> +	KUNIT_CASE_SLOW(dmb_test_reserve_geometry),
> +	KUNIT_CASE(dmb_test_reserve_straddle),
> +	KUNIT_CASE(dmb_test_reserve_documented_sizing),
> +	KUNIT_CASE(dmb_test_reserve_inert),
> +	KUNIT_CASE(dmb_test_withdrawn_feature),
> +	{}
> +};
> +
> +static struct kunit_suite virtio_dmb_test_suite = {
> +	.name = "virtio_dmb",
> +	.test_cases = virtio_dmb_test_cases,
> +};
> +
> +kunit_test_suite(virtio_dmb_test_suite);
> diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c
> index c43c1fc6e843..635ff8012f4c 100644
> --- a/drivers/virtio/virtio_pci_modern.c
> +++ b/drivers/virtio/virtio_pci_modern.c
> @@ -19,6 +19,7 @@
>  #define VIRTIO_PCI_NO_LEGACY
>  #define VIRTIO_RING_NO_LEGACY
>  #include "virtio_pci_common.h"
> +#include "virtio_dmb.h"
>  
>  #define VIRTIO_AVQ_SGS_MAX	4
>  
> @@ -803,8 +804,23 @@ static int vp_modern_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
>  {
>  	struct virtio_pci_device *vp_dev = to_vp_device(vdev);
>  	struct virtqueue *vq;
> -	int rc = vp_find_vqs(vdev, nvqs, vqs, vqs_info, desc);
> +	int rc;
>  
> +	/*
> +	 * Before vp_find_vqs(), so that no virtqueue exists yet: the ring
> +	 * allocations it makes are ordinary claims and cannot land in the
> +	 * range this withholds.
> +	 *
> +	 * nvqs counts the virtqueues the driver asked for.  vp_find_vqs()
> +	 * creates one more when the device offers an administration
> +	 * virtqueue, and that one draws on the guarantee like any other -- its
> +	 * only response to a refusal is a cpu_relax() spin -- so count it here
> +	 * rather than leave it as the one virtqueue without a floor.
> +	 */
> +	virtio_dmb_note_vqs(vdev, nvqs +
> +			    virtio_has_feature(vdev, VIRTIO_F_ADMIN_VQ));
> +
> +	rc = vp_find_vqs(vdev, nvqs, vqs, vqs_info, desc);
>  	if (rc)
>  		return rc;
>  
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index 641ab07be931..6cd82d418c37 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -527,10 +527,44 @@ static int vring_map_errno(const struct vring_virtqueue *vq)
>  	return -ENOMEM;
>  }
>  
> +/*
> + * Whether this mapping may draw on capacity a bounded map implementation
> + * withholds, and account it against the chain's allowance if so.
> + *
> + * Only while the ring is empty, which is exactly the window in which this
> + * virtqueue has no completion of its own to be woken by: the device cannot
> + * signal a used buffer on a queue with no buffers posted.  Every
> + * virtqueue_add_*() path decrements num_free after its mapping loop, so the
> + * test is true for every mapping of a chain going into an empty ring and
> + * false for every mapping after it, with nothing to keep in step.
> + *
> + * @chain_pages accumulates within one virtqueue_add_*() call and dies with
> + * it.  It is charged whether or not the mapping later fails, and a chain that
> + * fails half way therefore over-counts its own allowance and nothing else, so
> + * no bookkeeping can leak.
> + */
> +static unsigned long vring_map_reserve_attr(const struct vring_virtqueue *vq,
> +					    unsigned int *chain_pages,
> +					    size_t size)
> +{
> +	unsigned int pages = DIV_ROUND_UP(size, PAGE_SIZE);
> +
> +	if (*chain_pages + pages > VIRTIO_MAP_RESERVE_PAGES)
> +		return 0;
> +
> +	if (vq->vq.num_free != vring_num(vq))
> +		return 0;
> +
> +	*chain_pages += pages;
> +
> +	return VIRTIO_MAP_ATTR_RESERVE;
> +}
> +
>  /* Map one sg entry. */
>  static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist *sg,
>  			    enum dma_data_direction direction, dma_addr_t *addr,
> -			    u32 *len, bool premapped, unsigned long attr)
> +			    u32 *len, bool premapped, unsigned long attr,
> +			    unsigned int *chain_pages)
>  {
>  	if (premapped) {
>  		*addr = sg_dma_address(sg);
> @@ -577,7 +611,9 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist
>  	 */
>  	*addr = virtqueue_map_page_attrs(&vq->vq, sg_page(sg),
>  					 sg->offset, sg->length,
> -					 direction, attr);
> +					 direction,
> +					 attr | vring_map_reserve_attr(vq, chain_pages,
> +								       sg->length));
>  
>  	if (vring_mapping_error(vq, *addr))
>  		return vring_map_errno(vq);
> @@ -587,13 +623,15 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist
>  
>  static dma_addr_t vring_map_single(const struct vring_virtqueue *vq,
>  				   void *cpu_addr, size_t size,
> -				   enum dma_data_direction direction)
> +				   enum dma_data_direction direction,
> +				   unsigned int *chain_pages)
>  {
>  	if (!vq->use_map_api)
>  		return (dma_addr_t)virt_to_phys(cpu_addr);
>  
> -	return virtqueue_map_single_attrs(&vq->vq, cpu_addr,
> -					  size, direction, 0);
> +	return virtqueue_map_single_attrs(&vq->vq, cpu_addr, size, direction,
> +					  vring_map_reserve_attr(vq, chain_pages,
> +								 size));
>  }
>  
>  static void virtqueue_init(struct vring_virtqueue *vq, u32 num)
> @@ -717,6 +755,7 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
>  	unsigned int i, n, avail, descs_used, err_idx, sg_count = 0;
>  	/* Total length for in-order */
>  	unsigned int total_in_len = 0;
> +	unsigned int chain_pages = 0;
>  	int head;
>  	bool indirect;
>  	int err;
> @@ -783,7 +822,8 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
>  				flags |= VRING_DESC_F_NEXT;
>  
>  			err = vring_map_one_sg(vq, sg, DMA_TO_DEVICE, &addr,
> -					       &len, premapped, attr);
> +					       &len, premapped, attr,
> +				       &chain_pages);
>  			if (err)
>  				goto unmap_release;
>  
> @@ -804,7 +844,8 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
>  				flags |= VRING_DESC_F_NEXT;
>  
>  			err = vring_map_one_sg(vq, sg, DMA_FROM_DEVICE, &addr,
> -					       &len, premapped, attr);
> +					       &len, premapped, attr,
> +				       &chain_pages);
>  			if (err)
>  				goto unmap_release;
>  
> @@ -821,7 +862,8 @@ static inline int virtqueue_add_split(struct vring_virtqueue *vq,
>  		/* Now that the indirect table is filled in, map it. */
>  		dma_addr_t addr = vring_map_single(
>  			vq, desc, total_sg * sizeof(struct vring_desc),
> -			DMA_TO_DEVICE);
> +			DMA_TO_DEVICE,
> +			&chain_pages);
>  		if (vring_mapping_error(vq, addr)) {
>  			err = vring_map_errno(vq);
>  			goto unmap_release;
> @@ -1601,6 +1643,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
>  	struct vring_packed_desc *desc;
>  	struct scatterlist *sg;
>  	unsigned int i, n, err_idx, len, total_in_len = 0;
> +	unsigned int chain_pages = 0;
>  	u16 head;
>  	dma_addr_t addr;
>  
> @@ -1624,7 +1667,8 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
>  		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
>  			if (vring_map_one_sg(vq, sg, n < out_sgs ?
>  					     DMA_TO_DEVICE : DMA_FROM_DEVICE,
> -					     &addr, &len, premapped, attr))
> +					     &addr, &len, premapped, attr,
> +				       &chain_pages))
>  				goto unmap_release;
>  
>  			desc[i].flags = cpu_to_le16(n < out_sgs ?
> @@ -1647,7 +1691,8 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
>  	/* Now that the indirect table is filled in, map it. */
>  	addr = vring_map_single(vq, desc,
>  			total_sg * sizeof(struct vring_packed_desc),
> -			DMA_TO_DEVICE);
> +			DMA_TO_DEVICE,
> +			&chain_pages);
>  	if (vring_mapping_error(vq, addr))
>  		goto unmap_release;
>  
> @@ -1740,6 +1785,7 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq,
>  	struct vring_packed_desc *desc;
>  	struct scatterlist *sg;
>  	unsigned int i, n, c, descs_used, err_idx, len;
> +	unsigned int chain_pages = 0;
>  	__le16 head_flags, flags;
>  	u16 head, id, prev, curr, avail_used_flags;
>  	int err;
> @@ -1799,7 +1845,8 @@ static inline int virtqueue_add_packed(struct vring_virtqueue *vq,
>  
>  			err = vring_map_one_sg(vq, sg, n < out_sgs ?
>  					       DMA_TO_DEVICE : DMA_FROM_DEVICE,
> -					       &addr, &len, premapped, attr);
> +					       &addr, &len, premapped, attr,
> +				       &chain_pages);
>  			if (err)
>  				goto unmap_release;
>  
> @@ -1899,6 +1946,7 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
>  	struct vring_packed_desc *desc;
>  	struct scatterlist *sg;
>  	unsigned int i, n, sg_count, err_idx, total_in_len = 0;
> +	unsigned int chain_pages = 0;
>  	__le16 head_flags, flags;
>  	u16 head, avail_used_flags;
>  	bool avail_wrap_counter;
> @@ -1961,7 +2009,8 @@ static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
>  
>  			err = vring_map_one_sg(vq, sg, n < out_sgs ?
>  					       DMA_TO_DEVICE : DMA_FROM_DEVICE,
> -					       &addr, &len, premapped, attr);
> +					       &addr, &len, premapped, attr,
> +				       &chain_pages);
>  			if (err)
>  				goto unmap_release;
>  
> @@ -3880,9 +3929,15 @@ dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
>  					   page, offset, size,
>  					   dir, attrs);
>  
> +	/*
> +	 * Strip the bits virtio owns before the DMA API sees them: it defines
> +	 * DMA_ATTR_* over the same word, and a bit outside that set has no
> +	 * defined meaning there.  A map implementation is the only reader of
> +	 * them.
> +	 */
>  	return dma_map_page_attrs(vring_dma_dev(vq),
>  				  page, offset, size,
> -				  dir, attrs);
> +				  dir, attrs & ~VIRTIO_MAP_ATTR_MASK);
>  }
>  EXPORT_SYMBOL_GPL(virtqueue_map_page_attrs);
>  
> @@ -3907,7 +3962,8 @@ void virtqueue_unmap_page_attrs(const struct virtqueue *_vq,
>  				      map_handle, size, dir, attrs);
>  	else
>  		dma_unmap_page_attrs(vring_dma_dev(vq), map_handle,
> -				     size, dir, attrs);
> +				     size, dir,
> +				     attrs & ~VIRTIO_MAP_ATTR_MASK);
>  }
>  EXPORT_SYMBOL_GPL(virtqueue_unmap_page_attrs);
>  
> diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h
> index a6780aa85966..a25552862ec4 100644
> --- a/include/linux/virtio_config.h
> +++ b/include/linux/virtio_config.h
> @@ -3,6 +3,7 @@
>  #define _LINUX_VIRTIO_CONFIG_H
>  
>  #include <linux/err.h>
> +#include <linux/bits.h>
>  #include <linux/bug.h>
>  #include <linux/virtio.h>
>  #include <linux/virtio_byteorder.h>
> @@ -219,6 +220,38 @@ struct virtio_map_ops {
>  	size_t (*max_mapping_size)(union virtio_map map);
>  };
>  
> +/*
> + * Pages one virtqueue at a time is guaranteed to be able to map through a map
> + * implementation with a bounded pool, whatever the other virtqueues of the
> + * device have mapped.  Enough for one page-granular descriptor chain at the
> + * default CONFIG_MAX_SKB_FRAGS: a network receive buffer in the non-mergeable
> + * case is MAX_SKB_FRAGS + 2 scatterlist entries plus an indirect table, which
> + * is 20 pages where a page is 4 KiB and MAX_SKB_FRAGS is 17.  A chain larger
> + * than this -- a raised CONFIG_MAX_SKB_FRAGS, or entries spanning more than a
> + * page each -- draws on the reserve for as much of itself as fits and is not
> + * guaranteed.  Neither is a second chain of more than one page concurrent
> + * with the first: what every virtqueue is guaranteed is the single page that
> + * takes its ring from empty to non-empty.  Both hold on any pool of eight
> + * times this many pages or more, which is where an implementation withholds
> + * anything at all; below that there is no range and no guarantee.
> + */
> +#define VIRTIO_MAP_RESERVE_PAGES	32u
> +
> +/*
> + * map_page() attrs bits owned by virtio rather than by the DMA API.
> + * DMA_ATTR_* occupies bits 1 to 13 today; these sit above it, and
> + * virtqueue_map_page_attrs() masks them off before any dma_map_*() call, so
> + * the DMA API never sees one.  The gap is deliberate headroom rather than a
> + * partition: whoever grows either range has to check the other.
> + *
> + * VIRTIO_MAP_ATTR_RESERVE: this mapping is part of the first descriptor chain
> + * a virtqueue is publishing, and an implementation with a bounded pool should
> + * satisfy it from capacity withheld for that purpose if it has no other.
> + * An implementation that ignores the bit behaves exactly as before.
> + */
> +#define VIRTIO_MAP_ATTR_RESERVE		BIT(24)
> +#define VIRTIO_MAP_ATTR_MASK		VIRTIO_MAP_ATTR_RESERVE
> +
>  /* If driver didn't advertise the feature, it will never appear. */
>  void virtio_check_driver_offered_feature(const struct virtio_device *vdev,
>  					 unsigned int fbit);