Re: [PATCH v3] Add VIRTIO_F_DMB (Device Memory Buffer)

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 10755 bytes --]

On 8/9/26 17:48, Michael S. Tsirkin wrote:
> On Tue, Aug 04, 2026 at 04:12:02PM +0000, Alexander Graf wrote:
>> When a transport uses a shared bounce region for virtqueue memory today,
>> that region is global and shared by multiple devices at once. Devices
>> then contend over one small region, so a busy device can starve the
>> others, and one device's emulation cannot be isolated or torn down on its
>> own because its state lives in memory everyone else is using too.
>>
>> VIRTIO_F_DMB gives each device its own shared memory region, the Device
>> Memory Buffer, that holds that device's virtqueues (Descriptor, Driver
>> and Device Areas) and the buffers the descriptors reference. When the
>> feature is negotiated, every address the driver places in a virtqueue is
>> an offset into that region rather than a physical or bus address, so the
>> device only ever touches memory the driver placed there. Because each
>> device owns its region, its emulation can run in a separate host process
>> and be terminated independently, and devices no longer contend over one
>> shared region.
>>
>> The device reports the region's shmid through a transport-specific
>> mechanism (dmb_shm_id in the PCI common configuration structure, DMBSHMId
>> on MMIO), so the shmid does not collide with device-specific or future
>> transport-specific shared memory regions. The driver locates the region
>> through the existing Shared Memory Regions facility. VIRTIO_F_DMB refines
>> VIRTIO_F_ACCESS_PLATFORM and is negotiated only together with it.
>>
>> To provide the confinement guarantee, the device bounds-checks every
>> driver-supplied offset against the region and sets DEVICE_NEEDS_RESET on
>> violation.
>>
>> The general requirement on shared memory regions forbids a device to
>> expose a region used to control its operation or to stream data, and a
>> Device Memory Buffer is used for both, so it is named as that
>> requirement's exception.
>>
>> The Device Memory Buffer is memory the driver shares with the device on
>> platforms that distinguish shared from private mappings, and the device
>> provides coherency without cache maintenance by the driver. A device
>> that requires the platform's barriers offers VIRTIO_F_ORDER_PLATFORM,
>> and a driver accepts VIRTIO_F_DMB only together with it. Offset 0 in
>> the region is reserved: a device that predates the feature reads a
>> Descriptor Area address of 0 as a queue that is not in use, as the
>> legacy QueuePFN register states.
>>
>> Signed-off-by: Alexander Graf <graf@amazon.com>
>> ---
>>
>> Link: https://lore.kernel.org/virtio-comment/20260702040006.65669-1-graf@amazon.com/ [v1]
>> Link: https://lore.kernel.org/virtio-comment/20260716215426.46037-1-graf@amazon.com/ [v2]
>>
>> v1 -> v2:
>>  - Replace the fixed reserved shmid 2 and its registry with a shmid the
>>    device reports per transport, as Stefan Hajnoczi suggested on v1.
>>  - Spell out per-transport compatibility for dmb_shm_id and DMBSHMId.
>>  - Set FAILED when the driver cannot find the region, rather than refuse
>>    to negotiate.
>>  - List VIRTIO_F_DMB under CCW's features reserved for future use.
>>
>> v2 -> v3:
>>  - Require the region to be memory the driver shares with the device.
>>  - Require coherency; couple VIRTIO_F_DMB to VIRTIO_F_ORDER_PLATFORM,
>>    replacing v2's sentence on the areas' interaction with it.
>>  - Reserve offset 0 in the region.
>>  - Except the Device Memory Buffer from the shared-region prohibition.
>>  - Allow a device to expose further shared memory regions.
>>  - Move bit 44 into the queue and feature negotiation reserved list.
>>  - Aim two cross-references at the Reserved Feature Bits chapter.
>>
>>
>>  conformance.tex    |   2 +
>>  content.tex        |  28 ++++++-
>>  shared-mem.tex     | 199 ++++++++++++++++++++++++++++++++++++++++++++-
>>  transport-ccw.tex  |   1 +
>>  transport-mmio.tex |  15 ++++
>>  transport-pci.tex  |  20 +++++
>>  6 files changed, 262 insertions(+), 3 deletions(-)
>>
>> diff --git a/conformance.tex b/conformance.tex
>> index 9af31e2..cf4259d 100644
>> --- a/conformance.tex
>> +++ b/conformance.tex
>> @@ -104,6 +104,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>  \item \ref{drivernormative:General Initialization And Device Operation / Device Initialization}
>>  \item \ref{drivernormative:General Initialization And Device Operation / Device Cleanup}
>>  \item \ref{drivernormative:Reserved Feature Bits}
>> +\item \ref{drivernormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
>>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
>>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
>>  \item \ref{drivernormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device parts}
>> @@ -191,6 +192,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>>  \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / The Virtqueue Descriptor Table}
>>  \item \ref{devicenormative:Basic Facilities of a Virtio Device / Packed Virtqueues / Scatter-Gather Support}
>>  \item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions}
>> +\item \ref{devicenormative:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}
>>  \item \ref{devicenormative:Reserved Feature Bits}
>>  \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device and driver capabilities}
>>  \item \ref{devicenormative:Basic Facilities of a Virtio Device / Device groups / Group administration commands / Device resource objects}
>> diff --git a/content.tex b/content.tex
>> index 243ce2d..8c5a844 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -102,10 +102,10 @@ \section{Feature Bits}\label{sec:Basic Facilities of a Virtio Device / Feature B
>>  \begin{description}
>>  \item[0 to 23, 41, 42 and 50 to 127] Feature bits for the specific device type
>>  
>> -\item[24 to 40, and 43] Feature bits reserved for extensions to the queue and
>> +\item[24 to 40, 43 and 44] Feature bits reserved for extensions to the queue and
>>    feature negotiation mechanisms, see \ref{sec:Reserved Feature Bits}
>>  
>> -\item[44 to 49, and 128 and above] Feature bits reserved for future extensions.
>> +\item[45 to 49, and 128 and above] Feature bits reserved for future extensions.
>>  \end{description}
>>  
>>  \begin{note}
>> @@ -946,6 +946,15 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>     suspend the device by set the SUSPEND bit to 1.
>>     See \ref{sec:Basic Facilities of a Virtio Device / Device Status Field}.
>>  
>> +  \item[VIRTIO_F_DMB(44)] This feature indicates that the device offers a
>> +  Device Memory Buffer: a shared memory region, owned by the device,
>> +  that holds the virtqueues and the buffers they reference.  When this feature
>> +  is negotiated, every address that the driver places in a virtqueue is an
>> +  offset into the Device Memory Buffer region rather than a physical address
>> +  or a bus address.  The device is thereby confined to memory the driver has
>> +  published in that region.
>> +  See \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
>> +
>>  \end{description}
>>  
>>  \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>> @@ -982,6 +991,13 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>  
>>  A driver SHOULD accept VIRTIO_F_NOTIF_CONFIG_DATA if it is offered.
>>  
>> +A driver MUST NOT accept VIRTIO_F_DMB unless it also accepts
>> +VIRTIO_F_ACCESS_PLATFORM.  If VIRTIO_F_ORDER_PLATFORM is offered, a driver
>> +MUST NOT accept VIRTIO_F_DMB unless it also accepts VIRTIO_F_ORDER_PLATFORM.
> 
> Why is that? access platform is for DMA from device of which
> there's none here. Both it and order platform are useful to disable
> to speed up software configs.

I agree: combining VIRTIO_F_DMB and VIRTIO_F_ACCESS_PLATFORM shouldn't
be combined.  I'd treat the combination as an error.

>> +For the driver requirements that apply once VIRTIO_F_DMB has been
>> +negotiated, see
>> +\ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}~\nameref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions / Device Memory Buffer}.
>> +
>>  \devicenormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
>>  
>>  A device MUST offer VIRTIO_F_VERSION_1.  A device MAY fail to operate further
>> @@ -1011,6 +1027,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
>>  and presents a PCI SR-IOV capability structure, otherwise
>>  it MUST NOT offer VIRTIO_F_SR_IOV.
>>  
>> +A device that offers VIRTIO_F_DMB MUST also offer VIRTIO_F_ACCESS_PLATFORM.
>> +A device that offers VIRTIO_F_DMB MUST also offer VIRTIO_F_ORDER_PLATFORM if
>> +the device requires the driver to use memory barriers suitable for devices
>> +described by the platform when accessing the Device Memory Buffer region.
> 
> This is really vague. On most platforms drivers can map
> in any way they want. What is expected here? What barriers to
> use will very much depend on that.
> 
> In particular, how do you expect this mapped? If cached what forces
> the data out of the CPU cache?

My understanding is that VIRTIO_F_DMB is only a good choice for
untrusted devices implemented in software.  Both the original use-case
(protected VMs on Android) and my use-case (Spectrum VMs using untrusted
virtio-vhost-user devices) fall into this category.  In this case,
the DMB is just ordinary RAM, so VIRTIO_F_ORDER_PLATFORM isn't needed.

On the other hand, hardware devices shouldn't implement VIRTIO_F_DMB.
This would require the CPU to read data over PCIe, which is slow.
Instead, the driver should use a bounce buffer in system RAM, which
provides the same level of security (assuming hardware DMA protection
of some sort).

I'd be fine with making VIRTIO_F_DMB and VIRTIO_F_ORDER_PLATFORM
mutually exclusive.  Forbid devices from offering both, and require
drivers to refuse to use a device that does.  Reconsider once a
sensible use-case appears.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]