Re: [RFC PATCH 00/12] virtio: support devices that own their virtqueue memory

["Graf (AWS), Alexander" <graf@amazon.de>]

On 10.08.26 22:39, Stefan Hajnoczi wrote:
> On Sun, Aug 09, 2026 at 06:19:58PM +0000, Alexander Graf wrote:
>> Virtio drivers use guest memory to back virtqueues and their buffers.
>> That means a VMM needs to be able to map guest memory. That is ok in the
>> normal virt case. It gets icky with confidential computing (where we use
>> swiotlb as workaround) and it defeats the purpose of isolated vhost-user
>> backing devices, because they end up with full RAM access to the guest.
>>
>> So instead, I'm proposing an extension to virtio which allows it to give
>> each virtio device its own dedicated memory region to communicate with the
>> host, called DMB (Device Memory Buffer). A trusted hypervisor can force
>> DMB to be present, which then enables safer, more isolated and resilient
>> communication between guest and host.
>>
>> With DMB, the device provides a shared memory region that both parties
>> agree is the full memory map both have access to. All memory offsets
>> that previously would have been into guest RAM, are then offsets into
>> this shared memory buffer region. One nice property of this is that it
>> is a generic mechanism in the virtio transport layer, so higher level
>> drivers work unmodified.
> This is similar to VIRTIO's Shared Memory Regions. A problem with this
> kind of approach is that guest software that depends on zero-copy,
> O_DIRECT, the ability to mmap, etc may break when passing an address
> from one device to another device.
>
> For example, a guest userspace application writing to a virtio-blk
> device with O_DIRECT can pass any source memory buffer. One VIRTIO
> device will be unable to address another VIRTIO device's DMB. This
> problem also extends to vhost-user where one back-end cannot access
> another back-end's DMB or Shared Memory Regions.
>
> Maybe your use case will never hit this problem because you can rely on
> the guest software never to assume zero-copy/O_DIRECT/etc works.
> virtiofs hit it with its DAX Shared Memory Regions.


Virtiofs is a bit trickier. It assumes reverse mapping order (host maps 
memory into guest address space's SHM region) which is something I'm not 
looking to support. The main reason you did run into it is because the 
page cache is actually implemented by the SHM region, so Linux 
(rightfully) assumes that it has direct access to it.

For normal virtio device operations (like O_DIRECT on virtio-blk), I am 
not sure whether we can ever run into a case where anything in the 
driver assumes that both devices share the same IOVA space. Thanks a lot 
for the heads-up though, I'll take a deeper look to make sure that this 
is the case. But even if it is, IMHO we can consider it a guest bug and 
fix it in the guest code when DMB is active.


> I'm mentioned it in case this is something you want to think about
> before deploying this approach.
>
>> I was exploring to use swiotlb instead to create individual pools. But
>> that approach has multiple downsides:
>>
>> 1. Swiotlb is an OS primitive which is not available in all Operating
>> Systems. DMB however lives in the virtio transport layer, which means we
>> can add support for it in any OS independent of generic layers. This
>> helps with Windows support.
>>
>> 2. We munge DMA space together. DMB provides a separate DMA space per
>> virtio device. This means we can for example implement a device in
>> vhost-user and give the implementing process only visibility to the DMB
>> region, not all of guest memory. That reduces the exposure the
>> vhost-user provider has, improving security.
> Connor Kite is working on a different approach for vhost-user memory
> isolation here:
> https://lore.kernel.org/qemu-devel/20260723-vhost-user-isolated-memory-v1-0-6b97c439eb28@gmail.com/T/#t
>
> It involves a bounce buffer at the VMM level. Unmodified vhost-user
> back-ends never sees guest RAM. Guest drivers are also unmodified. The
> cost of doing this is that the VMM has to intercept kick and call
> eventfds in order to copy between the bounce buffer and guest RAM.
>
> I don't see DMB or vhost-user memory isolation as conflicting features.
> There can be two ways of solving the same problem with different
> trade-offs. I just wanted to share a link to Connor's ongoing work.


Thanks a bunch :). My design motivation is a bit different, but it's 
great to see more people interested in isolation!


Alex