Re: [RFC PATCH 00/12] virtio: support devices that own their virtqueue memory

["Graf (AWS), Alexander" <graf@amazon.de>]

On 10.08.26 10:25, Graf (AWS), Alexander wrote:
> On 10.08.26 10:04, Michael S. Tsirkin wrote:
>> On Mon, Aug 10, 2026 at 07:39:02AM +0000, Graf (AWS), Alexander wrote:
>>> Hey Michael,
>>>
>>> Thanks a bunch for having a detailed and quick look!
>>>
>>> On 10.08.26 08:23, Michael S. Tsirkin wrote:
>>>> On Sun, Aug 09, 2026 at 06:19:58PM +0000, Alexander Graf wrote:
>>>>> Virtio drivers use guest memory to back virtqueues and their buffers.
>>>>> That means a VMM needs to be able to map guest memory. That is ok in the
>>>>> normal virt case. It gets icky with confidential computing (where we use
>>>>> swiotlb as workaround) and it defeats the purpose of isolated vhost-user
>>>>> backing devices, because they end up with full RAM access to the guest.
>>>>>
>>>>> So instead, I'm proposing an extension to virtio which allows it to give
>>>>> each virtio device its own dedicated memory region to communicate with the
>>>>> host, called DMB (Device Memory Buffer). A trusted hypervisor can force
>>>>> DMB to be present, which then enables safer, more isolated and resilient
>>>>> communication between guest and host.
>>>>>
>>>>> With DMB, the device provides a shared memory region that both parties
>>>>> agree is the full memory map both have access to. All memory offsets
>>>>> that previously would have been into guest RAM, are then offsets into
>>>>> this shared memory buffer region. One nice property of this is that it
>>>>> is a generic mechanism in the virtio transport layer, so higher level
>>>>> drivers work unmodified.
>>>>>
>>>>> I was exploring to use swiotlb instead to create individual pools. But
>>>>> that approach has multiple downsides:
>>>>>
>>>>> 1. Swiotlb is an OS primitive which is not available in all Operating
>>>>> Systems. DMB however lives in the virtio transport layer, which means we
>>>>> can add support for it in any OS independent of generic layers. This
>>>>> helps with Windows support.
>>>> How does it help, if you are going to put a pool in
>>>> the driver, put a pool in the driver. Maybe with virtio mem to
>>>> simplify allocation.
>>> I'm not sure I understand the suggestion :)
>>>
>> I'm not sure what the problem is for windows :)
>>
>> But if you want a chunk of contiguos memory that windows
>> does not poke at without a driver, virtio mem is that :)
>>
>>>>> 2. We munge DMA space together. DMB provides a separate DMA space per
>>>>> virtio device. This means we can for example implement a device in
>>>>> vhost-user and give the implementing process only visibility to the DMB
>>>>> region, not all of guest memory. That reduces the exposure the
>>>>> vhost-user provider has, improving security.
>>>>>
>>>>> 3. Devices can opt-in. A hypervisor can choose to use standard virtio
>>>>> semantics for self-implemented devices (e.g. NSM), while requiring DMB
>>>>> for devices implemented by less trustworthy providers. The
>>>>> non-trustworthy devices do not get any visibility into the trustworthy
>>>>> ones, even with DMB in place for both.
>>>> So I am not sure whether the implication is that it's purely a software
>>>> construct. But if it is, can we extend virtio iommu to
>>>> add a way to discover and enforce trust boundaries?
>>>> And maybe translate offsets to BARs, if that is desired?
>>>>
>>>> It seems to be that the result would be that we don't need fiddly
>>>> special casing in virtio ring specifically, and a lot of things like
>>>> pre-mapped dma will begin to work.
>>> On thing I'm trying to avoid is dynamicity. Anything that dynamically
>>> changes visibility or needs state tracking is something that can go
>>> wrong. By keeping everything self-contained within the guest, I can
>>> reason about what is visible and what is not easily. Especially for
>>> confidential computing, IMHO static wins over dynamic in general.
>>>
>>> Or did I misunderstand your suggestion?
>> I get this part. But we can absolutely make it static.
>>
>>
>> Let's start by replicating your functionality with virtio-iommu.  So:
>>
>> -device is behind virtio-iommu
>> -virtio-iommu tells guest "this device can only consume memory from that range,"
>> -and maybe: ...and translation is 1:1 with this offset
>>
>> if guest does not acknowledge it will just fail?
>>
>>
>> this covers the proposal here simply by using a dedicated range
>> per device, right?
>>
>>
>>
>> But look what we can easily add later: share a range
>> between devices, now you can move data between them with zero copies.
>>
>>
>> Isn't that better?
>
> I don't know if "Add device emulation for virtio-mem and virtio-iommu,
> including all of its state machine and tracking" is necessarily an
> improvement, but let me try and prototype it.


I've been pondering about the virtio-mem idea a bit. I think you're 
trying to address 2 concerns:

1) RAM should be annotated as RAM.
2) We may in the future want to allow zero-copy DMA between virtio 
devices with DMB active

For 1, I tend to agree. That's why SHM is such a good fit. SHM already 
has RAM properties today and virtio devices use it that way. I don't 
think we need a special virtio-mem addition for that semantic.

However, virtio-mem brings in an interesting (and dangerous) other 
concept into the mix: Treating shared memory and private memory as a 
potentially common pool. Depending on how we implement the new 
virtio-mem mode, we may end up accidentally treating that shared memory 
as RAM.

That's a very undesirable design property: I want that a guest can rest 
fairly assured that it only every passes data into the shared memory 
area that it actively wants to. It's one of the properties that most of 
today's confidential compute technologies get wrong IMHO. I've seen way 
too many cases where you end up with a generic framework that happens to 
declare a page shared when in reality the page actually contains a mix 
of shared and private data. By clearly separating private and shared 
memory, I have a clear gate of when data passes between them and can 
easily reason about the secrecy of data.


for 2, I think it's an interesting use case. But it's nothing I'm 
worried about today. If it comes for free, I'm happy to take it. But 
looking at how much of a complicated monster we'd be creating with 
virtio-mem and virtio-iommu, I am convinced it's the wrong thing to 
optimize for :).

However, DMB as spec'ed does reference a "target SHM id". If we really 
later see a need for shared DMA, I think we can fairly easily create an 
extension that picks up your virtio-mem idea (or maybe something 
different) to implement a shared target identifier instead.


As for pure virtio-iommu, I genuinely fail to see how it is an 
improvement to either the code flow or the design principles I'm trying 
to achieve with this. We would still need a special purpose allocator. 
Or new zones, which Linux hates. And all we're gaining is yet another 
device that maintains useless state which eats up resources and which 
requires additional inter-connection between components to properly 
describe and tie up.

Maybe I'm missing the real point you're trying to make? :)


Alex