Re: Mapping non-pinned memory from one Xen domain into another

Teddy Astie
2026-03-26T18:26:53Z

Le 26/03/2026 à 18:18, Demi Marie Obenour a écrit :
> On 3/24/26 14:00, Teddy Astie wrote:
>>> ## Restrictions on lent memory>>
>>> Lent memory is still considered to belong to the lending domain.
>>> The borrowing domain can only access it via its p2m. Hypercalls made
>>> by the borrowing domain act as if the borrowed memory was not present.
>>> This includes, but is not limited to:
>>>
>>> - Using pointers to borrowed memory in hypercall arguments.
>>> - Granting borrowed memory to other VMs.
>>> - Any other operation that depends on whether a page is accessible
>>> by a domain.
>>
>> What about emulated instructions that refers to this memory ?
>
> This would be allowed if (and only if) it can trigger paging as you
> wrote above.
>
>>> Furthermore:
>>>
>>> - Borrowed memory isn't mapped into the IOMMU of any PCIe devices
>>> the guest has attached, because IOTLB faults generally are not
>>> replayable.
>>>
>>
>> Given that (as written bellow) Borrowed memory is a part of some form of
>> emulated BAR or special region, there is no guarantee that DMA will work
>> properly anyway (unless P2P DMA support is advertised).
>>
>> Splitting the IOMMU side from the P2M is not a good idea as it rules out
>> the "IOMMU HAP PT Share" optimization.
>
> If the pages are mapped in the IOMMU, paging them out requires an
> IOTLB invalidation. My understanding is that these are far too slow.
>

yes (aside specific cases like with paravirtualized IOMMU), but only if
you have a device in the guest.

The problem is that that would force us to modify the ABI to have
"non-DMA-able" memory in the guest, which doesn't exist yet aside
specific cases like grants in PV.

> How important is sharing the HAP and IOMMU page tables?
>
>>> - Foreign mapping hypercalls that reference lent memory will fail.
>>> Otherwise, the domain making the foreign mapping hypercall could
>>> continue to access the borrowed memory after the lease had been
>>> revoked. This is true even if the domain performing the foreign
>>> mapping is an all-powerful dom0. Otherwise, an emulated device
>>> could access memory whose lease had been revoked.
>>>
>>> This also means that live migration of a domain that has borrowed
>>> memory requires cooperation from the lending domain. For now, it
>>> will be considered out of scope. Live migration is typically used
>>> with server workloads, and accelerators for server hardware often
>>> support SR-IOV.
>>>
>>> ## Where will lent memory appear in a guest's address space?
>>>
>>> Typically, lent memory will be an emulated PCI BAR. It may be emulated
>>> by dom0 or an alternate ioreq server. However, it is not *required*
>>> to be a PCI BAR.
>>>
>>
>> ---
>>
>> While the design could work (albeit the implied complexity), I'm not a
>> big fan of it, or at least, it needs to consider some constraints for
>> having reasonable performance.
>> One of the big issue is that a performance-sensitive system (virtualized
>> GPU) is interlocking with several "hard to optimize" subsystem like P2M
>> or Dom0 having to process a paging event.
>>
>> Modifying the P2M (especially removing entries) is a fairly expensive
>> operation as it sometimes requires pausing all the vCPUs each time it's
>> done.
>
> Not every GPU supports recoverable page faults. Even when they
> are supported, they are extremely expensive. Each of them involves
> a round-trip from the GPU to the CPU and back, which means that a
> potentially very large number of GPU cores are blocked until the
> CPU can respond. Therefore, GPU driver developers avoid relying on
> GPU page faults whenever possible. Instead, data is moved in large
> chunks using a dedicated DMA engine in the GPU.
> As a result, I'm not too concerned with the cost of P2M manipulation.
> Anything that requires making a GPU buffer temporarily inaccessible
> is already an expensive process, and driver developers have strong
> incentives to keep the time the buffer is unmapped as short as
> possible.
> If performance turns out to be a problem, something like KVM's
> asynchronous page faults might be a better solution.
>

Asynchronous page fault looks like a interesting and potentially easier
to implement.

IIUC, the idea is to make the pages disappears on the guest behalf, and
the guest would have to deal with the eventual page fault. Currently in
Xen, a unhandled #NPF is fatal, but that could be tuned down for
specific regions and transformed into a #PF or another exception for the
guest to handle.

We have actually a similar need for SEV-ES MMIO handling, as we need to
distinguish "MMIO-related NPF" (to paravirtualize through GHCB) to the
other NPF; which needs to be configured in advance in page-tables (so
that the CPU choose between #VC and VMEXIT#NPF).

It would also need some form of para-virtualization coming from virtio
or a new Xen PV driver for the guest to be made aware of this mechanism.
I also assume that the guest handles properly that kind of event.

>> If it's done at 4k granularity, it would also lack superpage support,
>> which wouldn't help either. (doing things at the 2M+ scale would help,
>> but I don't know enough how MMU notifier does things.
>>
>> While I agree that grants is not a adequate mechanism for this (for
>> multiples reasons), I'm not fully convinced of the proposal.
>> I would prefer a strategy where we map a fixed amount of RAM+VRAM as a
>> blob, along with some form of cooperative hotplug mechanism to
>> dynamically provision the amount.
>
> I asked the GPU driver developers about pinning VRAM like this a couple
> years ago or so. The response I got was that it isn't supported.
> I suspect that anyone needing VRAM pinning for graphics workloads is
> using non-upstreamable hacks, most likely specific to a single driver.
>
> More generally, the entire graphics stack receives essentially no
> testing under Xen. There have been bugs that have affected Qubes OS
> users for months or more, and they went unfixed because they couldn't
> be reproduced outside of Xen. To the upstream graphics developers,
> Xen might as well not exist. This means that any solution that
> requires changing the graphics stack is not a practical option,
> and I do not expect this to change in the foreseeable future.

--
Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech