Re: [RFC PATCH 14/18] KVM: Add asynchronous userfaults, KVM_READ_USERFAULT

[Nikita Kalyazin <kalyazin@amazon.com>]

On 26/07/2024 19:00, James Houghton wrote:
> If it would be useful, we could absolutely have a flag to have all
> faults go through the asynchronous mechanism. :) It's meant to just be
> an optimization. For me, it is a necessary optimization.
> 
> Userfaultfd doesn't scale particularly well: we have to grab two locks
> to work with the wait_queues. You could create several userfaultfds,
> but the underlying issue is still there. KVM Userfault, if it uses a
> wait_queue for the async fault mechanism, will have the same
> bottleneck. Anish and I worked on making userfaults more scalable for
> KVM[1], and we ended up with a scheme very similar to what we have in
> this KVM Userfault series.
Yes, I see your motivation. Does this approach support async pagefaults 
[1]? Ie would all the guest processes on the vCPU need to stall until a 
fault is resolved or is there a way to let the vCPU run and only block 
the faulted process?

A more general question is, it looks like Userfaultfd's main purpose was 
to support the postcopy use case [2], yet it fails to do that 
efficiently for large VMs. Would it be ideologically better to try to 
improve Userfaultfd's performance (similar to how it was attempted in 
[3]) or is that something you have already looked into and reached a 
dead end as a part of [4]?

[1] https://lore.kernel.org/lkml/4AEFB823.4040607@redhat.com/T/
[2] https://lwn.net/Articles/636226/
[3] https://lore.kernel.org/lkml/20230905214235.320571-1-peterx@redhat.com/
[4] 
https://lore.kernel.org/linux-mm/CADrL8HVDB3u2EOhXHCrAgJNLwHkj2Lka1B_kkNb0dNwiWiAN_Q@mail.gmail.com/

> My use case already requires using a reasonably complex API for
> interacting with a separate userland process for fetching memory, and
> it's really fast. I've never tried to hook userfaultfd into this other
> process, but I'm quite certain that [1] + this process's interface
> scale better than userfaultfd does. Perhaps userfaultfd, for
> not-so-scaled-up cases, could be *slightly* faster, but I mostly care
> about what happens when we scale to hundreds of vCPUs.
> 
> [1]: https://lore.kernel.org/kvm/20240215235405.368539-1-amoorthy@google.com/
Do I understand it right that in your setup, when an EPT violation occurs,
  - VMM shares the fault information with the other process via a 
userspace protocol
  - the process fetches the memory, installs it (?) and notifies VMM
  - VMM calls KVM run to resume execution
?
Would you be ok to share an outline of the API you mentioned?

>> How do you envision resolving faults in userspace? Copying the page in
>> (provided that userspace mapping of guest_memfd is supported [3]) and
>> clearing the KVM_MEMORY_ATTRIBUTE_USERFAULT alone do not look
>> sufficient to resolve the fault because an attempt to copy the page
>> directly in userspace will trigger a fault on its own
> 
> This is not true for KVM Userfault, at least for right now. Userspace
> accesses to guest memory will not trigger KVM Userfaults. (I know this
> name is terrible -- regular old userfaultfd() userfaults will indeed
> get triggered, provided you've set things up properly.)
> 
> KVM Userfault is merely meant to catch KVM's own accesses to guest
> memory (including vCPU accesses). For non-guest_memfd memslots,
> userspace can totally just write through the VMA it has made (KVM
> Userfault *cannot*, by virtue of being completely divorced from mm,
> intercept this access). For guest_memfd, userspace could write to
> guest memory through a VMA if that's where guest_memfd is headed, but
> perhaps it will rely on exact details of how userspace is meant to
> populate guest_memfd memory.
True, it isn't the case right now. I think I fast-forwarded to a state 
where notifications about VMM-triggered faults to the guest_memfd are 
also sent asynchronously.

> In case it's interesting or useful at all, we actually use
> UFFDIO_CONTINUE for our live migration use case. We mmap() memory
> twice -- one of them we register with userfaultfd and also give to
> KVM. The other one we use to install memory -- our non-faulting view
> of guest memory!
That is interesting. You're replacing UFFDIO_COPY (vma1) with a memcpy 
(vma2) + UFFDIO_CONTINUE (vma1), IIUC. Are both mappings created by the 
same process? What benefits does it bring?