Re: Virtio in Xen on Arm (based on IOREQ concept)

["Roger Pau Monné" <roger.pau@citrix.com>]

On Tue, Jul 21, 2020 at 10:12:40PM +0100, Julien Grall wrote:
> Hi Oleksandr,
> 
> On 21/07/2020 19:16, Oleksandr wrote:
> > 
> > On 21.07.20 17:27, Julien Grall wrote:
> > > On a similar topic, I am a bit surprised you didn't encounter memory
> > > exhaustion when trying to use virtio. Because on how Linux currently
> > > works (see XSA-300), the backend domain as to have a least as much
> > > RAM as the domain it serves. For instance, you have serve two
> > > domains with 1GB of RAM each, then your backend would need at least
> > > 2GB + some for its own purpose.
> > > 
> > > This probably wants to be resolved by allowing foreign mapping to be
> > > "paging" out as you would for memory assigned to a userspace.
> > 
> > Didn't notice the last sentence initially. Could you please explain your
> > idea in detail if possible. Does it mean if implemented it would be
> > feasible to map all guest memory regardless of how much memory the guest
> > has?
> >
> > Avoiding map/unmap memory each guest request would allow us to have
> > better performance (of course with taking care of the fact that guest
> > memory layout could be changed)...
> 
> I will explain that below. Before let me comment on KVM first.
> 
> > Actually what I understand looking at kvmtool is the fact it does not
> > map/unmap memory dynamically, just calculate virt addresses according to
> > the gfn provided.
> 
> The memory management between KVM and Xen is quite different. In the case of
> KVM, the guest RAM is effectively memory from the userspace (allocated via
> mmap) and then shared with the guest.
> 
> From the userspace PoV, the guest memory will always be accessible from the
> same virtual region. However, behind the scene, the pages may not always
> reside in memory. They are basically managed the same way as "normal"
> userspace memory.
> 
> In the case of Xen, we are basically stealing a guest physical page
> allocated via kmalloc() and provide no facilities for Linux to reclaim the
> page if it needs to do it before the userspace decide to unmap the foreign
> mapping.
> 
> I think it would be good to handle the foreing mapping the same way as
> userspace memory. By that I mean, that Linux could reclaim the physical page
> used by the foreing mapping if it needs to.
> 
> The process for reclaiming the page would look like:
>     1) Unmap the foreign page
>     2) Ballon in the backend domain physical address used by the foreing
> mapping (allocate the page in the physmap)
> 
> The next time the userspace is trying to access the foreign page, Linux will
> receive a data abort that would result to:
>     1) Allocate a backend domain physical page
>     2) Balloon out the physical address (remove the page from the physmap)
>     3) Map the foreing mapping at the new guest physical address
>     4) Map the guest physical page in the userspace address space

This is going to shatter all the super pages in the stage-2
translation.

> With this approach, we should be able to have backend domain that can handle
> frontend domain without require a lot of memory.

Linux on x86 has the option to use empty hotplug memory ranges to map
foreign memory: the balloon driver hotplugs an unpopulated physical
memory range that's not made available to the OS free memory allocator
and it's just used as scratch space to map foreign memory. Not sure
whether Arm has something similar, or if it could be implemented.

You can still use the map-on-fault behaviour as above, but I would
recommend that you try to limit the number of hypercalls issued.
Having to issue a single hypercall for each page fault it's going to
be slow, so I would instead use mmap batch to map the hole range in
unpopulated physical memory and then the OS fault handler just needs to
fill the page tables with the corresponding address.

Roger.