Re: [PATCH] hw/misc: Add a virtual pci device to dynamically attach memory to QEMU

[David Hildenbrand <david@redhat.com>]

>> virito-mem currently relies on having a single sparse memory region (anon
>> mmap, mmaped file, mmaped huge pages, mmap shmem) per VM. Although we can
>> share memory with other processes, sharing with other VMs is not intended.
>> Instead of actually mmaping parts dynamically (which can be quite
>> expensive), virtio-mem relies on punching holes into the backend and
>> dynamically allocating memory/file blocks/... on access.
>>
>> So the easy way to make it work is:
>>
>> a) Exposing the CXL memory to the buddy via dax/kmem, esulting in device
>> memory getting managed by the buddy on a separate NUMA node.
>>
> 
> Linux kernel buddy system? how to guarantee other applications don't apply memory
> from it

Excellent question. Usually, you would online the memory to ZONE_MOVABLE,
such that even if some other allocation ended up there, that it could
get migrated somewhere else.

For example, "daxctl reconfigure-device" tries doing that as default:

https://pmem.io/ndctl/daxctl-reconfigure-device.html

However, I agree that we might actually want to tell the system to not
use this CPU-less node as fallback for other allocations, and that we
might not want to swap out such memory etc.


But, in the end all that virtio-mem needs to work in the hypervisor is

a) A sparse memmap (anonymous RAM, memfd, file)
b) A way to populate memory within that sparse memmap (e.g., on fault,
    using madvise(MADV_POPULATE_WRITE), fallocate())
c) A way to discard memory (madvise(MADV_DONTNEED),
    fallocate(FALLOC_FL_PUNCH_HOLE))

So instead of using anonymous memory+mbind, you can also mmap a sparse file
and rely on populate-on-demand. One alternative for your use case would be
to create a DAX  filesystem on that CXL memory (IIRC that should work) and
simply providing virtio-mem with a sparse file located on that filesystem.

Of course, you can also use some other mechanism as you might have in
your approach, as long as it supports a,b,c.

> 
>>
>> b) (optional) allocate huge pages on that separate NUMA node.
>> c) Use ordinary memory-device-ram or memory-device-memfd (for huge pages),
>> *bidning* the memory backend to that special NUMA node.
>>
>   
> "-object memory-backend/device-ram or memory-device-memfd, id=mem0, size=768G"
> How to bind backend memory to NUMA node
> 

I think the syntax is "policy=bind,host-nodes=X"

whereby X is a node mask. So for node "0" you'd use "host-nodes=0x1" for "5"
"host-nodes=0x20" etc.

>>
>> This will dynamically allocate memory from that special NUMA node, resulting
>> in the virtio-mem device completely being backed by that device memory,
>> being able to dynamically resize the memory allocation.
>>
>>
>> Exposing an actual devdax to the virtio-mem device, shared by multiple VMs
>> isn't really what we want and won't work without major design changes. Also,
>> I'm not so sure it's a very clean design: exposing memory belonging to other
>> VMs to unrelated QEMU processes. This sounds like a serious security hole:
>> if you managed to escalate to the QEMU process from inside the VM, you can
>> access unrelated VM memory quite happily. You want an abstraction
>> in-between, that makes sure each VM/QEMU process only sees private memory:
>> for example, the buddy via dax/kmem.
>>
> Hi David
> Thanks for your suggestion, also sorry for my delayed reply due to my long vacation.
> How does current virtio-mem dynamically attach memory to guest, via page fault?

Essentially you have a large sparse mmap. Withing that mmap, memory is
populated on demand. Instead if mmap/munmap you perform a single large
mmap and then dynamically populate memory/discard memory.

Right now, memory is populated via page faults on access. This is
sub-optimal when dealing with limited resources (i.e., hugetlbfs,
file blocks) and you might run out of backend memory.

I'm working on a "prealloc" mode, which will preallocate/populate memory
necessary for exposing the next block of memory to the VM, and which
fails gracefully if preallocation/population fails in the case of such
limited resources.

The patch resides on:
	https://github.com/davidhildenbrand/qemu/tree/virtio-mem-next

commit ded0e302c14ae1b68bdce9059dcca344e0a5f5f0
Author: David Hildenbrand <david@redhat.com>
Date:   Mon Aug 2 19:51:36 2021 +0200

     virtio-mem: support "prealloc=on" option
     
     Especially for hugetlb, but also for file-based memory backends, we'd
     like to be able to prealloc memory, especially to make user errors less
     severe: crashing the VM when there are not sufficient huge pages around.
     
     A common option for hugetlb will be using "reserve=off,prealloc=off" for
     the memory backend and "prealloc=on" for the virtio-mem device. This
     way, no huge pages will be reserved for the process, but we can recover
     if there are no actual huge pages when plugging memory.
     
     Signed-off-by: David Hildenbrand <david@redhat.com>


-- 
Thanks,

David / dhildenb