Re: [PATCH] hw/misc: Add a virtual pci device to dynamically attach memory to QEMU

[David Hildenbrand <david@redhat.com>]

On 27.09.21 14:28, david.dai wrote:
> On Mon, Sep 27, 2021 at 11:07:43AM +0200, David Hildenbrand (david@redhat.com) wrote:
>>
>> CAUTION: This email originated from outside of the organization. Do not
>> click links or open attachments unless you recognize the sender and know the
>> content is safe.
>>
>>
>> On 27.09.21 10:27, Stefan Hajnoczi wrote:
>>> On Sun, Sep 26, 2021 at 10:16:14AM +0800, David Dai wrote:
>>>> Add a virtual pci to QEMU, the pci device is used to dynamically attach memory
>>>> to VM, so driver in guest can apply host memory in fly without virtualization
>>>> management software's help, such as libvirt/manager. The attached memory is
>>
>> We do have virtio-mem to dynamically attach memory to a VM. It could be
>> extended by a mechanism for the VM to request more/less memory, that's
>> already a planned feature. But yeah, virito-mem memory is exposed as
>> ordinary system RAM, not only via a BAR to mostly be managed by user space
>> completely.

There is a virtio-pmem spec proposal to expose the memory region via a 
PCI BAR. We could do something similar for virtio-mem, however, we would 
have to wire that new model up differently in QEMU (it's no longer a 
"memory device" like a DIMM then).

>>
> 
> I wish virtio-mem can solve our problem, but it is a dynamic allocation mechanism
> for system RAM in virtualization. In heterogeneous computing environments, the
> attached memory usually comes from computing device, it should be managed separately.
> we doesn't hope Linux MM controls it.

If that heterogeneous memory would have a dedicated node (which usually 
is the case IIRC) , and you let it manage by the Linux kernel 
(dax/kmem), you can bind the memory backend of virtio-mem to that 
special NUMA node. So all memory managed by that virtio-mem device would 
come from that heterogeneous memory.

You could then further use a separate NUMA node for that virtio-mem 
device inside the VM. But to the VM it would look like System memory 
with different performance characteristics. That would work fore some 
use cases I guess, but not sure for which not (I assume you can tell :) ).

We could even write an alternative virtio-mem mode, where device manage 
isn't exposed to the buddy but using some different way to user space.

>   
>>>> isolated from System RAM, it can be used in heterogeneous memory management for
>>>> virtualization. Multiple VMs dynamically share same computing device memory
>>>> without memory overcommit.
>>
>> This sounds a lot like MemExpand/MemLego ... am I right that this is the
>> original design? I recall that VMs share a memory region and dynamically
>> agree upon which part of the memory region a VM uses. I further recall that
>> there were malloc() hooks that would dynamically allocate such memory in
>> user space from the shared memory region.
>>
> 
> Thank you for telling me about Memexpand/MemLego, I have carefully read the paper.
> some ideas from it are same as this patch, such as software model and stack, but
> it may have a security risk that whole shared memory is visible to all VMs.

How will you make sure that not all shared memory can be accessed by the 
other VMs? IOW, emulate !shared memory on shared memory?

> -----------------------
>       application
> -----------------------
> memory management driver
> -----------------------
>       pci driver
> -----------------------
>     virtual pci device
> -----------------------
> 
>> I can see some use cases for it, although the shared memory design isn't
>> what you typically want in most VM environments.
>>
> 
> The original design for this patch is to share a computing device among multipile
> VMs. Each VM runs a computing application(for example, OpenCL application)
> Our computing device can support a few applications in parallel. In addition, it
> supports SVM(shared virtual memory) via IOMMU/ATS/PASID/PRI. Device exposes its
> memory to host vis PCIe bar or CXL.mem, host constructs memory pool to uniformly
> manage device memory, then attach device memory to VM via a virtual PCI device.

How exactly is that memory pool created/managed? Simply dax/kmem and 
handling it via the buddy in a special NUMA node.

> but we don't know how much memory should be assigned when creating VM, so we hope
> memory is attached to VM on-demand. driver in guest triggers memory attaching, but
> not outside virtualization management software. so the original requirements are:
> 1> The managed memory comes from device, it should be isolated from system RAM
> 2> The memory can be dynamically attached to VM in fly
> 3> The attached memory supports SVM and DMA operation with IOMMU
> 
> Thank you very much.

Thanks for the info. If virtio-mem is not applicable and cannot be 
modified for this use case, would it make sense to create a new virtio 
device type?


-- 
Thanks,

David / dhildenb