Re: [PATCH] hw/misc: Add a virtual pci device to dynamically attach memory to QEMU

["david.dai" <david.dai@montage-tech.com>]

On Mon, Sep 27, 2021 at 11:07:43AM +0200, David Hildenbrand (david@redhat.com) wrote:
> 
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know the
> content is safe.
> 
> 
> On 27.09.21 10:27, Stefan Hajnoczi wrote:
> > On Sun, Sep 26, 2021 at 10:16:14AM +0800, David Dai wrote:
> > > Add a virtual pci to QEMU, the pci device is used to dynamically attach memory
> > > to VM, so driver in guest can apply host memory in fly without virtualization
> > > management software's help, such as libvirt/manager. The attached memory is
> 
> We do have virtio-mem to dynamically attach memory to a VM. It could be
> extended by a mechanism for the VM to request more/less memory, that's
> already a planned feature. But yeah, virito-mem memory is exposed as
> ordinary system RAM, not only via a BAR to mostly be managed by user space
> completely.
>

I wish virtio-mem can solve our problem, but it is a dynamic allocation mechanism
for system RAM in virtualization. In heterogeneous computing environments, the
attached memory usually comes from computing device, it should be managed separately.
we doesn't hope Linux MM controls it.
 
> > > isolated from System RAM, it can be used in heterogeneous memory management for
> > > virtualization. Multiple VMs dynamically share same computing device memory
> > > without memory overcommit.
> 
> This sounds a lot like MemExpand/MemLego ... am I right that this is the
> original design? I recall that VMs share a memory region and dynamically
> agree upon which part of the memory region a VM uses. I further recall that
> there were malloc() hooks that would dynamically allocate such memory in
> user space from the shared memory region.
>

Thank you for telling me about Memexpand/MemLego, I have carefully read the paper.
some ideas from it are same as this patch, such as software model and stack, but
it may have a security risk that whole shared memory is visible to all VMs.
-----------------------
     application
-----------------------
memory management driver
-----------------------
     pci driver
-----------------------
   virtual pci device
-----------------------

> I can see some use cases for it, although the shared memory design isn't
> what you typically want in most VM environments.
>

The original design for this patch is to share a computing device among multipile
VMs. Each VM runs a computing application(for example, OpenCL application)
Our computing device can support a few applications in parallel. In addition, it
supports SVM(shared virtual memory) via IOMMU/ATS/PASID/PRI. Device exposes its
memory to host vis PCIe bar or CXL.mem, host constructs memory pool to uniformly
manage device memory, then attach device memory to VM via a virtual PCI device.
but we don't know how much memory should be assigned when creating VM, so we hope
memory is attached to VM on-demand. driver in guest triggers memory attaching, but
not outside virtualization management software. so the original requirements are:
1> The managed memory comes from device, it should be isolated from system RAM
2> The memory can be dynamically attached to VM in fly
3> The attached memory supports SVM and DMA operation with IOMMU

Thank you very much. 


Best Regards,
David Dai

> -- 
> Thanks,
> 
> David / dhildenb
> 
>