Re: [PATCH] Add shared memory PCI device that shares a memory object betweens VMs

[Cam Macdonell <cam@cs.ualberta.ca>]

Avi Kivity wrote:
> Cam Macdonell wrote:
>> I think there is value for static memory sharing.   It can be used for 
>> fast, simple synchronization and communication between guests (and the 
>> host) that use need to share data that needs to be updated frequently 
>> (such as a simple cache or notification system).  It may not be a 
>> common task, but I think static sharing has its place and that's what 
>> this device is for at this point.
> 
> It would be good to detail a use case for reference.

I'll try my best...

We are using the (static) shared memory region for fast, interprocess 
communications (IPC).  Of course, shared-memory IPC is an old idea, and 
the IPC here is actually between VMs (i.e., ivshmem), not processes 
inside a single VM.  But, fast IPC is useful for shared caches, OS 
bypass (guest-to-guest, and host-to-guest), and low-latency IPC use-cases.

For example, one use of ivshmem is as as a file cache between VMs.  Note 
that, unlike stream-oriented IPC, this file cache can be shared between, 
say, four VMs simultaneously.  In using VMs as sandboxes for distributed 
computing (condor, cloud, etc.), if two (or more) VMs are co-located on 
the same server, they can effectively share a single, unified cache. 
Any VM can bring in the data, and other VMs can use it.  Otherwise, two 
VMs might transfer (over the WAN, in the worst case, as in a cloud) and 
buffer cache the same file in multiple VMs.  In some ways, the 
configuration would look like an in-memory cluster file system, but 
instead of shared disks, we have shared memory.

Alternative forms of file sharing between VMs (e.g., via SAMBA or NFS) 
are possible, but also results in multiple cached copies of the same 
file data on the same physical server.  Furthermore, ivshmem has the 
(usual, planned) latency (e.g., for file metadata stats) and bandwidth 
advantages between most forms of stream-oriented IPC for file sharing 
protocols.

Other (related) use cases include bulk-data movement between the host 
and guest VMs, due to the OS bypass properties of the  ivshmem.  Since 
static shared memory shares a file (or memory object) on the host, 
host-guest sharing is simpler than with dynamic shared memory.

We acknowledge that work has to be done with thread/process scheduling 
to truly gain low IPC latency; that is to come, possibly with
PCI interrupts.  And, as the VMware experience shows (see below), VM 
migration *is* affected by ivshmem, but we think a good (but 
non-trivial) attach-to-ivshmem and detach-from-ivshmem protocol (in the 
future) can mostly address that issue.

As an aside, VMware ditched shared memory as part of their VMCI 
interface.  We emailed with some of their people who suggested to use 
sockets since shared memory "de-virtualizes" the VM (i.e. it breaks 
migration).  But on their forums there were users that used shared 
memory for their work and were disappointed to see it go.  One person I 
emailed with used shared memory for simulations running across VMs. 
Using shared memory freed him from having to come up with a protocol to 
exchange updates and having a central VM responsible for receiving and 
broadcasting updates.  When he did try to use a socket-based approach, 
the performance dropped substantially due to the communication overhead.

>>> Then you need a side channel to communicate the information to the 
>>> guest.
>>
>> Couldn't one of the registers in BAR0 be used to store the actual 
>> (non-power-of-two) size?
> 
> The PCI config space (where the BARs reside) is a good place for it.  
> Registers 0x40+ are device specific IIRC.
> 

Ok.

Cam