Re: [PATCH v2] Driver for Inter-VM shared memory device for KVM supporting interrupts.

["Christian Bornträger" <borntraeger@de.ibm.com>]

Am Mittwoch 20 Mai 2009 04:58:38 schrieb Rusty Russell:
> On Wed, 20 May 2009 02:21:08 am Cam Macdonell wrote:
> > Avi Kivity wrote:
> > > Christian Bornträger wrote:
> > >>> To summarize, Anthony thinks it should use virtio, while I believe
> > >>> virtio is useful for exporting guest memory, not for importing host
> > >>> memory.
>
> Yes, precisely.
>
> But what's it *for*, this shared memory?  Implementing shared memory is
> trivial.  Using it is harder.  For example, inter-guest networking: you'd
> have to copy packets in and out, making it slow as well as losing
> abstraction.
>
> The only interesting idea I can think of is exposing it to userspace, and
> having that run some protocol across it for fast app <-> app comms.  But if
> that's your plan, you still have a lot of code the write!
>
> So I guess I'm missing the big picture here?

I can give some insights about shared memory usage in z/VM. z/VM uses so-
called discontiguous saved segments (DCSS) to shared memory between guests.
	(naming side note:
	o discontigous because these segments can have holes and different access
      rights, e.g. you can build DCSS that go from 800M-801M read only and
      900M-910M exclusive-write.
	o segments because the 2nd level of our page tables is called segment table.
     )

z/VM uses these segments for several purposes:
o The monitoring subsystem uses a DCSS to get data from several components
o shared guest kernels: The CMS operating system is build as a bootable DCSS
  (called named-saved-segments NSS). All guests have the same host pages for
  the read-only parts of the CMS kernel. The local data is stored in
  exclusive-write parts of the same NSS. Linux on System z is also capable of
  using this feature (CONFIG_SHARED_KERNEL). The kernel linkage is changed in
  a way to separate the read-only text segment from the other parts with
  segment size alignment
o execute-in-place: This is a Linux feature to exploit the DCSS technology.
  The goal is to shared identical guest pages without the additional overhead
  of KSM etc. We have a block device driver for DCSS. This block device driver
  supports the direct_access function and therefore allows to use the xip
  option of ext2. The idea is to put  binaries into an read-only ext2
  filesystem. Whenever an mmap is made on this file system, the page is not
  mapped into the page cache. The ptes point into the DCSS memory instead.
  Since the DCSS is demand-paged by the host no memory is wasted for unused
  parts of the binaries. In case of COW the page is copied as usual. It turned
  out that installations with many similar guests (lets say 400 guests) will
  profit in terms of memory saving and quicker application startups (not the
  first guest of course). There is a downside: this requires a skilled
  administrator to setup.

We have also experimented with network, Posix shared memory, and shared caches 
via DCSS. Most of these ideas turned out to be not very useful or hard to 
implement proper.