Re: [Qemu-devel] KVM "fake DAX" device flushing

Pankaj Gupta
2017-05-12T06:56:22Z

>
> On Wed, May 10, 2017 at 09:26:00PM +0530, Pankaj Gupta wrote:
> > We are sharing initial project proposal for
> > 'KVM "fake DAX" device flushing' project for feedback.
> > Got the idea during discussion with 'Rik van Riel'.
>
> CCing NVDIMM folks.
>
> >
> > Also, request answers to 'Questions' section.
> >
> > Abstract :
> > ----------
> > Project idea is to use fake persistent memory with direct
> > access(DAX) in virtual machines. Overall goal of project
> > is to increase the number of virtual machines that can be
> > run on a physical machine, in order to increase the density
> > of customer virtual machines.
> >
> > The idea is to avoid the guest page cache, and minimize the
> > memory footprint of virtual machines. By presenting a disk
> > image as a nvdimm direct access (DAX) memory region in a
> > virtual machine, the guest OS can avoid using page cache
> > memory for most file accesses.
> >
> > Problem Statement :
> > ------------------
> > * Guest uses page cache in memory to process fast requests
> > for disk read/write. This results in big memory footprint
> > of guests without host knowing much details of the guest
> > memory.
> >
> > * If guests use direct access(DAX) with fake persistent
> > storage, the host manages the page cache for guests,
> > allowing the host to easily reclaim/evict less frequently
> > used page cache pages without requiring guest cooperation,
> > like ballooning would.
> >
> > * Host manages guest cache as ‘mmaped’ disk image area in
> > qemu address space. This region is passed to guest as fake
> > persistent memory range. We need a new flushing interface
> > to flush this cache to secondary storage to persist guest
> > writes.
> >
> > * New asynchronous flushing interface will allow guests to
> > cause the host flush the dirty data to backup storage file.
> > Systems with pmem storage make use of CLFLUSH instruction
> > to flush single cache line to persistent storage and it
> > takes care of flushing. With fake persistent storage in
> > guest we cannot depend on CLFLUSH instruction to flush entire
> > dirty cache to backing storage. Even If we trap and emulate
> > CLFLUSH instruction guest vCPU has to wait till we flush all
> > the dirty memory. Instead of this we need to implement a new
> > asynchronous guest flushing interface, which allows the guest
> > to specify a larger range to be flushed at once, and allows
> > the vCPU to run something else while the data is being synced
> > to disk.
> >
> > * New flushing interface will consists of a para virt driver to
> > new fake nvdimm like device which will process guest flushing
> > requests like fsync/msync etc instead of pmem library calls
> > like clflush. The corresponding device at host side will be
> > responsible for flushing requests for guest dirty pages.
> > Guest can put current task in sleep and vCPU can run any other
> > task while host side flushing of guests pages is in progress.
> >
> > Host controlled fake nvdimm DAX to avoid guest page cache :
> > -------------------------------------------------------------
> > * Bypass guest page cache by using a fake persistent storage
> > like nvdimm & DAX. Guest Read/Write is directly done on
> > fake persistent storage without involving guest kernel for
> > caching data.
> >
> > * Fake nvdimm device passed to guest is backed by a regular
> > file in host stored in secondary storage.
> >
> > * Qemu has implementation of fake NVDIMM/DAX device. Use this
> > capability of passing regular host file(disk) as nvdimm device
> > to guest.
> >
> > * Nvdimm with DAX works for ext4/xfs filesystem. Supported
> > filesystem should be DAX compatible.
> >
> > * As we are using guest disk as fake DAX/NVDIMM device, we
> > need a mechanism for persistence of data backed on regular
> > host storage file.
> >
> > * For live migration use case, if host side backing file is
> > shared storage, we need to flush the page cache for the disk
> > image at the destination (new fadvise interface, FADV_INVALIDATE_CACHE?)
> > before starting execution of the guest on the destination host.
>
> Good point. QEMU currently only supports live migration with O_DIRECT.
> I think the problem was that userspace cannot guarantee consistency in
> the general case. If you find a solution to this problem for fake
> NVDIMM then maybe the QEMU block layer can also begin supporting live
> migration with buffered I/O.
>
> >
> > Design :
> > ---------
> > * In order to not have page cache inside the guest, qemu would:
> >
> > 1) mmap the guest's disk image and present that disk image to
> > the guest as a persistent memory range.
> >
> > 2) Present information to the guest telling it that the persistent
> > memory range is not physical persistent memory.
>
> Steps 1 & 2 are already supported by QEMU NVDIMM emulation today.

Yes. I have also tested guest 'fake DAX' device using QEMU NVDIMM emulation.
>
> > 3) Present an additional paravirt device alongside the persistent
> > memory range, that can be used to sync (ranges of) data to disk.
> >
> > * Guest would use the disk image mostly like a persistent memory
> > device, with two exceptions:
> >
> > 1) It would not tell userspace that the files on that device are
> > persistent memory. This is done so userspace knows to call
> > fsync/msync, instead of the pmem clflush library call.
>
> Not sure I agree with hiding the nvdimm nature of the device. Instead I
> think you need to build this capability into the Linux nvdimm code.
> libpmem will detect these types of devices and issue fsync/msync when
> the application wants to flush.
>
> > 2) When userspace calls fsync/msync on files on the fake persistent
> > memory device, issue a request through the paravirt device that
> > causes the host to flush the device back end.
> >
> > * Guest uses fake persistent storage data updates can be still in
> > qemu memory. We need a way to flush cached data in host to backed
>
> s/qemu memory/host memory/
>
> I guess you mean that host userspace needs a way to reliably flush an
> address range to the underlying storage.

right.
>
> > secondary storage.
> >
> > * Once the guest receives a completion event from the host, it will
> > allow userspace programs that were waiting on the fsync/msync to
> > continue running.
> >
> > * Host is responsible for paging in pages in host backing area for
> > guest persistent memory as they are accessed by the guest, and
> > for evicting pages as host memory fills up.
> >
> > Questions :
> > -----------
> > * What should the flushing interface between guest and host look
> > like?
>
> A simple hack for prototyping is to instantiate an virtio-blk-pci for
> the mmapped host file. The guest can send flush commands on the
> virtio-blk-pci device but will otherwise use the mapped memory directly.

okay. I will check this.
>
> > * Any suggestions to hook the IO caching code with KVM/Qemu or
> > thoughts on how we should do it?
> >
> > * Thinking of implementing a guest para virt driver which will send
> > guest requests to Qemu to flush data to disk. Not sure at this
> > point how to tell userspace to work on this device as any regular
> > device without considering it as persistent device. Any suggestions
> > on this?
> >
> > * Not thought yet about ballooning impact. But feel this solution
> > could be better than ballooning in long term? As we will be
> > managing all guests cache from host side.
> >
> > * Not sure this solution works for ARM and other architectures and
> > Windows?
>