Re: KVM "fake DAX" device flushing

Stefan Hajnoczi
2017-05-11T18:17:05Z

[-- Attachment #1: Type: text/plain, Size: 7490 bytes --]

On Wed, May 10, 2017 at 09:26:00PM +0530, Pankaj Gupta wrote:
> We are sharing initial project proposal for
> 'KVM "fake DAX" device flushing' project for feedback.
> Got the idea during discussion with 'Rik van Riel'.

CCing NVDIMM folks.

>
> Also, request answers to 'Questions' section.
>
> Abstract :
> ----------
> Project idea is to use fake persistent memory with direct
> access(DAX) in virtual machines. Overall goal of project
> is to increase the number of virtual machines that can be
> run on a physical machine, in order to increase the density
> of customer virtual machines.
>
> The idea is to avoid the guest page cache, and minimize the
> memory footprint of virtual machines. By presenting a disk
> image as a nvdimm direct access (DAX) memory region in a
> virtual machine, the guest OS can avoid using page cache
> memory for most file accesses.
>
> Problem Statement :
> ------------------
> * Guest uses page cache in memory to process fast requests
> for disk read/write. This results in big memory footprint
> of guests without host knowing much details of the guest
> memory.
>
> * If guests use direct access(DAX) with fake persistent
> storage, the host manages the page cache for guests,
> allowing the host to easily reclaim/evict less frequently
> used page cache pages without requiring guest cooperation,
> like ballooning would.
>
> * Host manages guest cache as ‘mmaped’ disk image area in
> qemu address space. This region is passed to guest as fake
> persistent memory range. We need a new flushing interface
> to flush this cache to secondary storage to persist guest
> writes.
>
> * New asynchronous flushing interface will allow guests to
> cause the host flush the dirty data to backup storage file.
> Systems with pmem storage make use of CLFLUSH instruction
> to flush single cache line to persistent storage and it
> takes care of flushing. With fake persistent storage in
> guest we cannot depend on CLFLUSH instruction to flush entire
> dirty cache to backing storage. Even If we trap and emulate
> CLFLUSH instruction guest vCPU has to wait till we flush all
> the dirty memory. Instead of this we need to implement a new
> asynchronous guest flushing interface, which allows the guest
> to specify a larger range to be flushed at once, and allows
> the vCPU to run something else while the data is being synced
> to disk.
>
> * New flushing interface will consists of a para virt driver to
> new fake nvdimm like device which will process guest flushing
> requests like fsync/msync etc instead of pmem library calls
> like clflush. The corresponding device at host side will be
> responsible for flushing requests for guest dirty pages.
> Guest can put current task in sleep and vCPU can run any other
> task while host side flushing of guests pages is in progress.
>
> Host controlled fake nvdimm DAX to avoid guest page cache :
> -------------------------------------------------------------
> * Bypass guest page cache by using a fake persistent storage
> like nvdimm & DAX. Guest Read/Write is directly done on
> fake persistent storage without involving guest kernel for
> caching data.
>
> * Fake nvdimm device passed to guest is backed by a regular
> file in host stored in secondary storage.
>
> * Qemu has implementation of fake NVDIMM/DAX device. Use this
> capability of passing regular host file(disk) as nvdimm device
> to guest.
>
> * Nvdimm with DAX works for ext4/xfs filesystem. Supported
> filesystem should be DAX compatible.
>
> * As we are using guest disk as fake DAX/NVDIMM device, we
> need a mechanism for persistence of data backed on regular
> host storage file.
>
> * For live migration use case, if host side backing file is
> shared storage, we need to flush the page cache for the disk
> image at the destination (new fadvise interface, FADV_INVALIDATE_CACHE?)
> before starting execution of the guest on the destination host.

Good point. QEMU currently only supports live migration with O_DIRECT.
I think the problem was that userspace cannot guarantee consistency in
the general case. If you find a solution to this problem for fake
NVDIMM then maybe the QEMU block layer can also begin supporting live
migration with buffered I/O.

>
> Design :
> ---------
> * In order to not have page cache inside the guest, qemu would:
>
> 1) mmap the guest's disk image and present that disk image to
> the guest as a persistent memory range.
>
> 2) Present information to the guest telling it that the persistent
> memory range is not physical persistent memory.

Steps 1 & 2 are already supported by QEMU NVDIMM emulation today.

> 3) Present an additional paravirt device alongside the persistent
> memory range, that can be used to sync (ranges of) data to disk.
>
> * Guest would use the disk image mostly like a persistent memory
> device, with two exceptions:
>
> 1) It would not tell userspace that the files on that device are
> persistent memory. This is done so userspace knows to call
> fsync/msync, instead of the pmem clflush library call.

Not sure I agree with hiding the nvdimm nature of the device. Instead I
think you need to build this capability into the Linux nvdimm code.
libpmem will detect these types of devices and issue fsync/msync when
the application wants to flush.

> 2) When userspace calls fsync/msync on files on the fake persistent
> memory device, issue a request through the paravirt device that
> causes the host to flush the device back end.
>
> * Guest uses fake persistent storage data updates can be still in
> qemu memory. We need a way to flush cached data in host to backed

s/qemu memory/host memory/

I guess you mean that host userspace needs a way to reliably flush an
address range to the underlying storage.

> secondary storage.
>
> * Once the guest receives a completion event from the host, it will
> allow userspace programs that were waiting on the fsync/msync to
> continue running.
>
> * Host is responsible for paging in pages in host backing area for
> guest persistent memory as they are accessed by the guest, and
> for evicting pages as host memory fills up.
>
> Questions :
> -----------
> * What should the flushing interface between guest and host look
> like?

A simple hack for prototyping is to instantiate an virtio-blk-pci for
the mmapped host file. The guest can send flush commands on the
virtio-blk-pci device but will otherwise use the mapped memory directly.

> * Any suggestions to hook the IO caching code with KVM/Qemu or
> thoughts on how we should do it?
>
> * Thinking of implementing a guest para virt driver which will send
> guest requests to Qemu to flush data to disk. Not sure at this
> point how to tell userspace to work on this device as any regular
> device without considering it as persistent device. Any suggestions
> on this?
>
> * Not thought yet about ballooning impact. But feel this solution
> could be better than ballooning in long term? As we will be
> managing all guests cache from host side.
>
> * Not sure this solution works for ARM and other architectures and
> Windows?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]