Re: [Qemu-devel] KVM "fake DAX" flushing interface - discussion

[Haozhong Zhang <haozhong.zhang@intel.com>]

On 07/21/17 02:56 -0400, Pankaj Gupta wrote:
> 
> Hello,
> 
> We shared a proposal for 'KVM fake DAX flushing interface'.
> 
> https://lists.gnu.org/archive/html/qemu-devel/2017-05/msg02478.html
>

In above link,
  "Overall goal of project 
   is to increase the number of virtual machines that can be 
   run on a physical machine, in order to *increase the density*
   of customer virtual machines"

Is the fake persistent memory used as normal RAM in guest? If no, how
is it expected to be used in guest?

> We did initial POC in which we used 'virtio-blk' device to perform 
> a device flush on pmem fsync on ext4 filesystem. They are few hacks 
> to make things work. We need suggestions on below points before we 
> start actual implementation.
>
> A] Problems to solve:
> ------------------
> 
> 1] We are considering two approaches for 'fake DAX flushing interface'.
>     
>  1.1] fake dax with NVDIMM flush hints & KVM async page fault
> 
>      - Existing interface.
> 
>      - The approach to use flush hint address is already nacked upstream.
> 
>      - Flush hint not queued interface for flushing. Applications might 
>        avoid to use it.
> 
>      - Flush hint address traps from guest to host and do an entire fsync 
>        on backing file which itself is costly.
> 
>      - Can be used to flush specific pages on host backing disk. We can 
>        send data(pages information) equal to cache-line size(limitation) 
>        and tell host to sync corresponding pages instead of entire disk sync.
> 
>      - This will be an asynchronous operation and vCPU control is returned 
>        quickly.
> 
> 
>  1.2] Using additional para virt device in addition to pmem device(fake dax with device flush)
> 
>      - New interface
> 
>      - Guest maintains information of DAX dirty pages as exceptional entries in 
>        radix tree.
> 
>      - If we want to flush specific pages from guest to host, we need to send 
>        list of the dirty pages corresponding to file on which we are doing fsync.
> 
>      - This will require implementation of new interface, a new paravirt device 
>        for sending flush requests.
> 
>      - Host side will perform fsync/fdatasync on list of dirty pages or entire 
>        block device backed file.
> 
> 2] Questions:
> -----------
> 
>  2.1] Not sure why WPQ flush is not a queued interface? We can force applications 
>       to call this? device DAX neither calls fsync/msync?
> 
>  2.2] Depending upon interface we decide, we need optimal solution to sync 
>       range of pages?
> 
>      - Send range of pages from guest to host to sync asynchronously instead 
>        of syncing entire block device?

e.g. a new virtio device to deliver sync requests to host?

> 
>      - Other option is to sync entire disk backing file to make sure all the 
>        writes are persistent. In our case, backing file is a regular file on 
>        non NVDIMM device so host page cache has list of dirty pages which
>        can be used either with fsync or similar interface.

As the amount of dirty pages can be variant, the latency of each host
fsync is likely to vary in a large range.

> 
>  2.3] If we do host fsync on entire disk we will be flushing all the dirty data
>       to backend file. Just thinking what would be better approach, flushing 
>       pages on corresponding guest file fsync or entire block device?
> 
>  2.4] If we decide to choose one of the above approaches, we need to consider 
>       all DAX supporting filesystems(ext4/xfs). Would hooking code to corresponding
>       fsync code of fs seems reasonable? Just thinking for flush hint address use-case?
>       Or how flush hint addresses would be invoked with fsync or similar api?
> 
>  2.5] Also with filesystem journalling and other mount options like barriers, 
>       ordered etc, how we decide to use page flush hint or regular fsync on file?
>  
>  2.6] If at guest side we have PFN of all the dirty pages in radixtree? and we send 
>       these to to host? At host side would we able to find corresponding page and flush 
>       them all?

That may require the host file system provides API to flush specified
blocks/extents and their meta data in the file system. I'm not
familiar with this part and don't know whether such API exists.

Haozhong