Re: [Qemu-devel] [PATCHv3] QEMU(upstream): Disable xen's use of O_DIRECT by default as it results in crashes.

[Alex Bligh <alex@alex.org.uk>]

Paolo,

--On 18 March 2013 15:49:48 +0100 Paolo Bonzini <pbonzini@redhat.com> wrote:

>>> The problem is that the target's page cache might host image data from a
>>> previous run.  If you do not use O_DIRECT, it will not see the changes
>>> made on the source.
>>
>> I was under the impression that with cache=writeback, qemu doesn't
>> use O_DIRECT, in which case why isn't there the same danger under
>> kvm, i.e. that the target page cache contains data from a previous
>> run?
>
> KVM in fact only supports migration using cache=none.  This does not
> apply of course if you're using cache-coherent storage, such as rbd or
> gluster; or if you're using one of the userspace backends that bypass
> the page cache, like NBD or libiscsi.

Well right now emulated devices under xen with qemu-upstream
dm do not use O_DIRECT. So that's going to hit this problem anyway.

Let me understand the problem a little more. Let's say a VM has a
disk backed by NFS. It runs on node A, at which point pages are
introduced to the page cache. It then migrates to node B, with
node A flushing its dirty pages to disk before the migration
completed, but not emptying the page cache completely.
It's then migrated back to node A in which case I think you are
saying that node A may still hold the page cache entries from
prior to the first migration.

However, node A has closed and reopened the file. Doesn't NFSv4
notice that the file has changed at this point and invalidate
all page cache entries? Else (re)opening a file on one NFS client
that has been changed on another would never work. I know this
is unpredictable if you haven't closed the file, but in this
instance A has closed the file.

>> Disabling migration seems a bit excessive when migration isn't disabled
>> with cache=unsafe (AFAIK)
>
> It is not QEMU's task.  There are cases where the cache= options are
> unnecessary or ignored.  But indeed libvirt warns (or blocks, I don't
> remember) in this case.

Well we are kvm user too, and I understand hat kvm (as opposed to
libvirt) does not error or warn if you live migrate with cache=writeback.

I've no problem if xl or libvirt or whatever error or warn. My usage
is API based, rather than xl / libvirt based.

> If libxl does migration without O_DIRECT, then that's a bug in libxl.
> What about blkback?  IIRC it uses bios, so it also bypasses the page
> cache.

Possibly a bug in xl rather than libxl, but as no emulated devices
use O_DIRECT, that bug is already there, and isn't in QEMU.

>> Stefano did ack the patch, and for a one line change it's been
>> through a pretty extensive discussion on xen-devel ...
>
> It may be a one-line change, but it completely changes the paths that
> I/O goes through.  Apparently the discussion was not enough.

What would you suggest?

-- 
Alex Bligh