Re: [Qemu-devel] Re: [PATCH][v2] Align file accesses with cache=off (O_DIRECT)

[Avi Kivity <avi@qumranet.com>]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Anthony Liguori wrote:
>>>
>>> "cached" is not a terribly accurate term.  O_DIRECT avoids the host 
>>> page cache but it doesn't guarantee that the disk is using 
>>> write-through.  For that, you need to use hdparm.
>>>
>>> O_SYNC basically turns the host page cache into a write-through 
>>> cache.  In terms of data integrity, the only question that matters 
>>> is whether you're misleading the guest into thinking data is on the 
>>> disk when it isn't.  Both O_DIRECT and O_SYNC accomplish this.
>>>
>>> If you just are concerned with data integrity, O_SYNC is probably 
>>> better because you get the benefits of host caching.  O_DIRECT is 
>>> really for circumstances where you know that using the host page 
>>> cache is going to reduce performance.
>>
>> In one specific circumstance O_SYNC has data integrity problems: 
>> shared disks with guests running on different hosts (or even a guest 
>> on one host, sharing a disk with another host).  In these cases, two 
>> reads can return different values without an intervening write.
>
> Are you assuming the underlying disk sharing protocol does not keep 
> the page cache coherent?
>

I was assuming access to a raw partition.  But yes, with a cluster file 
system this objection goes away.

At a significant cost, though.  You're now running two cache coherency 
protocols on top of each other.

>> In the general case, O_DIRECT gives better performance.  It avoids 
>> copying from the host pagecache to guest memory, and if you have 
>> spare memory to benefit from caching, give it to the guest; the 
>> nearer to the data consumer the cache is, the faster it performs.
>
> This assumes the only thing you're running on the machine is VMs.  If 
> you're just running one VM on your desktop, it is probably preferable 
> to go through the host page cache.  In particular, the host page cache 
> can be automatically aged and adjusted whereas, in general, you cannot 
> reduce the guests page cache size.

Agreed.  For casual uses, O_DIRECT is overkill.  It does get rid of the 
data copies, though.

>
>> In one specific case O_SYNC (or regular read/write cached operation) 
>> is better, rebooting the same guest over and over, as the guest cache 
>> is flushed on reboot.  Not a very interesting case, but unfortunately 
>> one that is very visible.
>>
>> (if you have a backing file for a COW disk, then opening the backing 
>> file without O_DIRECT may be a good idea too, as the file can be 
>> shared among many guests).
>
> FWIW, we really only need to use O_SYNC when the guest has disabled 
> write-back.  I think we should do that unconditionally too as it's an 
> issue of correctness. 

If the guest is used for non critical applications (like testing distro 
installers), then it's just a slowdown.

Even if the guest did not disable disk writeback, pagecache and disk 
write caches have vastly different characteristics, so I think we should 
set O_SYNC there as well.

Here's a summary of the use cases I saw so far:

- casual use, no critical data: write back cache

- backing file shared among many guests: read-only, cached

- desktop system, but don't lose my data: O_SYNC
(significant resources on the host)

- dedicated virtualization engine: O_DIRECT
(most host resources assigned to guests)

-- 
error compiling committee.c: too many arguments to function