Re: [Qemu-devel] Re: [PATCH][v2] Align file accesses with cache=off (O_DIRECT)

[Anthony Liguori <anthony@codemonkey.ws>]

Avi Kivity wrote:
> Anthony Liguori wrote:
>>
>> "cached" is not a terribly accurate term.  O_DIRECT avoids the host 
>> page cache but it doesn't guarantee that the disk is using 
>> write-through.  For that, you need to use hdparm.
>>
>> O_SYNC basically turns the host page cache into a write-through 
>> cache.  In terms of data integrity, the only question that matters is 
>> whether you're misleading the guest into thinking data is on the disk 
>> when it isn't.  Both O_DIRECT and O_SYNC accomplish this.
>>
>> If you just are concerned with data integrity, O_SYNC is probably 
>> better because you get the benefits of host caching.  O_DIRECT is 
>> really for circumstances where you know that using the host page 
>> cache is going to reduce performance.
>
> In one specific circumstance O_SYNC has data integrity problems: 
> shared disks with guests running on different hosts (or even a guest 
> on one host, sharing a disk with another host).  In these cases, two 
> reads can return different values without an intervening write.

Are you assuming the underlying disk sharing protocol does not keep the 
page cache coherent?

> In the general case, O_DIRECT gives better performance.  It avoids 
> copying from the host pagecache to guest memory, and if you have spare 
> memory to benefit from caching, give it to the guest; the nearer to 
> the data consumer the cache is, the faster it performs.

This assumes the only thing you're running on the machine is VMs.  If 
you're just running one VM on your desktop, it is probably preferable to 
go through the host page cache.  In particular, the host page cache can 
be automatically aged and adjusted whereas, in general, you cannot 
reduce the guests page cache size.

> In one specific case O_SYNC (or regular read/write cached operation) 
> is better, rebooting the same guest over and over, as the guest cache 
> is flushed on reboot.  Not a very interesting case, but unfortunately 
> one that is very visible.
>
> (if you have a backing file for a COW disk, then opening the backing 
> file without O_DIRECT may be a good idea too, as the file can be 
> shared among many guests).

FWIW, we really only need to use O_SYNC when the guest has disabled 
write-back.  I think we should do that unconditionally too as it's an 
issue of correctness.

Regards,

Anthony Liguori