Re: [Qemu-devel] [4367] Align file accesses with cache=off (Kevin Wolf, Laurent Vivier)

[Jamie Lokier <jamie@shareable.org>]

Fabrice Bellard wrote:
> A note: in order to avoid uncontrolled recursions, it is better to call
> the read/write AIO callback outside the aio_read/write (see
> bdrv_aio_read_em).

Definitely, I've been bitten by equivalent recursions and deadlocks in
my own code recently.

> Personally I would not trust the OS to correctly handle the mix of
> O_DIRECT and buffered operations, especially if the corresponding file
> regions intersect !

I agree.  Several OSes document that _any_ open buffered descriptors
suppress O_DIRECT, causing all descriptors to do buffered I/O, and
(explicitly or implied) opening a descriptor with O_DIRECT, and no
buffered descriptors at the time, flushes any buffers for that file so
that switching between them is coherent.  (Though I picked up hints
that some OSes don't even manage that coherence.)

Because it's not merely a flag change, it requires some cache flushing
for coherence, I would not trust fcntl(fd, F_SETFL, x) where x changes
between O_DIRECT and non-O_DIRECT to do the right thing on some OSes -
even for _synchronous_ I/O on separate regions.

-- Jamie