On Thu, 2017-11-16 at 15:33 +0000, Harris, James R wrote:
> It still seems like O_DSYNC should be a nop on NVMe SSDs that do not have a
> volatile write cache.  Certainly, adding O_DSYNC should not improve
> performance as Cunyin showed in his latest data.  I suspect some of these
> differences must still be related to preconditioning.

I agree - there has to be something else going on here. Sending flushes after
every I/O should make the drive slower (or have no impact), not faster.


> On 11/15/17, 5:39 PM, "SPDK on behalf of 松本周平 / MATSUMOTO，SHUUHEI" <spdk-bounc
> es(a)lists.01.org on behalf of shuhei.matsumoto.xt(a)hitachi.com> wrote:
>     I want to add our team's experience about AIO + only O_DIRECT.
>     We have observed that
>     - if only O_DIRECT is set, any IO succeeded even if HDD was hot removed.
>     - if both O_DIRECT and O_DSYNC, any IO failed if HDD was hot removed.

Is this experience based on experiments with HDDs? I assume those HDDs have a
volatile write cache, since most do. What if another explanation is that AIO
reported the I/O complete after the write to the HDD completed successfully,
then the drive was hot removed prior to a flush being sent? In that case, the
data would be lost even though the write I/O completed successfully. This would
also explain why O_DIRECT plus O_DSYNC fixes the problem - it first sends the
write which succeeds, but then tries to send a flush immediately which fails
because the device was hot removed between the two commands.

I really would like to avoid adding O_DSYNC if we can find a way to make the
code strictly correct in terms of data integrity. That option should have a
large negative impact on performance for devices that have a volatile write
cache (HDDs, consumer SSDs).