On Tue, 2017-11-14 at 07:41 +0000, 松本周平 / MATSUMOTO，SHUUHEI wrote:
> Hi,
>  
> Current BDEV AIO uses O_DIRECT to avoid IO cache effects but does not use
> O_DSYNC.
> O_DSYNC assures that IO is written to persistent storage.
>  
> About the difference of IO command sequence,
> for SCSI disk O_DSYNC issues extra IO command and it may affect IO performance
> but
> for NVMe-SSD O_DSYNC issues no extra IO command.
> Hence I estimate this is the reason of indifference of performance.
>  

O_DIRECT avoids using operating system caches in system memory. O_DSYNC avoids
using volatile caches in the SSD itself by setting the FUA bit on I/O (Force
Unit Access) and additionally by issuing a SCSI synchronize cache command after
each I/O on devices that report having a volatile write cache. This is why you
see an extra command for SCSI devices - they are reporting that they support a
volatile write cache. That's very common for a SAS/SATA HDD.

The SPDK API provides the user with a mechanism to query whether a block device
has a volatile write cache (spdk_bdev_has_write_cache) and an API to instruct
the device to make data in its volatile caches persistent (spdk_bdev_flush). The
existence of volatile write caches and the semantics around flushing are well
established in traditional block stacks and provide significant performance
benefits to some types of devices (particularly lower end consumer grade
devices), so we've chosen to provide those same semantics in SPDK. Altering the
spdk bdev aio module to always specify O_DSYNC will greatly reduce the
performance on these types of devices, so I think choosing just O_DIRECT and not
O_DSYNC is the correct choice for the flag. This provides the user the
traditional semantics of flushing block devices that they expect.

Note that the Intel P3700 does not have a volatile write cache, so sending flush
requests does nothing (it has a write cache, it just isn't volatile). The reason
Jim was asking about preconditioning in another branch of this thread is because
I don't think either of us expect to see any performance difference on the Intel
P3700 when the O_DSYNC flag is added, regardless of workload. If there is a
difference even after preconditioning, then it certainly warrants investigation.

Thanks Shuhei!

Ben