Re: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4653 bytes --]

Hi Shuhei, Cunyin, Ziye,

Do you see a difference between sequential write and random write?  Sequential writes with 40 jobs should behave similarly to random writes – even though each job is issuing writes sequentially, the 40 jobs combined will look like a random-like workload to the SSD.

A few additional questions:


1)       How long were these tests run for?  Especially for random write tests, they should be quite long (20-30 minutes) to ensure steady state.

2)       How much preconditioning was done on the SSD before starting the tests?  Intel recommends 90 minutes before running any type of random write workload.

Thanks,

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of 松本周平 / MATSUMOTO，SHUUHEI <shuhei.matsumoto.xt(a)hitachi.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, November 14, 2017 at 1:26 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO

Hi Cunyin, Ziye,

The following is the sequential IO test.
The queue depth is 64 (not 128) and NVMe-SSD is not in the complete steady state too.
But it looks sure that O_DIRECT and O_DSYNC is better by both results.

Thank you,
Shuhei

O_DIRECT
4K sequential read, 40jobs,
read: IOPS=473k, BW=1850MiB/s (1939MB/s)(18.1GiB/10007msec)
read: IOPS=485k, BW=1896MiB/s (1988MB/s)(18.5GiB/10006msec)

4K sequential write, 40jobs
write: IOPS=60.2k, BW=236MiB/s (248MB/s)(2380MiB/10080msec)
write: IOPS=69.4k, BW=272MiB/s (285MB/s)(2741MiB/10077msec)

O_DIRECT|O_DSYNC
4K sequential read, 40jobs,
read: IOPS=469k, BW=1834MiB/s (1923MB/s)(17.9GiB/10006msec)
read: IOPS=485k, BW=1895MiB/s (1987MB/s)(18.5GiB/10006msec)

4K sequential write, 40jobs
write: IOPS=160k, BW=627MiB/s (657MB/s)(6280MiB/10019msec)
write: IOPS=118k, BW=461MiB/s (484MB/s)(4624MiB/10025msec)

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Chang, Cunyin
Sent: Tuesday, November 14, 2017 5:07 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [!]Re: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO

Could you please also try the sequential write test:

This is my test result with P3700:
O_DIRECT
-bs=4K --iodepth=128 --rw=write
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/445.7MB/0KB /s] [0/114K/0 iops]
O_DIRECT + O_DSYNC
-bs=4K --iodepth=128 --rw=write
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/676.6MB/0KB /s] [0/173K/0 iops]

-Cunyin

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of ???? / MATSUMOTO,SHUUHEI
Sent: Tuesday, November 14, 2017 3:42 PM
To: 'spdk(a)lists.01.org' <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO

Hi,

This may be related with the current being fixed issue Anirudh found.
Current BDEV AIO uses O_DIRECT to avoid IO cache effects but does not use O_DSYNC.
O_DSYNC assures that IO is written to persistent storage.

SPDK is for storage and hence I think O_DIRECT and O_DSYNC will be better for SPDK BDEV AIO.

Cunyin understood O_DSYNC but asked me the performance difference between O_DIRECT and (O_DIRECT|O_DSYNC).
Since our team evaluated only functionality, I did simple performance test by FIO + NVMe-SSD (P3700) x 1.
I modified FIO slightly (O_SYNC -> O_DSYNC) and rebuild it.

I cannot create the complete steady state of NVMe-SSD to get the real world performance yet due to lack of time,
I did not notice major difference between O_DIRECT and (O_DIRECT|O_DSYNC).
It looks that NVMe-SSD is saturated and CPU utilization was less than 10% for all cases as long as I checked mpstat.
(Our performance team created the steady state and got the stable IO write performance 100K but I have not got it yet.
If I can create the complete steady state by taking more time I will be able to get 100K for write.)

O_DIRECT
4K random read, 40jobs,
   read: IOPS=478k, BW=1869MiB/s (1960MB/s)(18.3GiB/10007msec)

4K random write, 40jobs
  write: IOPS=68.4k, BW=268MiB/s (281MB/s)(2689MiB/10031msec)

O_DIRECT|O_DSYNC
4K random read, 40jobs,
   read: IOPS=477k, BW=1864MiB/s (1954MB/s)(18.2GiB/10007msec)

4K random write, 40jobs
  write: IOPS=72.0k, BW=286MiB/s (300MB/s)(2871MiB/10038msec)



About the difference of IO command sequence,
for SCSI disk O_DSYNC issues extra IO command and it may affect IO performance but
for NVMe-SSD O_DSYNC issues no extra IO command.
Hence I estimate this is the reason of indifference of performance.

I would appreciate your any feedback.

Thank you,
Shuhei Matsumoto

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 26513 bytes --]