Hi Shuhei, Cunyin, Ziye,
Do you see a difference between sequential write and random write? Sequential writes with 40 jobs should behave similarly to random writes – even though each job is issuing writes sequentially,
the 40 jobs combined will look like a random-like workload to the SSD.
A few additional questions:
1)
How long were these tests run for? Especially for random write tests, they should be quite long (20-30 minutes) to ensure steady state.
2)
How much preconditioning was done on the SSD before starting the tests? Intel recommends 90 minutes before running any type of random write workload.
Thanks,
-Jim
From:
SPDK <spdk-bounces@lists.01.org> on behalf of
松本周平 / MATSUMOTO，SHUUHEI
<shuhei.matsumoto.xt@hitachi.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Tuesday, November 14, 2017 at 1:26 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: Re: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO
Hi Cunyin, Ziye,
The following is the sequential IO test.
The queue depth is 64 (not 128) and NVMe-SSD is not in the complete steady state too.
But it looks sure that O_DIRECT and O_DSYNC is better by both results.
Thank you,
Shuhei
O_DIRECT
4K sequential read, 40jobs,
read: IOPS=473k, BW=1850MiB/s (1939MB/s)(18.1GiB/10007msec)
read: IOPS=485k, BW=1896MiB/s (1988MB/s)(18.5GiB/10006msec)
4K sequential write, 40jobs
write: IOPS=60.2k, BW=236MiB/s (248MB/s)(2380MiB/10080msec)
write: IOPS=69.4k, BW=272MiB/s (285MB/s)(2741MiB/10077msec)
O_DIRECT|O_DSYNC
4K sequential read, 40jobs,
read: IOPS=469k, BW=1834MiB/s (1923MB/s)(17.9GiB/10006msec)
read: IOPS=485k, BW=1895MiB/s (1987MB/s)(18.5GiB/10006msec)
4K sequential write, 40jobs
write: IOPS=160k, BW=627MiB/s (657MB/s)(6280MiB/10019msec)
write: IOPS=118k, BW=461MiB/s (484MB/s)(4624MiB/10025msec)
From: SPDK [mailto:spdk-bounces@lists.01.org]
On Behalf Of Chang, Cunyin
Sent: Tuesday, November 14, 2017 5:07 PM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [!]Re: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO
Could you please also try the sequential write test:
This is my test result with P3700:
O_DIRECT
-bs=4K --iodepth=128 --rw=write
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/445.7MB/0KB /s] [0/114K/0 iops]
O_DIRECT + O_DSYNC
-bs=4K --iodepth=128 --rw=write
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/676.6MB/0KB /s] [0/173K/0 iops]
-Cunyin
From: SPDK [mailto:spdk-bounces@lists.01.org]
On Behalf Of ???? / MATSUMOTO,SHUUHEI
Sent: Tuesday, November 14, 2017 3:42 PM
To: 'spdk@lists.01.org' <spdk@lists.01.org>
Subject: [SPDK] Set not only O_DIRECT but also O_DSYNC to BDEV_AIO
Hi,
This may be related with the current being fixed issue Anirudh found.
Current BDEV AIO uses O_DIRECT to avoid IO cache effects but does not use O_DSYNC.
O_DSYNC assures that IO is written to persistent storage.
SPDK is for storage and hence I think O_DIRECT and O_DSYNC will be better for SPDK BDEV AIO.
Cunyin understood O_DSYNC but asked me the performance difference between O_DIRECT and (O_DIRECT|O_DSYNC).
Since our team evaluated only functionality, I did simple performance test by FIO + NVMe-SSD (P3700) x 1.
I modified FIO slightly (O_SYNC -> O_DSYNC) and rebuild it.
I cannot create the complete steady state of NVMe-SSD to get the real world performance yet due to lack of time,
I did not notice major difference between O_DIRECT and (O_DIRECT|O_DSYNC).
It looks that NVMe-SSD is saturated and CPU utilization was less than 10% for all cases as long as I checked mpstat.
(Our performance team created the steady state and got the stable IO write performance 100K but I have not got it yet.
If I can create the complete steady state by taking more time I will be able to get 100K for write.)
O_DIRECT
4K random read, 40jobs,
read: IOPS=478k, BW=1869MiB/s (1960MB/s)(18.3GiB/10007msec)
4K random write, 40jobs
write: IOPS=68.4k, BW=268MiB/s (281MB/s)(2689MiB/10031msec)
O_DIRECT|O_DSYNC
4K random read, 40jobs,
read: IOPS=477k, BW=1864MiB/s (1954MB/s)(18.2GiB/10007msec)
4K random write, 40jobs
write: IOPS=72.0k, BW=286MiB/s (300MB/s)(2871MiB/10038msec)
About the difference of IO command sequence,
for SCSI disk O_DSYNC issues extra IO command and it may affect IO performance but
for NVMe-SSD O_DSYNC issues no extra IO command.
Hence I estimate this is the reason of indifference of performance.
I would appreciate your any feedback.
Thank you,
Shuhei Matsumoto