[SPDK] NVMeoF IO latency performance apps

[SPDK] NVMeoF IO latency performance apps

[Rajat Maheshwari]

[-- Attachment #1: Type: text/plain, Size: 200 bytes --]

Hi,

Which all supported IO utilities are available for NVMe-over-Fabric latency
analysis?
Also, does all these applications support parallel exploitation of NVMe
Queues?

---
Regards
Rajat

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 289 bytes --]

Re: [SPDK] NVMeoF IO latency performance apps

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

On Mon, 2017-03-06 at 23:04 +0530, Rajat Maheshwari wrote:
> Hi,
> 
> Which all supported IO utilities are available for NVMe-over-Fabric
> latency analysis?
> Also, does all these applications support parallel exploitation of
> NVMe Queues?

The majority of people use fio. Are you planning to use the SPDK NVMe-
oF target, the NVMe-oF host, or both? If you are using just the SPDK
NVMe-oF target with the kernel host, fio is an excellent tool.

> 
> ---
> Regards
> Rajat
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]

Re: [SPDK] NVMeoF IO latency performance apps

[Rajat Maheshwari]

[-- Attachment #1: Type: text/plain, Size: 2475 bytes --]

Thanks.

We are using SPDK on target end, but kernel Initiator. We used fio, but
found some unexpected results. (latency figures almost similar for SPDK
target & Kernel Target, besides we are using NULL block devices too, so
media latency is not involved here)

So, I would like to check for certain fio parameters, if in case we are
missing something.

For ex.:
couple of samples from fio "read I/O" run (SPDK NVMe-oF target, Kernel host)

(fio command line:
fio --bs=4k --numjobs=16 --iodepth=4 --loops=1 --ioengine=libaio --direct=1
--invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based
--runtime=10 --filename=/dev/nvme0n1  --name=read-phase --rw=read )

sample #1
    slat (usec): min=1 , max=43 , avg= 3.61, stdev= 1.77
    clat (usec): min=40 , max=414 , avg=233.02, stdev=28.75
     lat (usec): min=47 , max=416 , avg=236.73, stdev=28.79

sample #2
    slat (usec): min=1 , max=41 , avg= 3.63, stdev= 1.77
    clat (usec): min=71 , max=473 , avg=232.60, stdev=28.65
     lat (usec): min=74 , max=476 , avg=236.33, stdev=28.70

sample #3
    slat (usec): min=1 , max=48 , avg= 3.64, stdev= 1.82
    clat (usec): min=17 , max=477 , avg=233.10, stdev=28.87
     lat (usec): min=21 , max=479 , avg=236.84, stdev=28.93


All above samples (infact every sample) shows maximum clat *above 380us*.
Also, the average clat is *never below 230us*.


We also compared it using kernel target (& Kernel host) and apparently we
didn't see much difference between two runs.
Max clat range was almost in the range (only 10-20usec higher than SDPK
target), but average clat was, almost, exactly the same (230-240us).

Is it really expected? (I believe, its not!)
So, are we missing any fio flag(s)?


On Mon, 2017-03-06 at 23:04 +0530, Rajat Maheshwari wrote:
> Hi,
>
> Which all supported IO utilities are available for NVMe-over-Fabric
> latency analysis?
> Also, does all these applications support parallel exploitation of
> NVMe Queues?

The majority of people use fio. Are you planning to use the SPDK NVMe-
oF target, the NVMe-oF host, or both? If you are using just the SPDK
NVMe-oF target with the kernel host, fio is an excellent tool.

>
> ---
> Regards
> Rajat
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2913 bytes --]

Re: [SPDK] NVMeoF IO latency performance apps

[Raj Pandurangan]

[-- Attachment #1: Type: text/plain, Size: 2976 bytes --]

Our results were also pretty much align with your findings.  But CPU utilization was way less in SPDK target compared to Kernel mode target.
It would be nice to learn if anyone else has seen better results on kernel mode initiator with SPDK target.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Rajat Maheshwari
Sent: Thursday, March 09, 2017 12:53 AM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] NVMeoF IO latency performance apps


Thanks.

We are using SPDK on target end, but kernel Initiator. We used fio, but found some unexpected results. (latency figures almost similar for SPDK target & Kernel Target, besides we are using NULL block devices too, so media latency is not involved here)

So, I would like to check for certain fio parameters, if in case we are missing something.

For ex.:
couple of samples from fio "read I/O" run (SPDK NVMe-oF target, Kernel host)

(fio command line:
fio --bs=4k --numjobs=16 --iodepth=4 --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --time_based --runtime=10 --filename=/dev/nvme0n1  --name=read-phase --rw=read )

sample #1
    slat (usec): min=1 , max=43 , avg= 3.61, stdev= 1.77
    clat (usec): min=40 , max=414 , avg=233.02, stdev=28.75
     lat (usec): min=47 , max=416 , avg=236.73, stdev=28.79

sample #2
    slat (usec): min=1 , max=41 , avg= 3.63, stdev= 1.77
    clat (usec): min=71 , max=473 , avg=232.60, stdev=28.65
     lat (usec): min=74 , max=476 , avg=236.33, stdev=28.70

sample #3
    slat (usec): min=1 , max=48 , avg= 3.64, stdev= 1.82
    clat (usec): min=17 , max=477 , avg=233.10, stdev=28.87
     lat (usec): min=21 , max=479 , avg=236.84, stdev=28.93


All above samples (infact every sample) shows maximum clat above 380us. Also, the average clat is never below 230us.


We also compared it using kernel target (& Kernel host) and apparently we didn't see much difference between two runs.
Max clat range was almost in the range (only 10-20usec higher than SDPK target), but average clat was, almost, exactly the same (230-240us).

Is it really expected? (I believe, its not!)
So, are we missing any fio flag(s)?


On Mon, 2017-03-06 at 23:04 +0530, Rajat Maheshwari wrote:
> Hi,
>
> Which all supported IO utilities are available for NVMe-over-Fabric
> latency analysis?
> Also, does all these applications support parallel exploitation of
> NVMe Queues?

The majority of people use fio. Are you planning to use the SPDK NVMe-
oF target, the NVMe-oF host, or both? If you are using just the SPDK
NVMe-oF target with the kernel host, fio is an excellent tool.

>
> ---
> Regards
> Rajat
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk

SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6157 bytes --]