Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
> Dear all,
> 
> I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 CPU
> cores). I noticed that the IOPS is around 1K when the queue depth is 1. By
> simply increase the queue length to 2, the IOPS grows to over 60K. May I ask
> if this is normal?

Is this an EC2 dedicated server (i.e. physical hardware) or a virtual machine?
Is the NVMe device you're connected to backed by a local NVMe device, a remote
NVMe device, or an EBS volume? Amazon presents basically all storage as NVMe
devices these days, but the actual backing storage can perform differently
depending on the circumstances.

In all cases I'm aware of, the NVMe device presented is not the actual NVMe SSD.
It's either a virtual NVMe device emulated in software, or it's implemented on a
special card that Amazon puts in between the platform and the real SSD. SPDK
hasn't been tested thoroughly with Amazon's emulated NVMe devices, so there
could be some issues there. Most NVMe devices historically were not validated
against polled mode, ultra high performance drivers so when SPDK was first
released we found a number of firmware bugs in existing SSDs that were only
exposed because SPDK was so fast. We're seeing less and less of that now, but
you could be hitting something like that here.

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 12058 bytes --]

Dear all,

I started the same EC2 instance with the same image this morning, and run the perf commands. This time, the IOPS with 1 thread and 1 queue depth is around 40K, while that with 2 queue depth or two threads is approximately doubled, which is the same as my expectation. 

So I believethe reason linked to the wired performance before is highly likely to be an issue or bug on EC2 instance. 

Thanks you all for providing feedbacks and help. 

Regards,
Li

> On 3 Jan 2018, at 07:53, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
> 
> Yeah, clearly there's still something else going on likely related to the fact that it's an EC instance...
> 
> Thx
> Paul
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Brijesh Singh
> Sent: Tuesday, January 2, 2018 1:47 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth
> 
> Hi Paul,
> 
> Correct me if I am wrong, even if it is less efficient in terms of cpu, tight polling should lead to lower latency and higher throughput.
> I am not sure why the results are better when a sleep is injected in between.
> 
> I see 1 more anomaly in Li's results. The latencies are way higher for queue depth 1 than queue depth 2. That doesn't make sense to me.  I have tested this with my NVMe with SPDK/kernel driver. Latencies are always lowest at queue depth 1.
> 
> 
> Regards,
> Brijesh
> 
>> On Mon, Jan 1, 2018 at 8:20 AM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
>> 
>> 
>> Happy New Year!
>> 
>> 
>> 
>> I can’t comment on all points in your email, I’m sure some others will 
>> chime in as well but wrt your comment/observation that makes sense I 
>> think. All you are seeing is the expected relationship between polling 
>> frequency, Q depth and IOPs.  There’s no magic formula that is going 
>> to be optimal in all cases, if you poll too frequently and there’s not 
>> enough work to do you are not going to be as efficient as if you can 
>> be but polling less frequently in that case will give you sort of a 
>> harmonic effect where every time you poll you’ll have a much better 
>> chance of catching real work. You can easily add some counters to look 
>> at the ratio of the number of times you call the completion routine vs 
>> how many are completed each time and you should see this.
>> 
>> 
>> 
>> On changing the core mask, I ran these same two sets of parms on bare 
>> metal w/a real SSD and see very similar data between the two 
>> (collectively) which is what I would expect at least so I’d assume 
>> your disparity has to do with running on an EC2 instnce.
>> 
>> 
>> 
>> Core mask 0x3:
>> 
>> 
>> Latency(us)
>> 
>> Device Information                                     :       IOPS
>> MB/s    Average        min        max
>> 
>> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 1:   21790.00
>> 10.64      45.86       7.29    4061.01
>> 
>> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   22155.00
>> 10.82      45.09       6.59    4266.28
>> 
>> ========================================================
>> 
>> Total                                                  :   43945.00
>> 21.46      45.47       6.94    4163.65
>> 
>> 
>> 
>> Core mask 0x1:
>> 
>> ========================================================
>> 
>> 
>> Latency(us)
>> 
>> Device Information                                     :       IOPS
>> MB/s    Average        min        max
>> 
>> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   39795.00
>> 19.43      25.12       7.10    3395.01
>> 
>> ========================================================
>> 
>> Total                                                  :   39795.00
>> 19.43      25.12       7.10    3395.01
>> 
>> 
>> 
>> -Paul
>> 
>> 
>> 
>> 
>> 
>> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Li Wang
>> Sent: Monday, January 1, 2018 6:48 AM
>> To: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 
>> 100K/s at 2 queue depth
>> 
>> 
>> 
>> Hi All,
>> 
>> 
>> 
>> I run fio with kernel nvme. The IOPS with queue depth = 1 is 25K, two 
>> orders of magnitude higher than with spdk. The results are as follows:
>> 
>> 
>> 
>> $ fio --filename=/dev/nvme0n1 --name=randwrite --ioengine=libaio 
>> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 
>> —group_reporting
>> 
>> 
>> 
>> read : io=1002.8MB, bw=102665KB/s, iops=25666, runt= 10001msec
>> 
>> 
>> 
>> I write a test which create 1 thread that performs a large number of 
>> random writes to nvme in a loop using spdk with queue depth =1. In 
>> each round of the loop, the program submits a random write command to 
>> the queue pairs and then call repeatedly call 
>> spdk_nvme_qpair_process_completions() until the submitted IO is 
>> processed. I found that when the depth is 1, the delay between the 
>> calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and the 
>> calling of spdk_nvme_qpair_process_completions() has huge effects on 
>> the IOPS. More preciously, I found that if I call
>> spdk_nvme_qpair_process_completions() immediately after the submission 
>> of write command (via spdk_nvme_ns_cmd_write()), the IOPS will be 
>> extremely low, e.g., ~1KB. However, if the program sleeps for several 
>> microseconds, a much higher IOPS, say 10KB, can be achieved. My 
>> feeling is that calling
>> spdk_nvme_qpair_process_completions() too soon and too often could 
>> severely increase the processing time of an IO command.
>> 
>> 
>> 
>> I also found that "perf -q 1 -s 512 -w write -t 1 -c 3” achieved over 
>> 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1” achieved only ~1K IOPS.
>> 
>> 
>> 
>> Do you know the reason behind this wired performance? Also, I’d like 
>> to know if the wired performance is highly related to EC2 instance. I 
>> will appreciate if you can share the results of the same commands on a 
>> hardware rather than EC2 i3 instance?
>> 
>> 
>> 
>> 
>> 
>> Thank you all for your time and your feedback is highly appreciated.
>> 
>> 
>> 
>> Sincerely,
>> 
>> Li Wang
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On Dec 31, 2017, at 12:56 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>> 
>> 
>> 
>> Dear Brijesh,
>> 
>> Thanks for your step by step guidance. I will let you know when I get 
>> the results.
>> 
>> Regards,
>> Li Wang
>> 
>> 
>> On Dec 30, 2017, at 10:47 PM, Brijesh Singh 
>> <brijesh.s.singh(a)gmail.com>
>> wrote:
>> 
>> Hi Li,
>> 
>> Try following steps:
>> - Switch from SPDK to NVMe kernel module.
>> #rmmod uio_pci_generic && modprobe nvme && echo 1 > 
>> /sys/bus/pci/rescan
>> - Install fio
>> # yum install fio
>> - Run fio with queue depth 1
>> # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
>> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 
>> --group_reporting
>> 
>> When all tests are done, run following command to switch back to SPDK stack.
>> # rmmod nvme && modprobe uio_pci_generic && echo 1 > 
>> /sys/bus/pci/rescan
>> 
>> Regards,
>> Brijesh
>> 
>> 
>> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>> 
>> Hi Brijesh,
>> 
>> Thanks for your suggestion. May I know how I can run fio with kernel 
>> nvme driver?
>> 
>> Thanks,
>> Li
>> 
>> On 30 December 2017 at 12:06, Brijesh Singh 
>> <brijesh.s.singh(a)gmail.com>
>> wrote:
>> 
>> 
>> Hi Li,
>> 
>> Do you see similar results when you run fio with kernel nvme driver?
>> 
>> Regards,
>> Brijesh
>> 
>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>> 
>> Dear Crane,
>> 
>> Following you advice, I run a variety of random-read and random-write 
>> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 
>> is consistently around 1K under various workloads and settings.
>> 
>> Thanks,
>> Li
>> 
>> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>> 
>> 
>> Hi, Li,
>> 
>> Maybe you can try 4K random read to simplify the test. 512byte 
>> seq-read has too much possibilties and surprises in ssd firmware. :)
>> 
>> -Crane
>> 
>> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>> 
>> 
>> Dear Benjamin,
>> 
>> Thanks for you help.
>> 
>> I just allocate a dedicated host and rerun the test, but I got the 
>> similar results. By creating a instance on the dedicated host, I am 
>> sure that the NVMe device I connected is backed by a local NVMe 
>> device.
>> It's not
>> an EBS volume, although I don't whether the device is emulated.
>> 
>> Thanks,
>> Li
>> 
>> On 30 December 2017 at 04:42, Walker, Benjamin 
>> <benjamin.walker(a)intel.com> wrote:
>> 
>> 
>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>> 
>> Dear all,
>> 
>> I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 
>> CPU cores). I noticed that the IOPS is around 1K when the queue depth 
>> is 1. By simply increase the queue length to 2, the IOPS grows to over 
>> 60K.
>> May
>> I ask
>> if this is normal?
>> 
>> 
>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual 
>> machine?
>> Is the NVMe device you're connected to backed by a local NVMe device, 
>> a remote NVMe device, or an EBS volume? Amazon presents basically all 
>> storage as NVMe devices these days, but the actual backing storage can 
>> perform differently depending on the circumstances.
>> 
>> In all cases I'm aware of, the NVMe device presented is not the actual 
>> NVMe SSD.
>> It's either a virtual NVMe device emulated in software, or it's 
>> implemented on a special card that Amazon puts in between the platform 
>> and the real SSD.
>> SPDK
>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so 
>> there could be some issues there. Most NVMe devices historically were 
>> not validated against polled mode, ultra high performance drivers so 
>> when SPDK was first released we found a number of firmware bugs in 
>> existing SSDs that were only exposed because SPDK was so fast. We're 
>> seeing less and less of that now, but you could be hitting something 
>> like that here.
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 10833 bytes --]

Yeah, clearly there's still something else going on likely related to the fact that it's an EC instance...

Thx
Paul

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Brijesh Singh
Sent: Tuesday, January 2, 2018 1:47 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

Hi Paul,

Correct me if I am wrong, even if it is less efficient in terms of cpu, tight polling should lead to lower latency and higher throughput.
I am not sure why the results are better when a sleep is injected in between.

I see 1 more anomaly in Li's results. The latencies are way higher for queue depth 1 than queue depth 2. That doesn't make sense to me.  I have tested this with my NVMe with SPDK/kernel driver. Latencies are always lowest at queue depth 1.


Regards,
Brijesh

On Mon, Jan 1, 2018 at 8:20 AM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
>
>
> Happy New Year!
>
>
>
> I can’t comment on all points in your email, I’m sure some others will 
> chime in as well but wrt your comment/observation that makes sense I 
> think. All you are seeing is the expected relationship between polling 
> frequency, Q depth and IOPs.  There’s no magic formula that is going 
> to be optimal in all cases, if you poll too frequently and there’s not 
> enough work to do you are not going to be as efficient as if you can 
> be but polling less frequently in that case will give you sort of a 
> harmonic effect where every time you poll you’ll have a much better 
> chance of catching real work. You can easily add some counters to look 
> at the ratio of the number of times you call the completion routine vs 
> how many are completed each time and you should see this.
>
>
>
> On changing the core mask, I ran these same two sets of parms on bare 
> metal w/a real SSD and see very similar data between the two 
> (collectively) which is what I would expect at least so I’d assume 
> your disparity has to do with running on an EC2 instnce.
>
>
>
> Core mask 0x3:
>
>
> Latency(us)
>
> Device Information                                     :       IOPS
> MB/s    Average        min        max
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 1:   21790.00
> 10.64      45.86       7.29    4061.01
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   22155.00
> 10.82      45.09       6.59    4266.28
>
> ========================================================
>
> Total                                                  :   43945.00
> 21.46      45.47       6.94    4163.65
>
>
>
> Core mask 0x1:
>
> ========================================================
>
>
> Latency(us)
>
> Device Information                                     :       IOPS
> MB/s    Average        min        max
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   39795.00
> 19.43      25.12       7.10    3395.01
>
> ========================================================
>
> Total                                                  :   39795.00
> 19.43      25.12       7.10    3395.01
>
>
>
> -Paul
>
>
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Li Wang
> Sent: Monday, January 1, 2018 6:48 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 
> 100K/s at 2 queue depth
>
>
>
> Hi All,
>
>
>
> I run fio with kernel nvme. The IOPS with queue depth = 1 is 25K, two 
> orders of magnitude higher than with spdk. The results are as follows:
>
>
>
> $ fio --filename=/dev/nvme0n1 --name=randwrite --ioengine=libaio 
> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 
> —group_reporting
>
>
>
> read : io=1002.8MB, bw=102665KB/s, iops=25666, runt= 10001msec
>
>
>
> I write a test which create 1 thread that performs a large number of 
> random writes to nvme in a loop using spdk with queue depth =1. In 
> each round of the loop, the program submits a random write command to 
> the queue pairs and then call repeatedly call 
> spdk_nvme_qpair_process_completions() until the submitted IO is 
> processed. I found that when the depth is 1, the delay between the 
> calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and the 
> calling of spdk_nvme_qpair_process_completions() has huge effects on 
> the IOPS. More preciously, I found that if I call
> spdk_nvme_qpair_process_completions() immediately after the submission 
> of write command (via spdk_nvme_ns_cmd_write()), the IOPS will be 
> extremely low, e.g., ~1KB. However, if the program sleeps for several 
> microseconds, a much higher IOPS, say 10KB, can be achieved. My 
> feeling is that calling
> spdk_nvme_qpair_process_completions() too soon and too often could 
> severely increase the processing time of an IO command.
>
>
>
> I also found that "perf -q 1 -s 512 -w write -t 1 -c 3” achieved over 
> 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1” achieved only ~1K IOPS.
>
>
>
> Do you know the reason behind this wired performance? Also, I’d like 
> to know if the wired performance is highly related to EC2 instance. I 
> will appreciate if you can share the results of the same commands on a 
> hardware rather than EC2 i3 instance?
>
>
>
>
>
> Thank you all for your time and your feedback is highly appreciated.
>
>
>
> Sincerely,
>
> Li Wang
>
>
>
>
>
>
>
>
>
> On Dec 31, 2017, at 12:56 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>
>
>
> Dear Brijesh,
>
> Thanks for your step by step guidance. I will let you know when I get 
> the results.
>
> Regards,
> Li Wang
>
>
> On Dec 30, 2017, at 10:47 PM, Brijesh Singh 
> <brijesh.s.singh(a)gmail.com>
> wrote:
>
> Hi Li,
>
> Try following steps:
> - Switch from SPDK to NVMe kernel module.
>   #rmmod uio_pci_generic && modprobe nvme && echo 1 > 
> /sys/bus/pci/rescan
> - Install fio
>   # yum install fio
> - Run fio with queue depth 1
>  # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 
> --group_reporting
>
> When all tests are done, run following command to switch back to SPDK stack.
> # rmmod nvme && modprobe uio_pci_generic && echo 1 > 
> /sys/bus/pci/rescan
>
> Regards,
> Brijesh
>
>
> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>
> Hi Brijesh,
>
> Thanks for your suggestion. May I know how I can run fio with kernel 
> nvme driver?
>
> Thanks,
> Li
>
> On 30 December 2017 at 12:06, Brijesh Singh 
> <brijesh.s.singh(a)gmail.com>
> wrote:
>
>
> Hi Li,
>
> Do you see similar results when you run fio with kernel nvme driver?
>
> Regards,
> Brijesh
>
> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>
> Dear Crane,
>
> Following you advice, I run a variety of random-read and random-write 
> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 
> is consistently around 1K under various workloads and settings.
>
> Thanks,
> Li
>
> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>
>
> Hi, Li,
>
> Maybe you can try 4K random read to simplify the test. 512byte 
> seq-read has too much possibilties and surprises in ssd firmware. :)
>
> -Crane
>
> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>
>
> Dear Benjamin,
>
> Thanks for you help.
>
> I just allocate a dedicated host and rerun the test, but I got the 
> similar results. By creating a instance on the dedicated host, I am 
> sure that the NVMe device I connected is backed by a local NVMe 
> device.
> It's not
> an EBS volume, although I don't whether the device is emulated.
>
> Thanks,
> Li
>
> On 30 December 2017 at 04:42, Walker, Benjamin 
> <benjamin.walker(a)intel.com> wrote:
>
>
> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>
> Dear all,
>
> I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 
> CPU cores). I noticed that the IOPS is around 1K when the queue depth 
> is 1. By simply increase the queue length to 2, the IOPS grows to over 
> 60K.
> May
> I ask
> if this is normal?
>
>
> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual 
> machine?
> Is the NVMe device you're connected to backed by a local NVMe device, 
> a remote NVMe device, or an EBS volume? Amazon presents basically all 
> storage as NVMe devices these days, but the actual backing storage can 
> perform differently depending on the circumstances.
>
> In all cases I'm aware of, the NVMe device presented is not the actual 
> NVMe SSD.
> It's either a virtual NVMe device emulated in software, or it's 
> implemented on a special card that Amazon puts in between the platform 
> and the real SSD.
> SPDK
> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so 
> there could be some issues there. Most NVMe devices historically were 
> not validated against polled mode, ultra high performance drivers so 
> when SPDK was first released we found a number of firmware bugs in 
> existing SSDs that were only exposed because SPDK was so fast. We're 
> seeing less and less of that now, but you could be hitting something 
> like that here.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Brijesh Singh]

[-- Attachment #1: Type: text/plain, Size: 10240 bytes --]

Hi Paul,

Correct me if I am wrong, even if it is less efficient in terms of
cpu, tight polling should lead to lower latency and higher throughput.
I am not sure why the results are better when a sleep is injected in
between.

I see 1 more anomaly in Li's results. The latencies are way higher for
queue depth 1 than queue depth 2. That doesn't make sense to me.  I
have tested this with my NVMe with SPDK/kernel driver. Latencies are
always lowest at queue depth 1.


Regards,
Brijesh

On Mon, Jan 1, 2018 at 8:20 AM, Luse, Paul E <paul.e.luse(a)intel.com> wrote:
>
>
> Happy New Year!
>
>
>
> I can’t comment on all points in your email, I’m sure some others will chime
> in as well but wrt your comment/observation that makes sense I think. All
> you are seeing is the expected relationship between polling frequency, Q
> depth and IOPs.  There’s no magic formula that is going to be optimal in all
> cases, if you poll too frequently and there’s not enough work to do you are
> not going to be as efficient as if you can be but polling less frequently in
> that case will give you sort of a harmonic effect where every time you poll
> you’ll have a much better chance of catching real work. You can easily add
> some counters to look at the ratio of the number of times you call the
> completion routine vs how many are completed each time and you should see
> this.
>
>
>
> On changing the core mask, I ran these same two sets of parms on bare metal
> w/a real SSD and see very similar data between the two (collectively) which
> is what I would expect at least so I’d assume your disparity has to do with
> running on an EC2 instnce.
>
>
>
> Core mask 0x3:
>
>
> Latency(us)
>
> Device Information                                     :       IOPS
> MB/s    Average        min        max
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 1:   21790.00
> 10.64      45.86       7.29    4061.01
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   22155.00
> 10.82      45.09       6.59    4266.28
>
> ========================================================
>
> Total                                                  :   43945.00
> 21.46      45.47       6.94    4163.65
>
>
>
> Core mask 0x1:
>
> ========================================================
>
>
> Latency(us)
>
> Device Information                                     :       IOPS
> MB/s    Average        min        max
>
> INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   39795.00
> 19.43      25.12       7.10    3395.01
>
> ========================================================
>
> Total                                                  :   39795.00
> 19.43      25.12       7.10    3395.01
>
>
>
> -Paul
>
>
>
>
>
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Li Wang
> Sent: Monday, January 1, 2018 6:48 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s
> at 2 queue depth
>
>
>
> Hi All,
>
>
>
> I run fio with kernel nvme. The IOPS with queue depth = 1 is 25K, two orders
> of magnitude higher than with spdk. The results are as follows:
>
>
>
> $ fio --filename=/dev/nvme0n1 --name=randwrite --ioengine=libaio --iodepth=1
> --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 —group_reporting
>
>
>
> read : io=1002.8MB, bw=102665KB/s, iops=25666, runt= 10001msec
>
>
>
> I write a test which create 1 thread that performs a large number of random
> writes to nvme in a loop using spdk with queue depth =1. In each round of
> the loop, the program submits a random write command to the queue pairs and
> then call repeatedly call spdk_nvme_qpair_process_completions() until the
> submitted IO is processed. I found that when the depth is 1, the delay
> between the calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and
> the calling of spdk_nvme_qpair_process_completions() has huge effects on the
> IOPS. More preciously, I found that if I call
> spdk_nvme_qpair_process_completions() immediately after the submission of
> write command (via spdk_nvme_ns_cmd_write()), the IOPS will be extremely
> low, e.g., ~1KB. However, if the program sleeps for several microseconds, a
> much higher IOPS, say 10KB, can be achieved. My feeling is that calling
> spdk_nvme_qpair_process_completions() too soon and too often could severely
> increase the processing time of an IO command.
>
>
>
> I also found that "perf -q 1 -s 512 -w write -t 1 -c 3” achieved over 50K
> IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1” achieved only ~1K IOPS.
>
>
>
> Do you know the reason behind this wired performance? Also, I’d like to know
> if the wired performance is highly related to EC2 instance. I will
> appreciate if you can share the results of the same commands on a hardware
> rather than EC2 i3 instance?
>
>
>
>
>
> Thank you all for your time and your feedback is highly appreciated.
>
>
>
> Sincerely,
>
> Li Wang
>
>
>
>
>
>
>
>
>
> On Dec 31, 2017, at 12:56 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>
>
>
> Dear Brijesh,
>
> Thanks for your step by step guidance. I will let you know when I get the
> results.
>
> Regards,
> Li Wang
>
>
> On Dec 30, 2017, at 10:47 PM, Brijesh Singh <brijesh.s.singh(a)gmail.com>
> wrote:
>
> Hi Li,
>
> Try following steps:
> - Switch from SPDK to NVMe kernel module.
>   #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
> - Install fio
>   # yum install fio
> - Run fio with queue depth 1
>  # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
> --group_reporting
>
> When all tests are done, run following command to switch back to SPDK stack.
> # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan
>
> Regards,
> Brijesh
>
>
> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>
> Hi Brijesh,
>
> Thanks for your suggestion. May I know how I can run fio with kernel nvme
> driver?
>
> Thanks,
> Li
>
> On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
> wrote:
>
>
> Hi Li,
>
> Do you see similar results when you run fio with kernel nvme driver?
>
> Regards,
> Brijesh
>
> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>
> Dear Crane,
>
> Following you advice, I run a variety of random-read and random-write
> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
> consistently around 1K under various workloads and settings.
>
> Thanks,
> Li
>
> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>
>
> Hi, Li,
>
> Maybe you can try 4K random read to simplify the test. 512byte seq-read
> has too much possibilties and surprises in ssd firmware. :)
>
> -Crane
>
> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>
>
> Dear Benjamin,
>
> Thanks for you help.
>
> I just allocate a dedicated host and rerun the test, but I got the
> similar results. By creating a instance on the dedicated host, I am
> sure
> that the NVMe device I connected is backed by a local NVMe device.
> It's not
> an EBS volume, although I don't whether the device is emulated.
>
> Thanks,
> Li
>
> On 30 December 2017 at 04:42, Walker, Benjamin
> <benjamin.walker(a)intel.com> wrote:
>
>
> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>
> Dear all,
>
> I am running nvme/perf with master branch on a EC2 instance
> (Redhat, 8
> CPU
> cores). I noticed that the IOPS is around 1K when the queue depth
> is
> 1. By
> simply increase the queue length to 2, the IOPS grows to over 60K.
> May
> I ask
> if this is normal?
>
>
> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
> machine?
> Is the NVMe device you're connected to backed by a local NVMe device,
> a
> remote
> NVMe device, or an EBS volume? Amazon presents basically all storage
> as
> NVMe
> devices these days, but the actual backing storage can perform
> differently
> depending on the circumstances.
>
> In all cases I'm aware of, the NVMe device presented is not the
> actual
> NVMe SSD.
> It's either a virtual NVMe device emulated in software, or it's
> implemented on a
> special card that Amazon puts in between the platform and the real
> SSD.
> SPDK
> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
> there
> could be some issues there. Most NVMe devices historically were not
> validated
> against polled mode, ultra high performance drivers so when SPDK was
> first
> released we found a number of firmware bugs in existing SSDs that
> were
> only
> exposed because SPDK was so fast. We're seeing less and less of that
> now, but
> you could be hitting something like that here.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>
>
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 9478 bytes --]


Happy New Year!

I can’t comment on all points in your email, I’m sure some others will chime in as well but wrt your comment/observation that makes sense I think. All you are seeing is the expected relationship between polling frequency, Q depth and IOPs.  There’s no magic formula that is going to be optimal in all cases, if you poll too frequently and there’s not enough work to do you are not going to be as efficient as if you can be but polling less frequently in that case will give you sort of a harmonic effect where every time you poll you’ll have a much better chance of catching real work. You can easily add some counters to look at the ratio of the number of times you call the completion routine vs how many are completed each time and you should see this.

On changing the core mask, I ran these same two sets of parms on bare metal w/a real SSD and see very similar data between the two (collectively) which is what I would expect at least so I’d assume your disparity has to do with running on an EC2 instnce.

Core mask 0x3:
                                                                                            Latency(us)
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 1:   21790.00      10.64      45.86       7.29    4061.01
INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   22155.00      10.82      45.09       6.59    4266.28
========================================================
Total                                                  :   43945.00      21.46      45.47       6.94    4163.65

Core mask 0x1:
========================================================
                                                                                            Latency(us)
Device Information                                     :       IOPS       MB/s    Average        min        max
INTEL SSDPEDMD400G4  (CVFT7203002B400LGN  ) from core 0:   39795.00      19.43      25.12       7.10    3395.01
========================================================
Total                                                  :   39795.00      19.43      25.12       7.10    3395.01

-Paul


From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Li Wang
Sent: Monday, January 1, 2018 6:48 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

Hi All,

I run fio with kernel nvme. The IOPS with queue depth = 1 is 25K, two orders of magnitude higher than with spdk. The results are as follows:

$ fio --filename=/dev/nvme0n1 --name=randwrite --ioengine=libaio --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 —group_reporting


read : io=1002.8MB, bw=102665KB/s, iops=25666, runt= 10001msec

I write a test which create 1 thread that performs a large number of random writes to nvme in a loop using spdk with queue depth =1. In each round of the loop, the program submits a random write command to the queue pairs and then call repeatedly call spdk_nvme_qpair_process_completions() until the submitted IO is processed. I found that when the depth is 1, the delay between the calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and the calling of spdk_nvme_qpair_process_completions() has huge effects on the IOPS. More preciously, I found that if I call spdk_nvme_qpair_process_completions() immediately after the submission of write command (via spdk_nvme_ns_cmd_write()), the IOPS will be extremely low, e.g., ~1KB. However, if the program sleeps for several microseconds, a much higher IOPS, say 10KB, can be achieved. My feeling is that calling spdk_nvme_qpair_process_completions() too soon and too often could severely increase the processing time of an IO command.

I also found that "perf -q 1 -s 512 -w write -t 1 -c 3” achieved over 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1” achieved only ~1K IOPS.

Do you know the reason behind this wired performance? Also, I’d like to know if the wired performance is highly related to EC2 instance. I will appreciate if you can share the results of the same commands on a hardware rather than EC2 i3 instance?


Thank you all for your time and your feedback is highly appreciated.

Sincerely,
Li Wang





On Dec 31, 2017, at 12:56 PM, Li Wang <wangli1426(a)gmail.com<mailto:wangli1426(a)gmail.com>> wrote:

Dear Brijesh,

Thanks for your step by step guidance. I will let you know when I get the results.

Regards,
Li Wang


On Dec 30, 2017, at 10:47 PM, Brijesh Singh <brijesh.s.singh(a)gmail.com<mailto:brijesh.s.singh(a)gmail.com>> wrote:

Hi Li,

Try following steps:
- Switch from SPDK to NVMe kernel module.
  #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
- Install fio
  # yum install fio
- Run fio with queue depth 1
 # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
--iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
--group_reporting

When all tests are done, run following command to switch back to SPDK stack.
# rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan

Regards,
Brijesh


On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com<mailto:wangli1426(a)gmail.com>> wrote:

Hi Brijesh,

Thanks for your suggestion. May I know how I can run fio with kernel nvme
driver?

Thanks,
Li

On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com<mailto:brijesh.s.singh(a)gmail.com>>
wrote:


Hi Li,

Do you see similar results when you run fio with kernel nvme driver?

Regards,
Brijesh

On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com<mailto:wangli1426(a)gmail.com>> wrote:

Dear Crane,

Following you advice, I run a variety of random-read and random-write
workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
consistently around 1K under various workloads and settings.

Thanks,
Li

On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com<mailto:cranechu(a)gmail.com>> wrote:


Hi, Li,

Maybe you can try 4K random read to simplify the test. 512byte seq-read
has too much possibilties and surprises in ssd firmware. :)

-Crane

2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com<mailto:wangli1426(a)gmail.com>>写道：


Dear Benjamin,

Thanks for you help.

I just allocate a dedicated host and rerun the test, but I got the
similar results. By creating a instance on the dedicated host, I am
sure
that the NVMe device I connected is backed by a local NVMe device.
It's not
an EBS volume, although I don't whether the device is emulated.

Thanks,
Li

On 30 December 2017 at 04:42, Walker, Benjamin
<benjamin.walker(a)intel.com<mailto:benjamin.walker(a)intel.com>> wrote:


On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:

Dear all,

I am running nvme/perf with master branch on a EC2 instance
(Redhat, 8
CPU
cores). I noticed that the IOPS is around 1K when the queue depth
is
1. By
simply increase the queue length to 2, the IOPS grows to over 60K.
May
I ask
if this is normal?

Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
machine?
Is the NVMe device you're connected to backed by a local NVMe device,
a
remote
NVMe device, or an EBS volume? Amazon presents basically all storage
as
NVMe
devices these days, but the actual backing storage can perform
differently
depending on the circumstances.

In all cases I'm aware of, the NVMe device presented is not the
actual
NVMe SSD.
It's either a virtual NVMe device emulated in software, or it's
implemented on a
special card that Amazon puts in between the platform and the real
SSD.
SPDK
hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
there
could be some issues there. Most NVMe devices historically were not
validated
against polled mode, ultra high performance drivers so when SPDK was
first
released we found a number of firmware bugs in existing SSDs that
were
only
exposed because SPDK was so fast. We're seeing less and less of that
now, but
you could be hitting something like that here.
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk



_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk



_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 23035 bytes --]

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 7584 bytes --]

Hi All,

I run fio with kernel nvme. The IOPS with queue depth = 1 is 25K, two orders of magnitude higher than with spdk. The results are as follows:

$ fio --filename=/dev/nvme0n1 --name=randwrite --ioengine=libaio --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10 —group_reporting

read : io=1002.8MB, bw=102665KB/s, iops=25666, runt= 10001msec

I write a test which create 1 thread that performs a large number of random writes to nvme in a loop using spdk with queue depth =1. In each round of the loop, the program submits a random write command to the queue pairs and then call repeatedly call spdk_nvme_qpair_process_completions() until the submitted IO is processed. I found that when the depth is 1, the delay between the calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and the calling of spdk_nvme_qpair_process_completions() has huge effects on the IOPS. More preciously, I found that if I call spdk_nvme_qpair_process_completions() immediately after the submission of write command (via spdk_nvme_ns_cmd_write()), the IOPS will be extremely low, e.g., ~1KB. However, if the program sleeps for several microseconds, a much higher IOPS, say 10KB, can be achieved. My feeling is that calling spdk_nvme_qpair_process_completions() too soon and too often could severely increase the processing time of an IO command. 

I also found that "perf -q 1 -s 512 -w write -t 1 -c 3” achieved over 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1” achieved only ~1K IOPS.

Do you know the reason behind this wired performance? Also, I’d like to know if the wired performance is highly related to EC2 instance. I will appreciate if you can share the results of the same commands on a hardware rather than EC2 i3 instance?


Thank you all for your time and your feedback is highly appreciated.

Sincerely,
Li Wang




> On Dec 31, 2017, at 12:56 PM, Li Wang <wangli1426(a)gmail.com> wrote:
> 
> Dear Brijesh,
> 
> Thanks for your step by step guidance. I will let you know when I get the results.
> 
> Regards,
> Li Wang
> 
>> On Dec 30, 2017, at 10:47 PM, Brijesh Singh <brijesh.s.singh(a)gmail.com> wrote:
>> 
>> Hi Li,
>> 
>> Try following steps:
>> - Switch from SPDK to NVMe kernel module.
>>   #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
>> - Install fio
>>   # yum install fio
>> - Run fio with queue depth 1
>>  # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
>> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
>> --group_reporting
>> 
>> When all tests are done, run following command to switch back to SPDK stack.
>> # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan
>> 
>> Regards,
>> Brijesh
>> 
>> 
>> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>>> Hi Brijesh,
>>> 
>>> Thanks for your suggestion. May I know how I can run fio with kernel nvme
>>> driver?
>>> 
>>> Thanks,
>>> Li
>>> 
>>> On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
>>> wrote:
>>>> 
>>>> Hi Li,
>>>> 
>>>> Do you see similar results when you run fio with kernel nvme driver?
>>>> 
>>>> Regards,
>>>> Brijesh
>>>> 
>>>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>>>>> Dear Crane,
>>>>> 
>>>>> Following you advice, I run a variety of random-read and random-write
>>>>> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
>>>>> consistently around 1K under various workloads and settings.
>>>>> 
>>>>> Thanks,
>>>>> Li
>>>>> 
>>>>> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>>>>>> 
>>>>>> Hi, Li,
>>>>>> 
>>>>>> Maybe you can try 4K random read to simplify the test. 512byte seq-read
>>>>>> has too much possibilties and surprises in ssd firmware. :)
>>>>>> 
>>>>>> -Crane
>>>>>> 
>>>>>> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>>>>>>> 
>>>>>>> Dear Benjamin,
>>>>>>> 
>>>>>>> Thanks for you help.
>>>>>>> 
>>>>>>> I just allocate a dedicated host and rerun the test, but I got the
>>>>>>> similar results. By creating a instance on the dedicated host, I am
>>>>>>> sure
>>>>>>> that the NVMe device I connected is backed by a local NVMe device.
>>>>>>> It's not
>>>>>>> an EBS volume, although I don't whether the device is emulated.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Li
>>>>>>> 
>>>>>>> On 30 December 2017 at 04:42, Walker, Benjamin
>>>>>>> <benjamin.walker(a)intel.com> wrote:
>>>>>>>> 
>>>>>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>>>>>>>>> Dear all,
>>>>>>>>> 
>>>>>>>>> I am running nvme/perf with master branch on a EC2 instance
>>>>>>>>> (Redhat, 8
>>>>>>>>> CPU
>>>>>>>>> cores). I noticed that the IOPS is around 1K when the queue depth
>>>>>>>>> is
>>>>>>>>> 1. By
>>>>>>>>> simply increase the queue length to 2, the IOPS grows to over 60K.
>>>>>>>>> May
>>>>>>>>> I ask
>>>>>>>>> if this is normal?
>>>>>>>> 
>>>>>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>>>>>>>> machine?
>>>>>>>> Is the NVMe device you're connected to backed by a local NVMe device,
>>>>>>>> a
>>>>>>>> remote
>>>>>>>> NVMe device, or an EBS volume? Amazon presents basically all storage
>>>>>>>> as
>>>>>>>> NVMe
>>>>>>>> devices these days, but the actual backing storage can perform
>>>>>>>> differently
>>>>>>>> depending on the circumstances.
>>>>>>>> 
>>>>>>>> In all cases I'm aware of, the NVMe device presented is not the
>>>>>>>> actual
>>>>>>>> NVMe SSD.
>>>>>>>> It's either a virtual NVMe device emulated in software, or it's
>>>>>>>> implemented on a
>>>>>>>> special card that Amazon puts in between the platform and the real
>>>>>>>> SSD.
>>>>>>>> SPDK
>>>>>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>>>>>>>> there
>>>>>>>> could be some issues there. Most NVMe devices historically were not
>>>>>>>> validated
>>>>>>>> against polled mode, ultra high performance drivers so when SPDK was
>>>>>>>> first
>>>>>>>> released we found a number of firmware bugs in existing SSDs that
>>>>>>>> were
>>>>>>>> only
>>>>>>>> exposed because SPDK was so fast. We're seeing less and less of that
>>>>>>>> now, but
>>>>>>>> you could be hitting something like that here.
>>>>>>>> _______________________________________________
>>>>>>>> SPDK mailing list
>>>>>>>> SPDK(a)lists.01.org
>>>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> SPDK mailing list
>>>>>>> SPDK(a)lists.01.org
>>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> SPDK mailing list
>>>>>> SPDK(a)lists.01.org
>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>> 
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> SPDK mailing list
>>>>> SPDK(a)lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>> 
>>>> _______________________________________________
>>>> SPDK mailing list
>>>> SPDK(a)lists.01.org
>>>> https://lists.01.org/mailman/listinfo/spdk
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> 


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12107 bytes --]

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 5448 bytes --]

Dear Brijesh,

Thanks for your step by step guidance. I will let you know when I get the results.

Regards,
Li Wang

> On Dec 30, 2017, at 10:47 PM, Brijesh Singh <brijesh.s.singh(a)gmail.com> wrote:
> 
> Hi Li,
> 
> Try following steps:
> - Switch from SPDK to NVMe kernel module.
>    #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
> - Install fio
>    # yum install fio
> - Run fio with queue depth 1
>   # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
> --group_reporting
> 
> When all tests are done, run following command to switch back to SPDK stack.
>  # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan
> 
> Regards,
> Brijesh
> 
> 
> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>> Hi Brijesh,
>> 
>> Thanks for your suggestion. May I know how I can run fio with kernel nvme
>> driver?
>> 
>> Thanks,
>> Li
>> 
>> On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
>> wrote:
>>> 
>>> Hi Li,
>>> 
>>> Do you see similar results when you run fio with kernel nvme driver?
>>> 
>>> Regards,
>>> Brijesh
>>> 
>>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>>>> Dear Crane,
>>>> 
>>>> Following you advice, I run a variety of random-read and random-write
>>>> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
>>>> consistently around 1K under various workloads and settings.
>>>> 
>>>> Thanks,
>>>> Li
>>>> 
>>>> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>>>>> 
>>>>> Hi, Li,
>>>>> 
>>>>> Maybe you can try 4K random read to simplify the test. 512byte seq-read
>>>>> has too much possibilties and surprises in ssd firmware. :)
>>>>> 
>>>>> -Crane
>>>>> 
>>>>> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>>>>>> 
>>>>>> Dear Benjamin,
>>>>>> 
>>>>>> Thanks for you help.
>>>>>> 
>>>>>> I just allocate a dedicated host and rerun the test, but I got the
>>>>>> similar results. By creating a instance on the dedicated host, I am
>>>>>> sure
>>>>>> that the NVMe device I connected is backed by a local NVMe device.
>>>>>> It's not
>>>>>> an EBS volume, although I don't whether the device is emulated.
>>>>>> 
>>>>>> Thanks,
>>>>>> Li
>>>>>> 
>>>>>> On 30 December 2017 at 04:42, Walker, Benjamin
>>>>>> <benjamin.walker(a)intel.com> wrote:
>>>>>>> 
>>>>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>>>>>>>> Dear all,
>>>>>>>> 
>>>>>>>> I am running nvme/perf with master branch on a EC2 instance
>>>>>>>> (Redhat, 8
>>>>>>>> CPU
>>>>>>>> cores). I noticed that the IOPS is around 1K when the queue depth
>>>>>>>> is
>>>>>>>> 1. By
>>>>>>>> simply increase the queue length to 2, the IOPS grows to over 60K.
>>>>>>>> May
>>>>>>>> I ask
>>>>>>>> if this is normal?
>>>>>>> 
>>>>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>>>>>>> machine?
>>>>>>> Is the NVMe device you're connected to backed by a local NVMe device,
>>>>>>> a
>>>>>>> remote
>>>>>>> NVMe device, or an EBS volume? Amazon presents basically all storage
>>>>>>> as
>>>>>>> NVMe
>>>>>>> devices these days, but the actual backing storage can perform
>>>>>>> differently
>>>>>>> depending on the circumstances.
>>>>>>> 
>>>>>>> In all cases I'm aware of, the NVMe device presented is not the
>>>>>>> actual
>>>>>>> NVMe SSD.
>>>>>>> It's either a virtual NVMe device emulated in software, or it's
>>>>>>> implemented on a
>>>>>>> special card that Amazon puts in between the platform and the real
>>>>>>> SSD.
>>>>>>> SPDK
>>>>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>>>>>>> there
>>>>>>> could be some issues there. Most NVMe devices historically were not
>>>>>>> validated
>>>>>>> against polled mode, ultra high performance drivers so when SPDK was
>>>>>>> first
>>>>>>> released we found a number of firmware bugs in existing SSDs that
>>>>>>> were
>>>>>>> only
>>>>>>> exposed because SPDK was so fast. We're seeing less and less of that
>>>>>>> now, but
>>>>>>> you could be hitting something like that here.
>>>>>>> _______________________________________________
>>>>>>> SPDK mailing list
>>>>>>> SPDK(a)lists.01.org
>>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> SPDK mailing list
>>>>>> SPDK(a)lists.01.org
>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> SPDK mailing list
>>>>> SPDK(a)lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> SPDK mailing list
>>>> SPDK(a)lists.01.org
>>>> https://lists.01.org/mailman/listinfo/spdk
>>>> 
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Brijesh Singh]

[-- Attachment #1: Type: text/plain, Size: 5006 bytes --]

Hi Li,

Try following steps:
 - Switch from SPDK to NVMe kernel module.
    #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
 - Install fio
    # yum install fio
 - Run fio with queue depth 1
   # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
--iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
--group_reporting

 When all tests are done, run following command to switch back to SPDK stack.
  # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan

Regards,
Brijesh


On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
> Hi Brijesh,
>
> Thanks for your suggestion. May I know how I can run fio with kernel nvme
> driver?
>
> Thanks,
> Li
>
> On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
> wrote:
>>
>> Hi Li,
>>
>> Do you see similar results when you run fio with kernel nvme driver?
>>
>> Regards,
>> Brijesh
>>
>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>> > Dear Crane,
>> >
>> > Following you advice, I run a variety of random-read and random-write
>> > workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
>> > consistently around 1K under various workloads and settings.
>> >
>> > Thanks,
>> > Li
>> >
>> > On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>> >>
>> >> Hi, Li,
>> >>
>> >> Maybe you can try 4K random read to simplify the test. 512byte seq-read
>> >> has too much possibilties and surprises in ssd firmware. :)
>> >>
>> >> -Crane
>> >>
>> >> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>> >>>
>> >>> Dear Benjamin,
>> >>>
>> >>> Thanks for you help.
>> >>>
>> >>> I just allocate a dedicated host and rerun the test, but I got the
>> >>> similar results. By creating a instance on the dedicated host, I am
>> >>> sure
>> >>> that the NVMe device I connected is backed by a local NVMe device.
>> >>> It's not
>> >>> an EBS volume, although I don't whether the device is emulated.
>> >>>
>> >>> Thanks,
>> >>> Li
>> >>>
>> >>> On 30 December 2017 at 04:42, Walker, Benjamin
>> >>> <benjamin.walker(a)intel.com> wrote:
>> >>>>
>> >>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>> >>>> > Dear all,
>> >>>> >
>> >>>> > I am running nvme/perf with master branch on a EC2 instance
>> >>>> > (Redhat, 8
>> >>>> > CPU
>> >>>> > cores). I noticed that the IOPS is around 1K when the queue depth
>> >>>> > is
>> >>>> > 1. By
>> >>>> > simply increase the queue length to 2, the IOPS grows to over 60K.
>> >>>> > May
>> >>>> > I ask
>> >>>> > if this is normal?
>> >>>>
>> >>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>> >>>> machine?
>> >>>> Is the NVMe device you're connected to backed by a local NVMe device,
>> >>>> a
>> >>>> remote
>> >>>> NVMe device, or an EBS volume? Amazon presents basically all storage
>> >>>> as
>> >>>> NVMe
>> >>>> devices these days, but the actual backing storage can perform
>> >>>> differently
>> >>>> depending on the circumstances.
>> >>>>
>> >>>> In all cases I'm aware of, the NVMe device presented is not the
>> >>>> actual
>> >>>> NVMe SSD.
>> >>>> It's either a virtual NVMe device emulated in software, or it's
>> >>>> implemented on a
>> >>>> special card that Amazon puts in between the platform and the real
>> >>>> SSD.
>> >>>> SPDK
>> >>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>> >>>> there
>> >>>> could be some issues there. Most NVMe devices historically were not
>> >>>> validated
>> >>>> against polled mode, ultra high performance drivers so when SPDK was
>> >>>> first
>> >>>> released we found a number of firmware bugs in existing SSDs that
>> >>>> were
>> >>>> only
>> >>>> exposed because SPDK was so fast. We're seeing less and less of that
>> >>>> now, but
>> >>>> you could be hitting something like that here.
>> >>>> _______________________________________________
>> >>>> SPDK mailing list
>> >>>> SPDK(a)lists.01.org
>> >>>> https://lists.01.org/mailman/listinfo/spdk
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> SPDK mailing list
>> >>> SPDK(a)lists.01.org
>> >>> https://lists.01.org/mailman/listinfo/spdk
>> >>>
>> >>
>> >> _______________________________________________
>> >> SPDK mailing list
>> >> SPDK(a)lists.01.org
>> >> https://lists.01.org/mailman/listinfo/spdk
>> >>
>> >
>> >
>> > _______________________________________________
>> > SPDK mailing list
>> > SPDK(a)lists.01.org
>> > https://lists.01.org/mailman/listinfo/spdk
>> >
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 4017 bytes --]

Hi Brijesh,

Thanks for your suggestion. May I know how I can run fio with kernel nvme
driver?

Thanks,
Li

On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
wrote:

> Hi Li,
>
> Do you see similar results when you run fio with kernel nvme driver?
>
> Regards,
> Brijesh
>
> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
> > Dear Crane,
> >
> > Following you advice, I run a variety of random-read and random-write
> > workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
> > consistently around 1K under various workloads and settings.
> >
> > Thanks,
> > Li
> >
> > On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
> >>
> >> Hi, Li,
> >>
> >> Maybe you can try 4K random read to simplify the test. 512byte seq-read
> >> has too much possibilties and surprises in ssd firmware. :)
> >>
> >> -Crane
> >>
> >> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
> >>>
> >>> Dear Benjamin,
> >>>
> >>> Thanks for you help.
> >>>
> >>> I just allocate a dedicated host and rerun the test, but I got the
> >>> similar results. By creating a instance on the dedicated host, I am
> sure
> >>> that the NVMe device I connected is backed by a local NVMe device.
> It's not
> >>> an EBS volume, although I don't whether the device is emulated.
> >>>
> >>> Thanks,
> >>> Li
> >>>
> >>> On 30 December 2017 at 04:42, Walker, Benjamin
> >>> <benjamin.walker(a)intel.com> wrote:
> >>>>
> >>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
> >>>> > Dear all,
> >>>> >
> >>>> > I am running nvme/perf with master branch on a EC2 instance
> (Redhat, 8
> >>>> > CPU
> >>>> > cores). I noticed that the IOPS is around 1K when the queue depth is
> >>>> > 1. By
> >>>> > simply increase the queue length to 2, the IOPS grows to over 60K.
> May
> >>>> > I ask
> >>>> > if this is normal?
> >>>>
> >>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
> >>>> machine?
> >>>> Is the NVMe device you're connected to backed by a local NVMe device,
> a
> >>>> remote
> >>>> NVMe device, or an EBS volume? Amazon presents basically all storage
> as
> >>>> NVMe
> >>>> devices these days, but the actual backing storage can perform
> >>>> differently
> >>>> depending on the circumstances.
> >>>>
> >>>> In all cases I'm aware of, the NVMe device presented is not the actual
> >>>> NVMe SSD.
> >>>> It's either a virtual NVMe device emulated in software, or it's
> >>>> implemented on a
> >>>> special card that Amazon puts in between the platform and the real
> SSD.
> >>>> SPDK
> >>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
> >>>> there
> >>>> could be some issues there. Most NVMe devices historically were not
> >>>> validated
> >>>> against polled mode, ultra high performance drivers so when SPDK was
> >>>> first
> >>>> released we found a number of firmware bugs in existing SSDs that were
> >>>> only
> >>>> exposed because SPDK was so fast. We're seeing less and less of that
> >>>> now, but
> >>>> you could be hitting something like that here.
> >>>> _______________________________________________
> >>>> SPDK mailing list
> >>>> SPDK(a)lists.01.org
> >>>> https://lists.01.org/mailman/listinfo/spdk
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> SPDK mailing list
> >>> SPDK(a)lists.01.org
> >>> https://lists.01.org/mailman/listinfo/spdk
> >>>
> >>
> >> _______________________________________________
> >> SPDK mailing list
> >> SPDK(a)lists.01.org
> >> https://lists.01.org/mailman/listinfo/spdk
> >>
> >
> >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6307 bytes --]

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Brijesh Singh]

[-- Attachment #1: Type: text/plain, Size: 3442 bytes --]

Hi Li,

Do you see similar results when you run fio with kernel nvme driver?

Regards,
Brijesh

On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
> Dear Crane,
>
> Following you advice, I run a variety of random-read and random-write
> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
> consistently around 1K under various workloads and settings.
>
> Thanks,
> Li
>
> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>>
>> Hi, Li,
>>
>> Maybe you can try 4K random read to simplify the test. 512byte seq-read
>> has too much possibilties and surprises in ssd firmware. :)
>>
>> -Crane
>>
>> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>>>
>>> Dear Benjamin,
>>>
>>> Thanks for you help.
>>>
>>> I just allocate a dedicated host and rerun the test, but I got the
>>> similar results. By creating a instance on the dedicated host, I am sure
>>> that the NVMe device I connected is backed by a local NVMe device. It's not
>>> an EBS volume, although I don't whether the device is emulated.
>>>
>>> Thanks,
>>> Li
>>>
>>> On 30 December 2017 at 04:42, Walker, Benjamin
>>> <benjamin.walker(a)intel.com> wrote:
>>>>
>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>>>> > Dear all,
>>>> >
>>>> > I am running nvme/perf with master branch on a EC2 instance (Redhat, 8
>>>> > CPU
>>>> > cores). I noticed that the IOPS is around 1K when the queue depth is
>>>> > 1. By
>>>> > simply increase the queue length to 2, the IOPS grows to over 60K. May
>>>> > I ask
>>>> > if this is normal?
>>>>
>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>>>> machine?
>>>> Is the NVMe device you're connected to backed by a local NVMe device, a
>>>> remote
>>>> NVMe device, or an EBS volume? Amazon presents basically all storage as
>>>> NVMe
>>>> devices these days, but the actual backing storage can perform
>>>> differently
>>>> depending on the circumstances.
>>>>
>>>> In all cases I'm aware of, the NVMe device presented is not the actual
>>>> NVMe SSD.
>>>> It's either a virtual NVMe device emulated in software, or it's
>>>> implemented on a
>>>> special card that Amazon puts in between the platform and the real SSD.
>>>> SPDK
>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>>>> there
>>>> could be some issues there. Most NVMe devices historically were not
>>>> validated
>>>> against polled mode, ultra high performance drivers so when SPDK was
>>>> first
>>>> released we found a number of firmware bugs in existing SSDs that were
>>>> only
>>>> exposed because SPDK was so fast. We're seeing less and less of that
>>>> now, but
>>>> you could be hitting something like that here.
>>>> _______________________________________________
>>>> SPDK mailing list
>>>> SPDK(a)lists.01.org
>>>> https://lists.01.org/mailman/listinfo/spdk
>>>
>>>
>>>
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 3014 bytes --]

Dear Crane,

Following you advice, I run a variety of random-read and random-write
workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
consistently around 1K under various workloads and settings.

Thanks,
Li

On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:

> Hi, Li,
>
> Maybe you can try 4K random read to simplify the test. 512byte seq-read
> has too much possibilties and surprises in ssd firmware. :)
>
> -Crane
>
> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>
>> Dear Benjamin,
>>
>> Thanks for you help.
>>
>> I just allocate a dedicated host and rerun the test, but I got the
>> similar results. By creating a instance on the dedicated host, I am sure
>> that the NVMe device I connected is backed by a local NVMe device. It's not
>> an EBS volume, although I don't whether the device is emulated.
>>
>> Thanks,
>> Li
>>
>> On 30 December 2017 at 04:42, Walker, Benjamin <benjamin.walker(a)intel.com
>> > wrote:
>>
>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>>> > Dear all,
>>> >
>>> > I am running nvme/perf with master branch on a EC2 instance (Redhat, 8
>>> CPU
>>> > cores). I noticed that the IOPS is around 1K when the queue depth is
>>> 1. By
>>> > simply increase the queue length to 2, the IOPS grows to over 60K. May
>>> I ask
>>> > if this is normal?
>>>
>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>>> machine?
>>> Is the NVMe device you're connected to backed by a local NVMe device, a
>>> remote
>>> NVMe device, or an EBS volume? Amazon presents basically all storage as
>>> NVMe
>>> devices these days, but the actual backing storage can perform
>>> differently
>>> depending on the circumstances.
>>>
>>> In all cases I'm aware of, the NVMe device presented is not the actual
>>> NVMe SSD.
>>> It's either a virtual NVMe device emulated in software, or it's
>>> implemented on a
>>> special card that Amazon puts in between the platform and the real SSD.
>>> SPDK
>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>>> there
>>> could be some issues there. Most NVMe devices historically were not
>>> validated
>>> against polled mode, ultra high performance drivers so when SPDK was
>>> first
>>> released we found a number of firmware bugs in existing SSDs that were
>>> only
>>> exposed because SPDK was so fast. We're seeing less and less of that
>>> now, but
>>> you could be hitting something like that here.
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>>
>>
>>
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
>>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4546 bytes --]

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Crane Chu]

[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]

Hi, Li,

Maybe you can try 4K random read to simplify the test. 512byte seq-read has
too much possibilties and surprises in ssd firmware. :)

-Crane

2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：

> Dear Benjamin,
>
> Thanks for you help.
>
> I just allocate a dedicated host and rerun the test, but I got the similar
> results. By creating a instance on the dedicated host, I am sure that the
> NVMe device I connected is backed by a local NVMe device. It's not an EBS
> volume, although I don't whether the device is emulated.
>
> Thanks,
> Li
>
> On 30 December 2017 at 04:42, Walker, Benjamin <benjamin.walker(a)intel.com>
> wrote:
>
>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>> > Dear all,
>> >
>> > I am running nvme/perf with master branch on a EC2 instance (Redhat, 8
>> CPU
>> > cores). I noticed that the IOPS is around 1K when the queue depth is 1.
>> By
>> > simply increase the queue length to 2, the IOPS grows to over 60K. May
>> I ask
>> > if this is normal?
>>
>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>> machine?
>> Is the NVMe device you're connected to backed by a local NVMe device, a
>> remote
>> NVMe device, or an EBS volume? Amazon presents basically all storage as
>> NVMe
>> devices these days, but the actual backing storage can perform differently
>> depending on the circumstances.
>>
>> In all cases I'm aware of, the NVMe device presented is not the actual
>> NVMe SSD.
>> It's either a virtual NVMe device emulated in software, or it's
>> implemented on a
>> special card that Amazon puts in between the platform and the real SSD.
>> SPDK
>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>> there
>> could be some issues there. Most NVMe devices historically were not
>> validated
>> against polled mode, ultra high performance drivers so when SPDK was first
>> released we found a number of firmware bugs in existing SSDs that were
>> only
>> exposed because SPDK was so fast. We're seeing less and less of that now,
>> but
>> you could be hitting something like that here.
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>>
>
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3470 bytes --]

Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 2006 bytes --]

Dear Benjamin,

Thanks for you help.

I just allocate a dedicated host and rerun the test, but I got the similar
results. By creating a instance on the dedicated host, I am sure that the
NVMe device I connected is backed by a local NVMe device. It's not an EBS
volume, although I don't whether the device is emulated.

Thanks,
Li

On 30 December 2017 at 04:42, Walker, Benjamin <benjamin.walker(a)intel.com>
wrote:

> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
> > Dear all,
> >
> > I am running nvme/perf with master branch on a EC2 instance (Redhat, 8
> CPU
> > cores). I noticed that the IOPS is around 1K when the queue depth is 1.
> By
> > simply increase the queue length to 2, the IOPS grows to over 60K. May I
> ask
> > if this is normal?
>
> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
> machine?
> Is the NVMe device you're connected to backed by a local NVMe device, a
> remote
> NVMe device, or an EBS volume? Amazon presents basically all storage as
> NVMe
> devices these days, but the actual backing storage can perform differently
> depending on the circumstances.
>
> In all cases I'm aware of, the NVMe device presented is not the actual
> NVMe SSD.
> It's either a virtual NVMe device emulated in software, or it's
> implemented on a
> special card that Amazon puts in between the platform and the real SSD.
> SPDK
> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so there
> could be some issues there. Most NVMe devices historically were not
> validated
> against polled mode, ultra high performance drivers so when SPDK was first
> released we found a number of firmware bugs in existing SSDs that were only
> exposed because SPDK was so fast. We're seeing less and less of that now,
> but
> you could be hitting something like that here.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2576 bytes --]

[SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

Dear all,

I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 CPU
cores). I noticed that the IOPS is around 1K when the queue depth is 1. By
simply increase the queue length to 2, the IOPS grows to over 60K. May I
ask if this is normal?

$ perf -q 1 -s 512 -w read -t 5

========================================================


                Latency(us)

Device Information                                     :       IOPS
MB/s    Average        min        max

Amazon EC2 NVMe Inst (AWS6F57178D12C6C4DC1) from core 0:    1116.80
0.55     895.40      19.53    1969.98

========================================================

Total                                                  :    1116.80
0.55     895.40      19.53    1969.98



$ perf -q 2 -s 512 -w read -t 5

========================================================


                Latency(us)

Device Information                                     :       IOPS
MB/s    Average        min        max

Amazon EC2 NVMe Inst (AWS6F57178D12C6C4DC1) from core 0:   61094.60
 29.83      32.71      13.71     136.63

========================================================

Total                                                  :   61094.60
 29.83      32.71      13.71     136.63


Regards,

Li Wang

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4871 bytes --]