From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0465471928984125388==" MIME-Version: 1.0 From: Li Wang Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth Date: Wed, 03 Jan 2018 21:05:46 +0800 Message-ID: In-Reply-To: 82C9F782B054C94B9FC04A331649C77A9D489B97@fmsmsx104.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============0465471928984125388== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Dear all, I started the same EC2 instance with the same image this morning, and run t= he perf commands. This time, the IOPS with 1 thread and 1 queue depth is ar= ound 40K, while that with 2 queue depth or two threads is approximately dou= bled, which is the same as my expectation. = So I believethe reason linked to the wired performance before is highly lik= ely to be an issue or bug on EC2 instance. = Thanks you all for providing feedbacks and help. = Regards, Li > On 3 Jan 2018, at 07:53, Luse, Paul E wrote: > = > Yeah, clearly there's still something else going on likely related to the= fact that it's an EC instance... > = > Thx > Paul > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Brijesh Singh > Sent: Tuesday, January 2, 2018 1:47 PM > To: Storage Performance Development Kit > Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/= s at 2 queue depth > = > Hi Paul, > = > Correct me if I am wrong, even if it is less efficient in terms of cpu, t= ight polling should lead to lower latency and higher throughput. > I am not sure why the results are better when a sleep is injected in betw= een. > = > I see 1 more anomaly in Li's results. The latencies are way higher for qu= eue depth 1 than queue depth 2. That doesn't make sense to me. I have test= ed this with my NVMe with SPDK/kernel driver. Latencies are always lowest a= t queue depth 1. > = > = > Regards, > Brijesh > = >> On Mon, Jan 1, 2018 at 8:20 AM, Luse, Paul E w= rote: >> = >> = >> Happy New Year! >> = >> = >> = >> I can=E2=80=99t comment on all points in your email, I=E2=80=99m sure so= me others will = >> chime in as well but wrt your comment/observation that makes sense I = >> think. All you are seeing is the expected relationship between polling = >> frequency, Q depth and IOPs. There=E2=80=99s no magic formula that is g= oing = >> to be optimal in all cases, if you poll too frequently and there=E2=80= =99s not = >> enough work to do you are not going to be as efficient as if you can = >> be but polling less frequently in that case will give you sort of a = >> harmonic effect where every time you poll you=E2=80=99ll have a much bet= ter = >> chance of catching real work. You can easily add some counters to look = >> at the ratio of the number of times you call the completion routine vs = >> how many are completed each time and you should see this. >> = >> = >> = >> On changing the core mask, I ran these same two sets of parms on bare = >> metal w/a real SSD and see very similar data between the two = >> (collectively) which is what I would expect at least so I=E2=80=99d assu= me = >> your disparity has to do with running on an EC2 instnce. >> = >> = >> = >> Core mask 0x3: >> = >> = >> Latency(us) >> = >> Device Information : IOPS >> MB/s Average min max >> = >> INTEL SSDPEDMD400G4 (CVFT7203002B400LGN ) from core 1: 21790.00 >> 10.64 45.86 7.29 4061.01 >> = >> INTEL SSDPEDMD400G4 (CVFT7203002B400LGN ) from core 0: 22155.00 >> 10.82 45.09 6.59 4266.28 >> = >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> = >> Total : 43945.00 >> 21.46 45.47 6.94 4163.65 >> = >> = >> = >> Core mask 0x1: >> = >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> = >> = >> Latency(us) >> = >> Device Information : IOPS >> MB/s Average min max >> = >> INTEL SSDPEDMD400G4 (CVFT7203002B400LGN ) from core 0: 39795.00 >> 19.43 25.12 7.10 3395.01 >> = >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >> = >> Total : 39795.00 >> 19.43 25.12 7.10 3395.01 >> = >> = >> = >> -Paul >> = >> = >> = >> = >> = >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Li Wang >> Sent: Monday, January 1, 2018 6:48 AM >> To: Storage Performance Development Kit >> Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over = >> 100K/s at 2 queue depth >> = >> = >> = >> Hi All, >> = >> = >> = >> I run fio with kernel nvme. The IOPS with queue depth =3D 1 is 25K, two = >> orders of magnitude higher than with spdk. The results are as follows: >> = >> = >> = >> $ fio --filename=3D/dev/nvme0n1 --name=3Drandwrite --ioengine=3Dlibaio = >> --iodepth=3D1 --rw=3Drandread --bs=3D4k --direct=3D1 --numjobs=3D1 --run= time=3D10 = >> =E2=80=94group_reporting >> = >> = >> = >> read : io=3D1002.8MB, bw=3D102665KB/s, iops=3D25666, runt=3D 10001msec >> = >> = >> = >> I write a test which create 1 thread that performs a large number of = >> random writes to nvme in a loop using spdk with queue depth =3D1. In = >> each round of the loop, the program submits a random write command to = >> the queue pairs and then call repeatedly call = >> spdk_nvme_qpair_process_completions() until the submitted IO is = >> processed. I found that when the depth is 1, the delay between the = >> calling of spdk_nvme_ns_cmd_write()/spdk_nvme_ns_cmd_write() and the = >> calling of spdk_nvme_qpair_process_completions() has huge effects on = >> the IOPS. More preciously, I found that if I call >> spdk_nvme_qpair_process_completions() immediately after the submission = >> of write command (via spdk_nvme_ns_cmd_write()), the IOPS will be = >> extremely low, e.g., ~1KB. However, if the program sleeps for several = >> microseconds, a much higher IOPS, say 10KB, can be achieved. My = >> feeling is that calling >> spdk_nvme_qpair_process_completions() too soon and too often could = >> severely increase the processing time of an IO command. >> = >> = >> = >> I also found that "perf -q 1 -s 512 -w write -t 1 -c 3=E2=80=9D achieved= over = >> 50K IOPS, but "perf -q 1 -s 512 -w write -t 1 -c 1=E2=80=9D achieved onl= y ~1K IOPS. >> = >> = >> = >> Do you know the reason behind this wired performance? Also, I=E2=80=99d = like = >> to know if the wired performance is highly related to EC2 instance. I = >> will appreciate if you can share the results of the same commands on a = >> hardware rather than EC2 i3 instance? >> = >> = >> = >> = >> = >> Thank you all for your time and your feedback is highly appreciated. >> = >> = >> = >> Sincerely, >> = >> Li Wang >> = >> = >> = >> = >> = >> = >> = >> = >> = >> On Dec 31, 2017, at 12:56 PM, Li Wang wrote: >> = >> = >> = >> Dear Brijesh, >> = >> Thanks for your step by step guidance. I will let you know when I get = >> the results. >> = >> Regards, >> Li Wang >> = >> = >> On Dec 30, 2017, at 10:47 PM, Brijesh Singh = >> >> wrote: >> = >> Hi Li, >> = >> Try following steps: >> - Switch from SPDK to NVMe kernel module. >> #rmmod uio_pci_generic && modprobe nvme && echo 1 > = >> /sys/bus/pci/rescan >> - Install fio >> # yum install fio >> - Run fio with queue depth 1 >> # fio --filename=3D/dev/nvme0n1 --name=3Drandread --ioengine=3Dlibaio >> --iodepth=3D1 --rw=3Drandread --bs=3D4k --direct=3D1 --numjobs=3D1 --run= time=3D10 = >> --group_reporting >> = >> When all tests are done, run following command to switch back to SPDK st= ack. >> # rmmod nvme && modprobe uio_pci_generic && echo 1 > = >> /sys/bus/pci/rescan >> = >> Regards, >> Brijesh >> = >> = >> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang wrote: >> = >> Hi Brijesh, >> = >> Thanks for your suggestion. May I know how I can run fio with kernel = >> nvme driver? >> = >> Thanks, >> Li >> = >> On 30 December 2017 at 12:06, Brijesh Singh = >> >> wrote: >> = >> = >> Hi Li, >> = >> Do you see similar results when you run fio with kernel nvme driver? >> = >> Regards, >> Brijesh >> = >> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang wrote: >> = >> Dear Crane, >> = >> Following you advice, I run a variety of random-read and random-write = >> workloads, with -s ranging from 512 to 51200. The IOPS with depth =3D 1 = >> is consistently around 1K under various workloads and settings. >> = >> Thanks, >> Li >> = >> On 30 December 2017 at 11:32, Crane Chu wrote: >> = >> = >> Hi, Li, >> = >> Maybe you can try 4K random read to simplify the test. 512byte = >> seq-read has too much possibilties and surprises in ssd firmware. :) >> = >> -Crane >> = >> 2017=E5=B9=B412=E6=9C=8830=E6=97=A5 =E4=B8=8A=E5=8D=889:57=EF=BC=8C"Li W= ang" =E5=86=99=E9=81=93=EF=BC=9A >> = >> = >> Dear Benjamin, >> = >> Thanks for you help. >> = >> I just allocate a dedicated host and rerun the test, but I got the = >> similar results. By creating a instance on the dedicated host, I am = >> sure that the NVMe device I connected is backed by a local NVMe = >> device. >> It's not >> an EBS volume, although I don't whether the device is emulated. >> = >> Thanks, >> Li >> = >> On 30 December 2017 at 04:42, Walker, Benjamin = >> wrote: >> = >> = >> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote: >> = >> Dear all, >> = >> I am running nvme/perf with master branch on a EC2 instance (Redhat, 8 = >> CPU cores). I noticed that the IOPS is around 1K when the queue depth = >> is 1. By simply increase the queue length to 2, the IOPS grows to over = >> 60K. >> May >> I ask >> if this is normal? >> = >> = >> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual = >> machine? >> Is the NVMe device you're connected to backed by a local NVMe device, = >> a remote NVMe device, or an EBS volume? Amazon presents basically all = >> storage as NVMe devices these days, but the actual backing storage can = >> perform differently depending on the circumstances. >> = >> In all cases I'm aware of, the NVMe device presented is not the actual = >> NVMe SSD. >> It's either a virtual NVMe device emulated in software, or it's = >> implemented on a special card that Amazon puts in between the platform = >> and the real SSD. >> SPDK >> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so = >> there could be some issues there. Most NVMe devices historically were = >> not validated against polled mode, ultra high performance drivers so = >> when SPDK was first released we found a number of firmware bugs in = >> existing SSDs that were only exposed because SPDK was so fast. We're = >> seeing less and less of that now, but you could be hitting something = >> like that here. >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> = >> = >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============0465471928984125388==--