From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============4982079490448340929==" MIME-Version: 1.0 From: Li Wang Subject: Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth Date: Sun, 31 Dec 2017 12:56:02 +0800 Message-ID: In-Reply-To: CAAQ1jEEzc9PjLUeBFoiMxww0nT7-ZRz4kTaizS2EgcEWxZnJ9A@mail.gmail.com List-ID: To: spdk@lists.01.org --===============4982079490448340929== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Dear Brijesh, Thanks for your step by step guidance. I will let you know when I get the r= esults. Regards, Li Wang > On Dec 30, 2017, at 10:47 PM, Brijesh Singh = wrote: > = > Hi Li, > = > Try following steps: > - Switch from SPDK to NVMe kernel module. > #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan > - Install fio > # yum install fio > - Run fio with queue depth 1 > # fio --filename=3D/dev/nvme0n1 --name=3Drandread --ioengine=3Dlibaio > --iodepth=3D1 --rw=3Drandread --bs=3D4k --direct=3D1 --numjobs=3D1 --runt= ime=3D10 > --group_reporting > = > When all tests are done, run following command to switch back to SPDK sta= ck. > # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan > = > Regards, > Brijesh > = > = > On Sat, Dec 30, 2017 at 1:53 AM, Li Wang wrote: >> Hi Brijesh, >> = >> Thanks for your suggestion. May I know how I can run fio with kernel nvme >> driver? >> = >> Thanks, >> Li >> = >> On 30 December 2017 at 12:06, Brijesh Singh >> wrote: >>> = >>> Hi Li, >>> = >>> Do you see similar results when you run fio with kernel nvme driver? >>> = >>> Regards, >>> Brijesh >>> = >>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang wrote: >>>> Dear Crane, >>>> = >>>> Following you advice, I run a variety of random-read and random-write >>>> workloads, with -s ranging from 512 to 51200. The IOPS with depth =3D = 1 is >>>> consistently around 1K under various workloads and settings. >>>> = >>>> Thanks, >>>> Li >>>> = >>>> On 30 December 2017 at 11:32, Crane Chu wrote: >>>>> = >>>>> Hi, Li, >>>>> = >>>>> Maybe you can try 4K random read to simplify the test. 512byte seq-re= ad >>>>> has too much possibilties and surprises in ssd firmware. :) >>>>> = >>>>> -Crane >>>>> = >>>>> 2017=E5=B9=B412=E6=9C=8830=E6=97=A5 =E4=B8=8A=E5=8D=889:57=EF=BC=8C"L= i Wang" =E5=86=99=E9=81=93=EF=BC=9A >>>>>> = >>>>>> Dear Benjamin, >>>>>> = >>>>>> Thanks for you help. >>>>>> = >>>>>> I just allocate a dedicated host and rerun the test, but I got the >>>>>> similar results. By creating a instance on the dedicated host, I am >>>>>> sure >>>>>> that the NVMe device I connected is backed by a local NVMe device. >>>>>> It's not >>>>>> an EBS volume, although I don't whether the device is emulated. >>>>>> = >>>>>> Thanks, >>>>>> Li >>>>>> = >>>>>> On 30 December 2017 at 04:42, Walker, Benjamin >>>>>> wrote: >>>>>>> = >>>>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote: >>>>>>>> Dear all, >>>>>>>> = >>>>>>>> I am running nvme/perf with master branch on a EC2 instance >>>>>>>> (Redhat, 8 >>>>>>>> CPU >>>>>>>> cores). I noticed that the IOPS is around 1K when the queue depth >>>>>>>> is >>>>>>>> 1. By >>>>>>>> simply increase the queue length to 2, the IOPS grows to over 60K. >>>>>>>> May >>>>>>>> I ask >>>>>>>> if this is normal? >>>>>>> = >>>>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtu= al >>>>>>> machine? >>>>>>> Is the NVMe device you're connected to backed by a local NVMe devic= e, >>>>>>> a >>>>>>> remote >>>>>>> NVMe device, or an EBS volume? Amazon presents basically all storage >>>>>>> as >>>>>>> NVMe >>>>>>> devices these days, but the actual backing storage can perform >>>>>>> differently >>>>>>> depending on the circumstances. >>>>>>> = >>>>>>> In all cases I'm aware of, the NVMe device presented is not the >>>>>>> actual >>>>>>> NVMe SSD. >>>>>>> It's either a virtual NVMe device emulated in software, or it's >>>>>>> implemented on a >>>>>>> special card that Amazon puts in between the platform and the real >>>>>>> SSD. >>>>>>> SPDK >>>>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, = so >>>>>>> there >>>>>>> could be some issues there. Most NVMe devices historically were not >>>>>>> validated >>>>>>> against polled mode, ultra high performance drivers so when SPDK was >>>>>>> first >>>>>>> released we found a number of firmware bugs in existing SSDs that >>>>>>> were >>>>>>> only >>>>>>> exposed because SPDK was so fast. We're seeing less and less of that >>>>>>> now, but >>>>>>> you could be hitting something like that here. >>>>>>> _______________________________________________ >>>>>>> SPDK mailing list >>>>>>> SPDK(a)lists.01.org >>>>>>> https://lists.01.org/mailman/listinfo/spdk >>>>>> = >>>>>> = >>>>>> = >>>>>> _______________________________________________ >>>>>> SPDK mailing list >>>>>> SPDK(a)lists.01.org >>>>>> https://lists.01.org/mailman/listinfo/spdk >>>>>> = >>>>> = >>>>> _______________________________________________ >>>>> SPDK mailing list >>>>> SPDK(a)lists.01.org >>>>> https://lists.01.org/mailman/listinfo/spdk >>>>> = >>>> = >>>> = >>>> _______________________________________________ >>>> SPDK mailing list >>>> SPDK(a)lists.01.org >>>> https://lists.01.org/mailman/listinfo/spdk >>>> = >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >> = >> = >> = >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk >> = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============4982079490448340929==--