Re: [SPDK] IOPS of NVMe is ~1K/s at 1 queue depth but over 100K/s at 2 queue depth

[Li Wang <wangli1426 at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 5448 bytes --]

Dear Brijesh,

Thanks for your step by step guidance. I will let you know when I get the results.

Regards,
Li Wang

> On Dec 30, 2017, at 10:47 PM, Brijesh Singh <brijesh.s.singh(a)gmail.com> wrote:
> 
> Hi Li,
> 
> Try following steps:
> - Switch from SPDK to NVMe kernel module.
>    #rmmod uio_pci_generic && modprobe nvme && echo 1 > /sys/bus/pci/rescan
> - Install fio
>    # yum install fio
> - Run fio with queue depth 1
>   # fio --filename=/dev/nvme0n1 --name=randread --ioengine=libaio
> --iodepth=1 --rw=randread --bs=4k --direct=1 --numjobs=1 --runtime=10
> --group_reporting
> 
> When all tests are done, run following command to switch back to SPDK stack.
>  # rmmod nvme && modprobe uio_pci_generic && echo 1 > /sys/bus/pci/rescan
> 
> Regards,
> Brijesh
> 
> 
> On Sat, Dec 30, 2017 at 1:53 AM, Li Wang <wangli1426(a)gmail.com> wrote:
>> Hi Brijesh,
>> 
>> Thanks for your suggestion. May I know how I can run fio with kernel nvme
>> driver?
>> 
>> Thanks,
>> Li
>> 
>> On 30 December 2017 at 12:06, Brijesh Singh <brijesh.s.singh(a)gmail.com>
>> wrote:
>>> 
>>> Hi Li,
>>> 
>>> Do you see similar results when you run fio with kernel nvme driver?
>>> 
>>> Regards,
>>> Brijesh
>>> 
>>> On Fri, Dec 29, 2017 at 7:39 PM, Li Wang <wangli1426(a)gmail.com> wrote:
>>>> Dear Crane,
>>>> 
>>>> Following you advice, I run a variety of random-read and random-write
>>>> workloads, with -s ranging from 512 to 51200. The IOPS with depth = 1 is
>>>> consistently around 1K under various workloads and settings.
>>>> 
>>>> Thanks,
>>>> Li
>>>> 
>>>> On 30 December 2017 at 11:32, Crane Chu <cranechu(a)gmail.com> wrote:
>>>>> 
>>>>> Hi, Li,
>>>>> 
>>>>> Maybe you can try 4K random read to simplify the test. 512byte seq-read
>>>>> has too much possibilties and surprises in ssd firmware. :)
>>>>> 
>>>>> -Crane
>>>>> 
>>>>> 2017年12月30日 上午9:57，"Li Wang" <wangli1426(a)gmail.com>写道：
>>>>>> 
>>>>>> Dear Benjamin,
>>>>>> 
>>>>>> Thanks for you help.
>>>>>> 
>>>>>> I just allocate a dedicated host and rerun the test, but I got the
>>>>>> similar results. By creating a instance on the dedicated host, I am
>>>>>> sure
>>>>>> that the NVMe device I connected is backed by a local NVMe device.
>>>>>> It's not
>>>>>> an EBS volume, although I don't whether the device is emulated.
>>>>>> 
>>>>>> Thanks,
>>>>>> Li
>>>>>> 
>>>>>> On 30 December 2017 at 04:42, Walker, Benjamin
>>>>>> <benjamin.walker(a)intel.com> wrote:
>>>>>>> 
>>>>>>> On Fri, 2017-12-29 at 23:48 +0800, Li Wang wrote:
>>>>>>>> Dear all,
>>>>>>>> 
>>>>>>>> I am running nvme/perf with master branch on a EC2 instance
>>>>>>>> (Redhat, 8
>>>>>>>> CPU
>>>>>>>> cores). I noticed that the IOPS is around 1K when the queue depth
>>>>>>>> is
>>>>>>>> 1. By
>>>>>>>> simply increase the queue length to 2, the IOPS grows to over 60K.
>>>>>>>> May
>>>>>>>> I ask
>>>>>>>> if this is normal?
>>>>>>> 
>>>>>>> Is this an EC2 dedicated server (i.e. physical hardware) or a virtual
>>>>>>> machine?
>>>>>>> Is the NVMe device you're connected to backed by a local NVMe device,
>>>>>>> a
>>>>>>> remote
>>>>>>> NVMe device, or an EBS volume? Amazon presents basically all storage
>>>>>>> as
>>>>>>> NVMe
>>>>>>> devices these days, but the actual backing storage can perform
>>>>>>> differently
>>>>>>> depending on the circumstances.
>>>>>>> 
>>>>>>> In all cases I'm aware of, the NVMe device presented is not the
>>>>>>> actual
>>>>>>> NVMe SSD.
>>>>>>> It's either a virtual NVMe device emulated in software, or it's
>>>>>>> implemented on a
>>>>>>> special card that Amazon puts in between the platform and the real
>>>>>>> SSD.
>>>>>>> SPDK
>>>>>>> hasn't been tested thoroughly with Amazon's emulated NVMe devices, so
>>>>>>> there
>>>>>>> could be some issues there. Most NVMe devices historically were not
>>>>>>> validated
>>>>>>> against polled mode, ultra high performance drivers so when SPDK was
>>>>>>> first
>>>>>>> released we found a number of firmware bugs in existing SSDs that
>>>>>>> were
>>>>>>> only
>>>>>>> exposed because SPDK was so fast. We're seeing less and less of that
>>>>>>> now, but
>>>>>>> you could be hitting something like that here.
>>>>>>> _______________________________________________
>>>>>>> SPDK mailing list
>>>>>>> SPDK(a)lists.01.org
>>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> _______________________________________________
>>>>>> SPDK mailing list
>>>>>> SPDK(a)lists.01.org
>>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> SPDK mailing list
>>>>> SPDK(a)lists.01.org
>>>>> https://lists.01.org/mailman/listinfo/spdk
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> SPDK mailing list
>>>> SPDK(a)lists.01.org
>>>> https://lists.01.org/mailman/listinfo/spdk
>>>> 
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>> 
>> 
>> 
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
>> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk