Re: [SPDK] SPDK performance benchmarking guidelines and using FIO_plugin

[Kiran Dikshit <kdikshit at cloudsimple.com>]

[-- Attachment #1: Type: text/plain, Size: 3753 bytes --]

Hi Jim, 

Thank you for the pointers relating to workload. Is these tests ( throughput test , latency test ) are done on single drives or are they done on bunch of drives. 

Attached is FIO job file which i have used and getting the error. 


Thank you
Kiran

> On 31-Mar-2017, at 12:09 AM, Harris, James R <james.r.harris(a)intel.com> wrote:
> 
>> 
>> On Mar 30, 2017, at 5:43 AM, Kiran Dikshit <kdikshit(a)cloudsimple.com <mailto:kdikshit(a)cloudsimple.com>> wrote:
>> 
>> Hi All,
>> 
>> 
>> I have 2 part question relating to performance benchmarking of the SPDK and fio_plugin. 
>> 
>> 1. Is there any reference document specifying the workload types , queue depth and block size used for benchmarking the SPDK performance numbers. 
>>     Is there any OS level performance tuning to be done. It will be great if we can get some insight on the performance testbed used.
>> 
>> Note : 
>>      I did find the DPDK  performance optimisation guidelines https://lists.01.org/pipermail/spdk/2016-June/000035.html <https://lists.01.org/pipermail/spdk/2016-June/000035.html>. Which is useful. 
>> 
> 
> Hi Kiran,
> 
> Most of the tests we run for driver performance comparisons are with 4KB random reads.  This puts the most stress on the driver, since on NAND SSDs, random read performance is typically much higher than random write performance.  For throughput tests, queue depth is typically tested at 128.  For latency tests, queue depth is typically tested at 1.
> 
>> 
>> 2.  I am trying fio_plugin for benchmarking the performance, the jobs are completing with following error 
>> 
>> “nvme_pcie.c: 996:nvme_pcie_qpair_complete_pending_admin_request: ***ERROR*** the active process (pid 6700) is not found for this controller.”
>> 
>> I found that the  nvme_pcie_qpair_complete_pending_admin_request()is checking if the process exist, this is where the error message is coming from.
>> I am not sure how this process is getting killed even before completion. There is no there operation done on this system apart from running fio plugin.
>> Was any similar issue seen in the past, which might be of help to get around this error.
> 
> Could you post the contents of your fio job file?  Note that the SPDK fio plugin is currently limited to a single job.  I would expect to see an error like this if specifying multiple jobs and without -thread (meaning a separate process per job).
> 
> Thanks,
> 
> -Jim
> 
>> 
>> Below is my setup details 
>> 
>> OS : fedora 25 ( server edition )
>> Kernel version : 4.8.6-300
>> DPDK version : 6.11
>> 
>> I have attached a single nvme drive of 745GB from Intel on which i am running the FIO 
>> 
>> Below workaround were tried and the issue still persists not sure how to get around this. 
>> 
>> workaround
>> 
>> 1. Tried different workloads in FIO 
>> 2. I did detach the NVMe and attach a new NVMe drive 
>> 3. Re-installed the DPDK , SPDK and FIO tool
>> 
>> Note :
>> Following links were used to install and setup SPDK and FIO plugin
>> https://github.com/spdk/spdk <https://github.com/spdk/spdk> —> SPDK 
>> https://github.com/spdk/spdk/tree/master/examples/nvme/fio_plugin <https://github.com/spdk/spdk/tree/master/examples/nvme/fio_plugin> —> FIO_plugin
>> 
>> 
>> Thank you
>> Kiran
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
>> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk>
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org <mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk <https://lists.01.org/mailman/listinfo/spdk>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 590 bytes --]

[-- Attachment #3: seq_read.4KiB_csi.fio --]
[-- Type: application/octet-stream, Size: 305 bytes --]

[global]
time_based
size=700g
runtime=30
filename=0000.03.00.0/1
ioengine=./fio_plugin
randrepeat=0
direct=1
invalidate=1
verify=0
verify_fatal=0
ramp_time=5
blocksize=4k
zero_buffers
rw=read
numjobs=1
bwavgtime=1000
iopsavgtime=1000
log_avg_msec=1000
group_reporting

[job00-sr4k-1q]
stonewall
iodepth=1

[-- Attachment #4: attachment.html --]
[-- Type: text/html, Size: 16025 bytes --]