Re: [SPDK] SPDK fio benchmark support

Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1642 bytes --]

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 7968 bytes --]

Glad you figured it out. If you find any problems while testing with Samsung SSDs, patches are most
welcome!

On Thu, 2016-06-16 at 01:39 +0000, Raj (Rajinikanth) Pandurangan wrote:
> Apparently all 4 SSDs were sharing same PCIe that's why we were not scaling.  It's been fixed and
> able to scale now.
> 
> Thanks for your help.
> -Raj P
> 
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, June 15, 2016 4:30 PM
> To: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2
> SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there
> must be something else impacting the performance. Can you provide the following information:
> 
> 1) The Xeon E5 SKU you are using
> 2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be
> attached to the same socket that you are running perf on).
> 3) The command line parameters you are passing to the perf tool.
> 
> Thanks,
> Ben
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
> Sent: Wednesday, June 15, 2016 3:16 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com; Robert.Clevelan
> d(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> Hello Ben,
> 
> Thanks for the details.
> 
> I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1
> core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I
> introduce extra cores with 4 SSDs and it did not.
> 
> Thanks,
> -Raj P
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Wednesday, June 15, 2016 1:33 PM
> To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms
> .com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make
> it span more cores when given more jobs if someone is interested in doing the work here. At some
> point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing
> right now (NVMf target). The perf example we provide definitely does scale to many cores though,
> and it can optionally use libaio as its backend, so it is a great alternative to using fio that
> works today with no extra effort.
> 
> As far as expected performance scaling across cores with our perf tool, are you seeing the maximum
> quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding
> more cores won't improve performance. If you are not getting the performance you expect from the
> hardware I'd love to know the details of the devices and the platform.
> 
> To provide a quick example from my own system (and none of the following should be treated as
> official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe
> SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core,
> which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec
> IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the
> SSDs can provide) no matter how many cores I give the tool.
> 
> Thanks,
> Ben
> 
> -----Original Message-----
> From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
> Sent: Wednesday, June 15, 2016 11:52 AM
> To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.
> Cleveland(a)skhms.com
> Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
> Subject: RE: [SPDK] SPDK fio benchmark support
> 
> Hello All,
> 
> 1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than
> 1 core being used even when I increased numjobs?
> 
> Here is my config file.
> =============================================================
> 
> [global]
> ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
> group_reporting=1
> direct=1
> verify=0
> time_based=1
> ramp_time=0
> runtime=10
> 
> 
> [test1]
> iodepth=128
> rw=randread
> bs=4k
> #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
> filename=0000.09.00.0/1
> numjobs=2
> 
> [test2]
> iodepth=128
> rw=randread
> bs=4k
> numjobs=1
> filename=0000.08.00.0/1
> ================================================================
> 
> Has anyone succeeded using multiple cores?
> 
> 2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but
> performance numbers didn't scale.  Seemed like similar IOPS when I used single core.
> 
> Any insights would really be appreciated.
> 
> Thanks,
> -Raj P
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Friday, February 26, 2016 2:30 PM
> To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
> Subject: Re: [SPDK] SPDK fio benchmark support
> 
> As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more
> easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver
> and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.
> 
> Thanks all.
> 
> On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
> Hi Robert,
> 
> We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe
> driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme
> 
> (SPDK fio plugin is ioengine/spdk_fio.c)
> 
> If you have any questions on how to use it or interpret benchmark results, feel free to ask.
> 
> -Sam
> 
> From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
> Sent: Thursday, February 18, 2016 10:24 AM
> To: spdk(a)lists.01.org
> Subject: [SPDK] SPDK fio benchmark support
> 
> Hello all,
> 
> Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?
> 
> Thanks,
> Robert Cleveland
> The information contained in this e-mail is considered confidential of SK hynix memory solutions
> Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use,
> dissemination of the information, or copying of this message is strictly prohibited. If you are
> not the intended recipient, please contact the sender immediately and permanently delete the
> original and any copies of this email.
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> https://lists.01.org/mailman/listinfo/spdk
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Raj Pandurangan]

[-- Attachment #1: Type: text/plain, Size: 7221 bytes --]

Apparently all 4 SSDs were sharing same PCIe that's why we were not scaling.  It's been fixed and able to scale now.

Thanks for your help.
-Raj P


-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 4:30 PM
To: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there must be something else impacting the performance. Can you provide the following information:

1) The Xeon E5 SKU you are using
2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be attached to the same socket that you are running perf on).
3) The command line parameters you are passing to the perf tool.

Thanks,
Ben

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
Sent: Wednesday, June 15, 2016 3:16 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

Hello Ben,

Thanks for the details.

I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it did not.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 1:33 PM
To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 6641 bytes --]

The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there must be something else impacting the performance. Can you provide the following information:

1) The Xeon E5 SKU you are using
2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be attached to the same socket that you are running perf on).
3) The command line parameters you are passing to the perf tool.

Thanks,
Ben

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
Sent: Wednesday, June 15, 2016 3:16 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

Hello Ben,

Thanks for the details.

I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it did not.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 1:33 PM
To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Raj Pandurangan]

[-- Attachment #1: Type: text/plain, Size: 5621 bytes --]

Hello Ben,

Thanks for the details.

I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it did not.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 1:33 PM
To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4884 bytes --]

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com] 
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skhms.com
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Raj Pandurangan]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] SPDK fio benchmark support

[Sam Bradshaw]

[-- Attachment #1: Type: text/plain, Size: 1101 bytes --]

Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4309 bytes --]

Re: [SPDK] SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1379 bytes --]

Hi Robert,

FIO runs on devices exported by the Linux kernel, so significant changes would need to be made to use a user-mode stack. Instead of supporting FIO, we've written a new performance tool that understands user space NVMe devices and how to correctly leverage an asynchronous, polled mode driver. It so happens that this tool can optionally run against kernel devices as well using Linux AIO.

The tool is at examples/nvme/perf in the SPDK repository. It is built automatically when you build the NVMe driver. Does that meet your benchmarking needs?

Thanks,
Ben Walker

On Thu, 2016-02-18 at 18:23 +0000, Robert Cleveland wrote:
Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

[SPDK] SPDK fio benchmark support

[Robert Cleveland]

[-- Attachment #1: Type: text/plain, Size: 566 bytes --]

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 2277 bytes --]