[SPDK] 答复: 答复: SPDK fio benchmark support

[SPDK] 答复: 答复: SPDK fio benchmark support

[Li Tianxiang]

[-- Attachment #1: Type: text/plain, Size: 10007 bytes --]

Thx,

Actually，my question is Could one thread & one qpair achieve the

max  performance？Perf application adopts  the manner of one

threads per namespace，and many NVMe SSDs not support

multi-ns yet.


It makes sense that perf gets the performance better or comparable to

Kernel driver,because the Submission Queues(or Qpair) in NVMe are

still software queues, Venders still  hide the real parallelism as before.


If there is any general benchmarks such as fio, the result could be more

convincible.

And the mentioned fio plug-in needs to add multi jobs support.


-----

It could be interesting if NVMe CMD sets could manage the mapping

between Qpair and real parallelism.

-----


________________________________
发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <benjamin.walker(a)intel.com>
发送时间: 2016年6月16日 15:40
收件人: spdk(a)lists.01.org
主题: Re: [SPDK] 答复: SPDK fio benchmark support

The perf tool allocates 1 queue pair per thread (worker_fn is the thread entry point in the code) and there is 1 thread per core. Our findings on all of the devices we've tested has been that using multiple queue pairs does not increase the total performance of the SSD. The full quoted SSD hardware performance can be achieved with a single queue pair. This finding, however, is completely device specific. Different SSD controllers may have different behavior.

Using multiple queue pairs is a big advantage for the host software (i.e. SPDK) though. Using 1 queue pair per thread allows the host software to submit I/O in parallel with no locks.

On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote:

Hi，

I noticed that the perf  app.  only  allocates one qpair.

using :
"ns_ctx->u.nvme.qpair = spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.nvme.ctrlr, 0);"

Does more  qpair help to the  performance?

And P3700  has about 30 submission  queues default.


________________________________________
发件人: SPDK <spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>> 代表 Walker, Benjamin <benjamin.walker(a)intel.com<mailto:benjamin.walker(a)intel.com>>
发送时间: 2016年6月15日 23:29
收件人: Storage Performance Development Kit; sbradshaw(a)micron.com<mailto:sbradshaw(a)micron.com>; Robert.Cleveland(a)skhms.com<mailto:Robert.Cleveland(a)skhms.com>
主题: Re: [SPDK] SPDK fio benchmark support

The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't, there must be something else impacting the performance. Can you provide the following information:

1) The Xeon E5 SKU you are using
2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be attached to the same socket that you are running perf on).
3) The command line parameters you are passing to the perf tool.

Thanks,
Ben

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
Sent: Wednesday, June 15, 2016 3:16 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>; sbradshaw(a)micron.com<mailto:sbradshaw(a)micron.com>; Robert.Cleveland(a)skhms.com<mailto:Robert.Cleveland(a)skhms.com>
Subject: Re: [SPDK] SPDK fio benchmark support

Hello Ben,

Thanks for the details.

I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1 core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I introduce extra cores with 4 SSDs and it did not.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, June 15, 2016 1:33 PM
To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com<mailto:sbradshaw(a)micron.com>; spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>; Robert.Cleveland(a)skhms.com<mailto:Robert.Cleveland(a)skhms.com>
Subject: Re: [SPDK] SPDK fio benchmark support

The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to make it span more cores when given more jobs if someone is interested in doing the work here. At some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work we're doing right now (NVMf target). The perf example we provide definitely does scale to many cores though, and it can optionally use libaio as its backend, so it is a great alternative to using fio that works today with no extra effort.

As far as expected performance scaling across cores with our perf tool, are you seeing the maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are, adding more cores won't improve performance. If you are not getting the performance you expect from the hardware I'd love to know the details of the devices and the platform.

To provide a quick example from my own system (and none of the following should be treated as official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core, which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that the SSDs can provide) no matter how many cores I give the tool.

Thanks,
Ben

-----Original Message-----
From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]
Sent: Wednesday, June 15, 2016 11:52 AM
To: Walker, Benjamin <benjamin.walker(a)intel.com<mailto:benjamin.walker(a)intel.com>>; sbradshaw(a)micron.com<mailto:sbradshaw(a)micron.com>; spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>; Robert.Cleveland(a)skhms.com<mailto:Robert.Cleveland(a)skhms.com>
Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com<mailto:rajini.pandu(a)samsung.com>>
Subject: RE: [SPDK] SPDK fio benchmark support

Hello All,

1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more than 1 core being used even when I increased numjobs?

Here is my config file.
=============================================================

[global]
ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
group_reporting=1
direct=1
verify=0
time_based=1
ramp_time=0
runtime=10


[test1]
iodepth=128
rw=randread
bs=4k
#filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
filename=0000.09.00.0/1
numjobs=2

[test2]
iodepth=128
rw=randread
bs=4k
numjobs=1
filename=0000.08.00.0/1
================================================================

Has anyone succeeded using multiple cores?

2. I have also tried with 'perf' and I noticed that even though multiple cores were being used but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.

Any insights would really be appreciated.

Thanks,
-Raj P

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Friday, February 26, 2016 2:30 PM
To: sbradshaw(a)micron.com<mailto:sbradshaw(a)micron.com>; spdk(a)ml01.01.org<mailto:spdk(a)ml01.01.org>; Robert.Cleveland(a)skhms.com<mailto:Robert.Cleveland(a)skhms.com>
Subject: Re: [SPDK] SPDK fio benchmark support

As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more easily support vfio. That makes it a lot easier to switch back and forth between the unvme driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has suggestions.

Thanks all.

On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
Hi Robert,

We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme

(SPDK fio plugin is ioengine/spdk_fio.c)

If you have any questions on how to use it or interpret benchmark results, feel free to ask.

-Sam

From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
Sent: Thursday, February 18, 2016 10:24 AM
To: spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>
Subject: [SPDK] SPDK fio benchmark support

Hello all,

Has anyone done any work to make it easy to benchmark polling mode driver with something like FIO?

Thanks,
Robert Cleveland
The information contained in this e-mail is considered confidential of SK hynix memory solutions Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use, dissemination of the information, or copying of this message is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email.

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org><mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk
_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 12143 bytes --]

Re: [SPDK] 答复: 答复: SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 13226 bytes --]

On Fri, 2016-06-17 at 00:57 +0000, Li Tianxiang wrote:
> Thx,
> Actually，my question is Could one thread & one qpair achieve the
> max  performance？

Yes, on the devices we've tested with (Intel). There are people from basically all of the major SSD
vendors on this mailing list too and as far as I'm aware, they can get the full performance from
their devices with SPDK using only one thread and one queue pair. It ultimately depends on the
device, though. The one queue pair per thread model is just how we chose to do things in our perf
tool as well - a user application can choose any mapping of queues to devices to threads that they
want.

> Perf application adopts  the manner of one 
> threads per namespace，and many NVMe SSDs not support 
> multi-ns yet.

I suspect on most (or all) devices, namespaces are purely logical abstractions and have nothing to
do with physical NAND dies or channels. Therefore, one namespace or many namespaces won't have any
impact on performance.

> 
> It makes sense that perf gets the performance better or comparable to 
> Kernel driver,because the Submission Queues(or Qpair) in NVMe are 
> still software queues, Venders still  hide the real parallelism as before.

SPDK gets better performance than the kernel for a number of reasons (polling, zero copy, no
syscalls, no locking, streamlined I/O path), but I'm not sure what you are saying here. The queue
pair we expose from our NVMe driver really is the hardware queue from the NVMe controller - it is
not a software abstraction that we're presenting. The receiving end of the queue we are exposing is
really being processed by the NVMe controller ASIC on the other side. You can even elect to allocate
the queue memory from the NVMe controller's memory, if the controller supports that feature. The
queues are not physical NAND channels, if that's what you mean, but NAND doesn't support block-
granularity rewriting, so there must be some flash translation layer to translate logical block
writes to open physical locations on the NAND.

I also disagree that vendors are hiding any parallelism. If you think of an SSD as having two
distinct processing steps - a flash translation layer and a backend NAND layer - and you assume that
the available parallelism is bounded by that backend NAND layer, then it doesn't really matter if
the flash translation layer is parallel or not, as long as it can fully saturate the backend with
tasks. I would posit that you can't even experimentally tell the difference between an SSD with a
parallel flash translation layer and one with a serialized one, as long as they are both able to
keep all of the NAND channels full.

> 
> If there is any general benchmarks such as fio, the result could be more 
> convincible.
> And the mentioned fio plug-in needs to add multi jobs support.

Is there some inherent reason that fio is a more convincing test than our perf tool? I'm typically
convinced by benchmarks that are open and easy to understand. Our perf tool is ~1100 lines of code
and fio contains ~71k. I'm a big fan of fio as a benchmarking tool and I recommend its use, but
isn't it much easier to audit exactly what our perf tool is doing to build trust?

At some point we'll definitely add multi-core support to our fio plugin though, and if someone
volunteers to do it sooner we'd happily accept the patch.

> 
> -----
> It could be interesting if NVMe CMD sets could manage the mapping
> between Qpair and real parallelism.

If by "real parallelism" you mean actual NAND channels, it's impossible to expose those directly and
still present a block interface. The rewrite rules are totally different.

> -----
> 
> 
> 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <benjamin.walker(a)intel.com>
> 发送时间: 2016年6月16日 15:40
> 收件人: spdk(a)lists.01.org
> 主题: Re: [SPDK] 答复: SPDK fio benchmark support
>  
> The perf tool allocates 1 queue pair per thread (worker_fn is the thread entry point in the code)
> and there is 1 thread per core. Our findings on all of the devices we've tested has been that
> using multiple queue pairs does not increase the total performance of the SSD. The full quoted SSD
> hardware performance can be achieved with a single queue pair. This finding, however, is
> completely device specific. Different SSD controllers may have different behavior.
> 
> Using multiple queue pairs is a big advantage for the host software (i.e. SPDK) though. Using 1
> queue pair per thread allows the host software to submit I/O in parallel with no locks.
> 
> On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote:
> > Hi，
> > 
> > I noticed that the perf  app.  only  allocates one qpair.
> > 
> > using : 
> > "ns_ctx->u.nvme.qpair = spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.nvme.ctrlr, 0);"
> > 
> > Does more  qpair help to the  performance?
> > 
> > And P3700  has about 30 submission  queues default.
> > 
> > 
> > ________________________________________
> > 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <benjamin.walker(a)intel.com>
> > 发送时间: 2016年6月15日 23:29
> > 收件人: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
> > 主题: Re: [SPDK] SPDK fio benchmark support
> > 
> > The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS with
> > 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't,
> > there must be something else impacting the performance. Can you provide the following
> > information:
> > 
> > 1) The Xeon E5 SKU you are using
> > 2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be
> > attached to the same socket that you are running perf on).
> > 3) The command line parameters you are passing to the perf tool.
> > 
> > Thanks,
> > Ben
> > 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
> > Sent: Wednesday, June 15, 2016 3:16 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com; Robert.Clevel
> > and(a)skhms.com
> > Subject: Re: [SPDK] SPDK fio benchmark support
> > 
> > Hello Ben,
> > 
> > Thanks for the details.
> > 
> > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1
> > core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when I
> > introduce extra cores with 4 SSDs and it did not.
> > 
> > Thanks,
> > -Raj P
> > 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> > Sent: Wednesday, June 15, 2016 1:33 PM
> > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland(a)skh
> > ms.com
> > Subject: Re: [SPDK] SPDK fio benchmark support
> > 
> > The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to
> > make it span more cores when given more jobs if someone is interested in doing the work here. At
> > some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work
> > we're doing right now (NVMf target). The perf example we provide definitely does scale to many
> > cores though, and it can optionally use libaio as its backend, so it is a great alternative to
> > using fio that works today with no extra effort.
> > 
> > As far as expected performance scaling across cores with our perf tool, are you seeing the
> > maximum quoted IOPS for all of the device(s) on the system with only a single core? If you are,
> > adding more cores won't improve performance. If you are not getting the performance you expect
> > from the hardware I'd love to know the details of the devices and the platform.
> > 
> > To provide a quick example from my own system (and none of the following should be treated as
> > official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700 NVMe
> > SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1 core,
> > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware
> > spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum that
> > the SSDs can provide) no matter how many cores I give the tool.
> > 
> > Thanks,
> > Ben
> > 
> > -----Original Message-----
> > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]
> > Sent: Wednesday, June 15, 2016 11:52 AM
> > To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org; Rober
> > t.Cleveland(a)skhms.com
> > Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
> > Subject: RE: [SPDK] SPDK fio benchmark support
> > 
> > Hello All,
> > 
> > 1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more
> > than 1 core being used even when I increased numjobs?
> > 
> > Here is my config file.
> > =============================================================
> > 
> > [global]
> > ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
> > group_reporting=1
> > direct=1
> > verify=0
> > time_based=1
> > ramp_time=0
> > runtime=10
> > 
> > 
> > [test1]
> > iodepth=128
> > rw=randread
> > bs=4k
> > #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
> > filename=0000.09.00.0/1
> > numjobs=2
> > 
> > [test2]
> > iodepth=128
> > rw=randread
> > bs=4k
> > numjobs=1
> > filename=0000.08.00.0/1
> > ================================================================
> > 
> > Has anyone succeeded using multiple cores?
> > 
> > 2. I have also tried with 'perf' and I noticed that even though multiple cores were being used
> > but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.
> > 
> > Any insights would really be appreciated.
> > 
> > Thanks,
> > -Raj P
> > 
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> > Sent: Friday, February 26, 2016 2:30 PM
> > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
> > Subject: Re: [SPDK] SPDK fio benchmark support
> > 
> > As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now more
> > easily support vfio. That makes it a lot easier to switch back and forth between the unvme
> > driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has
> > suggestions.
> > 
> > Thanks all.
> > 
> > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
> > Hi Robert,
> > 
> > We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe
> > driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme
> > 
> > (SPDK fio plugin is ioengine/spdk_fio.c)
> > 
> > If you have any questions on how to use it or interpret benchmark results, feel free to ask.
> > 
> > -Sam
> > 
> > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
> > Sent: Thursday, February 18, 2016 10:24 AM
> > To: spdk(a)lists.01.org
> > Subject: [SPDK] SPDK fio benchmark support
> > 
> > Hello all,
> > 
> > Has anyone done any work to make it easy to benchmark polling mode driver with something like
> > FIO?
> > 
> > Thanks,
> > Robert Cleveland
> > The information contained in this e-mail is considered confidential of SK hynix memory solutions
> > Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized use,
> > dissemination of the information, or copying of this message is strictly prohibited. If you are
> > not the intended recipient, please contact the sender immediately and permanently delete the
> > original and any copies of this email.
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> > https://lists.01.org/mailman/listinfo/spdk
> > 
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] 答复: 答复: SPDK fio benchmark support

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 12116 bytes --]

On Fri, Jun 17, 2016 at 7:59 PM, Walker, Benjamin <benjamin.walker(a)intel.com
> wrote:

> On Fri, 2016-06-17 at 00:57 +0000, Li Tianxiang wrote:
> > Thx,
>
>
> > If there is any general benchmarks such as fio, the result could be more
> > convincible.
> > And the mentioned fio plug-in needs to add multi jobs support.
>
> Is there some inherent reason that fio is a more convincing test than our
> perf tool?


Because fio has de-facto became an industry-standard benchmark in the
recent years, and thus is definitely more convincing than any single-vendor
tool. Furthermore, in flexibility/coverage fio delivers what one would
expect from a full-fledged storage benchmark, while perf is essentially an
spdk stress tool (no offense meant).

I'm typically
> convinced by benchmarks that are open and easy to understand. Our perf
> tool is ~1100 lines of code
> and fio contains ~71k. I'm a big fan of fio as a benchmarking tool and I
> recommend its use, but
> isn't it much easier to audit exactly what our perf tool is doing to build
> trust?
>
> At some point we'll definitely add multi-core support to our fio plugin
> though,


While debatable with a single device where one can argue, as you do above,
that single thread suffices (although my mileage is different), it's
clearly a must for multiple device case.


> and if someone
> volunteers to do it sooner we'd happily accept the patch.
>
> >
> > -----
> > It could be interesting if NVMe CMD sets could manage the mapping
> > between Qpair and real parallelism.
>
> If by "real parallelism" you mean actual NAND channels, it's impossible to
> expose those directly and
> still present a block interface.


Non-FTL SSDs do expose the internal SSD composition and thus allow the
interested/capable application to fully control/exploit the internal SSD
parallelism.

Regards,
Andrey



> The rewrite rules are totally different.
>
> > -----
> >
> >
> > 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <
> benjamin.walker(a)intel.com>
> > 发送时间: 2016年6月16日 15:40
> > 收件人: spdk(a)lists.01.org
> > 主题: Re: [SPDK] 答复: SPDK fio benchmark support
> >
> > The perf tool allocates 1 queue pair per thread (worker_fn is the thread
> entry point in the code)
> > and there is 1 thread per core. Our findings on all of the devices we've
> tested has been that
> > using multiple queue pairs does not increase the total performance of
> the SSD. The full quoted SSD
> > hardware performance can be achieved with a single queue pair. This
> finding, however, is
> > completely device specific. Different SSD controllers may have different
> behavior.
> >
> > Using multiple queue pairs is a big advantage for the host software
> (i.e. SPDK) though. Using 1
> > queue pair per thread allows the host software to submit I/O in parallel
> with no locks.
> >
> > On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote:
> > > Hi，
> > >
> > > I noticed that the perf  app.  only  allocates one qpair.
> > >
> > > using :
> > > "ns_ctx->u.nvme.qpair =
> spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.nvme.ctrlr, 0);"
> > >
> > > Does more  qpair help to the  performance?
> > >
> > > And P3700  has about 30 submission  queues default.
> > >
> > >
> > > ________________________________________
> > > 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <
> benjamin.walker(a)intel.com>
> > > 发送时间: 2016年6月15日 23:29
> > > 收件人: Storage Performance Development Kit; sbradshaw(a)micron.com;
> Robert.Cleveland(a)skhms.com
> > > 主题: Re: [SPDK] SPDK fio benchmark support
> > >
> > > The perf application should scale linearly with CPU cores - so if you
> get 1.6 million IOPS with
> > > 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2
> cores. Since you aren't,
> > > there must be something else impacting the performance. Can you
> provide the following
> > > information:
> > >
> > > 1) The Xeon E5 SKU you are using
> > > 2) The number of CPU sockets and which socket the devices are attached
> to (all 4 need to be
> > > attached to the same socket that you are running perf on).
> > > 3) The command line parameters you are passing to the perf tool.
> > >
> > > Thanks,
> > > Ben
> > >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj
> (Rajinikanth) Pandurangan
> > > Sent: Wednesday, June 15, 2016 3:16 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>;
> sbradshaw(a)micron.com; Robert.Clevel
> > > and(a)skhms.com
> > > Subject: Re: [SPDK] SPDK fio benchmark support
> > >
> > > Hello Ben,
> > >
> > > Thanks for the details.
> > >
> > > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs
> (~1.6 million) on 1
> > > core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect
> the IOPS to go up when I
> > > introduce extra cores with 4 SSDs and it did not.
> > >
> > > Thanks,
> > > -Raj P
> > >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> Benjamin
> > > Sent: Wednesday, June 15, 2016 1:33 PM
> > > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com;
> spdk(a)lists.01.org; Robert.Cleveland(a)skh
> > > ms.com
> > > Subject: Re: [SPDK] SPDK fio benchmark support
> > >
> > > The fio plugin for SPDK is new and was only tested for a single job.
> We'd welcome patches to
> > > make it span more cores when given more jobs if someone is interested
> in doing the work here. At
> > > some point, I'm sure we'll get to it ourselves, but it's lower
> priority than the other work
> > > we're doing right now (NVMf target). The perf example we provide
> definitely does scale to many
> > > cores though, and it can optionally use libaio as its backend, so it
> is a great alternative to
> > > using fio that works today with no extra effort.
> > >
> > > As far as expected performance scaling across cores with our perf
> tool, are you seeing the
> > > maximum quoted IOPS for all of the device(s) on the system with only a
> single core? If you are,
> > > adding more cores won't improve performance. If you are not getting
> the performance you expect
> > > from the hardware I'd love to know the details of the devices and the
> platform.
> > >
> > > To provide a quick example from my own system (and none of the
> following should be treated as
> > > official benchmarking numbers for any reason), I have a Haswell Xeon
> E7 with 8 Intel P3700 NVMe
> > > SSDs attached. Running the perf tool on my system yields about 3
> million 4k read IOPs on 1 core,
> > > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores
> gives me the full hardware
> > > spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same
> performance (the maximum that
> > > the SSDs can provide) no matter how many cores I give the tool.
> > >
> > > Thanks,
> > > Ben
> > >
> > > -----Original Message-----
> > > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]
> > > Sent: Wednesday, June 15, 2016 11:52 AM
> > > To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com;
> spdk(a)lists.01.org; Rober
> > > t.Cleveland(a)skhms.com
> > > Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
> > > Subject: RE: [SPDK] SPDK fio benchmark support
> > >
> > > Hello All,
> > >
> > > 1. I tried to run SPDK fio with multiple nvme drives.  But for some
> reason, I don't see more
> > > than 1 core being used even when I increased numjobs?
> > >
> > > Here is my config file.
> > > =============================================================
> > >
> > > [global]
> > >
> ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
> > > group_reporting=1
> > > direct=1
> > > verify=0
> > > time_based=1
> > > ramp_time=0
> > > runtime=10
> > >
> > >
> > > [test1]
> > > iodepth=128
> > > rw=randread
> > > bs=4k
> > > #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
> > > filename=0000.09.00.0/1
> > > numjobs=2
> > >
> > > [test2]
> > > iodepth=128
> > > rw=randread
> > > bs=4k
> > > numjobs=1
> > > filename=0000.08.00.0/1
> > > ================================================================
> > >
> > > Has anyone succeeded using multiple cores?
> > >
> > > 2. I have also tried with 'perf' and I noticed that even though
> multiple cores were being used
> > > but performance numbers didn't scale.  Seemed like similar IOPS when I
> used single core.
> > >
> > > Any insights would really be appreciated.
> > >
> > > Thanks,
> > > -Raj P
> > >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker,
> Benjamin
> > > Sent: Friday, February 26, 2016 2:30 PM
> > > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
> > > Subject: Re: [SPDK] SPDK fio benchmark support
> > >
> > > As a follow up to this, I pushed some improvements to SPDK's setup
> scripts so that we now more
> > > easily support vfio. That makes it a lot easier to switch back and
> forth between the unvme
> > > driver and the SPDK driver. We'd happily accept more patches to
> improve vfio if anyone has
> > > suggestions.
> > >
> > > Thanks all.
> > >
> > > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
> > > Hi Robert,
> > >
> > > We built a fio ioengine plugin for benchmarking SPDK as well as our
> own userspace NVMe
> > > driver.  Source and documentation are available here:
> https://github.com/MicronSSD/unvme
> > >
> > > (SPDK fio plugin is ioengine/spdk_fio.c)
> > >
> > > If you have any questions on how to use it or interpret benchmark
> results, feel free to ask.
> > >
> > > -Sam
> > >
> > > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert
> Cleveland
> > > Sent: Thursday, February 18, 2016 10:24 AM
> > > To: spdk(a)lists.01.org
> > > Subject: [SPDK] SPDK fio benchmark support
> > >
> > > Hello all,
> > >
> > > Has anyone done any work to make it easy to benchmark polling mode
> driver with something like
> > > FIO?
> > >
> > > Thanks,
> > > Robert Cleveland
> > > The information contained in this e-mail is considered confidential of
> SK hynix memory solutions
> > > Inc. and intended only for the persons addressed or copied in this
> e-mail. Any unauthorized use,
> > > dissemination of the information, or copying of this message is
> strictly prohibited. If you are
> > > not the intended recipient, please contact the sender immediately and
> permanently delete the
> > > original and any copies of this email.
> > >
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 17930 bytes --]

Re: [SPDK] 答复: 答复: SPDK fio benchmark support

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 13236 bytes --]

On Sat, 2016-06-18 at 22:09 +0300, Andrey Kuzmin wrote:
> On Fri, Jun 17, 2016 at 7:59 PM, Walker, Benjamin <benjamin.walker(a)intel.com> wrote:
> > On Fri, 2016-06-17 at 00:57 +0000, Li Tianxiang wrote:
> > > Thx,
> 
> >
> > If there is any general benchmarks such as fio, the result could be more 
> > convincible.
> > And the mentioned fio plug-in needs to add multi jobs support.
> 
> Is there some inherent reason that fio is a more convincing test than our perf tool? 
> Because fio has de-facto became an industry-standard benchmark in the recent years, and thus is
> definitely more convincing than any single-vendor tool. Furthermore, in flexibility/coverage fio
> delivers what one would expect from a full-fledged storage benchmark, while perf is essentially an
> spdk stress tool (no offense meant).
> 
> > I'm typically
> > convinced by benchmarks that are open and easy to understand. Our perf tool is ~1100 lines of
> > code
> > and fio contains ~71k. I'm a big fan of fio as a benchmarking tool and I recommend its use, but
> > isn't it much easier to audit exactly what our perf tool is doing to build trust?
> > 
> > At some point we'll definitely add multi-core support to our fio plugin though, 
> While debatable with a single device where one can argue, as you do above, that single thread
> suffices (although my mileage is different), it's clearly a must for multiple device case.
>  
> > and if someone
> > volunteers to do it sooner we'd happily accept the patch.
> > 
> > >
> > > -----
> > > It could be interesting if NVMe CMD sets could manage the mapping
> > > between Qpair and real parallelism.
> > 
> > If by "real parallelism" you mean actual NAND channels, it's impossible to expose those directly
> > and
> > still present a block interface. 
> Non-FTL SSDs do expose the internal SSD composition and thus allow the interested/capable
> application to fully control/exploit the internal SSD parallelism.

I encourage you or other interested parties to consider implementing the LightNVM commands as a
header file (at include/spdk/lightnvm.h), doing any minor enabling work so that SPDK understands
alternate command sets (may be no work at all), and submitting a patch back to us. That would make
SPDK a great platform for FTL research and we'd support that effort. The current team that works on
SPDK is focused on production deployments of block devices though, so the only avenue I see right
now for this to get done is through community contributions.

> 
> Regards,
> Andrey
> 
>  
> > The rewrite rules are totally different.
> > 
> > > -----
> > >
> > >
> > > 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <benjamin.walker(a)intel.com>
> > > 发送时间: 2016年6月16日 15:40
> > > 收件人: spdk(a)lists.01.org
> > > 主题: Re: [SPDK] 答复: SPDK fio benchmark support
> > >  
> > > The perf tool allocates 1 queue pair per thread (worker_fn is the thread entry point in the
> > code)
> > > and there is 1 thread per core. Our findings on all of the devices we've tested has been that
> > > using multiple queue pairs does not increase the total performance of the SSD. The full quoted
> > SSD
> > > hardware performance can be achieved with a single queue pair. This finding, however, is
> > > completely device specific. Different SSD controllers may have different behavior.
> > >
> > > Using multiple queue pairs is a big advantage for the host software (i.e. SPDK) though. Using
> > 1
> > > queue pair per thread allows the host software to submit I/O in parallel with no locks.
> > >
> > > On Thu, 2016-06-16 at 14:24 +0000, Li Tianxiang wrote:
> > > > Hi，
> > > >
> > > > I noticed that the perf  app.  only  allocates one qpair.
> > > >
> > > > using : 
> > > > "ns_ctx->u.nvme.qpair = spdk_nvme_ctrlr_alloc_io_qpair(ns_ctx->entry->u.nvme.ctrlr, 0);"
> > > >
> > > > Does more  qpair help to the  performance?
> > > >
> > > > And P3700  has about 30 submission  queues default.
> > > >
> > > >
> > > > ________________________________________
> > > > 发件人: SPDK <spdk-bounces(a)lists.01.org> 代表 Walker, Benjamin <benjamin.walker(a)intel.com>
> > > > 发送时间: 2016年6月15日 23:29
> > > > 收件人: Storage Performance Development Kit; sbradshaw(a)micron.com; Robert.Cleveland(a)skhms.com
> > > > 主题: Re: [SPDK] SPDK fio benchmark support
> > > >
> > > > The perf application should scale linearly with CPU cores - so if you get 1.6 million IOPS
> > with
> > > > 2 SSDs on 1 core, you should get 3.2 million IOPS with 4 SSDs on 2 cores. Since you aren't,
> > > > there must be something else impacting the performance. Can you provide the following
> > > > information:
> > > >
> > > > 1) The Xeon E5 SKU you are using
> > > > 2) The number of CPU sockets and which socket the devices are attached to (all 4 need to be
> > > > attached to the same socket that you are running perf on).
> > > > 3) The command line parameters you are passing to the perf tool.
> > > >
> > > > Thanks,
> > > > Ben
> > > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Raj (Rajinikanth) Pandurangan
> > > > Sent: Wednesday, June 15, 2016 3:16 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>; sbradshaw(a)micron.com;
> > Robert.Clevel
> > > > and(a)skhms.com
> > > > Subject: Re: [SPDK] SPDK fio benchmark support
> > > >
> > > > Hello Ben,
> > > >
> > > > Thanks for the details.
> > > >
> > > > I have Intel Xeon E5 system and able to saturate 2 Samsung NVMe SSDs (~1.6 million) on 1
> > > > core.   IOPS are same even when I had 4 SSDs on 1 core.  But I expect the IOPS to go up when
> > I
> > > > introduce extra cores with 4 SSDs and it did not.
> > > >
> > > > Thanks,
> > > > -Raj P
> > > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> > > > Sent: Wednesday, June 15, 2016 1:33 PM
> > > > To: Raj (Rajinikanth) Pandurangan; sbradshaw(a)micron.com; spdk(a)lists.01.org; Robert.Cleveland
> > @skh
> > > > ms.com
> > > > Subject: Re: [SPDK] SPDK fio benchmark support
> > > >
> > > > The fio plugin for SPDK is new and was only tested for a single job. We'd welcome patches to
> > > > make it span more cores when given more jobs if someone is interested in doing the work
> > here. At
> > > > some point, I'm sure we'll get to it ourselves, but it's lower priority than the other work
> > > > we're doing right now (NVMf target). The perf example we provide definitely does scale to
> > many
> > > > cores though, and it can optionally use libaio as its backend, so it is a great alternative
> > to
> > > > using fio that works today with no extra effort.
> > > >
> > > > As far as expected performance scaling across cores with our perf tool, are you seeing the
> > > > maximum quoted IOPS for all of the device(s) on the system with only a single core? If you
> > are,
> > > > adding more cores won't improve performance. If you are not getting the performance you
> > expect
> > > > from the hardware I'd love to know the details of the devices and the platform.
> > > >
> > > > To provide a quick example from my own system (and none of the following should be treated
> > as
> > > > official benchmarking numbers for any reason), I have a Haswell Xeon E7 with 8 Intel P3700
> > NVMe
> > > > SSDs attached. Running the perf tool on my system yields about 3 million 4k read IOPs on 1
> > core,
> > > > which is the equivalent of 6 or 7 of my SSDs. Moving to two cores gives me the full hardware
> > > > spec IOPS for 8 SSDs. If I only test against 4 SSDs I get the same performance (the maximum
> > that
> > > > the SSDs can provide) no matter how many cores I give the tool.
> > > >
> > > > Thanks,
> > > > Ben
> > > >
> > > > -----Original Message-----
> > > > From: Raj (Rajinikanth) Pandurangan [mailto:rajini.pandu(a)samsung.com]
> > > > Sent: Wednesday, June 15, 2016 11:52 AM
> > > > To: Walker, Benjamin <benjamin.walker(a)intel.com>; sbradshaw(a)micron.com; spdk(a)lists.01.org;
> > Rober
> > > > t.Cleveland(a)skhms.com
> > > > Cc: Raj (Rajinikanth) Pandurangan <rajini.pandu(a)samsung.com>
> > > > Subject: RE: [SPDK] SPDK fio benchmark support
> > > >
> > > > Hello All,
> > > >
> > > > 1. I tried to run SPDK fio with multiple nvme drives.  But for some reason, I don't see more
> > > > than 1 core being used even when I increased numjobs?
> > > >
> > > > Here is my config file.
> > > > =============================================================
> > > >
> > > > [global]
> > > > ioengine=/spdk-master_temp_experiment/examples/nvme/fio_plugin/fio_plugin
> > > > group_reporting=1
> > > > direct=1
> > > > verify=0
> > > > time_based=1
> > > > ramp_time=0
> > > > runtime=10
> > > >
> > > >
> > > > [test1]
> > > > iodepth=128
> > > > rw=randread
> > > > bs=4k
> > > > #filename=0000.09.00.0/1:0000.07.00.0/1:0000.0a.00.0/1
> > > > filename=0000.09.00.0/1
> > > > numjobs=2
> > > >
> > > > [test2]
> > > > iodepth=128
> > > > rw=randread
> > > > bs=4k
> > > > numjobs=1
> > > > filename=0000.08.00.0/1
> > > > ================================================================
> > > >
> > > > Has anyone succeeded using multiple cores?
> > > >
> > > > 2. I have also tried with 'perf' and I noticed that even though multiple cores were being
> > used
> > > > but performance numbers didn't scale.  Seemed like similar IOPS when I used single core.
> > > >
> > > > Any insights would really be appreciated.
> > > >
> > > > Thanks,
> > > > -Raj P
> > > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> > > > Sent: Friday, February 26, 2016 2:30 PM
> > > > To: sbradshaw(a)micron.com; spdk(a)ml01.01.org; Robert.Cleveland(a)skhms.com
> > > > Subject: Re: [SPDK] SPDK fio benchmark support
> > > >
> > > > As a follow up to this, I pushed some improvements to SPDK's setup scripts so that we now
> > more
> > > > easily support vfio. That makes it a lot easier to switch back and forth between the unvme
> > > > driver and the SPDK driver. We'd happily accept more patches to improve vfio if anyone has
> > > > suggestions.
> > > >
> > > > Thanks all.
> > > >
> > > > On Thu, 2016-02-18 at 18:59 +0000, Sam Bradshaw (sbradshaw) wrote:
> > > > Hi Robert,
> > > >
> > > > We built a fio ioengine plugin for benchmarking SPDK as well as our own userspace NVMe
> > > > driver.  Source and documentation are available here:  https://github.com/MicronSSD/unvme
> > > >
> > > > (SPDK fio plugin is ioengine/spdk_fio.c)
> > > >
> > > > If you have any questions on how to use it or interpret benchmark results, feel free to ask.
> > > >
> > > > -Sam
> > > >
> > > > From: SPDK [mailto:spdk-bounces(a)ml01.01.org] On Behalf Of Robert Cleveland
> > > > Sent: Thursday, February 18, 2016 10:24 AM
> > > > To: spdk(a)lists.01.org
> > > > Subject: [SPDK] SPDK fio benchmark support
> > > >
> > > > Hello all,
> > > >
> > > > Has anyone done any work to make it easy to benchmark polling mode driver with something
> > like
> > > > FIO?
> > > >
> > > > Thanks,
> > > > Robert Cleveland
> > > > The information contained in this e-mail is considered confidential of SK hynix memory
> > solutions
> > > > Inc. and intended only for the persons addressed or copied in this e-mail. Any unauthorized
> > use,
> > > > dissemination of the information, or copying of this message is strictly prohibited. If you
> > are
> > > > not the intended recipient, please contact the sender immediately and permanently delete the
> > > > original and any copies of this email.
> > > >
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >