[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 2834 bytes --]

On Wed, Nov 20, 2019, 19:03 <nufosmatic(a)nufosmatic.com> wrote:

> So, the engineers in our group think SPDK/DPDK/NVMe is the right solution
> for the performance storage component on the system we are designing.
> That's not the same thing as evidence, or a convincing argument with some
> simple numbers that management (and the prospect) can understand. I


You might want to review spdk performance reports at spdk.io.

've now discovered the FIO tool and the "plugin" for NVMe in SPDK and it
> looks like the right objective performance measurement tool for the job. (I
> wish I'd found FIO for a post-mortum of a rather disasterous storage
> solution we took way to long to analyze. IOmeter finally showed up the
> problem, which was never solved).
>
> So we would like to be able to walk management and the prospect through
> the world before and the world after SPDK and be able to demonstrate
> features and benefits using FIO.
>
> Here's the walk:
>
> 1 [X] FIO => NVMEe native block device
> 2 [X] FIO => "plugin" SPDK NVMEe
> 3 [   ] FIO => "block device emulation" SPDK NVMEe
> 4 [   ] FIO => Host NVMEe-oF   <=network=> Target NVMe-oF SPDK NVMe
>

I believe you're missing some basic understanding of the spdk architecture
since 2 and 3 above are very much the same thing.

>
> In prose:
>
> 1 - basic performance of any given NVMe device
> 2 - performance advantage of SPDK for NVMe device(s)
> 3 - non-network overhead of SPDK as a means to NVMe device(s)
> 4 - network-connected storage accelerated by NVMe
>
> So I think I understand the cases 1 and 2. The 'X' means I think I've
> actually done it. I think I understand the case 4 (waiting for the rest of
> the networking hardware to show up).
>
> I suspect that there is some way to present the SPDK NVMe device(s) as
> block device locally so that I might be able to do case 3, but I just have
> not found something that states plainly that this is supported or
> unsupported.
>
> 5 - Soft-RoCE-based NVMe-oF implementation to work through software issues
> when hardware is not available
>

Nvme over tcp is another option.

Regards,
Andrey

>
> In my shop the RDMA stuff is rare as hens teeth. Or getting a system with
> NVMe and RDMA in the same chassis is rarer. I have a bucketload of software
> issues to understand that don't involve the hardware. Has anybody danced
> Soft-RoCE around with SPDK and NVMe-oF? (It's a wonderful way to get your
> head wrapped around RDMA programming when you just can't get the right
> hardware to play with.)
>
>
> http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/
> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org
>

[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 4874 bytes --]

On Wed, 2019-11-20 at 16:03 +0000, nufosmatic(a)nufosmatic.com wrote:
> So, the engineers in our group think SPDK/DPDK/NVMe is the right solution for
> the performance storage component on the system we are designing. That's not
> the same thing as evidence, or a convincing argument with some simple numbers
> that management (and the prospect) can understand. I've now discovered the FIO
> tool and the "plugin" for NVMe in SPDK and it looks like the right objective
> performance measurement tool for the job. (I wish I'd found FIO for a post-
> mortum of a rather disasterous storage solution we took way to long to
> analyze. IOmeter finally showed up the problem, which was never solved).
> 
> So we would like to be able to walk management and the prospect through the
> world before and the world after SPDK and be able to demonstrate features and
> benefits using FIO.
> 
> Here's the walk:
> 
> 1 [X] FIO => NVMEe native block device
> 2 [X] FIO => "plugin" SPDK NVMEe
> 3 [   ] FIO => "block device emulation" SPDK NVMEe
> 4 [   ] FIO => Host NVMEe-oF   <=network=> Target NVMe-oF SPDK NVMe
> 
> In prose:
> 
> 1 - basic performance of any given NVMe device
> 2 - performance advantage of SPDK for NVMe device(s)
> 3 - non-network overhead of SPDK as a means to NVMe device(s)
> 4 - network-connected storage accelerated by NVMe
> 
> So I think I understand the cases 1 and 2. The 'X' means I think I've actually
> done it. I think I understand the case 4 (waiting for the rest of the
> networking hardware to show up).
> 
> I suspect that there is some way to present the SPDK NVMe device(s) as block
> device locally so that I might be able to do case 3, but I just have not found
> something that states plainly that this is supported or unsupported.

Do you mean presenting a device managed by SPDK through the Linux kernel block
stack? Like under /dev? This is possible using things like NBD or the Linux
kernel NVMe-oF initiator in loopback, but there are significant performance
downsides.

But you may also mean presenting an SPDK block device directly to fio. SPDK has
its own block storage stack (lib/bdev) with a "module" system where different
types of devices can be plugged in. The Linux kernel would call a "module" a
device driver. One of the modules is "NVMe" but there's a dozen or more of these
that can talk to all sorts of things (iSCSI, libaio, Ceph RBD, PMEM, etc. etc.).

SPDK has two fio plugins. One (the one in examples/nvme/fio_plugin) talks
directly to the SPDK NVMe driver - i.e. the very lowest part of the SPDK stack.
I think this is what you're doing with #2 above. The other plugin (the one in
examples/bdev/fio_plugin) talks to the SPDK block layer. You can use that plugin
to talk to any SPDK block device, including an NVMe device. The reason we have
both is sometimes we want to benchmark just the lower layer NVMe driver, and
sometimes we want to benchmark the whole block stack.

Note that fio itself has some performance issues. You will not be able to
measure the full performance of SPDK via fio - the tool itself is just too slow.
We provide plugins primarily because it is the industry standard tool, so it's a
quick way for people to get up and running. We recommend you use the SPDK 'perf'
tool in examples/nvme/perf to benchmark the SPDK NVMe driver instead. It has
fewer features than fio, but has much lower overhead.

Also note that using the NVMe-oF initiator is exactly the same as using the
local NVMe driver in these benchmark tools. You simply configure it with a
different transport type (tcp or rdma instead of pcie) and transport address (an
IP/port instead of a PCI bus:device:function).


> 
> 5 - Soft-RoCE-based NVMe-oF implementation to work through software issues
> when hardware is not available
> 
> In my shop the RDMA stuff is rare as hens teeth. Or getting a system with NVMe
> and RDMA in the same chassis is rarer. I have a bucketload of software issues
> to understand that don't involve the hardware. Has anybody danced Soft-RoCE
> around with SPDK and NVMe-oF? (It's a wonderful way to get your head wrapped
> around RDMA programming when you just can't get the right hardware to play
> with.)
> 
> http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience-rdma/

We use Soft-RoCE all the time and it works well enough. It is not representative
of the way RoCE actually behaves, especially during failure scenarios, and the
performance is quite bad. But it certainly is fine for basic testing.

However, as Andrey suggested, I've mostly switched to using the tcp transport
for NVMe-oF local development and testing instead.

> _______________________________________________
> SPDK mailing list -- spdk(a)lists.01.org
> To unsubscribe send an email to spdk-leave(a)lists.01.org

[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[nufosmatic]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

My use of the word "management" has to do with the guys in ties (you know, the people who if you replace "manager" with "paranoid" in their job title, it all starts to make sense) as opposed to hurding cats... uh... devices. Selling management is always harder than selling prospects. I'm just looking for that single number that will convince them. ("The purpose of computing is insight, not numbers." - RW Hamming - clearly Hamming never had to deal with managers.)

I think I just came around to understanding how to get to the NBD device with NVMe. I have not found an explicit example, but I think I have enough to give it a go.

SPDK bdev fio_plugin - would be scenario 2a, someplace between the NVMe direct and the NVME-as-block device-through-SPDK.

Soft-RoCE is only good for software testing - but if you don't have the hardware, that's what you've got.

TCP is a fine protocol for addressing the world... for the next rack over, not so much... and never, EVER for satellites...

Thanks.

[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[nufosmatic]

[-- Attachment #1: Type: text/plain, Size: 243 bytes --]

Now I think I understand why the NDB stuff was not clear. I am directed to use CentOS 7.5, which is based upon Red Hat 7.5, which does not ship with the NDB driver built, and apparently does not even maintain the NDB driver.

Hurding cats...

[SPDK] Re: Management benchmarking of NVMe in SPDK with FIO

[Luse, Paul E]

[-- Attachment #1: Type: text/plain, Size: 728 bytes --]

FYI you won't want to use NBD for anything performance related. It's very slow and was implemented simply to provide a way to use some basic linux tools that you'd otherwise not be able to use with SPDK

On 11/21/19, 9:54 AM, "nufosmatic(a)nufosmatic.com" <nufosmatic(a)nufosmatic.com> wrote:

    Now I think I understand why the NDB stuff was not clear. I am directed to use CentOS 7.5, which is based upon Red Hat 7.5, which does not ship with the NDB driver built, and apparently does not even maintain the NDB driver.
    
    Hurding cats...
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org