From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3745629690151786889==" MIME-Version: 1.0 From: Walker, Benjamin Subject: [SPDK] Re: Management benchmarking of NVMe in SPDK with FIO Date: Wed, 20 Nov 2019 18:54:35 +0000 Message-ID: <9b7b57779c665981820c7df31d17501bcd979e4e.camel@intel.com> In-Reply-To: 20191120160357.2967.65251@new-ml01.vlan13.01.org List-ID: To: spdk@lists.01.org --===============3745629690151786889== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, 2019-11-20 at 16:03 +0000, nufosmatic(a)nufosmatic.com wrote: > So, the engineers in our group think SPDK/DPDK/NVMe is the right solution= for > the performance storage component on the system we are designing. That's = not > the same thing as evidence, or a convincing argument with some simple num= bers > that management (and the prospect) can understand. I've now discovered th= e FIO > tool and the "plugin" for NVMe in SPDK and it looks like the right object= ive > performance measurement tool for the job. (I wish I'd found FIO for a pos= t- > mortum of a rather disasterous storage solution we took way to long to > analyze. IOmeter finally showed up the problem, which was never solved). > = > So we would like to be able to walk management and the prospect through t= he > world before and the world after SPDK and be able to demonstrate features= and > benefits using FIO. > = > Here's the walk: > = > 1 [X] FIO =3D> NVMEe native block device > 2 [X] FIO =3D> "plugin" SPDK NVMEe > 3 [ ] FIO =3D> "block device emulation" SPDK NVMEe > 4 [ ] FIO =3D> Host NVMEe-oF <=3Dnetwork=3D> Target NVMe-oF SPDK NVMe > = > In prose: > = > 1 - basic performance of any given NVMe device > 2 - performance advantage of SPDK for NVMe device(s) > 3 - non-network overhead of SPDK as a means to NVMe device(s) > 4 - network-connected storage accelerated by NVMe > = > So I think I understand the cases 1 and 2. The 'X' means I think I've act= ually > done it. I think I understand the case 4 (waiting for the rest of the > networking hardware to show up). > = > I suspect that there is some way to present the SPDK NVMe device(s) as bl= ock > device locally so that I might be able to do case 3, but I just have not = found > something that states plainly that this is supported or unsupported. Do you mean presenting a device managed by SPDK through the Linux kernel bl= ock stack? Like under /dev? This is possible using things like NBD or the Linux kernel NVMe-oF initiator in loopback, but there are significant performance downsides. But you may also mean presenting an SPDK block device directly to fio. SPDK= has its own block storage stack (lib/bdev) with a "module" system where differe= nt types of devices can be plugged in. The Linux kernel would call a "module" a device driver. One of the modules is "NVMe" but there's a dozen or more of = these that can talk to all sorts of things (iSCSI, libaio, Ceph RBD, PMEM, etc. e= tc.). SPDK has two fio plugins. One (the one in examples/nvme/fio_plugin) talks directly to the SPDK NVMe driver - i.e. the very lowest part of the SPDK st= ack. I think this is what you're doing with #2 above. The other plugin (the one = in examples/bdev/fio_plugin) talks to the SPDK block layer. You can use that p= lugin to talk to any SPDK block device, including an NVMe device. The reason we h= ave both is sometimes we want to benchmark just the lower layer NVMe driver, and sometimes we want to benchmark the whole block stack. Note that fio itself has some performance issues. You will not be able to measure the full performance of SPDK via fio - the tool itself is just too = slow. We provide plugins primarily because it is the industry standard tool, so i= t's a quick way for people to get up and running. We recommend you use the SPDK '= perf' tool in examples/nvme/perf to benchmark the SPDK NVMe driver instead. It has fewer features than fio, but has much lower overhead. Also note that using the NVMe-oF initiator is exactly the same as using the local NVMe driver in these benchmark tools. You simply configure it with a different transport type (tcp or rdma instead of pcie) and transport addres= s (an IP/port instead of a PCI bus:device:function). > = > 5 - Soft-RoCE-based NVMe-oF implementation to work through software issues > when hardware is not available > = > In my shop the RDMA stuff is rare as hens teeth. Or getting a system with= NVMe > and RDMA in the same chassis is rarer. I have a bucketload of software is= sues > to understand that don't involve the hardware. Has anybody danced Soft-Ro= CE > around with SPDK and NVMe-oF? (It's a wonderful way to get your head wrap= ped > around RDMA programming when you just can't get the right hardware to play > with.) > = > http://www.roceinitiative.org/software-based-roce-a-new-way-to-experience= -rdma/ We use Soft-RoCE all the time and it works well enough. It is not represent= ative of the way RoCE actually behaves, especially during failure scenarios, and = the performance is quite bad. But it certainly is fine for basic testing. However, as Andrey suggested, I've mostly switched to using the tcp transpo= rt for NVMe-oF local development and testing instead. > _______________________________________________ > SPDK mailing list -- spdk(a)lists.01.org > To unsubscribe send an email to spdk-leave(a)lists.01.org --===============3745629690151786889==--