Re: [SPDK] vhost-user-blk client?

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Wed, 2019-02-13 at 09:44 -0700, Michael Haeuptle wrote:
> Hello,
> 
> I was wondering if there is a client side API for vhost-user-blk.
> Basically, I'd like to make block IO requests from another process to my
> vhost app. In other words, my process would be replacing QEMU.
> 
> Is bdev_virtio_blk.c a good starting point?

Darek is answering your questions on another thread, but I want to take this
opportunity to say that using vhost-user-* as a mechanism for fast block storage
IPC between processes is a great architecture for a number of reasons (that you
probably already see, but this note is for everyone else!) and we very highly
recommend this strategy to anyone out there laying out an architecture for use
with SPDK.

There are a two challenges in particular that come up when using SPDK as part of
a large system. First, the SSD often needs to be shared between multiple
different applications. When using it via the kernel, partitions or the
filesystem can be leveraged, but for SPDK you can't use the block stack from
outside of the process it is running in. One way to solve it would be to run all
of the applications and SPDK in the same process, but for robustness you really
want to be able to handle an application crash without taking everything else
down with it. The solution here is to run a "block stack service" process that
is essentially the SPDK vhost target. Then, each application can connect to it
using the vhost-user initiators we have and perform fast IPC. The vhost target
can expose the SPDK logical volumes it creates as separate disks (or it can do
GPT partitions), so SSDs can be shared effectively.

The second main challenge is that NVMe SSDs have a fairly limited number of
queue pairs in practice. When the NVMe SSD is shared between several
applications, each of which wants to do I/O on lots of threads, you quickly end
up running out. Using the vhost target to expose a block service lifts this
restriction, because it can expose any number of software virtqueues and it will
correctly mux them onto the real NVMe queue pairs for each device.

This is more or less why we created the new unified target application
(app/spdk_tgt) which is both a vhost target as well as an NVMe-oF and iSCSI
target. It can be deployed as a user space block service on each system and can
expose bdevs locally via virtio-user and remotely via NVMe-oF or iSCSI, all
configurable at run time via JSON-RPC.

> 
> Thanks.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk