From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============9036222410326398269==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] vhost-user-blk client? Date: Fri, 15 Feb 2019 17:36:17 +0000 Message-ID: <6d102c4e167ae8669bafdf4ce904154e15aaad8c.camel@intel.com> In-Reply-To: CAHUk8cRZ3W5bffUW_ZMD5ZJQuMjGw0aU6UrHg_OixtBbRbx28g@mail.gmail.com List-ID: To: spdk@lists.01.org --===============9036222410326398269== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, 2019-02-13 at 09:44 -0700, Michael Haeuptle wrote: > Hello, > = > I was wondering if there is a client side API for vhost-user-blk. > Basically, I'd like to make block IO requests from another process to my > vhost app. In other words, my process would be replacing QEMU. > = > Is bdev_virtio_blk.c a good starting point? Darek is answering your questions on another thread, but I want to take this opportunity to say that using vhost-user-* as a mechanism for fast block st= orage IPC between processes is a great architecture for a number of reasons (that= you probably already see, but this note is for everyone else!) and we very high= ly recommend this strategy to anyone out there laying out an architecture for = use with SPDK. There are a two challenges in particular that come up when using SPDK as pa= rt of a large system. First, the SSD often needs to be shared between multiple different applications. When using it via the kernel, partitions or the filesystem can be leveraged, but for SPDK you can't use the block stack from outside of the process it is running in. One way to solve it would be to ru= n all of the applications and SPDK in the same process, but for robustness you re= ally want to be able to handle an application crash without taking everything el= se down with it. The solution here is to run a "block stack service" process t= hat is essentially the SPDK vhost target. Then, each application can connect to= it using the vhost-user initiators we have and perform fast IPC. The vhost tar= get can expose the SPDK logical volumes it creates as separate disks (or it can= do GPT partitions), so SSDs can be shared effectively. The second main challenge is that NVMe SSDs have a fairly limited number of queue pairs in practice. When the NVMe SSD is shared between several applications, each of which wants to do I/O on lots of threads, you quickly= end up running out. Using the vhost target to expose a block service lifts this restriction, because it can expose any number of software virtqueues and it= will correctly mux them onto the real NVMe queue pairs for each device. This is more or less why we created the new unified target application (app/spdk_tgt) which is both a vhost target as well as an NVMe-oF and iSCSI target. It can be deployed as a user space block service on each system and= can expose bdevs locally via virtio-user and remotely via NVMe-oF or iSCSI, all configurable at run time via JSON-RPC. > = > Thanks. > = > -- Michael > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============9036222410326398269==--