Re: [SPDK] vhost-user-blk client?

Re: [SPDK] vhost-user-blk client?

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

On Wed, 2019-02-13 at 09:44 -0700, Michael Haeuptle wrote:
> Hello,
> 
> I was wondering if there is a client side API for vhost-user-blk.
> Basically, I'd like to make block IO requests from another process to my
> vhost app. In other words, my process would be replacing QEMU.
> 
> Is bdev_virtio_blk.c a good starting point?

Darek is answering your questions on another thread, but I want to take this
opportunity to say that using vhost-user-* as a mechanism for fast block storage
IPC between processes is a great architecture for a number of reasons (that you
probably already see, but this note is for everyone else!) and we very highly
recommend this strategy to anyone out there laying out an architecture for use
with SPDK.

There are a two challenges in particular that come up when using SPDK as part of
a large system. First, the SSD often needs to be shared between multiple
different applications. When using it via the kernel, partitions or the
filesystem can be leveraged, but for SPDK you can't use the block stack from
outside of the process it is running in. One way to solve it would be to run all
of the applications and SPDK in the same process, but for robustness you really
want to be able to handle an application crash without taking everything else
down with it. The solution here is to run a "block stack service" process that
is essentially the SPDK vhost target. Then, each application can connect to it
using the vhost-user initiators we have and perform fast IPC. The vhost target
can expose the SPDK logical volumes it creates as separate disks (or it can do
GPT partitions), so SSDs can be shared effectively.

The second main challenge is that NVMe SSDs have a fairly limited number of
queue pairs in practice. When the NVMe SSD is shared between several
applications, each of which wants to do I/O on lots of threads, you quickly end
up running out. Using the vhost target to expose a block service lifts this
restriction, because it can expose any number of software virtqueues and it will
correctly mux them onto the real NVMe queue pairs for each device.

This is more or less why we created the new unified target application
(app/spdk_tgt) which is both a vhost target as well as an NVMe-oF and iSCSI
target. It can be deployed as a user space block service on each system and can
expose bdevs locally via virtio-user and remotely via NVMe-oF or iSCSI, all
configurable at run time via JSON-RPC.

> 
> Thanks.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost-user-blk client?

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 6861 bytes --]

Dariusz,

thanks for the pointers! I really appreciate your feedback. I got a simple
example program to work.

-- Michael

On Fri, Feb 15, 2019 at 12:53 AM Stojaczyk, Dariusz <
dariusz.stojaczyk(a)intel.com> wrote:

>
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> > Sent: Friday, February 15, 2019 7:38 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] vhost-user-blk client?
> >
> > On Thu, Feb 14, 2019, 20:01 Stojaczyk, Dariusz <
> dariusz.stojaczyk(a)intel.com
> > wrote:
> >
> > > Andrey,
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> > Kuzmin
> > > > Sent: Thursday, February 14, 2019 11:19 AM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: Re: [SPDK] vhost-user-blk client?
> > > >
> > > > On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> > > dariusz.stojaczyk(a)intel.com
> > > > wrote:
> > > >
> > > > > Hi Michael,
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of
> Michael
> > > > > > Haeuptle
> > > > > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > > > Subject: [SPDK] vhost-user-blk client?
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I was wondering if there is a client side API for vhost-user-blk.
> > > > > > Basically, I'd like to make block IO requests from another
> process
> > > to my
> > > > > > vhost app. In other words, my process would be replacing QEMU.
> > > > > >
> > > > > > Is bdev_virtio_blk.c a good starting point?
> > > > >
> > > > > Yes, that's the right place. This file implements the Virtio Block
> bdev
> > > > > driver
> > > > > that's briefly described in
> > > > > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > > > > It can be used like any other bdev available in SPDK. You could
> send
> > > I/O
> > > > > requests with spdk_bdev_read/write(). See
> > > > https://spdk.io/doc/bdev_pg.html
> > > > >
> > > >
> > > >
> > > > Pease feel free to correct me if I'm wrong, but I believe the
> original
> > > > question was re what one would call vhost_blk initiator library, the
> code
> > > > one can link to a standalone app for it to talk to spdk vhost target
> > > > process. That's an intriguing use case, and I'd be glad to learn more
> > > about
> > > > a possibility of it being supported.
> > >
> > > Right, that's what I was trying to describe. SPDK fio_plugin is a fair
> > > example
> > > of how this can be done. The fio_plugin (examples/bdev/fio_plugin)
> > > can use Virtio Block bdevs with roughly a few function calls:
> > >
> > > ```
> > > struct spdk_conf *config = spdk_conf_allocate();
> > > spdk_conf_read(config, "/path/to/legacy_cfg_file"); # the file defines
> a
> > > vhost-blk bdev named MyBdev
> > > spdk_conf_set_as_default(config);
> > > spdk_env_init(...);
> > > spdk_bdev_initialize(fini_callback, ...);
> > >
> >
> > Thanks for pointing that out, I'll take a look at fio_plugin under this
> > specific perspective.
> >
> > ```
> > >
> > > And later on:
> > >
> > > ```
> > > struct spdk_bdev *vhost_bdev = spdk_bdev_get_by_name("MyBdev");
> > > spdk_bdev_open(vhost_bdev, ...);
> > > spdk_bdev_get_io_channel(vhost_bdev);
> > >
> >
> > And do/will the calls above take care of looking up vhost target,
> > allocating virtio queues etc? If they do, that basically answers my
> > question.
>
> Yup, they do.
>
> >
> > spdk_bdev_read(...); # send READ I/O to a vhost block target
> > > ```
> > >
> > > The above still uses a legacy config file for creating bdevs and we
> will
> > > be trying to deprecate it for SPDK 19.04. It will be replaced it with
> > > at least a JSON config file that has a different structure, but serves
> the
> > > same purpose.
> > >
> >
> > That's a really good news, legacy config replacement with json is long
> > overdue.
> >
> > >
> > > We don't really support creating particular SPDK bdevs at runtime
> through
> > > some APIs yet - maybe that's something we could do along switching to
> > > those JSON config files.
> >
> >
> > What's wring with RPCs? They nicely complement json config, and should do
> > the trick. One minor point that might be worth considering is the ability
> > for the app (say, spdk target) to update its json config at runtime,
> > something I'm not sure is possible at the moment.
>
> RPCs are fine, but if you look into e.g. bdev virtio RPCs implementation
> you'll
> find that they simply call functions from lib/bdev/virtio/bdev_virtio.h -
> an
> interface that's perfectly suitable of being public, but isn't public. The
> same
> goes for the most of other bdevs.
>
> It'd be handy to create vhost bdevs as follows:
>
> ```
> struct spdk_bdev *vhost_bdev = bdev_virtio_user_blk_dev_create(
> "MyBdev", "/path/to/socket",
> 5 /*< number of queues */,
> 128 /*< queue size */);
> ```
>
> But bdev_virtio_user_blk_dev_create is not an exported symbol.
>
> D.
>
> >
> > Thanks,
> > Andrey
> >
> > If you have any suggestions on that, we'll be glad
> > > to hear them.
> > >
> > > D.
> > >
> > > >
> > > > Thanks,
> > > > Andrey
> > > >
> > > >
> > > > > If you're looking for something more low-level, e.g. putting virtio
> > > > > descriptors directly into a virtqueue, you might be interested in
> our
> > > > > generic vhost / virtio lib at include/spdk_internal/virtio.h.
> That's
> > > what
> > > > > the Virtio Block bdev driver uses underneath.
> > > > >
> > > > > D.
> > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > -- Michael
> > > > > > _______________________________________________
> > > > > > SPDK mailing list
> > > > > > SPDK(a)lists.01.org
> > > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > >
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost-user-blk client?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 6098 bytes --]



> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Friday, February 15, 2019 7:38 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost-user-blk client?
> 
> On Thu, Feb 14, 2019, 20:01 Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com
> wrote:
> 
> > Andrey,
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey
> Kuzmin
> > > Sent: Thursday, February 14, 2019 11:19 AM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: Re: [SPDK] vhost-user-blk client?
> > >
> > > On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> > dariusz.stojaczyk(a)intel.com
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > > -----Original Message-----
> > > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > > > Haeuptle
> > > > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > > Subject: [SPDK] vhost-user-blk client?
> > > > >
> > > > > Hello,
> > > > >
> > > > > I was wondering if there is a client side API for vhost-user-blk.
> > > > > Basically, I'd like to make block IO requests from another process
> > to my
> > > > > vhost app. In other words, my process would be replacing QEMU.
> > > > >
> > > > > Is bdev_virtio_blk.c a good starting point?
> > > >
> > > > Yes, that's the right place. This file implements the Virtio Block bdev
> > > > driver
> > > > that's briefly described in
> > > > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > > > It can be used like any other bdev available in SPDK. You could send
> > I/O
> > > > requests with spdk_bdev_read/write(). See
> > > https://spdk.io/doc/bdev_pg.html
> > > >
> > >
> > >
> > > Pease feel free to correct me if I'm wrong, but I believe the original
> > > question was re what one would call vhost_blk initiator library, the code
> > > one can link to a standalone app for it to talk to spdk vhost target
> > > process. That's an intriguing use case, and I'd be glad to learn more
> > about
> > > a possibility of it being supported.
> >
> > Right, that's what I was trying to describe. SPDK fio_plugin is a fair
> > example
> > of how this can be done. The fio_plugin (examples/bdev/fio_plugin)
> > can use Virtio Block bdevs with roughly a few function calls:
> >
> > ```
> > struct spdk_conf *config = spdk_conf_allocate();
> > spdk_conf_read(config, "/path/to/legacy_cfg_file"); # the file defines a
> > vhost-blk bdev named MyBdev
> > spdk_conf_set_as_default(config);
> > spdk_env_init(...);
> > spdk_bdev_initialize(fini_callback, ...);
> >
> 
> Thanks for pointing that out, I'll take a look at fio_plugin under this
> specific perspective.
> 
> ```
> >
> > And later on:
> >
> > ```
> > struct spdk_bdev *vhost_bdev = spdk_bdev_get_by_name("MyBdev");
> > spdk_bdev_open(vhost_bdev, ...);
> > spdk_bdev_get_io_channel(vhost_bdev);
> >
> 
> And do/will the calls above take care of looking up vhost target,
> allocating virtio queues etc? If they do, that basically answers my
> question.

Yup, they do.

> 
> spdk_bdev_read(...); # send READ I/O to a vhost block target
> > ```
> >
> > The above still uses a legacy config file for creating bdevs and we will
> > be trying to deprecate it for SPDK 19.04. It will be replaced it with
> > at least a JSON config file that has a different structure, but serves the
> > same purpose.
> >
> 
> That's a really good news, legacy config replacement with json is long
> overdue.
> 
> >
> > We don't really support creating particular SPDK bdevs at runtime through
> > some APIs yet - maybe that's something we could do along switching to
> > those JSON config files.
> 
> 
> What's wring with RPCs? They nicely complement json config, and should do
> the trick. One minor point that might be worth considering is the ability
> for the app (say, spdk target) to update its json config at runtime,
> something I'm not sure is possible at the moment.

RPCs are fine, but if you look into e.g. bdev virtio RPCs implementation you'll
find that they simply call functions from lib/bdev/virtio/bdev_virtio.h - an
interface that's perfectly suitable of being public, but isn't public. The same
goes for the most of other bdevs.

It'd be handy to create vhost bdevs as follows:

```
struct spdk_bdev *vhost_bdev = bdev_virtio_user_blk_dev_create(
"MyBdev", "/path/to/socket",
5 /*< number of queues */,
128 /*< queue size */);
```

But bdev_virtio_user_blk_dev_create is not an exported symbol.

D.

> 
> Thanks,
> Andrey
> 
> If you have any suggestions on that, we'll be glad
> > to hear them.
> >
> > D.
> >
> > >
> > > Thanks,
> > > Andrey
> > >
> > >
> > > > If you're looking for something more low-level, e.g. putting virtio
> > > > descriptors directly into a virtqueue, you might be interested in our
> > > > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's
> > what
> > > > the Virtio Block bdev driver uses underneath.
> > > >
> > > > D.
> > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > -- Michael
> > > > > _______________________________________________
> > > > > SPDK mailing list
> > > > > SPDK(a)lists.01.org
> > > > > https://lists.01.org/mailman/listinfo/spdk
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > >
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost-user-blk client?

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 4395 bytes --]

On Thu, Feb 14, 2019, 18:42 Michael Haeuptle <michaelhaeuptle(a)gmail.com
wrote:

> Hi Dariusz, Andrey,
>
> thanks for the replies. Yes, Andrey, you are correct. I want to use
> virtio-blk as an IPC between my app and our SPDK based storage backend. We
> are already using vhost-user-scsi between a VM and our storage backend
> process but now we have a requirement to also talk to the storage backend
> from a regular process. So instead of rolling my own IPC, I was thinking
> about reusing virtio-blk.
>
> I've started with virtio block bdev code and copied the init functions to a
> separate program (see attached). Unfortunately, I'm getting the following
> error:
> virtio_user/vhost_user.c: 242:get_hugepage_file_info: *ERROR*: Exceed
> maximum of 8
> virtio_user/vhost_user.c: 275:prepare_vhost_memory_user: *ERROR*: Failed to
> prepare memory for vhost-user
>
> Interestingly, I'm getting the same error when I use the example in
> https://spdk.io/doc/bdev.html#bdev_config_virtio_blk:
> ./scripts/rpc.py construct_virtio_dev --dev-type blk --trtype user --traddr
> /home/mhae/src/spdk/vhost.0 --vq-count 2 --vq-size 512 VirtioBlk0
>
> So I don't think this is related to me missing some initialization steps.
> Is there a know problem?


Dariusz will certainly know better, but my recollection is that the limit
is somehow related/specific to qemu, so you might want to work around
get_hugepage_file for your specific use case.

Regards,
Andrey

I'm using master and not a released version.
> I briefly looked at the get_hugepage_file function and it seems to expect
> exactly 8 huge page files (e.g.  /dev/hugepages/spdk_pid17289map_0) but I
> have more than 8...
>
> Thanks for your help.
>
> -- Michael
>
>
> On Thu, Feb 14, 2019 at 3:19 AM Andrey Kuzmin <akuzmin(a)jetstreamsoft.com>
> wrote:
>
> > On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> > dariusz.stojaczyk(a)intel.com
> > wrote:
> >
> > > Hi Michael,
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > > Haeuptle
> > > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: [SPDK] vhost-user-blk client?
> > > >
> > > > Hello,
> > > >
> > > > I was wondering if there is a client side API for vhost-user-blk.
> > > > Basically, I'd like to make block IO requests from another process to
> > my
> > > > vhost app. In other words, my process would be replacing QEMU.
> > > >
> > > > Is bdev_virtio_blk.c a good starting point?
> > >
> > > Yes, that's the right place. This file implements the Virtio Block bdev
> > > driver
> > > that's briefly described in
> > > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > > It can be used like any other bdev available in SPDK. You could send
> I/O
> > > requests with spdk_bdev_read/write(). See
> > https://spdk.io/doc/bdev_pg.html
> > >
> >
> >
> > Pease feel free to correct me if I'm wrong, but I believe the original
> > question was re what one would call vhost_blk initiator library, the code
> > one can link to a standalone app for it to talk to spdk vhost target
> > process. That's an intriguing use case, and I'd be glad to learn more
> about
> > a possibility of it being supported.
> >
> > Thanks,
> > Andrey
> >
> >
> > > If you're looking for something more low-level, e.g. putting virtio
> > > descriptors directly into a virtqueue, you might be interested in our
> > > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's
> what
> > > the Virtio Block bdev driver uses underneath.
> > >
> > > D.
> > >
> > > >
> > > > Thanks.
> > > >
> > > > -- Michael
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost-user-blk client?

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 4807 bytes --]

On Thu, Feb 14, 2019, 20:01 Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com
wrote:

> Andrey,
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> > Sent: Thursday, February 14, 2019 11:19 AM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: Re: [SPDK] vhost-user-blk client?
> >
> > On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> dariusz.stojaczyk(a)intel.com
> > wrote:
> >
> > > Hi Michael,
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > > Haeuptle
> > > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: [SPDK] vhost-user-blk client?
> > > >
> > > > Hello,
> > > >
> > > > I was wondering if there is a client side API for vhost-user-blk.
> > > > Basically, I'd like to make block IO requests from another process
> to my
> > > > vhost app. In other words, my process would be replacing QEMU.
> > > >
> > > > Is bdev_virtio_blk.c a good starting point?
> > >
> > > Yes, that's the right place. This file implements the Virtio Block bdev
> > > driver
> > > that's briefly described in
> > > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > > It can be used like any other bdev available in SPDK. You could send
> I/O
> > > requests with spdk_bdev_read/write(). See
> > https://spdk.io/doc/bdev_pg.html
> > >
> >
> >
> > Pease feel free to correct me if I'm wrong, but I believe the original
> > question was re what one would call vhost_blk initiator library, the code
> > one can link to a standalone app for it to talk to spdk vhost target
> > process. That's an intriguing use case, and I'd be glad to learn more
> about
> > a possibility of it being supported.
>
> Right, that's what I was trying to describe. SPDK fio_plugin is a fair
> example
> of how this can be done. The fio_plugin (examples/bdev/fio_plugin)
> can use Virtio Block bdevs with roughly a few function calls:
>
> ```
> struct spdk_conf *config = spdk_conf_allocate();
> spdk_conf_read(config, "/path/to/legacy_cfg_file"); # the file defines a
> vhost-blk bdev named MyBdev
> spdk_conf_set_as_default(config);
> spdk_env_init(...);
> spdk_bdev_initialize(fini_callback, ...);
>

Thanks for pointing that out, I'll take a look at fio_plugin under this
specific perspective.

```
>
> And later on:
>
> ```
> struct spdk_bdev *vhost_bdev = spdk_bdev_get_by_name("MyBdev");
> spdk_bdev_open(vhost_bdev, ...);
> spdk_bdev_get_io_channel(vhost_bdev);
>

And do/will the calls above take care of looking up vhost target,
allocating virtio queues etc? If they do, that basically answers my
question.

spdk_bdev_read(...); # send READ I/O to a vhost block target
> ```
>
> The above still uses a legacy config file for creating bdevs and we will
> be trying to deprecate it for SPDK 19.04. It will be replaced it with
> at least a JSON config file that has a different structure, but serves the
> same purpose.
>

That's a really good news, legacy config replacement with json is long
overdue.

>
> We don't really support creating particular SPDK bdevs at runtime through
> some APIs yet - maybe that's something we could do along switching to
> those JSON config files.


What's wring with RPCs? They nicely complement json config, and should do
the trick. One minor point that might be worth considering is the ability
for the app (say, spdk target) to update its json config at runtime,
something I'm not sure is possible at the moment.

Thanks,
Andrey

If you have any suggestions on that, we'll be glad
> to hear them.
>
> D.
>
> >
> > Thanks,
> > Andrey
> >
> >
> > > If you're looking for something more low-level, e.g. putting virtio
> > > descriptors directly into a virtqueue, you might be interested in our
> > > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's
> what
> > > the Virtio Block bdev driver uses underneath.
> > >
> > > D.
> > >
> > > >
> > > > Thanks.
> > > >
> > > > -- Michael
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost-user-blk client?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 3717 bytes --]

Andrey,

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Andrey Kuzmin
> Sent: Thursday, February 14, 2019 11:19 AM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost-user-blk client?
> 
> On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com
> wrote:
> 
> > Hi Michael,
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > Haeuptle
> > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: [SPDK] vhost-user-blk client?
> > >
> > > Hello,
> > >
> > > I was wondering if there is a client side API for vhost-user-blk.
> > > Basically, I'd like to make block IO requests from another process to my
> > > vhost app. In other words, my process would be replacing QEMU.
> > >
> > > Is bdev_virtio_blk.c a good starting point?
> >
> > Yes, that's the right place. This file implements the Virtio Block bdev
> > driver
> > that's briefly described in
> > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > It can be used like any other bdev available in SPDK. You could send I/O
> > requests with spdk_bdev_read/write(). See
> https://spdk.io/doc/bdev_pg.html
> >
> 
> 
> Pease feel free to correct me if I'm wrong, but I believe the original
> question was re what one would call vhost_blk initiator library, the code
> one can link to a standalone app for it to talk to spdk vhost target
> process. That's an intriguing use case, and I'd be glad to learn more about
> a possibility of it being supported.

Right, that's what I was trying to describe. SPDK fio_plugin is a fair example
of how this can be done. The fio_plugin (examples/bdev/fio_plugin)
can use Virtio Block bdevs with roughly a few function calls:

```
struct spdk_conf *config = spdk_conf_allocate();
spdk_conf_read(config, "/path/to/legacy_cfg_file"); # the file defines a vhost-blk bdev named MyBdev
spdk_conf_set_as_default(config);
spdk_env_init(...);
spdk_bdev_initialize(fini_callback, ...);
```

And later on:

```
struct spdk_bdev *vhost_bdev = spdk_bdev_get_by_name("MyBdev");
spdk_bdev_open(vhost_bdev, ...);
spdk_bdev_get_io_channel(vhost_bdev);
spdk_bdev_read(...); # send READ I/O to a vhost block target
```

The above still uses a legacy config file for creating bdevs and we will
be trying to deprecate it for SPDK 19.04. It will be replaced it with
at least a JSON config file that has a different structure, but serves the
same purpose.

We don't really support creating particular SPDK bdevs at runtime through
some APIs yet - maybe that's something we could do along switching to
those JSON config files. If you have any suggestions on that, we'll be glad
to hear them.

D.

> 
> Thanks,
> Andrey
> 
> 
> > If you're looking for something more low-level, e.g. putting virtio
> > descriptors directly into a virtqueue, you might be interested in our
> > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's what
> > the Virtio Block bdev driver uses underneath.
> >
> > D.
> >
> > >
> > > Thanks.
> > >
> > > -- Michael
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] vhost-user-blk client?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 5153 bytes --]

Michael,

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> Haeuptle
> Sent: Thursday, February 14, 2019 4:43 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: Re: [SPDK] vhost-user-blk client?
> 
> Hi Dariusz, Andrey,
> 
> thanks for the replies. Yes, Andrey, you are correct. I want to use
> virtio-blk as an IPC between my app and our SPDK based storage backend.
> We
> are already using vhost-user-scsi between a VM and our storage backend
> process but now we have a requirement to also talk to the storage backend
> from a regular process. So instead of rolling my own IPC, I was thinking
> about reusing virtio-blk.
> 
> I've started with virtio block bdev code and copied the init functions to a
> separate program (see attached). Unfortunately, I'm getting the following
> error:
> virtio_user/vhost_user.c: 242:get_hugepage_file_info: *ERROR*: Exceed
> maximum of 8
> virtio_user/vhost_user.c: 275:prepare_vhost_memory_user: *ERROR*:
> Failed to
> prepare memory for vhost-user

We should definitely make this error message more descriptive. This is a
limitation of the vhost-user protocol, which only allows 8 memory regions
(and 8 shared memory file descriptors) to be shared between vhost-user
master and slave. DPDK - responsible for the memory management in
SPDK - creates one hugetlbfs file for each hugepage by default and that
just doesn't work with vhost when you're using 2MB hugepages. You
could either switch to 1GB hugepages, or initialize DPDK with a
" --single-file-segments" param. This can be done either by passing the
exact same command line param to an SPDK application, or by initializing
the SPDK env with spdk_env_opts->hugepage_single_segments = true.
It will make DPDK allocate all memory using a single hugetlbfs file. Starting
with DPDK 18.05.1, using this mode doesn't cause any significant drawbacks
and simply enables vhost-user to work.

D.

> 
> Interestingly, I'm getting the same error when I use the example in
> https://spdk.io/doc/bdev.html#bdev_config_virtio_blk:
> ./scripts/rpc.py construct_virtio_dev --dev-type blk --trtype user --traddr
> /home/mhae/src/spdk/vhost.0 --vq-count 2 --vq-size 512 VirtioBlk0
> 
> So I don't think this is related to me missing some initialization steps.
> Is there a know problem? I'm using master and not a released version.
> I briefly looked at the get_hugepage_file function and it seems to expect
> exactly 8 huge page files (e.g.  /dev/hugepages/spdk_pid17289map_0) but I
> have more than 8...
> 
> Thanks for your help.
> 
> -- Michael
> 
> 
> On Thu, Feb 14, 2019 at 3:19 AM Andrey Kuzmin
> <akuzmin(a)jetstreamsoft.com>
> wrote:
> 
> > On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> > dariusz.stojaczyk(a)intel.com
> > wrote:
> >
> > > Hi Michael,
> > >
> > > > -----Original Message-----
> > > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > > Haeuptle
> > > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > > Subject: [SPDK] vhost-user-blk client?
> > > >
> > > > Hello,
> > > >
> > > > I was wondering if there is a client side API for vhost-user-blk.
> > > > Basically, I'd like to make block IO requests from another process to
> > my
> > > > vhost app. In other words, my process would be replacing QEMU.
> > > >
> > > > Is bdev_virtio_blk.c a good starting point?
> > >
> > > Yes, that's the right place. This file implements the Virtio Block bdev
> > > driver
> > > that's briefly described in
> > > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > > It can be used like any other bdev available in SPDK. You could send I/O
> > > requests with spdk_bdev_read/write(). See
> > https://spdk.io/doc/bdev_pg.html
> > >
> >
> >
> > Pease feel free to correct me if I'm wrong, but I believe the original
> > question was re what one would call vhost_blk initiator library, the code
> > one can link to a standalone app for it to talk to spdk vhost target
> > process. That's an intriguing use case, and I'd be glad to learn more about
> > a possibility of it being supported.
> >
> > Thanks,
> > Andrey
> >
> >
> > > If you're looking for something more low-level, e.g. putting virtio
> > > descriptors directly into a virtqueue, you might be interested in our
> > > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's what
> > > the Virtio Block bdev driver uses underneath.
> > >
> > > D.
> > >
> > > >
> > > > Thanks.
> > > >
> > > > -- Michael
> > > > _______________________________________________
> > > > SPDK mailing list
> > > > SPDK(a)lists.01.org
> > > > https://lists.01.org/mailman/listinfo/spdk
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > >
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >

Re: [SPDK] vhost-user-blk client?

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 3740 bytes --]

Hi Dariusz, Andrey,

thanks for the replies. Yes, Andrey, you are correct. I want to use
virtio-blk as an IPC between my app and our SPDK based storage backend. We
are already using vhost-user-scsi between a VM and our storage backend
process but now we have a requirement to also talk to the storage backend
from a regular process. So instead of rolling my own IPC, I was thinking
about reusing virtio-blk.

I've started with virtio block bdev code and copied the init functions to a
separate program (see attached). Unfortunately, I'm getting the following
error:
virtio_user/vhost_user.c: 242:get_hugepage_file_info: *ERROR*: Exceed
maximum of 8
virtio_user/vhost_user.c: 275:prepare_vhost_memory_user: *ERROR*: Failed to
prepare memory for vhost-user

Interestingly, I'm getting the same error when I use the example in
https://spdk.io/doc/bdev.html#bdev_config_virtio_blk:
./scripts/rpc.py construct_virtio_dev --dev-type blk --trtype user --traddr
/home/mhae/src/spdk/vhost.0 --vq-count 2 --vq-size 512 VirtioBlk0

So I don't think this is related to me missing some initialization steps.
Is there a know problem? I'm using master and not a released version.
I briefly looked at the get_hugepage_file function and it seems to expect
exactly 8 huge page files (e.g.  /dev/hugepages/spdk_pid17289map_0) but I
have more than 8...

Thanks for your help.

-- Michael


On Thu, Feb 14, 2019 at 3:19 AM Andrey Kuzmin <akuzmin(a)jetstreamsoft.com>
wrote:

> On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <
> dariusz.stojaczyk(a)intel.com
> wrote:
>
> > Hi Michael,
> >
> > > -----Original Message-----
> > > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > > Haeuptle
> > > Sent: Wednesday, February 13, 2019 5:44 PM
> > > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > > Subject: [SPDK] vhost-user-blk client?
> > >
> > > Hello,
> > >
> > > I was wondering if there is a client side API for vhost-user-blk.
> > > Basically, I'd like to make block IO requests from another process to
> my
> > > vhost app. In other words, my process would be replacing QEMU.
> > >
> > > Is bdev_virtio_blk.c a good starting point?
> >
> > Yes, that's the right place. This file implements the Virtio Block bdev
> > driver
> > that's briefly described in
> > https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> > It can be used like any other bdev available in SPDK. You could send I/O
> > requests with spdk_bdev_read/write(). See
> https://spdk.io/doc/bdev_pg.html
> >
>
>
> Pease feel free to correct me if I'm wrong, but I believe the original
> question was re what one would call vhost_blk initiator library, the code
> one can link to a standalone app for it to talk to spdk vhost target
> process. That's an intriguing use case, and I'd be glad to learn more about
> a possibility of it being supported.
>
> Thanks,
> Andrey
>
>
> > If you're looking for something more low-level, e.g. putting virtio
> > descriptors directly into a virtqueue, you might be interested in our
> > generic vhost / virtio lib at include/spdk_internal/virtio.h. That's what
> > the Virtio Block bdev driver uses underneath.
> >
> > D.
> >
> > >
> > > Thanks.
> > >
> > > -- Michael
> > > _______________________________________________
> > > SPDK mailing list
> > > SPDK(a)lists.01.org
> > > https://lists.01.org/mailman/listinfo/spdk
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> >
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: mhae-blk.c --]
[-- Type: text/plain, Size: 5238 bytes --]


#include "spdk/stdinc.h"

#include "spdk/string.h"
#include "spdk/util.h"

#include <linux/virtio_ring.h>
#include <linux/virtio_pci.h>
#include <linux/virtio_config.h>

#include "spdk_internal/log.h"
#include "spdk_internal/log.h"
#include "spdk_internal/virtio.h"

#include <linux/virtio_blk.h>

/* Features desired/implemented by this driver. */
#define VIRTIO_BLK_DEV_SUPPORTED_FEATURES		\
	(1ULL << VIRTIO_BLK_F_BLK_SIZE		|	\
	 1ULL << VIRTIO_BLK_F_TOPOLOGY		|	\
	 1ULL << VIRTIO_BLK_F_MQ		|	\
	 1ULL << VIRTIO_BLK_F_RO		|	\
	 1ULL << VIRTIO_BLK_F_DISCARD		|	\
	 1ULL << VIRTIO_RING_F_EVENT_IDX	|	\
	 1ULL << VHOST_USER_F_PROTOCOL_FEATURES)

static struct virtio_dev *
virtio_user_blk_dev_create(const char *name, const char *path,
			   uint16_t num_queues, uint32_t queue_size)
{
	struct virtio_dev *vdev;
	int rc;

	vdev = calloc(1, sizeof(*vdev));
	if (vdev == NULL) {
		SPDK_ERRLOG("calloc failed for virtio device %s: %s\n", name, path);
		return NULL;
	}

	rc = virtio_user_dev_init(vdev, name, path, queue_size);
	if (rc != 0) {
		SPDK_ERRLOG("Failed to create virito device %s: %s\n", name, path);
		free(vdev);
		return NULL;
	}

	rc = virtio_dev_reset(vdev, VIRTIO_BLK_DEV_SUPPORTED_FEATURES);
	if (rc != 0) {
		virtio_dev_destruct(vdev);
		free(vdev);
		return NULL;
	}

	// init
	uint64_t capacity, num_blocks;
	uint32_t block_size;
	uint16_t host_max_queues;

	if (virtio_dev_has_feature(vdev, VIRTIO_BLK_F_BLK_SIZE)) {
		rc = virtio_dev_read_dev_config(vdev, offsetof(struct virtio_blk_config, blk_size),
						&block_size, sizeof(block_size));
		if (rc) {
			SPDK_ERRLOG("%s: config read failed: %s\n", vdev->name, spdk_strerror(-rc));
			return NULL;
		}

		if (block_size == 0 || block_size % 512 != 0) {
			SPDK_ERRLOG("%s: invalid block size (%"PRIu32"). Must be "
				    "a multiple of 512.\n", vdev->name, block_size);
			return NULL;
		}
	} else {
		block_size = 512;
	}
	printf("block_size=%d\n", block_size);

	rc = virtio_dev_read_dev_config(vdev, offsetof(struct virtio_blk_config, capacity),
					&capacity, sizeof(capacity));
	if (rc) {
		SPDK_ERRLOG("%s: config read failed: %s\n", vdev->name, spdk_strerror(-rc));
		return NULL;
	}
	printf("capacity=%ld\n", capacity);

	/* `capacity` is a number of 512-byte sectors. */
	num_blocks = capacity * 512 / block_size;
	if (num_blocks == 0) {
		SPDK_ERRLOG("%s: size too small (size: %"PRIu64", blocksize: %"PRIu32").\n",
			    vdev->name, capacity * 512, block_size);
		return NULL;
	}

	if ((capacity * 512) % block_size != 0) {
		SPDK_WARNLOG("%s: size has been rounded down to the nearest block size boundary. "
			     "(block size: %"PRIu32", previous size: %"PRIu64", new size: %"PRIu64")\n",
			     vdev->name, block_size, capacity * 512, num_blocks * block_size);
	}

	if (virtio_dev_has_feature(vdev, VIRTIO_BLK_F_MQ)) {
		rc = virtio_dev_read_dev_config(vdev, offsetof(struct virtio_blk_config, num_queues),
						&host_max_queues, sizeof(host_max_queues));
		if (rc) {
			SPDK_ERRLOG("%s: config read failed: %s\n", vdev->name, spdk_strerror(-rc));
			return NULL;
		}
	} else {
		host_max_queues = 1;
	}

	if (virtio_dev_has_feature(vdev, VIRTIO_BLK_F_RO)) {
		printf("dev is read-only\n");
	}

	if (virtio_dev_has_feature(vdev, VIRTIO_BLK_F_DISCARD)) {
		printf("dev has unmap\n");
	}

	if (num_queues == 0) {
		SPDK_ERRLOG("%s: requested 0 request queues (%"PRIu16" available).\n",
			    vdev->name, host_max_queues);
		return NULL;
	}

	if (num_queues > host_max_queues) {
		SPDK_WARNLOG("%s: requested %"PRIu16" request queues "
			     "but only %"PRIu16" available.\n",
			     vdev->name, num_queues, host_max_queues);
		num_queues = host_max_queues;
	}

	printf("dev num_queues=%d\n", num_queues);

	rc = virtio_dev_start(vdev, num_queues, 0);
	if (rc != 0) {
		return NULL;
	}

	return vdev;
}


static int
spdk_app_setup_env(void)
{
	struct spdk_env_opts env_opts = {};
	int rc;

	spdk_env_opts_init(&env_opts);

	env_opts.name = "mhae-blk";
	env_opts.core_mask = "0x1";
	env_opts.shm_id = -1;
	env_opts.mem_channel = -1;
	env_opts.master_core = -1;
	env_opts.mem_size = -1;
	//env_opts.hugepage_single_segments = opts->hugepage_single_segments;
	//env_opts.unlink_hugepage = opts->unlink_hugepage;
	//env_opts.hugedir = opts->hugedir;
	//env_opts.no_pci = opts->no_pci;
	//env_opts.num_pci_addr = opts->num_pci_addr;
	//env_opts.pci_blacklist = opts->pci_blacklist;
	//env_opts.pci_whitelist = opts->pci_whitelist;
	

	rc = spdk_env_init(&env_opts);
	// free(env_opts.pci_blacklist);
	// free(env_opts.pci_whitelist);

	if (rc < 0) {
		fprintf(stderr, "Unable to initialize SPDK env\n");
	}

	return rc;
}


static void
run(struct virtio_dev *vdev)
{
	struct virtqueue *vq;
	int32_t queue_idx;

	queue_idx = virtio_dev_find_and_acquire_queue(vdev, 0);
	if (queue_idx < 0) {
		SPDK_ERRLOG("Couldn't get an unused queue for the io_channel.\n");
		return;
	}

	vq = vdev->vqs[queue_idx];

	printf("vq=%p\n", vq);
}

int
main(int argc, char *argv[])
{
	int rc = 0;

	spdk_app_setup_env();

	struct virtio_dev *vdev = virtio_user_blk_dev_create("mhae", "/home/mhae/src/spdk/vhost.0", 2, 512);
	printf("dev=%p\n", vdev);
	if (vdev == NULL) {
		return 1;
	}

	spdk_thread_lib_init(NULL);

	printf("Run\n");
	run(vdev);


	return rc;
}

Re: [SPDK] vhost-user-blk client?

[Andrey Kuzmin]

[-- Attachment #1: Type: text/plain, Size: 1984 bytes --]

On Thu, Feb 14, 2019, 13:02 Stojaczyk, Dariusz <dariusz.stojaczyk(a)intel.com
wrote:

> Hi Michael,
>
> > -----Original Message-----
> > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> > Haeuptle
> > Sent: Wednesday, February 13, 2019 5:44 PM
> > To: Storage Performance Development Kit <spdk(a)lists.01.org>
> > Subject: [SPDK] vhost-user-blk client?
> >
> > Hello,
> >
> > I was wondering if there is a client side API for vhost-user-blk.
> > Basically, I'd like to make block IO requests from another process to my
> > vhost app. In other words, my process would be replacing QEMU.
> >
> > Is bdev_virtio_blk.c a good starting point?
>
> Yes, that's the right place. This file implements the Virtio Block bdev
> driver
> that's briefly described in
> https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
> It can be used like any other bdev available in SPDK. You could send I/O
> requests with spdk_bdev_read/write(). See https://spdk.io/doc/bdev_pg.html
>


Pease feel free to correct me if I'm wrong, but I believe the original
question was re what one would call vhost_blk initiator library, the code
one can link to a standalone app for it to talk to spdk vhost target
process. That's an intriguing use case, and I'd be glad to learn more about
a possibility of it being supported.

Thanks,
Andrey


> If you're looking for something more low-level, e.g. putting virtio
> descriptors directly into a virtqueue, you might be interested in our
> generic vhost / virtio lib at include/spdk_internal/virtio.h. That's what
> the Virtio Block bdev driver uses underneath.
>
> D.
>
> >
> > Thanks.
> >
> > -- Michael
> > _______________________________________________
> > SPDK mailing list
> > SPDK(a)lists.01.org
> > https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

Re: [SPDK] vhost-user-blk client?

[Stojaczyk, Dariusz]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Hi Michael,

> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael
> Haeuptle
> Sent: Wednesday, February 13, 2019 5:44 PM
> To: Storage Performance Development Kit <spdk(a)lists.01.org>
> Subject: [SPDK] vhost-user-blk client?
> 
> Hello,
> 
> I was wondering if there is a client side API for vhost-user-blk.
> Basically, I'd like to make block IO requests from another process to my
> vhost app. In other words, my process would be replacing QEMU.
> 
> Is bdev_virtio_blk.c a good starting point?

Yes, that's the right place. This file implements the Virtio Block bdev driver
that's briefly described in https://spdk.io/doc/bdev.html#bdev_config_virtio_blk
It can be used like any other bdev available in SPDK. You could send I/O
requests with spdk_bdev_read/write(). See https://spdk.io/doc/bdev_pg.html 

If you're looking for something more low-level, e.g. putting virtio
descriptors directly into a virtqueue, you might be interested in our
generic vhost / virtio lib at include/spdk_internal/virtio.h. That's what
the Virtio Block bdev driver uses underneath.

D.

> 
> Thanks.
> 
> -- Michael
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] vhost-user-blk client?

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

Hello,

I was wondering if there is a client side API for vhost-user-blk.
Basically, I'd like to make block IO requests from another process to my
vhost app. In other words, my process would be replacing QEMU.

Is bdev_virtio_blk.c a good starting point?

Thanks.

-- Michael