Re: [SPDK] Understanding io_channel

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 5616 bytes --]

Hi Fenggang,

> On Oct 9, 2017, at 12:04 PM, Fenggang Wu <fenggang(a)cs.umn.edu> wrote:
> 
> Hi,
> 
> I am new to SPDK and trying to develop an aggregated virtual block device module (vbdev_agg.c) that stripes across multiple base devices. I am having difficulty understanding 

Welcome to SPDK!  An aggregated virtual block device module is interesting - will this do striping and/or concatenation?

> the general physical meaning of the io_channel. And particularly in the vbdev_agg case, how can I define the io_channel for this aggregated device? Or more specifically, what is the right way to implementing the get_io_channel function in the vbdev_agg module?
> 
> My current understanding is each io_device can have many separate io_channels, each allocated for one thread. However, I/O requests issued to the bdev_agg will be forwarded to the base device’s io_channel anyway (after some offset translation), where the io_channel of the vbdev_agg is not used.

The vbdev_agg I/O channel is basically a place for you to store the I/O channels for the base device *for that thread*.

For example, an I/O channel for an nvme block device corresponds to an NVMe queue pair.  If this nvme block device is accessed from two different threads, those two threads will have two separate I/O channels, each channel associated with its own NVMe queue pair.  This ensures that all NVMe hardware accesses are done completely lock-free, since only one thread operates on any given queue pair.

If you roll this up to your vbdev_agg block device, you may (and likely will) have it accessed from two or more different threads.  So you will need vbdev_agg I/O channels which will effectively just be a placeholder for the underlying base block device I/O channels.  You may have an I/O channel on thread 0, which contains pointers to the base bdev channels for thread 0.  Another I/O channel on thread 1 will contain pointers to the base bdev channels for thread 1.

> 
> I’ve tried to return NULL in the get_io_channel function of the vbdev_agg module. It works fine for the read, write, unmap, and flush functions I implemented in the vbdev_agg module. The vbdev_agg I/O functions (read, write, unmap, flush) forward the I/O request to the underlying base device by calling spdk_bdev_{read, write, unmap, flush} again to the corresponding base device after the offset translation (defined by striping). In the call back of the completion of the base devices, I call the completion of the io request for the agg device.

I suspect that NULL is working in some of these cases, because you have already allocated I/O channels for the base device and have stored them in an internal global data structure.  But this is just a hypothesis.  This works OK for single-threaded use cases, but when a vbdev_agg block device gets accessed from multiple threads, you will need separate I/O channels for the base device too - one for each thread.

> 
> However, other part of the code sometime generate segment fault because of this NULL channel (e.g. in spdk_put_io_channel(), when accessing ch->channel, as ch is null pointer). So I am just wondering what is the right way of defining the vbdev_agg’s io_channel. 
> 
> Also I found in the comments in struct spdk_io_channel: 
>  "Modules will allocate extra memory off the end of this structure to store references to hardware-specific references (i.e. NVMe queue pairs, or references to child device spdk_io_channels (i.e. virtual bdevs)."
> However, I haven't figure out how to use it. Is there any code that exemplifies this?

nvme is a good example - lib/bdev/nvme/bdev_nvme.c.  Look for struct nvme_io_channel.  This is the context buffer for an I/O channel for an nvme block device.

Next look for the call to spdk_io_device_register().  The first parameter is a unique pointer (it just uses the address of the nvme_ctrlr structure) - it just needs to be a pointer value that we know is unique within the application.  The next two parameters are function pointers for creating and destroying I/O channels for this device.  The last parameter is the size of the per-I/O channel context buffer.

The SPDK io_channel module keeps a reference count on all of the I/O channels.  If there are multiple requests for an I/O channel for the NVMe controller on one thread, we do not need or want to allocate a separate NVMe queue pair for each request - we want them to share the same I/O channel.  So the create function pointer gets called when the I/O channel is first allocated, and the destroy function pointer is not called until the last reference to the I/O channel is released.

You will probably also want to look at bdev_nvme_create_cb and bdev_nvme_destroy_cb.  The former shows how an I/O channel for this device is created - it allocates an I/O qpair for the NVMe controller and then starts a poller to poll for completions on that queue pair.  The latter frees the I/O queue pair and stops the poller once the I/O channel is destroyed.

> 
> Any suggestions/hints will be appreciated. Thank you very much!

If you would like to post your module to GerritHub, I’m sure you’d get some good review feedback from myself and others.  Please note that this is a very active area of development right now.  Your questions are really appreciated and will help us clarify where we need to improve on example code and documentation.

Thanks,

-Jim


> 
> Regards,
> Fenggang
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk