From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============4863327981785814562==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Understanding io_channel Date: Tue, 10 Oct 2017 00:04:37 +0000 Message-ID: <1EA21786-C408-473B-AC06-53AAC1585C47@intel.com> In-Reply-To: CALOt4iezk+OBcifrDwoeKqcu4tjkEfyc03x6j1O=6v6NtiCvJA@mail.gmail.com List-ID: To: spdk@lists.01.org --===============4863327981785814562== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Fenggang, > On Oct 9, 2017, at 12:04 PM, Fenggang Wu wrote: > = > Hi, > = > I am new to SPDK and trying to develop an aggregated virtual block device= module (vbdev_agg.c) that stripes across multiple base devices. I am havin= g difficulty understanding = Welcome to SPDK! An aggregated virtual block device module is interesting = - will this do striping and/or concatenation? > the general physical meaning of the io_channel. And particularly in the v= bdev_agg case, how can I define the io_channel for this aggregated device? = Or more specifically, what is the right way to implementing the get_io_chan= nel function in the vbdev_agg module? > = > My current understanding is each io_device can have many separate io_chan= nels, each allocated for one thread. However, I/O requests issued to the bd= ev_agg will be forwarded to the base device=E2=80=99s io_channel anyway (af= ter some offset translation), where the io_channel of the vbdev_agg is not = used. The vbdev_agg I/O channel is basically a place for you to store the I/O cha= nnels for the base device *for that thread*. For example, an I/O channel for an nvme block device corresponds to an NVMe= queue pair. If this nvme block device is accessed from two different thre= ads, those two threads will have two separate I/O channels, each channel as= sociated with its own NVMe queue pair. This ensures that all NVMe hardware= accesses are done completely lock-free, since only one thread operates on = any given queue pair. If you roll this up to your vbdev_agg block device, you may (and likely wil= l) have it accessed from two or more different threads. So you will need v= bdev_agg I/O channels which will effectively just be a placeholder for the = underlying base block device I/O channels. You may have an I/O channel on = thread 0, which contains pointers to the base bdev channels for thread 0. = Another I/O channel on thread 1 will contain pointers to the base bdev chan= nels for thread 1. > = > I=E2=80=99ve tried to return NULL in the get_io_channel function of the v= bdev_agg module. It works fine for the read, write, unmap, and flush functi= ons I implemented in the vbdev_agg module. The vbdev_agg I/O functions (rea= d, write, unmap, flush) forward the I/O request to the underlying base devi= ce by calling spdk_bdev_{read, write, unmap, flush} again to the correspond= ing base device after the offset translation (defined by striping). In the = call back of the completion of the base devices, I call the completion of t= he io request for the agg device. I suspect that NULL is working in some of these cases, because you have alr= eady allocated I/O channels for the base device and have stored them in an = internal global data structure. But this is just a hypothesis. This works= OK for single-threaded use cases, but when a vbdev_agg block device gets a= ccessed from multiple threads, you will need separate I/O channels for the = base device too - one for each thread. > = > However, other part of the code sometime generate segment fault because o= f this NULL channel (e.g. in spdk_put_io_channel(), when accessing ch->chan= nel, as ch is null pointer). So I am just wondering what is the right way o= f defining the vbdev_agg=E2=80=99s io_channel. = > = > Also I found in the comments in struct spdk_io_channel: = > "Modules will allocate extra memory off the end of this structure to sto= re references to hardware-specific references (i.e. NVMe queue pairs, or re= ferences to child device spdk_io_channels (i.e. virtual bdevs)." > However, I haven't figure out how to use it. Is there any code that exemp= lifies this? nvme is a good example - lib/bdev/nvme/bdev_nvme.c. Look for struct nvme_i= o_channel. This is the context buffer for an I/O channel for an nvme block= device. Next look for the call to spdk_io_device_register(). The first parameter i= s a unique pointer (it just uses the address of the nvme_ctrlr structure) -= it just needs to be a pointer value that we know is unique within the appl= ication. The next two parameters are function pointers for creating and de= stroying I/O channels for this device. The last parameter is the size of t= he per-I/O channel context buffer. The SPDK io_channel module keeps a reference count on all of the I/O channe= ls. If there are multiple requests for an I/O channel for the NVMe control= ler on one thread, we do not need or want to allocate a separate NVMe queue= pair for each request - we want them to share the same I/O channel. So th= e create function pointer gets called when the I/O channel is first allocat= ed, and the destroy function pointer is not called until the last reference= to the I/O channel is released. You will probably also want to look at bdev_nvme_create_cb and bdev_nvme_de= stroy_cb. The former shows how an I/O channel for this device is created -= it allocates an I/O qpair for the NVMe controller and then starts a poller= to poll for completions on that queue pair. The latter frees the I/O queu= e pair and stops the poller once the I/O channel is destroyed. > = > Any suggestions/hints will be appreciated. Thank you very much! If you would like to post your module to GerritHub, I=E2=80=99m sure you=E2= =80=99d get some good review feedback from myself and others. Please note = that this is a very active area of development right now. Your questions a= re really appreciated and will help us clarify where we need to improve on = example code and documentation. Thanks, -Jim > = > Regards, > Fenggang > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============4863327981785814562==--