Re: [SPDK] Rocksdb test case issue

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3305 bytes --]

On Wed, 2018-06-06 at 23:16 +0000, Sablok, Kunal wrote:
> Thanks a lot Ben.
> Earlier I did it the same way you described i.e. in create_cb function but
> only issue I found was I required to create poller only for per core (or per
> channel) but create_cb was calling this for per core per io device. That way
> it was creating too many pollers per core per io device, so thought of
> creating per core only to limit it.
> Is there any other better way where we can create pollers only per core, one I
> can think is create poller in create_cb function only but check if it is
> already created for this core don't create more, similarly in destroy_cb but
> here I need to track of if this is the last call of destroy_cb for this core?

For each io_device registered, only one channel per thread will be constructed.
Multiple calls to get a channel for a given io_device on the same thread will
just return the existing channel. So if you register some global object as an
io_device, getting channels on that will yield just one per thread. You are
currently registering the bdevs as your io_device, so you'll get one channel per
bdev per thread.

> 
> Regards,
> Kunal
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
> Sent: Thursday, June 7, 2018 3:55 AM
> To: spdk(a)lists.01.org
> Subject: Re: [SPDK] Rocksdb test case issue
> 
> On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote:
> > Hi,
> > 
> > * Module init and exit functions have a requirement to be asynchronous 
> > in nature where module init needs to initialize some data structures 
> > per core, so it uses spdk_for_each_thread() call for this purpose.
> > * In module init(), spdk_for_each_thread() is used and it executes one 
> > function which initializes some data structures on that core. When 
> > init on all cores is done, callback is called, then it calls 
> > spdk_bdev_module_init_done() to notify module init is done. Similar 
> > things are there in module exit function, where it uses 
> > spdk_for_each_thread() to deallocate some data structures per core and 
> > when all cores are done processing later callback function calls 
> > spdk_bdev_module_finish_done(). This is what
> > spdk_for_each_thread() behavior.
> 
> 
> The root of the problem here is that you have a global array, indexed by the
> core number. Instead of storing data structures per core, you should be
> leveraging SPDK's I/O channel functionality. Some background information is
> available at http://www.spdk.io/doc/concurrency.html.
> 
> Specifically, inside pvol_bdev_create_cb you should be allocating a new
> pvol_bdev_io_waitq data structure and stuffing it into the channel. Similarly
> in pvol_bdev_destroy_cb you should be releasing it. Then, on start up and
> shutdown, instead of using spdk_for_each_thread, call spdk_for_each_channel.
> With that design, threads will be able to come and go as needed.
> 
> Thanks,
> Ben
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk