On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote:
> Hi,
> 
> * Module init and exit functions have a requirement to be asynchronous in
> nature where module init needs to initialize some data structures per core, so
> it uses spdk_for_each_thread() call for this purpose.
> * In module init(), spdk_for_each_thread() is used and it executes one
> function which initializes some data structures on that core. When init on all
> cores is done, callback is called, then it calls spdk_bdev_module_init_done()
> to notify module init is done. Similar things are there in module exit
> function, where it uses spdk_for_each_thread() to deallocate some data
> structures per core and when all cores are done processing later callback
> function calls spdk_bdev_module_finish_done(). This is what
> spdk_for_each_thread() behavior.


The root of the problem here is that you have a global array, indexed by the
core number. Instead of storing data structures per core, you should be
leveraging SPDK's I/O channel functionality. Some background information is
available at http://www.spdk.io/doc/concurrency.html.

Specifically, inside pvol_bdev_create_cb you should be allocating a new
pvol_bdev_io_waitq data structure and stuffing it into the channel. Similarly in
pvol_bdev_destroy_cb you should be releasing it. Then, on start up and shutdown,
instead of using spdk_for_each_thread, call spdk_for_each_channel. With that
design, threads will be able to come and go as needed.

Thanks,
Ben