From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============3961216095912289736=="
MIME-Version: 1.0
From: Walker, Benjamin <benjamin.walker at intel.com>
Subject: Re: [SPDK] Rocksdb test case issue
Date: Wed, 06 Jun 2018 22:25:27 +0000
Message-ID: <1528323925.5846.55.camel@intel.com>
In-Reply-To: F1151624ABEE794C8A34A4CA571EEAA8509E72@BGSMSX105.gar.corp.intel.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============3961216095912289736==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote:
> Hi,
> =

> * Module init and exit functions have a requirement to be asynchronous in
> nature where module init needs to initialize some data structures per cor=
e, so
> it uses spdk_for_each_thread() call for this purpose.
> * In module init(), spdk_for_each_thread() is used and it executes one
> function which initializes some data structures on that core. When init o=
n all
> cores is done, callback is called, then it calls spdk_bdev_module_init_do=
ne()
> to notify module init is done. Similar things are there in module exit
> function, where it uses spdk_for_each_thread() to deallocate some data
> structures per core and when all cores are done processing later callback
> function calls spdk_bdev_module_finish_done(). This is what
> spdk_for_each_thread() behavior.


The root of the problem here is that you have a global array, indexed by the
core number. Instead of storing data structures per core, you should be
leveraging SPDK's I/O channel functionality. Some background information is
available at http://www.spdk.io/doc/concurrency.html.

Specifically, inside pvol_bdev_create_cb you should be allocating a new
pvol_bdev_io_waitq data structure and stuffing it into the channel. Similar=
ly in
pvol_bdev_destroy_cb you should be releasing it. Then, on start up and shut=
down,
instead of using spdk_for_each_thread, call spdk_for_each_channel. With that
design, threads will be able to come and go as needed.

Thanks,
Ben


--===============3961216095912289736==--