From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0380327092452263580==" MIME-Version: 1.0 From: Harris, James R Subject: Re: [SPDK] Rocksdb test case issue Date: Mon, 11 Jun 2018 23:46:56 +0000 Message-ID: In-Reply-To: F1151624ABEE794C8A34A4CA571EEAA850A39C@BGSMSX105.gar.corp.intel.com List-ID: To: spdk@lists.01.org --===============0380327092452263580== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Hi Kunal, I=E2=80=99m working on this spdk_bdev_io_reserve API. I will keep you post= ed as I make progress. Part of this effort will be adding the ability to configure the number of s= pdk_bdev_io structures in the global pool and the per-thread caches to help= us stress these types of cases more extensively. -Jim On 6/11/18, 8:30 AM, "SPDK on behalf of Sablok, Kunal" wrote: Thanks Ben, As I understand from one of your recent review comment, som= ebody is working on SPDK API spdk_bdev_io_reserve(). I will be using this A= PI and then waitq and poller will be removed, also the RocksDB test case is= sue should go away with this. Please let me know when spdk_bdev_io_reserve(= ) is up in SPDK, I will merge with this. = Regards, Kunal = -----Original Message----- From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Be= njamin Sent: Friday, June 8, 2018 10:12 PM To: spdk(a)lists.01.org Subject: Re: [SPDK] Rocksdb test case issue = On Wed, 2018-06-06 at 23:16 +0000, Sablok, Kunal wrote: > Thanks a lot Ben. > Earlier I did it the same way you described i.e. in create_cb functio= n = > but only issue I found was I required to create poller only for per = > core (or per > channel) but create_cb was calling this for per core per io device. = > That way it was creating too many pollers per core per io device, so = > thought of creating per core only to limit it. > Is there any other better way where we can create pollers only per = > core, one I can think is create poller in create_cb function only but = > check if it is already created for this core don't create more, = > similarly in destroy_cb but here I need to track of if this is the la= st call of destroy_cb for this core? = For each io_device registered, only one channel per thread will be cons= tructed. Multiple calls to get a channel for a given io_device on the same threa= d will just return the existing channel. So if you register some global obj= ect as an io_device, getting channels on that will yield just one per threa= d. You are currently registering the bdevs as your io_device, so you'll get= one channel per bdev per thread. = > = > Regards, > Kunal > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, = > Benjamin > Sent: Thursday, June 7, 2018 3:55 AM > To: spdk(a)lists.01.org > Subject: Re: [SPDK] Rocksdb test case issue > = > On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote: > > Hi, > > = > > * Module init and exit functions have a requirement to be = > > asynchronous in nature where module init needs to initialize some = > > data structures per core, so it uses spdk_for_each_thread() call fo= r this purpose. > > * In module init(), spdk_for_each_thread() is used and it executes = > > one function which initializes some data structures on that core. = > > When init on all cores is done, callback is called, then it calls > > spdk_bdev_module_init_done() to notify module init is done. Similar = > > things are there in module exit function, where it uses > > spdk_for_each_thread() to deallocate some data structures per core = > > and when all cores are done processing later callback function call= s = > > spdk_bdev_module_finish_done(). This is what > > spdk_for_each_thread() behavior. > = > = > The root of the problem here is that you have a global array, indexed = > by the core number. Instead of storing data structures per core, you = > should be leveraging SPDK's I/O channel functionality. Some backgroun= d = > information is available at http://www.spdk.io/doc/concurrency.html. > = > Specifically, inside pvol_bdev_create_cb you should be allocating a = > new pvol_bdev_io_waitq data structure and stuffing it into the = > channel. Similarly in pvol_bdev_destroy_cb you should be releasing it= . = > Then, on start up and shutdown, instead of using spdk_for_each_thread= , call spdk_for_each_channel. > With that design, threads will be able to come and go as needed. > = > Thanks, > Ben > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk _______________________________________________ SPDK mailing list SPDK(a)lists.01.org https://lists.01.org/mailman/listinfo/spdk = --===============0380327092452263580==--