From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============6775953647304691750==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Rocksdb test case issue Date: Fri, 08 Jun 2018 16:41:51 +0000 Message-ID: <1528476110.5846.66.camel@intel.com> In-Reply-To: F1151624ABEE794C8A34A4CA571EEAA8509FD5@BGSMSX105.gar.corp.intel.com List-ID: To: spdk@lists.01.org --===============6775953647304691750== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Wed, 2018-06-06 at 23:16 +0000, Sablok, Kunal wrote: > Thanks a lot Ben. > Earlier I did it the same way you described i.e. in create_cb function but > only issue I found was I required to create poller only for per core (or = per > channel) but create_cb was calling this for per core per io device. That = way > it was creating too many pollers per core per io device, so thought of > creating per core only to limit it. > Is there any other better way where we can create pollers only per core, = one I > can think is create poller in create_cb function only but check if it is > already created for this core don't create more, similarly in destroy_cb = but > here I need to track of if this is the last call of destroy_cb for this c= ore? For each io_device registered, only one channel per thread will be construc= ted. Multiple calls to get a channel for a given io_device on the same thread wi= ll just return the existing channel. So if you register some global object as = an io_device, getting channels on that will yield just one per thread. You are currently registering the bdevs as your io_device, so you'll get one channe= l per bdev per thread. > = > Regards, > Kunal > = > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benj= amin > Sent: Thursday, June 7, 2018 3:55 AM > To: spdk(a)lists.01.org > Subject: Re: [SPDK] Rocksdb test case issue > = > On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote: > > Hi, > > = > > * Module init and exit functions have a requirement to be asynchronous = > > in nature where module init needs to initialize some data structures = > > per core, so it uses spdk_for_each_thread() call for this purpose. > > * In module init(), spdk_for_each_thread() is used and it executes one = > > function which initializes some data structures on that core. When = > > init on all cores is done, callback is called, then it calls = > > spdk_bdev_module_init_done() to notify module init is done. Similar = > > things are there in module exit function, where it uses = > > spdk_for_each_thread() to deallocate some data structures per core and = > > when all cores are done processing later callback function calls = > > spdk_bdev_module_finish_done(). This is what > > spdk_for_each_thread() behavior. > = > = > The root of the problem here is that you have a global array, indexed by = the > core number. Instead of storing data structures per core, you should be > leveraging SPDK's I/O channel functionality. Some background information = is > available at http://www.spdk.io/doc/concurrency.html. > = > Specifically, inside pvol_bdev_create_cb you should be allocating a new > pvol_bdev_io_waitq data structure and stuffing it into the channel. Simil= arly > in pvol_bdev_destroy_cb you should be releasing it. Then, on start up and > shutdown, instead of using spdk_for_each_thread, call spdk_for_each_chann= el. > With that design, threads will be able to come and go as needed. > = > Thanks, > Ben > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============6775953647304691750==--