Re: [SPDK] Rocksdb test case issue

[Harris, James R <james.r.harris at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 4962 bytes --]

Hi Kunal,

I’m working on this spdk_bdev_io_reserve API.  I will keep you posted as I make progress.

Part of this effort will be adding the ability to configure the number of spdk_bdev_io structures in the global pool and the per-thread caches to help us stress these types of cases more extensively.

-Jim


On 6/11/18, 8:30 AM, "SPDK on behalf of Sablok, Kunal" <spdk-bounces(a)lists.01.org on behalf of kunal.sablok(a)intel.com> wrote:

    Thanks Ben, As I understand from one of your recent review comment, somebody is working on SPDK API spdk_bdev_io_reserve(). I will be using this API and then waitq and poller will be removed, also the RocksDB test case issue should go away with this. Please let me know when spdk_bdev_io_reserve() is up in SPDK, I will merge with this.
    
    Regards,
    Kunal
    
    -----Original Message-----
    From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
    Sent: Friday, June 8, 2018 10:12 PM
    To: spdk(a)lists.01.org
    Subject: Re: [SPDK] Rocksdb test case issue
    
    On Wed, 2018-06-06 at 23:16 +0000, Sablok, Kunal wrote:
    > Thanks a lot Ben.
    > Earlier I did it the same way you described i.e. in create_cb function 
    > but only issue I found was I required to create poller only for per 
    > core (or per
    > channel) but create_cb was calling this for per core per io device. 
    > That way it was creating too many pollers per core per io device, so 
    > thought of creating per core only to limit it.
    > Is there any other better way where we can create pollers only per 
    > core, one I can think is create poller in create_cb function only but 
    > check if it is already created for this core don't create more, 
    > similarly in destroy_cb but here I need to track of if this is the last call of destroy_cb for this core?
    
    For each io_device registered, only one channel per thread will be constructed.
    Multiple calls to get a channel for a given io_device on the same thread will just return the existing channel. So if you register some global object as an io_device, getting channels on that will yield just one per thread. You are currently registering the bdevs as your io_device, so you'll get one channel per bdev per thread.
    
    > 
    > Regards,
    > Kunal
    > 
    > -----Original Message-----
    > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, 
    > Benjamin
    > Sent: Thursday, June 7, 2018 3:55 AM
    > To: spdk(a)lists.01.org
    > Subject: Re: [SPDK] Rocksdb test case issue
    > 
    > On Tue, 2018-06-05 at 08:59 +0000, Sablok, Kunal wrote:
    > > Hi,
    > > 
    > > * Module init and exit functions have a requirement to be 
    > > asynchronous in nature where module init needs to initialize some 
    > > data structures per core, so it uses spdk_for_each_thread() call for this purpose.
    > > * In module init(), spdk_for_each_thread() is used and it executes 
    > > one function which initializes some data structures on that core. 
    > > When init on all cores is done, callback is called, then it calls
    > > spdk_bdev_module_init_done() to notify module init is done. Similar 
    > > things are there in module exit function, where it uses
    > > spdk_for_each_thread() to deallocate some data structures per core 
    > > and when all cores are done processing later callback function calls 
    > > spdk_bdev_module_finish_done(). This is what
    > > spdk_for_each_thread() behavior.
    > 
    > 
    > The root of the problem here is that you have a global array, indexed 
    > by the core number. Instead of storing data structures per core, you 
    > should be leveraging SPDK's I/O channel functionality. Some background 
    > information is available at http://www.spdk.io/doc/concurrency.html.
    > 
    > Specifically, inside pvol_bdev_create_cb you should be allocating a 
    > new pvol_bdev_io_waitq data structure and stuffing it into the 
    > channel. Similarly in pvol_bdev_destroy_cb you should be releasing it. 
    > Then, on start up and shutdown, instead of using spdk_for_each_thread, call spdk_for_each_channel.
    > With that design, threads will be able to come and go as needed.
    > 
    > Thanks,
    > Ben
    > 
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    > _______________________________________________
    > SPDK mailing list
    > SPDK(a)lists.01.org
    > https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk
    _______________________________________________
    SPDK mailing list
    SPDK(a)lists.01.org
    https://lists.01.org/mailman/listinfo/spdk