From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============7372989803544500360==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Using spdk_thread_poll with custom scheduler Date: Wed, 01 May 2019 17:41:45 +0000 Message-ID: <01df4c56298fd5741dbc33a66530c06867739b40.camel@intel.com> In-Reply-To: AT5PR8401MB09326230635493255D75B297D93A0@AT5PR8401MB0932.NAMPRD84.PROD.OUTLOOK.COM List-ID: To: spdk@lists.01.org --===============7372989803544500360== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Tue, 2019-04-30 at 21:50 +0000, Gupta, Sumit wrote: > Hi > = > I am looking for using spdk_threads with a custom scheduler as presented = by > Ben in the recent summit. I understand that I can do that by calling > spdk_thread_poll from within my scheduler's poller. But I see that there = are > thread local variables used by various modules (such as bdev) which wont = work > if I move my poller to a different core. One example is spdk_bdev_open wh= ich > gets spdk_thread from the tls variable. Thoughts/ideas ? I just did a search for all occurrences of __thread in the code. I see seve= ral, but if I filter out the ones in tests or example applications (where we can deduce they're safe due to the structure of the example or test) then only = a few remain. The most important one is the one in lib/thread which stores the current th= read. This thread local variable is set when the user calls spdk_thread_poll() an= d is unset when that function returns. This allows us to implement an efficient version of spdk_get_thread() for use within pollers or events. All code wit= hin SPDK is driven by calls to spdk_thread_poll(), so all calls to spdk_get_thr= ead() are necessarily inside of an spdk_thread_poll() call while the thread local variable is set. Between calls to spdk_thread_poll(), you are free to move = the spdk_thread to a different system thread and everything continues to functi= on as expected. There's two additional ones that require some consideration. 1. nvme_pcie.c: There is a thread local variable used when reading/writing = the PCI BAR. This is part of hot remove handling. If the device is hot-removed, writes to the PCI BAR generate a SIGBUS. We install a SIGBUS handler to deal with this (by swapping in a memory region mapped with all 0xF). That SIGBUS handler needs some information to function correctly, which we store in a t= hread local variable. The SIGBUS handler is guaranteed to run on the same system thread as where the signal was generated and the system thread resumes exec= ution afterward. Because all of these PCI BAR writes are done within the context = of an spdk_thread_poll(), and the lifetime of the thread local variable does not extend across calls to spdk_thread_poll(), this one is safe. 2. strerror_tls.cs: We have a utility function for converting error codes to strings that uses a thread local variable as the pre-allocated string outpu= t. The typical use here is that you are going to convert the errnum to a strin= g and print it out immediately. This whole process also occurs entirely within the context of one spdk_thread_poll() call, so the system thread never switches during the lifetime of the thread local variable unless you were to hang on= to that string output for an extended period of time. We deal with this by sim= ply saying to not do that - make a copy or print it out and be done with it. So looking it over, I think all uses of thread local variables are safe. Yo= u can move spdk_threads around between calls to spdk_thread_poll() and everything= will work as expected. Thanks, Ben > = > Thanks > Sumit > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============7372989803544500360==--