From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2158244966392844549==" MIME-Version: 1.0 From: Walker, Benjamin Subject: Re: [SPDK] Using spdk_thread_poll with custom scheduler Date: Fri, 03 May 2019 16:45:08 +0000 Message-ID: <4b39f44d078d7909b63dde2c7ae41f7a7356492f.camel@intel.com> In-Reply-To: CAHUk8cRyG3HOUFtmhVPVps9+kXhVOi96pNOBDKNvwevLz_JJyw@mail.gmail.com List-ID: To: spdk@lists.01.org --===============2158244966392844549== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable On Fri, 2019-05-03 at 08:27 -0600, Michael Haeuptle wrote: > Piggybacking on this question, I wonder how we can run the reactor in our > custom scheduler. I know I've asked this before but it is still unclear to > me how a customer scheduler can be integrated and make use of the reactor > without modifying SPDK code. > While there are great improvements in making spdk_thread/poll more > reusable, there are still dependencies to the reactor from various > components. For example, spdk_event_allocate (implemented in reactor.c) is > used by lib/event/subsystems/nvmf, lib/scsi, lib/rocksdb, and lib/vhost. > This pretty much prevents me from creating our own reactor. > = > Are there any plans to make changes to these modules? This is exactly the plan and the work is underway right now. For example, s= ee this NVMe-oF series: https://review.gerrithub.io/c/spdk/spdk/+/451760 and this vhost series: https://review.gerrithub.io/c/spdk/spdk/+/452206 and this iSCSI series: https://review.gerrithub.io/c/spdk/spdk/+/452780 These are all in-flight at the moment, but the plan for the next release is= to have removed all uses of spdk_event_* functions from the SPDK libraries. Everything will use the abstractions in include/spdk/thread.h instead. > = > Any good ideas what to do in the meantime? In essence, I need to make > _spdk_reactor_run a single function call without the while loop. At this point we're getting close to having the full conversion done, so I'= m not sure it would be worthwhile to do anything temporary. I think that NVMe-oF = patch series above is the last one required for NVMe-oF, for example. So I'd reco= mmend trying to incorporate NVMe-oF into your framework using that patch series a= nd then report back any issues you encounter so we can get this properly docum= ented and make sure any required fixes get merged as quickly as possible. I can start walking you through how to do that if you want to go that direc= tion - but the best example is example/bdev/fio_plugin/fio_plugin.c, which is al= ready converted entirely and doesn't link to the SPDK event framework at all. Thanks, Ben > = > Thanks. > = > -- Michael > = > = > On Wed, May 1, 2019 at 11:41 AM Walker, Benjamin > wrote: > = > > On Tue, 2019-04-30 at 21:50 +0000, Gupta, Sumit wrote: > > > Hi > > > = > > > I am looking for using spdk_threads with a custom scheduler as presen= ted > > by > > > Ben in the recent summit. I understand that I can do that by calling > > > spdk_thread_poll from within my scheduler's poller. But I see that th= ere > > are > > > thread local variables used by various modules (such as bdev) which w= ont > > work > > > if I move my poller to a different core. One example is spdk_bdev_open > > which > > > gets spdk_thread from the tls variable. Thoughts/ideas ? > > = > > I just did a search for all occurrences of __thread in the code. I see > > several, > > but if I filter out the ones in tests or example applications (where we= can > > deduce they're safe due to the structure of the example or test) then o= nly > > a few > > remain. > > = > > The most important one is the one in lib/thread which stores the current > > thread. > > This thread local variable is set when the user calls spdk_thread_poll() > > and is > > unset when that function returns. This allows us to implement an effici= ent > > version of spdk_get_thread() for use within pollers or events. All code > > within > > SPDK is driven by calls to spdk_thread_poll(), so all calls to > > spdk_get_thread() > > are necessarily inside of an spdk_thread_poll() call while the thread l= ocal > > variable is set. Between calls to spdk_thread_poll(), you are free to m= ove > > the > > spdk_thread to a different system thread and everything continues to > > function as > > expected. > > = > > There's two additional ones that require some consideration. > > = > > 1. nvme_pcie.c: There is a thread local variable used when reading/writ= ing > > the > > PCI BAR. This is part of hot remove handling. If the device is hot-remo= ved, > > writes to the PCI BAR generate a SIGBUS. We install a SIGBUS handler to > > deal > > with this (by swapping in a memory region mapped with all 0xF). That SI= GBUS > > handler needs some information to function correctly, which we store in= a > > thread > > local variable. The SIGBUS handler is guaranteed to run on the same sys= tem > > thread as where the signal was generated and the system thread resumes > > execution > > afterward. Because all of these PCI BAR writes are done within the cont= ext > > of an > > spdk_thread_poll(), and the lifetime of the thread local variable does = not > > extend across calls to spdk_thread_poll(), this one is safe. > > = > > 2. strerror_tls.cs: We have a utility function for converting error cod= es > > to > > strings that uses a thread local variable as the pre-allocated string > > output. > > The typical use here is that you are going to convert the errnum to a > > string and > > print it out immediately. This whole process also occurs entirely within > > the > > context of one spdk_thread_poll() call, so the system thread never swit= ches > > during the lifetime of the thread local variable unless you were to hang > > on to > > that string output for an extended period of time. We deal with this by > > simply > > saying to not do that - make a copy or print it out and be done with it. > > = > > So looking it over, I think all uses of thread local variables are safe. > > You can > > move spdk_threads around between calls to spdk_thread_poll() and > > everything will > > work as expected. > > = > > Thanks, > > Ben > > = > > = > > > Thanks > > > Sumit > > > _______________________________________________ > > > SPDK mailing list > > > SPDK(a)lists.01.org > > > https://lists.01.org/mailman/listinfo/spdk > > = > > _______________________________________________ > > SPDK mailing list > > SPDK(a)lists.01.org > > https://lists.01.org/mailman/listinfo/spdk > > = > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============2158244966392844549==--