Re: [SPDK] Using spdk_thread_poll with custom scheduler

[Walker, Benjamin <benjamin.walker at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 3261 bytes --]

On Tue, 2019-04-30 at 21:50 +0000, Gupta, Sumit wrote:
> Hi
> 
> I am looking for using spdk_threads with a custom scheduler as presented by
> Ben in the recent summit. I understand that I can do that by calling
> spdk_thread_poll from within my scheduler's poller. But I see that there are
> thread local variables used by various modules (such as bdev) which wont work
> if I move my poller to a different core. One example is spdk_bdev_open which
> gets spdk_thread from the tls variable. Thoughts/ideas ?

I just did a search for all occurrences of __thread in the code. I see several,
but if I filter out the ones in tests or example applications (where we can
deduce they're safe due to the structure of the example or test) then only a few
remain.

The most important one is the one in lib/thread which stores the current thread.
This thread local variable is set when the user calls spdk_thread_poll() and is
unset when that function returns. This allows us to implement an efficient
version of spdk_get_thread() for use within pollers or events. All code within
SPDK is driven by calls to spdk_thread_poll(), so all calls to spdk_get_thread()
are necessarily inside of an spdk_thread_poll() call while the thread local
variable is set. Between calls to spdk_thread_poll(), you are free to move the
spdk_thread to a different system thread and everything continues to function as
expected.

There's two additional ones that require some consideration.

1. nvme_pcie.c: There is a thread local variable used when reading/writing the
PCI BAR. This is part of hot remove handling. If the device is hot-removed,
writes to the PCI BAR generate a SIGBUS. We install a SIGBUS handler to deal
with this (by swapping in a memory region mapped with all 0xF). That SIGBUS
handler needs some information to function correctly, which we store in a thread
local variable. The SIGBUS handler is guaranteed to run on the same system
thread as where the signal was generated and the system thread resumes execution
afterward. Because all of these PCI BAR writes are done within the context of an
spdk_thread_poll(), and the lifetime of the thread local variable does not
extend across calls to spdk_thread_poll(), this one is safe.

2. strerror_tls.cs: We have a utility function for converting error codes to
strings that uses a thread local variable as the pre-allocated string output.
The typical use here is that you are going to convert the errnum to a string and
print it out immediately. This whole process also occurs entirely within the
context of one spdk_thread_poll() call, so the system thread never switches
during the lifetime of the thread local variable unless you were to hang on to
that string output for an extended period of time. We deal with this by simply
saying to not do that - make a copy or print it out and be done with it.

So looking it over, I think all uses of thread local variables are safe. You can
move spdk_threads around between calls to spdk_thread_poll() and everything will
work as expected.

Thanks,
Ben


> 
> Thanks
> Sumit
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk