On Thu, 2018-06-07 at 11:08 -0600, Michael Haeuptle wrote:
> I'm using a vhost-user-scsi device and I'm running the pollers for this device
> on core 0 and the event loop on core 1. I basically split the existing
> reactor.c:_spdk_reactor_run() apart in a poller and event processor. The
> vhost-user-scsi device is associated with core 1.
> During the vhost-user negotiation we stop/start the device for each config
> message that comes in over the vhost-user socket. One race condition (there
> may be more) seems to happen when the poller is trying to access potentially
> half-initialized components (e.g. tasks) of spdk_vhost_scsi_dev. The
> initialization of some of the  spdk_vhost_scsi_dev is happening via the
> callbacks that are processed on the event loop.

There are two levels of abstraction in SPDK - the threading abstraction (in
include/spdk/io_channel.h) and the event framework implementation. All of the
code in the SPDK libraries only relies on the threading abstraction, since users
may replace the event framework with their own thing. However, in the threading
abstraction, an object called an spdk_thread is defined. Pollers and messages
are associated with a particular spdk_thread. This is very intentional - the
poller and messages associated with a particular thread need to execute in a
mutually exclusive manner (i.e. run on the same native thread). This is one of
the key ways that SPDK avoids taking locks.

> From your answer it seems that we really need to have the strict sequence of
> processing all events before we start the polling and we can't really run the
> two parts in parallel to avoid potential locking, correct?

Correct. What problem were you solving by attempting to separate them? Maybe
there's some other way to approach the problem that we could figure out that
might give you better results.

Note that right now there is lots of active work to clarify and better abstract
SPDK's expectations of the underlying application threading model. We're
currently working through breaking the strict one to one mapping of spdk_threads
to native system threads. But the requirement that pollers and messages
associated with the same thread run mutually exclusively will remain.

Thanks,
Ben