From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============3738928363987440866=="
MIME-Version: 1.0
From: Walker, Benjamin <benjamin.walker at intel.com>
Subject: Re: [SPDK] Reactor/vhost dependencies
Date: Thu, 07 Jun 2018 18:10:36 +0000
Message-ID: <1528395034.5846.60.camel@intel.com>
In-Reply-To: CAHUk8cQNUp0cjiUu=c8KqVCQdatDL5S5SCa2TXykuvRhoAVJoQ@mail.gmail.com
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============3738928363987440866==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

On Thu, 2018-06-07 at 11:08 -0600, Michael Haeuptle wrote:
> I'm using a vhost-user-scsi device and I'm running the pollers for this d=
evice
> on core 0 and the event loop on core 1. I basically split the existing
> reactor.c:_spdk_reactor_run() apart in a poller and event processor. The
> vhost-user-scsi device is associated with core 1.
> During the vhost-user negotiation we stop/start the device for each config
> message that comes in over the vhost-user socket. One race condition (the=
re
> may be more) seems to happen when the poller is trying to access potentia=
lly
> half-initialized components (e.g. tasks) of spdk_vhost_scsi_dev. The
> initialization of some of the  spdk_vhost_scsi_dev is happening via the
> callbacks that are processed on the event loop.

There are two levels of abstraction in SPDK - the threading abstraction (in
include/spdk/io_channel.h) and the event framework implementation. All of t=
he
code in the SPDK libraries only relies on the threading abstraction, since =
users
may replace the event framework with their own thing. However, in the threa=
ding
abstraction, an object called an spdk_thread is defined. Pollers and messag=
es
are associated with a particular spdk_thread. This is very intentional - the
poller and messages associated with a particular thread need to execute in a
mutually exclusive manner (i.e. run on the same native thread). This is one=
 of
the key ways that SPDK avoids taking locks.

> From your answer it seems that we really need to have the strict sequence=
 of
> processing all events before we start the polling and we can't really run=
 the
> two parts in parallel to avoid potential locking, correct?

Correct. What problem were you solving by attempting to separate them? Maybe
there's some other way to approach the problem that we could figure out that
might give you better results.

Note that right now there is lots of active work to clarify and better abst=
ract
SPDK's expectations of the underlying application threading model. We're
currently working through breaking the strict one to one mapping of spdk_th=
reads
to native system threads. But the requirement that pollers and messages
associated with the same thread run mutually exclusively will remain.

Thanks,
Ben
--===============3738928363987440866==--