Re: [SPDK] Reactor/vhost dependencies

Re: [SPDK] Reactor/vhost dependencies

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2273 bytes --]

On Thu, 2018-06-07 at 11:08 -0600, Michael Haeuptle wrote:
> I'm using a vhost-user-scsi device and I'm running the pollers for this device
> on core 0 and the event loop on core 1. I basically split the existing
> reactor.c:_spdk_reactor_run() apart in a poller and event processor. The
> vhost-user-scsi device is associated with core 1.
> During the vhost-user negotiation we stop/start the device for each config
> message that comes in over the vhost-user socket. One race condition (there
> may be more) seems to happen when the poller is trying to access potentially
> half-initialized components (e.g. tasks) of spdk_vhost_scsi_dev. The
> initialization of some of the  spdk_vhost_scsi_dev is happening via the
> callbacks that are processed on the event loop.

There are two levels of abstraction in SPDK - the threading abstraction (in
include/spdk/io_channel.h) and the event framework implementation. All of the
code in the SPDK libraries only relies on the threading abstraction, since users
may replace the event framework with their own thing. However, in the threading
abstraction, an object called an spdk_thread is defined. Pollers and messages
are associated with a particular spdk_thread. This is very intentional - the
poller and messages associated with a particular thread need to execute in a
mutually exclusive manner (i.e. run on the same native thread). This is one of
the key ways that SPDK avoids taking locks.

> From your answer it seems that we really need to have the strict sequence of
> processing all events before we start the polling and we can't really run the
> two parts in parallel to avoid potential locking, correct?

Correct. What problem were you solving by attempting to separate them? Maybe
there's some other way to approach the problem that we could figure out that
might give you better results.

Note that right now there is lots of active work to clarify and better abstract
SPDK's expectations of the underlying application threading model. We're
currently working through breaking the strict one to one mapping of spdk_threads
to native system threads. But the requirement that pollers and messages
associated with the same thread run mutually exclusively will remain.

Thanks,
Ben

Re: [SPDK] Reactor/vhost dependencies

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 3164 bytes --]

Hi Pawel,

thanks for the quick reply!

My code might be more confusing than helpful at this point. Let me provide
some more details.

I'm using a vhost-user-scsi device and I'm running the pollers for this
device on core 0 and the event loop on core 1. I basically split the
existing reactor.c:_spdk_reactor_run() apart in a poller and event
processor. The vhost-user-scsi device is associated with core 1.
During the vhost-user negotiation we stop/start the device for each config
message that comes in over the vhost-user socket. One race condition (there
may be more) seems to happen when the poller is trying to access
potentially half-initialized components (e.g. tasks)
of spdk_vhost_scsi_dev. The initialization of some of the  spdk_vhost_scsi_dev
is happening via the callbacks that are processed on the event loop.

From your answer it seems that we really need to have the strict sequence
of processing all events before we start the polling and we can't really
run the two parts in parallel to avoid potential locking, correct?

Note, it seems we'd only need the locking during the vhost-user negotiation
phase... once the device is ready we potentially should be able to pick up
requests from the virtio queue via the poller in parallel to the event
processing.

Thanks.
-- Michael




On Thu, Jun 7, 2018 at 9:47 AM, Wodkowski, PawelX <
pawelx.wodkowski(a)intel.com> wrote:

> I don’t know if fully understand what you try to do and why but don’t
> think that will be possible. If some vhost device is polled on reactor core
> X any date that is used internally  by this poller can’t be access outside
> of this reactor core X without locking and we don’t lock. This is why we
> events are pumped through reactor on which vhost device is polled.
>
>
>
> Ofcourse this discussion would be much easier if you could share some code.
>
>
>
> Pawel
>
>
>
> *From:* SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Michael
> Haeuptle
> *Sent:* Thursday, June 7, 2018 5:21 PM
> *To:* spdk(a)lists.01.org
> *Subject:* [SPDK] Reactor/vhost dependencies
>
>
>
> I would like to separate the event processing and the polling in the
> reactor to make it more flexible for our execution environment.
>
>
>
> Basically, the event processing and the poller execution should run in two
> separate threads or cores. I created a "reactor" that just deals with
> polling and a reactor that just executes events.
>
>
>
> Unfortunately, I'm running into some race conditions during the vhost
> device configuration phase (vhost-user-scsi).
>
> In some instances, the vhost_virtqueues are not initialized completely as
> far as I can tell.
>
>
>
> Before I continue down the path of troubleshooting this further, I was
> wondering if this separation is even supported or if there are too many
> assumptions currently in the code base where event processing must occur
> before polling.
>
>
>
> Thanks.
>
> -- Michael
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6046 bytes --]

Re: [SPDK] Reactor/vhost dependencies

[Wodkowski, PawelX]

[-- Attachment #1: Type: text/plain, Size: 1450 bytes --]

I don’t know if fully understand what you try to do and why but don’t think that will be possible. If some vhost device is polled on reactor core X any date that is used internally  by this poller can’t be access outside of this reactor core X without locking and we don’t lock. This is why we events are pumped through reactor on which vhost device is polled.

Ofcourse this discussion would be much easier if you could share some code.

Pawel

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Michael Haeuptle
Sent: Thursday, June 7, 2018 5:21 PM
To: spdk(a)lists.01.org
Subject: [SPDK] Reactor/vhost dependencies

I would like to separate the event processing and the polling in the reactor to make it more flexible for our execution environment.

Basically, the event processing and the poller execution should run in two separate threads or cores. I created a "reactor" that just deals with polling and a reactor that just executes events.

Unfortunately, I'm running into some race conditions during the vhost device configuration phase (vhost-user-scsi).
In some instances, the vhost_virtqueues are not initialized completely as far as I can tell.

Before I continue down the path of troubleshooting this further, I was wondering if this separation is even supported or if there are too many assumptions currently in the code base where event processing must occur before polling.

Thanks.
-- Michael

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 5296 bytes --]

[SPDK] Reactor/vhost dependencies

[Michael Haeuptle]

[-- Attachment #1: Type: text/plain, Size: 810 bytes --]

I would like to separate the event processing and the polling in the
reactor to make it more flexible for our execution environment.

Basically, the event processing and the poller execution should run in two
separate threads or cores. I created a "reactor" that just deals with
polling and a reactor that just executes events.

Unfortunately, I'm running into some race conditions during the vhost
device configuration phase (vhost-user-scsi).
In some instances, the vhost_virtqueues are not initialized completely as
far as I can tell.

Before I continue down the path of troubleshooting this further, I was
wondering if this separation is even supported or if there are too many
assumptions currently in the code base where event processing must occur
before polling.

Thanks.
-- Michael

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 954 bytes --]