Re: [SPDK] core_mask and spawning new threads

[E.W.Z. <kreuzerkrieg at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 3548 bytes --]

I think I’ve found a way to overcome this. In my test application when thread which were created to produce I/O were spawned from the “main” thread in which the SPDK was initialized. Of course the cpu mask given to SPDK environment was affecting affinity of thread I was creating. So I tried to do the following, create a temporary thread, in this thread create an instance of my object which instantiates the SPDK environment, and then the temporary thread is destroyed and I have fully working SPDK machinery (which in turn spawned its own threads affected by affinity set in SPDK initialization) with threads doing IO created from main thread, which, in turn, knows nothing about SPDK machinery and its affinity. It successfully completed the run. Looks like a hack, but working as intended. What do you say?

Sent from Mail for Windows 10

From: Stojaczyk, DariuszX
Sent: Wednesday, February 21, 2018 5:33 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] core_mask and spawning new threads

> in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
That's correct.
Regards,
D.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 1:18 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] core_mask and spawning new threads

Thanks for the explanation. Just to make it clear, in case I'm integrating the SPDK into my existing application, I have to change the way I'm spawning thread in the whole application, right?
Sincerely,
E.

On Wed, Feb 21, 2018 at 1:43 PM, Stojaczyk, DariuszX <dariuszx.stojaczyk(a)intel.com> wrote:
Hi Ernest,
This is expected behavior. SPDK starts a fixed number of affinitized threads on startup, and the core mask parameter you've mentioned is what specifies the affinity and number of those. Each thread is bound to a single CPU. The one thread that called `spdk_env_init` becomes a master thread and is affinitized to the `opts.master_core`. To spawn additional POSIX threads from any SPDK-affinitized thread, you should use `spdk_call_unaffinitized()` (include/spdk/env.h). 
 
(There's also an `spdk_unaffinitize_thread()`, but it's restricted to very specific workloads and probably shouldn't be used at all.)
Best regards,
D.
 
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Ernest Zed
Sent: Wednesday, February 21, 2018 10:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] core_mask and spawning new threads
 
Hi all,
I see that all threads I create within my application are running on the same core. Looks like core_mask is the right parameter to adjust, however I dont see core usage affected by changing the core_mask on spdk_env_opts
The code:
    spdk_env_opts opts;
    spdk_env_opts_init(&opts);
    opts.name = "blah";
    opts.core_mask = "0xFFFF";
    opts.master_core = 1;
    opts.shm_id = 0;
    if (spdk_env_init(&opts) < 0) {
        throw std::runtime_error("Unable to initialize SPDK env");
    }
Whatever value I assign to the core_mask everything runs on the same core defined in master_core. If I set affinity to my threads manually everything works as expected. Is it a spdk_env_init bug? Am I doing something wrong?
Sincerely,
Ernest

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org
https://lists.01.org/mailman/listinfo/spdk



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 13009 bytes --]