[SPDK] 答复: Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

[Sunshihao (Euler OS) <sunshihao at huawei.com>]

[-- Attachment #1: Type: text/plain, Size: 5002 bytes --]

Dear maintainer Jim,

We have modified spdk for some requirement, our io workflow is :

Frist we do init wok:
Spdk Init : spdk_env_init-> spdk_reactors_init-> spdk_thread_create(g_masterThread, &tmp_cpumask) -> spdk_reactors_start(we have modify code to make the g_masterThread never reactor run, just return to do io submit job, so we execte every msg send to g_masterThread)

And then, construct io resource:
Io resource construct: get a spdk_thread point thread_1-> spdk_bdev_open_ext(open by name)-> spdk_bdev_get_io_channel(send msg to thread_n to get io channel)

After finish that, submit io to thread_1 
Submit io: spdk_thread_send_msg(thread_n, LaunchIo,  io); 
LaunchIo will call bdev submit io function given by spdk;
Io is a struct have all args needed by bdev submit fuction.

We do init work one time and do io resource construct and submit work many times for io read/write.

I have read the bdevperf code, and at the frist time call spdk_thread_send_msg in spdk_app_start, the local thread is not a spdk_thread, we get msg form pool; after that, in the reactor_run to call
spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache.

In our submit work, we only get the spdk_thread  thread_1 to send msg; I set the local thread to thread_1 and create two or more thread to do submit work; so, it cause the problem when get msg.

Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, LaunchIo,  io);  the problem has been solved.

Above is what i used spdk to submit io, maybe I used wrong way somewhere, please give me some advice.

Thank you!


-----邮件原件-----
发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 
发送时间: 2020年9月17日 1:33
收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
主题: [SPDK] Re: 答复: Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

Hi,

Can you describe more about your application and when and where it submits IO?  Applications using the SPDK application framework (i.e. spdk_app_start()) typically do not need to create their own spdk_threads - they submit IO from the spdk_threads created by the reactors. 

Thanks,

Jim


On 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" <sunshihao(a)huawei.com> wrote:

    Dear maintainer Jim, thank you for your reply.

    After reading your email, I know I have used the wrong way to submiting io.

    When I create a thread to submit io, I don't create spdk_thread structure for the thread, this structure only be created in spdk_reactors_start api for every lcore;
    Must I create spdk_thread structure for every submit thread?


    Please give me some advice, thank you!


    -----邮件原件-----
    发件人: Harris, James R [mailto:james.r.harris(a)intel.com] 
    发送时间: 2020年9月16日 3:32
    收件人: Storage Performance Development Kit <spdk(a)lists.01.org>
    主题: [SPDK] Re: thread->msg_cache is not safe when multiple threads send messages to same spdk_thread

    Hi,

    local_thread gets its value from _get_thread().  _get_thread() returns the thread-local spdk_thread structure for the calling thread. So if multiple threads call spdk_thread_send_msg() in parallel, they will each use their own thread-local spdk_thread structure and no lock or synchronization is needed.

    -Jim

    On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" <sunshihao(a)huawei.com> wrote:

        hello，maintainer：
        when i use spdk api spdk_thread_send_msg to send io read/write msg，i found a bug：
        multiple threads send msg to one spdk_thread，they may get msg form local_thread->msg_cache，there is not a lock or cas to 
        ensure the safety of the linked list。 

                 msg = NULL;

        	if (local_thread != NULL) {
        		if (local_thread->msg_cache_count > 0) {
        				msg = SLIST_FIRST(&local_thread->msg_cache);    /* here maybe not thread safe */
        				assert(msg != NULL);
        				SLIST_REMOVE_HEAD(&local_thread->msg_cache, link);
        				local_thread->msg_cache_count--;
        		}
        	}

        i don't konw whether is my wrong way to use spdk_thread_send_msg  or the list is not safe，so please give me some advice。
        Thank you！
        _______________________________________________
        SPDK mailing list -- spdk(a)lists.01.org
        To unsubscribe send an email to spdk-leave(a)lists.01.org

    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org
    _______________________________________________
    SPDK mailing list -- spdk(a)lists.01.org
    To unsubscribe send an email to spdk-leave(a)lists.01.org

_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org