From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============3494012566019730892==" MIME-Version: 1.0 From: Sunshihao (Euler OS) Subject: =?utf-8?q?=5BSPDK=5D_=E7=AD=94=E5=A4=8D=3A_Re=3A_=E7=AD=94=E5=A4=8D=3A_Re?= =?utf-8?q?=3A_thread-=3Emsg=5Fcache_is_not_safe_when_multiple_threads_sen?= =?utf-8?q?d_messages_to_same_spdk=5Fthread?= Date: Thu, 17 Sep 2020 02:07:23 +0000 Message-ID: <944b683bedea4146ac4c82e7d863db59@huawei.com> In-Reply-To: 26BD67A4-C32A-4011-9BA2-456629A8FEF0@intel.com List-ID: To: spdk@lists.01.org --===============3494012566019730892== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Dear maintainer Jim, We have modified spdk for some requirement, our io workflow is : Frist we do init wok: Spdk Init : spdk_env_init-> spdk_reactors_init-> spdk_thread_create(g_maste= rThread, &tmp_cpumask) -> spdk_reactors_start(we have modify code to make t= he g_masterThread never reactor run, just return to do io submit job, so we= execte every msg send to g_masterThread) And then, construct io resource: Io resource construct: get a spdk_thread point thread_1-> spdk_bdev_open_ex= t(open by name)-> spdk_bdev_get_io_channel(send msg to thread_n to get io c= hannel) After finish that, submit io to thread_1 = Submit io: spdk_thread_send_msg(thread_n, LaunchIo, io); = LaunchIo will call bdev submit io function given by spdk; Io is a struct have all args needed by bdev submit fuction. We do init work one time and do io resource construct and submit work many = times for io read/write. I have read the bdevperf code, and at the frist time call spdk_thread_send_= msg in spdk_app_start, the local thread is not a spdk_thread, we get msg fo= rm pool; after that, in the reactor_run to call spdk_thread_send_msg, it is spdk_thread, so msg get from thread->msg_cache. In our submit work, we only get the spdk_thread thread_1 to send msg; I se= t the local thread to thread_1 and create two or more thread to do submit w= ork; so, it cause the problem when get msg. Now, I call spdk_set_thread(NULL) befor spdk_thread_send_msg(thread_n, Laun= chIo, io); the problem has been solved. Above is what i used spdk to submit io, maybe I used wrong way somewhere, p= lease give me some advice. Thank you! -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6----- =E5=8F=91=E4=BB=B6=E4=BA=BA: Harris, James R [mailto:james.r.harris(a)intel= .com] = =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2020=E5=B9=B49=E6=9C=8817=E6=97=A5 1:= 33 =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit =E4=B8=BB=E9=A2=98: [SPDK] Re: =E7=AD=94=E5=A4=8D: Re: thread->msg_cache is= not safe when multiple threads send messages to same spdk_thread Hi, Can you describe more about your application and when and where it submits = IO? Applications using the SPDK application framework (i.e. spdk_app_start= ()) typically do not need to create their own spdk_threads - they submit IO= from the spdk_threads created by the reactors. = Thanks, Jim =EF=BB=BFOn 9/15/20, 7:08 PM, "Sunshihao (Euler OS)" wrote: Dear maintainer Jim, thank you for your reply. After reading your email, I know I have used the wrong way to submiting= io. When I create a thread to submit io, I don't create spdk_thread structu= re for the thread, this structure only be created in spdk_reactors_start ap= i for every lcore; Must I create spdk_thread structure for every submit thread? Please give me some advice, thank you! -----=E9=82=AE=E4=BB=B6=E5=8E=9F=E4=BB=B6----- =E5=8F=91=E4=BB=B6=E4=BA=BA: Harris, James R [mailto:james.r.harris(a)i= ntel.com] = =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2020=E5=B9=B49=E6=9C=8816=E6=97= =A5 3:32 =E6=94=B6=E4=BB=B6=E4=BA=BA: Storage Performance Development Kit =E4=B8=BB=E9=A2=98: [SPDK] Re: thread->msg_cache is not safe when multi= ple threads send messages to same spdk_thread Hi, local_thread gets its value from _get_thread(). _get_thread() returns = the thread-local spdk_thread structure for the calling thread. So if multip= le threads call spdk_thread_send_msg() in parallel, they will each use thei= r own thread-local spdk_thread structure and no lock or synchronization is = needed. -Jim On 9/15/20, 6:33 AM, "sunshihao(a)huawei.com" = wrote: hello=EF=BC=8Cmaintainer=EF=BC=9A when i use spdk api spdk_thread_send_msg to send io read/write msg= =EF=BC=8Ci found a bug=EF=BC=9A multiple threads send msg to one spdk_thread=EF=BC=8Cthey may get m= sg form local_thread->msg_cache=EF=BC=8Cthere is not a lock or cas to = ensure the safety of the linked list=E3=80=82 = msg =3D NULL; if (local_thread !=3D NULL) { if (local_thread->msg_cache_count > 0) { msg =3D SLIST_FIRST(&local_thread->msg_cache); /* here maybe= not thread safe */ assert(msg !=3D NULL); SLIST_REMOVE_HEAD(&local_thread->msg_cache, link); local_thread->msg_cache_count--; } } i don't konw whether is my wrong way to use spdk_thread_send_msg o= r the list is not safe=EF=BC=8Cso please give me some advice=E3=80=82 Thank you=EF=BC=81 _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org --===============3494012566019730892==--