From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2228018123031469372==" MIME-Version: 1.0 From: Niu, Yawei Subject: [SPDK] Re: About "core_mask" for DPDK EAL Date: Thu, 27 Feb 2020 16:06:34 +0000 Message-ID: <8409B0DB-5BDF-4579-9CED-7B0701CEE0AD@intel.com> In-Reply-To: 74C855EC-AEB9-4089-9099-D9F5F5F2E774@intel.com List-ID: To: spdk@lists.01.org --===============2228018123031469372== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable I see, I'll reorg the code a bit. Thanks for your help! =EF=BB=BFOn 2020/2/28, 12:02 AM, "Harris, James R" wrote: Hi Niu, = Do you have the ability to call spdk_env_init() before setting the CPU = affinity? = If not, you'll likely need to save the CPU affinity before calling spdk= _env_init(), and then set it back again after spdk_env_init() returns. = -Jim = = On 2/27/20, 8:54 AM, "Niu, Yawei" wrote: = Hi, Jim = Our threading model looks like: On server start, the service proces= s creates all service threads one by one, each thread function will set CPU= affinity before doing anything, and only the first created thread will cal= l spdk_env_init(). So in my case, will the following created threads inherit the affin= ity set by spdk_env_init() from the first thread? And I suppose calling spd= k_unaffinitize_thread() after spdk_env_init() could break our CPU affinity = as well, so we need re-set affinity after the spdk_unaffinitize_thread() ca= ll? = Thanks -Niu = On 2020/2/27, 11:11 PM, "Harris, James R" wrote: = Hi Niu, = The core mask is used to specify which cores will be used for p= olling threads managed by the SPDK/DPDK application framework. There reall= y isn't a way to disable the core mask, but there are some ways to minimize= its effects. = I'm curious - in your environment, when do you create your DAOS= threads? And you create them from the same thread that is calling spdk_en= v_init()? = Note that the thread calling spdk_env_init() will be pinned to = that single core you've specified in your core mask. And threads created a= fter spdk_env_init() will inherit that affinity. You can call spdk_unaffin= itize_thread() immediately after spdk_env_init() returns to eliminate that = affinity. Could you try that and see if it helps when specifying core_mask= 0x1 in your environment? = -Jim = = = On 2/27/20, 2:02 AM, "Niu, Yawei" wrote: = Hi, = We (DAOS) used to call spdk_env_init() with default =E2=80= =98core_mask=E2=80=99 =E2=80=9C0x1=E2=80=9D to initialize DPDK EAL, during = recent tests, we realized that performance can be boosted a lot by changing= the =E2=80=98core_mask=E2=80=99 to some other value like =E2=80=980x8=E2= =80=99, so I=E2=80=99m wondering if it because some sort of DPDK service th= read was stealing CPU cycles from DAOS service thread, and when the =E2=80= =9Ccore_mask=E2=80=9D is changed, the DPDK thread is happen to be created o= n some unused core? But I didn=E2=80=99t observe any suspicious thread by = =E2=80=98ps=E2=80=99 command (with core_mask =E2=80=980x1=E2=80=99 or =E2= =80=980x8=E2=80=99). = So my question is what this =E2=80=98core_mask=E2=80=99 is = used for? I suppose it=E2=80=99s not useful for our server (we have our own= polling thread and user level scheduler), is there any way to disable it? = What=E2=80=99s the best practice of this =E2=80=98core_mask=E2=80=99 config= uration for SPDK apps? = Thanks -Niu _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org = = _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org = = _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org = = _______________________________________________ SPDK mailing list -- spdk(a)lists.01.org To unsubscribe send an email to spdk-leave(a)lists.01.org = --===============2228018123031469372==--