From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============0520368932933381086==" MIME-Version: 1.0 From: Zhengyu Zhang Subject: Re: [SPDK] how to use blobstore with pthread Date: Tue, 16 Jan 2018 18:03:51 +0800 Message-ID: <6308d230-918f-bac0-b799-e37150c27b78@gmail.com> In-Reply-To: 2BFA8F2383C3784C90698C10BC0963195EF52490@SHSMSX103.ccr.corp.intel.com List-ID: To: spdk@lists.01.org --===============0520368932933381086== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable sure. For Malloc backend, I tried again with IOAT disabled, the bs_init/load is a success now, but init/load callback won't be called. I guess I shouldn't mix use SPDK app framework and pthread? It would be helpful if someone point out some example code I can refer to. For Nvme backend, with or without IOAT, the results are the same: (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00000000004302a0 in spdk_poller_register (fn=3Dfn(a)entry=3D0x4097f0 , arg=3Darg(a)entry=3D0x7f0d0000ec70, period_microseconds = =3D period_microseconds(a)entry=3D0) at io_channel.c:226 #2 0x0000000000408e82 in bdev_nvme_create_cb (io_device=3D, ctx_buf=3D0x7f0d0000ec70) at bdev_nvme.c:539 #3 0x000000000043075c in spdk_get_io_channel (io_device=3D0x7f0efebd8f40) at io_channel.c:453 #4 0x0000000000429b79 in spdk_bdev_channel_create (io_device=3D, ctx_buf=3D0x7f0d0000ebc0) at bdev.c:847 #5 0x000000000043075c in spdk_get_io_channel (io_device=3D0x10ce260) at io_channel.c:453 #6 0x000000000041c436 in _spdk_bs_channel_create (io_device=3D0x7f0d00000a00, ctx_buf=3D0x7f0d00000b40) at blobstore.c:1425 #7 0x000000000043075c in spdk_get_io_channel (io_device=3Dio_device(a)entry=3D0x7f0d00000a00) at io_channel.c:453 #8 0x000000000041c598 in spdk_bs_register_md_thread (bs=3D0x7f0d00000a00) at blobstore.c:2649 #9 _spdk_bs_alloc (dev=3Ddev(a)entry=3D0x7f0d00000940, opts=3Dopts(a)entry=3D0x7f0f0e463c10) at blobstore.c:1551 #10 0x000000000041f1fe in spdk_bs_init (dev=3D0x7f0d00000940, o=3Do(a)entry=3D0x0, cb_fn=3Dcb_fn(a)entry=3D0x405e80 , cb_arg=3Dcb_arg(a)entry=3D0xbce270) at blobstore.c:2279 #11 0x0000000000405dca in hello_work (hello_context=3D0xbce270) at hello_blob.c:412 #12 0x00007f0f0f799dc5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007f0f0f4c876d in clone () from /lib64/libc.so.6 Thanks! Zhengyu On 1/16/18 3:51 PM, Chang, Cunyin wrote: > Could you please disable the IOAT in config file and try again? > = > -Cunyin > = >> -----Original Message----- >> From: Zhengyu Zhang [mailto:freeman.zhang1992(a)gmail.com] >> Sent: Tuesday, January 16, 2018 9:30 AM >> To: Chang, Cunyin >> Cc: Storage Performance Development Kit >> Subject: Re: [SPDK] how to use blobstore with pthread >> >> Hi Cunyin, >> >> Thanks for your reply! I do call spdk_bs_alloc_io_channle() before the >> read/write. But since the function needs a `struct spdk_blob_store *bs` >> parameter, we have to call it after spdk_bs_init/load, usually in init/l= oad >> callbacks. But spdk_bs_init/load is where segmentation fault >> happens: >> >> (gdb) bt >> #0 0x0000000000000000 in ?? () >> #1 0x0000000000430290 in spdk_poller_register (fn=3Dfn(a)entry=3D0x4065= f0 >> , arg=3D0x2629080, period_microseconds =3D >> period_microseconds(a)entry=3D0)at io_channel.c:226 >> #2 0x00000000004065c1 in ioat_create_cb (io_device=3D, >> ctx_buf=3D0x7f16ac00ecc0) at copy_engine_ioat.c:206 >> #3 0x000000000043074c in spdk_get_io_channel (io_device=3D0x67d020 >> ) at io_channel.c:453 >> #4 0x000000000042d11b in copy_create_cb (io_device=3D, >> ctx_buf=3D0x7f16ac00ec70) at copy_engine.c:186 >> #5 0x000000000043074c in spdk_get_io_channel (io_device=3D0x67d9d0 >> ) at io_channel.c:453 >> #6 0x0000000000429b69 in spdk_bdev_channel_create >> (io_device=3D> out>, ctx_buf=3D0x7f16ac00ebc0) at bdev.c:847 >> #7 0x000000000043074c in spdk_get_io_channel >> (io_device=3D0x7f18abbeb5c0) at io_channel.c:453 >> #8 0x000000000041c426 in _spdk_bs_channel_create >> (io_device=3D0x7f16ac000a00, ctx_buf=3D0x7f16ac000b40) at blobstore.c:14= 25 >> #9 0x000000000043074c in spdk_get_io_channel >> (io_device=3Dio_device(a)entry=3D0x7f16ac000a00) at io_channel.c:453 >> #10 0x000000000041c588 in spdk_bs_register_md_thread >> (bs=3D0x7f16ac000a00) at blobstore.c:2649 >> #11 _spdk_bs_alloc (dev=3Ddev(a)entry=3D0x7f16ac000940, >> opts=3Dopts(a)entry=3D0x7f18bd414c10) at blobstore.c:1551 >> #12 0x000000000041f1ee in spdk_bs_init (dev=3D0x7f16ac000940, >> o=3Do(a)entry=3D0x0, cb_fn=3Dcb_fn(a)entry=3D0x405e70 , >> cb_arg=3Dcb_arg(a)entry=3D0x2622270) >> at blobstore.c:2279 >> #13 0x0000000000405dca in hello_work (hello_context=3D0x2622270) at >> hello_blob.c:412 >> #14 0x00007f18be74adc5 in start_thread () from /lib64/libpthread.so.0 >> >> >> Thanks! >> Zhengyu >> >> On 1/16/18 9:00 AM, Chang, Cunyin wrote: >>> Hi Zhengyu, >>> >>> >>> >>> for new threads: >>> >>> after you allocate thread, before you do the create, read, write, you >>> need allocate io channel by calling: >>> >>> spdk_bs_alloc_io_channel(), >>> >>> the blobs read/write need one IO channel. >>> >>> >>> >>> -Cunyin >>> >>> >>> >>> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Zhengyu >>> Zhang >>> *Sent:* Monday, January 15, 2018 11:20 PM >>> *To:* Storage Performance Development Kit >>> *Subject:* [SPDK] how to use blobstore with pthread >>> >>> >>> >>> Hi list, >>> >>> >>> >>> Could someone tell me how to integrate the=C2=A0blobstore=C2=A0into mul= tithread >>> programs? I saw some examples using SPDK event framework to do >>> multithreading, but how to do that with the pthread=C2=A0library or C++ >>> std::thread is still mysterious to me... >>> >>> >>> >>> My work flow is shown below. It gives me segmentation fault when it >>> init/load blobstore: >>> >>> >>> >>> [main]: spdk_app_start -> get bdev -> pthread_create -> ... >>> >>> >>> >>> [new threads]:=C2=A0 spdk_allocate_thread -> spdk_bdev_create_bs_dev -> >>> spdk_bs_init/load[SEG FAULT] -> create/open/write/ blobs >>> >>> >>> >>> >>> >>> Thanks! >>> >>> Zhengyu >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> --- >>> >>> My thread routine: >>> >>> >>> >>> + 393 #define THREAD_NUM=C2=A0 =C2=A0 (1) >>> >>> + 394 >>> >>> + 395 static void _spdk_send_msg(spdk_thread_fn fn, void *ctx, void >>> *thread_ctx) >>> >>> + 396 { >>> >>> + 397=C2=A0 =C2=A0 =C2=A0assert(false); >>> >>> + 398 } >>> >>> + 399 >>> >>> + 400 int thread_routine(struct hello_context_t *hello_context) >>> >>> + 401 { >>> >>> + 402=C2=A0 =C2=A0 =C2=A0spdk_allocate_thread(_spdk_send_msg, NULL, NUL= L, NULL, >>> + NULL); >>> >>> + 403 >>> >>> + 404=C2=A0 =C2=A0 =C2=A0struct spdk_bs_dev *bs_dev =3D NULL; >>> >>> + 405=C2=A0 =C2=A0 =C2=A0bs_dev =3D spdk_bdev_create_bs_dev(hello_conte= xt->bdev, NULL, >>> + NULL); >>> >>> + 406=C2=A0 =C2=A0 =C2=A0if (bs_dev =3D=3D NULL) { >>> >>> + 407=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SPDK_ERRLOG("Could not cr= eate blob bdev!!\n"); >>> >>> + 408=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0spdk_app_stop(-1); >>> >>> + 409=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return; >>> >>> + 410=C2=A0 =C2=A0 =C2=A0} >>> >>> + 411 >>> >>> + 412=C2=A0 =C2=A0 =C2=A0spdk_bs_init(bs_dev, NULL, bs_init_complete, h= ello_context); >>> // SEG FAULT >>> >>> + 413 } >>> >>> >>> >>> _______________________________________________ >>> SPDK mailing list >>> SPDK(a)lists.01.org >>> https://lists.01.org/mailman/listinfo/spdk >>> --===============0520368932933381086==--