Re: [SPDK] how to use blobstore with pthread

[Zhengyu Zhang <freeman.zhang1992 at gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 7011 bytes --]

sure. For Malloc backend, I tried again with IOAT disabled, the
bs_init/load is a success now, but init/load callback won't be called. I
guess I shouldn't mix use SPDK app framework and pthread?

It would be helpful if someone point out some example code I can refer to.

For Nvme backend, with or without IOAT, the results are the same:

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00000000004302a0 in spdk_poller_register (fn=fn(a)entry=0x4097f0
<bdev_nvme_poll>, arg=arg(a)entry=0x7f0d0000ec70, period_microseconds =
period_microseconds(a)entry=0) at io_channel.c:226
#2  0x0000000000408e82 in bdev_nvme_create_cb (io_device=<optimized
out>, ctx_buf=0x7f0d0000ec70) at bdev_nvme.c:539
#3  0x000000000043075c in spdk_get_io_channel (io_device=0x7f0efebd8f40)
at io_channel.c:453
#4  0x0000000000429b79 in spdk_bdev_channel_create (io_device=<optimized
out>, ctx_buf=0x7f0d0000ebc0) at bdev.c:847
#5  0x000000000043075c in spdk_get_io_channel (io_device=0x10ce260) at
io_channel.c:453
#6  0x000000000041c436 in _spdk_bs_channel_create
(io_device=0x7f0d00000a00, ctx_buf=0x7f0d00000b40) at blobstore.c:1425
#7  0x000000000043075c in spdk_get_io_channel
(io_device=io_device(a)entry=0x7f0d00000a00) at io_channel.c:453
#8  0x000000000041c598 in spdk_bs_register_md_thread (bs=0x7f0d00000a00)
at blobstore.c:2649
#9  _spdk_bs_alloc (dev=dev(a)entry=0x7f0d00000940,
opts=opts(a)entry=0x7f0f0e463c10) at blobstore.c:1551
#10 0x000000000041f1fe in spdk_bs_init (dev=0x7f0d00000940,
o=o(a)entry=0x0, cb_fn=cb_fn(a)entry=0x405e80 <bs_init_complete>,
cb_arg=cb_arg(a)entry=0xbce270) at blobstore.c:2279
#11 0x0000000000405dca in hello_work (hello_context=0xbce270) at
hello_blob.c:412
#12 0x00007f0f0f799dc5 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f0f0f4c876d in clone () from /lib64/libc.so.6


Thanks!
Zhengyu

On 1/16/18 3:51 PM, Chang, Cunyin wrote:
> Could you please disable the IOAT in config file and try again?
> 
> -Cunyin
> 
>> -----Original Message-----
>> From: Zhengyu Zhang [mailto:freeman.zhang1992(a)gmail.com]
>> Sent: Tuesday, January 16, 2018 9:30 AM
>> To: Chang, Cunyin <cunyin.chang(a)intel.com>
>> Cc: Storage Performance Development Kit <spdk(a)lists.01.org>
>> Subject: Re: [SPDK] how to use blobstore with pthread
>>
>> Hi Cunyin,
>>
>> Thanks for your reply! I do call spdk_bs_alloc_io_channle() before the
>> read/write. But since the function needs a `struct spdk_blob_store *bs`
>> parameter, we have to call it after spdk_bs_init/load, usually in init/load
>> callbacks. But spdk_bs_init/load is where segmentation fault
>> happens:
>>
>> (gdb) bt
>> #0  0x0000000000000000 in ?? ()
>> #1  0x0000000000430290 in spdk_poller_register (fn=fn(a)entry=0x4065f0
>> <ioat_poll>, arg=0x2629080, period_microseconds =
>> period_microseconds(a)entry=0)at io_channel.c:226
>> #2  0x00000000004065c1 in ioat_create_cb (io_device=<optimized out>,
>> ctx_buf=0x7f16ac00ecc0) at copy_engine_ioat.c:206
>> #3  0x000000000043074c in spdk_get_io_channel (io_device=0x67d020
>> <ioat_copy_engine>) at io_channel.c:453
>> #4  0x000000000042d11b in copy_create_cb (io_device=<optimized out>,
>> ctx_buf=0x7f16ac00ec70) at copy_engine.c:186
>> #5  0x000000000043074c in spdk_get_io_channel (io_device=0x67d9d0
>> <spdk_copy_module_list>) at io_channel.c:453
>> #6  0x0000000000429b69 in spdk_bdev_channel_create
>> (io_device=<optimized
>> out>, ctx_buf=0x7f16ac00ebc0) at bdev.c:847
>> #7  0x000000000043074c in spdk_get_io_channel
>> (io_device=0x7f18abbeb5c0) at io_channel.c:453
>> #8  0x000000000041c426 in _spdk_bs_channel_create
>> (io_device=0x7f16ac000a00, ctx_buf=0x7f16ac000b40) at blobstore.c:1425
>> #9  0x000000000043074c in spdk_get_io_channel
>> (io_device=io_device(a)entry=0x7f16ac000a00) at io_channel.c:453
>> #10 0x000000000041c588 in spdk_bs_register_md_thread
>> (bs=0x7f16ac000a00) at blobstore.c:2649
>> #11 _spdk_bs_alloc (dev=dev(a)entry=0x7f16ac000940,
>> opts=opts(a)entry=0x7f18bd414c10) at blobstore.c:1551
>> #12 0x000000000041f1ee in spdk_bs_init (dev=0x7f16ac000940,
>> o=o(a)entry=0x0, cb_fn=cb_fn(a)entry=0x405e70 <bs_init_complete>,
>> cb_arg=cb_arg(a)entry=0x2622270)
>>     at blobstore.c:2279
>> #13 0x0000000000405dca in hello_work (hello_context=0x2622270) at
>> hello_blob.c:412
>> #14 0x00007f18be74adc5 in start_thread () from /lib64/libpthread.so.0
>>
>>
>> Thanks!
>> Zhengyu
>>
>> On 1/16/18 9:00 AM, Chang, Cunyin wrote:
>>> Hi Zhengyu,
>>>
>>>
>>>
>>> for new threads:
>>>
>>> after you allocate thread, before you do the create, read, write, you
>>> need allocate io channel by calling:
>>>
>>> spdk_bs_alloc_io_channel(),
>>>
>>> the blobs read/write need one IO channel.
>>>
>>>
>>>
>>> -Cunyin
>>>
>>>
>>>
>>> *From:*SPDK [mailto:spdk-bounces(a)lists.01.org] *On Behalf Of *Zhengyu
>>> Zhang
>>> *Sent:* Monday, January 15, 2018 11:20 PM
>>> *To:* Storage Performance Development Kit <spdk(a)lists.01.org>
>>> *Subject:* [SPDK] how to use blobstore with pthread
>>>
>>>
>>>
>>> Hi list,
>>>
>>>
>>>
>>> Could someone tell me how to integrate the blobstore into multithread
>>> programs? I saw some examples using SPDK event framework to do
>>> multithreading, but how to do that with the pthread library or C++
>>> std::thread is still mysterious to me...
>>>
>>>
>>>
>>> My work flow is shown below. It gives me segmentation fault when it
>>> init/load blobstore:
>>>
>>>
>>>
>>> [main]: spdk_app_start -> get bdev -> pthread_create -> ...
>>>
>>>
>>>
>>> [new threads]:  spdk_allocate_thread -> spdk_bdev_create_bs_dev ->
>>> spdk_bs_init/load[SEG FAULT] -> create/open/write/ blobs
>>>
>>>
>>>
>>>
>>>
>>> Thanks!
>>>
>>> Zhengyu
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---
>>>
>>> My thread routine:
>>>
>>>
>>>
>>> + 393 #define THREAD_NUM    (1)
>>>
>>> + 394
>>>
>>> + 395 static void _spdk_send_msg(spdk_thread_fn fn, void *ctx, void
>>> *thread_ctx)
>>>
>>> + 396 {
>>>
>>> + 397     assert(false);
>>>
>>> + 398 }
>>>
>>> + 399
>>>
>>> + 400 int thread_routine(struct hello_context_t *hello_context)
>>>
>>> + 401 {
>>>
>>> + 402     spdk_allocate_thread(_spdk_send_msg, NULL, NULL, NULL,
>>> + NULL);
>>>
>>> + 403
>>>
>>> + 404     struct spdk_bs_dev *bs_dev = NULL;
>>>
>>> + 405     bs_dev = spdk_bdev_create_bs_dev(hello_context->bdev, NULL,
>>> + NULL);
>>>
>>> + 406     if (bs_dev == NULL) {
>>>
>>> + 407           SPDK_ERRLOG("Could not create blob bdev!!\n");
>>>
>>> + 408           spdk_app_stop(-1);
>>>
>>> + 409           return;
>>>
>>> + 410     }
>>>
>>> + 411
>>>
>>> + 412     spdk_bs_init(bs_dev, NULL, bs_init_complete, hello_context);
>>> // SEG FAULT
>>>
>>> + 413 }
>>>
>>>
>>>
>>> _______________________________________________
>>> SPDK mailing list
>>> SPDK(a)lists.01.org
>>> https://lists.01.org/mailman/listinfo/spdk
>>>