Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA Memory Regions

[JD Zheng <jiandong.zheng at broadcom.com>]

[-- Attachment #1: Type: text/plain, Size: 6023 bytes --]

Hi Seth,

Thanks for the prompt reply!

Please find answers inline.

JD

On 7/30/19 5:01 PM, Howell, Seth wrote:
> Hi JD,
> 
> Thanks for the report. I want to ask a few questions to start getting to the bottom of this. Since this issue doesn't currently reproduce on our per-patch or nightly tests, I would like to understand what's unique about your setup so that we can replicate it in a per patch test to prevent future regressions.
I am running it on aarch64 platform. I tried x86 platform and I can see 
same buffer alignment in memory pool but can't run the real test to 
reproduce it due to other missing pieces.

> 
> What options are you passing when you create the rdma transport? Are you creating it over RPC or in a configuration file?
I am using conf file. Pls let me know if you'd like to look into conf file.

> 
> Are you using the current DPDK submodule as your environment abstraction layer?
No. Our project uses specific version of DPDK, which is v18.11. I did 
quick test using latest and DPDK submodule on x86, and the buffer 
alignment is the same, i.e. 64B aligned.

> 
> I notice that your error log is printing from spdk_nvmf_transport_poll_group_create, which value exactly are you printing out?
Here is patch to add dbg print. Pls note that SPDK version is v19.04

@@ -215,6 +222,7 @@ spdk_nvmf_transport_poll_group_create(st
                                 SPDK_NOTICELOG("Unable to reserve the 
full number of buffers for the pg buffer cache.\n");
                                 break;
                         }
+                       SPDK_ERRLOG("%p %d(%d)\n", buf, 
group->buf_cache_count, group->buf_cache_size);
                         STAILQ_INSERT_HEAD(&group->buf_cache, buf, link);
                         group->buf_cache_count++;
                 }

> 
> Can you run your target with the -L rdma option to get a dump of the memory regions registered with the NIC?
Let me test and get back to you soon.

> 
> We made a couple of changes to this code when dynamic memory allocations were added to DPDK. There were some safeguards that we added to try and make sure this case wouldn't hit, so I'd like to make sure you are running on the latest DPDK submodule as well as the latest SPDK to narrow down where we need to look.
Unfortunately I can't easily update DPDK because other team maintains it 
internally. But if it can be repro and fixed in latest, I will try to 
pull in the fix.

> 
> Thanks,
> 
> Seth
> 
> -----Original Message-----
> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of JD Zheng via SPDK
> Sent: Wednesday, July 31, 2019 3:00 AM
> To: spdk(a)lists.01.org
> Cc: JD Zheng <jiandong.zheng(a)broadcom.com>
> Subject: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA Memory Regions
> 
> Hello,
> 
> When I run nvmf_tgt over RDMA using latest SPDK code, I occasionally ran into this errors:
> "rdma.c:1505:nvmf_rdma_fill_buffers: *ERROR*: Data buffer split over multiple RDMA Memory Regions"
> 
> After digging into the code, I found that nvmf_rdma_fill_buffers() calls spdk_mem_map_translate() to check if a data buffer sit on 2 2MB pages, and if it is the case, it reports this error.
> 
> The following commit added change to use data buffer start address to calculate the size between buffer start address and 2MB boundary. The caller nvmf_rdma_fill_buffers() uses the size to compare with IO Unit size (which is 8KB in my conf) to determine if the buffer passes 2MB boundary.
> 
> commit 37b7a308941b996f0e69049358a6119ed90d70a2
> Author: Darek Stojaczyk <dariusz.stojaczyk(a)intel.com>
> Date:   Tue Nov 13 17:43:46 2018 +0100
> 
>       memory: fix contiguous memory calculation for unaligned buffers
> 
> In nvmf_tgt, the buffers are pre-allocated as a memory pool and new request will use free buffer from that pool and the buffer start address is passed to nvmf_rdma_fill_buffers(). But I found that these buffers are not 2MB aligned and not IOUnitSize aligned (8KB in my case) either, instead, they are 64Byte aligned so that some buffers will fail the checking and leads to this problem.
> 
> The corresponding code snippets are as following:
> spdk_nvmf_transport_create()
> {
> ...
>       transport->data_buf_pool = pdk_mempool_create(spdk_mempool_name,
>                                  opts->num_shared_buffers,
>                                  opts->io_unit_size + NVMF_DATA_BUFFER_ALIGNMENT,
>                                  SPDK_MEMPOOL_DEFAULT_CACHE_SIZE,
>                                  SPDK_ENV_SOCKET_ID_ANY); ...
> }
> 
> Also some debug print I added shows the start address of the buffers:
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x200019258800 0(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x2000192557c0 1(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x200019252780 2(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x20001924f740 3(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x20001924c700 4(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x2000192496c0 5(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x200019246680 6(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x200019243640 7(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x200019240600 8(32)
> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*:
> 0x20001923d5c0 9(32)
> ...
> 
> It looks like either the buffer allocation has alignment issue or the checking is not correct.
> 
> Please advice how to fix this problem.
> 
> Thanks,
> JD Zheng
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>