From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============8331044361375524294==" MIME-Version: 1.0 From: Rao, Anu H Subject: Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA Memory Regions Date: Wed, 31 Jul 2019 02:34:06 +0000 Message-ID: <55CC7B4B-C060-4639-BE2B-D2C8F2552DD7@intel.com> In-Reply-To: b8f51f34-2242-9df7-ff44-4a313c599dcb@broadcom.com List-ID: To: spdk@lists.01.org --===============8331044361375524294== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Bobi Sent from my iPhone > On Jul 30, 2019, at 5:28 PM, JD Zheng via SPDK wrot= e: > = > Hi Seth, > = > Thanks for the prompt reply! > = > Please find answers inline. > = > JD > = >> On 7/30/19 5:01 PM, Howell, Seth wrote: >> Hi JD, >> Thanks for the report. I want to ask a few questions to start getting to= the bottom of this. Since this issue doesn't currently reproduce on our pe= r-patch or nightly tests, I would like to understand what's unique about yo= ur setup so that we can replicate it in a per patch test to prevent future = regressions. > I am running it on aarch64 platform. I tried x86 platform and I can see s= ame buffer alignment in memory pool but can't run the real test to reproduc= e it due to other missing pieces. > = >> What options are you passing when you create the rdma transport? Are you= creating it over RPC or in a configuration file? > I am using conf file. Pls let me know if you'd like to look into conf fil= e. > = >> Are you using the current DPDK submodule as your environment abstraction= layer? > No. Our project uses specific version of DPDK, which is v18.11. I did qui= ck test using latest and DPDK submodule on x86, and the buffer alignment is= the same, i.e. 64B aligned. > = >> I notice that your error log is printing from spdk_nvmf_transport_poll_g= roup_create, which value exactly are you printing out? > Here is patch to add dbg print. Pls note that SPDK version is v19.04 > = > @@ -215,6 +222,7 @@ spdk_nvmf_transport_poll_group_create(st > SPDK_NOTICELOG("Unable to reserve the full= number of buffers for the pg buffer cache.\n"); > break; > } > + SPDK_ERRLOG("%p %d(%d)\n", buf, group->buf_cache_= count, group->buf_cache_size); > STAILQ_INSERT_HEAD(&group->buf_cache, buf, link); > group->buf_cache_count++; > } > = >> Can you run your target with the -L rdma option to get a dump of the mem= ory regions registered with the NIC? > Let me test and get back to you soon. > = >> We made a couple of changes to this code when dynamic memory allocations= were added to DPDK. There were some safeguards that we added to try and ma= ke sure this case wouldn't hit, so I'd like to make sure you are running on= the latest DPDK submodule as well as the latest SPDK to narrow down where = we need to look. > Unfortunately I can't easily update DPDK because other team maintains it = internally. But if it can be repro and fixed in latest, I will try to pull = in the fix. > = >> Thanks, >> Seth >> -----Original Message----- >> From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of JD Zheng vi= a SPDK >> Sent: Wednesday, July 31, 2019 3:00 AM >> To: spdk(a)lists.01.org >> Cc: JD Zheng >> Subject: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA M= emory Regions >> Hello, >> When I run nvmf_tgt over RDMA using latest SPDK code, I occasionally ran= into this errors: >> "rdma.c:1505:nvmf_rdma_fill_buffers: *ERROR*: Data buffer split over mul= tiple RDMA Memory Regions" >> After digging into the code, I found that nvmf_rdma_fill_buffers() calls= spdk_mem_map_translate() to check if a data buffer sit on 2 2MB pages, and= if it is the case, it reports this error. >> The following commit added change to use data buffer start address to ca= lculate the size between buffer start address and 2MB boundary. The caller = nvmf_rdma_fill_buffers() uses the size to compare with IO Unit size (which = is 8KB in my conf) to determine if the buffer passes 2MB boundary. >> commit 37b7a308941b996f0e69049358a6119ed90d70a2 >> Author: Darek Stojaczyk >> Date: Tue Nov 13 17:43:46 2018 +0100 >> memory: fix contiguous memory calculation for unaligned buffers >> In nvmf_tgt, the buffers are pre-allocated as a memory pool and new requ= est will use free buffer from that pool and the buffer start address is pas= sed to nvmf_rdma_fill_buffers(). But I found that these buffers are not 2MB= aligned and not IOUnitSize aligned (8KB in my case) either, instead, they = are 64Byte aligned so that some buffers will fail the checking and leads to= this problem. >> The corresponding code snippets are as following: >> spdk_nvmf_transport_create() >> { >> ... >> transport->data_buf_pool =3D pdk_mempool_create(spdk_mempool_name, >> opts->num_shared_buffers, >> opts->io_unit_size + NVMF_DATA_BUFFER_AL= IGNMENT, >> SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, >> SPDK_ENV_SOCKET_ID_ANY); ... >> } >> Also some debug print I added shows the start address of the buffers: >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x200019258800 0(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x2000192557c0 1(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x200019252780 2(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x20001924f740 3(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x20001924c700 4(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x2000192496c0 5(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x200019246680 6(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x200019243640 7(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x200019240600 8(32) >> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >> 0x20001923d5c0 9(32) >> ... >> It looks like either the buffer allocation has alignment issue or the ch= ecking is not correct. >> Please advice how to fix this problem. >> Thanks, >> JD Zheng >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============8331044361375524294==--