From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: multipart/mixed; boundary="===============2824513023477061483==" MIME-Version: 1.0 From: Sasha Kotchubievsky Subject: Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA Memory Regions Date: Wed, 21 Aug 2019 16:15:01 +0300 Message-ID: <40df7b59-6423-0ebc-232b-adc12c569fa2@dev.mellanox.co.il> In-Reply-To: EA913ED399BBA34AA4EAC2EDC24CDD00BC69A45D@FMSMSX105.amr.corp.intel.com List-ID: To: spdk@lists.01.org --===============2824513023477061483== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Seth, Thanks for update ! Sasha On 20-Aug-19 5:15 PM, Howell, Seth wrote: > Hi, > >> I think, dynamic memory allocation doesn't really work for RDMA case. > Dynamic memory allocation does work for the RDMA case on the latest maste= r. That is specifically why we use the match-allocations flag in DPDK. The = Broadcom case is distinct from stock SPDK in that they are using an older v= ersion of DPDK than the submodule which doesn't support this flag and has t= o use mitigations such as the one you mentioned below to attempt to work ar= ound the problems we faced before DPDK was updated. > >> Commit 9cec99b84b9a08e9122ada4f4455172e40ff6c06 already removes memory "= free" for dynamically allocated memory. > True, but that is a mitigation for DPDK submodules between 18.05 and 19.0= 2 which don't support the match-allocations flag. If you look at the prepro= cessor directives around this flag on master, it is only applicable if the = RTE_VERSION is >=3D 18.05 and < 19.02. > > Anyone using the stock SPDK with the DPDK submodule should be able to rel= y on DPDK dynamic allocations with the RDMA case. > > Thanks, > > Seth > > > -----Original Message----- > From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sasha Kotchu= bievsky > Sent: Tuesday, August 20, 2019 5:22 AM > To: spdk(a)lists.01.org > Subject: Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDM= A Memory Regions > > Hi, > > I think, dynamic memory allocation doesn't really work for RDMA case. > > Commit 9cec99b84b9a08e9122ada4f4455172e40ff6c06 already removes memory "f= ree" for dynamically allocated memory. > > I'd suggest to pre-allocate enough memory for nvmf target using "-s" opti= on. > > > Best regards > > Sasha > > On 20-Aug-19 12:42 AM, JD Zheng via SPDK wrote: >> Hi Seth, >> >> It sometimes triggered seg fault but I couldn't get backtrace due to >> likely corrupted stack. With my workaround, this is no longer seen. >> >> Let me submit my change as RFC to gerrit. It probably isn't necessary >> to upstream as DPDK 19.02 should fix this problem properly. >> >> Thanks, >> JD >> >> On 8/19/19 2:16 PM, Howell, Seth wrote: >>> Hi JD, >>> >>> What issue specifically did you see? If there is something measurable >>> happening (other than the error message) then I think it should be >>> high priority to get a more permanent workaround into upstream SPDK. >>> >>> Thanks, >>> >>> Seth >>> >>> -----Original Message----- >>> From: JD Zheng [mailto:jiandong.zheng(a)broadcom.com] >>> Sent: Monday, August 19, 2019 2:03 PM >>> To: Howell, Seth >>> Cc: Storage Performance Development Kit ; Jonathan >>> Richardson >>> Subject: Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple >>> RDMA Memory Regions >>> >>> Hi Seth, >>> >>> =C2=A0 > Unfortunately, the only way >>> =C2=A0 > to protect fully against this happening is by using the DPDK = flag >>>> --match-allocations which was introduced in DPDK 19.02. >>> Then I need to use DPDK 19.02. Do I need to enable this flag >>> explicitly when moving DPDK 19.02? >>> >>> =C2=A0 > The good news is that the SPDK target will skip these buffers >>> without=C2=A0 > bricking, causing data corruption or doing any otherwise >>> bad things. >>> >>> Unfortunately this is not what I saw. It appeared that SPDK gave up >>> this split buffer, but it still causes issue, maybe because it was >>> tried too many times(?). >>> >>> Currently, I have to use DPDK 18.11 so that I added a couple of >>> workarounds to prevent the split buffer from being used before >>> reaching fill_buffers(). I did a little trick there to call >>> spdk_mempool_get() but not mempool_put later, so that this buffer is >>> set as "allocated" in mempool and will not be tried again and again. >>> It does look like small memory leak though. We can usually see 2-3 >>> split buffers during overnight run, btw. >>> >>> This seems working OK. >>> >>> For sure, I will measure performance later. >>> >>> Thanks, >>> JD >>> >>> >>> On 8/19/19 1:12 PM, Howell, Seth wrote: >>>> Hi JD, >>>> >>>> Thanks for performing that experiment. With this new information, I >>>> think we can be pretty much 100% sure that the problem is related to >>>> the mempool being split over two DPDK memzones. Unfortunately, the >>>> only way to protect fully against this happening is by using the >>>> DPDK flag --match-allocations which was introduced in DPDK 19.02. >>>> Jim helped advocate for this flag specifically because of this >>>> problem with mempools and RPMA. >>>> >>>> The good news is that the SPDK target will skip these buffers >>>> without bricking, causing data corruption or doing any otherwise bad t= hings. >>>> What ends up happening is that the nvmf_rdma_fill_buffers function >>>> will print the error message and then return NULL which will trigger >>>> the target to retry the I/O again. By that time, there will be >>>> another buffer there for the request to use and it won=E2=80=99t fail = the >>>> second time around. So the code currently handles the problem in a >>>> technically correct way i.e. It=E2=80=99s not going to brick the targe= t or >>>> initiator by trying to use a buffer that spans multiple Memory >>>> Regions. Instead, it properly recognizes that it is trying to use a >>>> bad buffer and reschedules the request buffer parsing. >>>> >>>> However, I am a little worried over the fact that these buffers >>>> remain in the mempool and can be repeatedly used by the application. >>>> I can picture a scenario where this could possibly have a >>>> performance impact. >>>> Take for example a mempool with 128 entries in it in which one of >>>> them is split over a memzone. Since this split buffer will never >>>> find its way into a request, it=E2=80=99s possible that this split buf= fer >>>> gets pulled up into requests more often than other buffers and >>>> subsequently fails in nvmf_rdma_fill_buffers causing requests to >>>> have to be rescheduled to the next time the poller runs. Depending >>>> on how frequently this happens, the performance impact *could possibly= * add up. >>>> >>>> I have as yet been unable to replicate the split buffer error. One >>>> thing you could try to see if there is any measurable performance >>>> impact is try starting the NVMe-oF target with DPDK legacy memory >>>> mode which will move all memory allocations to startup and prevent >>>> you from splitting buffers. Then run a benchmark with a lot of >>>> connections at high queue depth and see what the performance looks >>>> like compared to the dynamic memory model. If there is a significant >>>> performance impact, we may have to modify how we handle this error cas= e. >>>> >>>> Thanks, >>>> >>>> Seth >>>> >>>> *From:*JD Zheng [mailto:jiandong.zheng(a)broadcom.com] >>>> *Sent:* Monday, August 12, 2019 4:17 PM >>>> *To:* Howell, Seth >>>> *Cc:* Storage Performance Development Kit ; >>>> Jonathan Richardson >>>> *Subject:* Re: [SPDK] nvmf_tgt *ERROR*: Data buffer split over >>>> multiple RDMA Memory Regions >>>> >>>> + Jonanthan >>>> >>>> Hi Seth, >>>> >>>> We finally got chance to test with more logs enabled. You are >>>> correct that that problematic buffer does sit on 2 registered memory r= egions: >>>> >>>> The problematic buffer is "Buffer address:*200019bfeb00", *actual >>>> used buffer pointer is "*200019bff000*" (SPDK makes it 4KiB >>>> aligned), size is >>>> 8KiB(0x2000) so it does sit on 2 registered memory region. >>>> >>>> However, looks like SPDK/DPDK allocates buffers starting from end of >>>> a region and going up, but due to the extra room and alignment of >>>> each buffer and there is chance that one buffer can exceed memory >>>> region boundary? >>>> >>>> In this case, the buffers are between 0x200019997800 and >>>> 0x200019c5320 so that last buffer exceeds one region and goes to next = one. >>>> >>>> Some logs for your information: >>>> >>>> rdma.c:1279:spdk_nvmf_rdma_mem_notify: *NOTICE*: memory region start: >>>> 200019800000, memory region length: 400000 >>>> rdma.c:1279:spdk_nvmf_rdma_mem_notify: *NOTICE*: memory region start: >>>> 200019c00000, memory region length: 400000 >>>> >>>> ... >>>> >>>> transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: >>>> 0x200019bfeb00 27(32) >>>> >>>> ... >>>> >>>> rdma.c:1508:nvmf_rdma_fill_buffers: *NOTICE*: Buffer >>>> address:**200019bfeb00**iov_base address 200019bff000 >>>> >>>> Thanks, >>>> >>>> JD >>>> >>>> On Thu, Aug 1, 2019 at 2:22 PM Howell, Seth >>> > wrote: >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 There are two different assignments that you= need to look at. >>>> I'll >>>> =C2=A0=C2=A0=C2=A0=C2=A0 detail the cases below based on line numbers= from the latest >>>> master. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Memory.c:656 *size =3D spdk_min(*size, cur_s= ize): >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 This assig= nment is inside of the conditional "if(size >>>> =3D=3D >>>> =C2=A0=C2=A0=C2=A0=C2=A0 NULL || map->ops.are_contiguous =3D=3D NULL)" >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 So in othe= r words, at the offset, we figure out how >>>> much >>>> =C2=A0=C2=A0=C2=A0=C2=A0 space we have left in the current translatio= n. Then, if there >>>> is no >>>> =C2=A0=C2=A0=C2=A0=C2=A0 callback to tell us whether the next transla= tion will be >>>> contiguous >>>> =C2=A0=C2=A0=C2=A0=C2=A0 to this one, we fill the size variable with = the remaining >>>> length of >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that 2 MiB buffer. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Memory.c:682 *size =3D spdk_min(*size, cur_s= ize): >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 This assig= nment comes after the while loop guarded by >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 condition "while (cur_size < *size)". This w= hile loop assumes >>>> that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 we have supplied some desired length for our= buffer. This is >>>> true in >>>> =C2=A0=C2=A0=C2=A0=C2=A0 the RDMA case. Now this while loop will only= break on two >>>> =C2=A0=C2=A0=C2=A0=C2=A0 conditions. 1. Cur_size becomes larger than = *size, or the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 are_contiguous function returns false, meani= ng that the two >>>> =C2=A0=C2=A0=C2=A0=C2=A0 translations cannot be considered together. = In the case of the >>>> RDMA >>>> =C2=A0=C2=A0=C2=A0=C2=A0 memory map, the only time are_contiguous ret= urns false is when >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 two memory regions correspond to two distinc= t RDMA MRs. Notice >>>> that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 in this case - the one where are_contiguous = is defined and we >>>> =C2=A0=C2=A0=C2=A0=C2=A0 supplied a size variable - the *size variabl= e is not >>>> overwritten >>>> =C2=A0=C2=A0=C2=A0=C2=A0 with cur_size until 1. cur_size is >=3D *siz= e or 2. The >>>> are_contiguous >>>> =C2=A0=C2=A0=C2=A0=C2=A0 check fails. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 In the second case detailed above, you can s= ee how one could >>>> pass >>>> =C2=A0=C2=A0=C2=A0=C2=A0 in a buffer that spanned a 2 MiB page and st= ill get a >>>> translation >>>> =C2=A0=C2=A0=C2=A0=C2=A0 value equal to the size of the buffer. This = second case is the >>>> one >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that the rdma.c code should be using since w= e have a registered >>>> =C2=A0=C2=A0=C2=A0=C2=A0 are_contiguous function with the NIC and we = have supplied a >>>> size >>>> =C2=A0=C2=A0=C2=A0=C2=A0 pointer filled with the length of our buffer. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 -----Original Message----- >>>> =C2=A0=C2=A0=C2=A0=C2=A0 From: JD Zheng [mailto:jiandong.zheng(a)broa= dcom.com >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ] >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Sent: Thursday, August 1, 2019 2:01 PM >>>> =C2=A0=C2=A0=C2=A0=C2=A0 To: Howell, Seth >>> =C2=A0=C2=A0=C2=A0=C2=A0 >; Storage P= erformance >>>> Development Kit >>>> =C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Subject: Re: [SPDK] nvmf_tgt *ERROR*: Data b= uffer split over >>>> =C2=A0=C2=A0=C2=A0=C2=A0 multiple RDMA Memory Regions >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Hi Seth, >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0> Just because a buffer extends = past a 2 MiB boundary doesn't >>>> mean >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that it exists in two different Memory Regio= ns. It also won't >>>> fail >>>> =C2=A0=C2=A0=C2=A0=C2=A0 the translation for being over two memory re= gions. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 This makes sense. However, spdk_mem_map_tran= slate() does >>>> following >>>> =C2=A0=C2=A0=C2=A0=C2=A0 to calculate translation_len: >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 cur_size =3D VALUE_2MB - _2MB_OFFSET(vaddr);= ... >>>> =C2=A0=C2=A0=C2=A0=C2=A0 *size =3D spdk_min(*size, cur_size); // *siz= e is the >>>> translation_len >>>> =C2=A0=C2=A0=C2=A0=C2=A0 from caller nvmf_rdma_fill_buffers() >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 In nvmf_rdma_fill_buffers(), >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 if (translation_len < rdma_req->req.iov[iovc= nt].iov_len) { >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SPDK_ERRLOG("Data buffer spli= t over >>>> =C2=A0=C2=A0=C2=A0=C2=A0 multiple RDMA Memory Regions\n"); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return -EINVAL; >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 } >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 This just checks if buffer sits on 2 2MB pag= es, not about 2 >>>> RDMA >>>> =C2=A0=C2=A0=C2=A0=C2=A0 memory regions. Is my understanding correct? >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 I still need some time to test. I will updat= e you the result >>>> with -s >>>> =C2=A0=C2=A0=C2=A0=C2=A0 as well. >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0 JD >>>> >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0 On 8/1/19 1:28 PM, Howell, Seth wrote: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Hi JD, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > The 2 MiB check is just because we a= lways do memory >>>> registrations >>>> =C2=A0=C2=A0=C2=A0=C2=A0 at at least 2 MiB granularity (the minimum h= ugepage size). Just >>>> =C2=A0=C2=A0=C2=A0=C2=A0 because a buffer extends past a 2 MiB bounda= ry doesn't mean >>>> that it >>>> =C2=A0=C2=A0=C2=A0=C2=A0 exists in two different Memory Regions. It a= lso won't fail the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 translation for being over two memory region= s. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > If you look at the definition of spd= k_mem_map_translate we >>>> call >>>> =C2=A0=C2=A0=C2=A0=C2=A0 map->ops->are_contiguous every time we cross= a 2 MiB boundary. >>>> For >>>> =C2=A0=C2=A0=C2=A0=C2=A0 RDMA, this function is registered to >>>> =C2=A0=C2=A0=C2=A0=C2=A0 spdk_nvmf_rdma_check_contiguous_entries. IF = this function >>>> returns >>>> =C2=A0=C2=A0=C2=A0=C2=A0 true, then even if the buffer crosses a 2 Mi= B boundary, the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 translation will still be valid. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > The problem you are running into is = not related to the >>>> buffer >>>> =C2=A0=C2=A0=C2=A0=C2=A0 alignment, it is related to the fact that th= e two pages across >>>> which >>>> =C2=A0=C2=A0=C2=A0=C2=A0 the buffer is split are registered to two di= fferent MRs in the >>>> NIC. >>>> =C2=A0=C2=A0=C2=A0=C2=A0 This can only happen if those two pages are = allocated >>>> independently >>>> =C2=A0=C2=A0=C2=A0=C2=A0 and trigger two distinct memory event callba= cks. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > That is why I am so interested in se= eing the results from >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 noticelog above ibv_reg_mr. It will tell me = how your target >>>> =C2=A0=C2=A0=C2=A0=C2=A0 application is allocating memory. Also, when= you start the SPDK >>>> =C2=A0=C2=A0=C2=A0=C2=A0 target, are you using the -s option? Somethi= ng like >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ./app/nvmf_tgt/nvmf_tgt -s 512 or something = like that (I don't >>>> know >>>> =C2=A0=C2=A0=C2=A0=C2=A0 if it'll make a difference, it's more of a c= uriosity thing for >>>> me)? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Seth >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > -----Original Message----- >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > From: JD Zheng [mailto:jiandong.zhen= g(a)broadcom.com >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ] >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Sent: Thursday, August 1, 2019 11:24= AM >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > To: Howell, Seth >>> =C2=A0=C2=A0=C2=A0=C2=A0 >; Storage P= erformance >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Development Kit >>> > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Subject: Re: [SPDK] nvmf_tgt *ERROR*= : Data buffer split over >>>> =C2=A0=C2=A0=C2=A0=C2=A0 multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > RDMA Memory Regions >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Hi Seth, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Thanks for the detailed description,= now I understand the >>>> reason >>>> =C2=A0=C2=A0=C2=A0=C2=A0 behind the checking. But I have a question, = why checking >>>> against >>>> =C2=A0=C2=A0=C2=A0=C2=A0 2MiB? Is it because DPDK uses 2MiB page size= by default so that >>>> one >>>> =C2=A0=C2=A0=C2=A0=C2=A0 RDMA memory region should not cross 2 pages? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >=C2=A0 =C2=A0> Once I see what your m= emory registrations look like and >>>> what >>>> =C2=A0=C2=A0=C2=A0=C2=A0 addresses you're failing on, it will help me= understand what is >>>> =C2=A0=C2=A0=C2=A0=C2=A0 going on better. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > I've added some print in nvmf_rdma_f= ill_buffers() @@ -1502,7 >>>> =C2=A0=C2=A0=C2=A0=C2=A0 +1503,11 @@ nvmf_rdma_fill_buffers(struct >>>> spdk_nvmf_rdma_transport >>>> =C2=A0=C2=A0=C2=A0=C2=A0 *rtransport, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0remaining_length -=3D >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > rdma_req->req.iov[iovcnt].iov_len; >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (translation_len < >>>> =C2=A0=C2=A0=C2=A0=C2=A0 rdma_req->req.iov[iovcnt].iov_len) { >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > -=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SPDK_ERRLOG("Data buffer sp= lit over >>>> multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > RDMA Memory Regions\n"); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > +=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SPDK_ERRLOG("Data buffer sp= lit over >>>> multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > RDMA Memory Regions %p %d (%d) (%d) = (%d) (%d)\n", >>>> =C2=A0=C2=A0=C2=A0=C2=A0 rdma_req->buffers[iovcnt], iovcnt, length, r= emaining_length, >>>> =C2=A0=C2=A0=C2=A0=C2=A0 translation_len, rdma_req->req.iov[iovcnt].i= ov_len); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0return -EINVA= L; >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0} >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > With this I can see which buffer fai= led the checking. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > For example, when SPKD initializes t= he memory pool, one of >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 buffers starts with 0x2000193feb00, and when= failed, I got >>>> following: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > rdma.c:1510:nvmf_rdma_fill_buffers: = *ERROR*: Data buffer >>>> split over >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > multiple RDMA Memory Regions 0x20001= 93feb00 0 (8192) (0) >>>> (5376) >>>> =C2=A0=C2=A0=C2=A0=C2=A0 (8192) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > This buffer has 5376B on one 2MB pag= e and the rest of it >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > (8192-5376=3D2816B) is on another pa= ge. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > The change https://review.gerrithub.= io/c/spdk/spdk/+/463893 >>>> to >>>> =C2=A0=C2=A0=C2=A0=C2=A0 use iov base should make it better as iov ba= se is 4KiB aligned. >>>> In >>>> =C2=A0=C2=A0=C2=A0=C2=A0 above case, iov_base is 0x2000193feb00 & 0xf= ff =3D 0x2000193fe000 >>>> and >>>> =C2=A0=C2=A0=C2=A0=C2=A0 it should pass the checking. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > However, another buffer in the pool = is 0x2000192010c0 and >>>> =C2=A0=C2=A0=C2=A0=C2=A0 iov_base is 0x200019201000, which would fail= the checking >>>> because it >>>> =C2=A0=C2=A0=C2=A0=C2=A0 is only 4KiB to 2MB boundary and IOUnitSize = is 8KiB. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > I will add the change from >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > https://review.gerrithub.io/c/spdk/s= pdk/+/463892 and rerun >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 test to get more information. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > I also attached the conf file too. T= he cmd line is "nvmf_tgt >>>> -m 0xff >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > -j >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > 0x90000000:0x20000000 -c 16disk_1ns.= conf" >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > JD >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 > On 8/1/19 7:52 AM, Howell, Seth wrot= e: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Hi JD, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> I was doing a little bit of digging= in the dpdk >>>> documentation >>>> =C2=A0=C2=A0=C2=A0=C2=A0 around this process, and I have a little bit= more information. >>>> We >>>> =C2=A0=C2=A0=C2=A0=C2=A0 were pretty worried about the whole dynamic = memory allocations >>>> thing >>>> =C2=A0=C2=A0=C2=A0=C2=A0 a few releases ago, so Jim helped add a flag= into DPDK that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 prevented allocations from being allocated a= nd freed in >>>> different >>>> =C2=A0=C2=A0=C2=A0=C2=A0 granularities. This flag also prevents mallo= c heap allocations >>>> from >>>> =C2=A0=C2=A0=C2=A0=C2=A0 spanning multiple memory events. However, th= is flag didn't make >>>> it >>>> =C2=A0=C2=A0=C2=A0=C2=A0 into DPDK until 19.02 (More documentation at >>>> https://doc.dpdk.org/guides/prog_guide/env_abstraction_layer.html#en >>>> vironment-abstraction-layer >>>> =C2=A0=C2=A0=C2=A0=C2=A0 if you're interested). We have some code in = the SPDK >>>> environment >>>> =C2=A0=C2=A0=C2=A0=C2=A0 layer that tries to deal with that (see >>>> =C2=A0=C2=A0=C2=A0=C2=A0 lib/env_dpdk/memory.c:memory_hotplug_cb) but= I don't know that >>>> that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 function is entirely capable of handling the= heap allocations >>>> =C2=A0=C2=A0=C2=A0=C2=A0 spanning multiple memory events part of the = problem. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Since you are using dpdk 18.11, the= memory callback inside >>>> of >>>> =C2=A0=C2=A0=C2=A0=C2=A0 lib/env_dpdk looks like a good candidate for= our issue. My best >>>> =C2=A0=C2=A0=C2=A0=C2=A0 guess is that somehow a heap allocation from= the buffer mempool >>>> is >>>> =C2=A0=C2=A0=C2=A0=C2=A0 hitting across addresses from two dynamic me= mory allocation >>>> events. >>>> =C2=A0=C2=A0=C2=A0=C2=A0 I'd still appreciate it if you could send me= the information in >>>> my >>>> =C2=A0=C2=A0=C2=A0=C2=A0 last e-mail, but I think we're onto somethin= g here. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Seth >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> -----Original Message----- >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> From: SPDK [mailto:spdk-bounces(a)l= ists.01.org >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ] On Beh= alf Of Howell, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Seth >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Sent: Thursday, August 1, 2019 5:26= AM >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> To: JD Zheng >>> =C2=A0=C2=A0=C2=A0=C2=A0 >; Sto= rage Performance >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Development Kit >>> > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Subject: Re: [SPDK] nvmf_tgt *ERROR= *: Data buffer split >>>> over >>>> =C2=A0=C2=A0=C2=A0=C2=A0 multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> RDMA Memory Regions >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Hi JD, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Thanks for doing that. Yeah, I am m= ainly looking to see how >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 mempool addresses are mapped into the NIC wi= th ibv_reg_mr. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> I think it's odd that we are using = the buffer base for the >>>> memory >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> check, we should be using the iov b= ase, but I don't believe >>>> that >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> would cause the issue you are seein= g. Pushed a change to >>>> modify >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> behavior anyways though: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> https://review.gerrithub.io/c/spdk/= spdk/+/463893 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> There was one registration that I w= asn't able to catch from >>>> your >>>> =C2=A0=C2=A0=C2=A0=C2=A0 last log. Sorry about that, I forgot there w= asn=E2=80=99t a debug log >>>> for >>>> =C2=A0=C2=A0=C2=A0=C2=A0 it. Can you try it again with this change wh= ich adds noticelogs >>>> for >>>> =C2=A0=C2=A0=C2=A0=C2=A0 the relevant registrations. >>>> =C2=A0=C2=A0=C2=A0=C2=A0 https://review.gerrithub.io/c/spdk/spdk/+/46= 3892 You should be >>>> able >>>> =C2=A0=C2=A0=C2=A0=C2=A0 to run your test without the -Lrdma argument= this time to avoid >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 extra bloat in the logs. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> The underlying assumption of the co= de is that any given >>>> object >>>> =C2=A0=C2=A0=C2=A0=C2=A0 is not going to cross a dynamic memory alloc= ation from DPDK. >>>> For a >>>> =C2=A0=C2=A0=C2=A0=C2=A0 little background, when the mempool gets cre= ated, the dpdk code >>>> =C2=A0=C2=A0=C2=A0=C2=A0 allocates some number of memzones to accommo= date those buffer >>>> =C2=A0=C2=A0=C2=A0=C2=A0 objects. Then it passes those memzones down = one at a time and >>>> places >>>> =C2=A0=C2=A0=C2=A0=C2=A0 objects inside the mempool from the given me= mzone until the >>>> memzone >>>> =C2=A0=C2=A0=C2=A0=C2=A0 is exhausted. Then it goes back and grabs an= other memzone. This >>>> =C2=A0=C2=A0=C2=A0=C2=A0 process continues until all objects are acco= unted for. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> This only works if each memzone cor= responds to a single >>>> memory >>>> =C2=A0=C2=A0=C2=A0=C2=A0 event when using dynamic memory allocation. = My understanding >>>> was >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that this was always the case, but this erro= r makes me think >>>> that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 it's possible that that's not true. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Once I see what your memory registr= ations look like and >>>> what >>>> =C2=A0=C2=A0=C2=A0=C2=A0 addresses you're failing on, it will help me= understand what is >>>> =C2=A0=C2=A0=C2=A0=C2=A0 going on better. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Can you also provide the command li= ne you are using to >>>> start the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 nvmf_tgt application and attach your configu= ration file? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Seth >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> -----Original Message----- >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> From: JD Zheng [mailto:jiandong.zhe= ng(a)broadcom.com >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ] >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Sent: Wednesday, July 31, 2019 3:13= PM >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> To: Howell, Seth >>> =C2=A0=C2=A0=C2=A0=C2=A0 >; Storage P= erformance >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Development Kit >>> > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Subject: Re: [SPDK] nvmf_tgt *ERROR= *: Data buffer split >>>> over >>>> =C2=A0=C2=A0=C2=A0=C2=A0 multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> RDMA Memory Regions >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Hi Seth, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> After I enabled debug and ran nvmf_= tgt with -L rdma, I got >>>> some >>>> =C2=A0=C2=A0=C2=A0=C2=A0 logs like: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 746:nvmf_rdma_resources_cre= ate: *DEBUG*: Command >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x2000084bf000 Length: 40000 LKey: = e601 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 749:nvmf_rdma_resources_cre= ate: *DEBUG*: Completion >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x200008621000 Length: 10000 LKey: = e701 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 753:nvmf_rdma_resources_cre= ate: *DEBUG*: In Capsule >>>> Data >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x200018600000 Length: 1000000 LKey= : e801 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 746:nvmf_rdma_resources_cre= ate: *DEBUG*: Command >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20000847e000 Length: 40000 LKey: = e701 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 749:nvmf_rdma_resources_cre= ate: *DEBUG*: Completion >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20000846d000 Length: 10000 LKey: = e801 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 753:nvmf_rdma_resources_cre= ate: *DEBUG*: In Capsule >>>> Data >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x200019800000 Length: 1000000 LKey= : e901 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 746:nvmf_rdma_resources_cre= ate: *DEBUG*: Command >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x200016ebb000 Length: 40000 LKey: = e801 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 749:nvmf_rdma_resources_cre= ate: *DEBUG*: Completion >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20000845c000 Length: 10000 LKey: = e901 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 753:nvmf_rdma_resources_cre= ate: *DEBUG*: In Capsule >>>> Data >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20001aa00000 Length: 1000000 LKey= : ea01 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 746:nvmf_rdma_resources_cre= ate: *DEBUG*: Command >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x200016e7a000 Length: 40000 LKey: = e901 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 749:nvmf_rdma_resources_cre= ate: *DEBUG*: Completion >>>> Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20000844b000 Length: 10000 LKey: = ea01 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> rdma.c: 753:nvmf_rdma_resources_cre= ate: *DEBUG*: In Capsule >>>> Data >>>> =C2=A0=C2=A0=C2=A0=C2=A0 Array: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> 0x20001bc00000 Length: 1000000 LKey= : eb01 ... >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Is this you are look for as memory = regions registered for NIC? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> I attached the complete log. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> JD >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> On 7/30/19 5:28 PM, JD Zheng wrote: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Hi Seth, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Thanks for the prompt reply! >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Please find answers inline. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> JD >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> On 7/30/19 5:01 PM, Howell, Seth w= rote: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Hi JD, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Thanks for the report. I want to = ask a few questions to >>>> start >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> getting to the bottom of this. Si= nce this issue doesn't >>>> currently >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> reproduce on our per-patch or nig= htly tests, I would like >>>> to >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> understand what's unique about yo= ur setup so that we can >>>> =C2=A0=C2=A0=C2=A0=C2=A0 replicate >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> it in a per patch test to prevent= future regressions. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> I am running it on aarch64 platfor= m. I tried x86 platform >>>> and I >>>> =C2=A0=C2=A0=C2=A0=C2=A0 can >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> see same buffer alignment in memor= y pool but can't run the >>>> real >>>> =C2=A0=C2=A0=C2=A0=C2=A0 test >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> to reproduce it due to other missi= ng pieces. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> What options are you passing when= you create the rdma >>>> transport? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Are you creating it over RPC or i= n a configuration file? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> I am using conf file. Pls let me k= now if you'd like to >>>> look >>>> =C2=A0=C2=A0=C2=A0=C2=A0 into conf file. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Are you using the current DPDK su= bmodule as your >>>> environment >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> abstraction layer? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> No. Our project uses specific vers= ion of DPDK, which is >>>> v18.11. I >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> did quick test using latest and DP= DK submodule on x86, and >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> buffer alignment is the same, i.e.= 64B aligned. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> I notice that your error log is p= rinting from >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> spdk_nvmf_transport_poll_group_cr= eate, which value >>>> exactly are >>>> =C2=A0=C2=A0=C2=A0=C2=A0 you >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> printing out? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Here is patch to add dbg print. Pl= s note that SPDK version >>>> is >>>> =C2=A0=C2=A0=C2=A0=C2=A0 v19.04 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> @@ -215,6 +222,7 @@ >>>> spdk_nvmf_transport_poll_group_create(st >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> SPDK_NOTICELOG("Unable to >>>> =C2=A0=C2=A0=C2=A0=C2=A0 reserve >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> the full number of buffers for the= pg buffer cache.\n"); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>=C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 break; >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>=C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 SPDK_ERRLOG("%p %d(%d)\n", buf, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> group->buf_cache_count, group->buf= _cache_size); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> STAILQ_INSERT_HEAD(&group->buf_cac= he, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> buf, link); >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> group->buf_cache_count++; >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>=C2=A0 =C2=A0 =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Can you run your target with the = -L rdma option to get a >>>> dump of >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> the memory regions registered wit= h the NIC? >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Let me test and get back to you so= on. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> We made a couple of changes to th= is code when dynamic >>>> memory >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> allocations were added to DPDK. T= here were some >>>> safeguards >>>> =C2=A0=C2=A0=C2=A0=C2=A0 that we >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> added to try and make sure this c= ase wouldn't hit, so I'd >>>> like to >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> make sure you are running on the = latest DPDK submodule as >>>> well as >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> the latest SPDK to narrow down wh= ere we need to look. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> Unfortunately I can't easily updat= e DPDK because other >>>> team >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> maintains it internally. But if it= can be repro and fixed >>>> in >>>> =C2=A0=C2=A0=C2=A0=C2=A0 latest, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> I will try to pull in the fix. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Seth >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> -----Original Message----- >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> From: SPDK [mailto:spdk-bounces(a= )lists.01.org >>>> =C2=A0=C2=A0=C2=A0=C2=A0 ] On Beh= alf Of JD Zheng >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> via SPDK >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Sent: Wednesday, July 31, 2019 3:= 00 AM >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> To: spdk(a)lists.01.org >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Cc: JD Zheng >>> =C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Subject: [SPDK] nvmf_tgt *ERROR*:= Data buffer split over >>>> multiple >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> RDMA Memory Regions >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Hello, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> When I run nvmf_tgt over RDMA usi= ng latest SPDK code, I >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> occasionally ran into this errors: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> "rdma.c:1505:nvmf_rdma_fill_buffe= rs: *ERROR*: Data buffer >>>> split >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> over multiple RDMA Memory Regions" >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> After digging into the code, I fo= und that >>>> =C2=A0=C2=A0=C2=A0=C2=A0 nvmf_rdma_fill_buffers() >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> calls=C2=A0spdk_mem_map_translate= () to check if a data buffer >>>> sit on 2 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 2MB pages, and if it is the case,= it reports this error. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> The following commit added change= to use data buffer >>>> start >>>> =C2=A0=C2=A0=C2=A0=C2=A0 address >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> to calculate the size between buf= fer start address and >>>> 2MB >>>> =C2=A0=C2=A0=C2=A0=C2=A0 boundary. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> The caller=C2=A0nvmf_rdma_fill_bu= ffers() uses the size to >>>> compare with >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> IO Unit size (which is 8KB in my = conf) to determine if >>>> the buffer >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> passes 2MB boundary. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> commit 37b7a308941b996f0e69049358= a6119ed90d70a2 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Author: Darek Stojaczyk >>> =C2=A0=C2=A0=C2=A0=C2=A0 > >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Date: =C2=A0 Tue Nov 13 17:43:46 = 2018 +0100 >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>>=C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 = memory: fix contiguous memory calculation for >>>> unaligned >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> buffers >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> In nvmf_tgt, the buffers are pre-= allocated as a memory >>>> pool >>>> =C2=A0=C2=A0=C2=A0=C2=A0 and new >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> request will use free buffer from= that pool and the >>>> buffer start >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> address is passed to=C2=A0nvmf_rd= ma_fill_buffers(). But I >>>> found that >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> these buffers are not 2MB aligned= and not IOUnitSize >>>> aligned (8KB >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> in my >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> case) either, instead, they are 6= 4Byte aligned so that >>>> some >>>> =C2=A0=C2=A0=C2=A0=C2=A0 buffers >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> will fail the checking and leads = to this problem. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> The corresponding code snippets a= re as following: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> spdk_nvmf_transport_create() >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> { >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> ... >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>>=C2=A0 =C2=A0=C2=A0 =C2=A0 =C2=A0 = transport->data_buf_pool =3D >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> pdk_mempool_create(spdk_mempool_n= ame, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> =C2=A0opts->num_shared_buffers, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> =C2=A0opts->io_unit_size + >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> NVMF_DATA_BUFFER_ALIGNMENT, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =C2=A0SPDK_MEMPOOL_DEFAULT_CACHE_SIZE, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> =C2=A0SPDK_ENV_SOCKET_ID_ANY); ... >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> } >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Also some debug print I added sho= ws the start address of >>>> the >>>> =C2=A0=C2=A0=C2=A0=C2=A0 buffers: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x200019258800 0(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x2000192557c0 1(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x200019252780 2(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x20001924f740 3(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x20001924c700 4(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x2000192496c0 5(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x200019246680 6(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x200019243640 7(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x200019240600 8(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> transport.c: 218:spdk_nvmf_transp= ort_poll_group_create: >>>> *ERROR*: >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> 0x20001923d5c0 9(32) >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> ... >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> It looks like either the buffer a= llocation has alignment >>>> issue or >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> the checking is not correct. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Please advice how to fix this pro= blem. >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> Thanks, >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> JD Zheng >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> _________________________________= ______________ >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> SPDK mailing list >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> SPDK(a)lists.01.org >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> https://lists.01.org/mailman/list= info/spdk >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >>>> >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> ___________________________________= ____________ >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> SPDK mailing list >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> SPDK(a)lists.01.org >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> https://lists.01.org/mailman/listin= fo/spdk >>>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 >> >>>> >> _______________________________________________ >> SPDK mailing list >> SPDK(a)lists.01.org >> https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk > _______________________________________________ > SPDK mailing list > SPDK(a)lists.01.org > https://lists.01.org/mailman/listinfo/spdk --===============2824513023477061483==--