From mboxrd@z Thu Jan  1 00:00:00 1970
Content-Type: multipart/mixed; boundary="===============0717912568673414360=="
MIME-Version: 1.0
From: JD Zheng <jiandong.zheng at broadcom.com>
Subject: [SPDK] nvmf_tgt *ERROR*: Data buffer split over multiple RDMA Memory
 Regions
Date: Tue, 30 Jul 2019 11:59:41 -0700
Message-ID: <0e490d33-0c63-f386-3e15-ad157cc3aaf6@broadcom.com>
List-ID: <spdk@lists.01.org>
To: spdk@lists.01.org

--===============0717912568673414360==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable

Hello,

When I run nvmf_tgt over RDMA using latest SPDK code, I occasionally ran =

into this errors:
"rdma.c:1505:nvmf_rdma_fill_buffers: *ERROR*: Data buffer split over =

multiple RDMA Memory Regions"

After digging into the code, I found that nvmf_rdma_fill_buffers() =

calls=C2=A0spdk_mem_map_translate() to check if a data buffer sit on 2 2MB =

pages, and if it is the case, it reports this error.

The following commit added change to use data buffer start address to =

calculate the size between buffer start address and 2MB boundary. The =

caller=C2=A0nvmf_rdma_fill_buffers() uses the size to compare with IO Unit =

size (which is 8KB in my conf) to determine if the buffer passes 2MB =

boundary.

commit 37b7a308941b996f0e69049358a6119ed90d70a2
Author: Darek Stojaczyk <dariusz.stojaczyk(a)intel.com>
Date: =C2=A0 Tue Nov 13 17:43:46 2018 +0100

 =C2=A0 =C2=A0 memory: fix contiguous memory calculation for unaligned buff=
ers

In nvmf_tgt, the buffers are pre-allocated as a memory pool and new =

request will use free buffer from that pool and the buffer start address =

is passed to=C2=A0nvmf_rdma_fill_buffers(). But I found that these buffers =

are not 2MB aligned and not IOUnitSize aligned (8KB in my case) either, =

instead, they are 64Byte aligned so that some buffers will fail the =

checking and leads to this problem.

The corresponding code snippets are as following:
spdk_nvmf_transport_create()
{
...
 =C2=A0 =C2=A0 transport->data_buf_pool =3D pdk_mempool_create(spdk_mempool=
_name,
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0opts->num_shared_buffers,
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0opts->io_unit_size + =

NVMF_DATA_BUFFER_ALIGNMENT,
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SPDK_MEMPOOL_DEFAULT_CACHE_SIZE,
 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0SPDK_ENV_SOCKET_ID_ANY);
...
}

Also some debug print I added shows the start address of the buffers:
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x200019258800 0(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x2000192557c0 1(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x200019252780 2(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x20001924f740 3(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x20001924c700 4(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x2000192496c0 5(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x200019246680 6(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x200019243640 7(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x200019240600 8(32)
transport.c: 218:spdk_nvmf_transport_poll_group_create: *ERROR*: =

0x20001923d5c0 9(32)
...

It looks like either the buffer allocation has alignment issue or the =

checking is not correct.

Please advice how to fix this problem.

Thanks,
JD Zheng

--===============0717912568673414360==--