[SPDK] Memory Buffer Allocation in SPDK

[SPDK] Memory Buffer Allocation in SPDK

[yzhu]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

Hi All,

Is there any way to directly use the memory allocated by the malloc() 
function? The spdk_malloc() allocates a pinned memory buffer in huge 
page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for 
write. However, if we allocate a memory buffer by malloc(), how could we 
directly write data to NVMe device by SPDK without copying data to the 
buffer allocated by spdk_malloc()?

Thanks,
Yue

Re: [SPDK] Memory Buffer Allocation in SPDK

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]


> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
> 
> Hi All,
> 
> Is there any way to directly use the memory allocated by the malloc() function? The spdk_malloc() allocates a pinned memory buffer in huge page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for write. However, if we allocate a memory buffer by malloc(), how could we directly write data to NVMe device by SPDK without copying data to the buffer allocated by spdk_malloc()?
> 

Hi Yue,

There is no safe way to use a malloc() buffer for DMA, since the buffer could be swapped out at any time.  mlock() can be used to ensure the buffer stays present in RAM, but it does not guarantee the buffer will not get migrated to a different physical location.  mlock() also has a lot of software overhead and is not something you would want to call for every I/O.

-Jim

Re: [SPDK] Memory Buffer Allocation in SPDK

[yzhu]

[-- Attachment #1: Type: text/plain, Size: 1805 bytes --]

Hi Jim,

Thanks for your quick reply. So I was wondering if spdk_malloc() is the 
only safe and correct way to allocate buffer for writing to NVMe device 
via spdk_nvme_ns_cmd_write.

Previously, I tried to lock the buffer allocated by malloc() in RAM by 
mlock(), and immediately write the buffer to NVMe device by 
spdk_nvme_ns_cmd_write. However, the write failed because of the failure 
of converting virtual address to physical address. The error was "could 
not find 2MB vfn 0x11 in DPDK mem config". Based on this, I was also 
wondering how to correctly write to NVMe device via spdk if we still 
have to use the buffer allocated by malloc()?

Thank you,
Yue





On 2016-12-12 23:46, Harris, James R wrote:
>> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
>> 
>> Hi All,
>> 
>> Is there any way to directly use the memory allocated by the malloc() 
>> function? The spdk_malloc() allocates a pinned memory buffer in huge 
>> page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for 
>> write. However, if we allocate a memory buffer by malloc(), how could 
>> we directly write data to NVMe device by SPDK without copying data to 
>> the buffer allocated by spdk_malloc()?
>> 
> 
> Hi Yue,
> 
> There is no safe way to use a malloc() buffer for DMA, since the
> buffer could be swapped out at any time.  mlock() can be used to
> ensure the buffer stays present in RAM, but it does not guarantee the
> buffer will not get migrated to a different physical location.
> mlock() also has a lot of software overhead and is not something you
> would want to call for every I/O.
> 
> -Jim
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

Re: [SPDK] Memory Buffer Allocation in SPDK

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 2713 bytes --]


> On Dec 13, 2016, at 8:49 AM, yzhu <yzhu(a)cs.fsu.edu> wrote:
> 
> Hi Jim,
> 
> Thanks for your quick reply. So I was wondering if spdk_malloc() is the only safe and correct way to allocate buffer for writing to NVMe device via spdk_nvme_ns_cmd_write.

spdk_malloc() is the best way to allocate the buffer for use with the SPDK NVMe API.  One alternative is to pass your own 2MB-aligned pinned memory buffer, consisting of >= 2MB huge pages, to the spdk_vtophys_register() function - this will insert the correct vtophys mappings into the SPDK userspace page table.  But spdk_vtophys_register() will not work with buffers allocated from standard malloc() which will not be 2MB aligned, pinned, nor guaranteed to be allocated from huge pages.

> 
> Previously, I tried to lock the buffer allocated by malloc() in RAM by mlock(), and immediately write the buffer to NVMe device by spdk_nvme_ns_cmd_write. However, the write failed because of the failure of converting virtual address to physical address. The error was "could not find 2MB vfn 0x11 in DPDK mem config". Based on this, I was also wondering how to correctly write to NVMe device via spdk if we still have to use the buffer allocated by malloc()?

SPDK is not designed to handle buffers allocated by standard malloc().  Would using glibc malloc hooks or overriding malloc() such that malloc()/free() calls get forwarded to spdk_malloc()/spdk_free() help in your situation?

> 
> Thank you,
> Yue
> 
> 
> 
> 
> 
> On 2016-12-12 23:46, Harris, James R wrote:
>>> On Dec 12, 2016, at 9:13 PM, yzhu <yzhu(a)cs.fsu.edu> wrote:
>>> Hi All,
>>> Is there any way to directly use the memory allocated by the malloc() function? The spdk_malloc() allocates a pinned memory buffer in huge page, and the spdk_nvme_ns_cmd_write also uses the pinned memory for write. However, if we allocate a memory buffer by malloc(), how could we directly write data to NVMe device by SPDK without copying data to the buffer allocated by spdk_malloc()?
>> Hi Yue,
>> There is no safe way to use a malloc() buffer for DMA, since the
>> buffer could be swapped out at any time.  mlock() can be used to
>> ensure the buffer stays present in RAM, but it does not guarantee the
>> buffer will not get migrated to a different physical location.
>> mlock() also has a lot of software overhead and is not something you
>> would want to call for every I/O.
>> -Jim
>> _______________________________________________
>> SPDK mailing list
>> SPDK(a)lists.01.org
>> https://lists.01.org/mailman/listinfo/spdk
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] Memory buffer allocation in SPDK

[Ernest Zaslavsky]

[-- Attachment #1: Type: text/plain, Size: 592 bytes --]

Hi,
I’ve found this discussion https://lists.01.org/pipermail/spdk/2016-December/000251.html on memory allocation. I have similar situation – our application uses some allocator to create IO buffers, but of course, SPDK would not accept such a buffer. So I have two questions:
1) I cant find in code the spdk_vtophys_register() mentioned in the above link, is it removed? If not, where is it? How to use it?
2) In case our allocator uses hugepages and it is aligned on 4Kib boundary, is there a way to make the SPDK believe that this buffer could be used for NVMe IO?
Sincerely,
E.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3859 bytes --]

Re: [SPDK] Memory buffer allocation in SPDK

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 1315 bytes --]



From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zaslavsky <kreuzerkrieg(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, February 6, 2018 at 6:32 AM
To: "spdk(a)lists.01.org" <spdk(a)lists.01.org>
Subject: [SPDK] Memory buffer allocation in SPDK

Hi,
I’ve found this discussion https://lists.01.org/pipermail/spdk/2016-December/000251.html on memory allocation. I have similar situation – our application uses some allocator to create IO buffers, but of course, SPDK would not accept such a buffer. So I have two questions:
1)       I cant find in code the spdk_vtophys_register() mentioned in the above link, is it removed? If not, where is it? How to use it?

Hi Ernest,

spdk_mem_register() is the equivalent function now.  This will register the specified buffer not only for virtual-to-physical address translation, but also for RDMA operations using the SPDK NVMe-oF driver.

2)       In case our allocator uses hugepages and it is aligned on 4Kib boundary, is there a way to make the SPDK believe that this buffer could be used for NVMe IO?

spdk_mem_register() is the function to use.  Note that currently for vtophys translation, the specified buffer must be 2MiB aligned and an even 2MiB multiple.

Regards,

-Jim


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6530 bytes --]

Re: [SPDK] Memory buffer allocation in SPDK

[Ernest Zaslavsky]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

Hi James,
Thanks for the info, sounds like we have a way to solve our problem. I have some additional questions. The spdk_mem_register() function, it just adds the region to the SPDK allocation map, right? It does not touch or modifies the memory in any way, right? Is it safe to say that we may register with SPDK all our pre-allocated memory when the application starts and just keep it that way?
Sincerely,
E.


From: Harris, James R
Sent: Wednesday, February 7, 2018 2:11 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Memory buffer allocation in SPDK



From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zaslavsky <kreuzerkrieg(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, February 6, 2018 at 6:32 AM
To: "spdk(a)lists.01.org" <spdk(a)lists.01.org>
Subject: [SPDK] Memory buffer allocation in SPDK

Hi,
I’ve found this discussion https://lists.01.org/pipermail/spdk/2016-December/000251.html on memory allocation. I have similar situation – our application uses some allocator to create IO buffers, but of course, SPDK would not accept such a buffer. So I have two questions:
1) I cant find in code the spdk_vtophys_register() mentioned in the above link, is it removed? If not, where is it? How to use it?

Hi Ernest,

spdk_mem_register() is the equivalent function now.  This will register the specified buffer not only for virtual-to-physical address translation, but also for RDMA operations using the SPDK NVMe-oF driver.

2) In case our allocator uses hugepages and it is aligned on 4Kib boundary, is there a way to make the SPDK believe that this buffer could be used for NVMe IO?

spdk_mem_register() is the function to use.  Note that currently for vtophys translation, the specified buffer must be 2MiB aligned and an even 2MiB multiple.

Regards,

-Jim



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6716 bytes --]

Re: [SPDK] Memory buffer allocation in SPDK

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 910 bytes --]



From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zaslavsky <kreuzerkrieg(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, February 6, 2018 at 10:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Memory buffer allocation in SPDK

Hi James,
Thanks for the info, sounds like we have a way to solve our problem. I have some additional questions. The spdk_mem_register() function, it just adds the region to the SPDK allocation map, right? It does not touch or modifies the memory in any way, right? Is it safe to say that we may register with SPDK all our pre-allocated memory when the application starts and just keep it that way?

Hi Ernest,

Correct – SPDK does not touch the buffer registered with spdk_mem_register() in any way – it only adds it to any registered translation maps.

-Jim


[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4991 bytes --]

Re: [SPDK] Memory buffer allocation in SPDK

[Ernest Zaslavsky]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

Great! Thanks!


Sent from Mail for Windows 10

From: Harris, James R
Sent: Wednesday, February 7, 2018 5:00 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] Memory buffer allocation in SPDK



From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Ernest Zaslavsky <kreuzerkrieg(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, February 6, 2018 at 10:51 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] Memory buffer allocation in SPDK

Hi James,
Thanks for the info, sounds like we have a way to solve our problem. I have some additional questions. The spdk_mem_register() function, it just adds the region to the SPDK allocation map, right? It does not touch or modifies the memory in any way, right? Is it safe to say that we may register with SPDK all our pre-allocated memory when the application starts and just keep it that way?

Hi Ernest,

Correct – SPDK does not touch the buffer registered with spdk_mem_register() in any way – it only adds it to any registered translation maps.

-Jim



[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3736 bytes --]