Re: [SPDK] Hugepages deallocation

Re: [SPDK] Hugepages deallocation

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

On Sun, 2019-05-26 at 15:23 +0000, Gal Shalom wrote:
> Hello,
> 
> I encounter a initialization failure of SPDK v18.04, which I suspect the
> reason is the process of hugepages deallocation in SPDK.
> During my initialization I try to allocate a buffer of 1GB.
> The SPDK is able to initialized on the first few tries after a fresh reboot,
> but after I have killed it and started it over several times - it is unable to
> allocate the 1GB buffer. I suspect the SPDK isn't properly release the
> hugepages and they remain fragmented which is the reason for the failure of
> the allocation.
> 
> I am using spdk_dma_zmalloc for the buffer allocation and hugepages of 512MB,
> and would like any thoughts or suggestions about the error I get.

One suggestion I can come up with would be to debug this a bit by making a few
DPDK calls after initialization to walk through the allocated hugepages ("memory
segments") and test whether they are physically contiguous enough to perform a
1GB allocation. DPDK has facilities for walking through the memory map - with
the version shipped in SPDK v18.04, I think the code would look like this:

	struct rte_mem_config *mcfg;
	size_t seg_idx;

	mcfg = rte_eal_get_configuration()->mem_config;
	for (seg_idx = 0; seg_idx < RTE_MAX_MEMSEG; seg_idx++) {
		struct rte_memseg *seg = &mcfg->memseg[seg_idx];

		if (seg->addr == NULL) {
			break;
		}

		seg->phys_addr; // Physical address of memory segment here.
		seg->len; // Length of this segment
	}

Alternatively, the memory allocator has been rewritten in a more recent DPDK.
You could try upgrading to SPDK 19.04 (which uses DPDK 19.02) and see if the
problem persists.

> 
> Thanks,
> Gal
> 
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk

[SPDK] Hugepages deallocation

[Gal Shalom]

[-- Attachment #1: Type: text/plain, Size: 687 bytes --]

Hello,

I encounter a initialization failure of SPDK v18.04, which I suspect the reason is the process of hugepages deallocation in SPDK.
During my initialization I try to allocate a buffer of 1GB.
The SPDK is able to initialized on the first few tries after a fresh reboot, but after I have killed it and started it over several times - it is unable to allocate the 1GB buffer. I suspect the SPDK isn't properly release the hugepages and they remain fragmented which is the reason for the failure of the allocation.

I am using spdk_dma_zmalloc for the buffer allocation and hugepages of 512MB, and would like any thoughts or suggestions about the error I get.

Thanks,
Gal