numa_alloc_interleaved and expectation of round-robin allocation

numa_alloc_interleaved and expectation of round-robin allocation

[Ananya Muddukrishna]

Hi,

I am using numactl-2.0.8-rc3 on a large server machine with 8 NUMA nodes and 
8Gbyte of memory per node.

I ran a test application compiled using numactl-2.0.8-rc3/test/tshared.c. I 
understand that this piece of code allocates memory pages as interleaved on all 
numa nodes. Later the application gets the memory policy 
(MPOL_F_NODE|MPOL_F_ADDR) for the pages allocated and prints out deviations from 
round-robin allocation if any.

In my tests I find a huge bias and deviation from the expected round-robin 
allocation. Around 90% of the pages are allocated on node#0. I tried 
numa_set_strict(1) but numa_alloc_interleaved does not honor this flag in 
agreement with the documentation.

Please explain this behavior to me. Must I patch numactl to enforce round-robin 
strictness?

Thanks!
-- Ananya

Re: numa_alloc_interleaved and expectation of round-robin allocation

[Cliff Wickman]

Hi Ananya,

Yes, test/tshared.c does not work.
It is not executed in the regression tests so I never noticed.

But the fault is in the test code, not in libnuma.
If you change it as below it works.
The curious thing (to me) is that the interleaving with a mask of 0xf causes
allocations to begin on node 1.


#include <numa.h>
#include <numaif.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/wait.h>
#include <unistd.h>

#define err(x) perror(x),exit(1)

enum SZ { 
	MEMSZ = 100<<20, 
	NTHR = 10,
}; 

/* test if shared interleaving state works. */
int main(void)
{ 
	int i, k;
	char *mem;
	int pagesz = getpagesize();
	int max_node;
	int errs = 0;

	if (numa_available() < 0) {
		printf("no NUMA API available\n"); 
		exit(1);
	}
	max_node = numa_max_node(); 
	mem = numa_alloc_interleaved(MEMSZ); 
	for (i = 0; i < MEMSZ; i += pagesz) { 
		*(mem+i) = 1;
	} 
	k = 1; 
	for (i = 0; i < MEMSZ; i += pagesz) { 
		int nd; 
		if (get_mempolicy(&nd, NULL, 0, mem + i,
						MPOL_F_NODE|MPOL_F_ADDR) < 0) {
			err("get_mempolicy");
		}
		if (nd != k) {
			printf("%p offset %d node %d expected %d\n",
							mem+i, i, nd, k); 
			errs++;
		}
		k = (k+1)%(max_node+1); 
	} 
	if (errs == 0)
		printf("Success\n");
	
	return 0;
} 

-Cliff


On Thu, Jan 05, 2012 at 09:31:00AM +0000, Ananya Muddukrishna wrote:
> Hi,
> 
> I am using numactl-2.0.8-rc3 on a large server machine with 8 NUMA nodes and 
> 8Gbyte of memory per node.
> 
> I ran a test application compiled using numactl-2.0.8-rc3/test/tshared.c. I 
> understand that this piece of code allocates memory pages as interleaved on all 
> numa nodes. Later the application gets the memory policy 
> (MPOL_F_NODE|MPOL_F_ADDR) for the pages allocated and prints out deviations from 
> round-robin allocation if any.
> 
> In my tests I find a huge bias and deviation from the expected round-robin 
> allocation. Around 90% of the pages are allocated on node#0. I tried 
> numa_set_strict(1) but numa_alloc_interleaved does not honor this flag in 
> agreement with the documentation.
> 
> Please explain this behavior to me. Must I patch numactl to enforce round-robin 
> strictness?
> 
> Thanks!
> -- Ananya
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-numa" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824