Re: [PATCH] x86_64: Make the NUMA hash function nodemap allocation dynamic and remove NODEMAPSIZE

[Eric Dumazet <dada1@cosmosbay.com>]

On Friday 10 November 2006 07:48, Andi Kleen wrote:

> Have you checked how much the code .text size changes because
> of the pointer reference? If it's a lot phys_to_nid might need to
> be out of lined.

Here I have also big numbers on pfn_to_page(), on a machine with mapsize=1  
(NUMA kernel, but one node)

oprofile results L1_AND_L2_DTLB_MISSES /usr/src/linux-2.6.18/vmlinux

Counted L1_AND_L2_DTLB_MISSES events (L1 and L2 DTLB misses) with a unit mask 
of 0x00 (No unit mask) count 10000 

ffffffff80258fa0 <pfn_to_page>: /* pfn_to_page total:  48433  0.4914 */

So adding yet another indirection (to a another cache line) might hurt.

Therefore I suggest to use a structure like that :

struct memnode {
 	int shift;
	unsigned int mapsize; /* no need to use 8 bytes here */
	u8 *map;
	u8 embedded_map[64-8]; /* total size = 64 bytes */
 } ____cacheline_aligned;

and make memnode.map point to memnode.embedded_map if mapsize <= 56 ?

This way, most AMD64 dual/quad processors wont waste a full PAGE to store few 
bytes in it, and should use only one cache line.

Thank you

Eric