[PATCH] numactl: Support dumping nodes of pages in shared memory segments

[PATCH] numactl: Support dumping nodes of pages in shared memory segments

[Andi Kleen]

I got a request to be able to dump the nodes individually of a shared memory
segment. This can be very verbose for interleaving on large segments, but is still useful
for some analysis.

This patch implements that with a new --dump-nodes option 

Against numactl 2.0.3-rc3

-Andi

---
 numactl.8 |    5 +++++
 numactl.c |   13 ++++++++++---
 shm.c     |   31 +++++++++++++++++++++++++++++++
 shm.h     |    1 +
 4 files changed, 47 insertions(+), 3 deletions(-)

Index: numactl-2.0.3-rc3/numactl.c
===================================================================
--- numactl-2.0.3-rc3.orig/numactl.c
+++ numactl-2.0.3-rc3/numactl.c
@@ -49,6 +49,7 @@ struct option opts[] = {
 	{"strict", 0, 0, 't'},
 	{"shmmode", 1, 0, 'M'},
 	{"dump", 0, 0, 'd'},
+	{"dump-nodes", 0, 0, 'D'},
 	{"shmid", 1, 0, 'I'},
 	{"huge", 0, 0, 'u'},
 	{"touch", 0, 0, 'T'},
@@ -67,8 +68,8 @@ void usage(void)
 		"       numactl [--length length] [--offset offset] [--shmmode shmmode]\n"
 		"               [--strict]\n"
 		"               [--shmid id] --shm shmkeyfile | --file tmpfsfile\n"
-		"               [--huge] [--touch]\n" 
-		"               memory policy\n"
+		"               [--huge] [--touch] \n"
+		"               memory policy | --dump | --dump-nodes\n"
 		"\n"
 		"memory policy is --interleave, --preferred, --membind, --localalloc\n"
 		"nodes is a comma delimited list of node numbers or A-B ranges or all.\n"
@@ -466,6 +467,12 @@ int main(int ac, char **av)
 			dump_shm();
 			do_dump = 1;
 			break;
+		case 'D': /* --dump */
+			if (shmfd < 0)
+				complain("Cannot do --dump-nodes without shared memory.\n");
+			dump_shm_nodes();
+			do_dump = 1;
+			break;
 		case 't': /* --strict */
 			did_strict = 1;
 			numa_set_strict(1);
@@ -520,7 +527,7 @@ int main(int ac, char **av)
 		fprintf(stderr,"numactl: warning. Strict flag for process ignored.\n");
 
 	if (do_dump)
-		usage_msg("cannot do --dump for process");
+		usage_msg("cannot do --dump|--dump-shm for process");
 
 	if (shmoption)
 		usage_msg("shm related option %s for process", shmoption);
Index: numactl-2.0.3-rc3/shm.c
===================================================================
--- numactl-2.0.3-rc3.orig/shm.c
+++ numactl-2.0.3-rc3/shm.c
@@ -202,6 +202,37 @@ void dump_shm(void)
 	dumppol(start, c, prevpol, prevnodes);
 } 
 
+static void dumpnode(unsigned long long start, unsigned long long end, int node)
+{
+	printf("%016Lx-%016Lx: %d\n", shmoffset+start, shmoffset+end, node);
+}
+
+/* Dump nodes in a shared memory segment. */
+void dump_shm_nodes(void)
+{
+	int prevnode = -1, node;
+	unsigned long long c, start;
+
+	start = 0;
+	if (shmlen == 0) {
+		printf("nothing to dump\n");
+		return;
+	}
+
+	for (c = 0; c < shmlen; c += shm_pagesize) {
+		if (get_mempolicy(&node, NULL, 0, c+shmptr,
+						MPOL_F_ADDR|MPOL_F_NODE) < 0)
+			err("get_mempolicy on shm");
+		if (node == prevnode)
+			continue;
+		if (prevnode != -1)
+			dumpnode(start, c, prevnode);
+		prevnode = node;
+		start = c;
+	}
+	dumpnode(start, c, prevnode);
+}
+
 static void vwarn(char *ptr, char *fmt, ...) 
 { 
 	va_list ap;
Index: numactl-2.0.3-rc3/shm.h
===================================================================
--- numactl-2.0.3-rc3.orig/shm.h
+++ numactl-2.0.3-rc3/shm.h
@@ -8,6 +8,7 @@ extern unsigned long long shmoffset;
 extern int shmflags;
 
 extern void dump_shm(void);
+extern void dump_shm_nodes(void);
 extern void attach_shared(char *);
 extern void attach_sysvshm(char *);
 extern void verify_shm(int policy, struct bitmask *);
Index: numactl-2.0.3-rc3/numactl.8
===================================================================
--- numactl-2.0.3-rc3.orig/numactl.8
+++ numactl-2.0.3-rc3/numactl.8
@@ -64,6 +64,8 @@ numactl \- Control NUMA policy for proce
 .B \-\-touch
 ] [
 .B \-\-dump
+] [
+.B \-\-dump-nodes
 ]
 memory policy
 .SH DESCRIPTION
@@ -225,6 +227,9 @@ is applied when an applications maps and
 .B \-\-dump
 Dump policy in the specified range.
 .TP
+.B \-\-dump-nodes
+Dump all nodes of the specific range (very verbose!)
+.TP
 Valid node specifiers
 .TS
 tab(:);

-- 
ak@linux.intel.com -- Speaking for myself only.

Re: [PATCH] numactl: Support dumping nodes of pages in shared memory segments

[Cliff Wickman]

Thanks Andi.  I tested and applied your patch.
It's part of numactl-2.0.3-rc4.tar.gz
(ftp://oss.sgi.com/www/projects/libnuma/download/)



It's been a year(!) since the 2.0.2 release.

I think it's about time to release the current libnuma/numactl as 2.0.3.
Does anyone have other fixes or enhancements forthcoming?

-Cliff



On Fri, Jun 05, 2009 at 11:33:24AM +0200, Andi Kleen wrote:
> 
> I got a request to be able to dump the nodes individually of a shared memory
> segment. This can be very verbose for interleaving on large segments, but is still useful
> for some analysis.
> 
> This patch implements that with a new --dump-nodes option 
> 
> Against numactl 2.0.3-rc3
> 
> -Andi
> 
> ---
>  numactl.8 |    5 +++++
>  numactl.c |   13 ++++++++++---
>  shm.c     |   31 +++++++++++++++++++++++++++++++
>  shm.h     |    1 +
>  4 files changed, 47 insertions(+), 3 deletions(-)
> 
> Index: numactl-2.0.3-rc3/numactl.c
> ===================================================================
> --- numactl-2.0.3-rc3.orig/numactl.c
> +++ numactl-2.0.3-rc3/numactl.c
> @@ -49,6 +49,7 @@ struct option opts[] = {
>  	{"strict", 0, 0, 't'},
>  	{"shmmode", 1, 0, 'M'},
>  	{"dump", 0, 0, 'd'},
> +	{"dump-nodes", 0, 0, 'D'},
>  	{"shmid", 1, 0, 'I'},
>  	{"huge", 0, 0, 'u'},
>  	{"touch", 0, 0, 'T'},
> @@ -67,8 +68,8 @@ void usage(void)
>  		"       numactl [--length length] [--offset offset] [--shmmode shmmode]\n"
>  		"               [--strict]\n"
>  		"               [--shmid id] --shm shmkeyfile | --file tmpfsfile\n"
> -		"               [--huge] [--touch]\n" 
> -		"               memory policy\n"
> +		"               [--huge] [--touch] \n"
> +		"               memory policy | --dump | --dump-nodes\n"
>  		"\n"
>  		"memory policy is --interleave, --preferred, --membind, --localalloc\n"
>  		"nodes is a comma delimited list of node numbers or A-B ranges or all.\n"
> @@ -466,6 +467,12 @@ int main(int ac, char **av)
>  			dump_shm();
>  			do_dump = 1;
>  			break;
> +		case 'D': /* --dump */
> +			if (shmfd < 0)
> +				complain("Cannot do --dump-nodes without shared memory.\n");
> +			dump_shm_nodes();
> +			do_dump = 1;
> +			break;
>  		case 't': /* --strict */
>  			did_strict = 1;
>  			numa_set_strict(1);
> @@ -520,7 +527,7 @@ int main(int ac, char **av)
>  		fprintf(stderr,"numactl: warning. Strict flag for process ignored.\n");
>  
>  	if (do_dump)
> -		usage_msg("cannot do --dump for process");
> +		usage_msg("cannot do --dump|--dump-shm for process");
>  
>  	if (shmoption)
>  		usage_msg("shm related option %s for process", shmoption);
> Index: numactl-2.0.3-rc3/shm.c
> ===================================================================
> --- numactl-2.0.3-rc3.orig/shm.c
> +++ numactl-2.0.3-rc3/shm.c
> @@ -202,6 +202,37 @@ void dump_shm(void)
>  	dumppol(start, c, prevpol, prevnodes);
>  } 
>  
> +static void dumpnode(unsigned long long start, unsigned long long end, int node)
> +{
> +	printf("%016Lx-%016Lx: %d\n", shmoffset+start, shmoffset+end, node);
> +}
> +
> +/* Dump nodes in a shared memory segment. */
> +void dump_shm_nodes(void)
> +{
> +	int prevnode = -1, node;
> +	unsigned long long c, start;
> +
> +	start = 0;
> +	if (shmlen == 0) {
> +		printf("nothing to dump\n");
> +		return;
> +	}
> +
> +	for (c = 0; c < shmlen; c += shm_pagesize) {
> +		if (get_mempolicy(&node, NULL, 0, c+shmptr,
> +						MPOL_F_ADDR|MPOL_F_NODE) < 0)
> +			err("get_mempolicy on shm");
> +		if (node == prevnode)
> +			continue;
> +		if (prevnode != -1)
> +			dumpnode(start, c, prevnode);
> +		prevnode = node;
> +		start = c;
> +	}
> +	dumpnode(start, c, prevnode);
> +}
> +
>  static void vwarn(char *ptr, char *fmt, ...) 
>  { 
>  	va_list ap;
> Index: numactl-2.0.3-rc3/shm.h
> ===================================================================
> --- numactl-2.0.3-rc3.orig/shm.h
> +++ numactl-2.0.3-rc3/shm.h
> @@ -8,6 +8,7 @@ extern unsigned long long shmoffset;
>  extern int shmflags;
>  
>  extern void dump_shm(void);
> +extern void dump_shm_nodes(void);
>  extern void attach_shared(char *);
>  extern void attach_sysvshm(char *);
>  extern void verify_shm(int policy, struct bitmask *);
> Index: numactl-2.0.3-rc3/numactl.8
> ===================================================================
> --- numactl-2.0.3-rc3.orig/numactl.8
> +++ numactl-2.0.3-rc3/numactl.8
> @@ -64,6 +64,8 @@ numactl \- Control NUMA policy for proce
>  .B \-\-touch
>  ] [
>  .B \-\-dump
> +] [
> +.B \-\-dump-nodes
>  ]
>  memory policy
>  .SH DESCRIPTION
> @@ -225,6 +227,9 @@ is applied when an applications maps and
>  .B \-\-dump
>  Dump policy in the specified range.
>  .TP
> +.B \-\-dump-nodes
> +Dump all nodes of the specific range (very verbose!)
> +.TP
>  Valid node specifiers
>  .TS
>  tab(:);
> 
> -- 
> ak@linux.intel.com -- Speaking for myself only.

-- 
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824

Re: [PATCH] numactl: Support dumping nodes of pages in shared memory segments

[Andi Kleen]

On Fri, Jun 05, 2009 at 08:54:25AM -0500, Cliff Wickman wrote:
> 
> Thanks Andi.  I tested and applied your patch.
> It's part of numactl-2.0.3-rc4.tar.gz
> (ftp://oss.sgi.com/www/projects/libnuma/download/)
> 
> 
> 
> It's been a year(!) since the 2.0.2 release.
> 
> I think it's about time to release the current libnuma/numactl as 2.0.3.
> Does anyone have other fixes or enhancements forthcoming?

I got a request to display large pages in --hardware. Haven't had time
to code that up yet. I'll be also gone for the next two weeks.

You probably don't want to delay a release for that long, i can submit it after
I'm back.

-Andi

-- 
ak@linux.intel.com -- Speaking for myself only.