* [PATCH] Enable hashdist by default on PowerPC
@ 2009-02-18 5:11 Anton Blanchard
2009-02-18 5:41 ` Benjamin Herrenschmidt
2009-02-18 9:19 ` David Miller
0 siblings, 2 replies; 9+ messages in thread
From: Anton Blanchard @ 2009-02-18 5:11 UTC (permalink / raw)
To: linuxppc-dev
On PowerPC we allocate large boot time hashes on node 0. This leads to
an imbalance in the free memory, for example on a 64GB box (4 x 16GB nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more
balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on
the smallest node, so even though the same amount of memory is used the
better balancing helps.
Signed-off-by: Anton Blanchard <anton@samba.org>
---
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index 95837bf..c0c63ee 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
#define HASH_EARLY 0x00000001 /* Allocating during early boot? */
/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+ * IA64, x86_64 and PowerPC have sufficient vmalloc space.
*/
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
+ defined(CONFIG_PPC64))
#define HASHDIST_DEFAULT 1
#else
#define HASHDIST_DEFAULT 0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on PowerPC
2009-02-18 5:11 [PATCH] Enable hashdist by default on PowerPC Anton Blanchard
@ 2009-02-18 5:41 ` Benjamin Herrenschmidt
2009-02-18 6:20 ` Anton Blanchard
2009-02-18 9:19 ` David Miller
1 sibling, 1 reply; 9+ messages in thread
From: Benjamin Herrenschmidt @ 2009-02-18 5:41 UTC (permalink / raw)
To: Anton Blanchard; +Cc: linuxppc-dev
> For many HPC applications we are limited by the free available memory on
> the smallest node, so even though the same amount of memory is used the
> better balancing helps.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
You have numbers ? :-) I'm asking mostly because I've been wondering
whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT
impact.
Cheers,
Ben.
> diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
> index 95837bf..c0c63ee 100644
> --- a/include/linux/bootmem.h
> +++ b/include/linux/bootmem.h
> @@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
> #define HASH_EARLY 0x00000001 /* Allocating during early boot? */
>
> /* Only NUMA needs hash distribution.
> - * IA64 and x86_64 have sufficient vmalloc space.
> + * IA64, x86_64 and PowerPC have sufficient vmalloc space.
> */
> -#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
> +#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
> + defined(CONFIG_PPC64))
> #define HASHDIST_DEFAULT 1
> #else
> #define HASHDIST_DEFAULT 0
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on PowerPC
2009-02-18 5:41 ` Benjamin Herrenschmidt
@ 2009-02-18 6:20 ` Anton Blanchard
0 siblings, 0 replies; 9+ messages in thread
From: Anton Blanchard @ 2009-02-18 6:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev
Hi Ben,
> You have numbers ? :-) I'm asking mostly because I've been wondering
> whether it offsets the 16M pages vs. 4K or 64K pages in term of TLB/ERAT
> impact.
The speedup is application dependent. Things like linpack usually
improve when you throw more memmory at them.
The potential slowdown will be in heavy dcache use (eg fileserving). We
originally added the large boot time hash code when we were benchmarking
SPECsfs (an NFS benchmark).
We can go back to the old behaviour with the hashdist=0 boot option, so
it's mostly a question of what the default should be.
Anton
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on PowerPC
2009-02-18 5:11 [PATCH] Enable hashdist by default on PowerPC Anton Blanchard
2009-02-18 5:41 ` Benjamin Herrenschmidt
@ 2009-02-18 9:19 ` David Miller
2009-02-20 5:19 ` Anton Blanchard
1 sibling, 1 reply; 9+ messages in thread
From: David Miller @ 2009-02-18 9:19 UTC (permalink / raw)
To: anton; +Cc: linuxppc-dev
From: Anton Blanchard <anton@samba.org>
Date: Wed, 18 Feb 2009 16:11:12 +1100
> @@ -145,9 +145,10 @@ extern void *alloc_large_system_hash(const char *tablename,
> #define HASH_EARLY 0x00000001 /* Allocating during early boot? */
>
> /* Only NUMA needs hash distribution.
> - * IA64 and x86_64 have sufficient vmalloc space.
> + * IA64, x86_64 and PowerPC have sufficient vmalloc space.
> */
> -#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
> +#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64) || \
> + defined(CONFIG_PPC64))
> #define HASHDIST_DEFAULT 1
> #else
> #define HASHDIST_DEFAULT 0
I should probably do this on sparc64 too.
Why don't we just change this thing to CONFIG_64BIT?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on PowerPC
2009-02-18 9:19 ` David Miller
@ 2009-02-20 5:19 ` Anton Blanchard
2009-02-20 8:03 ` David Miller
0 siblings, 1 reply; 9+ messages in thread
From: Anton Blanchard @ 2009-02-20 5:19 UTC (permalink / raw)
To: David Miller; +Cc: linuxppc-dev
Hi David,
> I should probably do this on sparc64 too.
>
> Why don't we just change this thing to CONFIG_64BIT?
I agree. How does this look?
Anton
--
On PowerPC we allocate large boot time hashes on node 0. This leads to
an imbalance in the free memory, for example on a 64GB box (4 x 16GB
nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more
balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on
the smallest node, so even though the same amount of memory is used the
better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient
vmalloc space, it makes sense to enable it via CONFIG_64BIT.
Signed-off-by: Anton Blanchard <anton@samba.org>
---
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index 95837bf..aa861d5 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
#define HASH_EARLY 0x00000001 /* Allocating during early boot? */
-/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+/* Only NUMA needs hash distribution. 64bit NUMA architectures have
+ * sufficient vmalloc space.
*/
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_64BIT)
#define HASHDIST_DEFAULT 1
#else
#define HASHDIST_DEFAULT 0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on PowerPC
2009-02-20 5:19 ` Anton Blanchard
@ 2009-02-20 8:03 ` David Miller
2009-02-26 11:24 ` [PATCH] Enable hashdist by default on 64bit NUMA Anton Blanchard
0 siblings, 1 reply; 9+ messages in thread
From: David Miller @ 2009-02-20 8:03 UTC (permalink / raw)
To: anton; +Cc: linuxppc-dev
From: Anton Blanchard <anton@samba.org>
Date: Fri, 20 Feb 2009 16:19:56 +1100
>
> Hi David,
>
> > I should probably do this on sparc64 too.
> >
> > Why don't we just change this thing to CONFIG_64BIT?
>
> I agree. How does this look?
Hmmm... my bad, I think you need to keep the CONFIG_NUMA
there too as there is a TLB usage penalty for non-NUMA
systems if you only use CONFIG_64BIT there.
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] Enable hashdist by default on 64bit NUMA
2009-02-20 8:03 ` David Miller
@ 2009-02-26 11:24 ` Anton Blanchard
2009-02-26 11:34 ` David Miller
2009-03-03 5:27 ` Benjamin Herrenschmidt
0 siblings, 2 replies; 9+ messages in thread
From: Anton Blanchard @ 2009-02-26 11:24 UTC (permalink / raw)
To: David Miller; +Cc: linuxppc-dev
Hi David,
> Hmmm... my bad, I think you need to keep the CONFIG_NUMA
> there too as there is a TLB usage penalty for non-NUMA
> systems if you only use CONFIG_64BIT there.
Sorry that was my screwup, here's a fixed version.
Anton
--
On PowerPC we allocate large boot time hashes on node 0. This leads to
an imbalance in the free memory, for example on a 64GB box (4 x 16GB
nodes):
Free memory:
Node 0: 97.03%
Node 1: 98.54%
Node 2: 98.42%
Node 3: 98.53%
If we switch to using vmalloc (like ia64 and x86-64) things are more
balanced:
Free memory:
Node 0: 97.53%
Node 1: 98.35%
Node 2: 98.33%
Node 3: 98.33%
For many HPC applications we are limited by the free available memory on
the smallest node, so even though the same amount of memory is used the
better balancing helps.
Since all 64bit NUMA capable architectures should have sufficient
vmalloc space, it makes sense to enable it via CONFIG_64BIT.
Signed-off-by: Anton Blanchard <anton@samba.org>
---
diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
index 95837bf..0c4d4b7 100644
--- a/include/linux/bootmem.h
+++ b/include/linux/bootmem.h
@@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
#define HASH_EARLY 0x00000001 /* Allocating during early boot? */
-/* Only NUMA needs hash distribution.
- * IA64 and x86_64 have sufficient vmalloc space.
+/* Only NUMA needs hash distribution. 64bit NUMA architectures have
+ * sufficient vmalloc space.
*/
-#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
+#if defined(CONFIG_NUMA) && defined(CONFIG_64BIT)
#define HASHDIST_DEFAULT 1
#else
#define HASHDIST_DEFAULT 0
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on 64bit NUMA
2009-02-26 11:24 ` [PATCH] Enable hashdist by default on 64bit NUMA Anton Blanchard
@ 2009-02-26 11:34 ` David Miller
2009-03-03 5:27 ` Benjamin Herrenschmidt
1 sibling, 0 replies; 9+ messages in thread
From: David Miller @ 2009-02-26 11:34 UTC (permalink / raw)
To: anton; +Cc: linuxppc-dev
From: Anton Blanchard <anton@samba.org>
Date: Thu, 26 Feb 2009 22:24:32 +1100
> On PowerPC we allocate large boot time hashes on node 0. This leads to
> an imbalance in the free memory, for example on a 64GB box (4 x 16GB
> nodes):
>
> Free memory:
> Node 0: 97.03%
> Node 1: 98.54%
> Node 2: 98.42%
> Node 3: 98.53%
>
> If we switch to using vmalloc (like ia64 and x86-64) things are more
> balanced:
>
> Free memory:
> Node 0: 97.53%
> Node 1: 98.35%
> Node 2: 98.33%
> Node 3: 98.33%
>
> For many HPC applications we are limited by the free available memory on
> the smallest node, so even though the same amount of memory is used the
> better balancing helps.
>
> Since all 64bit NUMA capable architectures should have sufficient
> vmalloc space, it makes sense to enable it via CONFIG_64BIT.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
Acked-by: David S. Miller <davem@davemloft.net>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] Enable hashdist by default on 64bit NUMA
2009-02-26 11:24 ` [PATCH] Enable hashdist by default on 64bit NUMA Anton Blanchard
2009-02-26 11:34 ` David Miller
@ 2009-03-03 5:27 ` Benjamin Herrenschmidt
1 sibling, 0 replies; 9+ messages in thread
From: Benjamin Herrenschmidt @ 2009-03-03 5:27 UTC (permalink / raw)
To: Anton Blanchard; +Cc: linuxppc-dev, David Miller
On Thu, 2009-02-26 at 22:24 +1100, Anton Blanchard wrote:
> Hi David,
>
> > Hmmm... my bad, I think you need to keep the CONFIG_NUMA
> > there too as there is a TLB usage penalty for non-NUMA
> > systems if you only use CONFIG_64BIT there.
>
> Sorry that was my screwup, here's a fixed version.
Sounds good, how do we proceed for merging that ? Andrew ? Should it hop
by linux-mm ?
Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
> Anton
>
> --
>
> On PowerPC we allocate large boot time hashes on node 0. This leads to
> an imbalance in the free memory, for example on a 64GB box (4 x 16GB
> nodes):
>
> Free memory:
> Node 0: 97.03%
> Node 1: 98.54%
> Node 2: 98.42%
> Node 3: 98.53%
>
> If we switch to using vmalloc (like ia64 and x86-64) things are more
> balanced:
>
> Free memory:
> Node 0: 97.53%
> Node 1: 98.35%
> Node 2: 98.33%
> Node 3: 98.33%
>
> For many HPC applications we are limited by the free available memory on
> the smallest node, so even though the same amount of memory is used the
> better balancing helps.
>
> Since all 64bit NUMA capable architectures should have sufficient
> vmalloc space, it makes sense to enable it via CONFIG_64BIT.
>
> Signed-off-by: Anton Blanchard <anton@samba.org>
> ---
>
> diff --git a/include/linux/bootmem.h b/include/linux/bootmem.h
> index 95837bf..0c4d4b7 100644
> --- a/include/linux/bootmem.h
> +++ b/include/linux/bootmem.h
> @@ -144,10 +144,10 @@ extern void *alloc_large_system_hash(const char *tablename,
>
> #define HASH_EARLY 0x00000001 /* Allocating during early boot? */
>
> -/* Only NUMA needs hash distribution.
> - * IA64 and x86_64 have sufficient vmalloc space.
> +/* Only NUMA needs hash distribution. 64bit NUMA architectures have
> + * sufficient vmalloc space.
> */
> -#if defined(CONFIG_NUMA) && (defined(CONFIG_IA64) || defined(CONFIG_X86_64))
> +#if defined(CONFIG_NUMA) && defined(CONFIG_64BIT)
> #define HASHDIST_DEFAULT 1
> #else
> #define HASHDIST_DEFAULT 0
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-03-03 5:27 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-02-18 5:11 [PATCH] Enable hashdist by default on PowerPC Anton Blanchard
2009-02-18 5:41 ` Benjamin Herrenschmidt
2009-02-18 6:20 ` Anton Blanchard
2009-02-18 9:19 ` David Miller
2009-02-20 5:19 ` Anton Blanchard
2009-02-20 8:03 ` David Miller
2009-02-26 11:24 ` [PATCH] Enable hashdist by default on 64bit NUMA Anton Blanchard
2009-02-26 11:34 ` David Miller
2009-03-03 5:27 ` Benjamin Herrenschmidt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).