[RFC] prefer allocations from nodes w/o CPUs

[RFC] prefer allocations from nodes w/o CPUs

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

Small patch for the ia64 version of topology.h to make the zonelist building 
code prefer nodes w/o CPUs for allocations.  Do any platforms aside from 
Altix have nodes with only memory?  If not, then this will only affect sn2, 
otherwise I'm interested in hearing feedback.

Thanks,
Jesse

[-- Attachment #2: prefer-headless-nodes.patch --]
[-- Type: text/plain, Size: 437 bytes --]

===== include/asm-ia64/topology.h 1.13 vs edited =====
--- 1.13/include/asm-ia64/topology.h	2004-10-20 11:20:44 -07:00
+++ edited/include/asm-ia64/topology.h	2004-11-02 12:02:48 -08:00
@@ -18,6 +18,10 @@
 #include <asm/smp.h>
 
 #ifdef CONFIG_NUMA
+
+/* Nodes w/o CPUs are preferred for memory allocations, see build_zonelists */
+#define PENALTY_FOR_NODE_WITH_CPUS 255
+
 /*
  * Returns the number of the node containing CPU 'cpu'
  */

Re: [RFC] prefer allocations from nodes w/o CPUs

[Alex Williamson]

On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> Small patch for the ia64 version of topology.h to make the zonelist building 
> code prefer nodes w/o CPUs for allocations.  Do any platforms aside from 
> Altix have nodes with only memory?  If not, then this will only affect sn2, 
> otherwise I'm interested in hearing feedback.

   Yes, hp sx1000 systems represent interleaved memory in a separate
node w/o cpus.

	Alex

-- 
Alex Williamson                             HP Linux & Open Source Lab

Re: [RFC] prefer allocations from nodes w/o CPUs

[Jesse Barnes]

On Thursday, November 4, 2004 9:11 am, Alex Williamson wrote:
> On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> > Small patch for the ia64 version of topology.h to make the zonelist
> > building code prefer nodes w/o CPUs for allocations.  Do any platforms
> > aside from Altix have nodes with only memory?  If not, then this will
> > only affect sn2, otherwise I'm interested in hearing feedback.
>
>    Yes, hp sx1000 systems represent interleaved memory in a separate
> node w/o cpus.

So does this patch make sense for those platforms too?  Or do we need 
something fancier?  It'll make nodes w/o CPUs come before nodes with CPUs in 
the zonelist (after the local node of course), even if the nodes with CPUs 
are closer.

Jesse

Re: [RFC] prefer allocations from nodes w/o CPUs

[Alex Williamson]

On Thu, 2004-11-04 at 09:16 -0800, Jesse Barnes wrote:
> On Thursday, November 4, 2004 9:11 am, Alex Williamson wrote:
> > On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> > > Small patch for the ia64 version of topology.h to make the zonelist
> > > building code prefer nodes w/o CPUs for allocations.  Do any platforms
> > > aside from Altix have nodes with only memory?  If not, then this will
> > > only affect sn2, otherwise I'm interested in hearing feedback.
> >
> >    Yes, hp sx1000 systems represent interleaved memory in a separate
> > node w/o cpus.
> 
> So does this patch make sense for those platforms too?  Or do we need 
> something fancier?  It'll make nodes w/o CPUs come before nodes with CPUs in 
> the zonelist (after the local node of course), even if the nodes with CPUs 
> are closer.

   That makes sense for us.  The interleaved memory should be somewhat
faster than truly remote memory.  Depending on the box configuration,
the interleaved memory node could be quite small and we'll run out
quickly, but better to make what use we can from it.  Thanks,

	Alex

-- 
Alex Williamson                             HP Linux & Open Source Lab

Re: [RFC] prefer allocations from nodes w/o CPUs

[Alex Tsariounov]

On Thu, Nov 04, 2004 at 10:11:19AM -0700, Alex Williamson wrote:
> On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> > Small patch for the ia64 version of topology.h to make the zonelist building 
> > code prefer nodes w/o CPUs for allocations.  Do any platforms aside from 
> > Altix have nodes with only memory?  If not, then this will only affect sn2, 
> > otherwise I'm interested in hearing feedback.
> 
>    Yes, hp sx1000 systems represent interleaved memory in a separate
> node w/o cpus.

So what's the ramifications of this patch?  Does this mean that
interleaved memory will now be preferably allocated over cell
local memory on sx1000 systems?

Thanks,
Alex Tsariounov

Re: [RFC] prefer allocations from nodes w/o CPUs

[Jesse Barnes]

On Thursday, November 4, 2004 9:32 am, Alex Tsariounov wrote:
> On Thu, Nov 04, 2004 at 10:11:19AM -0700, Alex Williamson wrote:
> > On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> > > Small patch for the ia64 version of topology.h to make the zonelist
> > > building code prefer nodes w/o CPUs for allocations.  Do any platforms
> > > aside from Altix have nodes with only memory?  If not, then this will
> > > only affect sn2, otherwise I'm interested in hearing feedback.
> >
> >    Yes, hp sx1000 systems represent interleaved memory in a separate
> > node w/o cpus.
>
> So what's the ramifications of this patch?  Does this mean that
> interleaved memory will now be preferably allocated over cell
> local memory on sx1000 systems?

Memory local to the node will come first, followed by the closest memory that 
doesn't have associated CPUs, followed by memory belonging to other CPUs.

Jesse

Re: [RFC] prefer allocations from nodes w/o CPUs

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

On Thursday, November 4, 2004 9:33 am, Jesse Barnes wrote:
> On Thursday, November 4, 2004 9:32 am, Alex Tsariounov wrote:
> > On Thu, Nov 04, 2004 at 10:11:19AM -0700, Alex Williamson wrote:
> > > On Thu, 2004-11-04 at 08:57 -0800, Jesse Barnes wrote:
> > > > Small patch for the ia64 version of topology.h to make the zonelist
> > > > building code prefer nodes w/o CPUs for allocations.  Do any
> > > > platforms aside from Altix have nodes with only memory?  If not, then
> > > > this will only affect sn2, otherwise I'm interested in hearing
> > > > feedback.
> > >
> > >    Yes, hp sx1000 systems represent interleaved memory in a separate
> > > node w/o cpus.
> >
> > So what's the ramifications of this patch?  Does this mean that
> > interleaved memory will now be preferably allocated over cell
> > local memory on sx1000 systems?
>
> Memory local to the node will come first, followed by the closest memory
> that doesn't have associated CPUs, followed by memory belonging to other
> CPUs.

Actually, I lied.  There's a bug in find_next_best_node that prevents the 
local node from coming first if the PENALTY_FOR_NODE_WITH_CPUS is set high 
enough.  Here's the fix (I'll send it to Andrew shortly).

Jesse

[-- Attachment #2: local-node-with-penalty-fix.patch --]
[-- Type: text/plain, Size: 446 bytes --]

===== mm/page_alloc.c 1.239 vs edited =====
--- 1.239/mm/page_alloc.c	2004-10-25 13:06:48 -07:00
+++ edited/mm/page_alloc.c	2004-11-04 09:41:23 -08:00
@@ -1213,6 +1213,12 @@
 		if (test_bit(n, used_node_mask))
 			continue;
 
+		/* Use the local node if we haven't already */
+		if (!test_bit(node, used_node_mask)) {
+			best_node = node;
+			break;
+		}
+
 		/* Use the distance array to find the distance */
 		val = node_distance(node, n);
 

Re: [RFC] prefer allocations from nodes w/o CPUs

[Alex Tsariounov]

On Thu, Nov 04, 2004 at 09:49:53AM -0800, Jesse Barnes wrote:
> On Thursday, November 4, 2004 9:33 am, Jesse Barnes wrote:
> > Memory local to the node will come first, followed by the closest memory
> > that doesn't have associated CPUs, followed by memory belonging to other
> > CPUs.
> 
> Actually, I lied.  There's a bug in find_next_best_node that prevents the 
> local node from coming first if the PENALTY_FOR_NODE_WITH_CPUS is set high 
> enough.  Here's the fix (I'll send it to Andrew shortly).

Great!  Thanks for the review.

Alex