topology api confusion

topology api confusion

[Nathan Lynch]

We need some clarity on how asm-generic/topology.h is intended to be
used.  I suspect that it's supposed to be unconditionally included at
the end of the architecture's topology.h so that any elements which
are undefined by the arch have sensible default definitions.  Looking
at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
however i386 does not (i386 pulls in the generic version only when
!CONFIG_NUMA).

The #ifndef guards around each element of the topology api
cannot serve their apparent intended purpose when the architecture
implements a given bit as a function instead of a macro
(e.g. cpu_to_node in ppc64):

----

asm-generic/topology.h:

#ifndef cpu_to_node
#define cpu_to_node(cpu)        (0)
#endif

----

asm-ppc64/topology.h:

static inline int cpu_to_node(int cpu)
{
        int node;

        node = numa_cpu_lookup_table[cpu];
	....

----

Since ppc64 unconditionally includes asm-generic/topology.h, all uses
of cpu_to_node are preprocessed to (0).  Similar damage occurs with
every other topology function which happens to be a real function
instead of a macro.  I'm surprised my ppc64 numa systems even boot ;)

If the intent is that the architecture is free to define only a subset
of the api and include the generic header for fallback definitions,
then we need to do the #ifndef __HAVE_ARCH_FOO thing, no?  That is,
the code above would look like:

----

asm-generic/topology.h:

#ifndef __HAVE_ARCH_CPU_TO_NODE
#define cpu_to_node(cpu)        (0)
#endif

----

asm-ppc64/topology.h:

#define __HAVE_ARCH_CPU_TO_NODE
static inline int cpu_to_node(int cpu)
{
        int node;

        node = numa_cpu_lookup_table[cpu];
	....

----

Thought I'd ask for input first since this would involve a sweep of
include/asm-*.


Nathan

Re: topology api confusion

[Matthew Dobson]

Nathan Lynch wrote:
> We need some clarity on how asm-generic/topology.h is intended to be
> used.  I suspect that it's supposed to be unconditionally included at
> the end of the architecture's topology.h so that any elements which
> are undefined by the arch have sensible default definitions.  Looking
> at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
> however i386 does not (i386 pulls in the generic version only when
> !CONFIG_NUMA).
> 
> The #ifndef guards around each element of the topology api
> cannot serve their apparent intended purpose when the architecture
> implements a given bit as a function instead of a macro
> (e.g. cpu_to_node in ppc64):

When I originally wrote this all up, I had planned it to be used the way
i386 does: offer a full implementation of topology if appropriate, else
include the generic "sane" default.  It has since changed to work more like
you described: implement some (or all) of the topology functions, then
include the generic one to define any you missed.


> Since ppc64 unconditionally includes asm-generic/topology.h, all uses
> of cpu_to_node are preprocessed to (0).  Similar damage occurs with
> every other topology function which happens to be a real function
> instead of a macro.  I'm surprised my ppc64 numa systems even boot ;)
> 
> If the intent is that the architecture is free to define only a subset
> of the api and include the generic header for fallback definitions,
> then we need to do the #ifndef __HAVE_ARCH_FOO thing, no?  That is,
> the code above would look like:

You are correct in noticing that things should (though apparently don't?)
go wonky when arches define their topology functions as *functions*, rather
than macros.  It hasn't really seemed to bite anyone yet, so I've left it
alone, though to be honest, it is as surprising to me that it works as it
is to you.  I've resisted creating #ifdef ARCH_HAVE_XXX all over the place,
though maybe it is finally time?


> Thought I'd ask for input first since this would involve a sweep of
> include/asm-*.

It would indeed...  I'd be more than happy to look at any patches you care
to generate.  As I said, it seems to work, though I'm certain it's all held
together by GCC black magic & voodoo, and probably a little duct-tape.  A
more obviously correct solution would not be a bad thing. :)

-Matt

Re: topology api confusion

[Nathan Lynch]

Matthew Dobson wrote:
> Nathan Lynch wrote:
> > We need some clarity on how asm-generic/topology.h is intended to be
> > used.  I suspect that it's supposed to be unconditionally included at
> > the end of the architecture's topology.h so that any elements which
> > are undefined by the arch have sensible default definitions.  Looking
> > at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
> > however i386 does not (i386 pulls in the generic version only when
> > !CONFIG_NUMA).
> > 
> > The #ifndef guards around each element of the topology api
> > cannot serve their apparent intended purpose when the architecture
> > implements a given bit as a function instead of a macro
> > (e.g. cpu_to_node in ppc64):
> 
> When I originally wrote this all up, I had planned it to be used the way
> i386 does: offer a full implementation of topology if appropriate, else
> include the generic "sane" default.  It has since changed to work more like
> you described: implement some (or all) of the topology functions, then
> include the generic one to define any you missed.

OK.

> You are correct in noticing that things should (though apparently don't?)
> go wonky when arches define their topology functions as *functions*, rather
> than macros.  It hasn't really seemed to bite anyone yet, so I've left it
> alone, though to be honest, it is as surprising to me that it works as it
> is to you.  I've resisted creating #ifdef ARCH_HAVE_XXX all over the place,
> though maybe it is finally time?

Things _do_ go wonky, but likely only on ppc64 -- all cpus show up in
all nodes' cpumaps in sysfs.  The other architectures which provide
overrides and unconditionally include the generic topology.h define
only macros iirc.  If i386 were to include the generic topology.h it
would have similar issues since it uses functions for some things too.

> > Thought I'd ask for input first since this would involve a sweep of
> > include/asm-*.
> 
> It would indeed...  I'd be more than happy to look at any patches you care
> to generate.  As I said, it seems to work, though I'm certain it's all held
> together by GCC black magic & voodoo, and probably a little duct-tape.  A
> more obviously correct solution would not be a bad thing. :)

I've got the changes ready, just need to test them a little more.

Thanks.

Nathan

Re: topology api confusion

[Bill Davidsen]

Matthew Dobson wrote:
> Nathan Lynch wrote:
> 
>>We need some clarity on how asm-generic/topology.h is intended to be
>>used.  I suspect that it's supposed to be unconditionally included at
>>the end of the architecture's topology.h so that any elements which
>>are undefined by the arch have sensible default definitions.  Looking
>>at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
>>however i386 does not (i386 pulls in the generic version only when
>>!CONFIG_NUMA).
>>
>>The #ifndef guards around each element of the topology api
>>cannot serve their apparent intended purpose when the architecture
>>implements a given bit as a function instead of a macro
>>(e.g. cpu_to_node in ppc64):
> 
> 
> When I originally wrote this all up, I had planned it to be used the way
> i386 does: offer a full implementation of topology if appropriate, else
> include the generic "sane" default.  It has since changed to work more like
> you described: implement some (or all) of the topology functions, then
> include the generic one to define any you missed.
> 
> 
> 
>>Since ppc64 unconditionally includes asm-generic/topology.h, all uses
>>of cpu_to_node are preprocessed to (0).  Similar damage occurs with
>>every other topology function which happens to be a real function
>>instead of a macro.  I'm surprised my ppc64 numa systems even boot ;)
>>
>>If the intent is that the architecture is free to define only a subset
>>of the api and include the generic header for fallback definitions,
>>then we need to do the #ifndef __HAVE_ARCH_FOO thing, no?  That is,
>>the code above would look like:
> 
> 
> You are correct in noticing that things should (though apparently don't?)
> go wonky when arches define their topology functions as *functions*, rather
> than macros.  It hasn't really seemed to bite anyone yet, so I've left it
> alone, though to be honest, it is as surprising to me that it works as it
> is to you.  I've resisted creating #ifdef ARCH_HAVE_XXX all over the place,
> though maybe it is finally time?

If I understand the problem, is it amenable to just defining the macros 
and using another name for a function? In other words, if most arch 
define xx_generic_add as a function, can you just
  #define xx_generic_add xx_local_arch_add
which would satisfy the #ifndef, allow use of a function, etc? Then 
xx_local_arch_add can be the function. Then the common include would not 
generate macros for things which exist as function.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: topology api confusion

[Anton Blanchard]

Hi,

> We need some clarity on how asm-generic/topology.h is intended to be
> used.  I suspect that it's supposed to be unconditionally included at
> the end of the architecture's topology.h so that any elements which
> are undefined by the arch have sensible default definitions.  Looking
> at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
> however i386 does not (i386 pulls in the generic version only when
> !CONFIG_NUMA).
> 
> The #ifndef guards around each element of the topology api
> cannot serve their apparent intended purpose when the architecture
> implements a given bit as a function instead of a macro
> (e.g. cpu_to_node in ppc64):

Since it doesnt look like this will be resolved by 2.6.13 and NUMA is
currently completely broken on ppc64, how does this patch look?

--

Dont include asm-generic/topology.h unconditionally, we end up
overriding all the ppc64 specific functions when NUMA is on.

Signed-off-by: Anton Blanchard <anton@samba.org>

Index: linux-2.6.git-work/include/asm-ppc64/topology.h
===================================================================
--- linux-2.6.git-work.orig/include/asm-ppc64/topology.h	2005-07-30 23:49:56.000000000 +1000
+++ linux-2.6.git-work/include/asm-ppc64/topology.h	2005-08-01 14:43:49.000000000 +1000
@@ -33,6 +33,7 @@
 	return first_cpu(tmp);
 }
 
+#define pcibus_to_node(node)    (-1)
 #define pcibus_to_cpumask(bus)	(cpu_online_map)
 
 #define nr_cpus_node(node)	(nr_cpus_in_node[node])
@@ -59,8 +60,10 @@
 	.nr_balance_failed	= 0,			\
 }
 
-#endif /* CONFIG_NUMA */
+#else
 
 #include <asm-generic/topology.h>
 
+#endif /* CONFIG_NUMA */
+
 #endif /* _ASM_PPC64_TOPOLOGY_H */

Re: topology api confusion

[Nathan Lynch]

Anton Blanchard wrote:
> 
> Hi,
> 
> > We need some clarity on how asm-generic/topology.h is intended to be
> > used.  I suspect that it's supposed to be unconditionally included at
> > the end of the architecture's topology.h so that any elements which
> > are undefined by the arch have sensible default definitions.  Looking
> > at 2.6.13-rc3, this is what ppc64, ia64, and x86_64 currently do,
> > however i386 does not (i386 pulls in the generic version only when
> > !CONFIG_NUMA).
> > 
> > The #ifndef guards around each element of the topology api
> > cannot serve their apparent intended purpose when the architecture
> > implements a given bit as a function instead of a macro
> > (e.g. cpu_to_node in ppc64):
> 
> Since it doesnt look like this will be resolved by 2.6.13 and NUMA is
> currently completely broken on ppc64, how does this patch look?

Yes, this change is the least risk for now, thanks.

> 
> --
> 
> Dont include asm-generic/topology.h unconditionally, we end up
> overriding all the ppc64 specific functions when NUMA is on.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>
> 
> Index: linux-2.6.git-work/include/asm-ppc64/topology.h
> ===================================================================
> --- linux-2.6.git-work.orig/include/asm-ppc64/topology.h	2005-07-30 23:49:56.000000000 +1000
> +++ linux-2.6.git-work/include/asm-ppc64/topology.h	2005-08-01 14:43:49.000000000 +1000
> @@ -33,6 +33,7 @@
>  	return first_cpu(tmp);
>  }
>  
> +#define pcibus_to_node(node)    (-1)
>  #define pcibus_to_cpumask(bus)	(cpu_online_map)
>  
>  #define nr_cpus_node(node)	(nr_cpus_in_node[node])
> @@ -59,8 +60,10 @@
>  	.nr_balance_failed	= 0,			\
>  }
>  
> -#endif /* CONFIG_NUMA */
> +#else
>  
>  #include <asm-generic/topology.h>
>  
> +#endif /* CONFIG_NUMA */
> +
>  #endif /* _ASM_PPC64_TOPOLOGY_H */

Re: topology api confusion

[Paul Mackerras]

Anton Blanchard writes:

> Dont include asm-generic/topology.h unconditionally, we end up
> overriding all the ppc64 specific functions when NUMA is on.
> 
> Signed-off-by: Anton Blanchard <anton@samba.org>

Acked-by: Paul Mackerras <paulus@samba.org>

It looks like this should go into 2.6.13.