[Patch] fix ia64 build failure when CONFIG_SFC=m

[Patch] fix ia64 build failure when CONFIG_SFC=m

[Robin Holt]

CONFIG_SFC=m uses topology_core_siblings() which, for ia64, expects
cpu_core_map to be exported.  It is not.  This patch exports the needed
symbol.

Signed-off-by: Robin Holt <holt@sgi.com>

Index: contention_unroll/arch/ia64/kernel/smpboot.c
===================================================================
--- contention_unroll.orig/arch/ia64/kernel/smpboot.c	2008-07-28 18:19:56.000000000 -0500
+++ contention_unroll/arch/ia64/kernel/smpboot.c	2008-07-28 20:25:54.240198328 -0500
@@ -138,6 +138,7 @@ cpumask_t cpu_possible_map = CPU_MASK_NO
 EXPORT_SYMBOL(cpu_possible_map);
 
 cpumask_t cpu_core_map[NR_CPUS] __cacheline_aligned;
+EXPORT_SYMBOL(cpu_core_map);
 DEFINE_PER_CPU_SHARED_ALIGNED(cpumask_t, cpu_sibling_map);
 EXPORT_PER_CPU_SYMBOL(cpu_sibling_map);
 

RE: [Patch] fix ia64 build failure when CONFIG_SFC=m

[Luck, Tony]

> CONFIG_SFC=m uses topology_core_siblings() which, for ia64, expects
> cpu_core_map to be exported.  It is not.  This patch exports the needed
> symbol.

Ben,

Before I rush to apply this (or one of the other identical patches that
I've received) ... I'd like to ponder on whether this is the right thing
to do.

Looking at the code in drivers/net/sfc/efx.c I see that you are using
this to compute the number of RX queues based on the number of packages
with the git commit comment saying:

    Using multiple cores in the same package to handle received traffic
    does not appear to provide a performance benefit.  Therefore use CPU
    topology information to count CPU packages and use that as the default
    number of RX queues and interrupts.  We rely on interrupt balancing to
    spread the interrupts across packages.

I have some questions on this:

1) Did you measure on FSB based systems? ... It's kind of sad that the extra
cores are not helping and I wonder why. Do AMD and QPI based systems have
this same limitation?

2) Should Linux have an API to give you a useful number of threads, rather
than make you dig through the topology data structures? At some point cpumask_t
data structure is going to be too big for the kernel stack.

3) Does interrupt balancing always do the right thing to spread your interrupts
across packages?

4) In a hotplug system would you want to adjust the number of threads if cpus
were added or removed?

-Tony

Re: [Patch] fix ia64 build failure when CONFIG_SFC=m

[Ben Hutchings]

Luck, Tony wrote:
> > CONFIG_SFC=m uses topology_core_siblings() which, for ia64, expects
> > cpu_core_map to be exported.  It is not.  This patch exports the needed
> > symbol.
> 
> Ben,
> 
> Before I rush to apply this (or one of the other identical patches that
> I've received) ... I'd like to ponder on whether this is the right thing
> to do.
> 
> Looking at the code in drivers/net/sfc/efx.c I see that you are using
> this to compute the number of RX queues based on the number of packages
> with the git commit comment saying:
> 
>     Using multiple cores in the same package to handle received traffic
>     does not appear to provide a performance benefit.  Therefore use CPU
>     topology information to count CPU packages and use that as the default
>     number of RX queues and interrupts.  We rely on interrupt balancing to
>     spread the interrupts across packages.
> 
> I have some questions on this:
> 
> 1) Did you measure on FSB based systems? ... It's kind of sad that the extra
> cores are not helping and I wonder why. Do AMD and QPI based systems have
> this same limitation?

This heuristic has been in the out-of-tree driver for some time, mostly
based on experience with Intel x86 systems but apparently resulting in good
performance on both Intel and AMD multi-core systems tested here.  We do not
have any IA64 multi-core systems.  I think a single core in each package can
generally saturate the memory bus and this is why spreading the load wider
is not useful.

> 2) Should Linux have an API to give you a useful number of threads, rather
> than make you dig through the topology data structures? At some point cpumask_t
> data structure is going to be too big for the kernel stack.

Since that's all we want to know, it would certainly simplify this driver
and might be useful to others that implement RSS.

> 3) Does interrupt balancing always do the right thing to spread your interrupts
> across packages?

No, if we want to be sure then we do interrupt pinning afterwards.

> 4) In a hotplug system would you want to adjust the number of threads if cpus
> were added or removed?

That would make sense, but I don't think it can be done without disrupting
network traffic.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [Patch] fix ia64 build failure when CONFIG_SFC=m

[David Miller]

From: Ben Hutchings <bhutchings@solarflare.com>
Date: Tue, 29 Jul 2008 18:48:41 +0100

> I think a single core in each package can generally saturate the
> memory bus and this is why spreading the load wider is not useful.

I disagree, especially for routing and firewall setups.

You want as many cpu threads as possible, even on the same core,
doing the routing and firewall lookups through the various
datastructures, in parallel.

Re: [Patch] fix ia64 build failure when CONFIG_SFC=m

[Ben Hutchings]

David Miller wrote:
> From: Ben Hutchings <bhutchings@solarflare.com>
> Date: Tue, 29 Jul 2008 18:48:41 +0100
> 
> > I think a single core in each package can generally saturate the
> > memory bus and this is why spreading the load wider is not useful.
> 
> I disagree, especially for routing and firewall setups.
> 
> You want as many cpu threads as possible, even on the same core,
> doing the routing and firewall lookups through the various
> datastructures, in parallel.

So far as I can see, hardly anyone is doing routing at 10G speeds on Linux.
If they were, the horrible interaction with LRO would presumably have been
found and fixed earlier.  Besides which, internal benchmarking showed that we
could route 1500-byte packets bidirectionally at very near line rate using one
core on each of two multi-core packages.  (I can give more details if you
want.)  The default can in any case be overridden using the rss_cpus module
parameter.  If you want to add a core network setting for that, I think we'd
be happy to use it.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: [Patch] fix ia64 build failure when CONFIG_SFC=m

[David Miller]

From: Ben Hutchings <bhutchings@solarflare.com>
Date: Wed, 30 Jul 2008 14:35:29 +0100

> So far as I can see, hardly anyone is doing routing at 10G speeds on Linux.
> If they were, the horrible interaction with LRO would presumably have been
> found and fixed earlier.

This assumes a precise intersection between those doing 10G routing
and their using chips with LRO capable drivers.

All of the 10G equipment I have here and do testing on use drivers
that lack LRO support.