Re: Fw: 2.6.15-rc2-mm1 problems (get_cpu / NUMA optimizations)

Re: Fw: 2.6.15-rc2-mm1 problems (get_cpu / NUMA optimizations)

[Harald Welte]

[-- Attachment #1: Type: text/plain, Size: 1406 bytes --]

On Wed, Nov 23, 2005 at 03:56:03PM -0800, David S. Miller wrote:
> 
> I've seen this reported multiple times.
> 
> It seems that module load is attempted by IP-tables while
> some locks are held, or similar.

yes, this was introduced by the recent NUMA patches that introduced
get_cpu() / put_cpu() all over the place.  

Seems like I only now realize the implications that this has all over
the code, and to be honest, I'm not confident that there is a short-term
fix since a lot of code needs to be audited.  

Also, apart from just the iptables core,  we actually call the
checkentry() and destroy() functions of the match/target extensions
under get_cpu() now.  Plugin authors always assumed that they'd be able
to do stuff like kmalloc(GFP_KERNEL) since they are guaranteed to always
only be called from userspace context under the ipt_mutex.

I'm further analyzing our options, but we might have to pull out that
NUMA change, especially since I'm heading for FOSS.in/linux-bangalore in
a couple of days.

-- 
- Harald Welte <laforge@netfilter.org>                 http://netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Fw: 2.6.15-rc2-mm1 problems (get_cpu / NUMA optimizations)

[Eric Dumazet]

Harald Welte a écrit :
> On Wed, Nov 23, 2005 at 03:56:03PM -0800, David S. Miller wrote:
> 
>>I've seen this reported multiple times.
>>
>>It seems that module load is attempted by IP-tables while
>>some locks are held, or similar.
> 
> 
> yes, this was introduced by the recent NUMA patches that introduced
> get_cpu() / put_cpu() all over the place.  
> 
>

Hum... I was not aware that disabling preemption (in get_cpu()) had this 
effect (ie not allowing to sleep). I was using get_cpu() to bind the thread to 
the current cpu, maybe there is something better ?

If not, we can zap get_cpu()/pu_cpu() and use raw_smp_processor_id() as a hint 
only, (and dont care of preemption/cpu migration), as the value is only used 
as a hint to speedup operations in do_replace(), do_add_counters(), 
ipt_unregister_table(), translate_table()...

Eric

Re: Fw: 2.6.15-rc2-mm1 problems (get_cpu / NUMA optimizations)

[Patrick McHardy]

Eric Dumazet wrote:
> Hum... I was not aware that disabling preemption (in get_cpu()) had this 
> effect (ie not allowing to sleep). I was using get_cpu() to bind the 
> thread to the current cpu, maybe there is something better ?

set_cpus_allowed() might work.

> If not, we can zap get_cpu()/pu_cpu() and use raw_smp_processor_id() as 
> a hint only, (and dont care of preemption/cpu migration), as the value 
> is only used as a hint to speedup operations in do_replace(), 
> do_add_counters(), ipt_unregister_table(), translate_table()...

That sounds like a good way. iptables ruleset replacement is and
will always be slow anyway.