Re: [PATCH 0/3] smp: reduce stack requirements for smp_call_function_mask

Ingo Molnar
2008-09-06T18:22:20Z

* Mike Travis <travis@sgi.com> wrote:

> Ingo Molnar wrote:
> > * Mike Travis <travis@sgi.com> wrote:
> >
> >> * Cleanup cpumask_t usages in smp_call_function_mask function chain
> >> to prevent stack overflow problem when NR_CPUS=4096.
> >>
> >> * Reduce the number of passed cpumask_t variables in the following
> >> call chain for x86_64:
> >>
> >> smp_call_function_mask -->
> >> arch_send_call_function_ipi->
> >> smp_ops.send_call_func_ipi -->
> >> genapic->send_IPI_mask
> >>
> >> Since the smp_call_function_mask() is an EXPORTED function, we
> >> cannot change it's calling interface for a patch to 2.6.27.
> >>
> >> The smp_ops.send_call_func_ipi interface is internal only and
> >> has two arch provided functions:
> >>
> >> arch/x86/kernel/smp.c: .send_call_func_ipi = native_send_call_func_ipi
> >> arch/x86/xen/smp.c: .send_call_func_ipi = xen_smp_send_call_function_ipi
> >> arch/x86/mach-voyager/voyager_smp.c: (uses native_send_call_func_ipi)
> >>
> >> Therefore modifying the internal interface to use a cpumask_t pointer
> >> is straight-forward.
> >>
> >> The changes to genapic are much more extensive and are affected by the
> >> recent additions of the x2apic modes, so they will be done for 2.6.28 only.
> >>
> >> Based on 2.6.27-rc5-git6.
> >>
> >> Applies to linux-2.6.tip/master (with FUZZ).
> >
> > applied to tip/cpus4096, thanks Mike.
>
> Thanks Ingo! Could you send me the git id for the merge?

the commits are:

363a5e3: x86: add MAXSMP
01f569c: x86: restore 4096 limit for NR_CPUS
ae74da3: x86: reduce stack requirements for send_call_func_ipi
562d8c2: smp: reduce stack requirements for smp_call_function_mask

the merge into tip/master is:

| commit 7f5d26f9425851e20ca9774acbd13d0e3b96d9dd
| Merge: da5e209... 363a5e3...
| Author: Ingo Molnar <mingo@elte.hu>
| Date: Sat Sep 6 15:29:18 2008 +0200
|
| Merge branch 'cpus4096'

That merge commit will go away on the next integration run though.

your changes seem to be largely problem-free so far - with two dozen
MAXSMP=y random bootups already.

> > I'm still wondering whether we should get rid of non-reference based
> > cpumask_t altogether ...
>
> I've got a whole slew of "get-ready-to-remove-cpumask_t's" coming
> soon. There are two phases, one completely within the x86 arch and the
> 2nd hits the generic smp_call_function_mask ABI (won't be doable as a
> back-ported patch to 2.6.27.)

ok. None of this can go into v2.6.27 obviously - the stack corruptions
were rather nasty. But it's looking good for v2.6.28 - especially if you
are removing cpumask_t.

> > Did you have a chance to look at the ftrace/stacktrace tracer in
> > latest tip/master, which will show the maximum stack footprint that
> > can occur?
>
> Hmm, no. I'm using a default config right now as I can boot that
> pretty easily. I'll turn on the ftrace thing and check it out.

it's CONFIG_STACK_TRACER=y and rather nifty.

> > Also, i've applied the patch below as well to restore MAXSMP in a
> > muted form - with big warning signs added as well.
>
> The main thing is to allow the distros to set it manually for their QA
> testing of 2.6.27. I'm sure I'll get back bugs because of just that.
>
> (Is there a way to have them know to assign bugzilla's to me if
> NR_CPUS=4k is the root of the problem? This is an extremely serious
> issue for SGI and I'd like to avoid any delays in me finding out about
> problems.)

i dont think there's any easy mapping.

Ingo