* Re: CMCI vector not initialized on BP
2013-03-05 23:05 CMCI vector not initialized on BP Hartnett, Fred
@ 2013-03-11 22:29 ` Tony Luck
2013-03-11 22:42 ` Hartnett, Fred
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Tony Luck @ 2013-03-11 22:29 UTC (permalink / raw)
To: linux-ia64
On Tue, Mar 5, 2013 at 3:05 PM, Hartnett, Fred <fred.hartnett@hp.com> wrote:
> but the BP vector is not initialized since the BP is already online
...
> Is this by design or an oops?
Looks like we handle BSP in ia64_mca_late_init() [see below]. Are
you seeing something that indicates that this isn't doing what the
comment says it is doing?
-Tony
static int __init
ia64_mca_late_init(void)
{
if (!mca_init)
return 0;
/*
* Configure the CMCI/P vector and handler. Interrupts for CMC are
* per-processor, so AP CMC interrupts are setup in
smp_callin() (smpboot.c).
*/
register_percpu_irq(IA64_CMC_VECTOR, &cmci_irqaction);
register_percpu_irq(IA64_CMCP_VECTOR, &cmcp_irqaction);
ia64_mca_cmc_vector_setup(); /* Setup vector on BSP */
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: CMCI vector not initialized on BP
2013-03-05 23:05 CMCI vector not initialized on BP Hartnett, Fred
2013-03-11 22:29 ` Tony Luck
@ 2013-03-11 22:42 ` Hartnett, Fred
2013-03-15 20:21 ` Luck, Tony
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Hartnett, Fred @ 2013-03-11 22:42 UTC (permalink / raw)
To: linux-ia64
Hi Tony - thanks for the response... appreciate this. If you follow register_percpu_irq you will discover it does not operate on CPU0 and the line ' ia64_mca_cmc_vector_setup()' operates on the current cpu which is typically not CPU0. It took some hacks in my kernel to prove this and ultimately show CPU CMCV is always 0x0.
smp_callin looks like the routine to iniatialize the vector table and the CMCI vector as well when the cpu goes online. However smp_callin is not called for the BP.
Thanks - Fred
-----Original Message-----
From: Tony Luck [mailto:tony.luck@gmail.com]
Sent: Monday, March 11, 2013 5:30 PM
To: Hartnett, Fred
Cc: linux-ia64@vger.kernel.org
Subject: Re: CMCI vector not initialized on BP
On Tue, Mar 5, 2013 at 3:05 PM, Hartnett, Fred <fred.hartnett@hp.com> wrote:
> but the BP vector is not initialized since the BP is already online
...
> Is this by design or an oops?
Looks like we handle BSP in ia64_mca_late_init() [see below]. Are you seeing something that indicates that this isn't doing what the comment says it is doing?
-Tony
static int __init
ia64_mca_late_init(void)
{
if (!mca_init)
return 0;
/*
* Configure the CMCI/P vector and handler. Interrupts for CMC are
* per-processor, so AP CMC interrupts are setup in
smp_callin() (smpboot.c).
*/
register_percpu_irq(IA64_CMC_VECTOR, &cmci_irqaction);
register_percpu_irq(IA64_CMCP_VECTOR, &cmcp_irqaction);
ia64_mca_cmc_vector_setup(); /* Setup vector on BSP */
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: CMCI vector not initialized on BP
2013-03-05 23:05 CMCI vector not initialized on BP Hartnett, Fred
2013-03-11 22:29 ` Tony Luck
2013-03-11 22:42 ` Hartnett, Fred
@ 2013-03-15 20:21 ` Luck, Tony
2013-03-19 23:13 ` Tony Luck
2013-03-20 12:49 ` Hartnett, Fred
4 siblings, 0 replies; 6+ messages in thread
From: Luck, Tony @ 2013-03-15 20:21 UTC (permalink / raw)
To: linux-ia64
How about if we make sure that the relevent parts of
ia64_mca_late_init() are run on CPU0. See patch below.
Does that fix it ... or do we need to do more surgery
to make the relevent pieces of smp_callin() callable
from CPU0.
-Tony
---
diff --git a/arch/ia64/kernel/mca.c b/arch/ia64/kernel/mca.c
index 65bf9cd..bfb39de 100644
--- a/arch/ia64/kernel/mca.c
+++ b/arch/ia64/kernel/mca.c
@@ -2074,22 +2074,8 @@ ia64_mca_init(void)
printk(KERN_INFO "MCA related initialization done\n");
}
-/*
- * ia64_mca_late_init
- *
- * Opportunity to setup things that require initialization later
- * than ia64_mca_init. Setup a timer to poll for CPEs if the
- * platform doesn't support an interrupt driven mechanism.
- *
- * Inputs : None
- * Outputs : Status
- */
-static int __init
-ia64_mca_late_init(void)
+static void __init ia64_mca_late_init_bsp(void *dummy)
{
- if (!mca_init)
- return 0;
-
/*
* Configure the CMCI/P vector and handler. Interrupts for CMC are
* per-processor, so AP CMC interrupts are setup in smp_callin() (smpboot.c).
@@ -2108,6 +2094,29 @@ ia64_mca_late_init(void)
/* Setup the CPEI/P handler */
register_percpu_irq(IA64_CPEP_VECTOR, &mca_cpep_irqaction);
#endif
+}
+
+/*
+ * ia64_mca_late_init
+ *
+ * Opportunity to setup things that require initialization later
+ * than ia64_mca_init. Setup a timer to poll for CPEs if the
+ * platform doesn't support an interrupt driven mechanism.
+ *
+ * Inputs : None
+ * Outputs : Status
+ */
+static int __init
+ia64_mca_late_init(void)
+{
+ if (!mca_init)
+ return 0;
+
+ /*
+ * We are probably not running on cpu0, but we have some
+ * things that must be set up there
+ */
+ smp_call_function_single(0, ia64_mca_late_init_bsp, NULL, 1);
register_hotcpu_notifier(&mca_cpu_notifier);
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: CMCI vector not initialized on BP
2013-03-05 23:05 CMCI vector not initialized on BP Hartnett, Fred
` (2 preceding siblings ...)
2013-03-15 20:21 ` Luck, Tony
@ 2013-03-19 23:13 ` Tony Luck
2013-03-20 12:49 ` Hartnett, Fred
4 siblings, 0 replies; 6+ messages in thread
From: Tony Luck @ 2013-03-19 23:13 UTC (permalink / raw)
To: linux-ia64
On Fri, Mar 15, 2013 at 1:21 PM, Luck, Tony <tony.luck@intel.com> wrote:
> How about if we make sure that the relevent parts of
> ia64_mca_late_init() are run on CPU0.
To answer my own quesontion ... because it will splat all sorts of
ugly messages on the console. Like these:
WARNING: at kernel/mutex.c:199 __mutex_lock_slowpath+0x6d0/0x700()
Hardware name: I8QBH
Modules linked in:
Call Trace:
[<a000000100016020>] show_stack+0x80/0xa0
spà00000301b5f620 bspà00000301b51530
[<a000000100b36ad0>] dump_stack+0x30/0x50
spà00000301b5f7f0 bspà00000301b51518
[<a000000100080920>] warn_slowpath_common+0xc0/0x100
spà00000301b5f7f0 bspà00000301b514d8
[<a0000001000809a0>] warn_slowpath_null+0x40/0x60
spà00000301b5f7f0 bspà00000301b514b0
[<a000000100b38ed0>] __mutex_lock_slowpath+0x6d0/0x700
spà00000301b5f7f0 bspà00000301b51420
[<a000000100b38f30>] mutex_lock+0x30/0x60
spà00000301b5f810 bspà00000301b51400
[<a0000001001495d0>] irq_reserve_irqs+0x90/0x140
spà00000301b5f810 bspà00000301b513c0
[<a000000100152db0>] irq_set_chip+0xb0/0xe0
spà00000301b5f810 bspà00000301b51398
[<a0000001000134b0>] ia64_native_register_percpu_irq+0xf0/0x1a0
spà00000301b5f820 bspà00000301b51368
[<a000000100e0e650>] ia64_mca_late_init_bsp+0x20/0xf0
spà00000301b5fbf0 bspà00000301b51350
[<a00000010011cff0>] generic_smp_call_function_single_interrupt+0x230/0x440
spà00000301b5fbf0 bspà00000301b51318
[<a000000100067bc0>] handle_IPI+0x1a0/0x2a0
spà00000301b5fc00 bspà00000301b512c8
:-(
-Tony
^ permalink raw reply [flat|nested] 6+ messages in thread* RE: CMCI vector not initialized on BP
2013-03-05 23:05 CMCI vector not initialized on BP Hartnett, Fred
` (3 preceding siblings ...)
2013-03-19 23:13 ` Tony Luck
@ 2013-03-20 12:49 ` Hartnett, Fred
4 siblings, 0 replies; 6+ messages in thread
From: Hartnett, Fred @ 2013-03-20 12:49 UTC (permalink / raw)
To: linux-ia64
Hi Tony - I tried a similar experiment of...
smp_call_function_single(0, ia64_mca_cmc_vector_setup, NULL, 0);
in ia64_mca_late_init and the kernel hung with and without the wait flag (forth argument) set.
I was thinking about the problem from a different angle. We know the BP is always cpu0, and it does not go through smp_callin like the APs. But at some point during the boot, the kernel gets assigned to one seemingly random cpu (it changes each time I boot the kernel). Seems to me if we forced the kernel to always run on cpu0, then by the time it gets to ia64_mca_late_init, the proper vector initialization would occur via the...
ia64_mca_cmc_vector_setup(); /* Setup vector on BSP */
call. Question is how/where is the kernel thread assigned to a cpu number? I can see the advantage of floating the kernel thread to any available cpu in the event another cpu does not come online. So perhaps the default is force to cpu0 if online, otherwise any available cpu. The user has other problems to deal with if a cpu does not initialize. Thoughts about this approach?
Thanks - Fred
-----Original Message-----
From: Tony Luck [mailto:tony.luck@gmail.com]
Sent: Tuesday, March 19, 2013 6:13 PM
To: Hartnett, Fred
Cc: linux-ia64@vger.kernel.org
Subject: Re: CMCI vector not initialized on BP
On Fri, Mar 15, 2013 at 1:21 PM, Luck, Tony <tony.luck@intel.com> wrote:
> How about if we make sure that the relevent parts of
> ia64_mca_late_init() are run on CPU0.
To answer my own quesontion ... because it will splat all sorts of ugly messages on the console. Like these:
WARNING: at kernel/mutex.c:199 __mutex_lock_slowpath+0x6d0/0x700()
Hardware name: I8QBH
Modules linked in:
Call Trace:
[<a000000100016020>] show_stack+0x80/0xa0
spà00000301b5f620 bspà00000301b51530 [<a000000100b36ad0>] dump_stack+0x30/0x50
spà00000301b5f7f0 bspà00000301b51518 [<a000000100080920>] warn_slowpath_common+0xc0/0x100
spà00000301b5f7f0 bspà00000301b514d8 [<a0000001000809a0>] warn_slowpath_null+0x40/0x60
spà00000301b5f7f0 bspà00000301b514b0 [<a000000100b38ed0>] __mutex_lock_slowpath+0x6d0/0x700
spà00000301b5f7f0 bspà00000301b51420 [<a000000100b38f30>] mutex_lock+0x30/0x60
spà00000301b5f810 bspà00000301b51400 [<a0000001001495d0>] irq_reserve_irqs+0x90/0x140
spà00000301b5f810 bspà00000301b513c0 [<a000000100152db0>] irq_set_chip+0xb0/0xe0
spà00000301b5f810 bspà00000301b51398 [<a0000001000134b0>] ia64_native_register_percpu_irq+0xf0/0x1a0
spà00000301b5f820 bspà00000301b51368 [<a000000100e0e650>] ia64_mca_late_init_bsp+0x20/0xf0
spà00000301b5fbf0 bspà00000301b51350 [<a00000010011cff0>] generic_smp_call_function_single_interrupt+0x230/0x440
spà00000301b5fbf0 bspà00000301b51318 [<a000000100067bc0>] handle_IPI+0x1a0/0x2a0
spà00000301b5fc00 bspà00000301b512c8
:-(
-Tony
^ permalink raw reply [flat|nested] 6+ messages in thread