[PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[Zwane Mwaikambo]

Some timer interrupt vectors were queued on the Local APIC and were being
serviced when we enabled interrupts again in fixup_irqs(), so we need to
mask the APIC timer, enable interrupts so that any queued interrupts get
processed whilst the processor is still on the online map and then clear
ourselves from the online map. 1ms is a nice safe number even under heavy
interrupt load with higher priority vectors queued. Andrew this is
the patch i promised, Rusty, i'm not sure if you find
__attribute__((weak)) offensive...

Unable to handle kernel NULL pointer dereference at virtual address 00000008
 printing eip:
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in:
CPU:    1
EIP:    0060:[<c01200cd>]    Not tainted VLI
EFLAGS: 00010006   (2.6.8.1-mm2)
EIP is at find_busiest_group+0x1cd/0x350
eax: ddca7ef0   ebx: 00000000   ecx: 00000080   edx: 00000000
esi: 0000007d   edi: c05e3460   ebp: ddca7ec8   esp: ddca7e7c
ds: 007b   es: 007b   ss: 0068
Process kstopmachine (pid: 29711, threadinfo=ddca7000 task=de688250)
Stack: ddca7e9c c013b2f1 c05c7000 00000000 00000001 00000080 00000000 00008000
       00000100 c05e346c 00000000 c05e3460 ddca7ef0 00000001 c14119e0 00000001
       c14119e0 c1410f60 00000001 ddca7f00 c0120350 00000001 00800f00 00007410
Call Trace:
 [<c01082ea>] show_stack+0x7a/0x90
 [<c0108472>] show_registers+0x152/0x1c0
 [<c01086a0>] die+0x110/0x200
 [<c011d0f5>] do_page_fault+0x235/0x5d3
 [<c0107ed9>] error_code+0x2d/0x38
 [<c0120350>] load_balance+0x40/0x2c0
 [<c0120961>] rebalance_tick+0xa1/0xb0
 [<c0119722>] smp_apic_timer_interrupt+0xf2/0x100
 [<c0107e5e>] apic_timer_interrupt+0x1a/0x20
 [<c0241adc>] __delay+0xc/0x10
 [<c010afd3>] fixup_irqs+0xe3/0x130
 [<c0118f66>] __cpu_disable+0x16/0x30
 [<c0140bb4>] take_cpu_down+0x24/0x50
 [<c01472f3>] do_stop+0x63/0x70
 [<c013f715>] kthread+0xa5/0xb0
 [<c0105375>] kernel_thread_helper+0x5/0x10
Code: 02 0f af d6 39 d0 0f 86 05 01 00 00 8b 5d cc 89 ca 8b 45 d4 29 da 29 c8 3

Signed-off-by: Zwane Mwaikambo <zwane@fsmlabs.com>

Index: linux-2.6.8.1-mm2/kernel/cpu.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/kernel/cpu.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 cpu.c
--- linux-2.6.8.1-mm2/kernel/cpu.c	19 Aug 2004 20:52:08 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/kernel/cpu.c	21 Aug 2004 14:10:18 -0000
@@ -84,11 +84,18 @@ static int cpu_run_sbin_hotplug(unsigned
 	return call_usermodehelper(argv[0], argv, envp, 0);
 }

+void __attribute__((weak)) __cpu_prep_disable(void)
+{
+}
+
 /* Take this CPU down. */
 static int take_cpu_down(void *unused)
 {
 	int err;

+	/* Prepatory work before clearing the cpu off the online map */
+	__cpu_prep_disable();
+
 	/* Take offline: makes arch_cpu_down somewhat easier. */
 	cpu_clear(smp_processor_id(), cpu_online_map);

Index: linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c
--- linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c	19 Aug 2004 20:51:36 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c	21 Aug 2004 12:56:52 -0000
@@ -1159,6 +1159,18 @@ static int __devinit cpu_enable(unsigned
 	return 0;
 }

+void __cpu_prep_disable(void)
+{
+	/* We enable the timer again on the exit path of the death loop */
+	if (smp_processor_id() != 0) {
+		disable_APIC_timer();
+		/* Allow any queued timer interrupts to get serviced */
+		local_irq_enable();
+		mdelay(1);
+		local_irq_disable();
+	}
+}
+
 int __cpu_disable(void)
 {
 	/*
@@ -1173,9 +1185,6 @@ int __cpu_disable(void)
 		return -EBUSY;

 	fixup_irqs();
-
-	/* We enable the timer again on the exit path of the death loop */
-	disable_APIC_timer();
 	return 0;
 }

Re: [PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[Rusty Russell]

On Sun, 2004-08-22 at 00:10, Zwane Mwaikambo wrote:
> Some timer interrupt vectors were queued on the Local APIC and were being
> serviced when we enabled interrupts again in fixup_irqs(), so we need to
> mask the APIC timer, enable interrupts so that any queued interrupts get
> processed whilst the processor is still on the online map and then clear
> ourselves from the online map. 1ms is a nice safe number even under heavy
> interrupt load with higher priority vectors queued. Andrew this is
> the patch i promised, Rusty, i'm not sure if you find
> __attribute__((weak)) offensive...

It's horrible.  Please move the unsetting of the cpu_online bit into the
arch-specific __cpu_disable() code for each arch, which is consistent
and also simplifies things.

Thanks,
Rusty.
-- 
Anyone who quotes me in their signature is an idiot -- Rusty Russell

Re: [PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[Zwane Mwaikambo]

On Sun, 22 Aug 2004, Rusty Russell wrote:

> On Sun, 2004-08-22 at 00:10, Zwane Mwaikambo wrote:
> > Some timer interrupt vectors were queued on the Local APIC and were being
> > serviced when we enabled interrupts again in fixup_irqs(), so we need to
> > mask the APIC timer, enable interrupts so that any queued interrupts get
> > processed whilst the processor is still on the online map and then clear
> > ourselves from the online map. 1ms is a nice safe number even under heavy
> > interrupt load with higher priority vectors queued. Andrew this is
> > the patch i promised, Rusty, i'm not sure if you find
> > __attribute__((weak)) offensive...
>
> It's horrible.  Please move the unsetting of the cpu_online bit into the
> arch-specific __cpu_disable() code for each arch, which is consistent
> and also simplifies things.

Alright this should do it then;

Thanks

Signed-off-by: Zwane Mwaikambo <zwane@linuxpower.ca>

 arch/i386/kernel/smpboot.c |   10 ++++++++--
 arch/ia64/kernel/smpboot.c |    1 +
 arch/ppc64/kernel/smp.c    |    4 +++-
 arch/s390/kernel/smp.c     |    4 +++-
 kernel/cpu.c               |    3 ---
 5 files changed, 15 insertions(+), 7 deletions(-)

Index: linux-2.6.8.1-mm2/kernel/cpu.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/kernel/cpu.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 cpu.c
--- linux-2.6.8.1-mm2/kernel/cpu.c	19 Aug 2004 20:52:08 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/kernel/cpu.c	22 Aug 2004 14:28:17 -0000
@@ -89,9 +89,6 @@ static int take_cpu_down(void *unused)
 {
 	int err;

-	/* Take offline: makes arch_cpu_down somewhat easier. */
-	cpu_clear(smp_processor_id(), cpu_online_map);
-
 	/* Ensure this CPU doesn't handle any more interrupts. */
 	err = __cpu_disable();
 	if (err < 0)
Index: linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c
--- linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c	19 Aug 2004 20:51:36 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/arch/i386/kernel/smpboot.c	22 Aug 2004 14:30:04 -0000
@@ -1172,10 +1172,16 @@ int __cpu_disable(void)
 	if (smp_processor_id() == 0)
 		return -EBUSY;

-	fixup_irqs();
-
 	/* We enable the timer again on the exit path of the death loop */
 	disable_APIC_timer();
+	/* Allow any queued timer interrupts to get serviced */
+	local_irq_enable();
+	mdelay(1);
+	local_irq_disable();
+
+	/* It's now safe to remove this processor from the online map */
+	cpu_clear(smp_processor_id(), cpu_online_map);
+	fixup_irqs();
 	return 0;
 }

Index: linux-2.6.8.1-mm2/arch/ia64/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/arch/ia64/kernel/smpboot.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c
--- linux-2.6.8.1-mm2/arch/ia64/kernel/smpboot.c	19 Aug 2004 20:51:37 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/arch/ia64/kernel/smpboot.c	22 Aug 2004 14:32:22 -0000
@@ -591,6 +591,7 @@ int __cpu_disable(void)
 	if (cpu == 0)
 		return -EBUSY;

+	cpu_clear(cpu, cpu_online_map);
 	fixup_irqs();
 	local_flush_tlb_all();
 	printk ("Disabled cpu %u\n", smp_processor_id());
Index: linux-2.6.8.1-mm2/arch/ppc64/kernel/smp.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/arch/ppc64/kernel/smp.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smp.c
--- linux-2.6.8.1-mm2/arch/ppc64/kernel/smp.c	19 Aug 2004 20:51:50 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/arch/ppc64/kernel/smp.c	22 Aug 2004 14:39:31 -0000
@@ -254,11 +254,13 @@ int __cpu_disable(void)
 {
 	/* FIXME: go put this in a header somewhere */
 	extern void xics_migrate_irqs_away(void);
+	int cpu = smp_processor_id();

+	cpu_clear(cpu, cpu_online_map);
 	systemcfg->processorCount--;

 	/*fix boot_cpuid here*/
-	if (smp_processor_id() == boot_cpuid)
+	if (cpu == boot_cpuid)
 		boot_cpuid = any_online_cpu(cpu_online_map);

 	/* FIXME: abstract this to not be platform specific later on */
Index: linux-2.6.8.1-mm2/arch/s390/kernel/smp.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm2/arch/s390/kernel/smp.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smp.c
--- linux-2.6.8.1-mm2/arch/s390/kernel/smp.c	19 Aug 2004 20:51:50 -0000	1.1.1.1
+++ linux-2.6.8.1-mm2/arch/s390/kernel/smp.c	22 Aug 2004 14:39:13 -0000
@@ -682,9 +682,11 @@ __cpu_disable(void)
 {
 	unsigned long flags;
 	ec_creg_mask_parms cr_parms;
+	int cpu = smp_processor_id();

+	cpu_clear(cpu, cpu_online_map);
 	spin_lock_irqsave(&smp_reserve_lock, flags);
-	if (smp_cpu_reserved[smp_processor_id()] != 0) {
+	if (smp_cpu_reserved[cpu] != 0) {
 		spin_unlock_irqrestore(&smp_reserve_lock, flags);
 		return -EBUSY;
 	}

Re: [PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[Zwane Mwaikambo]

On Sun, 22 Aug 2004, Zwane Mwaikambo wrote:

> On Sun, 22 Aug 2004, Rusty Russell wrote:
>
> > On Sun, 2004-08-22 at 00:10, Zwane Mwaikambo wrote:
> > > Some timer interrupt vectors were queued on the Local APIC and were being
> > > serviced when we enabled interrupts again in fixup_irqs(), so we need to
> > > mask the APIC timer, enable interrupts so that any queued interrupts get
> > > processed whilst the processor is still on the online map and then clear
> > > ourselves from the online map. 1ms is a nice safe number even under heavy
> > > interrupt load with higher priority vectors queued. Andrew this is
> > > the patch i promised, Rusty, i'm not sure if you find
> > > __attribute__((weak)) offensive...
> >
> > It's horrible.  Please move the unsetting of the cpu_online bit into the
> > arch-specific __cpu_disable() code for each arch, which is consistent
> > and also simplifies things.
>
> Alright this should do it then;

I'll sync against -mm shortly

Re: [PATCH][2.6] Hotplug cpu: Fix APIC queued timer vector race

[Rusty Russell]

On Mon, 2004-08-23 at 00:47, Zwane Mwaikambo wrote:
> On Sun, 22 Aug 2004, Rusty Russell wrote:
> 
> > On Sun, 2004-08-22 at 00:10, Zwane Mwaikambo wrote:
> > > Some timer interrupt vectors were queued on the Local APIC and were being
> > > serviced when we enabled interrupts again in fixup_irqs(), so we need to
> > > mask the APIC timer, enable interrupts so that any queued interrupts get
> > > processed whilst the processor is still on the online map and then clear
> > > ourselves from the online map. 1ms is a nice safe number even under heavy
> > > interrupt load with higher priority vectors queued. Andrew this is
> > > the patch i promised, Rusty, i'm not sure if you find
> > > __attribute__((weak)) offensive...
> >
> > It's horrible.  Please move the unsetting of the cpu_online bit into the
> > arch-specific __cpu_disable() code for each arch, which is consistent
> > and also simplifies things.
> 
> Alright this should do it then;
> 
> Thanks
> 
> Signed-off-by: Zwane Mwaikambo <zwane@linuxpower.ca>
> 
>  arch/i386/kernel/smpboot.c |   10 ++++++++--
>  arch/ia64/kernel/smpboot.c |    1 +
>  arch/ppc64/kernel/smp.c    |    4 +++-
>  arch/s390/kernel/smp.c     |    4 +++-
>  kernel/cpu.c               |    3 ---
>  5 files changed, 15 insertions(+), 7 deletions(-)
> 
> Index: linux-2.6.8.1-mm2/kernel/cpu.c
> ===================================================================
> RCS file: /home/cvsroot/linux-2.6.8.1-mm2/kernel/cpu.c,v
> retrieving revision 1.1.1.1
> diff -u -p -B -r1.1.1.1 cpu.c
> --- linux-2.6.8.1-mm2/kernel/cpu.c	19 Aug 2004 20:52:08 -0000	1.1.1.1
> +++ linux-2.6.8.1-mm2/kernel/cpu.c	22 Aug 2004 14:28:17 -0000
> @@ -89,9 +89,6 @@ static int take_cpu_down(void *unused)
>  {
>  	int err;
> 
> -	/* Take offline: makes arch_cpu_down somewhat easier. */
> -	cpu_clear(smp_processor_id(), cpu_online_map);
> -
>  	/* Ensure this CPU doesn't handle any more interrupts. */
>  	err = __cpu_disable();
>  	if (err < 0)

Hmm, I actually mean you to remove the cpu_set just after this, too, and
make the archs re-set th online_map if they failed.

A little more invasive, but more coherent.

Thanks,
Rusty.
-- 
Anyone who quotes me in their signature is an idiot -- Rusty Russell

[PATCH] Hotplug cpu: Fix APIC queued timer vector race #2

[Zwane Mwaikambo]

On Mon, 23 Aug 2004, Rusty Russell wrote:

> On Mon, 2004-08-23 at 00:47, Zwane Mwaikambo wrote:
> > On Sun, 22 Aug 2004, Rusty Russell wrote:
> >
> > > On Sun, 2004-08-22 at 00:10, Zwane Mwaikambo wrote:
> > > > Some timer interrupt vectors were queued on the Local APIC and were being
> > > > serviced when we enabled interrupts again in fixup_irqs(), so we need to
> > > > mask the APIC timer, enable interrupts so that any queued interrupts get
> > > > processed whilst the processor is still on the online map and then clear
> > > > ourselves from the online map. 1ms is a nice safe number even under heavy
> > > > interrupt load with higher priority vectors queued. Andrew this is
> > > > the patch i promised, Rusty, i'm not sure if you find
> > > > __attribute__((weak)) offensive...
> > >
> > > It's horrible.  Please move the unsetting of the cpu_online bit into the
> > > arch-specific __cpu_disable() code for each arch, which is consistent
> > > and also simplifies things.

Ok this should be the last one, against 2.6.8.1-mm4.

 arch/i386/kernel/smpboot.c |   21 +++++++++------------
 arch/ia64/kernel/smpboot.c |    3 ++-
 arch/ppc64/kernel/smp.c    |    4 +++-
 arch/s390/kernel/smp.c     |    4 +++-
 kernel/cpu.c               |   21 +++++----------------
 5 files changed, 22 insertions(+), 31 deletions(-)

Signed-off-by: Zwane Mwaikambo <zwane@linuxpower.ca>

Index: linux-2.6.8.1-mm4/kernel/cpu.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm4/kernel/cpu.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 cpu.c
--- linux-2.6.8.1-mm4/kernel/cpu.c	23 Aug 2004 08:24:54 -0000	1.1.1.1
+++ linux-2.6.8.1-mm4/kernel/cpu.c	23 Aug 2004 08:32:01 -0000
@@ -84,31 +84,20 @@ static int cpu_run_sbin_hotplug(unsigned
 	return call_usermodehelper(argv[0], argv, envp, 0);
 }

-void __attribute__((weak)) __cpu_prep_disable(void)
-{
-}
-
 /* Take this CPU down. */
 static int take_cpu_down(void *unused)
 {
 	int err;

-	/* Prepatory work before clearing the cpu off the online map */
-	__cpu_prep_disable();
-
-	/* Take offline: makes arch_cpu_down somewhat easier. */
-	cpu_clear(smp_processor_id(), cpu_online_map);
-
 	/* Ensure this CPU doesn't handle any more interrupts. */
 	err = __cpu_disable();
 	if (err < 0)
-		cpu_set(smp_processor_id(), cpu_online_map);
-	else
-		/* Force idle task to run as soon as we yield: it should
-		   immediately notice cpu is offline and die quickly. */
-		sched_idle_next();
+		return err;

-	return err;
+	/* Force idle task to run as soon as we yield: it should
+	   immediately notice cpu is offline and die quickly. */
+	sched_idle_next();
+	return 0;
 }

 int cpu_down(unsigned int cpu)
Index: linux-2.6.8.1-mm4/arch/i386/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm4/arch/i386/kernel/smpboot.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c
--- linux-2.6.8.1-mm4/arch/i386/kernel/smpboot.c	23 Aug 2004 08:23:22 -0000	1.1.1.1
+++ linux-2.6.8.1-mm4/arch/i386/kernel/smpboot.c	23 Aug 2004 08:32:01 -0000
@@ -1159,18 +1159,6 @@ static int __devinit cpu_enable(unsigned
 	return 0;
 }

-void __cpu_prep_disable(void)
-{
-	/* We enable the timer again on the exit path of the death loop */
-	if (smp_processor_id() != 0) {
-		disable_APIC_timer();
-		/* Allow any queued timer interrupts to get serviced */
-		local_irq_enable();
-		mdelay(1);
-		local_irq_disable();
-	}
-}
-
 int __cpu_disable(void)
 {
 	/*
@@ -1184,6 +1172,15 @@ int __cpu_disable(void)
 	if (smp_processor_id() == 0)
 		return -EBUSY;

+	/* We enable the timer again on the exit path of the death loop */
+	disable_APIC_timer();
+	/* Allow any queued timer interrupts to get serviced */
+	local_irq_enable();
+	mdelay(1);
+	local_irq_disable();
+
+	/* It's now safe to remove this processor from the online map */
+	cpu_clear(smp_processor_id(), cpu_online_map);
 	fixup_irqs();
 	return 0;
 }
Index: linux-2.6.8.1-mm4/arch/ia64/kernel/smpboot.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm4/arch/ia64/kernel/smpboot.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smpboot.c
--- linux-2.6.8.1-mm4/arch/ia64/kernel/smpboot.c	23 Aug 2004 08:23:31 -0000	1.1.1.1
+++ linux-2.6.8.1-mm4/arch/ia64/kernel/smpboot.c	23 Aug 2004 08:32:01 -0000
@@ -591,9 +591,10 @@ int __cpu_disable(void)
 	if (cpu == 0)
 		return -EBUSY;

+	cpu_clear(cpu, cpu_online_map);
 	fixup_irqs();
 	local_flush_tlb_all();
-	printk ("Disabled cpu %u\n", smp_processor_id());
+	printk("Disabled cpu %u\n", cpu);
 	return 0;
 }

Index: linux-2.6.8.1-mm4/arch/ppc64/kernel/smp.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm4/arch/ppc64/kernel/smp.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smp.c
--- linux-2.6.8.1-mm4/arch/ppc64/kernel/smp.c	23 Aug 2004 08:23:34 -0000	1.1.1.1
+++ linux-2.6.8.1-mm4/arch/ppc64/kernel/smp.c	23 Aug 2004 08:32:01 -0000
@@ -254,11 +254,13 @@ int __cpu_disable(void)
 {
 	/* FIXME: go put this in a header somewhere */
 	extern void xics_migrate_irqs_away(void);
+	int cpu = smp_processor_id();

+	cpu_clear(cpu, cpu_online_map);
 	systemcfg->processorCount--;

 	/*fix boot_cpuid here*/
-	if (smp_processor_id() == boot_cpuid)
+	if (cpu == boot_cpuid)
 		boot_cpuid = any_online_cpu(cpu_online_map);

 	/* FIXME: abstract this to not be platform specific later on */
Index: linux-2.6.8.1-mm4/arch/s390/kernel/smp.c
===================================================================
RCS file: /home/cvsroot/linux-2.6.8.1-mm4/arch/s390/kernel/smp.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 smp.c
--- linux-2.6.8.1-mm4/arch/s390/kernel/smp.c	23 Aug 2004 08:23:39 -0000	1.1.1.1
+++ linux-2.6.8.1-mm4/arch/s390/kernel/smp.c	23 Aug 2004 08:32:01 -0000
@@ -682,13 +682,15 @@ __cpu_disable(void)
 {
 	unsigned long flags;
 	ec_creg_mask_parms cr_parms;
+	int cpu = smp_processor_id();

 	spin_lock_irqsave(&smp_reserve_lock, flags);
-	if (smp_cpu_reserved[smp_processor_id()] != 0) {
+	if (smp_cpu_reserved[cpu] != 0) {
 		spin_unlock_irqrestore(&smp_reserve_lock, flags);
 		return -EBUSY;
 	}

+	cpu_clear(cpu, cpu_online_map);
 	/* disable all external interrupts */

 	cr_parms.start_ctl = 0;