NUMA-Q breakage 2/7 xquad_portio ioremap deadlock

NUMA-Q breakage 2/7 xquad_portio ioremap deadlock

[William Lee Irwin III]

The cpu_online_map stuff for hotplug cpu created a brand new bootstrap
ordering problem for NUMA-Q. The mmapped portio region needs to be
ioremapped early but ioremap attempts to do TLB shootdown, and
smp_call_function() (called by flush_tlb_all()) deadlocks when
cpu_online_map is uninitialized.

Workaround (due to Matt Dobson) below.



diff -Nur linux-2.5.23-vanilla/arch/i386/kernel/smp.c linux-2.5.23-patched/arch/i386/kernel/smp.c
--- linux-2.5.23-vanilla/arch/i386/kernel/smp.c	Tue Jun 18 19:11:47 2002
+++ linux-2.5.23-patched/arch/i386/kernel/smp.c	Mon Jul  8 14:52:32 2002
@@ -569,7 +569,7 @@
 	struct call_data_struct data;
 	int cpus = num_online_cpus()-1;
 
-	if (!cpus)
+	if (cpus <= 0)
 		return 0;
 
 	data.func = func;

Re: NUMA-Q breakage 2/7 xquad_portio ioremap deadlock

[Martin J. Bligh]

> The cpu_online_map stuff for hotplug cpu created a brand new bootstrap
> ordering problem for NUMA-Q. The mmapped portio region needs to be
> ioremapped early but ioremap attempts to do TLB shootdown, and
> smp_call_function() (called by flush_tlb_all()) deadlocks when
> cpu_online_map is uninitialized.
> 
> Workaround (due to Matt Dobson) below.
> 
> 
> 
> diff -Nur linux-2.5.23-vanilla/arch/i386/kernel/smp.c linux-2.5.23-patched/arch/i386/kernel/smp.c
> --- linux-2.5.23-vanilla/arch/i386/kernel/smp.c	Tue Jun 18 19:11:47 2002
> +++ linux-2.5.23-patched/arch/i386/kernel/smp.c	Mon Jul  8 14:52:32 2002
> @@ -569,7 +569,7 @@
>  	struct call_data_struct data;
>  	int cpus = num_online_cpus()-1;
>  
> -	if (!cpus)
> +	if (cpus <= 0)
>  		return 0;
>  
>  	data.func = func;

Would it be slightly less of a hack if we just move the ioremap down below
set_bit(0, &cpu_online_map); later on in smp_boot_cpus ? Untested patch
below. As long as we set up the xquad_portio remap before any other cpus 
are online, I can't see it matters exactly when we do it ....

Either that, or we just define cpu_online_map to be =1 to start with.

M.

--- virgin-2.5.25/arch/i386/kernel/smpboot.c	Fri Jul  5 16:42:23 2002
+++ linux-2.5.25-ioremap/arch/i386/kernel/smpboot.c	Fri Jul 12 15:55:20 2002
@@ -1019,16 +1019,6 @@
 {
 	int apicid, cpu, bit;
 
-        if (clustered_apic_mode && (numnodes > 1)) {
-                printk("Remapping cross-quad port I/O for %d quads\n",
-			numnodes);
-                printk("xquad_portio vaddr 0x%08lx, len %08lx\n",
-                        (u_long) xquad_portio, 
-			(u_long) numnodes * XQUAD_PORTIO_LEN);
-                xquad_portio = ioremap (XQUAD_PORTIO_BASE, 
-			numnodes * XQUAD_PORTIO_LEN);
-        }
-
 #ifdef CONFIG_MTRR
 	/*  Must be done before other processors booted  */
 	mtrr_init_boot_cpu ();
@@ -1126,6 +1116,16 @@
 
 	if (GET_APIC_ID(apic_read(APIC_ID)) != boot_cpu_physical_apicid)
 		BUG();
+
+        if (clustered_apic_mode && (numnodes > 1)) {
+                printk("Remapping cross-quad port I/O for %d quads\n",
+			numnodes);
+                printk("xquad_portio vaddr 0x%08lx, len %08lx\n",
+                        (u_long) xquad_portio, 
+			(u_long) numnodes * XQUAD_PORTIO_LEN);
+                xquad_portio = ioremap (XQUAD_PORTIO_BASE, 
+			numnodes * XQUAD_PORTIO_LEN);
+        }
 
 	/*
 	 * Scan the CPU present map and fire up the other CPUs via do_boot_cpu

Re: NUMA-Q breakage 2/7 xquad_portio ioremap deadlock

[William Lee Irwin III]

>> The cpu_online_map stuff for hotplug cpu created a brand new bootstrap
>> ordering problem for NUMA-Q. The mmapped portio region needs to be
>> ioremapped early but ioremap attempts to do TLB shootdown, and
>> smp_call_function() (called by flush_tlb_all()) deadlocks when
>> cpu_online_map is uninitialized.

On Fri, Jul 12, 2002 at 03:58:50PM -0700, Martin J. Bligh wrote:
> Would it be slightly less of a hack if we just move the ioremap down below
> set_bit(0, &cpu_online_map); later on in smp_boot_cpus ? Untested patch
> below. As long as we set up the xquad_portio remap before any other cpus 
> are online, I can't see it matters exactly when we do it ....
> Either that, or we just define cpu_online_map to be =1 to start with.
> M.

This looks better than fiddling with smp_call_function().


Cheers,
Bill