Re: 2.6-bk: cpu hotplug + preempt = smp_processor_id warnings galore

[Nathan Lynch <ntl@pobox.com>]

On Sat, Feb 12, 2005 at 11:59:04AM -0700, Zwane Mwaikambo wrote:
> How about;
> 
> Index: linux-2.6.11-rc3-mm2/kernel/softirq.c
> ===================================================================
> RCS file: /home/cvsroot/linux-2.6.11-rc3-mm2/kernel/softirq.c,v
> retrieving revision 1.1.1.1
> diff -u -p -B -r1.1.1.1 softirq.c
> --- linux-2.6.11-rc3-mm2/kernel/softirq.c	11 Feb 2005 05:14:57 -0000	1.1.1.1
> +++ linux-2.6.11-rc3-mm2/kernel/softirq.c	12 Feb 2005 18:24:54 -0000
> @@ -355,8 +355,12 @@ static int ksoftirqd(void * __bind_cpu)
>  	set_current_state(TASK_INTERRUPTIBLE);
>  
>  	while (!kthread_should_stop()) {
> -		if (!local_softirq_pending())
> +		preempt_disable();
> +		if (!local_softirq_pending()) {
> +			preempt_enable_no_resched();
>  			schedule();
> +			preempt_disable();
> +		}
>  
>  		__set_current_state(TASK_RUNNING);
>  
> @@ -364,14 +368,14 @@ static int ksoftirqd(void * __bind_cpu)
>  			/* Preempt disable stops cpu going offline.
>  			   If already offline, we'll be on wrong CPU:
>  			   don't process */
> -			preempt_disable();
>  			if (cpu_is_offline((long)__bind_cpu))
>  				goto wait_to_die;
>  			do_softirq();
> -			preempt_enable();
> +			preempt_enable_no_resched();
>  			cond_resched();
> +			preempt_disable();
>  		}
> -
> +		preempt_enable();
>  		set_current_state(TASK_INTERRUPTIBLE);
>  	}
>  	__set_current_state(TASK_RUNNING);
> 

Works ok here.

It looks as if we need to explicitly bind worker threads to a newly
onlined cpu.  This gets rid of the smp_processor_id warnings from
cache_reap.  Adding a little more instrumentation to the debug
smp_processor_id showed that new worker threads were actually running
on the wrong cpu...

Does this look ok?

Index: linux-2.6.11-rc4-bk2/kernel/workqueue.c
===================================================================
--- linux-2.6.11-rc4-bk2.orig/kernel/workqueue.c	2005-02-14 11:13:08.000000000 -0600
+++ linux-2.6.11-rc4-bk2/kernel/workqueue.c	2005-02-14 15:18:35.000000000 -0600
@@ -485,8 +485,10 @@
 
 	case CPU_ONLINE:
 		/* Kick off worker threads. */
-		list_for_each_entry(wq, &workqueues, list)
+		list_for_each_entry(wq, &workqueues, list) {
+			kthread_bind(wq->cpu_wq[hotcpu].thread, hotcpu);
 			wake_up_process(wq->cpu_wq[hotcpu].thread);
+		}
 		break;
 
 	case CPU_UP_CANCELED: