Re: IO CPU affinity test results

["Alan D. Brunelle" <Alan.Brunelle@hp.com>]

Jens Axboe wrote:
> On Fri, Mar 14 2008, Jens Axboe wrote:
>> I think that is encouraging, for such a small setup. The make results
>> are particularly nice. The hangs are a bother, I have no good ideas on
>> why the occur. The fact that it happens on both archs indicates that
>> this is perhaps a generic problem, which is good. The code to support
>> this is relatively simple, so it should be possible to go over it with a
>> fine toothed comb and see if anything shows up.
>>
>> You didn't get any watchdog triggers on the serial console, or anything
>> like that?
>
> Here's something that may explain it - if interrupts aren't disabled
> when generic_smp_call_function_single() is called, we could deadlock
> on the dst->lock. I think that the IPI invoke will have interrupt
> disabled, but I'm not 100% certain.
>
> Can you see if this passes the muster?
>
> diff --git a/kernel/smp.c b/kernel/smp.c
> index 852abd3..65808df 100644
> --- a/kernel/smp.c
> +++ b/kernel/smp.c
> @@ -24,12 +24,13 @@ void __cpuinit generic_init_call_single_data(void)
>  void generic_smp_call_function_single_interrupt(void)
>  {
>  	struct call_single_queue *q;
> +	unsigned long flags;
>  	LIST_HEAD(list);
> 
>  	q = &__get_cpu_var(call_single_queue);
> -	spin_lock(&q->lock);
> +	spin_lock_irqsave(&q->lock);
>  	list_replace_init(&q->list, &list);
> -	spin_unlock(&q->lock);
> +	spin_unlock_irqrestore(&q->lock);
> 
>  	while (!list_empty(&list)) {
>  		struct call_single_data *data;
>

Well, putting in these two small patches (/always/ invoke send_IPI_single + this one, modified to include flags on the two spin calls), things look better after two runs (without profiling enabled), this was done on the 4-way ia64 box:

Part  RQ   MIN     AVG     MAX      Dev
----- --  ------  ------  ------  ------
 mkfs  0  18.151  18.311  18.472   0.227
 mkfs  1  18.420  18.456  18.492   0.051

untar  0  18.260  18.420  18.581   0.228
untar  1  18.594  19.477  20.360   1.249

 make  0  23.730  24.149  24.567   0.592
 make  1  23.401  23.599  23.797   0.280

 comb  0  60.141  60.881  61.620   1.046
 comb  1  60.810  61.532  62.253   1.020

 psys  0   4.16%   4.17%   4.18%   0.011
 psys  1   3.80%   3.93%   4.06%   0.180

Will do a longer set of runs and report out on that. As you had noticed earlier Jens, the make results look good...

Alan