Re: [RFC][PATCH -rt] irqd starvation on SMP by a single process?

[Ingo Molnar <mingo@elte.hu>]

* john stultz <johnstul@us.ibm.com> wrote:

> -		if (!cpu_isset(smp_processor_id(), irq_affinity[irq])) {
> -			mask = cpumask_of_cpu(any_online_cpu(irq_affinity[irq]));
> -			set_cpus_allowed(current, mask);
> -		}

this is intentional, not a bug. The point of the code above is to ensure 
that every IRQ handler is executed on one CPU. I.e. the irq threads are 
semi-affine - they are strictly affine when executing a handler, but 
they may switch CPUs if the affinity mask points it elsewhere.

but ... we might be able to relax that. Potentially some IRQ handlers 
might assume per-cpu-ness, but that should be uncovered by 
DEBUG_PREEMPT's smp_processor_id() checks.

> +		if(!cpus_equal(current->cpus_allowed, irq_affinity[irq]));
> +			set_cpus_allowed(current, irq_affinity[irq]);

> The patch below appears to correct this issue, however it also
> repeatedly(on different irqs) causes the following BUG:

ah. This actually uncovered a real bug. We were calling __do_softirq() 
with interrupts enabled (and being preemptible) - which is certainly 
bad.

this was hidden before because the smp_processor_id() debugging code 
handles tasks bound to a single CPU as per-cpu-safe.

could you check the (totally untested) patch below and see if that fixes 
things for you? I've also added your affinity change.

	Ingo

Index: linux-rt.q/kernel/irq/manage.c
===================================================================
--- linux-rt.q.orig/kernel/irq/manage.c
+++ linux-rt.q/kernel/irq/manage.c
@@ -717,24 +717,21 @@ static int do_irqd(void * __desc)
 	if (param.sched_priority > 25)
 		curr_irq_prio = param.sched_priority - 1;
 
-//	param.sched_priority = 1;
 	sys_sched_setscheduler(current->pid, SCHED_FIFO, &param);
 
 	while (!kthread_should_stop()) {
 		set_current_state(TASK_INTERRUPTIBLE);
 		do_hardirq(desc);
 		cond_resched_all();
+		local_irq_disable();
 		__do_softirq();
-//		do_softirq_from_hardirq();
 		local_irq_enable();
 #ifdef CONFIG_SMP
 		/*
 		 * Did IRQ affinities change?
 		 */
-		if (!cpu_isset(smp_processor_id(), irq_affinity[irq])) {
-			mask = cpumask_of_cpu(any_online_cpu(irq_affinity[irq]));
-			set_cpus_allowed(current, mask);
-		}
+		if (!cpus_equal(current->cpus_allowed, irq_affinity[irq]));
+			set_cpus_allowed(current, irq_affinity[irq]);
 #endif
 		schedule();
 	}