ksoftirqd doesn't really become active

[Andreas Arnez <arnez@vnet.ibm.com>]

While testing with arpflood, I stumbled across what looks like a
general flaw in the softirq processing: softirqs were keeping one CPU
busy; only 1% of this CPU capacity was left to other applications.
But still ksoftirqd didn't get scheduled much, most of the time
softirqs were processed in the context of irq_exit.  Hence the
scheduler didn't really influence the balancing between softirq
processing and other tasks, as probably intended.  This looks like a
general flaw which should affect every platform in one way or the
other.

Here's what seems to happen: an arp storm leads to the NAPI activating
softirqs all the time.  After 10 softirqs in a row the softirq
processing is suspended and ksoftirqd is woken up.  Now the scheduler
influences the remaining time until the next interrupt, after which 10
softirqs are processed in the context of irq_exit again.

Let
  s = average time spent on processing 10 softirqs;
  t = average time distance between hardirqs

If hardirqs come in regularly, then this CPU spends
ca. s/(t+t*int(s/t)) of its time on softirq processing in irq_exit
context:

                    s
   _________________/\________________
  /                                   \
  |=============|=============|=======>-----|
  \_____________/
         t

Once s approaches or exceeds t, the CPU is at least 50% busy
processing softirqs, regardless of the scheduler's decisions.

What are useful ways to increase the scheduler's influence?  One way
might be to suspend softirq processing in irq_exit context as long as
ksoftirqd feels responsible.  I made a test with a quick hack, and
this looks promising.  Are there other suggestions?  Or should the
whole issue be dealt with in a completely different way?