Re: [PATCH 1/3 v2] softirq: Introduce statistics for softirq

Keika Kobayashi
2008-11-24T23:46:22Z

On Sat, 22 Nov 2008 14:36:34 +0900 (JST)
KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com> wrote:

> > Statistics for softirq doesn't exist.
> > It will be helpful like statistics for interrupts.
> > This patch introduces counting the number of softirq,
> > which will be exported in /proc/softirqs.
> >
> > When softirq handler consumes much CPU time,
> > /proc/stat is like the following.
> >
> > $ while :; do cat /proc/stat | head -n1 ; sleep 10 ; done
> > cpu 88 0 408 739665 583 28 2 0 0
> > cpu 450 0 1090 740970 594 28 1294 0 0
> > ^^^^
> > softirq
> >
> > In such a situation,
> > /proc/softirqs shows us which softirq handler is invoked.
> > We can see the increase rate of softirqs.
> >
> > <before>
> > $ cat /proc/softirqs
> > CPU0 CPU1 CPU2 CPU3
> > HI 0 0 0 0
> > TIMER 462850 462805 462782 462718
> > NET_TX 0 0 0 365
> > NET_RX 2472 2 2 40
> > BLOCK 0 0 381 1164
> > TASKLET 0 0 0 224
> > SCHED 462654 462689 462698 462427
> > RCU 3046 2423 3367 3173
> >
> > <after>
> > $ cat /proc/softirqs
> > CPU0 CPU1 CPU2 CPU3
> > HI 0 0 0 0
> > TIMER 463361 465077 465056 464991
> > NET_TX 53 0 1 365
> > NET_RX 3757 2 2 40
> > BLOCK 0 0 398 1170
> > TASKLET 0 0 0 224
> > SCHED 463074 464318 464612 463330
> > RCU 3505 2948 3947 3673
>
> Maybe, this printing format don't works well on >100 CPUS.
>
> Do you have any experience of trouble-shooting by this patch?
> Actually, I don't understand this patch usuful working situation.

When reported that CPU TIME of softirq went up,
we had to investigate the cause.
In that case, kind of network looked suspicious.
We added such a function to confirm.

Original patch shows ticks of softirqs.
But /proc/interrups have counts of interrupts.
So we changed from ticks to counts.

> Honestly, I think HI and TASKLET are not beneficialness.
> end user can't know what tasklet spent time.
> TIMER also are not so much useful because it always have the same rate.
>
> So.. guessing...
>
> ultra band-width networking problem?
>
>
<snip>
>
> > /*
> > * Number of interrupts per specific IRQ source, since bootup
> > */
> > diff --git a/kernel/softirq.c b/kernel/softirq.c
> > index e7c69a7..088e179 100644
> > --- a/kernel/softirq.c
> > +++ b/kernel/softirq.c
> > @@ -208,6 +208,7 @@ restart:
> > do {
> > if (pending & 1) {
> > int prev_count = preempt_count();
> > + kstat_incr_softirqs_this_cpu(h - softirq_vec);
> >
> > h->action(h);
>
> you should explain this patch don't decrease system performance.

In this pach, only kstat_incr_softirqs_this_cpu() may
influence system performance.
But this just increments one variable.
I don't think we can detect the decrease in the performance.