CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Fushen Chen]

[-- Attachment #1: Type: text/plain, Size: 853 bytes --]

On APM82181,  "vmstat" (/proc/stat)  doesn't show correct idle percent, if
kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic Tick).

When I run wireless throughput test with heavy traffic, "vmstat" shows very
high idle percent while "oprofile" shows very low idle percent. During the
test, the system is idle, but network traffic uses a lot of hard IRQ and
soft-irq time. "vmstat" would have the correct stats if
account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more idle
time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ" in kernel,
idle percent in "vmstat" and "oprofile" would match.

My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
"CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".

My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
report correct stats.

Thanks,
Fushen

[-- Attachment #2: Type: text/html, Size: 1037 bytes --]

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Benjamin Herrenschmidt]

On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
> On APM82181,  "vmstat" (/proc/stat)  doesn't show correct idle
> percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
> Tick).
> 
> When I run wireless throughput test with heavy traffic, "vmstat" shows
> very high idle percent while "oprofile" shows very low idle percent.
> During the test, the system is idle, but network traffic uses a lot of
> hard IRQ and soft-irq time. "vmstat" would have the correct stats if
> account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
> idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
> in kernel, idle percent in "vmstat" and "oprofile" would match.
> 
> My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
> "CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
> 
> My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
> report correct stats.

Hi Thomas ! Any idea what we're doing wrong ? :-)

Cheers,
Ben.

> Thanks,
> Fushen
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Thomas Gleixner]

On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:

> On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
> > On APM82181,  "vmstat" (/proc/stat)  doesn't show correct idle
> > percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
> > Tick).
> > 
> > When I run wireless throughput test with heavy traffic, "vmstat" shows
> > very high idle percent while "oprofile" shows very low idle percent.
> > During the test, the system is idle, but network traffic uses a lot of
> > hard IRQ and soft-irq time. "vmstat" would have the correct stats if
> > account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
> > idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
> > in kernel, idle percent in "vmstat" and "oprofile" would match.
> > 
> > My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
> > "CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
> > 
> > My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
> > report correct stats.
> 
> Hi Thomas ! Any idea what we're doing wrong ? :-)

Not really, that had been an issue before and had been fixed. Peter ????

Thanks,

	tglx

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Benjamin Herrenschmidt]

On Wed, 2011-12-14 at 00:28 +0100, Thomas Gleixner wrote:
> On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:
> 
> > On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
> > > On APM82181,  "vmstat" (/proc/stat)  doesn't show correct idle
> > > percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
> > > Tick).
> > > 
> > > When I run wireless throughput test with heavy traffic, "vmstat" shows
> > > very high idle percent while "oprofile" shows very low idle percent.
> > > During the test, the system is idle, but network traffic uses a lot of
> > > hard IRQ and soft-irq time. "vmstat" would have the correct stats if
> > > account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add more
> > > idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
> > > in kernel, idle percent in "vmstat" and "oprofile" would match.
> > > 
> > > My APM82181 kernel configuration is "CONFIG_NO_HZ", "CONFIG_HZ_250=y",
> > > "CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
> > > 
> > > My question is that if kernel enables "CONFIG_NO_HZ", how would kernel
> > > report correct stats.
> > 
> > Hi Thomas ! Any idea what we're doing wrong ? :-)
> 
> Not really, that had been an issue before and had been fixed. Peter ????

Fusen, what kernel version is this ?

Cheers,
Ben.

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Andreas Schwab]

Does this help?

<http://permalink.gmane.org/gmane.linux.ports.ppc.embedded/47530>

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Fushen Chen]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

This is 2.6.32, but I think 2.6.36 is the same.
Thanks,
Fushen

On Tue, Dec 13, 2011 at 3:34 PM, Benjamin Herrenschmidt <
benh@kernel.crashing.org> wrote:

> On Wed, 2011-12-14 at 00:28 +0100, Thomas Gleixner wrote:
> > On Wed, 14 Dec 2011, Benjamin Herrenschmidt wrote:
> >
> > > On Tue, 2011-12-13 at 12:42 -0800, Fushen Chen wrote:
> > > > On APM82181,  "vmstat" (/proc/stat)  doesn't show correct idle
> > > > percent, if kernel enables "CONFIG_NO_HZ" (Tickless System / Dynamic
> > > > Tick).
> > > >
> > > > When I run wireless throughput test with heavy traffic, "vmstat"
> shows
> > > > very high idle percent while "oprofile" shows very low idle percent.
> > > > During the test, the system is idle, but network traffic uses a lot
> of
> > > > hard IRQ and soft-irq time. "vmstat" would have the correct stats if
> > > > account_idle_ticks(ticks) in kernel/time/tick-sched.c doesn't add
> more
> > > > idle time in "vmstat". In the same test, if I disable "CONFIG_NO_HZ"
> > > > in kernel, idle percent in "vmstat" and "oprofile" would match.
> > > >
> > > > My APM82181 kernel configuration is "CONFIG_NO_HZ",
> "CONFIG_HZ_250=y",
> > > > "CONFIG_HZ=250", and "CONFIG_HIGH_RES_TIMERS".
> > > >
> > > > My question is that if kernel enables "CONFIG_NO_HZ", how would
> kernel
> > > > report correct stats.
> > >
> > > Hi Thomas ! Any idea what we're doing wrong ? :-)
> >
> > Not really, that had been an issue before and had been fixed. Peter ????
>
> Fusen, what kernel version is this ?
>
> Cheers,
> Ben.
>
>
>

[-- Attachment #2: Type: text/html, Size: 2237 bytes --]

Re: CONFIG_NO_HZ added too much idle time in /proc/stat during throughput test.

[Anton Blanchard]

Hi,

> This is 2.6.32, but I think 2.6.36 is the same.

Sounds a bit like this, merged in 2.6.39.

Anton
--

commit ad5d1c888e556bc00c4e86f452cad4a3a87d22c1
Author: Anton Blanchard <anton@samba.org>
Date:   Sun Mar 20 15:28:03 2011 +0000

    powerpc: Fix accounting of softirq time when idle
    
    commit cf9efce0ce31 (powerpc: Account time using timebase rather
    than PURR) used in_irq() to detect if the time was spent in
    interrupt processing. This only catches hardirq context so if we
    are in softirq context and in the idle loop we end up accounting it
    as idle time. If we instead use in_interrupt() we catch both softirq
    and hardirq time.
    
    The issue was found when running a network intensive workload. top
    showed the following:
    
    0.0%us,  1.1%sy,  0.0%ni, 85.7%id,  0.0%wa,  9.9%hi,  3.3%si,  0.0%st
    
    85.7% idle. But this was wildly different to the perf events data.
    To confirm the suspicion I ran something to keep the core busy:
    
    # yes > /dev/null &
    
    8.2%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa, 10.3%hi, 81.4%si,  0.0%st
    
    We only got 8.2% of the CPU for the userspace task and softirq has
    shot up to 81.4%.
    
    With the patch below top shows the correct stats:
    
    0.0%us,  0.0%sy,  0.0%ni,  5.3%id,  0.0%wa, 13.3%hi, 81.3%si,  0.0%st
    
    Signed-off-by: Anton Blanchard <anton@samba.org>
    Cc: stable@kernel.org
    Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>