a bug on NO_HZ_FULL_ALL

a bug on NO_HZ_FULL_ALL

[Alex Shi]

CC to kernel mailing list.

What's you expected result for twd?


On 11/14/2013 08:36 AM, Shaojie Sun wrote:
> Hi all
>      When I enble NO_HZ_FULL and NO_HZ_FULL_ALL option in K3V2 board.
> I find that the twd interrupt will be trigged in a large time pre
> second.
>      In our landing team branch, we used lsk branch.
> 
>      Do you find this bugs on your board? And how to resolve it?
> 
> 
>>> cat /proc/interrupts
> 
>              CPU0     CPU1
>  29:        293        395       GIC  twd
> 


-- 
Thanks
    Alex

a bug on NO_HZ_FULL_ALL

[Alex Shi]

On 11/14/2013 10:41 AM, Alex Shi wrote:
> CC to kernel mailing list.
> 
> What's you expected result for twd?

BTW, the upstream kernel has same results for twd.
> 
> 
> On 11/14/2013 08:36 AM, Shaojie Sun wrote:
>> Hi all
>>      When I enble NO_HZ_FULL and NO_HZ_FULL_ALL option in K3V2 board.
>> I find that the twd interrupt will be trigged in a large time pre
>> second.
>>      In our landing team branch, we used lsk branch.
>>
>>      Do you find this bugs on your board? And how to resolve it?
>>
>>
>>>> cat /proc/interrupts
>>
>>              CPU0     CPU1
>>  29:        293        395       GIC  twd
>>
> 
> 


-- 
Thanks
    Alex

a bug on NO_HZ_FULL_ALL

[Vincent Guittot]

Hi Shaojie,

AFAICT, It's a none issue. In full nohz, a timer fires periodically
(around 4sec period on ARM IIRC) on one cpu (cpu0).

CC'ing Frederic

Regards,
Vincent



On 14 November 2013 08:46, Alex Shi <alex.shi@linaro.org> wrote:
> On 11/14/2013 10:41 AM, Alex Shi wrote:
>> CC to kernel mailing list.
>>
>> What's you expected result for twd?
>
> BTW, the upstream kernel has same results for twd.
>>
>>
>> On 11/14/2013 08:36 AM, Shaojie Sun wrote:
>>> Hi all
>>>      When I enble NO_HZ_FULL and NO_HZ_FULL_ALL option in K3V2 board.
>>> I find that the twd interrupt will be trigged in a large time pre
>>> second.
>>>      In our landing team branch, we used lsk branch.
>>>
>>>      Do you find this bugs on your board? And how to resolve it?
>>>
>>>
>>>>> cat /proc/interrupts
>>>
>>>              CPU0     CPU1
>>>  29:        293        395       GIC  twd
>>>
>>
>>
>
>
> --
> Thanks
>     Alex

a bug on NO_HZ_FULL_ALL

[viresh kumar]

On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
> AFAICT, It's a none issue. In full nohz, a timer fires periodically
> (around 4sec period on ARM IIRC) on one cpu (cpu0).

Timer should always be running on CPU0, its out of nohz-full domain. Its
cpu 1, where it will fire after long delays..


https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTickless

a bug on NO_HZ_FULL_ALL

[Shaojie Sun]

No, I think it is a bug.

Because I tested the option with NO_HZ_FULL and without
NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
With same code, I added NO_HZ_FULL_ALL option. It had too many
interruptes on CPU0 twd.

So the sumbitter just didn't test twd interrupts, when he expanded
NO_HZ_FULL option to all cpu.

On Thu, Nov 14, 2013 at 4:09 PM, viresh kumar <viresh.kumar@linaro.org> wrote:
> On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
>> AFAICT, It's a none issue. In full nohz, a timer fires periodically
>> (around 4sec period on ARM IIRC) on one cpu (cpu0).
>
> Timer should always be running on CPU0, its out of nohz-full domain. Its
> cpu 1, where it will fire after long delays..
>
>
> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTickless

a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

On Thu, Nov 14, 2013 at 05:54:10PM +0800, Shaojie Sun wrote:
> No, I think it is a bug.
> 
> Because I tested the option with NO_HZ_FULL and without
> NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
> With same code, I added NO_HZ_FULL_ALL option. It had too many
> interruptes on CPU0 twd.

If you select:

   NO_HZ_FULL=y
   NO_HZ_FULL_ALL=n

then you need to pass a nohz_full= cpu range in the boot parameter, otherwise it's
simply going to behave like NO_HZ_FULL=n

OTOH, if you select NO_HZ_FULL_ALL=y, the "nohz_full=" parameter is not needed and
all CPUs will be full dynticks except CPU 0 where you should see more tick than usual
because it's handling the timekeeping for every other CPUs. Don't forget to select
CONFIG_NO_HZ_FULL_SYSIDLE=y or CPU 0 will never shutdown its tick even if the entire
system is idle.

答复: a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

On Thu, Nov 14, 2013 at 12:39:24PM +0000, Sunshaojie wrote:
> Do you mean I must select CONFIG_NO_HZ_FULL_SYSIDLE option and merge the patch of "[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"?

Yes you need to select CONFIG_NO_HZ_FULL_SYSIDLE=y. But latest linus tree should be fine
as it contains that patch.

Thanks.

> 
> -----????-----
> ???: Frederic Weisbecker [mailto:fweisbec at gmail.com] 
> ????: 2013?11?14? 19:50
> ???: Shaojie Sun
> ??: viresh kumar; Vincent Guittot; Alex Shi; Fathi Boudra; Guodong Xu; Amit Kucheria; Chris Redpath; Linaro Kernel; LAK; Kevin Hilman; Sunshaojie
> ??: Re: a bug on NO_HZ_FULL_ALL
> 
> On Thu, Nov 14, 2013 at 05:54:10PM +0800, Shaojie Sun wrote:
> > No, I think it is a bug.
> > 
> > Because I tested the option with NO_HZ_FULL and without 
> > NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
> > With same code, I added NO_HZ_FULL_ALL option. It had too many 
> > interruptes on CPU0 twd.
> 
> If you select:
> 
>    NO_HZ_FULL=y
>    NO_HZ_FULL_ALL=n
> 
> then you need to pass a nohz_full= cpu range in the boot parameter, otherwise it's simply going to behave like NO_HZ_FULL=n
> 
> OTOH, if you select NO_HZ_FULL_ALL=y, the "nohz_full=" parameter is not needed and all CPUs will be full dynticks except CPU 0 where you should see more tick than usual because it's handling the timekeeping for every other CPUs. Don't forget to select CONFIG_NO_HZ_FULL_SYSIDLE=y or CPU 0 will never shutdown its tick even if the entire system is idle.

a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

BTW, support for ARM's full dynticks is uncomplete without
"[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"

...which I'm not sure is applied upstream, or even any ARM tree yet.

On Thu, Nov 14, 2013 at 05:54:10PM +0800, Shaojie Sun wrote:
> No, I think it is a bug.
> 
> Because I tested the option with NO_HZ_FULL and without
> NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
> With same code, I added NO_HZ_FULL_ALL option. It had too many
> interruptes on CPU0 twd.
> 
> So the sumbitter just didn't test twd interrupts, when he expanded
> NO_HZ_FULL option to all cpu.
> 
> On Thu, Nov 14, 2013 at 4:09 PM, viresh kumar <viresh.kumar@linaro.org> wrote:
> > On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
> >> AFAICT, It's a none issue. In full nohz, a timer fires periodically
> >> (around 4sec period on ARM IIRC) on one cpu (cpu0).
> >
> > Timer should always be running on CPU0, its out of nohz-full domain. Its
> > cpu 1, where it will fire after long delays..
> >
> >
> > https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTickless

a bug on NO_HZ_FULL_ALL

[Russell King - ARM Linux]

On Thu, Nov 14, 2013 at 12:53:15PM +0100, Frederic Weisbecker wrote:
> BTW, support for ARM's full dynticks is uncomplete without
> "[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"
> 
> ...which I'm not sure is applied upstream, or even any ARM tree yet.

It's in mainline as of last night, along with a fix to the above patch.

a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

On Thu, Nov 14, 2013 at 12:08:46PM +0000, Russell King - ARM Linux wrote:
> On Thu, Nov 14, 2013 at 12:53:15PM +0100, Frederic Weisbecker wrote:
> > BTW, support for ARM's full dynticks is uncomplete without
> > "[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"
> > 
> > ...which I'm not sure is applied upstream, or even any ARM tree yet.
> 
> It's in mainline as of last night, along with a fix to the above patch.

Great, thanks!

a bug on NO_HZ_FULL_ALL

[Alex Shi]

On 11/14/2013 08:08 PM, Russell King - ARM Linux wrote:
> On Thu, Nov 14, 2013 at 12:53:15PM +0100, Frederic Weisbecker wrote:
>> BTW, support for ARM's full dynticks is uncomplete without
>> "[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"
>>
>> ...which I'm not sure is applied upstream, or even any ARM tree yet.
> 
> It's in mainline as of last night, along with a fix to the above patch.
> 

I saw this in linus tree and in tip/master. According to content, it
seems no effort on this issue. The following testing base on latest code.

btw, the full_all do cause more interrupts.

#
# Timers subsystem
#
CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
# CONFIG_NO_HZ_IDLE is not set
CONFIG_NO_HZ_FULL=y
# CONFIG_NO_HZ_FULL_ALL is not set
CONFIG_NO_HZ_FULL_SYSIDLE=y
CONFIG_NO_HZ_FULL_SYSIDLE_SMALL=2
CONFIG_NO_HZ=y
CONFIG_HIGH_RES_TIMERS=y

alexs at alex-panda:~$ head -2 /proc/interrupts; sleep 10 ; head -2
/proc/interrupts
           CPU0       CPU1
 29:      12567       9783       GIC  29  twd
           CPU0       CPU1
 29:      12814       9942       GIC  29  twd

Then enabled CONFIG_NO_HZ_FULL_ALL. more than 200/second interrupt
increased.

alexs at alex-panda:~$ head -2 /proc/interrupts; sleep 10 ; head -2
/proc/interrupts
           CPU0       CPU1
 29:      34969       6384       GIC  29  twd
           CPU0       CPU1
 29:      37172       6646       GIC  29  twd


-- 
Thanks
    Alex

a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

On Thu, Nov 14, 2013 at 09:08:29PM +0800, Alex Shi wrote:
> On 11/14/2013 08:08 PM, Russell King - ARM Linux wrote:
> > On Thu, Nov 14, 2013 at 12:53:15PM +0100, Frederic Weisbecker wrote:
> >> BTW, support for ARM's full dynticks is uncomplete without
> >> "[PATCH] ARM: Support arch_irq_work_raise() via self IPIs"
> >>
> >> ...which I'm not sure is applied upstream, or even any ARM tree yet.
> > 
> > It's in mainline as of last night, along with a fix to the above patch.
> > 
> 
> I saw this in linus tree and in tip/master. According to content, it
> seems no effort on this issue. The following testing base on latest code.
> 
> btw, the full_all do cause more interrupts.
> 
> #
> # Timers subsystem
> #
> CONFIG_TICK_ONESHOT=y
> CONFIG_NO_HZ_COMMON=y
> # CONFIG_HZ_PERIODIC is not set
> # CONFIG_NO_HZ_IDLE is not set
> CONFIG_NO_HZ_FULL=y
> # CONFIG_NO_HZ_FULL_ALL is not set
> CONFIG_NO_HZ_FULL_SYSIDLE=y
> CONFIG_NO_HZ_FULL_SYSIDLE_SMALL=2
> CONFIG_NO_HZ=y
> CONFIG_HIGH_RES_TIMERS=y
> 
> alexs at alex-panda:~$ head -2 /proc/interrupts; sleep 10 ; head -2
> /proc/interrupts
>            CPU0       CPU1
>  29:      12567       9783       GIC  29  twd
>            CPU0       CPU1
>  29:      12814       9942       GIC  29  twd
> 
> Then enabled CONFIG_NO_HZ_FULL_ALL. more than 200/second interrupt
> increased.
> 
> alexs at alex-panda:~$ head -2 /proc/interrupts; sleep 10 ; head -2
> /proc/interrupts
>            CPU0       CPU1
>  29:      34969       6384       GIC  29  twd
>            CPU0       CPU1
>  29:      37172       6646       GIC  29  twd
> 

There are various context requirements to let a CPU shut down its tick most of the
time: having at most 1 task running, no posix cpu timers, not use perf events based
on frequency (which is the default behaviour of perf record), etc...

Try this selftest if you want to see how it behaves it your system:

    git://git.kernel.org/pub/scm/linux/kernel/git/frederic/dynticks-testing.git

a bug on NO_HZ_FULL_ALL

[Alex Shi]

On 11/14/2013 05:54 PM, Shaojie Sun wrote:
> No, I think it is a bug.
> 
> Because I tested the option with NO_HZ_FULL and without
> NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
> With same code, I added NO_HZ_FULL_ALL option. It had too many
> interruptes on CPU0 twd.
> 
> So the sumbitter just didn't test twd interrupts, when he expanded
> NO_HZ_FULL option to all cpu.
> 
> On Thu, Nov 14, 2013 at 4:09 PM, viresh kumar <viresh.kumar@linaro.org> wrote:
>> On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
>>> AFAICT, It's a none issue. In full nohz, a timer fires periodically
>>> (around 4sec period on ARM IIRC) on one cpu (cpu0).
>>
>> Timer should always be running on CPU0, its out of nohz-full domain. Its
>> cpu 1, where it will fire after long delays..
>>
>>
>> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTickless


Shaojie,
which kernel version has this bug?
I just find only the v3.13 kernel has opened the NO_HZ_FULL for 32 bit
kernel. and my testing base it.

but for lsk, it is 3.10 kernel, so unless you are using 64bit kernel to
do testing, is it right?

-- 
Thanks
    Alex

答复: a bug on NO_HZ_FULL_ALL

[Alex Shi]

On 11/19/2013 11:10 AM, Sunshaojie wrote:
> Hi Alex
> 
> 	I tested it on kernel V3.10 and on 32bits ARM system.
> 	Where did you see this option opened only for 64bits?
> 

kernel/time/Kconfig:

config NO_HZ_FULL
        bool "Full dynticks system (tickless)"
        # NO_HZ_COMMON dependency
        depends on !ARCH_USES_GETTIMEOFFSET && GENERIC_CLOCKEVENTS
        # We need at least one periodic CPU for timekeeping
        depends on SMP
        # RCU_USER_QS dependency
        depends on HAVE_CONTEXT_TRACKING
        # VIRT_CPU_ACCOUNTING_GEN dependency
        depends on 64BIT	<=======
        select NO_HZ_COMMON
        select RCU_USER_QS
        select RCU_NOCB_CPU
        select VIRT_CPU_ACCOUNTING_GEN
        select IRQ_WORK

> 
> -----????-----
> ???: Alex Shi [mailto:alex.shi at linaro.org] 
> ????: 2013?11?19? 11:03
> ???: Shaojie Sun; viresh kumar
> ??: Vincent Guittot; Fathi Boudra; Guodong Xu; Amit Kucheria; Chris Redpath; Linaro Kernel; LAK; Fr?d?ric Weisbecker; Kevin Hilman; Sunshaojie
> ??: Re: a bug on NO_HZ_FULL_ALL
> 
> On 11/14/2013 05:54 PM, Shaojie Sun wrote:
>> No, I think it is a bug.
>>
>> Because I tested the option with NO_HZ_FULL and without 
>> NO_HZ_FULL_ALL. It had only little interruptes on CPU0 twd.
>> With same code, I added NO_HZ_FULL_ALL option. It had too many 
>> interruptes on CPU0 twd.
>>
>> So the sumbitter just didn't test twd interrupts, when he expanded 
>> NO_HZ_FULL option to all cpu.
>>
>> On Thu, Nov 14, 2013 at 4:09 PM, viresh kumar <viresh.kumar@linaro.org> wrote:
>>> On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
>>>> AFAICT, It's a none issue. In full nohz, a timer fires periodically 
>>>> (around 4sec period on ARM IIRC) on one cpu (cpu0).
>>>
>>> Timer should always be running on CPU0, its out of nohz-full domain. 
>>> Its cpu 1, where it will fire after long delays..
>>>
>>>
>>> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTic
>>> kless
> 
> 
> Shaojie,
> which kernel version has this bug?
> I just find only the v3.13 kernel has opened the NO_HZ_FULL for 32 bit kernel. and my testing base it.
> 
> but for lsk, it is 3.10 kernel, so unless you are using 64bit kernel to do testing, is it right?
> 
> --
> Thanks
>     Alex
> 


-- 
Thanks
    Alex

a bug on NO_HZ_FULL_ALL

[Frederic Weisbecker]

On Thu, Nov 14, 2013 at 01:39:43PM +0530, viresh kumar wrote:
> On Thursday 14 November 2013 01:35 PM, Vincent Guittot wrote:
> > AFAICT, It's a none issue. In full nohz, a timer fires periodically
> > (around 4sec period on ARM IIRC) on one cpu (cpu0).
> 
> Timer should always be running on CPU0, its out of nohz-full domain. Its
> cpu 1, where it will fire after long delays..
> 
> 
> https://wiki.linaro.org/WorkingGroups/PowerManagement/Doc/AdaptiveTickless

Right. CPU 0 stays pure periodic unless you select CONFIG_NO_HZ_FULL_SYSIDLE=y
then it can be also nohz when the full system is idle. I strongly recommend for power
consumption.