[PATCH RESEND] nohz: Restart tick before do softirq

[PATCH RESEND] nohz: Restart tick before do softirq

[chenshiyan]

When the cpu is continuously idle, tick_irq_exit() will count next tick
expiry with maybe several periodic ticks, but if it enters softirq
before next tick, invoke_softirq() is called before tick_irq_exit(),
there will be no ticks during softirq. So with sched_clock_irqtime
closing, sirq is very low in nohz idle even if the softirq costs much
time beause of the sampling ticks missing(sirq will be much higher and
seem exact if the cpu is busy).

Signed-off-by: chenshiyan <chenshiyan@linux.alibaba.com>
---
 kernel/softirq.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/softirq.c b/kernel/softirq.c
index 9d71046..8f3f71b9 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -210,6 +210,7 @@ static inline void invoke_softirq(void)
 		return;
 
 	if (!force_irqthreads) {
+		tick_nohz_idle_restart_tick();
 #ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
 		/*
 		 * We can safely execute softirq on the current stack if
-- 
1.8.3.1

Re: [PATCH RESEND] nohz: Restart tick before do softirq

[Frederic Weisbecker]

Hi,

I'm a bit confused with the changelog, let's try to clarify:

On Mon, Feb 01, 2021 at 11:45:43AM +0800, chenshiyan wrote:
> When the cpu is continuously idle, tick_irq_exit() will count next tick
> expiry

What do you mean by counting next tick expiry here? Is is when
tick_nohz_full_update_tick() reprogramms the next tick?

> with maybe several periodic ticks

you mean that perhaps it's not going to stop it right now?


> but if it enters softirq before next tick

Not sure what you mean here.

> invoke_softirq() is called before tick_irq_exit(),
> there will be no ticks during softirq.

Ok the issue happens when an IRQ fires while the tick is stopped, right?


> So with sched_clock_irqtime
> closing, sirq is very low in nohz idle even if the softirq costs much
> time beause of the sampling ticks missing(sirq will be much higher and
> seem exact if the cpu is busy).

But this only happens when sched_clock() is based on jiffies, right?
Are you working on an architecture that doesn't implement it?

Thanks.

> 
> Signed-off-by: chenshiyan <chenshiyan@linux.alibaba.com>
> ---
>  kernel/softirq.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/kernel/softirq.c b/kernel/softirq.c
> index 9d71046..8f3f71b9 100644
> --- a/kernel/softirq.c
> +++ b/kernel/softirq.c
> @@ -210,6 +210,7 @@ static inline void invoke_softirq(void)
>  		return;
>  
>  	if (!force_irqthreads) {
> +		tick_nohz_idle_restart_tick();
>  #ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
>  		/*
>  		 * We can safely execute softirq on the current stack if
> -- 
> 1.8.3.1
> 

Re: [PATCH RESEND] nohz: Restart tick before do softirq

[chenshiyan]

Hi Frederic,

Thanks for replying. Let’s try to clarify:

> On Mon, Feb 01, 2021 at 11:45:43AM +0800, chenshiyan wrote:
>> When the cpu is continuously idle, tick_irq_exit() will count next tick
>> expiry
> 
> What do you mean by counting next tick expiry here? Is is when
> tick_nohz_full_update_tick() reprogramms the next tick?

Yes, tick_nohz_full_update_tick() will reprograms the next tick.

> 
>> with maybe several periodic ticks
> 
> you mean that perhaps it's not going to stop it right now?
> 
I mean after reprogramming, maybe next tick will come in a long time .
That’s normal and the problem is not here.

> 
>> but if it enters softirq before next tick
> 
> Not sure what you mean here.
In irq_exit(), it  may do  softirq handlings for long time, and during that time, there’s no ticks.
That’s the problem.

> 
>> invoke_softirq() is called before tick_irq_exit(),
>> there will be no ticks during softirq.
> 
> Ok the issue happens when an IRQ fires while the tick is stopped, right?

Right.

> 
> 
>> So with sched_clock_irqtime
>> closing, sirq is very low in nohz idle even if the softirq costs much
>> time beause of the sampling ticks missing(sirq will be much higher and
>> seem exact if the cpu is busy).
> 
> But this only happens when sched_clock() is based on jiffies, right?
> Are you working on an architecture that doesn't implement it?

But this only happens when sched_clock() is based on jiffies, right?
-More specifically, this only happens when CONFIG_IRQ_TIME_ACCOUNTING is close.
As above say, there’s no ticks during softirq, so we can’t count how long it takes.

Are you working on an architecture that doesn't implement it?
-No, we test it on x86.

The problem is , when it's doing softirq, but there’s no tick, so we don’t know how long 
It takes. The TOP or some other procs can not show the right value of sirq.

This patch tries to restart tick before softirq. So far it works and we get an accurate value of sirq.
Maybe it’s not the best way to solve this problem and hope you would like to help me.

Thanks.


> 2021年2月1日 下午8:38，Frederic Weisbecker <frederic@kernel.org> 写道：
> 
> Hi,
> 
> I'm a bit confused with the changelog, let's try to clarify:
> 
> On Mon, Feb 01, 2021 at 11:45:43AM +0800, chenshiyan wrote:
>> When the cpu is continuously idle, tick_irq_exit() will count next tick
>> expiry
> 
> What do you mean by counting next tick expiry here? Is is when
> tick_nohz_full_update_tick() reprogramms the next tick?
> 
>> with maybe several periodic ticks
> 
> you mean that perhaps it's not going to stop it right now?
> 
> 
>> but if it enters softirq before next tick
> 
> Not sure what you mean here.
> 
>> invoke_softirq() is called before tick_irq_exit(),
>> there will be no ticks during softirq.
> 
> Ok the issue happens when an IRQ fires while the tick is stopped, right?
> 
> 
>> So with sched_clock_irqtime
>> closing, sirq is very low in nohz idle even if the softirq costs much
>> time beause of the sampling ticks missing(sirq will be much higher and
>> seem exact if the cpu is busy).
> 
> But this only happens when sched_clock() is based on jiffies, right?
> Are you working on an architecture that doesn't implement it?
> 
> Thanks.
> 
>> 
>> Signed-off-by: chenshiyan <chenshiyan@linux.alibaba.com>
>> ---
>> kernel/softirq.c | 1 +
>> 1 file changed, 1 insertion(+)
>> 
>> diff --git a/kernel/softirq.c b/kernel/softirq.c
>> index 9d71046..8f3f71b9 100644
>> --- a/kernel/softirq.c
>> +++ b/kernel/softirq.c
>> @@ -210,6 +210,7 @@ static inline void invoke_softirq(void)
>> 		return;
>> 
>> 	if (!force_irqthreads) {
>> +		tick_nohz_idle_restart_tick();
>> #ifdef CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK
>> 		/*
>> 		 * We can safely execute softirq on the current stack if
>> -- 
>> 1.8.3.1

Re: [PATCH RESEND] nohz: Restart tick before do softirq

[Frederic Weisbecker]

On Wed, Feb 03, 2021 at 09:04:05PM +0800, chenshiyan wrote:
> Are you working on an architecture that doesn't implement it?
> -No, we test it on x86.
> 
> The problem is , when it's doing softirq, but there’s no tick, so we don’t know how long 
> It takes. The TOP or some other procs can not show the right value of sirq.

But IRQTIME counting is based on sched_clock() which shouldn't use jiffies
in the case in x86. In fact jiffies based sched_clock() isn't suitable for
IRQTIME at all. So I don't understand the problem.