Question on the scheduling of timer interrupt and FIO interrupt

Question on the scheduling of timer interrupt and FIO interrupt

[wangwudi]

Hi, all
When running some FIO tests on ARM64 server(Kunpeng), frequent NVMe interrupts occupy the
CPU, and the CPU's hardirq load is 100%. The watchdog feed interrupt arch_timer cannot be
responded, triggering the hardlockup.
GIC driver uses GICV3_PRIO_IRQ to set the same priority for arch_timer interrupt and NVMe
interrupt. In GIC spec, "If, on a particular CPU interface, multiple pending interrupts
have the same priority, and have sufficient priority for the interface to signal them to
the PE, it is IMPLEMENTATION DEFINED how the interface selects which interrupt to signal."
Shell we consider setting a higher priority for the arch_timer interrupt to fix this case?

Thanks for your help.
Wangwudi

Re: Question on the scheduling of timer interrupt and FIO interrupt

[Marc Zyngier]

+ Zenghui, in case he has seen this before.

On Fri, 01 Aug 2025 07:26:20 +0100,
wangwudi <wangwudi@hisilicon.com> wrote:
> 
> Hi, all
> When running some FIO tests on ARM64 server(Kunpeng), frequent NVMe interrupts occupy the
> CPU, and the CPU's hardirq load is 100%. The watchdog feed interrupt arch_timer cannot be
> responded, triggering the hardlockup.

I am extremely surprised that even with a screaming NVMe (or even
several of them), you end up in a situation where you don't have the
resource to take the timer interrupt.

>
> GIC driver uses GICV3_PRIO_IRQ to set the same priority for arch_timer interrupt and NVMe
> interrupt. In GIC spec, "If, on a particular CPU interface, multiple pending interrupts
> have the same priority, and have sufficient priority for the interface to signal them to
> the PE, it is IMPLEMENTATION DEFINED how the interface selects which interrupt to signal."
> Shell we consider setting a higher priority for the arch_timer interrupt to fix this case?

Linux only deals with two priorities: the normal interrupt priority,
and NMI, where the NMI can preempt any other interrupt. obviously, we
don't want to make the timer an NMI, as it would break a lot of
things.

Which means that even if you were to give the timer a higher priority,
it should not be allowed to preempt any other interrupt. Which means
that you'd need to set the binary point so that both the NVMe and
timer priorities fall into the same preemption bucket.

But it also means that you now are eating into the few bits of
priority that we have, and that will cause problems with the NMI
priority. Also, how to you decide what interrupts should be of a
higher priority?

I find it surprising that your GIC doesn't have some form of
round-robin scheme to pick the next HPPI, because that's clearly a
fairness problem, and punting that on SW is pretty ugly.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question on the scheduling of timer interrupt and FIO interrupt

[Zenghui Yu]

Hi Marc,

On 2025/8/1 20:01, Marc Zyngier wrote:
> + Zenghui, in case he has seen this before.

I haven't heard of it before.

> On Fri, 01 Aug 2025 07:26:20 +0100,
> wangwudi <wangwudi@hisilicon.com> wrote:
> >
> > Hi, all
> > When running some FIO tests on ARM64 server(Kunpeng), frequent NVMe interrupts occupy the
> > CPU, and the CPU's hardirq load is 100%. The watchdog feed interrupt arch_timer cannot be
> > responded, triggering the hardlockup.
> 
> I am extremely surprised that even with a screaming NVMe (or even
> several of them), you end up in a situation where you don't have the
> resource to take the timer interrupt.

+1. I will probably have an offline discussion with Wudi today, or a bit
later, to dig out more clues about it.

> > GIC driver uses GICV3_PRIO_IRQ to set the same priority for arch_timer interrupt and NVMe
> > interrupt. In GIC spec, "If, on a particular CPU interface, multiple pending interrupts
> > have the same priority, and have sufficient priority for the interface to signal them to
> > the PE, it is IMPLEMENTATION DEFINED how the interface selects which interrupt to signal."
> > Shell we consider setting a higher priority for the arch_timer interrupt to fix this case?
> 
> Linux only deals with two priorities: the normal interrupt priority,
> and NMI, where the NMI can preempt any other interrupt. obviously, we
> don't want to make the timer an NMI, as it would break a lot of
> things.
> 
> Which means that even if you were to give the timer a higher priority,
> it should not be allowed to preempt any other interrupt. Which means
> that you'd need to set the binary point so that both the NVMe and
> timer priorities fall into the same preemption bucket.
> 
> But it also means that you now are eating into the few bits of
> priority that we have, and that will cause problems with the NMI
> priority. Also, how to you decide what interrupts should be of a
> higher priority?
> 
> I find it surprising that your GIC doesn't have some form of
> round-robin scheme to pick the next HPPI, because that's clearly a
> fairness problem, and punting that on SW is pretty ugly.
> 
> Thanks,
> 
> 	M.
>