Question about interrupt prioriyt of ARM GICv3/4

Question about interrupt prioriyt of ARM GICv3/4

[richard clark]

Hi,
Currently seems the GICv3/4 irqchip configures all the interrupts as
the same priority, I am thinking about to minimize the latency of the
interrupt for a particular device, e.g, the arm arch_timer in the RTL
system. The question is,
1. Why don't we provide a /proc or /sys interface for the enduser to
set the priority of a specific interrupt(SPI/PPI)?
2. Is there any way to verify the higher priority interrupt will have
more dominant to be selected to the CPU (IOW, the priority is really
working) in case of multiple different interrupts asserted to the GIC
at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
to record higher priority wins)?

Thanks

Re: Question about interrupt prioriyt of ARM GICv3/4

[Marc Zyngier]

On Fri, 06 Dec 2024 08:33:11 +0000,
richard clark <richard.xnu.clark@gmail.com> wrote:
> 
> Hi,
> Currently seems the GICv3/4 irqchip configures all the interrupts as
> the same priority, I am thinking about to minimize the latency of the
> interrupt for a particular device, e.g, the arm arch_timer in the RTL
> system. The question is,
> 1. Why don't we provide a /proc or /sys interface for the enduser to
> set the priority of a specific interrupt(SPI/PPI)?

I'm afraid this really has nothing to do with any particular interrupt
architecture.

Before thinking of exposing the interrupt priority to userspace, you
should look at what this translates into for the kernel once you allow
interrupts to be preempted by another one with a higher priority.

This means that at every point where you would normally see a
local_irq_save(), spinlock_irqsave() or equivalent, you would need to
explicitly specify the priority that you allow for preemption. You
should then make sure that any code that can be run during an
interrupt is reentrant. You need to define which data structures can
be manipulated at which priority level... The list goes on.

If you want a small taste of the complexity, just look at what
handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
generalise it to not just two, but an arbitrary range of priorities.

If you want the full blown experience, look at the BSDs and their use
of spl*(). I don't think anyone has any plan to get there, and the RT
patches have shown that there is little need for it.

> 2. Is there any way to verify the higher priority interrupt will have
> more dominant to be selected to the CPU (IOW, the priority is really
> working) in case of multiple different interrupts asserted to the GIC
> at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> to record higher priority wins)?

The GIC architecture makes no promise that the interrupt you
acknowledge is the highest priority pending interrupt, because this is
by definition a very racy process.

Also, even on busy systems, you will very rarely see two interrupts
targeting the same CPU being made pending at the same time, so that
the interrupt delivery system would have to arbitrate between the two.
That's because interrupts are vanishingly rare in the grand scheme of
things.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question about interrupt prioriyt of ARM GICv3/4

[richard clark]

Hi M,

On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
>
> On Fri, 06 Dec 2024 08:33:11 +0000,
> richard clark <richard.xnu.clark@gmail.com> wrote:
> >
> > Hi,
> > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > the same priority, I am thinking about to minimize the latency of the
> > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > system. The question is,
> > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > set the priority of a specific interrupt(SPI/PPI)?
>
> I'm afraid this really has nothing to do with any particular interrupt
> architecture.
>
> Before thinking of exposing the interrupt priority to userspace, you
> should look at what this translates into for the kernel once you allow
> interrupts to be preempted by another one with a higher priority.
>
Interrupt priority doesn't necessarily mean the preemption, seems
you're talking about the interrupt preemption harm according to your
statement followed.Frankly I am just thinking higher priority will win
the lower ones in case massive external interrupts received in the GIC
level (you see I am still talking about GIC, not kernel)
>
> This means that at every point where you would normally see a
> local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> explicitly specify the priority that you allow for preemption. You
> should then make sure that any code that can be run during an
> interrupt is reentrant. You need to define which data structures can
> be manipulated at which priority level... The list goes on.
>
irqsave just masks the interrupt from the point of cpu, I don't think
it will mess things up if preemption really happens (no? then what the
side-effect is for the nested interrupt handling in the softirq part.
damage the semantic of the lock primitives?)
>
> If you want a small taste of the complexity, just look at what
> handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> generalise it to not just two, but an arbitrary range of priorities.
>
> If you want the full blown experience, look at the BSDs and their use
> of spl*(). I don't think anyone has any plan to get there, and the RT
> patches have shown that there is little need for it.
>
As supplement，the fiq is suggested to be used as an alternative to the
higher priority in the RT area...
>
> > 2. Is there any way to verify the higher priority interrupt will have
> > more dominant to be selected to the CPU (IOW, the priority is really
> > working) in case of multiple different interrupts asserted to the GIC
> > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > to record higher priority wins)?
>
> The GIC architecture makes no promise that the interrupt you
> acknowledge is the highest priority pending interrupt, because this is
> by definition a very racy process.
>
> Also, even on busy systems, you will very rarely see two interrupts
> targeting the same CPU being made pending at the same time, so that
> the interrupt delivery system would have to arbitrate between the two.
> That's because interrupts are vanishingly rare in the grand scheme of
> things.
>
1. I am trying to stress the external interrupts to the core#0 via the
stress-ng tool with one of interrupt being configured as higher
priority to see the benchmark data, it's time consuming as you can
image, still is in progress(BTW, I can't see any lockup similar hang
in the system with a higher priority configured)
2. This raises a very interesting question and I am also very curious
about is, what is the purpose for the GIC to introduce the interrupt
priority features, a placeholder feature reserved for the future? Ah,
hardware prefer to provide more functionalities than its being
actually used by software, any other justification except that?
>
> Thanks,
>
>         M.
>
> --
> Without deviation from the norm, progress is not possible.

Re: Question about interrupt prioriyt of ARM GICv3/4

[Marc Zyngier]

On Thu, 12 Dec 2024 09:18:56 +0000,
richard clark <richard.xnu.clark@gmail.com> wrote:
> 
> Hi M,

Hi r,

> 
> On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
> >
> > On Fri, 06 Dec 2024 08:33:11 +0000,
> > richard clark <richard.xnu.clark@gmail.com> wrote:
> > >
> > > Hi,
> > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > the same priority, I am thinking about to minimize the latency of the
> > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > system. The question is,
> > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > set the priority of a specific interrupt(SPI/PPI)?
> >
> > I'm afraid this really has nothing to do with any particular interrupt
> > architecture.
> >
> > Before thinking of exposing the interrupt priority to userspace, you
> > should look at what this translates into for the kernel once you allow
> > interrupts to be preempted by another one with a higher priority.
> >
> Interrupt priority doesn't necessarily mean the preemption, seems
> you're talking about the interrupt preemption harm according to your
> statement followed.Frankly I am just thinking higher priority will win
> the lower ones in case massive external interrupts received in the GIC
> level (you see I am still talking about GIC, not kernel)

As I stated at the end of my email, the GIC only gives guarantee that
you will ack the highest priority interrupt in finite time. Not right
when it is made pending. Yes, it has the concept of HPPI. But that
from the PoV of the CPU interface, not that of the distributor. Factor
in the Stream interface, and you realise that expecting to always ack
the highest priority pending interrupt is akin to expecting no
reordering of packets in a network.

> >
> > This means that at every point where you would normally see a
> > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > explicitly specify the priority that you allow for preemption. You
> > should then make sure that any code that can be run during an
> > interrupt is reentrant. You need to define which data structures can
> > be manipulated at which priority level... The list goes on.
> >
> irqsave just masks the interrupt from the point of cpu, I don't think
> it will mess things up if preemption really happens (no? then what the
> side-effect is for the nested interrupt handling in the softirq part.
> damage the semantic of the lock primitives?)
> >
> > If you want a small taste of the complexity, just look at what
> > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > generalise it to not just two, but an arbitrary range of priorities.
> >
> > If you want the full blown experience, look at the BSDs and their use
> > of spl*(). I don't think anyone has any plan to get there, and the RT
> > patches have shown that there is little need for it.
> >
> As supplement，the fiq is suggested to be used as an alternative to the
> higher priority in the RT area...

<PulpFiction>
FIQ's dead, baby. FIQ's dead.
</PulpFiction>

> >
> > > 2. Is there any way to verify the higher priority interrupt will have
> > > more dominant to be selected to the CPU (IOW, the priority is really
> > > working) in case of multiple different interrupts asserted to the GIC
> > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > to record higher priority wins)?
> >
> > The GIC architecture makes no promise that the interrupt you
> > acknowledge is the highest priority pending interrupt, because this is
> > by definition a very racy process.
> >
> > Also, even on busy systems, you will very rarely see two interrupts
> > targeting the same CPU being made pending at the same time, so that
> > the interrupt delivery system would have to arbitrate between the two.
> > That's because interrupts are vanishingly rare in the grand scheme of
> > things.
> >
> 1. I am trying to stress the external interrupts to the core#0 via the
> stress-ng tool with one of interrupt being configured as higher
> priority to see the benchmark data, it's time consuming as you can
> image, still is in progress(BTW, I can't see any lockup similar hang
> in the system with a higher priority configured)

If you don't have preemption, I don't think anything wrong will
happen. But I don't expect any benefit either.

> 2. This raises a very interesting question and I am also very curious
> about is, what is the purpose for the GIC to introduce the interrupt
> priority features, a placeholder feature reserved for the future? Ah,
> hardware prefer to provide more functionalities than its being
> actually used by software, any other justification except that?

You realise that the HW is not exclusively designed for Linux, right?

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: Question about interrupt prioriyt of ARM GICv3/4

[Mark Kettenis]

> Date: Thu, 12 Dec 2024 10:12:32 +0000
> From: Marc Zyngier <maz@kernel.org>

Hi Marc, Richard,

> On Thu, 12 Dec 2024 09:18:56 +0000,
> richard clark <richard.xnu.clark@gmail.com> wrote:
> > 
> > Hi M,
> 
> Hi r,
> 
> > 
> > On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Fri, 06 Dec 2024 08:33:11 +0000,
> > > richard clark <richard.xnu.clark@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > > the same priority, I am thinking about to minimize the latency of the
> > > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > > system. The question is,
> > > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > > set the priority of a specific interrupt(SPI/PPI)?
> > >
> > > I'm afraid this really has nothing to do with any particular interrupt
> > > architecture.
> > >
> > > Before thinking of exposing the interrupt priority to userspace, you
> > > should look at what this translates into for the kernel once you allow
> > > interrupts to be preempted by another one with a higher priority.
> > >
> > Interrupt priority doesn't necessarily mean the preemption, seems
> > you're talking about the interrupt preemption harm according to your
> > statement followed.Frankly I am just thinking higher priority will win
> > the lower ones in case massive external interrupts received in the GIC
> > level (you see I am still talking about GIC, not kernel)
> 
> As I stated at the end of my email, the GIC only gives guarantee that
> you will ack the highest priority interrupt in finite time. Not right
> when it is made pending. Yes, it has the concept of HPPI. But that
> from the PoV of the CPU interface, not that of the distributor. Factor
> in the Stream interface, and you realise that expecting to always ack
> the highest priority pending interrupt is akin to expecting no
> reordering of packets in a network.
> 
> > >
> > > This means that at every point where you would normally see a
> > > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > > explicitly specify the priority that you allow for preemption. You
> > > should then make sure that any code that can be run during an
> > > interrupt is reentrant. You need to define which data structures can
> > > be manipulated at which priority level... The list goes on.
> > >
> > irqsave just masks the interrupt from the point of cpu, I don't think
> > it will mess things up if preemption really happens (no? then what the
> > side-effect is for the nested interrupt handling in the softirq part.
> > damage the semantic of the lock primitives?)
> > >
> > > If you want a small taste of the complexity, just look at what
> > > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > > generalise it to not just two, but an arbitrary range of priorities.
> > >
> > > If you want the full blown experience, look at the BSDs and their use
> > > of spl*(). I don't think anyone has any plan to get there, and the RT
> > > patches have shown that there is little need for it.
> > >
> > As supplement，the fiq is suggested to be used as an alternative to the
> > higher priority in the RT area...
> 
> <PulpFiction>
> FIQ's dead, baby. FIQ's dead.
> </PulpFiction>

Hah, tell that to Apple! ;).

> > > > 2. Is there any way to verify the higher priority interrupt will have
> > > > more dominant to be selected to the CPU (IOW, the priority is really
> > > > working) in case of multiple different interrupts asserted to the GIC
> > > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > > to record higher priority wins)?
> > >
> > > The GIC architecture makes no promise that the interrupt you
> > > acknowledge is the highest priority pending interrupt, because this is
> > > by definition a very racy process.
> > >
> > > Also, even on busy systems, you will very rarely see two interrupts
> > > targeting the same CPU being made pending at the same time, so that
> > > the interrupt delivery system would have to arbitrate between the two.
> > > That's because interrupts are vanishingly rare in the grand scheme of
> > > things.
> > >
> > 1. I am trying to stress the external interrupts to the core#0 via the
> > stress-ng tool with one of interrupt being configured as higher
> > priority to see the benchmark data, it's time consuming as you can
> > image, still is in progress(BTW, I can't see any lockup similar hang
> > in the system with a higher priority configured)
> 
> If you don't have preemption, I don't think anything wrong will
> happen. But I don't expect any benefit either.

Based on my experience with OpenBSD, I'm not even sure there is much
benefit even if you have preemtion.

And regarding anything wrong happening: there is an interesting bug in
the RK3399 GIC integration where it gets the priorities wrong:

https://github.com/openbsd/src/blob/feb3ea439d8f49b3c0e33f54c34631a611b98e21/sys/arch/arm64/dev/agintc.c#L395

(that comment is my interpretation of what's happening; I might be
misinterpreting what's really going on)

As far as I can tell the Linux code doesn't handle that quirk.
Probably it doesn't matter because Linux only uses the priority
mechanisms to implement pseudo-NMI functionality and/or doesn't do
preemption of interrupts.

> > 2. This raises a very interesting question and I am also very curious
> > about is, what is the purpose for the GIC to introduce the interrupt
> > priority features, a placeholder feature reserved for the future? Ah,
> > hardware prefer to provide more functionalities than its being
> > actually used by software, any other justification except that?
> 
> You realise that the HW is not exclusively designed for Linux, right?
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.
> 
> 

Re: Question about interrupt prioriyt of ARM GICv3/4

[Richard Clark]

Hi Mark,

On Thu, Dec 12, 2024 at 02:02:45PM +0100, Mark Kettenis wrote:
> > Date: Thu, 12 Dec 2024 10:12:32 +0000
> > From: Marc Zyngier <maz@kernel.org>
> 
> Hi Marc, Richard,
> 
> > On Thu, 12 Dec 2024 09:18:56 +0000,
> > richard clark <richard.xnu.clark@gmail.com> wrote:
> > > 
> > > Hi M,
> > 
> > Hi r,
> > 
> > > 
> > > On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > On Fri, 06 Dec 2024 08:33:11 +0000,
> > > > richard clark <richard.xnu.clark@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > > > the same priority, I am thinking about to minimize the latency of the
> > > > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > > > system. The question is,
> > > > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > > > set the priority of a specific interrupt(SPI/PPI)?
> > > >
> > > > I'm afraid this really has nothing to do with any particular interrupt
> > > > architecture.
> > > >
> > > > Before thinking of exposing the interrupt priority to userspace, you
> > > > should look at what this translates into for the kernel once you allow
> > > > interrupts to be preempted by another one with a higher priority.
> > > >
> > > Interrupt priority doesn't necessarily mean the preemption, seems
> > > you're talking about the interrupt preemption harm according to your
> > > statement followed.Frankly I am just thinking higher priority will win
> > > the lower ones in case massive external interrupts received in the GIC
> > > level (you see I am still talking about GIC, not kernel)
> > 
> > As I stated at the end of my email, the GIC only gives guarantee that
> > you will ack the highest priority interrupt in finite time. Not right
> > when it is made pending. Yes, it has the concept of HPPI. But that
> > from the PoV of the CPU interface, not that of the distributor. Factor
> > in the Stream interface, and you realise that expecting to always ack
> > the highest priority pending interrupt is akin to expecting no
> > reordering of packets in a network.
> > 
> > > >
> > > > This means that at every point where you would normally see a
> > > > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > > > explicitly specify the priority that you allow for preemption. You
> > > > should then make sure that any code that can be run during an
> > > > interrupt is reentrant. You need to define which data structures can
> > > > be manipulated at which priority level... The list goes on.
> > > >
> > > irqsave just masks the interrupt from the point of cpu, I don't think
> > > it will mess things up if preemption really happens (no? then what the
> > > side-effect is for the nested interrupt handling in the softirq part.
> > > damage the semantic of the lock primitives?)
> > > >
> > > > If you want a small taste of the complexity, just look at what
> > > > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > > > generalise it to not just two, but an arbitrary range of priorities.
> > > >
> > > > If you want the full blown experience, look at the BSDs and their use
> > > > of spl*(). I don't think anyone has any plan to get there, and the RT
> > > > patches have shown that there is little need for it.
> > > >
> > > As supplement，the fiq is suggested to be used as an alternative to the
> > > higher priority in the RT area...
> > 
> > <PulpFiction>
> > FIQ's dead, baby. FIQ's dead.
> > </PulpFiction>
> 
> Hah, tell that to Apple! ;).
>
Suppose you're kiding, seems neither Apple employee nor working on its HW :)
> 
> > > > > 2. Is there any way to verify the higher priority interrupt will have
> > > > > more dominant to be selected to the CPU (IOW, the priority is really
> > > > > working) in case of multiple different interrupts asserted to the GIC
> > > > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > > > to record higher priority wins)?
> > > >
> > > > The GIC architecture makes no promise that the interrupt you
> > > > acknowledge is the highest priority pending interrupt, because this is
> > > > by definition a very racy process.
> > > >
> > > > Also, even on busy systems, you will very rarely see two interrupts
> > > > targeting the same CPU being made pending at the same time, so that
> > > > the interrupt delivery system would have to arbitrate between the two.
> > > > That's because interrupts are vanishingly rare in the grand scheme of
> > > > things.
> > > >
> > > 1. I am trying to stress the external interrupts to the core#0 via the
> > > stress-ng tool with one of interrupt being configured as higher
> > > priority to see the benchmark data, it's time consuming as you can
> > > image, still is in progress(BTW, I can't see any lockup similar hang
> > > in the system with a higher priority configured)
> > 
> > If you don't have preemption, I don't think anything wrong will
> > happen. But I don't expect any benefit either.
> 
> Based on my experience with OpenBSD, I'm not even sure there is much
> benefit even if you have preemtion.
>
is OpenBSD has this priority feature supported? or do you have some related perf
data on BSP...
> 
> And regarding anything wrong happening: there is an interesting bug in
> the RK3399 GIC integration where it gets the priorities wrong:
> 
> https://github.com/openbsd/src/blob/feb3ea439d8f49b3c0e33f54c34631a611b98e21/sys/arch/arm64/dev/agintc.c#L395
> 
> (that comment is my interpretation of what's happening; I might be
> misinterpreting what's really going on)
> 
> As far as I can tell the Linux code doesn't handle that quirk.
> Probably it doesn't matter because Linux only uses the priority
> mechanisms to implement pseudo-NMI functionality and/or doesn't do
> preemption of interrupts.
>
seems the BSP has the priority support but encounter the bug/quirk, correct me if I am wrong. Frankly I have no
time to read the code of your link

	r.

> 
> > > 2. This raises a very interesting question and I am also very curious
> > > about is, what is the purpose for the GIC to introduce the interrupt
> > > priority features, a placeholder feature reserved for the future? Ah,
> > > hardware prefer to provide more functionalities than its being
> > > actually used by software, any other justification except that?
> > 
> > You realise that the HW is not exclusively designed for Linux, right?
> > 
> > 	M.
> > 
> > -- 
> > Without deviation from the norm, progress is not possible.
> > 
> > 

Re: Question about interrupt prioriyt of ARM GICv3/4

[Marc Zyngier]

On Thu, 12 Dec 2024 13:02:45 +0000,
Mark Kettenis <mark.kettenis@xs4all.nl> wrote:
> 
> Based on my experience with OpenBSD, I'm not even sure there is much
> benefit even if you have preemtion.
> 
> And regarding anything wrong happening: there is an interesting bug in
> the RK3399 GIC integration where it gets the priorities wrong:
> 
> https://github.com/openbsd/src/blob/feb3ea439d8f49b3c0e33f54c34631a611b98e21/sys/arch/arm64/dev/agintc.c#L395
> 
> (that comment is my interpretation of what's happening; I might be
> misinterpreting what's really going on)
> 
> As far as I can tell the Linux code doesn't handle that quirk.
> Probably it doesn't matter because Linux only uses the priority
> mechanisms to implement pseudo-NMI functionality and/or doesn't do
> preemption of interrupts.

Ah, beautiful! We actually do preemption with pseudo-NMI, and as it
turns out, I just had a report of 6.11 being broken on that SoC when
pNMIs are enabled.

My "solution" for this is to just disable security at the distributor
level, and let things rip, see [1].

Thanks for the heads up!

	M.

[1] https://lore.kernel.org/r/20241213141037.3995049-1-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

Re: Question about interrupt prioriyt of ARM GICv3/4

[Richard Clark]

Hi M,

On Thu, Dec 12, 2024 at 10:12:32AM +0000, Marc Zyngier wrote:
> On Thu, 12 Dec 2024 09:18:56 +0000,
> richard clark <richard.xnu.clark@gmail.com> wrote:
> > 
> > Hi M,
> 
> Hi r,
> 
> > 
> > On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > On Fri, 06 Dec 2024 08:33:11 +0000,
> > > richard clark <richard.xnu.clark@gmail.com> wrote:
> > > >
> > > > Hi,
> > > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > > the same priority, I am thinking about to minimize the latency of the
> > > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > > system. The question is,
> > > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > > set the priority of a specific interrupt(SPI/PPI)?
> > >
> > > I'm afraid this really has nothing to do with any particular interrupt
> > > architecture.
> > >
> > > Before thinking of exposing the interrupt priority to userspace, you
> > > should look at what this translates into for the kernel once you allow
> > > interrupts to be preempted by another one with a higher priority.
> > >
> > Interrupt priority doesn't necessarily mean the preemption, seems
> > you're talking about the interrupt preemption harm according to your
> > statement followed.Frankly I am just thinking higher priority will win
> > the lower ones in case massive external interrupts received in the GIC
> > level (you see I am still talking about GIC, not kernel)
> 
> As I stated at the end of my email, the GIC only gives guarantee that
> you will ack the highest priority interrupt in finite time. Not right
> when it is made pending. Yes, it has the concept of HPPI. But that
> from the PoV of the CPU interface, not that of the distributor. Factor
> in the Stream interface, and you realise that expecting to always ack
> the highest priority pending interrupt is akin to expecting no
> reordering of packets in a network.
> 

I am trying to figure out a meticulous way to verify if the priority really work
or not, all we're here is talk abou like the 'technically', we'd show the
statistic data collected. Except that the stress-ng to stress massive interrupts
to the cpu0, another way I can see is a kmod to remote cross cpu call targeting
the cpu0 from other cpus. From the current data collected, I do see some gain of
latency, still it's not finalized yet. Be patient baby :)

> > >
> > > This means that at every point where you would normally see a
> > > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > > explicitly specify the priority that you allow for preemption. You
> > > should then make sure that any code that can be run during an
> > > interrupt is reentrant. You need to define which data structures can
> > > be manipulated at which priority level... The list goes on.
> > >
> > irqsave just masks the interrupt from the point of cpu, I don't think
> > it will mess things up if preemption really happens (no? then what the
> > side-effect is for the nested interrupt handling in the softirq part.
> > damage the semantic of the lock primitives?)
> > >
> > > If you want a small taste of the complexity, just look at what
> > > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > > generalise it to not just two, but an arbitrary range of priorities.
> > >
> > > If you want the full blown experience, look at the BSDs and their use
> > > of spl*(). I don't think anyone has any plan to get there, and the RT
> > > patches have shown that there is little need for it.
> > >
> > As supplement，the fiq is suggested to be used as an alternative to the
> > higher priority in the RT area...
> 
> <PulpFiction>
> FIQ's dead, baby. FIQ's dead.
> </PulpFiction>
> 

As an advice too...

> > >
> > > > 2. Is there any way to verify the higher priority interrupt will have
> > > > more dominant to be selected to the CPU (IOW, the priority is really
> > > > working) in case of multiple different interrupts asserted to the GIC
> > > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > > to record higher priority wins)?
> > >
> > > The GIC architecture makes no promise that the interrupt you
> > > acknowledge is the highest priority pending interrupt, because this is
> > > by definition a very racy process.
> > >
> > > Also, even on busy systems, you will very rarely see two interrupts
> > > targeting the same CPU being made pending at the same time, so that
> > > the interrupt delivery system would have to arbitrate between the two.
> > > That's because interrupts are vanishingly rare in the grand scheme of
> > > things.
> > >
> > 1. I am trying to stress the external interrupts to the core#0 via the
> > stress-ng tool with one of interrupt being configured as higher
> > priority to see the benchmark data, it's time consuming as you can
> > image, still is in progress(BTW, I can't see any lockup similar hang
> > in the system with a higher priority configured)
> 
> If you don't have preemption, I don't think anything wrong will
> happen. But I don't expect any benefit either.
>

As a said before in this eamil, real statistic data is needed as evaluation,
WIP through now.

> 
> > 2. This raises a very interesting question and I am also very curious
> > about is, what is the purpose for the GIC to introduce the interrupt
> > priority features, a placeholder feature reserved for the future? Ah,
> > hardware prefer to provide more functionalities than its being
> > actually used by software, any other justification except that?
> 
> You realise that the HW is not exclusively designed for Linux, right?
> 
I realize, but it will be better if you can point is there any other OS
kernels make use of these interrupt priority of GIC but Linux, if no then
what the rational is for GIC to support these priority features...

	r. 
>
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.