Why no interrupt priorities?

Why no interrupt priorities?

[Tim Bird]

What's the rationale for not supporting interrupt priorities
in the kernel?

We're having a discussion of this in one of our CELF working
groups, and it would help if someone could explain why an
interrupt priority system has never been adopted in the
mainstream Linux kernel. (I know that some implementations
have been written and proposed).

Is there a strong policy against such a thing, or is it just
that the right implementation has not come along?

Thanks,

=============================
Tim Bird
Architecture Group Co-Chair
CE Linux Forum
Senior Staff Engineer
Sony Electronics
E-mail: Tim.Bird@am.sony.com
=============================

Re: Why no interrupt priorities?

[Richard B. Johnson]

On Thu, 26 Feb 2004, Tim Bird wrote:

> What's the rationale for not supporting interrupt priorities
> in the kernel?

Interrupt priorities are supported and have been supported
since the first cascaded interrupt controllers and, now
with the APIC. The interrupt priorities are enforced by
hardware. There are no "software interrupt priorities"
because we have more than one interrupt, already prioritized
by the hardware. The basic PC/AT has IRQ0 through IRQ15 interrupt
sources. The IO-APIC code emulates this. The priorites go like this:

Highest priority
  |
IRQ0	PIT channel 0
IRQ1	Keyboard
IRQ2	Cascade to second controller
IRQ8
IRQ9
IRQ10
IRQ11
IRQ12
IRQ13
IRQ14
IRQ15
IRQ3	Serial 1, Serial 3
IRQ4	Serial 0, Serial 2
IRQ5	Floppy disk
IRQ6
IRQ7	Printer
 |
Lowest priority

You can't do software interrupt priorities with hardware interrupt
controllers unless you funnel everything into one master ISR that
ACKs the hardware, then sorts through some priority lists. The
result is an abortion that wastes CPU cycles and throws away
the hardware advantage that you already have.

If you have an architecture that has only one hardware interrupt,
then you have no choice but to impliment some sort of software
priority scheme. This is not what we have on the ix86.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Why no interrupt priorities?

[Tim Bird]

Richard B. Johnson wrote:
> On Thu, 26 Feb 2004, Tim Bird wrote:
> 
>>What's the rationale for not supporting interrupt priorities
>>in the kernel?
> 
> Interrupt priorities are supported and have been supported
> since the first cascaded interrupt controllers and, now
> with the APIC. 

Please forgive my ignorance.  I'm not sure what's going
on with 2.6 and work queues, but do the hardware priorities
allow you to control scheduling of interrupt bottom halves?

=============================
Tim Bird
Architecture Group Co-Chair
CE Linux Forum
Senior Staff Engineer
Sony Electronics
E-mail: Tim.Bird@am.sony.com
=============================

Re: Why no interrupt priorities?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 1494 bytes --]

On Thu, 2004-02-26 at 22:02, Tim Bird wrote:
> Richard B. Johnson wrote:
> > On Thu, 26 Feb 2004, Tim Bird wrote:
> > 
> >>What's the rationale for not supporting interrupt priorities
> >>in the kernel?
> > 
> > Interrupt priorities are supported and have been supported
> > since the first cascaded interrupt controllers and, now
> > with the APIC. 
> 
> Please forgive my ignorance.  I'm not sure what's going
> on with 2.6 and work queues, but do the hardware priorities
> allow you to control scheduling of interrupt bottom halves?


hardware IRQ priorities are useless for the linux model. In linux, the
hardirq runs *very* briefly and then lets the softirq context do the
longer taking work. hardware irq priorities then don't matter really
because the hardirq's are hardly ever interrupted really, and when they
are they cause a performance *loss* due to cache trashing. The latency
added by waiting briefly is going to be really really short for any sane
hardware.

Now doing priorities in softirq context... well... here again it's a
case of a tiny latency hit vs a lot of cache trashing. If your softirq
handler runs in 10 cachemisses (it's useless to talk about cpu cycles
since most of teh time you'll be cache bound) that's not too long
latency, but if you interrupt it it might get 15 or more cachemisses
instead. That again will increase the delay the user context gets from
irq's.... so from a userspace pov you actually increased irq latency....

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why no interrupt priorities?

[Mark Gross]

On Thursday 26 February 2004 13:30, Arjan van de Ven wrote:
> On Thu, 2004-02-26 at 22:02, Tim Bird wrote:
> > Richard B. Johnson wrote:
> > > On Thu, 26 Feb 2004, Tim Bird wrote:
> > >>What's the rationale for not supporting interrupt priorities
> > >>in the kernel?
> > >
> > > Interrupt priorities are supported and have been supported
> > > since the first cascaded interrupt controllers and, now
> > > with the APIC.
> >
> > Please forgive my ignorance.  I'm not sure what's going
> > on with 2.6 and work queues, but do the hardware priorities
> > allow you to control scheduling of interrupt bottom halves?
>
> hardware IRQ priorities are useless for the linux model. In linux, the
> hardirq runs *very* briefly and then lets the softirq context do the
> longer taking work. hardware irq priorities then don't matter really
> because the hardirq's are hardly ever interrupted really, and when they
> are they cause a performance *loss* due to cache trashing. The latency
> added by waiting briefly is going to be really really short for any sane
> hardware.

Keep in mind the context is Linux running on non-sane hardware, sloooow CPUs, 
latency sensitive small io buffers etc. Losing system wide throughput to have 
the hardware codec not be starved is a happy trade off to make.

So far all the comments are very PC centric. I suspect the history of the no 
interrupt priorities goes back a lot of years.  Perhaps the answer is that it 
never seemed to help having priorities much on PC hardware so lets not deal 
with the complexity in the software.

Any more historical insights?
I wonder how hard it would be add for embedded types of 
applications/architectures?

--mgross

>
> Now doing priorities in softirq context... well... here again it's a
> case of a tiny latency hit vs a lot of cache trashing. If your softirq
> handler runs in 10 cachemisses (it's useless to talk about cpu cycles
> since most of teh time you'll be cache bound) that's not too long
> latency, but if you interrupt it it might get 15 or more cachemisses
> instead. That again will increase the delay the user context gets from
> irq's.... so from a userspace pov you actually increased irq latency....

Re: Why no interrupt priorities?

[Albert Cahalan]

Tim Bird writes:

> What's the rationale for not supporting interrupt priorities
> in the kernel?
>
> We're having a discussion of this in one of our CELF working
> groups, and it would help if someone could explain why an
> interrupt priority system has never been adopted in the
> mainstream Linux kernel. (I know that some implementations
> have been written and proposed).
>
> Is there a strong policy against such a thing, or is it just
> that the right implementation has not come along?

Simple reason: shared interrupts

Besides that, what you'd really want is a mask, not
a level. You might like to block out all network
interrupts while still allowing SCSI interrupts.
At some other time, you'd like the opposite.
None of the common hardware (PC, Mac, etc.) is at
all friendly to using a mask, even if you don't
have shared interrupts.

So the policy is: get in, shut the hardware up,
set a flag or wake something up, and get out.
Interrupt handlers are supposed to be very fast
and simple. Put the real work elsewhere.

RE: Why no interrupt priorities?

[Grover, Andrew]

> On Thursday 26 February 2004 13:30, Arjan van de Ven wrote:
> > hardware IRQ priorities are useless for the linux model. In 
> linux, the
> > hardirq runs *very* briefly and then lets the softirq context do the
> > longer taking work. hardware irq priorities then don't matter really
> > because the hardirq's are hardly ever interrupted really, 
> and when they
> > are they cause a performance *loss* due to cache trashing. 
> The latency
> > added by waiting briefly is going to be really really short 
> for any sane
> > hardware.

Is the assumption that hardirq handlers are superfast also the reason
why Linux calls all handlers on a shared interrupt, even if the first
handler reports it was for its device?

-- Andy

Re: Why no interrupt priorities?

[Randy.Dunlap]

On Thu, 26 Feb 2004 17:36:34 -0800 "Grover, Andrew" <andrew.grover@intel.com> wrote:

| > On Thursday 26 February 2004 13:30, Arjan van de Ven wrote:
| > > hardware IRQ priorities are useless for the linux model. In 
| > linux, the
| > > hardirq runs *very* briefly and then lets the softirq context do the
| > > longer taking work. hardware irq priorities then don't matter really
| > > because the hardirq's are hardly ever interrupted really, 
| > and when they
| > > are they cause a performance *loss* due to cache trashing. 
| > The latency
| > > added by waiting briefly is going to be really really short 
| > for any sane
| > > hardware.
| 
| Is the assumption that hardirq handlers are superfast also the reason
| why Linux calls all handlers on a shared interrupt, even if the first
| handler reports it was for its device?

Somehow I don't think that's the reasoning.

Is there a safe method to determine that there are no other pending
interrupts on one shared interrupt?  i.e., that other devices don't
also have interrupts pending?

--
~Randy

RE: Why no interrupt priorities?

[Benjamin Herrenschmidt]

> Is the assumption that hardirq handlers are superfast also the reason
> why Linux calls all handlers on a shared interrupt, even if the first
> handler reports it was for its device?

With level irqs only, it would be possible to return at this
point. But with edge irqs, we could miss it completely if another
device had an irq at the same time

Ben.

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 16:32:48 +1100, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

>
>> Is the assumption that hardirq handlers are superfast also the reason
>> why Linux calls all handlers on a shared interrupt, even if the first
>> handler reports it was for its device?
>
> With level irqs only, it would be possible to return at this
> point. But with edge irqs, we could miss it completely if another
> device had an irq at the same time

Is this to imply that edge triggered shared interrupts are used anywhere?

Never occured to me to use shared IRQ's edge triggered as this mode
_cannot_ work reliably for HW limitations.

Consider this scenario:

Two devices A) having it's IRQa and B) with it's IRQB.
Both devices "ored" on IRQBUS.

For simplicity lets assume these timings:
	IRQa,IRQB > IRQBUS: 		10ns
	IRQx transition time 		10ns
	PIC IRQ latch recovery time	10ns
	  This is the time IRQBUS input must be passive
	  for another edge on it to be detected

Here, by murphy IRQb arrives at the same time IRQa is reset.

Lets say one step (hyphen) is 5 ns
~ is a break in time
^ is a instable state glitch

           /-------~--------\	
IRQa  ---/                  \-------~------

                             /-------~------
IRQb  ------------~--------/

             /-----~--------^^-------~------
IRQBUS-----/


So, as you can see, IRQb gets lost as the PIC IRQ latch recovery
time is violated.

A work around would be to poll the _entire_ chain once more before
exiting the ISR - this would be rather clumsy though.

Regards
Michael

Re: Why no interrupt priorities?

[Benjamin Herrenschmidt]

> Never occured to me to use shared IRQ's edge triggered as this mode
> _cannot_ work reliably for HW limitations.

You are right, I wouldn't expect them to be shared indeed..

Ben.

Re: Why no interrupt priorities?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Thu, Feb 26, 2004 at 02:21:34PM -0800, Mark Gross wrote:
> > hardware IRQ priorities are useless for the linux model. In linux, the
> > hardirq runs *very* briefly and then lets the softirq context do the
> > longer taking work. hardware irq priorities then don't matter really
> > because the hardirq's are hardly ever interrupted really, and when they
> > are they cause a performance *loss* due to cache trashing. The latency
> > added by waiting briefly is going to be really really short for any sane
> > hardware.
> 
> Keep in mind the context is Linux running on non-sane hardware, sloooow CPUs, 

50Mhz is already really really fast in this context.

> latency sensitive small io buffers etc. Losing system wide throughput to have 
> the hardware codec not be starved is a happy trade off to make.

The point I tried to make was that it would INCREASE latency. Unless you
have misdesigned device drivers, which is something that is fixable :)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Why no interrupt priorities?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]


On Thu, Feb 26, 2004 at 05:36:34PM -0800, Grover, Andrew wrote:
> Is the assumption that hardirq handlers are superfast also the reason
> why Linux calls all handlers on a shared interrupt, even if the first
> handler reports it was for its device?

I guess so; and in addition it may avoid future irq's in a NAPI like way :)
Or it's just plain dead silly :)

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Why no interrupt priorities?

[Pete Zaitcev]

On Thu, 26 Feb 2004 11:05:07 -0800
Tim Bird <tim.bird@am.sony.com> wrote:

> What's the rationale for not supporting interrupt priorities
> in the kernel?

Interrupts interrupt other interrupts today, so this aspect of nested
interrupts is functional. In fact, it's a major headache, because
with explosion of the number of interrupt sources in modern x86 servers
were are pushed against the stack size.

The only thing remaining is something like UNIX spl(). You can overload
enable_irq, disable_irq with enabling IRQs numbered lower than the IRQ
being disabled, thus turning them into spl(). This is correct, but stupid.

Nobody writes drivers and arches like so, because if your _rely_ on such
implementation of enable_irq and disable_irq, the driver or framwork
becomes silently API dependant, and breaks silently.

If you want PILs like on sparc, well, your architecture can provide them.
They make no sense on x86, of course, so they are called some arch-specific
name, like __spl().

Why do you want "interrupt priorities", anyway? Is there a consumer
electronics application which requires that? No, seriously, what is it?

-- Pete

Re: Why no interrupt priorities?

[Russell King]

On Fri, Feb 27, 2004 at 02:26:31PM +0800, Michael Frank wrote:
> Is this to imply that edge triggered shared interrupts are used anywhere?

It is (or used to be) rather common with serial ports.  Remember that
COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
IRQ3.

> Never occured to me to use shared IRQ's edge triggered as this mode
> _cannot_ work reliably for HW limitations.

The serial driver takes great care with this - when we service such an
interrupt, we keep going until we have scanned all the devices until
such time that we can say "all devices are no longer signalling an
interrupt".

This is something it has always done - it's nothing new.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: Why no interrupt priorities?

[Helge Hafting]

Grover, Andrew wrote:
> 
> Is the assumption that hardirq handlers are superfast also the reason
> why Linux calls all handlers on a shared interrupt, even if the first
> handler reports it was for its device?
> 
No, it is the other way around.  hardirq handlers have to be superfast
because linux usually _have to_ call all the handlers of a shared irq.

The fact that one device did indeed have an interrupt for us doesn't mean
that the others didn't.  So all of them have to be checked to be safe.

If this becomes a performance problem, make sure that no _busy_ irqs
are shared.  The easy way is to shuffle pci cards around, or set
jumpers/switches or software controlled options.  Or resort to
reprogramming the APIC in extreme cases.

Helge Hafting

Re: Why no interrupt priorities?

[Ingo Oeser]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 27 February 2004 08:14, Arjan van de Ven wrote:
> The point I tried to make was that it would INCREASE latency. Unless you
> have misdesigned device drivers, which is something that is fixable :)
 
Not if you bought them as IP from some company. Or you did some
outsourcing and external development games.

Then you are basically lost and must workaround your problems instead of
fixing them.

Regards

Ingo Oeser

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFAPymkU56oYWuOrkARAsWUAKCTI3ikuVsV2xfFWS/frDBYCOJetACggQfr
NQcCvIS5I8jH4zWCPL5obYA=
=CoLz
-----END PGP SIGNATURE-----

Re: Why no interrupt priorities?

[Etienne Lorrain]

Russell King wrote:
> Michael Frank wrote:
> > Is this to imply that edge triggered shared interrupts
> > are used anywhere?
>
> It is (or used to be) rather common with serial ports.  Remember that
> COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
> IRQ3.

  If you are talking of old ISA serial cards, the IRQ were "shared" in
 the sense that the IRQ pin was not connected to the microprocessor
 when the device was not open (under Linux).
 So you could reliably have COM1 and COM3 on IRQ4 if you did not use
 the two devices at the same time - or use one of them in the polling
 mode (IRQ 0: handle by the timer at low speed).

  Some unusual hardware could handle two ports on the same IRQ (the
 same card mixing itself the IRQ of the two COM port present on the
 same ISA card) but the mixing could not be reliably handled at the
 ISA level. Using a resistor and/or a diode instead of a jumper to
 select the IRQ could also do the trick. So the software was ready
 to handle two interrupts from two UART on the same IRQ - but you
 had to have special hardware.

  Some people had success having a modem on COM1 and a serial _mouse_
 on COM3 (or the other way around) because the COM1/modem was winning
 the control of the ISA IRQ  and the COM3/modem was loosing but because
 of 1200 bauds transmission speed the timer interrupt was recovering the
 char received. So they could not tell the difference and did not know
 why inverting the two serial plug did not work.

  Now if you are talking of newer motherboard with COM port integrated,
 you may have been able to share (but unusual) - if you are talking
 of PCI card serial ports then that is another story.

  Etienne.


	

	
		
Yahoo! Mail : votre e-mail personnel et gratuit qui vous suit partout ! 
Créez votre Yahoo! Mail sur http://fr.benefits.yahoo.com/

Dialoguez en direct avec vos amis grâce à Yahoo! Messenger !Téléchargez Yahoo! Messenger sur http://fr.messenger.yahoo.com

Re: Why no interrupt priorities?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 479 bytes --]

On Fri, Feb 27, 2004 at 12:27:32PM +0100, Ingo Oeser wrote:
> On Friday 27 February 2004 08:14, Arjan van de Ven wrote:
> > The point I tried to make was that it would INCREASE latency. Unless you
> > have misdesigned device drivers, which is something that is fixable :)
>  
> Not if you bought them as IP from some company. Or you did some
> outsourcing and external development games.

I think that's a case of "Doctor Doctor it hurts when I do this" .. " Don't
do that then"

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Why no interrupt priorities?

[Christoph Hellwig]

On Thu, Feb 26, 2004 at 11:05:07AM -0800, Tim Bird wrote:
> What's the rationale for not supporting interrupt priorities
> in the kernel?

What do you actually want to do them?  Linux doesn't do the traditional
unix spl scheme for coplexity and performance reasons, see

www.usenix.org/publications/library/proceedings/ana97/full_papers/small/small.ps

for a related paper.  Give that linux hardirq handlers should be very small
there's no performance gain in that area for sane architectures, too.

Re: Why no interrupt priorities?

[Richard B. Johnson]

On Thu, 26 Feb 2004, Tim Bird wrote:

> Richard B. Johnson wrote:
> > On Thu, 26 Feb 2004, Tim Bird wrote:
> >
> >>What's the rationale for not supporting interrupt priorities
> >>in the kernel?
> >
> > Interrupt priorities are supported and have been supported
> > since the first cascaded interrupt controllers and, now
> > with the APIC.
>
> Please forgive my ignorance.  I'm not sure what's going
> on with 2.6 and work queues, but do the hardware priorities
> allow you to control scheduling of interrupt bottom halves?
>

Only to the extent that the bottom halves will naturally
occur after the hardware interrupt. There is no explicit
"hold execution until something else is done" in the queue.

It is a queue, i.e., one task after another.

> =============================
> Tim Bird
> Architecture Group Co-Chair
> CE Linux Forum
> Senior Staff Engineer
> Sony Electronics
> E-mail: Tim.Bird@am.sony.com
> =============================
>

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 12:37:25 +0100 (CET), Etienne Lorrain <etienne_lorrain@yahoo.fr> wrote:

> Russell King wrote:
>> Michael Frank wrote:
>> > Is this to imply that edge triggered shared interrupts
>> > are used anywhere?
>>
>> It is (or used to be) rather common with serial ports.  Remember that
>> COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
>> IRQ3.
>
>   Some unusual hardware could handle two ports on the same IRQ (the
>  same card mixing itself the IRQ of the two COM port present on the
>  same ISA card) but the mixing could not be reliably handled at the
>  ISA level. Using a resistor and/or a diode instead of a jumper to
>  select the IRQ could also do the trick. So the software was ready
>  to handle two interrupts from two UART on the same IRQ - but you
>  had to have special hardware.

I used those cheapy 8 Channel 8250 serial cards and modified them
by oring 8 interrupts to irq 3.

The IRQ was set to level triggered and on an IRQ one polled the
status regs of all 8 channels. That was _just_ OK on a 8MHz 286
at 9600 baud in a custom designed low latency application
(no FIFOS here).

>
>   Some people had success having a modem on COM1 and a serial _mouse_
>  on COM3 (or the other way around) because the COM1/modem was winning
>  the control of the ISA IRQ  and the COM3/modem was loosing but because
>  of 1200 bauds transmission speed the timer interrupt was recovering the
>  char received. So they could not tell the difference and did not know
>  why inverting the two serial plug did not work.

:-(

Regards
Michael

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 09:05:48 +0000, Russell King <rmk+lkml@arm.linux.org.uk> wrote:

> On Fri, Feb 27, 2004 at 02:26:31PM +0800, Michael Frank wrote:
>> Is this to imply that edge triggered shared interrupts are used anywhere?
>
> It is (or used to be) rather common with serial ports.  Remember that
> COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
> IRQ3.
>
>> Never occured to me to use shared IRQ's edge triggered as this mode
>> _cannot_ work reliably for HW limitations.
>
> The serial driver takes great care with this - when we service such an
> interrupt, we keep going until we have scanned all the devices until
> such time that we can say "all devices are no longer signalling an
> interrupt".
>
> This is something it has always done - it's nothing new.
>

Sorry, i think the serial driver IRQ is level triggered :)

Regards
Michael

Re: Why no interrupt priorities?

[Richard B. Johnson]

On Fri, 27 Feb 2004, Michael Frank wrote:

> On Fri, 27 Feb 2004 09:05:48 +0000, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>
> > On Fri, Feb 27, 2004 at 02:26:31PM +0800, Michael Frank wrote:
> >> Is this to imply that edge triggered shared interrupts are used anywhere?
> >
> > It is (or used to be) rather common with serial ports.  Remember that
> > COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
> > IRQ3.
> >
> >> Never occured to me to use shared IRQ's edge triggered as this mode
> >> _cannot_ work reliably for HW limitations.
> >
> > The serial driver takes great care with this - when we service such an
> > interrupt, we keep going until we have scanned all the devices until
> > such time that we can say "all devices are no longer signalling an
> > interrupt".
> >
> > This is something it has always done - it's nothing new.
> >
>
> Sorry, i think the serial driver IRQ is level triggered :)

           CPU0       CPU1
  0:    7904936    7922491    IO-APIC-edge  timer
  1:      74277      70096    IO-APIC-edge  keyboard
  2:          0          0          XT-PIC  cascade
  3:        535        556    IO-APIC-edge  serial <--- EDGE
  4:          2          1    IO-APIC-edge  serial <--- EDGE
 10:    3435637    3440251   IO-APIC-level  eth0
 11:     295018     296838   IO-APIC-level  BusLogic BT-958
 13:       4567       4591   IO-APIC-level  Analogic(tm) AMX/AFF Driver
 14:       1331       1239   IO-APIC-level  Analogic(tm) DSP/DSS Driver
 15:          2          3   IO-APIC-level  Analogic(tm) VXI/MSG Driver
NMI:          0          0
LOC:   15826338   15826337
ERR:          0
MIS:          2

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Why no interrupt priorities?

[Russell King]

On Fri, Feb 27, 2004 at 09:31:43PM +0800, Michael Frank wrote:
> On Fri, 27 Feb 2004 09:05:48 +0000, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
> > On Fri, Feb 27, 2004 at 02:26:31PM +0800, Michael Frank wrote:
> >> Is this to imply that edge triggered shared interrupts are used anywhere?
> >
> > It is (or used to be) rather common with serial ports.  Remember that
> > COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
> > IRQ3.
> >
> >> Never occured to me to use shared IRQ's edge triggered as this mode
> >> _cannot_ work reliably for HW limitations.
> >
> > The serial driver takes great care with this - when we service such an
> > interrupt, we keep going until we have scanned all the devices until
> > such time that we can say "all devices are no longer signalling an
> > interrupt".
> >
> > This is something it has always done - it's nothing new.
> >
> 
> Sorry, i think the serial driver IRQ is level triggered :)

That's actually incorrect.  Serial devices are (were) connected to the
old ISA PICs which are definitely edge triggered.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 PCMCIA      - http://pcmcia.arm.linux.org.uk/
                 2.6 Serial core

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 13:50:19 +0000, Russell King <rmk+lkml@arm.linux.org.uk> wrote:

> On Fri, Feb 27, 2004 at 09:31:43PM +0800, Michael Frank wrote:
>> On Fri, 27 Feb 2004 09:05:48 +0000, Russell King <rmk+lkml@arm.linux.org.uk> wrote:
>> > On Fri, Feb 27, 2004 at 02:26:31PM +0800, Michael Frank wrote:
>> >> Is this to imply that edge triggered shared interrupts are used anywhere?
>> >
>> > It is (or used to be) rather common with serial ports.  Remember that
>> > COM1 and COM3 were both defined to use IRQ4 and COM2 and COM4 to use
>> > IRQ3.
>> >
>> >> Never occured to me to use shared IRQ's edge triggered as this mode
>> >> _cannot_ work reliably for HW limitations.
>> >
>> > The serial driver takes great care with this - when we service such an
>> > interrupt, we keep going until we have scanned all the devices until
>> > such time that we can say "all devices are no longer signalling an
>> > interrupt".
>> >
>> > This is something it has always done - it's nothing new.
>> >
>>
>> Sorry, i think the serial driver IRQ is level triggered :)
>
> That's actually incorrect.  Serial devices are (were) connected to the
> old ISA PICs which are definitely edge triggered.
>

I was under the impression that the PIC's are historically set to
level triggered, certainly was the case with (IBM) PC's/AT's and
with embedded system I am working with.

At least it explains why I was never able to share IRQ's on hardware
with PIC's under linux.

Regards
Michael

RE: Why no interrupt priorities?

[Grover, Andrew]

> From: Helge Hafting [mailto:helgehaf@aitel.hist.no] 
> Grover, Andrew wrote:
> > Is the assumption that hardirq handlers are superfast also 
> the reason
> > why Linux calls all handlers on a shared interrupt, even if 
> the first
> > handler reports it was for its device?
> > 
> No, it is the other way around.  hardirq handlers have to be superfast
> because linux usually _have to_ call all the handlers of a shared irq.
> 
> The fact that one device did indeed have an interrupt for us 
> doesn't mean
> that the others didn't.  So all of them have to be checked to be safe.

If a device later in the handler chain is also interrupting, then the
interrupt will immediately trigger again. The irq line will remain
asserted until nobody is asserting it.

If the LAST guy in the chain is the one with the interrupt, then you
basically get today's ISR "call each handler" behavior, but it should be
possible to in some cases to get less time spent in do_IRQ.

It is a trivial change to implement this behavior, but benchmarking
would have to be done to verify it really would be a worthwhile
optimization.

Regards -- Andy

Re: Why no interrupt priorities?

[Chris Friesen]

Grover, Andrew wrote:

> If a device later in the handler chain is also interrupting, then the
> interrupt will immediately trigger again. The irq line will remain
> asserted until nobody is asserting it.

I thought I saw examples of edge-triggered shared interrupts earlier in 
the thread.  Doesn't that give the reason for this behaviour?

Chris

-- 
Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: Why no interrupt priorities?

[Mike Fedyk]

Helge Hafting wrote:
> Or resort to
> reprogramming the APIC in extreme cases.

Would this typically require kernel code, or is there an interface 
exposed to userspace?

Re: Why no interrupt priorities?

[Richard B. Johnson]

On Fri, 27 Feb 2004, Chris Friesen wrote:

> Grover, Andrew wrote:
>
> > If a device later in the handler chain is also interrupting, then the
> > interrupt will immediately trigger again. The irq line will remain
> > asserted until nobody is asserting it.
>
> I thought I saw examples of edge-triggered shared interrupts earlier in
> the thread.  Doesn't that give the reason for this behaviour?
>
> Chris
>
> --
> Chris Friesen                    | MailStop: 043/33/F10
> Nortel Networks                  | work: (613) 765-0557
> 3500 Carling Avenue              | fax:  (613) 765-2986
> Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

In the early IBM/AT, there was a port to which a user of
a shared "edge" interrupt could write. If the interrupt
line was still asserted, this would generate another edge.

This meant that any ISR needed to know about other users
of the same interrupt. This is probably why it didn't
catch on.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Why no interrupt priorities?

[Matt Mackall]

On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no] 
> > Grover, Andrew wrote:
> > > Is the assumption that hardirq handlers are superfast also 
> > the reason
> > > why Linux calls all handlers on a shared interrupt, even if 
> > the first
> > > handler reports it was for its device?
> > > 
> > No, it is the other way around.  hardirq handlers have to be superfast
> > because linux usually _have to_ call all the handlers of a shared irq.
> > 
> > The fact that one device did indeed have an interrupt for us 
> > doesn't mean
> > that the others didn't.  So all of them have to be checked to be safe.
> 
> If a device later in the handler chain is also interrupting, then the
> interrupt will immediately trigger again. The irq line will remain
> asserted until nobody is asserting it.
> 
> If the LAST guy in the chain is the one with the interrupt, then you
> basically get today's ISR "call each handler" behavior, but it should be
> possible to in some cases to get less time spent in do_IRQ.

Let's imagine you have n sources simultaneously interrupting on a
given descriptor. Check the first, it's happening, acknowledge it,
exit, notice interrupt still asserted, check the first, nope, check
the second, yep, exit, etc. By the time we've made it to the nth ISR,
we've banged on the first one n times, the second n-1 times, etc. In
other words, early chain termination has an O(n^2) worst case.

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: Why no interrupt priorities?

[Tim Hockin]

On Fri, Feb 27, 2004 at 12:55:55PM -0600, Matt Mackall wrote:
> Let's imagine you have n sources simultaneously interrupting on a
> given descriptor. Check the first, it's happening, acknowledge it,
> exit, notice interrupt still asserted, check the first, nope, check
> the second, yep, exit, etc. By the time we've made it to the nth ISR,
> we've banged on the first one n times, the second n-1 times, etc. In
> other words, early chain termination has an O(n^2) worst case.

That is a pretty pathological worst case, and n is (almost?) always small.
I don't know if it would make a lick of difference, or if it is worth the
risk. Someone who has a lot of shared interrupts ought to try it.

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 13:15:40 -0500, Chris Friesen <cfriesen@nortelnetworks.com> wrote:

> Grover, Andrew wrote:
>
>> If a device later in the handler chain is also interrupting, then the
>> interrupt will immediately trigger again. The irq line will remain
>> asserted until nobody is asserting it.
>
> I thought I saw examples of edge-triggered shared interrupts earlier in
> the thread.  Doesn't that give the reason for this behaviour?

To make it more clear

                 /---active--~--\
IRQa	---------/                \-------------------------

                                 /-------------------------
IRQb  ----------------------~--/

                  /----------~--\/\/-----------------------
IRQBUS ---------/

Program  ....Main.....|.ISR...........|.Main............
Flow Edge trig                          Interrupt _dead_

Program  ....Main.....|.ISR...........|.Main.|ISR.....
Flow Level trig

When IRQBUS is level triggered, ISR is reeentered fairly quickly
(within one instruction) after IRET (depends on CPU).

The only way to "fix" edge-trig shared IRQ's is to

a) perform blind extra poll of the _entire_ chain until _no_ devices
    were serviced
b) Check IRQBUS and poll _entire_ chain when active.

And I find it mind boggling that IRQ's other than Timer and RTC are
edge triggered which also explains the troubles with sharing IRQ's.

It would be better to just program them level.

I good rule is:

An IRQ should be edge triggered when the ISR has no influence on the
state of the IRQ line. This is usually the case with periodic signals
such as timer outputs or strobes by rotary encoders and the like.

All other IRQs - in particular those "reset" by a software generated
events such as reading or writing a register - should be level triggered.

Regards
Michael

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <mpm@selenic.com> wrote:

> On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
>> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no]
>> > Grover, Andrew wrote:
>> > > Is the assumption that hardirq handlers are superfast also
>> > the reason
>> > > why Linux calls all handlers on a shared interrupt, even if
>> > the first
>> > > handler reports it was for its device?
>> > >
>> > No, it is the other way around.  hardirq handlers have to be superfast
>> > because linux usually _have to_ call all the handlers of a shared irq.
>> >
>> > The fact that one device did indeed have an interrupt for us
>> > doesn't mean
>> > that the others didn't.  So all of them have to be checked to be safe.
>>
>> If a device later in the handler chain is also interrupting, then the
>> interrupt will immediately trigger again. The irq line will remain
>> asserted until nobody is asserting it.
>>
>> If the LAST guy in the chain is the one with the interrupt, then you
>> basically get today's ISR "call each handler" behavior, but it should be
>> possible to in some cases to get less time spent in do_IRQ.
>
> Let's imagine you have n sources simultaneously interrupting on a
> given descriptor. Check the first, it's happening, acknowledge it,
> exit, notice interrupt still asserted, check the first, nope, check
> the second, yep, exit, etc. By the time we've made it to the nth ISR,
> we've banged on the first one n times, the second n-1 times, etc. In
> other words, early chain termination has an O(n^2) worst case.
>

With level triggered you can just walk the chain, exit at the end of the
first cycle and  should the IRQ still be asserted you just incur the
overhead of exit and reentry of the ISR.

Even with edge, I would not check alwasy from the beginning of the chain...

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 13:42:12 -0500 (EST), Richard B. Johnson <root@chaos.analogic.com> wrote:

>
> In the early IBM/AT, there was a port to which a user of
> a shared "edge" interrupt could write. If the interrupt
> line was still asserted, this would generate another edge.
>
> This meant that any ISR needed to know about other users
> of the same interrupt. This is probably why it didn't
> catch on.

Oops, never seen that in the circuit diagrams. Was that an internal prototype?

Regards
Michael

Ooops2 sorry for priv msg, time to have a nap....

Re: Why no interrupt priorities?

[Matt Mackall]

On Fri, Feb 27, 2004 at 11:09:14AM -0800, Tim Hockin wrote:
> On Fri, Feb 27, 2004 at 12:55:55PM -0600, Matt Mackall wrote:
> > Let's imagine you have n sources simultaneously interrupting on a
> > given descriptor. Check the first, it's happening, acknowledge it,
> > exit, notice interrupt still asserted, check the first, nope, check
> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
> > we've banged on the first one n times, the second n-1 times, etc. In
> > other words, early chain termination has an O(n^2) worst case.
> 
> That is a pretty pathological worst case, and n is (almost?) always small.
> I don't know if it would make a lick of difference, or if it is worth the
> risk. Someone who has a lot of shared interrupts ought to try it.

For small n, it shouldn't make a difference. With early exit, best
case you end up walking half the chain on average, worst case you hit
the quadratic behavior. Given that the likelihood of contention rises
with chain length, we tend to lean towards the latter for larger n.
For try-them-all, we test n in all cases. So we've got [n/2] < n <
n^2/2. The try-them-all approach wins by virtue of being deterministic
and nicely bounded.

Oh, another concern is that early-exit lets sources in the front of
the chain starve the remainder and doing bookkeeping on
last-ISR-succeeded so that we can have some sort of fairness is just
not worth the trouble. So nix on the whole idea.

-- 
Matt Mackall : http://www.selenic.com : Linux development and consulting

Re: Why no interrupt priorities?

[Jesse Pollard]

On Friday 27 February 2004 13:19, Michael Frank wrote:
> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <mpm@selenic.com> wrote:
> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
> >> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no]
> >> >
> >> > Grover, Andrew wrote:
> >> > > Is the assumption that hardirq handlers are superfast also
> >> >
> >> > the reason
> >> >
> >> > > why Linux calls all handlers on a shared interrupt, even if
> >> >
> >> > the first
> >> >
> >> > > handler reports it was for its device?
> >> >
> >> > No, it is the other way around.  hardirq handlers have to be superfast
> >> > because linux usually _have to_ call all the handlers of a shared irq.
> >> >
> >> > The fact that one device did indeed have an interrupt for us
> >> > doesn't mean
> >> > that the others didn't.  So all of them have to be checked to be safe.
> >>
> >> If a device later in the handler chain is also interrupting, then the
> >> interrupt will immediately trigger again. The irq line will remain
> >> asserted until nobody is asserting it.
> >>
> >> If the LAST guy in the chain is the one with the interrupt, then you
> >> basically get today's ISR "call each handler" behavior, but it should be
> >> possible to in some cases to get less time spent in do_IRQ.
> >
> > Let's imagine you have n sources simultaneously interrupting on a
> > given descriptor. Check the first, it's happening, acknowledge it,
> > exit, notice interrupt still asserted, check the first, nope, check
> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
> > we've banged on the first one n times, the second n-1 times, etc. In
> > other words, early chain termination has an O(n^2) worst case.
>
> With level triggered you can just walk the chain, exit at the end of the
> first cycle and  should the IRQ still be asserted you just incur the
> overhead of exit and reentry of the ISR.
>
> Even with edge, I would not check alwasy from the beginning of the chain...

You should... after all that first entry in the chain has the highest priority

Re: Why no interrupt priorities?

[Michael Frank]

On Thu, 26 Feb 2004 19:02:59 -0800, Randy.Dunlap <rddunlap@osdl.org> wrote:

> On Thu, 26 Feb 2004 17:36:34 -0800 "Grover, Andrew" <andrew.grover@intel.com> wrote:
>
> | > On Thursday 26 February 2004 13:30, Arjan van de Ven wrote:
> | > > hardware IRQ priorities are useless for the linux model. In
> | > linux, the
> | > > hardirq runs *very* briefly and then lets the softirq context do the
> | > > longer taking work. hardware irq priorities then don't matter really
> | > > because the hardirq's are hardly ever interrupted really,
> | > and when they
> | > > are they cause a performance *loss* due to cache trashing.
> | > The latency
> | > > added by waiting briefly is going to be really really short
> | > for any sane
> | > > hardware.
> |
> | Is the assumption that hardirq handlers are superfast also the reason
> | why Linux calls all handlers on a shared interrupt, even if the first
> | handler reports it was for its device?
>
> Somehow I don't think that's the reasoning.
>
> Is there a safe method to determine that there are no other pending
> interrupts on one shared interrupt?  i.e., that other devices don't
> also have interrupts pending?
>

Most interrupt controllers can read back IRQ's to see whether it is
active. A shared IRQ would be readback active while any device
connected to it desires service.

x86 example for 8259A AT-PIC's Returns the state of IRQ0-15 in ax
Note that jmp $+2 is only needed on some old 286/386 hardware
to meet (real) 8259A cycle time requirements.

- Intel syntax :)

	mov	al,0ah
	out	0a0h,al
	jmp	$+2
	in	al,0a0h
	mov	ah,al
	mov 	al,0ah
	jmp	$+2
	out 	20h,al
	jmp	$+2
	in 	al,20h

Regards
Michael

Re: Why no interrupt priorities?

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Sun, Feb 29, 2004 at 04:32:54PM +0800, Michael Frank wrote:
> 
> Most interrupt controllers can read back IRQ's to see whether it is
> active. A shared IRQ would be readback active while any device
> connected to it desires service.
> 
> x86 example for 8259A AT-PIC's Returns the state of IRQ0-15 in ax
> Note that jmp $+2 is only needed on some old 286/386 hardware
> to meet (real) 8259A cycle time requirements.
> 
> - Intel syntax :)
> 
> 	mov	al,0ah
> 	out	0a0h,al
> 	jmp	$+2
> 	in	al,0a0h
> 	mov	ah,al
> 	mov 	al,0ah
> 	jmp	$+2
> 	out 	20h,al
> 	jmp	$+2
> 	in 	al,20h

interesting; however with modern cpus I suspect that a series of in/outs
like that is more expensive than one or two "surious" hardirq handler
calls...


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Why no interrupt priorities?

[Michael Frank]

On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard <jesse@cats-chateau.net> wrote:

> On Friday 27 February 2004 13:19, Michael Frank wrote:
>> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <mpm@selenic.com> wrote:
>> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
>> >> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no]
>> >> >
>> >> > Grover, Andrew wrote:
>> >> > > Is the assumption that hardirq handlers are superfast also
>> >> >
>> >> > the reason
>> >> >
>> >> > > why Linux calls all handlers on a shared interrupt, even if
>> >> >
>> >> > the first
>> >> >
>> >> > > handler reports it was for its device?
>> >> >
>> >> > No, it is the other way around.  hardirq handlers have to be superfast
>> >> > because linux usually _have to_ call all the handlers of a shared irq.
>> >> >
>> >> > The fact that one device did indeed have an interrupt for us
>> >> > doesn't mean
>> >> > that the others didn't.  So all of them have to be checked to be safe.
>> >>
>> >> If a device later in the handler chain is also interrupting, then the
>> >> interrupt will immediately trigger again. The irq line will remain
>> >> asserted until nobody is asserting it.
>> >>
>> >> If the LAST guy in the chain is the one with the interrupt, then you
>> >> basically get today's ISR "call each handler" behavior, but it should be
>> >> possible to in some cases to get less time spent in do_IRQ.
>> >
>> > Let's imagine you have n sources simultaneously interrupting on a
>> > given descriptor. Check the first, it's happening, acknowledge it,
>> > exit, notice interrupt still asserted, check the first, nope, check
>> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
>> > we've banged on the first one n times, the second n-1 times, etc. In
>> > other words, early chain termination has an O(n^2) worst case.
>>
>> With level triggered you can just walk the chain, exit at the end of the
>> first cycle and  should the IRQ still be asserted you just incur the
>> overhead of exit and reentry of the ISR.
>>
>> Even with edge, I would not check alwasy from the beginning of the chain...
>
> You should... after all that first entry in the chain has the highest priority
>

Please also consider that physcial IRQ's are in practice assigned "semi randomly".

In a way IRQ priorities in general purpose computing applications are irrelevant :)

Lets say you have a bunch of devices demand service, all have to be serviced but
is either not significant which gets done first or the practival IRQ priorities
are "less than optimal":

Keyboard	IRQ1		Well buffered
NIC1		IRQ10		Buffered
NIC2		IRQ10		Buffered
USB		IRQ11		Buffered
serial port	IRQ3		Small FIFO (assuming '550)
serial port	IRQ4			"-"

Here, serial ports would would be most critical, however the priorities of IRQ3
and IRQ4 are below priorities of IRQ10 and IRQ11...

IMO, the best practical approach is to keep efficiency by walking the chain only
once. If the chain is longer, it may be  worthwhile to check the IRQ and exit
walking the chain if it is inactive.

Priorities in chains would make sense only in specialized applications under
controlled circumstances wrt IRQ and linking devices into chain at the priority
desired.

Regards
Michael

Re: Why no interrupt priorities?

[Michael Frank]

On Sun, 29 Feb 2004 09:36:57 +0100, Arjan van de Ven <arjanv@redhat.com> wrote:

> On Sun, Feb 29, 2004 at 04:32:54PM +0800, Michael Frank wrote:
>>
>> Most interrupt controllers can read back IRQ's to see whether it is
>> active. A shared IRQ would be readback active while any device
>> connected to it desires service.
>>
>> x86 example for 8259A AT-PIC's Returns the state of IRQ0-15 in ax
>> Note that jmp $+2 is only needed on some old 286/386 hardware
>> to meet (real) 8259A cycle time requirements.
>>
>> - Intel syntax :)
>>
>> 	mov	al,0ah
>> 	out	0a0h,al
>> 	jmp	$+2
>> 	in	al,0a0h
>> 	mov	ah,al
>> 	mov 	al,0ah
>> 	jmp	$+2
>> 	out 	20h,al
>> 	jmp	$+2
>> 	in 	al,20h
>
> interesting; however with modern cpus I suspect that a series of in/outs
> like that is more expensive than one or two "surious" hardirq handler
> calls...
>

Yes, Four 8259A IO cycles would take almost 2us, which is several 1000
instructions worth of burning electricity.

Racehorse is still best at going straight :)

However on-chipset PIC's may be better and in on-CPU APICs should be
much better in this regard, but I have not studied data(sheet).

Regards
Michael

Re: Why no interrupt priorities?

[Jesse Pollard]

On Sunday 29 February 2004 03:43, Michael Frank wrote:
> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard <jesse@cats-chateau.net> 
wrote:
> > On Friday 27 February 2004 13:19, Michael Frank wrote:
> >> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <mpm@selenic.com> wrote:
> >> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
> >> >> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no]
> >> >> >
> >> >> > Grover, Andrew wrote:
> >> >> > > Is the assumption that hardirq handlers are superfast also
> >> >> >
> >> >> > the reason
> >> >> >
> >> >> > > why Linux calls all handlers on a shared interrupt, even if
> >> >> >
> >> >> > the first
> >> >> >
> >> >> > > handler reports it was for its device?
> >> >> >
> >> >> > No, it is the other way around.  hardirq handlers have to be
> >> >> > superfast because linux usually _have to_ call all the handlers of
> >> >> > a shared irq.
> >> >> >
> >> >> > The fact that one device did indeed have an interrupt for us
> >> >> > doesn't mean
> >> >> > that the others didn't.  So all of them have to be checked to be
> >> >> > safe.
> >> >>
> >> >> If a device later in the handler chain is also interrupting, then the
> >> >> interrupt will immediately trigger again. The irq line will remain
> >> >> asserted until nobody is asserting it.
> >> >>
> >> >> If the LAST guy in the chain is the one with the interrupt, then you
> >> >> basically get today's ISR "call each handler" behavior, but it should
> >> >> be possible to in some cases to get less time spent in do_IRQ.
> >> >
> >> > Let's imagine you have n sources simultaneously interrupting on a
> >> > given descriptor. Check the first, it's happening, acknowledge it,
> >> > exit, notice interrupt still asserted, check the first, nope, check
> >> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
> >> > we've banged on the first one n times, the second n-1 times, etc. In
> >> > other words, early chain termination has an O(n^2) worst case.
> >>
> >> With level triggered you can just walk the chain, exit at the end of the
> >> first cycle and  should the IRQ still be asserted you just incur the
> >> overhead of exit and reentry of the ISR.
> >>
> >> Even with edge, I would not check alwasy from the beginning of the
> >> chain...
> >
> > You should... after all that first entry in the chain has the highest
> > priority
>
> Please also consider that physcial IRQ's are in practice assigned "semi
> randomly".

I think that would be up to the installer.

> In a way IRQ priorities in general purpose computing applications are
> irrelevant :)

I would say that they have a priority, just that the priority isn't being
used.

> Lets say you have a bunch of devices demand service, all have to be
> serviced but is either not significant which gets done first or the
> practival IRQ priorities are "less than optimal":
>
> Keyboard	IRQ1		Well buffered
> NIC1		IRQ10		Buffered
> NIC2		IRQ10		Buffered
> USB		IRQ11		Buffered
> serial port	IRQ3		Small FIFO (assuming '550)
> serial port	IRQ4			"-"
>
> Here, serial ports would would be most critical, however the priorities of
> IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...

I don't think these are all using "shared IRQ...". I was my understanding that
each IRQ had its own chain. And the priority of the drivers in that chain
is determined by the order.

> IMO, the best practical approach is to keep efficiency by walking the chain
> only once. If the chain is longer, it may be  worthwhile to check the IRQ
> and exit walking the chain if it is inactive.

That depends entirely on the chain. If there is one or more high speed devices
such that walking the chain is faster that responding to an interrupt, then it
is DEFINITELY worth it to walk the chain.

> Priorities in chains would make sense only in specialized applications
> under controlled circumstances wrt IRQ and linking devices into chain at
> the priority desired.

No necessarily specialized applications - just ones that should/must not
loose data due to starvation. Consider the problems caused by relatively
slow devices such as a mouse. Resyncronization is a real problem, which is
one of the reasons to give it a nonshared IRQ. The same could be said of
a PPP connection over a serial line...

If they did share an interrupt, you make the PPP serial line higher priority
than the mouse... Yet you still don't want to loose mouse syncronization - so
check it anyway. Handling it is still faster than an additional interrupt,
and with less overhead.

Granted, this is more important on slower hardware or basic interactive 
service, but it all contributes to overhead. Properly organized (and used)
chains will reduce the overhead.

Re: Why no interrupt priorities?

[Michael Frank]

On Mon, 1 Mar 2004 10:57:42 -0600, Jesse Pollard <jesse@cats-chateau.net> wrote:

> On Sunday 29 February 2004 03:43, Michael Frank wrote:
>> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard <jesse@cats-chateau.net>
> wrote:
>> > On Friday 27 February 2004 13:19, Michael Frank wrote:
>> >> On Fri, 27 Feb 2004 12:55:55 -0600, Matt Mackall <mpm@selenic.com> wrote:
>> >> > On Fri, Feb 27, 2004 at 09:44:44AM -0800, Grover, Andrew wrote:
>> >> >> > From: Helge Hafting [mailto:helgehaf@aitel.hist.no]
>> >> >> >
>> >> >> > Grover, Andrew wrote:
>> >> >> > > Is the assumption that hardirq handlers are superfast also
>> >> >> >
>> >> >> > the reason
>> >> >> >
>> >> >> > > why Linux calls all handlers on a shared interrupt, even if
>> >> >> >
>> >> >> > the first
>> >> >> >
>> >> >> > > handler reports it was for its device?
>> >> >> >
>> >> >> > No, it is the other way around.  hardirq handlers have to be
>> >> >> > superfast because linux usually _have to_ call all the handlers of
>> >> >> > a shared irq.
>> >> >> >
>> >> >> > The fact that one device did indeed have an interrupt for us
>> >> >> > doesn't mean
>> >> >> > that the others didn't.  So all of them have to be checked to be
>> >> >> > safe.
>> >> >>
>> >> >> If a device later in the handler chain is also interrupting, then the
>> >> >> interrupt will immediately trigger again. The irq line will remain
>> >> >> asserted until nobody is asserting it.
>> >> >>
>> >> >> If the LAST guy in the chain is the one with the interrupt, then you
>> >> >> basically get today's ISR "call each handler" behavior, but it should
>> >> >> be possible to in some cases to get less time spent in do_IRQ.
>> >> >
>> >> > Let's imagine you have n sources simultaneously interrupting on a
>> >> > given descriptor. Check the first, it's happening, acknowledge it,
>> >> > exit, notice interrupt still asserted, check the first, nope, check
>> >> > the second, yep, exit, etc. By the time we've made it to the nth ISR,
>> >> > we've banged on the first one n times, the second n-1 times, etc. In
>> >> > other words, early chain termination has an O(n^2) worst case.
>> >>
>> >> With level triggered you can just walk the chain, exit at the end of the
>> >> first cycle and  should the IRQ still be asserted you just incur the
>> >> overhead of exit and reentry of the ISR.
>> >>
>> >> Even with edge, I would not check alwasy from the beginning of the
>> >> chain...
>> >
>> > You should... after all that first entry in the chain has the highest
>> > priority
>>
>> Please also consider that physcial IRQ's are in practice assigned "semi
>> randomly".
>
> I think that would be up to the installer.

How?

>
>> In a way IRQ priorities in general purpose computing applications are
>> irrelevant :)
>
> I would say that they have a priority, just that the priority isn't being
> used.

Not usable in 2 dimensions - A) the IRQ can't be controlled, B) the location
in the chain can't be contrlolled.

>
>> Lets say you have a bunch of devices demand service, all have to be
>> serviced but is either not significant which gets done first or the
>> practival IRQ priorities are "less than optimal":
>>
>> Keyboard	IRQ1		Well buffered

>> NIC1		IRQ10		Buffered
>> NIC2		IRQ10		Buffered
>> USB		IRQ11		Buffered
    MOUSE          IRQ12       ?
>> serial port	IRQ3		Small FIFO (assuming '550)
>> serial port	IRQ4			"-"
>>
>> Here, serial ports would would be most critical, however the priorities of
>> IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...
>
> I don't think these are all using "shared IRQ...". I was my understanding that
> each IRQ had its own chain. And the priority of the drivers in that chain
> is determined by the order.

The NICs's are both shared on IRQ10.

>
>> IMO, the best practical approach is to keep efficiency by walking the chain
>> only once. If the chain is longer, it may be  worthwhile to check the IRQ
>> and exit walking the chain if it is inactive.
>
> That depends entirely on the chain. If there is one or more high speed devices
> such that walking the chain is faster that responding to an interrupt, then it
> is DEFINITELY worth it to walk the chain.
>
>> Priorities in chains would make sense only in specialized applications
>> under controlled circumstances wrt IRQ and linking devices into chain at
>> the priority desired.
>
> No necessarily specialized applications - just ones that should/must not
> loose data due to starvation. Consider the problems caused by relatively
> slow devices such as a mouse. Resyncronization is a real problem, which is
> one of the reasons to give it a nonshared IRQ. The same could be said of
> a PPP connection over a serial line...
>
> If they did share an interrupt, you make the PPP serial line higher priority
> than the mouse... Yet you still don't want to loose mouse syncronization - so
> check it anyway. Handling it is still faster than an additional interrupt,
> and with less overhead.

Right, the mouse is important too. A major practical system level issue
is shown in above example:  The PPP serial line could be easily starved by
those 2 NIC's+USB+Mouse with higher priority.

Baudrate 115,200 / 11bit is about 10000 chars/s peak. Given a FIFO size
of 16 + 1 RxHR (best case), after 1ms-1.5ms characters get lost.

A fix would require reprogram PIC IRQ priorities here to make IRQ3 and IRQ4
higher priority than what comes in from the cascaded controller (via IRQ2).

Obviously more APIC's abound but there still is need for a facility to
manage priorities, and this (primary priority) is more significant than
what happens within a chain.

>
> Granted, this is more important on slower hardware or basic interactive
> service, but it all contributes to overhead. Properly organized (and used)
> chains will reduce the overhead.
>

More control over physical IRQ priority would be an initial step helping
with non-shared legacy hardware and device with small buffers.

More modern hardware like USB is using large buffers, DMA and is less critical.

And perhaps a config-bit per chain to tell it whether to rescan from scratch
would be useful to address the concerns mentioned, _and_ then there should
be a facility to link a interrupt into a specific location in a chain.

Regards
Michael

Re: Why no interrupt priorities?

[Jesse Pollard]

On Monday 01 March 2004 11:35, Michael Frank wrote:
> On Mon, 1 Mar 2004 10:57:42 -0600, Jesse Pollard <jesse@cats-chateau.net> 
wrote:
> > On Sunday 29 February 2004 03:43, Michael Frank wrote:
> >> On Fri, 27 Feb 2004 14:53:45 -0600, Jesse Pollard
> >> <jesse@cats-chateau.net>
> >
> > wrote:
[snip]
> >> >
> >> > You should... after all that first entry in the chain has the highest
> >> > priority
> >>
> >> Please also consider that physcial IRQ's are in practice assigned "semi
> >> randomly".
> >
> > I think that would be up to the installer.
>
> How?

Usually (at least in my boxes, anyway) the IRQ tends to be associated with the
PCI slot number. suitable ordering of the interfaces in the slots was what I 
was thinking of. But then, I don't overload the IRQs either.

>
> >> In a way IRQ priorities in general purpose computing applications are
> >> irrelevant :)
> >
> > I would say that they have a priority, just that the priority isn't being
> > used.
>
> Not usable in 2 dimensions - A) the IRQ can't be controlled, B) the
> location in the chain can't be contrlolled.

The location in the chain should be controled - even if it is nothing
other than a driver based priority number used to insert in the chain
at the appropriate location.

> >> Lets say you have a bunch of devices demand service, all have to be
> >> serviced but is either not significant which gets done first or the
> >> practival IRQ priorities are "less than optimal":
> >>
> >> Keyboard	IRQ1		Well buffered
> >>
> >> NIC1		IRQ10		Buffered
> >> NIC2		IRQ10		Buffered
> >> USB		IRQ11		Buffered
>
>     MOUSE          IRQ12       ?
>
> >> serial port	IRQ3		Small FIFO (assuming '550)
> >> serial port	IRQ4			"-"
> >>
> >> Here, serial ports would would be most critical, however the priorities
> >> of IRQ3 and IRQ4 are below priorities of IRQ10 and IRQ11...
> >
> > I don't think these are all using "shared IRQ...". I was my understanding
> > that each IRQ had its own chain. And the priority of the drivers in that
> > chain is determined by the order.
>
> The NICs's are both shared on IRQ10.
>
[snip]
> >
> > If they did share an interrupt, you make the PPP serial line higher
> > priority than the mouse... Yet you still don't want to loose mouse
> > syncronization - so check it anyway. Handling it is still faster than an
> > additional interrupt, and with less overhead.
>
> Right, the mouse is important too. A major practical system level issue
> is shown in above example:  The PPP serial line could be easily starved by
> those 2 NIC's+USB+Mouse with higher priority.
>
> Baudrate 115,200 / 11bit is about 10000 chars/s peak. Given a FIFO size
> of 16 + 1 RxHR (best case), after 1ms-1.5ms characters get lost.
>
> A fix would require reprogram PIC IRQ priorities here to make IRQ3 and IRQ4
> higher priority than what comes in from the cascaded controller (via IRQ2).
>
> Obviously more APIC's abound but there still is need for a facility to
> manage priorities, and this (primary priority) is more significant than
> what happens within a chain.

Yes.

>
> > Granted, this is more important on slower hardware or basic interactive
> > service, but it all contributes to overhead. Properly organized (and
> > used) chains will reduce the overhead.
>
> More control over physical IRQ priority would be an initial step helping
> with non-shared legacy hardware and device with small buffers.
>
> More modern hardware like USB is using large buffers, DMA and is less
> critical.
>
> And perhaps a config-bit per chain to tell it whether to rescan from
> scratch would be useful to address the concerns mentioned, _and_ then there
> should be a facility to link a interrupt into a specific location in a
> chain.

A round-robin IRQ scheduler would also help those at the same priority. Again,
software implementation sucks (I used to do embeded systems).

It really looks like the old arch is being pushed way beyond what is 
reasonable. It is really time for vectored interrupts to resurface (re: the 
old PDP 11 style, allowing each interrupt to have it's own software priority,
AND interrupt routine... which also happens to be where "spl" started from :-)

It really looks like the PCI can't handle more than about 3 interrupt sources
at a time, in spite of the APIC/cascade faking it to look like more.

The busyist box I have (a server) has two SCSI controllers and two IDE (one 
IRQ each, and the IDE is used only for primary boot, and backup), and an 
ethernet. The PS2 mouse and keyboard don't get used (much). The second 
busyist has three ethernet connections (1 from DSL, 1 from wireless, 1 from
the internal net - it is a firewall) and an IDE boot. Again, the
keyboard/mouse are not used.