Interrupt prioritization on linux for ppc440

Interrupt prioritization on linux for ppc440

[Shawn Jin]

Hi,

The home-made interrupt controller PIC supports interrupt priorities
and critical/non-critical interrupts. I found that the current kernel
doesn't support interrupt priorities. Is this observation true? Is
there any existing patch to support that? Can RTAI or RTLinux
prioritize interrupts?

I noticed that the implementation of ppc4xx_pic.c disables all
critical interrupts during initialization. To support critical
interrupts, is it so simple that we change the handler of critical
exception from CriticalInput to do_IRQ in head_44x.S?

Would you please share some of your inputs with me while I'm googling
this issue?

Thanks,
-Shawn.

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Fri, Apr 15, 2005 at 02:36:51PM -0700, Shawn Jin wrote:
> 
> The home-made interrupt controller PIC supports interrupt priorities
> and critical/non-critical interrupts. I found that the current kernel
> doesn't support interrupt priorities. 

Yes, we don't have IRQ priorities on 4xx. Theoretically, they can be 
emulated in get_irq, but I really don't think it's worth it.

> Is this observation true? Is
> there any existing patch to support that? 

I'm not aware of such patch existence.

> I noticed that the implementation of ppc4xx_pic.c disables all
> critical interrupts during initialization. To support critical
> interrupts, is it so simple that we change the handler of critical
> exception from CriticalInput to do_IRQ in head_44x.S?

No, it's not that simple. Linux doesn't support a notion of critical 
IRQs versus normal ones. Until there is an infrastructure for this, it 
doesn't make any sense to implement 4xx support.

--
Eugene

Re: Interrupt prioritization on linux for ppc440

[Shawn Jin]

On 4/15/05, Eugene Surovegin <ebs@ebshome.net> wrote:
> On Fri, Apr 15, 2005 at 02:36:51PM -0700, Shawn Jin wrote:
> >
> > The home-made interrupt controller PIC supports interrupt priorities
> > and critical/non-critical interrupts. I found that the current kernel
> > doesn't support interrupt priorities.
>=20
> Yes, we don't have IRQ priorities on 4xx. Theoretically, they can be
> emulated in get_irq, but I really don't think it's worth it.

Hmmm...that's one way I thought of to implement IRQ priority. Why
isn't it worth it? ppc4xx_pic gets the first irq from the least
significant bit by calling ffs(). So theoretically and maybe
practically some external interrupts will keep UART's interrupt from
being served.

> > Is this observation true? Is
> > there any existing patch to support that?
>=20
> I'm not aware of such patch existence.

By googling the Internet I found this patch for i386 architecture
http://home.t-online.de/home/Bernhard_Kuhn/rtirq/20040304/rtirq.html.
This mustn't have been caught sight of by linux mainstream.

Anybody knows if RTAI or RTLinux supports IRQ priorities?

> > I noticed that the implementation of ppc4xx_pic.c disables all
> > critical interrupts during initialization. To support critical
> > interrupts, is it so simple that we change the handler of critical
> > exception from CriticalInput to do_IRQ in head_44x.S?
>=20
> No, it's not that simple. Linux doesn't support a notion of critical
> IRQs versus normal ones. Until there is an infrastructure for this, it
> doesn't make any sense to implement 4xx support.

What kind of infrastructure can you think of to support this? ppc440
at least provides some interrupt processing registers (CSSR0/1) to
differentiate critical and non-critical interrupts.

Another confusion about 440GP/GX UIC is the registers VCR (Vector
Configuration Reg) and VR (Vector Reg). They seem to be totally
useless on linux. The interrupt vector is already in the table of
irq_desc[]. Why does the controller have to generate the address?

Thanks for the inputs. Welcome more.

Regards,
-Shawn.

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Fri, Apr 15, 2005 at 04:09:08PM -0700, Shawn Jin wrote:
> On 4/15/05, Eugene Surovegin <ebs@ebshome.net> wrote:
> > On Fri, Apr 15, 2005 at 02:36:51PM -0700, Shawn Jin wrote:
> > >
> > > The home-made interrupt controller PIC supports interrupt priorities
> > > and critical/non-critical interrupts. I found that the current kernel
> > > doesn't support interrupt priorities.
> > 
> > Yes, we don't have IRQ priorities on 4xx. Theoretically, they can be
> > emulated in get_irq, but I really don't think it's worth it.
> 
> Hmmm...that's one way I thought of to implement IRQ priority. Why
> isn't it worth it? 

It will significantly complicate PIC code, so if we want to go down 
this road, there should be some _real_ reasons and examples, that such 
change will be beneficial. For all the time I have been working with 
4xx based designs, I never needed such functionality.

> ppc4xx_pic gets the first irq from the least
> significant bit by calling ffs(). So theoretically and maybe
> practically some external interrupts will keep UART's interrupt from
> being served.

Yeah, _theoretically_ this is possible, although, in this case I think 
UART IRQ starvation will be your least problem :).

> > > I noticed that the implementation of ppc4xx_pic.c disables all
> > > critical interrupts during initialization. To support critical
> > > interrupts, is it so simple that we change the handler of critical
> > > exception from CriticalInput to do_IRQ in head_44x.S?
> > 
> > No, it's not that simple. Linux doesn't support a notion of critical
> > IRQs versus normal ones. Until there is an infrastructure for this, it
> > doesn't make any sense to implement 4xx support.
> 
> What kind of infrastructure can you think of to support this? ppc440
> at least provides some interrupt processing registers (CSSR0/1) to
> differentiate critical and non-critical interrupts.

OK, think about local_irq_disable/enable, and all code which is 
affected by this. Should these function disable all IRQs or just 
normal ones? If just normal one, what will happen if your critical IRQ 
handler calls the code, which assumes it's non-reentrant, because it 
called local_irq_disable()? Etc..

I really think that if you need some special higher-priority IRQ 
context you'd better use RTAI or RTLinux. At least, they make clear 
(IIRC) that these IRQ contexts are special.

> 
> Another confusion about 440GP/GX UIC is the registers VCR (Vector
> Configuration Reg) and VR (Vector Reg). They seem to be totally
> useless on linux. The interrupt vector is already in the table of
> irq_desc[]. Why does the controller have to generate the address?

It's being generated only for critical IRQs, so yes, because we don't 
use them, it's completely useless for us.

What could be interesting, though, is to to make all 4xx IRQs 
critical, in this case we could use VR to quickly determine which IRQ 
was asserted, instead of current implementation when we use bit 
operations. Not sure, if performance gain is really worth the 
effort :)

-- 
Eugene

Re: Interrupt prioritization on linux for ppc440

[Lawrence E. Bakst]

I just want to say that in certain applications critical interrupts can be put to good use. There are issues you have to deal with when you use them. I wonder if any of the people saying that they "really don't think it's worth it" have ever had to meet any real time interrupt latency constraints?

Best,

leb


At 4:09 PM -0700 4/15/05, Shawn Jin wrote:
>On 4/15/05, Eugene Surovegin <ebs@ebshome.net> wrote:
>> On Fri, Apr 15, 2005 at 02:36:51PM -0700, Shawn Jin wrote:
>> >
>> > The home-made interrupt controller PIC supports interrupt priorities
>> > and critical/non-critical interrupts. I found that the current kernel
>> > doesn't support interrupt priorities.
>>
>> Yes, we don't have IRQ priorities on 4xx. Theoretically, they can be
> > emulated in get_irq, but I really don't think it's worth it.
>
>Hmmm...that's one way I thought of to implement IRQ priority. Why
>isn't it worth it? ppc4xx_pic gets the first irq from the least
>significant bit by calling ffs(). So theoretically and maybe
>practically some external interrupts will keep UART's interrupt from
>being served.
>
>> > Is this observation true? Is
>> > there any existing patch to support that?
>>
>> I'm not aware of such patch existence.
>
>By googling the Internet I found this patch for i386 architecture
>http://home.t-online.de/home/Bernhard_Kuhn/rtirq/20040304/rtirq.html.
>This mustn't have been caught sight of by linux mainstream.
>
>Anybody knows if RTAI or RTLinux supports IRQ priorities?
>
>> > I noticed that the implementation of ppc4xx_pic.c disables all
>> > critical interrupts during initialization. To support critical
>> > interrupts, is it so simple that we change the handler of critical
>> > exception from CriticalInput to do_IRQ in head_44x.S?
>>
>> No, it's not that simple. Linux doesn't support a notion of critical
>> IRQs versus normal ones. Until there is an infrastructure for this, it
>> doesn't make any sense to implement 4xx support.
>
>What kind of infrastructure can you think of to support this? ppc440
>at least provides some interrupt processing registers (CSSR0/1) to
>differentiate critical and non-critical interrupts.
>
>Another confusion about 440GP/GX UIC is the registers VCR (Vector
>Configuration Reg) and VR (Vector Reg). They seem to be totally
>useless on linux. The interrupt vector is already in the table of
>irq_desc[]. Why does the controller have to generate the address?
>
>Thanks for the inputs. Welcome more.
>
>Regards,
>-Shawn.
>_______________________________________________
>Linuxppc-embedded mailing list
>Linuxppc-embedded@ozlabs.org
>https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Mon, Apr 18, 2005 at 10:01:02AM -0700, Lawrence E. Bakst wrote:
> I just want to say that in certain applications critical interrupts 
> can be put to good use. There are issues you have to deal with when 
> you use them. I wonder if any of the people saying that they "really 
> don't think it's worth it" have ever had to meet any real time 
> interrupt latency constraints?

You know, it'd help if instead of some general statements, you gave 
real example and explained, why default stuff doesn't work for you.

Also, it'd be nice if you explained how you dealt with infrastructure 
problems I mentioned in my previous e-mail.

Answering your question about "real time interrupt latency 
constraints", I have to ask what _exact_ latencies you need, and why 
do you think Linux can provide hard real-time guarantees? 

And yes, I work for company which makes VoIP hardware, and soft 
real-time qualities of 2.4 Linux kernel (preempt, low-lat) are good 
enough for us (~1-5 ms average scheduling latency), without any need 
of critical interrupts.

-- 
Eugene

Re: Interrupt prioritization on linux for ppc440

[Lawrence E. Bakst]

At 10:25 AM -0700 4/18/05, Eugene Surovegin wrote:
>On Mon, Apr 18, 2005 at 10:01:02AM -0700, Lawrence E. Bakst wrote:
>> I just want to say that in certain applications critical interrupts
>> can be put to good use. There are issues you have to deal with when
>> you use them. I wonder if any of the people saying that they "really
>> don't think it's worth it" have ever had to meet any real time
>> interrupt latency constraints?
>
>You know, it'd help if instead of some general statements, you gave
>real example and explained, why default stuff doesn't work for you.

I'd point out that your own statements were pretty general to start off with. I would have liked to elaborate but unfortunately the work is an ongoing research effort for a client under NDA so I can't say very much.

>Also, it'd be nice if you explained how you dealt with infrastructure
>problems I mentioned in my previous e-mail.

We didn't deal with them as we have a kind of special case. I am not trying to minimize the importance of that issue in the general case.

>
>Answering your question about "real time interrupt latency
>constraints", I have to ask what _exact_ latencies you need, and why
>do you think Linux can provide hard real-time guarantees?

We are trying to get to an interrupt response time of substantially less than what you mention below. In this case "substantially" means between 2 and 3 decimal orders of magnitude.

>
>And yes, I work for company which makes VoIP hardware, and soft
>real-time qualities of 2.4 Linux kernel (preempt, low-lat) are good
>enough for us (~1-5 ms average scheduling latency), without any need
>of critical interrupts.

We're operating in different environments and doing different things. With the kind of response time we're trying to achieve we can't afford to schedule anything.

The only point I was trying to make is that the critical interrupt facility was useful to us. That was it. It's not clear to me that within the constraints of how Linux works, the various infrastructure issues, as well as politics, that it could be made useful to others in a generalized form.

Best,

leb


>
>--
>Eugene

Re: Interrupt prioritization on linux for ppc440

[Shawn Jin]

On 4/15/05, Eugene Surovegin <ebs@ebshome.net> wrote:
> I really think that if you need some special higher-priority IRQ
> context you'd better use RTAI or RTLinux. At least, they make clear
> (IIRC) that these IRQ contexts are special.

OK, I'll look into RTAI or RTLinux for priority issue.

> What could be interesting, though, is to to make all 4xx IRQs
> critical, in this case we could use VR to quickly determine which IRQ
> was asserted, instead of current implementation when we use bit
> operations. Not sure, if performance gain is really worth the
> effort :)

If we use VR to determine which IRQ was asserted, it's kind of
reverse. We usually fetch a handler by its IRQ number. It could
require to change irq_desc[] and request_irq().

VR can be calculated automatically provided that VCR is set. What
should be the vector base address? One possible way is similar to the
way how IVPR is set to interrupt_base in head_44x.S but leave the
actual interrupt handler blank, which will be filled in later with a
jump instruction which jumps to the actual ISR once a device driver
request its irq. This means that request_irq() needs to modify kernel
code. Kinda of dangerous, isn't it? This might be part of
infrastructure change you mentioned?

Regards,
-Shawn.

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Mon, Apr 18, 2005 at 02:14:23PM -0700, Shawn Jin wrote:
> 
> > What could be interesting, though, is to to make all 4xx IRQs
> > critical, in this case we could use VR to quickly determine which IRQ
> > was asserted, instead of current implementation when we use bit
> > operations. Not sure, if performance gain is really worth the
> > effort :)
> 
> If we use VR to determine which IRQ was asserted, it's kind of
> reverse. We usually fetch a handler by its IRQ number. It could
> require to change irq_desc[] and request_irq().

You seems don't undestand what I'm talking about. VR can help us with 
determining what IRQ was asserted. It has nothing to do with irq_desc.

Curently, we use some bit searching functions like ffs to get IRQ 
number, VR can be used instead.

-- 
Eugene

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Mon, Apr 18, 2005 at 01:54:44PM -0700, Lawrence E. Bakst wrote:
> I'd point out that your own statements were pretty general to start 
> off with. 

Sure, because I wasn't the one who needed critical IRQs :). It really 
works this way: people who need some new facility either implement it 
themselves and submit patch or present their case which can, in 
theory, motivate someone else to implement this.

So far _none_ of the people whining about absence of critical IRQs did 
either.

Things like "It'd be nice to have this" aren't good enough :)


> I would have liked to elaborate but unfortunately the work 
> is an ongoing research effort for a client under NDA so I can't say 
> very much.

Nice.

> 
> >Also, it'd be nice if you explained how you dealt with infrastructure
> >problems I mentioned in my previous e-mail.
> 
> We didn't deal with them as we have a kind of special case. I am not 
> trying to minimize the importance of that issue in the general case.
> 
> >
> >Answering your question about "real time interrupt latency
> >constraints", I have to ask what _exact_ latencies you need, and why
> >do you think Linux can provide hard real-time guarantees?
> 
> We are trying to get to an interrupt response time of substantially 
> less than what you mention below. In this case "substantially" means 
> between 2 and 3 decimal orders of magnitude.

Are you really sure that Linux on 440GX is a good platform for 
microsecond or less IRQ latency? At this speeds, you have to worry 
about TLB misses, cache issues, etc. 

> The only point I was trying to make is that the critical interrupt 
> facility was useful to us. That was it. It's not clear to me that 
> within the constraints of how Linux works, the various 
> infrastructure issues, as well as politics, that it could be made 
> useful to others in a generalized form.

IMO, it'd be nice to have something like super-high-priority IRQs for 
some specific tasks in the kernel. However, this needs some generic 
support first, to be useful. Also, not all CPUs have something like 
Critical interrupts, so maybe simple IRQ virtualization will be more 
generic approach.

So far, I see one use for critical interrupts, which doesn't require 
such generic support - make all IRQS critical, so we can use some 
basic prioritization and vector generation.

-- 
Eugene

Re: Interrupt prioritization on linux for ppc440

[Shawn Jin]

On 4/18/05, Eugene Surovegin <ebs@ebshome.net> wrote:
> On Mon, Apr 18, 2005 at 02:14:23PM -0700, Shawn Jin wrote:
> >
> > > What could be interesting, though, is to to make all 4xx IRQs
> > > critical, in this case we could use VR to quickly determine which IRQ
> > > was asserted, instead of current implementation when we use bit
> > > operations. Not sure, if performance gain is really worth the
> > > effort :)
> >
> > If we use VR to determine which IRQ was asserted, it's kind of
> > reverse. We usually fetch a handler by its IRQ number. It could
> > require to change irq_desc[] and request_irq().
>=20
> You seems don't undestand what I'm talking about. VR can help us with
> determining what IRQ was asserted. It has nothing to do with irq_desc.

I guess I understand your statement of using VR to quickly determin
IRQ number. (VR - VCR) >> 9 is the irq number if UIC0_SR0 has the
highest priority, which  is quicker than bit searching.

However there could be a possible problem when a 2nd critical
interrupt is asserted before the 1st one is served. Then fetching the
VR returns the address for the 2nd interrupt. Of course if UIC can
queue interrupts when calculating VRs, there wouldn't be such a
problem.

Regards,
-Shawn.

Re: Interrupt prioritization on linux for ppc440

[Eugene Surovegin]

On Mon, Apr 18, 2005 at 04:11:02PM -0700, Shawn Jin wrote:

[snip]

> However there could be a possible problem when a 2nd critical
> interrupt is asserted before the 1st one is served. Then fetching the
> VR returns the address for the 2nd interrupt. Of course if UIC can
> queue interrupts when calculating VRs, there wouldn't be such a
> problem.

According to the user manual, UIC indeed queues IRQs based on their 
priorities, so I guess there shouldn't be any problems.

Look at paragraphs titled "Vector generation scenarios".

-- 
Eugene