Routable IRQs

Routable IRQs

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3041 bytes --]


Hi Thomas & Jason,

I'm dealing with an interesting situation which I'm wondering if Linux
already support for.

Basically, in some TI SoCs we have what's referred to as Programmable
Real-Time Unit SubSystem (PRUSS). That's essentially a really simple,
low latency, single cycle architecture which is pretty darn good for bit
banging peripherals (ETH, SPI, I2C, UART, etc). It's very predicatable
because every instruction takes 5ns and interrupts don't cause
exceptions, they just get registered.

Anyway, the interesting part is that PRUSS has 64 events (on current
incarnations at least) and PRUSS has 10 physical IRQ lines to the ARM
land. Each of these 64 events can be routed to any of these 10 IRQ
lines. This might not be very useful on UP (AM335x & AM437x) other than
the fact that soft-IP drivers running on Linux would need to guarantee
they are the ones who should handle the IRQ. However, on SMP (AM57xx) we
could have real tangible benefits by means of IRQ affinity, etc.

So, the question is, what is there in IRQ subsystem today for routable
IRQ support ?

If a Diagram helps here's a simple one. Note that I'm not showing
details on the PRUSS side, but that side can also map events pretty much
any way it wants.

 .--------------------.             .--------------------.
 |      HOST CPU      |             |       PRUSS        |
 |--------------------|             |--------------------|
 |                    |             |                    |
 |               irq0 |<-.----------|evt0                |
 |                    |  |          |                    |
 |               irq1 |  |  .-------|evt1                |
 |                    |  |  |       |                    |
 |               irq2 |  '----------|evt2                |
 |                    |     |       |                    |
 |               irq3 |     |       |                    |
 |                    |     |       |                    |
 |               irq4 |     |       | .                  |
 |                    |     |       |                    |
 |               irq5 |     |       | .                  |
 |                    |     |       |                    |
 |               irq6 |     |       | .                  |
 |                    |     |       |                    |
 |               irq7 |<----'       |                    |
 |                    |             |                    |
 |               irq8 |             |                    |
 |                    |             |                    |
 |               irq9 |<------------|evtN                |
 '--------------------'             '--------------------'

Given this setup, what I want to do, is let soft-IP drivers running on
linux rely on standard *request_*irq() calls and DTS descrition. But I'm
still considering how/if we should describe the routing itself or just
go round-robin (i.o.w. irq0 -> evt0, irq1 -> evt1, ..., irq9 -> evt9,
irq0 -> evt10, ...).

Thoughts ?

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Routable IRQs

[Thomas Gleixner]

Felipe,

On Tue, 29 Dec 2015, Felipe Balbi wrote:
> Anyway, the interesting part is that PRUSS has 64 events (on current
> incarnations at least) and PRUSS has 10 physical IRQ lines to the ARM
> land. Each of these 64 events can be routed to any of these 10 IRQ
> lines. This might not be very useful on UP (AM335x & AM437x) other than
> the fact that soft-IP drivers running on Linux would need to guarantee
> they are the ones who should handle the IRQ. However, on SMP (AM57xx) we
> could have real tangible benefits by means of IRQ affinity, etc.
> 
> So, the question is, what is there in IRQ subsystem today for routable
> IRQ support ?
> 
> If a Diagram helps here's a simple one. Note that I'm not showing
> details on the PRUSS side, but that side can also map events pretty much
> any way it wants.
> 
>  .--------------------.             .--------------------.
>  |      HOST CPU      |             |       PRUSS        |
>  |--------------------|             |--------------------|
>  |                    |             |                    |
>  |               irq0 |<-.----------|evt0                |
>  |                    |  |          |                    |
>  |               irq1 |  |  .-------|evt1                |
>  |                    |  |  |       |                    |
>  |               irq2 |  '----------|evt2                |
>  |                    |     |       |                    |
>  |               irq3 |     |       |                    |
>  |                    |     |       |                    |
>  |               irq4 |     |       | .                  |
>  |                    |     |       |                    |
>  |               irq5 |     |       | .                  |
>  |                    |     |       |                    |
>  |               irq6 |     |       | .                  |
>  |                    |     |       |                    |
>  |               irq7 |<----'       |                    |
>  |                    |             |                    |
>  |               irq8 |             |                    |
>  |                    |             |                    |
>  |               irq9 |<------------|evtN                |
>  '--------------------'             '--------------------'
> 
> Given this setup, what I want to do, is let soft-IP drivers running on
> linux rely on standard *request_*irq() calls and DTS descrition. But I'm
> still considering how/if we should describe the routing itself or just
> go round-robin (i.o.w. irq0 -> evt0, irq1 -> evt1, ..., irq9 -> evt9,
> irq0 -> evt10, ...).
> 
> Thoughts ?

I have a few questions:

 - Is there a "mapping" block between PRUSS and the host interrupt controller
   or is this "mapping" block part of PRUSS?

 - We all know how well shared interrupts work. Is there a point of supporting
   64 interrupts when you only have 10 irq lines available?

 - I assume that the PRUSS interrupt mapping is more or less a question of the
   firmware implementation. So you either have a fixed association in the
   firmware which is reflected in the DT description of the IP block or you
   need an interface to tell the PRUSS firmware which event it should map to
   which irq line. Is there actually a value in doing the latter?

Thanks,

	tglx

Re: Routable IRQs

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 4387 bytes --]


Hi Thomas,

Thomas Gleixner <tglx@linutronix.de> writes:
> On Tue, 29 Dec 2015, Felipe Balbi wrote:
>> Anyway, the interesting part is that PRUSS has 64 events (on current
>> incarnations at least) and PRUSS has 10 physical IRQ lines to the ARM
>> land. Each of these 64 events can be routed to any of these 10 IRQ
>> lines. This might not be very useful on UP (AM335x & AM437x) other than
>> the fact that soft-IP drivers running on Linux would need to guarantee
>> they are the ones who should handle the IRQ. However, on SMP (AM57xx) we
>> could have real tangible benefits by means of IRQ affinity, etc.
>> 
>> So, the question is, what is there in IRQ subsystem today for routable
>> IRQ support ?
>> 
>> If a Diagram helps here's a simple one. Note that I'm not showing
>> details on the PRUSS side, but that side can also map events pretty much
>> any way it wants.
>> 
>>  .--------------------.             .--------------------.
>>  |      HOST CPU      |             |       PRUSS        |
>>  |--------------------|             |--------------------|
>>  |                    |             |                    |
>>  |               irq0 |<-.----------|evt0                |
>>  |                    |  |          |                    |
>>  |               irq1 |  |  .-------|evt1                |
>>  |                    |  |  |       |                    |
>>  |               irq2 |  '----------|evt2                |
>>  |                    |     |       |                    |
>>  |               irq3 |     |       |                    |
>>  |                    |     |       |                    |
>>  |               irq4 |     |       | .                  |
>>  |                    |     |       |                    |
>>  |               irq5 |     |       | .                  |
>>  |                    |     |       |                    |
>>  |               irq6 |     |       | .                  |
>>  |                    |     |       |                    |
>>  |               irq7 |<----'       |                    |
>>  |                    |             |                    |
>>  |               irq8 |             |                    |
>>  |                    |             |                    |
>>  |               irq9 |<------------|evtN                |
>>  '--------------------'             '--------------------'
>> 
>> Given this setup, what I want to do, is let soft-IP drivers running on
>> linux rely on standard *request_*irq() calls and DTS descrition. But I'm
>> still considering how/if we should describe the routing itself or just
>> go round-robin (i.o.w. irq0 -> evt0, irq1 -> evt1, ..., irq9 -> evt9,
>> irq0 -> evt10, ...).
>> 
>> Thoughts ?
>
> I have a few questions:
>
>  - Is there a "mapping" block between PRUSS and the host interrupt controller
>    or is this "mapping" block part of PRUSS?

The description in TRM is a bit "poor", but from what I can gather, the
mapping is done on an interrupt controller inside the PRUSS. However,
Linux is the one who's got the driver for that INTC (well, Linux will be
the one with the soft ethernet/uart/whatever IP to talk to). All of its
(INTC's) registers are memory mapped to the ARM side.

>  - We all know how well shared interrupts work. Is there a point of supporting
>    64 interrupts when you only have 10 irq lines available?

I'm looking at these 64 events more like MSI kind of events. It's just
that the events themselves can be routed to any of the 10 available HW
IRQ lines.

>  - I assume that the PRUSS interrupt mapping is more or less a question of the
>    firmware implementation. So you either have a fixed association in the
>    firmware which is reflected in the DT description of the IP block or you
>    need an interface to tell the PRUSS firmware which event it should map to
>    which irq line. Is there actually a value in doing the latter?

right, I'd say the mapping is pretty static. Unless Suman has some extra
information which I don't. I guess the question was really to see if
there was an easy way for doing this so we don't have to mess with DTS
for every other FW and their neighbor.

Chances are (or at least I'm speculating) in most cases we won't use
more than 10 events anyway (Suman ?) so we might not run into any
troubles.

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Routable IRQs

[Thomas Gleixner]

Felipe,

On Wed, 30 Dec 2015, Felipe Balbi wrote:
> Thomas Gleixner <tglx@linutronix.de> writes:
> >  - Is there a "mapping" block between PRUSS and the host interrupt controller
> >    or is this "mapping" block part of PRUSS?
> 
> The description in TRM is a bit "poor", but from what I can gather, the
> mapping is done on an interrupt controller inside the PRUSS. However,
> Linux is the one who's got the driver for that INTC (well, Linux will be
> the one with the soft ethernet/uart/whatever IP to talk to). All of its
> (INTC's) registers are memory mapped to the ARM side.

Ok. And the INTC registers include the "mapping" configuration, right?
 
> >  - We all know how well shared interrupts work. Is there a point of supporting
> >    64 interrupts when you only have 10 irq lines available?
> 
> I'm looking at these 64 events more like MSI kind of events. It's just

Well, that's fine to look at them this way, but they will end up shared no
matter what.

> that the events themselves can be routed to any of the 10 available HW
> IRQ lines.
> 
> >  - I assume that the PRUSS interrupt mapping is more or less a question of the
> >    firmware implementation. So you either have a fixed association in the
> >    firmware which is reflected in the DT description of the IP block or you
> >    need an interface to tell the PRUSS firmware which event it should map to
> >    which irq line. Is there actually a value in doing the latter?
> 
> right, I'd say the mapping is pretty static. Unless Suman has some extra
> information which I don't. I guess the question was really to see if
> there was an easy way for doing this so we don't have to mess with DTS
> for every other FW and their neighbor.

Well, you will need information about every other firmware simply because you
need to know which events the firmware is actually using and what the purpose
of the particular event is.

Assume you have a simple uart with 3 events (RX, TX, status). So how will the
firmware tell you which event is which? You have a few options:

 1) DT + fixed mapping scheme: 

    Describe the PRUSS event number in DT and have a fixed mapping scheme like
    the one you mentioned evt0 -> irq0 .....

 2) DT + DT mapping scheme

    Describe the PRUSS event number in DT and describe the mapping scheme in
    DT as well

 3) DT + dynamic mapping scheme

    Describe the PRUSS event number in DT and let your interrupt controller
    associate the irq number dynamically. That's kind of similar to MSI with
    the exception that it needs to support shared interrupts.

 4) Fully dynamic association

    Have a query interface to the firmware which tells you which event it uses
    for which particular purpose (RX, TX ...) and then establish a dynamic
    mapping to one of the interrupts.

Not sure which level of complexity you want :)

Thanks,

	tglx



 

Re: Routable IRQs

[Felipe Balbi]

[-- Attachment #1: Type: text/plain, Size: 3574 bytes --]


Hi,

Thomas Gleixner <tglx@linutronix.de> writes:
> Felipe,
>
> On Wed, 30 Dec 2015, Felipe Balbi wrote:
>> Thomas Gleixner <tglx@linutronix.de> writes:
>> >  - Is there a "mapping" block between PRUSS and the host interrupt controller
>> >    or is this "mapping" block part of PRUSS?
>> 
>> The description in TRM is a bit "poor", but from what I can gather, the
>> mapping is done on an interrupt controller inside the PRUSS. However,
>> Linux is the one who's got the driver for that INTC (well, Linux will be
>> the one with the soft ethernet/uart/whatever IP to talk to). All of its
>> (INTC's) registers are memory mapped to the ARM side.
>
> Ok. And the INTC registers include the "mapping" configuration, right?

right. A bunch of 32 bit registers each with several 4 bit fields (one
for each of the 64 events) where we write the physical IRQ number.

>> >  - We all know how well shared interrupts work. Is there a point of supporting
>> >    64 interrupts when you only have 10 irq lines available?
>> 
>> I'm looking at these 64 events more like MSI kind of events. It's just
>
> Well, that's fine to look at them this way, but they will end up
> shared no matter what.

sure :-)

>> that the events themselves can be routed to any of the 10 available HW
>> IRQ lines.
>> 
>> >  - I assume that the PRUSS interrupt mapping is more or less a question of the
>> >    firmware implementation. So you either have a fixed association in the
>> >    firmware which is reflected in the DT description of the IP block or you
>> >    need an interface to tell the PRUSS firmware which event it should map to
>> >    which irq line. Is there actually a value in doing the latter?
>> 
>> right, I'd say the mapping is pretty static. Unless Suman has some extra
>> information which I don't. I guess the question was really to see if
>> there was an easy way for doing this so we don't have to mess with DTS
>> for every other FW and their neighbor.
>
> Well, you will need information about every other firmware simply because you
> need to know which events the firmware is actually using and what the purpose
> of the particular event is.
>
> Assume you have a simple uart with 3 events (RX, TX, status). So how will the
> firmware tell you which event is which? You have a few options:
>
>  1) DT + fixed mapping scheme: 
>
>     Describe the PRUSS event number in DT and have a fixed mapping scheme like
>     the one you mentioned evt0 -> irq0 .....
>
>  2) DT + DT mapping scheme
>
>     Describe the PRUSS event number in DT and describe the mapping scheme in
>     DT as well
>
>  3) DT + dynamic mapping scheme
>
>     Describe the PRUSS event number in DT and let your interrupt controller
>     associate the irq number dynamically. That's kind of similar to MSI with
>     the exception that it needs to support shared interrupts.
>
>  4) Fully dynamic association
>
>     Have a query interface to the firmware which tells you which event it uses
>     for which particular purpose (RX, TX ...) and then establish a dynamic
>     mapping to one of the interrupts.
>
> Not sure which level of complexity you want :)

I guess only 1, 2 are anything worth considering, most likely. 4 would
just be too much headache :-p

3 might be doable too, though a bit more complex. Suman (who has been
working on this for much longer than I have) might have some extra info
to add, but he's on vacations for now. Hopefully, he'll add to this
thread once he's back.

cheers

-- 
balbi

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]