[parisc-linux] Proposal for implementing IRQ affinity

[parisc-linux] Proposal for implementing IRQ affinity

[James Bottomley]

IRQ affinity really only applies to SMP systems.  However, this proposal
will alter the interrupt layout even on UP systems, so if you care about
that sort of thing, read on. 

Introduction 
============ 

External interrupts on parisc processors are triggered by writing to the
memory mapped control register EIRR (cr23).  Since the mapping of the
EIRR into memory is per processor, and usually programmed into external
devices, the upshot is that each external device usually sents an
interrupt to a specific processor.  The implementation of IRQ affinity
would allow us to redesignate a given interrupt to go to a different CPU
(this would mean, for instance, that we could run the user daemon
irqbalanced to balance out all our interrupts among all the processors).

This arrangement means that parisc has 32 distinct interrupts for a 32
bit kernel and 64 on a 64 bit kernel. 

There is, however, a bus based complication: not all parisc busses allow
an arbitrary device to send and interrupt to a CPU directly.  In
general, the older bus controllers: dino, cujo, etc cannot do this, with
the result that processing interrupts in these busses is two-phase. 
When a device interrupts, the irq is fielded by the dino controller,
which has an interrupt register to interrogate the dino specific lines
and see which actual device interrupted and execute the appropriate irq
handling routine.  The newer iosapic interrupt controllers don't require
this because every device attached to the iosapic can be programmed
directly with a CPU EIRR address and bit number. 

The current parisc scheme involves IRQ regions.  Each region is tied to
a particular EIRR bit (CPU irq number) and also contains all the
interrupt designations for the older bus multiplexing as components of
the irq_region.  This is ideal for the older busses, but a bit wasteful
in the iosapic which doesn't need the intermediate interruption
information. 

The new proposal is to sweep away irq regions completely and instead use
the generic struct hw_interrupt_type from linux/irq.h 

The way this would work is that we'd designate a new cpu_irq array
statically for 32 or 64 of these structures (one per bit in the EIRR). 

The iosapic would be allowed simply to allocate a free one of these
(according to the usual IRQ sharing rules) for any device that needed an
interrupt. 

An older bus would allocate a single one, but would then register a
separate vector of interrupts (also a vector of struct
hw_interrupt_type) along with a callback to select the correct vector
for subsequent execution. 

The hw_interrupt_type structure contains an affinity setting callback
(set_affinity) which can be used to adjust the affinity either
internally or via the /proc interface.  We would expose only the CPU
interrupts (first 32 or 64) as capable of having altered affinity (the
remaining older bus interrupts, being effectively slaved to the affinity
of the corresponding CPU interrupts would have a NULL set_affinity
callback). 

This scheme should move us entirely over to the use of generic interrupt
descriptors and allow the affinity setting of those interrupts which are
susceptible to it (namely only the directly accessible EIRR interrupts).
Any comments before I actually try this?

James 


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Bjorn Helgaas]

On Monday 30 August 2004 9:31 pm, James Bottomley wrote:
> This arrangement means that parisc has 32 distinct interrupts for a 32
> bit kernel and 64 on a 64 bit kernel. 

You could support 32 (or 64) interrupts per processor if you wanted to,
though I think right now it's only 32 or 64 for the whole system.

> The newer iosapic interrupt controllers don't require
> this because every device attached to the iosapic can be programmed
> directly with a CPU EIRR address and bit number. 

Strictly speaking, iosapics are programmed with a 16-bit ID/EID (not
an actual EIRR address) and an 8-bit vector (not a 5- or 6-bit bit
number.)  I think the chipset is responsible for routing the ID/EID
to the correct processor, and on parisc, probably only 32 (or 64) of
the 256 possible vectors are usable.  I would guess that in addition
to routing based on ID/EID, the chipset also translates the vector
number to a 2^vector mask in the form the EIRR expects.

I think it would be good if parisc adopted the more generic concepts,
because it's a shame that ia64 and parisc have such totally different
code to drive the identical iosapic hardware.

> This scheme should move us entirely over to the use of generic interrupt
> descriptors and allow the affinity setting of those interrupts which are
> susceptible to it (namely only the directly accessible EIRR interrupts).
> Any comments before I actually try this?

I think this is great (though I don't know all the details of parisc
interrupt controllers).
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Mon, Aug 30, 2004 at 11:31:30PM -0400, James Bottomley wrote:
> IRQ affinity really only applies to SMP systems.  However, this proposal
> will alter the interrupt layout even on UP systems, so if you care about
> that sort of thing, read on. 

*sigh*...that's me, written in big letters on my forehead. :^)

> External interrupts on parisc processors are triggered by writing to the
> memory mapped control register EIRR (cr23).

I don't think this is techinically accurate. The data value written to
the CPU's memory mapped register is transmitted across the bus as is.
This external facing register then converts the (5 or 6 bits for PARISC)
value to a bit mask before it reaches CR23 (EIRR).
IIRC, the PA2.0 I/O ACD defines how the CPU MMIO space is laid out.

> EIRR into memory is per processor, and usually programmed into external
> devices, the upshot is that each external device usually sents an
> interrupt to a specific processor.  The implementation of IRQ affinity
> would allow us to redesignate a given interrupt to go to a different CPU
> (this would mean, for instance, that we could run the user daemon
> irqbalanced to balance out all our interrupts among all the processors).

Thibaut and I started on this last year but didn't get to finish it.
The net result is we have to dump the struct irq_region everywhere 
and replace it with a global IRQ array.

> There is, however, a bus based complication: not all parisc busses allow
> an arbitrary device to send and interrupt to a CPU directly.  In
> general, the older bus controllers: dino, cujo, etc cannot do this,

AFAIK, this is only true for HP V-class.
And only because of some deficiency in the EPAC (CPU to X-bar chip?).
It's not true for every other parisc platform I'm aware of.

PCI and GSC devices can master their own Transaction Based Interrupt (TBI).
GSC device's EIM register are programmed directly with the target address
and vector. PCI devices can master their own TBI if they can be told
which address/data pair to use. E.g. clever scripting for NCR/SYM
scsi chips would allow this and in fact HPUX 10.x c720 SCSI driver does.
Otherwise PCI 2.2 (and later) devices can use MSI or MSI-X to
the same effect. We just need to write the support glue
to make it work for parisc.

>  with
> the result that processing interrupts in these busses is two-phase. 
> When a device interrupts, the irq is fielded by the dino controller,
> which has an interrupt register to interrogate the dino specific lines
> and see which actual device interrupted and execute the appropriate irq
> handling routine. The newer iosapic interrupt controllers don't require
> this because every device attached to the iosapic can be programmed
> directly with a CPU EIRR address and bit number. 

Only one nit: s/every device/every IRQ line/

In all cases, the interrupt controller is a surrogate which converts
the line based interrupt into a TBI.

> The current parisc scheme involves IRQ regions.  Each region is tied to
> a particular EIRR bit (CPU irq number)

Not exactly. An IRQ region groups a bunch of IRQ sources which are managed
by one instance of the interrupt controller driver.

Simple examples to look are the CPU, dino, or lasi IRQ support.
In those cases, a single register maps bits to "downstream" interrupt
sources. We read the register and call the Interrupt Service Routines (ISR)
which correspond to specific bits.

SAPIC is a more complicated since it involves multiple parents.
But the same idea applies: each interrupt source was one entry
in the "region". 

> and also contains all the
> interrupt designations for the older bus multiplexing as components of
> the irq_region.  This is ideal for the older busses, but a bit wasteful
> in the iosapic which doesn't need the intermediate interruption
> information. 

Well, as long as the IRQ line isn't shared, that's true...

> The new proposal is to sweep away irq regions completely and instead use
> the generic struct hw_interrupt_type from linux/irq.h 
>
> The way this would work is that we'd designate a new cpu_irq array
> statically for 32 or 64 of these structures (one per bit in the EIRR). 
> 
> The iosapic would be allowed simply to allocate a free one of these
> (according to the usual IRQ sharing rules) for any device that needed an
> interrupt. 
> 
> An older bus would allocate a single one, but would then register a
> separate vector of interrupts (also a vector of struct
> hw_interrupt_type) along with a callback to select the correct vector
> for subsequent execution. 

I'd rather see one global array (at least 256 entries) with the CPU
(and similar devices) getting a fixed number of consecutive entries.
We should probably reserve the first 16 entries for EISA/ISA support
like we did before.

> The hw_interrupt_type structure contains an affinity setting callback
> (set_affinity) which can be used to adjust the affinity either
> internally or via the /proc interface.  We would expose only the CPU
> interrupts (first 32 or 64) as capable of having altered affinity (the
> remaining older bus interrupts, being effectively slaved to the affinity
> of the corresponding CPU interrupts would have a NULL set_affinity
> callback). 

Sounds good. That will work.

> This scheme should move us entirely over to the use of generic interrupt
> descriptors and allow the affinity setting of those interrupts which are
> susceptible to it (namely only the directly accessible EIRR interrupts).
> Any comments before I actually try this?

It's alot of work. I think that's why thibaut and I ran out of steam
before we finished it. I expect much of the work we did before would
still apply.

Thibaut, you still have a diff or source laying around from that effort?
I might but can't find it right now.

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[James Bottomley]

On Tue, 2004-08-31 at 11:13, Bjorn Helgaas wrote:
> You could support 32 (or 64) interrupts per processor if you wanted to,
> though I think right now it's only 32 or 64 for the whole system.

Yes, but in order to do the affinity thing, we have to make it appear as
if a single interrupt vector potentially belongs to all CPUs...

> Strictly speaking, iosapics are programmed with a 16-bit ID/EID (not
> an actual EIRR address) and an 8-bit vector (not a 5- or 6-bit bit
> number.)  I think the chipset is responsible for routing the ID/EID
> to the correct processor, and on parisc, probably only 32 (or 64) of
> the 256 possible vectors are usable.  I would guess that in addition
> to routing based on ID/EID, the chipset also translates the vector
> number to a 2^vector mask in the form the EIRR expects.

Yes, I'm sorry, sloppy terminology.  I don't actually have any iosapic
based machines, so I only had to muck with the iosapic once getting an
a500 SMP working...I was going from memory instead of actually looking
in the code.

> I think it would be good if parisc adopted the more generic concepts,
> because it's a shame that ia64 and parisc have such totally different
> code to drive the identical iosapic hardware.

I agree.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Bjorn Helgaas]

On Tuesday 31 August 2004 11:29 am, James Bottomley wrote:
> On Tue, 2004-08-31 at 11:13, Bjorn Helgaas wrote:
> > You could support 32 (or 64) interrupts per processor if you wanted to,
> > though I think right now it's only 32 or 64 for the whole system.
> 
> Yes, but in order to do the affinity thing, we have to make it appear as
> if a single interrupt vector potentially belongs to all CPUs...

Ah, right.  But for large machines, it might be useful to have multiple
interrupt domains.  We're already pushing the ia64 limit of about 180
available vectors.  All the CPUs in a node could share a vector space,
so you could have affinity at least to CPUs close to the device.  Dunno
how that fits with the current affinity interfaces, though.
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Matthew Wilcox]

On Tue, Aug 31, 2004 at 11:14:21AM -0600, Grant Grundler wrote:
> > There is, however, a bus based complication: not all parisc busses allow
> > an arbitrary device to send and interrupt to a CPU directly.  In
> > general, the older bus controllers: dino, cujo, etc cannot do this,
> 
> AFAIK, this is only true for HP V-class.
> And only because of some deficiency in the EPAC (CPU to X-bar chip?).
> It's not true for every other parisc platform I'm aware of.

But it's true for several of the subdevices of Lasi/Dino/Wax.

> PCI and GSC devices can master their own Transaction Based Interrupt (TBI).
> GSC device's EIM register are programmed directly with the target address
> and vector. PCI devices can master their own TBI if they can be told
> which address/data pair to use. E.g. clever scripting for NCR/SYM
> scsi chips would allow this and in fact HPUX 10.x c720 SCSI driver does.
> Otherwise PCI 2.2 (and later) devices can use MSI or MSI-X to
> the same effect. We just need to write the support glue
> to make it work for parisc.

That's all true, but doesn't help for Apricot lan, PS/2, serial, Harmony
audio, etc.

> > An older bus would allocate a single one, but would then register a
> > separate vector of interrupts (also a vector of struct
> > hw_interrupt_type) along with a callback to select the correct vector
> > for subsequent execution. 
> 
> I'd rather see one global array (at least 256 entries) with the CPU
> (and similar devices) getting a fixed number of consecutive entries.
> We should probably reserve the first 16 entries for EISA/ISA support
> like we did before.

I'm not sure that's a good idea.  It would give the impression that all
Lasi interrupts could be independently retargetted, which isn't true.
Note that Dino is a little more advanced and has two IRR registers,
so we could theoretically do some cool stuff with that ... but it's not
important enough, IMO.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[James Bottomley]

On Tue, 2004-08-31 at 13:14, Grant Grundler wrote:
> I don't think this is techinically accurate. The data value written to
> the CPU's memory mapped register is transmitted across the bus as is.
> This external facing register then converts the (5 or 6 bits for PARISC)
> value to a bit mask before it reaches CR23 (EIRR).
> IIRC, the PA2.0 I/O ACD defines how the CPU MMIO space is laid out.

Yes, already apologised to Bjorn for that one.

> Thibaut and I started on this last year but didn't get to finish it.
> The net result is we have to dump the struct irq_region everywhere 
> and replace it with a global IRQ array.

Yes, I looked at your code.  what I'd like to do is slightly different
because I would have a static array of the actual CPU interrupts.

> > There is, however, a bus based complication: not all parisc busses allow
> > an arbitrary device to send and interrupt to a CPU directly.  In
> > general, the older bus controllers: dino, cujo, etc cannot do this,
> 
> AFAIK, this is only true for HP V-class.
> And only because of some deficiency in the EPAC (CPU to X-bar chip?).
> It's not true for every other parisc platform I'm aware of.

Actually, I meant the way the dino works: My dino device interrupts, the
dino hits the cpu interrupt, but the interrupt has to be routed back
into generic dino code to find out who actually interrupted and then
call their interrupt routine.

> PCI and GSC devices can master their own Transaction Based Interrupt (TBI).
> GSC device's EIM register are programmed directly with the target address
> and vector. PCI devices can master their own TBI if they can be told
> which address/data pair to use. E.g. clever scripting for NCR/SYM
> scsi chips would allow this and in fact HPUX 10.x c720 SCSI driver does.
> Otherwise PCI 2.2 (and later) devices can use MSI or MSI-X to
> the same effect. We just need to write the support glue
> to make it work for parisc.

Yes, I was basing the above on how we currently work rather than how we
could work.

> >  with
> > the result that processing interrupts in these busses is two-phase. 
> > When a device interrupts, the irq is fielded by the dino controller,
> > which has an interrupt register to interrogate the dino specific lines
> > and see which actual device interrupted and execute the appropriate irq
> > handling routine. The newer iosapic interrupt controllers don't require
> > this because every device attached to the iosapic can be programmed
> > directly with a CPU EIRR address and bit number. 
> 
> Only one nit: s/every device/every IRQ line/
> 
> In all cases, the interrupt controller is a surrogate which converts
> the line based interrupt into a TBI.
> 
> > The current parisc scheme involves IRQ regions.  Each region is tied to
> > a particular EIRR bit (CPU irq number)
> 
> Not exactly. An IRQ region groups a bunch of IRQ sources which are managed
> by one instance of the interrupt controller driver.
> 
> Simple examples to look are the CPU, dino, or lasi IRQ support.
> In those cases, a single register maps bits to "downstream" interrupt
> sources. We read the register and call the Interrupt Service Routines (ISR)
> which correspond to specific bits.
> 
> SAPIC is a more complicated since it involves multiple parents.
> But the same idea applies: each interrupt source was one entry
> in the "region". 

Yes, I was oversimplifying.  Basically the abstraction is unnecessary
for the SAPIC, which is why it looks slightly odd there.

> I'd rather see one global array (at least 256 entries) with the CPU
> (and similar devices) getting a fixed number of consecutive entries.
> We should probably reserve the first 16 entries for EISA/ISA support
> like we did before.

Well, I'd really rather see an IRQ hierarchy.  Currently our irq list is
a total fiction ... if we display what we actually have (namely the 32
or 64 interrupt lines with everything else hanging off them) it's more
intuitively obvious what's going on.

> It's alot of work. I think that's why thibaut and I ran out of steam
> before we finished it. I expect much of the work we did before would
> still apply.
> 
> Thibaut, you still have a diff or source laying around from that effort?
> I might but can't find it right now.

Well, moving to a single 32/64 global array and treating the older
busses as special cases hopefully simplifies this ... I 'll see though
as I get into it.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Tue, Aug 31, 2004 at 06:43:57PM +0100, Matthew Wilcox wrote:
...
> > I'd rather see one global array (at least 256 entries) with the CPU
> > (and similar devices) getting a fixed number of consecutive entries.
> > We should probably reserve the first 16 entries for EISA/ISA support
> > like we did before.
> 
> I'm not sure that's a good idea.

In linux-ia64-test-2.6.8.1/arch/ia64/kernel/iosapic.c:
        unsigned int    gsi_base;       /* first GSI assigned to this IOSAPIC */

Looks like that's what ia64 is doing and I don't see a problem with it.

One of the drawbacks of IRQ regions is they are 32 or 64 entries wide
regardless of many are actually used.

>  It would give the impression that all
> Lasi interrupts could be independently retargetted, which isn't true.

James mentioned this is a problem in his proprosal as well.
If we don't provide /proc/irq/<IRQ#>/smp_affinity, I don't see how
someone could be confused about it. Each IRQ entry will have
it's own function pointer to indicate if it supports redirection
or not.

I don't see how this related to single global array vs multiple arrays.

> Note that Dino is a little more advanced and has two IRR registers,
> so we could theoretically do some cool stuff with that ... but it's not
> important enough, IMO.

I agree.

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Tue, Aug 31, 2004 at 01:49:07PM -0400, James Bottomley wrote:
> > Thibaut and I started on this last year but didn't get to finish it.
> > The net result is we have to dump the struct irq_region everywhere 
> > and replace it with a global IRQ array.
> 
> Yes, I looked at your code.

Ah good.

> what I'd like to do is slightly different
> because I would have a static array of the actual CPU interrupts.

Ok. Why?
I'd really like one method to convert GSI to local interrupt numbers.
Doing away with local translations and directly indexing into a global
array seems the most obvious to me.

> > AFAIK, this is only true for HP V-class.
> > And only because of some deficiency in the EPAC (CPU to X-bar chip?).
> > It's not true for every other parisc platform I'm aware of.
> 
> Actually, I meant the way the dino works: My dino device interrupts, the
> dino hits the cpu interrupt, but the interrupt has to be routed back
> into generic dino code to find out who actually interrupted and then
> call their interrupt routine.

Yes , that's correct. Sorry, I wasn't clear. I was quibbling over this bit:
| not all parisc busses allow
| an arbitrary device to send and interrupt to a CPU directly.

The "AFAIK" comment I wrote above really only applies to this statment.
MSI capable PCI devices can master it's own interrupt transaction
below dino.  That does not mean *every* PCI is MSI capable.
In contrast, every GSC device *must* master it's TBI.
And LASI sub devices are neither.

Does that make more sense?

> > SAPIC is a more complicated since it involves multiple parents.
> > But the same idea applies: each interrupt source was one entry
> > in the "region". 
> 
> Yes, I was oversimplifying.  Basically the abstraction is unnecessary
> for the SAPIC, which is why it looks slightly odd there.

But we still want dino/PCI to work. So we have to replace the
existing abstraction with another one...

> 
> > I'd rather see one global array (at least 256 entries) with the CPU
> > (and similar devices) getting a fixed number of consecutive entries.
> > We should probably reserve the first 16 entries for EISA/ISA support
> > like we did before.
> 
> Well, I'd really rather see an IRQ hierarchy.

A single global array can implement a hierarchy as well.
Entries just contain indexes into other parts of the array.

>   Currently our irq list is
> a total fiction ... if we display what we actually have (namely the 32
> or 64 interrupt lines with everything else hanging off them) it's more
> intuitively obvious what's going on.

*nod*

> > It's alot of work. I think that's why thibaut and I ran out of steam
> > before we finished it. I expect much of the work we did before would
> > still apply.
> > 
> > Thibaut, you still have a diff or source laying around from that effort?
> > I might but can't find it right now.
> 
> Well, moving to a single 32/64 global array and treating the older
> busses as special cases hopefully simplifies this ... I 'll see though
> as I get into it.

Ok. Pushing everything into a global array would make the /proc/irq support
alot easier IMHO.

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Tue, Aug 31, 2004 at 07:57:50PM +0100, Matthew Wilcox wrote:
> > Looks like that's what ia64 is doing and I don't see a problem with it.
> 
> ia64 doesn't have Lasi.

No, but ia64 supports ISA. :^)
(They have their own evils to deal with)

Seriously, what problem does LASI bring that can't be handled with
a global array?

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Matthew Wilcox]

On Tue, Aug 31, 2004 at 01:21:12PM -0600, Grant Grundler wrote:
> On Tue, Aug 31, 2004 at 07:57:50PM +0100, Matthew Wilcox wrote:
> > > Looks like that's what ia64 is doing and I don't see a problem with it.
> > 
> > ia64 doesn't have Lasi.
> 
> No, but ia64 supports ISA. :^)
> (They have their own evils to deal with)

Er.  Says who?

$ grep ISA arch/ia64/Kconfig 
          your box. Other bus systems are ISA, EISA, MicroChannel (MCA) or
config DISABLE_VHPT
$ grep ISA drivers/parisc/Kconfig 
menu "Bus options (PCI, PCMCIA, EISA, GSC, ISA)"
          715/100.  Wax includes an EISA adapter, a serial port (not always
config EISA
        bool "EISA support"
          Say Y here if you have an EISA bus in your machine.  This code
          supports both the Mongoose & Wax EISA adapters.  It is sadly
config ISA
        bool "ISA support"
        depends on EISA
          If you want to plug an ISA card into your EISA bus, say Y here.

So PA-RISC supports ISA and ia64 doesn't :-P

In any case, ISA devices don't have subdevices.

> Seriously, what problem does LASI bring that can't be handled with
> a global array?

The subdevices that can't have their own processor interrupt.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[James Bottomley]

On Tue, 2004-08-31 at 14:34, Grant Grundler wrote:
> Ok. Why?

Because the only interrupts we can control the affinity of are the true
32/64 CPU interrupts. Everything else is effectively hanging off one of
these. If we try to pretend that we can change the affinity of the
subordinate interrupts, we'll end up confusing irqbalanced

> I'd really like one method to convert GSI to local interrupt numbers.
> Doing away with local translations and directly indexing into a global
> array seems the most obvious to me.

I know, but our interrupt structure isn't a flat array, it's a tree with
the root being a given CPU interrupt.  Obviously, since we have to give
some cookie to the devices to identify the interrupt, we end up having a
numeric array.

> But we still want dino/PCI to work. So we have to replace the
> existing abstraction with another one...

Yes, the abstraction for dino/PCI is that we allocate extra interrupts
to the region, but they're unequivocally tied to the give CPU interrupt
line the dino happens to be using.

> A single global array can implement a hierarchy as well.
> Entries just contain indexes into other parts of the array.

Yes, but we need to expose only the cpu interrupts to irqbalanced.  The
problem is that for the purposes of affinity, we can't treat the
fictitious subordinate interrupts as being ordinary independent
interrupts.

James


_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Tue, Aug 31, 2004 at 08:28:11PM +0100, Matthew Wilcox wrote:
> So PA-RISC supports ISA and ia64 doesn't :-P

yes, thanks for the correction. You are right.
I was staring at arch/ia64/kernel/iosapic.c...only it has ISA support.

> > Seriously, what problem does LASI bring that can't be handled with
> > a global array?
> 
> The subdevices that can't have their own processor interrupt.

That has nothing to do with using a global IRQ array.
It just means LASI subdevices can't be redirected individually.
They will still be assigned a global IRQ number.
PCI devices using IRQ lines below DINO will have the same problem.
Ditto for E/ISA support under WAX.

thanks for the correction,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Grant Grundler]

On Tue, Aug 31, 2004 at 03:44:14PM -0400, James Bottomley wrote:
> On Tue, 2004-08-31 at 14:34, Grant Grundler wrote:
> > Ok. Why?
> 
> Because the only interrupts we can control the affinity of are the true
> 32/64 CPU interrupts. Everything else is effectively hanging off one of
> these.

Yes - agreed.

>  If we try to pretend that we can change the affinity of the
> subordinate interrupts, we'll end up confusing irqbalanced

We don't need to pretend anything.
irqbalance-0.9 looks for smp_affinity entry in /proc:

        for (i = 0; i < MAX_INTERRUPTS; i++) {
                /* shortcut: if old and new cpu are identical, don't do a thing
*/
                if (interrupts[i].oldcpu == interrupts[i].cpu)
                        continue;
                if (interrupts[i].type == IRQ_INACTIVE)
                        continue;
                snprintf(filename,255,"/proc/irq/%i/smp_affinity",i);
                file = fopen(filename, "w");
                if (file==NULL)
                        continue;
                if (interrupts[i].cpu < MAX_CPU)
                        fprintf(file,"%x",1<<interrupts[i].cpu);
                else
                        fprintf(file,"ffffffff");
                fclose(file);
        }

We can just not create smp_affinity entries for anything that doesn't
talk directly to a CPU.

On a related issue:
Can we represent per CPU EIRR switch tables with this scheme?
I know we can with existing IRQ regions.
I think not with the new scheme since only one global IRQ maps to
one EIRR bit on all CPUs.

I think we need to disassociate a global IRQ table from the CPU EIRR
switch tables. I'm ok with this for a future project but want to
sketch out what needs to change in order to make it happen.
I suspect this would be a cool project for ESIEE or someone
like that. :^)

> > I'd really like one method to convert GSI to local interrupt numbers.
> > Doing away with local translations and directly indexing into a global
> > array seems the most obvious to me.
> 
> I know, but our interrupt structure isn't a flat array, it's a tree with
> the root being a given CPU interrupt.  Obviously, since we have to give
> some cookie to the devices to identify the interrupt, we end up having a
> numeric array.

Yes. A numeric array isn't a flat array?
Global IRQ #s still makes the most sense to me.

I implemented the IRQ region support thinking the existing IRQ hierarchy
required a hierarchial data structure. But that was heavily influenced
by my previous experience with HPUX which does not use global IRQ numbers
to represent IRQ sources. The arch specific IRQ support really needs to
reflect whatever "core" identifier is used and today that is the global
IRQ number.

> > But we still want dino/PCI to work. So we have to replace the
> > existing abstraction with another one...
> 
> Yes, the abstraction for dino/PCI is that we allocate extra interrupts
> to the region, but they're unequivocally tied to the give CPU interrupt
> line the dino happens to be using.

Can you expand on how PCI IRQs will be generated/decoded?
or want to whack at an implementation first?

> > A single global array can implement a hierarchy as well.
> > Entries just contain indexes into other parts of the array.
> 
> Yes, but we need to expose only the cpu interrupts to irqbalanced.

No. We only need to expose smp_affinity "property".
I still want LASI/PCI IRQs to show up in /proc/interrupts statistics.
ie, they must have a globally assigned IRQ "cookie".

>   The
> problem is that for the purposes of affinity, we can't treat the
> fictitious subordinate interrupts as being ordinary independent
> interrupts.

We can. We just can't indicate they support SMP affinity.

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Michael S. Zick]

On Tue August 31 2004 14:28, Matthew Wilcox wrote:
> On Tue, Aug 31, 2004 at 01:21:12PM -0600, Grant Grundler wrote:
> 
> In any case, ISA devices don't have subdevices.
> 
Perhaps not in the sense you are speaking of (not
in the terms that PCI does)...

But, for instance, DigiBoard has 4, 8, (16?) subdevices
which are distingushed in the driver in cooperation with
the on-board processor.

Probably would fall into your irq tree just like an hp
buss device that doesn't let you diddle the interrupt
sources/assignment the other side of the device/
controller.

My point;
Perhaps you'll need to allow for a next
level tree at each EISA/ISA interrupt to support multi-port
serial boards and other 'intelligent' controllers.

Mike
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Proposal for implementing IRQ affinity

[Matthew Wilcox]

On Tue, Aug 31, 2004 at 04:26:16PM -0500, Michael S. Zick wrote:
> On Tue August 31 2004 14:28, Matthew Wilcox wrote:
> > In any case, ISA devices don't have subdevices.
> 
> But, for instance, DigiBoard has 4, 8, (16?) subdevices
> which are distingushed in the driver in cooperation with
> the on-board processor.

But they're 4/8/16 of the same thing, not a set of completely different
devices with independent drivers that all expect to call request_irq()
and get reasonable results.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux