[parisc-linux] Thoughts on arch/parisc/irq.c

[parisc-linux] Thoughts on arch/parisc/irq.c

[Grant Grundler]

linux/arch/parisc/kernel/irq.c:irq_alloc()/request_irq() seems to be
the method to allocate EIRR bits and register ISRs with PA ext_intr
handler. Dino will call this to initialize it's own EIM register (IAR0).
But to program IAR0 Dino also needs the processor HPA of whatever
processor it is supposed to interrupt.
Is there an interface to return HPAs?

(BTW - anyone else modifying gecko/dino.c?)

Support for PCI 2.2 Message Signalled Interrupts requires
a similar interface - Dino can support this if the interface
were present and PCI drivers wanted to use it.


More thoughts on basic PA interrupt handling:

o Do GBD or other psuedo drivers need to reserve EIRR bits?
  Ie soft interrupts to reschedule work at lower SPL levels.

o associating SPL levels to specific EIRR bits allows changing SPL
  much faster. ie. if EIRR bit 1 is used for SPL2, then setting
  bit 1 in the SPL5 bit mask to disable EIRR bit 1 from generating
  an interrupt makes it very easy to change from SPL2 to SPL5.
  Eg. SPLHI just disables all EIRR bits.

o PA2.0 architecture defines EIRR to be 64-bits wide.
  irq_alloc() and request_irq() hard code 31.
  Using a #define with "ifdef" around it for 32/64 bit
  differences would be better. Just trying to make the
  transition that direction easier.

o Each processor can have it's own EIRR switch table.
  Thus, "irq_action[]" could be an an array hanging off a per processor
  data structure. This is interesting for large configurations where
  the 31 bits aren't enough and sharing isn't supported.

o EIRR bits can be shared just like IRQ lines. A wrapper gets
  put into the irq_action[] field when sharing is required
  (eg run out of EIRR bits to hand out). The wrapper function
  simple calls all ISR's registered for a given EIRR bit.
  This can be ignored for now since A-class/712/715 etc
  configurations probably won't need this.

grant

Grant Grundler
Communications Infrastructure Computer Operations
+1.408.447.7253

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Alan Cox]

> (BTW - anyone else modifying gecko/dino.c?)

Not just at the moment

> o Do GBD or other psuedo drivers need to reserve EIRR bits?
>   Ie soft interrupts to reschedule work at lower SPL levels.

Linux has no notion of SPL levels at all. The model is

Hardware interrupt. These live in a unified space described by a cookie.
We map the cookie to the irq number on a PC but that is optional. Drivers
only know about irqs as a cookie.

Bottom half handlers. We invoke these on the path out of an irq before
returning to the normal scheduled universe. mark_bh() sets a bh to be run
on the next return. BH's are not run if they are already running - they
are single threaded with respect to each other right now - that may change
to be 'with respect to self' one day.

Interrupts may occur during a bh, we normally dont allow interrupts during
an IRQ handler on the same CPU. 

> o Each processor can have it's own EIRR switch table.
>   Thus, "irq_action[]" could be an an array hanging off a per processor
>   data structure. This is interesting for large configurations where
>   the 31 bits aren't enough and sharing isn't supported.

Ok

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Grant Grundler]

Alan Cox wrote:
...
> > o Do GBD or other psuedo drivers need to reserve EIRR bits?
> >   Ie soft interrupts to reschedule work at lower SPL levels.
> 
> Linux has no notion of SPL levels at all. The model is
> 

Alan - looks like something got dropped here. Try again?


> Hardware interrupt. These live in a unified space described by a cookie.
> We map the cookie to the irq number on a PC but that is optional. Drivers
> only know about irqs as a cookie.

I think there is going to a problem with "cookies" on PA.
The problem is GSC devices need to program an "EIM" register.
(For Dino, this is IAR0). The contents of the EIM register
are a processor HPA (bits 11-31) and EIRR bit number (bits 0-4;
ie a value 0-31). The location of this register is NOT
defined by PA I/O architecture - or at least not well defined.
Is this a real problem or not?

Another tidbit: I know of only one device which can use "6-bit"
(0-63) EIRR vectors. GSC and PCI devices which are capable of mastering
their own interrupt transactions can also use 6-bits. 6-bits is
obviously only supportable running a 64-bit kernel binary.


> Bottom half handlers. We invoke these on the path out of an irq before
> returning to the normal scheduled universe. mark_bh() sets a bh to be run
> on the next return. BH's are not run if they are already running - they
> are single threaded with respect to each other right now - that may change
> to be 'with respect to self' one day.

For single CPU is doesn't matter. For SMP scalability it might.

> Interrupts may occur during a bh, we normally dont allow interrupts during
> an IRQ handler on the same CPU. 

SPL levels allow the system clock to always get service regardless of how
badly an ISR behaves on the interrupt stack. SPL levels also allow
the clock to "reschedule" work at a lower priority on the interrupt stack
to guarantee the timeout is called "pretty soon". SPL levels also
allow device drivers to service IRQs while the bottom half is active.
Allowing this nesting of SPL levels permits scheduling on the interrupt
stack. Any particular reason for not allowing this?
(eg abuse by driver writers)

thanks,
grant

Grant Grundler
Communications Infrastructure Computer Operations
+1.408.447.7253

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Alan Cox]

> Alan - looks like something got dropped here. Try again?

The perils of 2am posting - add the word "different"

> I think there is going to a problem with "cookies" on PA.
> The problem is GSC devices need to program an "EIM" register.
> (For Dino, this is IAR0). The contents of the EIM register
> are a processor HPA (bits 11-31) and EIRR bit number (bits 0-4;
> ie a value 0-31). The location of this register is NOT
> defined by PA I/O architecture - or at least not well defined.
> Is this a real problem or not?

It isnt. If need be you hand out irq numbers to drivers that are
indexes into an array that gives all the real gsc/pci mappings, registers
and other goodies.

> > on the next return. BH's are not run if they are already running - they
> > are single threaded with respect to each other right now - that may change
> > to be 'with respect to self' one day.
> 
> For single CPU is doesn't matter. For SMP scalability it might.

Thats why it might change 

> SPL levels allow the system clock to always get service regardless of how
> badly an ISR behaves on the interrupt stack. SPL levels also allow

We flame people who write badly behaving irq handlers instead. 

> to guarantee the timeout is called "pretty soon". SPL levels also
> allow device drivers to service IRQs while the bottom half is active.

We service IRQs with the bottom half active. The idea is that all non
immediate processing occurs at bh layer - think of the BH system as a BSD
like kernel with only one SPL level.

Having one level reflects the realities of more machines and doesn't really
lose us anything. A typical example flow is

	user space
	[IRQ arrives]
		myether_irq
			handle ring buffer
			netif_rx(buffer)
		 	(marks the NET bh to run)
		end irq
		runs net_bh if not running, irqs are enabled
		process packets
		done
	user space


Alan

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Philipp Rumpf]

> > Hardware interrupt. These live in a unified space described by a cookie.
> > We map the cookie to the irq number on a PC but that is optional. Drivers
> > only know about irqs as a cookie.

> I think there is going to a problem with "cookies" on PA.
> The problem is GSC devices need to program an "EIM" register.
> (For Dino, this is IAR0). The contents of the EIM register
> are a processor HPA (bits 11-31) and EIRR bit number (bits 0-4;

the Dino documentations says bits 5-31 are HPA but as long as 5-10 are 0 we
don't have to care.

> ie a value 0-31). The location of this register is NOT
> defined by PA I/O architecture - or at least not well defined.
> Is this a real problem or not?

No.  You just do
gsc_writel(0xfffe0000 + dev->irq, dev->hpa + DEVICE_SPECIFIC_OFFSET);
right after request_irq(dev->irq, ...);

> Another tidbit: I know of only one device which can use "6-bit"
> (0-63) EIRR vectors. GSC and PCI devices which are capable of mastering
> their own interrupt transactions can also use 6-bits. 6-bits is
> obviously only supportable running a 64-bit kernel binary.

Is it ok with you if we worry about SMP and PA2.0 boxes later ?

	Philipp Rumpf

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[LaMont Jones]

> No.  You just do
> gsc_writel(0xfffe0000 + dev->irq, dev->hpa + DEVICE_SPECIFIC_OFFSET);
> right after request_irq(dev->irq, ...);
> Is it ok with you if we worry about SMP and PA2.0 boxes later ?

Writing to the specific HPA is most likely a SMP issue as well, but if we
go the route of using global broadcast interrupts, we should leave some
__really__ large handwriting on the wall for when SMP gets dealt with.
The mad dash to handle an interrupt is a great way to thrash the machine.

Likewise, if there's a clean way to get the processor HPA in the IRQ value,
there is no reason not to do it now, and avoid tracking down the problems
later.

lamont

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Grant Grundler]

Philipp Rumpf:
> the Dino documentations says bits 5-31 are HPA but as long as 5-10 are 0 we
> don't have to care.

FYI - All PA device addresses are 4k page aligned.
I've seen devices alias within a page but the IODC can only
report the number of 4k pages a devices uses - ergo 4K aligned.

> 
> > ie a value 0-31). The location of this register is NOT
> > defined by PA I/O architecture - or at least not well defined.
> > Is this a real problem or not?
> 
> No.  You just do
> gsc_writel(0xfffe0000 + dev->irq, dev->hpa + DEVICE_SPECIFIC_OFFSET);
> right after request_irq(dev->irq, ...);

Ok - this is another SMP issue...I'll hardcode 0xfffe0000 for now. 

Platform	Processor HPA
--------	-------------
712/80		0xfffbe000
715/100		0xfffbe000
B132L		0xfffbe000
B180L		0xfffbe000
C200+		0xfffa0000

> > Another tidbit: I know of only one device which can use "6-bit"
> > (0-63) EIRR vectors. GSC and PCI devices which are capable of mastering
> > their own interrupt transactions can also use 6-bits. 6-bits is
> > obviously only supportable running a 64-bit kernel binary.
> 
> Is it ok with you if we worry about SMP and PA2.0 boxes later ?

Absolutely. I brought this up so it doesn't have to get redesigned
by whoever feels like tackling those issues.  Note the subject line.
I'm just not into doing thing two or three times in a row.

grant

> 
> 	Philipp Rumpf

Grant Grundler
Communications Infrastructure Computer Operations
+1.408.447.7253

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Alex deVries]

Grant Grundler wrote:
> 
> Ok - this is another SMP issue...I'll hardcode 0xfffe0000 for now.
> 
> Platform        Processor HPA
> --------        -------------
> 712/80          0xfffbe000
> 715/100         0xfffbe000
> B132L           0xfffbe000
> B180L           0xfffbe000
> C200+           0xfffa0000

For what it's worth, I have code in my soon-to-be-committed inventory
code that detects this.  The A180C is at 0xfffbe000 also.

I'm just going through the difficulty of merging my code into the
tree...

Here's the output of my code:

Doing the inventory of the machine
     HPA of the processor: fffbe000
Checking CPU's bus: 
Located Staccato L2 180 (A Class 180) (0) on bus 0x0 at 0xfffbe000,
versions 0x0
Located Staccato L2 180 Memory (1) on bus 0x1 at 0xfffbf000, versions
0x8b, 0x00
Bus does not implement On Line Replacement.
Checking for peripherals...
Located Dino PCI Bridge (13) on bus 0x0 at 0xfff80000, versions 0x680,
0x3, 0xa0
Located Dino PS2 Keyboard (10) on bus 0x1 at 0xfff81000, versions 0x7,
0x0, 0x90
Located unknown device (11) on bus 0x0 at 0xffd00000, versions 0x5f,
0x0, 0x81,0
Located Phantom PseudoBC GSC+ Port (7) on bus 0x0 at 0xffc00000,
versions 0x5040
Located Phantom PseudoBC GSC+ Port (7) on bus 0x1 at 0xffc01000,
versions 0x5050


- Alex

-- 
Alex deVries
Vice President of Engineering
The Puffin Group

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Philipp Rumpf]

> > the Dino documentations says bits 5-31 are HPA but as long as 5-10 are 0 we
> > don't have to care.
> FYI - All PA device addresses are 4k page aligned.

Thanks.  This does not conflict with dino decoding the not defined bits too,
though.
> > No.  You just do
> > gsc_writel(0xfffe0000 + dev->irq, dev->hpa + DEVICE_SPECIFIC_OFFSET);
> > right after request_irq(dev->irq, ...);
> 
> Ok - this is another SMP issue...I'll hardcode 0xfffe0000 for now. 
> 
> Platform	Processor HPA
> --------	-------------
> 712/80	0xfffbe000
> 715/100	0xfffbe000
> B132L		0xfffbe000
> B180L		0xfffbe000

If this is a 7300LC, the processor EIR depends on the bus id.

	Philipp Rumpf

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Philipp Rumpf]

> linux/arch/parisc/kernel/irq.c:irq_alloc()/request_irq() seems to be
> the method to allocate EIRR bits 

alloc_irq is just a quick hack to try to avoid sharing interrupt handlers
if possible.  It returns the next unused IRQ but isn't atomic right now.

> and register ISRs with PA ext_intr
> handler. Dino will call this to initialize it's own EIM register (IAR0).
> But to program IAR0 Dino also needs the processor HPA of whatever
> processor it is supposed to interrupt.

Not exactly right.  You can just use the broadcast EIR address (which might
be GSC-specific but we don't have docs on other hardware yet).

I am not sure how GSC devices will be managed, but if we just chose to be lame
and do it like PCI, there will be a list of GSC devices and the Dino driver
just can do gsc_writel(0xfffe0000+dev->irq, dev->hpa + DINO_IAR0);

> (BTW - anyone else modifying gecko/dino.c?)

I think I am pretty synched right now.  The GSC handling is not in place yet so
perhaps you should hardwire everything until it is.

> Support for PCI 2.2 Message Signalled Interrupts requires
> a similar interface - Dino can support this if the interface
> were present and PCI drivers wanted to use it.

I have no idea what that is.  I don't think it is in the Dino docs I have either
so I am afraid I cannot say anything about it.

> o Do GBD or other psuedo drivers need to reserve EIRR bits?
>   Ie soft interrupts to reschedule work at lower SPL levels.

I don't think so.

> o PA2.0 architecture defines EIRR to be 64-bits wide.
>   irq_alloc() and request_irq() hard code 31.
>   Using a #define with "ifdef" around it for 32/64 bit
>   differences would be better. Just trying to make the
>   transition that direction easier.

Dino at least does not support 6-bit interrupt code according to our
documentation.

> o Each processor can have it's own EIRR switch table.
>   Thus, "irq_action[]" could be an an array hanging off a per processor
>   data structure. This is interesting for large configurations where
>   the 31 bits aren't enough and sharing isn't supported.

I don't want to try things before they get tested by other architectures first
unless necessary.  I don't think per-processor irq_actions are a way to go, but
we can have another look at it when it's time to.


> o EIRR bits can be shared just like IRQ lines. A wrapper gets
>   put into the irq_action[] field when sharing is required
>   (eg run out of EIRR bits to hand out). The wrapper function

Once again, not yet.  There is no way I want to know of to run out of EIRR bits
on 712, 715 or A180s, so I'd like to get us up and running on those first.

	Philipp Rumpf

Re: [parisc-linux] Thoughts on arch/parisc/irq.c

[Alan Cox]

> I'm still convinced PA Processor HPAs should be passed to GSC drivers
> seperately (eg dino.c).  On IA64 platforms, I/O SAPIC and Local Sapic
> also seperate the target address and "vector".  I left David Mossberger
> voice mail about this - I suspect/hope the problem has already
> been solved.

Passing that kind of info out to arch specific drivers is ok. The rest of
the system wants a single opaque long cookie (with or without milk) . You
can even make that cookie a cast on a struct addr.

> Not HP's style. Too hostile. The process is to file "Change Requests"
> - a kind of black hole where managers pull statistics.  :^)

I've dealt with HP in another project. Im glad TPG are doing the HP interfacing
not me 8)

> I see three levels: block all IRQs, enable IRQs, and bh.
> It sounds like those correspond to SPLHI/SPL7, SPL5, and SPL2.
> Am I still on the right track?

Yes I think so. Im not that hot on the old BSD terminology

> implement more than one level of SPL. But I now agree not much is lost
> since two important levels are supported.  As long as the clock gets
> serviced, the timeouts it generates would run at about the same time.
> More on this below.

We never do timeouts in loops in an IRQ handler by checking versus the
clock. We also tend to use RDTSC or similar features if available - apic
clock, etc so we recover lost ticks too

> in this context. Knowing that Intel is eager to support UDI on linux,
> I would be worried about how UDI is made to work. (Hint: I worked on

I don't know any major Linux hacker who views UDI with anything but 
mild amusement. Linux people tend to be latency freaks.

> After doing this exercise, I agree with Alan - not much is lost.
> The ATT flavor Unixes need SPL6 because streams processing (another
> message passing subsystem) can take a really long time in the IRQ
> context and thus block the clock.

The crazies at gcom who did some streams based stuff for Linux (not in the
standard kernel) do their streams processing in the bh context for some
reason. 

Where we get drivers for hardware that is really horrible we sometimes do

	my_irq
		mark_bh
		mask irq
		return from irq

as the entire IRQ handler - the aha152x scsi controller is an example of
that. Thankfully for the devices people buy when they want speed you don't
have that problem.

Alan