PCI interrupt question

PCI interrupt question

[Jeff Hane]

Hello, 

 I'm trying to get interrupts working for my PCI device on a 460ex and
am having problem.  My ISR never triggers.

I'm new to PCI(and ppc) and LDD said that I could read the config reg
INTURRUPT_LINE to get the interrupt assigned to my PCI device.  Well,
this always reads zero. 

After reading through the code it appears that the interrupt is being
assigned after reading some information out of the device tree and then
filling in the irg in the pci_dev structure.  

I'm just looking for confirmation that I should be calling request_irq
with the irq that I found in the pci_dev struct.

thanks,
jeff

Re: PCI interrupt question

[Stefan Roese]

Hi Jeff,

On Thursday 03 December 2009 07:21:56 Jeff Hane wrote:
>  I'm trying to get interrupts working for my PCI device on a 460ex and
> am having problem.  My ISR never triggers.

Which 460EX board is this? Canyonlands or some custom board?
 
> I'm new to PCI(and ppc) and LDD said that I could read the config reg
> INTURRUPT_LINE to get the interrupt assigned to my PCI device.  Well,
> this always reads zero.
> 
> After reading through the code it appears that the interrupt is being
> assigned after reading some information out of the device tree and then
> filling in the irg in the pci_dev structure.

Yes. The device tree has to describe the PCI interrupt layout.
 
> I'm just looking for confirmation that I should be calling request_irq
> with the irq that I found in the pci_dev struct.

Yes.

Canyonlands dts correctly describes the PCI interrupts assignments. If you 
have a custom 460EX board, then you probably have a different interrupt routed 
to the PCI slot. This needs to be described in your device tree file.
 
Cheers,
Stefan

Re: PCI interrupt question

[David Hawkins]

Hi Jeff,

>  I'm trying to get interrupts working for my PCI device on a 460ex and
> am having problem.  My ISR never triggers.
> 
> I'm new to PCI(and ppc) and LDD said that I could read the config reg
> INTURRUPT_LINE to get the interrupt assigned to my PCI device.  Well,
> this always reads zero. 
> 
> After reading through the code it appears that the interrupt is being
> assigned after reading some information out of the device tree and then
> filling in the irg in the pci_dev structure.  
> 
> I'm just looking for confirmation that I should be calling request_irq
> with the irq that I found in the pci_dev struct.

Can you clarify:

1. 460EX is your PCI host CPU?

2. You have some PCI device - what? Have you tested it works in a
    standard PCI bus? Eg. ran lspci from an x86 host.

3. When your host processor boots, it assigns the PCI resources.
    On an x86, its the BIOS that assigns the PCI addresses and IRQs
    on the PCI devices. For your 460EX, it could be your bootloader,
    or it could be your host OS (Linux).

4. If you boot Linux on your 460EX and run lspci, and the device
    configuration space register for the IRQ line is 0, then your
    host has not setup the PCI interface on your device correctly.

5. Once you have the PCI IRQ assigned to the device correctly,
    you need to know which IRQ line on the host this corresponds to.
    I'm pretty sure that needs to be described in the DTS as
    commented by Stefan.

These comments might not be 100% correct, but the list of things
to check should be close enough for you to track down your
problem.

Cheers,
Dave

Re: PCI interrupt question

[Jeff Hane]

On Thu, 2009-12-03 at 09:11 -0800, David Hawkins wrote:
> Hi Jeff,
> 
> >  I'm trying to get interrupts working for my PCI device on a 460ex and
> > am having problem.  My ISR never triggers.
> > 
> > I'm new to PCI(and ppc) and LDD said that I could read the config reg
> > INTURRUPT_LINE to get the interrupt assigned to my PCI device.  Well,
> > this always reads zero. 
> > 
> > After reading through the code it appears that the interrupt is being
> > assigned after reading some information out of the device tree and then
> > filling in the irg in the pci_dev structure.  
> > 
> > I'm just looking for confirmation that I should be calling request_irq
> > with the irq that I found in the pci_dev struct.
> 
> Can you clarify:
> 
> 1. 460EX is your PCI host CPU?

yes.  We are using a canyonlands board with 460ex.

> 
> 2. You have some PCI device - what? Have you tested it works in a
>     standard PCI bus? Eg. ran lspci from an x86 host.

It is a FPGA board that we are using to develop a video codec/SOC.  We
have not connected it an x86; however, we know the the PCI interface is
working because we can talk to the FPGA board.

> 
> 3. When your host processor boots, it assigns the PCI resources.
>     On an x86, its the BIOS that assigns the PCI addresses and IRQs
>     on the PCI devices. For your 460EX, it could be your bootloader,
>     or it could be your host OS (Linux).

We are using the vanilla versions of u-boot and linux for the
canyonlands boards and have made few changes and definitely not to the
PCI setup.

> 
> 4. If you boot Linux on your 460EX and run lspci, and the device
>     configuration space register for the IRQ line is 0, then your
>     host has not setup the PCI interface on your device correctly.
>
> 5. Once you have the PCI IRQ assigned to the device correctly,
>     you need to know which IRQ line on the host this corresponds to.
>     I'm pretty sure that needs to be described in the DTS as
>     commented by Stefan.
> 

 So are you saying linux should be writing the irq number to the
INTERRUPT_LINE config reg?  This is what I expected but I do not see
it.  
 I believe the DTS is being parsed properly and the connection is made
to the correct interrupt line on the device.  But somebody still needs
to assign and IRQ number, right?  This is the part that is not clear,
there is an irq field in pci_dev structure which is filled in after
looking at the DTS and I just want to be sure this is the irq number to
be used when calling request_irq.

thanks,
jeff



> These comments might not be 100% correct, but the list of things
> to check should be close enough for you to track down your
> problem.
> 
> Cheers,
> Dave
> 

Re: PCI interrupt question

[Ira W. Snyder]

On Thu, Dec 03, 2009 at 11:39:00AM -0800, Jeff Hane wrote:
> On Thu, 2009-12-03 at 09:11 -0800, David Hawkins wrote:
> > Hi Jeff,
> > 
> > >  I'm trying to get interrupts working for my PCI device on a 460ex and
> > > am having problem.  My ISR never triggers.
> > > 
> > > I'm new to PCI(and ppc) and LDD said that I could read the config reg
> > > INTURRUPT_LINE to get the interrupt assigned to my PCI device.  Well,
> > > this always reads zero. 
> > > 
> > > After reading through the code it appears that the interrupt is being
> > > assigned after reading some information out of the device tree and then
> > > filling in the irg in the pci_dev structure.  
> > > 
> > > I'm just looking for confirmation that I should be calling request_irq
> > > with the irq that I found in the pci_dev struct.
> > 
> > Can you clarify:
> > 
> > 1. 460EX is your PCI host CPU?
> 
> yes.  We are using a canyonlands board with 460ex.
> 
> > 
> > 2. You have some PCI device - what? Have you tested it works in a
> >     standard PCI bus? Eg. ran lspci from an x86 host.
> 
> It is a FPGA board that we are using to develop a video codec/SOC.  We
> have not connected it an x86; however, we know the the PCI interface is
> working because we can talk to the FPGA board.
> 
> > 
> > 3. When your host processor boots, it assigns the PCI resources.
> >     On an x86, its the BIOS that assigns the PCI addresses and IRQs
> >     on the PCI devices. For your 460EX, it could be your bootloader,
> >     or it could be your host OS (Linux).
> 
> We are using the vanilla versions of u-boot and linux for the
> canyonlands boards and have made few changes and definitely not to the
> PCI setup.
> 
> > 
> > 4. If you boot Linux on your 460EX and run lspci, and the device
> >     configuration space register for the IRQ line is 0, then your
> >     host has not setup the PCI interface on your device correctly.
> >
> > 5. Once you have the PCI IRQ assigned to the device correctly,
> >     you need to know which IRQ line on the host this corresponds to.
> >     I'm pretty sure that needs to be described in the DTS as
> >     commented by Stefan.
> > 
> 
>  So are you saying linux should be writing the irq number to the
> INTERRUPT_LINE config reg?  This is what I expected but I do not see
> it.  
>  I believe the DTS is being parsed properly and the connection is made
> to the correct interrupt line on the device.  But somebody still needs
> to assign and IRQ number, right?  This is the part that is not clear,
> there is an irq field in pci_dev structure which is filled in after
> looking at the DTS and I just want to be sure this is the irq number to
> be used when calling request_irq.
> 

Last time I checked, Linux did not write the assigned IRQ to the card's
PCI configuration space. However, the struct pci_dev irq member
contained the correct value. You should be using the value from the
structure for request_irq(). That's how all PCI drivers I've ever seen
work.

Ira

Re: PCI interrupt question

[David Hawkins]

Hi Jeff,

> yes.  We are using a canyonlands board with 460ex.

Ok.

>> 2. You have some PCI device - what? Have you tested it works in a
>>     standard PCI bus? Eg. ran lspci from an x86 host.
> 
> It is a FPGA board that we are using to develop a video codec/SOC.  We
> have not connected it an x86; however, we know the the PCI interface is
> working because we can talk to the FPGA board.

Ok, so that means the BARs are setup.

>> 3. When your host processor boots, it assigns the PCI resources.
>>     On an x86, its the BIOS that assigns the PCI addresses and IRQs
>>     on the PCI devices. For your 460EX, it could be your bootloader,
>>     or it could be your host OS (Linux).
> 
> We are using the vanilla versions of u-boot and linux for the
> canyonlands boards and have made few changes and definitely not to the
> PCI setup.
> 
>> 4. If you boot Linux on your 460EX and run lspci, and the device
>>     configuration space register for the IRQ line is 0, then your
>>     host has not setup the PCI interface on your device correctly.

So what does the output of lspci look like?

>> 5. Once you have the PCI IRQ assigned to the device correctly,
>>     you need to know which IRQ line on the host this corresponds to.
>>     I'm pretty sure that needs to be described in the DTS as
>>     commented by Stefan.
>>
> 
>  So are you saying linux should be writing the irq number to the
> INTERRUPT_LINE config reg?  This is what I expected but I do not see
> it.

Yes. U-Boot or Linux needs to configure the PCI interface.
The output of lspci, or a logic analyzer trace of the PCI
activity on boot would help tell you what is happening.

> I believe the DTS is being parsed properly and the connection is made
> to the correct interrupt line on the device. 

Right, so if someone asserted the interrupt the host would
understand it was from the PCI system.

> But somebody still needs to assign and IRQ number, right? 

Yep. It might require enabling a PCI subsystem in U-Boot.
I recall playing with a Yosemite board and a PCI card,
but I forget whether I had to do anything to enable
PCI configuration space setup. Post a question to the
U-Boot list.

> This is the part that is not clear, there is an irq field in
> pci_dev structure which is filled in after looking at the
> DTS and I just want to be sure this is the irq number to
> be used when calling request_irq.

Really? I thought the pci_dev structures represent the
devices found on the PCI bus, and the IRQ line in
that structure was merely copied from the configuration
space registers. When you request that interrupt, the
code would have to remap it to a host IRQ line, which
it would use the DTS for. (But I'm just speculating).

Bottom line is; if the IRQ field of lspci is 0, then you
need to figure out that problem first :)

Cheers,
Dave

Re: PCI interrupt question

[Ira W. Snyder]

On Thu, Dec 03, 2009 at 11:52:25AM -0800, David Hawkins wrote:
> Hi Jeff,
> 
> > yes.  We are using a canyonlands board with 460ex.
> 
> Ok.
> 
> >> 2. You have some PCI device - what? Have you tested it works in a
> >>     standard PCI bus? Eg. ran lspci from an x86 host.
> > 
> > It is a FPGA board that we are using to develop a video codec/SOC.  We
> > have not connected it an x86; however, we know the the PCI interface is
> > working because we can talk to the FPGA board.
> 
> Ok, so that means the BARs are setup.
> 
> >> 3. When your host processor boots, it assigns the PCI resources.
> >>     On an x86, its the BIOS that assigns the PCI addresses and IRQs
> >>     on the PCI devices. For your 460EX, it could be your bootloader,
> >>     or it could be your host OS (Linux).
> > 
> > We are using the vanilla versions of u-boot and linux for the
> > canyonlands boards and have made few changes and definitely not to the
> > PCI setup.
> > 
> >> 4. If you boot Linux on your 460EX and run lspci, and the device
> >>     configuration space register for the IRQ line is 0, then your
> >>     host has not setup the PCI interface on your device correctly.
> 
> So what does the output of lspci look like?
> 
> >> 5. Once you have the PCI IRQ assigned to the device correctly,
> >>     you need to know which IRQ line on the host this corresponds to.
> >>     I'm pretty sure that needs to be described in the DTS as
> >>     commented by Stefan.
> >>
> > 
> >  So are you saying linux should be writing the irq number to the
> > INTERRUPT_LINE config reg?  This is what I expected but I do not see
> > it.
> 
> Yes. U-Boot or Linux needs to configure the PCI interface.
> The output of lspci, or a logic analyzer trace of the PCI
> activity on boot would help tell you what is happening.
> 
> > I believe the DTS is being parsed properly and the connection is made
> > to the correct interrupt line on the device. 
> 
> Right, so if someone asserted the interrupt the host would
> understand it was from the PCI system.
> 
> > But somebody still needs to assign and IRQ number, right? 
> 
> Yep. It might require enabling a PCI subsystem in U-Boot.
> I recall playing with a Yosemite board and a PCI card,
> but I forget whether I had to do anything to enable
> PCI configuration space setup. Post a question to the
> U-Boot list.
> 
> > This is the part that is not clear, there is an irq field in
> > pci_dev structure which is filled in after looking at the
> > DTS and I just want to be sure this is the irq number to
> > be used when calling request_irq.
> 
> Really? I thought the pci_dev structures represent the
> devices found on the PCI bus, and the IRQ line in
> that structure was merely copied from the configuration
> space registers. When you request that interrupt, the
> code would have to remap it to a host IRQ line, which
> it would use the DTS for. (But I'm just speculating).
> 

The DTS shouldn't need to specify PCI cards, only the PCI controller.
Linux can probe the PCI bus to figure out what is plugged in, just like
it does on x86.

When Linux is assigning BARs and IRQs, it writes the BAR addresses to
the PCI cards, but does not write the IRQ. This was my experience with
PCI hotplug.

> Bottom line is; if the IRQ field of lspci is 0, then you
> need to figure out that problem first :)
> 

In the PCI hotplug case mentioned above, lspci would show the IRQ as 0
until I called request_irq() in the kernel. Afterwards, the correct IRQ
showed up in the lspci output.

Try request_irq(pdev->irq, ....) and see if it works.

Ira

Re: PCI interrupt question

[David Hawkins]

Hi Ira,

> When Linux is assigning BARs and IRQs, it writes the BAR addresses to
> the PCI cards, but does not write the IRQ. This was my experience with
> PCI hotplug.

Ooh, subtle.

>> Bottom line is; if the IRQ field of lspci is 0, then you
>> need to figure out that problem first :)
>>
> 
> In the PCI hotplug case mentioned above, lspci would show the IRQ as 0
> until I called request_irq() in the kernel. Afterwards, the correct IRQ
> showed up in the lspci output.
> 
> Try request_irq(pdev->irq, ....) and see if it works.

You can also try lscpi -xxx, and setpci for writing the
registers. So try cheating and writing to the IRQ register
in the PCI interface.

Another possible problem, you may have your FPGA PCI core
configured in host mode (which can still be a master or target),
and then the IRQ register can not be written.

Cheers,
Dave

Re: PCI interrupt question

[Wolfgang Denk]

Dear David Hawkins,

In message <4B1816F9.1020601@ovro.caltech.edu> you wrote:
> 
> Yep. It might require enabling a PCI subsystem in U-Boot.
> I recall playing with a Yosemite board and a PCI card,
> but I forget whether I had to do anything to enable
> PCI configuration space setup. Post a question to the
> U-Boot list.

PCI is enabled by default on the Canyonlands board, both in U-Boot and
Linux.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"In the long run, every program becomes rococo, and then rubble."
- Alan Perlis

Re: PCI interrupt question

[David Hawkins]

Hi Wolfgang,

> In message <4B1816F9.1020601@ovro.caltech.edu> you wrote:
>> Yep. It might require enabling a PCI subsystem in U-Boot.
>> I recall playing with a Yosemite board and a PCI card,
>> but I forget whether I had to do anything to enable
>> PCI configuration space setup. Post a question to the
>> U-Boot list.
> 
> PCI is enabled by default on the Canyonlands board, both in U-Boot and
> Linux.

Great, thanks!

So Jeff, check out whether or not you can write to the
IRQ register in the PCI configuration space of the device
(using setpci from your host).

If you cannot write to the register, then I'd check your
FPGA core settings.

Cheers,
Dave

PS. Here's the output of lspci for a couple of machines I
have in the lab:

----------------------------------------------------------------------
PLX PCI-9054 master/target interface, with Trenton x86 host board
-----------------------------------------------------------------

[dwh@labslcor6 ~]$ lspci -s 03:0c.0 -v
03:0c.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus 
Bridge (
rev 0a)
         Subsystem: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> 
IOBus Bridg
e
         Flags: bus master, medium devsel, latency 64, IRQ 169
         Memory at fe9ff000 (32-bit, non-prefetchable) [size=256]
         Memory at fb000000 (32-bit, prefetchable) [size=8M]
         Capabilities: <access denied>

[dwh@labslcor6 ~]$ lspci -s 03:0c.0 -x
03:0c.0 Bridge: PLX Technology, Inc. PCI9054 32-bit 33MHz PCI <-> IOBus 
Bridge (
rev 0a)
00: b5 10 54 90 17 01 90 02 0a 00 80 06 08 40 00 00
10: 00 f0 9f fe 08 00 00 fb 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b5 10 54 90
30: 00 00 00 00 40 00 00 00 00 00 00 00 0b 01 00 00


0x3c = IRQ 11, whereas the lspci output indicates IRQ 169,
so there is an IRQ mapping going on there.

----------------------------------------------------------------------
MPC8349EA-based board, with Force Computers x86 host CPU
--------------------------------------------------------

[dwh@labslcor4 ~]$ lspci -s 01:0b.0 -v
01:0b.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
         Flags: 66MHz, fast devsel, IRQ 9
         Memory at f9100000 (32-bit, non-prefetchable) [size=1M]
         Memory at f9600000 (32-bit, non-prefetchable) [size=4K]
         Memory at f9500000 (32-bit, non-prefetchable) [size=1M]
         Memory at f9400000 (64-bit, non-prefetchable) [size=1M]
         Capabilities: <access denied>

[dwh@labslcor4 ~]$ lspci -s 01:0b.0 -x
01:0b.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
00: 57 19 80 00 02 01 b0 00 30 00 20 0b 08 40 00 00
10: 00 00 10 f9 00 00 60 f9 00 00 50 f9 00 00 00 00
20: 04 00 40 f9 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 09 01 00 00


0x3c = IRQ 9, and that matches the lspci output, so there is
no extra mapping here.

Re: PCI interrupt question

[Stefan Roese]

On Thursday 03 December 2009 20:39:00 Jeff Hane wrote:
> > Can you clarify:
> >
> > 1. 460EX is your PCI host CPU?
> 
> yes.  We are using a canyonlands board with 460ex.

PCI works on Canyonlands without any problems. I just tested latest Linux 
release (2.6.32) with an PCI USB card:

-bash-3.2# uname -a
Linux canyonlands 2.6.32 #1 Fri Dec 4 05:42:35 CET 2009 ppc ppc ppc GNU/Linux
-bash-3.2# cat /proc/cpuinfo
processor       : 0
cpu             : 460EX
clock           : 800.000010MHz
revision        : 24.162 (pvr 1302 18a2)
bogomips        : 1600.00
timebase        : 800000010
platform        : PowerPC 44x Platform
model           : amcc,canyonlands
Memory          : 512 MB
-bash-3.2# cat /proc/interrupts
           CPU0
 18:          0   UIC   Edge      L2C
 19:         22   UIC   Level     ohci_hcd:usb2, ohci_hcd:usb3
 20:      15220   UIC   Level     serial
 23:       8038   UIC   Level     MAL TX EOB
 24:      15510   UIC   Level     MAL RX EOB
 25:          0   UIC   Level     MAL SERR
 26:          0   UIC   Level     MAL TX DE
 27:          0   UIC   Level     MAL RX DE
 28:          0   UIC   Level     EMAC
 34:          1   UIC   Level     ohci_hcd:usb1
 35:       2926   UIC   Level     IBM IIC
 38:          0   UIC   Level     IBM IIC
BAD:          0
-bash-3.2#

I can't see any problems with PCI interrupts here.

Cheers,
Stefan

Re: PCI interrupt question

[Benjamin Herrenschmidt]

>  So are you saying linux should be writing the irq number to the
> INTERRUPT_LINE config reg?  This is what I expected but I do not see
> it.  

No it won't necessarily touch it, this is not terribly useful anyways.
Linux will assign an interrupt based on the informations from the
device-tree and you should be able to retrieve it in pci_dev->irq.

>  I believe the DTS is being parsed properly and the connection is made
> to the correct interrupt line on the device.  But somebody still needs
> to assign and IRQ number, right?  

What do you mean ? If the connection is made properly, the code will
obtain an HW IRQ input number on the UIC and will map it to a linux
virtual IRQ number which you can find in pci_dev->irq.
 
> This is the part that is not clear,
> there is an irq field in pci_dev structure which is filled in after
> looking at the DTS and I just want to be sure this is the irq number to
> be used when calling request_irq.

Yes, it is.

If things don't work, it's possible that you assigned the wrong number
in the device-tree ?

Cheers,
Ben.

> thanks,
> jeff
> 
> 
> 
> > These comments might not be 100% correct, but the list of things
> > to check should be close enough for you to track down your
> > problem.
> > 
> > Cheers,
> > Dave
> > 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: PCI interrupt question

[Benjamin Herrenschmidt]

On Thu, 2009-12-03 at 11:52 -0800, David Hawkins wrote:
> 
> Really? I thought the pci_dev structures represent the
> devices found on the PCI bus, and the IRQ line in
> that structure was merely copied from the configuration
> space registers. 

No, it's not. In fact it's mostly irrelevant. It has room for only 8
bits and we commonly manipulate a lot more interrupts on some modern
systems.

Linux will obtain from the device-tree the routing for the interrupt
line(s) of the device. If you don't have a node for the device (which is
allowed for PCI), linux will use the INTERRUPT_PIN register to query
which interrupt line is the device output, and will use the host bridge
device-tree node "interrupt-map" property to find where it's connected
to on the PIC.

It will then map that to a linux virtual IRQ number which is what you
find in pci_dev. At least that's how it works on powerpc :-)

> When you request that interrupt, the
> code would have to remap it to a host IRQ line, which
> it would use the DTS for. (But I'm just speculating).

No, it's already remapped in pci_dev. The mapping happen when the PCI
devices are discovered by the kernel.

> Bottom line is; if the IRQ field of lspci is 0, then you
> need to figure out that problem first :)

I wouldn't trust lspci too much, I'm not sure we bother writing back the
number to the PCI_INTERRUPT_LINE register anymore anyways.

If your pci_dev->irq is non-0 then Linux found -something- but it might
not be right, it depends on your device-tree.

Cheers,
Ben.