FPGA access over PCI-E on MPC8536

FPGA access over PCI-E on MPC8536

[Felix Radensky]

Hi,

On my custom MPC8536 based board running 2.6.31 kernel
FPGA is connected via x2 PCI-E lane. FPGA is identified
during PCI scan and is visible via lspci.

0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
         Subsystem: Altera Corporation Unknown device 0004
         Flags: fast devsel, IRQ 16
         Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
         Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ 
Queue=0/5 Enable-
         Capabilities: [78] Power Management version 3
         Capabilities: [80] Express Endpoint IRQ 0
         Capabilities: [100] Virtual Channel


However when I attempt to access FPGA memory my mmapping it in userspace 
the read hangs. The same happens in kernel space. Does it happen because 
FPGA memory is marked as disabled, or because FPGA code is doing 
something wrong ?

Another question is what can cause PCI device memory be marked as
disabled.

Thanks a lot.

Felix.

Re: FPGA access over PCI-E on MPC8536

[Kumar Gala]

On Sep 16, 2009, at 11:17 PM, Felix Radensky wrote:

> Hi,
>
> On my custom MPC8536 based board running 2.6.31 kernel
> FPGA is connected via x2 PCI-E lane. FPGA is identified
> during PCI scan and is visible via lspci.
>
> 0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev  
> 01)
>        Subsystem: Altera Corporation Unknown device 0004
>        Flags: fast devsel, IRQ 16
>        Memory at 98000000 (32-bit, non-prefetchable) [disabled]  
> [size=1M]
>        Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+  
> Queue=0/5 Enable-
>        Capabilities: [78] Power Management version 3
>        Capabilities: [80] Express Endpoint IRQ 0
>        Capabilities: [100] Virtual Channel
>
>
> However when I attempt to access FPGA memory my mmapping it in  
> userspace the read hangs. The same happens in kernel space. Does it  
> happen because FPGA memory is marked as disabled, or because FPGA  
> code is doing something wrong ?

Can you access the device in u-boot?  That would possible tell you if  
the HW is functioning or not.

> Another question is what can cause PCI device memory be marked as
> disabled.

Good question, no idea how lspci decided to print [disabled].  Take a  
look at lspci source and see :)

- k

Re: FPGA access over PCI-E on MPC8536

[Benjamin Herrenschmidt]

> > However when I attempt to access FPGA memory my mmapping it in  
> > userspace the read hangs. The same happens in kernel space. Does it  
> > happen because FPGA memory is marked as disabled, or because FPGA  
> > code is doing something wrong ?
> 
> Can you access the device in u-boot?  That would possible tell you if  
> the HW is functioning or not.
> 
> > Another question is what can cause PCI device memory be marked as
> > disabled.
> 
> Good question, no idea how lspci decided to print [disabled].  Take a  
> look at lspci source and see :)

Maybe the memory enable in the PCI command register isn't set ?

If you don't have a kernel driver that sets it (by calling
pci_enable_device) and u-boot doesn't set it then it's going
to be off and you'll get hangs or machine checks trying to
access the device...

Just an idea...

Cheers,
Ben.

Re: FPGA access over PCI-E on MPC8536

[Felix Radensky]

Hi, Benjamin

Benjamin Herrenschmidt wrote:
>>> However when I attempt to access FPGA memory my mmapping it in  
>>> userspace the read hangs. The same happens in kernel space. Does it  
>>> happen because FPGA memory is marked as disabled, or because FPGA  
>>> code is doing something wrong ?
>> Can you access the device in u-boot?  That would possible tell you if  
>> the HW is functioning or not.
>>
>>> Another question is what can cause PCI device memory be marked as
>>> disabled.
>> Good question, no idea how lspci decided to print [disabled].  Take a  
>> look at lspci source and see :)
> 
> Maybe the memory enable in the PCI command register isn't set ?
> 
> If you don't have a kernel driver that sets it (by calling
> pci_enable_device) and u-boot doesn't set it then it's going
> to be off and you'll get hangs or machine checks trying to
> access the device...
> 
> Just an idea...

Yes, that was my problem exactly ! Setting this bit via setpci
fixed it.

Thanks a lot for your help.

Felix

Re: FPGA access over PCI-E on MPC8536

[David Hawkins]

Hi Felix,

>> On my custom MPC8536 based board running 2.6.31 kernel
>> FPGA is connected via x2 PCI-E lane. FPGA is identified
>> during PCI scan and is visible via lspci.
>>
>> 0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
>>        Subsystem: Altera Corporation Unknown device 0004
>>        Flags: fast devsel, IRQ 16
>>        Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
>>        Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+ 
>> Queue=0/5 Enable-
>>        Capabilities: [78] Power Management version 3
>>        Capabilities: [80] Express Endpoint IRQ 0
>>        Capabilities: [100] Virtual Channel
>>
>>
>> However when I attempt to access FPGA memory my mmapping it in 
>> userspace the read hangs. The same happens in kernel space. Does it 
>> happen because FPGA memory is marked as disabled, or because FPGA code 
>> is doing something wrong ?
> 
> Can you access the device in u-boot?  That would possible tell you if 
> the HW is functioning or not.
> 
>> Another question is what can cause PCI device memory be marked as
>> disabled.
> 
> Good question, no idea how lspci decided to print [disabled].  Take a 
> look at lspci source and see :)


lspci has a -x and -xxx options to print the bytes in the
PCI configuration space header. You should use that to
see what the settings are, and then you'll be able
to interpret what disabled means.

For example, here's lspci run from an x86 host CPU looking
at an MPC8349EA processor peripheral board:

# lspci -s 04:0d.0 -v
04:0d.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
         Flags: 66Mhz, fast devsel, IRQ 10
         Memory at fc300000 (32-bit, non-prefetchable) [size=1M]
         Memory at fe400000 (32-bit, prefetchable) [size=4K]
         Memory at fe300000 (64-bit, prefetchable) [size=1M]
         Memory at fe200000 (64-bit, prefetchable) [size=1M]
         Capabilities: [48] #06 [0080]

# lspci -s 04:0d.0 -x
04:0d.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
00: 57 19 80 00 42 01 b0 00 30 00 20 0b 08 40 00 00
10: 00 00 30 fc 08 00 40 fe 0c 00 30 fe 00 00 00 00
20: 0c 00 20 fe 00 00 00 00 00 00 00 00 00 00 00 00
30: 00 00 00 00 48 00 00 00 00 00 00 00 0a 01 00 00

The memory, non-prefetchable, and size are decoded from bits in
the base address registers. I'm pretty sure there is an
enabled/disabled bit in there too. Thats probably where your
lspci output is getting it from.

I suspect that you could perform a configuration space
write to enable that bit, or that the FPGA initialization
code should be setting that bit. It really depends on
what the PCIe component inside the FPGA was designed to
do.

For example, on the MPC8349EA processor configured as a
peripheral board, the PCI bridge can come up enabled, or
disabled. If the bridge is disabled, then it is up to the
PowerPC processor core (U-boot code) to enable the PCI
interface prior to the host CPU BIOS configuring the base
addresses in the interface. Its possible you've missed
something like this when you configured the core,
eg. an 'enable on boot option' (I haven't used the PCIe
core, so have no idea if this is an option - but you
could submit an Altera service request).

Since you're playing with an FPGA, you should be able to
connect Altera's logic analyzer core to the PCI interface
and capture traffic as well, so that could be used to
debug your problem.

Cheers,
Dave

Re: FPGA access over PCI-E on MPC8536

[Grant Likely]

[-- Attachment #1: Type: text/plain, Size: 1594 bytes --]

Can you access fpga memory from u-boot?

Could very well be an fpga problem.  Same thing can happen on xilinx parts
when accessing an undefined register address via pci bridge --> plb bus -->
ipif register block --> bad address.  The pci core goes into some kind of
retry loop with the plb bus (I'm wasn't the one to look at the pci analyzer,
so I don't know the details), and it completely locks up.

g.

On Sep 16, 2009 11:02 PM, "Felix Radensky" <felix@embedded-sol.com> wrote:

Hi,

On my custom MPC8536 based board running 2.6.31 kernel
FPGA is connected via x2 PCI-E lane. FPGA is identified
during PCI scan and is visible via lspci.

0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
       Subsystem: Altera Corporation Unknown device 0004
       Flags: fast devsel, IRQ 16
       Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
       Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/5 Enable-
       Capabilities: [78] Power Management version 3
       Capabilities: [80] Express Endpoint IRQ 0
       Capabilities: [100] Virtual Channel


However when I attempt to access FPGA memory my mmapping it in userspace the
read hangs. The same happens in kernel space. Does it happen because FPGA
memory is marked as disabled, or because FPGA code is doing something wrong
?

Another question is what can cause PCI device memory be marked as
disabled.

Thanks a lot.

Felix.

_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

[-- Attachment #2: Type: text/html, Size: 2048 bytes --]

Re: FPGA access over PCI-E on MPC8536

[Leon Woestenberg]

Hello Felix,

On Thu, Sep 17, 2009 at 6:17 AM, Felix Radensky <felix@embedded-sol.com> wrote:
> On my custom MPC8536 based board running 2.6.31 kernel
> FPGA is connected via x2 PCI-E lane. FPGA is identified
> during PCI scan and is visible via lspci.
>

I committed a PCI Express device driver for an Altera FPGA (chaining
DMA reference) design upstream that resides in the upstream Linux
kernel at drivers/staging/altpciechdma/

It can act as a reference for the generic part of your design.

Regards,
-- 
Leon