orion/kirkwood pcie issue still open with 2.6.32-rc6 (marvell stock 2.6.22.18 works!)

[dk-arm-linux@gmx.de (Dieter Kiermaier)]

Am Mittwoch 11 November 2009 16:21:34 schrieb Lennert Buytenhek:
> On Wed, Nov 11, 2009 at 06:29:22AM -0800, Dieter Kiermaier wrote:
> 
> > > > > Ronan Shitrit from marvell gave me the information to clear bit 2 of physical
> > > > > address 0xf1020100 to enable bus scanning.
> > > > > I don't know what this really does but it helped to get my kernel up and running.
> > > >
> > 
> > I've tested again:
> > 
> > with 2.6.22.18 marvell stock kernel my fpga behing the pcie->pci bridge works as expected.
> > Here is the output of lscpi:
> > 
> > sh-3.2# uname -a
> > Linux DB88FXX81 2.6.22.18 #1 Wed Apr 22 20:31:28 IST 2009 armv5tejl GNU/Linux
> > sh-3.2#
> > 
> > 
> > sh-3.2# lspci -vv
> > 00:01.0 Class 0604: Device 11ab:2211 (rev 01)
> >         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> >         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Latency: 0, Cache Line Size: 32 bytes
> >         Bus: primary=00, secondary=01, subordinate=01, sec-latency=240
> >         I/O behind bridge: 00001000-00004fff
> >         Memory behind bridge: e8000000-ebffffff
> >         Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
> >         BridgeCtl: Parity+ SERR- NoISA+ VGA- MAbort+ >Reset- FastB2B-
> >                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> >         Capabilities: [40] Power Management version 2
> >                 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
> >                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> >                 Bridge: PM- B3+
> >         Capabilities: [48] Express (v1) PCI/PCI-X Bridge, MSI 00
> >                 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> >                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
> >                 DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> >                         RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
> >                         MaxPayload 128 bytes, MaxReadReq 512 bytes
> >                 DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr- TransPend-
> >                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <256ns, L1 unlimited
> >                         ClockPM- Surprise- LLActRep- BwNot-
> >                 LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
> >                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> >                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> >         Capabilities: [100] #11ab
> >         Capabilities: [221] #f0e8
> >         Capabilities: [f1e] #2a0
> >         Capabilities: [e80] #187
> > 
> > 01:08.0 Class ff00: Device 1731:0101 (rev 10)
> >         Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
> >         Status: Cap- 66MHz- UDF- FastB2B- ParErr+ DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> >         Region 0: Memory at e8000000 (32-bit, non-prefetchable) [size=64M]
> >         Region 1: I/O ports at <ignored>
> >         Kernel driver in use: ArtistaNET-III frame buffer driver
> > 
> > 
> > If I switch to mainline linux kernel (there is no difference between openrd, vanilla, orion.git kernels!) my problems start:
> > I have to patch openrd_init() as follows:
> > 
> > diff --git a/arch/arm/mach-kirkwood/openrd_base-setup.c b/arch/arm/mach-kirkwood/openrd_base-setup.c
> > index 77617c7..670312b 100644
> > --- a/arch/arm/mach-kirkwood/openrd_base-setup.c
> > +++ b/arch/arm/mach-kirkwood/openrd_base-setup.c
> > @@ -14,6 +14,7 @@
> >  #include <linux/mtd/partitions.h>
> >  #include <linux/ata_platform.h>
> >  #include <linux/mv643xx_eth.h>
> > +#include <asm/io.h>
> >  #include <asm/mach-types.h>
> >  #include <asm/mach/arch.h>
> >  #include <mach/kirkwood.h>
> > @@ -76,9 +77,21 @@ static void __init openrd_base_init(void)
> > 
> >  static int __init openrd_base_pci_init(void)
> >  {
> > +       u32 cpu_config_reg;
> > +       void __iomem *base;
> > +
> > +       base = ioremap(0xf1020100, 4);
> > +       if (base)
> > +       {
> > +               cpu_config_reg = readl(base);
> > +               cpu_config_reg &= ~(1 << 2);
> > +               writel(cpu_config_reg, base);
> > +               printk("register 0x20100: %x\n", readl(base));
> > +       }
> > +       iounmap(base);
> >         if (machine_is_openrd_base())
> >                 kirkwood_pcie_init();
> > -
> > +
> >         return 0;
> >   }
> >  subsys_initcall(openrd_base_pci_init);
> > ----
> > to allow my system to boot up
> 
> What's likely happening is that your boot loader either enables or
> does not disable this bit, and the 2.6.22.18 kernel disables the bit,
> while the upstream kernel leaves the bit untouched.
> 
> What this bit does is to decide whether or not aborts on the PCI
> interface are translated into processor aborts.  It's not really
> necessary to have this enabled, as the transaction will return
> 0xffffffff to the CPU anyway, which is then handled appropriately
> as well.
> 
> What uboot version are you using?  The uboot versions I have on my
> Kirkwood boards all jump to the OS with this bit already cleared.
> Perhaps we should clear it explicitly from Linux.
> 
> 
> > and after succesfully boot my pci device isn't reachable - without any errors / warnings!
> > I've enabled printk (echo 8 > /proc/sys/kernel/printk) and still no console output.
> 
> The device shows up on the bus, the bridge primary/secondary bus numbers
> look good, and the secondary memory address range on the bridge looks
> properly programmed.  It all looks good to me.
> 
> What do you mean by 'not reachable'?  I'm guessing that you're trying
> to access the memory BAR on the 01:08.0 device directly from userland
> by reading from the e000_0000 - e3ff_ffff address range from /dev/mem
> and only getting 0xffffffff back because you don't have an actual kernel
> driver for this FPGA board and thus you're not calling pci_enable_device()
> on your device, causing MEM/IO decoding not to have been enabled on
> the device as seems to be the case in your dump?
> 

I've tested once again (I really forgotten to load my kernel module at the last mail, sorry again!).
Here you can see the result (with a non working driver!);

sh-3.2# uname -a
Linux DB88FXX81 2.6.32-rc6-00014-g92bdfb9-dirty #3 PREEMPT Wed Nov 11 15:10:49 CET 2009 armv5tel GNU/Linux
sh-3.2# cat /proc/iomem
00000000-0fffffff : System RAM
  00026000-00464fff : Kernel text
  0048a000-00579c7b : Kernel data
10000000-1fffffff : System RAM
e0000000-e7ffffff : PCIe Memory Space
  e0000000-e3ffffff : PCI Bus 0000:01
    e0000000-e3ffffff : 0000:01:08.0
      e0000000-e00fffff : mv_video MMIO
f1010300-f101030f : rtc-mv
  f1010300-f101030f : rtc-mv
f1011000-f101101f : mv64xxx_i2c.0
  f1011000-f101101f : mv64xxx_i2c adapter
f1012000-f10120ff : serial8250.0
  f1012000-f101201f : serial
f1030000-f103ffff : regs
f1050000-f1050fff : orion-ehci.0
  f1050000-f1050fff : ehci_hcd
f1060800-f10608ff : xor 0 low
f1060900-f10609ff : xor 1 low
f1060a00-f1060aff : xor 0 high
f1060b00-f1060bff : xor 1 high
f1072000-f1073fff : ge00 base
f1080000-f1084fff : sata base
f1090000-f10903ff : mvsdio
  f1090000-f10903ff : mvsdio
f3000000-f30003ff : orion_nand
f4000000-f40007ff : sram
sh-3.2# lspci -vv
00:00.0 Class 0580: Device 11ab:6281 (rev 02)
        Subsystem: Device 11ab:11ab
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 9
        Region 0: Memory@<ignored> (64-bit, prefetchable)
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [60] Express (v1) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <256ns, L1 <1us
                        ExtTag- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <256ns, L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
        Capabilities: [100] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-

00:01.0 Class 0604: Device 11ab:2211 (rev 01)
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00004fff
        Memory behind bridge: e0000000-e3ffffff
        Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort+ >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
                Bridge: PM- B3+
        Capabilities: [48] Express (v1) PCI/PCI-X Bridge, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Latency L0 <256ns, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr+ BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-

01:08.0 Class ff00: Device 1731:0101 (rev 10)
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap- 66MHz- UDF- FastB2B- ParErr+ DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Region 0: Memory at e0000000 (32-bit, non-prefetchable) [size=64M]
        Region 1: I/O ports@<unassigned> [disabled]
        Kernel driver in use: ArtistaNET-III frame buffer driver

sh-3.2#
sh-3.2# lsmod
mv_video 3616 0 - Live 0xbf000000
sh-3.2#

Again, many thanks for helping!

Dieter


> For what it's worth, I have various mv78xx0 and Kirkwood boards with
> 88SB2211 PCIe-to-PCI bridges on them, and they all work fine, and
> the devices behind those bridges do too.
> 
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>