* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation [not found] <CAFJTrDt7OrtE4B52419F2KGbmXeLR9NEcgF6yMTMUBe5vdKwfg@mail.gmail.com> @ 2014-03-26 17:06 ` Thomas Petazzoni 2014-03-26 17:18 ` Jason Gunthorpe 0 siblings, 1 reply; 13+ messages in thread From: Thomas Petazzoni @ 2014-03-26 17:06 UTC (permalink / raw) To: linux-arm-kernel Dear Matthew Minter, Thanks for your report. I'm adding Jason Gunthorpe in Cc, as his insights are always very helpful when investigating PCIe issues on Marvell platforms. Best regards, Thomas On Wed, 26 Mar 2014 16:50:38 +0000, Matthew Minter wrote: > Using a custom board based around the Armada XP 230 (that's the dual core > one) we are trying to talk to some PCIe devices via a PLX PCIe switch. > However I have been able to reproduce these same results using an Armada XP > GP dev board as to show it is not a board defect causing the issue. > > We are trying to use a PCIe based ahci SSD to store some data on this board > (actually an m.2 card, but electronically identical to PCIe). This is > working well, we can see the switch, its upstream and downstream ports and > the SSD itself in the /sys/bus/pci file-system, we can even mount the SSD > and store and read data on it. This all seems fine. We are using kernel > version 3.14.0-rc7, I considered trying a more upstream version but could > not see any applicable patches in next, or in the mvebu repo) > > However the issues arrive when we try and use the hotplug features of the > board we are using (the same applies with the GP dev board and a PLX dev > board connected to it). If the SSD is connected at boot time, we can safely > hot-swap it but following the procedure: unbind the ahci driver, remove it > from the PCIe map using /sys/bus/pci/device/xxxxx/remove, switch off the > slot's power supply, isolate the slot electronically, perform the swap, > un-isolate the slot, re-power the slot, rescan the pci bus. This also works > great and we can talk to the new SSD. > > If the SSD is not plugged at boot however, this does not work properly > (This is definitely a kernel, not boot-loader issue as we can disconnect > the SSD while the bootloader is running and still have it work as long as > it gets plugged in before the kernel's PCIe initialization). We perform the > same sequence, this appears at first to be ok, however after rescanning the > pci bus, dmesg shows the following errors (0000:01:00.0 is the switch > upstream port, 0000:02:02.0 is the switch downstream port and 0000:04:00.0 > is the SSD): > > [ 83.135273] pcieport 0000:01:00.0: BAR 14: can't assign mem (size > 0x100000) > [ 83.135282] pcieport 0000:01:00.0: BAR 15: can't assign mem pref (size > 0x100000) > [ 83.135292] pcieport 0000:02:02.0: BAR 14: can't assign mem (size > 0x100000) > [ 83.135300] pcieport 0000:02:02.0: BAR 15: can't assign mem pref (size > 0x100000) > [ 83.135309] pci 0000:04:00.0: BAR 6: can't assign mem pref (size 0x10000) > [ 83.135317] pci 0000:04:00.0: BAR 5: can't assign mem (size 0x2000) > > Then when we try and attach the ahci driver, we get (predictably): > > [ 83.235749] ahci 0000:04:00.0: version 3.0 > [ 83.235802] ahci: probe of 0000:04:00.0 failed with error -22 > > It appears to me from a glance (though I have not examined the code paths > in detail) that performing a PCI rescan is not completely reinitializing > the bus, specifically it is not performing any re-allocation of the PCI bar > memory spaces which means the rescanned devices will not be able to connect > properly. Do we need to call mvebu_pcie_setup_wins again or some other > function in the kernel after the rescan? I am a little surprised that the > kernel is not reconfiguring the BARs/windows after a rescan to fit the > changed config. Is there any way I can make this work? Any advice here > would be greatly appreciated, I would also be happy to try and write some > patches if it is really just the need to call the window and bar > initialization functions again after a rescan but would greatly appreciate > any opinions first. > > Info dump ahead: > > Boot log with SSD connected: > > Uncompressing Linux... done, booting the kernel. > Booting Linux on physical CPU 0x0 > Linux version 3.14.0-rc7 (root at orangebox) (gcc version 4.7.3 (Gentoo > 4.7.3-r1 p1.3, pie-0.5.5) ) #9 SMP PREEMPT Mon Mar 24 17:47:43 GMT 2014 > CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d > CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache > Machine model: Xyratex Porter-v1 > bootconsole [earlycon0] enabled > Truncating RAM at 00000000-3fffffff to -2f7fffff (vmalloc region overlap). > Memory policy: Data cache writealloc > PERCPU: Embedded 7 pages/cpu @ef1ee000 s5952 r8192 d14528 u32768 > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 193040 > Kernel command line: console=ttyS0,115200 earlyprintk=ttys0 rw pm_disable > mtdparts=armada-nand:8m(boot),8m(kernel),-(rootfs) ubi.mtd=2 > root=ubi0:rootfs rootfstype=ubifs > PID hash table entries: 4096 (order: 2, 16384 bytes) > Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) > Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) > Memory: 766932K/778240K available (2979K kernel code, 197K rwdata, 912K > rodata, 137K init, 56K bss, 11308K reserved) > Virtual kernel memory layout: > vector : 0xffff0000 - 0xffff1000 ( 4 kB) > fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB) > vmalloc : 0xf0000000 - 0xff000000 ( 240 MB) > lowmem : 0xc0000000 - 0xef800000 ( 760 MB) > modules : 0xbf000000 - 0xc0000000 ( 16 MB) > .text : 0xc0008000 - 0xc03d4fbc (3892 kB) > .init : 0xc03d5000 - 0xc03f7740 ( 138 kB) > .data : 0xc03f8000 - 0xc04296d0 ( 198 kB) > .bss : 0xc04296dc - 0xc04378b8 ( 57 kB) > Preemptible hierarchical RCU implementation. > Dump stacks of tasks blocking RCU-preempt GP. > RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2. > RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 > NR_IRQS:16 nr_irqs:16 16 > sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798691800ns > Initializing Coherency fabric > Aurora cache controller enabled > l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1024 kB > Console: colour dummy device 80x30 > Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256) > pid_max: default: 32768 minimum: 301 > Mount-cache hash table entries: 512 > CPU: Testing write buffer coherency: ok > CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 > Setting up static identity map for 0x2ef2f0 - 0x2ef324 > Initializing Power Management Service Unit > Booting CPU 1 > CPU1: Booted secondary processor > CPU1: thread -1, cpu 1, socket 0, mpidr 80000001 > Brought up 2 CPUs > SMP: Total of 2 processors activated. > CPU: All CPU(s) started in SVC mode. > devtmpfs: initialized > VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6 > mvebu-soc-id: MVEBU SoC ID=0x7826, Rev=0x2 > pinctrl core: initialized pinctrl subsystem > DMA: preallocated 256 KiB pool for atomic coherent allocations > irq: Cannot allocate irq_descs @ IRQ32, assuming pre-allocated > irq: Cannot allocate irq_descs @ IRQ67, assuming pre-allocated > bio: create slab <bio-0> at 0 > SCSI subsystem initialized > usbcore: registered new interface driver usbfs > usbcore: registered new interface driver hub > usbcore: registered new device driver usb > Switched to clocksource armada_370_xp_clocksource > futex hash table entries: 512 (order: 3, 32768 bytes) > msgmni has been set to 1497 > Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) > io scheduler noop registered > io scheduler deadline registered > io scheduler cfq registered (default) > armada-xp-pinctrl f1018000.pinctrl: registered pinctrl driver > mvebu-pcie pcie-controller.2: PCI host bridge to bus 0000:00 > pci_bus 0000:00: root bus resource [io 0x1000-0xfffff] > pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe8000000] > pci_bus 0000:00: root bus resource [bus 00-ff] > PCI: bus0: Fast back to back transfers disabled > pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus1: Fast back to back transfers disabled > pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus2: Fast back to back transfers disabled > pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring > pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus3: Fast back to back transfers enabled > PCI: bus4: Fast back to back transfers disabled > PCI: bus5: Fast back to back transfers enabled > pci 0000:00:01.0: BAR 14: assigned [mem 0xe0000000-0xe02fffff] > pci 0000:01:00.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff] > pci 0000:01:00.0: BAR 15: assigned [mem 0xe0100000-0xe01fffff pref] > pci 0000:01:00.0: BAR 0: assigned [mem 0xe0200000-0xe021ffff] > pci 0000:02:02.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff] > pci 0000:02:02.0: BAR 15: assigned [mem 0xe0100000-0xe01fffff pref] > pci 0000:02:01.0: PCI bridge to [bus 03] > pci 0000:04:00.0: BAR 6: assigned [mem 0xe0100000-0xe010ffff pref] > pci 0000:04:00.0: BAR 5: assigned [mem 0xe0000000-0xe0001fff] > pci 0000:02:02.0: PCI bridge to [bus 04] > pci 0000:02:02.0: bridge window [mem 0xe0000000-0xe00fffff] > pci 0000:02:02.0: bridge window [mem 0xe0100000-0xe01fffff pref] > pci 0000:02:03.0: PCI bridge to [bus 05] > pci 0000:01:00.0: PCI bridge to [bus 02-05] > pci 0000:01:00.0: bridge window [mem 0xe0000000-0xe00fffff] > pci 0000:01:00.0: bridge window [mem 0xe0100000-0xe01fffff pref] > pci 0000:00:01.0: PCI bridge to [bus 01-05] > pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe02fffff] > PCI: enabling device 0000:00:01.0 (0140 -> 0143) > PCI: enabling device 0000:01:00.0 (0140 -> 0143) > PCI: enabling device 0000:02:01.0 (0140 -> 0143) > PCI: enabling device 0000:02:02.0 (0140 -> 0143) > PCI: enabling device 0000:02:03.0 (0140 -> 0143) > mv_xor f1060900.xor: Marvell shared XOR driver > mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) > mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) > mv_xor f10f0900.xor: Marvell shared XOR driver > mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) > mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) > Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled > f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 19, base_baud = 15625000) > is a 16550A > console [ttyS0] enabled > console [ttyS0] enabled > bootconsole [earlycon0] disabled > bootconsole [earlycon0] disabled > f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 20, base_baud = 15625000) > is a 16550A > mtip32xx Version 1.3.0 > pxa3xx-nand f10d0000.nand: This platform can't do DMA on this device > nand: device found, Manufacturer ID: 0x2c, Chip ID: 0x38 > nand: Micron MT29F8G08ABABAWP > nand: 1024MiB, SLC, page size: 4096, OOB size: 224 > Bad block table found at page 262016, version 0x01 > Bad block table found at page 261888, version 0x01 > 3 ofpart partitions found on MTD device pxa3xx_nand-0 > Creating 3 MTD partitions on "pxa3xx_nand-0": > 0x000000000000-0x000000800000 : "boot" > 0x000000800000-0x000001000000 : "kernel" > 0x000001000000-0x000040000000 : "rootfs" > ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver > ehci-pci: EHCI PCI platform driver > ehci-orion: EHCI orion driver > orion-ehci f1050000.usb: EHCI Host Controller > orion-ehci f1050000.usb: new USB bus registered, assigned bus number 1 > orion-ehci f1050000.usb: irq 26, io mem 0xf1050000 > orion-ehci f1050000.usb: USB 2.0 started, EHCI 1.00 > hub 1-0:1.0: USB hub found > hub 1-0:1.0: 1 port detected > usbcore: registered new interface driver usb-storage > rtc-mv f1010300.rtc: rtc core: registered f1010300.rtc as rtc0 > UBI: attaching mtd2 to ubi0 > random: nonblocking pool is initialized > UBI: scanning is finished > UBI: attached mtd2 (name "rootfs", size 1008 MiB) to ubi0 > UBI: PEB size: 524288 bytes (512 KiB), LEB size: 516096 bytes > UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096 > UBI: VID header offset: 4096 (aligned 4096), data offset: 8192 > UBI: good PEBs: 2008, bad PEBs: 8, corrupted PEBs: 0 > UBI: user volume: 1, internal volumes: 1, max. volumes count: 128 > UBI: max/mean erase counter: 2/1, WL threshold: 4096, image sequence > number: 3531581739 > UBI: available PEBs: 0, total reserved PEBs: 2008, PEBs reserved for bad > PEB handling: 32 > UBI: background thread "ubi_bgt0d" started, PID 885 > rtc-mv f1010300.rtc: setting system clock to 2000-07-04 13:58:42 UTC > (962719122) > UBIFS: background thread "ubifs_bgt0_0" started, PID 887 > UBIFS: recovery needed > UBIFS: recovery completed > UBIFS: mounted UBI device 0, volume 0, name "rootfs" > UBIFS: LEB size: 516096 bytes (504 KiB), min./max. I/O unit sizes: 4096 > bytes/4096 bytes > UBIFS: FS size: 1012064256 bytes (965 MiB, 1961 LEBs), journal size > 33546240 bytes (31 MiB, 65 LEBs) > UBIFS: reserved for root: 4952683 bytes (4836 KiB) > UBIFS: media format: w4/r0 (latest is w4/r0), UUID > 8BD77AAC-E543-41A2-842F-7011ADC039E0, small LPT model > VFS: Mounted root (ubifs filesystem) on device 0:10. > devtmpfs: mounted > Freeing unused kernel memory: 136K (c03d5000 - c03f7000) > > > Boot log with SSD disconnected: > > Uncompressing Linux... done, booting the kernel. > Booting Linux on physical CPU 0x0 > Linux version 3.14.0-rc7 (root at orangebox) (gcc version 4.7.3 (Gentoo > 4.7.3-r1 p1.3, pie-0.5.5) ) #9 SMP PREEMPT Mon Mar 24 17:47:43 GMT 2014 > CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d > CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache > Machine model: Xyratex Porter-v1 > bootconsole [earlycon0] enabled > Truncating RAM at 00000000-3fffffff to -2f7fffff (vmalloc region overlap). > Memory policy: Data cache writealloc > PERCPU: Embedded 7 pages/cpu @ef1ee000 s5952 r8192 d14528 u32768 > Built 1 zonelists in Zone order, mobility grouping on. Total pages: 193040 > Kernel command line: console=ttyS0,115200 earlyprintk=ttys0 rw pm_disable > mtdparts=armada-nand:8m(boot),8m(kernel),-(rootfs) ubi.mtd=2 > root=ubi0:rootfs rootfstype=ubifs > PID hash table entries: 4096 (order: 2, 16384 bytes) > Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) > Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) > Memory: 766932K/778240K available (2979K kernel code, 197K rwdata, 912K > rodata, 137K init, 56K bss, 11308K reserved) > Virtual kernel memory layout: > vector : 0xffff0000 - 0xffff1000 ( 4 kB) > fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB) > vmalloc : 0xf0000000 - 0xff000000 ( 240 MB) > lowmem : 0xc0000000 - 0xef800000 ( 760 MB) > modules : 0xbf000000 - 0xc0000000 ( 16 MB) > .text : 0xc0008000 - 0xc03d4fbc (3892 kB) > .init : 0xc03d5000 - 0xc03f7740 ( 138 kB) > .data : 0xc03f8000 - 0xc04296d0 ( 198 kB) > .bss : 0xc04296dc - 0xc04378b8 ( 57 kB) > Preemptible hierarchical RCU implementation. > Dump stacks of tasks blocking RCU-preempt GP. > RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2. > RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 > NR_IRQS:16 nr_irqs:16 16 > sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798691800ns > Initializing Coherency fabric > Aurora cache controller enabled > l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1024 kB > Console: colour dummy device 80x30 > Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256) > pid_max: default: 32768 minimum: 301 > Mount-cache hash table entries: 512 > CPU: Testing write buffer coherency: ok > CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 > Setting up static identity map for 0x2ef2f0 - 0x2ef324 > Initializing Power Management Service Unit > Booting CPU 1 > CPU1: Booted secondary processor > CPU1: thread -1, cpu 1, socket 0, mpidr 80000001 > Brought up 2 CPUs > SMP: Total of 2 processors activated. > CPU: All CPU(s) started in SVC mode. > devtmpfs: initialized > VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6 > mvebu-soc-id: MVEBU SoC ID=0x7826, Rev=0x2 > pinctrl core: initialized pinctrl subsystem > DMA: preallocated 256 KiB pool for atomic coherent allocations > irq: Cannot allocate irq_descs @ IRQ32, assuming pre-allocated > irq: Cannot allocate irq_descs @ IRQ67, assuming pre-allocated > bio: create slab <bio-0> at 0 > SCSI subsystem initialized > usbcore: registered new interface driver usbfs > usbcore: registered new interface driver hub > usbcore: registered new device driver usb > Switched to clocksource armada_370_xp_clocksource > futex hash table entries: 512 (order: 3, 32768 bytes) > msgmni has been set to 1497 > Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) > io scheduler noop registered > io scheduler deadline registered > io scheduler cfq registered (default) > armada-xp-pinctrl f1018000.pinctrl: registered pinctrl driver > mvebu-pcie pcie-controller.2: PCI host bridge to bus 0000:00 > pci_bus 0000:00: root bus resource [io 0x1000-0xfffff] > pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe8000000] > pci_bus 0000:00: root bus resource [bus 00-ff] > PCI: bus0: Fast back to back transfers disabled > pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus1: Fast back to back transfers disabled > pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus2: Fast back to back transfers disabled > pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring > pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring > pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring > PCI: bus3: Fast back to back transfers enabled > PCI: bus4: Fast back to back transfers enabled > PCI: bus5: Fast back to back transfers enabled > pci 0000:00:01.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff] > pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe001ffff] > pci 0000:02:01.0: PCI bridge to [bus 03] > pci 0000:02:02.0: PCI bridge to [bus 04] > pci 0000:02:03.0: PCI bridge to [bus 05] > pci 0000:01:00.0: PCI bridge to [bus 02-05] > pci 0000:00:01.0: PCI bridge to [bus 01-05] > pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe00fffff] > PCI: enabling device 0000:00:01.0 (0140 -> 0143) > PCI: enabling device 0000:01:00.0 (0140 -> 0143) > PCI: enabling device 0000:02:01.0 (0140 -> 0143) > PCI: enabling device 0000:02:02.0 (0140 -> 0143) > PCI: enabling device 0000:02:03.0 (0140 -> 0143) > mv_xor f1060900.xor: Marvell shared XOR driver > mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) > mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) > mv_xor f10f0900.xor: Marvell shared XOR driver > mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) > mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) > Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled > f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 19, base_baud = 15625000) > is a 16550A > console [ttyS0] enabled > console [ttyS0] enabled > bootconsole [earlycon0] disabled > bootconsole [earlycon0] disabled > f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 20, base_baud = 15625000) > is a 16550A > mtip32xx Version 1.3.0 > pxa3xx-nand f10d0000.nand: This platform can't do DMA on this device > nand: device found, Manufacturer ID: 0x2c, Chip ID: 0x38 > nand: Micron MT29F8G08ABABAWP > nand: 1024MiB, SLC, page size: 4096, OOB size: 224 > Bad block table found at page 262016, version 0x01 > Bad block table found at page 261888, version 0x01 > 3 ofpart partitions found on MTD device pxa3xx_nand-0 > Creating 3 MTD partitions on "pxa3xx_nand-0": > 0x000000000000-0x000000800000 : "boot" > 0x000000800000-0x000001000000 : "kernel" > 0x000001000000-0x000040000000 : "rootfs" > ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver > ehci-pci: EHCI PCI platform driver > ehci-orion: EHCI orion driver > orion-ehci f1050000.usb: EHCI Host Controller > orion-ehci f1050000.usb: new USB bus registered, assigned bus number 1 > orion-ehci f1050000.usb: irq 26, io mem 0xf1050000 > orion-ehci f1050000.usb: USB 2.0 started, EHCI 1.00 > hub 1-0:1.0: USB hub found > hub 1-0:1.0: 1 port detected > usbcore: registered new interface driver usb-storage > rtc-mv f1010300.rtc: rtc core: registered f1010300.rtc as rtc0 > UBI: attaching mtd2 to ubi0 > random: nonblocking pool is initialized > UBI: scanning is finished > UBI: attached mtd2 (name "rootfs", size 1008 MiB) to ubi0 > UBI: PEB size: 524288 bytes (512 KiB), LEB size: 516096 bytes > UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096 > UBI: VID header offset: 4096 (aligned 4096), data offset: 8192 > UBI: good PEBs: 2008, bad PEBs: 8, corrupted PEBs: 0 > UBI: user volume: 1, internal volumes: 1, max. volumes count: 128 > UBI: max/mean erase counter: 2/1, WL threshold: 4096, image sequence > number: 3531581739 > UBI: available PEBs: 0, total reserved PEBs: 2008, PEBs reserved for bad > PEB handling: 32 > UBI: background thread "ubi_bgt0d" started, PID 884 > rtc-mv f1010300.rtc: setting system clock to 2000-07-04 13:58:35 UTC > (962719115) > UBIFS: background thread "ubifs_bgt0_0" started, PID 886 > UBIFS: recovery needed > UBIFS: recovery completed > UBIFS: mounted UBI device 0, volume 0, name "rootfs" > UBIFS: LEB size: 516096 bytes (504 KiB), min./max. I/O unit sizes: 4096 > bytes/4096 bytes > UBIFS: FS size: 1012064256 bytes (965 MiB, 1961 LEBs), journal size > 33546240 bytes (31 MiB, 65 LEBs) > UBIFS: reserved for root: 4952683 bytes (4836 KiB) > UBIFS: media format: w4/r0 (latest is w4/r0), UUID > 8BD77AAC-E543-41A2-842F-7011ADC039E0, small LPT model > VFS: Mounted root (ubifs filesystem) on device 0:10. > devtmpfs: mounted > Freeing unused kernel memory: 136K (c03d5000 - c03f7000) > > Full dmesg output after failed rescan and ahci bind: > > [ 46.224192] pci_bus 0000:02: scanning bus > [ 46.224259] pcieport 0000:02:01.0: scanning [bus 03-03] behind bridge, > pass 0 > [ 46.224275] pcieport 0000:02:02.0: scanning [bus 04-04] behind bridge, > pass 0 > [ 46.224290] pcieport 0000:02:03.0: scanning [bus 05-05] behind bridge, > pass 0 > [ 46.224305] pcieport 0000:02:01.0: scanning [bus 00-00] behind bridge, > pass 1 > [ 46.224318] pci_bus 0000:03: scanning bus > [ 46.224328] pci_bus 0000:03: bus scan returning with max=03 > [ 46.224340] pci_bus 0000:03: busn_res: [bus 03] end is updated to 03 > [ 46.224353] pcieport 0000:02:02.0: scanning [bus 00-00] behind bridge, > pass 1 > [ 46.224367] pci_bus 0000:04: scanning bus > [ 46.224410] pci 0000:04:00.0: [144d:a800] type 00 class 0x010601 > [ 46.224495] pci 0000:04:00.0: reg 0x24: [mem 0x00000000-0x00001fff] > [ 46.224515] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref] > [ 46.224536] pci 0000:04:00.0: calling pci_fixup_ide_bases+0x0/0x58 > [ 46.224660] pci 0000:04:00.0: PME# supported from D3hot D3cold > [ 46.224672] pci 0000:04:00.0: PME# disabled > [ 46.225090] pci_bus 0000:04: bus scan returning with max=04 > [ 46.225101] pci_bus 0000:04: busn_res: [bus 04] end is updated to 04 > [ 46.225116] pcieport 0000:02:03.0: scanning [bus 00-00] behind bridge, > pass 1 > [ 46.225130] pci_bus 0000:05: scanning bus > [ 46.225140] pci_bus 0000:05: bus scan returning with max=05 > [ 46.225148] pci_bus 0000:05: busn_res: [bus 05] end is updated to 05 > [ 46.225159] pci_bus 0000:02: bus scan returning with max=05 > [ 46.225167] pci_bus 0000:02: busn_res: [bus 02-05] end is updated to 05 > [ 46.225178] pci_bus 0000:01: bus scan returning with max=05 > [ 46.225186] pci_bus 0000:01: busn_res: [bus 01-05] end is updated to 05 > [ 46.225195] pci_bus 0000:00: bus scan returning with max=05 > [ 46.225267] pcieport 0000:01:00.0: BAR 14: can't assign mem (size > 0x100000) > [ 46.225277] pcieport 0000:01:00.0: BAR 15: can't assign mem pref (size > 0x100000) > [ 46.225287] pcieport 0000:02:02.0: BAR 14: can't assign mem (size > 0x100000) > [ 46.225295] pcieport 0000:02:02.0: BAR 15: can't assign mem pref (size > 0x100000) > [ 46.225304] pci 0000:04:00.0: BAR 6: can't assign mem pref (size 0x10000) > [ 46.225312] pci 0000:04:00.0: BAR 5: can't assign mem (size 0x2000) > [ 46.325621] ahci 0000:04:00.0: no of_node; not parsing pinctrl DT > [ 46.325656] ahci 0000:04:00.0: version 3.0 > [ 46.325707] ahci: probe of 0000:04:00.0 failed with error -22 > -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-03-26 17:06 ` Armada XP (mvebu) PCIe memory (BAR/window) re-allocation Thomas Petazzoni @ 2014-03-26 17:18 ` Jason Gunthorpe 2014-03-31 14:17 ` Matthew Minter 0 siblings, 1 reply; 13+ messages in thread From: Jason Gunthorpe @ 2014-03-26 17:18 UTC (permalink / raw) To: linux-arm-kernel On Wed, Mar 26, 2014 at 06:06:34PM +0100, Thomas Petazzoni wrote: > Thanks for your report. I'm adding Jason Gunthorpe in Cc, as his > insights are always very helpful when investigating PCIe issues on > Marvell platforms. To me, this looks like standard problems with hot plug - unrelated to Marvell. Actually supporting the full generality of hot plug requires someone pre-allocate and reserve bridge window address space. On x86 the BIOS knows which slots are hot plug capable and ensures that address space is set aside for that slot when doing window assignment. Linux will just keep the BIOS assignment and when the hot plug happens the extra space is used. What you seem to be seeing is that Linux does an optimal allocation at boot, and the PCI-E switch port that connects to your hot plug slot is not allocated any spare address space for a future device. During rescan there is no free address space to assign to the new device so everything blows up. I know there are some provisions in the PCI core to solve this issue, I know on x86 the BIOS plays a role, but I'm not familiar enough with the details to give you an exact path to solve your problem, other than to say, at first boot you need to arrange for there to be extra address space available. A really simple dumb hack that might work is to change the order of the PCI-E ports on the Armada (I think you can do this through the DT). What you want is to place the port connected to the PLX last. The goal here is to have it do address assignment last, so it will get the last chunk of address space, and will have room to grow into unused space, instead of being blocked by another port. Regards, Jason ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-03-26 17:18 ` Jason Gunthorpe @ 2014-03-31 14:17 ` Matthew Minter 2014-04-01 16:41 ` Jason Gunthorpe 0 siblings, 1 reply; 13+ messages in thread From: Matthew Minter @ 2014-03-31 14:17 UTC (permalink / raw) To: linux-arm-kernel Hi guys, Just to give an update of what I have tried. I noticed that there was a PCI quirk available for a similar PLX chip which forced the kernel to see the slots as hot-plug slots (unfortunately some PLX chips falsely report themselves to be non hotplug capable bridges from my understanding). I tried extending this quirk to run on my chip also. This significantly changed the results in that along with the use of pci=hpmemsize=xM I managed to force the chip to pre-allocate some memory for the hotplug slot. Specifically I added this line: DECLARE_PCI_FIXUP_HEADER(0x10b5, 0x8617, quirk_hotplug_bridge); to linux-3.14-rc7/drivers/pci/quirks.c However the issue is that when trying to then rescan the PCI bus I have a crash (I am not entirely sure if it is an oops or something else) but it displays: Internal error: : 1008 [#1] PREEMPT SMP ARM then dumps a stack trace and my shell dies. From then on attempting to perform any PCI operation will make the system hang and need a reboot. I am not sure why this error occurs but think it may relate somehow to the chips memory window allocations, however it could be something completely different. I am currently working on a way to modify the scan order as Jason as suggested, as soon as I have worked out exactly how the scan order is determined I will try modifying the DT or any code needed to force the port with the PLX chip to be scanned last. However I first need to read some more of the code to understand what it orders on, eg: order in DT, bus number, memory address. Thanks again guys, Matt In case anyone is interested, here is the dump I get when i have a crash with pre-allocated memory: Internal error: : 1008 [#1] PREEMPT SMP ARM Modules linked in: CPU: 0 PID: 1746 Comm: bash Not tainted 3.14.0-rc7 #18 task: eed72840 ti: ee794000 task.ti: ee794000 PC is at ahci_enable_ahci+0x10/0x7c LR is at ahci_save_initial_config+0x28/0x24c pc : [<c02510d8>] lr : [<c02512cc>] psr: 60070013 sp : ee795cb0 ip : ee795cd0 fp : ee795ccc r10: 00000000 r9 : 00000000 r8 : 00000000 r7 : ee73b068 r6 : f0460000 r5 : ee6c9290 r4 : f0460004 r3 : 00000000 r2 : 00000000 r1 : ee6c9290 r0 : f0460000 Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user Control: 10c53c7d Table: 2e7e406a DAC: 00000015 Process bash (pid: 1746, stack limit = 0xee794240) Stack: (0xee795cb0 to 0xee796000) 5ca0: ee73b000 ee6c9290 f0460000 ee73b068 5cc0: ee795cfc ee795cd0 c02512cc c02510d4 00000000 ee73b000 ee73b068 ee6c9290 5ce0: 00000000 00000000 ee776c88 00000005 ee795d64 ee795d00 c024e7c4 c02512b0 5d00: ee795d34 ee795d10 c00fa3ec ee4355a0 c043f20d ee795d1c 00000000 00060082 5d20: 00000000 0000001f 00000000 0000007f c04d3a04 00000000 ee795d64 ee73b068 5d40: ee73b000 c04d36f4 00000000 c04d36c0 ee776c88 ee776c80 ee795d8c ee795d68 5d60: c01dc224 c024e44c c01dc1c4 ee73b068 c04d36f4 00000000 c0218c0c 00000002 5d80: ee795db4 ee795d90 c0218968 c01dc1d0 ee73b068 c0218c0c ee795db4 c04d36f4 5da0: ee73b068 ee73b068 ee795dcc ee795db8 c0218c3c c0218844 00000000 ee795dd0 5dc0: ee795df4 ee795dd0 c0216d24 c0218c18 eec6c7b0 ee487538 ee73b068 ee73b09c 5de0: eee5e600 ee795f78 ee795e14 ee795df8 c02187c4 c0216cdc 00000001 ee73b000 5e00: eee5e614 eee5e600 ee795e2c ee795e18 c01d4804 c0218760 00000000 ee73b000 5e20: ee795e4c ee795e30 c01d4990 c01d47d4 00000004 eeec9400 eee5ea14 eee5ea00 5e40: ee795e6c ee795e50 c01d49e0 c01d4964 00000004 eeec9c00 eee5ec14 eee5ec00 5e60: ee795e8c ee795e70 c01d49e0 c01d4964 00000004 eedd3000 eee5ee14 eee5ee00 5e80: ee795eac ee795e90 c01d49e0 c01d4964 ee795e88 eee5ee00 00000005 ee776380 5ea0: ee795ec4 ee795eb0 c037ceb4 c01d4964 eee5ee00 00000002 ee795ee4 ee795ec8 5ec0: c01dd170 c037ce98 00000041 00000001 eec69a68 ee776380 ee795ef4 ee795ee8 5ee0: c0216af4 c01dd134 ee795f0c ee795ef8 c00f6a4c c0216ad8 00000000 00000000 5f00: ee795f44 ee795f10 c00f9cd0 c00f6a14 00000000 00000000 c00a1430 eee47e00 5f20: 00000002 b6f9e000 ee795f78 00000002 ee794000 b6f9e000 ee795f74 ee795f48 5f40: c00a3830 c00f9be8 ee795f74 ee795f58 00000000 00000000 eee47e00 eee47e00 5f60: 00000002 b6f9e000 ee795fa4 ee795f78 c00a3bc8 c00a376c 00000000 00000000 5f80: 00000002 b6f9e000 b6edda78 00000004 c000ed84 00000000 00000000 ee795fa8 5fa0: c000ec00 c00a3b90 00000002 b6f9e000 00000001 b6f9e000 00000002 00000000 5fc0: 00000002 b6f9e000 b6edda78 00000004 00000002 00000002 b6f9e000 00000000 5fe0: 00000000 beb9da1c b6e1d390 b6e6eaac 60070010 00000001 00000000 00000000 Backtrace: [<c02510c8>] (ahci_enable_ahci) from [<c02512cc>] (ahci_save_initial_config+0x28/0x24c) r7:ee73b068 r6:f0460000 r5:ee6c9290 r4:ee73b000 [<c02512a4>] (ahci_save_initial_config) from [<c024e7c4>] (ahci_init_one+0x384/0x850) r10:00000005 r9:ee776c88 r8:00000000 r7:00000000 r6:ee6c9290 r5:ee73b068 r4:ee73b000 r3:00000000 [<c024e440>] (ahci_init_one) from [<c01dc224>] (pci_device_probe+0x60/0xa4) r10:ee776c80 r9:ee776c88 r8:c04d36c0 r7:00000000 r6:c04d36f4 r5:ee73b000 r4:ee73b068 [<c01dc1c4>] (pci_device_probe) from [<c0218968>] (driver_probe_device+0x130/0x340) r8:00000002 r7:c0218c0c r6:00000000 r5:c04d36f4 r4:ee73b068 r3:c01dc1c4 [<c0218838>] (driver_probe_device) from [<c0218c3c>] (__device_attach+0x30/0x4c) r6:ee73b068 r5:ee73b068 r4:c04d36f4 [<c0218c0c>] (__device_attach) from [<c0216d24>] (bus_for_each_drv+0x54/0x9c) r5:ee795dd0 r4:00000000 [<c0216cd0>] (bus_for_each_drv) from [<c02187c4>] (device_attach+0x70/0x88) r7:ee795f78 r6:eee5e600 r5:ee73b09c r4:ee73b068 [<c0218754>] (device_attach) from [<c01d4804>] (pci_bus_add_device+0x3c/0x68) r6:eee5e600 r5:eee5e614 r4:ee73b000 r3:00000001 [<c01d47c8>] (pci_bus_add_device) from [<c01d4990>] (pci_bus_add_devices+0x38/0x9c) r4:ee73b000 r3:00000000 [<c01d4958>] (pci_bus_add_devices) from [<c01d49e0>] (pci_bus_add_devices+0x88/0x9c) r6:eee5ea00 r5:eee5ea14 r4:eeec9400 r3:00000004 [<c01d4958>] (pci_bus_add_devices) from [<c01d49e0>] (pci_bus_add_devices+0x88/0x9c) r6:eee5ec00 r5:eee5ec14 r4:eeec9c00 r3:00000004 [<c01d4958>] (pci_bus_add_devices) from [<c01d49e0>] (pci_bus_add_devices+0x88/0x9c) r6:eee5ee00 r5:eee5ee14 r4:eedd3000 r3:00000004 [<c01d4958>] (pci_bus_add_devices) from [<c037ceb4>] (pci_rescan_bus+0x28/0x30) r6:ee776380 r5:00000005 r4:eee5ee00 r3:ee795e88 [<c037ce8c>] (pci_rescan_bus) from [<c01dd170>] (bus_rescan_store+0x48/0x70) r5:00000002 r4:eee5ee00 [<c01dd128>] (bus_rescan_store) from [<c0216af4>] (bus_attr_store+0x28/0x34) r5:ee776380 r4:eec69a68 [<c0216acc>] (bus_attr_store) from [<c00f6a4c>] (sysfs_kf_write+0x44/0x50) [<c00f6a08>] (sysfs_kf_write) from [<c00f9cd0>] (kernfs_fop_write+0xf4/0x148) r5:00000000 r4:00000000 [<c00f9bdc>] (kernfs_fop_write) from [<c00a3830>] (vfs_write+0xd0/0x17c) r10:b6f9e000 r9:ee794000 r8:00000002 r7:ee795f78 r6:b6f9e000 r5:00000002 r4:eee47e00 [<c00a3760>] (vfs_write) from [<c00a3bc8>] (SyS_write+0x44/0x84) r10:b6f9e000 r8:00000002 r7:eee47e00 r6:eee47e00 r5:00000000 r4:00000000 [<c00a3b84>] (SyS_write) from [<c000ec00>] (ret_fast_syscall+0x0/0x30) r10:00000000 r8:c000ed84 r7:00000004 r6:b6edda78 r5:b6f9e000 r4:00000002 Code: e1a0c00d e92dd8f0 e24cb004 e2804004 (e5906004) ---[ end trace 87a520011e442d68 ]--- And here is the boot log from the same kernel: Uncompressing Linux... done, booting the kernel. Booting Linux on physical CPU 0x0 Linux version 3.14.0-rc7 (root at orangebox) (gcc version 4.7.3 (Gentoo 4.7.3-r1 p1.3, pie-0.5.5) ) #18 SMP PREEMPT Mon Mar 31 11:22:04 BST 2014 CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache Machine model: Xyratex Porter-v1 bootconsole [earlycon0] enabled Truncating RAM at 00000000-3fffffff to -2f7fffff (vmalloc region overlap). Memory policy: Data cache writealloc PERCPU: Embedded 7 pages/cpu @ef1ee000 s7040 r8192 d13440 u32768 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 193040 Kernel command line: console=ttyS0,115200 earlyprintk=ttys0 rw pm_disable mtdparts=armada-nand:8m(boot),8m(kernel),-(rootfs) ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs pci=hpiosize=0 PID hash table entries: 4096 (order: 2, 16384 bytes) Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Memory: 766128K/778240K available (3579K kernel code, 225K rwdata, 1052K rodata, 154K init, 74K bss, 12112K reserved) Virtual kernel memory layout: vector : 0xffff0000 - 0xffff1000 ( 4 kB) fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB) vmalloc : 0xf0000000 - 0xff000000 ( 240 MB) lowmem : 0xc0000000 - 0xef800000 ( 760 MB) modules : 0xbf000000 - 0xc0000000 ( 16 MB) .text : 0xc0008000 - 0xc048dfd4 (4632 kB) .init : 0xc048e000 - 0xc04b4b80 ( 155 kB) .data : 0xc04b6000 - 0xc04ee438 ( 226 kB) .bss : 0xc04ee444 - 0xc0500ca8 ( 75 kB) Preemptible hierarchical RCU implementation. Dump stacks of tasks blocking RCU-preempt GP. RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2. RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 NR_IRQS:16 nr_irqs:16 16 sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798691800ns Initializing Coherency fabric Aurora cache controller enabled l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1024 kB Console: colour dummy device 80x30 Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256) pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 512 CPU: Testing write buffer coherency: ok CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 Setting up static identity map for 0x385020 - 0x385054 Initializing Power Management Service Unit Booting CPU 1 CPU1: Booted secondary processor CPU1: thread -1, cpu 1, socket 0, mpidr 80000001 Brought up 2 CPUs SMP: Total of 2 processors activated. CPU: All CPU(s) started in SVC mode. devtmpfs: initialized VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6 mvebu-soc-id: MVEBU SoC ID=0x7826, Rev=0x2 pinctrl core: initialized pinctrl subsystem NET: Registered protocol family 16 DMA: preallocated 256 KiB pool for atomic coherent allocations irq: Cannot allocate irq_descs @ IRQ32, assuming pre-allocated irq: Cannot allocate irq_descs @ IRQ67, assuming pre-allocated bio: create slab <bio-0> at 0 SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb Switched to clocksource armada_370_xp_clocksource NET: Registered protocol family 2 TCP established hash table entries: 8192 (order: 3, 32768 bytes) TCP bind hash table entries: 8192 (order: 4, 65536 bytes) TCP: Hash tables configured (established 8192 bind 8192) TCP: reno registered UDP hash table entries: 512 (order: 2, 16384 bytes) UDP-Lite hash table entries: 512 (order: 2, 16384 bytes) NET: Registered protocol family 1 futex hash table entries: 512 (order: 3, 32768 bytes) msgmni has been set to 1496 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253) io scheduler noop registered io scheduler deadline registered io scheduler cfq registered (default) armada-xp-pinctrl f1018000.pinctrl: registered pinctrl driver mvebu-pcie pcie-controller.2: PCI host bridge to bus 0000:00 pci_bus 0000:00: root bus resource [io 0x1000-0xfffff] pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe8000000] pci_bus 0000:00: root bus resource [bus 00-ff] PCI: bus0: Fast back to back transfers disabled pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring PCI: bus1: Fast back to back transfers disabled pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring PCI: bus2: Fast back to back transfers disabled pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring PCI: bus3: Fast back to back transfers enabled PCI: bus4: Fast back to back transfers enabled PCI: bus5: Fast back to back transfers enabled pci 0000:00:01.0: BAR 8: assigned [mem 0xe0000000-0xe0cfffff] pci 0000:01:00.0: BAR 8: assigned [mem 0xe0000000-0xe05fffff] pci 0000:01:00.0: BAR 9: assigned [mem 0xe0600000-0xe0bfffff 64bit pref] pci 0000:01:00.0: BAR 0: assigned [mem 0xe0c00000-0xe0c1ffff] pci 0000:02:01.0: BAR 8: assigned [mem 0xe0000000-0xe01fffff] pci 0000:02:01.0: BAR 9: assigned [mem 0xe0600000-0xe07fffff 64bit pref] pci 0000:02:02.0: BAR 8: assigned [mem 0xe0200000-0xe03fffff] pci 0000:02:02.0: BAR 9: assigned [mem 0xe0800000-0xe09fffff 64bit pref] pci 0000:02:03.0: BAR 8: assigned [mem 0xe0400000-0xe05fffff] pci 0000:02:03.0: BAR 9: assigned [mem 0xe0a00000-0xe0bfffff 64bit pref] pci 0000:02:01.0: PCI bridge to [bus 03] pci 0000:02:01.0: bridge window [mem 0xe0000000-0xe01fffff] pci 0000:02:01.0: bridge window [mem 0xe0600000-0xe07fffff 64bit pref] pci 0000:02:02.0: PCI bridge to [bus 04] pci 0000:02:02.0: bridge window [mem 0xe0200000-0xe03fffff] pci 0000:02:02.0: bridge window [mem 0xe0800000-0xe09fffff 64bit pref] pci 0000:02:03.0: PCI bridge to [bus 05] pci 0000:02:03.0: bridge window [mem 0xe0400000-0xe05fffff] pci 0000:02:03.0: bridge window [mem 0xe0a00000-0xe0bfffff 64bit pref] pci 0000:01:00.0: PCI bridge to [bus 02-05] pci 0000:01:00.0: bridge window [mem 0xe0000000-0xe05fffff] pci 0000:01:00.0: bridge window [mem 0xe0600000-0xe0bfffff 64bit pref] pci 0000:00:01.0: PCI bridge to [bus 01-05] pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe0cfffff] PCI: enabling device 0000:00:01.0 (0140 -> 0143) PCI: enabling device 0000:01:00.0 (0140 -> 0143) PCI: enabling device 0000:02:01.0 (0140 -> 0143) PCI: enabling device 0000:02:02.0 (0140 -> 0143) PCI: enabling device 0000:02:03.0 (0140 -> 0143) mv_xor f1060900.xor: Marvell shared XOR driver mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) mv_xor f1060900.xor: Marvell XOR: ( xor cpy ) mv_xor f10f0900.xor: Marvell shared XOR driver mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) mv_xor f10f0900.xor: Marvell XOR: ( xor cpy ) Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 19, base_baud = 15625000) is a 16550A console [ttyS0] enabled console [ttyS0] enabled bootconsole [earlycon0] disabled bootconsole [earlycon0] disabled f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 20, base_baud = 15625000) is a 16550A pxa3xx-nand f10d0000.nand: This platform can't do DMA on this device nand: device found, Manufacturer ID: 0x2c, Chip ID: 0x38 nand: Micron MT29F8G08ABABAWP nand: 1024MiB, SLC, page size: 4096, OOB size: 224 Bad block table found at page 262016, version 0x01 Bad block table found at page 261888, version 0x01 3 ofpart partitions found on MTD device pxa3xx_nand-0 Creating 3 MTD partitions on "pxa3xx_nand-0": 0x000000000000-0x000000800000 : "boot" 0x000000800000-0x000001000000 : "kernel" 0x000001000000-0x000040000000 : "rootfs" libphy: orion_mdio_bus: probed ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci-pci: EHCI PCI platform driver ehci-orion: EHCI orion driver orion-ehci f1050000.usb: EHCI Host Controller orion-ehci f1050000.usb: new USB bus registered, assigned bus number 1 orion-ehci f1050000.usb: irq 26, io mem 0xf1050000 orion-ehci f1050000.usb: USB 2.0 started, EHCI 1.00 hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected usbcore: registered new interface driver usb-storage rtc-mv f1010300.rtc: rtc core: registered f1010300.rtc as rtc0 TCP: cubic registered NET: Registered protocol family 17 UBI: attaching mtd2 to ubi0 random: nonblocking pool is initialized UBI: scanning is finished UBI: attached mtd2 (name "rootfs", size 1008 MiB) to ubi0 UBI: PEB size: 524288 bytes (512 KiB), LEB size: 516096 bytes UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096 UBI: VID header offset: 4096 (aligned 4096), data offset: 8192 UBI: good PEBs: 2008, bad PEBs: 8, corrupted PEBs: 0 UBI: user volume: 1, internal volumes: 1, max. volumes count: 128 UBI: max/mean erase counter: 2/1, WL threshold: 4096, image sequence number: 1257185697 UBI: available PEBs: 0, total reserved PEBs: 2008, PEBs reserved for bad PEB handling: 32 UBI: background thread "ubi_bgt0d" started, PID 920 rtc-mv f1010300.rtc: setting system clock to 2017-07-24 12:40:46 UTC (1500900046) UBIFS: background thread "ubifs_bgt0_0" started, PID 922 UBIFS: recovery needed UBIFS: recovery completed UBIFS: mounted UBI device 0, volume 0, name "rootfs" UBIFS: LEB size: 516096 bytes (504 KiB), min./max. I/O unit sizes: 4096 bytes/4096 bytes UBIFS: FS size: 1012064256 bytes (965 MiB, 1961 LEBs), journal size 33546240 bytes (31 MiB, 65 LEBs) UBIFS: reserved for root: 4952683 bytes (4836 KiB) UBIFS: media format: w4/r0 (latest is w4/r0), UUID 4EA889AC-C944-413F-A363-33F57F5CAE9A, small LPT model VFS: Mounted root (ubifs filesystem) on device 0:12. devtmpfs: mounted Freeing unused kernel memory: 152K (c048e000 - c04b4000) On 26 March 2014 17:18, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Wed, Mar 26, 2014 at 06:06:34PM +0100, Thomas Petazzoni wrote: > >> Thanks for your report. I'm adding Jason Gunthorpe in Cc, as his >> insights are always very helpful when investigating PCIe issues on >> Marvell platforms. > > To me, this looks like standard problems with hot plug - unrelated to > Marvell. > > Actually supporting the full generality of hot plug requires someone > pre-allocate and reserve bridge window address space. On x86 the BIOS > knows which slots are hot plug capable and ensures that address space > is set aside for that slot when doing window assignment. Linux will > just keep the BIOS assignment and when the hot plug happens the extra > space is used. > > What you seem to be seeing is that Linux does an optimal allocation at > boot, and the PCI-E switch port that connects to your hot plug slot is > not allocated any spare address space for a future device. > > During rescan there is no free address space to assign to the new > device so everything blows up. > > I know there are some provisions in the PCI core to solve this issue, > I know on x86 the BIOS plays a role, but I'm not familiar enough with > the details to give you an exact path to solve your problem, other > than to say, at first boot you need to arrange for there to be extra > address space available. > > A really simple dumb hack that might work is to change the order of > the PCI-E ports on the Armada (I think you can do this through the > DT). What you want is to place the port connected to the PLX last. > The goal here is to have it do address assignment last, so it will get > the last chunk of address space, and will have room to grow into > unused space, instead of being blocked by another port. > > Regards, > Jason -- ------------------------------ For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com ------------------------------ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-03-31 14:17 ` Matthew Minter @ 2014-04-01 16:41 ` Jason Gunthorpe 2014-04-03 9:31 ` Matthew Minter [not found] ` <CAFJTrDu1=Hoa6NS3DaFphMUXgJYunybbdCoGiBVe1qKNQ4s3MA@mail.gmail.com> 0 siblings, 2 replies; 13+ messages in thread From: Jason Gunthorpe @ 2014-04-01 16:41 UTC (permalink / raw) To: linux-arm-kernel On Mon, Mar 31, 2014 at 03:17:24PM +0100, Matthew Minter wrote: > Just to give an update of what I have tried. I noticed that there was > a PCI quirk available for a similar PLX chip which forced the kernel > to see the slots as hot-plug slots (unfortunately some PLX chips > falsely report themselves to be non hotplug capable bridges from my > understanding). I tried extending this quirk to run on my chip also. > This significantly changed the results in that along with the use of > pci=hpmemsize=xM I managed to force the chip to pre-allocate some > memory for the hotplug slot. Specifically I added this line: > DECLARE_PCI_FIXUP_HEADER(0x10b5, 0x8617, quirk_hotplug_bridge); > to linux-3.14-rc7/drivers/pci/quirks.c This all sounds right to me, it is probably the proper approach, things could be refined a bit by adding a DT property to indicate hot plug, but that won't help your oops. The oops kinda looks like something went wrong during a PCI memory read. > Internal error: : 1008 [#1] PREEMPT SMP ARM Which is a call to die, but the string that should tell us where is "" There are cases like that in the kernel, but all are prefixed by another line - did a line go missing (Unhandled Fault perhaps?) First guess would be something went wrong when the mbus windows were created after the rescan. Some debugging prints from that part of the flow might be informative, as well as a dump from the mbus debugfs.. Second guess, something went wrong during the hot plug, and the PLX is not forwarding TLPs.. lpsci -v might give a clue. > I am currently working on a way to modify the scan order as Jason as > suggested, as soon as I have worked out exactly how the scan order is > determined I will try modifying the DT or any code needed to force the > port with the PLX chip to be scanned last. However I first need to > read some more of the code to understand what it orders on, eg: order > in DT, bus number, memory address. That suggestion was a hack, you have found the 'proper' way to do hot plug. If you can resolve this oops you should be golden :) Jason ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-01 16:41 ` Jason Gunthorpe @ 2014-04-03 9:31 ` Matthew Minter [not found] ` <CAFJTrDu1=Hoa6NS3DaFphMUXgJYunybbdCoGiBVe1qKNQ4s3MA@mail.gmail.com> 1 sibling, 0 replies; 13+ messages in thread From: Matthew Minter @ 2014-04-03 9:31 UTC (permalink / raw) To: linux-arm-kernel Sorry, I just realized I missed the lspci results in my last email. They are as follows: Before the oops: 00:01.0 Class 0604: Device 11ab:7826 (rev 02) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e0cfffff Prefetchable memory behind bridge: 00000000-000fffff 01:00.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Memory at e0c00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=01, secondary=02, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e05fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e0bfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:01.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 00010000-00010fff Memory behind bridge: e0000000-e01fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e07fffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:02.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 00011000-00011fff Memory behind bridge: e0200000-e03fffff Prefetchable memory behind bridge: 00000000e0800000-00000000e09fffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:03.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 00012000-00012fff Memory behind bridge: e0400000-e05fffff Prefetchable memory behind bridge: 00000000e0a00000-00000000e0bfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport After the oops: 00:01.0 Class 0604: Device 11ab:7826 (rev 02) Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e0cfffff Prefetchable memory behind bridge: 00000000-000fffff 01:00.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Memory at e0c00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=01, secondary=02, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e05fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e0bfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Upstream Port, MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [138] Power Budgeting <?> Capabilities: [148] Virtual Channel Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:01.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 00010000-00010fff Memory behind bridge: e0000000-e01fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e07fffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:02.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 00011000-00011fff Memory behind bridge: e0200000-e03fffff Prefetchable memory behind bridge: 00000000e0800000-00000000e09fffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:03.0 Class 0604: Device 10b5:8617 (rev ba) Flags: bus master, fast devsel, latency 0 Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 00012000-00012fff Memory behind bridge: e0400000-e05fffff Prefetchable memory behind bridge: 00000000e0a00000-00000000e0bfffff Capabilities: [40] Power Management version 3 Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Capabilities: [68] Express Downstream Port (Slot+), MSI 00 Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4] Advanced Error Reporting Capabilities: [148] Virtual Channel Capabilities: [520] Access Control Services Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 04:00.0 Class 0106: Device 144d:a800 (rev 01) (prog-if 01) Subsystem: Device 144d:a811 Flags: fast devsel, IRQ 255 Memory at e0200000 (32-bit, non-prefetchable) [size=8K] [virtual] Expansion ROM at e0800000 [disabled] [size=64K] Capabilities: [40] Power Management version 3 Capabilities: [50] MSI: Enable- Count=1/2 Maskable+ 64bit+ Capabilities: [70] Express Endpoint, MSI 00 Capabilities: [d0] Vital Product Data Capabilities: [100] Advanced Error Reporting Capabilities: [140] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [150] Power Budgeting <?> Capabilities: [160] Latency Tolerance Reporting Kernel driver in use: ahci Finally just in case it helps vv: 00:01.0 Class 0604: Device 11ab:7826 (rev 02) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e0cfffff Prefetchable memory behind bridge: 00000000-000fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- 01:00.0 Class 0604: Device 10b5:8617 (rev ba) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Region 0: Memory@e0c00000 (32-bit, non-prefetchable) [size=128K] Bus: primary=01, secondary=02, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e05fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e0bfffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Upstream Port, MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [138 v1] Power Budgeting <?> Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=4 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=06 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32+ WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=WRR32 TC/VC=ff Status: NegoPending- InProgress- Port Arbitration Table <?> VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [448 v1] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?> Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:01.0 Class 0604: Device 10b5:8617 (rev ba) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=03, subordinate=03, sec-latency=0 I/O behind bridge: 00010000-00010fff Memory behind bridge: e0000000-e01fffff Prefetchable memory behind bridge: 00000000e0600000-00000000e07fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #1, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #1, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending+ InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:02.0 Class 0604: Device 10b5:8617 (rev ba) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=04, subordinate=04, sec-latency=0 I/O behind bridge: 00011000-00011fff Memory behind bridge: e0200000-e03fffff Prefetchable memory behind bridge: 00000000e0800000-00000000e09fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #2, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock- Changed: MRL- PresDet+ LinkState+ DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending- InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 02:03.0 Class 0604: Device 10b5:8617 (rev ba) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=02, secondary=05, subordinate=05, sec-latency=0 I/O behind bridge: 00012000-00012fff Memory behind bridge: e0400000-e05fffff Prefetchable memory behind bridge: 00000000e0a00000-00000000e0bfffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR- BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/4 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00 DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #3, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <4us ClockPM- Surprise+ LLActRep+ BwNot+ LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- SltCap: AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise- Slot #3, PowerLimit 25.000W; Interlock- NoCompl- SltCtl: Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg- Control: AttnInd Off, PwrInd Off, Power- Interlock- SltSta: Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock- Changed: MRL- PresDet- LinkState- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported ARIFwd+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a4] Subsystem: Device 10b5:8617 Capabilities: [100 v1] Device Serial Number ba-86-01-10-b5-df-0e-00 Capabilities: [fb4 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- Bahci 0000:04:00.0: vpd r/w failed. This is likely a firmware bug on this device. Contact the card vendor for a firmware update. adDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [148 v1] Virtual Channel Caps: LPEVC=1 RefClk=100ns PATEntryBits=1 Arb: Fixed+ WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff Status: NegoPending+ InProgress- VC1: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256- Ctrl: Enable- ID=1 ArbSelect=Fixed TC/VC=00 Status: NegoPending+ InProgress- Capabilities: [520 v1] Access Control Services ACSCap: SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+ ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [950 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?> Kernel driver in use: pcieport 04:00.0 Class 0106: Device 144d:a800 (rev 01) (prog-if 01) Subsystem: Device 144d:a811 Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 255 Region 5: Memory at e0200000 (32-bit, non-prefetchable) [size=8K] [virtual] Expansion ROM@e0800000 [disabled] [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/2 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Latency L0 <4us, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x2, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [d0] Vital Product Data pcilib: sysfs_read_vpd: read failed: Connection timed out Not readable Capabilities: [100 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 00-00-00-00-00-00-00-00 Capabilities: [150 v1] Power Budgeting <?> Capabilities: [160 v1] Latency Tolerance Reporting Max snoop latency: 256ns Max no snoop latency: 256ns Kernel driver in use: ahci On 1 April 2014 17:41, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Mon, Mar 31, 2014 at 03:17:24PM +0100, Matthew Minter wrote: >> Just to give an update of what I have tried. I noticed that there was >> a PCI quirk available for a similar PLX chip which forced the kernel >> to see the slots as hot-plug slots (unfortunately some PLX chips >> falsely report themselves to be non hotplug capable bridges from my >> understanding). I tried extending this quirk to run on my chip also. >> This significantly changed the results in that along with the use of >> pci=hpmemsize=xM I managed to force the chip to pre-allocate some >> memory for the hotplug slot. Specifically I added this line: >> DECLARE_PCI_FIXUP_HEADER(0x10b5, 0x8617, quirk_hotplug_bridge); >> to linux-3.14-rc7/drivers/pci/quirks.c > > This all sounds right to me, it is probably the proper approach, > things could be refined a bit by adding a DT property to indicate hot > plug, but that won't help your oops. > > The oops kinda looks like something went wrong during a PCI memory > read. > >> Internal error: : 1008 [#1] PREEMPT SMP ARM > > Which is a call to die, but the string that should tell us where is "" > There are cases like that in the kernel, but all are prefixed by > another line - did a line go missing (Unhandled Fault perhaps?) > > First guess would be something went wrong when the mbus windows were > created after the rescan. Some debugging prints from that part of the > flow might be informative, as well as a dump from the mbus debugfs.. > > Second guess, something went wrong during the hot plug, and the PLX is > not forwarding TLPs.. lpsci -v might give a clue. > >> I am currently working on a way to modify the scan order as Jason as >> suggested, as soon as I have worked out exactly how the scan order is >> determined I will try modifying the DT or any code needed to force the >> port with the PLX chip to be scanned last. However I first need to >> read some more of the code to understand what it orders on, eg: order >> in DT, bus number, memory address. > > That suggestion was a hack, you have found the 'proper' way to do hot > plug. If you can resolve this oops you should be golden :) > > Jason -- ------------------------------ For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com ------------------------------ ^ permalink raw reply [flat|nested] 13+ messages in thread
[parent not found: <CAFJTrDu1=Hoa6NS3DaFphMUXgJYunybbdCoGiBVe1qKNQ4s3MA@mail.gmail.com>]
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation [not found] ` <CAFJTrDu1=Hoa6NS3DaFphMUXgJYunybbdCoGiBVe1qKNQ4s3MA@mail.gmail.com> @ 2014-04-03 18:23 ` Jason Gunthorpe 2014-04-03 18:24 ` Thomas Petazzoni 0 siblings, 1 reply; 13+ messages in thread From: Jason Gunthorpe @ 2014-04-03 18:23 UTC (permalink / raw) To: linux-arm-kernel On Wed, Apr 02, 2014 at 05:32:04PM +0100, Matthew Minter wrote: > Ok, I have managed to gather some more info. The problem of the lost > line was with my serial console acting and losing lines of data > oddly around the time of the the oops and I have indeed gained some > extra info (I logged dmesg to a file while the issue was happening > and pulled the full transcript from there). Your oops seem to suggest that the very first MMIO read from the AHCI is causing this: > [? 191.654496] Unhandled fault: external abort on non-linefetch > (0x1008) at 0xf0460004 And the address 0xf0460004 is certainly funny - that doesn't look like it is valid at all, but prehaps it is a bogus report (?) On the other hand, the low bits are right: drivers/ata/ahci.h: HOST_CTL = 0x04, /* global host control */ If you belive the Unhandled fault report - the driver is accessing the wrong memory region (0xf0460000, instead of 0xe0200000) You may want to double check that the mmio pointer in the ahci is actually being mapped to 0xe0200000... >> Second guess, something went wrong during the hot plug, and the >> PLX is not forwarding TLPs.. lpsci -v might give a clue. The lspci -vv looks OK for the AHCI: 04:00.0 Class 0106: Device 144d:a800 (rev 01) (prog-if 01) Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- ^^^^^^^ Region 5: Memory at e0200000 (32-bit, non-prefetchable) [size=8K] MMIO is enabled, address looks OK The PLX upstream bridge looks OK: 02:02.0 Class 0604: Device 10b5:8617 (rev ba) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Memory behind bridge: e0200000-e03fffff Prefetchable memory behind bridge: 00000000e0800000-00000000e09fffff The mvebu bridge looks OK: 00:01.0 Class 0604: Device 11ab:7826 (rev 02) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Bus: primary=00, secondary=01, subordinate=05, sec-latency=0 I/O behind bridge: 00010000-00013fff Memory behind bridge: e0000000-e0cfffff Matches the mbus debug: [09] 00000000e0000000 - 00000000e0d00000 : 0004:00e8 Jason ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-03 18:23 ` Jason Gunthorpe @ 2014-04-03 18:24 ` Thomas Petazzoni 2014-04-07 13:03 ` Matthew Minter 0 siblings, 1 reply; 13+ messages in thread From: Thomas Petazzoni @ 2014-04-03 18:24 UTC (permalink / raw) To: linux-arm-kernel Dear Jason Gunthorpe, On Thu, 3 Apr 2014 12:23:22 -0600, Jason Gunthorpe wrote: > Your oops seem to suggest that the very first MMIO read from the AHCI > is causing this: > > > [? 191.654496] Unhandled fault: external abort on non-linefetch > > (0x1008) at 0xf0460004 > > And the address 0xf0460004 is certainly funny - that doesn't look like > it is valid at all, but prehaps it is a bogus report (?) Notice that the address reported by the Unhandled fault message is a *virtual* address, not a *physical* address. So you have to look to what physical address it corresponds to draw some conclusions, I believe. Thomas -- Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-03 18:24 ` Thomas Petazzoni @ 2014-04-07 13:03 ` Matthew Minter 2014-04-07 16:45 ` Jason Gunthorpe 0 siblings, 1 reply; 13+ messages in thread From: Matthew Minter @ 2014-04-07 13:03 UTC (permalink / raw) To: linux-arm-kernel Hi Guys, This is not a proper update yet, however as a test to confirm if the issue is indeed relating to the address mapping, I modified the ahci code to use a hard coded ioremap to the correct address range 0xe0200000-0xe0201fff. This did not affect the error in any way and I still have the same problem. Thus I can conclude that the ahci driver pointing to the wrong address does not seem to be the problem. However I am somewhat stumped on where to look next. I am going to read over the PLX datasheet in case there is a known issue with my configuration. Again, I will post if I have any more information. I will also see if I can try jtaging the chip so I can check the state of the PCI bus and mmu at the time of failure. Many thanks, Matt On 3 April 2014 19:24, Thomas Petazzoni <thomas.petazzoni@free-electrons.com> wrote: > Dear Jason Gunthorpe, > > On Thu, 3 Apr 2014 12:23:22 -0600, Jason Gunthorpe wrote: > >> Your oops seem to suggest that the very first MMIO read from the AHCI >> is causing this: >> >> > [ 191.654496] Unhandled fault: external abort on non-linefetch >> > (0x1008) at 0xf0460004 >> >> And the address 0xf0460004 is certainly funny - that doesn't look like >> it is valid at all, but prehaps it is a bogus report (?) > > Notice that the address reported by the Unhandled fault message is a > *virtual* address, not a *physical* address. So you have to look to > what physical address it corresponds to draw some conclusions, I > believe. > > Thomas > -- > Thomas Petazzoni, CTO, Free Electrons > Embedded Linux, Kernel and Android engineering > http://free-electrons.com -- ------------------------------ For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com ------------------------------ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-07 13:03 ` Matthew Minter @ 2014-04-07 16:45 ` Jason Gunthorpe 2014-04-08 10:57 ` Matthew Minter 0 siblings, 1 reply; 13+ messages in thread From: Jason Gunthorpe @ 2014-04-07 16:45 UTC (permalink / raw) To: linux-arm-kernel On Mon, Apr 07, 2014 at 02:03:55PM +0100, Matthew Minter wrote: > However I am somewhat stumped on where to look next. I am going to Things to try: - Look at the PCI Interrupt Cause Register, see this message from Thomas and the related thread: http://www.spinics.net/lists/arm-kernel/msg318400.html Do this before and after the faulting instruction. If it doesn't change then the fault is generated internally to the SOC - diff the lspci -vv output against the non-hotplug case that works - Dump and verify all the mbus registers, not just through debugfs. - Check if the PLX has some special sequence required to hotplug Cheers, Jason ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-07 16:45 ` Jason Gunthorpe @ 2014-04-08 10:57 ` Matthew Minter 2014-04-08 16:34 ` Jason Gunthorpe 0 siblings, 1 reply; 13+ messages in thread From: Matthew Minter @ 2014-04-08 10:57 UTC (permalink / raw) To: linux-arm-kernel Hi Guys, I have already posted this on the other thread, however I can confirm the first suggestion fixes this. It seems I am having the same issue regarding the power of two memory windows. Is it still helpful for me to do any more checks or should we consider this closed? Many thanks, Matt On 7 April 2014 17:45, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote: > On Mon, Apr 07, 2014 at 02:03:55PM +0100, Matthew Minter wrote: > >> However I am somewhat stumped on where to look next. I am going to > > Things to try: > - Look at the PCI Interrupt Cause Register, see this message > from Thomas and the related thread: > http://www.spinics.net/lists/arm-kernel/msg318400.html > Do this before and after the faulting instruction. If it doesn't > change then the fault is generated internally to the SOC > - diff the lspci -vv output against the non-hotplug case that works > - Dump and verify all the mbus registers, not just through debugfs. > - Check if the PLX has some special sequence required > to hotplug > > Cheers, > Jason -- ------------------------------ For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com ------------------------------ ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation 2014-04-08 10:57 ` Matthew Minter @ 2014-04-08 16:34 ` Jason Gunthorpe 0 siblings, 0 replies; 13+ messages in thread From: Jason Gunthorpe @ 2014-04-08 16:34 UTC (permalink / raw) To: linux-arm-kernel On Tue, Apr 08, 2014 at 11:57:59AM +0100, Matthew Minter wrote: > I have already posted this on the other thread, however I can confirm > the first suggestion fixes this. It seems I am having the same issue > regarding the power of two memory windows. Is it still helpful for me > to do any more checks or should we consider this closed? I think the mbus problem explains everything? Please try to find time to test Thomas's final fix when he posts it, but others I think it can be considered done :) Regards, Jason ^ permalink raw reply [flat|nested] 13+ messages in thread
* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation
@ 2014-04-03 8:57 Matthew Minter
0 siblings, 0 replies; 13+ messages in thread
From: Matthew Minter @ 2014-04-03 8:57 UTC (permalink / raw)
To: linux-arm-kernel
Ok, I have managed to gather some more info. The problem of the lost
line was with my serial console acting and losing lines of data oddly
around the time of the the oops and I have indeed gained some extra
info (I logged dmesg to a file while the issue was happening and
pulled the full transcript from there).
It appears the mbus windows are not being changed during the rescan, I
think this is a good thing however as it seems with this hotplug quirk
they seem to be correct to begin with (the bridge windows all seem to
fall within the range of the mbus windows).
Finally I have noticed that I get different results if I first remove
the bridge (by echoing to the remove file) then rescan after that (The
log of that seems to indicate some issue involving the pcieport
driver, please see the end of the email). However I shall continue
working on this issue as it is still oopsing either way, the log has
revealed a little more I think and I will send an update if I find
anything.
Dumps ahead...
Membus debugfs: (Never changes)
[00] 00000000e8010000 - 00000000e8020000 : 0004:00e0 (remap 0000000000010000)
[01] disabled
[02] disabled
[03] disabled
[04] disabled
[05] disabled
[06] disabled
[07] disabled
[08] 00000000fff00000 - 0000000100000000 : 0001:001d
[09] 00000000e0000000 - 00000000e0d00000 : 0004:00e8
[10] disabled
[11] disabled
[12] disabled
[13] disabled
[14] disabled
[15] disabled
[16] disabled
[17] disabled
[18] disabled
[19] disabled
Unabridged version of crash log (without removing ports):
[ 191.653052] pci_bus 0000:01: scanning bus
[ 191.653094] pcieport 0000:01:00.0: scanning [bus 02-05] behind bridge, pass 0
[ 191.653110] pcieport 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 191.653126] pci_bus 0000:02: scanning bus
[ 191.653202] pcieport 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 0
[ 191.653220] pcieport 0000:02:02.0: scanning [bus 04-04] behind bridge, pass 0
[ 191.653236] pcieport 0000:02:03.0: scanning [bus 05-05] behind bridge, pass 0
[ 191.653251] pcieport 0000:02:01.0: scanning [bus 00-00] behind bridge, pass 1
[ 191.653265] pci_bus 0000:03: scanning bus
[ 191.653276] pci_bus 0000:03: bus scan returning with max=03
[ 191.653288] pci_bus 0000:03: busn_res: [bus 03] end is updated to 03
[ 191.653302] pcieport 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 1
[ 191.653315] pci_bus 0000:04: scanning bus
[ 191.653361] pci 0000:04:00.0: [144d:a800] type 00 class 0x010601
[ 191.653452] pci 0000:04:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 191.653472] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 191.653496] pci 0000:04:00.0: calling pci_fixup_ide_bases+0x0/0x58
[ 191.653593] pci 0000:04:00.0: PME# supported from D3hot D3cold
[ 191.653611] pci 0000:04:00.0: PME# disabled
[ 191.654078] pci_bus 0000:04: bus scan returning with max=04
[ 191.654095] pci_bus 0000:04: busn_res: [bus 04] end is updated to 04
[ 191.654118] pcieport 0000:02:03.0: scanning [bus 00-00] behind bridge, pass 1
[ 191.654137] pci_bus 0000:05: scanning bus
[ 191.654154] pci_bus 0000:05: bus scan returning with max=05
[ 191.654169] pci_bus 0000:05: busn_res: [bus 05] end is updated to 05
[ 191.654187] pci_bus 0000:02: bus scan returning with max=05
[ 191.654198] pci_bus 0000:02: busn_res: [bus 02-05] end is updated to 05
[ 191.654211] pci_bus 0000:01: bus scan returning with max=05
[ 191.654219] pci_bus 0000:01: busn_res: [bus 01-05] end is updated to 05
[ 191.654229] pci_bus 0000:00: bus scan returning with max=05
[ 191.654296] pci 0000:04:00.0: BAR 6: assigned [mem
0xe0800000-0xe080ffff pref]
[ 191.654306] pci 0000:04:00.0: BAR 5: assigned [mem 0xe0200000-0xe0201fff]
[ 191.654321] pci 0000:04:00.0: BAR 5: set to [mem
0xe0200000-0xe0201fff] (PCI address [0xe0200000-0xe0201fff])
[ 191.654412] ahci 0000:04:00.0: no of_node; not parsing pinctrl DT
[ 191.654444] ahci 0000:04:00.0: version 3.0
[ 191.654460] PCI: enabling device 0000:04:00.0 (0000 -> 0002)
[ 191.654496] Unhandled fault: external abort on non-linefetch
(0x1008) at 0xf0460004
[ 191.654505] Internal error: : 1008 [#1] PREEMPT SMP ARM
[ 191.658441] Modules linked in:
[ 191.660224] CPU: 0 PID: 1746 Comm: bash Not tainted 3.14.0-rc7 #24
[ 191.665122] task: eee9b440 ti: edc38000 task.ti: edc38000
[ 191.669243] PC is at ahci_enable_ahci+0x10/0x7c
[ 191.672488] LR is at ahci_save_initial_config+0x28/0x24c
[ 191.676515] pc : [<c02510e4>] lr : [<c02512d8>] psr: 60070013
sp : edc39cb0 ip : edc39cd0 fp : edc39ccc
[ 191.685413] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 191.689350] r7 : ee7a9c68 r6 : f0460000 r5 : eeca2790 r4 : f0460004
r3 : 00000000 r2 : 00000000 r1 : eeca2790 r0 : f0460000
[ 191.699838] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 191.705690] Control: 10c53c7d Table: 2dc6406a DAC: 00000015
[ 191.710150] Process bash (pid: 1746, stack limit = 0xedc38240)
[ 191.714696] Stack: (0xedc39cb0 to 0xedc3a000)
[ 191.717765] 9ca0: ee7a9c00
eeca2790 f0460000 ee7a9c68
[ 191.724668] 9cc0: edc39cfc edc39cd0 c02512d8 c02510e0 00000000
ee7a9c00 ee7a9c68 eeca27
[ 191.731572] 9ce0: 00000000 00000000 eef2a2c8 00000005 edc39d64
edc39d00 c024e7d0 c02512bc
[ 191.738475] 9d00: edc39d34 edc39d10 c00fa3ec ee4987e0 c043f20d
edc39d1c 00000000 00060082
[ 191.745378] 9d20: 00000000 0000001f 00000000 0000007f c04d3a04
00000000 edc39d64 ee7a9c68
[ 191.752285] 9d40: ee7a9c00 c04d36f4 00000000 c04d36c0 eef2a2c8
eef2a2c0 edc39d8c edc39d68
9d60: c01dc230 c024e458 c01dc1d0 ee7a9c68 c04d36f4 00000000 c0218c18 00000002
[ 191.766098] 9d80: edc39db4 edc39d90 c0218974 c01dc1dc ee7a9c68
c0218c18 edc39db4 c04d36f4
[ 191.773005] 9da0: ee7a9c68 ee7a9c68 edc39dcc edc39db8 c0218c48
c0218850 00000000 edc39dd0
[ 191.779911] 9dc0: edc39df4 edc39dd0 c0216d30 c0218c24 eec6c7b0
ee492538 ee7a9c68 ee7a9c9c
9de0: eeecd600 edc39f78 edc39e14 edc39df8 c02187d0 c0216ce8 00000001 ee7a9c00
[ 191.793723] 9e00: eeecd614 eeecd600 edc39e2c edc39e18 c01d4810
c021876c 00000000 ee7a9c00
[ 191.800629] 9e20: edc39e4c edc39e30 c01d499c c01d47e0 00000004
eed38400 eeecda14 eeecda00
[ 191.807533] 9e40: edc39e6c edc39e50 c01d49ec c01d4970 00000004
eed38c00 eeecdc14 eeecdc00
[ 191.814439] 9e60: edc39e8c edc39e70 c01d49ec c01d4970 00000004
eeefe000 eeecde14 eeecde00
[ 191.821343] 9e80: edc39eac edc39e90 c01d49ec c01d4970 edc39e88
eeecde00 00000005 eef2a840
[ 191.828249] 9ea0: edc39ec4 edc39eb0 c037cec0 c01d4970 eeecde00
00000002 edc39ee4 edc39ec8
[ 191.835155] 9ec0: c01dd17c c037cea4 00000041 00000001 eec69a68
eef2a840 edc39ef4 edc39ee8
[ 191.842058] 9ee0: c0216b00 c01dd140 edc39f0c edc39ef8 c00f6a4c
c0216ae4 00000000 00000000
[ 191.848961] 9f00: edc39f44 edc39f10 c00f9cd0 c00f6a14 00000000
00000000 c00a1430 eecb7d40
[ 191.855866] 9f20: 00000002 b6f96000 edc39f78 00000002 edc38000
b6f96000 edc39f74 edc39f48
[ 191.862769] 9f40: c00a3830 c00f9be8 edc39f74 edc39f58 00000000
00000000 eecb7d40 eecb7d40
[ 191.869675] 9f60: 00000002 b6f96000 edc39fa4 edc39f78 c00a3bc8
c00a376c 00000000 00000000
[ 191.876581] 9f80: 00000002 b6f96000 b6ed5a78 00000004 c000ed84
00000000 00000000 edc39fa8
[ 191.883485] 9fa0: c000ec00 c00a3b90 00000002 b6f96000 00000001
b6f96000 00000002 00000000
[ 191.890387] 9fc0: 00000002 b6f96000 b6ed5a78 00000004 00000002
00000002 b6f96000 00000000
[ 191.897290] 9fe0: 00000000 bed43a1c b6e15390 b6e66aac 60070010
00000001 00000000 00000000
[ 191.904184] Backtrace:
[ 191.905365] [<c02510d4>] (ahci_enable_ahci) from [<c02512d8>]
(ahci_save_initial_config+0x28/0x24c)
[ 191.913132] r7:ee7a9c68 r6:f0460000 r5:eeca2790 r4:ee7a9c00
[ 191.917582] [<c02512b0>] (ahci_save_initial_config) from
[<c024e7d0>] (ahci_init_one+0x384/0x850)
[ 191.925175] r10:00000005 r9:eef2a2c8 r8:00000000 r7:00000000
r6:eeca2790 r5:ee7a9c68
[ 191.931809] r4:ee7a9c00 r3:00000000
[ 191.934137] [<c024e44c>] (ahci_init_one) from [<c01dc230>]
(pci_device_probe+0x60/0xa4)
[ 191.940860] r10:eef2a2c0 r9:eef2a2c8 r8:c04d36c0 r7:00000000
r6:c04d36f4 r5:ee7a9c00
[ 191.947497] r4:ee7a9c68
[ 191.948764] [<c01dc1d0>] (pci_device_probe) from [<c0218974>]
(driver_probe_device+0x130/0x340)
[ 191.956183] r8:00000002 r7:c0218c18 r6:00000000 r5:c04d36f4
r4:ee7a9c68 r3:c01dc1d0
[ 191.962735] [<c0218844>] (driver_probe_device) from [<c0218c48>]
(__device_attach+0x30/0x4c)
[ 191.969892] r6:ee7a9c68 r5:ee7a9c68 r4:c04d36f4
[ 191.973279] [<c0218c18>] (__device_attach) from [<c0216d30>]
(bus_for_each_drv+0x54/0x9c)
[ 191.980174] r5:edc39dd0 r4:00000000
[ 191.982501] [<c0216cdc>] (bus_for_each_drv) from [<c02187d0>]
(device_attach+0x70/0x88)
[ 191.989224] r7:edc39f78 r6:eeecd600 r5:ee7a9c9c r4:ee7a9c68
[ 191.993673] [<c0218760>] (device_attach) from [<c01d4810>]
(pci_bus_add_device+0x3c/0x68)
[ 192.000570] r6:eeecd600 r5:eeecd614 r4:ee7a9c00 r3:00000001
[ 192.005017] [<c01d47d4>] (pci_bus_add_device) from [<c01d499c>]
(pci_bus_add_devices+0x38/0x9c)
[ 192.012436] r4:ee7a9c00 r3:00000000
[ 192.014761] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 192.022265] r6:eeecda00 r5:eeecda14 r4:eed38400 r3:00000004
[ 192.026711] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 192.034216] r6:eeecdc00 r5:eeecdc14 r4:eed38c00 r3:00000004
[ 192.038659] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 192.046164] r6:eeecde00 r5:eeecde14 r4:eeefe000 r3:00000004
[ 192.050616] [<c01d4964>] (pci_bus_add_devices) from [<c037cec0>]
(pci_rescan_bus+0x28/0x30)
[ 192.057686] r6:eef2a840 r5:00000005 r4:eeecde00 r3:edc39e88
[ 192.062132] [<c037ce98>] (pci_rescan_bus) from [<c01dd17c>]
(bus_rescan_store+0x48/0x70)
[ 192.068940] r5:00000002 r4:eeecde00
[ 192.071260] [<c01dd134>] (bus_rescan_store) from [<c0216b00>]
(bus_attr_store+0x28/0x34)
r5:eef2a840 r4:eec69a68
[ 192.080393] [<c0216ad8>] (bus_attr_store) from [<c00f6a4c>]
(sysfs_kf_write+0x44/0x50)
[ 192.087037] [<c00f6a08>] (sysfs_kf_write) from [<c00f9cd0>]
(kernfs_fop_write+0xf4/0x148)
[ 192.093933] r5:00000000 r4:00000000
[ 192.096255] [<c00f9bdc>] (kernfs_fop_write) from [<c00a3830>]
(vfs_write+0xd0/0x17c)
[ 192.102715] r10:b6f96000 r9:edc38000 r8:00000002 r7:edc39f78
r6:b6f96000 r5:00000002
[ 192.109349] r4:eecb7d40
[ 192.110607] [<c00a3760>] (vfs_write) from [<c00a3bc8>] (SyS_write+0x44/0x84)
[ 192.116372] r10:b6f96000 r8:00000002 r7:eecb7d40 r6:eecb7d40
r5:00000000 r4:00000000
[ 192.123022] [<c00a3b84>] (SyS_write) from [<c000ec00>]
(ret_fast_syscall+0x0/0x30)
[ 192.129309] r10:00000000 r8:c000ed84 r7:00000004 r6:b6ed5a78
r5:b6f96000 r4:00000002
[ 192.135952] Code: e1a0c00d e92dd8f0 e24cb004 e2804004 (e5906004)
[ 192.140763] ---[ end trace 2c2666e38d6db7ad ]---
Crash log (When removing ports):
[ 190.947343] pci_bus 0000:00: scanning bus
[ 190.947389] pci 0000:00:01.0: scanning [bus 01-05] behind bridge, pass 0
[ 190.947402] pci 0000:00:01.0: scanning [bus 00-00] behind bridge, pass 1
[ 190.947413] pci_bus 0000:01: scanning bus
[ 190.947453] pci 0000:01:00.0: [10b5:8617] type 01 class 0x060400
[ 190.947484] pci 0000:01:00.0: reg 0x10: [mem 0xe0c00000-0xe0c1ffff]
[ 190.947538] pci 0000:01:00.0: calling pci_fixup_ide_bases+0x0/0x58
[ 190.947556] pci 0000:01:00.0: calling quirk_hotplug_bridge+0x0/0x1c
[ 190.947635] pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
[ 190.947646] pci 0000:01:00.0: PME# disabled
[ 190.948056] pci 0000:01:00.0: scanning [bus 02-05] behind bridge, pass 0
[ 190.948073] pci 0000:01:00.0: scanning [bus 00-00] behind bridge, pass 1
[ 190.948529] pci_bus 0000:02: busn_res: can not insert [bus 02-ff]
under [bus 01-05] (conflicts with (null) [bus 01-05])
[ 190.948540] pci_bus 0000:02: scanning bus
[ 190.948575] pci 0000:02:01.0: [10b5:8617] type 01 class 0x060400
[ 190.948644] pci 0000:02:01.0: calling pci_fixup_ide_bases+0x0/0x58
[ 190.948660] pci 0000:02:01.0: calling quirk_hotplug_bridge+0x0/0x1c
[ 190.948730] pci 0000:02:01.0: PME# supported from D0 D3hot D3cold
[ 190.948741] pci 0000:02:01.0: PME# disabled
[ 190.963600] pci 0000:02:02.0: [10b5:8617] type 01 class 0x060400
[ 190.963693] pci 0000:02:02.0: calling pci_fixup_ide_bases+0x0/0x58
[ 190.963717] pci 0000:02:02.0: calling quirk_hotplug_bridge+0x0/0x1c
[ 190.963804] pci 0000:02:02.0: PME# supported from D0 D3hot D3cold
[ 190.963819] pci 0000:02:02.0: PME# disabled
[ 190.977971] pci 0000:02:03.0: [10b5:8617] type 01 class 0x060400
[ 190.978060] pci 0000:02:03.0: calling pci_fixup_ide_bases+0x0/0x58
[ 190.978090] pci 0000:02:03.0: calling quirk_hotplug_bridge+0x0/0x1c
[ 190.978177] pci 0000:02:03.0: PME# supported from D0 D3hot D3cold
[ 190.978193] pci 0000:02:03.0: PME# disabled
[ 190.979859] pci_bus 0000:02: fixups for bus
[ 190.979895] PCI: bus2: Fast back to back transfers disabled
[ 190.979908] pci 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 0
[ 190.979925] pci 0000:02:02.0: scanning [bus 04-04] behind bridge, pass 0
[ 190.979940] pci 0000:02:03.0: scanning [bus 05-05] behind bridge, pass 0
[ 190.979955] pci 0000:02:01.0: scanning [bus 00-00] behind bridge, pass 1
[ 190.988273] pci_bus 0000:03: scanning bus
[ 190.988291] pci_bus 0000:03: fixups for bus
[ 190.988298] PCI: bus3: Fast back to back transfers enabled
[ 190.988306] pci_bus 0000:03: bus scan returning with max=03
[ 190.988318] pci_bus 0000:03: busn_res: [bus 03-ff] end is updated to 03
[ 190.988335] pci 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 1
[ 190.994159] pci_bus 0000:04: scanning bus
[ 190.994220] pci 0000:04:00.0: [144d:a800] type 00 class 0x010601
[ 190.994321] pci 0000:04:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 190.994348] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 190.994377] pci 0000:04:00.0: calling pci_fixup_ide_bases+0x0/0x58
[ 190.994474] pci 0000:04:00.0: PME# supported from D3hot D3cold
[ 190.994492] pci 0000:04:00.0: PME# disabled
[ 191.003803] pci_bus 0000:04: fixups for bus
[ 191.003824] PCI: bus4: Fast back to back transfers disabled
[ 191.003832] pci_bus 0000:04: bus scan returning with max=04
[ 191.003845] pci_bus 0000:04: busn_res: [bus 04-ff] end is updated to 04
[ 191.003861] pci 0000:02:03.0: scanning [bus 00-00] behind bridge, pass 1
[ 191.009765] pci 0000:02:02.0: BAR 7: assigned [io 0x11000-0x11fff]
[ 191.009773] pci 0000:02:03.0: BAR 7: assigned [io 0x12000-0x12fff]
[ 191.009785] pci 0000:02:02.0: BAR 8: assigned [mem 0xe0300000-0xe03fffff]
[ 191.009794] pci 0000:02:02.0: BAR 9: assigned [mem
0xe0100000-0xe01fffff pref]
[ 191.009803] pci 0000:02:03.0: BAR 8: assigned [mem 0xe0400000-0xe05fffff]
[ 191.009813] pci 0000:02:03.0: BAR 9: assigned [mem
0xe0600000-0xe07fffff 64bit pref]
[ 191.009821] pci 0000:02:02.0: BAR 8: can't assign mem (size 0x100000)
[ 191.009829] pci 0000:02:02.0: failed to add 200000 res[8]=[mem
0xe0300000-0xe03fffff]
[ 191.009839] pci 0000:02:02.0: BAR 9: can't assign mem pref (size 0x100000)
[ 191.009847] pci 0000:02:02.0: failed to add 200000 res[9]=[mem
0xe0100000-0xe01fffff pref]
[ 191.009856] pci 0000:02:01.0: BAR 8: assigned [mem 0xe0800000-0xe09fffff]
[ 191.009865] pci 0000:02:01.0: BAR 9: can't assign mem pref (size 0x200000)
[ 191.009873] pci 0000:02:01.0: PCI bridge to [bus 03]
[ 191.009885] pci 0000:02:01.0: bridge window [io 0x10000-0x10fff]
[ 191.009899] pci 0000:02:01.0: bridge window [mem 0xe0800000-0xe09fffff]
[ 191.009920] pci 0000:04:00.0: BAR 6: assigned [mem
0xe0100000-0xe010ffff pref]
[ 191.009929] pci 0000:04:00.0: BAR 5: assigned [mem 0xe0300000-0xe0301fff]
[ 191.009944] pci 0000:04:00.0: BAR 5: set to [mem
0xe0300000-0xe0301fff] (PCI address [0xe0300000-0xe0301fff])
[ 191.009951] pci 0000:02:02.0: PCI bridge to [bus 04]
[ 191.009960] pci 0000:02:02.0: bridge window [io 0x11000-0x11fff]
[ 191.009971] pci 0000:02:02.0: bridge window [mem 0xe0300000-0xe03fffff]
[ 191.009982] pci 0000:02:02.0: bridge window [mem
0xe0100000-0xe01fffff pref]
[ 191.009994] pci 0000:02:03.0: PCI bridge to [bus 05]
[ 191.010003] pci 0000:02:03.0: bridge window [io 0x12000-0x12fff]
[ 191.010014] pci 0000:02:03.0: bridge window [mem 0xe0400000-0xe05fffff]
[ 191.010024] pci 0000:02:03.0: bridge window [mem
0xe0600000-0xe07fffff 64bit pref]
[ 191.010037] pci 0000:01:00.0: PCI bridge to [bus 02-05]
[ 191.010045] pci 0000:01:00.0: bridge window [io 0x10000-0x12fff]
[ 191.010057] pci 0000:01:00.0: bridge window [mem 0xe0300000-0xe09fffff]
[ 191.010067] pci 0000:01:00.0: bridge window [mem
0xe0100000-0xe01fffff pref]
[ 191.010135] pcieport 0000:01:00.0: no of_node; not parsing pinctrl DT
[ 191.010177] pcieport 0000:01:00.0: enabling bus mastering
[ 191.020457] pcieport 0000:02:01.0: no of_node; not parsing pinctrl DT
[ 191.020506] pcieport 0000:02:01.0: enabling bus mastering
[ 191.031349] pcieport 0000:02:02.0: no of_node; not parsing pinctrl DT
[ 191.031408] pcieport 0000:02:02.0: enabling bus mastering
[ 191.032829] pcieport 0000:02:03.0: no of_node; not parsing pinctrl DT
[ 191.032876] pcieport 0000:02:03.0: enabling bus mastering
[ 191.041427] ahci 0000:04:00.0: no of_node; not parsing pinctrl DT
[ 191.041459] ahci 0000:04:00.0: version 3.0
[ 191.041474] PCI: enabling device 0000:04:00.0 (0140 -> 0142)
[ 191.041510] Unhandled fault: external abort on non-linefetch
(0x1008) at 0xf0460004
[ 191.041521] Internal error: : 1008 [#1] PREEMPT SMP ARM
[ 191.045460] Modules linked in:
[ 191.047237] CPU: 0 PID: 1746 Comm: bash Not tainted 3.14.0-rc7 #24
[ 191.052133] task: ee58b440 ti: ee760000 task.ti: ee760000
PC is at ahci_enable_ahci+0x10/0x7c
[ 191.059501] LR is at ahci_save_initial_config+0x28/0x24c
[ 191.063529] pc : [<c02510e4>] lr : [<c02512d8>] psr: 60070013
sp : ee761cb0 ip : ee761cd0 fp : ee761ccc
[ 191.072429] r10: 00000000 r9 : 00000000 r8 : 00000000
[ 191.076370] r7 : ee6ca468 r6 : f0460000 r5 : eee0dd10 r4 : f0460004
[ 191.081613] r3 : 00000000 r2 : 00000000 r1 : eee0dd10 r0 : f0460000
[ 191.086859] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[ 191.092714] Control: 10c53c7d Table: 2e6e006a DAC: 00000015
[ 191.097177] Process bash (pid: 1746, stack limit = 0xee760240)
[ 191.101725] Stack: (0xee761cb0 to 0xee762000)
[ 191.104795] 1ca0: ee6ca400
eee0dd10 f0460000 ee6ca468
[ 191.111699] 1cc0: ee761cfc ee761cd0 c02512d8 c02510e0 00000000
ee6ca400 ee6ca468 eee0dd10
[ 191.118602] 1ce0: 00000000 00000000 ee7e4ec8 00000005 ee761d64
ee761d00 c024e7d0 c02512bc
[ 191.125508] 1d00: ee761d34 ee761d10 c00fa3ec eedd67e0 c043f20d
ee761d1c 00000000 00060082
[ 191.132411] 1d20: 00000000 0000001f 00000000 0000007f c04d3a04
00000000 ee761d64 ee6ca468
[ 191.139316] 1d40: ee6ca400 c04d36f4 00000000 c04d36c0 ee7e4ec8
ee7e4ec0 ee761d8c ee761d68
[ 191.146220] 1d60: c01dc230 c024e458 c01dc1d0 ee6ca468 c04d36f4
00000000 c0218c18 00000002
[ 191.153124] 1d80: ee761db4 ee761d90 c0218974 c01dc1dc ee6ca468
c0218c18 ee761db4 c04d36f4
[ 191.160032] 1da0: ee6ca468 ee6ca468 ee761dcc ee761db8 c0218c48
c0218850 00000000 ee761dd0
[ 191.166936] 1dc0: ee761df4 ee761dd0 c0216d30 c0218c24 eec6c7b0
eedd0538 ee6ca468 ee6ca49c
[ 191.173840] 1de0: eee95a00 ee761f78 ee761e14 ee761df8 c02187d0
c0216ce8 00000001 ee6ca400
[ 191.180745] 1e00: eee95a14 eee95a00 ee761e2c ee761e18 c01d4810
c021876c 00000000 ee6ca400
[ 191.187651] 1e20: ee761e4c ee761e30 c01d499c c01d47e0 00000004
ee5fa400 eee95814 eee95800
[ 191.194556] 1e40: ee761e6c ee761e50 c01d49ec c01d4970 00000004
eeec8000 eee95c14 eee95c00
1e60: ee761e8c ee761e70 c01d49ec c01d4970 00000004 eed4e000 eee95e14 eee95e00
[ 191.208369] 1e80: ee761eac ee761e90 c01d49ec c01d4970 ee761e88
eee95e00 00000005 eee962c0
[ 191.215275] 1ea0: ee761ec4 ee761eb0 c037cec0 c01d4970 eee95e00
00000002 ee761ee4 ee761ec8
[ 191.222180] 1ec0: c01dd17c c037cea4 00000041 00000001 eec69a68
eee962c0 ee761ef4 ee761ee8
[ 191.229088] 1ee0: c0216b00 c01dd140 ee761f0c ee761ef8 c00f6a4c
c0216ae4 00000000 00000000
[ 191.235993] 1f00: ee761f44 ee761f10 c00f9cd0 c00f6a14 00000000
00000000 c00a1430 ee480140
[ 191.242898] 1f20: 00000002 b6f9e000 ee761f78 00000002 ee760000
b6f9e000 ee761f74 ee761f48
[ 191.249807] 1f40: c00a3830 c00f9be8 ee761f74 ee761f58 00000000
00000000 ee480140 ee480140
[ 191.256712] 1f60: 00000002 b6f9e000 ee761fa4 ee761f78 c00a3bc8
c00a376c 00000000 00000000
[ 191.263614] 1f80: 00000002 b6f9e000 b6edda78 00000004 c000ed84
00000000 00000000 ee761fa8
[ 191.270520] 1fa0: c000ec00 c00a3b90 00000002 b6f9e000 00000001
b6f9e000 00000002 00000000
[ 191.277424] 1fc0: 00000002 b6f9e000 b6edda78 00000004 00000002
00000002 b6f9e000 00000000
[ 191.284330] 1fe0: 00000000 bed35a1c b6e1d390 b6e6eaac 60070010
00000001 00000000 00000000
[ 191.291225] Backtrace:
[ 191.292402] [<c02510d4>] (ahci_enable_ahci) from [<c02512d8>]
(ahci_save_initial_config+0x28/0x24c)
[ 191.300171] r7:ee6ca468 r6:f0460000 r5:eee0dd10 r4:ee6ca400
[ 191.304623] [<c02512b0>] (ahci_save_initial_config) from
[<c024e7d0>] (ahci_init_one+0x384/0x850)
[ 191.312216] r10:00000005 r9:ee7e4ec8 r8:00000000 r7:00000000
r6:eee0dd10 r5:ee6ca468
[ 191.318850] r4:ee6ca400 r3:00000000
[ 191.321177] [<c024e44c>] (ahci_init_one) from [<c01dc230>]
(pci_device_probe+0x60/0xa4)
[ 191.327900] r10:ee7e4ec0 r9:ee7e4ec8 r8:c04d36c0 r7:00000000
r6:c04d36f4 r5:ee6ca400
[ 191.334530] r4:ee6ca468
[ 191.335792] [<c01dc1d0>] (pci_device_probe) from [<c0218974>]
(driver_probe_device+0x130/0x340)
[ 191.343211] r8:00000002 r7:c0218c18 r6:00000000 r5:c04d36f4
r4:ee6ca468 r3:c01dc1d0
[ 191.349766] [<c0218844>] (driver_probe_device) from [<c0218c48>]
(__device_attach+0x30/0x4c)
[ 191.356923] r6:ee6ca468 r5:ee6ca468 r4:c04d36f4
[ 191.360307] [<c0218c18>] (__device_attach) from [<c0216d30>]
(bus_for_each_drv+0x54/0x9c)
[ 191.367204] r5:ee761dd0 r4:00000000
[ 191.369531] [<c0216cdc>] (bus_for_each_drv) from [<c02187d0>]
(device_attach+0x70/0x88)
[ 191.376254] r7:ee761f78 r6:eee95a00 r5:ee6ca49c r4:ee6ca468
[ 191.380700] [<c0218760>] (device_attach) from [<c01d4810>]
(pci_bus_add_device+0x3c/0x68)
[ 191.387595] r6:eee95a00 r5:eee95a14 r4:ee6ca400 r3:00000001
[ 191.392044] [<c01d47d4>] (pci_bus_add_device) from [<c01d499c>]
(pci_bus_add_devices+0x38/0x9c)
[ 191.399463] r4:ee6ca400 r3:00000000
[ 191.401787] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 191.409293] r6:eee95800 r5:eee95814 r4:ee5fa400 r3:00000004
[ 191.413738] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 191.421244] r6:eee95c00 r5:eee95c14 r4:eeec8000 r3:00000004
[ 191.425692] [<c01d4964>] (pci_bus_add_devices) from [<c01d49ec>]
(pci_bus_add_devices+0x88/0x9c)
[ 191.433198] r6:eee95e00 r5:eee95e14 r4:eed4e000 r3:00000004
[ 191.437643] [<c01d4964>] (pci_bus_add_devices) from [<c037cec0>]
(pci_rescan_bus+0x28/0x30)
[ 191.444714] r6:eee962c0 r5:00000005 r4:eee95e00 r3:ee761e88
[ 191.449161] [<c037ce98>] (pci_rescan_bus) from [<c01dd17c>]
(bus_rescan_store+0x48/0x70)
[ 191.455971] r5:00000002 r4:eee95e00
[ 191.458292] [<c01dd134>] (bus_rescan_store) from [<c0216b00>]
(bus_attr_store+0x28/0x34)
[ 191.465102] r5:eee962c0 r4:eec69a68
[ 191.467429] [<c0216ad8>] (bus_attr_store) from [<c00f6a4c>]
(sysfs_kf_write+0x44/0x50)
[ 191.474076] [<c00f6a08>] (sysfs_kf_write) from [<c00f9cd0>]
(kernfs_fop_write+0xf4/0x148)
[ 191.480973] r5:00000000 r4:00000000
[ 191.483302] [<c00f9bdc>] (kernfs_fop_write) from [<c00a3830>]
(vfs_write+0xd0/0x17c)
[ 191.489764] r10:b6f9e000 r9:ee760000 r8:00000002 r7:ee761f78
r6:b6f9e000 r5:00000002
[ 191.496397] r4:ee480140
[ 191.497655] [<c00a3760>] (vfs_write) from [<c00a3bc8>] (SyS_write+0x44/0x84)
[ 191.503419] r10:b6f9e000 r8:00000002 r7:ee480140 r6:ee480140
r5:00000000 r4:00000000
[ 191.510063] [<c00a3b84>] (SyS_write) from [<c000ec00>]
(ret_fast_syscall+0x0/0x30)
[ 191.516350] r10:00000000 r8:c000ed84 r7:00000004 r6:b6edda78
r5:b6f9e000 r4:00000002
[ 191.522994] Code: e1a0c00d e92dd8f0 e24cb004 e2804004 (e5906004)
[ 191.527805] ---[ end trace cab2b46318ba201a ]---
On 1 April 2014 17:41, Jason Gunthorpe <jgunthorpe@obsidianresearch.com> wrote:
>
> On Mon, Mar 31, 2014 at 03:17:24PM +0100, Matthew Minter wrote:
> > Just to give an update of what I have tried. I noticed that there was
> > a PCI quirk available for a similar PLX chip which forced the kernel
> > to see the slots as hot-plug slots (unfortunately some PLX chips
> > falsely report themselves to be non hotplug capable bridges from my
> > understanding). I tried extending this quirk to run on my chip also.
> > This significantly changed the results in that along with the use of
> > pci=hpmemsize=xM I managed to force the chip to pre-allocate some
> > memory for the hotplug slot. Specifically I added this line:
> > DECLARE_PCI_FIXUP_HEADER(0x10b5, 0x8617, quirk_hotplug_bridge);
> > to linux-3.14-rc7/drivers/pci/quirks.c
>
> This all sounds right to me, it is probably the proper approach,
> things could be refined a bit by adding a DT property to indicate hot
> plug, but that won't help your oops.
>
> The oops kinda looks like something went wrong during a PCI memory
> read.
>
> > Internal error: : 1008 [#1] PREEMPT SMP ARM
>
> Which is a call to die, but the string that should tell us where is ""
> There are cases like that in the kernel, but all are prefixed by
> another line - did a line go missing (Unhandled Fault perhaps?)
>
> First guess would be something went wrong when the mbus windows were
> created after the rescan. Some debugging prints from that part of the
> flow might be informative, as well as a dump from the mbus debugfs..
>
> Second guess, something went wrong during the hot plug, and the PLX is
> not forwarding TLPs.. lpsci -v might give a clue.
>
> > I am currently working on a way to modify the scan order as Jason as
> > suggested, as soon as I have worked out exactly how the scan order is
> > determined I will try modifying the DT or any code needed to force the
> > port with the PLX chip to be scanned last. However I first need to
> > read some more of the code to understand what it orders on, eg: order
> > in DT, bus number, memory address.
>
> That suggestion was a hack, you have found the 'proper' way to do hot
> plug. If you can resolve this oops you should be golden :)
>
> Jason
--
------------------------------
For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com
------------------------------
^ permalink raw reply [flat|nested] 13+ messages in thread* Armada XP (mvebu) PCIe memory (BAR/window) re-allocation
@ 2014-03-26 16:54 Matthew Minter
0 siblings, 0 replies; 13+ messages in thread
From: Matthew Minter @ 2014-03-26 16:54 UTC (permalink / raw)
To: linux-arm-kernel
Using a custom board based around the Armada XP 230 (that's the dual
core one) we are trying to talk to some PCIe devices via a PLX PCIe
switch. However I have been able to reproduce these same results using
an Armada XP GP dev board as to show it is not a board defect causing
the issue.
We are trying to use a PCIe based ahci SSD to store some data on this
board (actually an m.2 card, but electronically identical to PCIe).
This is working well, we can see the switch, its upstream and
downstream ports and the SSD itself in the /sys/bus/pci file-system,
we can even mount the SSD and store and read data on it. This all
seems fine. We are using kernel version 3.14.0-rc7, I considered
trying a more upstream version but could not see any applicable
patches in next, or in the mvebu repo)
However the issues arrive when we try and use the hotplug features of
the board we are using (the same applies with the GP dev board and a
PLX dev board connected to it). If the SSD is connected at boot time,
we can safely hot-swap it but following the procedure: unbind the ahci
driver, remove it from the PCIe map using
/sys/bus/pci/device/xxxxx/remove, switch off the slot's power supply,
isolate the slot electronically, perform the swap, un-isolate the
slot, re-power the slot, rescan the pci bus. This also works great and
we can talk to the new SSD.
If the SSD is not plugged at boot however, this does not work properly
(This is definitely a kernel, not boot-loader issue as we can
disconnect the SSD while the bootloader is running and still have it
work as long as it gets plugged in before the kernel's PCIe
initialization). We perform the same sequence, this appears at first
to be ok, however after rescanning the pci bus, dmesg shows the
following errors (0000:01:00.0 is the switch upstream port,
0000:02:02.0 is the switch downstream port and 0000:04:00.0 is the
SSD):
[ 83.135273] pcieport 0000:01:00.0: BAR 14: can't assign mem (size 0x100000)
[ 83.135282] pcieport 0000:01:00.0: BAR 15: can't assign mem pref
(size 0x100000)
[ 83.135292] pcieport 0000:02:02.0: BAR 14: can't assign mem (size 0x100000)
[ 83.135300] pcieport 0000:02:02.0: BAR 15: can't assign mem pref
(size 0x100000)
[ 83.135309] pci 0000:04:00.0: BAR 6: can't assign mem pref (size 0x10000)
[ 83.135317] pci 0000:04:00.0: BAR 5: can't assign mem (size 0x2000)
Then when we try and attach the ahci driver, we get (predictably):
[ 83.235749] ahci 0000:04:00.0: version 3.0
[ 83.235802] ahci: probe of 0000:04:00.0 failed with error -22
It appears to me from a glance (though I have not examined the code
paths in detail) that performing a PCI rescan is not completely
reinitializing the bus, specifically it is not performing any
re-allocation of the PCI bar memory spaces which means the rescanned
devices will not be able to connect properly. Do we need to call
mvebu_pcie_setup_wins again or some other function in the kernel after
the rescan? I am a little surprised that the kernel is not
reconfiguring the BARs/windows after a rescan to fit the changed
config. Is there any way I can make this work? Any advice here would
be greatly appreciated, I would also be happy to try and write some
patches if it is really just the need to call the window and bar
initialization functions again after a rescan but would greatly
appreciate any opinions first.
Info dump ahead:
Boot log with SSD connected:
Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.14.0-rc7 (root at orangebox) (gcc version 4.7.3 (Gentoo
4.7.3-r1 p1.3, pie-0.5.5) ) #9 SMP PREEMPT Mon Mar 24 17:47:43 GMT
2014
CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Machine model: Xyratex Porter-v1
bootconsole [earlycon0] enabled
Truncating RAM at 00000000-3fffffff to -2f7fffff (vmalloc region overlap).
Memory policy: Data cache writealloc
PERCPU: Embedded 7 pages/cpu @ef1ee000 s5952 r8192 d14528 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 193040
Kernel command line: console=ttyS0,115200 earlyprintk=ttys0 rw
pm_disable mtdparts=armada-nand:8m(boot),8m(kernel),-(rootfs)
ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 766932K/778240K available (2979K kernel code, 197K rwdata,
912K rodata, 137K init, 56K bss, 11308K reserved)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
modules : 0xbf000000 - 0xc0000000 ( 16 MB)
.text : 0xc0008000 - 0xc03d4fbc (3892 kB)
.init : 0xc03d5000 - 0xc03f7740 ( 138 kB)
.data : 0xc03f8000 - 0xc04296d0 ( 198 kB)
.bss : 0xc04296dc - 0xc04378b8 ( 57 kB)
Preemptible hierarchical RCU implementation.
Dump stacks of tasks blocking RCU-preempt GP.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798691800ns
Initializing Coherency fabric
Aurora cache controller enabled
l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1024 kB
Console: colour dummy device 80x30
Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x2ef2f0 - 0x2ef324
Initializing Power Management Service Unit
Booting CPU 1
CPU1: Booted secondary processor
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated.
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6
mvebu-soc-id: MVEBU SoC ID=0x7826, Rev=0x2
pinctrl core: initialized pinctrl subsystem
DMA: preallocated 256 KiB pool for atomic coherent allocations
irq: Cannot allocate irq_descs @ IRQ32, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ67, assuming pre-allocated
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switched to clocksource armada_370_xp_clocksource
futex hash table entries: 512 (order: 3, 32768 bytes)
msgmni has been set to 1497
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
armada-xp-pinctrl f1018000.pinctrl: registered pinctrl driver
mvebu-pcie pcie-controller.2: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x1000-0xfffff]
pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe8000000]
pci_bus 0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus2: Fast back to back transfers disabled
pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus3: Fast back to back transfers enabled
PCI: bus4: Fast back to back transfers disabled
PCI: bus5: Fast back to back transfers enabled
pci 0000:00:01.0: BAR 14: assigned [mem 0xe0000000-0xe02fffff]
pci 0000:01:00.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:01:00.0: BAR 15: assigned [mem 0xe0100000-0xe01fffff pref]
pci 0000:01:00.0: BAR 0: assigned [mem 0xe0200000-0xe021ffff]
pci 0000:02:02.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:02:02.0: BAR 15: assigned [mem 0xe0100000-0xe01fffff pref]
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:04:00.0: BAR 6: assigned [mem 0xe0100000-0xe010ffff pref]
pci 0000:04:00.0: BAR 5: assigned [mem 0xe0000000-0xe0001fff]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:02.0: bridge window [mem 0xe0000000-0xe00fffff]
pci 0000:02:02.0: bridge window [mem 0xe0100000-0xe01fffff pref]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:01:00.0: PCI bridge to [bus 02-05]
pci 0000:01:00.0: bridge window [mem 0xe0000000-0xe00fffff]
pci 0000:01:00.0: bridge window [mem 0xe0100000-0xe01fffff pref]
pci 0000:00:01.0: PCI bridge to [bus 01-05]
pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe02fffff]
PCI: enabling device 0000:00:01.0 (0140 -> 0143)
PCI: enabling device 0000:01:00.0 (0140 -> 0143)
PCI: enabling device 0000:02:01.0 (0140 -> 0143)
PCI: enabling device 0000:02:02.0 (0140 -> 0143)
PCI: enabling device 0000:02:03.0 (0140 -> 0143)
mv_xor f1060900.xor: Marvell shared XOR driver
mv_xor f1060900.xor: Marvell XOR: ( xor cpy )
mv_xor f1060900.xor: Marvell XOR: ( xor cpy )
mv_xor f10f0900.xor: Marvell shared XOR driver
mv_xor f10f0900.xor: Marvell XOR: ( xor cpy )
mv_xor f10f0900.xor: Marvell XOR: ( xor cpy )
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 19, base_baud =
15625000) is a 16550A
console [ttyS0] enabled
console [ttyS0] enabled
bootconsole [earlycon0] disabled
bootconsole [earlycon0] disabled
f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 20, base_baud =
15625000) is a 16550A
mtip32xx Version 1.3.0
pxa3xx-nand f10d0000.nand: This platform can't do DMA on this device
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0x38
nand: Micron MT29F8G08ABABAWP
nand: 1024MiB, SLC, page size: 4096, OOB size: 224
Bad block table found at page 262016, version 0x01
Bad block table found at page 261888, version 0x01
3 ofpart partitions found on MTD device pxa3xx_nand-0
Creating 3 MTD partitions on "pxa3xx_nand-0":
0x000000000000-0x000000800000 : "boot"
0x000000800000-0x000001000000 : "kernel"
0x000001000000-0x000040000000 : "rootfs"
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ehci-orion: EHCI orion driver
orion-ehci f1050000.usb: EHCI Host Controller
orion-ehci f1050000.usb: new USB bus registered, assigned bus number 1
orion-ehci f1050000.usb: irq 26, io mem 0xf1050000
orion-ehci f1050000.usb: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
usbcore: registered new interface driver usb-storage
rtc-mv f1010300.rtc: rtc core: registered f1010300.rtc as rtc0
UBI: attaching mtd2 to ubi0
random: nonblocking pool is initialized
UBI: scanning is finished
UBI: attached mtd2 (name "rootfs", size 1008 MiB) to ubi0
UBI: PEB size: 524288 bytes (512 KiB), LEB size: 516096 bytes
UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
UBI: VID header offset: 4096 (aligned 4096), data offset: 8192
UBI: good PEBs: 2008, bad PEBs: 8, corrupted PEBs: 0
UBI: user volume: 1, internal volumes: 1, max. volumes count: 128
UBI: max/mean erase counter: 2/1, WL threshold: 4096, image sequence
number: 3531581739
UBI: available PEBs: 0, total reserved PEBs: 2008, PEBs reserved for
bad PEB handling: 32
UBI: background thread "ubi_bgt0d" started, PID 885
rtc-mv f1010300.rtc: setting system clock to 2000-07-04 13:58:42 UTC (962719122)
UBIFS: background thread "ubifs_bgt0_0" started, PID 887
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: LEB size: 516096 bytes (504 KiB), min./max. I/O unit sizes:
4096 bytes/4096 bytes
UBIFS: FS size: 1012064256 bytes (965 MiB, 1961 LEBs), journal size
33546240 bytes (31 MiB, 65 LEBs)
UBIFS: reserved for root: 4952683 bytes (4836 KiB)
UBIFS: media format: w4/r0 (latest is w4/r0), UUID
8BD77AAC-E543-41A2-842F-7011ADC039E0, small LPT model
VFS: Mounted root (ubifs filesystem) on device 0:10.
devtmpfs: mounted
Freeing unused kernel memory: 136K (c03d5000 - c03f7000)
Boot log with SSD disconnected:
Uncompressing Linux... done, booting the kernel.
Booting Linux on physical CPU 0x0
Linux version 3.14.0-rc7 (root at orangebox) (gcc version 4.7.3 (Gentoo
4.7.3-r1 p1.3, pie-0.5.5) ) #9 SMP PREEMPT Mon Mar 24 17:47:43 GMT
2014
CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d
CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
Machine model: Xyratex Porter-v1
bootconsole [earlycon0] enabled
Truncating RAM at 00000000-3fffffff to -2f7fffff (vmalloc region overlap).
Memory policy: Data cache writealloc
PERCPU: Embedded 7 pages/cpu @ef1ee000 s5952 r8192 d14528 u32768
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 193040
Kernel command line: console=ttyS0,115200 earlyprintk=ttys0 rw
pm_disable mtdparts=armada-nand:8m(boot),8m(kernel),-(rootfs)
ubi.mtd=2 root=ubi0:rootfs rootfstype=ubifs
PID hash table entries: 4096 (order: 2, 16384 bytes)
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 766932K/778240K available (2979K kernel code, 197K rwdata,
912K rodata, 137K init, 56K bss, 11308K reserved)
Virtual kernel memory layout:
vector : 0xffff0000 - 0xffff1000 ( 4 kB)
fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
modules : 0xbf000000 - 0xc0000000 ( 16 MB)
.text : 0xc0008000 - 0xc03d4fbc (3892 kB)
.init : 0xc03d5000 - 0xc03f7740 ( 138 kB)
.data : 0xc03f8000 - 0xc04296d0 ( 198 kB)
.bss : 0xc04296dc - 0xc04378b8 ( 57 kB)
Preemptible hierarchical RCU implementation.
Dump stacks of tasks blocking RCU-preempt GP.
RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798691800ns
Initializing Coherency fabric
Aurora cache controller enabled
l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1024 kB
Console: colour dummy device 80x30
Calibrating delay loop... 1196.85 BogoMIPS (lpj=5984256)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x2ef2f0 - 0x2ef324
Initializing Power Management Service Unit
Booting CPU 1
CPU1: Booted secondary processor
CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
Brought up 2 CPUs
SMP: Total of 2 processors activated.
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 56 architecture 2 part 20 variant 9 rev 6
mvebu-soc-id: MVEBU SoC ID=0x7826, Rev=0x2
pinctrl core: initialized pinctrl subsystem
DMA: preallocated 256 KiB pool for atomic coherent allocations
irq: Cannot allocate irq_descs @ IRQ32, assuming pre-allocated
irq: Cannot allocate irq_descs @ IRQ67, assuming pre-allocated
bio: create slab <bio-0> at 0
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Switched to clocksource armada_370_xp_clocksource
futex hash table entries: 512 (order: 3, 32768 bytes)
msgmni has been set to 1497
Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
armada-xp-pinctrl f1018000.pinctrl: registered pinctrl driver
mvebu-pcie pcie-controller.2: PCI host bridge to bus 0000:00
pci_bus 0000:00: root bus resource [io 0x1000-0xfffff]
pci_bus 0000:00: root bus resource [mem 0xe0000000-0xe8000000]
pci_bus 0000:00: root bus resource [bus 00-ff]
PCI: bus0: Fast back to back transfers disabled
pci 0000:00:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus1: Fast back to back transfers disabled
pci 0000:01:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus2: Fast back to back transfers disabled
pci 0000:02:01.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:02.0: bridge configuration invalid ([bus 00-00]), reconfiguring
pci 0000:02:03.0: bridge configuration invalid ([bus 00-00]), reconfiguring
PCI: bus3: Fast back to back transfers enabled
PCI: bus4: Fast back to back transfers enabled
PCI: bus5: Fast back to back transfers enabled
pci 0000:00:01.0: BAR 14: assigned [mem 0xe0000000-0xe00fffff]
pci 0000:01:00.0: BAR 0: assigned [mem 0xe0000000-0xe001ffff]
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:02.0: PCI bridge to [bus 04]
pci 0000:02:03.0: PCI bridge to [bus 05]
pci 0000:01:00.0: PCI bridge to [bus 02-05]
pci 0000:00:01.0: PCI bridge to [bus 01-05]
pci 0000:00:01.0: bridge window [mem 0xe0000000-0xe00fffff]
PCI: enabling device 0000:00:01.0 (0140 -> 0143)
PCI: enabling device 0000:01:00.0 (0140 -> 0143)
PCI: enabling device 0000:02:01.0 (0140 -> 0143)
PCI: enabling device 0000:02:02.0 (0140 -> 0143)
PCI: enabling device 0000:02:03.0 (0140 -> 0143)
mv_xor f1060900.xor: Marvell shared XOR driver
mv_xor f1060900.xor: Marvell XOR: ( xor cpy )
mv_xor f1060900.xor: Marvell XOR: ( xor cpy )
mv_xor f10f0900.xor: Marvell shared XOR driver
mv_xor f10f0900.xor: Marvell XOR: ( xor cpy )
mv_xor f10f0900.xor: Marvell XOR: ( xor cpy )
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
f1012000.serial: ttyS0 at MMIO 0xf1012000 (irq = 19, base_baud =
15625000) is a 16550A
console [ttyS0] enabled
console [ttyS0] enabled
bootconsole [earlycon0] disabled
bootconsole [earlycon0] disabled
f1012100.serial: ttyS1 at MMIO 0xf1012100 (irq = 20, base_baud =
15625000) is a 16550A
mtip32xx Version 1.3.0
pxa3xx-nand f10d0000.nand: This platform can't do DMA on this device
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0x38
nand: Micron MT29F8G08ABABAWP
nand: 1024MiB, SLC, page size: 4096, OOB size: 224
Bad block table found at page 262016, version 0x01
Bad block table found at page 261888, version 0x01
3 ofpart partitions found on MTD device pxa3xx_nand-0
Creating 3 MTD partitions on "pxa3xx_nand-0":
0x000000000000-0x000000800000 : "boot"
0x000000800000-0x000001000000 : "kernel"
0x000001000000-0x000040000000 : "rootfs"
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
ehci-orion: EHCI orion driver
orion-ehci f1050000.usb: EHCI Host Controller
orion-ehci f1050000.usb: new USB bus registered, assigned bus number 1
orion-ehci f1050000.usb: irq 26, io mem 0xf1050000
orion-ehci f1050000.usb: USB 2.0 started, EHCI 1.00
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
usbcore: registered new interface driver usb-storage
rtc-mv f1010300.rtc: rtc core: registered f1010300.rtc as rtc0
UBI: attaching mtd2 to ubi0
random: nonblocking pool is initialized
UBI: scanning is finished
UBI: attached mtd2 (name "rootfs", size 1008 MiB) to ubi0
UBI: PEB size: 524288 bytes (512 KiB), LEB size: 516096 bytes
UBI: min./max. I/O unit sizes: 4096/4096, sub-page size 4096
UBI: VID header offset: 4096 (aligned 4096), data offset: 8192
UBI: good PEBs: 2008, bad PEBs: 8, corrupted PEBs: 0
UBI: user volume: 1, internal volumes: 1, max. volumes count: 128
UBI: max/mean erase counter: 2/1, WL threshold: 4096, image sequence
number: 3531581739
UBI: available PEBs: 0, total reserved PEBs: 2008, PEBs reserved for
bad PEB handling: 32
UBI: background thread "ubi_bgt0d" started, PID 884
rtc-mv f1010300.rtc: setting system clock to 2000-07-04 13:58:35 UTC (962719115)
UBIFS: background thread "ubifs_bgt0_0" started, PID 886
UBIFS: recovery needed
UBIFS: recovery completed
UBIFS: mounted UBI device 0, volume 0, name "rootfs"
UBIFS: LEB size: 516096 bytes (504 KiB), min./max. I/O unit sizes:
4096 bytes/4096 bytes
UBIFS: FS size: 1012064256 bytes (965 MiB, 1961 LEBs), journal size
33546240 bytes (31 MiB, 65 LEBs)
UBIFS: reserved for root: 4952683 bytes (4836 KiB)
UBIFS: media format: w4/r0 (latest is w4/r0), UUID
8BD77AAC-E543-41A2-842F-7011ADC039E0, small LPT model
VFS: Mounted root (ubifs filesystem) on device 0:10.
devtmpfs: mounted
Freeing unused kernel memory: 136K (c03d5000 - c03f7000)
Full dmesg output after failed rescan and ahci bind:
[ 46.224192] pci_bus 0000:02: scanning bus
[ 46.224259] pcieport 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 0
[ 46.224275] pcieport 0000:02:02.0: scanning [bus 04-04] behind bridge, pass 0
[ 46.224290] pcieport 0000:02:03.0: scanning [bus 05-05] behind bridge, pass 0
[ 46.224305] pcieport 0000:02:01.0: scanning [bus 00-00] behind bridge, pass 1
[ 46.224318] pci_bus 0000:03: scanning bus
[ 46.224328] pci_bus 0000:03: bus scan returning with max=03
[ 46.224340] pci_bus 0000:03: busn_res: [bus 03] end is updated to 03
[ 46.224353] pcieport 0000:02:02.0: scanning [bus 00-00] behind bridge, pass 1
[ 46.224367] pci_bus 0000:04: scanning bus
[ 46.224410] pci 0000:04:00.0: [144d:a800] type 00 class 0x010601
[ 46.224495] pci 0000:04:00.0: reg 0x24: [mem 0x00000000-0x00001fff]
[ 46.224515] pci 0000:04:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[ 46.224536] pci 0000:04:00.0: calling pci_fixup_ide_bases+0x0/0x58
[ 46.224660] pci 0000:04:00.0: PME# supported from D3hot D3cold
[ 46.224672] pci 0000:04:00.0: PME# disabled
[ 46.225090] pci_bus 0000:04: bus scan returning with max=04
[ 46.225101] pci_bus 0000:04: busn_res: [bus 04] end is updated to 04
[ 46.225116] pcieport 0000:02:03.0: scanning [bus 00-00] behind bridge, pass 1
[ 46.225130] pci_bus 0000:05: scanning bus
[ 46.225140] pci_bus 0000:05: bus scan returning with max=05
[ 46.225148] pci_bus 0000:05: busn_res: [bus 05] end is updated to 05
[ 46.225159] pci_bus 0000:02: bus scan returning with max=05
[ 46.225167] pci_bus 0000:02: busn_res: [bus 02-05] end is updated to 05
[ 46.225178] pci_bus 0000:01: bus scan returning with max=05
[ 46.225186] pci_bus 0000:01: busn_res: [bus 01-05] end is updated to 05
[ 46.225195] pci_bus 0000:00: bus scan returning with max=05
[ 46.225267] pcieport 0000:01:00.0: BAR 14: can't assign mem (size 0x100000)
[ 46.225277] pcieport 0000:01:00.0: BAR 15: can't assign mem pref
(size 0x100000)
[ 46.225287] pcieport 0000:02:02.0: BAR 14: can't assign mem (size 0x100000)
[ 46.225295] pcieport 0000:02:02.0: BAR 15: can't assign mem pref
(size 0x100000)
[ 46.225304] pci 0000:04:00.0: BAR 6: can't assign mem pref (size 0x10000)
[ 46.225312] pci 0000:04:00.0: BAR 5: can't assign mem (size 0x2000)
[ 46.325621] ahci 0000:04:00.0: no of_node; not parsing pinctrl DT
[ 46.325656] ahci 0000:04:00.0: version 3.0
[ 46.325707] ahci: probe of 0000:04:00.0 failed with error -22
--
------------------------------
For additional information including the registered office and the treatment of Xyratex confidential information please visit www.xyratex.com
------------------------------
^ permalink raw reply [flat|nested] 13+ messages in threadend of thread, other threads:[~2014-04-08 16:34 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAFJTrDt7OrtE4B52419F2KGbmXeLR9NEcgF6yMTMUBe5vdKwfg@mail.gmail.com>
2014-03-26 17:06 ` Armada XP (mvebu) PCIe memory (BAR/window) re-allocation Thomas Petazzoni
2014-03-26 17:18 ` Jason Gunthorpe
2014-03-31 14:17 ` Matthew Minter
2014-04-01 16:41 ` Jason Gunthorpe
2014-04-03 9:31 ` Matthew Minter
[not found] ` <CAFJTrDu1=Hoa6NS3DaFphMUXgJYunybbdCoGiBVe1qKNQ4s3MA@mail.gmail.com>
2014-04-03 18:23 ` Jason Gunthorpe
2014-04-03 18:24 ` Thomas Petazzoni
2014-04-07 13:03 ` Matthew Minter
2014-04-07 16:45 ` Jason Gunthorpe
2014-04-08 10:57 ` Matthew Minter
2014-04-08 16:34 ` Jason Gunthorpe
2014-04-03 8:57 Matthew Minter
-- strict thread matches above, loose matches on Subject: below --
2014-03-26 16:54 Matthew Minter
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).