Broken PCI on Sequoia

Broken PCI on Sequoia

[Geert Uytterhoeven]

	Hi Ben, Josh,

I did some background bisecting to find out when PCI stopped working on the
AMCC EV-440EPX `Sequoia' Reference Board.

With ppc44x_defconfig + CONFIG_USB=y + CONFIG_USB_EHCI_HCD=y and a USB 2.0 PCI
card in one of the PCI slots, I get:

| ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
| ehci_hcd 0000:00:0a.2: enabling device (0000 -> 0002)
| ehci_hcd 0000:00:0a.2: EHCI Host Controller
| ehci_hcd 0000:00:0a.2: new USB bus registered, assigned bus number 1
| Machine check in kernel mode.
| Data Read PLB Error
| Oops: Machine check, sig: 7 [#1]
| PowerPC 44x Platform
| Modules linked in:
| NIP: c01acf24 LR: c019a2e0 CTR: c01ace34
| REGS: cfff7f10 TRAP: 0214   Not tainted  (2.6.29-rc3)
| MSR: 00029000 <EE,ME,CE>  CR: 28002224  XER: 20000000
| TASK = cf818400[1] 'swapper' THREAD: cf828000
| GPR00: 00000000 cf829d80 cf818400 cf9a8a00 0000002f 00000000 c024dd9c fffffffd 
| GPR08: 00000000 d10a8000 00000000 d10a8000 48002242 1001b1cc 0ffb2400 00000001 
| GPR16: 007fff13 00400458 00800000 ffffffff 007fff00 0ffadd68 00000000 00000001 
| GPR24: 00000000 c0322a88 00000010 000000a0 cf9fd000 cf83a800 cf9a8ab8 cf9a8a00 
| NIP [c01acf24] ehci_pci_setup+0xf0/0x5f0
| LR [c019a2e0] usb_add_hcd+0x1a8/0x5e8
| Call Trace:
| [cf829d80] [cf9a8a00] 0xcf9a8a00 (unreliable)
| [cf829db0] [c019a2e0] usb_add_hcd+0x1a8/0x5e8
| [cf829de0] [c01a59c4] usb_hcd_pci_probe+0x158/0x2e4
| [cf829e10] [c015c0cc] local_pci_probe+0x24/0x34
| [cf829e20] [c015c2f4] pci_device_probe+0x84/0xa8
| [cf829e50] [c017b388] driver_probe_device+0xb4/0x1e8
| [cf829e70] [c017b560] __driver_attach+0xa4/0xa8
| [cf829e90] [c017ab2c] bus_for_each_dev+0x5c/0xb0
| [cf829ec0] [c017b1a4] driver_attach+0x24/0x34
| [cf829ed0] [c017a44c] bus_add_driver+0x1d0/0x244
| [cf829ef0] [c017b780] driver_register+0x70/0x160
| [cf829f10] [c015c59c] __pci_register_driver+0x4c/0xac
| [cf829f30] [c02fb514] ehci_hcd_init+0xb0/0xf0
| [cf829f50] [c00013d8] do_one_initcall+0x34/0x1b0
| [cf829fc0] [c02e1178] kernel_init+0x94/0x100
| [cf829ff0] [c000da64] kernel_thread+0x50/0x6c
| Instruction dump:
| 2f8001b5 409eff68 801e00e0 64002000 901e00e0 74082000 813f008c 7d2b4b78 
| 913f00b8 40a2ff60 7c0004ac 7c004c2c <0c000000> 4c00012c 5400063e 7c0b0214 
| ---[ end trace 1bac7a35c80d2df5 ]---

I get a similar crash with a Sym53c876 SCSI card.

PCI seems to have stopped working between 2.6.28 and 2.6.29-rc1, due to the
following commit:

| commit 84d727a109081684c2e01b811cb0d6dc3b9380ca
| Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
| Date:   Thu Oct 9 16:58:19 2008 +0000
| 
|     powerpc/4xx: Add support for ISA holes on 4xx PCI/X/E
|     
|     This adds support for ISA memory holes on the PCI, PCI-X and
|     PCI-E busses of the 4xx platforms. The patch includes changes
|     to the Bamboo and Canyonlands device-trees to add such a hole,
|     others can be updated separately.
|     
|     The ISA memory hole is an additional outbound window configured
|     in the bridge to generate PCI cycles in the low memory addresses,
|     thus allowing to access things such as the hard-decoded VGA
|     aperture at 0xa0000..0xbffff or other similar things. It's made
|     accessible to userspace via the new legacy_mem file in sysfs for
|     which support was added by a previous patch.
|     
|     Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
|     Signed-off-by: Josh Boyer <jwboyer@linux.vnet.ibm.com>
| 
|  boot/dts/bamboo.dts      |    3 
|  boot/dts/canyonlands.dts |    3 
|  sysdev/ppc4xx_pci.c      |  306 ++++++++++++++++++++++++++++++---------------
|  3 files changed, 213 insertions(+), 99 deletions(-)

Git-reverting this commit on top of 2.6.29-rc3 makes the crash go away.

Perhaps sequoia.dts (and other 44x DTS files) had to be changed, too?

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010

Re: Broken PCI on Sequoia

[Benjamin Herrenschmidt]

On Thu, 2009-01-29 at 18:37 +0100, Geert Uytterhoeven wrote:
> 	Hi Ben, Josh,


 .../...

> Git-reverting this commit on top of 2.6.29-rc3 makes the crash go away.
> 
> Perhaps sequoia.dts (and other 44x DTS files) had to be changed, too?

Weird, maybe I have a bug when there is no ISA hole in the DT, I'll have
a look later today.

Cheers,
Ben.

Re: Broken PCI on Sequoia

[Josh Boyer]

On Fri, Jan 30, 2009 at 09:11:01AM +1100, Benjamin Herrenschmidt wrote:
>On Thu, 2009-01-29 at 18:37 +0100, Geert Uytterhoeven wrote:
>> 	Hi Ben, Josh,
>
>
> .../...
>
>> Git-reverting this commit on top of 2.6.29-rc3 makes the crash go away.
>> 
>> Perhaps sequoia.dts (and other 44x DTS files) had to be changed, too?
>
>Weird, maybe I have a bug when there is no ISA hole in the DT, I'll have
>a look later today.

Yeah.  In fact, I think you have that bug in almost every board.  You only
updated Bamboo and Canyonlands with the initial patch and the changelog
says "other boards can be updated separately."  Nobody did that.  So not
so weird after all.

josh

Re: Broken PCI on Sequoia

[Benjamin Herrenschmidt]

> Yeah.  In fact, I think you have that bug in almost every board.  You only
> updated Bamboo and Canyonlands with the initial patch and the changelog
> says "other boards can be updated separately."  Nobody did that.  So not
> so weird after all.

I still don't see off hand what's wrong in the code..

Geert, any chance you can sprinkle printk's in
ppc4xx_configure_pci_PMMs() ? I'd like to see the arguments to the
various calls to ppc4xx_setup_one_pci_PMM(), and the value of
hose->pci_mem_offset and hose->isa_mem_phys & size.

Cheers,
Ben.

Re: Broken PCI on Sequoia

[Geert Uytterhoeven]

On Fri, 30 Jan 2009, Benjamin Herrenschmidt wrote:
> > Yeah.  In fact, I think you have that bug in almost every board.  You only
> > updated Bamboo and Canyonlands with the initial patch and the changelog
> > says "other boards can be updated separately."  Nobody did that.  So not
> > so weird after all.
> 
> I still don't see off hand what's wrong in the code..
> 
> Geert, any chance you can sprinkle printk's in
> ppc4xx_configure_pci_PMMs() ? I'd like to see the arguments to the
> various calls to ppc4xx_setup_one_pci_PMM(), and the value of
> hose->pci_mem_offset and hose->isa_mem_phys & size.

| PCI host bridge /plb/pci@1ec000000 (primary) ranges:
|  MEM 0x0000000180000000..0x00000001bfffffff -> 0x0000000080000000 
|   IO 0x00000001e8000000..0x00000001e800ffff -> 0x0000000000000000
|   IO 0x00000001e8800000..0x00000001ebffffff -> 0x0000000000000000
|  \--> Skipped (too many) !
| 4xx PCI DMA offset set to 0x00000000
| ppc4xx_configure_pci_PMMs: i = 0, hose->pci_mem_offset = 0x100000000
| ppc4xx_setup_one_pci_PMM:     hose = 0xcf825000
| ppc4xx_setup_one_pci_PMM:     reg = 0xd1000000
| ppc4xx_setup_one_pci_PMM:     plb_addr = 0x180000000
| ppc4xx_setup_one_pci_PMM:     pci_addr = 0x80000000
| ppc4xx_setup_one_pci_PMM:     size = 0x40000000
| ppc4xx_setup_one_pci_PMM:     flags = 0x200
| ppc4xx_setup_one_pci_PMM:     index = 0
| /plb/pci@1ec000000: Resource out of range
                      ^^^^^^^^^^^^^^^^^^^^^
because plb_addr + size lies outside 32-bit space.

| ppc4xx_configure_pci_PMMs: hose->isa_mem_phys = 0x0, hose->isa_mem_size = 0x0
| PCI: Probing PCI hardware
| PCI: Hiding 4xx host bridge resources 0000:00:00.0
| pci 0000:00:0a.0: PME# supported from D0 D1 D2 D3hot
| pci 0000:00:0a.0: PME# disabled
| pci 0000:00:0a.1: PME# supported from D0 D1 D2 D3hot
| pci 0000:00:0a.1: PME# disabled
| pci 0000:00:0a.2: PME# supported from D0 D1 D2 D3hot
| pci 0000:00:0a.2: PME# disabled

diff --git a/arch/powerpc/sysdev/ppc4xx_pci.c b/arch/powerpc/sysdev/ppc4xx_pci.c
index 77fae5f..70684ee 100644
--- a/arch/powerpc/sysdev/ppc4xx_pci.c
+++ b/arch/powerpc/sysdev/ppc4xx_pci.c
@@ -16,6 +16,8 @@
  *
  */
 
+#define pr_fmt(fmt)  "%s: " fmt, __func__
+
 #undef DEBUG
 
 #include <linux/kernel.h>
@@ -204,6 +206,13 @@ static int __init ppc4xx_setup_one_pci_PMM(struct pci_controller	*hose,
 {
 	u32 ma, pcila, pciha;
 
+pr_info("    hose = 0x%p\n", hose);
+pr_info("    reg = 0x%p\n", reg);
+pr_info("    plb_addr = 0x%llx\n", plb_addr);
+pr_info("    pci_addr = 0x%llx\n", pci_addr);
+pr_info("    size = 0x%llx\n", size);
+pr_info("    flags = 0x%x\n", flags);
+pr_info("    index = %d\n", index);
 	if ((plb_addr + size) > 0xffffffffull || !is_power_of_2(size) ||
 	    size < 0x1000 || (plb_addr & (size - 1)) != 0) {
 		printk(KERN_WARNING "%s: Resource out of range\n",
@@ -244,6 +253,7 @@ static void __init ppc4xx_configure_pci_PMMs(struct pci_controller *hose,
 		}
 
 		/* Configure the resource */
+pr_info("i = %d, hose->pci_mem_offset = 0x%llx\n", i, hose->pci_mem_offset);
 		if (ppc4xx_setup_one_pci_PMM(hose, reg,
 					     res->start,
 					     res->start - hose->pci_mem_offset,
@@ -260,6 +270,7 @@ static void __init ppc4xx_configure_pci_PMMs(struct pci_controller *hose,
 		}
 	}
 
+pr_info("hose->isa_mem_phys = 0x%llx, hose->isa_mem_size = 0x%llx\n", hose->isa_mem_phys, hose->isa_mem_size);
 	/* Handle ISA memory hole if not already covered */
 	if (j <= 2 && !found_isa_hole && hose->isa_mem_size)
 		if (ppc4xx_setup_one_pci_PMM(hose, reg, hose->isa_mem_phys, 0,

With kind regards,

Geert Uytterhoeven
Software Architect

Sony Techsoft Centre Europe
The Corporate Village · Da Vincilaan 7-D1 · B-1935 Zaventem · Belgium

Phone:    +32 (0)2 700 8453
Fax:      +32 (0)2 700 8622
E-mail:   Geert.Uytterhoeven@sonycom.com
Internet: http://www.sony-europe.com/

A division of Sony Europe (Belgium) N.V.
VAT BE 0413.825.160 · RPR Brussels
Fortis · BIC GEBABEBB · IBAN BE41293037680010

Re: Broken PCI on Sequoia

[Benjamin Herrenschmidt]

> | PCI host bridge /plb/pci@1ec000000 (primary) ranges:
> |  MEM 0x0000000180000000..0x00000001bfffffff -> 0x0000000080000000 
> |   IO 0x00000001e8000000..0x00000001e800ffff -> 0x0000000000000000
> |   IO 0x00000001e8800000..0x00000001ebffffff -> 0x0000000000000000
> |  \--> Skipped (too many) !
> | 4xx PCI DMA offset set to 0x00000000
> | ppc4xx_configure_pci_PMMs: i = 0, hose->pci_mem_offset = 0x100000000
> | ppc4xx_setup_one_pci_PMM:     hose = 0xcf825000
> | ppc4xx_setup_one_pci_PMM:     reg = 0xd1000000
> | ppc4xx_setup_one_pci_PMM:     plb_addr = 0x180000000
> | ppc4xx_setup_one_pci_PMM:     pci_addr = 0x80000000
> | ppc4xx_setup_one_pci_PMM:     size = 0x40000000
> | ppc4xx_setup_one_pci_PMM:     flags = 0x200
> | ppc4xx_setup_one_pci_PMM:     index = 0
> | /plb/pci@1ec000000: Resource out of range
>                       ^^^^^^^^^^^^^^^^^^^^^
> because plb_addr + size lies outside 32-bit space.

Ok so the code was buggy already, the ISA hole patch just makes it
trigger...

For that sort of 4xx PHB (ie, the PCI 2.x ones, not the PCI-X nor the
PCI-E), we only know how to program 32-bit of PLB address. IE. The old
code would have cropped the plb_addr when writing to the register, the
new code complains.

I suspect some implementation support a register to put the "high" part
of the PLB address, and that it already contains 1, so the old code
would have worked by chance, the new code doesn't because it bails out.

I need to check the doco for your CPU or any other using that cell to
see who supports what regarding the location of the outbound windows in
PLB space. I think the original 440GP which I used as a basis for the
PCI 2.x host bridge only supports 32-bits here but maybe I'm just
confused.

I'll have a look next week.

Cheers,
Ben

Re: Broken PCI on Sequoia

[Benjamin Herrenschmidt]

> For that sort of 4xx PHB (ie, the PCI 2.x ones, not the PCI-X nor the
> PCI-E), we only know how to program 32-bit of PLB address. IE. The old
> code would have cropped the plb_addr when writing to the register, the
> new code complains.
> 
> I suspect some implementation support a register to put the "high" part
> of the PLB address, and that it already contains 1, so the old code
> would have worked by chance, the new code doesn't because it bails out.

Hrm... from the doco it's also one 32-bit register... I'm starting to
think that those guys always assume the top 1 bit is set or something
like that ...

The doc is unclear. Maybe somebody form AMCC can confirm ?

Cheers,
Ben.

RE: Broken PCI on Sequoia

[Feng Kan]

Hi:
   It looks like the top bit is hard coded to 1. There doesn't seem to
be anyway
Of changing it.=20

Feng Kan
AMCC Engineering

-----Original Message-----
From: linuxppc-dev-bounces+fkan=3Damcc.com@ozlabs.org
[mailto:linuxppc-dev-bounces+fkan=3Damcc.com@ozlabs.org] On Behalf Of
Benjamin Herrenschmidt
Sent: Friday, January 30, 2009 1:30 PM
To: Geert Uytterhoeven
Cc: Linux/PPC Development
Subject: Re: Broken PCI on Sequoia


> For that sort of 4xx PHB (ie, the PCI 2.x ones, not the PCI-X nor the
> PCI-E), we only know how to program 32-bit of PLB address. IE. The old
> code would have cropped the plb_addr when writing to the register, the
> new code complains.
>=20
> I suspect some implementation support a register to put the "high"
part
> of the PLB address, and that it already contains 1, so the old code
> would have worked by chance, the new code doesn't because it bails
out.

Hrm... from the doco it's also one 32-bit register... I'm starting to
think that those guys always assume the top 1 bit is set or something
like that ...

The doc is unclear. Maybe somebody form AMCC can confirm ?

Cheers,
Ben.


_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@ozlabs.org
https://ozlabs.org/mailman/listinfo/linuxppc-dev

RE: Broken PCI on Sequoia

[Benjamin Herrenschmidt]

On Fri, 2009-01-30 at 17:19 -0800, Feng Kan wrote:
> Hi:
>    It looks like the top bit is hard coded to 1. There doesn't seem to
> be anyway
> Of changing it. 

Thanks !

Would it be possible for you to check other 4xx parts using that PCI
controller as to whether the top bit is always hard-coded to 1 or it
changes from part to part ?

Thanks !

Cheers,
Ben.

> Feng Kan
> AMCC Engineering
> 
> -----Original Message-----
> From: linuxppc-dev-bounces+fkan=amcc.com@ozlabs.org
> [mailto:linuxppc-dev-bounces+fkan=amcc.com@ozlabs.org] On Behalf Of
> Benjamin Herrenschmidt
> Sent: Friday, January 30, 2009 1:30 PM
> To: Geert Uytterhoeven
> Cc: Linux/PPC Development
> Subject: Re: Broken PCI on Sequoia
> 
> 
> > For that sort of 4xx PHB (ie, the PCI 2.x ones, not the PCI-X nor the
> > PCI-E), we only know how to program 32-bit of PLB address. IE. The old
> > code would have cropped the plb_addr when writing to the register, the
> > new code complains.
> > 
> > I suspect some implementation support a register to put the "high"
> part
> > of the PLB address, and that it already contains 1, so the old code
> > would have worked by chance, the new code doesn't because it bails
> out.
> 
> Hrm... from the doco it's also one 32-bit register... I'm starting to
> think that those guys always assume the top 1 bit is set or something
> like that ...
> 
> The doc is unclear. Maybe somebody form AMCC can confirm ?
> 
> Cheers,
> Ben.
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-dev