[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Dave Littell]

Hi all,

I'm working on an AMCC PPC440EPx-based platform that's similar to the
Sequoia.  The board runs pretty well, but occasionally takes exceptions
both in U-Boot and while running Linux.  The exceptions vary from
Illegal Instructions to FP Unavailable to FP Unimplemented to ITLB and
DTLB Errors, to DSI, etc., which made us believe there's a problem with
SDRAM.  Sometimes there are simple hard lockups from which even the JTAG
can't regain control.  However, the SDRAM physical/electrical and DDR
Controller configurations have been investigated in detail (and some
corrections made) with no resulting improvement in the exception behavior.

We see problems primarily at 667 MHz but have noted some issues at 533
MHz as well.  Some boards seem particularly susceptible to this while
others rarely (if ever) exhibit the problem.

At this point all possibilities are on the table and I'm looking for any
input from anyone with experience (good, bad, or whatever) with this
processor and/or designs similar to the Sequoia.


Thanks very much,
Dave

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Stefan Roese]

On Wednesday 23 April 2008, Dave Littell wrote:
> I'm working on an AMCC PPC440EPx-based platform that's similar to the
> Sequoia.  The board runs pretty well, but occasionally takes exceptions
> both in U-Boot and while running Linux.  The exceptions vary from
> Illegal Instructions to FP Unavailable to FP Unimplemented to ITLB and
> DTLB Errors, to DSI, etc., which made us believe there's a problem with
> SDRAM.

Yes, this is my first idea too. Almost every time such "random" errors are 
seen, SDRAM setup/initialization is the cause for it.

> Sometimes there are simple hard lockups from which even the JTAG 
> can't regain control.  However, the SDRAM physical/electrical and DDR
> Controller configurations have been investigated in detail (and some
> corrections made) with no resulting improvement in the exception behavior.

Is the SDRAM termination similar to the one used on Sequoia too? Are you using 
soldered chips and not DIMM modules? Is ECC available?

> We see problems primarily at 667 MHz but have noted some issues at 533
> MHz as well.  Some boards seem particularly susceptible to this while
> others rarely (if ever) exhibit the problem.
>
> At this point all possibilities are on the table and I'm looking for any
> input from anyone with experience (good, bad, or whatever) with this
> processor and/or designs similar to the Sequoia.

If you scan the U-Boot mailing list for 440EPx/Denali DDR2 problems, you will 
most likely find some references.

Please note that I recently introduced a CFG_MEM_TOP_HIDE option for the 
440EPx CHIP 11 errata. I suggest you take a look at this too and see if this 
changes your behavior.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Josh Boyer]

On Wed, 2008-04-23 at 14:49 +0200, Stefan Roese wrote:
> On Wednesday 23 April 2008, Dave Littell wrote:
> > At this point all possibilities are on the table and I'm looking for any
> > input from anyone with experience (good, bad, or whatever) with this
> > processor and/or designs similar to the Sequoia.
> 
> If you scan the U-Boot mailing list for 440EPx/Denali DDR2 problems, you will 
> most likely find some references.
> 
> Please note that I recently introduced a CFG_MEM_TOP_HIDE option for the 
> 440EPx CHIP 11 errata. I suggest you take a look at this too and see if this 
> changes your behavior.

Explain this a bit more please?  Is a kernel change needed here?

josh

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Stefan Roese]

On Thursday 24 April 2008, Josh Boyer wrote:
> > Please note that I recently introduced a CFG_MEM_TOP_HIDE option for the
> > 440EPx CHIP 11 errata. I suggest you take a look at this too and see if
> > this changes your behavior.
>
> Explain this a bit more please?  Is a kernel change needed here?

This depends. When the bootwrapper version is used then yes, the kernel should 
get changed. This is because the bootwrapper detects the SDRAM size from the 
DDR2 controller and passes it to Linux.

Without bootwrapper no changes are needed, since U-Boot already passes the 
corrected memory size to Linux (totalsize-4k currently).

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Josh Boyer]

On Thu, 2008-04-24 at 06:36 +0200, Stefan Roese wrote:
> On Thursday 24 April 2008, Josh Boyer wrote:
> > > Please note that I recently introduced a CFG_MEM_TOP_HIDE option for the
> > > 440EPx CHIP 11 errata. I suggest you take a look at this too and see if
> > > this changes your behavior.
> >
> > Explain this a bit more please?  Is a kernel change needed here?
> 
> This depends. When the bootwrapper version is used then yes, the kernel should 
> get changed. This is because the bootwrapper detects the SDRAM size from the 
> DDR2 controller and passes it to Linux.
> 
> Without bootwrapper no changes are needed, since U-Boot already passes the 
> corrected memory size to Linux (totalsize-4k currently).

Hm.  Given that AMCC ships the boards with an older U-Boot that requires
cuImages, I think we'll need to patch the wrapper.  Does 440GRx share
this same errata?  I would think so.

josh

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Stefan Roese]

On Thursday 24 April 2008, Josh Boyer wrote:
> > This depends. When the bootwrapper version is used then yes, the kernel
> > should get changed. This is because the bootwrapper detects the SDRAM
> > size from the DDR2 controller and passes it to Linux.
> >
> > Without bootwrapper no changes are needed, since U-Boot already passes
> > the corrected memory size to Linux (totalsize-4k currently).
>
> Hm.  Given that AMCC ships the boards with an older U-Boot that requires
> cuImages, I think we'll need to patch the wrapper.

Yes, this should be done too. I forgot about it since I usually don't use the 
wrapper.

> Does 440GRx share 
> this same errata?  I would think so.

Yep. Same problem on 440GRx.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Dave Littell]

Stefan Roese wrote:
> On Wednesday 23 April 2008, Dave Littell wrote:
>> I'm working on an AMCC PPC440EPx-based platform that's similar to the
>> Sequoia.  The board runs pretty well, but occasionally takes exceptions
>> both in U-Boot and while running Linux.  The exceptions vary from
>> Illegal Instructions to FP Unavailable to FP Unimplemented to ITLB and
>> DTLB Errors, to DSI, etc., which made us believe there's a problem with
>> SDRAM.
> 
> Yes, this is my first idea too. Almost every time such "random" errors are 
> seen, SDRAM setup/initialization is the cause for it.
> 
>> Sometimes there are simple hard lockups from which even the JTAG 
>> can't regain control.  However, the SDRAM physical/electrical and DDR
>> Controller configurations have been investigated in detail (and some
>> corrections made) with no resulting improvement in the exception behavior.
> 
> Is the SDRAM termination similar to the one used on Sequoia too? Are you using 
> soldered chips and not DIMM modules? Is ECC available?
> 

Termination: very likely - they stayed very close to Sequoia, but I'll ask.

We're using soldered chips and there's no ECC.

>> We see problems primarily at 667 MHz but have noted some issues at 533
>> MHz as well.  Some boards seem particularly susceptible to this while
>> others rarely (if ever) exhibit the problem.
>>
>> At this point all possibilities are on the table and I'm looking for any
>> input from anyone with experience (good, bad, or whatever) with this
>> processor and/or designs similar to the Sequoia.
> 
> If you scan the U-Boot mailing list for 440EPx/Denali DDR2 problems, you will 
> most likely find some references.
> 

Will do, thanks for the pointer.

> Please note that I recently introduced a CFG_MEM_TOP_HIDE option for the 
> 440EPx CHIP 11 errata. I suggest you take a look at this too and see if this 
> changes your behavior.
> 

I remember this - PPC440EPx Revision A errata CHIP_11, right?


Thanks,
Dave

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Mike Nuss]

Stefan wrote:
> >
> > At this point all possibilities are on the table and I'm 
> looking for any
> > input from anyone with experience (good, bad, or whatever) with this
> > processor and/or designs similar to the Sequoia.
> 
> If you scan the U-Boot mailing list for 440EPx/Denali DDR2 
> problems, you will 
> most likely find some references.

I definitely recommend using the latest U-Boot with the SPD-based Denali
code, if you're using DIMM modules.

A smoking gun for us was the timing of the address clocking. Look at the
lines on a scope; you may find that the clock is hitting too soon (before
the address is stable). (If so, you'll need to delay the clock on your
board.)

Hope this helps,
Mike

[U-Boot-Users] AMCC PPC440EPx/sequoia stability question...

[Niklaus Giger]

Hi Dave

Dave Littell wrote:
> Hi all,
> 
> I'm working on an AMCC PPC440EPx-based platform that's similar to the
> Sequoia. ?The board runs pretty well, but occasionally takes exceptions
> both in U-Boot and while running Linux. ?The exceptions vary from
> Illegal Instructions to FP Unavailable to FP Unimplemented to ITLB and
> DTLB Errors, to DSI, etc., which made us believe there's a problem with
> SDRAM. ?Sometimes there are simple hard lockups from which even the JTAG
> can't regain control. ?However, the SDRAM physical/electrical and DDR
> Controller configurations have been investigated in detail (and some
> corrections made) with no resulting improvement in the exception behavior.
> 
> We see problems primarily at 667 MHz but have noted some issues at 533
> MHz as well. ?Some boards seem particularly susceptible to this while
> others rarely (if ever) exhibit the problem.
> 
> At this point all possibilities are on the table and I'm looking for any
> input from anyone with experience (good, bad, or whatever) with this
> processor and/or designs similar to the Sequoia.
> 
We at netstal.com have here a new PPC440EPx based board called HCU5,
which runs well u-boot and vxWorks 6.4. Last August I had also linux
running which compiled on a NFS root its own kernel for a weekend.

Our HW is stable (running at 667 MHz) and we never had exceptions, which we
could not explain after looking around for its case.

We had (and still have) problems getting our SW (about ?1 M LOC) correctly,
as the PPC440 has a lot of internal concurrency. We had a case were two
lines of C-Code which read some external HW-register were read in the
wrong order the loop was running twice or more times. Therefore we had
to look very carefully at how external devices were accessed and are using
memory barrier like in16/out16 a lot.

I have also seen JTAG lockups but could never figure out why.

Best regards

Niklaus