[U-Boot-Users] MPC8560 DDR controller

[U-Boot-Users] MPC8560 DDR controller

[Murray.Jensen at csiro.au]

Hi U-Boot Users, I noticed a patch go through CVS a few days ago which
updated the DDR DLL workaround for the tqm8560 and tqm8540 boards and it
piqued my interest - mainly because we are having trouble with the DDR
memory on our MPC8560 board.

I am interested in other people's experiences with the DDR controller
on their MPC8560 based boards - either positive or negative. Here is
our tail ...

It appears that noise internal to the processor is corrupting reads from
DDR memory (by causing the DDR DLL to drop out). We have played extensively
with the DDR configuration, following instructions in AN2583 (rev 2) and
the DDR11 errata, with some success - but I don't think we have ever been
able to completely eliminate the problem, only greatly reduce its frequency.

The things that seemed to help the most were setting the Reduced Drive
Strength extended operating mode in the DDR memory chips themselves, and
slowing the bus rate from 166MHz to 150Mhz. i.e. both of these really
have nothing to do with the processor's DDR controller, or any errata
workarounds, they obviously just combine to reduce noise on the bus.

Our processors are marked "PPC8560..." rather than "MPC8560.." and are
dated 0412 (week 12 of 2004 I believe ??). We were thinking that maybe
these were pre-production chips and it might help things if we replaced
them with "younger chips". By the way, I believe our chips are version
2.0.2.

The board is currently usable for development, but would be unacceptable
for production use - but even in development, if you get a crash, you can't
really be sure if there was a real problem or whether the memory read
simply failed.

Does anyone have any experiences and/or advise they could share with us?
Thank you for listening. Cheers!
								Murray...
-- 
Murray Jensen, CSIRO Manufacturing & Infra. Tech.      Phone: +61 3 9662 7763
Locked Bag No. 9, Preston, Vic, 3072, Australia.         Fax: +61 3 9662 7853
Internet: Murray.Jensen at csiro.au

To the extent permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been maintained or
that the communication is free of errors, virus, interception or interference.

The information contained in this e-mail may be confidential or privileged.
Any unauthorised use or disclosure is prohibited. If you have received this
e-mail in error, please delete it immediately and notify Murray Jensen on
+61 3 9662 7763. Thank you.

[U-Boot-Users] MPC8560 DDR controller

[Clemens Koller]

Hello, Murray!

Murray.Jensen at csiro.au wrote:
> Hi U-Boot Users, I noticed a patch go through CVS a few days ago which
> updated the DDR DLL workaround for the tqm8560 and tqm8540 boards and it
> piqued my interest - mainly because we are having trouble with the DDR
> memory on our MPC8560 board.

How do you test the DDR memory in your case?
It would be interesting to do the same over here.
Is there some code available?

> I am interested in other people's experiences with the DDR controller
> on their MPC8560 based boards - either positive or negative. Here is
> our tail ...

Over here, I am working on an MPC8540 board from MicroSys which is
able to do ECC. Currently, I haven't seen any errors looking at the
ECC status registers after several days uptime and lots of memory io,
so I think the DDR is okay.
I turned off ECC for some performance tests (no results yet)...
never had a problem there, too.

> Our processors are marked "PPC8560..." rather than "MPC8560.." and are
> dated 0412 (week 12 of 2004 I believe ??). We were thinking that maybe
> these were pre-production chips and it might help things if we replaced
> them with "younger chips". By the way, I believe our chips are version
> 2.0.2.

Mine says: PPC8540 PX833LB 2L71V MSIA QEAD0412
working fine @166MHz

Greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19

[U-Boot-Users] MPC8560 DDR controller

[Murray.Jensen at csiro.au]

On Fri, 09 Sep 2005 11:54:15 +0200, Clemens Koller writes:
>How do you test the DDR memory in your case?
>It would be interesting to do the same over here.
>Is there some code available?

Thanks very much for the quick response... I use the "mtest" command in
U-Boot. There is also a Linux memory tester called "memtester" (URL:
http://pyropus.ca/software/memtester/ - another page that might interest
you resides at: http://linuxquality.sunsite.dk/articles/testsuites/).

But the dead giveaway is while running in linux, getting an Illegal
Instruction exception on a legal instruction in a program that ran fine
a number of times before and after, doing identical tasks (also get
random Segmentation Violations) - plus I get random exceptions in kernel
mode that crashes Linux. I am mounting the root filesystem via NFS.

It appears that I can get the memory stable in the memory testers, but
something to do with the activity that Linux generates seems to trigger
the problem (although the boot code will hang randomly if left long
enough - the processor goes into a checkstop condition - I am guessing
it gets a memory error in the middle of handling an exception - would
this do it?).

I can't really rule out some problem in the Linux virtual memory system
I suppose, but it is too suspicious. I am running with all caches turned
off, and L2 is even disabled in HID1 (along with PCI and RIO which we
dont use - I was hoping that completely disabling these parts of the chip
in DEVDISR might reduce the internal noise).

A relevant point is that our DDR memory bus tracks are relatively short, I
believe - all are the same length @ 54mm (~300ps propagation delay). I base
this on various things I have read e.g. the worked CPO example in AN2583
uses 800-1000ps of propagation delay on the PCB - three times ours (but the
difference isn't enough to make the default CPO value not work). We also
do not use ECC - I was wondering if ECC might protect others in a similar
situation e.g. if they only get single bit errors which are corrected.

I am hoping someone will reply saying "you fool - you forgot xxxxx" :-)

>Over here, I am working on an MPC8540 board from MicroSys which is
>able to do ECC. Currently, I haven't seen any errors looking at the
>ECC status registers after several days uptime and lots of memory io,
>so I think the DDR is okay.
>I turned off ECC for some performance tests (no results yet)...
>never had a problem there, too.
...
>Mine says: PPC8540 PX833LB 2L71V MSIA QEAD0412
>working fine @166MHz

OK thanks for the info - it is much appreciated. Cheers!
								Murray...
--
Murray Jensen, CSIRO Manufacturing & Infra. Tech.      Phone: +61 3 9662 7763
Locked Bag No. 9, Preston, Vic, 3072, Australia.         Fax: +61 3 9662 7853
Internet: Murray.Jensen at csiro.au

To the extent permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been maintained or
that the communication is free of errors, virus, interception or interference.

The information contained in this e-mail may be confidential or privileged.
Any unauthorised use or disclosure is prohibited. If you have received this
e-mail in error, please delete it immediately and notify Murray Jensen on
+61 3 9662 7763. Thank you.

[U-Boot-Users] MPC8560 DDR controller

[Clemens Koller]

Hi, Murray!

(Sorry, this got OT now)

Murray.Jensen at csiro.au wrote:
> Thanks very much for the quick response... I use the "mtest" command in
> U-Boot. There is also a Linux memory tester called "memtester" (URL:
> http://pyropus.ca/software/memtester/ - another page that might interest
> you resides at: http://linuxquality.sunsite.dk/articles/testsuites/).

Good stuff. Thanks!

> But the dead giveaway is while running in linux, getting an Illegal
> Instruction exception on a legal instruction in a program that ran fine
> a number of times before and after, doing identical tasks (also get
> random Segmentation Violations) - plus I get random exceptions in kernel
> mode that crashes Linux. I am mounting the root filesystem via NFS.

Okay, so, you speak about an error rate of about 1..10 cph (crash per hour)?!
Never had this over here.

> I can't really rule out some problem in the Linux virtual memory system

Oh, I wouldn't blame the kernel in that case. 2.6.8 to 2.6.13 work really
fine over here. What linux do you use?

> A relevant point is that our DDR memory bus tracks are relatively short, I
> believe - all are the same length @ 54mm (~300ps propagation delay). I base
> this on various things I have read e.g. the worked CPO example in AN2583
> uses 800-1000ps of propagation delay on the PCB - three times ours (but the
> difference isn't enough to make the default CPO value not work). We also
> do not use ECC - I was wondering if ECC might protect others in a similar
> situation e.g. if they only get single bit errors which are corrected.

Check the datasheet according ECC. AFAIK it can correct single-bit errors
and detect double-bit errors. If you would have the chance to have an ECCable
system you can pretty easily check the reliability of the DDR looking at
the ECC status registers.
And just for completeness:
It's possible to turn off ECC during linux-runtime pretty reliable
with something like this:

	tmp=immap->im_ddr.sdram_cfg;
	mb();	/* sync */
        wmb();	/* eieio */
        immap->im_ddr.sdram_cfg = tmp & ~0x20000000;
        mb();	/* sync */
        wmb();	/* eieio*/

> I am hoping someone will reply saying "you fool - you forgot xxxxx" :-)

There is a story around here (from a guy who did a DDR for MPC8540)
that you might have to put on really good capacitors on some of the
ddr vref voltages on the CPU side because something is bouncing there
more than you would expect...

You also might want to check what happens if you increase/decrease
supply voltages here and there... well... you know, all these EE tricks.

Greets,

Clemens Koller
_______________________________
R&D Imaging Devices
Anagramm GmbH
Rupert-Mayer-Str. 45/1
81379 Muenchen
Germany

http://www.anagramm.de
Phone: +49-89-741518-50
Fax: +49-89-741518-19

[U-Boot-Users] MPC8560 DDR controller

[Murray.Jensen at csiro.au]

On Fri, 09 Sep 2005 14:37:21 +0200, Clemens Koller writes:
>> I can't really rule out some problem in the Linux virtual memory system
>
>Oh, I wouldn't blame the kernel in that case. 2.6.8 to 2.6.13 work really
>fine over here. What linux do you use?

We are using 2.6.13, with some patches from the ozlabs patch tracker.

>If you would have the chance to have an ECCable
>system you can pretty easily check the reliability of the DDR looking at
>the ECC status registers.

Unfortunately, this would require a board re-design.

>There is a story around here (from a guy who did a DDR for MPC8540)
>that you might have to put on really good capacitors on some of the
>ddr vref voltages on the CPU side because something is bouncing there
>more than you would expect...

Thanks - we will take a look at them ...

>You also might want to check what happens if you increase/decrease
>supply voltages here and there... well... you know, all these EE tricks.

Your response is very much appreciated. Cheers!
								Murray...
-- 
Murray Jensen, CSIRO Manufacturing & Infra. Tech.      Phone: +61 3 9662 7763
Locked Bag No. 9, Preston, Vic, 3072, Australia.         Fax: +61 3 9662 7853
Internet: Murray.Jensen at csiro.au

To the extent permitted by law, CSIRO does not represent, warrant and/or
guarantee that the integrity of this communication has been maintained or
that the communication is free of errors, virus, interception or interference.

The information contained in this e-mail may be confidential or privileged.
Any unauthorised use or disclosure is prohibited. If you have received this
e-mail in error, please delete it immediately and notify Murray Jensen on
+61 3 9662 7763. Thank you.