Linux 2.4.27-pre6-bk hangs on MPC8540 board

Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Cordes, Aaron M]

We are bringing up an 8540-based board, and are getting hangs at random
spots when booting Linux.  Sometimes the hangs occur while the kernel is
initializing, sometimes the boot gets far enough where I can log in and
run a couple of programs.  Eventually the board locks up with no oopses
or panics.  If I connect to the board with a debugger, the processor is
usually stopped in the Data miss exception handler, and there is usually
memory corruption in the kernel code.

I ran memory tests using both Edink and U-boot and they all passed, so
I'm reluctant to blame hardware for the memory corruption.  Has anyone
seen similar behavior, or have any suggestions of things I might try?

Thanks in advance
Aaron Cordes

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Marc Leeman]

> seen similar behavior, or have any suggestions of things I might try?

no, only that 2.4.26 is running rock solid on an MPC8245 based board.

 -marc.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Wolfgang Denk]

In message <EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you wrote:
>
> We are bringing up an 8540-based board, and are getting hangs at random
> spots when booting Linux.  Sometimes the hangs occur while the kernel is
> initializing, sometimes the boot gets far enough where I can log in and
> run a couple of programs.  Eventually the board locks up with no oopses
> or panics.  If I connect to the board with a debugger, the processor is
> usually stopped in the Data miss exception handler, and there is usually
> memory corruption in the kernel code.

That's it: memory problems.

> I ran memory tests using both Edink and U-boot and they all passed, so
> I'm reluctant to blame hardware for the memory corruption.  Has anyone
> seen similar behavior, or have any suggestions of things I might try?

All those memory tests only can test simple read and write  accesses;
they  all  fail to access burst mode accesses, which will happen when
Linux starts runing, and which are failing.

Your SDRAM initialization is bad, and  the  system  crashes  when  it
tries to fetch instructions from RAM. Note that simple read and write
accesses may still work, it's the burst mode that is failing.

It is NOT sufficient to program the memory controller  of  your  CPU;
each  SDRAM  chip  also  requires  a specific initialization sequence
which you must adhere  to  to  the  letter  -  check  with  the  chip
manufacturer's manual.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
If this is a service economy, why is the service so bad?

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Cordes, Aaron M]

Is there a way in Linux or the 8540 to prevent burst accesses, in order
to test whether they are the problem?  I scanned the processor's
reference manual, but I didn't see anything that would let me do this.

Thanks
Aaron

Aaron Cordes

-----Original Message-----
From: wd@denx.de [mailto:wd@denx.de]
Sent: Thursday, August 12, 2004 9:08 AM
To: Cordes, Aaron M
Cc: linuxppc-embedded@lists.linuxppc.org
Subject: Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board


In message
<EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you
wrote:
>
> We are bringing up an 8540-based board, and are getting hangs at
> random spots when booting Linux.  Sometimes the hangs occur while the
> kernel is initializing, sometimes the boot gets far enough where I can

> log in and run a couple of programs.  Eventually the board locks up
> with no oopses or panics.  If I connect to the board with a debugger,
> the processor is usually stopped in the Data miss exception handler,
> and there is usually memory corruption in the kernel code.

That's it: memory problems.

> I ran memory tests using both Edink and U-boot and they all passed, so

> I'm reluctant to blame hardware for the memory corruption.  Has anyone

> seen similar behavior, or have any suggestions of things I might try?

All those memory tests only can test simple read and write  accesses;
they  all  fail to access burst mode accesses, which will happen when
Linux starts runing, and which are failing.

Your SDRAM initialization is bad, and  the  system  crashes  when  it
tries to fetch instructions from RAM. Note that simple read and write
accesses may still work, it's the burst mode that is failing.

It is NOT sufficient to program the memory controller  of  your  CPU;
each  SDRAM  chip  also  requires  a specific initialization sequence
which you must adhere  to  to  the  letter  -  check  with  the  chip
manufacturer's manual.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de If
this is a service economy, why is the service so bad?

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Cordes, Aaron M]

I should mention that our SDRAM is connected to the 8540's DDR
controller, not LocalBus.

Aaron

-----Original Message-----
From: Cordes, Aaron M
Sent: Thursday, August 12, 2004 2:34 PM
To: 'wd@denx.de'
Cc: linuxppc-embedded@lists.linuxppc.org
Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board


Is there a way in Linux or the 8540 to prevent burst accesses, in order
to test whether they are the problem?  I scanned the processor's
reference manual, but I didn't see anything that would let me do this.

Thanks
Aaron

Aaron Cordes

-----Original Message-----
From: wd@denx.de [mailto:wd@denx.de]
Sent: Thursday, August 12, 2004 9:08 AM
To: Cordes, Aaron M
Cc: linuxppc-embedded@lists.linuxppc.org
Subject: Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board


In message
<EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you
wrote:
>
> We are bringing up an 8540-based board, and are getting hangs at
> random spots when booting Linux.  Sometimes the hangs occur while the
> kernel is initializing, sometimes the boot gets far enough where I can

> log in and run a couple of programs.  Eventually the board locks up
> with no oopses or panics.  If I connect to the board with a debugger,
> the processor is usually stopped in the Data miss exception handler,
> and there is usually memory corruption in the kernel code.

That's it: memory problems.

> I ran memory tests using both Edink and U-boot and they all passed, so
> I'm reluctant to blame hardware for the memory corruption.  Has anyone

> seen similar behavior, or have any suggestions of things I might try?

All those memory tests only can test simple read and write  accesses;
they  all  fail to access burst mode accesses, which will happen when
Linux starts runing, and which are failing.

Your SDRAM initialization is bad, and  the  system  crashes  when  it
tries to fetch instructions from RAM. Note that simple read and write
accesses may still work, it's the burst mode that is failing.

It is NOT sufficient to program the memory controller  of  your  CPU;
each  SDRAM  chip  also  requires  a specific initialization sequence
which you must adhere  to  to  the  letter  -  check  with  the  chip
manufacturer's manual.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de If
this is a service economy, why is the service so bad?

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Cordes, Aaron M]

I completely disabled L1 and L2 cache in the processor and I am not
seeing the memory corruption anymore.  So, good news is board is now
booting, bad news is it's dog slow and we have a memory design to debug.
Thanks to everyone for your prompt and helpful advice.

Aaron


-----Original Message-----
From: VanBaren, Gerald (AGRE)
[mailto:Gerald.VanBaren@smiths-aerospace.com]
Sent: Thursday, August 12, 2004 3:32 PM
To: Cordes, Aaron M
Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board



On 8xx and 82xx and, I presume, 85xx processors, disabling cache
inhibits burst mode.  It is the cache line filling  and flushing that
causes burst accesses.  Note that DMA also causes bursts, but that
becomes a problem much later.

gvb


> -----Original Message-----
> From: owner-linuxppc-embedded@lists.linuxppc.org
> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of
> Cordes, Aaron M
> Sent: Thursday, August 12, 2004 3:34 PM
> To: wd@denx.de
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>
>
>
> Is there a way in Linux or the 8540 to prevent burst accesses, in
> order to test whether they are the problem?  I scanned the processor's
> reference manual, but I didn't see anything that would let me do this.
>
> Thanks
> Aaron
>
> Aaron Cordes
>
> -----Original Message-----
> From: wd@denx.de [mailto:wd@denx.de]
> Sent: Thursday, August 12, 2004 9:08 AM
> To: Cordes, Aaron M
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>
>
> In message
> <EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you
> wrote:
> >
> > We are bringing up an 8540-based board, and are getting hangs at
> > random spots when booting Linux.  Sometimes the hangs occur
> while the
> > kernel is initializing, sometimes the boot gets far enough
> where I can
>
> > log in and run a couple of programs.  Eventually the board locks up
> > with no oopses or panics.  If I connect to the board with a
> debugger,
> > the processor is usually stopped in the Data miss exception handler,

> > and there is usually memory corruption in the kernel code.
>
> That's it: memory problems.
>
> > I ran memory tests using both Edink and U-boot and they all
> passed, so
>
> > I'm reluctant to blame hardware for the memory corruption.
> Has anyone
>
> > seen similar behavior, or have any suggestions of things I
> might try?
>
> All those memory tests only can test simple read and write  accesses;
> they  all  fail to access burst mode accesses, which will happen when
> Linux starts runing, and which are failing.
>
> Your SDRAM initialization is bad, and  the  system  crashes  when  it
> tries to fetch instructions from RAM. Note that simple read and write
> accesses may still work, it's the burst mode that is failing.
>
> It is NOT sufficient to program the memory controller  of  your  CPU;
> each  SDRAM  chip  also  requires  a specific initialization sequence
> which you must adhere  to  to  the  letter  -  check  with  the  chip
> manufacturer's manual.
>
> Best regards,
>
> Wolfgang Denk
>
> --
> Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
> Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
> If this is a service economy, why is the service so bad?
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Kumar Gala]

See you are still having issues.  A few questions:
1. What version of the silicon do you have?
2. What board is this on (customer, ADS, etc.)?
3. What firmware/bootloader (and if u-boot, what version)

thanks

- kumar

On Aug 12, 2004, at 2:38 PM, Cordes, Aaron M wrote:

>
> I should mention that our SDRAM is connected to the 8540's DDR
> controller, not LocalBus.
>
> Aaron
>
> -----Original Message-----
> From: Cordes, Aaron M
> Sent: Thursday, August 12, 2004 2:34 PM
> To: 'wd@denx.de'
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>
>
> Is there a way in Linux or the 8540 to prevent burst accesses, in order
> to test whether they are the problem?  I scanned the processor's
> reference manual, but I didn't see anything that would let me do this.
>
> Thanks
> Aaron
>
> Aaron Cordes
>
> -----Original Message-----
> From: wd@denx.de [mailto:wd@denx.de]
> Sent: Thursday, August 12, 2004 9:08 AM
> To: Cordes, Aaron M
> Cc: linuxppc-embedded@lists.linuxppc.org
> Subject: Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>
>
> In message
> <EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you
> wrote:
>>
>> We are bringing up an 8540-based board, and are getting hangs at
>> random spots when booting Linux.  Sometimes the hangs occur while the
>> kernel is initializing, sometimes the boot gets far enough where I can
>
>> log in and run a couple of programs.  Eventually the board locks up
>> with no oopses or panics.  If I connect to the board with a debugger,
>> the processor is usually stopped in the Data miss exception handler,
>> and there is usually memory corruption in the kernel code.
>
> That's it: memory problems.
>
>> I ran memory tests using both Edink and U-boot and they all passed, so
>> I'm reluctant to blame hardware for the memory corruption.  Has anyone
>
>> seen similar behavior, or have any suggestions of things I might try?
>
> All those memory tests only can test simple read and write  accesses;
> they  all  fail to access burst mode accesses, which will happen when
> Linux starts runing, and which are failing.
>
> Your SDRAM initialization is bad, and  the  system  crashes  when  it
> tries to fetch instructions from RAM. Note that simple read and write
> accesses may still work, it's the burst mode that is failing.
>
> It is NOT sufficient to program the memory controller  of  your  CPU;
> each  SDRAM  chip  also  requires  a specific initialization sequence
> which you must adhere  to  to  the  letter  -  check  with  the  chip
> manufacturer's manual.
>
> Best regards,
>
> Wolfgang Denk
>
> --
> Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
> Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
> If
> this is a service economy, why is the service so bad?
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Swarthout Edward L-SWARTHOU]

> We are bringing up an 8540-based board, and are getting hangs at
> random spots when booting Linux.  Sometimes the hangs occur while the

Indeed, classic memory corruption symptoms.

> Is there a way in Linux or the 8540 to prevent burst accesses,
> in order to test whether they are the problem?

Instead of trying to mask the problem by using crawl modes, I would suggest
proving that it is a DDR interface problem by enabling ECC (of course you
need to be able to plug in an ECC memory).  Most interface corruptions will
be detected by the ECC logic.

It is much easier to resolve board or DDR configuration issues when hardware clearly reports an ECC error instead of causing code to go wild.

u-boot supports enabling ECC, but it does not report its occurrence.

Ed dot Swarthout at Freescale dot com


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Conor McLoughlin]

Disable the cache. The processor uses a burst access to fill a cache line.

Conor

Cordes, Aaron M wrote:
> Is there a way in Linux or the 8540 to prevent burst accesses, in order
> to test whether they are the problem?  I scanned the processor's
> reference manual, but I didn't see anything that would let me do this.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Kumar Gala]

Try just enabling the L1 caches and see if the issue still exists.

- kumar

On Aug 12, 2004, at 5:07 PM, Cordes, Aaron M wrote:

>
> I completely disabled L1 and L2 cache in the processor and I am not
> seeing the memory corruption anymore.  So, good news is board is now
> booting, bad news is it's dog slow and we have a memory design to
> debug.
> Thanks to everyone for your prompt and helpful advice.
>
> Aaron
>
>
> -----Original Message-----
> From: VanBaren, Gerald (AGRE)
> [mailto:Gerald.VanBaren@smiths-aerospace.com]
> Sent: Thursday, August 12, 2004 3:32 PM
> To: Cordes, Aaron M
> Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>
>
>
> On 8xx and 82xx and, I presume, 85xx processors, disabling cache
> inhibits burst mode.  It is the cache line filling  and flushing that
> causes burst accesses.  Note that DMA also causes bursts, but that
> becomes a problem much later.
>
> gvb
>
>
>> -----Original Message-----
>> From: owner-linuxppc-embedded@lists.linuxppc.org
>> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of
>> Cordes, Aaron M
>> Sent: Thursday, August 12, 2004 3:34 PM
>> To: wd@denx.de
>> Cc: linuxppc-embedded@lists.linuxppc.org
>> Subject: RE: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>>
>>
>>
>> Is there a way in Linux or the 8540 to prevent burst accesses, in
>> order to test whether they are the problem?  I scanned the processor's
>> reference manual, but I didn't see anything that would let me do this.
>>
>> Thanks
>> Aaron
>>
>> Aaron Cordes
>>
>> -----Original Message-----
>> From: wd@denx.de [mailto:wd@denx.de]
>> Sent: Thursday, August 12, 2004 9:08 AM
>> To: Cordes, Aaron M
>> Cc: linuxppc-embedded@lists.linuxppc.org
>> Subject: Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board
>>
>>
>> In message
>> <EE7CAF713952AC4A970DD7CD8B4C12A042DCA7@emss09m06.us.lmco.com> you
>> wrote:
>>>
>>> We are bringing up an 8540-based board, and are getting hangs at
>>> random spots when booting Linux.  Sometimes the hangs occur
>> while the
>>> kernel is initializing, sometimes the boot gets far enough
>> where I can
>>
>>> log in and run a couple of programs.  Eventually the board locks up
>>> with no oopses or panics.  If I connect to the board with a
>> debugger,
>>> the processor is usually stopped in the Data miss exception handler,
>
>>> and there is usually memory corruption in the kernel code.
>>
>> That's it: memory problems.
>>
>>> I ran memory tests using both Edink and U-boot and they all
>> passed, so
>>
>>> I'm reluctant to blame hardware for the memory corruption.
>> Has anyone
>>
>>> seen similar behavior, or have any suggestions of things I
>> might try?
>>
>> All those memory tests only can test simple read and write  accesses;
>> they  all  fail to access burst mode accesses, which will happen when
>> Linux starts runing, and which are failing.
>>
>> Your SDRAM initialization is bad, and  the  system  crashes  when  it
>> tries to fetch instructions from RAM. Note that simple read and write
>> accesses may still work, it's the burst mode that is failing.
>>
>> It is NOT sufficient to program the memory controller  of  your  CPU;
>> each  SDRAM  chip  also  requires  a specific initialization sequence
>> which you must adhere  to  to  the  letter  -  check  with  the  chip
>> manufacturer's manual.
>>
>> Best regards,
>>
>> Wolfgang Denk
>>
>> --
>> Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
>> Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
>> If this is a service economy, why is the service so bad?
>>
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Linux 2.4.27-pre6-bk hangs on MPC8540 board

[Chris Zimman]

On Thu, Aug 12, 2004 at 05:07:30PM -0500, Cordes, Aaron M wrote:

> I completely disabled L1 and L2 cache in the processor and I am not
> seeing the memory corruption anymore.  So, good news is board is now
> booting, bad news is it's dog slow and we have a memory design to debug.
> Thanks to everyone for your prompt and helpful advice.

Before you spend too much time debugging the memory design, did you follow
Wolfgang's advice and take another look at your memory controller setup?

If it's working with cache disabled, and then failing in burst mode, 9 out of
10 times, it's because the SDRAM controller setup is wrong.  It only has to
be very subtly wrong to fail like this.

If you're using DIMMs versus having memory right on the board, are you
reading from the SPD for the config parameters?

--Chris

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/