[U-Boot] amcc kilauea odd crashes

[U-Boot] amcc kilauea odd crashes

[Markus Klotzbücher]

Hi List,

Just updated my kilauea to top of git (2008.10-00092-gd9d8c7c) and am
seeing odd crashes:

 - sometimes u-boot comes up normally, but printenv displays random
   crap.

 - sometimes I get machine checks such as here [1].

 - sometimes it resets without any message at all.

The major visible change since the last update is the "DRAM: auto
calibration" eye candy.

Does anyone see any similar problems? To me this smells like a faulty
DRAM configuration.

Best regards
Markus 

P.S.: I'll go back and test some prior version, but unfortunaltely the
board is bricked and I first need to find a BDI.


[1]:


U-Boot 2008.10-00092-gd9d8c7c (Oct 23 2008 - 10:59:45)
CPU:   AMCC PowerPC 405EX Rev. A at 400 MHz (PLB=200, OPB=100, EBC=100
MHz)
       Security support
       Bootstrap Option H - Boot ROM Location I2C (Addr 0x52)
       16 kB I-Cache 16 kB D-Cache
Board: Kilauea - AMCC PPC405EX Evaluation Board
I2C:   ready
DTT1:  44 C
DRAM:  256 MB
FLASH: NIP: 0FFADE30 XER: 00000000 LR: 0FFADEC8 REGS: 0fea0d20 TRAP:
0700 DEAR: 00000000
MSR: 00021000 EE: 0 PR: 0 FP: 0 ME: 1 IR/DR: 00

GPR00: 0FEA0020 0FEA0E10 0FEA0F44 0FFF8DFC 000000F0 0FEA0E2F 000000F0
5D396680
GPR08: 0FEA0E1C 0FEA0E1C 00000000 00000000 00002FAF 00000000 0FFF1800
10005000
GPR16: 0FFE6A14 0FFEC8E8 00000000 00000000 00000001 00000000 00000000
00000000
GPR24: 00000000 FC000000 00000000 00000000 0FFF8DFC 000000F0 0FEAAA00
17D7FFE8
** Illegal Instruction **
Call backtrace:
17D78400 0FFADEB4 0FFAED50 0FFAF398 0FFA8DC8 0FFA76A4
Program Check Exception

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de")

[U-Boot] amcc kilauea odd crashes

[Stefan Roese]

Hi Markus,

On Thursday 23 October 2008, Markus Klotzb?cher wrote:
> Just updated my kilauea to top of git (2008.10-00092-gd9d8c7c) and am
> seeing odd crashes:

Is this a 600MHz Kilauea? This is a known issue, that the new autocalibration 
code has a problem here. I reported this problem a few weeks ago. AMCC is 
currently working on a fix for this.

>  - sometimes u-boot comes up normally, but printenv displays random
>    crap.
>
>  - sometimes I get machine checks such as here [1].
>
>  - sometimes it resets without any message at all.
>
> The major visible change since the last update is the "DRAM: auto
> calibration" eye candy.
>
> Does anyone see any similar problems? To me this smells like a faulty
> DRAM configuration.

Yes.

> Best regards
> Markus
>
> P.S.: I'll go back and test some prior version, but unfortunaltely the
> board is bricked and I first need to find a BDI.

:-(

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot] amcc kilauea odd crashes

[Wolfgang Denk]

Dear Stefan,

In message <200810231251.36841.sr@denx.de> you wrote:
> 
> Is this a 600MHz Kilauea? This is a known issue, that the new autocalibration

No, this is a CPU Rev. A board at 400 Mhz.

> code has a problem here. I reported this problem a few weeks ago. AMCC is
> currently working on a fix for this.

Should we not backout the autocalib patches that cause the problem
until a stable working solution is found?

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
miracle:  an  extremely  outstanding  or  unusual  event,  thing,  or
accomplishment.                                - Webster's Dictionary

[U-Boot] amcc kilauea odd crashes

[Stefan Roese]

Hi Wolfgang,

On Thursday 23 October 2008, Wolfgang Denk wrote:
> In message <200810231251.36841.sr@denx.de> you wrote:
> > Is this a 600MHz Kilauea? This is a known issue, that the new
> > autocalibration
>
> No, this is a CPU Rev. A board at 400 Mhz.

Probably with the same DDR2 frequency.

> > code has a problem here. I reported this problem a few weeks ago. AMCC is
> > currently working on a fix for this.
>
> Should we not backout the autocalib patches that cause the problem
> until a stable working solution is found?

Not sure. My hope is that AMCC find a solution quickly. They should receive 
the failing board this week.

And they already did send a "fix" (more a workaround) for this problem:

[PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz

which you rejected. So I suggest to wait for a few days.

Victor, Adam, did you already receive my board?

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot] amcc kilauea odd crashes

[Wolfgang Denk]

Dear Stefan,

In message <200810231315.08807.sr@denx.de> you wrote:
> 
> > Should we not backout the autocalib patches that cause the problem
> > until a stable working solution is found?
> 
> Not sure. My hope is that AMCC find a solution quickly. They should receive 
> the failing board this week.
> 
> And they already did send a "fix" (more a workaround) for this problem:
> 
> [PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz
> 
> which you rejected. So I suggest to wait for a few days.

Well, that was one full month ago, and nothing happened since.1s

I see people running into problems with the current code, so I vote
to back out the culprit until a real fix has been found.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
You can fool some of the people all of the time, and You can fool all
of the people some of the time, but You can't fool mom.

[U-Boot] amcc kilauea odd crashes

[Stefan Roese]

Hi Wolfgang,

On Thursday 23 October 2008, Wolfgang Denk wrote:
> > > Should we not backout the autocalib patches that cause the problem
> > > until a stable working solution is found?
> >
> > Not sure. My hope is that AMCC find a solution quickly. They should
> > receive the failing board this week.
> >
> > And they already did send a "fix" (more a workaround) for this problem:
> >
> > [PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz
> >
> > which you rejected. So I suggest to wait for a few days.
>
> Well, that was one full month ago, and nothing happened since.1s

That's not correct. One patch got checked in which definitely made the 
situation better:

f8a00dea841d5d75de1f8e8107e90ee1beeddf5f

    ppc4xx: Reset and relock memory DLL after SDRAM_CLKTR change
    
    After changing SDRAM_CLKTR phase value rerun the memory preload
    initialization sequence (INITPLR) to reset and relock the memory
    DLL. Changing the SDRAM_CLKTR memory clock phase coarse timing
    adjustment effects the phase relationship of the internal, to the
    PPC chip, and external, to the PPC chip, versions of MEMCLK_OUT.
    
    Signed-off-by: Adam Graham <agraham@amcc.com>
    Signed-off-by: Victor Gallardo <vgallardo@amcc.com>
    Signed-off-by: Stefan Roese <sr@denx.de>

Unfortunately it didn't fix all problems. AMCC already provided another patch 
for testing purposes. Not to the list but to me (and you) directly. Please 
find it attached again. Would be great if Markus could test it on the failing 
Kilauea.

> I see people running into problems with the current code, so I vote
> to back out the culprit until a real fix has been found.

Hmmm, "people" have been running into this problem before too. That we me.

Again, let's please wait at least for Adam and/or Victor to comment on this 
issue. It should only be a few hours until they read their mails.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================
-------------- next part --------------
An embedded message was scrubbed...
From: Adam Graham <agraham@amcc.com>
Subject: [PATCH 1/1] Check for memory rank missed oversample during DDR autocalibration.
Date: Tue, 14 Oct 2008 19:24:13 -0700
Size: 4264
Url: http://lists.denx.de/pipermail/u-boot/attachments/20081023/c6bd7b79/attachment.eml 

[U-Boot] amcc kilauea odd crashes

[Markus Klotzbücher]

On Thu, Oct 23, 2008 at 01:30:08PM +0200, Stefan Roese wrote:
> Hi Wolfgang,
> 
> On Thursday 23 October 2008, Wolfgang Denk wrote:
> > > > Should we not backout the autocalib patches that cause the problem
> > > > until a stable working solution is found?
> > >
> > > Not sure. My hope is that AMCC find a solution quickly. They should
> > > receive the failing board this week.
> > >
> > > And they already did send a "fix" (more a workaround) for this problem:
> > >
> > > [PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz
> > >
> > > which you rejected. So I suggest to wait for a few days.
> >
> > Well, that was one full month ago, and nothing happened since.1s
> 
> That's not correct. One patch got checked in which definitely made the 
> situation better:
> 
> f8a00dea841d5d75de1f8e8107e90ee1beeddf5f
> 
>     ppc4xx: Reset and relock memory DLL after SDRAM_CLKTR change
>     
>     After changing SDRAM_CLKTR phase value rerun the memory preload
>     initialization sequence (INITPLR) to reset and relock the memory
>     DLL. Changing the SDRAM_CLKTR memory clock phase coarse timing
>     adjustment effects the phase relationship of the internal, to the
>     PPC chip, and external, to the PPC chip, versions of MEMCLK_OUT.
>     
>     Signed-off-by: Adam Graham <agraham@amcc.com>
>     Signed-off-by: Victor Gallardo <vgallardo@amcc.com>
>     Signed-off-by: Stefan Roese <sr@denx.de>
> 
> Unfortunately it didn't fix all problems. AMCC already provided another patch 
> for testing purposes. Not to the list but to me (and you) directly. Please 
> find it attached again. Would be great if Markus could test it on the failing 
> Kilauea.

I tested it and it's still failing. I dare say the patch makes things
worse. After about 20 hard resets the board didn't reach the u-boot
console a single time.

:-(

Thanks
Markus


-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de")

[U-Boot] amcc kilauea odd crashes

[Stefan Roese]

On Thursday 23 October 2008, Markus Klotzb?cher wrote:
> > Unfortunately it didn't fix all problems. AMCC already provided another
> > patch for testing purposes. Not to the list but to me (and you) directly.
> > Please find it attached again. Would be great if Markus could test it on
> > the failing Kilauea.
>
> I tested it and it's still failing. I dare say the patch makes things 
> worse. After about 20 hard resets the board didn't reach the u-boot
> console a single time.
>
> :-(

Too bad. Thanks for testing though.

Adam & Victor, any ideas? If you don't see a "quick" solution, please provide 
a temporary fix for this problem.

Thanks.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================

[U-Boot] amcc kilauea odd crashes

[prodyut hazarika]

> I tested it and it's still failing. I dare say the patch makes things
> worse. After about 20 hard resets the board didn't reach the u-boot
> console a single time.
>

Markus, have you got access to BDI?

[U-Boot] amcc kilauea odd crashes

[Markus Klotzbücher]

On Thu, Oct 23, 2008 at 07:51:30AM -0700, prodyut hazarika wrote:
> > I tested it and it's still failing. I dare say the patch makes things
> > worse. After about 20 hard resets the board didn't reach the u-boot
> > console a single time.
> >
> 
> Markus, have you got access to BDI?

I sure do!

Regards
Markus

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de")

[U-Boot] amcc kilauea odd crashes

[Adam Graham]

 

> -----Original Message-----
> From: Stefan Roese [mailto:sr at denx.de] 
> Sent: Thursday, October 23, 2008 4:15 AM
> To: u-boot at lists.denx.de
> Cc: Wolfgang Denk; Markus Klotzb?cher; Adam Graham; Victor Gallardo
> Subject: Re: [U-Boot] amcc kilauea odd crashes
> 
> Hi Wolfgang,
> 
> On Thursday 23 October 2008, Wolfgang Denk wrote:
> > In message <200810231251.36841.sr@denx.de> you wrote:
> > > Is this a 600MHz Kilauea? This is a known issue, that the new 
> > > autocalibration
> >
> > No, this is a CPU Rev. A board at 400 Mhz.
> 
> Probably with the same DDR2 frequency.

Correct, the important clocking is the DDR2 frequency and not the CPU frequency (i.e. the PLB of 200MHz set on this board).


> 
> > > code has a problem here. I reported this problem a few weeks ago. 
> > > AMCC is currently working on a fix for this.
> >
> > Should we not backout the autocalib patches that cause the problem 
> > until a stable working solution is found?
> 
> Not sure. My hope is that AMCC find a solution quickly. They 
> should receive the failing board this week.
> 
> And they already did send a "fix" (more a workaround) for 
> this problem:
> 
> [PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz
> 
> which you rejected. So I suggest to wait for a few days.
> 
> Victor, Adam, did you already receive my board?

We have not received your board yet.  Do you have a shipping tracking number we can check?

Thanks,
Adam

> 
> Best regards,
> Stefan
> 
> =====================================================================
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: 
> office at denx.de 
> =====================================================================
> 

[U-Boot] amcc kilauea odd crashes

[Adam Graham]

> 
> Hi Wolfgang,
> 
> On Thursday 23 October 2008, Wolfgang Denk wrote:
> > > > Should we not backout the autocalib patches that cause 
> the problem 
> > > > until a stable working solution is found?
> > >
> > > Not sure. My hope is that AMCC find a solution quickly. 
> They should 
> > > receive the failing board this week.

We have not received this board yet and hope to try out our unreleased patches on your Kilauea board.  (That being said, it looks like Markus' Kilauea configuration can also be used to test the unreleased patches - CPU 400MHz, PLB 200MHz.)


> > >
> > > And they already did send a "fix" (more a workaround) for 
> this problem:
> > >
> > > [PATCH v2] ppc4xx: Fix DDR2 auto calibration on Kilauea 600MHz
> > >
> > > which you rejected. So I suggest to wait for a few days.
> >
> > Well, that was one full month ago, and nothing happened since.1s
> 
> That's not correct. One patch got checked in which definitely 
> made the situation better:
> 
> f8a00dea841d5d75de1f8e8107e90ee1beeddf5f
> 
>     ppc4xx: Reset and relock memory DLL after SDRAM_CLKTR change
>     
>     After changing SDRAM_CLKTR phase value rerun the memory preload
>     initialization sequence (INITPLR) to reset and relock the memory
>     DLL. Changing the SDRAM_CLKTR memory clock phase coarse timing
>     adjustment effects the phase relationship of the internal, to the
>     PPC chip, and external, to the PPC chip, versions of MEMCLK_OUT.
>     
>     Signed-off-by: Adam Graham <agraham@amcc.com>
>     Signed-off-by: Victor Gallardo <vgallardo@amcc.com>
>     Signed-off-by: Stefan Roese <sr@denx.de>
> 
> Unfortunately it didn't fix all problems. AMCC already 
> provided another patch for testing purposes. Not to the list 
> but to me (and you) directly. Please find it attached again. 
> Would be great if Markus could test it on the failing Kilauea.

Stefan, thank you for send the test patch to Markus.

Markus, if you like to test out this patch and send us (AMCC) the results, that would be appreciated.  To get more DDR autocalibration information, you can set the U-Boot environment variable "autocalib" to "loop".  This will display all the passing and non-passing write-read-compare memory windows as well as the final result that was chosen.

=> setenv autocalib loop
=> saveenv
=> reset

To remove the "autocalib" verbosity, unset this "autocalib" environment variable.

Thanks,
Adam Graham
AMCC

[U-Boot] amcc kilauea odd crashes

[Adam Graham]

 

> > Unfortunately it didn't fix all problems. AMCC already provided 
> > another patch for testing purposes. Not to the list but to me (and 
> > you) directly. Please find it attached again. Would be 
> great if Markus 
> > could test it on the failing Kilauea.
> 
> I tested it and it's still failing. I dare say the patch 
> makes things worse. After about 20 hard resets the board 
> didn't reach the u-boot console a single time.
> 
> :-(
> 
> Thanks
> Markus
> 

Markus,

Sorry, ignore my prior request for you to try out those unreleased patches as I see that you already tried them out and the patches did not work.

Thanks,
Adam

[U-Boot] amcc kilauea odd crashes

[Adam Graham]

 

> Subject: Re: [U-Boot] amcc kilauea odd crashes
> 
> On Thursday 23 October 2008, Markus Klotzb?cher wrote:
> > > Unfortunately it didn't fix all problems. AMCC already provided 
> > > another patch for testing purposes. Not to the list but 
> to me (and you) directly.
> > > Please find it attached again. Would be great if Markus 
> could test 
> > > it on the failing Kilauea.
> >
> > I tested it and it's still failing. I dare say the patch 
> makes things 
> > worse. After about 20 hard resets the board didn't reach the u-boot 
> > console a single time.
> >
> > :-(
> 

Markus, yes, thank you for your testing and debug efforts.

> Too bad. Thanks for testing though.
> 
> Adam & Victor, any ideas? If you don't see a "quick" 
> solution, please provide a temporary fix for this problem.
> 

Markus, the only quick solution that we have for immediate application is to disable the DDR autocalibration and therefore run with the original Kilauea SDRAM static values.  This will require you to recompile U-Boot and download it onto your board.  

To disable the DDR autocalibration code, in the include/configs/kilauea.h file, undef the "CONFIG_PPC4xx_DDR_AUTOCALIBRATION" definition and recompile U-Boot.

Once we receive Stefan's Kilauea board we can debug his DDR autocalibration issue, which probably is a similar issue to what you are seeing on your board.  We will keep you in the loop.

As an aside I grabbed the top of git tree and tried it on all my Kilauea boards, including the latest board which supports (CPU=600MHz, PLB=200MHz) clocking and I can not get DDR autocalibration to fail.  I also successfully tried the PPC405EX chip built-in bootstraps of (CPU=333MHz PLB=166MHz, CPU=400MHz PLB=200MHz) and the bootstrap EEPROM strapping of (CPU=533MHz PLB=177MHz).

Thanks,
Adam


> Thanks.
> 
> Best regards,
> Stefan
> 
> =====================================================================
> DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: 
> office at denx.de 
> =====================================================================
> 

[U-Boot] amcc kilauea odd crashes

[Stefan Roese]

On Friday 24 October 2008, Adam Graham wrote:
> > Too bad. Thanks for testing though.
> >
> > Adam & Victor, any ideas? If you don't see a "quick"
> > solution, please provide a temporary fix for this problem.
>
> Markus, the only quick solution that we have for immediate application is
> to disable the DDR autocalibration and therefore run with the original
> Kilauea SDRAM static values.  This will require you to recompile U-Boot and
> download it onto your board.
>
> To disable the DDR autocalibration code, in the include/configs/kilauea.h
> file, undef the "CONFIG_PPC4xx_DDR_AUTOCALIBRATION" definition and
> recompile U-Boot.

I just sent a patch disabling the autocalibration to the list. I intend to 
push this to Wolfgang soon. We can enable the autocalibration again when the 
issue is fixed.

Best regards,
Stefan

=====================================================================
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-0 Fax: +49-8142-66989-80  Email: office at denx.de
=====================================================================