[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Ira Snyder]

Hello u-boot-users,

I am currently in the process of developing a new driver in U-Boot. It
is a software "virtual ethernet over PCI" driver. The driver itself is
largely irrelevant for this problem.

During the development, I noticed that adding and removing certain
pieces of debugging code (which did not change any program state) caused
the board to hang on boot, like so:
================================================================================

U-Boot 1.3.4-rc1-00001-gb89881c-dirty (Jul 16 2008 - 14:54:56) MPC83XX

Reset Status: External/Internal Soft, External/Internal Hard

CPU:   e300c1, MPC8349EA, Rev: 3.0 at 528 MHz, CSB: 264 MHz
Board: Freescale MPC8349EMDS
I2C:   



As opposed to a normal boot:
================================================================================

U-Boot 1.3.4-rc1-00001-g8ff17d3-dirty (Jul 16 2008 - 12:00:01) MPC83XX

Reset Status: External/Internal Soft, External/Internal Hard

CPU:   e300c1, MPC8349EA, Rev: 3.0 at 528 MHz, CSB: 264 MHz
Board: Freescale MPC8349EMDS
I2C:   ready
SPI:   ready
DRAM:  256 MB (DDR2, 64-bit, ECC on, 264 MHz)
FLASH: 32 MB
In:    serial
Out:   serial
Err:   serial
Net:   TSEC0, TSEC1

Type "run flash_nfs" to mount root filesystem over NFS



I have tried to narrow the problem down as much as possible, by removing
as much of my code as possible. I succeeded in reproducing the problem
using code that seemingly "does nothing wrong". The inlined patch is
against U-Boot v1.3.4-rc1

During my testing, it seemed that elf section alignment in the image
mattered. I took a broken, non-booting image and saved the section
offsets. They are reproduced below, generated with:

$ powerpc-linux-strip u-boot
$ powerpc-linux-readelf -e u-boot

Here is the output of the readelf command for my bad image:
================================================================================

ELF Header:
  Magic:   7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, big endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           PowerPC
  Version:                           0x1
  Entry point address:               0xfe000100
  Start of program headers:          52 (bytes into file)
  Start of section headers:          225328 (bytes into file)
  Flags:                             0x8000, relocatable-lib
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         2
  Size of section headers:           40 (bytes)
  Number of section headers:         12
  Section header string table index: 11

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        fe000000 000078 032c50 00  AX  0   0  4
  [ 2] .reloc            PROGBITS        fe033000 033078 0014c0 00  WA  0   0  1
  [ 3] .data             PROGBITS        fe0344c0 034538 000bdc 00  WA  0   0  4
  [ 4] .data.rel.ro.loca PROGBITS        fe03509c 035114 000078 00  WA  0   0  4
  [ 5] .data.rel         PROGBITS        fe035114 03518c 000710 00  WA  0   0  4
  [ 6] data              PROGBITS        fe035824 03589c 000004 00  WA  0   0  4
  [ 7] .data.rel.local   PROGBITS        fe035828 0358a0 000be4 00  WA  0   0  4
  [ 8] .u_boot_cmd       PROGBITS        fe03640c 036484 000570 00  WA  0   0  4
  [ 9] .bss              NOBITS          fe037000 000000 0068e0 00  WA  0   0  8
  [10] .comment          PROGBITS        00000000 0369f4 0005d6 00      0   0  1
  [11] .shstrtab         STRTAB          00000000 036fca 000065 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000078 0xfe000000 0xfe000000 0x3697c 0x3697c RWE 0x8
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4

 Section to Segment mapping:
  Segment Sections...
   00     .text .reloc .data .data.rel.ro.local .data.rel data .data.rel.local .u_boot_cmd 
   01     



The part I found important was the size of each section. The .bss,
.comment, and .shstrtab don't seem to matter (but I may be wrong!)

I wrote some code that went into the appropriate sections to get the
sizes I needed. This is why there are lots of "a" in the patch (to boost
the .data section) and the extern variables (to boost the .reloc
section) and the extern variables with values (to boost the data section).

I also did my best to trim down the code that was added in
board/freescale/mpc8349emds/mpc8394emds.c checkboard(). In my original
code, all of the code is there. I left it there in the patch, but
#if 0'd out all of the parts that made no difference.

Removing the BCSR read does seem to fix the problem (while concurrently
adjusting the padding bits until the sections are the correct size). I
see nothing wrong with the BCSR read, and it may just be another
unrelated symptom.

Here is the readelf output on my non-working U-Boot binary (produced
from v1.3.4-rc1 + the attached patch), produced using the exact same
procedure as above:
================================================================================

ELF Header:
  Magic:   7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 
  Class:                             ELF32
  Data:                              2's complement, big endian
  Version:                           1 (current)
  OS/ABI:                            UNIX - System V
  ABI Version:                       0
  Type:                              EXEC (Executable file)
  Machine:                           PowerPC
  Version:                           0x1
  Entry point address:               0xfe000100
  Start of program headers:          52 (bytes into file)
  Start of section headers:          225232 (bytes into file)
  Flags:                             0x8000, relocatable-lib
  Size of this header:               52 (bytes)
  Size of program headers:           32 (bytes)
  Number of program headers:         2
  Size of section headers:           40 (bytes)
  Number of section headers:         12
  Section header string table index: 11

Section Headers:
  [Nr] Name              Type            Addr     Off    Size   ES Flg Lk Inf Al
  [ 0]                   NULL            00000000 000000 000000 00      0   0  0
  [ 1] .text             PROGBITS        fe000000 000078 032c50 00  AX  0   0  4
  [ 2] .reloc            PROGBITS        fe033000 033078 0014c0 00  WA  0   0  1
  [ 3] .data             PROGBITS        fe0344c0 034538 000bdc 00  WA  0   0  4
  [ 4] .data.rel.ro.loca PROGBITS        fe03509c 035114 000078 00  WA  0   0  4
  [ 5] .data.rel         PROGBITS        fe035114 03518c 000710 00  WA  0   0  4
  [ 6] data              PROGBITS        fe035824 03589c 000004 00  WA  0   0  4
  [ 7] .data.rel.local   PROGBITS        fe035828 0358a0 000be4 00  WA  0   0  4
  [ 8] .u_boot_cmd       PROGBITS        fe03640c 036484 000558 00  WA  0   0  4
  [ 9] .bss              NOBITS          fe037000 000000 0068f0 00  WA  0   0  8
  [10] .comment          PROGBITS        00000000 0369dc 00058e 00      0   0  1
  [11] .shstrtab         STRTAB          00000000 036f6a 000065 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings)
  I (info), L (link order), G (group), x (unknown)
  O (extra OS processing required) o (OS specific), p (processor specific)

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000078 0xfe000000 0xfe000000 0x36964 0x36964 RWE 0x8
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RWE 0x4

 Section to Segment mapping:
  Segment Sections...
   00     .text .reloc .data .data.rel.ro.local .data.rel data .data.rel.local .u_boot_cmd 
   01     





I built U-Boot using ELDK-4.2, using the following:
$ make mrproper
$ make MPC8349EMDS_config

My board is a Freescale MPC8349EMDS evaluation board.

If you need any more information, I am happy to provide it.

I appreciate any help in tracking down this problem,
Ira Snyder

Here is the patch (I apologize for the 100 character "a" lines, but they
made counting the correct number much easier)
================================================================================

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Kim Phillips]

On Wed, 16 Jul 2008 15:28:21 -0700
Ira Snyder <iws@ovro.caltech.edu> wrote:

> During the development, I noticed that adding and removing certain
> pieces of debugging code (which did not change any program state) caused
> the board to hang on boot, like so:
> ================================================================================
> 
> U-Boot 1.3.4-rc1-00001-gb89881c-dirty (Jul 16 2008 - 14:54:56) MPC83XX
> 
> Reset Status: External/Internal Soft, External/Internal Hard
> 
> CPU:   e300c1, MPC8349EA, Rev: 3.0 at 528 MHz, CSB: 264 MHz
> Board: Freescale MPC8349EMDS
> I2C:   

try reverting this commit:

commit 1c3dd43338a077165e7e0309cb3994e65d2bdbf8
Author: Grant Likely <grant.likely@secretlab.ca>
Date:   Tue Nov 13 22:18:33 2007 -0700

    powerpc: Backout relocation changes.
    
    Ugh.  I *hate* to back this change out, but these compiler flags don't
    work for relocation on all versions of GCC.  I've not been able to
    reproduce the environment in my setup (and hence, not been able to
    find a combination that *does* work), so I've got no choice but to go
    back to the old gcc flags and linker script.
    
    Signed-off-by: Grant Likely <grant.likely@secretlab.ca>

Kim

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Ira Snyder]

On Thu, Jul 17, 2008 at 04:54:53PM -0500, Kim Phillips wrote:
> On Wed, 16 Jul 2008 15:28:21 -0700
> Ira Snyder <iws@ovro.caltech.edu> wrote:
> 
> > During the development, I noticed that adding and removing certain
> > pieces of debugging code (which did not change any program state) caused
> > the board to hang on boot, like so:
> > ================================================================================
> > 
> > U-Boot 1.3.4-rc1-00001-gb89881c-dirty (Jul 16 2008 - 14:54:56) MPC83XX
> > 
> > Reset Status: External/Internal Soft, External/Internal Hard
> > 
> > CPU:   e300c1, MPC8349EA, Rev: 3.0 at 528 MHz, CSB: 264 MHz
> > Board: Freescale MPC8349EMDS
> > I2C:   
> 
> try reverting this commit:
> 
> commit 1c3dd43338a077165e7e0309cb3994e65d2bdbf8
> Author: Grant Likely <grant.likely@secretlab.ca>
> Date:   Tue Nov 13 22:18:33 2007 -0700
> 
>     powerpc: Backout relocation changes.
>     
>     Ugh.  I *hate* to back this change out, but these compiler flags don't
>     work for relocation on all versions of GCC.  I've not been able to
>     reproduce the environment in my setup (and hence, not been able to
>     find a combination that *does* work), so I've got no choice but to go
>     back to the old gcc flags and linker script.
>     
>     Signed-off-by: Grant Likely <grant.likely@secretlab.ca>
> 
> Kim

Hello Kim,

Thanks for looking at this problem so quickly. I'll try reverting the
patch as soon as I'm done writing this email and post the results here.

I spent this afternoon trying to narrow down the problem exhibited by
the changes in my earlier email. I managed to narrow it down to the
.data section's start address being a multiple of 32 bytes, and the
bcsr code in checkboard().

I have attached a patch which causes my board to hang on boot in exactly
the same way that the earlier patch does. Removing the line from the
linker script makes the board boot perfectly, as long as the .data
section's start address is not on a 32 byte boundary by coincidence.

Ira



diff --git a/board/freescale/mpc8349emds/mpc8349emds.c b/board/freescale/mpc8349emds/mpc8349emds.c
index 9a312c3..f4792a9 100644
--- a/board/freescale/mpc8349emds/mpc8349emds.c
+++ b/board/freescale/mpc8349emds/mpc8349emds.c
@@ -165,6 +165,13 @@ int fixed_sdram(void)
 
 int checkboard (void)
 {
+	volatile immap_t *immr = (immap_t *)CFG_IMMR;
+	volatile u8 *bcsr = (u8 *)CFG_BCSR;
+	int in_pci_slot;
+
+	/* in a pci slot or standalone */
+	in_pci_slot = (bcsr[10] & 0x80) ? 1 : 0;
+
 	puts("Board: Freescale MPC8349EMDS\n");
 	return 0;
 }
diff --git a/cpu/mpc83xx/u-boot.lds b/cpu/mpc83xx/u-boot.lds
index 99ad675..f605eab 100644
--- a/cpu/mpc83xx/u-boot.lds
+++ b/cpu/mpc83xx/u-boot.lds
@@ -79,6 +79,7 @@ SECTIONS
   __got2_entries = (_FIXUP_TABLE_ - _GOT2_TABLE_) >> 2;
   __fixup_entries = (. - _FIXUP_TABLE_) >> 2;
 
+  . = ALIGN(32);
   .data    :
   {
     *(.data)

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Jerry Van Baren]

Ira Snyder wrote:
> On Thu, Jul 17, 2008 at 04:54:53PM -0500, Kim Phillips wrote:
>> On Wed, 16 Jul 2008 15:28:21 -0700
>> Ira Snyder <iws@ovro.caltech.edu> wrote:
>>
>>> During the development, I noticed that adding and removing certain
>>> pieces of debugging code (which did not change any program state) caused
>>> the board to hang on boot, like so:

[snip]

> 
> Hello Kim,
> 
> Thanks for looking at this problem so quickly. I'll try reverting the
> patch as soon as I'm done writing this email and post the results here.
> 
> I spent this afternoon trying to narrow down the problem exhibited by
> the changes in my earlier email. I managed to narrow it down to the
> .data section's start address being a multiple of 32 bytes, and the
> bcsr code in checkboard().
> 
> I have attached a patch which causes my board to hang on boot in exactly
> the same way that the earlier patch does. Removing the line from the
> linker script makes the board boot perfectly, as long as the .data
> section's start address is not on a 32 byte boundary by coincidence.
> 
> Ira
> 
> 
> 
> diff --git a/board/freescale/mpc8349emds/mpc8349emds.c b/board/freescale/mpc8349emds/mpc8349emds.c
> index 9a312c3..f4792a9 100644
> --- a/board/freescale/mpc8349emds/mpc8349emds.c
> +++ b/board/freescale/mpc8349emds/mpc8349emds.c
> @@ -165,6 +165,13 @@ int fixed_sdram(void)
>  
>  int checkboard (void)
>  {
> +	volatile immap_t *immr = (immap_t *)CFG_IMMR;
> +	volatile u8 *bcsr = (u8 *)CFG_BCSR;
> +	int in_pci_slot;
> +
> +	/* in a pci slot or standalone */
> +	in_pci_slot = (bcsr[10] & 0x80) ? 1 : 0;
> +
>  	puts("Board: Freescale MPC8349EMDS\n");
>  	return 0;
>  }
> diff --git a/cpu/mpc83xx/u-boot.lds b/cpu/mpc83xx/u-boot.lds
> index 99ad675..f605eab 100644
> --- a/cpu/mpc83xx/u-boot.lds
> +++ b/cpu/mpc83xx/u-boot.lds
> @@ -79,6 +79,7 @@ SECTIONS
>    __got2_entries = (_FIXUP_TABLE_ - _GOT2_TABLE_) >> 2;
>    __fixup_entries = (. - _FIXUP_TABLE_) >> 2;
>  
> +  . = ALIGN(32);
>    .data    :
>    {
>      *(.data)

Hi Ira,

This is a long shot, but could there be a problem with the DDR SDRAM 
and/or its initialization?  Are you using a factory-provided DIMM?  What 
if you try a different one (preferably a different brand/size)?

I'm not familiar with the board configuration, I presume instruction 
caches are enabled.  Are data caches enabled too?  What happens if you 
disable cache(s)?

Best regards,
gvb

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Ira Snyder]

On Fri, Jul 18, 2008 at 07:59:36AM -0400, Jerry Van Baren wrote:
>
> Hi Ira,
>
> This is a long shot, but could there be a problem with the DDR SDRAM  
> and/or its initialization?

I suppose it could be, but this seems unlikely. Why would aligning the
start of a section to a 32 byte boundary make the ram initialization fail,
but any other alignment works?

Also, I have two nearly identical boards here. One is a MPC8349EMDS
Rev-3.1 (it's an EA board, using DDR2) and another, older MPC8349EMDS
Rev-1.1 (it's an E board, using plain DDR). Both fail in exactly the
same way.

> Are you using a factory-provided DIMM?  What  
> if you try a different one (preferably a different brand/size)?
>

Yes, both boards have the factory provided memory. There are different
DIMMs on each of the two boards. The DDR is Micron, the DDR2 I don't
recognize the manufacturer for. Hopefully they're different enough.
Unfortunately, I don't have any more memory that fits these boards.

> I'm not familiar with the board configuration, I presume instruction  
> caches are enabled.  Are data caches enabled too?  What happens if you  
> disable cache(s)?
>

Yes, both caches are enabled. I'm not really sure how to turn them off
without effecting the bootup sequence. It looks like they are assumed to
enabled during parts of startup in cpu/mpc83xx/start.S

I checked include/configs/MPC8349EMDS.h and didn't find any obvious
config options for disabling the cache.

The hang happens before the relocation to RAM. The "I2C:" line where it
hangs is printed before the relocation. There are "SPI:" and "DRAM:"
lines that should also be printed before the relocation to RAM.

> Best regards,
> gvb

As some added information, I tried building with ELDK-4.1 (rather than
ELDK-4.2, which I have been using) and was unable to get the lockup.
However, in the past I did get several similar lockups using ELDK-4.1,
which went away when I upgraded to ELDK-4.2 (the problems prompted the
upgrade). Unfortunately, I just thought the problem was my code at the
time, and so I did not keep any images around.

Thanks for your help,
Ira

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Jerry Van Baren]

Ira Snyder wrote:
> On Fri, Jul 18, 2008 at 07:59:36AM -0400, Jerry Van Baren wrote:
>> Hi Ira,
>>
>> This is a long shot, but could there be a problem with the DDR SDRAM  
>> and/or its initialization?
> 
> I suppose it could be, but this seems unlikely. Why would aligning the
> start of a section to a 32 byte boundary make the ram initialization fail,
> but any other alignment works?

Not the initialization itself, but an incorrect initialization can cause 
funny things to happen when the SDRAM is used.  As you note here and 
below, doesn't look like this is the problem.

The thought was that your moving memory contents around ("misaligning" 
it) causes changes to the bus cycles which make it work or not.

> Also, I have two nearly identical boards here. One is a MPC8349EMDS
> Rev-3.1 (it's an EA board, using DDR2) and another, older MPC8349EMDS
> Rev-1.1 (it's an E board, using plain DDR). Both fail in exactly the
> same way.

That is useful to know (I think).  It doesn't sound like a hardware problem.

>> Are you using a factory-provided DIMM?  What  
>> if you try a different one (preferably a different brand/size)?
>>
> 
> Yes, both boards have the factory provided memory. There are different
> DIMMs on each of the two boards. The DDR is Micron, the DDR2 I don't
> recognize the manufacturer for. Hopefully they're different enough.
> Unfortunately, I don't have any more memory that fits these boards.

Good enough, especially since you are using DIMMs that are provided with 
the boards and thus (should be) used around the world without problems.

>> I'm not familiar with the board configuration, I presume instruction  
>> caches are enabled.  Are data caches enabled too?  What happens if you  
>> disable cache(s)?
> 
> Yes, both caches are enabled. I'm not really sure how to turn them off
> without effecting the bootup sequence. It looks like they are assumed to
> enabled during parts of startup in cpu/mpc83xx/start.S

That is the trick using data cache for the stack before the SDRAM is 
initialized, not really what I was referring to.  My question probably 
is immaterial because your failure occurs before (at?) SDRAM 
initialization, before the caches would be enabled as real caches.

> I checked include/configs/MPC8349EMDS.h and didn't find any obvious
> config options for disabling the cache.
> 
> The hang happens before the relocation to RAM. The "I2C:" line where it
> hangs is printed before the relocation. There are "SPI:" and "DRAM:"
> lines that should also be printed before the relocation to RAM.

I2C is used to read the SPD configuration off the memory DIMM. 
Interesting, but I'm not sure what it is telling us...

>> Best regards,
>> gvb
> 
> As some added information, I tried building with ELDK-4.1 (rather than
> ELDK-4.2, which I have been using) and was unable to get the lockup.
> However, in the past I did get several similar lockups using ELDK-4.1,
> which went away when I upgraded to ELDK-4.2 (the problems prompted the
> upgrade). Unfortunately, I just thought the problem was my code at the
> time, and so I did not keep any images around.
> 
> Thanks for your help,
> Ira

Weird behavior that changes with different builds and/or shifting things 
in memory quite often are build tools or memory problems.  It doesn't 
sound like you are having memory problems, but your build tools sound 
suspicious.  There are lots of people running ELDK4.1 and 4.2 (myself 
included - on debian, cross compiling for a MPC8360) so that probably 
isn't the problem.

What is your build host running (OS, distribution)?  Any chance of doing 
a clean OS and ELDK installation on a different machine?

Running low on theories,
gvb

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Ira Snyder]

On Fri, Jul 18, 2008 at 02:17:48PM -0400, Jerry Van Baren wrote:
> > I checked include/configs/MPC8349EMDS.h and didn't find any obvious
> > config options for disabling the cache.
> > 
> > The hang happens before the relocation to RAM. The "I2C:" line where it
> > hangs is printed before the relocation. There are "SPI:" and "DRAM:"
> > lines that should also be printed before the relocation to RAM.
> 
> I2C is used to read the SPD configuration off the memory DIMM. 
> Interesting, but I'm not sure what it is telling us...
> 

Yep. Note that there is an access to the BCSR registers that I added.
Removing the access makes the problem go away, even with the .data
section 32 byte aligned. There shouldn't be any problem reading the
BCSR registers, though.

What I'm getting at is maybe it is neither of these that is the real
problem, it could just be that the BCSR access plus the alignment of the
.data section triggers the real problem, whatever it may be.

> > 
> > As some added information, I tried building with ELDK-4.1 (rather than
> > ELDK-4.2, which I have been using) and was unable to get the lockup.
> > However, in the past I did get several similar lockups using ELDK-4.1,
> > which went away when I upgraded to ELDK-4.2 (the problems prompted the
> > upgrade). Unfortunately, I just thought the problem was my code at the
> > time, and so I did not keep any images around.
> > 
> > Thanks for your help,
> > Ira
> 
> Weird behavior that changes with different builds and/or shifting things 
> in memory quite often are build tools or memory problems.  It doesn't 
> sound like you are having memory problems, but your build tools sound 
> suspicious.  There are lots of people running ELDK4.1 and 4.2 (myself 
> included - on debian, cross compiling for a MPC8360) so that probably 
> isn't the problem.
> 
> What is your build host running (OS, distribution)?  Any chance of doing 
> a clean OS and ELDK installation on a different machine?
> 

I'm running Ubuntu Hardy. My test machine here has a completely clean,
not updated installation of Hardy on it as well, I could try installing
and building with that. I also have access to a CentOS 4 machine that I
could install ELDK-4.2 onto. It has ELDK-4.1 already installed. I'll try
ELDK-4.2 on the CentOS machine as soon as the install media is finished
downloading, which will take a while. It might be Monday before I have a
chance to get back to you with the results.

> Running low on theories,
> gvb

Yep, it's got me completely stumped as well. I tried running "objdump -d"
on a broken and working u-boot ELF file (not the bin file, I don't know how
to dissassemble that) and the only changes it showed was the compilation
timestamp at the beginning of the file.

Thanks again,
Ira

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Jerry Van Baren]

Ira Snyder wrote:
> On Fri, Jul 18, 2008 at 02:17:48PM -0400, Jerry Van Baren wrote:
>>> I checked include/configs/MPC8349EMDS.h and didn't find any obvious
>>> config options for disabling the cache.
>>>
>>> The hang happens before the relocation to RAM. The "I2C:" line where it
>>> hangs is printed before the relocation. There are "SPI:" and "DRAM:"
>>> lines that should also be printed before the relocation to RAM.
>> I2C is used to read the SPD configuration off the memory DIMM. 
>> Interesting, but I'm not sure what it is telling us...
>>
> 
> Yep. Note that there is an access to the BCSR registers that I added.
> Removing the access makes the problem go away, even with the .data
> section 32 byte aligned. There shouldn't be any problem reading the
> BCSR registers, though.
> 
> What I'm getting at is maybe it is neither of these that is the real
> problem, it could just be that the BCSR access plus the alignment of the
> .data section triggers the real problem, whatever it may be.

Hmmm, interesting point: reading the BCSR is implicated.  Are you sure 
that reading the BCSR is safe *at this point in time*?  Have you 
confirmed CS1 is configured this early in the boot sequence?

[snip]

>> What is your build host running (OS, distribution)?  Any chance of doing 
>> a clean OS and ELDK installation on a different machine?
>>
> 
> I'm running Ubuntu Hardy. My test machine here has a completely clean,
> not updated installation of Hardy on it as well, I could try installing
> and building with that. I also have access to a CentOS 4 machine that I
> could install ELDK-4.2 onto. It has ELDK-4.1 already installed. I'll try
> ELDK-4.2 on the CentOS machine as soon as the install media is finished
> downloading, which will take a while. It might be Monday before I have a
> chance to get back to you with the results.

That is so low a probability that I wouldn't bother unless I were really 
desperate.

> Thanks again,
> Ira

Have a good weekend,
gvb

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[David Hawkins]

Hi guys,

As noted in Ira's previous posts, he'd managed to get the
MPC8349E and MPC8349EA boards to fail consistently.

Starting with the U-Boot head, and applying Ira's minimal
BCSR access plus linker 32-byte alignment patch, we used a
BDI2000 to set break-points until we identified the flash
address at which the board died. The BDI could break
consistently at FE00_9B90, but stepping past that point
would result in failure.

FE00_9B90 is in the drivers/i2c/fsl_i2c.c code, which
explains why Ira would see the 'I2C: ' line before lockup.

I put a logic analyzer on the local bus and triggered
at the access to FE00_9B90, and found something weird ...

Right before the access to FE00_9B90 was a *write* to
FE03_03C8. This location is in the .data section, but
in *read-only* Flash.

The data following that write is then *not* the same as
the data shown in the ELF file (or binary file written
to flash).

My hypothesis is that this write-access to the flash triggers
an undocumented Flash command code, that causes further reads
from the  Flash to return manufacturer data *not code*. Hence,
following the write to Flash, the processor loads garbage
instructions and goes off into nowhere land.

Ira's alignment patch to the linker code causes the
write-address to change, so I suspect that the address for
the Flash write was critical in triggering the Flash code.

The offending write shown in the logic analyzer trace was
the value 0005_FE99 to address FE03_03C8, which results
in 16-bits writes of 0005 then FE99 to address FE03_03C8
and FE03_03CA.

On the MPC8349E board, which uses Micron Q-Flash, the Flash
reads after this write return 0, while on the MPC8349EA board,
which uses Spansion Flash, the reads return non-zero data, but
still invalid data.

We tried writing the offending command code to Flash and
then reading from FE00_9B90 to see if we could reproduce
what the logic analyzer shows. However, we could not reproduce
the problem; the write waveforms would match, but the
reads returned valid Flash data. So, its a pretty subtle
Flash command code sequence!

How's that for a 'trial by fire'?!

The offending line that produces the write to Flash, involves
the use of a

static unsigned int i2c_bus_speed[2]

on line 48 of drivers/i2c/fsl_i2c.c, which is written to in
the function i2c_init(). This code is running from Flash still,
so there should be no writes to .data section code.

git log d8c82db4

shows Timur added this code. So, Timur, you owe Ira a
beer for the stress this change caused :)

We haven't looked into the intent of the I2C speed tracking
variable, so can't provide a patch. However, I'm sure Timur
will come up with a nice solution.

Thanks for all the valuable feedback everyone supplied to Ira.

Kudos to Ira for debugging this issue.

Cheers,
Dave Hawkins.

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Timur Tabi]

On Fri, Jul 18, 2008 at 8:52 PM, David Hawkins <dwh@ovro.caltech.edu> wrote:

> FE00_9B90 is in the drivers/i2c/fsl_i2c.c code, which
> explains why Ira would see the 'I2C: ' line before lockup.
>

Someone reported this problem last week while I was on vacation.  The fix is
to check if relocation has occurred, and only write the speed if it has.  I
guess I need to post a patch ASAP.

-- 
Timur Tabi
Linux kernel developer at Freescale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.denx.de/pipermail/u-boot/attachments/20080719/5fc47d96/attachment.htm 

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[David Hawkins]

Hi Timur,

>     FE00_9B90 is in the drivers/i2c/fsl_i2c.c code, which
>     explains why Ira would see the 'I2C: ' line before lockup.
> 
> 
> Someone reported this problem last week while I was on vacation. 

Did they post directly to you, or the list? I don't recall
seeing anything ... but then again, we didn't know what
we were looking for until yesterday. I'd be interested in
seeing the post. Seems like the sourceforge pages are down
at the moment though.

> The fix is to check if relocation has occurred, and only write
> the speed if it has.  I guess I need to post a patch ASAP. 

Sounds good. Thanks!

Cheers,
Dave

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[David Hawkins]

Hi all,

We recently debugged a problem where a Flash write on
MPC8349E and MPC8349EA processor boards was accidentally
occurring during board initialization.

Under the right conditions, the write appears to put
the flash into a command-mode, rather than read-data mode,
and further reads from the flash return invalid data
(as far as the processor is concerned). The write was
coming from drivers code that we had not introduced,
and depends on the address of the flash write which
changes depending on the section sizes, so the cause
of the bug was not initially obvious.

The MPC8349 processor base register has the option;
BRn[WP] = 1, that could be used to disable writes to the
boot flash. This local bus controller register is likely
the same on all 83xx series processors. To see if this
type of feature was on any other PowerPC processors,
I looked in the AMCC 440EP manual. The 440EP local
bus peripheral bank configuration registers has
a bank-usage bit that can be used to set the
440EP local bus read-only, EBC0_B0CR[BU] = 01b.

Writes to boot Flash could be disabled early on in the
boot sequence, and then enabled again after relocation.
This would protect against writes to the boot flash.

The downside of this change, would be that flash writes
would not work if you stopped the processor between the
time writes were disabled, and then enabled. So if you
were debugging, stepped into code before relocation,
and decided to stop the processor and erase the flash
(using your favorite debugger), flash erase would fail.

Hence, the consensus may be that we don't want this type
of feature - it may generate too many messages;

'help I can't program Flash'

However, the debugger initialization file could be written to
enable the flash before attempting to erase or program it.

The fix will also not expose the accidental introduction
of flash writes in the future, it'll just stop those
writes from having any effect.

It would be nicer to generate an exception if a write to
flash occurs during the period before relocation, at least
that way the introduction of an accidental flash write
would be detected immediately. I could have a look at
the 83xx MMU settings during that time and see if there
was an alternative solution using that.

The 440EP manual indicates that a protect error would
occur if a write is attempted to a read-only bank.

I figured I'd gauge if there was any interest in adding
this feature before looking into it further.

Cheers,
Dave

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Wolfgang Denk]

In message <48822929.5050408@ovro.caltech.edu> you wrote:
> 
> We recently debugged a problem where a Flash write on
> MPC8349E and MPC8349EA processor boards was accidentally
> occurring during board initialization.

A write to flash should not matter at all.

> Under the right conditions, the write appears to put
> the flash into a command-mode, rather than read-data mode,

Hm... all flash devices I know of require a certain command  sequence
to be written, which should never happen "by accident". Agreed, Intel
flashes  are braindead by accepting the writes to arbitrary addresses
within the same sector, so writing a binary image sequentially  to  a
flash  memory  space  may  easily  trigger  such a problem - but this
should never happen in U-Boot code.

> and further reads from the flash return invalid data
> (as far as the processor is concerned). The write was
> coming from drivers code that we had not introduced,

Which exact driver are you referring to? If any driver has such a bug,
it should be fixed.

> The MPC8349 processor base register has the option;
> BRn[WP] = 1, that could be used to disable writes to the
> boot flash. This local bus controller register is likely
...
> Writes to boot Flash could be disabled early on in the
> boot sequence, and then enabled again after relocation.
> This would protect against writes to the boot flash.

This is not the way to go. Hushing up problems has never been a
solution. Instead, let's find and fix the culprit.

> I figured I'd gauge if there was any interest in adding
> this feature before looking into it further.

Frankly: I saee this as a non-issue. Let's rather fix the bug instead.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
A mouse is an elephant built by the Japanese.

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Timur Tabi]

Wolfgang Denk wrote:

> Which exact driver are you referring to? If any driver has such a bug,
> it should be fixed.

drivers/i2c/fsl_i2c.c

The function is i2c_init().  I recently posted a patch that added this line of code:

	i2c_bus_speed[0] = set_i2c_bus_speed(dev, gd->i2c1_clk, speed);

And this is what causes the problem.  i2c_bus_speed[] is a global variable, and
i2c_init() is called before relocation.  Therefore, this function writes to
flash the first time it's called.

> This is not the way to go. Hushing up problems has never been a
> solution. Instead, let's find and fix the culprit.

I'm working on a patch that blocks i2c_init() from writing to global variables
before relocation has occurred.  But if you're saying that this patch should not
be necessary, then that would be better.

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[David Hawkins]

Hi Timur,

> I'm working on a patch that blocks i2c_init() from writing to
> global variables before relocation has occurred.  But if
> you're saying that this patch should not be necessary,
> then that would be better.

The I2C patch is necessary.

Whatever was written to i2c_bus_speed[0] was never actually
written, since the write occurs to flash before relocation,
so whatever the intent of the write was, needs to be fixed.

Wolfgang did not like the idea of adding code to disable
writes to flash at the processor-level before relocation.
This has the unfortunate side-effect of masking errors
such as the one your code added.

Please don't feel bad about this discussion, bugs happen :)

Cheers,
Dave

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Timur Tabi]

David Hawkins wrote:

> Whatever was written to i2c_bus_speed[0] was never actually
> written, since the write occurs to flash before relocation,
> so whatever the intent of the write was, needs to be fixed.

Prior to relocation, the value of i2c_bus_speed[] is irrelevant.  All that
matter is that the I2C device is programmed for that speed.  No code ever looks
at i2c_bus_speed[] prior to relocation.

Regardless, I'm testing a patch that prevents writes before relocation, so I'll
be sending it out soon.

-- 
Timur Tabi
Linux kernel developer at Freescale

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Wolfgang Denk]

In message <4884AFE1.3080001@freescale.com> you wrote:
> 
> > Which exact driver are you referring to? If any driver has such a bug,
> > it should be fixed.
> 
> drivers/i2c/fsl_i2c.c
> 
> The function is i2c_init().  I recently posted a patch that added this line of code:
> 
> 	i2c_bus_speed[0] = set_i2c_bus_speed(dev, gd->i2c1_clk, speed);
> 
> And this is what causes the problem.  i2c_bus_speed[] is a global variable, and
> i2c_init() is called before relocation.  Therefore, this function writes to
> flash the first time it's called.

The driver is broken if it writes to the data segment before
relocation.

That's a bug that should be fixed.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
The only solution is ... a balance of power. We  arm  our  side  with
exactly  that  much  more.  A balance of power -- the trickiest, most
difficult, dirtiest game of them all. But the only one that preserves
both sides.
	-- Kirk, "A Private Little War", stardate 4211.8

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[David Hawkins]

Hi Wolfgang,

>> We recently debugged a problem where a Flash write on
>> MPC8349E and MPC8349EA processor boards was accidentally
>> occurring during board initialization.
> 
> A write to flash should not matter at all.

Thats what I thought, until I saw the logic analyzer
trace show otherwise ...

>> This would protect against writes to the boot flash.
> 
> This is not the way to go. Hushing up problems has never been a
> solution. Instead, let's find and fix the culprit.
> 
>> I figured I'd gauge if there was any interest in adding
>> this feature before looking into it further.
> 
> Frankly: I saee this as a non-issue. Let's rather fix the bug instead.

Yep, fair enough. I figured that might be the consensus,
hence I did not look into it too much.

Cheers,
Dave

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Jerry Van Baren]

David Hawkins wrote:
> Hi all,
> 
> We recently debugged a problem where a Flash write on
> MPC8349E and MPC8349EA processor boards was accidentally
> occurring during board initialization.
> 
> Under the right conditions, the write appears to put
> the flash into a command-mode, rather than read-data mode,
> and further reads from the flash return invalid data
> (as far as the processor is concerned). The write was
> coming from drivers code that we had not introduced,
> and depends on the address of the flash write which
> changes depending on the section sizes, so the cause
> of the bug was not initially obvious.

Thanks for the complete write up on your findings.  That was very helpful.

[snip]

> Writes to boot Flash could be disabled early on in the
> boot sequence, and then enabled again after relocation.
> This would protect against writes to the boot flash.
> 
> The downside of this change, would be that flash writes
> would not work if you stopped the processor between the
> time writes were disabled, and then enabled.

[snip]

> The fix will also not expose the accidental introduction
> of flash writes in the future, it'll just stop those
> writes from having any effect.

IOW, it simply hides the bug.  :-(

> It would be nicer to generate an exception if a write to
> flash occurs during the period before relocation, at least
> that way the introduction of an accidental flash write
> would be detected immediately. I could have a look at
> the 83xx MMU settings during that time and see if there
> was an alternative solution using that.

Using the MMU that early is going to be some work and has risks of other 
mysterious lockups when done wrong.  MMUs are different for different 
processors and, often, within different branches of the same family of 
processors.  This will add to the complexity.

MMU == complexity == risk.  :-(

Most processors available today have debug registers.  If the processor 
used on a given target has a debug register set and the registers can be 
set to trigger on a write to a range, that would give you an exception. 
  You would not necessarily have to handle the exception "properly", 
simply enter a spin loop so that the processor stops in a known state 
with enough information to identify the root cause.

Pros:
* Get an exception identifying a bad bug occurrence rather than silent 
pass (mostly) or failure (mysteriously).

Cons:
* More complexity == risk
* Debug capabilities, like the MMU, are different for different 
processors and families.  This could be complex and could turn into an 
ifdefhell.
* It may be easier and better to use a debugger (e.g. BDI-3000) to 
control the hardware breakpoint registers.  A debugger may get unhappy 
or may simply undo our doings if we directly control the hardware 
breakpoint registers.

> The 440EP manual indicates that a protect error would
> occur if a write is attempted to a read-only bank.
> 
> I figured I'd gauge if there was any interest in adding
> this feature before looking into it further.

It would be nice to have a technique to trap these pre-relocation bugs. 
  They don't happen often, but they *do* happen and they are hard to 
find (until they bite you and then they are hard to identify).

What are the capabilities of your debugger?  Can you set a hardware 
breakpoint range on your flash and have it trigger on start up?  If so, 
we should add it to our FAQ and add the technique to our toolbox.

> Cheers,
> Dave

Thanks,
gvb

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[David Hawkins]

Hi Jerry,

>> The fix will also not expose the accidental introduction
>> of flash writes in the future, it'll just stop those
>> writes from having any effect.
> 
> IOW, it simply hides the bug.  :-(

Yeah, I didn't like that as a solution either.

>> It would be nicer to generate an exception if a write to
>> flash occurs during the period before relocation, at least
>> that way the introduction of an accidental flash write
>> would be detected immediately. I could have a look at
>> the 83xx MMU settings during that time and see if there
>> was an alternative solution using that.
> 
> Using the MMU that early is going to be some work and has risks of other 
> mysterious lockups when done wrong.  MMUs are different for different 
> processors and, often, within different branches of the same family of 
> processors.  This will add to the complexity.
> 
> MMU == complexity == risk.  :-(

Yep, I agree.

The 440EP solution to generate an exception looked a bit
nicer, but its not portable either.

> Most processors available today have debug registers.  If the processor 
> used on a given target has a debug register set and the registers can be 
> set to trigger on a write to a range, that would give you an exception. 
> You would not necessarily have to handle the exception "properly", 
> simply enter a spin loop so that the processor stops in a known state 
> with enough information to identify the root cause.

Haven't seen that type of register on the MPC8349EA, it might
exist, I just didn't look :)

> Pros:
> * Get an exception identifying a bad bug occurrence rather than silent 
> pass (mostly) or failure (mysteriously).
> 
> Cons:
> * More complexity == risk
> * Debug capabilities, like the MMU, are different for different 
> processors and families.  This could be complex and could turn into an 
> ifdefhell.
> * It may be easier and better to use a debugger (e.g. BDI-3000) to 
> control the hardware breakpoint registers.  A debugger may get unhappy 
> or may simply undo our doings if we directly control the hardware 
> breakpoint registers.

Yep, a repeatable bug can be traced using a debugger. The
hard part is making the bug repeatable.

> It would be nice to have a technique to trap these pre-relocation bugs. 
> They don't happen often, but they *do* happen and they are hard to find 
> (until they bite you and then they are hard to identify).
> 
> What are the capabilities of your debugger?  Can you set a hardware 
> breakpoint range on your flash and have it trigger on start up?  If so, 
> we should add it to our FAQ and add the technique to our toolbox.

Its a BDI2000.

I don't recall seeing a trap on range feature.

This is a tricky one to put in the FAQ, as it really shouldn't
happen :)

We managed to get pretty far with the debugger; we eventually
found the address at which things died, however, the debugger
wasn't able to give us an explanation. It was the logic analyzer
on the flash/local-bus that showed the reason. So perhaps thats
something that can be added to the FAQ. Let me know if you
want something coherent, and I can write a 'logic analyzer tricks
and tips' section for the FAQ.

Cheers,
Dave

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[Jerry Van Baren]

David Hawkins wrote:
> Hi Jerry,

[snip]

>> Most processors available today have debug registers.  If the 
>> processor used on a given target has a debug register set and the 
>> registers can be set to trigger on a write to a range, that would give 
>> you an exception. You would not necessarily have to handle the 
>> exception "properly", simply enter a spin loop so that the processor 
>> stops in a known state with enough information to identify the root 
>> cause.
> 
> Haven't seen that type of register on the MPC8349EA, it might
> exist, I just didn't look :)

"e300 Power Architecture Core Family Reference Manual, Rev. 2" Chapter 
10 "Debug Features" Section 10.1.3 "Data Address Breakpoint Registers 
(DABR, DABR2)"

Unfortunately, it looks like it only handles exact address matches, not 
a range.

The e500 core appears to be able to do ranges.  Unfortunately, this is 
of limited value since it is an unusual capability.  (Chapter "8.4 Book 
E Debug Events" section 8.4.2.4 "Data Address Compare (DAC) Mode")

[snip]

> Yep, a repeatable bug can be traced using a debugger. The
> hard part is making the bug repeatable.
> 
>> It would be nice to have a technique to trap these pre-relocation 
>> bugs. They don't happen often, but they *do* happen and they are hard 
>> to find (until they bite you and then they are hard to identify).
>>
>> What are the capabilities of your debugger?  Can you set a hardware 
>> breakpoint range on your flash and have it trigger on start up?  If 
>> so, we should add it to our FAQ and add the technique to our toolbox.
> 
> Its a BDI2000.
> 
> I don't recall seeing a trap on range feature.

Nope, looks like it is beyond the capability of the e300 core.  The BDI 
hardware breakpoints are just playing with the core's debug registers, 
so it isn't going to do any better than the core (and the software 
breakpoints require software assist, so they don't work when running in 
flash).

> This is a tricky one to put in the FAQ, as it really shouldn't
> happen :)
> 
> We managed to get pretty far with the debugger; we eventually
> found the address at which things died, however, the debugger
> wasn't able to give us an explanation. It was the logic analyzer
> on the flash/local-bus that showed the reason. So perhaps thats
> something that can be added to the FAQ. Let me know if you
> want something coherent, and I can write a 'logic analyzer tricks
> and tips' section for the FAQ.
> 
> Cheers,
> Dave

One really good trick is how to hook a logic analyzer to those 
itty-bitty balls on the processor.  Back when I was a boy, processors 
had 40 legs and no balls.  :-D  You must have a board with a logic 
analyzer connector.

Thanks,
gvb

[U-Boot-Users] RFQ: disable flash writes until after relocation?

[David Hawkins]

Hi Jerry,

> One really good trick is how to hook a logic analyzer to those 
> itty-bitty balls on the processor.  Back when I was a boy, processors 
> had 40 legs and no balls.  :-D  You must have a board with a logic 
> analyzer connector.

Thankfully the Freescale MDS boards do have logic
analyzer headers on their boards.

My custom boards have 5000+ BGA balls ....

http://www.ovro.caltech.edu/~dwh/carma_board/

so I made sure to hang a few logic analyzer connectors on
that too :)

One interesting thing that came out of the board development
is that the PowerPC processor was RoHS, while the FPGAs were
not. The assembly company stripped the RoHS balls off the
PowerPC, reballed them non-RoHS, and then assembled the boards.
The boards were then X-rayed with what amounts to a CT scanner,
and each ball shape was observed at the bottom, middle, and top
to ensure that they all looked nice and pretty.

Amazing stuff :)

Cheers,
Dave

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[David Hawkins]

Hi all,

We're having a rough week with our Freescale MDS boards.
I've just submitted a service request through
Freescale's online system. However, I figured others
(eg. Kim, and Timur) might be interested in the following
problem:

We have an MPC8349E-MDS-PB and an MPC8349EA-MDS-PB that we
are using to develop PCI agent mode software. After
debugging the I2C flash write issue, we found that the
serial port would stop receiving shortly after boot (we
could see the U-Boot boot messages). We traced the issue
to the RS232 receiver enable bit in the BCSR register 0
being written to; disabling the serial port receiver
tri-state outputs.

The RS232 receiver enable bit is not being changed by a
write to the BCSR, but by a write to the *flash*.

Yikes!!

We were adding print statements that showed the value of
the BCSR code in U-Boot. We found the error occurred
during the Flash CFI code. I don't think the problem is
related to CFI code - thats just where the corruption is
triggered.

Starting with U-Boot head, we applied Timur's I2C patch,
and then added the following code sequence to
__flash_detect_cfi in U-boot:

  - write 0x2F (the reset value) to the
    BCSR[0] register at 0xE2400000
  - read BCSR[0] (to confirm the write)
  - write 0xCC to Flash (0xFE000000)
  - read BCSR[0] (to show the value changed)

(If you would like to repeat this test, please ask and Ira
can send a patch file).

The board was operated in stand-alone host-mode on the bench.

The following logic analyzer traces show:

a) the case where this sequence did nothing,

http://www.ovro.caltech.edu/~dwh/mpc8349e_bcsr_read_ok.pdf

and

(b) the case where the sequence changed the value read
back from the BCSR base address to 0xCC.

http://www.ovro.caltech.edu/~dwh/mpc8349e_bcsr_read_bad.pdf

We were able to trigger the error on both the E and EA board,
but it would not fail every single reset - it occurred about
1-in-3 tries - hence it seems like a timing issue.
It could be that the BCSR CPLD code has a decoding bug,
or a timing issue. Another possibility is that the CPLD code
that monitors CS#[0] for RCW access on boot is screwy and
is allowing a write to CS#[1] to occur.

We have had several other weird issues with the ethernet
interfaces failing too, and their enable registers are
in BCSR[0], so this would explain that too.

I figured that this may help others explain weird issues
they may be seeing on their development boards.

Cheers,
Dave

PS. The MDS board serial port is also driving the 3.3V
processor with 5V logic levels (4V measured) which is
bad-bad-bad, but thats not the cause of our current issue.

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[Dave Liu]

On Tue, 2008-07-22 at 16:14 -0700, David Hawkins wrote:
> Hi all,
> 
> We're having a rough week with our Freescale MDS boards.
> I've just submitted a service request through
> Freescale's online system. However, I figured others
> (eg. Kim, and Timur) might be interested in the following
> problem:
> 
> We have an MPC8349E-MDS-PB and an MPC8349EA-MDS-PB that we
> are using to develop PCI agent mode software. After
> debugging the I2C flash write issue, we found that the
> serial port would stop receiving shortly after boot (we
> could see the U-Boot boot messages). We traced the issue
> to the RS232 receiver enable bit in the BCSR register 0
> being written to; disabling the serial port receiver
> tri-state outputs.
> 
> The RS232 receiver enable bit is not being changed by a
> write to the BCSR, but by a write to the *flash*.
> 
> Yikes!!
> 
> We were adding print statements that showed the value of
> the BCSR code in U-Boot. We found the error occurred
> during the Flash CFI code. I don't think the problem is
> related to CFI code - thats just where the corruption is
> triggered.
> 
> Starting with U-Boot head, we applied Timur's I2C patch,
> and then added the following code sequence to
> __flash_detect_cfi in U-boot:
> 
>   - write 0x2F (the reset value) to the
>     BCSR[0] register at 0xE2400000
>   - read BCSR[0] (to confirm the write)
>   - write 0xCC to Flash (0xFE000000)
>   - read BCSR[0] (to show the value changed)
> 
> (If you would like to repeat this test, please ask and Ira
> can send a patch file).
> 
> The board was operated in stand-alone host-mode on the bench.
> 
> The following logic analyzer traces show:
> 
> a) the case where this sequence did nothing,
> 
> http://www.ovro.caltech.edu/~dwh/mpc8349e_bcsr_read_ok.pdf
> 
> and
> 
> (b) the case where the sequence changed the value read
> back from the BCSR base address to 0xCC.
> 
> http://www.ovro.caltech.edu/~dwh/mpc8349e_bcsr_read_bad.pdf
> 
> We were able to trigger the error on both the E and EA board,
> but it would not fail every single reset - it occurred about
> 1-in-3 tries - hence it seems like a timing issue.
> It could be that the BCSR CPLD code has a decoding bug,
> or a timing issue. Another possibility is that the CPLD code
> that monitors CS#[0] for RCW access on boot is screwy and
> is allowing a write to CS#[1] to occur.
> 
> We have had several other weird issues with the ethernet
> interfaces failing too, and their enable registers are
> in BCSR[0], so this would explain that too.
> 
> I figured that this may help others explain weird issues
> they may be seeing on their development boards.

I believe it is timing issue.
The BCSR read(#LOE) has not enough setup time.
You may try the two solutions:
(a) slow down local bus clock
    If you are using default configuration, the local bus clock is
66MHz. change the LCRR from div 4 to div 8 to make the local bus clock
as 33MHz. you can find it in the MPC8349EMDS.h

(b) Tuning the #CS1 timing, such as
change the OR1 from 0xFFFFE8F0 to 0xFFFFE9F7

Let us know the result.

Thanks,
Dave

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[Dave Liu]

On Wed, 2008-07-23 at 14:16 +0800, Dave Liu wrote:
> On Tue, 2008-07-22 at 16:14 -0700, David Hawkins wrote:
> > Hi all,
> > 
> > We're having a rough week with our Freescale MDS boards.
> > I've just submitted a service request through
> > Freescale's online system. However, I figured others
> > (eg. Kim, and Timur) might be interested in the following
> > problem:
> > 
> > We have an MPC8349E-MDS-PB and an MPC8349EA-MDS-PB that we
> > are using to develop PCI agent mode software. After
> > debugging the I2C flash write issue, we found that the
> > serial port would stop receiving shortly after boot (we
> > could see the U-Boot boot messages). We traced the issue
> > to the RS232 receiver enable bit in the BCSR register 0
> > being written to; disabling the serial port receiver
> > tri-state outputs.
> > 
> > The RS232 receiver enable bit is not being changed by a
> > write to the BCSR, but by a write to the *flash*.

Anyway,
I can not duplicated the issue on my MPC8349E-MDS-PB board.
I'm using the currently latest denx tree. the tool chain is
gcc-4.1.78-eglibc-2.5.78-1/powerpc-e300c3-linux-gnu


 

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[Ira Snyder]

On Wed, Jul 23, 2008 at 02:16:42PM +0800, Dave Liu wrote:
> 
> I believe it is timing issue.
> The BCSR read(#LOE) has not enough setup time.
> You may try the two solutions:
> (a) slow down local bus clock
>     If you are using default configuration, the local bus clock is
> 66MHz. change the LCRR from div 4 to div 8 to make the local bus clock
> as 33MHz. you can find it in the MPC8349EMDS.h
> 
> (b) Tuning the #CS1 timing, such as
> change the OR1 from 0xFFFFE8F0 to 0xFFFFE9F7
> 
> Let us know the result.
> 

I tried both changes, one at a time and both together. None of the
combinations helped.

I've attached a patch which demonstrates that the BCSR 0 value changes
with a write to the flash. The patch has Timur's i2c patch rolled in,
since it has not hit mainline yet. The modification to lib_ppc/board.c
writes the hard reset value back to BCSR 0, which makes the serial port
and ethernet start working again.

Here is the output of U-Boot head + the attached patch:
==========================================================================
U-Boot 1.3.4-rc1-00014-gcacee6b-dirty (Jul 23 2008 - 10:11:52) MPC83XX

Reset Status: Software Hard, External/Internal Soft, External/Internal
Hard

CPU:   e300c1, MPC8349E, Rev: 1.1 at 528 MHz, CSB: 264 MHz
Board: Freescale MPC8349EMDS
I2C:   ready
SPI:   ready
DRAM:  256 MB (DDR1, 64-bit, ECC off, 264 MHz)
FLASH: 
BCSR READ1 2f
BCSR READ2 f0

BCSR READ1 ff
BCSR READ2 f0

BCSR READ1 ff
BCSR READ2 00
 8 MB
In:    serial
Out:   serial
Err:   serial
Net:   TSEC0, TSEC1
BCSR FIXUP: was 00 now 2f

Type "run flash_nfs" to mount root filesystem over NFS

Hit any key to stop autoboot:  0 


Thanks for the help,
Ira


diff --git a/drivers/i2c/fsl_i2c.c b/drivers/i2c/fsl_i2c.c
index 9f2c1ec..9d5df8a 100644
--- a/drivers/i2c/fsl_i2c.c
+++ b/drivers/i2c/fsl_i2c.c
@@ -143,12 +143,15 @@ void
 i2c_init(int speed, int slaveadd)
 {
 	struct fsl_i2c *dev;
+	unsigned int temp;
 
 	dev = (struct fsl_i2c *) (CFG_IMMR + CFG_I2C_OFFSET);
 
 	writeb(0, &dev->cr);			/* stop I2C controller */
 	udelay(5);				/* let it shutdown in peace */
-	i2c_bus_speed[0] = set_i2c_bus_speed(dev, gd->i2c1_clk, speed);
+	temp = set_i2c_bus_speed(dev, gd->i2c1_clk, speed);
+	if (gd->flags & GD_FLG_RELOC)
+		i2c_bus_speed[0] = temp;
 	writeb(slaveadd << 1, &dev->adr);	/* write slave address */
 	writeb(0x0, &dev->sr);			/* clear status register */
 	writeb(I2C_CR_MEN, &dev->cr);		/* start I2C controller */
@@ -158,7 +161,9 @@ i2c_init(int speed, int slaveadd)
 
 	writeb(0, &dev->cr);			/* stop I2C controller */
 	udelay(5);				/* let it shutdown in peace */
-	i2c_bus_speed[1] = set_i2c_bus_speed(dev, gd->i2c2_clk, speed);
+	temp = set_i2c_bus_speed(dev, gd->i2c2_clk, speed);
+	if (gd->flags & GD_FLG_RELOC)
+		i2c_bus_speed[1] = temp;
 	writeb(slaveadd << 1, &dev->adr);	/* write slave address */
 	writeb(0x0, &dev->sr);			/* clear status register */
 	writeb(I2C_CR_MEN, &dev->cr);		/* start I2C controller */
diff --git a/drivers/mtd/cfi_flash.c b/drivers/mtd/cfi_flash.c
index 12647ef..12c794e 100644
--- a/drivers/mtd/cfi_flash.c
+++ b/drivers/mtd/cfi_flash.c
@@ -1617,7 +1617,11 @@ static int __flash_detect_cfi (flash_info_t * info, struct cfi_qry *qry)
 	/* We do not yet know what kind of commandset to use, so we issue
 	   the reset command in both Intel and AMD variants, in the hope
 	   that AMD flash roms ignore the Intel command. */
+	printf("\nBCSR READ1 %.2x\n", readb(CFG_BCSR));
+	udelay(1000);
 	flash_write_cmd (info, 0, 0, AMD_CMD_RESET);
+	printf("BCSR READ2 %.2x\n", readb(CFG_BCSR));
+	udelay(1000);
 	flash_write_cmd (info, 0, 0, FLASH_CMD_RESET);
 
 	for (cfi_offset=0;
diff --git a/lib_ppc/board.c b/lib_ppc/board.c
index 71a70db..1f10a0d 100644
--- a/lib_ppc/board.c
+++ b/lib_ppc/board.c
@@ -26,6 +26,7 @@
 #include <command.h>
 #include <malloc.h>
 #include <devices.h>
+#include <asm/io.h>
 #ifdef CONFIG_8xx
 #include <mpc8xx.h>
 #endif
@@ -1179,6 +1180,9 @@ void board_init_r (gd_t *id, ulong dest_addr)
  }
 #endif
 
+	printf("BCSR FIXUP: was %.2x now 2f\n", readb(CFG_BCSR));
+	writeb(0x2f, CFG_BCSR);
+
 	/* Initialization complete - start the monitor */
 
 	/* main_loop() can return to retry autoboot, if so just run it again. */

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[David Hawkins]

Hi Dave,

> I believe it is timing issue.
> The BCSR read(#LOE) has not enough setup time.
> You may try the two solutions:
> (a) slow down local bus clock
>     If you are using default configuration, the local bus clock is
> 66MHz. change the LCRR from div 4 to div 8 to make the local bus clock
> as 33MHz. you can find it in the MPC8349EMDS.h
> 
> (b) Tuning the #CS1 timing, such as
> change the OR1 from 0xFFFFE8F0 to 0xFFFFE9F7
> 
> Let us know the result.

Sorry for not getting back to you sooner, I've been out of
touch for the last week.

Freescale support responded that they were able to reproduce
the issue on their hardware. In case you are interested;
the support person in question was Fedor, and the SR# is
1-458670791. (I don't expect you to work on this, I'm just
providing the information in case you care to look at it).

I haven't talked in detail with Ira regarding the additional
tests he performed after I left last week, however, I believe
he changed the clock rates as you mentioned and the error
still occurred.

Our work-around that just writes the correct reset value to
BCSR[0] was sufficient to keep him working on productive code.

Since you were not able to reproduce the problem on your
hardware, I'm interested to learn how you determined that
the LOE# signal does not have enough setup time. Did you
look at the placed-and-routed BCSR project to determine
actual timing?

I have a copy of the Verilog code used in the BCSR, but not
the Xilinx ISE project. I'll take a more detailed look at the
code, but I won't be able to look at the timing, unless I ask
SR for an ISE project. I'll let them get back to me with
their solution before I go trying to get the project files.
Hopefully the fix will be to update the BCSR, and I'll just
have to build a Xilinx download cable, and use the free
web version of ISE to update the CPLD.

Thanks for helping look at this issue.

Cheers,
Dave

[U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[David Hawkins]

> I have a copy of the Verilog code used in the BCSR

I took a closer look at the Verilog code. Searching
for the I/O port nBCSR_CS within the code shows that
it is synchronized through a single register to the 33MHz
clock (which comes from on-board oscillator, not the
processor) to generate the signal nBCSRCSsyn.

This code is incorrect, the signal nBCSR_CS is asynchronous
relative to the 33MHz clock. As such, the signal should be
registered through two registers to synchronize it to the
33MHz clock domain.

This synchronization error would cause multiple transient issues.
These issues will not show up in simulation, as the
metastability of the register outputs is not simulated
(by Modelsim anyway). These issues will also show up on the
hardware, regardless of the PowerPC local clock setting, as
the BCSR is accessed in PowerPC LBC GPCM mode, and the CPLD
is using its own 33MHz clock to generate GPCM responses.

Cheers,
Dave

[U-Boot] [U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[David Hawkins]

Hi Dave and Kim,

Freescale support got back to me regarding the BCSR
corruption error. They determined there was a bug in
the v1.2 BCSR code, and now have a v1.3.

For anyone else with an MDS board, if you want to
update your BCSR, submit a SR to Freescale and
ask for the v1.3 EEPROM file, or ask me and I'll send
the file.

I didn't have any Xilinx FPGA boards, so didn't have
any programming cables. So I programmed an Altera
FPGA board to act as a Xilinx Parallel Cable III
(the web has the schematic), and downloaded the old
version of the Xilinx ISE 9.2i tools, since they has
support for that cable. I used the iMPACT tool to
detect the devices in the JTAG chain (EEPROM and
Spartan) and then used it to program the EEPROM.

The BCSR version register reads back as v1.3.

The erroneous over-write of the BCSR[0] register
no longer occurs.

Cheers,
Dave

[U-Boot] [U-Boot-Users] Freescale MPC8349EMDS BCSR corruption

[Liu Dave-R63238]

Thanks David Hawkins, 

> -----Original Message-----
> From: David Hawkins [mailto:dwh at ovro.caltech.edu] 
> Sent: 2008?10?8? 11:51 AM
> To: u-boot at lists.denx.de
> Cc: Liu Dave-R63238; Phillips Kim-R1AAHA
> Subject: Re: [U-Boot-Users] Freescale MPC8349EMDS BCSR corruption
> 
> 
> Hi Dave and Kim,
> 
> Freescale support got back to me regarding the BCSR
> corruption error. They determined there was a bug in
> the v1.2 BCSR code, and now have a v1.3.
> 
> For anyone else with an MDS board, if you want to
> update your BCSR, submit a SR to Freescale and
> ask for the v1.3 EEPROM file, or ask me and I'll send
> the file.
> 
> I didn't have any Xilinx FPGA boards, so didn't have
> any programming cables. So I programmed an Altera
> FPGA board to act as a Xilinx Parallel Cable III
> (the web has the schematic), and downloaded the old
> version of the Xilinx ISE 9.2i tools, since they has
> support for that cable. I used the iMPACT tool to
> detect the devices in the JTAG chain (EEPROM and
> Spartan) and then used it to program the EEPROM.
> 
> The BCSR version register reads back as v1.3.
> 
> The erroneous over-write of the BCSR[0] register
> no longer occurs.
> 
> Cheers,
> Dave
> 
> 
> 

[U-Boot-Users] Freescale MPC8349EMDS hang on boot

[Ira Snyder]

On Thu, Jul 17, 2008 at 04:54:53PM -0500, Kim Phillips wrote:
> On Wed, 16 Jul 2008 15:28:21 -0700
> Ira Snyder <iws@ovro.caltech.edu> wrote:
> 
> > During the development, I noticed that adding and removing certain
> > pieces of debugging code (which did not change any program state) caused
> > the board to hang on boot, like so:
> > ================================================================================
> > 
> > U-Boot 1.3.4-rc1-00001-gb89881c-dirty (Jul 16 2008 - 14:54:56) MPC83XX
> > 
> > Reset Status: External/Internal Soft, External/Internal Hard
> > 
> > CPU:   e300c1, MPC8349EA, Rev: 3.0 at 528 MHz, CSB: 264 MHz
> > Board: Freescale MPC8349EMDS
> > I2C:   
> 
> try reverting this commit:
> 
> commit 1c3dd43338a077165e7e0309cb3994e65d2bdbf8
> Author: Grant Likely <grant.likely@secretlab.ca>
> Date:   Tue Nov 13 22:18:33 2007 -0700
> 
>     powerpc: Backout relocation changes.
>     
>     Ugh.  I *hate* to back this change out, but these compiler flags don't
>     work for relocation on all versions of GCC.  I've not been able to
>     reproduce the environment in my setup (and hence, not been able to
>     find a combination that *does* work), so I've got no choice but to go
>     back to the old gcc flags and linker script.
>     
>     Signed-off-by: Grant Likely <grant.likely@secretlab.ca>
> 
> Kim

Ok, I have tried reverting this commit. With the patch from my previous
email that aligns the .data section start address to a multiple of 32
bytes, the board still hangs on boot.

I removed the ". = ALIGN(32);" from the mpc83xx linker script, and (by
coincidence the .data section start address was still aligned to 32
bytes (the board still hung on boot). I added ". = . + 4;" in the same
place as the ALIGN so I did not have to make any code changes. This
moved the start address of .data off of the 32 byte boundary. The board
booted up fine in this case.

So the revert didn't seem to help. The board still breaks when .data
starts on a 32 byte boundary.

I hope this helps narrow down the problem.

Thanks again for the help,
Ira