[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Scott Coulter]

Hi everyone,

I have a custom MPC8540-based board running U-boot 1.1.4.  The processor
core runs at 833MHz and the memory is DDR 333MHz with ECC.  With a
single banked SoDIMM memory module, 256 or 512Mbyte, the board can run a
memory test and boot/run linux 2.6.15.  With a multiple banked (2)
1Gbyte memory module, the board can boot/run linux 2.6.15, but, the
memory test hangs in the vicinity of the bank (512Mbyte) boundary.  The
uboot fill command also hangs.  We can also generate a hang condition by
doing a sequence of reads of a memory range that crosses the bank
boundary.  Individual peeks and pokes of memory work fine.  It seems
that a prolonged sequence of accesses is required to cause the problem.

If I run 'mtest 1fff0000 20010000' (over the 512MB boundary on a 1GB
module), the test hangs in seconds.

If I use a 512MB SoDIMM module which use two Chip Selects (2 banks).  I
can run 'mtest fff0000 10010000' from U-boot for hours without a
problem.

Investigation with a JTAG emulator shows that the processor is not hung,
but rather the code is stuck in a loop trying to output characters to
the serial port (polling the LSR in the DUART for the transmitter to be
empty).  According to the JTAG emulator, the data being read in the loop
during the "hang" condition was 0.  However, if I read the register
manually using the emulator, the value returned is 0x60 which should
have caused the loop to exit and the character to be written.

On the board, we have an LED register connected to the local bus.  If I
write to the LEDs prior to the DUART LSR polling loop the memory test
runs fine.  What effect should the local bus access have on the DUART
polling loop?  According to the code and the emulator, the DUART
registers do not show up in the data cache.

Since we are testing over a nearly 1Gbyte memory range (the exception
table and the u-boot area are excluded), and the code only writes to the
serial port before and after the memory fill operation and then before
and after the read/verify operation, I cannot see that the led register
write is having any significant impact on the DRAM access timing.

The following is the DRAM initialization data from Uboot:

DRAM:  Initializing
DDR: number of banks = 2
DDR: DDR I bank density = 0x20000000
Number of Row Address Bits - 13
Number of Col Address Bits - 11

DDR: cs0_bnds   = 0x0000001f
DDR: cs0_config = 0x80000103
DDR: cs1_bnds   = 0x0020003f
DDR: cs1_config = 0x80000103
DDR: caslat SPD bit is 3
DDR: tCKmin = 6000 ps
DDR: Module max data rate = 333 Mhz
DDR: effective data rate is 333 MHz
DDR: caslat SPD bit is 3, controller field is 0x4
DDR: ext_refrec = 0x00000000
DDR: timing_cfg_1  = 0x37344321
DDR: timing_cfg_2 = 0x00000c00
DDR: sdram_mode   = 0x00000062
DDR: sdram_mode_2 = 0x00000000
DDR: sdram_interval = 0x05130100
DDR: err_disable = 0x0000000d
DDR: err_sbe = 0x00ff0000
DDR: sdram_cfg_2  = 0x00000000
ECC enabled
DDR: sdram_cfg   = 0xe2008000
Setup for 256MB pagesize
Setup TLB index #8 for address 0x00000000
DDR: MAS0=0x10080000
DDR: MAS1=0xc0000900
DDR: MAS2=0x00000000
DDR: MAS3=0x00000015
Setup TLB index #9 for address 0x10000000
DDR: MAS0=0x10090000
DDR: MAS1=0xc0000900
DDR: MAS2=0x10000000
DDR: MAS3=0x10000015
Setup TLB index #10 for address 0x20000000
DDR: MAS0=0x100a0000
DDR: MAS1=0xc0000900
DDR: MAS2=0x20000000
DDR: MAS3=0x20000015
Setup TLB index #11 for address 0x30000000
DDR: MAS0=0x100b0000
DDR: MAS1=0xc0000900
DDR: MAS2=0x30000000
DDR: MAS3=0x30000015
DDR: LAWBAR1=0x00000000
DDR: LARAR1=0x80f0001d
DMA DDR: err_disable = 0x0000000d
DMA DDR: err_disable = 0x00000000
    DDR: 1024 MB

This is the uboot serial routine with the led writes:

void NS16550_putc (NS16550_t com_port, char c) {
	/* snc - debug */
	volatile unsigned char *led = (volatile unsigned char
*)0xffb00000;

	*led = 0xa;
	while ((com_port->lsr & LSR_THRE) == 0);
	com_port->thr = c;
}

I am pretty sure that the memory design is OK since even with a 2 bank,
1GB memory module will, the board will  boot and run linux, mount its
root filesystem over NFS, be able to mount an NFS partition containing
the linux 2.6.15 kernel source, and be able to repeatedly compile the
kernel over NFS overnight.

Anyone have any thoughts?

Thanks in advance,

Scott





___________________________________________________________________

  Scott N. Coulter
  Senior Software Engineer
  
  Cyclone Microsystems          
  370 James Street              Phone:  203.786.5536 ext. 118
  New Haven, CT 06513-3051      Email:  scott.coulter at cyclone.com
  U.S.A.                        Web:    http://www.cyclone.com
___________________________________________________________________

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Jerry Van Baren]

Scott Coulter wrote:
> Hi everyone,
> 
> I have a custom MPC8540-based board running U-boot 1.1.4.  The processor
> core runs at 833MHz and the memory is DDR 333MHz with ECC.  With a
> single banked SoDIMM memory module, 256 or 512Mbyte, the board can run a
> memory test and boot/run linux 2.6.15.  With a multiple banked (2)
> 1Gbyte memory module, the board can boot/run linux 2.6.15, but, the
> memory test hangs in the vicinity of the bank (512Mbyte) boundary.  The
> uboot fill command also hangs.  We can also generate a hang condition by
> doing a sequence of reads of a memory range that crosses the bank
> boundary.  Individual peeks and pokes of memory work fine.  It seems
> that a prolonged sequence of accesses is required to cause the problem.
> 
> If I run 'mtest 1fff0000 20010000' (over the 512MB boundary on a 1GB
> module), the test hangs in seconds.
> 
> If I use a 512MB SoDIMM module which use two Chip Selects (2 banks).  I
> can run 'mtest fff0000 10010000' from U-boot for hours without a
> problem.
> 
> Investigation with a JTAG emulator shows that the processor is not hung,
> but rather the code is stuck in a loop trying to output characters to
> the serial port (polling the LSR in the DUART for the transmitter to be
> empty).  According to the JTAG emulator, the data being read in the loop
> during the "hang" condition was 0.  However, if I read the register
> manually using the emulator, the value returned is 0x60 which should
> have caused the loop to exit and the character to be written.

OK, what is the system trying to send out the UART when the hang occurs? 
  Since you only send characters out the UART "when the test is done" 
either the test has been successfully completed or it is sending a 
failure message.  The two cases would cause different guesses.

> On the board, we have an LED register connected to the local bus.  If I
> write to the LEDs prior to the DUART LSR polling loop the memory test
> runs fine.  What effect should the local bus access have on the DUART
> polling loop?  According to the code and the emulator, the DUART
> registers do not show up in the data cache.

Are you missing "sync" instructions in your UART handling?  Sounds like 
the UART register operations are not being forced to run by the 
processor... when you hit the LED register, you are forcing the 
instructions to run as a side effect and all becomes well.

[snip]

> Thanks in advance,
> Scott

You are welcome in advance, ;-)
gvb

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Scott Coulter]

Hi Jerry,

> OK, what is the system trying to send out the UART when the hang
occurs? 
> Since you only send characters out the UART "when the test is done" 
> either the test has been successfully completed or it is sending a 
> failure message.  The two cases would cause different guesses.

I suspect that the "hang" is caused by a completion message since I can
also make the processor "hang" by only reading large chunks of memory in
a loop.

> Are you missing "sync" instructions in your UART handling?  Sounds
like 
> the UART register operations are not being forced to run by the 
> processor... when you hit the LED register, you are forcing the 
>instructions to run as a side effect and all becomes well.

The UART code that I am using is plain vanilla U-boot 1.1.4,
drivers/ns16550.c.  I dropped an asm("sync;isync;msync") in front of the
loop which polls the LSR and the behavior didn't change.  I dropped an
asm("sync;isync;msync") inside the loop which polls the LSR and the
behavior didn't change.

Scott

___________________________________________________________________

  Scott N. Coulter
  Senior Software Engineer
  
  Cyclone Microsystems          
  370 James Street              Phone:  203.786.5536 ext. 118
  New Haven, CT 06513-3051      Email:  scott.coulter at cyclone.com
  U.S.A.                        Web:    http://www.cyclone.com
___________________________________________________________________

-----Original Message-----
From: u-boot-users-bounces@lists.sourceforge.net
[mailto:u-boot-users-bounces at lists.sourceforge.net] On Behalf Of Jerry
Van Baren
Sent: Wednesday, July 05, 2006 12:39 PM
To: u-boot-users at lists.sourceforge.net
Subject: Re: [U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory
module

Scott Coulter wrote:
> Hi everyone,
> 
> I have a custom MPC8540-based board running U-boot 1.1.4.  The
processor
> core runs at 833MHz and the memory is DDR 333MHz with ECC.  With a
> single banked SoDIMM memory module, 256 or 512Mbyte, the board can run
a
> memory test and boot/run linux 2.6.15.  With a multiple banked (2)
> 1Gbyte memory module, the board can boot/run linux 2.6.15, but, the
> memory test hangs in the vicinity of the bank (512Mbyte) boundary.
The
> uboot fill command also hangs.  We can also generate a hang condition
by
> doing a sequence of reads of a memory range that crosses the bank
> boundary.  Individual peeks and pokes of memory work fine.  It seems
> that a prolonged sequence of accesses is required to cause the
problem.
> 
> If I run 'mtest 1fff0000 20010000' (over the 512MB boundary on a 1GB
> module), the test hangs in seconds.
> 
> If I use a 512MB SoDIMM module which use two Chip Selects (2 banks).
I
> can run 'mtest fff0000 10010000' from U-boot for hours without a
> problem.
> 
> Investigation with a JTAG emulator shows that the processor is not
hung,
> but rather the code is stuck in a loop trying to output characters to
> the serial port (polling the LSR in the DUART for the transmitter to
be
> empty).  According to the JTAG emulator, the data being read in the
loop
> during the "hang" condition was 0.  However, if I read the register
> manually using the emulator, the value returned is 0x60 which should
> have caused the loop to exit and the character to be written.

OK, what is the system trying to send out the UART when the hang occurs?

  Since you only send characters out the UART "when the test is done" 
either the test has been successfully completed or it is sending a 
failure message.  The two cases would cause different guesses.

> On the board, we have an LED register connected to the local bus.  If
I
> write to the LEDs prior to the DUART LSR polling loop the memory test
> runs fine.  What effect should the local bus access have on the DUART
> polling loop?  According to the code and the emulator, the DUART
> registers do not show up in the data cache.

Are you missing "sync" instructions in your UART handling?  Sounds like 
the UART register operations are not being forced to run by the 
processor... when you hit the LED register, you are forcing the 
instructions to run as a side effect and all becomes well.

[snip]

> Thanks in advance,
> Scott

You are welcome in advance, ;-)
gvb

Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your job
easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
U-Boot-Users mailing list
U-Boot-Users at lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/u-boot-users

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Joao, Nuno]

> ...
> > Are you missing "sync" instructions in your UART handling?  Sounds
> like 
> > the UART register operations are not being forced to run by the 
> > processor... when you hit the LED register, you are forcing the 
> >instructions to run as a side effect and all becomes well.
> 
> The UART code that I am using is plain vanilla U-boot 1.1.4,
> drivers/ns16550.c.  I dropped an asm("sync;isync;msync") in 
> front of the
> loop which polls the LSR and the behavior didn't change.  I dropped an
> asm("sync;isync;msync") inside the loop which polls the LSR and the
> behavior didn't change.

	I just want to mention that those instructions will only "sync"
	from the processor core's point of view. However, PQIII has this
	coeherency module between the processor core and the rest of
	the system, which adds another level of "asynchronism". For a
	"real sync" in this chip, specially when there are dependencies
	between different subsystems, you need to do a read after a
	write; that will force the processor to really sync with the
	device and it is the only way for you to make sure that a
	value has been written to the sub-system.

	Best Regards

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Joao, Nuno]

> It seems
> that a prolonged sequence of accesses is required to cause the
problem.

	Heating problem? It can cause strange problems.

> The following is the DRAM initialization data from Uboot:
> 
> DRAM:  Initializing
> DDR: number of banks = 2
> DDR: DDR I bank density = 0x20000000
> Number of Row Address Bits - 13
> Number of Col Address Bits - 11
> 
> DDR: cs0_bnds   = 0x0000001f
> DDR: cs0_config = 0x80000103
> (...)
> DDR: err_sbe = 0x00ff0000
> DDR: sdram_cfg_2  = 0x00000000

	The configuration of the DDR is rather complex, specially
	the timming parameters and, of course, it depends a bit on
	the DDR in question. This is usually done by the hardware
	team, they have to give the parameters. Where I work I have
	a hardware collegue specialized in this kind of configuration
	and it took him some time to fully analyse and optimize this
	parameters for a MPC8541E based custom board.

	Best Regards

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Jerry Van Baren]

Joao, Nuno (ext) wrote:
>> It seems
>> that a prolonged sequence of accesses is required to cause the
> problem.
> 
> 	Heating problem? It can cause strange problems.

Long shot.  Still, it could be...

>> The following is the DRAM initialization data from Uboot:
>>
>> DRAM:  Initializing
>> DDR: number of banks = 2
>> DDR: DDR I bank density = 0x20000000
>> Number of Row Address Bits - 13
>> Number of Col Address Bits - 11
>>
>> DDR: cs0_bnds   = 0x0000001f
>> DDR: cs0_config = 0x80000103
>> (...)
>> DDR: err_sbe = 0x00ff0000
>> DDR: sdram_cfg_2  = 0x00000000
> 
> 	The configuration of the DDR is rather complex, specially
> 	the timming parameters and, of course, it depends a bit on
> 	the DDR in question. This is usually done by the hardware
> 	team, they have to give the parameters. Where I work I have
> 	a hardware collegue specialized in this kind of configuration
> 	and it took him some time to fully analyse and optimize this
> 	parameters for a MPC8541E based custom board.

My guess is that the SDRAM is initialized OK.
1) Sam claims linux boots and runs as long as he doesn't do a SDRAM test.
2) Sam indicated that the SDRAM test actually runs or appears to run to 
completion, but the UART register read (Tx ready) has stale data 
(program sees "busy", debugger sees "ready").

This would indicate that the SDRAM size (and initialization) is 
incidental to the problem - it is a way to trigger/replicate the 
problem, but is not the cause of the problem.  Also, the problem is 
"fixed" by doing an unrelated register write (*and probably read* - 
setting a LED is probably a read-modify-write operation).

Stale data in registers is very often caused by:
* Caching - reading cached values instead of the register
   * If the larger memory size changes the MMU configuration, it could
       mess up and inadvertently cache the registers
* Write posting - the write never made it to the register
   * Forgetting to do a "sync" or "EIEIO" to force proper data ordering
       (allowing the processor's BIU to reorder reads and writes)
   * Not doing a read to flush an external write posting buffer (e.g.
       PCI, PQ-III).  Your (Nuno's) previous email guessing the PQIII
       coherency module sounds very likely to me.
* Forgetting to put the "volatile" keyword on the register pointer

gvb

[U-Boot-Users] MPC8540 mtest "hang" with 1GByte memory module

[Scott Coulter]

Hi all,

I think I found it.  I based my boards u-boot port on the SBC85xx board
port.  My CCSRBAR was setup at 0xff700000.  Kicking around in init.S was
code to setup LAWBAR2/LAWAR2 starting at 0xe0000000 for 512MB of LBC
access and code to setup TLB3 to cover a cache-inhibited area starting
at 0xf0000000 for 256MB.  I changed both the LAW and TLB definitions to
only cover my flash starting at 0xffc00000 for 4MB and the memory test
works OK.  Somehow the processor must have been getting confused as to
whether the UART registers were in the CCSRBAR region or on the local
bus.

BTW I have since changed init.S to follow that of the MPC8540ADS and the
TQM85xx boards along with moving CCSRBAR to 0xe0000000.

Thanks for the input,

Scott





___________________________________________________________________

  Scott N. Coulter
  Senior Software Engineer
  
  Cyclone Microsystems          
  370 James Street              Phone:  203.786.5536 ext. 118
  New Haven, CT 06513-3051      Email:  scott.coulter at cyclone.com
  U.S.A.                        Web:    http://www.cyclone.com
___________________________________________________________________