[U-Boot] at91 failures under temperature?

[Joey Oravec <joravec@drewtech.com>]

Hi -

I've been investigating a problem involving the at91sam9261 processor. 
Across 2 or 3 years we've manufactured two products with:

Crystal: 16.000 MHz
Processor Clock: 240 MHz (PLLA *15)
Master Clock: 120 MHz
VDDCORE: 1.2v

Based on testing history, about 10% of the assemblies fail starting 
above 30 degC. The simplest testcase to demonstrate is: 1) use uboot 
relocated to SDRAM as normal 2) enable caches 3) fill 16mb of SDRAM with 
a known pattern 4) use a test command to continually calculate and print 
CRC. As you apply heat to the processor the testcase will print an 
incorrect CRC. If the unit has not already crashed you can decrease the 
temperature and see the correct CRC again, which demonstrates that SDRAM 
hasn't flipped any bits.

Extending ram timings or lowering master clock (ie. sdram bus clock) has 
no impact. Lowering the processor clock to 224 MHz makes the problem go 
away -- at this speed I can heat the processor to the rated 85 C and the 
testcase above will not fail. Also replacing the processor usually makes 
the problem go away. You won't see the problem with any variable 
changed: caches disabled, any processor clock below 240 MHz, or VDDCORE 
at 1.3v.

I'm using PLL R/C calculated by the spreadsheet, and 240 MHz @ 1.2v 
should be within spec up to 85 C. I've requested failure analysis on 
some of the processors but I don't expect an answer any time soon.

Is anybody else successful with a 240 MHz processor clock, or are you 
using a lower processor clock for greater reliability? Can you recommend 
any other diagnostics that might identify the root cause? In the 
meantime for maximum reliability, I recommend using a processor clock 
less than 240 MHz and checking your hardware with a similar testcase.

-joey