MPC83xx watchdog reset board dead lock

MPC83xx watchdog reset board dead lock

[Leon Woestenberg]

Hello,

this is a hardware, even board issue, but I hope to find the right
target audience here.


In our MPC83xx design I would like to prevent dead lock in case where
a field upgrade is performed, i.e. NOR Flash is erased or written, and
the MPC83xx built-in hardware watchdog triggers.

In u-boot the scenario can be easily reproduced by running this
command (WARNING, erases some sectors!) on an MPC8313E-RDB:

erase_wdg=mw.l 0xe0000204 0x10000007 1;mw.w 0xe000020e 0x556c 1;mw.w
0xe000020e 0xaa39 1;erase 1:10-30

This sets up the watchdog to reset soonish, then starts erasing NOR
sectors. Watchdog triggers and resets -> Dead lock.


Most MPC8xxx board designs I have seen suffer from this possible dead lock:
- NOR Flash is put in erase mode or write mode
- Hardware watchdog triggers
- HRESET# is asserted by the processor, during which the configuration
words are read from NOR Flash.

Either
HRESET# is not attached to NOR, NOR stays in erase/write mode and
invalid words will be read -> dead lock

or either:
HRESET# is attached to NOR reset, NOR is reset, but stays in reset as
HRESET# stays asserted.



We have been looking at several solutions hardware wise that reset the
NOR flash on HRESET# going low, but the processors are stubborn,
read the config words only once, than dead lock.

I wonder if there are known-working designs for this.

Regards,
-- 
Leon

Re: MPC83xx watchdog reset board dead lock

[David Hawkins]

Hi Leon,

> Most MPC8xxx board designs I have seen suffer from this possible dead lock:
> - NOR Flash is put in erase mode or write mode
> - Hardware watchdog triggers
> - HRESET# is asserted by the processor, during which the configuration
> words are read from NOR Flash.
> 
> Either
> HRESET# is not attached to NOR, NOR stays in erase/write mode and
> invalid words will be read -> dead lock
> 
> or either:
> HRESET# is attached to NOR reset, NOR is reset, but stays in reset as
> HRESET# stays asserted.
> 
> We have been looking at several solutions hardware wise that reset the
> NOR flash on HRESET# going low, but the processors are stubborn,
> read the config words only once, than dead lock.
> 
> I wonder if there are known-working designs for this.

What do you do in the case of blank flash on a board?

I'm not sure that this will work for you, but it might ...
on my boards, the PowerPC connects to an FPGA on the local
bus, and the flash is connected to that FPGA (it made
routing easier to go through the FPGA). When the board
powers up, a small FSM in the FPGA reads from the first
word in Flash, and checks it is not blank. If it is
blank, the configuration pins are asserted such that
the processor uses a hard-coded reset configuration word,
and the processor reset is deasserted. If the flash is
not blank, the processor is allowed to boot from Flash.

An alternative method for firmware updates would be
to use the boot from high-mem versus low-mem flag
in the RCWs. On a number of the Freescale EVMs, you
can have two images in the Flash; one at low mem,
and one at high mem. Toggling a dip switch selects
the value of the BMS bit in the RCWs, and lets you boot
from one image or the other. This is a good trick that
avoids having to have a debugger to recover back
to a sane image - note that a CPLD delivers the RCWs,
not the flash.

So in either of these examples, the RCWs source is
sometimes not the Flash image ... which requires
that feature designed into the original hardware.

Those two ideas should get you thinking :)

Cheers,
Dave

Re: MPC83xx watchdog reset board dead lock

[Leon Woestenberg]

Hello,

On Tue, Jun 16, 2009 at 6:30 PM, David Hawkins<dwh@ovro.caltech.edu> wrote:
>> Most MPC8xxx board designs I have seen suffer from this possible dead
>> lock:
>> - NOR Flash is put in erase mode or write mode
>> - Hardware watchdog triggers
>> - HRESET# is asserted by the processor, during which the configuration
>> words are read from NOR Flash.
>>
>> Either
>> HRESET# is not attached to NOR, NOR stays in erase/write mode and
>> invalid words will be read -> dead lock
>>
>> or either:
>> HRESET# is attached to NOR reset, NOR is reset, but stays in reset as
>> HRESET# stays asserted.
>>
>> We have been looking at several solutions hardware wise that reset the
>> NOR flash on HRESET# going low, but the processors are stubborn,
>> read the config words only once, than dead lock.
>>
>> I wonder if there are known-working designs for this.
>
> What do you do in the case of blank flash on a board?
>
The problem is not with blank flash or firmware upgrades, we know how
to handle that.

Your solution is (a solution) to a different problem.

The problem lies in the fact that board dead lock can occur if the
watchdog triggers, for all reference designs I have seen.

Thanks for thinking along.  I would like to solve the original problem though.
BTW, we use CPLD/FPGAs on most of our boards, this one we do not for
cost reasons.


Regards,
-- 
Leon

Re: MPC83xx watchdog reset board dead lock

[David Hawkins]

Hi Leon,

>>> Most MPC8xxx board designs I have seen suffer from this possible dead
>>> lock:
>>> - NOR Flash is put in erase mode or write mode
>>> - Hardware watchdog triggers
>>> - HRESET# is asserted by the processor, during which the configuration
>>> words are read from NOR Flash.
>>>
>>> Either
>>> HRESET# is not attached to NOR, NOR stays in erase/write mode and
>>> invalid words will be read -> dead lock
>>>
>>> or either:
>>> HRESET# is attached to NOR reset, NOR is reset, but stays in reset as
>>> HRESET# stays asserted.
>>>
>>> We have been looking at several solutions hardware wise that reset the
>>> NOR flash on HRESET# going low, but the processors are stubborn,
>>> read the config words only once, than dead lock.
>>>
>>> I wonder if there are known-working designs for this.
>> What do you do in the case of blank flash on a board?
>>
> The problem is not with blank flash or firmware upgrades, we know how
> to handle that.
> 
> Your solution is (a solution) to a different problem.
> 
> The problem lies in the fact that board dead lock can occur if the
> watchdog triggers, for all reference designs I have seen.
> 
> Thanks for thinking along.  I would like to solve the original problem though.
> BTW, we use CPLD/FPGAs on most of our boards, this one we do not for
> cost reasons.

So we're talking about the sequence where HRESET# asserts as
in say the logic analyzer trace on p34:

http://www.ovro.caltech.edu/~dwh/carma_board/powerpc_mpc8349e.pdf

the LALE pulse where the processor reads the RCWs occurs very
soon after the falling edge of HRESET#.

So the Flash needs to be reset to ensure that it is in
read-array mode, so that the processor doesn't choke.
Since HRESET# is still low, thats no good. A pulse
generator that is based on HRESET# might work, but
the pulse would have to be long enough to meet any
reset requirement of the flash, yet short enough so
that the read of the first RCW would be valid. Since
the local bus is running really slow at this point,
I think that could be done ok.

How about a set-reset flip flop that is set on the
falling edge of HRESET# and cleared on the rising
edge of LALE. That'll produce a decent reset
pulse to the flash, and then there is plenty of
time for the first access to produce valid data
on the bus.

Cheers,
Dave

Re: MPC83xx watchdog reset board dead lock

[Norbert van Bolhuis]

Hi Leon,

I doubt if there are working designs for this.

In u-boot the watchdog (if enabled with CONFIG_WATCHDOG) is normally
strobed in the decrementer interrupt routine (timer_interrupt). So
I guess there's not a big chance it triggers a reset.

It is possible to configure the WD to issue a machine check interrupt
(i.s.o. HRESET). Maybe it's possible (or even done already) to put the
flash into READ-mode from the isr ?

---
N. van Bolhuis.



Leon Woestenberg wrote:
> Hello,
> 
> this is a hardware, even board issue, but I hope to find the right
> target audience here.
> 
> 
> In our MPC83xx design I would like to prevent dead lock in case where
> a field upgrade is performed, i.e. NOR Flash is erased or written, and
> the MPC83xx built-in hardware watchdog triggers.
> 
> In u-boot the scenario can be easily reproduced by running this
> command (WARNING, erases some sectors!) on an MPC8313E-RDB:
> 
> erase_wdg=mw.l 0xe0000204 0x10000007 1;mw.w 0xe000020e 0x556c 1;mw.w
> 0xe000020e 0xaa39 1;erase 1:10-30
> 
> This sets up the watchdog to reset soonish, then starts erasing NOR
> sectors. Watchdog triggers and resets -> Dead lock.
> 
> 
> Most MPC8xxx board designs I have seen suffer from this possible dead lock:
> - NOR Flash is put in erase mode or write mode
> - Hardware watchdog triggers
> - HRESET# is asserted by the processor, during which the configuration
> words are read from NOR Flash.
> 
> Either
> HRESET# is not attached to NOR, NOR stays in erase/write mode and
> invalid words will be read -> dead lock
> 
> or either:
> HRESET# is attached to NOR reset, NOR is reset, but stays in reset as
> HRESET# stays asserted.
> 
> 
> 
> We have been looking at several solutions hardware wise that reset the
> NOR flash on HRESET# going low, but the processors are stubborn,
> read the config words only once, than dead lock.
> 
> I wonder if there are known-working designs for this.
> 
> Regards,

Re: MPC83xx watchdog reset board dead lock

[Leon Woestenberg]

Hello all,

On Wed, Jun 17, 2009 at 10:35 AM, Norbert van
Bolhuis<nvbolhuis@aimvalley.nl> wrote:
> Hi Leon,
>
> I doubt if there are working designs for this.
> ...
> In u-boot the watchdog (if enabled with CONFIG_WATCHDOG) is normally
> strobed in the decrementer interrupt routine (timer_interrupt). So
> I guess there's not a big chance it triggers a reset.
> ...
>
Most designs do not care about the watchdog, or only pet in their
non-critical paths... That's not what the watchdog is for.
Also, I don't care about u-boot.

I care about a design where the Flash NOR could be in write mode at
any time when the watchdog triggers, when the hardware is running
critical software.
No lifes in danger when it happens, only jobs, so no biggy :-)


David has been helpful in thinking this through, but we followed-up
offline, and we independently came up with the following design, so
this must therefore work (disclaimer applies).

Note, it DOES require a NOR flash that has a RY/BUSY# pin.

Quoting David Hawkins, who gave a very clear explanation:
---
How about using the RDY/BUSY# pin on the Flash in conjunction
with PORESET#. If the flash is busy, then the processor gets
PORESET#, otherwise, the HRESET# just does its normal thing.

That way PORESET# only ever asserts when you have the
combo of the Flash being busy and HRESET# asserting.

<...>

If you have the Flash BUSY# signal, then this scheme works
great, since using HRESET# low and BUSY# low to create a
PORESET# source is only active until the Flash RESET#
is asserted long enough for it to get out of the BUSY#
state and back into read-array mode.
---

Kudos to David.

I'll be testing the design tomorrow on the reference board, I'll
report results in this thread.


Regards / Groeten,
-- 
Leon

Re: MPC83xx watchdog reset board dead lock

[Leon Woestenberg]

Hello,

On Wed, Jun 17, 2009 at 12:09 PM, Leon
Woestenberg<leon.woestenberg@gmail.com> wrote:
> Quoting David Hawkins, who gave a very clear explanation:
> <...>
> If you have the Flash BUSY# signal, then this scheme works
> great, since using HRESET# low and BUSY# low to create a
> PORESET# source is only active until the Flash RESET#
> is asserted long enough for it to get out of the BUSY#
> state and back into read-array mode.
>

I just found out from a Spansion datasheet that the RY/BUSY# of a
typical Flash NOR is enabled by CE#, and tri-state otherwise.
CE# in turn is driven by the LCS# from the PowerPC.

So basically, the first configuration access cycle while the NOR is in
write mode, will pull CE# low, which results in RY/BUSY#
being driven.

I have measured this pulse is ~1.9 us. So the reset circuitry needs a
maximum minimum pulse duration of 1.9 us.

Our reset controller (DS1818) fulfills this requirement, with a T,PB of 1 us.

A reset controller will extend the reset pulse. This is needed because:

for Flash NOR: I have seen a mininum of 35 us reset pulse.
(for the PowerPC: PORESET# should be asserted externally for at least
32 input clock cycles)


Regards,
-- 
Leon

Re: MPC83xx watchdog reset board dead lock

[Norbert van Bolhuis]

Hi Leon,

...

> Most designs do not care about the watchdog, or only pet in their
> non-critical paths... That's not what the watchdog is for.
> Also, I don't care about u-boot.
> 
> I care about a design where the Flash NOR could be in write mode at
> any time when the watchdog triggers, when the hardware is running
> critical software.
> No lifes in danger when it happens, only jobs, so no biggy :-)
> 


true, I was just looking from SW/u-boot perspective.
Ideally the dead-lock is prevented on board/HW level.


> 
> David has been helpful in thinking this through, but we followed-up
> offline, and we independently came up with the following design, so
> this must therefore work (disclaimer applies).
> 
> Note, it DOES require a NOR flash that has a RY/BUSY# pin.
> 
> Quoting David Hawkins, who gave a very clear explanation:
> ---
> How about using the RDY/BUSY# pin on the Flash in conjunction
> with PORESET#. If the flash is busy, then the processor gets
> PORESET#, otherwise, the HRESET# just does its normal thing.
> 
> That way PORESET# only ever asserts when you have the
> combo of the Flash being busy and HRESET# asserting.
> 
> <...>
> 
> If you have the Flash BUSY# signal, then this scheme works
> great, since using HRESET# low and BUSY# low to create a
> PORESET# source is only active until the Flash RESET#
> is asserted long enough for it to get out of the BUSY#
> state and back into read-array mode.
> ---
> 
> Kudos to David.
> 
> I'll be testing the design tomorrow on the reference board, I'll
> report results in this thread.
> 


Interesting.
Looking forward to the results.

---
N. van Bolhuis.

Re: MPC83xx watchdog reset board dead lock

[Leon Woestenberg]

Hello,

On Wed, Jun 17, 2009 at 2:16 PM, Norbert van
Bolhuis<nvbolhuis@aimvalley.nl> wrote:
>>
>> I'll be testing the design tomorrow on the reference board, I'll
>> report results in this thread.
>
> Interesting.
> Looking forward to the results.
>
Design works as expected on the now slightly modified MPC8313E-RDB
board. Kudos to David.

Cheers,
-- 
Leon

Re: MPC83xx watchdog reset board dead lock

[David Hawkins]

Hi Leon,

>>> I'll be testing the design tomorrow on the reference board, I'll
>>> report results in this thread.
>> Interesting.
>> Looking forward to the results.
>>
> Design works as expected on the now slightly modified MPC8313E-RDB
> board.

That's great!

> Kudos to David.

I'm sure you would have come up with a similar solution,
had you had a chance to sleep on it :)

Glad to help out.

Cheers,
Dave