Failsafe bootloader

Failsafe bootloader

[Sam Ravnborg]

Hi all.

We are developing an application for an embedded target that will be
located in some rural areas. We foresee a need to update the target
with a new version of the application or a new kernel from time to time.

A few of the added requirements on top of that are:
1) The target shall boot up in the old kernel, if a bogus kernel is loaded
2) The target shall boot up in the old application, if a bogus
	application is loaded
3) On the management side an update shall be handled as a single file,
	the target may decide to 'unpack' it when received.

We have looked all over the net, but none of the bootloaders found
so far could meet the above demands.
The boot loaders usually have an interactive mode used to select
between different configurations - and there are no feedback from
the application side if the reboot actually went well.
The interactiviness does not fit well with an embedded target.

Do you know of a boot loader that partially or fully meets the above
requirements?

Background information
Today we are using VxWorks - for which we have made our own boot loader.
The boot laoder allows the boot loader itself, and the application part
to be upgraded - and if a restart is failed the old version will
be activated on a subsequent build.
I want the same behaviour in the Linux based target.

	Thanks in advance,

	Sam

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Mark Hatle]

A bit off topic, but I'll jump in.  (The comments are based on experience
designing items for myself, and also information from looking at commercial
product designs. i.e. hacking things like my TiVo.)

Sam Ravnborg wrote:
> Hi all.
>
> We are developing an application for an embedded target that will be
> located in some rural areas. We foresee a need to update the target
> with a new version of the application or a new kernel from time to time.
>

First thing, make sure you have some mechanism for checksuming and possibly
signing updates.  The first assumption is the update made it over properly if
that stuff passes.. (and your company did QA it..)  ;)

> A few of the added requirements on top of that are:
> 1) The target shall boot up in the old kernel, if a bogus kernel is loaded

This will require a watchdog timer, and modification to a bootloader to do
counting and other items to instruct it to go to the backup kernel.

> 2) The target shall boot up in the old application, if a bogus
> 	application is loaded

A simple application daemon (init?) can do similar.

> 3) On the management side an update shall be handled as a single file,
> 	the target may decide to 'unpack' it when received.
>

cpio, tar, even RPM have been used for this.  These are standard field-upgrade
techniques..  the key is that every product is unique.

What _I_ personally would do is have two sets of "partitions" (memory, disk
doesn't matter) one set for two kernels, current and "updates" and one set for
userspace current and "updates".  Then you have magic values that tell the
bootloader what matches with what.  On an initial update, you make this magic
value "temporary" until the boot (application run) has succeeded and then you
make it perminant.

But all in all this will require you to make changes to an existing bootloader
or write your own.

--Mark


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Wolfgang Denk]

In message <20030603192851.GA1921@mars.ravnborg.org> you wrote:
>
> A few of the added requirements on top of that are:
> 1) The target shall boot up in the old kernel, if a bogus kernel is loaded

Define "bogus kernel"! Depending if you mean one which was  corrupted
during  download/installation  you can use a checkum; if you mean one
which fails to start the application you can use a watchdog.

Both is trivial and supporeted by several bootloaders.

> 2) The target shall boot up in the old application, if a bogus
> 	application is loaded

Same as above. Here it comes handy if you don't have  to  bundle  the
application with the kernel into one image.

> 3) On the management side an update shall be handled as a single file,
> 	the target may decide to 'unpack' it when received.

OK, not too difficult.

> We have looked all over the net, but none of the bootloaders found
> so far could meet the above demands.

Ummm... how comes you missed to look into U-Boot? It easily  performs
all your reqirements. Without modifying a single line of code.

> The boot loaders usually have an interactive mode used to select
> between different configurations - and there are no feedback from
> the application side if the reboot actually went well.
> The interactiviness does not fit well with an embedded target.

U-Boot allows for canned command sequences and even shell scripts.

> Do you know of a boot loader that partially or fully meets the above
> requirements?

U-Boot meets your rrequirements fully (and some more :-)

> Today we are using VxWorks - for which we have made our own boot loader.
> The boot laoder allows the boot loader itself, and the application part
> to be upgraded - and if a restart is failed the old version will
> be activated on a subsequent build.
> I want the same behaviour in the Linux based target.

==> http://sourceforge.net/projects/u-boot

See also: http://www.denx.de/re/DPLG.html

If you have questions, send an email to the u-boot-users mailing list
(or to me).

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Bus error -- please leave by the rear door.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Wolfgang Denk]

In message <3EDCFB6C.4040102@mvista.com> you wrote:
>
> But all in all this will require you to make changes to an existing bootloader
> or write your own.

This is not correct. U-Boot can  perform  all  these  things  without
modification.  It  can  be  easily  adjusted even to more complicated
requirements - if necessary, write a shell script :-)


Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Perl already has an IDE.  It's called Unix.
                      -- Tom Christiansen in 375bd509@cs.colorado.edu

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Jerry Van Baren]

Check out U-Boot
   http://sourceforge.net/projects/u-boot
The project page on sourceforge is pretty much non-existant other than the
CVS link.  Pull down the source -- there _is_ a good README file in it that
explains a lot about u-boot.
   http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/u-boot/u-boot/README?rev=1.30
Wolfgang Denk of Denx Computer Systems runs the project.  He doesn't appear
to have anything on U-Boot on his web site either.
   http://www.denx.de/

The predecessor was ppcboot, which at least has a home page on sourceforge
   http://ppcboot.sourceforge.net/
Pretty much everything said for ppcboot goes for u-boot as well, only
better :-).

U-boot has some scripting capabilities as well as variables that can be
used for rudimentary decision making.  I'm sure Wolfgang will welcome
improvements to the scripting :-).

The scripts can be accessed remotely or via your application (assuming your
application's "failure" mode is sufficient to allow it to write to
flash/eeprom to change the script to a backup mode).  You don't tell us
_how_ the application fails and how it and/or the boot loader is expected
to recover from that failure, so I cannot give you and direct suggestions.

gvb


At 09:28 PM 6/3/2003 -0400, sam@ravnborg.org wrote:

>Hi all.
>
>We are developing an application for an embedded target that will be
>located in some rural areas. We foresee a need to update the target
>with a new version of the application or a new kernel from time to time.
>
>A few of the added requirements on top of that are:
>1) The target shall boot up in the old kernel, if a bogus kernel is loaded
>2) The target shall boot up in the old application, if a bogus
>         application is loaded
>3) On the management side an update shall be handled as a single file,
>         the target may decide to 'unpack' it when received.
>
>We have looked all over the net, but none of the bootloaders found
>so far could meet the above demands.
>The boot loaders usually have an interactive mode used to select
>between different configurations - and there are no feedback from
>the application side if the reboot actually went well.
>The interactiviness does not fit well with an embedded target.
>
>Do you know of a boot loader that partially or fully meets the above
>requirements?
>
>Background information
>Today we are using VxWorks - for which we have made our own boot loader.
>The boot laoder allows the boot loader itself, and the application part
>to be upgraded - and if a restart is failed the old version will
>be activated on a subsequent build.
>I want the same behaviour in the Linux based target.
>
>         Thanks in advance,
>
>         Sam
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Wolfgang Denk]

In message <5.1.0.14.2.20030603154752.034f0e78@falcon.si.com> you wrote:
>
> Check out U-Boot
>    http://sourceforge.net/projects/u-boot
> The project page on sourceforge is pretty much non-existant other than the
> CVS link.  Pull down the source -- there _is_ a good README file in it that
> explains a lot about u-boot.
>    http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/u-boot/u-boot/README?rev=1.30
> Wolfgang Denk of Denx Computer Systems runs the project.  He doesn't appear

Actually this is Wolfgang Denk of DENX Software Engineering :-)

> to have anything on U-Boot on his web site either.
>    http://www.denx.de/

Well, we do have some documentation about PPCBoot at

	http://www.denx.de/re/DPLG.html

which can be used directly for U-Boot, too.

> The predecessor was ppcboot, which at least has a home page on sourceforge
>    http://ppcboot.sourceforge.net/
> Pretty much everything said for ppcboot goes for u-boot as well, only
> better :-).

Right :-)

> U-boot has some scripting capabilities as well as variables that can be
> used for rudimentary decision making.  I'm sure Wolfgang will welcome
> improvements to the scripting :-).

The "some scripting capabilities" includes a more  or  less  complete
bourne   shell   (the   "hush"   shell  from  busybox).  Things  like
if/then/else/fi,   for/do/done,   while/do/done,   until/do/done   or
operators like && and || are available.

For a trivial example: to recover from corrupted images (bad checksum
etc.) you could use

	setenv bootcmd 'bootm $addr_primary || bootm $addr_secondary'

etc.


Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
"A fractal is by definition a set for which the Hausdorff Besicovitch
dimension strictly exceeds the topological dimension."
- Mandelbrot, _The Fractal Geometry of Nature_

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Failsafe bootloader

[Darin.Johnson]

We had a similar scheme for upgrades.  There was an "approved" image, and a "current" image.  If the current image wasn't working it would revert to the approved image automatically.  If the running image appeared to be ok, the operator could then mark it as the "approved" version.  In addition, there was a "factory installed" image that could be used if the operator messed the system up badly.

The drawback is that this takes more Flash space, which can be important when the goals turn from "make a product that works" into "save an extra $5 on the board cost".  Thus, for a later product we got rid of the factory image.

It also could be very nice if you could upgrade the lowest level of software, the boot loader or system initializer (ppcboot, etc).  We had found a bug in our loader at one point, and because many registers are "write once" it was difficult to fix these settings from the application software.  But finding a foolproof and reliable way to upgrade the boot loader software is difficult, especially if you are relying on customers and operators to do the upgrading.

Darin Johnson

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Magnus Damm]

On Tue, 3 Jun 2003 23:47:22 -0700
Darin.Johnson@nokia.com wrote:

> It also could be very nice if you could upgrade the lowest level of software, the boot loader or system initializer (ppcboot, etc).  We had found a bug in our loader at one point, and because many registers are "write once" it was difficult to fix these settings from the application software.  But finding a foolproof and reliable way to upgrade the boot loader software is difficult, especially if you are relying on customers and operators to do the upgrading.

And that's why you should use Miniboot as your first bootloader. =)
It's not upgradeable itself, but it is used to load and update a
secondary bootloader. Miniboot itself should be very small and simple
to keep the number of bugs low. It only requires the internal memory
of the mpc8xx so almost all hardware initialization is made
after miniboot has run, and therefore is upgradeable.

Need to upgrade your hardware configuration?
No problem as long as you haven't changed CS0 or your SMC... =)
And yes, it runs with watchdog on and leaves that configuration
up to the secondary bootloader.

http://opensource.se/projects/miniboot/index.html

However, the number of boards / processors supported are limited at the moment.
I've also heard that uboot/ppcboot could be used to update itself, but I've never
tested it myself.

/ magnus

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Jerry Van Baren]

At 12:16 PM 6/4/2003 -0400, damm@opensource.se wrote:

>On Tue, 3 Jun 2003 23:47:22 -0700
>Darin.Johnson@nokia.com wrote:

[snipped intro to miniboot]

>http://opensource.se/projects/miniboot/index.html
>
>However, the number of boards / processors supported are limited at the
>moment.
>I've also heard that uboot/ppcboot could be used to update itself, but I've
>never
>tested it myself.

[snip]

>/ magnus

Uploading a new bootrom (which u-boot can do) is quite easy.  The hard part
is protecting the hardware from the users. :-/  The critical part is the
erasing of the flash holding the bootrom image and programming the new
bootrom in WITHOUT LOSING POWER.  If an idiot user unplugs the board or a
truck hits a power pole at the instant when the old bootrom has been erased
and before the new bootrom has been programmed, you have a dead
board.  Bad.  Real bad.

You can minimize the window, but it is impossible to close it entirely
(using EEPROM, a power supply with a 10mSec hold up and a power fail
warning can close the window, but that isn't a common
configuration).  Depending on your flash, the window can be anywhere from
milliseconds to seconds (usually tens to hundreds of milliseconds).

Using a two-step boot process via miniboot as Magnus suggests allows you to
upgrade your "bootrom" without ever erasing your reset vector and lowest
level bootrom executable.  This _does_ prevent the above mentioned window
from ever opening, making it a safe(r) upgrade process.  In addition, if
you have two bootrom images, you will be able to upgrade the one with a
fallback if an idiot user unplugs your board.  You then upgrade the backup
once the primary has been programmed successfully.

Of course, this still doesn't protect you from loading a new bootrom that
has a valid checksum but a fatal problem (doh-doh-doh).  Fortunately, in
that case the idiot is ourselves and we deserve the pain involved, and
hopefully learn from it ;-).

gvb


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Failsafe bootloader

[Darin.Johnson]

> Of course, this still doesn't protect you from loading a new
> bootrom that
> has a valid checksum but a fatal problem (doh-doh-doh).
> Fortunately, in
> that case the idiot is ourselves and we deserve the pain involved, and
> hopefully learn from it ;-).

The problems we had were first that upgrades could be done
transparently without the user knowing, so you can't
really protect from the user unplugging (though we were
supposed to be an always-on-don't-turn-off product).
Second, have an image with a good checksum that boots
up ok is only part of the story, we have to make sure that
it can communicate correctly on its network.  Ie, the application
software has to communicate to the boot loader that the new
image can be kept.

Also from experience, I've really lost a lot of hair in
situations with other products that say "do not turn off the
power until this update is finished", only to have the product
crash, or the computer that is talking to it crashes.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Failsafe bootloader

[Jerry Van Baren]

At 05:33 AM 6/4/2003 -0400, Darin.Johnson@nokia.com wrote:

> > Of course, this still doesn't protect you from loading a new
> > bootrom that
> > has a valid checksum but a fatal problem (doh-doh-doh).
> > Fortunately, in
> > that case the idiot is ourselves and we deserve the pain involved, and
> > hopefully learn from it ;-).
>
>The problems we had were first that upgrades could be done
>transparently without the user knowing, so you can't
>really protect from the user unplugging (though we were
>supposed to be an always-on-don't-turn-off product).
>Second, have an image with a good checksum that boots
>up ok is only part of the story, we have to make sure that
>it can communicate correctly on its network.  Ie, the application
>software has to communicate to the boot loader that the new
>image can be kept.
>
>Also from experience, I've really lost a lot of hair in
>situations with other products that say "do not turn off the
>power until this update is finished", only to have the product
>crash, or the computer that is talking to it crashes.

Yes, you REALLY need enough storage to have two copies: the working copy
and the new copy.  The bootrom must NEVER erase the working copy until the
new copy is verified to be completely and correctly loaded (checksums are a
pretty weak validation, CRCs are strongly preferred).

If you are trying to upload a new image in-place, your window vulnerability
(mentioned in my previous email) goes from the amount of time that it takes
to erase and reprogram a block of flash to the time it takes to upload a
program: typically this means going from hundreds of milliseconds
(reasonable risk) to hundreds of SECONDS (extremely high risk).

gvb


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Failsafe bootloader

[Johnson, Stephen]

We implemented a verification feature to ppcboot for products that performs
a crc32 check before branching to an image. If the crc32 fails to match a
stored checksum, it will check a secondary image, if that one fails too then
we usually default to a tftp from a well known tftp server that we have
accessible. One could take this to as much extreme as desired, but that's up
to the system designers to deem what's reasonable and what's overkill. The
key is the primitives to do such are in place.

Obviously, we put some resources into this for our products. You have to
expect to add some value to your products.

ppcboot allows this to be quite easy since the bootcmd can contain several
commands.




-----Original Message-----
From: Sam Ravnborg [mailto:sam@ravnborg.org]
Sent: Tuesday, June 03, 2003 2:29 PM
To: linuxppc-embedded@lists.linuxppc.org
Subject: Failsafe bootloader



Hi all.

We are developing an application for an embedded target that will be
located in some rural areas. We foresee a need to update the target
with a new version of the application or a new kernel from time to time.

A few of the added requirements on top of that are:
1) The target shall boot up in the old kernel, if a bogus kernel is loaded
2) The target shall boot up in the old application, if a bogus
	application is loaded
3) On the management side an update shall be handled as a single file,
	the target may decide to 'unpack' it when received.

We have looked all over the net, but none of the bootloaders found
so far could meet the above demands.
The boot loaders usually have an interactive mode used to select
between different configurations - and there are no feedback from
the application side if the reboot actually went well.
The interactiviness does not fit well with an embedded target.

Do you know of a boot loader that partially or fully meets the above
requirements?

Background information
Today we are using VxWorks - for which we have made our own boot loader.
The boot laoder allows the boot loader itself, and the application part
to be upgraded - and if a restart is failed the old version will
be activated on a subsequent build.
I want the same behaviour in the Linux based target.

	Thanks in advance,

	Sam


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Failsafe bootloader

[Callebaut Benoit]

Hi,
We use small part of our flash to hold config data (last sector of the flash)
and which image must be started. PPCBoot read this and start the right image.
when upgrading , we change the kernel option to the failsafe kernel, do the
update of the second kernel (the working one) and after that we change the
kernel option again and we try to start the new one. We use also a kind of
watchdog feature in case of system lockout.

Benoit Callebaut
Barco NV
www.barco.com


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Wolfgang Denk]

In message <F9102D41F595D311ACA7009027DE2C840527B347@c3po.heurikon.com> you wrote:
>
> We implemented a verification feature to ppcboot for products that performs
> a crc32 check before branching to an image. If the crc32 fails to match a
> stored checksum, it will check a secondary image, if that one fails too then
> we usually default to a tftp from a well known tftp server that we have
> accessible. One could take this to as much extreme as desired, but that's up
> to the system designers to deem what's reasonable and what's overkill. The
> key is the primitives to do such are in place.

What exactly did you "implement" for this?

All this is already in place with the standard PPCBoot / U-Boot.  And
has been there right from the first versions.

> ppcboot allows this to be quite easy since the bootcmd can contain several
> commands.

Right, this allows for simple applications like you described above.

More complex things are possible by enabling the hush shell.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
News is what a chap who doesn't care much  about  anything  wants  to
read. And it's only news until he's read it. After that it's dead.
                           - Evelyn Waugh _Scoop_ (1938) bk. 1, ch. 5

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Failsafe bootloader

[Wolfgang Denk]

In message <3EE8E5EA@mailandnews.com> you wrote:
>
> We use small part of our flash to hold config data (last sector of the flash)
> and which image must be started. PPCBoot read this and start the right image.
> when upgrading , we change the kernel option to the failsafe kernel, do the
> update of the second kernel (the working one) and after that we change the
> kernel option again and we try to start the new one. We use also a kind of
> watchdog feature in case of system lockout.

It may be worth to point out that  U-Boot  supports  redundand  flash
sectors  for exactly such purposes, so you will always have a working
backup copy in case the update  of  your  config  data  (=environment
variables) sector fails because of powerdown etc.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
God made the integers; all else is the work of Man.       - Kronecker

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/