Solutions for Fast Software Upgrade in Linux/PPC

Solutions for Fast Software Upgrade in Linux/PPC

[Grant Erickson]

I am embarking on a project in which there exists a requirement to take my
embedded system (Walnut board w/ PowerPC 405GP w/ raft of PCI devices) and
allow it to perform a near-zero downtime upgrade/downgrade of
software/firmware.

Of the belief that there are very few new problems, just solutions that
aren't widely known, I have to imagine that the telecommunications carrier
equipment people have solved this problem long ago--albeit probably not
with Linux.

Does anyone know of any commercial or public solutions addressing this
problem in Linux or Linux on the PowerPC?

Anyway, there are a few solution spaces I can envision:

1. Check point all of your driver and application state, reboot, and
   hope that you can warm-start with your check pointed state quickly.

   - This means massaging the boot code, the Linux kernel, the device
     drivers, and the applications to tweak their start-up time.

   - At best, you may just quell kernel printks and disable auto-boot
     countdown in the PROM, at most, buying you a few seconds, if that. If
     it's a few seconds out of five, great. If it's a few out of twenty,
     you've got a lot of work to do--maybe you'll get there...maybe not.

2. Check point all of your driver and application state, and bring up a
   parallel set of upgraded/downgraded applications in standby mode and
   then fail over to them (w/ check-pointed state).

   - What if you're also upgrading the drivers? You've got a name space
     problem because you can have two versions of the same driver
     loaded--one for the existing and one for the new/old apps.

   - Fixing this means lots of kernel symbol hacking...ick.

   - Create a framework for your drivers that'd allow multiple
     versions. What about the Linux drivers that you'd rather not
     modify? No help there.

   - What if you are upgrading/downgrading the kernel? You're
     stuck...reboot and take the downtime hit.

3. Run a virtual machine in software. Check point all of your driver and
   application state. Bring-up the new kernel, drivers, and applications
   along side the existing ones and fail-over to them with the
   check-pointed state.

   - Poor performance for the common case of not upgrading/downgrading

   - Preventing console and Ethernet access from second virtual image
     until fail-over.

4. Run a pseudo virtual machine in hardware. Establish a second "bank" of
   DRAM, run the minimal, pseudo hardware virtual machine using the 405GPs
   OCM, memory controller base, and interrupt vector base. Then follow the
   same steps for (3).

   - Better performance over (3) possibly.

   - Is the 4KB OCM enough to run the VM? Maybe. Probably not though.

   - Very processor-specific solution. I think the 405GP is the only
     PowerPC with the OCM. Even if the 8xx, 8xxx, 7xxx, or 8xxx had it,
     it'd like be structured differently.

5. Use suspend/resume memory image.

   - With every software release, you also ship a memory image of that
     release in standby mode.

   - To upgrade/downgrade, you stream that image from flash
     (CompactFlash) into a new memory bank and then "jump" to it.

   - Creates huge resource demands for storing the image. Increases time
     and storage requirements as installed memory increases.

   - Won't likely work because "on-disk" layout (inode numbers, etc) are
     almost guaranteed to not match the system in the field.

   - Dramatically increases time to download new firmware/software.

Thoughts, comments, input would be greatly appreciated.

Regards,

Grant


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Mark Hatle]

Grant Erickson wrote:
>
> I am embarking on a project in which there exists a requirement to take my
> embedded system (Walnut board w/ PowerPC 405GP w/ raft of PCI devices) and
> allow it to perform a near-zero downtime upgrade/downgrade of
> software/firmware.
>
> Of the belief that there are very few new problems, just solutions that
> aren't widely known, I have to imagine that the telecommunications carrier
> equipment people have solved this problem long ago--albeit probably not
> with Linux.
>
> Does anyone know of any commercial or public solutions addressing this
> problem in Linux or Linux on the PowerPC?
>
> Anyway, there are a few solution spaces I can envision:
>
> 1. Check point all of your driver and application state, reboot, and
>    hope that you can warm-start with your check pointed state quickly.
>
>    - This means massaging the boot code, the Linux kernel, the device
>      drivers, and the applications to tweak their start-up time.
>
>    - At best, you may just quell kernel printks and disable auto-boot
>      countdown in the PROM, at most, buying you a few seconds, if that. If
>      it's a few seconds out of five, great. If it's a few out of twenty,
>      you've got a lot of work to do--maybe you'll get there...maybe not.

One note.. this has been discussed a lot on the linux kernel mailing
list (well a lot in the past at least.)  If all you will be doing is
maintaining the SAME kernel version till the end of time, and not
changing "critical" datastructures this "upgrade in place" can work.
However, even when moving minor versions of the kernel internal data
structures can change you you run into massive problems...  I would
suggest you scan the linux kernel mailing list (in addition to whatever
someone on this list suggests of course.)

--Mark

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Jim Thompson]

Doesn't the 'linux bios' project claim that they boot linux (as the
bios) so quickly that the IDE drives haven't come on-line yet, so they
just *reboot* in order to give enough delay to get the drives online?

I'd concentrate on taking the delay() calls to their minimum as a
first step, then see just how bad a simple 'reboot' could be.

--
C++ is like jamming a helicopter inside a Miata and expecting some
sort of improvement.       -- Drew Olbrich

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Peter Desnoyers]

I would give some serious thought to the fast reboot idea.

[a related anecdote - a former colleague who worked for Caymam back in
the days when people bought Appletalk routers complained how they were
perceived as much less reliable than the Shiva product, even though the
Shiva crashed much more frequently.  It turns out that the Shiva
rebooted faster than the AppleTalk timeout, while the Cayman box
didn't...]

You can probably separate it into two halves, kernel and application.  I
can boot a 50MHz MBX860 up to /sbin/init in 6 seconds with an NFS root,
so I think 1 or 2 second kernel boot is quite feasible on a faster
machine with data in flash or (already spun up) local disk.  (PCI
scanning will take time, but you should be able to trim it significantly
with knowledge of your hardware)

Application startup depends.  You might investigate having all your
applications use a "checkpoint" or "pre-digested" format for their
configuration files, so that they can load config quickly at boot, and
save configuration before reboot.  This includes whatever application is
going to configure the kernel after boot, which I assume won't be
init+init.d/network+ifconfig etc.

--
.....................................................................
 Peter Desnoyers            (781) 457-1165   pdesnoyers@chinook.com
 Chinook Communications     (617) 661-1979   pjd@fred.cambridge.ma.us
 100 Hayden Ave, Lexington MA 02421

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Josh Huber]

Grant Erickson <erick205@umn.edu> writes:

[snip]

I'd also recommend the fast-reboot option, but since no one mentioned
yet I'll suggest Werner Almsberger's bootimg utility.

Take a look at:
<URL:ftp://icaftp.epfl.ch/pub/people/almesber/misc/>

specifically, bootimg-current.tar.gz

Also, in ../booting/ there is a paper describing the
method. (bootinglinux-current.ps.gz)

I've tested this many times (we use it at Mission Critical for our
kernel crash dump system), and if you're looking for fast reboots this
is the way to go.

Unfortunately, it doesn't solve the upgrading of the ROM issue.

For the lazy who don't want to look at the references, what this
allows you to do is boot a linux kernel, using an already loaded Linux
kernel.

ttyl,
--
Josh Huber

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Josh Huber]

Grant Erickson <erick205@umn.edu> writes:

[snip]

oh, and I forgot to mention one thing about bootimg.  Unfortunately it
only supports x86, but the pieces you'd need to rewrite are small.

ttyl,
--
Josh Huber

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Solutions for Fast Software Upgrade in Linux/PPC

[Magnus Damm]

I recommend a combination of a simple bootloader and
a reboot-linux-in-linux piece of software.

You can find miniboot - a safe and simple bootloader
at http://opensource.se/projects/
With a clever-designed upgrade application it's possible
to update the flash in a fail-safe way.

One neat feature is that miniboot runs without any external RAM,
which makes it possible to keep the RAM-code in a upgradeable
image.

I also have a driver, relf, (Reboot ELF) that makes it
possible to reboot an ELF directly from Linux.
It has booted Linux, vxWorks and pSOS. mpc8xx only.
If you're intrested - give me a mail and I will send it to you.

Good luck /

Magnus

Josh Huber wrote:
>
> Grant Erickson <erick205@umn.edu> writes:
>
> [snip]
>
> I'd also recommend the fast-reboot option, but since no one mentioned
> yet I'll suggest Werner Almsberger's bootimg utility.
>
> Take a look at:
> <URL:ftp://icaftp.epfl.ch/pub/people/almesber/misc/>
>
> specifically, bootimg-current.tar.gz
>
> Also, in ../booting/ there is a paper describing the
> method. (bootinglinux-current.ps.gz)
>
> I've tested this many times (we use it at Mission Critical for our
> kernel crash dump system), and if you're looking for fast reboots this
> is the way to go.
>
> Unfortunately, it doesn't solve the upgrading of the ROM issue.
>
> For the lazy who don't want to look at the references, what this
> allows you to do is boot a linux kernel, using an already loaded Linux
> kernel.
>
> ttyl,
> --
> Josh Huber
>

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/