Rootfs choice ideas

Rootfs choice ideas

[Michael]

Hi.  I was wondering if you could comment on some of my ideas
regarding a root filesystem choice.  I did some simple searches on
google, but didn't find what I was looking for.  I thought this
list was the most appropriate to post to.

I have an embedded system, using only NAND flash.  This stores my
bootloader, kernel, and root filesystem.  The bootloader and kernel
are protected by using the same block ECC mechanisms that JFFS2
uses. I write them (mtd0 and mtd1) with nandwrite -j, and read them
(correcting bits using the ECC as necessary) in my bootloader.  I
could further protect them (in case of powerfail during updating)
by having another copy (either mirrored or failsafe) of each, using
a CRC or similar to check the validity of the entire image.

Now, I'm unclear as to the best way to protect the root filesystem
and its files against accidental corruption; Either by flash errors
or by a logical error (e.g. in the update procedure).

Currently, we're using JFFS2 as the root filesystem, in read-write
mode.  It's nice, as it provides power-fail safety, wear levelling
and such.  And we can modify files freely.

However, there seem to be some disadvantages:
 1. Instead of versioning a fixed filesystem image (which you could
do  with cramfs or initrd), you now manage a collection of files
each of which could change independently.
 2. It would be hard or impractical to check the entire image for
consistency, as you could with a cramfs image.
 3. I could perhaps run into problems when updating it, for
instance if the update fails part-way through.
 4. There seems to be some overhead in scanning the rootfs at boot
time (yeah, there are recent changes that speed this up), and
garbage collection running in the background. (which is a seperate
issue)

So, it's acting very much like a desktop (which can possibly
develope cruft), and very little like an embedded device (which
typically can't change).

Does anyone share some of these same concerns (or perhaps others as
well)?  What schemes have you used or seen, and how do they
compare?

These are some ideas I've been throwing around:

1. Use a non-changing cramfs or initrd root filesystem, mounting
/var and /tmp in ram, using tmpfs.
Then I can verify and version this rootfs as a whole, perhaps
storing multiple copies.

2. Use JFFS2 only for logs, application data and config (and
perhaps the application itself).

3. Allow updating when booting (the static rootfs in idea #1 could
check for the existance of an update file (I'm going to use a USB
flash drive), and use that to update the system while it's safe to
overwrite anything and everything.

4. If any base OS files need to change (modules, /etc/*, ...), they
can be stored on the JFFS2 rootfs and overwrite the base ones.
(kind of like Knoppix or DSL do)


How do these ideas sound?
Am I too paranoid?  going about this the wrong way?  misinformed?

Any feedback would be great.
Thanks in advance.
Mike

Re: Rootfs choice ideas

[David Woodhouse]

On Fri, 2004-12-17 at 15:16 -0800, Michael wrote:
> And we can modify files freely.

Yes but you don't _have_ to modify files freely. You can exercise a
little restraint :)

> However, there seem to be some disadvantages:
>  1. Instead of versioning a fixed filesystem image (which you could
> do  with cramfs or initrd), you now manage a collection of files
> each of which could change independently.

Yes. This is usually done in packages -- by RPM or more usefully on an
embedded system by something smaller like ipkg. Take a look at the
Familiar distribution.

>  2. It would be hard or impractical to check the entire image for
> consistency, as you could with a cramfs image.

'rpm -Vva'

Not sure if ipkg stores checksums of the installed files.

>  3. I could perhaps run into problems when updating it, for
> instance if the update fails part-way through.

You reattempt the update after you come back up.

>  4. There seems to be some overhead in scanning the rootfs at boot
> time (yeah, there are recent changes that speed this up), and
> garbage collection running in the background. (which is a seperate
> issue)

These are true. We're trying to improve it.

> So, it's acting very much like a desktop (which can possibly
> develope cruft), and very little like an embedded device (which
> typically can't change).
> 
> Does anyone share some of these same concerns (or perhaps others as
> well)?  What schemes have you used or seen, and how do they
> compare?

Basically it's up to you -- if you can manage to live with a read-only
file system, as presumably you could since you seem to consider
writeability a disadvantage, then you might as well stick with cramfs.

-- 
dwmw2

RE: Rootfs choice ideas

[Michael]

Hi.  Thanks for your feedback, it is appreciated.

--- Fillod Stephane <stephane.fillod@thomson.net> wrote:

> One basic idea is to have partitioned flashed, for example, 
> one partition for kernel, one for "base" files (libc,
> busybox,..),
> and a couple for the application. The problem with this approach
> is that space is wasted because partitions tend to be static.
> Note: bootloader will always have to be in a separate partition.

Yes, I agree.  However, space is _definitely not an issue for me. 
The bootloader is less than 16KiB, the kernel is around 800KiB, and
the rootfs currently has 8MiB allocated. (and is using half that)

The smallest flash chip we can get is 32MiB, but we'll likely be
going with a 64MiB.

> A solution to this problem is to have cramfs images on top
> of loop devices, ie. files stored in a jffs2 big partition. 
> Ok, it may waste some RAM, but the page cache do it nice 
> with this stack.

> The /tmp and /var directories are tmpfs.
> I would not recomand using initrd for rootfs as it wastes 
> memory on non-used files.

Yes, we've been using tmpfs for /tmp and /var for a while now.  And
I think I could overcome much of that shortcoming by using symbolic
links to files on a cramfs.

> Instead, I have a patch witch let 
> Linux take its rootfs from an image (ex: cramfs,..) stored 
> in a file of a filesystem (here jffs2). 
> I've called this "init loopback", or initlo for short.
> initlo is setup from the Linux command line. Once rootfs
> is mounted, supplemental images can be mounted for application,
> etc.

So instead of mtd partitioning, you're using jffs2 to dynamically
partition the space (i.e. store files).  That makes sense.

The downside of this is that I want to be able to verify and update
separate from the jffs2 partition.  I suppose I don't necessarily
need to, but I thought it would be nice to do all this verification
in the bootloader.

A couple potential problems for me, though:  I don't think I want
to read a jffs2 filesystem in my bootloader.  It may be simpler
than I imagine, but the way I'm thinking of doing it is about the
most straightforward you can get.  As well, I was thinking of
decreasing the boot time by minimizing the partition scanning that
jffs2 does on mounting.

> Corollary: you have to reboot to "change" file of rootfs image,
> but as long as care is taken to choose what goes in "base",
> it's not a problem. Besides, soft links to /var files get away 
> with files in need of updating.

That's a good point; I'll run into this as well.  If I want to
modify the cramfs image, then I'll need to stop using it somehow.
(perhaps by pivot_root'ing to a ram based FS)

> 
> The CRC of cramfs images can be verified. Monitors like U-Boot
> are able to read and verify CRC of kernel image from a jffs2 fs. 
> Any addition of a firmware part (=file copy) is safe thanks to
> jffs2.
> No partition means any number (space permitting) of version of 
> firmwares, ex: one per directory, + a file or a link to tell the 
> version to use.

> Configuration files are stored in the jffs2 fs.
> Most of the time, the jffs2 fs is mounted RO. It is remounted RW
> only when adding a firmware or changing a config file.

I also want some persistent storage, to save configuration and
such.  I was thinking of using it mounted read/write the whole
time.

> All this brings unbreakability and peace of mind :-)

Yes, this was my exact motivation.  That, and simple, unbreakable
updating.

> IMHO, I don't think storing logs in flash is a good idea. For
> designs 
> having a NVRAM, it's better to store logs there, without JFFS2,
> just a simple mapped circular buffer like in busybox syslog.
> I haven't found an off-the-shelf solution yet, but something like
> a flight-recorder would be great here.

Is that because of the potential wearing?  Or the speed?  We like
this because we only need one memory device for our board (reduced
cost, which is important), and we have lots of spare space in NAND.

> REM: jffs2 can be changed in favor of yaffs. 
>      cramfs can be changed in favor of squashfs.

Do you have any experience with yaffs?  I've heard good things
about it, but from my impressions, I think jffs2 either better or
more suitable for me.
I recently read a short report about SquashFS.1.  It looked good to
me, especially with some of the reduced overhead.  I would consider
this, but I don't need the extra space it provides, and I like the
fact that cramfs is in the kernel and is probably simpler. (less
can go wrong)

> Comments are welcome on these ideas.

Ditto.  And, well, you got them.  =]

Thanks,
Mike

Re: Rootfs choice ideas

[xemc]

> > And we can modify files freely.
> 
> Yes but you don't _have_ to modify files freely. You can exercise a
> little restraint :)

Well, perhaps you don't know me too well.  =]  But you have a good point.

> Yes. This is usually done in packages -- by RPM or more usefully on an
> embedded system by something smaller like ipkg. Take a look at the
> Familiar distribution.

Thanks, I will do that.  I guess I just haven't thought along those lines.

> 'rpm -Vva'
> 
> Not sure if ipkg stores checksums of the installed files.

Aha.  Well, I'd have to say I've never used rpm before.  I usually use
Slackware, just starting with Debian (laptop) and Gentoo (x86-64
desktop)
 
> >  3. I could perhaps run into problems when updating it, for
> > instance if the update fails part-way through.
> 
> You reattempt the update after you come back up.

If I can come back up with a partially failed update.  But yes, the
same thing can be done that I was proposing, but using packages on a
JFFS2 filesystem as well.

> Basically it's up to you -- if you can manage to live with a read-only
> file system, as presumably you could since you seem to consider
> writeability a disadvantage, then you might as well stick with cramfs.

Well, I suppose I considered it a disadvantage when I don't need it. 
My idea was to partition read-only areas (maybe CRAMFS) separately
from a read-write area (JFFS2).  But there seem to be multiple ways of
doing this.

Thanks for your feedback.  It is appreciated, and I'll mull over it for a bit.

Mike