Question on the fs choice

Question on the fs choice

[Zajac Adam-AAZ004]

Hello,

We've built a custom board based on MPC8540 that runs Linux OS and utilizes
compact flash card to store the kernel on a raw-data partition (read-only)
and root file system on an Ext2 partition (read-write). As Ext2 is not a
journaling FS, in case of any power failure the file system is not cleanly
unmounted and the fscheck complains upon a system start-up.
Also, sometimes the file system partition gets corrupted when we power-cycle
the card instead of shutting the system down gracefully (the card is build
to be "hot-swappable" so we test it on purpose).

I saw Wolfgang Denx's post discouraging any use of the CF cards for an
embedded platform where write operations are performed under power-failure
prone environment. Unfortunately, this is the reality we're facing on our
card.

I'm seeking some help with selecting the file system that would survive
power failure if the CF card didn't get damaged during a write cycle (we're
gonna perform a thorough testing to assess the severity of that issue).

I've seen some articles suggesting one of the journaling systems, like Ext3,
XFS or ReiserFS.

Has anybody successfully implemented any of these FS systems on an embedded
platform equipped with a compact flash card working as a main storage
medium?

Any help would be greatly appreciated.

Adam Zajac
Motorola Inc, Global Telecom Solutions Sector
5555 N Beach St, Mailstop 8E, Fort Worth, TX 76137-2794
Phone: (817) 245-7963


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Question on the fs choice

[David Gardiner]

Sorry for the late post thought I'd just add my experiences which seem
to reflect the general discussion, and hopefully gain some more

>Has anybody successfully implemented any of these FS systems on an embedded
>platform equipped with a compact flash card working as a main storage
>medium?
>
>
Yes, not successfully yet though,

platform: mvme5100

ide controller: promise pmc
ide 0000:03:03.0 Unknown mass storage controller: Promise Technology,
Inc. 20269 (rev 02) (prog-if 85)
        Subsystem: Promise Technology, Inc. Ultra133TX2
        Flags: bus master, 66MHz, slow devsel, latency 128, IRQ 28
        I/O ports at 774fb8 [size=8]
        I/O ports at 774fb4 [size=4]
        I/O ports at 774fa8 [size=8]
        I/O ports at 774fa4 [size=4]
        I/O ports at 774f90 [size=16]
        Memory at f36fc000 (32-bit, non-prefetchable) [size=16K]
        Expansion ROM at <unassigned> [disabled] [size=16K]
        Capabilities: <available only to root>

compact flash: SanDisk SDCFB-512

dmesg:
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
PDC20269: IDE controller at PCI slot 03:03.0
PDC20269: chipset revision 2
PDC20269: not 100% native mode: will probe irqs later
    ide2: BM-DMA at 0x774f90-0x774f97, BIOS settings: hde:pio, hdf:pio
    ide3: BM-DMA at 0x774f98-0x774f9f, BIOS settings: hdg:pio, hdh:pio
Probing IDE interface ide2...
hde: SanDisk SDCFB-512, CFA DISK drive
blk: queue c01d7d70, I/O limit 4095Mb (mask 0xffffffff)
Probing IDE interface ide3...
ide2 at 0x774fb8-0x774fbf,0x774fb6 on irq 28
hde: attached ide-disk driver.
hde: task_no_data_intr: status=0x51 { DriveReady SeekComplete Error }
hde: task_no_data_intr: error=0x04 { DriveStatusError }
hde: 1000944 sectors (512 MB) w/1KiB Cache, CHS=993/16/63, BUG
Partition check:
 /dev/ide/host2/bus0/target0/lun0:hde: dma_intr: status=0x51 {
DriveReady SeekComplete Error }
hde: dma_intr: error=0x04 { DriveStatusError }
hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hde: dma_intr: error=0x04 { DriveStatusError }
hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hde: dma_intr: error=0x04 { DriveStatusError }
hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
hde: dma_intr: error=0x04 { DriveStatusError }
hde: DMA disabled
PDC202XX: Primary channel reset.
ide2: reset: success
 p1 p2


The Compact Flash works fine as long (if you can ignore the errors, like
above) as it's mounted read only through power failures, resets ....,
but if it's mounted rw a corrupted disk (a repartition "usually" fixes
this) can be the result after resets, power failures .....
I haven't had much time to spend on testing this but may have in the
near future, any suggestions on how I can get rid of the "hde:" errors
would be greatly appreciated, as it's probably obvious to some, but not
me ....

Ta,
dg

ps: Adam hope this helps


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Question on the fs choice

[Zajac Adam-AAZ004]

Hello,

We've built a custom board based on MPC8540 that runs Linux OS and utilizes
compact flash card to store the kernel on a raw-data partition (read-only)
and root file system on an Ext2 partition (read-write). As Ext2 is not a
journaling FS, in case of any power failure the file system is not cleanly
unmounted and the fscheck complains upon a system start-up.
Also, sometimes the file system partition gets corrupted when we power-cycle
the card instead of shutting the system down gracefully (the card is build
to be "hot-swappable" so we test it on purpose).

I saw Wolfgang Denx's post discouraging any use of the CF cards for an
embedded platform where write operations are performed under power-failure
prone environment. Unfortunately, this is the reality we're facing on our
card.

I'm seeking some help with selecting the file system that would survive
power failure if the CF card didn't get damaged during a write cycle (we're
gonna perform a thorough testing to assess the severity of that issue).

I've seen some articles suggesting one of the journaling systems, like Ext3,
XFS or ReiserFS.

Has anybody successfully implemented any of these FS systems on an embedded
platform equipped with a compact flash card working as a main storage
medium?

Any help would be greatly appreciated.

Adam Zajac
Motorola Inc, Global Telecom Solutions Sector
5555 N Beach St, Mailstop 8E, Fort Worth, TX 76137-2794
Phone: (817) 245-7963


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[VanBaren, Gerald (AGRE)]

Journaling file systems don't cure corruption on power fail, they just make the power fail recovery faster by eliminating the full fsck disk scan and significantly reduce the probability of corruption due to the transactional nature.  The assumption with a journaling file system is that the power fail only corrupted the last (partial) transaction.  With normal usage patterns this is a good assumption (spinning disk physics probably help, but write caches in disks are going to hurt).  With CF under heavy power failure cycles, this might not be as good an assumption.

There was a slashdot item yesterday pointing to a benchmark of various journaling file systems.  In the discussion, one point that was made was that ext3 journals the data as well as the metadata by default where the other journaling file systems journal only the metadata (directory info).  I presume this was accurate (hey, it was on ./ which guarantees accuracy, right? ;-).  Whichever fs you pick, you probably want to journal the data as well as the metadata.

A suggestion is to have multiple partitions on your CF and use RAID-3 or RAID-5.  The assumption here is that a power fail-induced physical corruption would corrupt only one of the partitions which could then be rebuilt due to the RAID redundancy.  The journaled file system would then eliminate the fsck on start up and make the file system itself consistant and the RAID redundancy would (presumably) fix any low level errors that the unexpected power fail could induce.

...or maybe a journaling file system is good enough.

gvb


> -----Original Message-----
> From: owner-linuxppc-embedded@lists.linuxppc.org
> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of Zajac
> Adam-AAZ004
> Sent: Tuesday, May 11, 2004 7:20 PM
> To: 'linuxppc-embedded@lists.linuxppc.org'
> Subject: Question on the fs choice
>
>
>
> Hello,
>
> We've built a custom board based on MPC8540 that runs Linux
> OS and utilizes
> compact flash card to store the kernel on a raw-data
> partition (read-only)
> and root file system on an Ext2 partition (read-write). As
> Ext2 is not a
> journaling FS, in case of any power failure the file system
> is not cleanly
> unmounted and the fscheck complains upon a system start-up.
> Also, sometimes the file system partition gets corrupted when
> we power-cycle
> the card instead of shutting the system down gracefully (the
> card is build
> to be "hot-swappable" so we test it on purpose).
>
> I saw Wolfgang Denx's post discouraging any use of the CF cards for an
> embedded platform where write operations are performed under
> power-failure
> prone environment. Unfortunately, this is the reality we're
> facing on our
> card.
>
> I'm seeking some help with selecting the file system that
> would survive
> power failure if the CF card didn't get damaged during a
> write cycle (we're
> gonna perform a thorough testing to assess the severity of
> that issue).
>
> I've seen some articles suggesting one of the journaling
> systems, like Ext3,
> XFS or ReiserFS.
>
> Has anybody successfully implemented any of these FS systems
> on an embedded
> platform equipped with a compact flash card working as a main storage
> medium?
>
> Any help would be greatly appreciated.
>
> Adam Zajac
> Motorola Inc, Global Telecom Solutions Sector
> 5555 N Beach St, Mailstop 8E, Fort Worth, TX 76137-2794
> Phone: (817) 245-7963
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[VanBaren, Gerald (AGRE)]

Sorry for following up my own post, but this is a very interesting talk on EXT3 which has application to all journaling file systems...
  http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html

A very interesting quote:
  Now, disks these days actually make these guarantees. If you start a write
  operation to a disk, then even if the power fails in the middle of that
  sector write, the disk has enough power available, and it can actually steal
  power from the rotational energy of the spindle; it has enough power to
  complete the write of the sector that's being written right now. In all
  cases, the disks make that guarantee. [23m, 41s]

This is Wolfgang's and my point about CF: the CF itself does not have sufficient energy storage inside it to guarantee completion of a write operation.  In addition, our local experience with flash chips is that a power fail during a write operation can scribble on RANDOM locations in the flash chip.  The intuitive assumption is that a power fail during a CF write will corrupt ONLY the sector that is the target of the write operation.  This is NOT a good assumption.

gvb

> -----Original Message-----
> From: owner-linuxppc-embedded@lists.linuxppc.org
> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of
> VanBaren, Gerald (AGRE)
> Sent: Wednesday, May 12, 2004 8:37 AM
> To: linuxppc-embedded@lists.linuxppc.org
> Subject: RE: Question on the fs choice
>
>
>
> Journaling file systems don't cure corruption on power fail,
> they just make the power fail recovery faster by eliminating
> the full fsck disk scan and significantly reduce the
> probability of corruption due to the transactional nature.
> The assumption with a journaling file system is that the
> power fail only corrupted the last (partial) transaction.
> With normal usage patterns this is a good assumption
> (spinning disk physics probably help, but write caches in
> disks are going to hurt).  With CF under heavy power failure
> cycles, this might not be as good an assumption.
>
> There was a slashdot item yesterday pointing to a benchmark
> of various journaling file systems.  In the discussion, one
> point that was made was that ext3 journals the data as well
> as the metadata by default where the other journaling file
> systems journal only the metadata (directory info).  I
> presume this was accurate (hey, it was on ./ which guarantees
> accuracy, right? ;-).  Whichever fs you pick, you probably
> want to journal the data as well as the metadata.
>
> A suggestion is to have multiple partitions on your CF and
> use RAID-3 or RAID-5.  The assumption here is that a power
> fail-induced physical corruption would corrupt only one of
> the partitions which could then be rebuilt due to the RAID
> redundancy.  The journaled file system would then eliminate
> the fsck on start up and make the file system itself
> consistant and the RAID redundancy would (presumably) fix any
> low level errors that the unexpected power fail could induce.
>
> ...or maybe a journaling file system is good enough.
>
> gvb
>
>
> > -----Original Message-----
> > From: owner-linuxppc-embedded@lists.linuxppc.org
> > [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On
> Behalf Of Zajac
> > Adam-AAZ004
> > Sent: Tuesday, May 11, 2004 7:20 PM
> > To: 'linuxppc-embedded@lists.linuxppc.org'
> > Subject: Question on the fs choice
> >
> >
> >
> > Hello,
> >
> > We've built a custom board based on MPC8540 that runs Linux
> > OS and utilizes
> > compact flash card to store the kernel on a raw-data
> > partition (read-only)
> > and root file system on an Ext2 partition (read-write). As
> > Ext2 is not a
> > journaling FS, in case of any power failure the file system
> > is not cleanly
> > unmounted and the fscheck complains upon a system start-up.
> > Also, sometimes the file system partition gets corrupted when
> > we power-cycle
> > the card instead of shutting the system down gracefully (the
> > card is build
> > to be "hot-swappable" so we test it on purpose).
> >
> > I saw Wolfgang Denx's post discouraging any use of the CF
> cards for an
> > embedded platform where write operations are performed under
> > power-failure
> > prone environment. Unfortunately, this is the reality we're
> > facing on our
> > card.
> >
> > I'm seeking some help with selecting the file system that
> > would survive
> > power failure if the CF card didn't get damaged during a
> > write cycle (we're
> > gonna perform a thorough testing to assess the severity of
> > that issue).
> >
> > I've seen some articles suggesting one of the journaling
> > systems, like Ext3,
> > XFS or ReiserFS.
> >
> > Has anybody successfully implemented any of these FS systems
> > on an embedded
> > platform equipped with a compact flash card working as a
> main storage
> > medium?
> >
> > Any help would be greatly appreciated.
> >
> > Adam Zajac
> > Motorola Inc, Global Telecom Solutions Sector
> > 5555 N Beach St, Mailstop 8E, Fort Worth, TX 76137-2794
> > Phone: (817) 245-7963
> >
> >
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[Zajac Adam-AAZ004]

Gerald, Wolfgang,

Thank you for your prompt response to my post.
I had been aware of the CF card corruption problem that exists for some
cards under power failure scenario prior to posting my questions but was
seeking some advice from experts on the subject.
Since building embedded systems equipped with a compact flash card as a main
storage medium is an interesting issue per se, I think it's at least
worthwhile to exchange some experience with the users of that group.

I've contacted SanDisk regarding their warranty of surviving a power
shortage in the middle of a write-cycle and received a document explaining
the expected behavior.
Here's an introductory quote:

"If power is removed while the flash memory cells are being programmed, it
could leave
them in a partially programmed state. The partially programmed state will be
very
different from the intended final state. In that type of situation, the
write operation will
result in a corrupt sector. This condition is referred to as a write abort
situation.1 It is
critical for the flash card to consistently report write-abort situations to
the host so that
the host can take appropriate action.
Write-Abort Situations
Three situations can arise when using write abort:
− Case 1 Write abort results in a correctable ECC error.
− Case 2 Write abort results in an uncorrectable ECC error.
− Case 3 Write abort results in an undetected ECC.
In the undetected ECC case, if a large number of bits are in the wrong
state, it may result
in a situation where the ECC circuitry fails to detect the error, or it may
detect an error
and attempt to correct it but instead result in the wrong data after
correction."

We've been using two types of SanDisk cards for our systems:
SDCFB-64-201-80  - 64MB
SDCFB-512-201-00 - 512MB

These particular SanDisk cards are equipped with an enhanced version of the
internal controller that is capable of surviving such scenario leaving the
card in a robust state.
At worst, the user loses the data that was being written to a particular
sector while the power failure happened.
Case 1: the data errors are corrected
Case 2: the old data is restored back to the sector
Case 3: the user loses the data on the sector

One of the recommendations was to implement a 100 millisecond power back-up
provided for the card to finish its internal write cycle. Of course, this
can only be applied  to the systems where the user cannot yank the card out
directly. For example, digital cameras have a sensor on the card's cover to
signal such event.

To cut the long story short I personally believe (based on the number of
stress tests we perform on our target platforms) that modern compact flash
cards (e.g. SDCFB-xx-201 series) can be safely used as a main storage medium
for commercial applications.

Resolving a file system corruption upon an unexpected power failure should
just be the matter of selecting one of the journaling systems (ext3, XFS,
ReiserFS etc.). BTW, we're still evaluating which FS to choose and by far
Ext3 looks like the best candidate due to its backward compatibility with
Ext2.

Coming back to the CF card discussion subject, does anybody know how the
"power failure" scenario is resolved at such commercial platforms as pocket
pc's and digital cameras utilizing CF cards?

Regards,
Adam


-----Original Message-----
From: owner-linuxppc-embedded@lists.linuxppc.org
[mailto:owner-linuxppc-embedded@lists.linuxppc.org] On Behalf Of VanBaren,
Gerald (AGRE)
Sent: Wednesday, May 12, 2004 8:14 AM
To: linuxppc-embedded@lists.linuxppc.org
Subject: RE: Question on the fs choice



Sorry for following up my own post, but this is a very interesting talk on
EXT3 which has application to all journaling file systems...
  http://olstrans.sourceforge.net/release/OLS2000-ext3/OLS2000-ext3.html

A very interesting quote:
  Now, disks these days actually make these guarantees. If you start a write
  operation to a disk, then even if the power fails in the middle of that
  sector write, the disk has enough power available, and it can actually
steal
  power from the rotational energy of the spindle; it has enough power to
  complete the write of the sector that's being written right now. In all
  cases, the disks make that guarantee. [23m, 41s]

This is Wolfgang's and my point about CF: the CF itself does not have
sufficient energy storage inside it to guarantee completion of a write
operation.  In addition, our local experience with flash chips is that a
power fail during a write operation can scribble on RANDOM locations in the
flash chip.  The intuitive assumption is that a power fail during a CF write
will corrupt ONLY the sector that is the target of the write operation.
This is NOT a good assumption.

gvb

> -----Original Message-----
> From: owner-linuxppc-embedded@lists.linuxppc.org
> [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of
> VanBaren, Gerald (AGRE)
> Sent: Wednesday, May 12, 2004 8:37 AM
> To: linuxppc-embedded@lists.linuxppc.org
> Subject: RE: Question on the fs choice
>
>
>
> Journaling file systems don't cure corruption on power fail,
> they just make the power fail recovery faster by eliminating
> the full fsck disk scan and significantly reduce the
> probability of corruption due to the transactional nature.
> The assumption with a journaling file system is that the
> power fail only corrupted the last (partial) transaction.
> With normal usage patterns this is a good assumption
> (spinning disk physics probably help, but write caches in
> disks are going to hurt).  With CF under heavy power failure
> cycles, this might not be as good an assumption.
>
> There was a slashdot item yesterday pointing to a benchmark
> of various journaling file systems.  In the discussion, one
> point that was made was that ext3 journals the data as well
> as the metadata by default where the other journaling file
> systems journal only the metadata (directory info).  I
> presume this was accurate (hey, it was on ./ which guarantees
> accuracy, right? ;-).  Whichever fs you pick, you probably
> want to journal the data as well as the metadata.
>
> A suggestion is to have multiple partitions on your CF and
> use RAID-3 or RAID-5.  The assumption here is that a power
> fail-induced physical corruption would corrupt only one of
> the partitions which could then be rebuilt due to the RAID
> redundancy.  The journaled file system would then eliminate
> the fsck on start up and make the file system itself
> consistant and the RAID redundancy would (presumably) fix any
> low level errors that the unexpected power fail could induce.
>
> ...or maybe a journaling file system is good enough.
>
> gvb
>
>
> > -----Original Message-----
> > From: owner-linuxppc-embedded@lists.linuxppc.org
> > [mailto:owner-linuxppc-embedded@lists.linuxppc.org]On
> Behalf Of Zajac
> > Adam-AAZ004
> > Sent: Tuesday, May 11, 2004 7:20 PM
> > To: 'linuxppc-embedded@lists.linuxppc.org'
> > Subject: Question on the fs choice
> >
> >
> >
> > Hello,
> >
> > We've built a custom board based on MPC8540 that runs Linux
> > OS and utilizes
> > compact flash card to store the kernel on a raw-data
> > partition (read-only)
> > and root file system on an Ext2 partition (read-write). As
> > Ext2 is not a
> > journaling FS, in case of any power failure the file system
> > is not cleanly
> > unmounted and the fscheck complains upon a system start-up.
> > Also, sometimes the file system partition gets corrupted when
> > we power-cycle
> > the card instead of shutting the system down gracefully (the
> > card is build
> > to be "hot-swappable" so we test it on purpose).
> >
> > I saw Wolfgang Denx's post discouraging any use of the CF
> cards for an
> > embedded platform where write operations are performed under
> > power-failure
> > prone environment. Unfortunately, this is the reality we're
> > facing on our
> > card.
> >
> > I'm seeking some help with selecting the file system that
> > would survive
> > power failure if the CF card didn't get damaged during a
> > write cycle (we're
> > gonna perform a thorough testing to assess the severity of
> > that issue).
> >
> > I've seen some articles suggesting one of the journaling
> > systems, like Ext3,
> > XFS or ReiserFS.
> >
> > Has anybody successfully implemented any of these FS systems
> > on an embedded
> > platform equipped with a compact flash card working as a
> main storage
> > medium?
> >
> > Any help would be greatly appreciated.
> >
> > Adam Zajac
> > Motorola Inc, Global Telecom Solutions Sector
> > 5555 N Beach St, Mailstop 8E, Fort Worth, TX 76137-2794
> > Phone: (817) 245-7963
> >
> >
>
>


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: Question on the fs choice

[Wolfgang Denk]

Dear Adam,

in message <EFB813091B18D511BD3600508B644F820A617EC3@tx14exm06.ftw.mot.com> you wrote:
>
> I've contacted SanDisk regarding their warranty of surviving a power
> shortage in the middle of a write-cycle and received a document explaining
> the expected behavior.
...
> These particular SanDisk cards are equipped with an enhanced version of the
> internal controller that is capable of surviving such scenario leaving the
> card in a robust state.
...
> One of the recommendations was to implement a 100 millisecond power back-up
> provided for the card to finish its internal write cycle. Of course, this

There may even be CF cards with integrated coldcap - at least  that's
what I've been told once.

> To cut the long story short I personally believe (based on the number of
> stress tests we perform on our target platforms) that modern compact flash
> cards (e.g. SDCFB-xx-201 series) can be safely used as a main storage medium
> for commercial applications.

Be careful here. The fact that one manufacturer gives you a  warranty
for some specific types of flash cards does not mean that all "modern
compact  flash  cards"  behave identically. You will need to ask each
vendor for such a warranty for the specific cards you  are  going  to
use.

Best regards,

Wolfgang Denk

--
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-4596-87  Fax: (+49)-8142-4596-88  Email: wd@denx.de
Voodoo Programming: Things programmers do that  they  know  shouldn't
work  but they try anyway, and which sometimes actually work, such as
recompiling everything.                             - Karl Lehenbauer

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[Marius Groeger]

Adam,

Thanks for the interesting report.

On Wed, 12 May 2004, Zajac Adam-AAZ004 wrote:

> Coming back to the CF card discussion subject, does anybody know how the
> "power failure" scenario is resolved at such commercial platforms as pocket
> pc's and digital cameras utilizing CF cards?

Usually such devices have some power monitoring which indicates power
failure long before the worst case scenarios you described actually occur. I
could imagine the manufacturers just live with that.

Provided you can spare a CF card, you could take any such device and
manually induce a power failure while there is a write operation in
progress.

Regards,
Marius

--
Marius Groeger <mgroeger@sysgo.com>           Project Manager
SYSGO AG                      Embedded and Real-Time Software
Voice: +49 6136 9948 0                  FAX: +49 6136 9948 10
www.sysgo.com | www.elinos.com | www.osek.de | www.imerva.com


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[David Woodhouse]

On Wed, 2004-05-12 at 20:00 -0500, Zajac Adam-AAZ004 wrote:
> Gerald, Wolfgang,
>
> Thank you for your prompt response to my post.

Adam, please do something about your mail software if you're going to
use it in public. It is not correctly inserting References: or
In-Reply-To: headers, and is thus causing your reply to be disassociated
from the thread to which you're replying. This isn't polite behaviour in
a public forum. Gerald, that goes for you too.

Please also ensure that you trim the context of your reply to include
only those parts of the original message which are _absolutely_
necessary. You seem to have included far more of the previous posting
than you needed to.

> To cut the long story short I personally believe (based on the number of
> stress tests we perform on our target platforms) that modern compact flash
> cards (e.g. SDCFB-xx-201 series) can be safely used as a main storage medium
> for commercial applications.

That's an interesting report -- thanks. It's useful to know that the
quality of implementation of CF devices is improving almost to the point
where one can consider using them in production.

> Resolving a file system corruption upon an unexpected power failure should
> just be the matter of selecting one of the journaling systems (ext3, XFS,
> ReiserFS etc.). BTW, we're still evaluating which FS to choose and by far
> Ext3 looks like the best candidate due to its backward compatibility with
> Ext2.

The problem here is that you reduce the lifetime of the flash
significantly. Each write of a journalled sector actually gets written
to the 'disk' twice -- once to the journal and then again to the real
location in the file system. If you're doing data journalling, you
therefore halve the lifetime.

This is a fundamental problem with the CF mode of operation. The CF
firmware implements a pseudo-filesystem on the flash, used to 'emulate'
a normal block device. The high-end cards, as you report, at least
manage to implement this pseudo-filesystem in a way which is resilient
to power failure. Other cards have been observed to do it less
competently.

On top of that journalling pseudo-fs, you are putting another
journalling file system. Whatever you do in this situation, you are not
making optimal use of the underlying flash.

You don't even have a mechanism to 'delete' a sector from the underlying
CF device. Once you've filled the device with data once, the underlying
pseudo-fs has to keep copying around the data from sectors which are
actually _unused_ by the top-level file system.

Whatever FS you choose to put at the top of this particular stack, it
can never be as efficient as a real file system directly on the flash
can be.

> Coming back to the CF card discussion subject, does anybody know how the
> "power failure" scenario is resolved at such commercial platforms as pocket
> pc's and digital cameras utilizing CF cards?

They offer a way to reformat the file system on the flash card :)

--
dwmw2


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[VanBaren, Gerald (AGRE)]

> From: Zajac Adam-AAZ004 [mailto:Adam.Zajac@motorola.com]
> Sent: Wednesday, May 12, 2004 9:00 PM
> To: VanBaren, Gerald (AGRE); linuxppc-embedded@lists.linuxppc.org;
> 'wd@denx.de'
> Subject: RE: Question on the fs choice
>
>
> Gerald, Wolfgang,

[snip]

>
> I've contacted SanDisk regarding their warranty of surviving a power
> shortage in the middle of a write-cycle and received a
> document explaining
> the expected behavior.
> Here's an introductory quote:

[snip]

> These particular SanDisk cards are equipped with an enhanced
> version of the
> internal controller that is capable of surviving such
> scenario leaving the
> card in a robust state.
> At worst, the user loses the data that was being written to a
> particular
> sector while the power failure happened.
> Case 1: the data errors are corrected
> Case 2: the old data is restored back to the sector
> Case 3: the user loses the data on the sector
>
> One of the recommendations was to implement a 100 millisecond
> power back-up
> provided for the card to finish its internal write cycle. Of
> course, this
> can only be applied  to the systems where the user cannot
> yank the card out
> directly. For example, digital cameras have a sensor on the
> card's cover to
> signal such event.

The 100mSec power holdup (implying a power fail interrupt to the processor and associated cleanup code so it does not write to the CF when the power goes away) is very important.  The case I was referring to previously had no power fail warning (it was raw flash, not CF, but unless your CF has BIG caps built in this is immaterial).  A very low but NON-zero percentage of the time, the processor would start a write sequence and power would fail before the start address was set properly: probably between the write unlock sequence and the actual write to the device.  I speculate that the flash is then vulnerable since the write sequence was successfully completed, and the processor did some random writes that happened to hit the flash address space as its address, data, and control lines spasmed due to the power loss.  In this case the flash chip would scribble on a random sector.  Bad, really bad.

> Coming back to the CF card discussion subject, does anybody
> know how the
> "power failure" scenario is resolved at such commercial
> platforms as pocket
> pc's and digital cameras utilizing CF cards?

They all (?) use the FAT (VFAT) file system and I would speculate that they do not do any write caching (almost universal FAT file system convention).  Then they close their eyes and hope for the best ;-).  Seriously, they all warn you in the manual to NOT remove the CF while it is busy.

Incidentally, you can mount linux file systems with inhibited caching (use the "sync" option for synchronous writing -- i.e. no write caching).  A journaling file system is probably better, but putting "sync" in your /etc/fstab would be a simple and easy fix to try.

gvb


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: Question on the fs choice

[Zajac Adam-AAZ004]

David,

Thank you for your brief report.

You're getting quite a number of errors. Let's attack'em one by one.
This is your CF card type:
> hde: SanDisk SDCFB-512, CFA DISK drive

SDCFB-512 cards are equipped with the 100 series internal controller and are
not DMA capable.
I think you should try to turn DMA off for your CF card. Not sure if you can
selectively specify such support for individual IDE devices, though (you may
still want it for your hard drives if you have any in that system) -
CONFIG_BLK_DEV_IDEDMA.
As far as SanDisk cards are concerned, only cards equipped with 201
controller fully support DMA transfers (SDCFB-512-201-00, SDCFB-512-201-80).

> Partition check:
 /dev/ide/host2/bus0/target0/lun0
The partition on your CF card should get detected as hde1 or p1. The fact
that you're getting the absolute path tells me that either you don't have
hde, hde1, hde2..entries created at /dev on your root file system or you're
not running DEVFS. At the beginning, try to enable DEVFS in your kernel.

> hde: dma_intr: error=0x04 { DriveStatusError }
> hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> hde: dma_intr: error=0x04 { DriveStatusError }
> hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> hde: dma_intr: error=0x04 { DriveStatusError }
> hde: dma_intr: status=0x51 { DriveReady SeekComplete Error }
> hde: dma_intr: error=0x04 { DriveStatusError }

Compact flash cards are relatively slow in comparison with e.g. hard drives.
You may want to play with the timeout at ide.h file for pending on READY,
DRQ, BUSY bits.

Regards,
Adam


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/