Writing large files onto sync mounted MMC corrupts the FS

Writing large files onto sync mounted MMC corrupts the FS

[Jarkko Lavinen]

Hi people.

I am seeing both VFAT and Ext3 becoming corrupted when using
synchronous mount on MMC card (kernel is 2.6.12-rc2-omap1, HW is a
test board with Omap 1710).  The problem is reproduced by mounting a
MMC card synchronously and copying large file onto it.

Typically I insert vfat/ext3 formatted MMC card and do:
  mount /dev/mmcblk0p1 /media/mmc1 -t vfat -o sync
  cp /lib/libc-2.3.2.so  /media/mmc1
or
  mount /dev/mmcblk0p1 /media/mmc1 -t ext2 -o noatime,sync
  cp /lib/libc-2.3.2.so  /media/mmc1

The results are further below. I have tried to reproduce VFAT problen
in desktop PC with loopback device but to no avail. I have used
different MMC cards but the problem persists. Disabling DMA reduces
the number of error messages or pushes the first occurence a bit
further, but does not remove the problem.

Very similar problem appears when writing a large file onto ext2
formatted MMC card. The problem disappears if I mount asynchronoysly.
Also similarly the problem on VFAT disappears when mounting
asynchronously.

The VFAT problem seems to have appeared around January -- February
this year but it is hard to tell the exact point and changeset. For
ext2 I haven't done similar search.

Because there are two so similar problems appearing with two separate
file-systems, I doubt this is VFAT or Ext3 only problem, but more
likely problem in the mmc driver. It would be good to try this out on
other on some other hardware using the same mmc core code. Has anybody
else seen these?

Regards
Jarkko Lavinen


VFAT problem:
su-18-18:~# mount /dev/mmcblk0p1 /media/mmc1 -t vfat -o sync,rw,noauto,nodev,noexec,nosuid 
ke-recv.c:132: iface: org.kernel.kevent, member: mount, path: /org/kernel/block/mmcblk0/mmcblk0p1
su-18-18:~# cp /lib/libc-2.3.2.so  /media/mmc1
FAT: Filesystem panic (dev mmcblk0p1)
    fat_get_cluster: invalid cluster chain (i_pos 7861)
    File system has been set read-only
FAT: Filesystem panic (dev mmcblk0p1)
    fat_free_clusters: deleting FAT entry beyond EOF
FAT: Filesystem panic (dev mmcblk0p1)
    fat_free: invalid cluster chain (i_pos 7861)
FAT: Filesystem panic (dev mmcblk0p1)
    fat_get_cluster: invalid cluster chain (i_pos 7861)
FAT: Filesystem panic (dev mmcblk0p1)
    fat_free_clusters: deleting FAT entry beyond EOF
FAT: Filesystem panic (dev mmcblk0p1)
    fat_free: invalid cluster chain (i_pos 7861)
cp: Write Error: Input/output error


EXT3 problem:

su-18-18:~# mount /dev/mmcblk0p1 /media/mmc1 -t ext2 -o noatime,sync
ke-recv.c:132: iface: org.kernel.kevent, member: mount, path: /org/kernel/block/mmcblk0/mmcblk0p1
su-18-18:~# cp /lib/libc-2.3.2.so  /media/mmc1
jffs2_get_inode_nodes(): Data CRC failed on node at 0x05eeaee4: Read 0x0e2f9681, calculated 0xbcd5ccf9 (Unclean reboot without syncing, usually harmless)
jffs2_get_inode_nodes(): Data CRC failed on node at 0x05ef66e0: Read 0x1d6fb47e, calculated 0xaf610aef (Unclean reboot without syncing, usually harmless)
EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6771
EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6772
EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6773
EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6774
EXT2-fs error (device mmcblk0p1): ext2_free_blocks: bit already cleared for block 6775
su-18-18:~# 

Re: Writing large files onto sync mounted MMC corrupts the FS

[Lennart Sorensen]

On Wed, Jun 01, 2005 at 12:13:21PM +0300, Jarkko Lavinen wrote:
> I am seeing both VFAT and Ext3 becoming corrupted when using
> synchronous mount on MMC card (kernel is 2.6.12-rc2-omap1, HW is a
> test board with Omap 1710).  The problem is reproduced by mounting a
> MMC card synchronously and copying large file onto it.
> 
> Typically I insert vfat/ext3 formatted MMC card and do:
>   mount /dev/mmcblk0p1 /media/mmc1 -t vfat -o sync
>   cp /lib/libc-2.3.2.so  /media/mmc1
> or
>   mount /dev/mmcblk0p1 /media/mmc1 -t ext2 -o noatime,sync
>   cp /lib/libc-2.3.2.so  /media/mmc1
> 
> The results are further below. I have tried to reproduce VFAT problen
> in desktop PC with loopback device but to no avail. I have used
> different MMC cards but the problem persists. Disabling DMA reduces
> the number of error messages or pushes the first occurence a bit
> further, but does not remove the problem.
> 
> Very similar problem appears when writing a large file onto ext2
> formatted MMC card. The problem disappears if I mount asynchronoysly.
> Also similarly the problem on VFAT disappears when mounting
> asynchronously.
> 
> The VFAT problem seems to have appeared around January -- February
> this year but it is hard to tell the exact point and changeset. For
> ext2 I haven't done similar search.
> 
> Because there are two so similar problems appearing with two separate
> file-systems, I doubt this is VFAT or Ext3 only problem, but more
> likely problem in the mmc driver. It would be good to try this out on
> other on some other hardware using the same mmc core code. Has anybody
> else seen these?

Go read the thread on lkml a few weeks ago about how sync + vfat (and
others) does nasty destructive things to flash media that doesn't have
excelent wear leveling (vfat updates the fat sector for every block of
data written when you use sync option, which means you do a lot of
writes to one sector of the device, which is not good for devices with a
limited number of writes).  Typically when flash starts to reach it's
limit in number of writes, it starts having trouble holding the value
written (some bits start to drift back to one state) which can explain
your crc errors and such).

In other words: Don't use sync with flash media.  It's a horribly bad
idea, and if you look into how flash works (at least the cheaper ones)
you will realize why.  Anything with a limited number of writes allowed
to each sector should avoid rewriting the same sector again and again,
and that is what sync does when you use filesystems designed for disks
rather than flash.  JFFS was designed for flash use and rotates the
sectors it uses to store filesystem meta data and has spare space in the
filesystem to do wearleveling at the filesystem level.  vfat and ext2/3
do not, and it shows.  At least write caching/delayed write back
elliminates the worst of the rewriting of the meta data sectors.

Len Sorensen

Re: Writing large files onto sync mounted MMC corrupts the FS

[Artem B. Bityuckiy]

Lennart Sorensen wrote:
> In other words: Don't use sync with flash media.  It's a horribly bad
> idea, and if you look into how flash works (at least the cheaper ones)
> you will realize why.  Anything with a limited number of writes allowed
> to each sector should avoid rewriting the same sector again and again,
> and that is what sync does when you use filesystems designed for disks
> rather than flash.  JFFS was designed for flash use and rotates the
> sectors it uses to store filesystem meta data and has spare space in the
> filesystem to do wearleveling at the filesystem level.  vfat and ext2/3
> do not, and it shows.  At least write caching/delayed write back
> elliminates the worst of the rewriting of the meta data sectors.
>
So, we should first ask whether this happens with *new* MMC cards or not.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: Writing large files onto sync mounted MMC corrupts the FS

[Jarkko Lavinen]

On Thu, Jun 02, 2005 at 10:52:57AM +0400, ext Artem B. Bityuckiy wrote:
> So, we should first ask whether this happens with *new* MMC cards or
not.

Flash manufactorers typically claim 100k erase cycles -- some even 1M.
If FAT is updated after each written 512 byte block, filling 64MiB
card in synchronous mode would cause over 100k FAT updates.  One
couldn't even fill a new 64MiB card without already breaking it.

With asynchronous mount, one could limit the dirty_ratio from proc to
reduce the sync time against sudden power loss. But wouldn't this have
the same adverse wearing effect as with synchronous mode once the
dirty_ratio is hit and writing still proceeds?

Jarkko Lavinen