superblock checksum mismatch after crash, cannot mount

superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

Hi there,

I think i just crashed my btrfs partition, is someone willing to guide 
me through the recovery steps?

The "crash" went like so: I was testing the watchdog ability of my 
Raspberry Pi, whose root filesystem is btrfs. To test if the watchdog 
works I started a fork bomb. Normally, the watchdog device would not be 
reset and the Pi restarts. My Pi actually shut down, but didn't start 
again. When inspecting the SD card in another computer, i could not 
mount the btrfs partition. The dmesg keeps saying:

BTRFS: device label RASPIROOT devid 1 transid 18320 /dev/sdf3
BTRFS: superblock checksum mismatch
BTRFS: open_ctree failed

When trying to mount in recovery mode, it says:

$ LANG=en_US sudo mount -t btrfs -o recovery,ro /dev/sdf3 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/sdf3,
        missing codepage or helper program, or other error

I also wanted to create a btrfs-image, but it went like:

checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238
checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238
Csum didn't match
Error reading metadata block
Error adding block -5
checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238
checksum verify failed on 65536 found DFE4B1C4 wanted 712D0238
Csum didn't match
Error reading metadata block
Error flushing pending -5
create failed (Success)

Do you have any advice, how I can get the partition up running? Of 
course I do have backups and it would be a matter of minutes to reset 
the whole card, but I want to know the "btrfs like" way to repair the 
partition.

--
Best wishes
FloGa

Re: superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

I am sorry, I totally forgot to post information about my systems. I use
Btrfs v3.14.2 on all systems. My Raspberry Pi, on which the crash occured,
runs on Linux Kernel 3.16.1, my other computer on which I try to recover the
file system, runs on 3.14.14.

Re: superblock checksum mismatch after crash, cannot mount

[Duncan]

Florian Gamböck posted on Sat, 23 Aug 2014 00:00:46 +0200 as excerpted:

> I think i just crashed my btrfs partition, is someone willing to guide
> me through the recovery steps?

A month or so ago I had a similar problem, and got some experience with 
real recovery, that I hadn't had previous despite being on the list for 
some time.

First thing, afaik right now, the recovery mount option only tries the 
first superblock.  If it's corrupted, as your case reads like it might 
be, there's currently no fallback to the first and second backup 
superblocks.  There has been some discussion and a patch that would allow 
the recovery mount option to check the backup superblocks too, if the 
primary superblock fails checksum as yours did.

So the next thing is to inspect the primary superblock and see how bad it 
is, and compare it to the others, then select a good one and rewrite it 
over the bad one (semi-)manually.

btrfs-show-super is your tool for inspecting the superblocks.  See the 
manpage for the details.

Then use btrfs rescue super-recover to overwrite the bad superblock with 
one that checks out as good (csum match).  See the btrfs-rescue manpage 
for the details there.

FWIW, the first superblock is @ 64 KiB, the second @ 64 MiB, the third @ 
256 GiB.  I didn't see the size of your SD card, but I'd assume it's big 
enough to at least have the second one, but presumably not the third.  
(This is from the wiki @ btrfs.wiki.kernel.org, documentation, problem 
faq, how to cleanup the old superblock, what if I don't have wipefs.)


In my case it wasn't the superblock which was fine, but a different 
problem.  Hopefully the above will fix things for you, but if not or if 
you're simply interested in fixes for other problems, what I ended up 
using was btrfs restore.  That doesn't write to the bad filesystem, 
instead using the restore tool to search for and restore files off the 
unmounted filesystem, writing them to a presumably good mounted 
filesystem (they're written as ordinary files so the filesystem can be 
anything, it was reiserfs, here).

In addition to the manpage, the wiki has a nice page on using btrfs 
restore along with btrfs-find-root (which has a manpage as well) to try 
to find a better tree root (btrfs keeps a history of roots going back 
quite some way), if the one listed in the superblock is too damaged or 
gives you generation-doesn't-match errors.

https://btrfs.wiki.kernel.org/index.php/Restore

One thing that's /not/ entirely clear from the wiki page, that ended up 
confusing me at first when I was actually trying to use the information 
in a recovery, is that generation number and transaction ID (transid) are 
the same thing.  Assuming a good superblock (either primary or a backup), 
you can read what the current generation/transid should be with btrfs-
show-super.  That ends up being your target, and you try to get something 
as close to that as possible that has all your trees.  See the link above 
for more.

Another hint is that you can use the recently added btrfs restore -D
(--dry-run) option, to see how much it'll recover (it'll normally scroll 
off all the files it'd recover).  Using -D (dry run) with -l (list tree 
roots) and -t (root tree location) lets you see which tree roots can be 
restored from each location (the bytenr corresponding to a transid).

Finally, a couple things to keep in mind if you do end up restoring files 
this way:

* File user and permissions are lost.  Everything is restored as root 
using umasked permissions.

I had a backup but it was a bit dated, so I choose to use btrfs restore 
instead of simply restoring from the backup.  Then I hacked up a find 
script to check to see if the restored file was in the backup as well, 
and if so, chown/chmode the restored file to the same permissions as the 
backed up file.  That took care of about 98% of the files, with the 
others being new since the backup so I had to manually set those 
permissions.

* Symlinks aren't restored.

I ended up recreating them as I missed them in the course of my normal 
work.

Hopefully between the above, the wiki page and the manpages, you can use 
restore if you need to.  But with a bit of luck, all you need is btrfs-
show-super and btrfs rescue super-recover, and you'll be all set.  Of 
course you said you had backups too, giving you the freedom to experiment 
a bit and get some recovery practice without danger of further loss, 
which is always nice. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

Hi Duncan,

thank you for your detailed explanations!

Am 23.08.2014 07:27, schrieb Duncan:
> btrfs-show-super is your tool for inspecting the superblocks.  See the
> manpage for the details.
>
> Then use btrfs rescue super-recover to overwrite the bad superblock with
> one that checks out as good (csum match).  See the btrfs-rescue manpage
> for the details there.

Yes, show-super told me, that the first super block did "NOT MATCH", but 
the second one seemed to be in order ("MATCHED"). The SD card is 16 GiB 
big, so there was no third super block.

Rescue also listed the first one as bad and the second one as good and 
it seemed to run successfully, no errors whatsoever.

However, after "recovering", every single partition on the card broke. 
There were three partitions: 1 = fat32 ("/boot"), 2 = swap, 3 = btrfs 
("/"). Naturally I ran the rescue command on partition 3, I hope that 
was not a mistake. Afterwards, as I said, none of the partitions were 
recognized anymore. Not even btrfs recognized the third partition.

Since I've thought I messed up entirely, I just gave up, re-formatted 
the card and let my backups do the job.

It's quite annoying though, how often I get those problems. May it be 
that the Raspberry Pi is too slow to let btrfs do its background magic, 
may it be that btrfs is not that suitable for SD cards (in my case it's 
a class 10 microSDHC, 16GiB), or may it be that btrfs is still not that 
reliable when it comes to "unclean" shutdowns.

Nevertheless, I will keep your explanations around for future reference, 
they were really helpful! Thanks again!

--
Best wishes
FloGa

Re: superblock checksum mismatch after crash, cannot mount

[Duncan]

Florian Gamböck posted on Sat, 23 Aug 2014 10:38:47 +0200 as excerpted:

> Am 23.08.2014 07:27, schrieb Duncan:
>> btrfs-show-super is your tool for inspecting the superblocks.  [...]
>> Then use btrfs rescue super-recover

> Yes, show-super told me, that the first super block did "NOT MATCH", but
> the second one seemed to be in order ("MATCHED"). The SD card is 16 GiB
> big, so there was no third super block.
> 
> Rescue also listed the first one as bad and the second one as good and
> it seemed to run successfully, no errors whatsoever.

Good so far, but...

> However, after "recovering", every single partition on the card broke.
> There were three partitions: 1 = fat32 ("/boot"), 2 = swap, 3 = btrfs
> ("/"). Naturally I ran the rescue command on partition 3, I hope that
> was not a mistake. Afterwards, as I said, none of the partitions were
> recognized anymore. Not even btrfs recognized the third partition.

That is strange.  Was the partition table still recognized?  Had you used 
mbr or gpt partitioning (I'm presuming the pi handles them, of course, I 
don't know a lot about pi/arm).

FWIW these days I use gpt because it keeps one checksummed copy at each 
end of the device for better reliability, and of course also because it 
does away with primary/secondary/logical partition worries and allows 
partition names/labels (as distinct from filesystem labels), but I have 
literally /no/ idea whether pi supports it as I've never worked with a pi.

I can't think of any scenario that would screw up all the partitions  
unless the partition table was damaged and/or unless the firmware was 
buggy and basically scrambled things.  It's almost as if the superblock 
fix landed in the middle of the partition table (maybe 64 KiB into the 
entire SD card, is that still partition table?) instead of at the 64 KiB 
mark into what would have been partition 3, the btrfs partition, but if 
it was reading the second one as good, it must have gotten the right 
partition to do that, in which case it couldn't be that you simply typoed 
and didn't include the digit.

I can't personally rule out a bug in btrfs rescue as I've never used that 
tool myself, but one this severe?  Hard to imagine, yet /something/ 
screwed up. 

Weird.

Anyway, were it me, I'd consider that sdcard suspect (bad firmware) until 
I could pin down what it did a bit better, because something implausible 
just happened, and that's as plausible an explanation as anything else 
we've got.

> Since I've thought I messed up entirely, I just gave up, re-formatted
> the card and let my backups do the job.

At that point, that's what I would have done too, with the caveat about 
considering the card suspect for the time being, and being sure it's 
/regularly/ backed up in case it scrambles again.

> It's quite annoying though, how often I get those problems. May it be
> that the Raspberry Pi is too slow to let btrfs do its background magic,
> may it be that btrfs is not that suitable for SD cards (in my case it's
> a class 10 microSDHC, 16GiB), or may it be that btrfs is still not that
> reliable when it comes to "unclean" shutdowns.

I'd guess a mix of the above.  There's some recent research at how poor 
certain solid state devices are at handling power cuts.  Someone here 
probably has a link or google it.  Having work on one partition scramble 
others certainly seems like bad firmware to me, tho I'm no expert.  But 
btrfs' write patterns are surely about as far from the fat many of these 
devices were designed for as it gets, so btrfs may well be stressing it 
quite a bit as well.

I'd also try to compare notes with other pi users and see if they're 
noting problems with the pi and btrfs as well.  I'm entirely clueless in 
that regard, but I read enough about them to at least know they have a 
rather large and active user community, so you can't be the first to have 
tried it.  Someone's gotta have some experience to share.

The other possibility I guess is to try a different brand and spec of 
card, to see if that makes any difference.  I just don't know at this 
point, so it's all in-play ATM.

> Nevertheless, I will keep your explanations around for future reference,
> they were really helpful! Thanks again!

=:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

Am 23.08.2014 11:34, schrieb Duncan:
>> Afterwards, as I said, none of the partitions were recognized anymore. Not even btrfs recognized the third partition.
> That is strange.  Was the partition table still recognized?  Had you used
> mbr or gpt partitioning (I'm presuming the pi handles them, of course, I
> don't know a lot about pi/arm).

I haven't run any tests on that, so to be safe, I use a MBR table. And 
yes, the table was still in order, all three partitions were there, but 
none of the filesystems were recognized. Sorry if I confused you. But 
still I think it's a big deal if a rescue on partition 3 breaks the 
filesystems on each partition of the device. It's like every single 
superblock was deleted.

> FWIW these days I use gpt because it keeps one checksummed copy at each
> end of the device for better reliability, and of course also because it
> does away with primary/secondary/logical partition worries and allows
> partition names/labels (as distinct from filesystem labels), but I have
> literally /no/ idea whether pi supports it as I've never worked with a pi.

This sounds interesting. Maybe that will be my next project to stress 
little Pi.

> I can't think of any scenario that would screw up all the partitions
> unless the partition table was damaged and/or unless the firmware was
> buggy and basically scrambled things.  It's almost as if the superblock
> fix landed in the middle of the partition table (maybe 64 KiB into the
> entire SD card, is that still partition table?) instead of at the 64 KiB
> mark into what would have been partition 3, the btrfs partition, but if
> it was reading the second one as good, it must have gotten the right
> partition to do that, in which case it couldn't be that you simply typoed
> and didn't include the digit.

So, I am almost absolutely sure that the table is in order, since there 
are no other problems regarding that. And like you said, rescue showed 
me the right superblocks and also here I am sure that I didn't forget 
the 3 when typing in the command.

> I can't personally rule out a bug in btrfs rescue as I've never used that
> tool myself, but one this severe?  Hard to imagine, yet /something/
> screwed up.

I don't want to believe that the SD firmware or the driver or whatever 
would redirect calls on certain partitions to the main device. So it 
seems more plausible to think that there is something wrong in the 
rescue command, like the calculation of the offset or something.

> Anyway, were it me, I'd consider that sdcard suspect (bad firmware) until
> I could pin down what it did a bit better, because something implausible
> just happened, and that's as plausible an explanation as anything else
> we've got.

These corruptions (although I can't say for sure that the problems were 
identical) happened both on a cheap class 4 SDHC and on my current class 
10 microSDHC, the first one on a Raspberry B, the latter on a Raspberry 
B+. You are right, at the moment we can only guess what the cause of the 
errors was, but it's hard to believe that both Pis and both SD cards 
share the same bugs. At least I don't want to believe that.

> At that point, that's what I would have done too, with the caveat about
> considering the card suspect for the time being, and being sure it's
> /regularly/ backed up in case it scrambles again.

I create btrfs snapshots every 6 hours on the same card and send them to 
an external flash drive every 24 hours. If I've done much work, then 
these intervals will be shorter, of course. I always work in a way that 
nothing will be lost, even if the Pi explodes. ;-)

> There's some recent research at how poor certain solid state devices are at handling power cuts. Someone here probably has a link or google it. Having work on one partition scramble others certainly seems like bad firmware to me, tho I'm no expert. But btrfs' write patterns are surely about as far from the fat many of these devices were designed for as it gets, so btrfs may well be stressing it quite a bit as well.

Yes, but that's exactly why we test and experiment. I try several mount 
options, I tried several SD cards and maybe some lucky day I can track 
down and reproduce the problems and can say for sure whether the card or 
btrfs was at fault.

> I'd also try to compare notes with other pi users and see if they're
> noting problems with the pi and btrfs as well.  I'm entirely clueless in
> that regard, but I read enough about them to at least know they have a
> rather large and active user community, so you can't be the first to have
> tried it.  Someone's gotta have some experience to share.

Regarding that, most times I just read posts like "just don't do that", 
"just use raspbian's default" and the like. No more explanations. So I'd 
rather go on my own journey, keep on using Gentoo, adjust my kernel, and 
stress btrfs to its limits. I like to experiment new things, after all.

But thanks again for your advice, it helps me to poke at the right 
edges. :-)

-- 
Best wishes

FloGa

Re: superblock checksum mismatch after crash, cannot mount

[Zygo Blaxell]

On Sat, Aug 23, 2014 at 09:34:10AM +0000, Duncan wrote:
> Florian Gamböck posted on Sat, 23 Aug 2014 10:38:47 +0200 as excerpted:
> 
> > Am 23.08.2014 07:27, schrieb Duncan:
> >> btrfs-show-super is your tool for inspecting the superblocks.  [...]
> >> Then use btrfs rescue super-recover
> 
> > Yes, show-super told me, that the first super block did "NOT MATCH", but
> > the second one seemed to be in order ("MATCHED"). The SD card is 16 GiB
> > big, so there was no third super block.
> > 
> > Rescue also listed the first one as bad and the second one as good and
> > it seemed to run successfully, no errors whatsoever.
> 
> Good so far, but...
> 
> > However, after "recovering", every single partition on the card broke.
> > There were three partitions: 1 = fat32 ("/boot"), 2 = swap, 3 = btrfs
> > ("/"). Naturally I ran the rescue command on partition 3, I hope that
> > was not a mistake. Afterwards, as I said, none of the partitions were
> > recognized anymore. Not even btrfs recognized the third partition.
> 
> That is strange.  Was the partition table still recognized?  Had you used 
> mbr or gpt partitioning (I'm presuming the pi handles them, of course, I 
> don't know a lot about pi/arm).
> 
> FWIW these days I use gpt because it keeps one checksummed copy at each 
> end of the device for better reliability, and of course also because it 
> does away with primary/secondary/logical partition worries and allows 
> partition names/labels (as distinct from filesystem labels), but I have 
> literally /no/ idea whether pi supports it as I've never worked with a pi.
> 
> I can't think of any scenario that would screw up all the partitions  
> unless the partition table was damaged and/or unless the firmware was 
> buggy and basically scrambled things.  It's almost as if the superblock 
> fix landed in the middle of the partition table (maybe 64 KiB into the 
> entire SD card, is that still partition table?) instead of at the 64 KiB 
> mark into what would have been partition 3, the btrfs partition, but if 
> it was reading the second one as good, it must have gotten the right 
> partition to do that, in which case it couldn't be that you simply typoed 
> and didn't include the digit.
> 
> I can't personally rule out a bug in btrfs rescue as I've never used that 
> tool myself, but one this severe?  Hard to imagine, yet /something/ 
> screwed up. 
> 
> Weird.

Sounds like perfectly normal SD card corruption.

> Anyway, were it me, I'd consider that sdcard suspect (bad firmware) until 
> I could pin down what it did a bit better, because something implausible 
> just happened, and that's as plausible an explanation as anything else 
> we've got.
> 
> > Since I've thought I messed up entirely, I just gave up, re-formatted
> > the card and let my backups do the job.
> 
> At that point, that's what I would have done too, with the caveat about 
> considering the card suspect for the time being, and being sure it's 
> /regularly/ backed up in case it scrambles again.
> 
> > It's quite annoying though, how often I get those problems. May it be
> > that the Raspberry Pi is too slow to let btrfs do its background magic,
> > may it be that btrfs is not that suitable for SD cards (in my case it's
> > a class 10 microSDHC, 16GiB), or may it be that btrfs is still not that
> > reliable when it comes to "unclean" shutdowns.
> 
> I'd guess a mix of the above.  There's some recent research at how poor 
> certain solid state devices are at handling power cuts.  Someone here 
> probably has a link or google it.  Having work on one partition scramble 
> others certainly seems like bad firmware to me, tho I'm no expert.  But 
> btrfs' write patterns are surely about as far from the fat many of these 
> devices were designed for as it gets, so btrfs may well be stressing it 
> quite a bit as well.

btrfs write patterns are probably closer to what SD cards are designed
for than ext4 or xfs.  btrfs COW tries to write data sequentially in
contiguous chunks like a camera sequentially storing JPEGs on a FAT
filesystem.  Contrast with ext4 which alternates between journal writes
and random 4K page writes.  ext4 also writes metadata twice (once in
the FS, once in the journal) to wear out the SD card faster.

> I'd also try to compare notes with other pi users and see if they're 
> noting problems with the pi and btrfs as well.  I'm entirely clueless in 
> that regard, but I read enough about them to at least know they have a 
> rather large and active user community, so you can't be the first to have 
> tried it.  Someone's gotta have some experience to share.

I have several Pis running btrfs on a local build of kernel 3.12.24.
I have no btrfs-specific problems with these...but there's a "but":

Consumer SD cards are /terrible/ storage devices.  Always back up all
data written to an SD card as soon as possible after writing it, and
develop a process to restore the backup to a new SD card conveniently
when--not if--the card fails.

Over the years I've burned my way through dozens of SD cards in Pis,
Beagles, x86 laptops, USB SD card readers, cameras and cell phones.
I have more bad or failed cards than good ones in my collection, but no
more than three of any specific model.  Brand, price, and specs don't
correlate to success or failure.  Even the good cards wear out after
heavy use.  The bad ones fail much faster, and are more likely to give
you garbage data instead of properly formed I/O errors as they fail.

btrfs helps with the backup/restore process, detects SD card failure
faster, and puts less write load on the SD card (COW instead of
journalling, and compression reduces write size).

Some might suggest "ext4 is better than btrfs because I can fix a
corrupt SD card with e2fsck that I can't fix with btrfsck"; however,
in my experience the Pis don't have enough RAM to run e2fsck when it
allocates extra memory to fix non-trivial FS corruption, and failing SD
cards just get worse when you rewrite them anyway.  Save some time and
sanity, and just skip ahead to building a new SD card (preferably not
the same brand and model as the old one) from the last backup.  You may
need to go through three or four models before you find one that works.

Bad firmware (i.e. anything less than the very /best/ firmware) can flip
bits instead of giving explicit I/O errors.  No filesystem can survive
this.  btrfs can turn flipped bits into I/O errors and try to limit
the damage by making the filesystem read-only, but it can't fix the
underlying problem.

You might still be able to use a bad SD card read-only in a Pi (e.g. to
boot and mount root from a USB device); however, any write to a bad
SD card can scramble its contents, so such a system is running on
borrowed time.  In the worst cases, a single write can make all data on
the card instantly and permanently irretrievable.  In typical cases a
write to one location effectively change data in several other locations
in unpredictable ways, including changes to the mapping between logical
block addresses and physical storage locations.

btrfs suppresses redundant metadata by default when it detects
non-rotating media, so on a single storage device you won't get two copies
of metadata and all corruption is unrecoverable.  Corrupt metadata
produces messages that look more like FS bugs than storage device
corruption--you won't get nice clean "checksum failed" error messages
when btrfs can't find the data checksums in a corrupt metadata tree to
verify them.

> The other possibility I guess is to try a different brand and spec of 
> card, to see if that makes any difference.  I just don't know at this 
> point, so it's all in-play ATM.
> 
> > Nevertheless, I will keep your explanations around for future reference,
> > they were really helpful! Thanks again!
> 
> =:^)
> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: superblock checksum mismatch after crash, cannot mount

[Duncan]

Zygo Blaxell posted on Sat, 23 Aug 2014 12:38:05 -0400 as excerpted:

> Consumer SD cards are /terrible/ storage devices.  Always back up all
> data written to an SD card as soon as possible after writing it, and
> develop a process to restore the backup to a new SD card conveniently
> when--not if--the card fails.
> 
> Over the years I've burned my way through dozens of SD cards in Pis,
> Beagles, x86 laptops, USB SD card readers, cameras and cell phones.
> I have more bad or failed cards than good ones in my collection, but no
> more than three of any specific model.  Brand, price, and specs don't
> correlate to success or failure.  Even the good cards wear out after
> heavy use.  The bad ones fail much faster, and are more likely to give
> you garbage data instead of properly formed I/O errors as they fail.

I had read that it was bad, but I didn't know it was /that/ bad.  The 
ones I've used have tended to be write-once, read for quite some time, 
often lose (or throw away as obsolete due to tiny size) before I write 
them again, or at least before I write them half a dozen times, and I've 
generally not has problems with them doing that, but I wouldn't tend to 
know about routine rewrite behavior.  Sounds like it's much worse than I 
might have thought.

Thanks.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: superblock checksum mismatch after crash, cannot mount

[Chris Murphy]

On Aug 23, 2014, at 6:56 PM, Duncan <1i5t5.duncan@cox.net> wrote:

> Zygo Blaxell posted on Sat, 23 Aug 2014 12:38:05 -0400 as excerpted:
> 
>> Consumer SD cards are /terrible/ storage devices.  Always back up all
>> data written to an SD card as soon as possible after writing it, and
>> develop a process to restore the backup to a new SD card conveniently
>> when--not if--the card fails.
>> 
>> Over the years I've burned my way through dozens of SD cards in Pis,
>> Beagles, x86 laptops, USB SD card readers, cameras and cell phones.
>> I have more bad or failed cards than good ones in my collection, but no
>> more than three of any specific model.  Brand, price, and specs don't
>> correlate to success or failure.  Even the good cards wear out after
>> heavy use.  The bad ones fail much faster, and are more likely to give
>> you garbage data instead of properly formed I/O errors as they fail.
> 
> I had read that it was bad, but I didn't know it was /that/ bad.  The 
> ones I've used have tended to be write-once, read for quite some time, 
> often lose (or throw away as obsolete due to tiny size) before I write 
> them again, or at least before I write them half a dozen times, and I've 
> generally not has problems with them doing that, but I wouldn't tend to 
> know about routine rewrite behavior.  Sounds like it's much worse than I 
> might have thought.

Their primary usage is photography and video. Specifically it means highly variable camera firmware doing only sequential writing with only FAT32 (SD/SDHC) or exFAT (SDXC) as the fs, and jpeg, raw and video for file formats. The computer's job is pretty much read-only. It's a really narrow use case and I think they have their hands full with the camera firmware as a variable. So I wouldn't be surprised if the use case discussed here doesn't get much attention.

It's so bad for years now with CF and SD Cards the card manufacturer's say basically to expect corruption or malfunctioning cameras if you move the card between cameras without reformatting. And the reformatting needs to be done in-camera. In amateur and esp. pro photography and video, it's long been suggests to not delete individual files in the camera UI. Take your shots. When full or convenient copy all of them off the card. Back them up. Open/process them to make sure they're not already corrupted. Then reformat the card in-camera. Do the editing, i.e. delete photos you don't want, on the computer.


Chris Murphy

Re: superblock checksum mismatch after crash, cannot mount

[Leen Besselink]

On Sun, Aug 24, 2014 at 12:56:47AM +0000, Duncan wrote:
> Zygo Blaxell posted on Sat, 23 Aug 2014 12:38:05 -0400 as excerpted:
> 
> > Consumer SD cards are /terrible/ storage devices.  Always back up all
> > data written to an SD card as soon as possible after writing it, and
> > develop a process to restore the backup to a new SD card conveniently
> > when--not if--the card fails.
> > 
> > Over the years I've burned my way through dozens of SD cards in Pis,
> > Beagles, x86 laptops, USB SD card readers, cameras and cell phones.
> > I have more bad or failed cards than good ones in my collection, but no
> > more than three of any specific model.  Brand, price, and specs don't
> > correlate to success or failure.  Even the good cards wear out after
> > heavy use.  The bad ones fail much faster, and are more likely to give
> > you garbage data instead of properly formed I/O errors as they fail.
> 
> I had read that it was bad, but I didn't know it was /that/ bad.  The 
> ones I've used have tended to be write-once, read for quite some time, 
> often lose (or throw away as obsolete due to tiny size) before I write 
> them again, or at least before I write them half a dozen times, and I've 
> generally not has problems with them doing that, but I wouldn't tend to 
> know about routine rewrite behavior.  Sounds like it's much worse than I 
> might have thought.
> 

tip for basically any Linux filesystem, especially on Flash-based storage and also btrfs:
- use noatime (if you aren't doing that already, don't know if that is the default in btrfs)

" Performance

noatime - as discussed in the mailing list noatime mount option might speed up your file system, especially in case you have lots of snapshots. Each read access to a file is supposed to update its unix access time. COW will happen and will make even more writes. Default is now relatime which updates access times less often (see: http://kerneltrap.org/node/14148). 

Note that noatime will break some applications like the venerable mutt (unless you use Maildir mailboxes). "

https://btrfs.wiki.kernel.org/index.php/Mount_options#Performance

How bad is it ? Well, look at this presentation:

https://www.youtube.com/watch?v=CPEzLNh5YIo

> Thanks.
> 
> -- 
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: superblock checksum mismatch after crash, cannot mount

[Chris Samuel]

On Sun, 24 Aug 2014 01:08:53 PM Leen Besselink wrote:

> tip for basically any Linux filesystem, especially on Flash-based storage
> and also btrfs: - use noatime (if you aren't doing that already, don't know
> if that is the default in btrfs)

Since 2.6.30 (5 years old now) the kernel has defaulted to mounting filesystems
with relatime not atime so it's not as bad as it used to be.  This is from the mount
manual page.

       relatime
              Update inode access times relative to modify or change time.  Access time is  only
              updated  if the previous access time was earlier than the current modify or change
              time. (Similar to noatime, but doesn't break mutt or other applications that  need
              to know if a file has been read since the last time it was modified.)

              Since  Linux  2.6.30,  the kernel defaults to the behavior provided by this option
              (unless noatime was  specified), and the strictatime option is required to  obtain
              traditional  semantics.  In  addition,  since Linux 2.6.30, the file's last access
              time is always  updated  if  it  is more than 1 day old.

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: superblock checksum mismatch after crash, cannot mount

[Duncan]

Leen Besselink posted on Sun, 24 Aug 2014 13:08:53 +0200 as excerpted:

> tip for basically any Linux filesystem, especially on Flash-based
> storage and also btrfs:
> - use noatime (if you aren't doing that already, don't know if that is
> the default in btrfs)

It's not the default for btrfs, but I have a patch that makes it the 
default on my system. =:^)



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

Am 24.08.2014 13:08, schrieb Leen Besselink:
> tip for basically any Linux filesystem, especially on Flash-based storage and also btrfs:
> - use noatime (if you aren't doing that already, don't know if that is the default in btrfs)

Yeah, I use this option per standard since I don't really need access times.

> How bad is it ? Well, look at this presentation:
>
> https://www.youtube.com/watch?v=CPEzLNh5YIo

Wow, this really surprised me. I never thought that SD cards are that 
unreliable. Maybe on my Pi I'll use the card only for booting from now 
on and move the system to an external hard drive ...

Re: superblock checksum mismatch after crash, cannot mount

[Flash ROM]

About SD cards and somesuch... 

TL;DR: THINK TWICE before formatting SD cards!!!

What is SD card? One or several NAND flash ICs + controller doing wear leveling and interface translation. It does wear leveling and handles flash blocks translation to show you what you expect, making it look like if it can deal with 512 byte sectors, while flash haves totally different blocking factors.

There'are some bumps on this way. Most of issues come from the fact that flash memory *lacks* "write" operation and uses very specific block operations. There is "page program" and "block erase". Page is typically 1 to 4KiB (+some bytes for ECC, "out of band" data). Erase block is much larger and can usually vary about 256KiB to about 16MiB. Advanced flash storages could prefer to do erase operations in larger erase groups, probably to speed things up by submitting commands to several flash IC at once. This can make preferred erase block size of flash-backed storage even larger.

Once flash page is programmed, it can't be filled with different data unless whole huge erase block is erased by erase operation, clearing all pages in block to their erased state. And then flash can withstand only limited amount of cycles, about several thousands for MLC (2 bit per cell) NAND and only few hundreds rewrites for TLC (3 bit per cell) nand. Then there could be bad blocks, both pre-existing at manufacture time and newly appeared due to wear. And some reserve space to cover new bad blocks. And software expects all these damn 512 byte sectors. 

Sounds a bit complicated, isn't it? So there is rather comples translation layer on the way which does translations, bad blocks mapping, wear leveling and so on. In fact even SD card haves rather smart firmware in controller which does many non-trivial operations. But it haves simplified routines, which makes it worse than SSDs. 

So what? There're some things you do not expect.

1) Formatting and repartitioning SD card? Generally WORST IDEA EVER. Unless you got idea about NAND flash, how it works and able to guess flash geometry and place filesystem adequately and hint it to use sane blocking factors (hard thing to do), you better do not repartition or format SD cards at all. Factory file sysmte comes pre-formatted in very special way. Generally if you look on new card, you will see fileystem follows special patterns. First, partition table looks a bit empty. There is partition table block and then ... some unused space. While it sounds dumb, this strange thing being done to put partition table in separate erase block, so it never read-modify-written when FAT entries are updated. Should something go wrong, FAR can recover from backup copy. But erased partition table just suxx. Then, FAT tables are aligned in way to fit well around erase block bounds. And of course filesystem blocks should map correctly over NAND pages, because getting filesystem block put into middle of two pages means 2 pages have to be touched when you write single filesystem block just because of bad layout. Incorrect layout can kill write performance of SD card by 2-3 times, especially on small files/small writes and cause extra wear. This is known as write amplification factor. 

This said, you can *try* to reformat, BUT no standard OS of firmware formatter will help you with default settings. They can't know geometry of underlying NAND and controller properties. There is no standard, widely accepted way to get such information from card. No matter if you use OS formatter, camera formatter or whatever. YOU WILL RUIN factory format (which is crafted in best possible way) and replace it with another, very likely suboptimal one. So you can easily reduce write speed, get increased wear and just make it unsafe. Up to sudden loss of partition table. So if you're about to try, remember at least the following: put partition table to separate erase block which is never touched. Sudden loss of partition table suxx. So, assuming most common 2-bit MLC NAND, first partition should start at least about 4-32MiB away from card beginning (more alignment does not hurts as long as it is power of 2 and you're ok with lost space). Large alignments of powers of 2 is your friend (in case TLC nand can be not a case due to strange block sizes). Filesystem start should be aligned to large power of 2 factors like units of 32MiB from begin of card. Filesytem blocks should be put on page boundary, avoiding case where filesystem block laid badly over pages boundary (if you lucky, can happen automatically if you align FS start in mentioned way). For these who is really inclined to try this and want to get sane results at cost of some card wearout, there are some tools like "flashbench" which will try to guess actual geometry and best alignment by actually doing some writes and looking on resulting speeds. Obviously, best blocking match leads to best speed as well due to least possible amplification factor.

2) Active writes? Especially small and random? Nice way to quickly kill your card. You see, wear leveling controller could be simplified. Then card often shown as mass storage and I'm not really sure this way card getting any hints about unused regions at all (shouldnt be huge issue for real sd/mmc hosts like Pi but can be problem with USB card readers and somesuch, I guess). If unused region hints are not used (no working DISCARD-like things in effect), wear leveller faces very heavy and suboptimal conditions, where it only haves few extra non-busy blocks for all operations and so it have to deal with read-modiwy-erase-write sequencing far more often than it could be in better conditions. Whole point of DISCARD was to improve operational conditions of wear leveler by giving hints which blocks are not used anymore so it can erase blocks in background and then use them to satisfy request in optimal ways, by just doing what has been requested with minimal overhead. This speeds up write (read-modify-erase-program sequence is slow) and reduces amplification factor, hence less wear. And why so many buzz about amplification? Imagine you'll write 64-bytes file. Due to sector nature it would be at least 512 bytes operation (8x amplification). Maybe more if filesystem only works with blocks. Controller will at least read something like 4KiB page, no matter what, and patch requested 512 bytes inside, and at least program it to new place, should there be free page. So now 64 bytes are 4KiB operation, huh? Buf if there was no free page, controller will have to do some read-patch-erase-write for whole erase block, often 4MiB or more. Or even several blocks + forced garbage collection, if there was no empty erase block to put patched data on (likely scenario if you haven't got DISCARD support on wheels). So, several 4MiB blocks were read, patched and at least one was erased? For that silly 64 byte write? That's what we call amplificaton! Needless to say, if you feed card with small random writes, it will die much, much faster 
than you can expect from card size multiplied by erase cycles NAND can withstand. Now you probably understand why your card will never reach rated write speed on small files. If you write 100MiB file, it will eventually self-sync on blocking border, making amplification factor close to 1. Not a case for 64-byte files. Btw same applies for SSDs, though their larger capacity, widespread discard support and better controllers and algos make it less daunting problem for sure. 

Solution? Use heavy RAM buffering (increase write barrier times?) and try to hint filesystem to aggregate writes to large sequential blocks, preferrably multiple of eraseblock size(in some FS its possible to "ab"use hinting intended for RAID stripes, etc). And DISCARD is your best friend, if you can get it working. Interestingly, CoW based designs are inherently more flash-friendly due to the fact wear leveler is kind of CoW itself and in lucly case it can be almost 1 to 1 mapping. In bad cases write amplification can occur of course. Best case is when file system is explicitly aware of flash blocking factors and uses this knowledge to optimise all sorts of things but its not what you can expect from SD cards and/or "generic" filesystems.

3) Attitude when NAND worn out. SSDs are usually relatively smart in this regard and will just go read only when they're close to expected maximum of erase-reprogram cycles of flash. And SSDs are quite good in ECC abilities. SD cards have cheap and weak controller and so much more dumb and what you see can wary in drastic ways. Sometimes critical structures in NAND could become corrupted to degree controller would fail to initialise card at power up, making data completely unavailable, card is pretty dead at this point. Sometimes controller would miss corrupt data. Sometimes it would be unable to handle it properly. In best case it would go readonly as SSDs do. SSDs have S.M.A.R.T. to help you to get idea how it performs. SD cards.... uhm, there was idea to do similar thing and newest standards added commands with similar meaning, but needless to say its not yet widespread and very optional thing, mostly meant for built-in eMMC "cards". So hell yeah, backup your data. You can't know when it will fail and you can't know how it would fail. Needless to say it makes it easy to suddenly loss all your data.

4) Unexpected power loss on write? Ouch, it hurts! Imagine power got lost in the middle of read-patch-erase-program sequence dealing with large 4+ MiB block, etc. It can happen data were not completely written back to new destination when power loss arrived, possibly resulting in data loss. SSDs are even nasty enough to keep interesting counter - "unsafe shutdown count", it increased each time you've turned off power without properly shutting down SSD first, hence causing risk of mentioned issues. That's why partition table *MUST* live in separate erase block. Else you can once figure out block has been erased but power lost before partition table data were written back. Hence, partition table loss. Sounds cool, isn't it? This case can also violate assumptions of most filesystems about how underlying layer behaves on power loss. Most filesystems just do not expect power loss can cause corruption of something like 4MiB eraseblock with all data it keeps instead of just data involved into current operation. Then, hidden geometry makes it hard to even predict which (meta)data will be damaged. This also affects SSDs to some degree. Toggling power of flash based storages when write is in progress and some time after it (where background logic can operate) is a really bad idea.

5) Sometimes you can face less common stuff like wear leveler hardcoded to deal with FAT32. Fat area can have smaller eraseblocks and wear levler can assume there is FAT and will adjust logic to better handle small frequent writes into this area. You never know what wear leveler will do. So once you trashed factory filesystem, you're really on your own and proceeding on your own risk.


Bottom line: THINK TWICE before formatting SD cards.ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: superblock checksum mismatch after crash, cannot mount

[Florian Gamböck]

Am 24.08.2014 18:59, schrieb Flash ROM:
> About SD cards and somesuch...
>
> TL;DR: THINK TWICE before formatting SD cards!!!

If you intended to make me hate SD cards then you did a really good job.

> Bottom line: THINK TWICE before formatting SD cards.

I didn't even think once, because actually I was never aware of those 
things you mentioned. In fact I still don't care about that, because I 
use my Pis mainly to experiment and to get experience with those 
failures, but if I ever use them in a real life situation then I'm 
definitely going to use an external drive and just put the bootloader on 
SD, without formatting. ;-)

Thank you for your detailed essay!

Re: superblock checksum mismatch after crash, cannot mount

[Chris Murphy]

On Aug 24, 2014, at 10:59 AM, Flash ROM <flashromguru@yandex.com> wrote:
> 
> 1) Formatting and repartitioning SD card? Generally WORST IDEA EVER.

All cameras have a format function, and this function both partitions and formats (creates a filesystem/volume format).  If they can get away with it, so can anyone else.


> While it sounds dumb, this strange thing being done to put partition table in separate erase block, so it never read-modify-written when FAT entries are updated. Should something go wrong, FAR can recover from backup copy. But erased partition table just suxx. Then, FAT tables are aligned in way to fit well around erase block bounds.

I think you seriously overestimate the knowledge of camera manufacturer's about the details of flash storage; and any ability to discover it; and any willingness on the part of the flash manufacturer to reveal such underlying details. The whole point of these cards is to completely abstract the reality of the underlying hardware from the application layer - in this case the camera or mobile device using it.

Also, with SDXC exFAT is now specified. And it has only one FAT there isn't a backup FAT. So they're even more difficult to recover data from should things go awry filesystem wise.


> This said, you can *try* to reformat, BUT no standard OS of firmware formatter will help you with default settings. They can't know geometry of underlying NAND and controller properties. There is no standard, widely accepted way to get such information from card. No matter if you use OS formatter, camera formatter or whatever. YOU WILL RUIN factory format (which is crafted in best possible way) and replace it with another, very likely suboptimal one.

It's recommended by the card manufacturers to reformat it in each camera its inserted into. It's the only recommended way to "erase" the sd card for re-use, they don't recommend selectively deleting images. And it's known that one camera's partition table and formatting can irritate another camera make/model if the card isn't reformatted by that camera.

> So once you trashed factory filesystem, you're really on your own and proceeding on your own risk.

Absurd. It's recommended *by* both device manufacturers (cameras and mobile devices) that consume such cards, as well as the cards' manufacturers. The workflow you're proposing for SD Cards is as a write once device with no practical way of deleting the files on it.


Chris Murphy

Re: superblock checksum mismatch after crash, cannot mount

[Chris Murphy]

On Aug 23, 2014, at 8:14 AM, Florian Gamböck <mail@floga.de> wrote:

> 
> I haven't run any tests on that, so to be safe, I use a MBR table. And yes, the table was still in order, all three partitions were there, but none of the filesystems were recognized. Sorry if I confused you. But still I think it's a big deal if a rescue on partition 3 breaks the filesystems on each partition of the device. It's like every single superblock was deleted.

This is on 32-bit ARM, so it is possible there's a bug no one else has run into yet. It's worth trying to reproduce the conditions. Maybe someone has a suggestion on how to corrupt just the first Btrfs superblock, it should be really straightforward. Then you can try to repair it again: both ways, with the partition specified and not specified to see if there are different results.


> So, I am almost absolutely sure that the table is in order, since there are no other problems regarding that. And like you said, rescue showed me the right superblocks and also here I am sure that I didn't forget the 3 when typing in the command.

Well it's a question then whether that should matter, and if it does then maybe that's a bug or feature request. You chose superblock 2 to overwrite superblock 1, which is a fixed LBA distance, so it should only be overwriting sectors inside the Btrfs partition in any case unless there isn't a check for this. In which case it should only affect some sectors starting 64KB offset from the start of the whole block device. That would only adversely affect one of the filesystems, not two. So it seems something else is going on here.

If you haven't changed anything on this SD Card you could look at a couple sectors 64KB offset from each /dev/sdX and /dev/sdX[12] - so three checks. And see what's there. Is it a Btrfs superblock? If none of them have a Btrfs superblock then definitely something else is going on.

> 
> I don't want to believe that the SD firmware or the driver or whatever would redirect calls on certain partitions to the main device. So it seems more plausible to think that there is something wrong in the rescue command, like the calculation of the offset or something.

It's possible. See the other thread about this "btrfs receive problem on ARM kirkwood NAS" in particular the reply from Hugo Mills.



Chris Murphy

Re: superblock checksum mismatch after crash, cannot mount

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

On 2014-08-24 15:48, Chris Murphy wrote:
> 
> On Aug 24, 2014, at 10:59 AM, Flash ROM <flashromguru@yandex.com> wrote:
>> While it sounds dumb, this strange thing being done to put partition table in separate erase block, so it never read-modify-written when FAT entries are updated. Should something go wrong, FAR can recover from backup copy. But erased partition table just suxx. Then, FAT tables are aligned in way to fit well around erase block bounds.
> 
> I think you seriously overestimate the knowledge of camera manufacturer's about the details of flash storage; and any ability to discover it; and any willingness on the part of the flash manufacturer to reveal such underlying details. The whole point of these cards is to completely abstract the reality of the underlying hardware from the application layer - in this case the camera or mobile device using it.
> 
If you really know what you are doing, it is possible to determine erase
block size by looking at device performance timings, with surprisingly
high accuracy (assuming you aren't trying to have software do it for
you).  I've actually done this before on several occasions, with nearly
100% success.
> Also, with SDXC exFAT is now specified. And it has only one FAT there isn't a backup FAT. So they're even more difficult to recover data from should things go awry filesystem wise.
> 
It's too bad that TFAT didn't catch on, as it would have been great for
SD cards if it could be configured to put each FAT on a different erase
block.
> 
>> This said, you can *try* to reformat, BUT no standard OS of firmware formatter will help you with default settings. They can't know geometry of underlying NAND and controller properties. There is no standard, widely accepted way to get such information from card. No matter if you use OS formatter, camera formatter or whatever. YOU WILL RUIN factory format (which is crafted in best possible way) and replace it with another, very likely suboptimal one.
> 
> It's recommended by the card manufacturers to reformat it in each camera its inserted into. It's the only recommended way to "erase" the sd card for re-use, they don't recommend selectively deleting images. And it's known that one camera's partition table and formatting can irritate another camera make/model if the card isn't reformatted by that camera.
> 
It's not just cameras that have this issue, a lot of other hardware
makes stupid assumptions about the format of media.  The first firmware
release for the Nintendo Wii for example, chocked if you tried to use an
SD card with more than one partition on it, and old desktop versions of
Windows won't ever show you anything other than the first partition on
an SD card (or most USB storage devices for that matter).



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 2455 bytes --]