Unrecoverable errors when the btrfs file system was modified outside the running OS

Unrecoverable errors when the btrfs file system was modified outside the running OS

[rocwhite168]

Hello,

I just had a very frustrating experience with btrfs, which I was only
able to resolve by rolling back to ext4 using the subvol btrfs-convert
created. The same type of situation occurred before when I was using
the ext file system and the result was far less disastrous.

The source of problem came from the fact that I have a Windows and
Ubuntu 14.04 dual-boot setup and within Windows I also use VirtualBox
to run the same Ubuntu with rawdisk. Today I updated Ubuntu to a new
kernel within VirtualBox. At this point, I would usually shutdown
VirtualBox before I let my machine go to hibernation. However, this
time I forgot. And when the machine started up, it went directly into
Ubuntu (because grub was updated and to avoid issues my VirtualBox
setup didn't allow Ubuntu to see my Windows partitions). I did a
grub-udpate, and rebooted back to Windows, where my VirtualBox was
still up and running fine. The tragedy happened when I now shut down
Ubuntu and VirtualBox. The btrfs file system was totally corrupted. I
tried various combinations ro, recovery, nospace_cache, and
clear_cache mount options, and it wouldn't mount. dmesg showed some
"transid verify failed", "open_ctree failed" error messages. btrfs
restore only retrieved three files.. btrfsck --repair and
btrfs-zero-log didn't help either.

To my very surprise, btrfs-convert -r was able to use the subvol it
created to roll back to ext4. But had I not converted from ext4 to
btrfs, this would be an unrecoverable situation. Whether or not I have
backups is a separate issue, being able to recover at least "somewhat"
in this situation seems to be a desired feature for any file system.

Roc

Re: Unrecoverable errors when the btrfs file system was modified outside the running OS

[Chris Murphy]

On Aug 4, 2014, at 12:29 AM, rocwhite168 <rocwhite168@163.com> wrote:

> Hello,
> 
> I just had a very frustrating experience with btrfs, which I was only
> able to resolve by rolling back to ext4 using the subvol btrfs-convert
> created. The same type of situation occurred before when I was using
> the ext file system and the result was far less disastrous.

The recoverability of a filesystem after it's been used simultaneously by two computers can't be a useful metric. This is so highly non-deterministic I just don't buy the one off comparison of two filesystems' survival rates.

This is sort of in the realm of if you're going to slam doors, make really certain first there aren't any fingers near the door frame.



Chris Murphy

Re: Unrecoverable errors when the btrfs file system was modified outside the running OS

[Justus Seifert]

[-- Attachment #1: Type: text/plain, Size: 3432 bytes --]

On Monday 04 August 2014 01:29:49 you wrote:
> Hello,
> 
> I just had a very frustrating experience with btrfs, which I was only
> able to resolve by rolling back to ext4 using the subvol btrfs-convert
> created. The same type of situation occurred before when I was using
> the ext file system and the result was far less disastrous.
> 
> The source of problem came from the fact that I have a Windows and
> Ubuntu 14.04 dual-boot setup and within Windows I also use VirtualBox
> to run the same Ubuntu with rawdisk. Today I updated Ubuntu to a new
> kernel within VirtualBox. At this point, I would usually shutdown
> VirtualBox before I let my machine go to hibernation. However, this
> time I forgot. And when the machine started up, it went directly into
> Ubuntu (because grub was updated and to avoid issues my VirtualBox
> setup didn't allow Ubuntu to see my Windows partitions). I did a
> grub-udpate, and rebooted back to Windows, where my VirtualBox was
> still up and running fine. The tragedy happened when I now shut down
> Ubuntu and VirtualBox. The btrfs file system was totally corrupted. I
> tried various combinations ro, recovery, nospace_cache, and
> clear_cache mount options, and it wouldn't mount. dmesg showed some
> "transid verify failed", "open_ctree failed" error messages. btrfs
> restore only retrieved three files.. btrfsck --repair and
> btrfs-zero-log didn't help either.
> 
> To my very surprise, btrfs-convert -r was able to use the subvol it
> created to roll back to ext4. But had I not converted from ext4 to
> btrfs, this would be an unrecoverable situation. Whether or not I have
> backups is a separate issue, being able to recover at least "somewhat"
> in this situation seems to be a desired feature for any file system.
> 
> Roc


btrfs is CoW
stale metadata means stale block info
stale block info means no idea where the files are on disk and no idea where 
free blocks are
btrfs does not know, that it does not know, where to read and where to write, 
it assumes the metadata cached in ram is correct and just reads and writes to 
the disk where it sees fit.
this is disastrous if the location of files changes
cow means the location of files and metadata changes with _every single write_ 
!

ext4 is not CoW
if you do not delete, grow, shrink or create files – only inplace edits or 
small appeds that do not allocate aditional blocks, then the location of the 
files on disk does not change.
this is often the case for most files and metadata on a root filesystem.
thus it is no surprise at all that the consequences of using stale metadata 
for a filesystem that was mounted elsewhere differs dramatically between ext4 
and btrfs.

please keep in mind that all filesystems assume that the ram is a safe place to 
keep metadataduring runtime,  since reading all the required metadata all the 
time would be a huge performance brake for modern filesystems.  old filesystems 
might be different.

if you want the metadata not to change then try a read-only filesystem for root 
for a while.  squashfs is the default root fs for openwrt and is ro.

sry kmail ate my draft for this mail once and i was too lazy to retype it 
completely thus only simple grammar and no spell checks.

if you are interested in more details, ask detailed questions.  duncan might 
also help you if you ask him.

regards
justus seifert

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Unrecoverable errors when the btrfs file system was modified outside the running OS

[Chris Samuel]

On Mon, 4 Aug 2014 01:29:49 AM rocwhite168 wrote:

> being able to recover at least "somewhat"
> in this situation seems to be a desired feature for any file system.

I would add that only specially designed clustered/distributed filesystems will 
survive this sort of accident.

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC