BTRFS disaster (of my own making). Is this recoverable?

BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

I have had bad dreams about this particular fat finger but after a few
years it has finally happened.

Scenario: 2 drives in a raw btrfs array (no previous partitions and
non-redundant) with various subvols as well. One was sdc the other was
sde, although sde never displays with mount command and blkid is the
same for both.

Thinking I was writing to a flash drive I sent 32MB via dd
===========================
dd if=file of=/dev/sde
===========================
to sde (instead of what I wanted - sdf) and now the volume nor any of
it's subvol's can mount (of course that seems entirely reasonable,
although you can imagine how unhappy I am).

With:
===========================
mount -t btrfs /mnt/butter/
===========================
I get:
===========================
[ 3421.193103] BTRFS info (device sde): disk space caching is enabled
[ 3421.193734] BTRFS (device sde): bad tree block start
8330001001141004672 20971520
[ 3421.193738] BTRFS: failed to read chunk root on sde
[ 3421.203221] BTRFS: open_ctree failed
===========================

If I specify /dev/sdc instead of relying on fstab I get:
===========================
mount -t btrfs -o degraded /dev/sdc /mnt/butter/
===========================
[ 3839.506766] BTRFS info (device sde): allowing degraded mounts
[ 3839.506769] BTRFS info (device sde): disk space caching is enabled
[ 3839.507154] BTRFS (device sde): bad tree block start
8330001001141004672 20971520
[ 3839.507159] BTRFS: failed to read chunk root on sde
[ 3839.515023] BTRFS: open_ctree failed
===========================

Obligatory details:
===========================
uname -a
Linux sartre 4.1.3-gentoo #1 SMP Sat Jul 25 22:34:14 EDT 2015 x86_64
Intel(R) Core(TM) i7-3770K CPU @ 3.50GHz GenuineIntel GNU/Linux
===========================
btrfs --version
btrfs-progs v4.1.2
===========================
btrfs fi show
Label: 'garage'  uuid: 896f615e-6437-41c6-8f2a-25f73ff1af7a
        Total devices 1 FS bytes used 89.33GiB
        devid    1 size 200.00GiB used 92.02GiB path /dev/sdb3

warning, device 2 is missing
bytenr mismatch, want=20971520, have=0
Couldn't read chunk root
Label: 'terrafirm'  uuid: 09024c28-7932-4ddb-960d-becc1ea839e5
        Total devices 2 FS bytes used 6.40TiB
        devid    1 size 3.64TiB used 3.21TiB path /dev/sdc
        *** Some devices missing

btrfs-progs v4.1.2
===========================
btrfs fi df /mnt/butter
ERROR: couldn't get space info - Inappropriate ioctl for device
ERROR: get_df failed Inappropriate ioctl for device
===========================

Please advise.

Thank you,

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Duncan]

Sonic posted on Sun, 02 Aug 2015 13:13:41 -0400 as excerpted:

> I have had bad dreams about this particular fat finger but after a few
> years it has finally happened.
> 
> Scenario: 2 drives in a raw btrfs array (no previous partitions and
> non-redundant) with various subvols as well. One was sdc the other was
> sde, although sde never displays with mount command and blkid is the
> same for both.
> 
> Thinking I was writing to a flash drive I sent 32MB via dd
> ===========================
> dd if=file of=/dev/sde
> ===========================
> to sde (instead of what I wanted - sdf) and now the volume nor any of
> it's subvol's can mount (of course that seems entirely reasonable,
> although you can imagine how unhappy I am).

> [...]

> Obligatory details:
> ===========================
> uname -a
> Linux sartre 4.1.3-gentoo [...]
> ===========================
> btrfs --version
> btrfs-progs v4.1.2
> ===========================
> btrfs fi show
> [...]
> warning, device 2 is missing
> bytenr mismatch, want=20971520, have=0
> Couldn't read chunk root
> Label: 'terrafirm'  [...]
>         Total devices 2 FS bytes used 6.40TiB
>         devid    1 size 3.64TiB used 3.21TiB path /dev/sdc
>         *** Some devices missing
> [...]

Of course the first thing is sysadmin's backups rule -- if you don't have 
it backed up, by definition and action, that data is worth less to you 
than the time/effort/media to back it up, any claims to the contrary not 
withstanding, as your actions spoke louder than your words.

And of course btrfs isn't fully stable yet, so that rule applies double, 
compared to a fully mature and stable filesystem.

Therefore, you're in luck.  Either you didn't lose anything of value 
because you had it backed up, or it wasn't of /that/ much value to you 
anyway, so again, you didn't lose anything of value. =:^)

(And actually, that's truer than you might think at the moment.  Consider 
my next-door neighbor, who a couple weeks ago escaped a fire with his 
life and the pair of shorts he was wearing.  No wallet, no keys, 
nothing.  I don't think he's going to be worrying much about that last 
fat-fingering he did on the computer for awhile!  If you have your life, 
your health, a roof over your head, and either food on the kitchen 
shelves or money to eat out with, you're a lucky one!  They say everybody 
fat-fingers at some point and has to learn the value of those backups the 
hard way.  Now you have that behind you. =:^)  I've been there too, and a 
bit of perspective really does help.)


Beyond that...

You're running current kernel and userspace/progs already, good job 
there. =:^)

The first thing you need to do in terms of trying to recover, is restore 
the superblock on the damaged device.  Since btrfs keeps multiple copies 
(up to three, once the filesystem is large enough, as yours is) per 
device, that's actually relatively easy.  Use...

btrfs rescue super-recover

... for that.  Being a gentooer (as am I), I'm confident you can read the 
manpage for the details.  =:^)  You can use...

btrfs-show-super

... to examine the various superblock copies in ordered to pick a good 
one to restore.


Beyond that, as a simple btrfs user and list regular who hasn't had that 
particular problem here (my brown-bag day was a different fat-fingering!
), not a dev, I'm a bit beyond my depth, but were it me, at that point 
I'd see what I had, then try...

btrfs restore

There's a page on the wiki with details on how to try to get it to work 
manually, if it's not going to work automatically for you:

https://btrfs.wiki.kernel.org/index.php/Restore

I've used it here (tho not in the situation you're in) when I had 
backups, but they were a bit stale.  Basically, I valued the data enough 
to have backups, but valued saving the hassle over keeping them current, 
given the amount of actually changed data.  Still, since restore worked 
for me and thus gave me the choice of somewhat stale backups, or rather 
less stale btrfs restore, I took less stale. =:^)

The nice thing about restore, other than that it often works when the 
filesystem won't mount, is that it doesn't write to the damaged 
filesystem at all, so there's no chance of making the problem worse 
instead of better.  Restore simply reads what files it can get, restoring 
them to some other (working and mounted) filesystem.  So you'll need some 
alternate place to store all those files...

Tho the wiki is somewhat behind current code, so read the manpage for 
more useful options.  In particular, the dry-run option can give you a 
good idea whether you're going to get much of anything useful with that 
generation/transid, or not, and there's now options to restore metadata 
(ownership and permissions) and symlinks, as well as the file data 
itself.  Also, where it talks about picking a tree root with as many of 
the filesystem trees as possible, use the list-roots option.  This is 
what displays all those trees it's talking about.


If restore worked for you, you have a choice.  What I've done at that 
point, since I was reasonably happy with the result as I got pretty much 
everything back using restore, is simply blow away the damaged 
filesystem, and start over, fresh mkfs.btrfs, etc.  For me, that was the 
simplest.  If it didn't work (still too damaged) or it did but you want 
to continue...

Next, if you want more than one chance at things, it's time to copy the 
filesystem's raw devices elsewhere, so you have a backup copy of the raw 
devices in case you screw up the one you're experimenting with even 
worse.  dd each filesystem component device to some other storage, for 
that raw backup.

This is where I start going beyond my depth...  If you had no backups and 
restore hasn't worked on it at this point, you may well want to wait for 
a second opinion.  If he answers, I'd put Hugo Mills' answers at 
priority, else Chris Murphy or others.

Once that's done, or if you decide to skip it and have only one go at 
things, if it were me, /I/ would try...

btrfs rescue chunk-recover

... next.  After that, hopefully at least restore will work, if it didn't 
before.   There's also some stuff possible with btrfs check, but again, 
this is getting beyond my level, and it's going pretty vague and hand-
wavy at this point.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS disaster (of my own making). Is this recoverable?

[Chris Murphy]

I can't tell what the data and metadata profile are? That it won't
mount degraded makes me think the metadata is not explicitly raid1;
and it either raid0 or accidentally single or dup which can happen at
mkfs time on single device, and just doing btrfs dev add to add
another device.

I think recovery is difficult but it depends on what sort of critical
information is in those first 32MB other than the superblock. There
are copies of the superblock so that can probably be reconstructed.


Chris Murphy

Re: BTRFS disaster (of my own making). Is this recoverable?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

On Sun, Aug 02, 2015 at 11:42:25PM -0600, Chris Murphy wrote:
> I can't tell what the data and metadata profile are? That it won't
> mount degraded makes me think the metadata is not explicitly raid1;
> and it either raid0 or accidentally single or dup which can happen at
> mkfs time on single device, and just doing btrfs dev add to add
> another device.
> 
> I think recovery is difficult but it depends on what sort of critical
> information is in those first 32MB other than the superblock. There
> are copies of the superblock so that can probably be reconstructed.

   It's probably that the small empty single chunks left behind by
mkfs are still there. I don't think we have a good solution to this
yet (other than fixing mkfs so it doesn't happen in the first place).

   Hugo.

-- 
Hugo Mills             | Our so-called leaders speak
hugo@... carfax.org.uk | with words they try to jail ya
http://carfax.org.uk/  | They subjugate the meek
PGP: E2AB1DE4          | but it's the rhetoric of failure.          The Police

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Mon, Aug 3, 2015 at 1:42 AM, Chris Murphy <lists@colorremedies.com> wrote:
> I can't tell what the data and metadata profile are? That it won't
> mount degraded makes me think the metadata is not explicitly raid1;
> and it either raid0 or accidentally single or dup which can happen at
> mkfs time on single device

It's been a long time since I created the two disk array but I'm sure
it's raid 0 and it may be single but my memory fails me on that point.

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Mon, Aug 3, 2015 at 1:17 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> The first thing you need to do in terms of trying to recover, is restore
> the superblock on the damaged device.  Since btrfs keeps multiple copies
> (up to three, once the filesystem is large enough, as yours is) per
> device, that's actually relatively easy.  Use...
>
> btrfs rescue super-recover

Not sure how to tell if there is a superblock issue:
===============================
btrfs-show-super -f /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
---------------------------------------------------------
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527267057664
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 16777216 owner 2 type SYSTEM|RAID0 num_stripes 2
                        stripe 0 devid 2 offset 1048576
                        stripe 1 devid 1 offset 20971520
backup_roots[4]:
        backup 0:
                backup_tree_root:       1517037699072   gen: 9025       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     576585728       gen: 9025       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       1517028753408   gen: 9025       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038625824768
                backup_num_devices:     2

        backup 1:
                backup_tree_root:       1517167755264   gen: 9026       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1517167771648   gen: 9026       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       2503711637504   gen: 9026       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038625824768
                backup_num_devices:     2

        backup 2:
                backup_tree_root:       980877312       gen: 9023       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1026768896      gen: 9023       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       1790377984      gen: 9023       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038617616384
                backup_num_devices:     2

        backup 3:
                backup_tree_root:       1960509440      gen: 9024       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1960525824      gen: 9024       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       2106736640      gen: 9024       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038617616384
                backup_num_devices:     2

btrfs-show-super -f /dev/sde
superblock: bytenr=65536, device=/dev/sde
---------------------------------------------------------
csum                    0x9634c164 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
chunk_root_generation   8957
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             8001574060032
bytes_used              7038625824768
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             2
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x21
                        ( MIXED_BACKREF |
                          BIG_METADATA )
csum_type               0
csum_size               4
cache_generation        9026
uuid_tree_generation    9026
dev_item.uuid           d63b1c46-b5a2-49ae-861f-804b2f97f427
dev_item.fsid           09024c28-7932-4ddb-960d-becc1ea839e5 [match]
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527250280448
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
sys_chunk_array[2048]:
        item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 20971520)
                chunk length 16777216 owner 2 type SYSTEM|RAID0 num_stripes 2
                        stripe 0 devid 2 offset 1048576
                        stripe 1 devid 1 offset 20971520
backup_roots[4]:
        backup 0:
                backup_tree_root:       1517037699072   gen: 9025       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     576585728       gen: 9025       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       1517028753408   gen: 9025       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038625824768
                backup_num_devices:     2

        backup 1:
                backup_tree_root:       1517167755264   gen: 9026       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1517167771648   gen: 9026       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       2503711637504   gen: 9026       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038625824768
                backup_num_devices:     2

        backup 2:
                backup_tree_root:       980877312       gen: 9023       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1026768896      gen: 9023       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       1790377984      gen: 9023       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038617616384
                backup_num_devices:     2

        backup 3:
                backup_tree_root:       1960509440      gen: 9024       level: 1
                backup_chunk_root:      20971520        gen: 8957       level: 1
                backup_extent_root:     1960525824      gen: 9024       level: 2
                backup_fs_root:         2056568832      gen: 1106       level: 0
                backup_dev_root:        52576256        gen: 9021       level: 1
                backup_csum_root:       2106736640      gen: 9024       level: 3
                backup_total_bytes:     8001574060032
                backup_bytes_used:      7038617616384
                backup_num_devices:     2

btrfs-show-super -a /dev/sdc
superblock: bytenr=65536, device=/dev/sdc
---------------------------------------------------------
csum                    0x510e90f6 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
chunk_root_generation   8957
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             8001574060032
bytes_used              7038625824768
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             2
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x21
                        ( MIXED_BACKREF |
                          BIG_METADATA )
csum_type               0
csum_size               4
cache_generation        9026
uuid_tree_generation    9026
dev_item.uuid           224c95e6-d71b-494a-a4f6-cc72a84f1e89
dev_item.fsid           09024c28-7932-4ddb-960d-becc1ea839e5 [match]
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527267057664
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

superblock: bytenr=67108864, device=/dev/sdc
---------------------------------------------------------
csum                    0xf16fb838 [match]
bytenr                  67108864
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527267057664
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

superblock: bytenr=274877906944, device=/dev/sdc
---------------------------------------------------------
csum                    0x0ce8ee09 [match]
bytenr                  274877906944
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
chunk_root_generation   8957
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             8001574060032
bytes_used              7038625824768
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             2
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x21
                        ( MIXED_BACKREF |
                          BIG_METADATA )
csum_type               0
csum_size               4
cache_generation        9026
uuid_tree_generation    9026
dev_item.uuid           224c95e6-d71b-494a-a4f6-cc72a84f1e89
dev_item.fsid           09024c28-7932-4ddb-960d-becc1ea839e5 [match]
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527267057664
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          1
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

btrfs-show-super -a /dev/sde
superblock: bytenr=65536, device=/dev/sde
---------------------------------------------------------
csum                    0x9634c164 [match]
bytenr                  65536
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
chunk_root_generation   8957
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             8001574060032
bytes_used              7038625824768
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             2
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x21
                        ( MIXED_BACKREF |
                          BIG_METADATA )
csum_type               0
csum_size               4
cache_generation        9026
uuid_tree_generation    9026
dev_item.uuid           d63b1c46-b5a2-49ae-861f-804b2f97f427
dev_item.fsid           09024c28-7932-4ddb-960d-becc1ea839e5 [match]
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527250280448
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

superblock: bytenr=67108864, device=/dev/sde
---------------------------------------------------------
csum                    0x3655e9aa [match]
bytenr                  67108864
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527250280448
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0

superblock: bytenr=274877906944, device=/dev/sde
---------------------------------------------------------
csum                    0xcbd2bf9b [match]
bytenr                  274877906944
flags                   0x1
                        ( WRITTEN )
magic                   _BHRfS_M [match]
fsid                    09024c28-7932-4ddb-960d-becc1ea839e5
label                   terrafirm
generation              9026
root                    1517167755264
sys_array_size          129
chunk_root_generation   8957
root_level              1
chunk_root              20971520
chunk_root_level        1
log_root                0
log_root_transid        0
log_root_level          0
total_bytes             8001574060032
bytes_used              7038625824768
sectorsize              4096
nodesize                16384
leafsize                16384
stripesize              4096
root_dir                6
num_devices             2
compat_flags            0x0
compat_ro_flags         0x0
incompat_flags          0x21
                        ( MIXED_BACKREF |
                          BIG_METADATA )
csum_type               0
csum_size               4
cache_generation        9026
uuid_tree_generation    9026
dev_item.uuid           d63b1c46-b5a2-49ae-861f-804b2f97f427
dev_item.fsid           09024c28-7932-4ddb-960d-becc1ea839e5 [match]
dev_item.type           0
dev_item.total_bytes    4000787030016
dev_item.bytes_used     3527250280448
dev_item.io_align       4096
dev_item.io_width       4096
dev_item.sector_size    4096
dev_item.devid          2
dev_item.dev_group      0
dev_item.seek_speed     0
dev_item.bandwidth      0
dev_item.generation     0
===============================

There's some difference between the output of /dev/sdc and /dev/sde
but they may be completely normal.
How does one determine from the above whether or not a superblock is bad?

Is "btrfs rescue super-recover" safe to run? IOW, will it ask before
doing anything possibly destructive (assuming I don't give it a -y)?

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Mon, Aug 3, 2015 at 10:43 AM, Sonic <sonicsmith@gmail.com> wrote:
> Is "btrfs rescue super-recover" safe to run? IOW, will it ask before
> doing anything possibly destructive (assuming I don't give it a -y)?

Seemed a bit safe so I went for it and:

sartre ~ # btrfs rescue super-recover /dev/sdc
All supers are valid, no need to recover
sartre ~ # btrfs rescue super-recover /dev/sde
All supers are valid, no need to recover

So it may not be a superblock issue.

>From the dmesg earlier:
[ 3421.193734] BTRFS (device sde): bad tree block start
8330001001141004672 20971520
[ 3421.193738] BTRFS: failed to read chunk root on sde
[ 3421.203221] BTRFS: open_ctree failed

I may need a chunk-recover and also wonder if zero-log is advisable.
Any ideas in those directions?

Thanks,

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

On Mon, Aug 03, 2015 at 11:05:46AM -0400, Sonic wrote:
> On Mon, Aug 3, 2015 at 10:43 AM, Sonic <sonicsmith@gmail.com> wrote:
> > Is "btrfs rescue super-recover" safe to run? IOW, will it ask before
> > doing anything possibly destructive (assuming I don't give it a -y)?
> 
> Seemed a bit safe so I went for it and:
> 
> sartre ~ # btrfs rescue super-recover /dev/sdc
> All supers are valid, no need to recover
> sartre ~ # btrfs rescue super-recover /dev/sde
> All supers are valid, no need to recover
> 
> So it may not be a superblock issue.
> 
> From the dmesg earlier:
> [ 3421.193734] BTRFS (device sde): bad tree block start
> 8330001001141004672 20971520
> [ 3421.193738] BTRFS: failed to read chunk root on sde
> [ 3421.203221] BTRFS: open_ctree failed
> 
> I may need a chunk-recover and also wonder if zero-log is advisable.
> Any ideas in those directions?

   Very unlikely and definitely not, respectively. There's nothing at
all here to indicate that you've got a broken log, so dropping it
would be at best pointless. The chunk tree is also most likely
undamaged on both copies.

   Hugo.

-- 
Hugo Mills             | The early bird gets the worm, but the second mouse
hugo@... carfax.org.uk | gets the cheese.
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Mon, Aug 3, 2015 at 11:28 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
>    Very unlikely and definitely not, respectively. There's nothing at
> all here to indicate that you've got a broken log, so dropping it
> would be at best pointless. The chunk tree is also most likely
> undamaged on both copies.

Will it hurt to try a chunk-recover? I'm guessing it wont do anything
unless I answer Y to some prompt.

Is there anything else you would suggest?

Thanks,

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

Also tried:
mount -o recovery,ro /mnt/butter/

And dmesg gives:
[88228.756622] BTRFS info (device sde): enabling auto recovery
[88228.756635] BTRFS info (device sde): disk space caching is enabled
[88228.757244] BTRFS (device sde): bad tree block start
8330001001141004672 20971520
[88228.757248] BTRFS: failed to read chunk root on sde
[88228.769877] BTRFS: open_ctree failed


On Mon, Aug 3, 2015 at 12:03 PM, Sonic <sonicsmith@gmail.com> wrote:
> On Mon, Aug 3, 2015 at 11:28 AM, Hugo Mills <hugo@carfax.org.uk> wrote:
>>    Very unlikely and definitely not, respectively. There's nothing at
>> all here to indicate that you've got a broken log, so dropping it
>> would be at best pointless. The chunk tree is also most likely
>> undamaged on both copies.
>
> Will it hurt to try a chunk-recover? I'm guessing it wont do anything
> unless I answer Y to some prompt.
>
> Is there anything else you would suggest?
>
> Thanks,
>
> Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

Output of btrfs check:

btrfs check /dev/sdc
warning, device 2 is missing
bytenr mismatch, want=20971520, have=0
Couldn't read chunk root
Couldn't open file system

btrfs check /dev/sde
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
bytenr mismatch, want=20971520, have=8330001001141004672
Couldn't read chunk root
Couldn't open file system

In case the above helps.

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

Are either one of these called for?

btrfs check --repair
or
btrfs check --repair --init-csum-tree

Seems like they might be a last ditch attempt. Is one preferred over the other?

Is:
btrfs rescue chunk-recover
a much less dangerous attempt (IOW it wont hurt to try it first)?

Thanks,

Chris


On Mon, Aug 3, 2015 at 12:22 PM, Sonic <sonicsmith@gmail.com> wrote:
> Output of btrfs check:
>
> btrfs check /dev/sdc
> warning, device 2 is missing
> bytenr mismatch, want=20971520, have=0
> Couldn't read chunk root
> Couldn't open file system
>
> btrfs check /dev/sde
> checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
> checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
> bytenr mismatch, want=20971520, have=8330001001141004672
> Couldn't read chunk root
> Couldn't open file system
>
> In case the above helps.

Re: BTRFS disaster (of my own making). Is this recoverable?

[Duncan]

Sonic posted on Mon, 03 Aug 2015 12:32:21 -0400 as excerpted:

> Are either one of these called for?
> 
> btrfs check --repair or btrfs check --repair --init-csum-tree
> 
> Seems like they might be a last ditch attempt. Is one preferred over the
> other?

The read-only check (without --repair) couldn't read the chunk tree, so 
check --repair probably should run into the same issue and not be able to 
do anything as well.

check --init-csum-tree reinitializes the checksums, but to do that you 
have to have something to reinitialize on, so that's unlikely to do 
anything, either.  (I did have that in mind as a possibility after the 
superblock recovery and chunk-recover operations, tho, if necessary.  
That's why I mentioned check at the end.  But the message was long enough 
already and that would have been getting ahead of things, so...)

> Is:
> btrfs rescue chunk-recover a much less dangerous attempt (IOW it wont
> hurt to try it first)?

I'd not call it less dangerous, but given that the chunk tree can't be 
read, you're already dealing with dangerous.  That's what I'd try here 
(tho Hugo's suggesting it won't help has me doubting, but that's where 
things do point from what I see).

Which is why I mentioned doing a raw device backup, before attempting 
it.  Chunk-recover is pretty invasive, and if it doesn't work, it could 
make the filesystem unrecoverable.  It's basically a one-shot deal.  A 
raw device backup can be used to avoid it being one-shot, but of course 
that does require that you have enough devices of sufficient size to 
backup to, which certainly in the terabyte range, is a potential problem 
unless you're a big-budget corporate user with a stack of spare devices 
on hand.

I believe Hugo's right on zero-log, tho.  That's a fix for a very 
specific situation, seen most often due to a specific bug that was short 
lived and has been fixed for quite some time, now, tho in certain very 
narrow post-crash situations the same fix has been known to work too.  
There's no evidence you're anywhere near that very specific situation, so 
zero-log's just not the tool for this job.  And it could make things 
slightly worse, too, tho in theory at least, all you're doing is cutting 
off the last 30 seconds or so of activity, so the chances of it doing 
major harm are small, unless you were actively rebalancing or something 
when the filesystem was last unmounted (gracefully or not).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Mon, Aug 3, 2015 at 12:32 PM, Sonic <sonicsmith@gmail.com> wrote:
> btrfs rescue chunk-recover

Ran this:
btrfs rescue chunk-recover -v /dev/sde |tee brcr.txt

Got this (very end of output):
==================================
Unrecoverable Chunks:

Total Chunks:           3292
  Recoverable:          3292
  Unrecoverable:        0

Orphan Block Groups:

Orphan Device Extents:

Fail to recover the chunk tree.
==================================
If earlier output of this process might be helpful I can provide it
(used tee to create a file).

Didn't work - seems all chunks found were recoverable - it's
apparently the missing chunks that are the problem.

I'm thinking this is a lost cause. Also thinking I need to provide for
some redundancy in the future :-)

Any other suggestions before I give up the ghost on this?

Thanks,

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

Out of frustration and vodka I tried:

btrfs check --repair /dev/sde

To instantly be met with:

enabling repair mode
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
bytenr mismatch, want=20971520, have=8330001001141004672
Couldn't read chunk root
Couldn't open file system



On Mon, Aug 3, 2015 at 10:20 PM, Sonic <sonicsmith@gmail.com> wrote:
> On Mon, Aug 3, 2015 at 12:32 PM, Sonic <sonicsmith@gmail.com> wrote:
>> btrfs rescue chunk-recover
>
> Ran this:
> btrfs rescue chunk-recover -v /dev/sde |tee brcr.txt
>
> Got this (very end of output):
> ==================================
> Unrecoverable Chunks:
>
> Total Chunks:           3292
>   Recoverable:          3292
>   Unrecoverable:        0
>
> Orphan Block Groups:
>
> Orphan Device Extents:
>
> Fail to recover the chunk tree.
> ==================================
> If earlier output of this process might be helpful I can provide it
> (used tee to create a file).
>
> Didn't work - seems all chunks found were recoverable - it's
> apparently the missing chunks that are the problem.
>
> I'm thinking this is a lost cause. Also thinking I need to provide for
> some redundancy in the future :-)
>
> Any other suggestions before I give up the ghost on this?
>
> Thanks,
>
> Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Duncan]

Sonic posted on Mon, 03 Aug 2015 22:20:38 -0400 as excerpted:

> I'm thinking this is a lost cause. Also thinking I need to provide for
> some redundancy in the future :-)
> 
> Any other suggestions before I give up the ghost on this?

I don't remember you saying anything about trying btrfs restore.  I'm not 
sure it can do anything for this case, and it definitely requires enough 
additional space on some other filesystem to store the data it restores, 
but if you had stuff on there you'd really like to get back, it's worth a 
shot, and if you're going to be backing up in the future, you'll need the 
additional devices for the backup in any case, so might as well get 'em 
now, mkfs them to whatever (doesn't have to be btrfs since all you're 
doing is using it as a place to store the files, it's the one you're 
restoring /from/ that's btrfs), mount 'em, and try restoring to them.

Worst-case, it doesn't work and you have your set of backup devices 
already prepped and ready to go.  Best-case, it recovers pretty much 
everything.

But I am guessing restore won't be able to do anything on it's own, 
you'll need to try btrfs-find-root, and feed the bytenr from roots it 
finds into restore until you get something useful.  Take a look at the 
link and brief instructions I posted in my first reply.

Of course, if btrfs-find-root doesn't show you any roots (these are the 
filesystem master roots), or if fed those roots using -t <bytenr>, btrfs 
restore --list-roots (these are the various subtree roots) doesn't give 
you anything worthwhile, then it /would/ appear you're looking at a lost-
cause... unless anyone else has a bright idea at this point.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Tue, Aug 4, 2015 at 1:10 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> I don't remember you saying anything about trying btrfs restore.

I did try it earlier in dry-run mode and it can't do anything:
=========================================
btrfs restore -D /dev/sdc /mnt/saved/
warning, device 2 is missing
bytenr mismatch, want=20971520, have=0
Couldn't read chunk root
Could not open root, trying backup super
warning, device 2 is missing
bytenr mismatch, want=20971520, have=0
Couldn't read chunk root
Could not open root, trying backup super
warning, device 2 is missing
bytenr mismatch, want=20971520, have=0
Couldn't read chunk root
Could not open root, trying backup super
=========================================

Find-roots, list-roots, none of it works.
=========================================
btrfs restore --list-roots /dev/sde
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
bytenr mismatch, want=20971520, have=8330001001141004672
Couldn't read chunk root
Could not open root, trying backup super
warning, device 1 is missing
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
bytenr mismatch, want=20971520, have=8330001001141004672
Couldn't read chunk root
Could not open root, trying backup super
warning, device 1 is missing
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
checksum verify failed on 20971520 found 8B1D9672 wanted 2F8A4238
bytenr mismatch, want=20971520, have=8330001001141004672
Couldn't read chunk root
Could not open root, trying backup super
=========================================
btrfs-find-root -a /dev/sdc
warning, device 2 is missing
Couldn't read chunk root

btrfs-find-root -a /dev/sde
Couldn't read chunk root
Open ctree failed
=========================================

In defense of BTRFS it's been absolutely trouble free keeping my data
through many power outages, etc. and I plan to keep using it. It was
my own doing that caused the problem.

Seems that if there was someway to edit something in those first
overwritten 32MB of disc 2 to say "hey, I'm really here, just a bit
screwed up" maybe some of the recovery tools could actually work.

Thanks to all.

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Tue, Aug 4, 2015 at 4:23 PM, Sonic <sonicsmith@gmail.com> wrote:
> Seems that if there was someway to edit something in those first
> overwritten 32MB of disc 2 to say "hey, I'm really here, just a bit
> screwed up" maybe some of the recovery tools could actually work.

Just want to reiterate this thought.

The basic error in most cases with the tools at hand is that Disc 2 is
missing so there's little the tools can do. Somewhere in those first
32MB should be something to properly identify the disc as part of the
array.

If the btrfs tools can't fix it maybe dd can. Is there anything can be
gained from the beginning of disc 1 (can dd this to a file) in order
to create the necessary bits needed at the beginning of disc2? Or some
other way to overwrite the beginning of disc 2 (using dd again) with
some identification information so that the automated btrfs tools can
take it from there?

Thanks,

Chris

Re: BTRFS disaster (of my own making). Is this recoverable?

[Sonic]

On Wed, Aug 5, 2015 at 8:31 AM, Sonic <sonicsmith@gmail.com> wrote:
> The basic error in most cases with the tools at hand is that Disc 2 is
> missing so there's little the tools can do. Somewhere in those first
> 32MB should be something to properly identify the disc as part of the
> array.

Somehow manually create the missing "chunk root" if this is the core problem??

Re: BTRFS disaster (of my own making). Is this recoverable?

[Chris Murphy]

On Wed, Aug 5, 2015 at 6:31 AM, Sonic <sonicsmith@gmail.com> wrote:
> On Tue, Aug 4, 2015 at 4:23 PM, Sonic <sonicsmith@gmail.com> wrote:
>> Seems that if there was someway to edit something in those first
>> overwritten 32MB of disc 2 to say "hey, I'm really here, just a bit
>> screwed up" maybe some of the recovery tools could actually work.
>
> Just want to reiterate this thought.
>
> The basic error in most cases with the tools at hand is that Disc 2 is
> missing so there's little the tools can do. Somewhere in those first
> 32MB should be something to properly identify the disc as part of the
> array.

Yes but it was probably  uniquely only on that disk, because there's
no redundancy for metadata or system chunks. Therefore there's no copy
on the other disk to use as a model. The btrfs check command has an
option to use other superblocks, so you could try that switch and see
if it makes a difference but it sounds like it's finding backup
superblocks automatically. That's the one thing that is pretty much
always duplicated on the same disk; for sure the first superblock is
munged and would need repair. But there's still other chunks
missing... so I don't think it'll help.


>
> If the btrfs tools can't fix it maybe dd can. Is there anything can be
> gained from the beginning of disc 1 (can dd this to a file) in order
> to create the necessary bits needed at the beginning of disc2?

Not if there's no metadata or system redundancy profile like raid1.

> Or some
> other way to overwrite the beginning of disc 2 (using dd again) with
> some identification information so that the automated btrfs tools can
> take it from there?

I think to have a viable reference, you need two disks (virtual or
real) and you need to exactly replicate how you got to this two disk
setup to find out what's in those 32MB that might get the file system
to mount even if it complaints of some corrupt files. That's work
that's way beyond my skill level. The tools don't do this right now as
far as I'm aware. You'd be making byte by byte insertions to multiple
sectors. Tedious. But I can't even guess how many steps it is. It
might be 10. It might be 10000.


-- 
Chris Murphy

RE: BTRFS disaster (of my own making). Is this recoverable?

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2004 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Chris Murphy
> Sent: Thursday, 6 August 2015 2:54 AM
> To: Sonic <sonicsmith@gmail.com>
> Cc: Btrfs BTRFS <linux-btrfs@vger.kernel.org>; Hugo Mills
> <hugo@carfax.org.uk>
> Subject: Re: BTRFS disaster (of my own making). Is this recoverable?
> 
> On Wed, Aug 5, 2015 at 6:31 AM, Sonic <sonicsmith@gmail.com> wrote:
> > On Tue, Aug 4, 2015 at 4:23 PM, Sonic <sonicsmith@gmail.com> wrote:
> >> Seems that if there was someway to edit something in those first
> >> overwritten 32MB of disc 2 to say "hey, I'm really here, just a bit
> >> screwed up" maybe some of the recovery tools could actually work.
> >
> > Just want to reiterate this thought.
> >
> > The basic error in most cases with the tools at hand is that Disc 2 is
> > missing so there's little the tools can do. Somewhere in those first
> > 32MB should be something to properly identify the disc as part of the
> > array.
> 
> Yes but it was probably  uniquely only on that disk, because there's no
> redundancy for metadata or system chunks. Therefore there's no copy on
> the other disk to use as a model. The btrfs check command has an option to
> use other superblocks, so you could try that switch and see if it makes a
> difference but it sounds like it's finding backup superblocks automatically.
> That's the one thing that is pretty much always duplicated on the same disk;
> for sure the first superblock is munged and would need repair. But there's
> still other chunks missing... so I don't think it'll help.

Would it be possible to store this type of critical information twice on each disk, at the beginning and end? I thought BTRFS already did that, but I might be thinking of some other filesystem. I've had my share of these types of oops! moments as well.
 


Paul.

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: BTRFS disaster (of my own making). Is this recoverable?

[Chris Murphy]

On Wed, Aug 5, 2015 at 6:45 PM, Paul Jones <paul@pauljones.id.au> wrote:
> Would it be possible to store this type of critical information twice on each disk, at the beginning and end? I thought BTRFS already did that, but I might be thinking of some other filesystem. I've had my share of these types of oops! moments as well.

That option is metadata profile raid1. To do an automatic
-mconvert=raid1 when the user does 'btrfs device add' breaks any use
case where you want to temporarily add a small device, maybe a USB
stick, and now hundreds of MiBs possibly GiBs of metadata have to be
copied over to this device without warning. It could be made smart,
autoconvert to raid1 when the added device is at least 4x the size of
metadata allocation, but then that makes it inconsistent. OK so it
could be made interactive, but now that breaks scripts. So... where do
you draw the line?

Maybe this would work if the system chunk only was raid1? I don't know
what the minimum necessary information is for such a case.

Possibly it make more sense if 'btrfs device add' always does
-dconvert=raid1 unless a --quick option is passed?

-- 
Chris Murphy

Re: BTRFS disaster (of my own making). Is this recoverable?

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

On 2015-08-05 22:13, Chris Murphy wrote:
> On Wed, Aug 5, 2015 at 6:45 PM, Paul Jones <paul@pauljones.id.au> wrote:
>> Would it be possible to store this type of critical information twice on each disk, at the beginning and end? I thought BTRFS already did that, but I might be thinking of some other filesystem. I've had my share of these types of oops! moments as well.
>
> That option is metadata profile raid1. To do an automatic
> -mconvert=raid1 when the user does 'btrfs device add' breaks any use
> case where you want to temporarily add a small device, maybe a USB
> stick, and now hundreds of MiBs possibly GiBs of metadata have to be
> copied over to this device without warning. It could be made smart,
> autoconvert to raid1 when the added device is at least 4x the size of
> metadata allocation, but then that makes it inconsistent. OK so it
> could be made interactive, but now that breaks scripts. So... where do
> you draw the line?
>
> Maybe this would work if the system chunk only was raid1? I don't know
> what the minimum necessary information is for such a case.
>
> Possibly it make more sense if 'btrfs device add' always does
> -dconvert=raid1 unless a --quick option is passed?
>
Perhaps we could print out a big noisy warning that could be silenced?


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]