read block failed check_tree_block / Couldn't read chunk tree

read block failed check_tree_block / Couldn't read chunk tree

[Rene Thomas]

Dear Developers / Maintainer,
I’ve set up a running RAID5 with three devices (sda1 / sdb1 /sdc1)

Mountpoint was /home, filesystem was mounted
A chain of unfortunal circumstances gives me the chance to run as root
in a terminal.

mkfs.ntfs /dev/sdb1

where /dev/sdb1 is device 1 in the Array.

After a while I’ve realised that the terminal I used was the wrong one
and killed the process where the progress still was 0%.
The filesystem was still in a readable state switched to read only.
Write access was not able.

After a restart the RAID failed to mount.

Tried several steps to reactivate the Array.

All btrfs commands leads to:

Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=65536
Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=0
read block failed check_tree_block
Couldn't read chunk tree

Started with:
btrfs fi show
mount -o revovery,ro /dev/sdc1 /home
failed with error (see dmesg log at [311])
btrfs-image -c 9 -t 8 /dev/sdb1 /media/storageplace/fs_image
btrfs-zero-log /dev/sdb1
btrfs check /dev/sdb1
btrfs check --repair /dev/sdb1

At least I tried
btrfs rescue chunk-recover /dev/sdb1

Works for several hours on the arry and end up in (huge log available > 100k):

“Fail to recover the chunk tree.”

 I’ve checked the super block there are no errors

My question is there any chance on a broken c tree to get the data
back? Not necessarily fix the Array only get the data.

Thanks in advanced

# uname -a
Linux engelserver 3.17.0-031700rc5-generic #201409151105 SMP Mon Sep
15 15:08:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

 # btrfs --version
Btrfs v3.17

 # btrfs fi show
Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
        Total devices 1 FS bytes used 2.36MiB
        devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1

 Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=65536
Check tree block failed, want=5845480062976, have=0
Check tree block failed, want=5845480062976, have=0
read block failed check_tree_block
Couldn't read chunk tree

Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
        Total devices 3 FS bytes used 3.02TiB
        devid    1 size 2.73TiB used 1.77TiB path /dev/sdb1
        devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
        devid    3 size 2.73TiB used 1.77TiB path /dev/sdc1
Btrfs v3.17

dmesg:
[  280.637338] init: plymouth-upstart-bridge main process ended, respawning
[  311.298682] BTRFS info (device sdc1): disk space caching is enabled
[  311.335674] BTRFS: bad tree block start 0 5845480062976
[  311.336215] BTRFS: bad tree block start 0 5845480062976
[  311.354660] BTRFS: bad tree block start 0 5845480054784
[  311.354762] BTRFS: bad tree block start 0 5845480054784
[  311.355024] BTRFS: bad tree block start 0 5845480067072
[  311.355145] BTRFS: bad tree block start 0 5845480067072
[  311.355496] BTRFS: bad tree block start 0 5845480050688
[  311.355612] BTRFS: bad tree block start 0 5845480050688
[  311.356347] BTRFS: bad tree block start 0 5845480071168
[  311.356458] BTRFS: bad tree block start 0 5845480071168
[  311.580203] BTRFS: Failed to read block groups: -5
[  311.605073] BTRFS: open_ctree failed

René

Re: read block failed check_tree_block / Couldn't read chunk tree

[Qu Wenruo]

-------- Original Message --------
Subject: read block failed check_tree_block / Couldn't read chunk tree
From: Rene Thomas <re.thomas@gmx.de>
To: <linux-btrfs@vger.kernel.org>
Date: 2014年10月25日 00:43
> Dear Developers / Maintainer,
> I’ve set up a running RAID5 with three devices (sda1 / sdb1 /sdc1)
>
> Mountpoint was /home, filesystem was mounted
> A chain of unfortunal circumstances gives me the chance to run as root
> in a terminal.
>
> mkfs.ntfs /dev/sdb1
>
> where /dev/sdb1 is device 1 in the Array.
>
> After a while I’ve realised that the terminal I used was the wrong one
> and killed the process where the progress still was 0%.
> The filesystem was still in a readable state switched to read only.
> Write access was not able.
>
> After a restart the RAID failed to mount.
>
> Tried several steps to reactivate the Array.
>
> All btrfs commands leads to:
>
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> read block failed check_tree_block
> Couldn't read chunk tree
Chunk tree corruption will always lead to disaster :(

What about remove sdb1 and mount sda1/sdc1 with -o degraded ?
I hope the sda1/sdc1 is still in correct status...
>
> Started with:
> btrfs fi show
> mount -o revovery,ro /dev/sdc1 /home
> failed with error (see dmesg log at [311])
> btrfs-image -c 9 -t 8 /dev/sdb1 /media/storageplace/fs_image
> btrfs-zero-log /dev/sdb1
> btrfs check /dev/sdb1
> btrfs check --repair /dev/sdb1
>
> At least I tried
> btrfs rescue chunk-recover /dev/sdb1
>
> Works for several hours on the arry and end up in (huge log available > 100k):
>
> “Fail to recover the chunk tree.”
If my previous defraded doesn't help, chunk-recover is the last chance 
for it.
If still failed, the chance to recover seems quite low.

However, can you upload the huge log for developers to investigate and 
improve the chunk-recover tools?

Thanks,
Qu
>
>   I’ve checked the super block there are no errors
>
> My question is there any chance on a broken c tree to get the data
> back? Not necessarily fix the Array only get the data.
>
> Thanks in advanced
>
> # uname -a
> Linux engelserver 3.17.0-031700rc5-generic #201409151105 SMP Mon Sep
> 15 15:08:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
>   # btrfs --version
> Btrfs v3.17
>
>   # btrfs fi show
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>          Total devices 1 FS bytes used 2.36MiB
>          devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>
>   Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> read block failed check_tree_block
> Couldn't read chunk tree
>
> Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
>          Total devices 3 FS bytes used 3.02TiB
>          devid    1 size 2.73TiB used 1.77TiB path /dev/sdb1
>          devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
>          devid    3 size 2.73TiB used 1.77TiB path /dev/sdc1
> Btrfs v3.17
>
> dmesg:
> [  280.637338] init: plymouth-upstart-bridge main process ended, respawning
> [  311.298682] BTRFS info (device sdc1): disk space caching is enabled
> [  311.335674] BTRFS: bad tree block start 0 5845480062976
> [  311.336215] BTRFS: bad tree block start 0 5845480062976
> [  311.354660] BTRFS: bad tree block start 0 5845480054784
> [  311.354762] BTRFS: bad tree block start 0 5845480054784
> [  311.355024] BTRFS: bad tree block start 0 5845480067072
> [  311.355145] BTRFS: bad tree block start 0 5845480067072
> [  311.355496] BTRFS: bad tree block start 0 5845480050688
> [  311.355612] BTRFS: bad tree block start 0 5845480050688
> [  311.356347] BTRFS: bad tree block start 0 5845480071168
> [  311.356458] BTRFS: bad tree block start 0 5845480071168
> [  311.580203] BTRFS: Failed to read block groups: -5
> [  311.605073] BTRFS: open_ctree failed
>
> René
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: read block failed check_tree_block / Couldn't read chunk tree

[Qu Wenruo]

-------- Original Message --------
Subject: Re: read block failed check_tree_block / Couldn't read chunk tree
From: Rene Thomas <re.thomas@gmx.de>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>, <linux-btrfs@vger.kernel.org>
Date: 2014年10月28日 14:59
> Yes, removed sdb as well. Got the same error.
>
> You are right. I miss to add the btrfs ml when I reply. Thanks.
>
> I'am looking forward to hearing from you.
>
> Thanks
> René
Oh, that's bad...

Also investigate the full output of chunk-recover.
The results is worse that I first think.

1. Missing chunk item in RAID5
Which caused so many orphan dev extents and orphan block groups.
This is especially bad when it comes to RAID0/RAID5 where the stripe 
sequence matters.
That's to say, there are at least 499 chunks unable to recover.
(In RAID1, it could be recovered)

Also, chunk-recover current won't continue if there are so many orphans.
In your case, 1300+ valid chunk vs 500- unrecoverable orphan chunks,
The chance is very low now.

2. Low amount of rebuildable chunks.
The 77 bad chunks seems god, except the first bad chunk, who has no 
corresponding dev extent nor
block group.
Other 76 chunks are rebuildable, but 1316 + 76 won't change much against 
499 unrecoverable chunks.

So overall, it is almost impossible to recover.
I'm very sorry that I can't provide much help.

Thanks,
Qu

>
> 2014-10-27 9:20 GMT+01:00 Qu Wenruo <quwenruo@cn.fujitsu.com 
> <mailto:quwenruo@cn.fujitsu.com>>:
>
>
>     -------- Original Message --------
>     Subject: Re: read block failed check_tree_block / Couldn't read
>     chunk tree
>     From: Rene Thomas <re.thomas@gmx.de <mailto:re.thomas@gmx.de>>
>     To: Qu Wenruo <quwenruo@cn.fujitsu.com
>     <mailto:quwenruo@cn.fujitsu.com>>
>     Date: 2014年10月27日 15:30
>
>         Tried to mount the not affected drives in several ways. Always
>         get:
>
>         sudo mount -t btrfs -o degraded /dev/sda1 /home
>         [sudo] password for rthomas:
>         mount: Falscher Dateisystemtyp, ungültige Optionen, der
>            Superblock von /dev/sda1 ist beschädigt, fehlende
>            Kodierungsseite oder ein anderer Fehler
>            Manchmal liefert das Systemprotokoll wertvolle Informationen,
>            versuchen Sie »dmesg | tail« oder so
>
>     Even plug off the /dev/sdb?
>     If even remove the /dev/sdb hardware, it still reproduce the same
>     thing,
>     I think there is little chance.
>
>
>         And dmesg shows:
>
>         [  311.335674] BTRFS: bad tree block start 0 5845480062976
>         [  311.336215] BTRFS: bad tree block start 0 5845480062976
>         [  311.354660] BTRFS: bad tree block start 0 5845480054784
>         [  311.354762] BTRFS: bad tree block start 0 5845480054784
>         [  311.355024] BTRFS: bad tree block start 0 5845480067072
>         [  311.355145] BTRFS: bad tree block start 0 5845480067072
>         [  311.355496] BTRFS: bad tree block start 0 5845480050688
>         [  311.355612] BTRFS: bad tree block start 0 5845480050688
>         [  311.356347] BTRFS: bad tree block start 0 5845480071168
>         [  311.356458] BTRFS: bad tree block start 0 5845480071168
>         [  311.580203] BTRFS: Failed to read block groups: -5
>         [  311.605073] BTRFS: open_ctree failed
>
>         Tried all non-destructive ways to get data back.
>
>         Please find attached btrfs rescue chunk-recover log
>
>         http://sebsauvage.net/paste/?baa6fa7fbb2056cb#RTUGry3LQlmVghpjiMkRRGsxpwzDzoWOsmjO0elHF5c=
>
>     Thanks for the log,
>     I'll investigate it.
>
>         uname -a:
>         Linux engelserver 3.17.0-031700rc5-generic #201409151105 SMP
>         Mon Sep 15 15:08:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
>         btrfs --version:
>         3.17
>
>         I'am happy to help to improve tool quality / functionality.
>
>         Maybe it's interisting why mkfs.ntfs have such an impact on a
>         btrfs RAID5 array. In my opinion mkfs should not touch a
>         mounted device even when it's part of a raid and not explicit
>         mentioned in the fstab as the drive to be mounted.
>
>     Yes, mkfs should never touch a mounted device.
>     However, not every fs utils handles it in the right way.
>     In my test, ext4/btrfs/xfs can detect it and fail to mkfs, but
>     ntfs utils seems not to handle it right.
>     That caused the tragedy and btrfs-progs can't help but only to see
>     it happens.
>
>     BTW, would you mind Cc the mail to btrfs ml?
>     More people may provide help.
>
>     Thanks,
>     Qu
>
>
>         If there is a way get more information on the mount process /
>         chunk-recover,(e.g. enhanced log level) please advice.
>
>         Thanks
>         René
>
>
>         2014-10-27 3:29 GMT+01:00 Qu Wenruo <quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>
>         <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>>>:
>
>
>             -------- Original Message --------
>             Subject: read block failed check_tree_block / Couldn't
>         read chunk tree
>             From: Rene Thomas <re.thomas@gmx.de
>         <mailto:re.thomas@gmx.de> <mailto:re.thomas@gmx.de
>         <mailto:re.thomas@gmx.de>>>
>             To: <linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>
>         <mailto:linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>>>
>             Date: 2014年10月25日 00:43
>
>                 Dear Developers / Maintainer,
>                 I’ve set up a running RAID5 with three devices (sda1 /
>         sdb1 /sdc1)
>
>                 Mountpoint was /home, filesystem was mounted
>                 A chain of unfortunal circumstances gives me the
>         chance to run
>                 as root
>                 in a terminal.
>
>                 mkfs.ntfs /dev/sdb1
>
>                 where /dev/sdb1 is device 1 in the Array.
>
>                 After a while I’ve realised that the terminal I used
>         was the
>                 wrong one
>                 and killed the process where the progress still was 0%.
>                 The filesystem was still in a readable state switched
>         to read
>                 only.
>                 Write access was not able.
>
>                 After a restart the RAID failed to mount.
>
>                 Tried several steps to reactivate the Array.
>
>                 All btrfs commands leads to:
>
>                 Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=65536
>                 Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=0
>                 read block failed check_tree_block
>                 Couldn't read chunk tree
>
>             Chunk tree corruption will always lead to disaster :(
>
>             What about remove sdb1 and mount sda1/sdc1 with -o degraded ?
>             I hope the sda1/sdc1 is still in correct status...
>
>
>                 Started with:
>                 btrfs fi show
>                 mount -o revovery,ro /dev/sdc1 /home
>                 failed with error (see dmesg log at [311])
>                 btrfs-image -c 9 -t 8 /dev/sdb1
>         /media/storageplace/fs_image
>                 btrfs-zero-log /dev/sdb1
>                 btrfs check /dev/sdb1
>                 btrfs check --repair /dev/sdb1
>
>                 At least I tried
>                 btrfs rescue chunk-recover /dev/sdb1
>
>                 Works for several hours on the arry and end up in
>         (huge log
>                 available > 100k):
>
>                 “Fail to recover the chunk tree.”
>
>             If my previous defraded doesn't help, chunk-recover is the
>         last
>             chance for it.
>             If still failed, the chance to recover seems quite low.
>
>             However, can you upload the huge log for developers to
>         investigate
>             and improve the chunk-recover tools?
>
>             Thanks,
>             Qu
>
>
>                   I’ve checked the super block there are no errors
>
>                 My question is there any chance on a broken c tree to
>         get the data
>                 back? Not necessarily fix the Array only get the data.
>
>                 Thanks in advanced
>
>                 # uname -a
>                 Linux engelserver 3.17.0-031700rc5-generic
>         #201409151105 SMP
>                 Mon Sep
>                 15 15:08:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
>                   # btrfs --version
>                 Btrfs v3.17
>
>                   # btrfs fi show
>                 Label: 'mythstorage'  uuid:
>         9b454272-6800-4b3c-b196-9e180407a6cb
>                          Total devices 1 FS bytes used 2.36MiB
>                          devid    1 size 931.51GiB used 10.04GiB path
>         /dev/sdd1
>
>                   Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=65536
>                 Check tree block failed, want=5845480062976, have=0
>                 Check tree block failed, want=5845480062976, have=0
>                 read block failed check_tree_block
>                 Couldn't read chunk tree
>
>                 Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
>                          Total devices 3 FS bytes used 3.02TiB
>                          devid    1 size 2.73TiB used 1.77TiB path
>         /dev/sdb1
>                          devid    2 size 2.73TiB used 1.77TiB path
>         /dev/sda1
>                          devid    3 size 2.73TiB used 1.77TiB path
>         /dev/sdc1
>                 Btrfs v3.17
>
>                 dmesg:
>                 [  280.637338] init: plymouth-upstart-bridge main process
>                 ended, respawning
>                 [  311.298682] BTRFS info (device sdc1): disk space
>         caching is
>                 enabled
>                 [  311.335674] BTRFS: bad tree block start 0 5845480062976
>                 [  311.336215] BTRFS: bad tree block start 0 5845480062976
>                 [  311.354660] BTRFS: bad tree block start 0 5845480054784
>                 [  311.354762] BTRFS: bad tree block start 0 5845480054784
>                 [  311.355024] BTRFS: bad tree block start 0 5845480067072
>                 [  311.355145] BTRFS: bad tree block start 0 5845480067072
>                 [  311.355496] BTRFS: bad tree block start 0 5845480050688
>                 [  311.355612] BTRFS: bad tree block start 0 5845480050688
>                 [  311.356347] BTRFS: bad tree block start 0 5845480071168
>                 [  311.356458] BTRFS: bad tree block start 0 5845480071168
>                 [  311.580203] BTRFS: Failed to read block groups: -5
>                 [  311.605073] BTRFS: open_ctree failed
>
>                 René
>                 --
>                 To unsubscribe from this list: send the line "unsubscribe
>                 linux-btrfs" in
>                 the body of a message to majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>
>                 <mailto:majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>>
>                 More majordomo info at
>         http://vger.kernel.org/majordomo-info.html
>
>
>
>
>

Re: read block failed check_tree_block / Couldn't read chunk tree

[Qu Wenruo]

-------- Original Message --------
Subject: Re: read block failed check_tree_block / Couldn't read chunk tree
From: Rene Thomas <re.thomas@gmx.de>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>
Date: 2014年10月28日 18:59
> Ok. That's not what I want to hear but bad thinks happens sometimes :-(
>
> Only for understanding. Recover means to get the RAID back in 
> functional mode or the use of a data recovery tool won't recover anything?
At least for me, recover means all chunks are OK, that's to say,btrfs 
can handle logical <-> disk offset mapping,
which is just the very fundamental base function to make btrfs works.

Since chunk tree is highly damaged, it won't surprise me if other 
tree's(metadata or data) is also damaged.
So in short, my "recover" may only refer to the logical <-> disk offset 
mapping part.

About the data recovery tool, you may want to try btrfs-restore.
Not quite familiar with it, but it may have a little chance.

Thanks,
Qu
>
> Thank you for your time and analysing the log. BTRFS is still my first 
> choice! Maybe I should remove / rename mkfs.ntfs ;-)
>
> Thanks to all for the hints and the help on that.
> René
>
>
> 2014-10-28 10:14 GMT+01:00 Qu Wenruo <quwenruo@cn.fujitsu.com 
> <mailto:quwenruo@cn.fujitsu.com>>:
>
>
>     -------- Original Message --------
>     Subject: Re: read block failed check_tree_block / Couldn't read
>     chunk tree
>     From: Rene Thomas <re.thomas@gmx.de <mailto:re.thomas@gmx.de>>
>     To: Qu Wenruo <quwenruo@cn.fujitsu.com
>     <mailto:quwenruo@cn.fujitsu.com>>, <linux-btrfs@vger.kernel.org
>     <mailto:linux-btrfs@vger.kernel.org>>
>     Date: 2014年10月28日 14:59
>
>         Yes, removed sdb as well. Got the same error.
>
>         You are right. I miss to add the btrfs ml when I reply. Thanks.
>
>         I'am looking forward to hearing from you.
>
>         Thanks
>         René
>
>     Oh, that's bad...
>
>     Also investigate the full output of chunk-recover.
>     The results is worse that I first think.
>
>     1. Missing chunk item in RAID5
>     Which caused so many orphan dev extents and orphan block groups.
>     This is especially bad when it comes to RAID0/RAID5 where the
>     stripe sequence matters.
>     That's to say, there are at least 499 chunks unable to recover.
>     (In RAID1, it could be recovered)
>
>     Also, chunk-recover current won't continue if there are so many
>     orphans.
>     In your case, 1300+ valid chunk vs 500- unrecoverable orphan chunks,
>     The chance is very low now.
>
>     2. Low amount of rebuildable chunks.
>     The 77 bad chunks seems god, except the first bad chunk, who has
>     no corresponding dev extent nor
>     block group.
>     Other 76 chunks are rebuildable, but 1316 + 76 won't change much
>     against 499 unrecoverable chunks.
>
>     So overall, it is almost impossible to recover.
>     I'm very sorry that I can't provide much help.
>
>     Thanks,
>     Qu
>
>
>         2014-10-27 9:20 GMT+01:00 Qu Wenruo <quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>
>         <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>>>:
>
>
>             -------- Original Message --------
>             Subject: Re: read block failed check_tree_block / Couldn't
>         read
>             chunk tree
>             From: Rene Thomas <re.thomas@gmx.de
>         <mailto:re.thomas@gmx.de> <mailto:re.thomas@gmx.de
>         <mailto:re.thomas@gmx.de>>>
>             To: Qu Wenruo <quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>
>             <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>>>
>             Date: 2014年10月27日 15:30
>
>                 Tried to mount the not affected drives in several
>         ways. Always
>                 get:
>
>                 sudo mount -t btrfs -o degraded /dev/sda1 /home
>                 [sudo] password for rthomas:
>                 mount: Falscher Dateisystemtyp, ungültige Optionen, der
>                    Superblock von /dev/sda1 ist beschädigt, fehlende
>                    Kodierungsseite oder ein anderer Fehler
>                    Manchmal liefert das Systemprotokoll wertvolle
>         Informationen,
>                    versuchen Sie »dmesg | tail« oder so
>
>             Even plug off the /dev/sdb?
>             If even remove the /dev/sdb hardware, it still reproduce
>         the same
>             thing,
>             I think there is little chance.
>
>
>                 And dmesg shows:
>
>                 [  311.335674] BTRFS: bad tree block start 0 5845480062976
>                 [  311.336215] BTRFS: bad tree block start 0 5845480062976
>                 [  311.354660] BTRFS: bad tree block start 0 5845480054784
>                 [  311.354762] BTRFS: bad tree block start 0 5845480054784
>                 [  311.355024] BTRFS: bad tree block start 0 5845480067072
>                 [  311.355145] BTRFS: bad tree block start 0 5845480067072
>                 [  311.355496] BTRFS: bad tree block start 0 5845480050688
>                 [  311.355612] BTRFS: bad tree block start 0 5845480050688
>                 [  311.356347] BTRFS: bad tree block start 0 5845480071168
>                 [  311.356458] BTRFS: bad tree block start 0 5845480071168
>                 [  311.580203] BTRFS: Failed to read block groups: -5
>                 [  311.605073] BTRFS: open_ctree failed
>
>                 Tried all non-destructive ways to get data back.
>
>                 Please find attached btrfs rescue chunk-recover log
>
>         http://sebsauvage.net/paste/?baa6fa7fbb2056cb#RTUGry3LQlmVghpjiMkRRGsxpwzDzoWOsmjO0elHF5c=
>
>             Thanks for the log,
>             I'll investigate it.
>
>                 uname -a:
>                 Linux engelserver 3.17.0-031700rc5-generic
>         #201409151105 SMP
>                 Mon Sep 15 15:08:10 UTC 2014 x86_64 x86_64 x86_64
>         GNU/Linux
>
>                 btrfs --version:
>                 3.17
>
>                 I'am happy to help to improve tool quality /
>         functionality.
>
>                 Maybe it's interisting why mkfs.ntfs have such an
>         impact on a
>                 btrfs RAID5 array. In my opinion mkfs should not touch a
>                 mounted device even when it's part of a raid and not
>         explicit
>                 mentioned in the fstab as the drive to be mounted.
>
>             Yes, mkfs should never touch a mounted device.
>             However, not every fs utils handles it in the right way.
>             In my test, ext4/btrfs/xfs can detect it and fail to mkfs, but
>             ntfs utils seems not to handle it right.
>             That caused the tragedy and btrfs-progs can't help but
>         only to see
>             it happens.
>
>             BTW, would you mind Cc the mail to btrfs ml?
>             More people may provide help.
>
>             Thanks,
>             Qu
>
>
>                 If there is a way get more information on the mount
>         process /
>                 chunk-recover,(e.g. enhanced log level) please advice.
>
>                 Thanks
>                 René
>
>
>                 2014-10-27 3:29 GMT+01:00 Qu Wenruo
>         <quwenruo@cn.fujitsu.com <mailto:quwenruo@cn.fujitsu.com>
>                 <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>>
>                 <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>
>                 <mailto:quwenruo@cn.fujitsu.com
>         <mailto:quwenruo@cn.fujitsu.com>>>>:
>
>
>                     -------- Original Message --------
>                     Subject: read block failed check_tree_block / Couldn't
>                 read chunk tree
>                     From: Rene Thomas <re.thomas@gmx.de
>         <mailto:re.thomas@gmx.de>
>                 <mailto:re.thomas@gmx.de <mailto:re.thomas@gmx.de>>
>         <mailto:re.thomas@gmx.de <mailto:re.thomas@gmx.de>
>                 <mailto:re.thomas@gmx.de <mailto:re.thomas@gmx.de>>>>
>                     To: <linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>
>                 <mailto:linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>>
>                 <mailto:linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>
>                 <mailto:linux-btrfs@vger.kernel.org
>         <mailto:linux-btrfs@vger.kernel.org>>>>
>                     Date: 2014年10月25日 00:43
>
>                         Dear Developers / Maintainer,
>                         I’ve set up a running RAID5 with three devices
>         (sda1 /
>                 sdb1 /sdc1)
>
>                         Mountpoint was /home, filesystem was mounted
>                         A chain of unfortunal circumstances gives me the
>                 chance to run
>                         as root
>                         in a terminal.
>
>                         mkfs.ntfs /dev/sdb1
>
>                         where /dev/sdb1 is device 1 in the Array.
>
>                         After a while I’ve realised that the terminal
>         I used
>                 was the
>                         wrong one
>                         and killed the process where the progress
>         still was 0%.
>                         The filesystem was still in a readable state
>         switched
>                 to read
>                         only.
>                         Write access was not able.
>
>                         After a restart the RAID failed to mount.
>
>                         Tried several steps to reactivate the Array.
>
>                         All btrfs commands leads to:
>
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=65536
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         read block failed check_tree_block
>                         Couldn't read chunk tree
>
>                     Chunk tree corruption will always lead to disaster :(
>
>                     What about remove sdb1 and mount sda1/sdc1 with -o
>         degraded ?
>                     I hope the sda1/sdc1 is still in correct status...
>
>
>                         Started with:
>                         btrfs fi show
>                         mount -o revovery,ro /dev/sdc1 /home
>                         failed with error (see dmesg log at [311])
>                         btrfs-image -c 9 -t 8 /dev/sdb1
>                 /media/storageplace/fs_image
>                         btrfs-zero-log /dev/sdb1
>                         btrfs check /dev/sdb1
>                         btrfs check --repair /dev/sdb1
>
>                         At least I tried
>                         btrfs rescue chunk-recover /dev/sdb1
>
>                         Works for several hours on the arry and end up in
>                 (huge log
>                         available > 100k):
>
>                         “Fail to recover the chunk tree.”
>
>                     If my previous defraded doesn't help,
>         chunk-recover is the
>                 last
>                     chance for it.
>                     If still failed, the chance to recover seems quite
>         low.
>
>                     However, can you upload the huge log for developers to
>                 investigate
>                     and improve the chunk-recover tools?
>
>                     Thanks,
>                     Qu
>
>
>                           I’ve checked the super block there are no errors
>
>                         My question is there any chance on a broken c
>         tree to
>                 get the data
>                         back? Not necessarily fix the Array only get
>         the data.
>
>                         Thanks in advanced
>
>                         # uname -a
>                         Linux engelserver 3.17.0-031700rc5-generic
>                 #201409151105 SMP
>                         Mon Sep
>                         15 15:08:10 UTC 2014 x86_64 x86_64 x86_64
>         GNU/Linux
>
>                           # btrfs --version
>                         Btrfs v3.17
>
>                           # btrfs fi show
>                         Label: 'mythstorage'  uuid:
>                 9b454272-6800-4b3c-b196-9e180407a6cb
>                                  Total devices 1 FS bytes used 2.36MiB
>                                  devid    1 size 931.51GiB used
>         10.04GiB path
>                 /dev/sdd1
>
>                           Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=65536
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         Check tree block failed, want=5845480062976,
>         have=0
>                         read block failed check_tree_block
>                         Couldn't read chunk tree
>
>                         Label: none  uuid:
>         8ef575d9-2465-479c-bf9c-067e2e417770
>                                  Total devices 3 FS bytes used 3.02TiB
>                                  devid    1 size 2.73TiB used 1.77TiB path
>                 /dev/sdb1
>                                  devid    2 size 2.73TiB used 1.77TiB path
>                 /dev/sda1
>                                  devid    3 size 2.73TiB used 1.77TiB path
>                 /dev/sdc1
>                         Btrfs v3.17
>
>                         dmesg:
>                         [  280.637338] init: plymouth-upstart-bridge
>         main process
>                         ended, respawning
>                         [  311.298682] BTRFS info (device sdc1): disk
>         space
>                 caching is
>                         enabled
>                         [  311.335674] BTRFS: bad tree block start 0
>         5845480062976
>                         [  311.336215] BTRFS: bad tree block start 0
>         5845480062976
>                         [  311.354660] BTRFS: bad tree block start 0
>         5845480054784
>                         [  311.354762] BTRFS: bad tree block start 0
>         5845480054784
>                         [  311.355024] BTRFS: bad tree block start 0
>         5845480067072
>                         [  311.355145] BTRFS: bad tree block start 0
>         5845480067072
>                         [  311.355496] BTRFS: bad tree block start 0
>         5845480050688
>                         [  311.355612] BTRFS: bad tree block start 0
>         5845480050688
>                         [  311.356347] BTRFS: bad tree block start 0
>         5845480071168
>                         [  311.356458] BTRFS: bad tree block start 0
>         5845480071168
>                         [  311.580203] BTRFS: Failed to read block
>         groups: -5
>                         [  311.605073] BTRFS: open_ctree failed
>
>                         René
>                         --
>                         To unsubscribe from this list: send the line
>         "unsubscribe
>                         linux-btrfs" in
>                         the body of a message to
>         majordomo@vger.kernel.org <mailto:majordomo@vger.kernel.org>
>                 <mailto:majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>>
>                         <mailto:majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>
>                 <mailto:majordomo@vger.kernel.org
>         <mailto:majordomo@vger.kernel.org>>>
>                         More majordomo info at
>         http://vger.kernel.org/majordomo-info.html
>
>
>
>
>
>
>

Re: read block failed check_tree_block / Couldn't read chunk tree

[Anand Jain]

this is (most likely) due to patch below,
----------------
commit 915902c5002485fb13d27c4b699a73fb66cc0f09

     btrfs-progs: fix device missing of btrfs fi show with seed devices
----------------

  Could you try to back out the patch from progs and give it a shot ?
  and pls report what you see. Thanks.




On 10/25/14 00:43, Rene Thomas wrote:
>   # btrfs --version
> Btrfs v3.17
>
>   # btrfs fi show
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>          Total devices 1 FS bytes used 2.36MiB
>          devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>
>   Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> read block failed check_tree_block
> Couldn't read chunk tree

Re: read block failed check_tree_block / Couldn't read chunk tree

[Rene Thomas]

Can't find commit in official repos

Get "fatal: bad object 915902c5002485fb13d27c4b699a73fb66cc0f09" from git show

Found
----------------------------
commit 2513077f2f830b4bc83d528bfb6979eb461918bd

btrfs-progs: fix device missing of btrfs fi show with seed devices
----------------------------

Thanks
René

2014-10-29 4:45 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>
> this is (most likely) due to patch below,
> ----------------
> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>
>     btrfs-progs: fix device missing of btrfs fi show with seed devices
> ----------------
>
>  Could you try to back out the patch from progs and give it a shot ?
>  and pls report what you see. Thanks.
>
>
>
>
>
> On 10/25/14 00:43, Rene Thomas wrote:
>>
>>   # btrfs --version
>> Btrfs v3.17
>>
>>   # btrfs fi show
>> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>>          Total devices 1 FS bytes used 2.36MiB
>>          devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>>
>>   Check tree block failed, want=5845480062976, have=0
>> Check tree block failed, want=5845480062976, have=0
>> Check tree block failed, want=5845480062976, have=65536
>> Check tree block failed, want=5845480062976, have=0
>> Check tree block failed, want=5845480062976, have=0
>> read block failed check_tree_block
>> Couldn't read chunk tree

Re: read block failed check_tree_block / Couldn't read chunk tree

[Anand Jain]

  yes that's the one.

 > btrfs-progs: fix device missing of btrfs fi show with seed devices

Thanks


On 10/29/2014 08:15 PM, Rene Thomas wrote:
> Can't find commit in official repos
>
> Get "fatal: bad object 915902c5002485fb13d27c4b699a73fb66cc0f09" from git show
>
> Found
> ----------------------------
> commit 2513077f2f830b4bc83d528bfb6979eb461918bd
>
> btrfs-progs: fix device missing of btrfs fi show with seed devices
> ----------------------------
>
> Thanks
> René
>
> 2014-10-29 4:45 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>
>> this is (most likely) due to patch below,
>> ----------------
>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>
>>      btrfs-progs: fix device missing of btrfs fi show with seed devices
>> ----------------
>>
>>   Could you try to back out the patch from progs and give it a shot ?
>>   and pls report what you see. Thanks.
>>
>>
>>
>>
>>
>> On 10/25/14 00:43, Rene Thomas wrote:
>>>
>>>    # btrfs --version
>>> Btrfs v3.17
>>>
>>>    # btrfs fi show
>>> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>>>           Total devices 1 FS bytes used 2.36MiB
>>>           devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>>>
>>>    Check tree block failed, want=5845480062976, have=0
>>> Check tree block failed, want=5845480062976, have=0
>>> Check tree block failed, want=5845480062976, have=65536
>>> Check tree block failed, want=5845480062976, have=0
>>> Check tree block failed, want=5845480062976, have=0
>>> read block failed check_tree_block
>>> Couldn't read chunk tree

Re: read block failed check_tree_block / Couldn't read chunk tree

[Rene Thomas]

With commit patched out.

All devices in Place

uname -a
Linux engelserver 3.18.0-031800rc1-generic #201410192135 SMP Mon Oct
20 01:37:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@engelserver:/# btrfs --version
Btrfs v3.17-dirty

root@engelserver:/# btrfs fi show
Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
Total devices 1 FS bytes used 4.27MiB
devid    1 size 931.51GiB used 2.04GiB path /dev/sdh1

Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
Total devices 3 FS bytes used 3.02TiB
devid    1 size 2.73TiB used 1.77TiB path /dev/sdf1
devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
devid    3 size 2.73TiB used 1.77TiB path /dev/sdg1

Btrfs v3.17-dirty

root@engelserver:/# dmesg | grep -i btrfs
[   12.144778] Btrfs loaded
[   12.155656] BTRFS: device label mythstorage devid 1 transid 633 /dev/sdh1
[   12.167277] BTRFS info (device sdh1): enabling auto defrag
[   12.178478] BTRFS info (device sdh1): disk space caching is enabled
[   13.008943] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
devid 1 transid 32270 /dev/sdf1
[   13.017482] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
devid 3 transid 32270 /dev/sdg1
[   13.020782] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
devid 2 transid 32270 /dev/sda1

root@engelserver:/# mount -t btrfs -o recovery,ro /dev/sdf1 /home
mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

[  440.600286] BTRFS info (device sda1): enabling auto recovery
[  440.600288] BTRFS info (device sda1): disk space caching is enabled
[  440.601304] BTRFS (device sda1): bad tree block start 0 5845480062976
[  440.601854] BTRFS (device sda1): bad tree block start 0 5845480054784
[  440.601971] BTRFS (device sda1): bad tree block start 0 5845480067072
[  440.602203] BTRFS (device sda1): bad tree block start 0 5845480050688
[  440.602467] BTRFS (device sda1): bad tree block start 0 5845480071168
[  440.602720] BTRFS (device sda1): bad tree block start 0 5845480058880
[  440.603000] BTRFS (device sda1): bad tree block start 0 5845480075264
[  440.603270] BTRFS (device sda1): bad tree block start 0 5845480079360
[  440.603558] BTRFS (device sda1): bad tree block start 0 5845480046592
[  440.603685] BTRFS (device sda1): bad tree block start 0 5845480091648
[  440.630760] BTRFS: Failed to read block groups: -5
[  440.651385] BTRFS: open_ctree failed

Remove affected device (reboot)

dmesg
[   11.809442] Btrfs loaded
[   11.835961] BTRFS: device label mythstorage devid 1 transid 634 /dev/sdc1
[   11.847034] BTRFS info (device sdc1): enabling auto defrag
[   11.857649] BTRFS info (device sdc1): disk space caching is enabled
[   12.586106] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
devid 2 transid 32270 /dev/sda1
[   12.599428] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
devid 3 transid 32270 /dev/sdb1

root@engelserver:/# btrfs fi sh
Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
Total devices 1 FS bytes used 4.27MiB
devid    1 size 931.51GiB used 2.04GiB path /dev/sdc1

Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
Total devices 3 FS bytes used 3.02TiB
devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
devid    3 size 2.73TiB used 1.77TiB path /dev/sdb1
*** Some devices missing

Btrfs v3.17-dirty

root@engelserver:/# mount -t btrfs -o ro,degraded /dev/sda1 /home
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

root@engelserver:/# dmesg | grep -i btrfs
[  299.792397] BTRFS info (device sdb1): allowing degraded mounts
[  299.792400] BTRFS info (device sdb1): disk space caching is enabled
[  299.964814] BTRFS (device sdb1): bad tree block start 0 5845002854400
[  299.964896] BTRFS (device sdb1): bad tree block start 0 5845002854400
[  299.964965] BTRFS: Failed to read block groups: -5
[  299.987542] BTRFS: open_ctree failed

root@engelserver:/# btrfs restore -i /dev/sda1 /media/mythstorage/
warning, device 1 is missing
warning, device 1 is missing
warning devid 1 not found already
Check tree block failed, want=5845480062976, have=65536
Check tree block failed, want=5845480062976, have=65536
read block failed check_tree_block
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 1 is missing
warning devid 1 not found already
Check tree block failed, want=5845480062976, have=65536
Check tree block failed, want=5845480062976, have=65536
read block failed check_tree_block
Couldn't read chunk tree
Could not open root, trying backup super
warning, device 1 is missing
warning, device 1 is missing
warning devid 1 not found already
Check tree block failed, want=5845480062976, have=65536
Check tree block failed, want=5845480062976, have=65536
read block failed check_tree_block
Couldn't read chunk tree
Could not open root, trying backup super

root@engelserver:/# btrfs rescue chunk-recover -v /dev/sdb1

http://sebsauvage.net/paste/?ed4437f335d9a563#Hy3yzV8fIrgEotLNs3dPg+xX4loQZ/DMxOhXQ/HqdVc=

root@engelserver:/# btrfs rescue super-recover -v /dev/sda1
All Devices:
Device: id = 3, name = /dev/sdc1
Device: id = 1, name = /dev/sdb1
Device: id = 2, name = /dev/sda1

Before Recovering:
[All good supers]:
device name = /dev/sdc1
superblock bytenr = 65536

device name = /dev/sdc1
superblock bytenr = 67108864

device name = /dev/sdc1
superblock bytenr = 274877906944

device name = /dev/sdb1
superblock bytenr = 65536

device name = /dev/sdb1
superblock bytenr = 67108864

device name = /dev/sdb1
superblock bytenr = 274877906944

device name = /dev/sda1
superblock bytenr = 65536

device name = /dev/sda1
superblock bytenr = 67108864

device name = /dev/sda1
superblock bytenr = 274877906944

[All bad supers]:

All supers are valid, no need to recover

Did I miss anything?

Thanks
René

2014-10-30 4:54 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>
>
>
>  yes that's the one.
>
>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>
> Thanks
>
>
>
> On 10/29/2014 08:15 PM, Rene Thomas wrote:
>>
>> Can't find commit in official repos
>>
>> Get "fatal: bad object 915902c5002485fb13d27c4b699a73fb66cc0f09" from git
>> show
>>
>> Found
>> ----------------------------
>> commit 2513077f2f830b4bc83d528bfb6979eb461918bd
>>
>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>> ----------------------------
>>
>> Thanks
>> René
>>
>> 2014-10-29 4:45 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>>
>>>
>>> this is (most likely) due to patch below,
>>> ----------------
>>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>>
>>>      btrfs-progs: fix device missing of btrfs fi show with seed devices
>>> ----------------
>>>
>>>   Could you try to back out the patch from progs and give it a shot ?
>>>   and pls report what you see. Thanks.
>>>
>>>
>>>
>>>
>>>
>>> On 10/25/14 00:43, Rene Thomas wrote:
>>>>
>>>>
>>>>    # btrfs --version
>>>> Btrfs v3.17
>>>>
>>>>    # btrfs fi show
>>>> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>>>>           Total devices 1 FS bytes used 2.36MiB
>>>>           devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>>>>
>>>>    Check tree block failed, want=5845480062976, have=0
>>>> Check tree block failed, want=5845480062976, have=0
>>>> Check tree block failed, want=5845480062976, have=65536
>>>> Check tree block failed, want=5845480062976, have=0
>>>> Check tree block failed, want=5845480062976, have=0
>>>> read block failed check_tree_block
>>>> Couldn't read chunk tree

Re: read block failed check_tree_block / Couldn't read chunk tree

[Anand Jain]

There might be more than one issue. I was trying solve errors

Check tree block failed, want=5845480062976, have=0

when 'btrfs fi show' was run.  Thanks for confirming patchout
helped to fix that.



On 31/10/2014 00:47, Rene Thomas wrote:
> With commit patched out.
>
> All devices in Place
>
> uname -a
> Linux engelserver 3.18.0-031800rc1-generic #201410192135 SMP Mon Oct
> 20 01:37:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> root@engelserver:/# btrfs --version
> Btrfs v3.17-dirty
>
> root@engelserver:/# btrfs fi show
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
> Total devices 1 FS bytes used 4.27MiB
> devid    1 size 931.51GiB used 2.04GiB path /dev/sdh1
>
> Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
> Total devices 3 FS bytes used 3.02TiB
> devid    1 size 2.73TiB used 1.77TiB path /dev/sdf1
> devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
> devid    3 size 2.73TiB used 1.77TiB path /dev/sdg1
>
> Btrfs v3.17-dirty
>
> root@engelserver:/# dmesg | grep -i btrfs
> [   12.144778] Btrfs loaded
> [   12.155656] BTRFS: device label mythstorage devid 1 transid 633 /dev/sdh1
> [   12.167277] BTRFS info (device sdh1): enabling auto defrag
> [   12.178478] BTRFS info (device sdh1): disk space caching is enabled
> [   13.008943] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 1 transid 32270 /dev/sdf1
> [   13.017482] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 3 transid 32270 /dev/sdg1
> [   13.020782] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 2 transid 32270 /dev/sda1
>
> root@engelserver:/# mount -t btrfs -o recovery,ro /dev/sdf1 /home
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>         missing codepage or helper program, or other error
>         In some cases useful info is found in syslog - try
>         dmesg | tail  or so
>
> [  440.600286] BTRFS info (device sda1): enabling auto recovery
> [  440.600288] BTRFS info (device sda1): disk space caching is enabled
> [  440.601304] BTRFS (device sda1): bad tree block start 0 5845480062976
> [  440.601854] BTRFS (device sda1): bad tree block start 0 5845480054784
> [  440.601971] BTRFS (device sda1): bad tree block start 0 5845480067072
> [  440.602203] BTRFS (device sda1): bad tree block start 0 5845480050688
> [  440.602467] BTRFS (device sda1): bad tree block start 0 5845480071168
> [  440.602720] BTRFS (device sda1): bad tree block start 0 5845480058880
> [  440.603000] BTRFS (device sda1): bad tree block start 0 5845480075264
> [  440.603270] BTRFS (device sda1): bad tree block start 0 5845480079360
> [  440.603558] BTRFS (device sda1): bad tree block start 0 5845480046592
> [  440.603685] BTRFS (device sda1): bad tree block start 0 5845480091648
> [  440.630760] BTRFS: Failed to read block groups: -5
> [  440.651385] BTRFS: open_ctree failed
>
> Remove affected device (reboot)
>
> dmesg
> [   11.809442] Btrfs loaded
> [   11.835961] BTRFS: device label mythstorage devid 1 transid 634 /dev/sdc1
> [   11.847034] BTRFS info (device sdc1): enabling auto defrag
> [   11.857649] BTRFS info (device sdc1): disk space caching is enabled
> [   12.586106] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 2 transid 32270 /dev/sda1
> [   12.599428] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 3 transid 32270 /dev/sdb1
>
> root@engelserver:/# btrfs fi sh
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
> Total devices 1 FS bytes used 4.27MiB
> devid    1 size 931.51GiB used 2.04GiB path /dev/sdc1
>
> Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
> Total devices 3 FS bytes used 3.02TiB
> devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
> devid    3 size 2.73TiB used 1.77TiB path /dev/sdb1
> *** Some devices missing
>
> Btrfs v3.17-dirty
>
> root@engelserver:/# mount -t btrfs -o ro,degraded /dev/sda1 /home
> mount: wrong fs type, bad option, bad superblock on /dev/sda1,
>         missing codepage or helper program, or other error
>         In some cases useful info is found in syslog - try
>         dmesg | tail  or so
>
> root@engelserver:/# dmesg | grep -i btrfs
> [  299.792397] BTRFS info (device sdb1): allowing degraded mounts
> [  299.792400] BTRFS info (device sdb1): disk space caching is enabled
> [  299.964814] BTRFS (device sdb1): bad tree block start 0 5845002854400
> [  299.964896] BTRFS (device sdb1): bad tree block start 0 5845002854400
> [  299.964965] BTRFS: Failed to read block groups: -5
> [  299.987542] BTRFS: open_ctree failed
>
> root@engelserver:/# btrfs restore -i /dev/sda1 /media/mythstorage/
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
>
> root@engelserver:/# btrfs rescue chunk-recover -v /dev/sdb1
>
> http://sebsauvage.net/paste/?ed4437f335d9a563#Hy3yzV8fIrgEotLNs3dPg+xX4loQZ/DMxOhXQ/HqdVc=
>
> root@engelserver:/# btrfs rescue super-recover -v /dev/sda1
> All Devices:
> Device: id = 3, name = /dev/sdc1
> Device: id = 1, name = /dev/sdb1
> Device: id = 2, name = /dev/sda1
>
> Before Recovering:
> [All good supers]:
> device name = /dev/sdc1
> superblock bytenr = 65536
>
> device name = /dev/sdc1
> superblock bytenr = 67108864
>
> device name = /dev/sdc1
> superblock bytenr = 274877906944
>
> device name = /dev/sdb1
> superblock bytenr = 65536
>
> device name = /dev/sdb1
> superblock bytenr = 67108864
>
> device name = /dev/sdb1
> superblock bytenr = 274877906944
>
> device name = /dev/sda1
> superblock bytenr = 65536
>
> device name = /dev/sda1
> superblock bytenr = 67108864
>
> device name = /dev/sda1
> superblock bytenr = 274877906944
>
> [All bad supers]:
>
> All supers are valid, no need to recover
>
> Did I miss anything?
>
> Thanks
> René
>
> 2014-10-30 4:54 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>
>>
>>
>>   yes that's the one.
>>
>>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>>
>> Thanks
>>
>>
>>
>> On 10/29/2014 08:15 PM, Rene Thomas wrote:
>>>
>>> Can't find commit in official repos
>>>
>>> Get "fatal: bad object 915902c5002485fb13d27c4b699a73fb66cc0f09" from git
>>> show
>>>
>>> Found
>>> ----------------------------
>>> commit 2513077f2f830b4bc83d528bfb6979eb461918bd
>>>
>>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>>> ----------------------------
>>>
>>> Thanks
>>> René
>>>
>>> 2014-10-29 4:45 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>>>
>>>>
>>>> this is (most likely) due to patch below,
>>>> ----------------
>>>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>>>
>>>>       btrfs-progs: fix device missing of btrfs fi show with seed devices
>>>> ----------------
>>>>
>>>>    Could you try to back out the patch from progs and give it a shot ?
>>>>    and pls report what you see. Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/25/14 00:43, Rene Thomas wrote:
>>>>>
>>>>>
>>>>>     # btrfs --version
>>>>> Btrfs v3.17
>>>>>
>>>>>     # btrfs fi show
>>>>> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>>>>>            Total devices 1 FS bytes used 2.36MiB
>>>>>            devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>>>>>
>>>>>     Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=65536
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> read block failed check_tree_block
>>>>> Couldn't read chunk tree

Re: read block failed check_tree_block / Couldn't read chunk tree

[Qu Wenruo]

-------- Original Message --------
Subject: Re: read block failed check_tree_block / Couldn't read chunk tree
From: Rene Thomas <re.thomas@gmx.de>
To: Anand Jain <Anand.Jain@oracle.com>
Date: 2014年10月31日 00:47
> With commit patched out.
>
> All devices in Place
>
> uname -a
> Linux engelserver 3.18.0-031800rc1-generic #201410192135 SMP Mon Oct
> 20 01:37:04 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
>
> root@engelserver:/# btrfs --version
> Btrfs v3.17-dirty
>
> root@engelserver:/# btrfs fi show
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
> Total devices 1 FS bytes used 4.27MiB
> devid    1 size 931.51GiB used 2.04GiB path /dev/sdh1
>
> Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
> Total devices 3 FS bytes used 3.02TiB
> devid    1 size 2.73TiB used 1.77TiB path /dev/sdf1
> devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
> devid    3 size 2.73TiB used 1.77TiB path /dev/sdg1
>
> Btrfs v3.17-dirty
>
> root@engelserver:/# dmesg | grep -i btrfs
> [   12.144778] Btrfs loaded
> [   12.155656] BTRFS: device label mythstorage devid 1 transid 633 /dev/sdh1
> [   12.167277] BTRFS info (device sdh1): enabling auto defrag
> [   12.178478] BTRFS info (device sdh1): disk space caching is enabled
> [   13.008943] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 1 transid 32270 /dev/sdf1
> [   13.017482] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 3 transid 32270 /dev/sdg1
> [   13.020782] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 2 transid 32270 /dev/sda1
>
> root@engelserver:/# mount -t btrfs -o recovery,ro /dev/sdf1 /home
> mount: wrong fs type, bad option, bad superblock on /dev/sdf1,
>         missing codepage or helper program, or other error
>         In some cases useful info is found in syslog - try
>         dmesg | tail  or so
>
> [  440.600286] BTRFS info (device sda1): enabling auto recovery
> [  440.600288] BTRFS info (device sda1): disk space caching is enabled
> [  440.601304] BTRFS (device sda1): bad tree block start 0 5845480062976
> [  440.601854] BTRFS (device sda1): bad tree block start 0 5845480054784
> [  440.601971] BTRFS (device sda1): bad tree block start 0 5845480067072
> [  440.602203] BTRFS (device sda1): bad tree block start 0 5845480050688
> [  440.602467] BTRFS (device sda1): bad tree block start 0 5845480071168
> [  440.602720] BTRFS (device sda1): bad tree block start 0 5845480058880
> [  440.603000] BTRFS (device sda1): bad tree block start 0 5845480075264
> [  440.603270] BTRFS (device sda1): bad tree block start 0 5845480079360
> [  440.603558] BTRFS (device sda1): bad tree block start 0 5845480046592
> [  440.603685] BTRFS (device sda1): bad tree block start 0 5845480091648
> [  440.630760] BTRFS: Failed to read block groups: -5
> [  440.651385] BTRFS: open_ctree failed
Fail to read block groups....
What about try btrfs check with --init-extent-tree to reset the block 
groups with all devices attached?

Thanks,
Qu

>
> Remove affected device (reboot)
>
> dmesg
> [   11.809442] Btrfs loaded
> [   11.835961] BTRFS: device label mythstorage devid 1 transid 634 /dev/sdc1
> [   11.847034] BTRFS info (device sdc1): enabling auto defrag
> [   11.857649] BTRFS info (device sdc1): disk space caching is enabled
> [   12.586106] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 2 transid 32270 /dev/sda1
> [   12.599428] BTRFS: device fsid 8ef575d9-2465-479c-bf9c-067e2e417770
> devid 3 transid 32270 /dev/sdb1
>
> root@engelserver:/# btrfs fi sh
> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
> Total devices 1 FS bytes used 4.27MiB
> devid    1 size 931.51GiB used 2.04GiB path /dev/sdc1
>
> Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
> Total devices 3 FS bytes used 3.02TiB
> devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
> devid    3 size 2.73TiB used 1.77TiB path /dev/sdb1
> *** Some devices missing
>
> Btrfs v3.17-dirty
>
> root@engelserver:/# mount -t btrfs -o ro,degraded /dev/sda1 /home
> mount: wrong fs type, bad option, bad superblock on /dev/sda1,
>         missing codepage or helper program, or other error
>         In some cases useful info is found in syslog - try
>         dmesg | tail  or so
>
> root@engelserver:/# dmesg | grep -i btrfs
> [  299.792397] BTRFS info (device sdb1): allowing degraded mounts
> [  299.792400] BTRFS info (device sdb1): disk space caching is enabled
> [  299.964814] BTRFS (device sdb1): bad tree block start 0 5845002854400
> [  299.964896] BTRFS (device sdb1): bad tree block start 0 5845002854400
> [  299.964965] BTRFS: Failed to read block groups: -5
> [  299.987542] BTRFS: open_ctree failed
>
> root@engelserver:/# btrfs restore -i /dev/sda1 /media/mythstorage/
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
> warning, device 1 is missing
> warning, device 1 is missing
> warning devid 1 not found already
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=65536
> read block failed check_tree_block
> Couldn't read chunk tree
> Could not open root, trying backup super
>
> root@engelserver:/# btrfs rescue chunk-recover -v /dev/sdb1
>
> http://sebsauvage.net/paste/?ed4437f335d9a563#Hy3yzV8fIrgEotLNs3dPg+xX4loQZ/DMxOhXQ/HqdVc=
>
> root@engelserver:/# btrfs rescue super-recover -v /dev/sda1
> All Devices:
> Device: id = 3, name = /dev/sdc1
> Device: id = 1, name = /dev/sdb1
> Device: id = 2, name = /dev/sda1
>
> Before Recovering:
> [All good supers]:
> device name = /dev/sdc1
> superblock bytenr = 65536
>
> device name = /dev/sdc1
> superblock bytenr = 67108864
>
> device name = /dev/sdc1
> superblock bytenr = 274877906944
>
> device name = /dev/sdb1
> superblock bytenr = 65536
>
> device name = /dev/sdb1
> superblock bytenr = 67108864
>
> device name = /dev/sdb1
> superblock bytenr = 274877906944
>
> device name = /dev/sda1
> superblock bytenr = 65536
>
> device name = /dev/sda1
> superblock bytenr = 67108864
>
> device name = /dev/sda1
> superblock bytenr = 274877906944
>
> [All bad supers]:
>
> All supers are valid, no need to recover
>
> Did I miss anything?
>
> Thanks
> René
>
> 2014-10-30 4:54 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>
>>
>>   yes that's the one.
>>
>>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>> Thanks
>>
>>
>>
>> On 10/29/2014 08:15 PM, Rene Thomas wrote:
>>> Can't find commit in official repos
>>>
>>> Get "fatal: bad object 915902c5002485fb13d27c4b699a73fb66cc0f09" from git
>>> show
>>>
>>> Found
>>> ----------------------------
>>> commit 2513077f2f830b4bc83d528bfb6979eb461918bd
>>>
>>> btrfs-progs: fix device missing of btrfs fi show with seed devices
>>> ----------------------------
>>>
>>> Thanks
>>> René
>>>
>>> 2014-10-29 4:45 GMT+01:00 Anand Jain <Anand.Jain@oracle.com>:
>>>>
>>>> this is (most likely) due to patch below,
>>>> ----------------
>>>> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
>>>>
>>>>       btrfs-progs: fix device missing of btrfs fi show with seed devices
>>>> ----------------
>>>>
>>>>    Could you try to back out the patch from progs and give it a shot ?
>>>>    and pls report what you see. Thanks.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On 10/25/14 00:43, Rene Thomas wrote:
>>>>>
>>>>>     # btrfs --version
>>>>> Btrfs v3.17
>>>>>
>>>>>     # btrfs fi show
>>>>> Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
>>>>>            Total devices 1 FS bytes used 2.36MiB
>>>>>            devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
>>>>>
>>>>>     Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=65536
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> Check tree block failed, want=5845480062976, have=0
>>>>> read block failed check_tree_block
>>>>> Couldn't read chunk tree
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [offlist]Re: read block failed check_tree_block / Couldn't read chunk tree

[Rene Thomas]

Hi Gui,

no worries. I'am happy with the btrfs fi sh (with or without patch)
I'll try your paches. At the moment I have started to recover the data
from an other device where some parts of the data was stored before.

Since there was no progress during mkfs I don't now what was affected
on the disc. After a short time I've closed the terminal I've used.

I guess the attemp to recover the data make things worse. At least the
Filesystem was in ro state and accesible before I shutdown the system.
Maybe this was the right time to save the data :-)

Thanks
René

Re: read block failed check_tree_block / Couldn't read chunk tree

[Gui Hecheng]

On Wed, 2014-10-29 at 11:45 +0800, Anand Jain wrote:
> this is (most likely) due to patch below,
> ----------------
> commit 915902c5002485fb13d27c4b699a73fb66cc0f09
> 
>      btrfs-progs: fix device missing of btrfs fi show with seed devices
> ----------------
> 
>   Could you try to back out the patch from progs and give it a shot ?
>   and pls report what you see. Thanks.

Hi Anand,
-------------------------------------------------------------------
 # btrfs fi show
Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
        Total devices 1 FS bytes used 2.36MiB
        devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1

[errors from open_ctree]

Label: none  uuid: 8ef575d9-2465-479c-bf9c-067e2e417770
        Total devices 3 FS bytes used 3.02TiB
        devid    1 size 2.73TiB used 1.77TiB path /dev/sdb1
        devid    2 size 2.73TiB used 1.77TiB path /dev/sda1
        devid    3 size 2.73TiB used 1.77TiB path /dev/sdc1
Btrfs v3.17
------------------------------------------------------------------

The original output of btrfs fi show actually detect all devices in all
btrfs successfully. I think the patch mentioned above really missed some
case in which num of devices found match the num of all devices in one
btrfs. In such case, we should not bother trying to detect seed devices.

And for now, we did. Unfortunately, the btrfs is some what corrupted by
mkfs.ntfs, so @open_ctree call complains about corruption on the chunk
root like the following:

> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=65536
> Check tree block failed, want=5845480062976, have=0
> Check tree block failed, want=5845480062976, have=0
> read block failed check_tree_block
> Couldn't read chunk tree

I'll fix this case by avoiding @open_ctree call when all devices in
btrfs are detected.

But if not all devices are detected, then there are 3 possible cases:
1) seed devices present
2) disk unpluged
3) disk reformatted

For case 1), *for now* we need the @open_ctree to detect them(After
your former proposal implemented, things may be different).

For case 2) & 3), @open_ctree complains as above. To clean the not so
helpful and annoying complains, I think we could keep @open_ctree silent
when called by btrfs fi show by introduce a new flag OPEN_CTREE_SILENT.

Thanks
Gui

> 
> 
> On 10/25/14 00:43, Rene Thomas wrote:
> >   # btrfs --version
> > Btrfs v3.17
> >
> >   # btrfs fi show
> > Label: 'mythstorage'  uuid: 9b454272-6800-4b3c-b196-9e180407a6cb
> >          Total devices 1 FS bytes used 2.36MiB
> >          devid    1 size 931.51GiB used 10.04GiB path /dev/sdd1
> >
> >   Check tree block failed, want=5845480062976, have=0
> > Check tree block failed, want=5845480062976, have=0
> > Check tree block failed, want=5845480062976, have=65536
> > Check tree block failed, want=5845480062976, have=0
> > Check tree block failed, want=5845480062976, have=0
> > read block failed check_tree_block
> > Couldn't read chunk tree
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html