Inconsistent reports after disk-error.

Inconsistent reports after disk-error.

[Ronny H. Kavli]

I've been running btrfs on some disks with Ubuntu kernel 2.6.31 for a
while now. Recently I had a bad USB-cable to one of those disks which
left the btrfs filesystem in a somewhat sad state. Fair enough.

What worries me is what btrfsck reports back to me during two successive
runs:
root@bollox:/home/kavli# btrfsck /dev/mapper/vg3-vbox1 
checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
btrfsck: disk-io.c:741: open_ctree_fd: Assertion `!(!tree_root->node)'
failed.
Aborted
root@bollox:/home/kavli# btrfsck /dev/mapper/vg3-vbox1 
checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
btrfsck: disk-io.c:741: open_ctree_fd: Assertion `!(!tree_root->node)'
failed.
Aborted

I'd expected the checksums for the two separate runs to be equal.

This is a vanilla filesystem that resides on one physical disk:
root@bollox:/home/kavli# btrfs-show /dev/mapper/vg3-vbox1 
failed to read /dev/sr0
Label: none  uuid: 08078f2f-22e0-4b79-8367-66528791afff
        Total devices 1 FS bytes used 106.79GB
        devid    1 size 232.88GB used 218.04GB
path /dev/mapper/vg3-vbox1

Btrfs Btrfs v0.19

I've googled a bit and found one case with a similar problem in a raid1
setup (I guess incorrectly stated as raid0):
 http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg03429.html

There were no followups beyond that point which stated the cause of this
behaviour.

BTW: Is there any hope of recovery of this filesystem, or is it just a
new mkfs that can help?

Regards,

 -- RHK

Re: Inconsistent reports after disk-error.

[Ronny H. Kavli]

Ah. I can reply to myself after checking the logs when I try to mount
with dmesg(1).

Two successive mount-attempts give indeed a consistent result:

[ 3712.163040] device fsid 794be0222f8f0708-ffaf918752666783 devid 1
transid 9044 /dev/mapper/vg3-vbox1
[ 3712.172059] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D181110 found 5D18D333 level 0
[ 3712.172931] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D181110 found 5D18D333 level 0
[ 3712.173433] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D185110 found 5D18D333 level 0
[ 3712.173440] btrfs: failed to read tree root on dm-7
[ 3712.173762] btrfs: open_ctree failed
[ 3714.210952] device fsid 794be0222f8f0708-ffaf918752666783 devid 1
transid 9044 /dev/mapper/vg3-vbox1
[ 3714.220220] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D181110 found 5D18D333 level 0
[ 3714.220718] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D181110 found 5D18D333 level 0
[ 3714.221217] btrfs: dm-7 checksum verify failed on 142837022720 wanted
5D185110 found 5D18D333 level 0
[ 3714.221224] btrfs: failed to read tree root on dm-7
[ 3714.221461] btrfs: open_ctree failed

I then make the the assumption that my copy of btrfsck is not giving
correct output.

 -- RHK

Re: Inconsistent reports after disk-error.

[Chris Mason]

On Mon, Jan 18, 2010 at 11:45:40PM +0100, Ronny H. Kavli wrote:
> I've been running btrfs on some disks with Ubuntu kernel 2.6.31 for a
> while now. Recently I had a bad USB-cable to one of those disks which
> left the btrfs filesystem in a somewhat sad state. Fair enough.
> 
> What worries me is what btrfsck reports back to me during two successive
> runs:
> root@bollox:/home/kavli# btrfsck /dev/mapper/vg3-vbox1 
> checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
> checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
> checksum verify failed on 142837022720 wanted 5D18D333 found 95085E8
> btrfsck: disk-io.c:741: open_ctree_fd: Assertion `!(!tree_root->node)'
> failed.
> Aborted
> root@bollox:/home/kavli# btrfsck /dev/mapper/vg3-vbox1 
> checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
> checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
> checksum verify failed on 142837022720 wanted 5D18D333 found 81D15E8
> btrfsck: disk-io.c:741: open_ctree_fd: Assertion `!(!tree_root->node)'
> failed.
> Aborted
> 
> I'd expected the checksums for the two separate runs to be equal.
> 
> This is a vanilla filesystem that resides on one physical disk:
> root@bollox:/home/kavli# btrfs-show /dev/mapper/vg3-vbox1 
> failed to read /dev/sr0
> Label: none  uuid: 08078f2f-22e0-4b79-8367-66528791afff
>         Total devices 1 FS bytes used 106.79GB
>         devid    1 size 232.88GB used 218.04GB
> path /dev/mapper/vg3-vbox1
> 
> Btrfs Btrfs v0.19
> 
> I've googled a bit and found one case with a similar problem in a raid1
> setup (I guess incorrectly stated as raid0):
>  http://www.mail-archive.com/linux-btrfs@vger.kernel.org/msg03429.html
> 
> There were no followups beyond that point which stated the cause of this
> behaviour.

I haven't been able to reproduce it locally, but I definitely think it
sounds like the same problem.  Have you tried the btrfs-map-logical
program in the unstable btrfs-progs repo?

-chris

Re: Inconsistent reports after disk-error.

[Chris Mason]

On Mon, Jan 18, 2010 at 11:45:40PM +0100, Ronny H. Kavli wrote:
> 
> BTW: Is there any hope of recovery of this filesystem, or is it just a
> new mkfs that can help?

I'd expect that we can help at least copy some data off.

-chris