Re: Can't mount array with super_total_bytes mismatch with fs_devices total_rw_bytes

[Asif Youssuff <yoasif@gmail.com>]

Thanks for the advice.

On 10/03/2017 09:38 PM, Qu Wenruo wrote:


>>
>> [210017.281912] BTRFS info (device sdb): disk space caching is enabled
>> [210017.281915] BTRFS info (device sdb): has skinny extents
>> [210017.402084] BTRFS error (device sdb): super_total_bytes 
>> 92017859088384
>> mismatch with fs_devices total_rw_bytes 92017859094528
> 
> One of your device size is not aligned to 4K.
> Which is fine, but recently enhanced validation checker does not allow it.
> (Which should be a regression, and there is some other WARN_ON related 
> to it)
> 
>> [210017.402126] BTRFS error (device sdb): failed to read chunk tree: -22
>> [210017.461473] BTRFS error (device sdb): open_ctree failed
>>
>> I've tried a few steps --
>>
>> btrfs-chunk-recover, super-recover and I have run a btrfs check 
>> --repair on two of the disks in the array (this takes a very long 
>> time, so I'm hoping I don't have to run this on all of the disks).
>>
>> I had run into this problem once before in the past, and I'm not sure 
>> how I recovered from it; I may have simply rolled back the booted 
>> kernel to escape the extra checks around this mismatch.
>>
>> I'm at a loss for ideas and am running a btrfs-image so I can also 
>> report an issue -- I'm not sure whether 'btrfs-image -c9 -t4 /dev/sdo 
>> btrfs.image' is  the right command to run if it is a multi-device array.
>>
>> Any ideas would be helpful, and I am happy to provide further 
>> information.
>>
>> root@ubuntu-server:~#   uname -a
>> Linux ubuntu-server 4.14.0-041400rc2-generic #201709242031 SMP Mon Sep 
>> 25 00:33:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> 
> You can rollback to an earlier kernel to mount the fs.
> 
> And manually find and resize the device with unaligned size:
> 
> # btrfs fi show -b
> And check each device for its size:
> Label: none  uuid: 839ddcfa-5701-4437-aff3-bcb2a26ae6dd
>      Total devices 1 FS bytes used 397312
>      devid    1 size <<10737418240>> used 2172649472 path 
> /dev/mapper/data-btrfs
> 
> If it's not align, round it down to 4K, and resize it using devid:
> 
> # btrfs fi resize <devid>:<rounded_size> <mount_point>
> 
> All device must be rounded. And the command should finish almost in no 
> time.

I was able to mount the fs using kernel version 4.4 and rounded down 
(took the size in bytes, and rounded down to a smaller number divisible 
by 4096).

This is what btrfs fi show looks like now:

asif@ubuntu-server:~$ sudo btrfs fi show --raw
[sudo] password for asif:
Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc
	Total devices 13 FS bytes used 44783241728000
	devid    4 size 6001141571584 used 5955045097472 path /dev/sdh
	devid    5 size 6001141571584 used 5955128983552 path /dev/sdg
	devid    7 size 6001141571584 used 5955709960192 path /dev/sdk
	devid    9 size 6001175126016 used 5955716186112 path /dev/sde
	devid   10 size 6001141571584 used 5955145760768 path /dev/sdc
	devid   11 size 8001563222016 used 7955416416256 path /dev/sdl
	devid   12 size 6001175126016 used 5956054286336 path /dev/sdf
	devid   14 size 8001563222016 used 7956009123840 path /dev/sdb
	devid   15 size 8001563222016 used 7956373831680 path /dev/sdj
	devid   17 size 8001563222016 used 6341094866944 path /dev/sdd
	devid   18 size 8001563222016 used 7955827064832 path /dev/sdn
	devid   20 size 8001563222016 used 7955378339840 path /dev/sdi
	devid   21 size 8001563222016 used 7955386728448 path /dev/sdo

> 
> Then check if latest kernel can mount it.

Unfortunately, the latest kernel still cannot mount it, showing  the 
same errors as before.

[  139.852862] BTRFS error (device sdj): super_total_bytes 
92017859086336 mismatch with fs_devices total_rw_bytes 92017859092480
[  139.852894] BTRFS error (device sdj): failed to read chunk tree: -22
[  139.916645] BTRFS error (device sdj): open_ctree failed

> 
> I think it can be made as part of "btrfs check" to fix it.
> (Although it should be handled by kernel well)
> 
> Thanks,
> Qu

Hope there are some other ideas (or please correct me if I have done 
something wrong!).

Thanks,
Asif

> 
>>
>> root@ubuntu-server:~#   btrfs --version
>> btrfs-progs v4.13.1
>>
>> root@ubuntu-server:~#   btrfs fi show
>> Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc
>> Total devices 13 FS bytes used 40.73TiB
>> devid    4 size 5.46TiB used 5.42TiB path /dev/sdo
>> devid    5 size 5.46TiB used 5.42TiB path /dev/sdn
>> devid    7 size 5.46TiB used 5.42TiB path /dev/sdc
>> devid    9 size 5.46TiB used 5.42TiB path /dev/sdk
>> devid   10 size 5.46TiB used 5.42TiB path /dev/sdj
>> devid   11 size 7.28TiB used 7.24TiB path /dev/sdd
>> devid   12 size 5.46TiB used 5.42TiB path /dev/sdm
>> devid   14 size 7.28TiB used 7.24TiB path /dev/sdh
>> devid   15 size 7.28TiB used 7.24TiB path /dev/sdb
>> devid   17 size 7.28TiB used 5.77TiB path /dev/sdl
>> devid   18 size 7.28TiB used 7.24TiB path /dev/sdf
>> devid   20 size 7.28TiB used 7.24TiB path /dev/sdi
>> devid   21 size 7.28TiB used 7.24TiB path /dev/sdg
>>
>> Thanks,
>> Asif