Re: Can't mount array with super_total_bytes mismatch with fs_devices total_rw_bytes

[Asif Youssuff <yoasif@gmail.com>]

On 10/04/2017 01:18 AM, Qu Wenruo wrote:
> 
> 
> On 2017年10月04日 12:00, Asif Youssuff wrote:
>> Thanks for the advice.
>>
>> On 10/03/2017 09:38 PM, Qu Wenruo wrote:
>>
>>
>>>>
>>>> [210017.281912] BTRFS info (device sdb): disk space caching is enabled
>>>> [210017.281915] BTRFS info (device sdb): has skinny extents
>>>> [210017.402084] BTRFS error (device sdb): super_total_bytes 
>>>> 92017859088384
>>>> mismatch with fs_devices total_rw_bytes 92017859094528
>>>
>>> One of your device size is not aligned to 4K.
>>> Which is fine, but recently enhanced validation checker does not 
>>> allow it.
>>> (Which should be a regression, and there is some other WARN_ON 
>>> related to it)
>>>
>>>> [210017.402126] BTRFS error (device sdb): failed to read chunk tree: 
>>>> -22
>>>> [210017.461473] BTRFS error (device sdb): open_ctree failed
>>>>
>>>> I've tried a few steps --
>>>>
>>>> btrfs-chunk-recover, super-recover and I have run a btrfs check 
>>>> --repair on two of the disks in the array (this takes a very long 
>>>> time, so I'm hoping I don't have to run this on all of the disks).
>>>>
>>>> I had run into this problem once before in the past, and I'm not 
>>>> sure how I recovered from it; I may have simply rolled back the 
>>>> booted kernel to escape the extra checks around this mismatch.
>>>>
>>>> I'm at a loss for ideas and am running a btrfs-image so I can also 
>>>> report an issue -- I'm not sure whether 'btrfs-image -c9 -t4 
>>>> /dev/sdo btrfs.image' is  the right command to run if it is a 
>>>> multi-device array.
>>>>
>>>> Any ideas would be helpful, and I am happy to provide further 
>>>> information.
>>>>
>>>> root@ubuntu-server:~#   uname -a
>>>> Linux ubuntu-server 4.14.0-041400rc2-generic #201709242031 SMP Mon 
>>>> Sep 25 00:33:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>> You can rollback to an earlier kernel to mount the fs.
>>>
>>> And manually find and resize the device with unaligned size:
>>>
>>> # btrfs fi show -b
>>> And check each device for its size:
>>> Label: none  uuid: 839ddcfa-5701-4437-aff3-bcb2a26ae6dd
>>>      Total devices 1 FS bytes used 397312
>>>      devid    1 size <<10737418240>> used 2172649472 path 
>>> /dev/mapper/data-btrfs
>>>
>>> If it's not align, round it down to 4K, and resize it using devid:
>>>
>>> # btrfs fi resize <devid>:<rounded_size> <mount_point>
>>>
>>> All device must be rounded. And the command should finish almost in 
>>> no time.
>>
>> I was able to mount the fs using kernel version 4.4 and rounded down 
>> (took the size in bytes, and rounded down to a smaller number 
>> divisible by 4096).
>>
>> This is what btrfs fi show looks like now:
>>
>> asif@ubuntu-server:~$ sudo btrfs fi show --raw
>> [sudo] password for asif:
>> Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc
>>      Total devices 13 FS bytes used 44783241728000
>>      devid    4 size 6001141571584 used 5955045097472 path /dev/sdh
>>      devid    5 size 6001141571584 used 5955128983552 path /dev/sdg
>>      devid    7 size 6001141571584 used 5955709960192 path /dev/sdk
>>      devid    9 size 6001175126016 used 5955716186112 path /dev/sde
>>      devid   10 size 6001141571584 used 5955145760768 path /dev/sdc
>>      devid   11 size 8001563222016 used 7955416416256 path /dev/sdl
>>      devid   12 size 6001175126016 used 5956054286336 path /dev/sdf
>>      devid   14 size 8001563222016 used 7956009123840 path /dev/sdb
>>      devid   15 size 8001563222016 used 7956373831680 path /dev/sdj
>>      devid   17 size 8001563222016 used 6341094866944 path /dev/sdd
>>      devid   18 size 8001563222016 used 7955827064832 path /dev/sdn
>>      devid   20 size 8001563222016 used 7955378339840 path /dev/sdi
>>      devid   21 size 8001563222016 used 7955386728448 path /dev/sdo
>>
>>>
>>> Then check if latest kernel can mount it.
>>
>> Unfortunately, the latest kernel still cannot mount it, showing  the 
>> same errors as before.
>>
>> [  139.852862] BTRFS error (device sdj): super_total_bytes 
>> 92017859086336 mismatch with fs_devices total_rw_bytes 92017859092480
>> [  139.852894] BTRFS error (device sdj): failed to read chunk tree: -22
>> [  139.916645] BTRFS error (device sdj): open_ctree failed
> 
> Then the problem is btrfs doesn't update its super_total_bytes correctly.
> 
>  From what I can see, grow/shrink only update its delta, not 
> re-calculate it.
> 
> Before we have a good way to fix it in kernel, the only way is to 
> manually modify the superblock to allow it pass kernel validation checker.
> 

A manual edit of the superblock sounds painful unless there is good 
documentation for that.

How can I follow up to get this fixed in the kernel? Do we have enough 
information in this thread to create a bug on bugzilla?  Should I open one?

Thanks!
> Thanks,
> Qu
>>
>>>
>>> I think it can be made as part of "btrfs check" to fix it.
>>> (Although it should be handled by kernel well)
>>>
>>> Thanks,
>>> Qu
>>
>> Hope there are some other ideas (or please correct me if I have done 
>> something wrong!).
>>
>> Thanks,
>> Asif
>>
>>>
>>>>
>>>> root@ubuntu-server:~#   btrfs --version
>>>> btrfs-progs v4.13.1
>>>>
>>>> root@ubuntu-server:~#   btrfs fi show
>>>> Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc
>>>> Total devices 13 FS bytes used 40.73TiB
>>>> devid    4 size 5.46TiB used 5.42TiB path /dev/sdo
>>>> devid    5 size 5.46TiB used 5.42TiB path /dev/sdn
>>>> devid    7 size 5.46TiB used 5.42TiB path /dev/sdc
>>>> devid    9 size 5.46TiB used 5.42TiB path /dev/sdk
>>>> devid   10 size 5.46TiB used 5.42TiB path /dev/sdj
>>>> devid   11 size 7.28TiB used 7.24TiB path /dev/sdd
>>>> devid   12 size 5.46TiB used 5.42TiB path /dev/sdm
>>>> devid   14 size 7.28TiB used 7.24TiB path /dev/sdh
>>>> devid   15 size 7.28TiB used 7.24TiB path /dev/sdb
>>>> devid   17 size 7.28TiB used 5.77TiB path /dev/sdl
>>>> devid   18 size 7.28TiB used 7.24TiB path /dev/sdf
>>>> devid   20 size 7.28TiB used 7.24TiB path /dev/sdi
>>>> devid   21 size 7.28TiB used 7.24TiB path /dev/sdg
>>>>
>>>> Thanks,
>>>> Asif
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html