From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-qt0-f195.google.com ([209.85.216.195]:50474 "EHLO mail-qt0-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750951AbdJDXN2 (ORCPT ); Wed, 4 Oct 2017 19:13:28 -0400 Received: by mail-qt0-f195.google.com with SMTP id f15so22338389qtf.7 for ; Wed, 04 Oct 2017 16:13:28 -0700 (PDT) Subject: Re: Can't mount array with super_total_bytes mismatch with fs_devices total_rw_bytes To: Qu Wenruo , linux-btrfs@vger.kernel.org References: <9dc8fe6e-bda5-2b19-b616-ab64696f2102@gmx.com> <7cbdb7e9-3c1b-c5cb-4ff0-4530de23f6e1@gmx.com> From: Asif Youssuff Message-ID: Date: Wed, 4 Oct 2017 19:13:26 -0400 MIME-Version: 1.0 In-Reply-To: <7cbdb7e9-3c1b-c5cb-4ff0-4530de23f6e1@gmx.com> Content-Type: text/plain; charset=utf-8; format=flowed Sender: linux-btrfs-owner@vger.kernel.org List-ID: On 10/04/2017 01:18 AM, Qu Wenruo wrote: > > > On 2017年10月04日 12:00, Asif Youssuff wrote: >> Thanks for the advice. >> >> On 10/03/2017 09:38 PM, Qu Wenruo wrote: >> >> >>>> >>>> [210017.281912] BTRFS info (device sdb): disk space caching is enabled >>>> [210017.281915] BTRFS info (device sdb): has skinny extents >>>> [210017.402084] BTRFS error (device sdb): super_total_bytes >>>> 92017859088384 >>>> mismatch with fs_devices total_rw_bytes 92017859094528 >>> >>> One of your device size is not aligned to 4K. >>> Which is fine, but recently enhanced validation checker does not >>> allow it. >>> (Which should be a regression, and there is some other WARN_ON >>> related to it) >>> >>>> [210017.402126] BTRFS error (device sdb): failed to read chunk tree: >>>> -22 >>>> [210017.461473] BTRFS error (device sdb): open_ctree failed >>>> >>>> I've tried a few steps -- >>>> >>>> btrfs-chunk-recover, super-recover and I have run a btrfs check >>>> --repair on two of the disks in the array (this takes a very long >>>> time, so I'm hoping I don't have to run this on all of the disks). >>>> >>>> I had run into this problem once before in the past, and I'm not >>>> sure how I recovered from it; I may have simply rolled back the >>>> booted kernel to escape the extra checks around this mismatch. >>>> >>>> I'm at a loss for ideas and am running a btrfs-image so I can also >>>> report an issue -- I'm not sure whether 'btrfs-image -c9 -t4 >>>> /dev/sdo btrfs.image' is  the right command to run if it is a >>>> multi-device array. >>>> >>>> Any ideas would be helpful, and I am happy to provide further >>>> information. >>>> >>>> root@ubuntu-server:~#   uname -a >>>> Linux ubuntu-server 4.14.0-041400rc2-generic #201709242031 SMP Mon >>>> Sep 25 00:33:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux >>> >>> You can rollback to an earlier kernel to mount the fs. >>> >>> And manually find and resize the device with unaligned size: >>> >>> # btrfs fi show -b >>> And check each device for its size: >>> Label: none  uuid: 839ddcfa-5701-4437-aff3-bcb2a26ae6dd >>>      Total devices 1 FS bytes used 397312 >>>      devid    1 size <<10737418240>> used 2172649472 path >>> /dev/mapper/data-btrfs >>> >>> If it's not align, round it down to 4K, and resize it using devid: >>> >>> # btrfs fi resize : >>> >>> All device must be rounded. And the command should finish almost in >>> no time. >> >> I was able to mount the fs using kernel version 4.4 and rounded down >> (took the size in bytes, and rounded down to a smaller number >> divisible by 4096). >> >> This is what btrfs fi show looks like now: >> >> asif@ubuntu-server:~$ sudo btrfs fi show --raw >> [sudo] password for asif: >> Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc >>      Total devices 13 FS bytes used 44783241728000 >>      devid    4 size 6001141571584 used 5955045097472 path /dev/sdh >>      devid    5 size 6001141571584 used 5955128983552 path /dev/sdg >>      devid    7 size 6001141571584 used 5955709960192 path /dev/sdk >>      devid    9 size 6001175126016 used 5955716186112 path /dev/sde >>      devid   10 size 6001141571584 used 5955145760768 path /dev/sdc >>      devid   11 size 8001563222016 used 7955416416256 path /dev/sdl >>      devid   12 size 6001175126016 used 5956054286336 path /dev/sdf >>      devid   14 size 8001563222016 used 7956009123840 path /dev/sdb >>      devid   15 size 8001563222016 used 7956373831680 path /dev/sdj >>      devid   17 size 8001563222016 used 6341094866944 path /dev/sdd >>      devid   18 size 8001563222016 used 7955827064832 path /dev/sdn >>      devid   20 size 8001563222016 used 7955378339840 path /dev/sdi >>      devid   21 size 8001563222016 used 7955386728448 path /dev/sdo >> >>> >>> Then check if latest kernel can mount it. >> >> Unfortunately, the latest kernel still cannot mount it, showing  the >> same errors as before. >> >> [  139.852862] BTRFS error (device sdj): super_total_bytes >> 92017859086336 mismatch with fs_devices total_rw_bytes 92017859092480 >> [  139.852894] BTRFS error (device sdj): failed to read chunk tree: -22 >> [  139.916645] BTRFS error (device sdj): open_ctree failed > > Then the problem is btrfs doesn't update its super_total_bytes correctly. > > From what I can see, grow/shrink only update its delta, not > re-calculate it. > > Before we have a good way to fix it in kernel, the only way is to > manually modify the superblock to allow it pass kernel validation checker. > A manual edit of the superblock sounds painful unless there is good documentation for that. How can I follow up to get this fixed in the kernel? Do we have enough information in this thread to create a bug on bugzilla? Should I open one? Thanks! > Thanks, > Qu >> >>> >>> I think it can be made as part of "btrfs check" to fix it. >>> (Although it should be handled by kernel well) >>> >>> Thanks, >>> Qu >> >> Hope there are some other ideas (or please correct me if I have done >> something wrong!). >> >> Thanks, >> Asif >> >>> >>>> >>>> root@ubuntu-server:~#   btrfs --version >>>> btrfs-progs v4.13.1 >>>> >>>> root@ubuntu-server:~#   btrfs fi show >>>> Label: none  uuid: 48ed8a66-731d-499b-829e-dd07dd7260cc >>>> Total devices 13 FS bytes used 40.73TiB >>>> devid    4 size 5.46TiB used 5.42TiB path /dev/sdo >>>> devid    5 size 5.46TiB used 5.42TiB path /dev/sdn >>>> devid    7 size 5.46TiB used 5.42TiB path /dev/sdc >>>> devid    9 size 5.46TiB used 5.42TiB path /dev/sdk >>>> devid   10 size 5.46TiB used 5.42TiB path /dev/sdj >>>> devid   11 size 7.28TiB used 7.24TiB path /dev/sdd >>>> devid   12 size 5.46TiB used 5.42TiB path /dev/sdm >>>> devid   14 size 7.28TiB used 7.24TiB path /dev/sdh >>>> devid   15 size 7.28TiB used 7.24TiB path /dev/sdb >>>> devid   17 size 7.28TiB used 5.77TiB path /dev/sdl >>>> devid   18 size 7.28TiB used 7.24TiB path /dev/sdf >>>> devid   20 size 7.28TiB used 7.24TiB path /dev/sdi >>>> devid   21 size 7.28TiB used 7.24TiB path /dev/sdg >>>> >>>> Thanks, >>>> Asif >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at  http://vger.kernel.org/majordomo-info.html