Help repairing a partition

Help repairing a partition

[Suvayu Ali]

Hi,

(please CC me in replies, I'm not subscribed)

I'm using kernel 4.7.7-100.fc23 with btrfs-progs v4.7.1.

I had my /home, /var, and /opt as subvolumes in a btrfs partition.
Last night btrfs failed, and I was unable to mount it normally
(leading to boot failures).  The journal had messages like this:

  BTRFS: open_ctree failed
  BTRFS error: super_total_bytes ... mismatch with fs_devices total_rw_bytes
  BTRFS error: failed to read chunk tree: -22

Finally I managed to mount it manually like this (after making a dd
image of the partition):

  # mount -t btrfs -o ro,recovery,nospace_cache /dev/sdb2 /mnt

and managed to recover my data. Initially "btrfs check" yielded a few

  parent transid verify failed on 101679726592 wanted 822619 found 822617

and

  checksum verify failed on 101756387328 found 78C8A0BC wanted B7C59D79

however after backing up my data, I mounted without the "-o ro" (I got
a transid related message, but it did mount).  "btrfs check" now spits
out a whole bunch of:

  Incorrect local backref count on 202118008832 root 259 owner 178928
offset 41181184 found 2 wanted 7 back 0x55713fbbf150
  Incorrect global backref count on 202118008832 found 2 wanted 7
  backpointer mismatch on [202118008832 376832]

followed by this summary:

checking csums
checking root refs
checking quota groups
Counts for qgroup id: 0/257 are different
our:            referenced 7746465792 referenced compressed 7746465792
disk:           referenced 7746461696 referenced compressed 7746461696
diff:           referenced 4096 referenced compressed 4096
our:            exclusive 7746465792 exclusive compressed 7746465792
disk:           exclusive 7746461696 exclusive compressed 7746461696
diff:           exclusive 4096 exclusive compressed 4096
Counts for qgroup id: 0/259 are different
our:            referenced 135641784320 referenced compressed 135641784320
disk:           referenced 135633862656 referenced compressed 135633862656
diff:           referenced 7921664 referenced compressed 7921664
our:            exclusive 135641784320 exclusive compressed 135641784320
disk:           exclusive 135633862656 exclusive compressed 135633862656
diff:           exclusive 7921664 exclusive compressed 7921664
found 167864082432 bytes used err is 0
total csum bytes: 161187492
total tree bytes: 2021015552
total fs tree bytes: 1725759488
total extent tree bytes: 86228992
btree space waste bytes: 386160897
file data blocks allocated: 1269363683328
 referenced 164438126592

How do I repair this?  Any thoughts and guidance would be greatly
appreciated.  I am not well versed with all the btrfs commands and
utilities, so I hope I have managed to provide all the right
information.

Thanks,

PS: I see that it now it mounts normally as well!  As in, with default
fstab options, so I guess I can boot.  I would still like to repair
the errors.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Help repairing a partition

[Chris Murphy]

On Thu, Oct 20, 2016 at 3:20 PM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
> Hi,
>
> (please CC me in replies, I'm not subscribed)
>
> I'm using kernel 4.7.7-100.fc23 with btrfs-progs v4.7.1.
>
> I had my /home, /var, and /opt as subvolumes in a btrfs partition.
> Last night btrfs failed, and I was unable to mount it normally
> (leading to boot failures).  The journal had messages like this:
>
>   BTRFS: open_ctree failed
>   BTRFS error: super_total_bytes ... mismatch with fs_devices total_rw_bytes
>   BTRFS error: failed to read chunk tree: -22
>
> Finally I managed to mount it manually like this (after making a dd
> image of the partition):
>
>   # mount -t btrfs -o ro,recovery,nospace_cache /dev/sdb2 /mnt
>
> and managed to recover my data. Initially "btrfs check" yielded a few
>
>   parent transid verify failed on 101679726592 wanted 822619 found 822617
>
> and
>
>   checksum verify failed on 101756387328 found 78C8A0BC wanted B7C59D79
>
> however after backing up my data, I mounted without the "-o ro" (I got
> a transid related message, but it did mount).  "btrfs check" now spits
> out a whole bunch of:
>
>   Incorrect local backref count on 202118008832 root 259 owner 178928
> offset 41181184 found 2 wanted 7 back 0x55713fbbf150
>   Incorrect global backref count on 202118008832 found 2 wanted 7
>   backpointer mismatch on [202118008832 376832]


This is a known problem with btrfs-progs 4.7.1 it should not be used.
https://btrfs.wiki.kernel.org/index.php/Changelog#btrfs-progs_4.7.1_.28Aug_2016.29

Upgrade to 4.7.3 or 4.8.1 is advised.




>
> followed by this summary:
>
> checking csums
> checking root refs
> checking quota groups
> Counts for qgroup id: 0/257 are different
> our:            referenced 7746465792 referenced compressed 7746465792
> disk:           referenced 7746461696 referenced compressed 7746461696
> diff:           referenced 4096 referenced compressed 4096
> our:            exclusive 7746465792 exclusive compressed 7746465792
> disk:           exclusive 7746461696 exclusive compressed 7746461696
> diff:           exclusive 4096 exclusive compressed 4096
> Counts for qgroup id: 0/259 are different
> our:            referenced 135641784320 referenced compressed 135641784320
> disk:           referenced 135633862656 referenced compressed 135633862656
> diff:           referenced 7921664 referenced compressed 7921664
> our:            exclusive 135641784320 exclusive compressed 135641784320
> disk:           exclusive 135633862656 exclusive compressed 135633862656
> diff:           exclusive 7921664 exclusive compressed 7921664
> found 167864082432 bytes used err is 0
> total csum bytes: 161187492
> total tree bytes: 2021015552
> total fs tree bytes: 1725759488
> total extent tree bytes: 86228992
> btree space waste bytes: 386160897
> file data blocks allocated: 1269363683328
>  referenced 164438126592
>
> How do I repair this?

Yeah good question. I can't tell from the message whether different
counts is a bad thing, or if it's just a notification, or what. Yet
again btrfs-progs does not help the user make informed decisions, it's
really frustrating. I think that part can be ignored though for now,
and see if btrfs check --repair can fix the problem now that you have
a backup.


-- 
Chris Murphy

Re: Help repairing a partition

[Suvayu Ali]

Hi Chris,

Thanks for your response :).

On 21 October 2016 at 05:18, Chris Murphy <lists@colorremedies.com> wrote:
> On Thu, Oct 20, 2016 at 3:20 PM, Suvayu Ali <fatkasuvayu+linux@gmail.com> wrote:
>> Hi,
>>
>> (please CC me in replies, I'm not subscribed)
>>
>> I'm using kernel 4.7.7-100.fc23 with btrfs-progs v4.7.1.
>>
>> I had my /home, /var, and /opt as subvolumes in a btrfs partition.
>> Last night btrfs failed, and I was unable to mount it normally
>> (leading to boot failures).  The journal had messages like this:
>>
>>   BTRFS: open_ctree failed
>>   BTRFS error: super_total_bytes ... mismatch with fs_devices total_rw_bytes
>>   BTRFS error: failed to read chunk tree: -22
>>
>> Finally I managed to mount it manually like this (after making a dd
>> image of the partition):
>>
>>   # mount -t btrfs -o ro,recovery,nospace_cache /dev/sdb2 /mnt
>>
>> and managed to recover my data. Initially "btrfs check" yielded a few
>>
>>   parent transid verify failed on 101679726592 wanted 822619 found 822617
>>
>> and
>>
>>   checksum verify failed on 101756387328 found 78C8A0BC wanted B7C59D79
>>
>> however after backing up my data, I mounted without the "-o ro" (I got
>> a transid related message, but it did mount).  "btrfs check" now spits
>> out a whole bunch of:
>>
>>   Incorrect local backref count on 202118008832 root 259 owner 178928
>> offset 41181184 found 2 wanted 7 back 0x55713fbbf150
>>   Incorrect global backref count on 202118008832 found 2 wanted 7
>>   backpointer mismatch on [202118008832 376832]
>
>
> This is a known problem with btrfs-progs 4.7.1 it should not be used.
> https://btrfs.wiki.kernel.org/index.php/Changelog#btrfs-progs_4.7.1_.28Aug_2016.29
>
> Upgrade to 4.7.3 or 4.8.1 is advised.

I had upgraded to 4.7.3 to test this issue:

  https://bugzilla.redhat.com/show_bug.cgi?id=1372910

It hadn't helped, but I didn't have time to debug it any further.
Since the Fedora 23 repos have 4.4.1, I guess downgrading is easier
for me.

Thanks for the pointer to the changelog; under 4.7.2 it mentions not
to repair with 4.7.1, so I'll try `btrfs check --repair` after the
downgrade.

>> followed by this summary:
>>
>> checking csums
>> checking root refs
>> checking quota groups
>> Counts for qgroup id: 0/257 are different
>> our:            referenced 7746465792 referenced compressed 7746465792
>> disk:           referenced 7746461696 referenced compressed 7746461696
>> diff:           referenced 4096 referenced compressed 4096
>> our:            exclusive 7746465792 exclusive compressed 7746465792
>> disk:           exclusive 7746461696 exclusive compressed 7746461696
>> diff:           exclusive 4096 exclusive compressed 4096
>> Counts for qgroup id: 0/259 are different
>> our:            referenced 135641784320 referenced compressed 135641784320
>> disk:           referenced 135633862656 referenced compressed 135633862656
>> diff:           referenced 7921664 referenced compressed 7921664
>> our:            exclusive 135641784320 exclusive compressed 135641784320
>> disk:           exclusive 135633862656 exclusive compressed 135633862656
>> diff:           exclusive 7921664 exclusive compressed 7921664
>> found 167864082432 bytes used err is 0
>> total csum bytes: 161187492
>> total tree bytes: 2021015552
>> total fs tree bytes: 1725759488
>> total extent tree bytes: 86228992
>> btree space waste bytes: 386160897
>> file data blocks allocated: 1269363683328
>>  referenced 164438126592
>>
>> How do I repair this?
>
> Yeah good question. I can't tell from the message whether different
> counts is a bad thing, or if it's just a notification, or what. Yet
> again btrfs-progs does not help the user make informed decisions, it's
> really frustrating. I think that part can be ignored though for now,
> and see if btrfs check --repair can fix the problem now that you have
> a backup.

Indeed, I have never been this confused about a file system before.

I tried repairing after the downgrade to 4.4.1, it says "Couldn't open
file system"!  Mounting now works without errors, I can also r/w files
as normal; go figure!

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Help repairing a partition

[Chris Murphy]

On Fri, Oct 21, 2016 at 12:36 AM, Suvayu Ali
<fatkasuvayu+linux@gmail.com> wrote:

> I had upgraded to 4.7.3 to test this issue:
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=1372910
>
> It hadn't helped, but I didn't have time to debug it any further.
> Since the Fedora 23 repos have 4.4.1, I guess downgrading is easier
> for me.

Better is to go to http://koji.fedoraproject.org/ and type in
btrfs-progs for the package, and find the most recent x.y-1.z version
- right now that's 4.7.3, although 4.8.1 is probably OK also - it has
no new features, mainly just a pile of bug fixes, which might be
useful. So that'd be either:
btrfs-progs-4.8.1-2.fc26
or
btrfs-progs-4.7.3-1.fc26

And rpmbuild --rebuild them for F23 and then install. I would not
downgrade to 4.4.1 - it's not that it's bad, it's just a waste of time
if it can't help fix the problem which is very likely the older progs
you have.


>
> Thanks for the pointer to the changelog; under 4.7.2 it mentions not
> to repair with 4.7.1, so I'll try `btrfs check --repair` after the
> downgrade.

No. The older the progs the less safe the repair is. And this
particular problem you have probably needs a newer progs to fix it
anyway. So you need to go newer not older. That's pretty much always
the case with Btrfs.


>
>>> followed by this summary:
>>>
>>> checking csums
>>> checking root refs
>>> checking quota groups
>>> Counts for qgroup id: 0/257 are different
>>> our:            referenced 7746465792 referenced compressed 7746465792
>>> disk:           referenced 7746461696 referenced compressed 7746461696
>>> diff:           referenced 4096 referenced compressed 4096
>>> our:            exclusive 7746465792 exclusive compressed 7746465792
>>> disk:           exclusive 7746461696 exclusive compressed 7746461696
>>> diff:           exclusive 4096 exclusive compressed 4096
>>> Counts for qgroup id: 0/259 are different
>>> our:            referenced 135641784320 referenced compressed 135641784320
>>> disk:           referenced 135633862656 referenced compressed 135633862656
>>> diff:           referenced 7921664 referenced compressed 7921664
>>> our:            exclusive 135641784320 exclusive compressed 135641784320
>>> disk:           exclusive 135633862656 exclusive compressed 135633862656
>>> diff:           exclusive 7921664 exclusive compressed 7921664
>>> found 167864082432 bytes used err is 0
>>> total csum bytes: 161187492
>>> total tree bytes: 2021015552
>>> total fs tree bytes: 1725759488
>>> total extent tree bytes: 86228992
>>> btree space waste bytes: 386160897
>>> file data blocks allocated: 1269363683328
>>>  referenced 164438126592
>>>
>>> How do I repair this?
>>
>> Yeah good question. I can't tell from the message whether different
>> counts is a bad thing, or if it's just a notification, or what. Yet
>> again btrfs-progs does not help the user make informed decisions, it's
>> really frustrating. I think that part can be ignored though for now,
>> and see if btrfs check --repair can fix the problem now that you have
>> a backup.
>
> Indeed, I have never been this confused about a file system before.
>
> I tried repairing after the downgrade to 4.4.1, it says "Couldn't open
> file system"!  Mounting now works without errors, I can also r/w files
> as normal; go figure!

Oh shit. That's hilarious. I'm not even going to edit what I wrote above.

Anyway, it looks like you have quotas enabled. There are a number of
quota related bug fixes in progs newer than 4.4, so you really ought
to use something newer, and if it breaks then it's a bug and needs a
good bug report write up so it can get fixed.

In the meantime I would be wary with this file system if it's the only
backup copy. (Actually I feel that way no matter the file system.) I'd
make sure btrfs check with progs 4.7.3 or 4.8.1 come up clean (i.e.
err is 0 is generally a good sign), and that a scrub also comes up
clean with no errors: either 'btrfs scrub start <mp>' and then later
check with 'btrfs scrub status' or use -BR flag to not background and
show stats after completion.



-- 
Chris Murphy