[Help] Errors found in extent allocation tree or chunk allocation

[Help] Errors found in extent allocation tree or chunk allocation

[Michael Witten]

Firstly, it should be noted that I can mount and use my Btrfs file
system with nary an error or warning; however, I'm uncomfortable
using it while it's in some kind of inconsistent state.

The `btrfsck' tool is telling me the following:

  Errors found in extent allocation tree or chunk allocation

In particular, I'm receiving 69158 messages of the following form:

  Extent back ref already exists for <a> parent <b> root <c>

All of those are followed by the same number of messages of the
following form:

  ref mismatch on [<d> 4096] extent item <e>, found <f>
  Incorrect global backref count on <d> found <e> wanted <f>
  backpointer mismatch on [<d> 4096]

I've run several commands in an attempt to repair/recover the file
system, but nothing seems to help.

What is the meaning of these errors, and how should I fix them?

Sincerely,
Michael Witten

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Duncan]

Michael Witten posted on Mon, 31 Mar 2014 17:39:05 +0000 as excerpted:

> Firstly, it should be noted that I can mount and use my Btrfs file
> system with nary an error or warning; however, I'm uncomfortable using
> it while it's in some kind of inconsistent state.
> 
> The `btrfsck' tool is telling me the following:
> 
>   Errors found in extent allocation tree or chunk allocation
> 
> In particular, I'm receiving 69158 messages of the following form:
> 
>   Extent back ref already exists for <a> parent <b> root <c>
> 
> All of those are followed by the same number of messages of the
> following form:
> 
>   ref mismatch on [<d> 4096] extent item <e>, found <f>
>   Incorrect global backref count on <d> found <e> wanted <f>
>   backpointer mismatch on [<d> 4096]

FWIW, while I'm not sure about those particular errors (I'll let someone 
else tackle that end), btrfsck doesn't yet know about some kinds of 
errors and can also misinterpret some structures it doesn't understand as 
errors when they actually aren't.

IOW, it's a useful tool for the devs and for the curious types, but don't 
be too alarmed by what it says, particularly if the filesystem seems to 
be operating fine in other respects.

That goes double if you're running raid5/6 mode, since support for it in 
general is known to be incomplete as yet -- routine operations work but 
there are known holes in its device drop recovery and scrub support, 
among other things, as the code simply isn't all there yet.  Since btrfsck 
is typically one of the later things on the implementation list, I'd 
expect it to not understand and thus report as errors quite a bit of the 
extended raid5/6 structure.

> I've run several commands in an attempt to repair/recover the file
> system, but nothing seems to help.

Hopefully, none of those commands included btrfsck --repair, because that 
is known to actually make things worse in some instances.  That's the 
last thing to try, generally after you've posted here and gotten 
confirmation that the errors it reports are actual errors and that it can 
actually fix them, without making anything else it's reporting worse.  
Alternatively, it's what you run right before you give up hope and do a 
mkfs, since if that's your next step, your risk is essentially zero 
anyway.

> What is the meaning of these errors, and how should I fix them?

I'd actually put more faith in the btrfs balance command (assuming you 
aren't running raid5/6 mode anyway, I'd consider it effectively raid0, 
only put stuff on it that you expect to be able to throw away and do a 
new mkfs over, if there's errors, tho once that's your expectation it 
doesn't hurt to experiment).  Balance won't fix everything either, but it 
shouldn't make things worse, and since it rewrites all chunks both data 
and metadata, if it finishes without complaining about anything and 
you're not seeing any problems elsewhere, you're reasonably safe in 
assuming you can ignore whatever "errors" btrfsck is 
/claiming/ you have.

Btrfs scrub is another alternative (altho as I said not for raid5/6 yet 
as the support there is known to be unfinished and it WILL complain about 
errors that aren't there!), validating checksums on everything, tho while 
it can spot data corruption it only actually fixes the problem on data/
metadata that has a second, checksum-valid copy, around.  On a default 
btrfs that'll be all metadata chunks since they're default-dup mode 
(except on SSD) on a single device btrfs, and default-raid1 (2-way-
mirrored) on a multi-device btrfs, while actual data always defaults to 
single mode, so it's checksummed but without a backup if there's 
corruption.  Of course, if you have both data and metadata in raid1 or 
raid10 mode (or if you're running a mixed-data/metadata chunk filesystem, 
the default for filesystems under a gig, which further defaults to dup 
mode if on a single device), there should be a second copy of data as 
well, and (assuming the checksum verifies on the second copy) scrub can 
restore from that.

FWIW, most of my btrfs are raid1 mode both data/metadata here, for just 
that reason.  And I'm on SSD (tho my media partition and second level 
backups are still on reiserfs on slower spinning rust) too, with the 
already reasonably small 256 GB SSDs partitioned up with all partitions 
50 GiB or under, so both full balances and scrubs only take a few (under 
10, under 1 on the smallest partitions) minutes, not the hours or even 
days they can take on single-partition multi-terabyte spinning rust.

Of course word of warning if it applies, that's why people with whole-
drive TB+ spinning-rust drives tend not to balance or scrub as often -- 
it takes too long.  Luckily the time is short enough here I don't worry 
about that.

Anyway, here's hoping you get a real answer on your specific errors, but 
until then, just know that some of the things btrfsck reports aren't 
actually errors at all, just things it doesn't understand yet.  So don't 
get too worried unless you either have other problems or one of the devs 
or other regulars explains and says you really SHOULD be worried about 
them.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Michael Witten]

On Mon, 31 Mar 2014 19:45:27 +0000 (UTC), Duncan wrote:

> Hopefully, none of those commands included btrfsck --repair

... Woops...

Fortunately, it doesn't seem to have changed anything.

> That goes double if you're running raid5/6 mode

For the record, I've essentially just put Btrfs on 1 block device
with no special RAID structuring (I suppose that would be, in
your parlance, `single mode').

For completeness, this is the actual structure:

  Raw disk 
    |
    +-> MSDOS partition
          |
          +-> LVM2 logical volume
                |
                +-> LUKS dm-crypt device
                      |
                      +-> Btrfs

I apologize if that ASCII art fails to render properly on your end.

> I'd actually put more faith in the btrfs balance command...
> Btrfs scrub is another alternative

I suppose, then, that my ultimate plan of action is thus:

  * Back up important data.

  * Give `btrfs scrub' a try.

  * Give `btrfs balance' a try.

  * Rebuild the file system if necessary.

> or if you're running a mixed-data/metadata chunk filesystem, the
> default for filesystems under a gig, which further defaults to
> dup mode if on a single device

Just as an aside, I find it odd that the default for such a small
system would be to duplicate user data.

--------

Thanks for your insights!

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Duncan]

Michael Witten posted on Mon, 31 Mar 2014 20:33:39 +0000 as excerpted:

> Just as an aside, I find it odd that the default for such a small system
> would be to duplicate user data.

I've wondered at that "logical accident" too, but the problem was that 
data chunks default to a gig in size and metadata to a quarter gig (but 
two chunks allocated at once since it's default-dup, so a half-gig).  
While allocations can and do get smaller to use the last bit of space 
left, at under a gig size, separate data/metadata blocks simply weren't 
flexible enough, and a shared data/metadata block type eliminated the 
separate data/metadata block flexibility issues at those tiny sizes.

Of course that did result in data being duped by default as well, since 
it was now sharing metadata chunks and metadata rules were applied to the 
shared chunks, but I guess that was figured to be the less bothersome 
problem, compared to the flexibility issues of separate data/metadata and 
the seriously increased risk of single metadata.

Of course one can specify single mode for shared/mixed too, if the extra 
metadata risk of just the single copy is considered less of an issue than 
the space usage of dup mode, but given the available tradeoffs, while I 
certainly appreciate the irony in the smallest btrfs being the only ones 
getting dup-data by default, I still agree with the choice and consider 
it the most sane one available for the general case, given the domain 
constraints they were working with.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Duncan]

Michael Witten posted on Mon, 31 Mar 2014 20:33:39 +0000 as excerpted:

> I apologize if that ASCII art fails to render properly on your end.

FWIW I make a point of configuring monospace fonts as default for 
internet message (Mail and news) bodies, precisely so I CAN properly view 
ascii-art.  So it came thru very nicely here. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Michael Witten]

On Mon, Mar 31, 2014 at 9:16 PM, Duncan wrote:

> FWIW I make a point of configuring monospace fonts as default for
> internet message (Mail and news) bodies, precisely so I CAN properly view
> ascii-art.  So it came thru very nicely here. =:^)


That's good to know; unfortunately, you can never be too sure these days.

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Michael Witten]

>> I'd actually put more faith in the btrfs balance command...
>> Btrfs scrub is another alternative
>
> I suppose, then, that my ultimate plan of action is thus:
>
>   * Back up important data.
>
>   * Give `btrfs scrub' a try.
>
>   * Give `btrfs balance' a try.
>
>   * Rebuild the file system if necessary.

The `btrfs scrub' completed successfully without error, but did not
solve my issues.

The `btrfs balance' completed successfully without error, and DID
solve my issues; it "relocated" every chunk, after which `btrfsck'
ran smoothly.

Thanks for the advice! You've put me at ease, and you've saved me
a lot of time and energy.

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Michael Witten]

On Tue, 1 Apr 2014 07:15:47 +0800, Shilong Wang wrote:

> hello, recently, i was working on btrfsck repair. i post some
> patches to make btrfsck to rebuild extent tree with snapshots.
> you can pull code from david latest branch and have a try with
> btrfsck --init-extent-tree.

Fortunately, the `btrfs balance' command solved the issue for me:

  http://mid.gmane.org/<8e4932bb29f743419660da1846c2363b-mfwitten@gmail.com>

However, I did notice `--init-extent-tree' when trying to figure
out what I might do to fix my troubles; however, I was uncertain
about exactly what it would do.

According to `man btrfsck':

  --init-csum-tree
       create a new CRC tree.

  --init-extent-tree
       create a new extent tree.

These sound potentially destructive, so I was afraid to use them;
what exactly does it mean to create a new tree? After your email,
I assume each action creates a new tree that is populated from the
data in the corresponding old tree.

Also, you wrote:

> rebuild extent tree with snapshots

Do you mean to say that your patches handle the previously unhandled
case in which both of the following are true?

  * There are snapshots in the file system.
  * The extent tree needs to be rebuilt.

Thanks!

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Wang Shilong]

Hello Michael,

On 04/02/2014 03:26 AM, Michael Witten wrote:
> On Tue, 1 Apr 2014 07:15:47 +0800, Shilong Wang wrote:
>
>> hello, recently, i was working on btrfsck repair. i post some
>> patches to make btrfsck to rebuild extent tree with snapshots.
>> you can pull code from david latest branch and have a try with
>> btrfsck --init-extent-tree.
> Fortunately, the `btrfs balance' command solved the issue for me:
>
>    http://mid.gmane.org/<8e4932bb29f743419660da1846c2363b-mfwitten@gmail.com>
>
> However, I did notice `--init-extent-tree' when trying to figure
> out what I might do to fix my troubles; however, I was uncertain
> about exactly what it would do.
>
> According to `man btrfsck':
>
>    --init-csum-tree
>         create a new CRC tree.
>
>    --init-extent-tree
>         create a new extent tree.
>
> These sound potentially destructive, so I was afraid to use them;
> what exactly does it mean to create a new tree? After your email,
> I assume each action creates a new tree that is populated from the
> data in the corresponding old tree.

We may have a totally broken Extent tree, then you could not even
mount your filesystem anymore.

'--init-extent-tree' will recreate a new tree and then re-insert 
everything according to other fs/file tree.

>
> Also, you wrote:
>
>> rebuild extent tree with snapshots
> Do you mean to say that your patches handle the previously unhandled
> case in which both of the following are true?

Previouly, '--init-extent-tree' did not handle it properly if there is 
snapshots in the filesystem.

Thanks,
Wang
>
>    * There are snapshots in the file system.
>    * The extent tree needs to be rebuilt.
>
> Thanks!
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [Help] Errors found in extent allocation tree or chunk allocation

[Duncan]

Michael Witten posted on Tue, 01 Apr 2014 19:05:16 +0000 as excerpted:

> The `btrfs balance' completed successfully without error, and DID solve
> my issues; it "relocated" every chunk, after which `btrfsck'
> ran smoothly.
> 
> Thanks for the advice! You've put me at ease, and you've saved me a lot
> of time and energy.

Very good! =:^)  Thanks for the fix-report.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman