Rebuilding chunk root?

Rebuilding chunk root?

[Sami Haahtinen]

Hi,

Due to certain unfortunate chain of events, I managed to overwrite a
small portion of my btrfs array which had only single redundancy for
metadata. The data itself is present and only a small portion (2.5%)
of the array was overwritten.

After quite a bit of debugging and tinkering, I realized that my chunk
root was in the portion that was overwritten. After reading through
the documentation I was able to pull together it's still unclear to me
whether chunk root is something that can be rebuilt.

A transcript of btrfsck trying to recover with superblock 2 which is
uncorrupted by itself:

root@sysresccd /root/btrfs-progs % ./btrfsck --super 2 /dev/patience/home
using SB copy 2, bytenr 274877906944
Check tree block failed, want=139264, have=0
Check tree block failed, want=139264, have=0
Check tree block failed, want=139264, have=0
read block failed check_tree_block
Couldn't read chunk root

If I'm interpreting the output correctly, it's trying to read bytes
from address 139264, which would fall into the corrupted area.

Regards,
--
Sami Haahtinen
Bad Wolf Oy
+358443302775

Re: Rebuilding chunk root?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2000 bytes --]

On Mon, Sep 24, 2012 at 04:28:08PM +0300, Sami Haahtinen wrote:
> Due to certain unfortunate chain of events, I managed to overwrite a
> small portion of my btrfs array which had only single redundancy for
> metadata. The data itself is present and only a small portion (2.5%)
> of the array was overwritten.
> 
> After quite a bit of debugging and tinkering, I realized that my chunk
> root was in the portion that was overwritten. After reading through
> the documentation I was able to pull together it's still unclear to me
> whether chunk root is something that can be rebuilt.

   Chris had some experimental code for doing it in btrfsck which
never saw the light of day (because it was too unreliable). He may
be able to offer you something to help, though.

> A transcript of btrfsck trying to recover with superblock 2 which is
> uncorrupted by itself:
> 
> root@sysresccd /root/btrfs-progs % ./btrfsck --super 2 /dev/patience/home
> using SB copy 2, bytenr 274877906944
> Check tree block failed, want=139264, have=0
> Check tree block failed, want=139264, have=0
> Check tree block failed, want=139264, have=0
> read block failed check_tree_block
> Couldn't read chunk root
> 
> If I'm interpreting the output correctly, it's trying to read bytes
> from address 139264, which would fall into the corrupted area.

   No, I believe the "want=, have=" text is referring to a generation
ID, not a block number. That's not to say that your chunk tree isn't
damaged, though -- I'm just clarifying your interpretation of the
numbers.

   Out of interest, does mounting with -o recovery help at all? (I'm
not expecting it to do much if your chunk tree's gone, but it might do
something).

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
      --- Eighth Army Push Bottles Up Germans -- WWII newspaper ---      
                     headline (possibly apocryphal)                      

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: Rebuilding chunk root?

[David Sterba]

On Mon, Sep 24, 2012 at 03:02:39PM +0100, Hugo Mills wrote:
> > root@sysresccd /root/btrfs-progs % ./btrfsck --super 2 /dev/patience/home
> > using SB copy 2, bytenr 274877906944
> > Check tree block failed, want=139264, have=0
> > Check tree block failed, want=139264, have=0
> > Check tree block failed, want=139264, have=0
> > read block failed check_tree_block
> > Couldn't read chunk root
> > 
> > If I'm interpreting the output correctly, it's trying to read bytes
> > from address 139264, which would fall into the corrupted area.
> 
>    No, I believe the "want=, have=" text is referring to a generation
> ID, not a block number. That's not to say that your chunk tree isn't
> damaged, though -- I'm just clarifying your interpretation of the
> numbers.

  40 static int check_tree_block(struct btrfs_root *root, struct extent_buffer *buf)
  41 {
  42
  43         struct btrfs_fs_devices *fs_devices;
  44         int ret = 1;
  45
  46         if (buf->start != btrfs_header_bytenr(buf)) {
  47                 printk("Check tree block failed, want=%Lu, have=%Lu\n",
  48                        buf->start, btrfs_header_bytenr(buf));
  49                 return ret;
  50         }

No, it's the block address in bytes, 4k-block number 34.

>    Out of interest, does mounting with -o recovery help at all? (I'm
> not expecting it to do much if your chunk tree's gone, but it might do
> something).

The -o recovery has access to the respective tree roots, but the
contents may be destroyed already. The chunk tree is not deep, I can see
height 1 on a 6 disk array (though lightly used, 1 node, 8 leaves) and 3
disk array (1/7 TB used, 1 node, 29 leaves). So it's quite a small
amount of data to destroy the chunktree completely, COW will lower the
chances a bit.

Rebuilding from scratch does not look simple, the available information
is stored in BLOCK_GROUP_ITEMs or INODE_ITEMs and covers portions of the
chunks. Given that the device tree would be probably damaged as well,
the amount of information to do cross-check is not high. Maybe replaying
the chunk creation logic can save some guesswork.


david

Re: Rebuilding chunk root?

[Sami Haahtinen]

On Mon, Sep 24, 2012 at 6:12 PM, David Sterba <dave@jikos.cz> wrote:
> On Mon, Sep 24, 2012 at 03:02:39PM +0100, Hugo Mills wrote:
> >    Out of interest, does mounting with -o recovery help at all? (I'm
> > not expecting it to do much if your chunk tree's gone, but it might do
> > something).
>
> The -o recovery has access to the respective tree roots, but the
> contents may be destroyed already. The chunk tree is not deep, I can see
> height 1 on a 6 disk array (though lightly used, 1 node, 8 leaves) and 3
> disk array (1/7 TB used, 1 node, 29 leaves). So it's quite a small
> amount of data to destroy the chunktree completely, COW will lower the
> chances a bit.

Yeah, the whole tree is gone, I'm pretty sure of it since the first
20-50GB has been wiped from the drive and the mentioned address is in
the beginning of that part. I just wonder if there is any chance of
the older versions of the chunk tree still being somewhere and how to
find them. I doubt it's an easy feat though.

> Rebuilding from scratch does not look simple, the available information
> is stored in BLOCK_GROUP_ITEMs or INODE_ITEMs and covers portions of the
> chunks. Given that the device tree would be probably damaged as well,
> the amount of information to do cross-check is not high. Maybe replaying
> the chunk creation logic can save some guesswork.

Replaying chunk creation logic would not help that much, since the
drive has been resized a few times and had other operations that have
modified the chunk tree as well. The array itself is not that complex
(2 drives), but it's still not as simple as a single drive array.

Regards,
--
Sami Haahtinen
Bad Wolf Oy
+358443302775