Problem with a filesystem

Problem with a filesystem

[Tovo Rabemanantsoa]

Hi all,
I have a server running debian 7.5 and meet issues for mounting a btrfs
filesystem. This fs had worked for almost one year now but crashed last
sunday. I'm unable to mount the fs and each btrfs-* commands ends with
something like :
Ignoring transid failure
btrfsck: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted

Ignoring transid failure
btrfs-zero-log: ctree.c:1550: leaf_space_used: Assertion `!(data_len <
0)' failed.
Aborted


Ignoring transid failure
btrfs: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
Aborted

So, I can't fsck, restore, repair ou erase logs on it.
I have followed what is told here [1] or there [2] but now, I'm out of
ideas.
Any help is appreciated.
Thanks in advance


[1]http://ram.kossboss.com/btrfs-transid-issue-explained-fix/
[2]https://btrfs.wiki.kernel.org/index.php/Restore

Re: Problem with a filesystem

[Chris Murphy]

On Sep 16, 2014, at 9:54 AM, Tovo Rabemanantsoa <tovo.rabemanantsoa@bordeaux.inra.fr> wrote:

> Hi all,
> I have a server running debian 7.5 and meet issues for mounting a btrfs
> filesystem. This fs had worked for almost one year now but crashed last
> sunday. I'm unable to mount the fs and each btrfs-* commands ends with
> something like :
> Ignoring transid failure
> btrfsck: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
> Aborted
> 
> Ignoring transid failure
> btrfs-zero-log: ctree.c:1550: leaf_space_used: Assertion `!(data_len <
> 0)' failed.
> Aborted
> 
> 
> Ignoring transid failure
> btrfs: ctree.c:1550: leaf_space_used: Assertion `!(data_len < 0)' failed.
> Aborted
> 
> So, I can't fsck, restore, repair ou erase logs on it.
> I have followed what is told here [1] or there [2] but now, I'm out of
> ideas.

Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?

Chris Murphy

Re: Problem with a filesystem

[Tovo Rabemanantsoa]

On 09/16/2014 07:17 PM, Chris Murphy wrote:
> 
> Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?
> 
> Chris Murphy
You're right,
In the beginning, I used the btrfs-progs provided by my debian (kernel
3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git
one (3.16).
More information, 'btrfs filesystem show' gives :

Label: none  uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3
	Total devices 1 FS bytes used 18.67TiB
	devid    1 size 25.34TiB used 22.47TiB path /dev/sda4

Btrfs v3.16

Re: Problem with a filesystem

[Chris Murphy]

On Sep 16, 2014, at 2:11 PM, Tovo Rabemanantsoa <tovo.rabemanantsoa@bordeaux.inra.fr> wrote:

> On 09/16/2014 07:17 PM, Chris Murphy wrote:
>> 
>> Debian 7.5 is based on kernel 3.2 which is very old in Btrfs terms. What kernel version are you using, and version of btrfs-progs?
>> 
>> Chris Murphy
> You're right,
> In the beginning, I used the btrfs-progs provided by my debian (kernel
> 3.2.0-4 and btrfs version 0.19) but later, I compiled and used the git
> one (3.16).
> More information, 'btrfs filesystem show' gives :
> 
> Label: none  uuid: 9ddd403a-2863-4e71-b28d-d2931a133af3
> 	Total devices 1 FS bytes used 18.67TiB
> 	devid    1 size 25.34TiB used 22.47TiB path /dev/sda4


Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.

Chris Murphy

Re: Problem with a filesystem

[Tovo Rabemanantsoa]

On 09/16/2014 11:27 PM, Chris Murphy wrote:
> 
> Try to mount normally, then with -o recovery, then with -o ro,recovery. Include dmesg showing any messages that appear for these attempts.
Thank you,
Please find below the messages shown during a normal mount and a mount
with -o ro,recovery.

======== Normal Mount ========
/var/log/messages
Sep 17 11:00:24 sdeeph1 kernel: [85051.529541] device fsid
9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4
Sep 17 11:00:24 sdeeph1 kernel: [85051.529917] btrfs: disk space caching
is enabled
Sep 17 11:01:09 sdeeph1 kernel: [85096.286741] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286756] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286764] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286770] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.286775] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:01:09 sdeeph1 kernel: [85096.408271] btrfs: open_ctree failed

/var/log/syslog adds this line :
Sep 17 11:01:09 sdeeph1 kernel: [85096.286782] Failed to read block
groups: -5


======== mount -o ro,recovery ========
/var/log/messages
Sep 17 11:05:10 sdeeph1 kernel: [85337.307267] device fsid
9ddd403a-2863-4e71-b28d-d2931a133af3 devid 1 transid 255446 /dev/sda4
Sep 17 11:05:10 sdeeph1 kernel: [85337.307635] btrfs: enabling auto recovery
Sep 17 11:05:10 sdeeph1 kernel: [85337.307639] btrfs: disk space caching
is enabled
Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify
failed on 24562568384512 wanted 255444 found 255446
Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed

/var/log/syslog adds this line :
Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read block
groups: -5

Re: Problem with a filesystem

[Chris Murphy]

On Sep 17, 2014, at 3:06 AM, Tovo Rabemanantsoa <tovo.rabemanantsoa@bordeaux.inra.fr> wrote:
> 
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548215] parent transid verify
> failed on 24562568384512 wanted 255444 found 255446
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548229] parent transid verify
> failed on 24562568384512 wanted 255444 found 255446
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548237] parent transid verify
> failed on 24562568384512 wanted 255444 found 255446
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548243] parent transid verify
> failed on 24562568384512 wanted 255444 found 255446
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548249] parent transid verify
> failed on 24562568384512 wanted 255444 found 255446
> Sep 17 11:05:55 sdeeph1 kernel: [85381.670344] btrfs: open_ctree failed
> 
> /var/log/syslog adds this line :
> Sep 17 11:05:55 sdeeph1 kernel: [85381.548255] Failed to read block
> groups: -5

This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a 
btrfs check <btrfsdev>
and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results

Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.

btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
btrfs-zero-log <btrfsdev>


Chris Murphy

Re: Problem with a filesystem

[Tovo Rabemanantsoa]

On 09/17/2014 08:49 PM, Chris Murphy wrote:
> 
> This isn't what I expect as a candidate for btrfs-zero-log, I don't know what the last message means. You could run a 
> btrfs check <btrfsdev>
> and then ask about both the failure to read block groups -5 and also the btrfs check (without --repair) results
> 
> Or take a leap of faith, take an image of the file system first as it might make recovery more difficult. If there are things you really need off this file system first you should look at https://btrfs.wiki.kernel.org/index.php/Restore before zeroing the log.
> 
> btrfs-image -c 9 -t 8 <btrfsdev> /path/to/file
> btrfs-zero-log <btrfsdev>
> 
> 
> Chris Murphy
Well,
btrfs check ends also with a failed message.
In any ways, I think this FS corruption is due to a weakness of my raid
card. Indeed, before it happens, the kernel panicked because of this
card. The card claims that he had lost 2 disks from the pool but, just
removing them and plugging them again solved the error.
My thinking is that I didn't plug the two disks on the proper sequence.
So it corrupted the raid reconstruction process.
By the way, I don't have enough structure to host 26TB of an image.
Moreover, as it is our backup server, we need it to get back as soon as
possible.
What I'm gonna do, now, is to make a new file system instead and set up
everything one more time.
Thank you so much for helping me.

Tovo