segmentation-fault in btrfsck (git-version)

segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Dear all,

thanks for developing btrfsck!
Now, I'd like to contribute -as far as I can. I'm not a developer, but I 
do have some linux-experience.
I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under 
Kernel 3.0. Now it's corrupt. I had some hard resets of the machine 
-which might have contributed. I do have a backup of the data -at least 
of the important stuff. Some TV-Recordings are missing. The reason I am 
writing is, to support the development.

Unfortunately, btrfsck (latest git-version) crashes with a segmentation 
fault, when trying to repair this.

Here's the backtrace:
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
btrfsck: disk-io.c:382: __commit_transaction: Assertion `!(!eb || 
eb->start != start)' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb)
(gdb) backtrace
#0  0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007ffff7845192 in __assert_fail () from 
/lib/x86_64-linux-gnu/libc.so.6
#4  0x000000000040d3ae in __commit_transaction (trans=0x62e010, 
root=0xb66ae0) at disk-io.c:382
#5  0x000000000040d4d8 in btrfs_commit_transaction (trans=0x62e010, 
root=0xb66ae0) at disk-io.c:415
#6  0x000000000040743d in main (ac=<optimized out>, av=<optimized out>) 
at btrfsck.c:3587


Now, here's where my debugging knowledge ends. Are you interested in 
debugging this further, or is it a known bug?

Regards,
Hendrik
-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Mitch Harder]

On Wed, Dec 5, 2012 at 2:50 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
> Dear all,
>
> thanks for developing btrfsck!
> Now, I'd like to contribute -as far as I can. I'm not a developer, but I do
> have some linux-experience.
> I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under
> Kernel 3.0. Now it's corrupt. I had some hard resets of the machine -which
> might have contributed. I do have a backup of the data -at least of the
> important stuff. Some TV-Recordings are missing. The reason I am writing is,
> to support the development.
>
> Unfortunately, btrfsck (latest git-version) crashes with a segmentation
> fault, when trying to repair this.
>
> Here's the backtrace:
> root 261 inode 64375 errors 400
> root 261 inode 64376 errors 400
> btrfsck: disk-io.c:382: __commit_transaction: Assertion `!(!eb || eb->start
> != start)' failed.
>
> Program received signal SIGABRT, Aborted.
> 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb)
> (gdb) backtrace
> #0  0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2  0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3  0x00007ffff7845192 in __assert_fail () from
> /lib/x86_64-linux-gnu/libc.so.6
> #4  0x000000000040d3ae in __commit_transaction (trans=0x62e010,
> root=0xb66ae0) at disk-io.c:382
> #5  0x000000000040d4d8 in btrfs_commit_transaction (trans=0x62e010,
> root=0xb66ae0) at disk-io.c:415
> #6  0x000000000040743d in main (ac=<optimized out>, av=<optimized out>) at
> btrfsck.c:3587
>
>
> Now, here's where my debugging knowledge ends. Are you interested in
> debugging this further, or is it a known bug?
>

Line 382 in disk-io.c is:

BUG_ON(!eb || eb->start != start);

So, basically, btrfsck is intentionally crashing because it doesn't
know how to handle this condition.

Future refinements of btrfsck will probably include proper error
messages for issues that can't be handled, or perhaps even fix the
error.

It might be interesting for you to try a newer kernel, and use scrub
on this volume if you have the two disks RAIDed.

Re: segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Hello,

thanks for letting me know.
Indeed it would be good to replace the segmentation Fault by " btrfs 
does not yet know how to handle this condition.
Future refinements of btrfsck will probably include proper error
messages for issues that can't be handled, or perhaps even fix the
error."


> It might be interesting for you to try a newer kernel, and use scrub
> on this volume if you have the two disks RAIDed.
I will try that.

Greetings,
Hendrik



-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Dear Mich,

thanks for your help and suggestion:
 > It might be interesting for you to try a newer kernel, and use scrub
 > on this volume if you have the two disks RAIDed.

I have now scrubbed the Disk:
./btrfs scrub status /mnt/other/
scrub status for a15eede9-1a92-47d8-940a-adc7cf97352d
         scrub started at Sun Dec  9 13:48:57 2012 and finished after 
3372 seconds
         total bytes scrubbed: 1.10TB with 0 errors


That's odd, as in one folder, data is missing (I could have deleted it, 
but I'd be very surprised...)

Also, when I run btrfsck, I get errors:
On sdc1:
root 261 inode 64370 errors 400
root 261 inode 64373 errors 400
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
found 1203899371520 bytes used err is 1
total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608
btree space waste bytes: 212383383
file data blocks allocated: 28032005304320
  referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eec

On sdb1:
root 261 inode 64373 errors 400
root 261 inode 64375 errors 400
root 261 inode 64376 errors 400
found 1203899371520 bytes used err is 1
total csum bytes: 1173983136
total tree bytes: 1740640256
total fs tree bytes: 280260608
btree space waste bytes: 212383383
file data blocks allocated: 28032005304320
  referenced 1190305632256
Btrfs v0.20-rc1-37-g91d9eec



And when I try to mount one of the two raided disks, I get:
[ 1173.773861] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 1 
transid 140194 /dev/sdb1
[ 1173.774695] btrfs: failed to read the system array on sdb1
[ 1173.774854] btrfs: open_ctree failed

while the other works:
[ 1177.927096] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 2 
transid 140194 /dev/sdc1

Do you have hints for me?
The Kernel now is 3.3.7-030307-generic (anything more recent, I would 
have to compile myself, which I will do, if you suggest to)

Greetings,
Hendrik



Am 06.12.2012 20:09, schrieb Mitch Harder:
> On Wed, Dec 5, 2012 at 2:50 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
>> Dear all,
>>
>> thanks for developing btrfsck!
>> Now, I'd like to contribute -as far as I can. I'm not a developer, but I do
>> have some linux-experience.
>> I've been using btrfsck on two 3TB HDDs (mirrored) for a while now under
>> Kernel 3.0. Now it's corrupt. I had some hard resets of the machine -which
>> might have contributed. I do have a backup of the data -at least of the
>> important stuff. Some TV-Recordings are missing. The reason I am writing is,
>> to support the development.
>>
>> Unfortunately, btrfsck (latest git-version) crashes with a segmentation
>> fault, when trying to repair this.
>>
>> Here's the backtrace:
>> root 261 inode 64375 errors 400
>> root 261 inode 64376 errors 400
>> btrfsck: disk-io.c:382: __commit_transaction: Assertion `!(!eb || eb->start
>> != start)' failed.
>>
>> Program received signal SIGABRT, Aborted.
>> 0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> (gdb)
>> (gdb) backtrace
>> #0  0x00007ffff784c425 in raise () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x00007ffff784fb8b in abort () from /lib/x86_64-linux-gnu/libc.so.6
>> #2  0x00007ffff78450ee in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #3  0x00007ffff7845192 in __assert_fail () from
>> /lib/x86_64-linux-gnu/libc.so.6
>> #4  0x000000000040d3ae in __commit_transaction (trans=0x62e010,
>> root=0xb66ae0) at disk-io.c:382
>> #5  0x000000000040d4d8 in btrfs_commit_transaction (trans=0x62e010,
>> root=0xb66ae0) at disk-io.c:415
>> #6  0x000000000040743d in main (ac=<optimized out>, av=<optimized out>) at
>> btrfsck.c:3587
>>
>>
>> Now, here's where my debugging knowledge ends. Are you interested in
>> debugging this further, or is it a known bug?
>>
>
> Line 382 in disk-io.c is:
>
> BUG_ON(!eb || eb->start != start);
>
> So, basically, btrfsck is intentionally crashing because it doesn't
> know how to handle this condition.
>
> Future refinements of btrfsck will probably include proper error
> messages for issues that can't be handled, or perhaps even fix the
> error.
>
> It might be interesting for you to try a newer kernel, and use scrub
> on this volume if you have the two disks RAIDed.
>


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Mitch Harder]

On Sun, Dec 9, 2012 at 1:06 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
> Dear Mich,
>
> thanks for your help and suggestion:
>
>> It might be interesting for you to try a newer kernel, and use scrub
>> on this volume if you have the two disks RAIDed.
>
> I have now scrubbed the Disk:
> ./btrfs scrub status /mnt/other/
> scrub status for a15eede9-1a92-47d8-940a-adc7cf97352d
>         scrub started at Sun Dec  9 13:48:57 2012 and finished after 3372
> seconds
>         total bytes scrubbed: 1.10TB with 0 errors
>
>
> That's odd, as in one folder, data is missing (I could have deleted it, but
> I'd be very surprised...)
>
> Also, when I run btrfsck, I get errors:
> On sdc1:
> root 261 inode 64370 errors 400
> root 261 inode 64373 errors 400
>
> root 261 inode 64375 errors 400
> root 261 inode 64376 errors 400
> found 1203899371520 bytes used err is 1
> total csum bytes: 1173983136
> total tree bytes: 1740640256
> total fs tree bytes: 280260608
> btree space waste bytes: 212383383
> file data blocks allocated: 28032005304320
>  referenced 1190305632256
> Btrfs v0.20-rc1-37-g91d9eec
>
> On sdb1:
> root 261 inode 64373 errors 400
>
> root 261 inode 64375 errors 400
> root 261 inode 64376 errors 400
> found 1203899371520 bytes used err is 1
> total csum bytes: 1173983136
> total tree bytes: 1740640256
> total fs tree bytes: 280260608
> btree space waste bytes: 212383383
> file data blocks allocated: 28032005304320
>  referenced 1190305632256
> Btrfs v0.20-rc1-37-g91d9eec
>
>
>
> And when I try to mount one of the two raided disks, I get:
> [ 1173.773861] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 1
> transid 140194 /dev/sdb1
> [ 1173.774695] btrfs: failed to read the system array on sdb1
> [ 1173.774854] btrfs: open_ctree failed
>
> while the other works:
> [ 1177.927096] device fsid a15eede9-1a92-47d8-940a-adc7cf97352d devid 2
> transid 140194 /dev/sdc1
>
> Do you have hints for me?
> The Kernel now is 3.3.7-030307-generic (anything more recent, I would have
> to compile myself, which I will do, if you suggest to)
>

Since btrfs has significant improvements and fixes in each kernel
release, and since very few of these changes are backported, it is
recommended to use the latest kernels available.

The "root ### inode ##### errors 400" are an indication that there is
an inconsistency in the inode size.  There was a patch included in the
3.1 or 3.2 kernel to address this issue
(http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
 But I don't believe this patch fixed existing occurrences of this
error.

At this point, the quickest solution for you may be to rebuild and
reformat this RAID assembly, and restore this data from backups.

If you don't have a backup of this data, and since your array seems to
be working pretty well in a degraded state, this would be a really
good time to look at a strategy of getting a backup of this data
before doing many more attempts at rescue.

Re: segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Hello Mitch, hello all,

 > Since btrfs has significant improvements and fixes in each kernel
> release, and since very few of these changes are backported, it is
> recommended to use the latest kernels available.

Ok, it's 3.7 now.

> The "root ### inode ##### errors 400" are an indication that there is
> an inconsistency in the inode size.  There was a patch included in the
> 3.1 or 3.2 kernel to address this issue
> (http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
>   But I don't believe this patch fixed existing occurrences of this
> error.

Apparently not. It's still there.

> At this point, the quickest solution for you may be to rebuild and
> reformat this RAID assembly, and restore this data from backups.

Yepp, I did that. But in fact, some data is missing. It is not 
essential, but nice to have.

> If you don't have a backup of this data, and since your array seems to
> be working pretty well in a degraded state, this would be a really
> good time to look at a strategy of getting a backup of this data
> before doing many more attempts at rescue.

Done. It's all save on another ext4 drive.

So, let's play ;-)
Could you please help me trying to restore the missing Data?

What I tried sofar was:
./btrfs-restore /dev/sdc1 /mnt/restore/

It worked, in a way that it restored what I already had.
What's odd aswell is, that btrfs scrub did run through without errors.
So, the missing data could have been (accidentally) deleted by me. But I 
don't think... nevertheless I cannot exclude.

What I know is the (original) Path of the Data.

Greetings,
Hendrik


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Mitch Harder]

On Sat, Dec 15, 2012 at 1:40 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
> Hello Mitch, hello all,
>
>
>> Since btrfs has significant improvements and fixes in each kernel
>>
>> release, and since very few of these changes are backported, it is
>> recommended to use the latest kernels available.
>
>
> Ok, it's 3.7 now.
>
>
>> The "root ### inode ##### errors 400" are an indication that there is
>> an inconsistency in the inode size.  There was a patch included in the
>> 3.1 or 3.2 kernel to address this issue
>>
>> (http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
>>   But I don't believe this patch fixed existing occurrences of this
>> error.
>
>
> Apparently not. It's still there.
>
>
>> At this point, the quickest solution for you may be to rebuild and
>> reformat this RAID assembly, and restore this data from backups.
>
>
> Yepp, I did that. But in fact, some data is missing. It is not essential,
> but nice to have.
>
>
>> If you don't have a backup of this data, and since your array seems to
>> be working pretty well in a degraded state, this would be a really
>> good time to look at a strategy of getting a backup of this data
>> before doing many more attempts at rescue.
>
>
> Done. It's all save on another ext4 drive.
>
> So, let's play ;-)
> Could you please help me trying to restore the missing Data?
>
> What I tried sofar was:
> ./btrfs-restore /dev/sdc1 /mnt/restore/
>
> It worked, in a way that it restored what I already had.
> What's odd aswell is, that btrfs scrub did run through without errors.
> So, the missing data could have been (accidentally) deleted by me. But I
> don't think... nevertheless I cannot exclude.
>
> What I know is the (original) Path of the Data.
>

You could try btrfs-debug-tree, and search for any traces of your
file.  However, be ready to sift through a massive amount of output.

Re: segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Hi Mitch, hi all,

thanks for your hint.

I used btrfs-debug-tree now.
With -e, the output is empty. But without -e I do get a bit output file.
When I search for Filenames that I am missing, I get:

grep Sting big_output_file  |grep Berlin
                 namelen 20 datalen 0 name: Sting_Live_in_Berlin
                 namelen 20 datalen 0 name: Sting_Live_in_Berlin
                 inode ref index 29 namelen 20 name: Sting_Live_in_Berlin

That looks good.
That raises two questions now: Can I restore the file?
And: Can I do that for a whole Path (e.g. ./Video/)

Greetings&Thanks!
Hendrik


Am 15.12.2012 23:24, schrieb Mitch Harder:
> On Sat, Dec 15, 2012 at 1:40 PM, Hendrik Friedel <hendrik@friedels.name> wrote:
>> Hello Mitch, hello all,
>>
>>
>>> Since btrfs has significant improvements and fixes in each kernel
>>>
>>> release, and since very few of these changes are backported, it is
>>> recommended to use the latest kernels available.
>>
>>
>> Ok, it's 3.7 now.
>>
>>
>>> The "root ### inode ##### errors 400" are an indication that there is
>>> an inconsistency in the inode size.  There was a patch included in the
>>> 3.1 or 3.2 kernel to address this issue
>>>
>>> (http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
>>>    But I don't believe this patch fixed existing occurrences of this
>>> error.
>>
>>
>> Apparently not. It's still there.
>>
>>
>>> At this point, the quickest solution for you may be to rebuild and
>>> reformat this RAID assembly, and restore this data from backups.
>>
>>
>> Yepp, I did that. But in fact, some data is missing. It is not essential,
>> but nice to have.
>>
>>
>>> If you don't have a backup of this data, and since your array seems to
>>> be working pretty well in a degraded state, this would be a really
>>> good time to look at a strategy of getting a backup of this data
>>> before doing many more attempts at rescue.
>>
>>
>> Done. It's all save on another ext4 drive.
>>
>> So, let's play ;-)
>> Could you please help me trying to restore the missing Data?
>>
>> What I tried sofar was:
>> ./btrfs-restore /dev/sdc1 /mnt/restore/
>>
>> It worked, in a way that it restored what I already had.
>> What's odd aswell is, that btrfs scrub did run through without errors.
>> So, the missing data could have been (accidentally) deleted by me. But I
>> don't think... nevertheless I cannot exclude.
>>
>> What I know is the (original) Path of the Data.
>>
>
> You could try btrfs-debug-tree, and search for any traces of your
> file.  However, be ready to sift through a massive amount of output.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Hendrik Friedel]

Hello,

I re-send this message, hoping that someone can give me a hint?

Regards,
Hendrik

Am 18.12.2012 23:17, schrieb Hendrik Friedel:
> Hi Mitch, hi all,
>
> thanks for your hint.
>
> I used btrfs-debug-tree now.
> With -e, the output is empty. But without -e I do get a bit output file.
> When I search for Filenames that I am missing, I get:
>
> grep Sting big_output_file  |grep Berlin
>                  namelen 20 datalen 0 name: Sting_Live_in_Berlin
>                  namelen 20 datalen 0 name: Sting_Live_in_Berlin
>                  inode ref index 29 namelen 20 name: Sting_Live_in_Berlin
>
> That looks good.
> That raises two questions now: Can I restore the file?
> And: Can I do that for a whole Path (e.g. ./Video/)
>
> Greetings&Thanks!
> Hendrik
>
>
> Am 15.12.2012 23:24, schrieb Mitch Harder:
>> On Sat, Dec 15, 2012 at 1:40 PM, Hendrik Friedel
>> <hendrik@friedels.name> wrote:
>>> Hello Mitch, hello all,
>>>
>>>
>>>> Since btrfs has significant improvements and fixes in each kernel
>>>>
>>>> release, and since very few of these changes are backported, it is
>>>> recommended to use the latest kernels available.
>>>
>>>
>>> Ok, it's 3.7 now.
>>>
>>>
>>>> The "root ### inode ##### errors 400" are an indication that there is
>>>> an inconsistency in the inode size.  There was a patch included in the
>>>> 3.1 or 3.2 kernel to address this issue
>>>>
>>>> (http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=commit;h=f70a9a6b94af86fca069a7552ab672c31b457786).
>>>>
>>>>    But I don't believe this patch fixed existing occurrences of this
>>>> error.
>>>
>>>
>>> Apparently not. It's still there.
>>>
>>>
>>>> At this point, the quickest solution for you may be to rebuild and
>>>> reformat this RAID assembly, and restore this data from backups.
>>>
>>>
>>> Yepp, I did that. But in fact, some data is missing. It is not
>>> essential,
>>> but nice to have.
>>>
>>>
>>>> If you don't have a backup of this data, and since your array seems to
>>>> be working pretty well in a degraded state, this would be a really
>>>> good time to look at a strategy of getting a backup of this data
>>>> before doing many more attempts at rescue.
>>>
>>>
>>> Done. It's all save on another ext4 drive.
>>>
>>> So, let's play ;-)
>>> Could you please help me trying to restore the missing Data?
>>>
>>> What I tried sofar was:
>>> ./btrfs-restore /dev/sdc1 /mnt/restore/
>>>
>>> It worked, in a way that it restored what I already had.
>>> What's odd aswell is, that btrfs scrub did run through without errors.
>>> So, the missing data could have been (accidentally) deleted by me. But I
>>> don't think... nevertheless I cannot exclude.
>>>
>>> What I know is the (original) Path of the Data.
>>>
>>
>> You could try btrfs-debug-tree, and search for any traces of your
>> file.  However, be ready to sift through a massive amount of output.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>


-- 
Hendrik Friedel
Auf dem Brink 12
28844 Weyhe
Mobil 0178 1874363

Re: segmentation-fault in btrfsck (git-version)

[Mitch Harder]

On Sat, Dec 29, 2012 at 5:28 AM, Hendrik Friedel <hendrik@friedels.name> wrote:
> Hello,
>
> I re-send this message, hoping that someone can give me a hint?
>
> Regards,
> Hendrik
>

Two possibilities come to mind (although there may be others).

(1)  The file still exists, but it is somewhere you did not expect.

(2)  Your filesystem tree has some sort of corruption.

For item (1), have you thoroughly searched the entire volume for this
file with something like:

find <path/to/volume/top/level/mount>  -iname 'Sting_Live_in_Berlin'

It is possible that the file exists in a snapshot or different
directory then you were expecting.

If the filesystem tree is corrupted, the task becomes tricky.

Perhaps you can look at the Wiki entry for how the filesystem tree is
constructed:

https://btrfs.wiki.kernel.org/index.php/Trees

Then examine the btrfs-debug-tree output around these entries, and try
to determine why the tree still has entries for these files, but does
not show these files nor report the problem with btrfsck.