BTRFS 3.8.7 Kernel Crash Report‏

BTRFS 3.8.7 Kernel Crash Report‏

[Mark Ridley]

Hi,

I have been having a kernel crash since kernel 3.3.2 and hoped it had been fixed 
by the time 3.8.7 
(FC18) but no joy.

I take backups of large image files (around 100GB) and btrfs snapshot them.

If I then use rsync --inplace to update the images, I get a btrfs stack trace 
and btrfs hangs:

This happens every night.

Here is the stack trace:

Any idea what is causing this?

Thanks,

Mark
Apr 22 13:29:29 bkserver kernel: [66320.561671] ------------[ cut here ]----
--------
Apr 22 13:29:29 bkserver kernel: [66320.561818] kernel BUG at fs/btrfs/
ctree.c:2952!
Apr 22 13:29:29 bkserver kernel: [66320.561906] invalid opcode: 0000 
[#1] SMP
Apr 22 13:29:29 bkserver kernel: [66320.562212] Modules linked in:
 ppdev parport_pc vmxnet3
 
coretemp vmw_balloon parport shpchp i2c_piix4 microcode btrfs 
zlib_deflate libcrc32c
 vmwgfx ttm 
drm vmw_pvscsi i2c_core
Apr 22 13:29:29 bkserver kernel: [66320.562538] CPU 6Apr 22 
13:29:29 bkserver kernel: 
[66320.562605] Pid: 30965, comm: btrfs-endio-wri Not tainted 
3.8.7-201.fc18.x86_64 #1 VMware, 
Inc. VMware Virtual Platform/440BX Desktop Reference Platform
Apr 22 13:29:29 bkserver kernel: [66320.562820] RIP: 0010:[<ffffffffa00bd5c1>]  
[<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]Apr 22 13:29:29
 bkserver kernel: 
[66320.563046] RSP: 0018:ffff88014763dae8  EFLAGS: 00010286
Apr 22 13:29:29 bkserver kernel: [66320.563147] RAX: 00000000ffffffff RBX:
 000000000000001a 
RCX: 0000000ce75ee000
Apr 22 13:29:29 bkserver kernel: [66320.563258] RDX: 00000000ffffffff RSI:
 ffff88014763dc16 RDI: 
ffff88014763dac7
Apr 22 13:29:29 bkserver kernel: [66320.563372] RBP: ffff88014763db48 R08: 
0000000000004b58 
R09: 00000ce75e80006c
Apr 22 13:29:29 bkserver kernel: [66320.563475] R10: 6c0000000000004b R11: 
0000000ce75e8000 R12: ffff88014763db07
Apr 22 13:29:29 bkserver kernel: [66320.563586] R13: ffff880101d151b0 R14: 
ffff88014763dc16 
R15: ffff8800afb69d90
Apr 22 13:29:29 bkserver kernel: [66320.563729] FS:  
0000000000000000(0000) 
GS:ffff8803bfd80000(0000) knlGS:0000000000000000
Apr 22 13:29:29 bkserver kernel: [66320.563842] CS:  0010 DS: 0000 ES: 0000 CR0: 
000000008005003b
Apr 22 13:29:29 bkserver kernel: [66320.563931] CR2: 0000000000440be0 
CR3: 
00000003ab6ab000 CR4: 00000000000007e0
Apr 22 13:29:29 bkserver kernel: [66320.564051] DR0: 0000000000000000
 DR1: 
0000000000000000 DR2: 0000000000000000

Apr 22 13:29:29 bkserver kernel: [66320.564185] DR3: 0000000000000000
 DR6: 00000000ffff0ff0 
DR7: 0000000000000400
Apr 22 13:29:29 bkserver kernel: [66320.564298] Process btrfs-endio-wri 
(pid: 30965, threadinfo 
ffff88014763c000, task ffff8803ab79aec0)
Apr 22 13:29:29 bkserver kernel: [66320.564455] Stack:
Apr 22 13:29:29 bkserver kernel: [66320.564558]  ffff88014763db48 
ffff8800066a9800 
ffff880131387000 5800000000000000
Apr 22 13:29:29 bkserver kernel: [66320.564662]  6c0000000000004b 
0000000ce75e8000 
ffff880100000000 ffff8800afb69d90
Apr 22 13:29:29 bkserver kernel: [66320.564755]  ffff880101d151b0 
0000000000000a69 
0000000000000000 0000000ce75d8000
Apr 22 13:29:29 bkserver kernel: [66320.564865] Call Trace:
Apr 22 13:29:29 bkserver kernel: [66320.564948]  [<ffffffffa00f1b50>] 
__btrfs_drop_extents+0x460/0xbc0 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.565048]  [<ffffffffa00cbe46>] ? 
run_clustered_refs+0x256/0xb50 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.565143]  [<ffffffffa00f740e>] ? 
alloc_extent_state+0x2e/0xc0 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.565300]  [<ffffffffa00f2d2b>] 
btrfs_drop_extents+0x6b/0xa0 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.565406]  [<ffffffffa00e31ab>] 
insert_reserved_file_extent.constprop.58+0x7b/0x2c0 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.566955]  [<ffffffffa00e06b5>] ? 
start_transaction+0x95/0x460 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.567792]  [<ffffffffa00e9004>] 
btrfs_finish_ordered_io+0x364/0x3f0 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.568534]  [<ffffffffa00e90a5>] 
finish_ordered_fn+0x15/0x20 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.569362]  [<ffffffffa0109e56>] 
worker_loop+0x136/0x580 
[btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.570101]  [<ffffffffa0109d20>] ? 
btrfs_queue_worker+0x300/0x300 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.570858]  [<ffffffff81081f40>]
 kthread+0xc0/0xd0
Apr 22 13:29:29 bkserver kernel: [66320.571561]  [<ffffffff81088f30>] ? 
set_security_override_from_ctx+0x50/0x50
Apr 22 13:29:29 bkserver kernel: [66320.572263]  [<ffffffff81010000>] ? 
perf_trace_xen_cpu_write_gdt_entry+0xa0/0x100
Apr 22 13:29:29 bkserver kernel: [66320.572924]  [<ffffffff81081e80>] ? 
kthread_create_on_node+0x120/0x120
Apr 22 13:29:29 bkserver kernel: [66320.573588]  [<ffffffff8165baac>] 
ret_from_fork+0x7c/0xb0
Apr 22 13:29:29 bkserver kernel: [66320.574225]  [<ffffffff81081e80>] ? 
kthread_create_on_node+0x120/0x120
Apr 22 13:29:29 bkserver kernel: [66320.574883] Code: 00 00 4c 89 e6 4c 
89 ff 48 98 48 8d 04 80 
48 8d 54 80 65 e8 02 0c 04 00 4c 89 f6 4c 89 e7 e8 07 f2 ff ff 85 c0 0f 
8f 5e ff ff ff <0f> 0b 0f 0b 
66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55
Apr 22 13:29:29 bkserver kernel: [66320.576813] RIP  [<ffffffffa00bd5c1>] 
btrfs_set_item_key_safe+0x141/0x150 [btrfs]
Apr 22 13:29:29 bkserver kernel: [66320.577476]  RSP <ffff88014763dae8>
Apr 22 13:29:29 bkserver kernel: [66320.579362] ---
[ end trace 0ae203d38f3d2f34 ]---

Re: BTRFS 3.8.7 Kernel Crash Report‏

[David Sterba]

On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
> If I then use rsync --inplace to update the images, I get a btrfs stack trace 
> and btrfs hangs:
> 
> This happens every night.

> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]

The check fails because it finds keys in reverted order. Given the
conditions under which it happens I think it's an on-disk corruption and
fsck should be able to at least detect it.

david

Re: BTRFS 3.8.7 Kernel Crash Report‏

[Mark Ridley]

Thanks, David.

What causes this corruption and how can I fix it?

I'm very worried about running btrfs.fsck as last time it made a slight
corruption like this worse and the whole volume had to be trashed.

After fsck the "available space" on df ended up being negative so nothing
could be written to the volume.

Mark

On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:

>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>> If I then use rsync --inplace to update the images, I get a btrfs stack
>>trace 
>> and btrfs hangs:
>> 
>> This happens every night.
>
>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>
>The check fails because it finds keys in reverted order. Given the
>conditions under which it happens I think it's an on-disk corruption and
>fsck should be able to at least detect it.
>
>david

Re: BTRFS 3.8.7 Kernel Crash Report

[Harald Glatt]

On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley <mark@backupsystems.co.uk> wrote:
> Thanks, David.
>
> What causes this corruption and how can I fix it?
>
> I'm very worried about running btrfs.fsck as last time it made a slight
> corruption like this worse and the whole volume had to be trashed.
>
> After fsck the "available space" on df ended up being negative so nothing
> could be written to the volume.
>
> Mark
>
> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>
>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>> If I then use rsync --inplace to update the images, I get a btrfs stack
>>>trace
>>> and btrfs hangs:
>>>
>>> This happens every night.
>>
>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>
>>The check fails because it finds keys in reverted order. Given the
>>conditions under which it happens I think it's an on-disk corruption and
>>fsck should be able to at least detect it.
>>
>>david
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

This happened without --repair?

Re: BTRFS 3.8.7 Kernel Crash Report

[Mark Ridley]

No.

Without --repair pointed out some problems, but whats the point of knowing
about the problems if they can't be fixed so I ran with --repair and it
broke the volume.

On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:

>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley <mark@backupsystems.co.uk>
>wrote:
>> Thanks, David.
>>
>> What causes this corruption and how can I fix it?
>>
>> I'm very worried about running btrfs.fsck as last time it made a slight
>> corruption like this worse and the whole volume had to be trashed.
>>
>> After fsck the "available space" on df ended up being negative so
>>nothing
>> could be written to the volume.
>>
>> Mark
>>
>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>
>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>stack
>>>>trace
>>>> and btrfs hangs:
>>>>
>>>> This happens every night.
>>>
>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>
>>>The check fails because it finds keys in reverted order. Given the
>>>conditions under which it happens I think it's an on-disk corruption and
>>>fsck should be able to at least detect it.
>>>
>>>david
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>This happened without --repair?

Re: BTRFS 3.8.7 Kernel Crash Report

[Harald Glatt]

Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
output just from checking would be helpful in this case.

On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley <mark@backupsystems.co.uk> wrote:
> No.
>
> Without --repair pointed out some problems, but whats the point of knowing
> about the problems if they can't be fixed so I ran with --repair and it
> broke the volume.
>
> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>
>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley <mark@backupsystems.co.uk>
>>wrote:
>>> Thanks, David.
>>>
>>> What causes this corruption and how can I fix it?
>>>
>>> I'm very worried about running btrfs.fsck as last time it made a slight
>>> corruption like this worse and the whole volume had to be trashed.
>>>
>>> After fsck the "available space" on df ended up being negative so
>>>nothing
>>> could be written to the volume.
>>>
>>> Mark
>>>
>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>
>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>>stack
>>>>>trace
>>>>> and btrfs hangs:
>>>>>
>>>>> This happens every night.
>>>>
>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>
>>>>The check fails because it finds keys in reverted order. Given the
>>>>conditions under which it happens I think it's an on-disk corruption and
>>>>fsck should be able to at least detect it.
>>>>
>>>>david
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>This happened without --repair?
>

Re: BTRFS 3.8.7 Kernel Crash Report

[Mark Ridley]

What does btrfs scrub do?

Is that meant to detect and fix problems?

Thanks,

Mark

On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:

>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>output just from checking would be helpful in this case.
>
>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley <mark@backupsystems.co.uk>
>wrote:
>> No.
>>
>> Without --repair pointed out some problems, but whats the point of
>>knowing
>> about the problems if they can't be fixed so I ran with --repair and it
>> broke the volume.
>>
>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>
>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>wrote:
>>>> Thanks, David.
>>>>
>>>> What causes this corruption and how can I fix it?
>>>>
>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>slight
>>>> corruption like this worse and the whole volume had to be trashed.
>>>>
>>>> After fsck the "available space" on df ended up being negative so
>>>>nothing
>>>> could be written to the volume.
>>>>
>>>> Mark
>>>>
>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>
>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>>>stack
>>>>>>trace
>>>>>> and btrfs hangs:
>>>>>>
>>>>>> This happens every night.
>>>>>
>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>
>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>conditions under which it happens I think it's an on-disk corruption
>>>>>and
>>>>>fsck should be able to at least detect it.
>>>>>
>>>>>david
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>This happened without --repair?
>>

Re: BTRFS 3.8.7 Kernel Crash Report

[Harald Glatt]

Only data errors (from CRC checks), maybe also some structure errors -
I'm not sure. A remount should fix all errors. If it doesn't I think
it's considered a bug - ultimately the goal is that no --repair should
ever be required... Scrub is generally safe though, unlike btrfsck
--repair.

On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley <mark@backupsystems.co.uk> wrote:
> What does btrfs scrub do?
>
> Is that meant to detect and fix problems?
>
> Thanks,
>
> Mark
>
> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>
>>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>>output just from checking would be helpful in this case.
>>
>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley <mark@backupsystems.co.uk>
>>wrote:
>>> No.
>>>
>>> Without --repair pointed out some problems, but whats the point of
>>>knowing
>>> about the problems if they can't be fixed so I ran with --repair and it
>>> broke the volume.
>>>
>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>
>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>>wrote:
>>>>> Thanks, David.
>>>>>
>>>>> What causes this corruption and how can I fix it?
>>>>>
>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>slight
>>>>> corruption like this worse and the whole volume had to be trashed.
>>>>>
>>>>> After fsck the "available space" on df ended up being negative so
>>>>>nothing
>>>>> could be written to the volume.
>>>>>
>>>>> Mark
>>>>>
>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>
>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>>>>stack
>>>>>>>trace
>>>>>>> and btrfs hangs:
>>>>>>>
>>>>>>> This happens every night.
>>>>>>
>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>>
>>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>>conditions under which it happens I think it's an on-disk corruption
>>>>>>and
>>>>>>fsck should be able to at least detect it.
>>>>>>
>>>>>>david
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs"
>>>>>in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>>This happened without --repair?
>>>
>

Re: BTRFS 3.8.7 Kernel Crash Report

[Mark Ridley]

Thanks for getting back to me.

Do you think remount fixes the keys reverted problem?

On 22/04/2013 15:18, "Harald Glatt" <mail@hachre.de> wrote:

>Only data errors (from CRC checks), maybe also some structure errors -
>I'm not sure. A remount should fix all errors. If it doesn't I think
>it's considered a bug - ultimately the goal is that no --repair should
>ever be required... Scrub is generally safe though, unlike btrfsck
>--repair.
>
>On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley <mark@backupsystems.co.uk>
>wrote:
>> What does btrfs scrub do?
>>
>> Is that meant to detect and fix problems?
>>
>> Thanks,
>>
>> Mark
>>
>> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>>
>>>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>>>output just from checking would be helpful in this case.
>>>
>>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>wrote:
>>>> No.
>>>>
>>>> Without --repair pointed out some problems, but whats the point of
>>>>knowing
>>>> about the problems if they can't be fixed so I ran with --repair and
>>>>it
>>>> broke the volume.
>>>>
>>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>>
>>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley
>>>>><mark@backupsystems.co.uk>
>>>>>wrote:
>>>>>> Thanks, David.
>>>>>>
>>>>>> What causes this corruption and how can I fix it?
>>>>>>
>>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>>slight
>>>>>> corruption like this worse and the whole volume had to be trashed.
>>>>>>
>>>>>> After fsck the "available space" on df ended up being negative so
>>>>>>nothing
>>>>>> could be written to the volume.
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>>
>>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>>>>>stack
>>>>>>>>trace
>>>>>>>> and btrfs hangs:
>>>>>>>>
>>>>>>>> This happens every night.
>>>>>>>
>>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>>>
>>>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>>>conditions under which it happens I think it's an on-disk corruption
>>>>>>>and
>>>>>>>fsck should be able to at least detect it.
>>>>>>>
>>>>>>>david
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>linux-btrfs"
>>>>>>in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>>This happened without --repair?
>>>>
>>

Re: BTRFS 3.8.7 Kernel Crash Report

[Harald Glatt]

I think it won't... That would just be the goal eventually. If scrub
sees no errors all the data should be in tact and your best bet to get
things working perfectly again is to create a new filesystem and
transfer the data into it.

I'm not a dev and I can't say if a btrfs-image for debugging purposes
would be helpful in this case or not, maybe this kind of corrution has
been fixed a long time ago and wouldn't happen again in up-to-date
versions of btrfs anyway.

On Mon, Apr 22, 2013 at 4:24 PM, Mark Ridley <mark@backupsystems.co.uk> wrote:
> Thanks for getting back to me.
>
> Do you think remount fixes the keys reverted problem?
>
> On 22/04/2013 15:18, "Harald Glatt" <mail@hachre.de> wrote:
>
>>Only data errors (from CRC checks), maybe also some structure errors -
>>I'm not sure. A remount should fix all errors. If it doesn't I think
>>it's considered a bug - ultimately the goal is that no --repair should
>>ever be required... Scrub is generally safe though, unlike btrfsck
>>--repair.
>>
>>On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley <mark@backupsystems.co.uk>
>>wrote:
>>> What does btrfs scrub do?
>>>
>>> Is that meant to detect and fix problems?
>>>
>>> Thanks,
>>>
>>> Mark
>>>
>>> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>>>
>>>>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>>>>output just from checking would be helpful in this case.
>>>>
>>>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>>wrote:
>>>>> No.
>>>>>
>>>>> Without --repair pointed out some problems, but whats the point of
>>>>>knowing
>>>>> about the problems if they can't be fixed so I ran with --repair and
>>>>>it
>>>>> broke the volume.
>>>>>
>>>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>
>>>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley
>>>>>><mark@backupsystems.co.uk>
>>>>>>wrote:
>>>>>>> Thanks, David.
>>>>>>>
>>>>>>> What causes this corruption and how can I fix it?
>>>>>>>
>>>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>>>slight
>>>>>>> corruption like this worse and the whole volume had to be trashed.
>>>>>>>
>>>>>>> After fsck the "available space" on df ended up being negative so
>>>>>>>nothing
>>>>>>> could be written to the volume.
>>>>>>>
>>>>>>> Mark
>>>>>>>
>>>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>>>
>>>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>>>> If I then use rsync --inplace to update the images, I get a btrfs
>>>>>>>>>stack
>>>>>>>>>trace
>>>>>>>>> and btrfs hangs:
>>>>>>>>>
>>>>>>>>> This happens every night.
>>>>>>>>
>>>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>>>>
>>>>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>>>>conditions under which it happens I think it's an on-disk corruption
>>>>>>>>and
>>>>>>>>fsck should be able to at least detect it.
>>>>>>>>
>>>>>>>>david
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>linux-btrfs"
>>>>>>>in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>>This happened without --repair?
>>>>>
>>>
>

Re: BTRFS 3.8.7 Kernel Crash Report

[Mark Ridley]

Any idea when the 3.9 kernel will be out.

I see there are lots of btrfs fixes going into it.

I'm currently running FC18 on 3.8.7, although 10 minutes ago it got
updated to 3.8.8.

On 22/04/2013 15:28, "Harald Glatt" <mail@hachre.de> wrote:

>I think it won't... That would just be the goal eventually. If scrub
>sees no errors all the data should be in tact and your best bet to get
>things working perfectly again is to create a new filesystem and
>transfer the data into it.
>
>I'm not a dev and I can't say if a btrfs-image for debugging purposes
>would be helpful in this case or not, maybe this kind of corrution has
>been fixed a long time ago and wouldn't happen again in up-to-date
>versions of btrfs anyway.
>
>On Mon, Apr 22, 2013 at 4:24 PM, Mark Ridley <mark@backupsystems.co.uk>
>wrote:
>> Thanks for getting back to me.
>>
>> Do you think remount fixes the keys reverted problem?
>>
>> On 22/04/2013 15:18, "Harald Glatt" <mail@hachre.de> wrote:
>>
>>>Only data errors (from CRC checks), maybe also some structure errors -
>>>I'm not sure. A remount should fix all errors. If it doesn't I think
>>>it's considered a bug - ultimately the goal is that no --repair should
>>>ever be required... Scrub is generally safe though, unlike btrfsck
>>>--repair.
>>>
>>>On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>wrote:
>>>> What does btrfs scrub do?
>>>>
>>>> Is that meant to detect and fix problems?
>>>>
>>>> Thanks,
>>>>
>>>> Mark
>>>>
>>>> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>>>>
>>>>>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>>>>>output just from checking would be helpful in this case.
>>>>>
>>>>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley
>>>>><mark@backupsystems.co.uk>
>>>>>wrote:
>>>>>> No.
>>>>>>
>>>>>> Without --repair pointed out some problems, but whats the point of
>>>>>>knowing
>>>>>> about the problems if they can't be fixed so I ran with --repair and
>>>>>>it
>>>>>> broke the volume.
>>>>>>
>>>>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>>
>>>>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley
>>>>>>><mark@backupsystems.co.uk>
>>>>>>>wrote:
>>>>>>>> Thanks, David.
>>>>>>>>
>>>>>>>> What causes this corruption and how can I fix it?
>>>>>>>>
>>>>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>>>>slight
>>>>>>>> corruption like this worse and the whole volume had to be trashed.
>>>>>>>>
>>>>>>>> After fsck the "available space" on df ended up being negative so
>>>>>>>>nothing
>>>>>>>> could be written to the volume.
>>>>>>>>
>>>>>>>> Mark
>>>>>>>>
>>>>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>>>>
>>>>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>>>>> If I then use rsync --inplace to update the images, I get a
>>>>>>>>>>btrfs
>>>>>>>>>>stack
>>>>>>>>>>trace
>>>>>>>>>> and btrfs hangs:
>>>>>>>>>>
>>>>>>>>>> This happens every night.
>>>>>>>>>
>>>>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>>>>>
>>>>>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>>>>>conditions under which it happens I think it's an on-disk
>>>>>>>>>corruption
>>>>>>>>>and
>>>>>>>>>fsck should be able to at least detect it.
>>>>>>>>>
>>>>>>>>>david
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>linux-btrfs"
>>>>>>>>in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>>This happened without --repair?
>>>>>>
>>>>
>>

Re: BTRFS 3.8.7 Kernel Crash Report

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 607 bytes --]

On Mon, Apr 22, 2013 at 03:40:00PM +0100, Mark Ridley wrote:
> Any idea when the 3.9 kernel will be out.

   Most likely next Sunday/Monday, unless something drastic shows up
this week (or there are too many fixes going in).

> I see there are lots of btrfs fixes going into it.
> 
> I'm currently running FC18 on 3.8.7, although 10 minutes ago it got
> updated to 3.8.8.

   Hugo.

-- 
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
  PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
       --- We believe in free will because we have no choice. ---        

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: BTRFS 3.8.7 Kernel Crash Report

[Harald Glatt]

I heard it's coming out next week.

On Mon, Apr 22, 2013 at 4:40 PM, Mark Ridley <mark@backupsystems.co.uk> wrote:
> Any idea when the 3.9 kernel will be out.
>
> I see there are lots of btrfs fixes going into it.
>
> I'm currently running FC18 on 3.8.7, although 10 minutes ago it got
> updated to 3.8.8.
>
> On 22/04/2013 15:28, "Harald Glatt" <mail@hachre.de> wrote:
>
>>I think it won't... That would just be the goal eventually. If scrub
>>sees no errors all the data should be in tact and your best bet to get
>>things working perfectly again is to create a new filesystem and
>>transfer the data into it.
>>
>>I'm not a dev and I can't say if a btrfs-image for debugging purposes
>>would be helpful in this case or not, maybe this kind of corrution has
>>been fixed a long time ago and wouldn't happen again in up-to-date
>>versions of btrfs anyway.
>>
>>On Mon, Apr 22, 2013 at 4:24 PM, Mark Ridley <mark@backupsystems.co.uk>
>>wrote:
>>> Thanks for getting back to me.
>>>
>>> Do you think remount fixes the keys reverted problem?
>>>
>>> On 22/04/2013 15:18, "Harald Glatt" <mail@hachre.de> wrote:
>>>
>>>>Only data errors (from CRC checks), maybe also some structure errors -
>>>>I'm not sure. A remount should fix all errors. If it doesn't I think
>>>>it's considered a bug - ultimately the goal is that no --repair should
>>>>ever be required... Scrub is generally safe though, unlike btrfsck
>>>>--repair.
>>>>
>>>>On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>>wrote:
>>>>> What does btrfs scrub do?
>>>>>
>>>>> Is that meant to detect and fix problems?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mark
>>>>>
>>>>> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>
>>>>>>Yeah, --repair is not recommended as of now. Maybe posting the btrfsck
>>>>>>output just from checking would be helpful in this case.
>>>>>>
>>>>>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley
>>>>>><mark@backupsystems.co.uk>
>>>>>>wrote:
>>>>>>> No.
>>>>>>>
>>>>>>> Without --repair pointed out some problems, but whats the point of
>>>>>>>knowing
>>>>>>> about the problems if they can't be fixed so I ran with --repair and
>>>>>>>it
>>>>>>> broke the volume.
>>>>>>>
>>>>>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>>>
>>>>>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley
>>>>>>>><mark@backupsystems.co.uk>
>>>>>>>>wrote:
>>>>>>>>> Thanks, David.
>>>>>>>>>
>>>>>>>>> What causes this corruption and how can I fix it?
>>>>>>>>>
>>>>>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>>>>>slight
>>>>>>>>> corruption like this worse and the whole volume had to be trashed.
>>>>>>>>>
>>>>>>>>> After fsck the "available space" on df ended up being negative so
>>>>>>>>>nothing
>>>>>>>>> could be written to the volume.
>>>>>>>>>
>>>>>>>>> Mark
>>>>>>>>>
>>>>>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>>>>>
>>>>>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>>>>>> If I then use rsync --inplace to update the images, I get a
>>>>>>>>>>>btrfs
>>>>>>>>>>>stack
>>>>>>>>>>>trace
>>>>>>>>>>> and btrfs hangs:
>>>>>>>>>>>
>>>>>>>>>>> This happens every night.
>>>>>>>>>>
>>>>>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150 [btrfs]
>>>>>>>>>>
>>>>>>>>>>The check fails because it finds keys in reverted order. Given the
>>>>>>>>>>conditions under which it happens I think it's an on-disk
>>>>>>>>>>corruption
>>>>>>>>>>and
>>>>>>>>>>fsck should be able to at least detect it.
>>>>>>>>>>
>>>>>>>>>>david
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>linux-btrfs"
>>>>>>>>>in
>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>>This happened without --repair?
>>>>>>>
>>>>>
>>>
>

Re: BTRFS 3.8.7 Kernel Crash Report

[Mark Ridley]

For me anyway, I hope the fedora team don't take too long to make it
available.

Thanks for all your help today.

On 22/04/2013 15:41, "Harald Glatt" <mail@hachre.de> wrote:

>I heard it's coming out next week.
>
>On Mon, Apr 22, 2013 at 4:40 PM, Mark Ridley <mark@backupsystems.co.uk>
>wrote:
>> Any idea when the 3.9 kernel will be out.
>>
>> I see there are lots of btrfs fixes going into it.
>>
>> I'm currently running FC18 on 3.8.7, although 10 minutes ago it got
>> updated to 3.8.8.
>>
>> On 22/04/2013 15:28, "Harald Glatt" <mail@hachre.de> wrote:
>>
>>>I think it won't... That would just be the goal eventually. If scrub
>>>sees no errors all the data should be in tact and your best bet to get
>>>things working perfectly again is to create a new filesystem and
>>>transfer the data into it.
>>>
>>>I'm not a dev and I can't say if a btrfs-image for debugging purposes
>>>would be helpful in this case or not, maybe this kind of corrution has
>>>been fixed a long time ago and wouldn't happen again in up-to-date
>>>versions of btrfs anyway.
>>>
>>>On Mon, Apr 22, 2013 at 4:24 PM, Mark Ridley <mark@backupsystems.co.uk>
>>>wrote:
>>>> Thanks for getting back to me.
>>>>
>>>> Do you think remount fixes the keys reverted problem?
>>>>
>>>> On 22/04/2013 15:18, "Harald Glatt" <mail@hachre.de> wrote:
>>>>
>>>>>Only data errors (from CRC checks), maybe also some structure errors -
>>>>>I'm not sure. A remount should fix all errors. If it doesn't I think
>>>>>it's considered a bug - ultimately the goal is that no --repair should
>>>>>ever be required... Scrub is generally safe though, unlike btrfsck
>>>>>--repair.
>>>>>
>>>>>On Mon, Apr 22, 2013 at 4:16 PM, Mark Ridley
>>>>><mark@backupsystems.co.uk>
>>>>>wrote:
>>>>>> What does btrfs scrub do?
>>>>>>
>>>>>> Is that meant to detect and fix problems?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mark
>>>>>>
>>>>>> On 22/04/2013 15:13, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>>
>>>>>>>Yeah, --repair is not recommended as of now. Maybe posting the
>>>>>>>btrfsck
>>>>>>>output just from checking would be helpful in this case.
>>>>>>>
>>>>>>>On Mon, Apr 22, 2013 at 4:04 PM, Mark Ridley
>>>>>>><mark@backupsystems.co.uk>
>>>>>>>wrote:
>>>>>>>> No.
>>>>>>>>
>>>>>>>> Without --repair pointed out some problems, but whats the point of
>>>>>>>>knowing
>>>>>>>> about the problems if they can't be fixed so I ran with --repair
>>>>>>>>and
>>>>>>>>it
>>>>>>>> broke the volume.
>>>>>>>>
>>>>>>>> On 22/04/2013 15:02, "Harald Glatt" <mail@hachre.de> wrote:
>>>>>>>>
>>>>>>>>>On Mon, Apr 22, 2013 at 4:01 PM, Mark Ridley
>>>>>>>>><mark@backupsystems.co.uk>
>>>>>>>>>wrote:
>>>>>>>>>> Thanks, David.
>>>>>>>>>>
>>>>>>>>>> What causes this corruption and how can I fix it?
>>>>>>>>>>
>>>>>>>>>> I'm very worried about running btrfs.fsck as last time it made a
>>>>>>>>>>slight
>>>>>>>>>> corruption like this worse and the whole volume had to be
>>>>>>>>>>trashed.
>>>>>>>>>>
>>>>>>>>>> After fsck the "available space" on df ended up being negative
>>>>>>>>>>so
>>>>>>>>>>nothing
>>>>>>>>>> could be written to the volume.
>>>>>>>>>>
>>>>>>>>>> Mark
>>>>>>>>>>
>>>>>>>>>> On 22/04/2013 14:42, "David Sterba" <dsterba@suse.cz> wrote:
>>>>>>>>>>
>>>>>>>>>>>On Mon, Apr 22, 2013 at 01:19:41PM +0000, Mark Ridley wrote:
>>>>>>>>>>>> If I then use rsync --inplace to update the images, I get a
>>>>>>>>>>>>btrfs
>>>>>>>>>>>>stack
>>>>>>>>>>>>trace
>>>>>>>>>>>> and btrfs hangs:
>>>>>>>>>>>>
>>>>>>>>>>>> This happens every night.
>>>>>>>>>>>
>>>>>>>>>>>> [<ffffffffa00bd5c1>] btrfs_set_item_key_safe+0x141/0x150
>>>>>>>>>>>>[btrfs]
>>>>>>>>>>>
>>>>>>>>>>>The check fails because it finds keys in reverted order. Given
>>>>>>>>>>>the
>>>>>>>>>>>conditions under which it happens I think it's an on-disk
>>>>>>>>>>>corruption
>>>>>>>>>>>and
>>>>>>>>>>>fsck should be able to at least detect it.
>>>>>>>>>>>
>>>>>>>>>>>david
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>>>>linux-btrfs"
>>>>>>>>>>in
>>>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>>>> More majordomo info at
>>>>>>>>>>http://vger.kernel.org/majordomo-info.html
>>>>>>>>>
>>>>>>>>>This happened without --repair?
>>>>>>>>
>>>>>>
>>>>
>>