Fwd: Cloning a Btrfs partition

Fwd: Cloning a Btrfs partition

[BJ Quinn]

(Apologies for the double post -- forgot to send as plain text the first time around, so the list rejected it.)

I see that there's now a btrfs send / receive and I've tried using it, but I'm getting the oops I've pasted below, after which the FS becomes unresponsive (no I/O to the drive, no CPU usage, but all attempts to access the FS results in a hang). I have an internal drive (single drive) that contains 82GB of compressed data with a couple hundred snapshots. I tried taking the first snapshot and making a read only copy (btrfs subvolume snapshot -r) and then I connected an external USB drive and ran btrfs send / receive to that external drive. It starts working and gets a couple of GB in (I'd expect the first snapshot to be about 20GB) and then gets the following error. I had to use the latest copy of btrfs-progs from git, because the package installed on my system (btrfs-progs-0.20-0.2.git91d9eec) simply returned "invalid argument" when trying to run btrfs send / receive. Thanks in advance for any info you may have. 

Jul 24 18:46:48 foxserver8 kernel: general protection fault: 0000 [#1] SMP 
Jul 24 18:46:48 foxserver8 kernel: Modules linked in: des_generic ecb md4 sha256_ssse3 sha256_generic nls_utf8 cifs fscache dns_resolver fuse drbd lru_cache autofs4 sunrpc bonding ipv6 btrfs raid6_pq xor libcrc32c uinput iTCO_wdt iTCO_vendor_support gpio_ich dcdbas coretemp freq_table mperf intel_powerclamp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel microcode pcspkr joydev sg sb_edac edac_core lpc_ich shpchp acpi_power_meter tg3 hwmon ptp pps_core ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci libahci wmi usb_storage mgag200 ttm drm_kms_helper dm_mirror dm_region_hash dm_log dm_mod 
Jul 24 18:46:48 foxserver8 kernel: CPU: 7 PID: 10170 Comm: btrfs Not tainted 3.10.2-1.el6.elrepo.x86_64 #1 
Jul 24 18:46:48 foxserver8 kernel: Hardware name: Dell Inc. PowerEdge R420/0CN7CM, BIOS 1.4.6 10/26/2012 
Jul 24 18:46:48 foxserver8 kernel: task: ffff880c0f1c9540 ti: ffff880beebae000 task.ti: ffff880beebae000 
Jul 24 18:46:48 foxserver8 kernel: RIP: 0010:[<ffffffffa03be4cd>] [<ffffffffa03be4cd>] ulist_add_merge+0x2d/0x190 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: RSP: 0018:ffff880beebaf958 EFLAGS: 00010202 
Jul 24 18:46:48 foxserver8 kernel: RAX: 4c415a4e5a4c4c45 RBX: 00000007c461c000 RCX: ffff880beebafa48 
Jul 24 18:46:48 foxserver8 kernel: RDX: 0000000000000000 RSI: ffff8802b232ed70 RDI: ffff880b0ba63400 
Jul 24 18:46:48 foxserver8 kernel: RBP: ffff880beebaf988 R08: 0000000000000050 R09: 0000000000000000 
Jul 24 18:46:48 foxserver8 kernel: R10: 0000000000000000 R11: dead000000200200 R12: ffff880b0ba63400 
Jul 24 18:46:48 foxserver8 kernel: R13: 0000000000000000 R14: ffff880beebafa28 R15: ffff880c0f6d8000 
Jul 24 18:46:48 foxserver8 kernel: FS: 00007fd121589740(0000) GS:ffff880c2fc60000(0000) knlGS:0000000000000000 
Jul 24 18:46:48 foxserver8 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 
Jul 24 18:46:48 foxserver8 kernel: CR2: ffffffffff600400 CR3: 000000060ca2c000 CR4: 00000000000407e0 
Jul 24 18:46:48 foxserver8 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 
Jul 24 18:46:48 foxserver8 kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 
Jul 24 18:46:48 foxserver8 kernel: Stack: 
Jul 24 18:46:48 foxserver8 kernel: 0000000000000000 0000000000000000 ffff880bd449d8c0 ffff880bcc39fbe0 
Jul 24 18:46:48 foxserver8 kernel: ffff880beebafa28 ffff880c0f6d8000 ffff880beebafa88 ffffffffa03bdae6 
Jul 24 18:46:48 foxserver8 kernel: 0000000000008046 ffff880beebafa38 ffff880beebafab8 ffff880c00b95800 
Jul 24 18:46:48 foxserver8 kernel: Call Trace: 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03bdae6>] find_parent_nodes+0x4f6/0x630 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03bdd99>] btrfs_find_all_roots+0x99/0x100 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c4810>] ? did_overwrite_ref+0x100/0x100 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03bdf0a>] iterate_extent_inodes+0x10a/0x1f0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa037dd01>] ? free_extent_buffer+0x61/0xc0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c431b>] find_extent_clone+0x26b/0x330 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c6b41>] process_extent+0x71/0xd0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c8360>] changed_cb+0xd0/0x130 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c851e>] full_send_tree+0x15e/0x2c0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c87b8>] send_subvol+0x138/0x150 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c8600>] ? full_send_tree+0x240/0x2c0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c87b8>] send_subvol+0x138/0x150 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c8600>] ? full_send_tree+0x240/0x2c0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa03c8b22>] btrfs_ioctl_send+0x352/0x560 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffffa0396e3b>] btrfs_ioctl+0x65b/0x8c0 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff8108bbb5>] ? check_preempt_curr+0x75/0xa0 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff810936bb>] ? wake_up_new_task+0xfb/0x160 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff811a95b9>] do_vfs_ioctl+0x89/0x350 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff811a9921>] SyS_ioctl+0xa1/0xb0 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff8105ab86>] ? SyS_clone+0x16/0x20 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff815fce79>] ? stub_clone+0x69/0x90 
Jul 24 18:46:48 foxserver8 kernel: [<ffffffff815fcb19>] system_call_fastpath+0x16/0x1b 
Jul 24 18:46:48 foxserver8 kernel: Code: 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 48 8b 47 18 49 89 fc 48 89 f3 49 89 d5 0f 1f 44 00 00 48 85 c0 74 13 <48> 3b 58 f0 48 8d 70 f0 76 79 48 8b 40 08 48 85 c0 75 ed 49 8b 
Jul 24 18:46:48 foxserver8 kernel: RIP [<ffffffffa03be4cd>] ulist_add_merge+0x2d/0x190 [btrfs] 
Jul 24 18:46:48 foxserver8 kernel: RSP <ffff880beebaf958> 
Jul 24 18:46:48 foxserver8 kernel: ---[ end trace a30ba65210ac4804 ]--- 

-BJ 

----- Forwarded Message -----

From: "Jan Schmidt" <list.btrfs@jan-o-sch.net> 
To: "BJ Quinn" <bj@placs.net> 
Cc: "Phillip Susi" <psusi@cfl.rr.com>, "Freddie Cash" <fjwcash@gmail.com>, linux-btrfs@vger.kernel.org 
Sent: Thursday, December 8, 2011 10:41:38 AM 
Subject: Re: Cloning a Btrfs partition 

On 08.12.2011 17:28, BJ Quinn wrote: 
>>> At any rate, was someone saying that some work had already started on something like btrfs send? 
> 
>> That's right. 
> 
> Google tells me that someone is you. :) 
> 
> What Google wouldn't tell me though was whether you have something I could test? 

Well, it's telling you the right thing :-) 

Currently I'm distracted by reliable backref walking, which turned out 
to be a prerequisite of btrfs send. Once I have that thing done, direct 
work on the send/receive functionality will continue. 

As soon as there's something that can be tested, you'll find it on this 
list. 

-Jan 

Re: Fwd: Cloning a Btrfs partition

[Jan Schmidt]

Hi BJ,

[original message rewrapped]

On Thu, July 25, 2013 at 18:32 (+0200), BJ Quinn wrote:
> (Apologies for the double post -- forgot to send as plain text the first time
> around, so the list rejected it.)
> 
> I see that there's now a btrfs send / receive and I've tried using it, but
> I'm getting the oops I've pasted below, after which the FS becomes
> unresponsive (no I/O to the drive, no CPU usage, but all attempts to access
> the FS results in a hang). I have an internal drive (single drive) that
> contains 82GB of compressed data with a couple hundred snapshots. I tried
> taking the first snapshot and making a read only copy (btrfs subvolume
> snapshot -r) and then I connected an external USB drive and ran btrfs send /
> receive to that external drive. It starts working and gets a couple of GB in
> (I'd expect the first snapshot to be about 20GB) and then gets the following
> error. I had to use the latest copy of btrfs-progs from git, because the
> package installed on my system (btrfs-progs-0.20-0.2.git91d9eec) simply
> returned "invalid argument" when trying to run btrfs send / receive. Thanks
> in advance for any info you may have.

The problem has been introduced with rbtree ulists in 3.10, commit

    Btrfs: add a rb_tree to improve performance of ulist search

You should be safe to revert that commit, it's a performance optimization
attempt. Alternatively, you can apply the published fix

    Btrfs: fix crash regarding to ulist_add_merge

It has not made it into 3.10 stable or 3.11, yet, but is contained in Josef's
btrfs-next

    git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git

Thanks,
-Jan

Re: Cloning a Btrfs partition

[BJ Quinn]

Thanks for the response!  Not sure I want to roll a custom kernel on this
particular system.  Any idea on when it might make it to 3.10 stable or 
3.11?  Or should I just revert back to 3.9?

Thanks!

-BJ

----- Original Message ----- 

From: "Jan Schmidt" <list.btrfs@jan-o-sch.net> 
Sent: Monday, July 29, 2013 3:21:51 AM 

Hi BJ, 

[original message rewrapped] 

On Thu, July 25, 2013 at 18:32 (+0200), BJ Quinn wrote: 
> (Apologies for the double post -- forgot to send as plain text the first time 
> around, so the list rejected it.) 
> 
> I see that there's now a btrfs send / receive and I've tried using it, but 
> I'm getting the oops I've pasted below, after which the FS becomes 
> unresponsive (no I/O to the drive, no CPU usage, but all attempts to access 
> the FS results in a hang). I have an internal drive (single drive) that 
> contains 82GB of compressed data with a couple hundred snapshots. I tried 
> taking the first snapshot and making a read only copy (btrfs subvolume 
> snapshot -r) and then I connected an external USB drive and ran btrfs send / 
> receive to that external drive. It starts working and gets a couple of GB in 
> (I'd expect the first snapshot to be about 20GB) and then gets the following 
> error. I had to use the latest copy of btrfs-progs from git, because the 
> package installed on my system (btrfs-progs-0.20-0.2.git91d9eec) simply 
> returned "invalid argument" when trying to run btrfs send / receive. Thanks 
> in advance for any info you may have. 

The problem has been introduced with rbtree ulists in 3.10, commit 

Btrfs: add a rb_tree to improve performance of ulist search 

You should be safe to revert that commit, it's a performance optimization 
attempt. Alternatively, you can apply the published fix 

Btrfs: fix crash regarding to ulist_add_merge 

It has not made it into 3.10 stable or 3.11, yet, but is contained in Josef's 
btrfs-next 

git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 

Thanks, 
-Jan 

Re: Cloning a Btrfs partition

[Jan Schmidt]

On Mon, July 29, 2013 at 17:32 (+0200), BJ Quinn wrote:
> Thanks for the response!  Not sure I want to roll a custom kernel on this
> particular system.  Any idea on when it might make it to 3.10 stable or 
> 3.11?  Or should I just revert back to 3.9?

I missed that it's in fact in 3.11 and if I got Liu Bo right he's going to send
it to 3.10 stable soon.

Thanks,
-Jan

> Thanks!
> 
> -BJ
> 
> ----- Original Message ----- 
> 
> From: "Jan Schmidt" <list.btrfs@jan-o-sch.net> 
> Sent: Monday, July 29, 2013 3:21:51 AM 
> 
> Hi BJ, 
> 
> [original message rewrapped] 
> 
> On Thu, July 25, 2013 at 18:32 (+0200), BJ Quinn wrote: 
>> (Apologies for the double post -- forgot to send as plain text the first time 
>> around, so the list rejected it.) 
>>
>> I see that there's now a btrfs send / receive and I've tried using it, but 
>> I'm getting the oops I've pasted below, after which the FS becomes 
>> unresponsive (no I/O to the drive, no CPU usage, but all attempts to access 
>> the FS results in a hang). I have an internal drive (single drive) that 
>> contains 82GB of compressed data with a couple hundred snapshots. I tried 
>> taking the first snapshot and making a read only copy (btrfs subvolume 
>> snapshot -r) and then I connected an external USB drive and ran btrfs send / 
>> receive to that external drive. It starts working and gets a couple of GB in 
>> (I'd expect the first snapshot to be about 20GB) and then gets the following 
>> error. I had to use the latest copy of btrfs-progs from git, because the 
>> package installed on my system (btrfs-progs-0.20-0.2.git91d9eec) simply 
>> returned "invalid argument" when trying to run btrfs send / receive. Thanks 
>> in advance for any info you may have. 
> 
> The problem has been introduced with rbtree ulists in 3.10, commit 
> 
> Btrfs: add a rb_tree to improve performance of ulist search 
> 
> You should be safe to revert that commit, it's a performance optimization 
> attempt. Alternatively, you can apply the published fix 
> 
> Btrfs: fix crash regarding to ulist_add_merge 
> 
> It has not made it into 3.10 stable or 3.11, yet, but is contained in Josef's 
> btrfs-next 
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
> 
> Thanks, 
> -Jan 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Cloning a Btrfs partition

[BJ Quinn]

Ok, so the fix is now in 3.10.6 and I'm using that.  I don't get the hang
anymore, but now I'm having a new problem.

Mount options --

rw,noatime,nodiratime,compress-force=zlib,space_cache,inode_cache,ssd

I need compression because I get a very high compression ratio with my data
and I have lots of snapshots, so it's the only way it can all fit. I have an
ssd and 24 cores anyway, so it should be fast. I need compress-force because
I have lots of files in my data which compress typically by a 10:1 or 20:1
ratio, but btrfs likes to see them as incompressible, so I need the
compress-force flag. I've just heard good things about space_cache
and inode_cache, so I've enabled them. The ssd option is because I do have an
ssd, but I have DRBD on top of it, and it looked like btrfs could not
automatically detect that it was an ssd (rotation speed was showing as "1").

Using newest btrfs-progs from git, because newest shipping btrfs-progs on
CentOS 6 returns an error for invalid argument.

I have a filesystem with maybe 1000 snapshots. They're daily snapshots of
a filesystem that is about 24GB compressed. The total space usage is 323GB
out of 469GB on an Intel SSD.

All the snapshots are writable, so I know I have to create a readonly
snapshot to copy to a backup drive.

btrfs subvolume snapshot -r /home/data/snapshots/storage\@NIGHTLY20101201 /home/data/snapshots\storageROTEMP

Then I send the snapshot to the backup drive, mounted with the same mount options.

btrfs send /home/data/snapshots/storageROTEMP | btrfs receive /mnt/backup/snapshots/

This takes about 5 hours to transfer 24GB compressed. Uncompressed it is about
150GB.  There is a "btrfs" process that takes 100% of one core during this 5
hour period.  There are some btrfs-endio and other processes that are using
small amounts of more than one core, but the "btrfs" process always takes 100%
and always only takes one core. And iostat clearly shows no significant disk
activity, so we're completely waiting on the btrfs command. Keep in mind that
the source filesystem is on an SSD, so it should be super fast. The destination
filesystem is on a hard drive connected via USB 2.0, but again, there's no
significant disk activity.  Processor is a dual socket Xeon E5-2420.

Then I try to copy another snapshot to the backup drive, hoping that it will
keep the space efficiency of the snapshots.

mv /mnt/backup/snapshots/storageROTEMP /mnt/backup/snapshots/storage\@NIGHTLY20101201
btrfs subvolume delete /home/data/snapshots/storageROTEMP
btrfs subvolume snapshot -r /home/data/snapshots/storage\@NIGHTLY20101202 /home/data/snapshots/storageROTEMP
btrfs send /home/data/snapshots/storageROTEMP | btrfs receive /mnt/backup/snapshots/

This results in a couple of problems. First of all, it takes 5 hours just like
the first snapshot did. Secondly, it takes up another ~20GB of data, so it's not
space efficient (I expect each snapshot should add far less than 500MB on
average due to the math on how many snapshots I have and how much total space
usage I have on the main filesystem). Finally, it doesn't even complete without
error. I get the following error after about 5 hours --

At subvol /home/data/snapshots/storageROTEMP
At subvol storageROTEMP
ERROR: send ioctl failed with -12: Cannot allocate memory
ERROR: unexpected EOF in stream.

So in the end, unless I'm doing something wrong, btrfs send is much slower
than just doing a full rsync of the first snapshot, and then incremental
rsyncs with the subsequent ones.  That and btrfs send doesn't seem to be
space efficient here (again, unless I'm using it incorrectly).

Thanks in advance for your help!

-BJ

----- Original Message ----- 

From: "Jan Schmidt" <mail@jan-o-sch.net> 
To: "BJ Quinn" <bj@placs.net> 
Cc: "Jan Schmidt" <list.btrfs@jan-o-sch.net>, linux-btrfs@vger.kernel.org, psusi@cfl.rr.com, "Freddie Cash" <fjwcash@gmail.com> 
Sent: Tuesday, July 30, 2013 5:28:00 AM 
Subject: Re: Cloning a Btrfs partition 

On Mon, July 29, 2013 at 17:32 (+0200), BJ Quinn wrote: 
> Thanks for the response! Not sure I want to roll a custom kernel on this 
> particular system. Any idea on when it might make it to 3.10 stable or 
> 3.11? Or should I just revert back to 3.9? 

I missed that it's in fact in 3.11 and if I got Liu Bo right he's going to send 
it to 3.10 stable soon. 

Thanks, 
-Jan 

> Thanks! 
> 
> -BJ 
> 
> ----- Original Message ----- 
> 
> From: "Jan Schmidt" <list.btrfs@jan-o-sch.net> 
> Sent: Monday, July 29, 2013 3:21:51 AM 
> 
> Hi BJ, 
> 
> [original message rewrapped] 
> 
> On Thu, July 25, 2013 at 18:32 (+0200), BJ Quinn wrote: 
>> (Apologies for the double post -- forgot to send as plain text the first time 
>> around, so the list rejected it.) 
>> 
>> I see that there's now a btrfs send / receive and I've tried using it, but 
>> I'm getting the oops I've pasted below, after which the FS becomes 
>> unresponsive (no I/O to the drive, no CPU usage, but all attempts to access 
>> the FS results in a hang). I have an internal drive (single drive) that 
>> contains 82GB of compressed data with a couple hundred snapshots. I tried 
>> taking the first snapshot and making a read only copy (btrfs subvolume 
>> snapshot -r) and then I connected an external USB drive and ran btrfs send / 
>> receive to that external drive. It starts working and gets a couple of GB in 
>> (I'd expect the first snapshot to be about 20GB) and then gets the following 
>> error. I had to use the latest copy of btrfs-progs from git, because the 
>> package installed on my system (btrfs-progs-0.20-0.2.git91d9eec) simply 
>> returned "invalid argument" when trying to run btrfs send / receive. Thanks 
>> in advance for any info you may have. 
> 
> The problem has been introduced with rbtree ulists in 3.10, commit 
> 
> Btrfs: add a rb_tree to improve performance of ulist search 
> 
> You should be safe to revert that commit, it's a performance optimization 
> attempt. Alternatively, you can apply the published fix 
> 
> Btrfs: fix crash regarding to ulist_add_merge 
> 
> It has not made it into 3.10 stable or 3.11, yet, but is contained in Josef's 
> btrfs-next 
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/josef/btrfs-next.git 
> 
> Thanks, 
> -Jan 
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in 
> the body of a message to majordomo@vger.kernel.org 
> More majordomo info at http://vger.kernel.org/majordomo-info.html 
> 

Re: Cloning a Btrfs partition

[Xavier Bassery]

On Mon, 19 Aug 2013 15:45:32 -0500 (CDT)
BJ Quinn <bj@placs.net> wrote:

> Ok, so the fix is now in 3.10.6 and I'm using that.  I don't get the
> hang anymore, but now I'm having a new problem.
> 
> Mount options --
> 
> rw,noatime,nodiratime,compress-force=zlib,space_cache,inode_cache,ssd
> 
> I need compression because I get a very high compression ratio with
> my data and I have lots of snapshots, so it's the only way it can all
> fit. I have an ssd and 24 cores anyway, so it should be fast. I need
> compress-force because I have lots of files in my data which compress
> typically by a 10:1 or 20:1 ratio, but btrfs likes to see them as
> incompressible, so I need the compress-force flag. I've just heard
> good things about space_cache and inode_cache, so I've enabled them.
> The ssd option is because I do have an ssd, but I have DRBD on top of
> it, and it looked like btrfs could not automatically detect that it
> was an ssd (rotation speed was showing as "1").
> 
> Using newest btrfs-progs from git, because newest shipping
> btrfs-progs on CentOS 6 returns an error for invalid argument.
> 
> I have a filesystem with maybe 1000 snapshots. They're daily
> snapshots of a filesystem that is about 24GB compressed. The total
> space usage is 323GB out of 469GB on an Intel SSD.
> 
> All the snapshots are writable, so I know I have to create a readonly
> snapshot to copy to a backup drive.

Hi BJ,

I am curious to know why you use writable snapshots instead of
read-only?
When I use snapshots as a base for backups, I create them read-only, so
that I don't need to worry something might have accidentally changed in
any of those.
I only use writable ones in cases when I actually need to write to them
(e.g. doing an experimental upgrade on a system root subvolume).
As a bonus, this would save you the need to:
1. create a ro snapshot of your rw one
2. rename the sent snapshot on the destination fs to a meaningful name.

> 
> btrfs subvolume snapshot
> -r /home/data/snapshots/storage\@NIGHTLY20101201 /home/data/snapshots\storageROTEMP
> 
> Then I send the snapshot to the backup drive, mounted with the same
> mount options.
> 
> btrfs send /home/data/snapshots/storageROTEMP | btrfs
> receive /mnt/backup/snapshots/
> 
> This takes about 5 hours to transfer 24GB compressed. Uncompressed it
> is about 150GB.  There is a "btrfs" process that takes 100% of one
> core during this 5 hour period.  There are some btrfs-endio and other
> processes that are using small amounts of more than one core, but the
> "btrfs" process always takes 100% and always only takes one core. And
> iostat clearly shows no significant disk activity, so we're
> completely waiting on the btrfs command. Keep in mind that the source
> filesystem is on an SSD, so it should be super fast. The destination
> filesystem is on a hard drive connected via USB 2.0, but again,
> there's no significant disk activity.  Processor is a dual socket
> Xeon E5-2420.

5 hours for 150GB, meaning you only get ~8MB/s to your USB2 external
HD (instead of the ~25MB/s you could expect from USB2) is indeed rather
slow. 
But as you have noticed, your bottleneck here is cpu-bound, which I
guess you find frustrating given how powerful your system is (2 x 6
cores cpu + hyperthreading = 24 threads).
Your case may illustrate the need for more parallelism...

My guess is that the poor performance stems from your choice of
'compress-force=zlib' mount option.
First, zlib compression is known to be slower than lzo while able to
give higher compression ratios. 
Secondly, 'compress-force' while giving you even better compression
means that your system will also compress already highly compressed
files (and potentially big and/or numerous).
To sum up, you have chosen space efficiency at the cost of performance
because of the lack of parallelism in this particular use case (so
your multi-core system cannot help).

> 
> Then I try to copy another snapshot to the backup drive, hoping that
> it will keep the space efficiency of the snapshots.
> 
> mv /mnt/backup/snapshots/storageROTEMP /mnt/backup/snapshots/storage\@NIGHTLY20101201
> btrfs subvolume delete /home/data/snapshots/storageROTEMP
> btrfs subvolume snapshot
> -r /home/data/snapshots/storage\@NIGHTLY20101202 /home/data/snapshots/storageROTEMP
> btrfs send /home/data/snapshots/storageROTEMP | btrfs
> receive /mnt/backup/snapshots/
> 
> This results in a couple of problems. First of all, it takes 5 hours
> just like the first snapshot did. Secondly, it takes up another ~20GB
> of data, so it's not space efficient (I expect each snapshot should
> add far less than 500MB on average due to the math on how many
> snapshots I have and how much total space usage I have on the main
> filesystem).

It is not surprising that it takes another 5 hours, because you've sent
a full copy of your new snapshot made at day+1! What you should have
done instead is :

btrfs send -p <path_of_parent_snapshot> <path_of_next_snapshot>, so in
your case that would be:

btrfs send -p [...]20101201 [...]20101202 | btrfs receive
<path_to_backup_volume>

(I have omitted your paths in the above for clarity).
For this to work, you need to use read-only dated snapshots.

> Finally, it doesn't even complete without error. I get
> the following error after about 5 hours --
> 
> At subvol /home/data/snapshots/storageROTEMP
> At subvol storageROTEMP
> ERROR: send ioctl failed with -12: Cannot allocate memory
> ERROR: unexpected EOF in stream.

I am not competent enough to explain this error.

> 
> So in the end, unless I'm doing something wrong, btrfs send is much
> slower than just doing a full rsync of the first snapshot, and then
> incremental rsyncs with the subsequent ones.  That and btrfs send
> doesn't seem to be space efficient here (again, unless I'm using it
> incorrectly).

At least you were right supposing you were not using it correctly :p

Best regards,
Xavier

Re: Cloning a Btrfs partition

[BJ Quinn]

The use of writable snapshots isn't necessary.  It's just what I had to
start with.  I'm sure I could switch to using read only snapshots
exclusively to skip the additional steps.

As for the throughput, the disparity between actual speed and a speed
I might expect to achieve is that much greater over USB 3.0 or
SATA.  I have sort of a strange data set.  It's primarily legacy
FoxPro DBF files, which are mostly empty space.  For some reason, btrfs
thinks they're incompressible when not using compress-force.  Nearly
everything on my filesystem is compressible.  There are a few directories
with lots of small files, but it's primarily large (100MB+) compressible
file-based databases.  Anyway, due to choices made by btrfs with
respect to detecting compressibility, I'm required to use the
compress-force option.

As for the compression method, I will go ahead and try lzo and see how
much space I lose.  It may be worth the tradeoff if I end up with
better performance.  Perhaps lzo in conjunction with readonly snaps
AND the proper syntax for sending incremental snaps will make btrfs
send work for my situation.  Thanks for the suggestions!!!

At any rate, it seems that btrfs send would benefit from parallelism
if it were at all reasonably possible to do so.  I'm surprised ANY
compression method could really tax modern hardware to that extent.

-BJ

----- Original Message ----- 

From: "Xavier Bassery" <xavier@bartica.org> 
To: "BJ Quinn" <bj@placs.net> 
Cc: psusi@cfl.rr.com, "Jan Schmidt" <list.btrfs@jan-o-sch.net>, linux-btrfs@vger.kernel.org, "Freddie Cash" <fjwcash@gmail.com>, "bo li liu" <bo.li.liu@oracle.com> 
Sent: Tuesday, August 20, 2013 4:59:23 AM 
Subject: Re: Cloning a Btrfs partition 

On Mon, 19 Aug 2013 15:45:32 -0500 (CDT) 
BJ Quinn <bj@placs.net> wrote: 

> Ok, so the fix is now in 3.10.6 and I'm using that. I don't get the 
> hang anymore, but now I'm having a new problem. 
> 
> Mount options -- 
> 
> rw,noatime,nodiratime,compress-force=zlib,space_cache,inode_cache,ssd 
> 
> I need compression because I get a very high compression ratio with 
> my data and I have lots of snapshots, so it's the only way it can all 
> fit. I have an ssd and 24 cores anyway, so it should be fast. I need 
> compress-force because I have lots of files in my data which compress 
> typically by a 10:1 or 20:1 ratio, but btrfs likes to see them as 
> incompressible, so I need the compress-force flag. I've just heard 
> good things about space_cache and inode_cache, so I've enabled them. 
> The ssd option is because I do have an ssd, but I have DRBD on top of 
> it, and it looked like btrfs could not automatically detect that it 
> was an ssd (rotation speed was showing as "1"). 
> 
> Using newest btrfs-progs from git, because newest shipping 
> btrfs-progs on CentOS 6 returns an error for invalid argument. 
> 
> I have a filesystem with maybe 1000 snapshots. They're daily 
> snapshots of a filesystem that is about 24GB compressed. The total 
> space usage is 323GB out of 469GB on an Intel SSD. 
> 
> All the snapshots are writable, so I know I have to create a readonly 
> snapshot to copy to a backup drive. 

Hi BJ, 

I am curious to know why you use writable snapshots instead of 
read-only? 
When I use snapshots as a base for backups, I create them read-only, so 
that I don't need to worry something might have accidentally changed in 
any of those. 
I only use writable ones in cases when I actually need to write to them 
(e.g. doing an experimental upgrade on a system root subvolume). 
As a bonus, this would save you the need to: 
1. create a ro snapshot of your rw one 
2. rename the sent snapshot on the destination fs to a meaningful name. 

> 
> btrfs subvolume snapshot 
> -r /home/data/snapshots/storage\@NIGHTLY20101201 /home/data/snapshots\storageROTEMP 
> 
> Then I send the snapshot to the backup drive, mounted with the same 
> mount options. 
> 
> btrfs send /home/data/snapshots/storageROTEMP | btrfs 
> receive /mnt/backup/snapshots/ 
> 
> This takes about 5 hours to transfer 24GB compressed. Uncompressed it 
> is about 150GB. There is a "btrfs" process that takes 100% of one 
> core during this 5 hour period. There are some btrfs-endio and other 
> processes that are using small amounts of more than one core, but the 
> "btrfs" process always takes 100% and always only takes one core. And 
> iostat clearly shows no significant disk activity, so we're 
> completely waiting on the btrfs command. Keep in mind that the source 
> filesystem is on an SSD, so it should be super fast. The destination 
> filesystem is on a hard drive connected via USB 2.0, but again, 
> there's no significant disk activity. Processor is a dual socket 
> Xeon E5-2420. 

5 hours for 150GB, meaning you only get ~8MB/s to your USB2 external 
HD (instead of the ~25MB/s you could expect from USB2) is indeed rather 
slow. 
But as you have noticed, your bottleneck here is cpu-bound, which I 
guess you find frustrating given how powerful your system is (2 x 6 
cores cpu + hyperthreading = 24 threads). 
Your case may illustrate the need for more parallelism... 

My guess is that the poor performance stems from your choice of 
'compress-force=zlib' mount option. 
First, zlib compression is known to be slower than lzo while able to 
give higher compression ratios. 
Secondly, 'compress-force' while giving you even better compression 
means that your system will also compress already highly compressed 
files (and potentially big and/or numerous). 
To sum up, you have chosen space efficiency at the cost of performance 
because of the lack of parallelism in this particular use case (so 
your multi-core system cannot help). 

> 
> Then I try to copy another snapshot to the backup drive, hoping that 
> it will keep the space efficiency of the snapshots. 
> 
> mv /mnt/backup/snapshots/storageROTEMP /mnt/backup/snapshots/storage\@NIGHTLY20101201 
> btrfs subvolume delete /home/data/snapshots/storageROTEMP 
> btrfs subvolume snapshot 
> -r /home/data/snapshots/storage\@NIGHTLY20101202 /home/data/snapshots/storageROTEMP 
> btrfs send /home/data/snapshots/storageROTEMP | btrfs 
> receive /mnt/backup/snapshots/ 
> 
> This results in a couple of problems. First of all, it takes 5 hours 
> just like the first snapshot did. Secondly, it takes up another ~20GB 
> of data, so it's not space efficient (I expect each snapshot should 
> add far less than 500MB on average due to the math on how many 
> snapshots I have and how much total space usage I have on the main 
> filesystem). 

It is not surprising that it takes another 5 hours, because you've sent 
a full copy of your new snapshot made at day+1! What you should have 
done instead is : 

btrfs send -p <path_of_parent_snapshot> <path_of_next_snapshot>, so in 
your case that would be: 

btrfs send -p [...]20101201 [...]20101202 | btrfs receive 
<path_to_backup_volume> 

(I have omitted your paths in the above for clarity). 
For this to work, you need to use read-only dated snapshots. 

> Finally, it doesn't even complete without error. I get 
> the following error after about 5 hours -- 
> 
> At subvol /home/data/snapshots/storageROTEMP 
> At subvol storageROTEMP 
> ERROR: send ioctl failed with -12: Cannot allocate memory 
> ERROR: unexpected EOF in stream. 

I am not competent enough to explain this error. 

> 
> So in the end, unless I'm doing something wrong, btrfs send is much 
> slower than just doing a full rsync of the first snapshot, and then 
> incremental rsyncs with the subsequent ones. That and btrfs send 
> doesn't seem to be space efficient here (again, unless I'm using it 
> incorrectly). 

At least you were right supposing you were not using it correctly :p 

Best regards, 
Xavier 

kernel BUG at fs/btrfs/ctree.c:2964!

[BJ Quinn]

I keep running into this issue --

Oct 31 19:45:26 localhost kernel: kernel BUG at fs/btrfs/ctree.c:2964!
Oct 31 19:45:26 localhost kernel: invalid opcode: 0000 [#2] SMP 
Oct 31 19:45:26 localhost kernel: Modules linked in: des_generic ecb md4 nls_utf8 cifs fscache dns_resolver usb_storage fuse ip6table_filter ip6_tables 
ebtable_nat ebtables autofs4 bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_t
gt 8021q garp sunrpc bridge stp llc ipv6 ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables btrfs raid6_pq xor libcrc32c vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support dcdbas i2c_i801 pcspkr microcode sg lpc_ich bnx2 freq_table mperf ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic pata_jmicron ahci libahci mgag200 ttm drm_kms_helper sysimgblt sysfillrect syscopyarea dm_mirror dm_region_hash dm_log dm_mod
Oct 31 19:45:26 localhost kernel: CPU: 3 PID: 6245 Comm: btrfs-endio-wri Tainted: G      D      3.11.4-1.el6.elrepo.x86_64 #1
Oct 31 19:45:26 localhost kernel: Hardware name: Dell Inc. PowerEdge R210 II/09T7VV, BIOS 1.2.3 07/21/2011
Oct 31 19:45:26 localhost kernel: task: ffff880428ed2040 ti: ffff8803b041c000 task.ti: ffff8803b041c000
Oct 31 19:45:26 localhost kernel: RIP: 0010:[<ffffffffa02cde10>]  [<ffffffffa02cde10>] btrfs_set_item_key_safe+0x190/0x1d0 [btrfs]
Oct 31 19:45:26 localhost kernel: RSP: 0018:ffff8803b041dac8  EFLAGS: 00010287
Oct 31 19:45:26 localhost kernel: RAX: 0000000000046738 RBX: 000000000000001d RCX: 0000000000b5b000
Oct 31 19:45:26 localhost kernel: RDX: 000000000000006c RSI: 0000000000b4b000 RDI: ffff8803b041dae9
Oct 31 19:45:26 localhost kernel: RBP: ffff8803b041db28 R08: 0000000000001000 R09: ffff8803b041dae0
Oct 31 19:45:26 localhost kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880325147e20
Oct 31 19:45:26 localhost kernel: R13: ffff8803b041dc08 R14: ffff8803b041dad8 R15: ffff8801785b4d00
Oct 31 19:45:26 localhost kernel: FS:  0000000000000000(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
Oct 31 19:45:26 localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct 31 19:45:26 localhost kernel: CR2: ffffffffff600400 CR3: 0000000001c0c000 CR4: 00000000000407e0
Oct 31 19:45:26 localhost kernel: Stack:
Oct 31 19:45:26 localhost kernel: ffff8803b041db28 ffff8804007ef000 0000000000046738 00000000b4b0006c
Oct 31 19:45:26 localhost kernel: 0000000000000000 ffff88022d617000 ffff8803b041db28 ffff8801785b4d00
Oct 31 19:45:26 localhost kernel: ffff880325147e20 0000000000b63000 0000000000b43000 0000000000000001
Oct 31 19:45:26 localhost kernel: Call Trace:
Oct 31 19:45:26 localhost kernel: [<ffffffffa030b23d>] __btrfs_drop_extents+0x59d/0xb90 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffff8118e7c5>] ? kmem_cache_alloc+0x275/0x280
Oct 31 19:45:26 localhost kernel: [<ffffffffa030c343>] btrfs_drop_extents+0x73/0xa0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa02fc33c>] insert_reserved_file_extent.clone.0+0x7c/0x290 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa02f7859>] ? start_transaction+0x99/0x4e0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa0304082>] btrfs_finish_ordered_io+0x452/0x4f0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa0304135>] finish_ordered_fn+0x15/0x20 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa03254fc>] worker_loop+0x15c/0x4b0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa03253a0>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffffa03253a0>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs]
Oct 31 19:45:26 localhost kernel: [<ffffffff8108a14e>] kthread+0xce/0xe0
Oct 31 19:45:26 localhost kernel: [<ffffffff8108a080>] ? kthread_freezable_should_stop+0x70/0x70
Oct 31 19:45:26 localhost kernel: [<ffffffff81604bac>] ret_from_fork+0x7c/0xb0
Oct 31 19:45:26 localhost kernel: [<ffffffff8108a080>] ? kthread_freezable_should_stop+0x70/0x70
Oct 31 19:45:26 localhost kernel: Code: 8b 7d f8 c9 c3 66 0f 1f 44 00 00 72 1e 41 0f b6 55 08 38 d1 76 0d 49 8b 4d 09 eb 8c 0f 1f 80 00 00 00 00 73 2e 66 0f 1f 44 00 00 <0f> 0b eb fe 0f 1f 40 00 49 3b 55 09 0f 1f 40 00 0f 87 c3 fe ff 
Oct 31 19:45:26 localhost kernel: RIP  [<ffffffffa02cde10>] btrfs_set_item_key_safe+0x190/0x1d0 [btrfs]
Oct 31 19:45:26 localhost kernel: RSP <ffff8803b041dac8>

All Google will tell me related to errors in btrfs_set_item_key_safe is some vague references to memory corruption.  As I have gotten this error on several machines with different types of hardware, memory, and storage, I don't think that's the issue in my case.  I've seen what looks to be the same issue (with slightly differing line numbers in ctree.c) over several kernel versions.  Currently I'm at 3.11.4, although I remember having this issue at least as far back as 3.7 or 3.8.  It's possible that the issue predates that as well, but I was having different issues at the time as well, so I can't be sure.

All I can tell that I might be doing differently is that I generally have a fairly large filesystem with lots of snapshots and my mount options, which are noatime,nodiratime,compress-force=zlib,inode_cache.  I did mount once with space_cache.  The system that is currently having the issue has a 4x3TB striped array (btrfs is handling the RAID) =12TB, but I also got it multiple times on a system with a single 500GB SSD, but there were lots of snapshots (read: 100s) of 100GB+ of compressed data.  Much of what I'm doing is backups, so I have a daily snapshot of large amounts of highly compressible data that changes <1% daily.

Sometimes I can work around the issue by deleting the offending snapshot (I get hangs that require a hard reboot during the backup process) and retrying.

-BJ Quinn

Re: kernel BUG at fs/btrfs/ctree.c:2964!

[Josef Bacik]

On Fri, Nov 01, 2013 at 07:28:41PM -0500, BJ Quinn wrote:
> I keep running into this issue --
> 
> Oct 31 19:45:26 localhost kernel: kernel BUG at fs/btrfs/ctree.c:2964!
> Oct 31 19:45:26 localhost kernel: invalid opcode: 0000 [#2] SMP 
> Oct 31 19:45:26 localhost kernel: Modules linked in: des_generic ecb md4 nls_utf8 cifs fscache dns_resolver usb_storage fuse ip6table_filter ip6_tables 
> ebtable_nat ebtables autofs4 bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_t
> gt 8021q garp sunrpc bridge stp llc ipv6 ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle xt_physdev ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables btrfs raid6_pq xor libcrc32c vhost_net macvtap macvlan vhost tun kvm_intel kvm uinput iTCO_wdt iTCO_vendor_support dcdbas i2c_i801 pcspkr microcode sg lpc_ich bnx2 freq_table mperf ext4 jbd2 mbcache sd_mod crc_t10dif sr_mod cdrom pata_acpi ata_generic pata_jmicron ahci libahci mgag200 ttm drm_kms_helper sysimgblt sysfillrect syscopyarea dm_mirror dm_region_hash dm_log dm_mod
> Oct 31 19:45:26 localhost kernel: CPU: 3 PID: 6245 Comm: btrfs-endio-wri Tainted: G      D      3.11.4-1.el6.elrepo.x86_64 #1
> Oct 31 19:45:26 localhost kernel: Hardware name: Dell Inc. PowerEdge R210 II/09T7VV, BIOS 1.2.3 07/21/2011
> Oct 31 19:45:26 localhost kernel: task: ffff880428ed2040 ti: ffff8803b041c000 task.ti: ffff8803b041c000
> Oct 31 19:45:26 localhost kernel: RIP: 0010:[<ffffffffa02cde10>]  [<ffffffffa02cde10>] btrfs_set_item_key_safe+0x190/0x1d0 [btrfs]
> Oct 31 19:45:26 localhost kernel: RSP: 0018:ffff8803b041dac8  EFLAGS: 00010287
> Oct 31 19:45:26 localhost kernel: RAX: 0000000000046738 RBX: 000000000000001d RCX: 0000000000b5b000
> Oct 31 19:45:26 localhost kernel: RDX: 000000000000006c RSI: 0000000000b4b000 RDI: ffff8803b041dae9
> Oct 31 19:45:26 localhost kernel: RBP: ffff8803b041db28 R08: 0000000000001000 R09: ffff8803b041dae0
> Oct 31 19:45:26 localhost kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff880325147e20
> Oct 31 19:45:26 localhost kernel: R13: ffff8803b041dc08 R14: ffff8803b041dad8 R15: ffff8801785b4d00
> Oct 31 19:45:26 localhost kernel: FS:  0000000000000000(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
> Oct 31 19:45:26 localhost kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> Oct 31 19:45:26 localhost kernel: CR2: ffffffffff600400 CR3: 0000000001c0c000 CR4: 00000000000407e0
> Oct 31 19:45:26 localhost kernel: Stack:
> Oct 31 19:45:26 localhost kernel: ffff8803b041db28 ffff8804007ef000 0000000000046738 00000000b4b0006c
> Oct 31 19:45:26 localhost kernel: 0000000000000000 ffff88022d617000 ffff8803b041db28 ffff8801785b4d00
> Oct 31 19:45:26 localhost kernel: ffff880325147e20 0000000000b63000 0000000000b43000 0000000000000001
> Oct 31 19:45:26 localhost kernel: Call Trace:
> Oct 31 19:45:26 localhost kernel: [<ffffffffa030b23d>] __btrfs_drop_extents+0x59d/0xb90 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffff8118e7c5>] ? kmem_cache_alloc+0x275/0x280
> Oct 31 19:45:26 localhost kernel: [<ffffffffa030c343>] btrfs_drop_extents+0x73/0xa0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa02fc33c>] insert_reserved_file_extent.clone.0+0x7c/0x290 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa02f7859>] ? start_transaction+0x99/0x4e0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa0304082>] btrfs_finish_ordered_io+0x452/0x4f0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa0304135>] finish_ordered_fn+0x15/0x20 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa03254fc>] worker_loop+0x15c/0x4b0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa03253a0>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffffa03253a0>] ? check_pending_worker_creates+0xe0/0xe0 [btrfs]
> Oct 31 19:45:26 localhost kernel: [<ffffffff8108a14e>] kthread+0xce/0xe0
> Oct 31 19:45:26 localhost kernel: [<ffffffff8108a080>] ? kthread_freezable_should_stop+0x70/0x70
> Oct 31 19:45:26 localhost kernel: [<ffffffff81604bac>] ret_from_fork+0x7c/0xb0
> Oct 31 19:45:26 localhost kernel: [<ffffffff8108a080>] ? kthread_freezable_should_stop+0x70/0x70
> Oct 31 19:45:26 localhost kernel: Code: 8b 7d f8 c9 c3 66 0f 1f 44 00 00 72 1e 41 0f b6 55 08 38 d1 76 0d 49 8b 4d 09 eb 8c 0f 1f 80 00 00 00 00 73 2e 66 0f 1f 44 00 00 <0f> 0b eb fe 0f 1f 40 00 49 3b 55 09 0f 1f 40 00 0f 87 c3 fe ff 
> Oct 31 19:45:26 localhost kernel: RIP  [<ffffffffa02cde10>] btrfs_set_item_key_safe+0x190/0x1d0 [btrfs]
> Oct 31 19:45:26 localhost kernel: RSP <ffff8803b041dac8>
> 
> All Google will tell me related to errors in btrfs_set_item_key_safe is some vague references to memory corruption.  As I have gotten this error on several machines with different types of hardware, memory, and storage, I don't think that's the issue in my case.  I've seen what looks to be the same issue (with slightly differing line numbers in ctree.c) over several kernel versions.  Currently I'm at 3.11.4, although I remember having this issue at least as far back as 3.7 or 3.8.  It's possible that the issue predates that as well, but I was having different issues at the time as well, so I can't be sure.
> 
> All I can tell that I might be doing differently is that I generally have a fairly large filesystem with lots of snapshots and my mount options, which are noatime,nodiratime,compress-force=zlib,inode_cache.  I did mount once with space_cache.  The system that is currently having the issue has a 4x3TB striped array (btrfs is handling the RAID) =12TB, but I also got it multiple times on a system with a single 500GB SSD, but there were lots of snapshots (read: 100s) of 100GB+ of compressed data.  Much of what I'm doing is backups, so I have a daily snapshot of large amounts of highly compressible data that changes <1% daily.
> 
> Sometimes I can work around the issue by deleting the offending snapshot (I get hangs that require a hard reboot during the backup process) and retrying.

On this box can you run

gdb btrfs.ko
list *(__btrfs_drop_extents+0x59d)

I want to know where exactly this is coming from so I can start trying to figure
out how it's happening.  Thanks,

Josef

Re: kernel BUG at fs/btrfs/ctree.c:2964!

[Ilari Stenroth]

On 2.11.2013 3.34, Josef Bacik wrote:
> 
> On this box can you run
> 
> gdb btrfs.ko
> list *(__btrfs_drop_extents+0x59d)
> 
> I want to know where exactly this is coming from so I can start trying to figure
> out how it's happening.  Thanks,

Hi,

I've got the same issue. Btrfs FS crashes on my box running Linux 3.11.7.

It seems that the crash happens when I start collectd daemon. Maybe a
corrupted RRD file somewhere.
I don't have time for debugging this issue further so I'm going to
re-format this box to ext4. Btrfs is statically compiled in kernel so
I'm unable to gdb btrfs.ko. No debug info in kernel image. Sorry not
being able to help.

/dev/sda2 on / type btrfs (rw,noatime,nodiratime,ssd,space_cache)

Mount options used to have also autodefrag but I removed it while hoping
it would make the box more stable.

# btrfs fi show
Label: 'root'  uuid: 87943e37-a134-4b8d-bcf2-989334d23b75
	Total devices 2 FS bytes used 6.22GB
	devid    2 size 10.00GB used 9.98GB path /dev/sdb2
	devid    1 size 10.00GB used 10.00GB path /dev/sda2


# btrfs fi df /
Data, RAID1: total=8.97GB, used=5.99GB
Data: total=8.00MB, used=0.00
System, RAID1: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, RAID1: total=1.00GB, used=234.92MB
Metadata: total=8.00MB, used=0.00

A screen grab from the console after the crash happened:
http://s3-eu-west-1.amazonaws.com/ilari.stenroth/btrfs-crash-201311/Screen+Shot+2013-11-07+at+9.19.27.png

--
Ilari Stenroth