btrfs and load (sys)

btrfs and load (sys)

[krzf83@gmail.com]

I've been using btrfs for two months now. Every day between 02:00 and
08:00 I rsync some 300GB data (milions of files) to btrfs device and
then make snapshot. Next day i rsync again 300GG little changed (rsync
"in place"). First days it worked perfectly. Then loadavg (sys load)
started to rise. Now, after 60 days of rsync+shapshots sys load is so
high that whole server becames quite unresponsive betweend 02:00 and
08:00. I never deleted any shapshot (yet) to its not rebalancing
thing.
What all those btrfs benchmark does not tell you that its performance
decreases (sys load increases) with growing size of btree. Creating
btrfs filesystem is instantaneous because initial tree is just
nothing...
I tried using btrfs and nilfs2 for a half year now and I would never
trust a byte of production data to it.

Re: btrfs and load (sys)

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1394 bytes --]

On Wed, 23 Nov 2011 15:43:14 +0100
"krzf83@gmail.com " <krzf83@gmail.com> wrote:

> I've been using btrfs for two months now. Every day between 02:00 and
> 08:00 I rsync some 300GB data (milions of files) to btrfs device and
> then make snapshot. Next day i rsync again 300GG little changed (rsync
> "in place"). First days it worked perfectly. Then loadavg (sys load)
> started to rise. Now, after 60 days of rsync+shapshots sys load is so
> high that whole server becames quite unresponsive betweend 02:00 and
> 08:00. I never deleted any shapshot (yet) to its not rebalancing
> thing.

Hello,

I employ the same scenario, except that my rsync-(actually mirrordir-) destination btrfs device is only mounted briefly for the copy-then-snapshot operation, and unmounted during the rest of the time. Haven't noticed any performance problems yet, but even though I have about 2 TB of data there, it's probably not millions of files.

Have you tried unmounting the FS and then mounting it again, to see if it solves the performance problem (at least for a while)? Or just keeping it mostly unmounted, I suppose what you have is a partition with backups, so keeping it mostly offline is better reliability-wise.


-- 
With respect,
Roman

~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: btrfs and load (sys)

[krzf83@gmail.com]

Yes, yesterday I've umouted partition and re-mounted it. Nothing has
changed this night.

You can look at my load graphs here:
http://img42.imageshack.us/img42/4661/33737291.png
http://img210.imageshack.us/img210/2742/46527625.png

On the second one blue is SYS load. I bet you can reasily spot time
when backup to btrfs partition was running :)

I doubt its a bug in btrfs. It is just broken concept :(


# uname -a
Linux sv15.vipserv.org 3.0.4-xxxx-std-ipv6-64 #1 SMP Mon Oct 10
01:07:40 CEST 2011 x86_64 x86_64 x86_64 GNU/Linux

Re: btrfs and load (sys)

[krzf83@gmail.com]

I see that this high sys load and loadavg did not appear gradualy. It
appeared first time about 7 days ago and were present every backup,
every night since then.

During backup top shows higher cpu load on

 4216 root      20   0     0    0    0 R 20.3  0.0 284:25.83 btrfs-delayed-m
 4222 root      20   0     0    0    0 D 19.6  0.0 203:23.89 btrfs-transacti

however one cannot really debug sys load so it does not tell much.
Scenario here is quite simple (rsync to working dir and then snapshot,
next night rsync (in place) to working dir and again snapshot). Rsync
is killed at 08:00.
I will try rebooting server this night but I doubt it will change
anything on those high loads during backup to btrfs partition.

Re: btrfs and load (sys)

[Phillip Susi]

On 11/23/2011 9:43 AM, krzf83@gmail.com wrote:
> What all those btrfs benchmark does not tell you that its performance
> decreases (sys load increases) with growing size of btree. Creating
> btrfs filesystem is instantaneous because initial tree is just
> nothing...

While something is clearly wrong, this isn't it.  Each snapshot is its 
own btree, and you said there is little churn each day, so they aren't 
getting significantly larger over time.  Each snapshot is tracked in the 
tree of tree roots, so technically it is growing each time you take a 
snapshot, but 60 items is nothing.

Re: btrfs and load (sys)

[krzf83@gmail.com]

I've rebooted server and run backup to btrfs partition again. It seems
that problem is gone, high sys load does not occur now. So it is some
bug in btrfs... Before reboot server had 30 days uptime so its really
not much.

Re: btrfs and load (sys)

[Chris Samuel]

On 24/11/11 02:08, krzf83@gmail.com wrote:

> Linux sv15.vipserv.org 3.0.4-xxxx-std-ipv6-64 #1 SMP Mon Oct 10
> 01:07:40 CEST 2011 x86_64 x86_64 x86_64 GNU/Linux

You're running quite an old kernel btrfs wise - it would be
very interesting to see if you have the same issue with Linus's
current git version (which has the latest btrfs code) or
wait for 3.2-rc3.

Another possibility I *think* is that you could try 3.1 with
Chris Mason's for-linus git branch pulled into it.  Hopefully
someone who knows the procedure better than I can correct me
on this! :-)

There's a lot of work gone on both in fixing up corruption
issues on power loss and also reducing a lot of unnecessary
I/O that was happening due to bugs since that release.

cheers!
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: btrfs and load (sys)

[Fajar A. Nugraha]

On Thu, Nov 24, 2011 at 8:00 AM, Chris Samuel <chris@csamuel.org> wrote=
:
> Another possibility I *think* is that you could try 3.1 with
> Chris Mason's for-linus git branch pulled into it. =A0Hopefully
> someone who knows the procedure better than I can correct me
> on this! :-)

My method is:
- use 3.1.1 (latest stable at the time I was compiling it), compiie
btrfs as module
- use only fs/btrfs directory from Chris' for-linus tree, compile the
module externally (make -C /lib/modules/`pwd`/build M=3D$(pwd) modules)
- put the resulting btrfs.ko on /lib/modules/`uname -r`/updates/manual/
- depmod -a
- verify the new module is selected by default (modinfo btrfs)
- rebuild initrd
- reboot

--=20
=46ajar
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs and load (sys)

[Chris Samuel]

On 24/11/11 12:00, Chris Samuel wrote:

> You're running quite an old kernel btrfs wise - it would be
> very interesting to see if you have the same issue with Linus's
> current git version (which has the latest btrfs code) or
> wait for 3.2-rc3.

3.2-rc3 is now out, Linus is after testers around the world
because of some holiday in the US.

cheers!
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

Re: btrfs and load (sys)

[krzf83@gmail.com]

I've upgraded to newest kernel 3.1.2 but problem is not fixed. After
reboot btrfs works fine to maybe few hours but after that system stats
to choke on sys load, system becames unresponsive, and backups to
btrfs becames extremly slow. Anything you want me to check so you can
diagnose the problem - before I will return to ext4?

blue on graph is sys load. It should not be visible at all. Something
is seriously wrong with btrfs
http://img265.imageshack.us/img265/1416/37248826.png

I use snapshots for every important directory (after each backup) so
there quite lot of them but this is a reason why I use btrfs.

# btrfs sub list /backup|wc -l
7651

Re: btrfs and load (sys)

[Chris Samuel]

On 02/12/11 15:14, krzf83@gmail.com wrote:

> I've upgraded to newest kernel 3.1.2 but problem is not fixed. 

As btrfs is an experimental filesystem I don't believe that fixes are
backported to those stable releases, they only go into Linus's tree for
his next release (3.2 in this case).  So I'm afraid if you want to pick
up fixes as they appear you'll need to track Linus's tree really (or one
of Chris's).

I guess once btrfs is stable then that'll start to happen, but for the
moment I get the feeling that fixes and features are so tightly
entangled that it would be an awful lot of work to backport them to the
older code in the last stable release.

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC