Rebalance makes BTRFS 10x slower

Rebalance makes BTRFS 10x slower

[Swâmi Petaramesh]

Hi there,

# uname -r
3.13.6-1-ARCH
# btrfs --version
Btrfs v3.12

After having read the recent discussion about rebalance, I ran it for a test 
on my laptop with a 1TB HD, which current situation (after rebalance) is :

# btrfs fi sh
Label: TETHYS  uuid: 9a1ca6f4-1c1b-4a62-a84b-b388066084dc
        Total devices 1 FS bytes used 575.56GiB
        devid    1 size 845.00GiB used 580.06GiB path /dev/dm-3

Label: BOOT  uuid: 6a16d133-4b99-47b2-876f-148a8266f58f
        Total devices 1 FS bytes used 67.99MiB
        devid    1 size 1.00GiB used 144.00MiB path /dev/dm-0

Btrfs v3.12

# btrfs fi df /
Data, single: total=574.00GiB, used=573.60GiB
System, DUP: total=32.00MiB, used=72.00KiB
Metadata, DUP: total=3.00GiB, used=1.96GiB

# df -h /boot /
Sys. de fichiers Taille Utilisé Dispo Uti% Monté sur
/dev/dm-0          1,0G     68M  924M   7% /boot
/dev/dm-3          845G    578G  266G  69% /


The rebalance for the biggest BTRFS took about 19 hours.

I was expecting either a speed improvement after rebalance, or no noticeable 
effect, but I am extremely disappointed to see that now (and after having 
rebooted), my system has become slow like hell, takes at least 10x longer to 
boot and operate, to the point it has become hardly usable :-(

I would have thought a rebalance would have improved the filesystem 
organization, looks like it's the absolute contrary :-(

-- 
Swâmi Petaramesh <swami@petaramesh.org> http://petaramesh.org PGP 9076E32E

Re: Rebalance makes BTRFS 10x slower

[Martin Steigerwald]

Am Sonntag, 23. März 2014, 15:51:34 schrieben Sie:
> I was expecting either a speed improvement after rebalance, or no
> noticeable  effect, but I am extremely disappointed to see that now (and
> after having rebooted), my system has become slow like hell, takes at least
> 10x longer to boot and operate, to the point it has become hardly usable
> 
> I would have thought a rebalance would have improved the filesystem 
> organization, looks like it's the absolute contrary

I also found this to be the case as I rebalanced the root filesystem of my 
Debian installation on this ThinkPad T520 on an Intel SSD 320.

It doubled the boot time that systemd-analayze rebootet back then.

apt-get dist-upgrade was noticably slower to.

Thus I avoid balance unless I really need it.

For migrating of /home to BTRFS RAID 1 on Intel SSD 320 + Crucial mSATA M500 
SSD I did a balance to switch to RAID 1. Probably /home would be faster 
recreating it from scratch and restoring from backup tough.

Especially MySQL based Akonadi seems to be slower than with /home on a single 
SSD, but that might also be due to that amount of mails in Linux kernel-ml 
folder raising to a high amount (and inefficiencies in Akonadi).

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: Rebalance makes BTRFS 10x slower

[Clemens Eisserer]

Hi,

> I also found this to be the case as I rebalanced the root filesystem of my
> Debian installation on this ThinkPad T520 on an Intel SSD 320.

Same here, I ran balance on kernel-3.12 some time ago and after
balancing performance dropped noticeable and stayed there.
When booting natively I can see the HDD activity led lid for 10+
seconds (Samsung 830 SSD), working with the same system in virtualbox
using direct device access is a real pain.

Non of the measures I took (re-balance, fstrim, defraging all files on
the volume, ...) had mentionable impact.

Regards, Clemens

Re: Rebalance makes BTRFS 10x slower

[Duncan]

Clemens Eisserer posted on Sun, 13 Apr 2014 10:16:22 -0400 as excerpted:

>> I also found this to be the case as I rebalanced the root filesystem of
>> my Debian installation on this ThinkPad T520 on an Intel SSD 320.
> 
> Same here, I ran balance on kernel-3.12 some time ago and after
> balancing performance dropped noticeable and stayed there.
> When booting natively I can see the HDD activity led lid for 10+ seconds
> (Samsung 830 SSD), working with the same system in virtualbox using
> direct device access is a real pain.
> 
> Non of the measures I took (re-balance, fstrim, defraging all files on
> the volume, ...) had mentionable impact.

This one is a real mystery to me, and only recently came up on my radar 
as an issue people are seeing.  It'd be interesting to get to the bottom 
of it as it's possible it has other bug and correctness implications as 
well.

I've seen nothing like it here, but then my btrfs are all relatively 
small (under 50 gig each), and I don't tend to do much snapshotting or 
subvolumes, which might make a difference.  Additionally, I don't have a 
lot of large "internal write" files such as databases and the like (not 
even the systemd journal file issue that some have reported, as while I 
recently switched to systemd, I deliberately chose to configure it to do 
tmpfs logs only to avoid that problem entirely, with all longer term 
logging going thru syslog-ng, which has more conventional append-only log 
files).

But as I said, I'd sure like to get to the bottom of this one, since I do 
believe it has other potential implications in terms of bugs, etc.  In 
theory, a balance should either not affect performance or should improve 
it, so getting to the bottom of why it's having such a bad performance 
impact for many really is something that needs to be done.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Rebalance makes BTRFS 10x slower

[Clemens Eisserer]

Hi Ducan,

> But as I said, I'd sure like to get to the bottom of this one, since I do
> believe it has other potential implications in terms of bugs, etc.  In
> theory, a balance should either not affect performance or should improve
> it, so getting to the bottom of why it's having such a bad performance
> impact for many really is something that needs to be done.

I agree - getting to the bottom of this would be interesting.
The question is how?

Regards, Clemens