From: Dave Hansen <dave@sr71.net>
To: Qu Wenruo <quwenruo@cn.fujitsu.com>,
linux-btrfs@vger.kernel.org, Chris Mason <clm@fb.com>
Subject: Re: qgroup code slowing down rebalance
Date: Thu, 17 Mar 2016 09:36:46 -0700 [thread overview]
Message-ID: <56EADD1E.3040202@sr71.net> (raw)
In-Reply-To: <56EA0A0F.8070008@cn.fujitsu.com>
On 03/16/2016 06:36 PM, Qu Wenruo wrote:
> Dave Hansen wrote on 2016/03/16 13:53 -0700:
>> I have a medium-sized multi-device btrfs filesystem (4 disks, 16TB
>> total) running under 4.5.0-rc5. I recently added a disk and needed to
>> rebalance. I started a rebalance operation three days ago. It was on
>> the order of 20% done after those three days. :)
...
> Perf record profile will help a lot.
> Please upload it if it's OK for you.
I'll send it privately.
But, I do see basically the same behavior when balancing data and
metadata. I profiled both.
> Also, the following data would help a lot:
> 1) btrfs fi df output
> To determine the metadata/data ratio
> Balancing metadata should be quite slow.
Data, RAID1: total=4.53TiB, used=4.53TiB
System, RAID1: total=32.00MiB, used=720.00KiB
Metadata, RAID1: total=17.00GiB, used=15.77GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
> 2) btrfs subvolume list output
> To determine how many tree blocks are shared against each other
> More shared tree blocks, slower quota routing is.
>
> Feel free to mask the output to avoid information leaking.
Here you go (pasted at the end). You can pretty clearly see that I'm
using this volume for incremental backups. My plan was to keep the old
snapshots around until the filesystem fills up.
> 3) perf record for balancing metadata and data respectively
> Although this is optional. Just to prove some assumption.
>
> btrfs_find_all_roots() is quite a slow operation, and in its worst case
> (tree blocks are shared by a lot of trees) it may be near O(2^n).
Yikes! But, that does look to be consistent with what I'm seeing.
>> ID 15183 gen 207620 top level 5 path blackbird_backups.enc
>> ID 15547 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455779116
>> ID 15548 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455779185
>> ID 15559 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455814382
>> ID 15560 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455814540
>> ID 15561 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455814739
>> ID 15562 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455814783
>> ID 15563 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455815180
>> ID 15564 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455815259
>> ID 15565 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455815314
>> ID 15566 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455815948
>> ID 15567 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816104
>> ID 15568 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816127
>> ID 15569 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816213
>> ID 15570 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816305
>> ID 15571 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816591
>> ID 15572 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816622
>> ID 15573 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816632
>> ID 15574 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816638
>> ID 15575 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1455816837
>> ID 15576 gen 207620 top level 5 path home-backup-on-btrfs.enc
>> ID 16634 gen 207620 top level 5 path homes-backup-nimitz-o2.enc
>> ID 17389 gen 207620 top level 5 path snapshots/root/1456271568
>> ID 17390 gen 207620 top level 5 path snapshots/root/1456272221
>> ID 17391 gen 207620 top level 5 path snapshots/root/1456416369
>> ID 17392 gen 207620 top level 5 path snapshots/root/1456435609
>> ID 17393 gen 207620 top level 5 path snapshots/root/1456539873
>> ID 17394 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456553298
>> ID 17395 gen 207620 top level 5 path snapshots/root/1456553448
>> ID 17396 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456553460
>> ID 17397 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456553461
>> ID 17398 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456602362
>> ID 17399 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456602405
>> ID 17400 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456602405
>> ID 17401 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456628682
>> ID 17402 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456628682
>> ID 17403 gen 207620 top level 5 path snapshots/root/1456628682
>> ID 17404 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456675846
>> ID 17405 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456675846
>> ID 17406 gen 207620 top level 5 path snapshots/root/1456675846
>> ID 17407 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456713915
>> ID 17408 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456720748
>> ID 17409 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456722793
>> ID 17410 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456722812
>> ID 17411 gen 207620 top level 5 path snapshots/root/1456722812
>> ID 17412 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456722812
>> ID 17413 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456731271
>> ID 17414 gen 207620 top level 5 path snapshots/root/1456731271
>> ID 17415 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456731271
>> ID 17416 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1456838946
>> ID 17417 gen 207620 top level 5 path snapshots/root/1456838946
>> ID 17418 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1456838946
>> ID 17419 gen 207620 top level 5 path snapshots/root/1457127404
>> ID 17420 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1457127404
>> ID 17421 gen 207620 top level 5 path snapshots/home-backup-on-btrfs.enc/1457127404
>> ID 17422 gen 207620 top level 5 path snapshots/root/1457138948
>> ID 17423 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1457138948
>> ID 17425 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1457451483
>> ID 17426 gen 207620 top level 5 path snapshots/blackbird_backups.enc/1457808380
next prev parent reply other threads:[~2016-03-17 16:36 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-03-16 20:53 qgroup code slowing down rebalance Dave Hansen
2016-03-17 1:36 ` Qu Wenruo
2016-03-17 16:36 ` Dave Hansen [this message]
2016-03-18 1:02 ` Qu Wenruo
2016-03-18 16:33 ` Dave Hansen
2016-03-21 1:44 ` Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56EADD1E.3040202@sr71.net \
--to=dave@sr71.net \
--cc=clm@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=quwenruo@cn.fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).