Re: qgroup code slowing down rebalance

[Qu Wenruo <quwenruo@cn.fujitsu.com>]

Dave Hansen wrote on 2016/03/18 09:33 -0700:
> On 03/17/2016 06:02 PM, Qu Wenruo wrote:
>> Dave Hansen wrote on 2016/03/17 09:36 -0700:
>>> On 03/16/2016 06:36 PM, Qu Wenruo wrote:
>>>> Dave Hansen wrote on 2016/03/16 13:53 -0700:
>>>>> I have a medium-sized multi-device btrfs filesystem (4 disks, 16TB
>>>>> total) running under 4.5.0-rc5.  I recently added a disk and needed to
>>>>> rebalance.  I started a rebalance operation three days ago.  It was on
>>>>> the order of 20% done after those three days. :)
>>> ...
>>> Data, RAID1: total=4.53TiB, used=4.53TiB
>>> System, RAID1: total=32.00MiB, used=720.00KiB
>>> Metadata, RAID1: total=17.00GiB, used=15.77GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>> Considering the size and the amount of metadata, even doing a quota
>> rescan will be quite slowing.
>>
>> Would you please try to do a quota rescan and see the CPU/IO usage?
>
> I did a quota rescan.  It uses about 80% of one CPU core, but also has
> some I/O wait time and pulls 1-20MB/s of data off the disk (the balance
> with quotas on was completely CPU-bound, and had very low I/O rates).
>
> It would seem that the "quota rescan" *does* have the same issue as the
> balance with quotas on, but to a much smaller extent than what I saw
> with the "balance" operation.

This is quite expected. Most CPU would be consumed by find_all_roots(), 
as your subvolume layout is the just the worst case for find_all_roots().

That's to say, remove unneeded snapshots and keep them under a limited 
amount would be the best case.
Btrfs snapshot is superfast to create, but the design implies a lot of 
overhead for some minor operation, especially for backref lookup.
And unfortunately, quota relies heavily on it.


The only difference that I doesn't expect is IO. As I expected rescan 
would be much the same with balance, with little IO.
Did you run rescan/balance after dropping all the caches?


Although for balance, I would add some patch to make them by-pass quota 
accounting as them seems to be OK.
But for rescan case, AFAIK that's the case.


BTW although it's quite risky, I hope you can run some old kernels and 
see the performance difference of rescan and balance.
For old kernels, I mean 4.1 which is the latest kernel that doesn't use 
the new quota framework.

I'm very interesting to see if the old but wrong code would have a 
better or worse performance on balance or rescan.

Thank you very much for all this quota related reports.
Qu


>
> I have a full profile recorded from the "quota rescan", but the most
> relevant parts are pasted below.  Basically btrfs_search_slot() and
> radix tree lookups are eating all the CPU time, but they're still doing
> enough I/O to see _some_ idle time on the processor.
>
>>      74.55%     3.10%  kworker/u8:0     [btrfs]                      [k] find_parent_nodes
>>                 |
>>                 ---find_parent_nodes
>>                    |
>>                    |--99.95%-- __btrfs_find_all_roots
>>                    |          btrfs_find_all_roots
>>                    |          btrfs_qgroup_rescan_worker
>>                    |          normal_work_helper
>>                    |          btrfs_qgroup_rescan_helper
>>                    |          process_one_work
>>                    |          worker_thread
>>                    |          kthread
>>                    |          ret_from_fork
>>                     --0.05%-- [...]
>>
>>      32.14%     4.16%  kworker/u8:0     [btrfs]                      [k] btrfs_search_slot
>>                 |
>>                 ---btrfs_search_slot
>>                    |
>>                    |--87.90%-- find_parent_nodes
>>                    |          __btrfs_find_all_roots
>>                    |          btrfs_find_all_roots
>>                    |          btrfs_qgroup_rescan_worker
>>                    |          normal_work_helper
>>                    |          btrfs_qgroup_rescan_helper
>>                    |          process_one_work
>>                    |          worker_thread
>>                    |          kthread
>>                    |          ret_from_fork
>>                    |
>>                    |--11.70%-- btrfs_search_old_slot
>>                    |          __resolve_indirect_refs
>>                    |          find_parent_nodes
>>                    |          __btrfs_find_all_roots
>>                    |          btrfs_find_all_roots
>>                    |          btrfs_qgroup_rescan_worker
>>                    |          normal_work_helper
>>                    |          btrfs_qgroup_rescan_helper
>>                    |          process_one_work
>>                    |          worker_thread
>>                    |          kthread
>>                    |          ret_from_fork
>>                     --0.39%-- [...]
>>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>