* Re: btrfs-cleaner causing heavy load
2017-07-17 8:03 btrfs-cleaner causing heavy load Brennecke, Simon
@ 2017-07-18 6:51 ` Duncan
[not found] ` <d117be0bcbf641fb8f254911d3bf7ef8@sap.com>
0 siblings, 1 reply; 4+ messages in thread
From: Duncan @ 2017-07-18 6:51 UTC (permalink / raw)
To: linux-btrfs
Brennecke, Simon posted on Mon, 17 Jul 2017 08:03:33 +0000 as excerpted:
> I understand that purging snapshots is a complex operation, but we
> somehow need to reduce the load this causes during working hours.
> Are there any ways to tell btrfs-cleaner to suspend or reduce its
> operations?
>
> Background:
> The file-server runs inside a XEN domU The backing disk is a Ceph
RDB
> with 50TiB capacity We employ a bcache with a local SSD to improve
> latency Files are served via NFS and Samba to a couple of hundred
> clients.
>
> Thanks & regards Simon
>
> uname -a
> Linux v2-fs 4.1.42-xen #2 SMP Wed Jul 12 14:06:37 CEST 2017 x86_64
> GNU/Linux
>
> btrfs --version
> Btrfs v3.17
Just another user and list regular here, and one that doesn't use either
ceph or btrfs snapshots so my ability to help will be limited, but some
things to try based on what I've seen go by on the list, the low hanging
fruit, if you will...
* If at all possible and you have it on, try turning off the btrfs quota
machinery when deleting snapshots. Attempting to keep quotas in sync
while deleting snapshots increases the load *tremendously* and it simply
doesn't scale well. Based on list reports, turning them off during
snapshot deletion can be like night and day, result-wise.
If you need quotas you can of course turn them back on when you're done
with the deletes, and it's quite possible that it's the big deletion
backlog that's the problem and deleting just one snapshot at a time with
quotas on may work at least tolerably well once you've caught up with the
backlog.
OTOH, quotas do dramatically increase scaling issues for a number of
btrfs maintenance tasks, and until recently, they were too buggy to be
reliable anyway, so if you don't really need them, keeping them off will
likely improve btrfs responsiveness for you, particularly when doing
snapshot deletion, balances or checks. And as buggy as quotas were in
4.1, I'd suggest either upgrading, or just leaving them off, as they're
just not worth the bother, in 4.1.
* Talking about "recently", on the LTS side this list does try to support
the last two LTS kernel series, but not really beyond that... as is
understandable for a forward focused development list covering a still
stabilizing filesystem such as btrfs. The two currently supported LTS
series are thus now 4.9 and 4.4, with the 4.1 you're running now off in
long ago btrfs development history.
Basically, while there are valid reasons to run kernels that old, in
general they tend to be incompatible with the reasons one may choose to
run a still stabilizing, not fully stable and mature filesystem like
btrfs. So the list recommendation tends to be to choose one or the
other, a more current kernel in keeping with a not yet fully stabilized
btrfs, or a different, fully stable filesystem, in keeping with the
reasons people usually give for choosing to run such old kernels.
Alternatively, some distros support btrfs on older kernels, but in that
case you're better off getting support from them, because theyknow what
changes they've backported and which ones they haven't, and are thus
better positioned to provide support for those kernels and what's running
in them.
Or you can stay with what you have and try to do the best with the
support you can get from this list, and we'll try to do our best too, but
realize there's a serious limit to how much work people are going to be
putting into stuff that old, is all.
Meanwhile, as hinted, quotas are one area that has had dramatic fixes
since 4.1, where they were definitely known to still be buggy. Tho
they'll still kill snapshot deletion performance in current kernels, but
at least you should be getting more reliable numbers with them in current
kernels. So if you use quotas, you _definitely_ want to upgrade...
unless of course your distro has backported those fixes, but then we're
back to them being in a better position to provide support as they know
what they've backported.
As for btrfs-tools, while in normal operation they're not as critical as
the kernel (since in normal operation most of the work is done by the
kernel, with userspace only calling the appropriate kernel
functionality), if there's problems, you want a newer userspace as well,
because then it's the userspace code doing the work. Regardless, 3.17 is
a /very/ long time ago in btrfs terms, and an upgrade to something even
/half/ modern, perhaps a 4.4 userspace to match a 4.4 LTS kernel upgrade,
should be considered.
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 4+ messages in thread