* Re: Kickstarting snapshot-aware defrag?
2017-10-03 13:32 Kickstarting snapshot-aware defrag? Niccolò Belli
@ 2017-10-04 7:01 ` Duncan
0 siblings, 0 replies; 2+ messages in thread
From: Duncan @ 2017-10-04 7:01 UTC (permalink / raw)
To: linux-btrfs
Niccolò Belli posted on Tue, 03 Oct 2017 15:32:22 +0200 as excerpted:
> Hi,
> It seems to me that the proposal[1] for a snapshot-aware defrag has long
> been abandoned. Since most peoples badly need this feature I tought
> about how to possibly speed up the achievement of this goal.
AFAIK it hasn't been /abandoned/, tho agreed it could seem that way on
the surface. Rather, it was tried, and it simply didn't scale, making
defrag slow to effective impracticality (a snapshot-aware defrag that
takes a month might as well not exist at all), because many of the
operations it had to do behind the scenes simply didn't scale.
So the first attempt was rolled back so people could at least defrag in
reasonable time again, even if it was no longer snapshot-aware.
Now before snapshot-aware defrag can work reasonably, all those various
scaling issues have to be addressed. That's happening, but it's not
something that can be done, at least not done /right/, in a day.
Now I'm not a dev, but if I've been following the patches and discussion
properly, perhaps the biggest scaling issue has been the lack of an
ability to reverse-trace something in particular -- the lists work one
way and that's fast, but reversing things means going thru the entire
list looking for the item(s) of interest... and snapshot-aware-defrag
isn't the only thing with the problem, just the easiest to disable as a
workaround. The same root problem is the reason TB-scale btrfs check
takes so much memory in normal mode, or so long in low-memory-mode, and
it's the reason behind the performance issues with qgroup-enabled
balance, thus the recommendation to disable qgroups if your balance is
taking unreasonably long, etc.
Without getting too far into stuff I'm not entirely sure about as a non-
dev, I believe a solution is known in general, but it's only recently
being looked at for implementation, because there were other more
pressing things that kept coming up, stuff that ended up needing fixed
first, etc. And this /might/ be one of the things that's going to take
an on-device format change as well, that they're saving up in a list so
they can do them all together, for one more major change before the final
push to official btrfs stability, something I've seen mentioned in other
contexts, tho I'm not positive that it applies to this particular issue.
So bottom line, they're working on it, but it's not something that's
going to be done in a day, or even a year, as time has demonstrated.
It's /not/ abandoned, and sure, it could be turned back on with little
effort in a kernel release cycle or two if they wanted, but until the
scaling issues are fixed, that wouldn't help, because while some of the
issues have been fixed and it might take a week to do an hour's work now
instead of a month, or even "only" a day now, that's not going to change
the fact that it's not going to be particularly practical, and a non-
snapshot-aware defrag that actually works in reasonable time remains far
more practical than a snapshot-aware defrag that takes that long.
> I know of several bounty-based kickstarting platforms, among them the
> best ones are probably bountysource.com[2] and freedomsponsors.org[3].
> With both platforms everyone interested can place a bounty on the issue
> and if/when there will be someone interested to implement it he will get
> the bounty.
> I created an issue on both of them just to show how the platform will
> handle it.
>
> Since btrfs is a small community, before actually placing bounties and
> sponsoring it I would like to know if there is someone against this
> development model or someone interested in implementing a feature
> because of a bounty.
Glad you asked first, because...
Honestly, this is going to go over like a lead balloon. Similar things
have been tried before, and invariably, they fail, because it horribly
clashes with existing kernel culture and practices, and because it
incentivizes the wrong thing.
What happens both with cash bounties, and with similar effect non-cash
bounties such as (bare) "get a patch accepted into the kernel"
requirements for development courses or resume puffery, is that it
clashes with the general kernel developer long view -- the patch series
may end up going thru 20 revisions over the course of a year or more,
fixing complaints from first one subsystem maintainer and then another,
only to finally be rejected in favor of someone else's different approach
because it's more maintainable, or works better with something else to be
built on top, or ... . Or elements of both approaches (or five or six
approaches) may ultimately be combined, and the person who first proposed
a feature may not be the one that gets final authorship even if they
cooperated with the person and approach that did.
Someone interested in a bounty simply isn't going to stick around for all
that.
Additionally, there's a huge interest in developing kernel devs that will
stick around, both to help maintain code they've already submitted, and
to continue to work on other things. Bounty-type solutions encourage
"one-shot-wonders" that disappear and are never seen or heard from
again. While that might be fine for a one-line or even 10-line fix, and
there's normally many such one-shot-fixes in every kernel, those are
trivial fixes that don't normally need a bounty, while the larger
features where a bounty might appear to be useful, really need someone
that's going to stick around to maintain it. But that's not what
bounties encourage!
So a bounty simply isn't useful for kernel work, and proposing such
things, as I said, goes over like a lead balloon, because the experience
with such things has been **BAD**.
OTOH, there's always room for individuals who are using the code
themselves, want to make it better, and take a longer term interest in
learning its ins and outs, and starting small, I've seen suggestions here
to start with the on-disk format and with btrfs-progs, since that doesn't
involve hard kernel issues such as machine-locking lock-inversions, or
data corrupting missing-lock concurrency issues. Similarly for companies
willing to sponsor devs to take the time to come up to speed and make
meaningful contributions, even if it's going to take several man-months
of work before there's going to be much to show for it, and even if those
"meaningful contributions" end up coming in under someone else's
authorship, because that turned out to be the better longer-term solution
to the problem, tho there /was/ significant contribution to the finally
accepted solution, just not on the authorship line.
But again, that's pretty much the utter antithesis of bounties.
> [1]https://www.spinics.net/lists/linux-btrfs/msg34539.html
> [2]https://www.bountysource.com/issues/50004702-feature-request-
snapshot-aware-defrag
> [3]https://freedomsponsors.org/issue/817/feature-request-snapshot-aware-
defrag?alert=KICKSTART
--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman
^ permalink raw reply [flat|nested] 2+ messages in thread