From: Mark Fasheh <mfasheh@suse.de>
To: Martin <m_btrfs@ml1.co.uk>
Cc: linux-btrfs@vger.kernel.org
Subject: Re: btrfs dedup - available or experimental? Or yet to be?
Date: Fri, 27 Mar 2015 13:51:55 -0700 [thread overview]
Message-ID: <20150327205155.GF17170@wotan.suse.de> (raw)
In-Reply-To: <mf2701$d5k$1@ger.gmane.org>
On Fri, Mar 27, 2015 at 12:07:29AM +0000, Martin wrote:
> Excellent and very rapid packaging, thanks!
>
>
> Already compiled, installed, and soon to be tried on a test subvolume...
>
>
> Anyone with any comments on how well duperemove performs for TB-sized
> volumes?
https://github.com/markfasheh/duperemove/wiki/Performance-Numbers
That page has some sample performance numbers. Keep in mind that the tests
were done on reasonably nice hardware.
TB-size is definitely on the larger end of what I expect it should handling
these days. The biggest problem you would see is memory usage - versions
0.09 and below will be storing all hashes in memory so if everything else is
fast enough that's likely the first bump you'll hit.
Master branch has some code which reduces our memory consumption
dramatically by using a bloom filter and temporarily storing them on disk.
That branch needs some more features and bug fixing before I'm ready to call
it stable.
> Does it work across subvolumes? (Presumably not...)
Yep it will dedupe across subvolumes for you!
--Mark
--
Mark Fasheh
next prev parent reply other threads:[~2015-03-27 20:51 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-23 23:10 btrfs dedup - available or experimental? Or yet to be? Martin
2015-03-23 23:22 ` Hugo Mills
2015-03-25 1:30 ` Rich Freeman
2015-03-27 0:07 ` Martin
2015-03-27 0:30 ` Rich Freeman
2015-03-29 11:43 ` Kai Krakow
2015-03-29 12:31 ` Rich Freeman
2015-03-29 14:44 ` Kai Krakow
2015-03-29 17:54 ` Christoph Anton Mitterer
2015-03-29 17:51 ` Christoph Anton Mitterer
2015-03-27 20:51 ` Mark Fasheh [this message]
2015-03-27 20:44 ` Mark Fasheh
2015-05-13 16:23 ` Learner Study
2015-05-13 21:08 ` Zygo Blaxell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150327205155.GF17170@wotan.suse.de \
--to=mfasheh@suse.de \
--cc=linux-btrfs@vger.kernel.org \
--cc=m_btrfs@ml1.co.uk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).