Defragmentation vw. Deduplication

Defragmentation vw. Deduplication

[Christian Völker]

Hi all,

might be a simple question but I did not find a trustable source for this.

BTRFS uses COW which might lead to fragmentation.
So using "btrfs fi defrag -r /mnt" will bring most file extend in a row 
and copy previously deduplicated extends.
Obviously this uses more disk space. This is not what I want, but I need 
to run "defrag" because I initially skipped the "compress=zstd" option 
when mounting. So many files are stored without compression. Therefor I 
neede to do the "defrag".

I am now unsure about the deduplication itself.  How does it work?
I create a file in a directory (ie on Monday). Some days later I create 
a file which has some extents with equal data. Does btrfs recon the 
equal extents and does it doe deduplication then? Or does it only do 
deduplication when ie "cp --reflink" is used?

However as I needed the compression and not the defragmentation is there 
any way to add compression and recreate deduplication later?

Sorry if this is a dumb question.

/KNEBB

Re: Defragmentation vw. Deduplication

[Zygo Blaxell]

On Tue, Mar 02, 2021 at 10:31:08PM +0100, Christian Völker wrote:
> Hi all,
> 
> might be a simple question but I did not find a trustable source for this.
> 
> BTRFS uses COW which might lead to fragmentation.
> So using "btrfs fi defrag -r /mnt" will bring most file extend in a row and
> copy previously deduplicated extends.
> Obviously this uses more disk space. This is not what I want, but I need to
> run "defrag" because I initially skipped the "compress=zstd" option when
> mounting. So many files are stored without compression. Therefor I neede to
> do the "defrag".
> 
> I am now unsure about the deduplication itself.  How does it work?
> I create a file in a directory (ie on Monday). Some days later I create a
> file which has some extents with equal data. Does btrfs recon the equal
> extents and does it doe deduplication then? 

You have to run a deduper like duperemove or bees to get that behavior.  See

	https://btrfs.wiki.kernel.org/index.php/Deduplication

There are a number of different tools that are also available for other
filesystems as well, e.g. jdupes, which dedupe entire identical files
as opposed to blocks.  If most of your duplicates are entire files then
this can significantly increase dedupe speed and reduce dedupe memory
compared to the block-oriented dedupers.

> Or does it only do deduplication when ie "cp --reflink" is used?

With cp --reflink there is no duplication in the first place.

You can choose a deduper depending on how much RAM and IO you want to
commit to the task, whether you need block- or file-oriented dedupe, etc.

If you don't run a deduper then duplicate data stays duplicated.
If you don't run defrag then reflinked data stays reflinked.

> However as I needed the compression and not the defragmentation is there any
> way to add compression and recreate deduplication later?

You can simply run a deduper after compression is done.

> Sorry if this is a dumb question.
> 
> /KNEBB
> 
>