Deduplication

Deduplication

[Florian Lindner]

Hello,

some questions regarding btrfs deduplication.

- What is the state of it? Is it "safe" to use?
https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield
much information.

- https://pypi.python.org/pypi/bedup says: "bedup looks for new and
changed files, making sure that multiple copies of identical files
share space on disk. It integrates deeply with btrfs so that scans are
incremental and low-impact."

Is is file-based deduplcation only or is there also a block-based
mode? Does not seem so according to the docs.

If so, what is the advantage to numerous other tools that replace
duplicates by hard links? Performance?

Thanks,

Florian

Re: Deduplication

[Liu Bo]

Hi,

On Thu, Aug 22, 2013 at 09:28:09PM +0200, Florian Lindner wrote:
> Hello,
> 
> some questions regarding btrfs deduplication.
> 
> - What is the state of it? Is it "safe" to use?
> https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield
> much information.

For inband dedup, it's experimental because it introduces some format changes.

> 
> - https://pypi.python.org/pypi/bedup says: "bedup looks for new and
> changed files, making sure that multiple copies of identical files
> share space on disk. It integrates deeply with btrfs so that scans are
> incremental and low-impact."
> 
> Is is file-based deduplcation only or is there also a block-based
> mode? Does not seem so according to the docs.

Both inband dedup and offband dedup patchset are floating on the list, and
inband is block-based, but I'm not sure about offband dedup.

-liubo

> 
> If so, what is the advantage to numerous other tools that replace
> duplicates by hard links? Performance?
> 
> Thanks,
> 
> Florian
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Deduplication

[mpe]

On 23/08/2013 09:42, Liu Bo wrote:
> Hi,
>
> On Thu, Aug 22, 2013 at 09:28:09PM +0200, Florian Lindner wrote:
>> Hello,
>>
>> some questions regarding btrfs deduplication.
>>
>> - What is the state of it? Is it "safe" to use?
>> https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield
>> much information.
>
> For inband dedup, it's experimental because it introduces some format changes.
>

In our tests we are trying btrfs for our internal storage backup backend 
and we like a lot compression! Now would like to test inband dedup.
Does those patchs (are v6 the last?) can be used on ubuntu lts? Do have 
a simple howt-to to follow?

Thanks,
Michele

Re: Deduplication

[Liu Bo]

On Fri, Aug 23, 2013 at 12:04:10PM +0200, mpe wrote:
> On 23/08/2013 09:42, Liu Bo wrote:
> >Hi,
> >
> >On Thu, Aug 22, 2013 at 09:28:09PM +0200, Florian Lindner wrote:
> >>Hello,
> >>
> >>some questions regarding btrfs deduplication.
> >>
> >>- What is the state of it? Is it "safe" to use?
> >>https://btrfs.wiki.kernel.org/index.php/Deduplication does not yield
> >>much information.
> >
> >For inband dedup, it's experimental because it introduces some format changes.
> >
> 
> In our tests we are trying btrfs for our internal storage backup
> backend and we like a lot compression! Now would like to test inband
> dedup.

dedup can cope with compression, too.

> Does those patchs (are v6 the last?) can be used on ubuntu lts? Do
> have a simple howt-to to follow?

I didn't try ubuntu lts yet, but the v6 patch set can be well applied on
3.11-rc4.

To enable dedup, you need to
1) mount DEV /PATH_OF_MNT -o dedup (or -o dedup_bs=xxxK)
2) btrfs dedup register /PATH_OF_MNT
3) write your data now :)


Feel free to tell me if any questions, thanks.

-liubo