Re: Started developing a deduplication feature

[Marcel Lauhoff <lauhoff@uni-mainz.de>]

Hi list,
short recap of the dedup topic from the CDM on Wednesday:


The main change from the original mail is not to add a PG backend, but
rather use Object Redirects (Tiering v2).
Another backend would have to implement its own replication for
recipes and increase the OSD code base just for dedup. Redirects are
useful beyond deduplication.


The CAS pool design was refined: An object class should handle the ref
counting and content addressing. The pool should also only
allow access through this object class to prevent collisions with
regular objects and support immutable objects.


There was also the idea of client-side deduplication by using metadata
that clients like RGW store. This would save the additional round trip
that object redirects add.


I'll be working on the CAS pool first, since there
is ongoing refactoring in the ReplicatedPG code base. I'll work out a
more detailed design document for the CAS pool soon.


~irq0
--
Marcel Lauhoff
Mail: lauhoff@uni-mainz.de
XMPP: mlauhoff@jabber.uni-mainz.de