erasure code software and papers and swift

erasure code software and papers and swift

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 1871 bytes --]

Hi Ceph,

For the record, here are a few erasure code related links (thanks Kevin :-), 

   https://bitbucket.org/tsg-/liberasurecode/
   liberasurecode is an Erasure Code API library written in C with pluggable Erasure Code backends.

the first commit is dated back april 2014 but its content suggests development happened elsewhere before that. 

It is the equivalent of a standalone 

    https://github.com/ceph/ceph/tree/master/src/erasure-code

Only with dynamically loaded jerasure/isa instead of an embedded copy. The idea of an external general purpose erasure code library is appealing. It would lower the maintenance effort, among other things. It's nice to see that Ceph is listed as a contributor in the readme. There is no commit to date in the liberasurecode library itself but that can be fixed ;-)

liberasurecode is already used as a dependency of https://bitbucket.org/kmgreen2/pyeclib , the library swift depends on for erasure coded storage policies.

And some interesting papers:

    Simple Regenerating Codes: http://arxiv.org/pdf/1109.0264v1.pdf

    LT Codes: http://en.wikipedia.org/wiki/Luby_transform_code

    Rateless codes and downloads: http://www.iptps.org/papers-2003/rateless_codes.pdf

Yesterday I attended Paul Luse presentation of the erasure code implementation in swift and it was enlightening. Very much like Samuel Just did in Ceph when he introduced the Placement Groups backends, Paul worked on Swift Storage Policies to make room for erasure coded "rings" (pools in the swift parlance, if I'm not mistaken).

http://docs-draft.openstack.org/90/125190/8/gate/gate-swift-specs-docs/2d2cf67/doc/build/html/specs/swift/erasure_coding.html

It looks like erasure code will be available in swift soon and the hard part of the work is complete.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]