OSD-Based Object Stubs

[Marcel Lauhoff <ml@irq0.org>]

Hi,

I wrote a prototype for an OSD-based object stub feature. An object stub
being an object with it's data moved /elsewhere/. I hope to get some
feedback, especially whether I'm on the right path here and if it
is a feature you are interested in.



Code is in my "osd-stubs" branch:
 https://github.com/ceph/ceph/compare/master...irq0:osd-stubs
 https://github.com/irq0/ceph/tree/osd-stubs

 Tools to toy around with osd-stubs + web server to send stubs to:
 https://github.com/irq0/ceph_osd-stub_tools



Related:
- https://wiki.ceph.com/Planning/Blueprints/%3CSIDEBOARD%3E/osd:_tiering:_object_redirects



Implementation:

Adds two new OSD OPs:
- STUB :: Move data away; Save location in xattr; Set
          object_info_t::stub_state to 'remote'
- UNSTUB :: Get data back; Remove xattr; Set object_info_t::stub_state
            to 'local'


STUB is meant to be called by an external archive agent. UNSTUB
implicitly when OPs come in that need the object's data. Operations are
classified as "may_use_obj_data" similar to how op->may_{write,read,cache} work.

The implicit UNSUB is implemented by prepending an UNSTUB operation to the
incoming OP list if "may_use_obj_data() and stub_state == REMOTE".
This sadly causes a waring in the client saying that
the reply doesn't match the request. He is of course right, but I found
it to be the simplest way to try that feature.

External storage in the prototype is just simple HTTP: PUT on STUB;
GET+DELETE on UNSTUB.

The Operations implement a kind of converter: STUB reads from the
primary OSD, store the object on the remote and then issue TRUNCATE(0)
and SETXATTR OPs. Similar UNSTUB retrieves the object, then does
WRITEFULL and RMXATTR.


~marcel

--
Marcel Lauhoff
Mail/XMPP: ml@irq0.org
http://irq0.org