Radosgw-agent data full sync implementation

Radosgw-agent data full sync implementation

[christophe courtaut]

Hi Joe,

I'm currently looking at the radosgw-agent, and i have seen your
recent job on full sync of data.

What will be the preferred way to list all objects to sync? Using
admin API or standard S3 API?

Moreover, we will need to do an exact copy of the object to preserve
attributes, as mentionned here
http://pad.ceph.com/p/rgw-multi-region-dr .

Will this be part of the admin API? or am i missing something?

Btw i recently posted an article here : http://blog.kri5.fr/?p=21, to
setup two cluster for testing purpose of the radosgw-agent.

Best regards.

-- 
Christophe Courtaut

Re: Radosgw-agent data full sync implementation

[Joe Buck]

On 08/28/2013 09:21 AM, christophe courtaut wrote:

> Hi Joe,
>
> I'm currently looking at the radosgw-agent, and i have seen your
> recent job on full sync of data.
>
> What will be the preferred way to list all objects to sync? Using
> admin API or standard S3 API?

It looks like we'll use the S3 API to list the objects in a bucket
(we'll actually use the boto library to do this).
It's worth noting that there's some locking that we need to do to
account for multiple agents running at the same time, so it's not quite
as easy as "list objects, sync objects, call it a day".

> Moreover, we will need to do an exact copy of the object to preserve
> attributes, as mentionned here
> http://pad.ceph.com/p/rgw-multi-region-dr  .
>
> Will this be part of the admin API? or am i missing something?

The following link on the wiki has most (all?) of the calls that are
being added for the sync work. There's an extended object GET and PUT in
there, towards the top.
http://wiki.ceph.com/index.php?title=RESTful_API_for_DR_%2F%2F_Geo-Replication
This link is also handy in terms of an overview:
http://www.spinics.net/lists/ceph-devel/msg11905.html

> Btw i recently posted an article here :http://blog.kri5.fr/?p=21, to
> setup two cluster for testing purpose of the radosgw-agent.

Cool.

> Best regards.
>