PowerDNS backend for Rados Gateway to direct bucket traffic

PowerDNS backend for Rados Gateway to direct bucket traffic

[Wido den Hollander]

Hi,

I'm playing with an idea, but I'm not sure if I'm taking the correct route.

When you create a bucket with Amazon S3 it's always reachable via 
bucketname.s3.amazonaws.com.

Depending on the region you created it in you'll get a different 
response from their DNS servers directing you to the correct HTTP endpoint.

When you use the RGW with multiple regions you get a 301 redirect from a 
region if the bucket is not located there.

This 301 is extra TCP traffic and adds extra latency. You could 
potentially have a developer who is mistaken and directs thousands of 
requests to the wrong region which are all answered with 301 redirects. 
A waste of resources.

I've worked with PowerDNS a lot and it supports various backends and one 
of them is the Pipe or HTTP (new!) backend [0]

It would be rather cool if you could have a global domainname like 
"s3.amazonaws.com", but behind it you have multiple regions, but based 
on the DNS request you do PowerDNS responds with a different CNAME.

global: o.myobject.store
EU:     eu.myobject.store
US:     us.myobject.store

Based on where the bucket is PowerDNS will respond with:

CNAME bucket.eu.myobject.store
or
CNAME bucket.us.myobject.store

Does it sound reasonable what I'm thinking of or am I completely wrong 
about this?

[0]: http://doc.powerdns.com/html/remotebackend.html

-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: PowerDNS backend for Rados Gateway to direct bucket traffic

[Kyle Bader]

> Does it sound reasonable what I'm thinking of or am I completely wrong about
> this?
>
> [0]: http://doc.powerdns.com/html/remotebackend.html

I was considering exactly what your describe, not being a programmer
though it was simply an idea. I could see myself using a powerdns
backend for bucket mapping, all other ways I've considered seem very
brittle. Let me know if you write it, I'd be willing to test it out.

-- 

Kyle

Re: PowerDNS backend for Rados Gateway to direct bucket traffic

[Sage Weil]

I think this is a good path forward, FWIW.  Want to write up a blueprint, 
Wido?  :)

sage


On Thu, 7 Nov 2013, Kyle Bader wrote:

> > Does it sound reasonable what I'm thinking of or am I completely wrong about
> > this?
> >
> > [0]: http://doc.powerdns.com/html/remotebackend.html
> 
> I was considering exactly what your describe, not being a programmer
> though it was simply an idea. I could see myself using a powerdns
> backend for bucket mapping, all other ways I've considered seem very
> brittle. Let me know if you write it, I'd be willing to test it out.
> 
> -- 
> 
> Kyle
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: PowerDNS backend for Rados Gateway to direct bucket traffic

[Wido den Hollander]

On 11/08/2013 05:37 AM, Sage Weil wrote:
> I think this is a good path forward, FWIW.  Want to write up a blueprint,
> Wido?  :)
>

Done! 
http://wiki.ceph.com/01Planning/02Blueprints/Firefly/PowerDNS_backend_for_RGW

This really shouldn't be that much work. I'll try to write a small POC 
in the coming weeks.

Wido

> sage
>
>
> On Thu, 7 Nov 2013, Kyle Bader wrote:
>
>>> Does it sound reasonable what I'm thinking of or am I completely wrong about
>>> this?
>>>
>>> [0]: http://doc.powerdns.com/html/remotebackend.html
>>
>> I was considering exactly what your describe, not being a programmer
>> though it was simply an idea. I could see myself using a powerdns
>> backend for bucket mapping, all other ways I've considered seem very
>> brittle. Let me know if you write it, I'd be willing to test it out.
>>
>> --
>>
>> Kyle
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on