Hardware Requirements for RADOS Gateway Cluster

Hardware Requirements for RADOS Gateway Cluster

[Brice Burgess]

I am planning the system architecture around a ceph cluster [my first!]. 
While resources for setting up the basic cluster (MON+MDS+OSD) are 
readily available*, I haven't come across anything outlining production 
recommendations for a RADOS Gateway Cluster. Specifically;

1. Is it preferable to run the RADOS Gateway on a MDS machine [for 
latency issues], or should the Gateway run on seperate hosts/VMs than 
the ODS/MON/MDS servers?

2. I've seen reference that multiple RADOS Gateway servers can be setup 
as a cluster "to scale". Is this possible? Is there an 
example/documentation for this? Is it just 2 standalone gateways with a 
load balancer in front?

My assumption is to provision a dedicated machine for the RADOS Gateway. 
I'd treat this machine as a "front end proxy/caching server" meaning it 
would have a lot of RAM for varnish/nginx and a low latency, high 
throughput network connection to the ODS/MDS machines as well as one to 
the public network.

Any thoughts and suggestions are welcome. Eventually I'd like to release 
a whitepaper of this setup if we can hit the budget to actually implement ;)

~ Brice

Re: Hardware Requirements for RADOS Gateway Cluster

[Mark Nelson]

Hi Brice!

On 09/24/2012 05:58 PM, Brice Burgess wrote:
>I am planning the system architecture around a ceph cluster [my first!].
> While resources for setting up the basic cluster (MON+MDS+OSD) are
> readily available*, I haven't come across anything outlining production
> recommendations for a RADOS Gateway Cluster. Specifically;
>
> 1. Is it preferable to run the RADOS Gateway on a MDS machine [for
> latency issues], or should the Gateway run on seperate hosts/VMs than
> the ODS/MON/MDS servers?
>

It's entirely possible to run the gateway on an OSD/MON/MDS.  I don't 
think we've done any kind of extensive analysis of where the best place 
to run one is.  My guess is that it will only matter in very specific cases.

> 2. I've seen reference that multiple RADOS Gateway servers can be setup
> as a cluster "to scale". Is this possible? Is there an
> example/documentation for this? Is it just 2 standalone gateways with a
> load balancer in front?
>

Yes!  It's entirely possible to run multiple RGWs with a load balancer 
in front.  I'm not sure how much documentation is out there for this 
yet, but it's something that customers of ours have successfully 
implemented.

> My assumption is to provision a dedicated machine for the RADOS Gateway.
> I'd treat this machine as a "front end proxy/caching server" meaning it
> would have a lot of RAM for varnish/nginx and a low latency, high
> throughput network connection to the ODS/MDS machines as well as one to
> the public network.

Sounds good to me.  Other people might chime in with more suggestions.

>
> Any thoughts and suggestions are welcome. Eventually I'd like to release
> a whitepaper of this setup if we can hit the budget to actually
> implement ;)

Some things you'll want to keep in mind:

You'll want to make sure that your testing (especially of small IO) is 
done across many buckets.  Performance of a single bucket can be limited 
by the OSD where it's log is stored (or optionally you can turn the log 
off).  You'll also want to keep in mind that RGW will likely incur 
larger performance overhead for small objects than large ones.

Thanks,
Mark

Re: Hardware Requirements for RADOS Gateway Cluster

[Tommi Virtanen]

On Mon, Sep 24, 2012 at 3:58 PM, Brice Burgess <briceburg@gmail.com> wrote:
> 1. Is it preferable to run the RADOS Gateway on a MDS machine [for latency
...
> My assumption is to provision a dedicated machine for the RADOS Gateway. I'd
> treat this machine as a "front end proxy/caching server" meaning it would
> have a lot of RAM for varnish/nginx and a low latency, high throughput
> network connection to the ODS/MDS machines as well as one to the public
> network.

Just to make sure you realize this: radosgw does not use MDS in any way.