Finding out via librados if a cluster is near full

Finding out via librados if a cluster is near full

[Wido den Hollander]

Hi,

Is there a way to find out if a cluster is near full via librados?

The reason I'm asking is that I'm working on a deployment of the RADOS 
Gateway and one of the potential problems I see is that the RADOS 
Gateway can keep writing and filling up the whole cluster.

It's obvious that it's the admin's responsibility to make sure that 
doesn't happen, but what I'd like to do is make the RADOS Gateway deny 
PUT requests when the cluster is near full.

That way normal meta data operations (thus writes) can continue, but new 
objects won't be accepted until the cluster has enough space available.

This might also be useful for librbd. Refuse the creation of new RBD 
images so that existing images can continue operating.

-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Finding out via librados if a cluster is near full

[Wido den Hollander]

On 05/29/2013 10:14 AM, Wido den Hollander wrote:
> Hi,
>
> Is there a way to find out if a cluster is near full via librados?
>

Yes, there is. (Thanks tnt on IRC!)

There is ofcourse rados_cluster_stat which will give you:

struct rados_cluster_stat_t {
   uint64_t kb, kb_used, kb_avail;
   uint64_t num_objects;
};


One thing here is however that you don't know to what the (near)full 
ratio has been set to. So you have to do your own guessing.

> The reason I'm asking is that I'm working on a deployment of the RADOS
> Gateway and one of the potential problems I see is that the RADOS
> Gateway can keep writing and filling up the whole cluster.
>

So, with the cluster stat information we could implement:

rgw_refuse_write_above_ratio DOUBLE 0.90

> It's obvious that it's the admin's responsibility to make sure that
> doesn't happen, but what I'd like to do is make the RADOS Gateway deny
> PUT requests when the cluster is near full.
>
> That way normal meta data operations (thus writes) can continue, but new
> objects won't be accepted until the cluster has enough space available.
>
> This might also be useful for librbd. Refuse the creation of new RBD
> images so that existing images can continue operating.
>


-- 
Wido den Hollander
42on B.V.

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: Finding out via librados if a cluster is near full

[Dan Mick]

On 05/29/2013 01:24 AM, Wido den Hollander wrote:
> On 05/29/2013 10:14 AM, Wido den Hollander wrote:
>> Hi,
>>
>> Is there a way to find out if a cluster is near full via librados?
>>
>
> Yes, there is. (Thanks tnt on IRC!)
>
> There is ofcourse rados_cluster_stat which will give you:
>
> struct rados_cluster_stat_t {
>    uint64_t kb, kb_used, kb_avail;
>    uint64_t num_objects;
> };
>
>
> One thing here is however that you don't know to what the (near)full
> ratio has been set to. So you have to do your own guessing.

Aside: Soon there will be a librados interface for issuing commands like
ceph pg dump and getting the response (in JSON if desired), if that's
going to be helpful.

>
>> The reason I'm asking is that I'm working on a deployment of the RADOS
>> Gateway and one of the potential problems I see is that the RADOS
>> Gateway can keep writing and filling up the whole cluster.
>>
>
> So, with the cluster stat information we could implement:
>
> rgw_refuse_write_above_ratio DOUBLE 0.90
>
>> It's obvious that it's the admin's responsibility to make sure that
>> doesn't happen, but what I'd like to do is make the RADOS Gateway deny
>> PUT requests when the cluster is near full.
>>
>> That way normal meta data operations (thus writes) can continue, but new
>> objects won't be accepted until the cluster has enough space available.
>>
>> This might also be useful for librbd. Refuse the creation of new RBD
>> images so that existing images can continue operating.
>>
>
>