status monitoring options for ceph cluster

status monitoring options for ceph cluster

[Ugis]

Hi all,

I am planning to set up monitoring of ceph cluster relying on output
of commands:

#ceph health
#ceph -s

The need is to retrieve certain values, for example %degraded;
proportion "X GB / Y GB avail" to calculate available space%. I plan
to graph those values via zabbix and set up triggers for certain
tresholds.
May be somebody already has written shell parser for getting values
from output of these commands?

Probably this ir wider question - what is the intended approach for
ceph cluster monitoring? What do you guys use?

BR,
Ugis

Re: status monitoring options for ceph cluster

[Wido den Hollander]

On 01/02/2013 04:05 PM, Ugis wrote:
> Hi all,
>
> I am planning to set up monitoring of ceph cluster relying on output
> of commands:
>
> #ceph health
> #ceph -s
>
> The need is to retrieve certain values, for example %degraded;
> proportion "X GB / Y GB avail" to calculate available space%. I plan
> to graph those values via zabbix and set up triggers for certain
> tresholds.
> May be somebody already has written shell parser for getting values
> from output of these commands?
>
> Probably this ir wider question - what is the intended approach for
> ceph cluster monitoring? What do you guys use?

You might want to take a look at: http://tracker.newdream.net/issues/685

In version 0.35 a json output was added. I've never tried it before, but 
you might want to look at that.

A blogpost was also recently added: 
http://ceph.com/community/monitoring-a-ceph-cluster/

Wido

>
> BR,
> Ugis
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: status monitoring options for ceph cluster

[Joao Eduardo Luis]

On 01/02/2013 03:08 PM, Wido den Hollander wrote:
> On 01/02/2013 04:05 PM, Ugis wrote:
>> Hi all,
>>
>> I am planning to set up monitoring of ceph cluster relying on output
>> of commands:
>>
>> #ceph health
>> #ceph -s
>>
>> The need is to retrieve certain values, for example %degraded;
>> proportion "X GB / Y GB avail" to calculate available space%. I plan
>> to graph those values via zabbix and set up triggers for certain
>> tresholds.
>> May be somebody already has written shell parser for getting values
>> from output of these commands?
>>
>> Probably this ir wider question - what is the intended approach for
>> ceph cluster monitoring? What do you guys use?
>
> You might want to take a look at: http://tracker.newdream.net/issues/685
>
> In version 0.35 a json output was added. I've never tried it before, but
> you might want to look at that.

Currently, the only command that will output useful json for this kind 
of thing is 'ceph report'. I am finishing up on a patch to add json 
output capabilities to 'ceph health' and 'ceph status' though, but 
wasn't aiming at doing it with such level of detail -- basically, just 
output the same info as their plain counterparts, but using json instead.

   -Joao

Re: status monitoring options for ceph cluster

[Dennis Jacobfeuerborn]

On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>> On 01/02/2013 04:05 PM, Ugis wrote:
>>> Hi all,
>>>
>>> I am planning to set up monitoring of ceph cluster relying on output
>>> of commands:
>>>
>>> #ceph health
>>> #ceph -s
>>>
>>> The need is to retrieve certain values, for example %degraded;
>>> proportion "X GB / Y GB avail" to calculate available space%. I plan
>>> to graph those values via zabbix and set up triggers for certain
>>> tresholds.
>>> May be somebody already has written shell parser for getting values
>>> from output of these commands?
>>>
>>> Probably this ir wider question - what is the intended approach for
>>> ceph cluster monitoring? What do you guys use?
>>
>> You might want to take a look at: http://tracker.newdream.net/issues/685
>>
>> In version 0.35 a json output was added. I've never tried it before, but
>> you might want to look at that.
> 
> Currently, the only command that will output useful json for this kind of
> thing is 'ceph report'. I am finishing up on a patch to add json output
> capabilities to 'ceph health' and 'ceph status' though, but wasn't aiming
> at doing it with such level of detail -- basically, just output the same
> info as their plain counterparts, but using json instead.

I'm interested in this topic as well. What is also needed is a way to query
the state of the individual daemons on each machine in the cluster.

Regards,
  Dennis

Re: status monitoring options for ceph cluster

[Dan Mick]

There are plans being discussed for RESTful interfaces for configuration 
and monitoring; no timetable yet but it's definitely on our radar, and 
we'll discuss design etc. here when it starts happening; anyone wanting 
to participate is of course welcome.

On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
> On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
>> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>>> On 01/02/2013 04:05 PM, Ugis wrote:
>>>> Hi all,
>>>>
>>>> I am planning to set up monitoring of ceph cluster relying on output
>>>> of commands:
>>>>
>>>> #ceph health
>>>> #ceph -s
>>>>
>>>> The need is to retrieve certain values, for example %degraded;
>>>> proportion "X GB / Y GB avail" to calculate available space%. I plan
>>>> to graph those values via zabbix and set up triggers for certain
>>>> tresholds.
>>>> May be somebody already has written shell parser for getting values
>>>> from output of these commands?
>>>>
>>>> Probably this ir wider question - what is the intended approach for
>>>> ceph cluster monitoring? What do you guys use?
>>>
>>> You might want to take a look at: http://tracker.newdream.net/issues/685
>>>
>>> In version 0.35 a json output was added. I've never tried it before, but
>>> you might want to look at that.
>>
>> Currently, the only command that will output useful json for this kind of
>> thing is 'ceph report'. I am finishing up on a patch to add json output
>> capabilities to 'ceph health' and 'ceph status' though, but wasn't aiming
>> at doing it with such level of detail -- basically, just output the same
>> info as their plain counterparts, but using json instead.
>
> I'm interested in this topic as well. What is also needed is a way to query
> the state of the individual daemons on each machine in the cluster.
>
> Regards,
>    Dennis
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: status monitoring options for ceph cluster

[Dan Mick]

(and of course it makes sense to build those on libraries that could
serve other access methods)

On 01/02/2013 03:02 PM, Dan Mick wrote:
> There are plans being discussed for RESTful interfaces for configuration
> and monitoring; no timetable yet but it's definitely on our radar, and
> we'll discuss design etc. here when it starts happening; anyone wanting
> to participate is of course welcome.
>
> On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
>> On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
>>> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>>>> On 01/02/2013 04:05 PM, Ugis wrote:
>>>>> Hi all,
>>>>>
>>>>> I am planning to set up monitoring of ceph cluster relying on output
>>>>> of commands:
>>>>>
>>>>> #ceph health
>>>>> #ceph -s
>>>>>
>>>>> The need is to retrieve certain values, for example %degraded;
>>>>> proportion "X GB / Y GB avail" to calculate available space%. I plan
>>>>> to graph those values via zabbix and set up triggers for certain
>>>>> tresholds.
>>>>> May be somebody already has written shell parser for getting values
>>>>> from output of these commands?
>>>>>
>>>>> Probably this ir wider question - what is the intended approach for
>>>>> ceph cluster monitoring? What do you guys use?
>>>>
>>>> You might want to take a look at:
>>>> http://tracker.newdream.net/issues/685
>>>>
>>>> In version 0.35 a json output was added. I've never tried it before,
>>>> but
>>>> you might want to look at that.
>>>
>>> Currently, the only command that will output useful json for this
>>> kind of
>>> thing is 'ceph report'. I am finishing up on a patch to add json output
>>> capabilities to 'ceph health' and 'ceph status' though, but wasn't
>>> aiming
>>> at doing it with such level of detail -- basically, just output the same
>>> info as their plain counterparts, but using json instead.
>>
>> I'm interested in this topic as well. What is also needed is a way to
>> query
>> the state of the individual daemons on each machine in the cluster.
>>
>> Regards,
>>    Dennis
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: status monitoring options for ceph cluster

[Wido den Hollander]

On 01/03/2013 12:05 AM, Dan Mick wrote:
> (and of course it makes sense to build those on libraries that could
> serve other access methods)

I'd prefer a library :-) This way you can build your own binaries, 
bindings for languages like Python, PHP and Java.

The binaries could just link against the library and output information 
in any way we want for Zabbix, Nagios, Icinga, etc.

Wido

>
> On 01/02/2013 03:02 PM, Dan Mick wrote:
>> There are plans being discussed for RESTful interfaces for configuration
>> and monitoring; no timetable yet but it's definitely on our radar, and
>> we'll discuss design etc. here when it starts happening; anyone wanting
>> to participate is of course welcome.
>>
>> On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
>>> On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
>>>> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>>>>> On 01/02/2013 04:05 PM, Ugis wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I am planning to set up monitoring of ceph cluster relying on output
>>>>>> of commands:
>>>>>>
>>>>>> #ceph health
>>>>>> #ceph -s
>>>>>>
>>>>>> The need is to retrieve certain values, for example %degraded;
>>>>>> proportion "X GB / Y GB avail" to calculate available space%. I plan
>>>>>> to graph those values via zabbix and set up triggers for certain
>>>>>> tresholds.
>>>>>> May be somebody already has written shell parser for getting values
>>>>>> from output of these commands?
>>>>>>
>>>>>> Probably this ir wider question - what is the intended approach for
>>>>>> ceph cluster monitoring? What do you guys use?
>>>>>
>>>>> You might want to take a look at:
>>>>> http://tracker.newdream.net/issues/685
>>>>>
>>>>> In version 0.35 a json output was added. I've never tried it before,
>>>>> but
>>>>> you might want to look at that.
>>>>
>>>> Currently, the only command that will output useful json for this
>>>> kind of
>>>> thing is 'ceph report'. I am finishing up on a patch to add json output
>>>> capabilities to 'ceph health' and 'ceph status' though, but wasn't
>>>> aiming
>>>> at doing it with such level of detail -- basically, just output the
>>>> same
>>>> info as their plain counterparts, but using json instead.
>>>
>>> I'm interested in this topic as well. What is also needed is a way to
>>> query
>>> the state of the individual daemons on each machine in the cluster.
>>>
>>> Regards,
>>>    Dennis
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: status monitoring options for ceph cluster

[Sage Weil]

On Thu, 3 Jan 2013, Wido den Hollander wrote:
> On 01/03/2013 12:05 AM, Dan Mick wrote:
> > (and of course it makes sense to build those on libraries that could
> > serve other access methods)
> 
> I'd prefer a library :-) This way you can build your own binaries, bindings
> for languages like Python, PHP and Java.
> 
> The binaries could just link against the library and output information in any
> way we want for Zabbix, Nagios, Icinga, etc.

A library is coming very soon, and shortly after that a REST interface 
that wraps it.

Dennis, Ugis:

More generally, the 'ceph report' command will give you a JSON dump of 
theh current cluster status.  You can get less detail from 'ceph health 
[detail]', which will also have a JSON option shortly.

You may also be interested in the admin socket, which lets you query 
individaul running daemons (instead of the monitor):

 ceph --admin-daemon /var/run/ceph/ceph-osd.123.asok help

sage

> 
> Wido
> 
> > 
> > On 01/02/2013 03:02 PM, Dan Mick wrote:
> > > There are plans being discussed for RESTful interfaces for configuration
> > > and monitoring; no timetable yet but it's definitely on our radar, and
> > > we'll discuss design etc. here when it starts happening; anyone wanting
> > > to participate is of course welcome.
> > > 
> > > On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
> > > > On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
> > > > > On 01/02/2013 03:08 PM, Wido den Hollander wrote:
> > > > > > On 01/02/2013 04:05 PM, Ugis wrote:
> > > > > > > Hi all,
> > > > > > > 
> > > > > > > I am planning to set up monitoring of ceph cluster relying on
> > > > > > > output
> > > > > > > of commands:
> > > > > > > 
> > > > > > > #ceph health
> > > > > > > #ceph -s
> > > > > > > 
> > > > > > > The need is to retrieve certain values, for example %degraded;
> > > > > > > proportion "X GB / Y GB avail" to calculate available space%. I
> > > > > > > plan
> > > > > > > to graph those values via zabbix and set up triggers for certain
> > > > > > > tresholds.
> > > > > > > May be somebody already has written shell parser for getting
> > > > > > > values
> > > > > > > from output of these commands?
> > > > > > > 
> > > > > > > Probably this ir wider question - what is the intended approach
> > > > > > > for
> > > > > > > ceph cluster monitoring? What do you guys use?
> > > > > > 
> > > > > > You might want to take a look at:
> > > > > > http://tracker.newdream.net/issues/685
> > > > > > 
> > > > > > In version 0.35 a json output was added. I've never tried it before,
> > > > > > but
> > > > > > you might want to look at that.
> > > > > 
> > > > > Currently, the only command that will output useful json for this
> > > > > kind of
> > > > > thing is 'ceph report'. I am finishing up on a patch to add json
> > > > > output
> > > > > capabilities to 'ceph health' and 'ceph status' though, but wasn't
> > > > > aiming
> > > > > at doing it with such level of detail -- basically, just output the
> > > > > same
> > > > > info as their plain counterparts, but using json instead.
> > > > 
> > > > I'm interested in this topic as well. What is also needed is a way to
> > > > query
> > > > the state of the individual daemons on each machine in the cluster.
> > > > 
> > > > Regards,
> > > >    Dennis
> > > > 
> > > > --
> > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > > > the body of a message to majordomo@vger.kernel.org
> > > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

RE: status monitoring options for ceph cluster

[Paul Pettigrew]

Happy new year all

Over the past year, our company has written an extensive guide on how we monitor ceph using Zabbix.

I would rather contribute it as a HOWTO or Guide into the Ceph wiki, rather than just this email thread. It is written in wiki format, and email would lose that formatting.

Hopefully it will benefit others, and I can continue to maintain it and update it by incorporating advice from others as time goes by.

I see this intro Blog post to monitoring:
http://ceph.com/community/monitoring-a-ceph-cluster/

That would be a good intro, and what I am proposing would be a follow-on "advanced" guide.

Should it go to the Ceph Blog? Or elsewhere?

If somebody can please advise how this contribution can be made, thanks!

Paul


==
Paul Pettigrew 
w: http://mach.com.au/
Offices & Data Centres: Noosa | Brisbane | Cooroy | USA

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Dan Mick
Sent: Thursday, 3 January 2013 9:03 AM
To: Dennis Jacobfeuerborn
Cc: Joao Eduardo Luis; Wido den Hollander; Ugis; ceph-devel@vger.kernel.org
Subject: Re: status monitoring options for ceph cluster

There are plans being discussed for RESTful interfaces for configuration and monitoring; no timetable yet but it's definitely on our radar, and we'll discuss design etc. here when it starts happening; anyone wanting to participate is of course welcome.

On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
> On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
>> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>>> On 01/02/2013 04:05 PM, Ugis wrote:
>>>> Hi all,
>>>>
>>>> I am planning to set up monitoring of ceph cluster relying on 
>>>> output of commands:
>>>>
>>>> #ceph health
>>>> #ceph -s
>>>>
>>>> The need is to retrieve certain values, for example %degraded; 
>>>> proportion "X GB / Y GB avail" to calculate available space%. I 
>>>> plan to graph those values via zabbix and set up triggers for 
>>>> certain tresholds.
>>>> May be somebody already has written shell parser for getting values 
>>>> from output of these commands?
>>>>
>>>> Probably this ir wider question - what is the intended approach for 
>>>> ceph cluster monitoring? What do you guys use?
>>>
>>> You might want to take a look at: 
>>> http://tracker.newdream.net/issues/685
>>>
>>> In version 0.35 a json output was added. I've never tried it before, 
>>> but you might want to look at that.
>>
>> Currently, the only command that will output useful json for this 
>> kind of thing is 'ceph report'. I am finishing up on a patch to add 
>> json output capabilities to 'ceph health' and 'ceph status' though, 
>> but wasn't aiming at doing it with such level of detail -- basically, 
>> just output the same info as their plain counterparts, but using json instead.
>
> I'm interested in this topic as well. What is also needed is a way to 
> query the state of the individual daemons on each machine in the cluster.
>
> Regards,
>    Dennis
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: status monitoring options for ceph cluster

[Ugis]

Hopefully there will be some place where community can share their
ceph "how-to", hope to see what Paul offers. Wiki could be suitable
solution.

As I see at present there seems no ready solution for fine grained
ceph monitoring. Therefore I can express what I would imagine ceph
monitoring look like.
From admin perspective at least we need possibility to get "key -
value" pairs at shell level. From that point admins can use their
prefered solutions with ease - zabbix, nagios, shell scripts etc.

I would expect "ceph variables" command to return list of all
available variables and "ceph query XXXX" to return value of specified
variable.
As an example(based on excerpt from "ceph report") would be:
#ceph query overall_status
#HEALTH_OK

With such approach there would be no problems in future to add
variables not destroying previous solutions. In case of zabbix it
would be the matter of developing and sharing ceph template.
In more distant future ceph could be queryable by SNMP, but first
things first :)

Ugis


2013/1/3 Paul Pettigrew <Paul.Pettigrew@mach.com.au>:
> Happy new year all
>
> Over the past year, our company has written an extensive guide on how we monitor ceph using Zabbix.
>
> I would rather contribute it as a HOWTO or Guide into the Ceph wiki, rather than just this email thread. It is written in wiki format, and email would lose that formatting.
>
> Hopefully it will benefit others, and I can continue to maintain it and update it by incorporating advice from others as time goes by.
>
> I see this intro Blog post to monitoring:
> http://ceph.com/community/monitoring-a-ceph-cluster/
>
> That would be a good intro, and what I am proposing would be a follow-on "advanced" guide.
>
> Should it go to the Ceph Blog? Or elsewhere?
>
> If somebody can please advise how this contribution can be made, thanks!
>
> Paul
>
>
> ==
> Paul Pettigrew
> w: http://mach.com.au/
> Offices & Data Centres: Noosa | Brisbane | Cooroy | USA
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Dan Mick
> Sent: Thursday, 3 January 2013 9:03 AM
> To: Dennis Jacobfeuerborn
> Cc: Joao Eduardo Luis; Wido den Hollander; Ugis; ceph-devel@vger.kernel.org
> Subject: Re: status monitoring options for ceph cluster
>
> There are plans being discussed for RESTful interfaces for configuration and monitoring; no timetable yet but it's definitely on our radar, and we'll discuss design etc. here when it starts happening; anyone wanting to participate is of course welcome.
>
> On 01/02/2013 08:34 AM, Dennis Jacobfeuerborn wrote:
>> On 01/02/2013 04:58 PM, Joao Eduardo Luis wrote:
>>> On 01/02/2013 03:08 PM, Wido den Hollander wrote:
>>>> On 01/02/2013 04:05 PM, Ugis wrote:
>>>>> Hi all,
>>>>>
>>>>> I am planning to set up monitoring of ceph cluster relying on
>>>>> output of commands:
>>>>>
>>>>> #ceph health
>>>>> #ceph -s
>>>>>
>>>>> The need is to retrieve certain values, for example %degraded;
>>>>> proportion "X GB / Y GB avail" to calculate available space%. I
>>>>> plan to graph those values via zabbix and set up triggers for
>>>>> certain tresholds.
>>>>> May be somebody already has written shell parser for getting values
>>>>> from output of these commands?
>>>>>
>>>>> Probably this ir wider question - what is the intended approach for
>>>>> ceph cluster monitoring? What do you guys use?
>>>>
>>>> You might want to take a look at:
>>>> http://tracker.newdream.net/issues/685
>>>>
>>>> In version 0.35 a json output was added. I've never tried it before,
>>>> but you might want to look at that.
>>>
>>> Currently, the only command that will output useful json for this
>>> kind of thing is 'ceph report'. I am finishing up on a patch to add
>>> json output capabilities to 'ceph health' and 'ceph status' though,
>>> but wasn't aiming at doing it with such level of detail -- basically,
>>> just output the same info as their plain counterparts, but using json instead.
>>
>> I'm interested in this topic as well. What is also needed is a way to
>> query the state of the individual daemons on each machine in the cluster.
>>
>> Regards,
>>    Dennis
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html