Preferred location for utility execution

Preferred location for utility execution

[Handzik, Joe]

Hey ceph-devel,

Gregory Meno (of Calamari fame) and I are working on what is now officially a blueprint for Jewel ( http://tracker.ceph.com/projects/ceph/wiki/Calamariapihardwarestorage ), and we'd like some feedback.

Some of this has been addressed via separate conversations about the feature that some of this work started out as (identifying drives in a cluster by toggling their LED states), but we wanted to ask a more direct question: What is the preferred location/mechanism to execute operations on storage hardware?

We see two clear options:

1. Make Calamari responsible for executing commands using various linux utilities (and /sys, when applicable).
2. Build a command set into RADOS to execute commands using various linux utilities. These commands could then be executed by Calamari using the rest api.

The big win for #1 is the ability to rapidly iterate on the capabilities of the Calamari toolset (it is almost certainly going to be faster to create a set of scripts similar to Gregory's initial commit for SMART polling than to add that functionality inside RADOS. See: https://github.com/ceph/calamari/pull/267 ). For #2, we'd pick up the ability to run those same commands via the cli, which would give users a lot more flexibility in how they troubleshoot their cluster (Calamari wouldn't be required, it would just make life easier).

The LED state conversation had people sitting on both sides of the fence on this one, there wasn't a clear winner. Gregory mentioned a hybrid option in a conversation we had this week: Implement first in Calamari, then consider migrating the feature into the Ceph source itself if users communicate that they want it there (the LED state might move for its debugging utility, whereas the SMART poll could stay in Calamari, for example).

Thoughts?

Joe

Re: Preferred location for utility execution

[John Spray]

On 26/06/2015 21:14, Handzik, Joe wrote:
> Hey ceph-devel,
>
> Gregory Meno (of Calamari fame) and I are working on what is now officially a blueprint for Jewel ( http://tracker.ceph.com/projects/ceph/wiki/Calamariapihardwarestorage ), and we'd like some feedback.
>
> Some of this has been addressed via separate conversations about the feature that some of this work started out as (identifying drives in a cluster by toggling their LED states), but we wanted to ask a more direct question: What is the preferred location/mechanism to execute operations on storage hardware?
>
> We see two clear options:
>
> 1. Make Calamari responsible for executing commands using various linux utilities (and /sys, when applicable).
> 2. Build a command set into RADOS to execute commands using various linux utilities. These commands could then be executed by Calamari using the rest api.
>
> The big win for #1 is the ability to rapidly iterate on the capabilities of the Calamari toolset (it is almost certainly going to be faster to create a set of scripts similar to Gregory's initial commit for SMART polling than to add that functionality inside RADOS. See: https://github.com/ceph/calamari/pull/267 ). For #2, we'd pick up the ability to run those same commands via the cli, which would give users a lot more flexibility in how they troubleshoot their cluster (Calamari wouldn't be required, it would just make life easier).

Hi Joe,

I'd reiterate my earlier comments[1] in favour of option 2.

I would be cautious about implementing any of this in Calamari until 
there are at least upstream packages available for folks to use, and 
broader uptake.  In the current situation, it's hard to ask people to 
try something out in Calamari, and much more straightforward to 
distribute something as part of Ceph.  Hardware is pretty varied, I 
would expect you'll need help from others in the community to ensure any 
hardware handling works as expected in diverse environments, which will 
be much simpler with ceph than calamari.

The part where some central python (calamari or otherwise) would really 
come into its own is in the fusion of information from multiple hosts, 
and exposing it to a user interface.  On that aspect, I left some 
comments last time this came up: 
http://lists.ceph.com/pipermail/ceph-calamari-ceph.com/2015-May/000073.html

Ceph itself is getting a bit smarter with some of this stuff, e.g. the 
new "node ls" stuff gives you metadata about hosts and services without 
the need for calamari.  Hanging device info off these new structures 
would be a pretty reasonable thing to do, and if someone later has a GUI 
that they want to pipe that into, they can grab it via the mon along 
with everything else.

Cheers,
John


1. https://www.mail-archive.com/ceph-devel@vger.kernel.org/msg23186.html

RE: Preferred location for utility execution

[Handzik, Joe]

Thanks for the reply, John. I knew where you stood. :)

I tend to agree with you for much of this, I should be able to pull a fair amount of drive metadata out of /sys, and now that I've added the device node to bootstrap off of, that sort of data collection should be easy to initiate alongside the rest of the metadata collection here: https://github.com/ceph/ceph/blob/master/src/osd/OSD.cc#L4566

Something I'm not sure of is, say, Gregory's commit to Calamari. Does your opinion extend to functionality like that? Should SMART be collected within Ceph, rather than via a salt script initiated by Calamari?

Joe

-----Original Message-----
From: John Spray [mailto:john.spray@redhat.com] 
Sent: Friday, June 26, 2015 5:24 PM
To: Handzik, Joe; ceph-devel
Cc: gmeno@redhat.com
Subject: Re: Preferred location for utility execution



On 26/06/2015 21:14, Handzik, Joe wrote:
> Hey ceph-devel,
>
> Gregory Meno (of Calamari fame) and I are working on what is now officially a blueprint for Jewel ( http://tracker.ceph.com/projects/ceph/wiki/Calamariapihardwarestorage ), and we'd like some feedback.
>
> Some of this has been addressed via separate conversations about the feature that some of this work started out as (identifying drives in a cluster by toggling their LED states), but we wanted to ask a more direct question: What is the preferred location/mechanism to execute operations on storage hardware?
>
> We see two clear options:
>
> 1. Make Calamari responsible for executing commands using various linux utilities (and /sys, when applicable).
> 2. Build a command set into RADOS to execute commands using various linux utilities. These commands could then be executed by Calamari using the rest api.
>
> The big win for #1 is the ability to rapidly iterate on the capabilities of the Calamari toolset (it is almost certainly going to be faster to create a set of scripts similar to Gregory's initial commit for SMART polling than to add that functionality inside RADOS. See: https://github.com/ceph/calamari/pull/267 ). For #2, we'd pick up the ability to run those same commands via the cli, which would give users a lot more flexibility in how they troubleshoot their cluster (Calamari wouldn't be required, it would just make life easier).

Hi Joe,

I'd reiterate my earlier comments[1] in favour of option 2.

I would be cautious about implementing any of this in Calamari until there are at least upstream packages available for folks to use, and broader uptake.  In the current situation, it's hard to ask people to try something out in Calamari, and much more straightforward to distribute something as part of Ceph.  Hardware is pretty varied, I would expect you'll need help from others in the community to ensure any hardware handling works as expected in diverse environments, which will be much simpler with ceph than calamari.

The part where some central python (calamari or otherwise) would really come into its own is in the fusion of information from multiple hosts, and exposing it to a user interface.  On that aspect, I left some comments last time this came up: 
http://lists.ceph.com/pipermail/ceph-calamari-ceph.com/2015-May/000073.html

Ceph itself is getting a bit smarter with some of this stuff, e.g. the new "node ls" stuff gives you metadata about hosts and services without the need for calamari.  Hanging device info off these new structures would be a pretty reasonable thing to do, and if someone later has a GUI that they want to pipe that into, they can grab it via the mon along with everything else.

Cheers,
John


1. https://www.mail-archive.com/ceph-devel@vger.kernel.org/msg23186.html

Re: Preferred location for utility execution

[John Spray]

On 29/06/2015 19:44, Handzik, Joe wrote:
> Thanks for the reply, John. I knew where you stood. :)
>
> I tend to agree with you for much of this, I should be able to pull a fair amount of drive metadata out of /sys, and now that I've added the device node to bootstrap off of, that sort of data collection should be easy to initiate alongside the rest of the metadata collection here: https://github.com/ceph/ceph/blob/master/src/osd/OSD.cc#L4566
>
> Something I'm not sure of is, say, Gregory's commit to Calamari. Does your opinion extend to functionality like that? Should SMART be collected within Ceph, rather than via a salt script initiated by Calamari?

Good question...  It depends on your position: if you're someone already 
using or developing Calamari, then it's a perfectly reasonable thing to 
add into Calamari.  If you're looking for a way to provide functionality 
for most ordinary ceph users, and enable involvement of most ceph 
developers, then Calamari brings quite a bit of baggage with it, and 
it's much more desirable to have it baked into Ceph.

The good news is that there's no rule against doing these things both 
places, so there's not really a wrong answer (at least in my opinion).  
We might find that while Calamari remains a minority thing, it's useful 
to have a whizz-bang version of a feature in Calamari, and something 
more basic baked into Ceph.  That's kind of already true for e.g. 
calamari's /server API vs. ceph's built in "node ls".

Finally, going a bit off topic: in an ideal world, the answer would be 
to get Calamari where it needs to be, and then add hardware monitoring 
functionality into Calamari while keeping the core of Ceph lightweight.  
This applies to other pieces of functionality, like the recent "node ls" 
functionality, continued existence of ceph-rest-api, some of the recent 
MDS health check stuff, all the kind of things that we would add in the 
python layer if we had one that was positioned to be present in all Ceph 
deployments, rather than only some.  Instead, we're writing these things 
into the C++ part because that's the only way to reach all the users.  
Calamari can only take up that role once it's:
  * Packaged (without any onerous dependencies)
  * Comprehensive (can do anything the CLI can do)
  * Lightweight (nothing to prevent us from enabling it by default)

John

RE: Preferred location for utility execution

[Handzik, Joe]

Sounds like a worthwhile discussion point for the blueprint session. Thanks for the feedback!

Joe

-----Original Message-----
From: John Spray [mailto:john.spray@redhat.com] 
Sent: Monday, June 29, 2015 3:10 PM
To: Handzik, Joe; ceph-devel
Cc: gmeno@redhat.com
Subject: Re: Preferred location for utility execution

On 29/06/2015 19:44, Handzik, Joe wrote:
> Thanks for the reply, John. I knew where you stood. :)
>
> I tend to agree with you for much of this, I should be able to pull a 
> fair amount of drive metadata out of /sys, and now that I've added the 
> device node to bootstrap off of, that sort of data collection should 
> be easy to initiate alongside the rest of the metadata collection 
> here: https://github.com/ceph/ceph/blob/master/src/osd/OSD.cc#L4566
>
> Something I'm not sure of is, say, Gregory's commit to Calamari. Does your opinion extend to functionality like that? Should SMART be collected within Ceph, rather than via a salt script initiated by Calamari?

Good question...  It depends on your position: if you're someone already using or developing Calamari, then it's a perfectly reasonable thing to add into Calamari.  If you're looking for a way to provide functionality for most ordinary ceph users, and enable involvement of most ceph developers, then Calamari brings quite a bit of baggage with it, and it's much more desirable to have it baked into Ceph.

The good news is that there's no rule against doing these things both places, so there's not really a wrong answer (at least in my opinion).  
We might find that while Calamari remains a minority thing, it's useful to have a whizz-bang version of a feature in Calamari, and something more basic baked into Ceph.  That's kind of already true for e.g. 
calamari's /server API vs. ceph's built in "node ls".

Finally, going a bit off topic: in an ideal world, the answer would be to get Calamari where it needs to be, and then add hardware monitoring functionality into Calamari while keeping the core of Ceph lightweight.  
This applies to other pieces of functionality, like the recent "node ls" 
functionality, continued existence of ceph-rest-api, some of the recent MDS health check stuff, all the kind of things that we would add in the python layer if we had one that was positioned to be present in all Ceph deployments, rather than only some.  Instead, we're writing these things into the C++ part because that's the only way to reach all the users.  
Calamari can only take up that role once it's:
  * Packaged (without any onerous dependencies)
  * Comprehensive (can do anything the CLI can do)
  * Lightweight (nothing to prevent us from enabling it by default)

John