disk enclosure LEDs

disk enclosure LEDs

[Sage Weil]

Hi all,

libstoragemgmt has made progress on a generic interface for twiddling 
enclosure LEDs!

>      * RHEL 7.3 or Fedora 24+
>      * sudo yum install libstoragemgmt
>      * sudo lsmcli local-disk-ident-led-on --path /dev/sdX
>      * sudo lsmcli local-disk-ident-led-off --path /dev/sdX
>      * sudo lsmcli local-disk-fault-led-on --path /dev/sdX
>      * sudo lsmcli local-disk-fault-led-off --path /dev/sdX

> Python API document:
> 
>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_on)'
>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_off)'
>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_on)'
>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_off)'
> 
> C API document:
> 
>      Check header file `libstoragemgmt_local_disk.h` in
>      `libstoragemgmt-devel` rpm package. The functions are:
> 
>      lsm_local_disk_ident_led_on()
>      lsm_local_disk_ident_led_off()
>      lsm_local_disk_fault_led_on()
>      lsm_local_disk_fault_led_off()

Since this is in a reasonably usable state, I think It's time for us to 
figure out how we are going to do this in ceph.

A few ideas:

 ceph osd identify osd.123    # blink for a few seconds?

or

 ceph osd ident-led-on osd.123  # turn on
 ceph osd ident-led-off osd.123  # turn off
 ceph osd fault-led-on osd.123  # turn on
 ceph osd fault-led-off osd.123  # turn off

This would mean persistently recording the LED state in the OSDMap.  And 
it would mean ceph-osd is the one twiddling the LEDs.  But that might not 
be the way to go in all cases.  For example, if we have an OSD that fails, 
once we confirm that we've healed (and don't need that OSDs data) we'd 
probably want to set the fault light so that the disk can be pulled 
safely.  In that case, ceph-osd isn't running (it's long-since failed), 
and we'd need some other agent on the node to twiddle the light.  Do we 
really want multiple things twiddling lights?

We also often have a N:M mapping of osds to devices (multiple devices per 
OSD, multiple OSDs per device), which means a per-OSD flag might not be 
the right way to think about this anyway.

Has anyone thought this through yet?

Thanks!
sage

Re: disk enclosure LEDs

[Brett Niver]

Besides the obvious disk pull, I really like the option to dynamically
identify mappings.  Ive used a very similar concept on fabric switch
enclosures as part of visually verifying proper cabling and path
mapping...

On Wed, Nov 30, 2016 at 11:51 AM, Sage Weil <sweil@redhat.com> wrote:
> Hi all,
>
> libstoragemgmt has made progress on a generic interface for twiddling
> enclosure LEDs!
>
>>      * RHEL 7.3 or Fedora 24+
>>      * sudo yum install libstoragemgmt
>>      * sudo lsmcli local-disk-ident-led-on --path /dev/sdX
>>      * sudo lsmcli local-disk-ident-led-off --path /dev/sdX
>>      * sudo lsmcli local-disk-fault-led-on --path /dev/sdX
>>      * sudo lsmcli local-disk-fault-led-off --path /dev/sdX
>
>> Python API document:
>>
>>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_on)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_off)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_on)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_off)'
>>
>> C API document:
>>
>>      Check header file `libstoragemgmt_local_disk.h` in
>>      `libstoragemgmt-devel` rpm package. The functions are:
>>
>>      lsm_local_disk_ident_led_on()
>>      lsm_local_disk_ident_led_off()
>>      lsm_local_disk_fault_led_on()
>>      lsm_local_disk_fault_led_off()
>
> Since this is in a reasonably usable state, I think It's time for us to
> figure out how we are going to do this in ceph.
>
> A few ideas:
>
>  ceph osd identify osd.123    # blink for a few seconds?
>
> or
>
>  ceph osd ident-led-on osd.123  # turn on
>  ceph osd ident-led-off osd.123  # turn off
>  ceph osd fault-led-on osd.123  # turn on
>  ceph osd fault-led-off osd.123  # turn off
>
> This would mean persistently recording the LED state in the OSDMap.  And
> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
> be the way to go in all cases.  For example, if we have an OSD that fails,
> once we confirm that we've healed (and don't need that OSDs data) we'd
> probably want to set the fault light so that the disk can be pulled
> safely.  In that case, ceph-osd isn't running (it's long-since failed),
> and we'd need some other agent on the node to twiddle the light.  Do we
> really want multiple things twiddling lights?
>
> We also often have a N:M mapping of osds to devices (multiple devices per
> OSD, multiple OSDs per device), which means a per-OSD flag might not be
> the right way to think about this anyway.
>
> Has anyone thought this through yet?
>
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Joao Eduardo Luis]

On 11/30/2016 04:51 PM, Sage Weil wrote:
> Since this is in a reasonably usable state, I think It's time for us to
> figure out how we are going to do this in ceph.
>
> A few ideas:
>
>  ceph osd identify osd.123    # blink for a few seconds?
>
> or
>
>  ceph osd ident-led-on osd.123  # turn on
>  ceph osd ident-led-off osd.123  # turn off
>  ceph osd fault-led-on osd.123  # turn on
>  ceph osd fault-led-off osd.123  # turn off
>
> This would mean persistently recording the LED state in the OSDMap.  And
> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
> be the way to go in all cases.  For example, if we have an OSD that fails,
> once we confirm that we've healed (and don't need that OSDs data) we'd
> probably want to set the fault light so that the disk can be pulled
> safely.  In that case, ceph-osd isn't running (it's long-since failed),
> and we'd need some other agent on the node to twiddle the light.  Do we
> really want multiple things twiddling lights?

Although this is really neat, I don't think ceph is the place to 
implement or even manage it.

This should fall under a management tool's purview.

That doesn't mean we couldn't keep some info on ceph itself, but in that 
case it should be kept either on the mon's kv config-key or ceph-mgr 
(e.g., set of osds that actually support this, which led to turn if 
something misbehaves, whatever).

And adding this sort of information to the osdmap is overkill. This is 
not something that the osds should be concerned about, but whatever 
tool/service monitors the osds.

Besides, we'd be taking a stand on how to do something completely ceph 
unrelated - "how to blink your enclosure's leds". I'm guessing this will 
be hardware dependent; availability of the required libraries will be 
distro dependent; and people may want to implement this in totally 
different ways from whatever we concoct here.

I don't see a good argument to delegate this to ceph.

   -Joao

Re: disk enclosure LEDs

[John Spray]

On Wed, Nov 30, 2016 at 7:37 PM, Joao Eduardo Luis <joao@suse.de> wrote:
> On 11/30/2016 04:51 PM, Sage Weil wrote:
>>
>> Since this is in a reasonably usable state, I think It's time for us to
>> figure out how we are going to do this in ceph.
>>
>> A few ideas:
>>
>>  ceph osd identify osd.123    # blink for a few seconds?
>>
>> or
>>
>>  ceph osd ident-led-on osd.123  # turn on
>>  ceph osd ident-led-off osd.123  # turn off
>>  ceph osd fault-led-on osd.123  # turn on
>>  ceph osd fault-led-off osd.123  # turn off
>>
>> This would mean persistently recording the LED state in the OSDMap.  And
>> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
>> be the way to go in all cases.  For example, if we have an OSD that fails,
>> once we confirm that we've healed (and don't need that OSDs data) we'd
>> probably want to set the fault light so that the disk can be pulled
>> safely.  In that case, ceph-osd isn't running (it's long-since failed),
>> and we'd need some other agent on the node to twiddle the light.  Do we
>> really want multiple things twiddling lights?
>
>
> Although this is really neat, I don't think ceph is the place to implement
> or even manage it.
>
> This should fall under a management tool's purview.

I would agree that any persistence/policy parts don't belong in the
OSD or the mons, but the remote access part where we reach across the
network and do something with libstoragemgmt is very useful to have
inside Ceph, because it means we can have management tools that just
speak librados and don't have to have their own remote access to all
the nodes in the system.

Put another way, I'd like it to be the case that people who want to
twiddle LEDs can write a ceph-mgr module that does that, rather than
having to push out a libstoragemgmt-enabled agent to the Ceph servers.

John

> That doesn't mean we couldn't keep some info on ceph itself, but in that
> case it should be kept either on the mon's kv config-key or ceph-mgr (e.g.,
> set of osds that actually support this, which led to turn if something
> misbehaves, whatever).
>
> And adding this sort of information to the osdmap is overkill. This is not
> something that the osds should be concerned about, but whatever tool/service
> monitors the osds.
>
> Besides, we'd be taking a stand on how to do something completely ceph
> unrelated - "how to blink your enclosure's leds". I'm guessing this will be
> hardware dependent; availability of the required libraries will be distro
> dependent; and people may want to implement this in totally different ways
> from whatever we concoct here.
>
> I don't see a good argument to delegate this to ceph.
>
>   -Joao
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: disk enclosure LEDs

[Allen Samuels]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of John Spray
> Sent: Wednesday, November 30, 2016 4:15 PM
> To: Joao Eduardo Luis <joao@suse.de>
> Cc: Sage Weil <sweil@redhat.com>; Ceph Development <ceph-
> devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> On Wed, Nov 30, 2016 at 7:37 PM, Joao Eduardo Luis <joao@suse.de> wrote:
> > On 11/30/2016 04:51 PM, Sage Weil wrote:
> >>
> >> Since this is in a reasonably usable state, I think It's time for us
> >> to figure out how we are going to do this in ceph.
> >>
> >> A few ideas:
> >>
> >>  ceph osd identify osd.123    # blink for a few seconds?
> >>
> >> or
> >>
> >>  ceph osd ident-led-on osd.123  # turn on  ceph osd ident-led-off
> >> osd.123  # turn off  ceph osd fault-led-on osd.123  # turn on  ceph
> >> osd fault-led-off osd.123  # turn off
> >>
> >> This would mean persistently recording the LED state in the OSDMap.
> >> And it would mean ceph-osd is the one twiddling the LEDs.  But that
> >> might not be the way to go in all cases.  For example, if we have an
> >> OSD that fails, once we confirm that we've healed (and don't need
> >> that OSDs data) we'd probably want to set the fault light so that the
> >> disk can be pulled safely.  In that case, ceph-osd isn't running
> >> (it's long-since failed), and we'd need some other agent on the node
> >> to twiddle the light.  Do we really want multiple things twiddling lights?
> >
> >
> > Although this is really neat, I don't think ceph is the place to
> > implement or even manage it.
> >
> > This should fall under a management tool's purview.
> 
> I would agree that any persistence/policy parts don't belong in the OSD or
> the mons, but the remote access part where we reach across the network
> and do something with libstoragemgmt is very useful to have inside Ceph,
> because it means we can have management tools that just speak librados
> and don't have to have their own remote access to all the nodes in the
> system.
> 
> Put another way, I'd like it to be the case that people who want to twiddle
> LEDs can write a ceph-mgr module that does that, rather than having to push
> out a libstoragemgmt-enabled agent to the Ceph servers.

+100

> 
> John
> 
> > That doesn't mean we couldn't keep some info on ceph itself, but in
> > that case it should be kept either on the mon's kv config-key or
> > ceph-mgr (e.g., set of osds that actually support this, which led to
> > turn if something misbehaves, whatever).
> >
> > And adding this sort of information to the osdmap is overkill. This is
> > not something that the osds should be concerned about, but whatever
> > tool/service monitors the osds.
> >
> > Besides, we'd be taking a stand on how to do something completely ceph
> > unrelated - "how to blink your enclosure's leds". I'm guessing this
> > will be hardware dependent; availability of the required libraries
> > will be distro dependent; and people may want to implement this in
> > totally different ways from whatever we concoct here.
> >
> > I don't see a good argument to delegate this to ceph.
> >
> >   -Joao
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Piotr Dałek]

On Thu, Dec 01, 2016 at 12:15:25AM +0000, John Spray wrote:
> On Wed, Nov 30, 2016 at 7:37 PM, Joao Eduardo Luis <joao@suse.de> wrote:
> > On 11/30/2016 04:51 PM, Sage Weil wrote:
> >>
> >> Since this is in a reasonably usable state, I think It's time for us to
> >> figure out how we are going to do this in ceph.
> >>
> >> A few ideas:
> >>
> >>  ceph osd identify osd.123    # blink for a few seconds?
> >>
> >> or
> >>
> >>  ceph osd ident-led-on osd.123  # turn on
> >>  ceph osd ident-led-off osd.123  # turn off
> >>  ceph osd fault-led-on osd.123  # turn on
> >>  ceph osd fault-led-off osd.123  # turn off
> >>
> >> This would mean persistently recording the LED state in the OSDMap.  And
> >> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
> >> be the way to go in all cases.  For example, if we have an OSD that fails,
> >> once we confirm that we've healed (and don't need that OSDs data) we'd
> >> probably want to set the fault light so that the disk can be pulled
> >> safely.  In that case, ceph-osd isn't running (it's long-since failed),
> >> and we'd need some other agent on the node to twiddle the light.  Do we
> >> really want multiple things twiddling lights?
> >
> >
> > Although this is really neat, I don't think ceph is the place to implement
> > or even manage it.
> >
> > This should fall under a management tool's purview.
> 
> I would agree that any persistence/policy parts don't belong in the
> OSD or the mons, but the remote access part where we reach across the
> network and do something with libstoragemgmt is very useful to have
> inside Ceph, because it means we can have management tools that just
> speak librados and don't have to have their own remote access to all
> the nodes in the system.
> 
> Put another way, I'd like it to be the case that people who want to
> twiddle LEDs can write a ceph-mgr module that does that, rather than
> having to push out a libstoragemgmt-enabled agent to the Ceph servers.

Major server vendors already provide tools to play with drive and/or server 
LEDs and many users of these machines already make use of these tools in
their own setups. Combining them with Ceph would be confusing if both these
tools and Ceph will try to manage lights.
On the other hand, I don't believe that libstoragemgmt is reliable enough to
cover enough users without bringing influx of Ceph bugs/issues that are
really a libstoragemgmt faults due to incompatibilities/bugs/lack of support.

-- 
Piotr Dałek
branch@predictor.org.pl
http://blog.predictor.org.pl

Re: disk enclosure LEDs

[Joao Eduardo Luis]

On 12/01/2016 12:15 AM, John Spray wrote:
> On Wed, Nov 30, 2016 at 7:37 PM, Joao Eduardo Luis <joao@suse.de> wrote:
>> On 11/30/2016 04:51 PM, Sage Weil wrote:
>>>
>>> Since this is in a reasonably usable state, I think It's time for us to
>>> figure out how we are going to do this in ceph.
>>>
>>> A few ideas:
>>>
>>>  ceph osd identify osd.123    # blink for a few seconds?
>>>
>>> or
>>>
>>>  ceph osd ident-led-on osd.123  # turn on
>>>  ceph osd ident-led-off osd.123  # turn off
>>>  ceph osd fault-led-on osd.123  # turn on
>>>  ceph osd fault-led-off osd.123  # turn off
>>>
>>> This would mean persistently recording the LED state in the OSDMap.  And
>>> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
>>> be the way to go in all cases.  For example, if we have an OSD that fails,
>>> once we confirm that we've healed (and don't need that OSDs data) we'd
>>> probably want to set the fault light so that the disk can be pulled
>>> safely.  In that case, ceph-osd isn't running (it's long-since failed),
>>> and we'd need some other agent on the node to twiddle the light.  Do we
>>> really want multiple things twiddling lights?
>>
>>
>> Although this is really neat, I don't think ceph is the place to implement
>> or even manage it.
>>
>> This should fall under a management tool's purview.
>
> I would agree that any persistence/policy parts don't belong in the
> OSD or the mons, but the remote access part where we reach across the
> network and do something with libstoragemgmt is very useful to have
> inside Ceph, because it means we can have management tools that just
> speak librados and don't have to have their own remote access to all
> the nodes in the system.
>
> Put another way, I'd like it to be the case that people who want to
> twiddle LEDs can write a ceph-mgr module that does that, rather than
> having to push out a libstoragemgmt-enabled agent to the Ceph servers.

One thing would be to provide some *optional* package with a small 
daemon, or even a plugin for osds, that would handle led twiddling for 
users if they so wished.

Another totally different thing is to tie this up to Ceph so tightly 
that you can't really have one without the other. That feels a lot like 
systemd's approach to do all things, regardless of whether you really 
want it doing them or not.

I don't believe Ceph should be in the LED twiddling business, no only 
because this should be a management/monitoring tool's business, but also 
because (as Piotr mentioned) we'd be opening a new vector for issues 
that should not have anything to do with Ceph.

But assuming that's not the prevailing opinion and this does go forth, 
at least make it pluggable enough that we don't end up depending on 
libstoragemgmt to build the osd, ceph-mgr or anything else.

   -Joao

Re: disk enclosure LEDs

[Lars Marowsky-Bree]

On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:

> Put another way, I'd like it to be the case that people who want to
> twiddle LEDs can write a ceph-mgr module that does that, rather than
> having to push out a libstoragemgmt-enabled agent to the Ceph servers.

I'm not sure I agree.

I understand some of the motivation, but the key use cases where I'd
want to "twiddle LEDs" involve enclosures that either don't yet have an
OSD deployed (e.g., where to plug in the new disk), or where the OSD has
failed (and may not even be able to start).

And I don't want to have to maintain multiple paths for interacting with
the enclosures.


-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

Re: disk enclosure LEDs

[Sage Weil]

On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
> On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
> 
> > Put another way, I'd like it to be the case that people who want to
> > twiddle LEDs can write a ceph-mgr module that does that, rather than
> > having to push out a libstoragemgmt-enabled agent to the Ceph servers.
> 
> I'm not sure I agree.
> 
> I understand some of the motivation, but the key use cases where I'd
> want to "twiddle LEDs" involve enclosures that either don't yet have an
> OSD deployed (e.g., where to plug in the new disk), or where the OSD has
> failed (and may not even be able to start).
> 
> And I don't want to have to maintain multiple paths for interacting with
> the enclosures.

I agree with John that it would be great to have ceph-mgr plugins doing 
this sort of thing.  But the OSD can't be the only path, and I also don't 
like the idea of multiple paths.

The problem is that there needs to be something on the hosts that's doing 
this, and I don't want to just ignore it and hope some other layer solves 
it; years have gone by and nobody has done it, and I'd like to 
avoid a fragmented approach if we can.

On the other hand, if we simply make an opinionated choice about some 
other per-host agent service, like we did in the original Calamari (which 
used salt and zeromq, IIRC), we'll probably end up offending more users 
than we please.

Perhaps the way forward is to pick *some* external host agent and service 
to start with and ensure that it is integrated into ceph-mgr in a 
pluggable way such that we have (1) the osd:device mappings available, (2) 
a standard API for working with LEDs etc, and (3) a working plugin for 
twiddling the LEDs, but ensure that other "disk services" backends can 
also be used?

sage

Re: disk enclosure LEDs

[Lars Marowsky-Bree]

On 2016-12-02T14:23:39, Sage Weil <sage@newdream.net> wrote:

> On the other hand, if we simply make an opinionated choice about some 
> other per-host agent service, like we did in the original Calamari (which 
> used salt and zeromq, IIRC), we'll probably end up offending more users 
> than we please.

> Perhaps the way forward is to pick *some* external host agent and service 
> to start with and ensure that it is integrated into ceph-mgr in a 
> pluggable way such that we have (1) the osd:device mappings available, (2) 
> a standard API for working with LEDs etc, and (3) a working plugin for 
> twiddling the LEDs, but ensure that other "disk services" backends can 
> also be used?

Yes, that makes sense. Making an opinionated choice about the API the
ceph-mgr plugin has and then we can patch in whatever transport we want.




-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

Re: disk enclosure LEDs

[Bassam Tabbara]

> On the other hand, if we simply make an opinionated choice about some 
> other per-host agent service, like we did in the original Calamari (which 
> used salt and zeromq, IIRC), we'll probably end up offending more users 
> than we please.

I think it fine for Ceph to have an opinionated story here, as long as it remains optional.
This would mean that we need to ensure a clean separation between “management” functions
(those that cater to administrators/users and are highly opinionated) and “data management” 
functions (those that ensure the data is stored safely in the cluster and are not opinionated).

> Perhaps the way forward is to pick *some* external host agent and service 
> to start with and ensure that it is integrated into ceph-mgr in a 
> pluggable way such that we have (1) the osd:device mappings available, (2) 
> a standard API for working with LEDs etc, and (3) a working plugin for 
> twiddling the LEDs, but ensure that other "disk services" backends can 
> also be used?

I believe there is a need for a ceph-mgr *agent* on every host, otherwise I don’t
see how one could implement some of the envisioned “management” functions for
ceph-mgr — this LED example is a good example. Turning the OSDs into this agent 
seems to violate the clean separation I mention above. If there was a ceph-mgr-agent 
I would recommend moving stat collection into it which currently has gone
into OSDs and MONs.

Bassam

RE: disk enclosure LEDs

[Allen Samuels]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, December 02, 2016 6:24 AM
> To: Lars Marowsky-Bree <lmb@suse.com>
> Cc: Ceph Development <ceph-devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
> > On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
> >
> > > Put another way, I'd like it to be the case that people who want to
> > > twiddle LEDs can write a ceph-mgr module that does that, rather than
> > > having to push out a libstoragemgmt-enabled agent to the Ceph servers.
> >
> > I'm not sure I agree.
> >
> > I understand some of the motivation, but the key use cases where I'd
> > want to "twiddle LEDs" involve enclosures that either don't yet have
> > an OSD deployed (e.g., where to plug in the new disk), or where the
> > OSD has failed (and may not even be able to start).
> >
> > And I don't want to have to maintain multiple paths for interacting
> > with the enclosures.
> 
> I agree with John that it would be great to have ceph-mgr plugins doing this
> sort of thing.  But the OSD can't be the only path, and I also don't like the idea
> of multiple paths.
> 
> The problem is that there needs to be something on the hosts that's doing
> this, and I don't want to just ignore it and hope some other layer solves it;
> years have gone by and nobody has done it, and I'd like to avoid a
> fragmented approach if we can.
> 
> On the other hand, if we simply make an opinionated choice about some
> other per-host agent service, like we did in the original Calamari (which used
> salt and zeromq, IIRC), we'll probably end up offending more users than we
> please.
> 
> Perhaps the way forward is to pick *some* external host agent and service
> to start with and ensure that it is integrated into ceph-mgr in a pluggable way
> such that we have (1) the osd:device mappings available, (2) a standard API
> for working with LEDs etc, and (3) a working plugin for twiddling the LEDs, but
> ensure that other "disk services" backends can also be used?

I think it's important to separate this problem into two pieces: ceph-specific and platform-specific.

The ceph-specific part of the problem is to create a single abstraction and mechanism so that upper-level tools (ceph-mgr, GUIs, workflow scripts, etc.) can manipulation the abstraction without worrying about the implementation details. I'm completely with John that ceph-mgr is the right place for the upper-layer to terminated. Ultimately, we'll have infrastructure that ties the physical state of the drives (power up/down, lights on/off, etc.) more closely to the operational state of the system -- this is an area where more standardized automation will remove a significant barrier to wide-spread deployments.

The platform-specific part definitely should be pluggable/configurable/soft/manipulable, etc. It'll be quite some time (if ever) before something like libstoragemgmt becomes a reliable, universal, standard interface. We can't let that hold back the development of the upper-level infrastructure stuff.

So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact the destination system (the OSD is *NOT* the right path) to perform the action. Then do something like invoke a configurable shell script (something that's configurable) to perform the actual operation. 

 
> 
> sage
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Lars Marowsky-Bree]

On 2016-12-02T16:57:22, Allen Samuels <Allen.Samuels@sandisk.com> wrote:

> So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact the destination system (the OSD is *NOT* the right path) to perform the action. Then do something like invoke a configurable shell script (something that's configurable) to perform the actual operation. 

+1

RE: disk enclosure LEDs

[Alan Johnson]

As a hardware vendor we would like some sort of API to hook into - we do have a proprietary way of identifying failing devices but would rather conform to something more standard within the Ceph arena.

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Allen Samuels
Sent: Friday, December 02, 2016 11:57 AM
To: Sage Weil; Lars Marowsky-Bree
Cc: Ceph Development
Subject: RE: disk enclosure LEDs

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
> owner@vger.kernel.org] On Behalf Of Sage Weil
> Sent: Friday, December 02, 2016 6:24 AM
> To: Lars Marowsky-Bree <lmb@suse.com>
> Cc: Ceph Development <ceph-devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
> > On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
> >
> > > Put another way, I'd like it to be the case that people who want 
> > > to twiddle LEDs can write a ceph-mgr module that does that, rather 
> > > than having to push out a libstoragemgmt-enabled agent to the Ceph servers.
> >
> > I'm not sure I agree.
> >
> > I understand some of the motivation, but the key use cases where I'd 
> > want to "twiddle LEDs" involve enclosures that either don't yet have 
> > an OSD deployed (e.g., where to plug in the new disk), or where the 
> > OSD has failed (and may not even be able to start).
> >
> > And I don't want to have to maintain multiple paths for interacting 
> > with the enclosures.
> 
> I agree with John that it would be great to have ceph-mgr plugins 
> doing this sort of thing.  But the OSD can't be the only path, and I 
> also don't like the idea of multiple paths.
> 
> The problem is that there needs to be something on the hosts that's 
> doing this, and I don't want to just ignore it and hope some other 
> layer solves it; years have gone by and nobody has done it, and I'd 
> like to avoid a fragmented approach if we can.
> 
> On the other hand, if we simply make an opinionated choice about some 
> other per-host agent service, like we did in the original Calamari 
> (which used salt and zeromq, IIRC), we'll probably end up offending 
> more users than we please.
> 
> Perhaps the way forward is to pick *some* external host agent and 
> service to start with and ensure that it is integrated into ceph-mgr 
> in a pluggable way such that we have (1) the osd:device mappings 
> available, (2) a standard API for working with LEDs etc, and (3) a 
> working plugin for twiddling the LEDs, but ensure that other "disk services" backends can also be used?

I think it's important to separate this problem into two pieces: ceph-specific and platform-specific.

The ceph-specific part of the problem is to create a single abstraction and mechanism so that upper-level tools (ceph-mgr, GUIs, workflow scripts, etc.) can manipulation the abstraction without worrying about the implementation details. I'm completely with John that ceph-mgr is the right place for the upper-layer to terminated. Ultimately, we'll have infrastructure that ties the physical state of the drives (power up/down, lights on/off, etc.) more closely to the operational state of the system -- this is an area where more standardized automation will remove a significant barrier to wide-spread deployments.

The platform-specific part definitely should be pluggable/configurable/soft/manipulable, etc. It'll be quite some time (if ever) before something like libstoragemgmt becomes a reliable, universal, standard interface. We can't let that hold back the development of the upper-level infrastructure stuff.

So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact the destination system (the OSD is *NOT* the right path) to perform the action. Then do something like invoke a configurable shell script (something that's configurable) to perform the actual operation. 

 
> 
> sage
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: disk enclosure LEDs

[Allen Samuels]

I agree. For the HW vendors, I think it's Ceph's job to execute some kind of command on the local node (presumably, some kind of shell command). The vendor should take over from there.

I'd assume that one version of the shell command would be a simple wrapper around libstoragemgmt. So you'd have your choice of either extending libstoragemgmt, or writing a command-line utility that did your special thing (whatever it was).


Allen Samuels
SanDisk |a Western Digital brand
2880 Junction Avenue, San Jose, CA 95134
T: +1 408 801 7030| M: +1 408 780 6416
allen.samuels@SanDisk.com


> -----Original Message-----
> From: Alan Johnson [mailto:alanj@supermicro.com]
> Sent: Friday, December 02, 2016 9:57 AM
> To: Allen Samuels <Allen.Samuels@sandisk.com>; Sage Weil
> <sage@newdream.net>; Lars Marowsky-Bree <lmb@suse.com>
> Cc: Ceph Development <ceph-devel@vger.kernel.org>; Alex Yin
> <alexy@supermicro.com>
> Subject: RE: disk enclosure LEDs
> 
> As a hardware vendor we would like some sort of API to hook into - we do
> have a proprietary way of identifying failing devices but would rather
> conform to something more standard within the Ceph arena.
> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Allen Samuels
> Sent: Friday, December 02, 2016 11:57 AM
> To: Sage Weil; Lars Marowsky-Bree
> Cc: Ceph Development
> Subject: RE: disk enclosure LEDs
> 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> > owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, December 02, 2016 6:24 AM
> > To: Lars Marowsky-Bree <lmb@suse.com>
> > Cc: Ceph Development <ceph-devel@vger.kernel.org>
> > Subject: Re: disk enclosure LEDs
> >
> > On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
> > > On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
> > >
> > > > Put another way, I'd like it to be the case that people who want
> > > > to twiddle LEDs can write a ceph-mgr module that does that, rather
> > > > than having to push out a libstoragemgmt-enabled agent to the Ceph
> servers.
> > >
> > > I'm not sure I agree.
> > >
> > > I understand some of the motivation, but the key use cases where I'd
> > > want to "twiddle LEDs" involve enclosures that either don't yet have
> > > an OSD deployed (e.g., where to plug in the new disk), or where the
> > > OSD has failed (and may not even be able to start).
> > >
> > > And I don't want to have to maintain multiple paths for interacting
> > > with the enclosures.
> >
> > I agree with John that it would be great to have ceph-mgr plugins
> > doing this sort of thing.  But the OSD can't be the only path, and I
> > also don't like the idea of multiple paths.
> >
> > The problem is that there needs to be something on the hosts that's
> > doing this, and I don't want to just ignore it and hope some other
> > layer solves it; years have gone by and nobody has done it, and I'd
> > like to avoid a fragmented approach if we can.
> >
> > On the other hand, if we simply make an opinionated choice about some
> > other per-host agent service, like we did in the original Calamari
> > (which used salt and zeromq, IIRC), we'll probably end up offending
> > more users than we please.
> >
> > Perhaps the way forward is to pick *some* external host agent and
> > service to start with and ensure that it is integrated into ceph-mgr
> > in a pluggable way such that we have (1) the osd:device mappings
> > available, (2) a standard API for working with LEDs etc, and (3) a
> > working plugin for twiddling the LEDs, but ensure that other "disk services"
> backends can also be used?
> 
> I think it's important to separate this problem into two pieces: ceph-specific
> and platform-specific.
> 
> The ceph-specific part of the problem is to create a single abstraction and
> mechanism so that upper-level tools (ceph-mgr, GUIs, workflow scripts, etc.)
> can manipulation the abstraction without worrying about the
> implementation details. I'm completely with John that ceph-mgr is the right
> place for the upper-layer to terminated. Ultimately, we'll have infrastructure
> that ties the physical state of the drives (power up/down, lights on/off, etc.)
> more closely to the operational state of the system -- this is an area where
> more standardized automation will remove a significant barrier to wide-
> spread deployments.
> 
> The platform-specific part definitely should be
> pluggable/configurable/soft/manipulable, etc. It'll be quite some time (if
> ever) before something like libstoragemgmt becomes a reliable, universal,
> standard interface. We can't let that hold back the development of the
> upper-level infrastructure stuff.
> 
> So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact
> the destination system (the OSD is *NOT* the right path) to perform the
> action. Then do something like invoke a configurable shell script (something
> that's configurable) to perform the actual operation.
> 
> 
> >
> > sage
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[John Spray]

On Fri, Dec 2, 2016 at 6:05 PM, Allen Samuels <Allen.Samuels@sandisk.com> wrote:
> I agree. For the HW vendors, I think it's Ceph's job to execute some kind of command on the local node (presumably, some kind of shell command). The vendor should take over from there.

Assuming that we want Ceph to act as a channel for remote execution,
there are two approaches:
 * Put a ceph-supervisor service on all our nodes, and use that to
terminate the remote commands.  This daemon would also be the place
for reporting per-host stats that we currently send multiple times
through each OSD.
 * Build a SSH/salt remote execution client into ceph-mgr, and modify
installation tools to store a privileged key in Ceph so that it can
remote to the nodes.

Both of these are possible.  They are essentially agent vs. agentless
approaches.

One can argue either way:
 * "sshd already exists, why would we write our own daemon that
enables remote execution?"
 * "we already have our own protocol and crypto, adding remote
execution is simple so why would we bring in a whole separate set of
keys?"

I have a preference for creating an agent, because we would use it for
monitoring as well as for remote execution.  However, this is really
only useful if somebody wants to use it: when management tools have
their own agents, it's likely they won't use whatever we build into
Ceph.

Question to the OpenAttic folks, who afaik are the only ones on the
thread actively building tools like this: if ceph-mgr modules could do
arbitrary command remote execution on OSD nodes, is that what you
would use for blinking LEDs?  How would you wire that up to the rest
of your stack?

John

> I'd assume that one version of the shell command would be a simple wrapper around libstoragemgmt. So you'd have your choice of either extending libstoragemgmt, or writing a command-line utility that did your special thing (whatever it was).
>
>
> Allen Samuels
> SanDisk |a Western Digital brand
> 2880 Junction Avenue, San Jose, CA 95134
> T: +1 408 801 7030| M: +1 408 780 6416
> allen.samuels@SanDisk.com
>
>
>> -----Original Message-----
>> From: Alan Johnson [mailto:alanj@supermicro.com]
>> Sent: Friday, December 02, 2016 9:57 AM
>> To: Allen Samuels <Allen.Samuels@sandisk.com>; Sage Weil
>> <sage@newdream.net>; Lars Marowsky-Bree <lmb@suse.com>
>> Cc: Ceph Development <ceph-devel@vger.kernel.org>; Alex Yin
>> <alexy@supermicro.com>
>> Subject: RE: disk enclosure LEDs
>>
>> As a hardware vendor we would like some sort of API to hook into - we do
>> have a proprietary way of identifying failing devices but would rather
>> conform to something more standard within the Ceph arena.
>>
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of Allen Samuels
>> Sent: Friday, December 02, 2016 11:57 AM
>> To: Sage Weil; Lars Marowsky-Bree
>> Cc: Ceph Development
>> Subject: RE: disk enclosure LEDs
>>
>> > -----Original Message-----
>> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> > owner@vger.kernel.org] On Behalf Of Sage Weil
>> > Sent: Friday, December 02, 2016 6:24 AM
>> > To: Lars Marowsky-Bree <lmb@suse.com>
>> > Cc: Ceph Development <ceph-devel@vger.kernel.org>
>> > Subject: Re: disk enclosure LEDs
>> >
>> > On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
>> > > On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
>> > >
>> > > > Put another way, I'd like it to be the case that people who want
>> > > > to twiddle LEDs can write a ceph-mgr module that does that, rather
>> > > > than having to push out a libstoragemgmt-enabled agent to the Ceph
>> servers.
>> > >
>> > > I'm not sure I agree.
>> > >
>> > > I understand some of the motivation, but the key use cases where I'd
>> > > want to "twiddle LEDs" involve enclosures that either don't yet have
>> > > an OSD deployed (e.g., where to plug in the new disk), or where the
>> > > OSD has failed (and may not even be able to start).
>> > >
>> > > And I don't want to have to maintain multiple paths for interacting
>> > > with the enclosures.
>> >
>> > I agree with John that it would be great to have ceph-mgr plugins
>> > doing this sort of thing.  But the OSD can't be the only path, and I
>> > also don't like the idea of multiple paths.
>> >
>> > The problem is that there needs to be something on the hosts that's
>> > doing this, and I don't want to just ignore it and hope some other
>> > layer solves it; years have gone by and nobody has done it, and I'd
>> > like to avoid a fragmented approach if we can.
>> >
>> > On the other hand, if we simply make an opinionated choice about some
>> > other per-host agent service, like we did in the original Calamari
>> > (which used salt and zeromq, IIRC), we'll probably end up offending
>> > more users than we please.
>> >
>> > Perhaps the way forward is to pick *some* external host agent and
>> > service to start with and ensure that it is integrated into ceph-mgr
>> > in a pluggable way such that we have (1) the osd:device mappings
>> > available, (2) a standard API for working with LEDs etc, and (3) a
>> > working plugin for twiddling the LEDs, but ensure that other "disk services"
>> backends can also be used?
>>
>> I think it's important to separate this problem into two pieces: ceph-specific
>> and platform-specific.
>>
>> The ceph-specific part of the problem is to create a single abstraction and
>> mechanism so that upper-level tools (ceph-mgr, GUIs, workflow scripts, etc.)
>> can manipulation the abstraction without worrying about the
>> implementation details. I'm completely with John that ceph-mgr is the right
>> place for the upper-layer to terminated. Ultimately, we'll have infrastructure
>> that ties the physical state of the drives (power up/down, lights on/off, etc.)
>> more closely to the operational state of the system -- this is an area where
>> more standardized automation will remove a significant barrier to wide-
>> spread deployments.
>>
>> The platform-specific part definitely should be
>> pluggable/configurable/soft/manipulable, etc. It'll be quite some time (if
>> ever) before something like libstoragemgmt becomes a reliable, universal,
>> standard interface. We can't let that hold back the development of the
>> upper-level infrastructure stuff.
>>
>> So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact
>> the destination system (the OSD is *NOT* the right path) to perform the
>> action. Then do something like invoke a configurable shell script (something
>> that's configurable) to perform the actual operation.
>>
>>
>> >
>> > sage
>> >
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> > in the body of a message to majordomo@vger.kernel.org More
>> majordomo
>> > info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>> body of a message to majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Lars Marowsky-Bree]

On 2016-12-05T12:44:51, John Spray <jspray@redhat.com> wrote:

> Question to the OpenAttic folks, who afaik are the only ones on the
> thread actively building tools like this: if ceph-mgr modules could do
> arbitrary command remote execution on OSD nodes, is that what you
> would use for blinking LEDs?  How would you wire that up to the rest
> of your stack?

So, in "our" world, we have Salt minions for remote execution
everywhere.

If Ceph requires an additional remote execution channel, aren't we
rebuilding work that's part of both Salt, Ansible, Puppet, ...?

The advantage of having a ceph remote agent is most prominent if you
view ceph as an isolated distributed system.


Regards,
    Lars

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

Re: disk enclosure LEDs

[Jeff Applewhite]

In the case of Tendrl we have Ceph node agents that communicate with
an etcd central store for all operations. In this particular case I
think we would likely trigger the local node agent to run a local
ansible playbook to execute the needed code (via etcd). So an API to
consume these events from Ceph efficiently would be the enabling
technology we'd be interested in.

Jeff

On Mon, Dec 5, 2016 at 7:58 AM, Lars Marowsky-Bree <lmb@suse.com> wrote:
> On 2016-12-05T12:44:51, John Spray <jspray@redhat.com> wrote:
>
>> Question to the OpenAttic folks, who afaik are the only ones on the
>> thread actively building tools like this: if ceph-mgr modules could do
>> arbitrary command remote execution on OSD nodes, is that what you
>> would use for blinking LEDs?  How would you wire that up to the rest
>> of your stack?
>
> So, in "our" world, we have Salt minions for remote execution
> everywhere.
>
> If Ceph requires an additional remote execution channel, aren't we
> rebuilding work that's part of both Salt, Ansible, Puppet, ...?
>
> The advantage of having a ceph remote agent is most prominent if you
> view ceph as an isolated distributed system.
>
>
> Regards,
>     Lars
>
> --
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Jeff Applewhite
Principal Product Manager

Re: disk enclosure LEDs

[John Spray]

On Mon, Dec 5, 2016 at 12:58 PM, Lars Marowsky-Bree <lmb@suse.com> wrote:
> On 2016-12-05T12:44:51, John Spray <jspray@redhat.com> wrote:
>
>> Question to the OpenAttic folks, who afaik are the only ones on the
>> thread actively building tools like this: if ceph-mgr modules could do
>> arbitrary command remote execution on OSD nodes, is that what you
>> would use for blinking LEDs?  How would you wire that up to the rest
>> of your stack?
>
> So, in "our" world, we have Salt minions for remote execution
> everywhere.
>
> If Ceph requires an additional remote execution channel, aren't we
> rebuilding work that's part of both Salt, Ansible, Puppet, ...?

Yes, although only a tiny part of what tools like ansible do -- and in
the process, we save the effort of integrating with all of the above.
IMHO it would actually be better value for money to write our own
agent than to maintain and test a pluggable infrastructure that could
use any of them (especially as some have different semantics), and
handle newer/older versions of them, etc.

John

>
> The advantage of having a ceph remote agent is most prominent if you
> view ceph as an isolated distributed system.
>
>
> Regards,
>     Lars
>
> --
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Bassam Tabbara]

> Yes, although only a tiny part of what tools like ansible do -- and in
> the process, we save the effort of integrating with all of the above.
> IMHO it would actually be better value for money to write our own
> agent than to maintain and test a pluggable infrastructure that could
> use any of them (especially as some have different semantics), and
> handle newer/older versions of them, etc.

Calmari was using salt stack (among other things). I agree with John that Ceph should 
have its own simple story for management. I just hope that it remains *optional* as
we’ve seen on this thread there are numerous approaches to this problem. Its also
not immediately obvious that the different solutions could be layered.

Should this be a topic for the next CDM?

RE: disk enclosure LEDs

[Allen Samuels]

I'm indifferent to agent vs. agent-less.

I *believe* that having a ceph-private distribution is easier/simpler/more reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I agree with John]. But this isn't a strongly held belief.

I'm *metaphysically certain* that whatever distribution scheme is adopted that it not be optional. A large barrier to adoption of Ceph today is the lack of "middle-ware" that handles infrequent operational events (node addition/removal, media failure/recovery, migration, etc.). IMO, this middle-ware will have to be a standard part of Ceph, i.e., fully functional "out of the  box" without site-specific twiddling (though having a mechanism to insert site-specific stuff is fine with me, it just can't be *required*). 

In my mind, the distribution scheme is the next step in the evolution of Ceph-mgr. It's what's missing :)


> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of Bassam Tabbara
> Sent: Monday, December 05, 2016 9:48 AM
> To: John Spray <jspray@redhat.com>
> Cc: Lars Marowsky-Bree <lmb@suse.com>; Ceph Development <ceph-
> devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> > Yes, although only a tiny part of what tools like ansible do -- and in
> > the process, we save the effort of integrating with all of the above.
> > IMHO it would actually be better value for money to write our own
> > agent than to maintain and test a pluggable infrastructure that could
> > use any of them (especially as some have different semantics), and
> > handle newer/older versions of them, etc.
> 
> Calmari was using salt stack (among other things). I agree with John that
> Ceph should have its own simple story for management. I just hope that it
> remains *optional* as we’ve seen on this thread there are numerous
> approaches to this problem. Its also not immediately obvious that the
> different solutions could be layered.
> 
> Should this be a topic for the next CDM?
> 
> 
> \x04 {.n +       +%  lzwm  b 맲  r  yǩ ׯzX  \x17  ܨ}   Ơz &j:+v        zZ+  +zf   h   ~    i   z \x1e w
> ?    & )ߢ^[f

Re: disk enclosure LEDs

[Lars Marowsky-Bree]

On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@sandisk.com> wrote:

> I'm indifferent to agent vs. agent-less.
> 
> I *believe* that having a ceph-private distribution is easier/simpler/more reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I agree with John]. But this isn't a strongly held belief.
> 
> I'm *metaphysically certain* that whatever distribution scheme is adopted that it not be optional. A large barrier to adoption of Ceph today is the lack of "middle-ware" that handles infrequent operational events (node addition/removal, media failure/recovery, migration, etc.). IMO, this middle-ware will have to be a standard part of Ceph, i.e., fully functional "out of the  box" without site-specific twiddling (though having a mechanism to insert site-specific stuff is fine with me, it just can't be *required*). 
> 
> In my mind, the distribution scheme is the next step in the evolution of Ceph-mgr. It's what's missing :)

I see the benefits of having a ceph-specific agent for hardware
interaction. However, that then shifts the problem for bootstrapping
said Ceph agent.

And when you open the can of worms that is server addition/removal, etc
we start hitting the question of either spinning up a distribution
mechanism as well.

When we want to look at container-izing Ceph in hyper-converged
environments, this gets even worse.

e.g., the cephalopod turns into a cephaloblob.  (Sorry. I'm terrible
with puns.)

I need a mechanism for interacting with enclosures (to stick with the
example), but I don't need it to be part of Ceph, since I need it for
other parts of my infrastructure too anyway.

If it's part of Ceph, I end up writing a special case for Ceph.

And I need a way to handle it when Ceph itself isn't around yet; how do
I blink an enclosure that receives a new disk? Ah, I pre-register a
given enclosure with Ceph, before an OSD is even created. I know Ceph
has many tentacles, but ... ;-)


Regards,
    Lars

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

Re: disk enclosure LEDs

[John Spray]

On Mon, Dec 5, 2016 at 6:41 PM, Lars Marowsky-Bree <lmb@suse.com> wrote:
> On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@sandisk.com> wrote:
>
>> I'm indifferent to agent vs. agent-less.
>>
>> I *believe* that having a ceph-private distribution is easier/simpler/more reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I agree with John]. But this isn't a strongly held belief.
>>
>> I'm *metaphysically certain* that whatever distribution scheme is adopted that it not be optional. A large barrier to adoption of Ceph today is the lack of "middle-ware" that handles infrequent operational events (node addition/removal, media failure/recovery, migration, etc.). IMO, this middle-ware will have to be a standard part of Ceph, i.e., fully functional "out of the  box" without site-specific twiddling (though having a mechanism to insert site-specific stuff is fine with me, it just can't be *required*).
>>
>> In my mind, the distribution scheme is the next step in the evolution of Ceph-mgr. It's what's missing :)
>
> I see the benefits of having a ceph-specific agent for hardware
> interaction. However, that then shifts the problem for bootstrapping
> said Ceph agent.

Bootstrapping would be the same as we already have for installing OSDs
and MDSs.  So ceph-deploy/ceph-ansible/whatever needs to be able to do
the same thing for the per-host agent that it currently does for OSDs,
no overall increase in complexity.

> And when you open the can of worms that is server addition/removal, etc
> we start hitting the question of either spinning up a distribution
> mechanism as well.
>
> When we want to look at container-izing Ceph in hyper-converged
> environments, this gets even worse.

I'm imagining that in a container-per-service model, where something
external has configured the OSD containers to have access to the block
device that they will run on, it doesn't seem unreasonable to have the
same configuration process set up the ceph agent container with access
to all the OSD block devices.  What are your thoughts about how this
would (or wouldn't) work?

>
> e.g., the cephalopod turns into a cephaloblob.  (Sorry. I'm terrible
> with puns.)
>
> I need a mechanism for interacting with enclosures (to stick with the
> example), but I don't need it to be part of Ceph, since I need it for
> other parts of my infrastructure too anyway.
>
>
> If it's part of Ceph, I end up writing a special case for Ceph.

I think this would cease to be a problem for you if we just had a flag
in Ceph to disable its own smartmontools type stuff?  That way when
someone was using an external tool there would be no conflict.

There is some duplication of effort, but I don't think that's
intrinsically problematic: I predict that we'll always have many users
who do not take up any of the external tools and will benefit from the
built-in Ceph bits.

> And I need a way to handle it when Ceph itself isn't around yet; how do
> I blink an enclosure that receives a new disk? Ah, I pre-register a
> given enclosure with Ceph, before an OSD is even created. I know Ceph
> has many tentacles, but ... ;-)

While at runtime we shouldn't have two agents competing to manage the
same device, I think it is reasonable to have a separate piece of
software that does installation vs. does the ongoing monitoring.  We
shouldn't let the constraints over installation (especially the need
to operate on cephless machines) restrict how we manage systems
through their life cycles.  Again, I don't think the built-in Ceph
functionality is mutually exclusive with having a good external
installation tool that touches some of the same functionality.

John


>
>
> Regards,
>     Lars
>
> --
> SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
> "Experience is the name everyone gives to their mistakes." -- Oscar Wilde
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: disk enclosure LEDs

[Allen Samuels]

> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> owner@vger.kernel.org] On Behalf Of John Spray
> Sent: Monday, December 05, 2016 11:28 AM
> To: Lars Marowsky-Bree <lmb@suse.com>
> Cc: Ceph Development <ceph-devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> On Mon, Dec 5, 2016 at 6:41 PM, Lars Marowsky-Bree <lmb@suse.com>
> wrote:
> > On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@sandisk.com>
> wrote:
> >
> >> I'm indifferent to agent vs. agent-less.
> >>
> >> I *believe* that having a ceph-private distribution is easier/simpler/more
> reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I
> agree with John]. But this isn't a strongly held belief.
> >>
> >> I'm *metaphysically certain* that whatever distribution scheme is
> adopted that it not be optional. A large barrier to adoption of Ceph today is
> the lack of "middle-ware" that handles infrequent operational events (node
> addition/removal, media failure/recovery, migration, etc.). IMO, this middle-
> ware will have to be a standard part of Ceph, i.e., fully functional "out of the
> box" without site-specific twiddling (though having a mechanism to insert
> site-specific stuff is fine with me, it just can't be *required*).
> >>
> >> In my mind, the distribution scheme is the next step in the evolution
> >> of Ceph-mgr. It's what's missing :)
> >
> > I see the benefits of having a ceph-specific agent for hardware
> > interaction. However, that then shifts the problem for bootstrapping
> > said Ceph agent.
> 
> Bootstrapping would be the same as we already have for installing OSDs and
> MDSs.  So ceph-deploy/ceph-ansible/whatever needs to be able to do the
> same thing for the per-host agent that it currently does for OSDs, no overall
> increase in complexity.
> 
> > And when you open the can of worms that is server addition/removal,
> > etc we start hitting the question of either spinning up a distribution
> > mechanism as well.
> >
> > When we want to look at container-izing Ceph in hyper-converged
> > environments, this gets even worse.
> 
> I'm imagining that in a container-per-service model, where something
> external has configured the OSD containers to have access to the block
> device that they will run on, it doesn't seem unreasonable to have the same
> configuration process set up the ceph agent container with access to all the
> OSD block devices.  What are your thoughts about how this would (or
> wouldn't) work?

The current OSD design is per-drive and not-reliable. We need a piece of software, running on the target system, that's NOT per-drive and NOT not-reliable (i.e., reliable :)). We need the management system to be able to dig out of the OSD's system why it crashed -- i.e., read logs and other types of status, etc. It's possible to mutate the OSD there, but I don't think it's easy nor soon.
> 
> >
> > e.g., the cephalopod turns into a cephaloblob.  (Sorry. I'm terrible
> > with puns.)
> >
> > I need a mechanism for interacting with enclosures (to stick with the
> > example), but I don't need it to be part of Ceph, since I need it for
> > other parts of my infrastructure too anyway.
> >
> >
> > If it's part of Ceph, I end up writing a special case for Ceph.
> 
> I think this would cease to be a problem for you if we just had a flag in Ceph
> to disable its own smartmontools type stuff?  That way when someone was
> using an external tool there would be no conflict.
> 
> There is some duplication of effort, but I don't think that's intrinsically
> problematic: I predict that we'll always have many users who do not take up
> any of the external tools and will benefit from the built-in Ceph bits.
> 
> > And I need a way to handle it when Ceph itself isn't around yet; how
> > do I blink an enclosure that receives a new disk? Ah, I pre-register a
> > given enclosure with Ceph, before an OSD is even created. I know Ceph
> > has many tentacles, but ... ;-)
> 
> While at runtime we shouldn't have two agents competing to manage the
> same device, I think it is reasonable to have a separate piece of software that
> does installation vs. does the ongoing monitoring.  We shouldn't let the
> constraints over installation (especially the need to operate on cephless
> machines) restrict how we manage systems through their life cycles.  Again, I
> don't think the built-in Ceph functionality is mutually exclusive with having a
> good external installation tool that touches some of the same functionality.
> 
> John
> 
> 
> >
> >
> > Regards,
> >     Lars
> >
> > --
> > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
> > HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to
> > their mistakes." -- Oscar Wilde
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> > in the body of a message to majordomo@vger.kernel.org More
> majordomo
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
> body of a message to majordomo@vger.kernel.org More majordomo info at
> http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[John Spray]

On Mon, Dec 5, 2016 at 10:50 PM, Allen Samuels
<Allen.Samuels@sandisk.com> wrote:
>> -----Original Message-----
>> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
>> owner@vger.kernel.org] On Behalf Of John Spray
>> Sent: Monday, December 05, 2016 11:28 AM
>> To: Lars Marowsky-Bree <lmb@suse.com>
>> Cc: Ceph Development <ceph-devel@vger.kernel.org>
>> Subject: Re: disk enclosure LEDs
>>
>> On Mon, Dec 5, 2016 at 6:41 PM, Lars Marowsky-Bree <lmb@suse.com>
>> wrote:
>> > On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@sandisk.com>
>> wrote:
>> >
>> >> I'm indifferent to agent vs. agent-less.
>> >>
>> >> I *believe* that having a ceph-private distribution is easier/simpler/more
>> reliable than trying to layer over some other system (ansible, salt, etc.) [i.e., I
>> agree with John]. But this isn't a strongly held belief.
>> >>
>> >> I'm *metaphysically certain* that whatever distribution scheme is
>> adopted that it not be optional. A large barrier to adoption of Ceph today is
>> the lack of "middle-ware" that handles infrequent operational events (node
>> addition/removal, media failure/recovery, migration, etc.). IMO, this middle-
>> ware will have to be a standard part of Ceph, i.e., fully functional "out of the
>> box" without site-specific twiddling (though having a mechanism to insert
>> site-specific stuff is fine with me, it just can't be *required*).
>> >>
>> >> In my mind, the distribution scheme is the next step in the evolution
>> >> of Ceph-mgr. It's what's missing :)
>> >
>> > I see the benefits of having a ceph-specific agent for hardware
>> > interaction. However, that then shifts the problem for bootstrapping
>> > said Ceph agent.
>>
>> Bootstrapping would be the same as we already have for installing OSDs and
>> MDSs.  So ceph-deploy/ceph-ansible/whatever needs to be able to do the
>> same thing for the per-host agent that it currently does for OSDs, no overall
>> increase in complexity.
>>
>> > And when you open the can of worms that is server addition/removal,
>> > etc we start hitting the question of either spinning up a distribution
>> > mechanism as well.
>> >
>> > When we want to look at container-izing Ceph in hyper-converged
>> > environments, this gets even worse.
>>
>> I'm imagining that in a container-per-service model, where something
>> external has configured the OSD containers to have access to the block
>> device that they will run on, it doesn't seem unreasonable to have the same
>> configuration process set up the ceph agent container with access to all the
>> OSD block devices.  What are your thoughts about how this would (or
>> wouldn't) work?
>
> The current OSD design is per-drive and not-reliable. We need a piece of software, running on the target system, that's NOT per-drive and NOT not-reliable (i.e., reliable :)). We need the management system to be able to dig out of the OSD's system why it crashed -- i.e., read logs and other types of status, etc. It's possible to mutate the OSD there, but I don't think it's easy nor soon.

I think I've lost you there -- what's the relation between what you've
just said and the issue of containerisation?

John

>>
>> >
>> > e.g., the cephalopod turns into a cephaloblob.  (Sorry. I'm terrible
>> > with puns.)
>> >
>> > I need a mechanism for interacting with enclosures (to stick with the
>> > example), but I don't need it to be part of Ceph, since I need it for
>> > other parts of my infrastructure too anyway.
>> >
>> >
>> > If it's part of Ceph, I end up writing a special case for Ceph.
>>
>> I think this would cease to be a problem for you if we just had a flag in Ceph
>> to disable its own smartmontools type stuff?  That way when someone was
>> using an external tool there would be no conflict.
>>
>> There is some duplication of effort, but I don't think that's intrinsically
>> problematic: I predict that we'll always have many users who do not take up
>> any of the external tools and will benefit from the built-in Ceph bits.
>>
>> > And I need a way to handle it when Ceph itself isn't around yet; how
>> > do I blink an enclosure that receives a new disk? Ah, I pre-register a
>> > given enclosure with Ceph, before an OSD is even created. I know Ceph
>> > has many tentacles, but ... ;-)
>>
>> While at runtime we shouldn't have two agents competing to manage the
>> same device, I think it is reasonable to have a separate piece of software that
>> does installation vs. does the ongoing monitoring.  We shouldn't let the
>> constraints over installation (especially the need to operate on cephless
>> machines) restrict how we manage systems through their life cycles.  Again, I
>> don't think the built-in Ceph functionality is mutually exclusive with having a
>> good external installation tool that touches some of the same functionality.
>>
>> John
>>
>>
>> >
>> >
>> > Regards,
>> >     Lars
>> >
>> > --
>> > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
>> > HRB 21284 (AG Nürnberg) "Experience is the name everyone gives to
>> > their mistakes." -- Oscar Wilde
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> > in the body of a message to majordomo@vger.kernel.org More
>> majordomo
>> > info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the
>> body of a message to majordomo@vger.kernel.org More majordomo info at
>> http://vger.kernel.org/majordomo-info.html

RE: disk enclosure LEDs

[Allen Samuels]

> -----Original Message-----
> From: John Spray [mailto:jspray@redhat.com]
> Sent: Monday, December 05, 2016 4:20 PM
> To: Allen Samuels <Allen.Samuels@sandisk.com>
> Cc: Lars Marowsky-Bree <lmb@suse.com>; Ceph Development <ceph-
> devel@vger.kernel.org>
> Subject: Re: disk enclosure LEDs
> 
> On Mon, Dec 5, 2016 at 10:50 PM, Allen Samuels
> <Allen.Samuels@sandisk.com> wrote:
> >> -----Original Message-----
> >> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-
> >> owner@vger.kernel.org] On Behalf Of John Spray
> >> Sent: Monday, December 05, 2016 11:28 AM
> >> To: Lars Marowsky-Bree <lmb@suse.com>
> >> Cc: Ceph Development <ceph-devel@vger.kernel.org>
> >> Subject: Re: disk enclosure LEDs
> >>
> >> On Mon, Dec 5, 2016 at 6:41 PM, Lars Marowsky-Bree <lmb@suse.com>
> >> wrote:
> >> > On 2016-12-05T18:02:08, Allen Samuels <Allen.Samuels@sandisk.com>
> >> wrote:
> >> >
> >> >> I'm indifferent to agent vs. agent-less.
> >> >>
> >> >> I *believe* that having a ceph-private distribution is
> >> >> easier/simpler/more
> >> reliable than trying to layer over some other system (ansible, salt,
> >> etc.) [i.e., I agree with John]. But this isn't a strongly held belief.
> >> >>
> >> >> I'm *metaphysically certain* that whatever distribution scheme is
> >> adopted that it not be optional. A large barrier to adoption of Ceph
> >> today is the lack of "middle-ware" that handles infrequent
> >> operational events (node addition/removal, media failure/recovery,
> >> migration, etc.). IMO, this middle- ware will have to be a standard
> >> part of Ceph, i.e., fully functional "out of the box" without
> >> site-specific twiddling (though having a mechanism to insert site-specific
> stuff is fine with me, it just can't be *required*).
> >> >>
> >> >> In my mind, the distribution scheme is the next step in the
> >> >> evolution of Ceph-mgr. It's what's missing :)
> >> >
> >> > I see the benefits of having a ceph-specific agent for hardware
> >> > interaction. However, that then shifts the problem for
> >> > bootstrapping said Ceph agent.
> >>
> >> Bootstrapping would be the same as we already have for installing
> >> OSDs and MDSs.  So ceph-deploy/ceph-ansible/whatever needs to be
> able
> >> to do the same thing for the per-host agent that it currently does
> >> for OSDs, no overall increase in complexity.
> >>
> >> > And when you open the can of worms that is server addition/removal,
> >> > etc we start hitting the question of either spinning up a
> >> > distribution mechanism as well.
> >> >
> >> > When we want to look at container-izing Ceph in hyper-converged
> >> > environments, this gets even worse.
> >>
> >> I'm imagining that in a container-per-service model, where something
> >> external has configured the OSD containers to have access to the
> >> block device that they will run on, it doesn't seem unreasonable to
> >> have the same configuration process set up the ceph agent container
> >> with access to all the OSD block devices.  What are your thoughts
> >> about how this would (or
> >> wouldn't) work?
> >
> > The current OSD design is per-drive and not-reliable. We need a piece of
> software, running on the target system, that's NOT per-drive and NOT not-
> reliable (i.e., reliable :)). We need the management system to be able to dig
> out of the OSD's system why it crashed -- i.e., read logs and other types of
> status, etc. It's possible to mutate the OSD there, but I don't think it's easy
> nor soon.
> 
> I think I've lost you there -- what's the relation between what you've just
> said and the issue of containerisation?

Perhaps none, just that the container world tends to want to ignore box boundaries, storage management doesn't have that luxury.

> 
> John
> 
> >>
> >> >
> >> > e.g., the cephalopod turns into a cephaloblob.  (Sorry. I'm
> >> > terrible with puns.)
> >> >
> >> > I need a mechanism for interacting with enclosures (to stick with
> >> > the example), but I don't need it to be part of Ceph, since I need
> >> > it for other parts of my infrastructure too anyway.
> >> >
> >> >
> >> > If it's part of Ceph, I end up writing a special case for Ceph.
> >>
> >> I think this would cease to be a problem for you if we just had a
> >> flag in Ceph to disable its own smartmontools type stuff?  That way
> >> when someone was using an external tool there would be no conflict.
> >>
> >> There is some duplication of effort, but I don't think that's
> >> intrinsically
> >> problematic: I predict that we'll always have many users who do not
> >> take up any of the external tools and will benefit from the built-in Ceph
> bits.
> >>
> >> > And I need a way to handle it when Ceph itself isn't around yet;
> >> > how do I blink an enclosure that receives a new disk? Ah, I
> >> > pre-register a given enclosure with Ceph, before an OSD is even
> >> > created. I know Ceph has many tentacles, but ... ;-)
> >>
> >> While at runtime we shouldn't have two agents competing to manage the
> >> same device, I think it is reasonable to have a separate piece of
> >> software that does installation vs. does the ongoing monitoring.  We
> >> shouldn't let the constraints over installation (especially the need
> >> to operate on cephless
> >> machines) restrict how we manage systems through their life cycles.
> >> Again, I don't think the built-in Ceph functionality is mutually
> >> exclusive with having a good external installation tool that touches some
> of the same functionality.
> >>
> >> John
> >>
> >>
> >> >
> >> >
> >> > Regards,
> >> >     Lars
> >> >
> >> > --
> >> > SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham
> >> > Norton, HRB 21284 (AG Nürnberg) "Experience is the name everyone
> >> > gives to their mistakes." -- Oscar Wilde
> >> >
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> > in the body of a message to majordomo@vger.kernel.org More
> >> majordomo
> >> > info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
> >> in the body of a message to majordomo@vger.kernel.org More
> majordomo
> >> info at http://vger.kernel.org/majordomo-info.html

RE: disk enclosure LEDs

[Sage Weil]

On Fri, 2 Dec 2016, Alan Johnson wrote:
> As a hardware vendor we would like some sort of API to hook into - we do 
> have a proprietary way of identifying failing devices but would rather 
> conform to something more standard within the Ceph arena.

https://libstorage.github.io/libstoragemgmt-doc/

I think this is the best place to invest your efforts!

sage



> 
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Allen Samuels
> Sent: Friday, December 02, 2016 11:57 AM
> To: Sage Weil; Lars Marowsky-Bree
> Cc: Ceph Development
> Subject: RE: disk enclosure LEDs
> 
> > -----Original Message-----
> > From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel- 
> > owner@vger.kernel.org] On Behalf Of Sage Weil
> > Sent: Friday, December 02, 2016 6:24 AM
> > To: Lars Marowsky-Bree <lmb@suse.com>
> > Cc: Ceph Development <ceph-devel@vger.kernel.org>
> > Subject: Re: disk enclosure LEDs
> > 
> > On Fri, 2 Dec 2016, Lars Marowsky-Bree wrote:
> > > On 2016-12-01T00:15:25, John Spray <jspray@redhat.com> wrote:
> > >
> > > > Put another way, I'd like it to be the case that people who want 
> > > > to twiddle LEDs can write a ceph-mgr module that does that, rather 
> > > > than having to push out a libstoragemgmt-enabled agent to the Ceph servers.
> > >
> > > I'm not sure I agree.
> > >
> > > I understand some of the motivation, but the key use cases where I'd 
> > > want to "twiddle LEDs" involve enclosures that either don't yet have 
> > > an OSD deployed (e.g., where to plug in the new disk), or where the 
> > > OSD has failed (and may not even be able to start).
> > >
> > > And I don't want to have to maintain multiple paths for interacting 
> > > with the enclosures.
> > 
> > I agree with John that it would be great to have ceph-mgr plugins 
> > doing this sort of thing.  But the OSD can't be the only path, and I 
> > also don't like the idea of multiple paths.
> > 
> > The problem is that there needs to be something on the hosts that's 
> > doing this, and I don't want to just ignore it and hope some other 
> > layer solves it; years have gone by and nobody has done it, and I'd 
> > like to avoid a fragmented approach if we can.
> > 
> > On the other hand, if we simply make an opinionated choice about some 
> > other per-host agent service, like we did in the original Calamari 
> > (which used salt and zeromq, IIRC), we'll probably end up offending 
> > more users than we please.
> > 
> > Perhaps the way forward is to pick *some* external host agent and 
> > service to start with and ensure that it is integrated into ceph-mgr 
> > in a pluggable way such that we have (1) the osd:device mappings 
> > available, (2) a standard API for working with LEDs etc, and (3) a 
> > working plugin for twiddling the LEDs, but ensure that other "disk services" backends can also be used?
> 
> I think it's important to separate this problem into two pieces: ceph-specific and platform-specific.
> 
> The ceph-specific part of the problem is to create a single abstraction and mechanism so that upper-level tools (ceph-mgr, GUIs, workflow scripts, etc.) can manipulation the abstraction without worrying about the implementation details. I'm completely with John that ceph-mgr is the right place for the upper-layer to terminated. Ultimately, we'll have infrastructure that ties the physical state of the drives (power up/down, lights on/off, etc.) more closely to the operational state of the system -- this is an area where more standardized automation will remove a significant barrier to wide-spread deployments.
> 
> The platform-specific part definitely should be pluggable/configurable/soft/manipulable, etc. It'll be quite some time (if ever) before something like libstoragemgmt becomes a reliable, universal, standard interface. We can't let that hold back the development of the upper-level infrastructure stuff.
> 
> So, I'd vote for having an API termination in ceph-mgr. Have the mgr contact the destination system (the OSD is *NOT* the right path) to perform the action. Then do something like invoke a configurable shell script (something that's configurable) to perform the actual operation. 
> 
>  
> > 
> > sage
> > 
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> > in the body of a message to majordomo@vger.kernel.org More majordomo 
> > info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: disk enclosure LEDs

[John Spray]

On Wed, Nov 30, 2016 at 4:51 PM, Sage Weil <sweil@redhat.com> wrote:
> Hi all,
>
> libstoragemgmt has made progress on a generic interface for twiddling
> enclosure LEDs!
>
>>      * RHEL 7.3 or Fedora 24+
>>      * sudo yum install libstoragemgmt
>>      * sudo lsmcli local-disk-ident-led-on --path /dev/sdX
>>      * sudo lsmcli local-disk-ident-led-off --path /dev/sdX
>>      * sudo lsmcli local-disk-fault-led-on --path /dev/sdX
>>      * sudo lsmcli local-disk-fault-led-off --path /dev/sdX
>
>> Python API document:
>>
>>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_on)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.ident_led_off)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_on)'
>>      python2 -c'import lsm; help(lsm.LocalDisk.fault_led_off)'
>>
>> C API document:
>>
>>      Check header file `libstoragemgmt_local_disk.h` in
>>      `libstoragemgmt-devel` rpm package. The functions are:
>>
>>      lsm_local_disk_ident_led_on()
>>      lsm_local_disk_ident_led_off()
>>      lsm_local_disk_fault_led_on()
>>      lsm_local_disk_fault_led_off()
>
> Since this is in a reasonably usable state, I think It's time for us to
> figure out how we are going to do this in ceph.
>
> A few ideas:
>
>  ceph osd identify osd.123    # blink for a few seconds?
>
> or
>
>  ceph osd ident-led-on osd.123  # turn on
>  ceph osd ident-led-off osd.123  # turn off
>  ceph osd fault-led-on osd.123  # turn on
>  ceph osd fault-led-off osd.123  # turn off
>
> This would mean persistently recording the LED state in the OSDMap.  And
> it would mean ceph-osd is the one twiddling the LEDs.  But that might not
> be the way to go in all cases.  For example, if we have an OSD that fails,
> once we confirm that we've healed (and don't need that OSDs data) we'd
> probably want to set the fault light so that the disk can be pulled
> safely.  In that case, ceph-osd isn't running (it's long-since failed),
> and we'd need some other agent on the node to twiddle the light.  Do we
> really want multiple things twiddling lights?
>
> We also often have a N:M mapping of osds to devices (multiple devices per
> OSD, multiple OSDs per device), which means a per-OSD flag might not be
> the right way to think about this anyway.
>
> Has anyone thought this through yet?

My preferences is to keep this as lightweight as possible and keeping
it in `tell` commands that the OSD can pass through almost completely
directly to libstoragemgmt.  This could include "fault on", "fault
off" and "blink"/"identify, but without putting any persistent state
in there: if you stopped ceph-osd the blinking would stop (but a
toggled fault light would stay on).  Using tell commands would also
make it straightforward to get an error back if the OSD can't identify
a device to blink an LED on, instead of setting something in the OSD
map and then wondering why nothing blinks.

The "what block device am I?" part in the OSD (especially given
many-to-one relations as you say) is probably harder than the calling
into libstoragemgmt.  We would probably also need all the LED commands
to have a flag to optionally target the journal drive instead of the
data drive.  Where multiple OSDs target the same drive, I don't see
that as a problem: it's reasonable to have the commands me "blink the
drive you use" and not "blink a drive and thereby claim you are the
only thing using it".

John

>
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: disk enclosure LEDs

[Lenz Grimmer]

[-- Attachment #1.1: Type: text/plain, Size: 1066 bytes --]

Hi,

On 12/01/2016 01:10 AM, John Spray wrote:

> The "what block device am I?" part in the OSD (especially given
> many-to-one relations as you say) is probably harder than the calling
> into libstoragemgmt.  We would probably also need all the LED commands
> to have a flag to optionally target the journal drive instead of the
> data drive.  Where multiple OSDs target the same drive, I don't see
> that as a problem: it's reasonable to have the commands me "blink the
> drive you use" and not "blink a drive and thereby claim you are the
> only thing using it".

I tend to agree to Joao's point that I don't think that the OSD should
be in the business of blinking disk LEDs.

Instead, I would be in favor if it could tell me which disks it actually
uses, which would allow me to pass this device name to any given tool I
choose for blinking the LED on that device.

There was a pull request about this, but I don't think it ever got
merged - http://tracker.ceph.com/issues/3120

Would it make sense to revisit this?

Thanks,

Lenz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: disk enclosure LEDs

[Jesse Williamson]

On Thu, 1 Dec 2016, Lenz Grimmer wrote:

Hi,

> Instead, I would be in favor if it could tell me which disks it actually
> uses, which would allow me to pass this device name to any given tool I
> choose for blinking the LED on that device.

This seems like a good path to me.

The suggestion Joao made of a plugin is my second choice. That itself 
opens up new opportunities for error, but I like that it doesn't tie 
another dependency into OSD.

> There was a pull request about this, but I don't think it ever got
> merged - http://tracker.ceph.com/issues/3120

Would exposing this through the console (if this has not been done 
already) enable enough to address John's concern?

-Jesse