Update on ceph-mgr

Update on ceph-mgr

[John Spray]

Hi all,

For those who are interested, I wanted to give a quick update on the
recently-started ceph-mgr work.  This is all still very changeable,
but now that some code exists I'm putting it out there.

Firstly, here it is: https://github.com/jcsp/ceph/tree/wip-ceph-mgr

The place I'm starting is with the python embedding.  Currently it's
running the REST API code that previously lived in Calamari, which you
can now see in src/pybind/mgr/ (entry point is rest.py).  As
previously discussed, the Python interface is pretty minimal:
 * Python modules expose a serve() method and a notify() method (see rest.py)
 * Python modules consume methods "get" (for cluster objects like
maps), get_server (for the list of servers+services) and send_command
(for sending rados commands)

The rest.py module is serving the webapp using cherrypy -- this is a
handy choice because it just uses plain python threads, which are
fairly easy to deal with from the C++ side (gevent has a whole
different model that C-land sometimes needs to be aware of, gunicorn
uses multiprocessing).

I had to update some of the rest api code to work with the latest
version of django-rest-framework because Calamari was using an older
one.  My code is working with the versions of python dependencies that
are present in Fedora 22.

To play with this:
 1. Build your tree with cmake
 2. make ceph-mgr
 3. CALAMARI_CONFIG=<whereever>/ceph/src/pybind/mgr/calamari.conf
PYTHONPATH=env/lib/python2.7/site-packages bin/ceph-mgr -f
 4. Point your browser at e.g.
http://localhost:8002/api/v2/cluster/osd and witness that you get
calamari-like API output

Next part is the integration with the mon so that we can run ceph-mgr
as an active/standby HA service, and let it store its configuration
inside the mon.  Once we have that, combined with the embedded python
parts, this will turn into a nicely cleaned up pull request for the
main ceph tree.

Cheers,
John

Re: Update on ceph-mgr

[Wido den Hollander]

> Op 18 mei 2016 om 13:02 schreef John Spray <jspray@redhat.com>:
> 
> 
> Hi all,
> 
> For those who are interested, I wanted to give a quick update on the
> recently-started ceph-mgr work.  This is all still very changeable,
> but now that some code exists I'm putting it out there.
> 
> Firstly, here it is: https://github.com/jcsp/ceph/tree/wip-ceph-mgr
> 
> The place I'm starting is with the python embedding.  Currently it's
> running the REST API code that previously lived in Calamari, which you
> can now see in src/pybind/mgr/ (entry point is rest.py).  As
> previously discussed, the Python interface is pretty minimal:
>  * Python modules expose a serve() method and a notify() method (see rest.py)
>  * Python modules consume methods "get" (for cluster objects like
> maps), get_server (for the list of servers+services) and send_command
> (for sending rados commands)
> 
> The rest.py module is serving the webapp using cherrypy -- this is a
> handy choice because it just uses plain python threads, which are
> fairly easy to deal with from the C++ side (gevent has a whole
> different model that C-land sometimes needs to be aware of, gunicorn
> uses multiprocessing).
> 
> I had to update some of the rest api code to work with the latest
> version of django-rest-framework because Calamari was using an older
> one.  My code is working with the versions of python dependencies that
> are present in Fedora 22.
> 
> To play with this:
>  1. Build your tree with cmake
>  2. make ceph-mgr
>  3. CALAMARI_CONFIG=<whereever>/ceph/src/pybind/mgr/calamari.conf
> PYTHONPATH=env/lib/python2.7/site-packages bin/ceph-mgr -f
>  4. Point your browser at e.g.
> http://localhost:8002/api/v2/cluster/osd and witness that you get
> calamari-like API output
> 
> Next part is the integration with the mon so that we can run ceph-mgr
> as an active/standby HA service, and let it store its configuration
> inside the mon.  Once we have that, combined with the embedded python
> parts, this will turn into a nicely cleaned up pull request for the
> main ceph tree.
> 

So is the idea that when you install 'ceph-mon' you get the API by default?

Having a REST API for Ceph enabled at all times or with a simple click would be great for integrations with for example Salt.

And why do you need HA at all, the API mainly talks to the mon, right?

Wido

> Cheers,
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Update on ceph-mgr

[John Spray]

On Wed, May 18, 2016 at 12:10 PM, Wido den Hollander <wido@42on.com> wrote:
>
> So is the idea that when you install 'ceph-mon' you get the API by default?

No, it's a separate service.

> Having a REST API for Ceph enabled at all times or with a simple click would be great for integrations with for example Salt.
>
> And why do you need HA at all, the API mainly talks to the mon, right?

ceph-mgr will talk to mons, but it will also talk directly to other
daemons, for e.g. gathering statistics.  Check out the earlier
discussions on where this fits in and what it's for:

http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/28944
1:51:00 here: https://www.youtube.com/watch?v=0gIqgxrmrJw

Cheers,
John

> Wido
>
>> Cheers,
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Update on ceph-mgr

[John Spray]

On Wed, May 18, 2016 at 12:21 PM, John Spray <jspray@redhat.com> wrote:
> On Wed, May 18, 2016 at 12:10 PM, Wido den Hollander <wido@42on.com> wrote:
>>
>> So is the idea that when you install 'ceph-mon' you get the API by default?
>
> No, it's a separate service.

...although I should add that it would be a default part of ceph
releases, and I expect many people would just put a ceph-mgr instance
alongside each ceph-mon instance.

>
>> Having a REST API for Ceph enabled at all times or with a simple click would be great for integrations with for example Salt.
>>
>> And why do you need HA at all, the API mainly talks to the mon, right?
>
> ceph-mgr will talk to mons, but it will also talk directly to other
> daemons, for e.g. gathering statistics.  Check out the earlier
> discussions on where this fits in and what it's for:
>
> http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/28944
> 1:51:00 here: https://www.youtube.com/watch?v=0gIqgxrmrJw
>
> Cheers,
> John
>
>> Wido
>>
>>> Cheers,
>>> John
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Update on ceph-mgr

[Tim Serong]

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

On 05/18/2016 09:02 PM, John Spray wrote:
> Hi all,
> 
> For those who are interested, I wanted to give a quick update on the
> recently-started ceph-mgr work.  This is all still very changeable,
> but now that some code exists I'm putting it out there.
> 
> Firstly, here it is: https://github.com/jcsp/ceph/tree/wip-ceph-mgr
> 
> [...]
> 
> To play with this:
>  1. Build your tree with cmake
>  2. make ceph-mgr
>  3. CALAMARI_CONFIG=<whereever>/ceph/src/pybind/mgr/calamari.conf
> PYTHONPATH=env/lib/python2.7/site-packages bin/ceph-mgr -f
>  4. Point your browser at e.g.
> http://localhost:8002/api/v2/cluster/osd and witness that you get
> calamari-like API output

Just for fun, I decided to play.  I built from a clone of your
wip-ceph-mgr branch, then grabbed a ceph.conf and admin keyring from an
existing Jewel cluster, dumped them in build/, and ran:

PYTHONPATH=../src/pybind/mgr
CALAMARI_CONFIG=../src/pybind/mgr/calamari.conf ./bin/ceph-mgr
--keyring=./ceph.client.admin.keyring --log-file=client.log

This immediately aborted with an assertion failure:

  /ceph/src/mgr/Mgr.cc: 241: FAILED assert(mds_cmd.r == 0)

I suspect the "mds metadata" command might want an MDS name.  Throwing
caution to the wind I commented out that assert, and the mon_cmd.r
assert as well (see attached patch), and got a bit further, but then it
failed with:

  ImportError: No module named rest_framework

TL;DR: I had to install pythond-datetime, django 1.8 and
djangorestframework.  The latter I let be the latest version available
via `pip install` (3.3.3).

I had earlier tried Django 1.9, which I randomly happened to have
installed, but that failed with:

RuntimeError(u"Model class django.contrib.auth.models.Permission doesn't
declare an explicit app_label and isn't in an application in
INSTALLED_APPS.",)

(Django 1.6 is no good either, BTW: AttributeError: 'module' object has
no attribute 'UUIDField')

Anyway, after all that, I can in fact get a nice looking chunk of JSON
out of http://localhost:8002/api/v2/cluster/osd :)

Regards,

Tim
-- 
Tim Serong
Senior Clustering Engineer
SUSE
tserong@suse.com

[-- Attachment #2: ditch-mon-and-mds.patch --]
[-- Type: text/x-patch, Size: 769 bytes --]

diff --git a/src/mgr/Mgr.cc b/src/mgr/Mgr.cc
index e8e17b5..73df961 100644
--- a/src/mgr/Mgr.cc
+++ b/src/mgr/Mgr.cc
@@ -238,10 +238,10 @@ void Mgr::load_all_metadata()
   osd_cmd.wait();
   mon_cmd.wait();
 
-  assert(mds_cmd.r == 0);
-  assert(mon_cmd.r == 0);
+//  assert(mds_cmd.r == 0);
+//  assert(mon_cmd.r == 0);
   assert(osd_cmd.r == 0);
-
+/*
   for (auto &metadata_val : mds_cmd.json_result.get_array()) {
     json_spirit::mObject daemon_meta = metadata_val.get_obj();
 
@@ -277,7 +277,7 @@ void Mgr::load_all_metadata()
 
     dmi.insert(dm);
   }
-
+*/
   for (auto &osd_metadata_val : osd_cmd.json_result.get_array()) {
     json_spirit::mObject osd_metadata = osd_metadata_val.get_obj();
     dout(4) << osd_metadata.at("hostname").get_str() << dendl;

Re: Update on ceph-mgr

[John Spray]

On Thu, May 19, 2016 at 12:17 PM, Tim Serong <tserong@suse.com> wrote:
> On 05/18/2016 09:02 PM, John Spray wrote:
>> Hi all,
>>
>> For those who are interested, I wanted to give a quick update on the
>> recently-started ceph-mgr work.  This is all still very changeable,
>> but now that some code exists I'm putting it out there.
>>
>> Firstly, here it is: https://github.com/jcsp/ceph/tree/wip-ceph-mgr
>>
>> [...]
>>
>> To play with this:
>>  1. Build your tree with cmake
>>  2. make ceph-mgr
>>  3. CALAMARI_CONFIG=<whereever>/ceph/src/pybind/mgr/calamari.conf
>> PYTHONPATH=env/lib/python2.7/site-packages bin/ceph-mgr -f
>>  4. Point your browser at e.g.
>> http://localhost:8002/api/v2/cluster/osd and witness that you get
>> calamari-like API output
>
> Just for fun, I decided to play.  I built from a clone of your
> wip-ceph-mgr branch, then grabbed a ceph.conf and admin keyring from an
> existing Jewel cluster, dumped them in build/, and ran:
>
> PYTHONPATH=../src/pybind/mgr
> CALAMARI_CONFIG=../src/pybind/mgr/calamari.conf ./bin/ceph-mgr
> --keyring=./ceph.client.admin.keyring --log-file=client.log
>
> This immediately aborted with an assertion failure:
>
>   /ceph/src/mgr/Mgr.cc: 241: FAILED assert(mds_cmd.r == 0)

Yeah, you're not in general going to be able to run this against an
older-versioned Ceph cluster; there are already mon changes (such as
improving the metadata commands) and there will also be more extensive
additions to all the services to enable them to talk to the mgr.  This
is going to be new functionality all around, as opposed to a new thing
that can sit on older clusters.

> I suspect the "mds metadata" command might want an MDS name.  Throwing
> caution to the wind I commented out that assert, and the mon_cmd.r
> assert as well (see attached patch), and got a bit further, but then it
> failed with:
>
>   ImportError: No module named rest_framework
>
> TL;DR: I had to install pythond-datetime, django 1.8 and
> djangorestframework.  The latter I let be the latest version available
> via `pip install` (3.3.3).
>
> I had earlier tried Django 1.9, which I randomly happened to have
> installed, but that failed with:
>
> RuntimeError(u"Model class django.contrib.auth.models.Permission doesn't
> declare an explicit app_label and isn't in an application in
> INSTALLED_APPS.",)
>
> (Django 1.6 is no good either, BTW: AttributeError: 'module' object has
> no attribute 'UUIDField')
>
> Anyway, after all that, I can in fact get a nice looking chunk of JSON
> out of http://localhost:8002/api/v2/cluster/osd :)

Awesome!

John

>
> Regards,
>
> Tim
> --
> Tim Serong
> Senior Clustering Engineer
> SUSE
> tserong@suse.com