Re: Ideas for new ceph-mgr service

[Matt Benjamin <mbenjamin@redhat.com>]

Sorry, ignore the part about ceph-mon, I misread that sentence in the original mail.

Matt

----- Original Message -----
> From: "Matt Benjamin" <mbenjamin@redhat.com>
> To: "John Spray" <jspray@redhat.com>
> Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
> Sent: Wednesday, January 13, 2016 10:53:50 AM
> Subject: Re: Ideas for new ceph-mgr service
> 
> Hi,
> 
> 
> ----- Original Message -----
> > From: "John Spray" <jspray@redhat.com>
> > To: "Ceph Development" <ceph-devel@vger.kernel.org>
> > Sent: Wednesday, January 13, 2016 8:13:27 AM
> > Subject: Ideas for new ceph-mgr service
> > 
> > Hi all,
> > 
> > We currently have an unfulfilled need for a high level
> > management/monitoring service that can take some of the non-essential
> > tasks away from the mon (like handling the volume of advisory pg
> > stats), and provide a place to implement new features (like
> > cluster-wide command and control of the new scrub stuff in cephfs).
> 
> I (and our group as a whole) think this will be a HUGE win, it's something
> we've talked about conceptually for years.  Thank you sincerely for
> proposing and prototyping this!
> 
> > 
> > We've had a couple of attempts in this area historically:
> >  * ceph-rest-api, which is a stateless HTTP wrapper around the
> > MonCommand interface.  All calls hit the mons directly, it's really
> > just a protocol converter, and it's really more RPC than REST.
> >  * Calamari, which has a very extensible architecture, but suffers
> > from being rather heavyweight, with lots of dependencies like its own
> > database, and requires its own separate agents running on all the Ceph
> > servers.
> > 
> > So, the idea is to create a new lightweight service (ceph-mgr) that
> > runs alongside the mon, and uses the existing Ceph network channels to
> > talk to remote hosts.  The address of this service would be published
> > in the OSDMap, and OSDs and other daemons would send their
> > non-essential stats to the mgr instead of the mon.  For HA we would
> > probably run a mgr alongside each mon, and use whichever mgr instance
> > lived with the current leader mon.
> > 
> > Internally, the mgr itself then has three main components:
> >  * The server (a Messenger), which receives telemetry from daemons
> > elsewhere in the system, and receives cluster map updates from the mon
> >  * A simple in memory store of all the structures that we receive from
> > the cluster (the maps, the daemon metadata, the pg stats)
> >  * An embedded python interpreter that hosts high level functionality
> > like a REST API.
> > 
> > The mgr embodies the interface between "C++ Ceph land" (cephx auth,
> > Messenger, and ::encode/::decode serialization) and "admin land"
> > (JSON-like structures, REST APIs, Python modules).  The reason for
> > doing this in one process, rather than putting the Python parts in a
> > separate service (like calamari) is twofold:
> >  * Code simplicity: avoid inventing a C++->Python network API that
> > re-implements things like cluster map subscription and incremental
> > OSDmaps.
> >  * Efficiency: transmit data in its native encoding, hold it in memory
> > in native structs, and only expose what's needed up into Python-land
> > at runtime.

> 
> I would -strongly- request that we not use a hybrid C++ & Python server as a
> production version of this capability.  If the proof of concept is as
> successful as I intuit, I think it would be highly desirable to design a
> scalable, native-code framework for the core management service runtime.
> 
> Any apparent advantage from flexibility of Cython interfacing is, honestly, I
> think, strongly outweighed by the drawbacks of supporting the hybrid
> interfaces, not to mention the pervasive serialization and latency
> properties of a Python-driven runtime model.  (That's not to say I think
> that Python shouldn't be used to implement routines called from a core
> management runtime, if you strongly prefer not to run such code
> out-of-process [as systems like Salt, iirc, do].)
> 
> Matt
> 
> > 
> > That last part involves a bit of a trick: because Python (specifically
> > the CPython interpreter) is so flexible, we can do neat things like
> > implementing functions in C++ that have access to our native Ceph data
> > structures, but are callable from high level Python code.  We can also
> > cast our C++ structures into Python dicts directly, without an
> > intermediate JSON step, using a magic Formatter subclass that
> > generates python objects instead of serializing.  In general the
> > PyFormatter is still not quite as efficient as writing full blown
> > wrappers for C++ structures, but it's way more efficient that
> > serializing stuff to JSON and sending it over the network.
> > 
> > Most of the business logic would then be written in python.  This
> > would include the obvious status/health REST APIs, but potentially
> > also things like pool management (similar to how the Calamari API
> > handles these).  As well as being accessible via a REST API, the stats
> > that live in the mgr could also be streamed on to a full featured time
> > series database like influxdb, for users that want to deploy that kind
> > of thing.  Our service would store some very recent history, so that
> > folks without a full featured TSDB can still load things like the last
> > 60s of bandwidth into a graph in their GUI, if they have a GUI that
> > uses our API.
> > 
> > I've written a small proof-of-concept service that just subscribes to
> > cluster maps, loads a python module that acts as an HTTP server, and
> > exposes the maps to the module.  It's here:
> > https://github.com/jcsp/ceph/tree/wip-pyfoo/src/pyfoo
> > 
> > I appreciate that this might not all be completely clear in text form,
> > probably some more detailed design and pictures will be needed in due
> > course, but I wanted to put this out there to get feedback.
> > 
> > Cheers,
> > John
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> 
> --
> --
> Matt Benjamin
> Red Hat, Inc.
> 315 West Huron Street, Suite 140A
> Ann Arbor, Michigan 48103
> 
> http://www.redhat.com/en/technologies/storage
> 
> tel.  734-707-0660
> fax.  734-769-8938
> cel.  734-216-5309
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309