RFC: progress bars

[John Spray <john.spray@redhat.com>]

Hi all,

[this is a re-send of a mail from yesterday that didn't make it, 
probably due to an attachment]

It has always annoyed me that we don't provide a simple progress bar 
indicator for things like the migration of data from an OSD when it's 
marked out, the rebalance that happens when we add a new OSD, or 
scrubbing the PGs on an OSD.

I've experimented a bit with adding user-visible progress bars for some 
of the simple cases (screenshot at http://imgur.com/OaifxMf). The code 
is here:
https://github.com/ceph/ceph/blob/wip-progress-events/src/mon/ProgressEvent.cc 


This is based on a series of "ProgressEvent" classes that are 
instantiated when certain things happen, like marking and OSD in or 
out.  They provide an init() hook that captures whatever state is needed 
at the start of the operation (generally noting which PGs are affected) 
and a tick() hook that checks whether the affected PGs have reached 
their final state.

Clearly, while this is simple for the simple cases, there are lots of 
instances where things will overlap: a PG can get moved again while it's 
being backfilled following a particular OSD going out. These progress 
indicators don't have to capture that complexity, but the goal would be 
to make sure they did complete eventually rather than getting 
stuck/confused in those cases.

This is just a rough cut to play with the idea, there's no persistence 
of the ProgressEvents, and the init/tick() methods are peppered with 
correctness issues.  Still, it gives a flavour of how we could add 
something friendlier like this to expose simplified progress indicators.

Ideas for further work:
  * Add in an MDS handler to capture the progress of an MDS rank as it 
goes through replay/reconnect/clientreplay
  * A handler for overall cluster restart, that noticed when the mon 
quorum was established and all the map timestamps were some time in the 
past, and then generated progress based on OSDs coming up and PGs peering.
  * Simple: a handler for PG creation after pool creation
  * Generate estimated completion times from the rate of progress so far
  * Friendlier PGMap output, by hiding all PG states that are explained 
by an ongoing ProgressEvent, to only indicate low level PG status for 
things that the ProgressEvents don't understand.

Cheers,
John