RBD mirroring CLI proposal ...

RBD mirroring CLI proposal ...

[Jason Dillaman]

The following describes a set of proposals to support the upcoming RBD mirroring feature [1] via the rbd cli.    The RBD mirroring feature will utilize a journal to allow modifications from a primary source to be replicated to one or more backup destinations. 

To keep configuration simple (no need for synchronizing mirroring metadata), RBD mirroring will be configured on a per-pool basis within a (conceptional) availability zone. For a given image, each zone will have a copy of the image and a journal.  When mirroring is enabled on an image (either at creation due to the default pool settings to explicitly via the rbd CLI), the image's journal will automatically register the known peer clusters.  If a peer is added or rebuilt after mirroring is enabled, the peer will be registered with the image and the RBD mirroring daemon on the remote peer will then be responsible for snapshotting the image, transferring the base image, deleting the snapshot, and starting journal replay.

A mirrored image can have the following state: primary, secondary (consistent), and secondary (inconsistent).  The images will also track an epoch associated with the primary state for
detecting when a secondary is now inconsistent due to a failover event. Any modification (IO writes, resize, snap create, etc) to a mirrored image will result in its status being automatically updated to primary.  Inconsistent secondary images will disallow any modifications.

Proposed CLI Updates

To configure basic journaling support for an RBD image:

* rbd feature enable <image-spec> journaling
                [--journal-object-pool <pool-name>]
                [--journal-splay-width <num>]
                [--journal-object-size <B/K/M>]
                [--journal-additional-tweakable-settings]
        This will enable the RBD journaling feature bit.  A new journal will be
        created using default settings if not overridden.  

* rbd feature disable <image-spec> journaling
        This will disable the RBD journaling feature bit. If there are associated RBD
        mirroring peers connected to this image's journal, this will fail if the mirror
        doesn't detach within a timeout.  If the image is not attached to a consistency
        group, the journal will be automatically deleted.

To configure consistency groups:

* rbd consistency-group create <group-spec>
                [--object-pool <pool-name>]
                [--splay-width <num>]
                [--object-size <B/K/M>]
                [--additional-journal-tweakable-settings]
        This will create an empty journal for use with consistency groups (i.e. attaching
        multiple RBD images to the same journal to ensure consistent replay).

* rbd consistency-group rename <group-spec>
        This will remove the named consistency group journal.  If one or more images
        are attached, this will fail.

* rbd consistency-group attach <image-spec> <journal-spec>
        This will enable the RBD journaling feature bit and will configure the image to
        record all journal entries to the specified journal.  If journaling is already
        enabled on the image, this will fail.

* rbd consistency-group detach <image-spec>
        This will detach the specified image from its journal and disable the RBD
        journaling feature. 

* rbd consistency-group ls
        This will list all consistency groups within the current pool.

* rbd consistency-group info <group-spec>
        This will display information about the specified consistent group

where <group-spec> is [<pool-name>/]<group-name>

To configure mirroring support for an RBD image:

* rbd feature enable <image-spec> mirroring
        This will enable mirroring for an existing image if it wasn't auto-enabled
        by the default pool policy.

* rbd feature disable <image-spec> mirroring
        This will disable mirroring for a specific image if enabled manually or
        automatically via the default pool policy.
        
* rbd mirror pool enable <pool-name>
        This will, by default, ensure that all images created in this
        pool have exclusive lock, journaling, and mirroring feature bits
        enabled.

* rbd mirror pool disable <pool-name>
        This will clear the default image features for new images in this
        pool.

* rbd mirror pool add <remote-pool-spec>
        This will register a remote cluster/pool as a peer to the current,
        local pool.  All existing mirrored images and all future mirrored
        images will have this peer registered as a journal client.
        
* rbd mirror pool remove <remote-pool-spec>
        This will deregister a remote cluster/pool as a peer to the current,
        local pool.  All existing mirrored images will have the remote
        deregistered from image journals.

* rbd mirror pool info
         This will show the current status of pool mirroring (if newly created
         images automatically have mirroring enabled) and will list any
         registered remote clusters/pools.

* rbd mirror image enable <image-spec>
        This is an alias for 'rbd feature enable mirroring'

* rbd mirror image disable <image-spec>
        This is an alias for 'rbd feature disable mirroring'

* rbd mirror image resync <image-spec>
        This will delete the local image and initiate a full resync
        with a remote primary image.  This can be used to fix an
        inconsistent secondary image.

where <pool-spec> is [<cluster name>/]<pool name>

To verify the operational status of mirroring for a given image:

* rbd status <image-spec>
        This command currently only shows the current watchers for the
        image. This will be expanded to include mirroring status.

To manage journals:

* rbd journal info <journal-spec>
        This will display information about the specified journal

* rbd journal inspect <journal-spec>
        This will inspect the journal for structural errors

* rbd journal export <journal-spec> <dest>
        This will export the journal entries into JSON

* rbd journal import <src> <journal-spec>
        This will import the JSON-formated journal

* rbd journal reset
        This will delete all entries from the journal

where <journal-spec> is [<pool name>/]journal name

-- 

Jason Dillaman 

[1] http://tracker.ceph.com/projects/ceph/wiki/RBD_Async_Mirroring

Re: RBD mirroring CLI proposal ...

[Mykola Golub]

On Mon, Sep 21, 2015 at 10:48:11AM -0400, Jason Dillaman wrote:
> The following describes a set of proposals to support the upcoming RBD mirroring feature [1] via the rbd cli.    The RBD mirroring feature will utilize a journal to allow modifications from a primary source to be replicated to one or more backup destinations. 
> 
> To keep configuration simple (no need for synchronizing mirroring metadata), RBD mirroring will be configured on a per-pool basis within a (conceptional) availability zone. For a given image, each zone will have a copy of the image and a journal.  When mirroring is enabled on an image (either at creation due to the default pool settings to explicitly via the rbd CLI), the image's journal will automatically register the known peer clusters.  If a peer is added or rebuilt after mirroring is enabled, the peer will be registered with the image and the RBD mirroring daemon on the remote peer will then be responsible for snapshotting the image, transferring the base image, deleting the snapshot, and starting journal replay.
> 
> A mirrored image can have the following state: primary, secondary (consistent), and secondary (inconsistent).  The images will also track an epoch associated with the primary state for
> detecting when a secondary is now inconsistent due to a failover event. Any modification (IO writes, resize, snap create, etc) to a mirrored image will result in its status being automatically updated to primary.  Inconsistent secondary images will disallow any modifications.
> 
> Proposed CLI Updates
> 
> To configure basic journaling support for an RBD image:
> 
> * rbd feature enable <image-spec> journaling
>                 [--journal-object-pool <pool-name>]
>                 [--journal-splay-width <num>]
>                 [--journal-object-size <B/K/M>]
>                 [--journal-additional-tweakable-settings]
>         This will enable the RBD journaling feature bit.  A new journal will be
>         created using default settings if not overridden.  
> 
> * rbd feature disable <image-spec> journaling
>         This will disable the RBD journaling feature bit. If there are associated RBD
>         mirroring peers connected to this image's journal, this will fail if the mirror
>         doesn't detach within a timeout.  If the image is not attached to a consistency
>         group, the journal will be automatically deleted.
> 
> To configure consistency groups:
> 
> * rbd consistency-group create <group-spec>
>                 [--object-pool <pool-name>]
>                 [--splay-width <num>]
>                 [--object-size <B/K/M>]
>                 [--additional-journal-tweakable-settings]
>         This will create an empty journal for use with consistency groups (i.e. attaching
>         multiple RBD images to the same journal to ensure consistent replay).

For 'rbd feature' commands the option names have "journal" prefix
(--journal-object-pool), while for 'rbd consistency-group' they
don't. Is it intentional? I would prefer having "journal" prefix for
both.

> 
> * rbd consistency-group rename <group-spec>

s/rename/remove/ ?

>         This will remove the named consistency group journal.  If one or more images
>         are attached, this will fail.
> 
> * rbd consistency-group attach <image-spec> <journal-spec>
>         This will enable the RBD journaling feature bit and will configure the image to
>         record all journal entries to the specified journal.  If journaling is already
>         enabled on the image, this will fail.
> 
> * rbd consistency-group detach <image-spec>
>         This will detach the specified image from its journal and disable the RBD
>         journaling feature. 
> 
> * rbd consistency-group ls
>         This will list all consistency groups within the current pool.
> 
> * rbd consistency-group info <group-spec>
>         This will display information about the specified consistent group
> 
> where <group-spec> is [<pool-name>/]<group-name>

Is a consistency-group just a journal (usually used for several images)
or is it something more? If I enable journaling feature for an image,
journal is automatically created, is it already a consistency-group?

> 
> To configure mirroring support for an RBD image:
> 
> * rbd feature enable <image-spec> mirroring
>         This will enable mirroring for an existing image if it wasn't auto-enabled
>         by the default pool policy.
> 
> * rbd feature disable <image-spec> mirroring
>         This will disable mirroring for a specific image if enabled manually or
>         automatically via the default pool policy.
>         
> * rbd mirror pool enable <pool-name>
>         This will, by default, ensure that all images created in this
>         pool have exclusive lock, journaling, and mirroring feature bits
>         enabled.
> 
> * rbd mirror pool disable <pool-name>
>         This will clear the default image features for new images in this
>         pool.

Will 'rbd mirror pool enable|disable' change behaviour only for newly
created images in the pool or will enable|disable mirroring for
existent images too?

> 
> * rbd mirror pool add <remote-pool-spec>
>         This will register a remote cluster/pool as a peer to the current,
>         local pool.  All existing mirrored images and all future mirrored
>         images will have this peer registered as a journal client.
>         
> * rbd mirror pool remove <remote-pool-spec>
>         This will deregister a remote cluster/pool as a peer to the current,
>         local pool.  All existing mirrored images will have the remote
>         deregistered from image journals.

In remote-pool-spec, if it is "cluster/pool", where it is expected to
find configuration for the cluster "cluster"? In /etc/ceph/cluster.conf?

> 
> * rbd mirror pool info
>          This will show the current status of pool mirroring (if newly created
>          images automatically have mirroring enabled) and will list any
>          registered remote clusters/pools.
> 
> * rbd mirror image enable <image-spec>
>         This is an alias for 'rbd feature enable mirroring'
> 
> * rbd mirror image disable <image-spec>
>         This is an alias for 'rbd feature disable mirroring'
> 
> * rbd mirror image resync <image-spec>
>         This will delete the local image and initiate a full resync
>         with a remote primary image.  This can be used to fix an
>         inconsistent secondary image.
> 
> where <pool-spec> is [<cluster name>/]<pool name>
> 
> To verify the operational status of mirroring for a given image:
> 
> * rbd status <image-spec>
>         This command currently only shows the current watchers for the
>         image. This will be expanded to include mirroring status.
> 
> To manage journals:
> 
> * rbd journal info <journal-spec>
>         This will display information about the specified journal
> 
> * rbd journal inspect <journal-spec>
>         This will inspect the journal for structural errors
> 
> * rbd journal export <journal-spec> <dest>
>         This will export the journal entries into JSON
> 
> * rbd journal import <src> <journal-spec>
>         This will import the JSON-formated journal
> 
> * rbd journal reset
>         This will delete all entries from the journal
> 
> where <journal-spec> is [<pool name>/]journal name


Apart from the list of new commands, it would be very helpful to see
typical use case examples. Are my examples below correct?

Enable mirroring for pool "volumes", using pool "journals" for journal
objects:

  rbd mirror pool enable volumes --journal-object-pool journals

Start mirroring to the remote cluster ceph-remote:

  Specify the remote cluster configuration in
  /etc/ceph/ceph-remote.conf

  rbd mirror pool add ceph-remote/volumes

  Start RBD mirroring daemon on the remote cluster. Both clusters
  should be specified in config or via command line parameters. No
  need to specify mirroring pools, I suppose?

To configure a consistency group cgroup1 for images volume1 and
volume2 in pool volumes:

  rbd consistency-group create volume/cgroup1 --object-pool journals
  rbd consistency-group attach volume/volume1 volume/cgroup1
  rbd consistency-group attach volume/volume2 volume/cgroup1

I guess before attaching I will need to disable journaling for volumes
(because it was automatically enabled previously with 'rbd mirror pool
enable')? Will it automatically enable mirroring feature for the
attached images?

> 
> -- 
> 
> Jason Dillaman 
> 
> [1] http://tracker.ceph.com/projects/ceph/wiki/RBD_Async_Mirroring
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Mykola Golub

Re: RBD mirroring CLI proposal ...

[Jason Dillaman]

> > * rbd consistency-group create <group-spec>
> >                 [--object-pool <pool-name>]
> >                 [--splay-width <num>]
> >                 [--object-size <B/K/M>]
> >                 [--additional-journal-tweakable-settings]
> >         This will create an empty journal for use with consistency groups
> >         (i.e. attaching
> >         multiple RBD images to the same journal to ensure consistent
> >         replay).
> 
> For 'rbd feature' commands the option names have "journal" prefix
> (--journal-object-pool), while for 'rbd consistency-group' they
> don't. Is it intentional? I would prefer having "journal" prefix for
> both.

It was intentional with the rationale that the 'journal-' prefix was added to the 'rbd feature enable' command because in the future it might support other configurable params.  I have no issue keeping them consistent between the two.   

> > 
> > * rbd consistency-group rename <group-spec>
> 
> s/rename/remove/ ?

Yup -- I think I merged the rename and remove commands by accident.  

> > * rbd consistency-group info <group-spec>
> >         This will display information about the specified consistent group
> > 
> > where <group-spec> is [<pool-name>/]<group-name>
> 
> Is a consistency-group just a journal (usually used for several images)
> or is it something more? If I enable journaling feature for an image,
> journal is automatically created, is it already a consistency-group?

Correct -- I'm just using the command name to describe the behavior you would receive if you attached multiple images to the same journal.  If you enable journaling for a given image (or if it is enabled by default due to the pool's mirroring policy), then you will have a journal created for you automatically that is only associated with the single image.  However, you wouldn't be able to attach another image to the same journal nor would you see such a journal listed as a named consistency group.  I was originally toying around with having all of these "rbd consistency-group" commands being just additional "rbd journal" commands and eliminating the semantic difference.

> > * rbd mirror pool enable <pool-name>
> >         This will, by default, ensure that all images created in this
> >         pool have exclusive lock, journaling, and mirroring feature bits
> >         enabled.
> > 
> > * rbd mirror pool disable <pool-name>
> >         This will clear the default image features for new images in this
> >         pool.
> 
> Will 'rbd mirror pool enable|disable' change behaviour only for newly
> created images in the pool or will enable|disable mirroring for
> existent images too?

Since the goal is to set default pool behavior, it would only apply to newly created images.  You can enable/disable on specific images using the 'rbd mirror image enable/disable' commands.

> > * rbd mirror pool add <remote-pool-spec>
> >         This will register a remote cluster/pool as a peer to the current,
> >         local pool.  All existing mirrored images and all future mirrored
> >         images will have this peer registered as a journal client.
> >         
> > * rbd mirror pool remove <remote-pool-spec>
> >         This will deregister a remote cluster/pool as a peer to the
> >         current,
> >         local pool.  All existing mirrored images will have the remote
> >         deregistered from image journals.
> 
> In remote-pool-spec, if it is "cluster/pool", where it is expected to
> find configuration for the cluster "cluster"? In /etc/ceph/cluster.conf?

Ceph conventions state that the configuration file for a given cluster should be named "<cluster-name>.conf" and placed in the search path (/etc/ceph being one such place). 

> Apart from the list of new commands, it would be very helpful to see
> typical use case examples. Are my examples below correct?
> 
> Enable mirroring for pool "volumes", using pool "journals" for journal
> objects:
> 
>   rbd mirror pool enable volumes --journal-object-pool journals

Yup -- I neglected to include journaling defaults in that command, but it would be good to include them.

> Start mirroring to the remote cluster ceph-remote:
> 
>   Specify the remote cluster configuration in
>   /etc/ceph/ceph-remote.conf
> 
>   rbd mirror pool add ceph-remote/volumes
> 
>   Start RBD mirroring daemon on the remote cluster. Both clusters
>   should be specified in config or via command line parameters. No
>   need to specify mirroring pools, I suppose?

Haven't thought about this level of detail, but I think you'd be fine only specifying the local cluster to the RBD mirroring daemon.  It could then scan all the pools, lookup peer cluster names, and connect.  The daemon should use the journal's primary / secondary state + epoch to know whether or not to replicate the events from the remote.

> To configure a consistency group cgroup1 for images volume1 and
> volume2 in pool volumes:
> 
>   rbd consistency-group create volume/cgroup1 --object-pool journals
>   rbd consistency-group attach volume/volume1 volume/cgroup1
>   rbd consistency-group attach volume/volume2 volume/cgroup1
> 
> I guess before attaching I will need to disable journaling for volumes
> (because it was automatically enabled previously with 'rbd mirror pool
> enable')? Will it automatically enable mirroring feature for the
> attached images?

I was planning to just fail the attach if journaling was already enabled, but it could just as easily (behind the scenes) disable journaling to kill off the old journal (and wean potential RBD mirroring daemons off the journal) and then just re-enable it using the new consistency group journal.  In this example, if the images volume1 and volume2 existed before you enable the pool mirroring property, they wouldn't have add journaling automatically enabled.

--

Jason Dillaman
 

Re: RBD mirroring CLI proposal ...

[Mykola Golub]

On Tue, Sep 22, 2015 at 01:32:49PM -0400, Jason Dillaman wrote:

> > > * rbd mirror pool enable <pool-name>
> > >         This will, by default, ensure that all images created in this
> > >         pool have exclusive lock, journaling, and mirroring feature bits
> > >         enabled.
> > > 
> > > * rbd mirror pool disable <pool-name>
> > >         This will clear the default image features for new images in this
> > >         pool.
> > 
> > Will 'rbd mirror pool enable|disable' change behaviour only for newly
> > created images in the pool or will enable|disable mirroring for
> > existent images too?
>
> Since the goal is to set default pool behavior, it would only apply
> to newly created images.  You can enable/disable on specific images
> using the 'rbd mirror image enable/disable' commands.

In this case the commands look a little confusing to me, as from their
names I would rather think they enable/disable mirror for existent
images too. Also, I don't see a command to check what current
behaviour is. And, I suppose it would be useful if we could configure
other default features for a pool (exclusive-lock, object-map, ...)
Also, I am not sure we should specify <pool-name> this way, as it is
not consistent with other rbd commands. By default rbd operates on
'rbd' pool, which can be changed by --pool option. So what do you
think if we have something similar to 'rbd feature' commands?

  rbd [--pool <pool>] default-feature enable <feature>
  rbd [--pool <pool>] default-feature disable <feature>
  rbd [--pool <pool>] default-feature show [<feature>]

(If <feature> is not specified in the last command, all features are
shown).

Similarly, it might be useful to have 'rbd feature show' command:

  rbd feature show <image-spec> [<feature>]

BTW, where do you think these default feature flags will be stored?
Storing in pg_pool_t::flags I suppose is the easiest but it looks like
a layering violation.

-- 
Mykola Golub

Re: RBD mirroring CLI proposal ...

[Mykola Golub]

On Wed, Sep 23, 2015 at 09:33:14AM +0300, Mykola Golub wrote:

> Also, I am not sure we should specify <pool-name> this way, as it is
> not consistent with other rbd commands. By default rbd operates on
> 'rbd' pool, which can be changed by --pool option.

The same reasoning for these commands:

> > * rbd mirror pool add <remote-pool-spec>
> >         This will register a remote cluster/pool as a peer to the current,
> >         local pool.  All existing mirrored images and all future mirrored
> >         images will have this peer registered as a journal client.
> >
> > * rbd mirror pool remove <remote-pool-spec>
> >         This will deregister a remote cluster/pool as a peer to the
> >         current,
> >         local pool.  All existing mirrored images will have the remote
> >         deregistered from image journals.

I am not sure we need 'pool' here. I would prefer:

  rbd mirror peer add <remote-peer-spec>
  rbd mirror peer remove <remote-peer-spec>
  rbd mirror peer show

Where, <remote-peer-spec> should not necessary contain pool, because
the default could be used or what is specified via '--pool' option.

-- 
Mykola Golub

Re: RBD mirroring CLI proposal ...

[Jason Dillaman]

> > > * rbd mirror pool add <remote-pool-spec>
> > >         This will register a remote cluster/pool as a peer to the
> > >         current,
> > >         local pool.  All existing mirrored images and all future mirrored
> > >         images will have this peer registered as a journal client.
> > >
> > > * rbd mirror pool remove <remote-pool-spec>
> > >         This will deregister a remote cluster/pool as a peer to the
> > >         current,
> > >         local pool.  All existing mirrored images will have the remote
> > >         deregistered from image journals.
> 
> I am not sure we need 'pool' here. I would prefer:
> 
>   rbd mirror peer add <remote-peer-spec>
>   rbd mirror peer remove <remote-peer-spec>
>   rbd mirror peer show
> 
> Where, <remote-peer-spec> should not necessary contain pool, because
> the default could be used or what is specified via '--pool' option.
> 

I used "pool" to signify that these commands operate on a per-pool basis, which is how mirroring will function.

Re: RBD mirroring CLI proposal ...

[Ilya Dryomov]

On Wed, Sep 23, 2015 at 9:33 AM, Mykola Golub <mgolub@mirantis.com> wrote:
> On Tue, Sep 22, 2015 at 01:32:49PM -0400, Jason Dillaman wrote:
>
>> > > * rbd mirror pool enable <pool-name>
>> > >         This will, by default, ensure that all images created in this
>> > >         pool have exclusive lock, journaling, and mirroring feature bits
>> > >         enabled.
>> > >
>> > > * rbd mirror pool disable <pool-name>
>> > >         This will clear the default image features for new images in this
>> > >         pool.
>> >
>> > Will 'rbd mirror pool enable|disable' change behaviour only for newly
>> > created images in the pool or will enable|disable mirroring for
>> > existent images too?
>>
>> Since the goal is to set default pool behavior, it would only apply
>> to newly created images.  You can enable/disable on specific images
>> using the 'rbd mirror image enable/disable' commands.
>
> In this case the commands look a little confusing to me, as from their
> names I would rather think they enable/disable mirror for existent
> images too. Also, I don't see a command to check what current
> behaviour is. And, I suppose it would be useful if we could configure
> other default features for a pool (exclusive-lock, object-map, ...)
> Also, I am not sure we should specify <pool-name> this way, as it is
> not consistent with other rbd commands. By default rbd operates on
> 'rbd' pool, which can be changed by --pool option. So what do you
> think if we have something similar to 'rbd feature' commands?
>
>   rbd [--pool <pool>] default-feature enable <feature>
>   rbd [--pool <pool>] default-feature disable <feature>
>   rbd [--pool <pool>] default-feature show [<feature>]
>
> (If <feature> is not specified in the last command, all features are
> shown).

I haven't read the discussion in full, so feel free to ignore, but I'd
much rather have rbd create create images with the same set of features
enabled no matter which pool it's pointed to.

I'm not clear on what exactly a pool policy is.  If it's just a set of
feature bits to enable at create time, I don't think it's worth
introducing at all.  If it's a set feature bits + a set of mirroring
related options, I think it should just be a set of those options.
Then, rbd create --enable-mirroring could be a way to create an image
with mirroring enabled.

My point is a plain old "rbd create foo" shouldn't depend an any new
pool-level metadata.  It's not that hard to remember which features you
want for which pools and rbd create shortcuts like --enable-object-map
and --enable-mirroring would hide feature bit dependencies and save
typing.  --enable-mirroring would also serve as a ticket to go look at
new metadata and pull out any mirroring related defaults.

Thanks,

                Ilya

Re: RBD mirroring CLI proposal ...

[Jason Dillaman]

> > In this case the commands look a little confusing to me, as from their
> > names I would rather think they enable/disable mirror for existent
> > images too. Also, I don't see a command to check what current
> > behaviour is. And, I suppose it would be useful if we could configure
> > other default features for a pool (exclusive-lock, object-map, ...)
> > Also, I am not sure we should specify <pool-name> this way, as it is
> > not consistent with other rbd commands. By default rbd operates on
> > 'rbd' pool, which can be changed by --pool option. So what do you
> > think if we have something similar to 'rbd feature' commands?
> >
> >   rbd [--pool <pool>] default-feature enable <feature>
> >   rbd [--pool <pool>] default-feature disable <feature>
> >   rbd [--pool <pool>] default-feature show [<feature>]
> >
> > (If <feature> is not specified in the last command, all features are
> > shown).
> 
> I haven't read the discussion in full, so feel free to ignore, but I'd
> much rather have rbd create create images with the same set of features
> enabled no matter which pool it's pointed to.
> 
> I'm not clear on what exactly a pool policy is.  If it's just a set of
> feature bits to enable at create time, I don't think it's worth
> introducing at all.  If it's a set feature bits + a set of mirroring
> related options, I think it should just be a set of those options.
> Then, rbd create --enable-mirroring could be a way to create an image
> with mirroring enabled.
> 
> My point is a plain old "rbd create foo" shouldn't depend an any new
> pool-level metadata.  It's not that hard to remember which features you
> want for which pools and rbd create shortcuts like --enable-object-map
> and --enable-mirroring would hide feature bit dependencies and save
> typing.  --enable-mirroring would also serve as a ticket to go look at
> new metadata and pull out any mirroring related defaults.
> 

While I agree that I don't necessarily want to go down the road of specifying per-pool default features, I think there is a definite use-case need to be able to enable mirroring on a per-pool basis by default.  Without such an ability, taking OpenStack for example, it would require changes to Glance, Cinder, and Nova in order to support DR configurations -- or you could get it automatically with a little pre-configuration.

Re: RBD mirroring CLI proposal ...

[Ilya Dryomov]

On Wed, Sep 23, 2015 at 4:08 PM, Jason Dillaman <dillaman@redhat.com> wrote:
>> > In this case the commands look a little confusing to me, as from their
>> > names I would rather think they enable/disable mirror for existent
>> > images too. Also, I don't see a command to check what current
>> > behaviour is. And, I suppose it would be useful if we could configure
>> > other default features for a pool (exclusive-lock, object-map, ...)
>> > Also, I am not sure we should specify <pool-name> this way, as it is
>> > not consistent with other rbd commands. By default rbd operates on
>> > 'rbd' pool, which can be changed by --pool option. So what do you
>> > think if we have something similar to 'rbd feature' commands?
>> >
>> >   rbd [--pool <pool>] default-feature enable <feature>
>> >   rbd [--pool <pool>] default-feature disable <feature>
>> >   rbd [--pool <pool>] default-feature show [<feature>]
>> >
>> > (If <feature> is not specified in the last command, all features are
>> > shown).
>>
>> I haven't read the discussion in full, so feel free to ignore, but I'd
>> much rather have rbd create create images with the same set of features
>> enabled no matter which pool it's pointed to.
>>
>> I'm not clear on what exactly a pool policy is.  If it's just a set of
>> feature bits to enable at create time, I don't think it's worth
>> introducing at all.  If it's a set feature bits + a set of mirroring
>> related options, I think it should just be a set of those options.
>> Then, rbd create --enable-mirroring could be a way to create an image
>> with mirroring enabled.
>>
>> My point is a plain old "rbd create foo" shouldn't depend an any new
>> pool-level metadata.  It's not that hard to remember which features you
>> want for which pools and rbd create shortcuts like --enable-object-map
>> and --enable-mirroring would hide feature bit dependencies and save
>> typing.  --enable-mirroring would also serve as a ticket to go look at
>> new metadata and pull out any mirroring related defaults.
>>
>
> While I agree that I don't necessarily want to go down the road of specifying per-pool default features, I think there is a definite use-case need to be able to enable mirroring on a per-pool basis by default.  Without such an ability, taking OpenStack for example, it would require changes to Glance, Cinder, and Nova in order to support DR configurations -- or you could get it automatically with a little pre-configuration.

So a pool policy is just a set of feature bits?

I think Cinder at least creates images with rbd_default_features from
ceph.conf and adds in layering if it's not set, meaning there is no
interface for passing through feature bits (or anything else really,
things like striping options, etc).  What pool-level default feature
bits infrastructure would do is replace a big (cluster-level) hammer
with a smaller (pool-level) hammer.  You'd have to add librbd APIs for
it and someone eventually will try to follow suit and add defaults for
other settings.  You said you weren't attempting to create a mechanism
to specify arbitrary default features for a given pool, but I think it
will come up in the future if we introduce this - it's only logical.

What we might want to do instead is use this mirroring milestone to add
support for a proper key-value interface for passing in features and
other settings for individual rbd images to OpenStack.  I assume it's
all python dicts with OpenStack, so it shouldn't be hard?  I know that
getting patches into OpenStack can be frustrating at times and I might
be underestimating the importance of the use case you have in mind, but
patching our OpenStack drivers rather than adding what essentially is
a workaround to librbd makes a lot more sense to me.

Thanks,

                Ilya

Re: RBD mirroring CLI proposal ...

[Jason Dillaman]

> So a pool policy is just a set of feature bits?

It would have to store additional details as well.

> I think Cinder at least creates images with rbd_default_features from
> ceph.conf and adds in layering if it's not set, meaning there is no
> interface for passing through feature bits (or anything else really,
> things like striping options, etc).  What pool-level default feature
> bits infrastructure would do is replace a big (cluster-level) hammer
> with a smaller (pool-level) hammer.  You'd have to add librbd APIs for
> it and someone eventually will try to follow suit and add defaults for
> other settings.  You said you weren't attempting to create a mechanism
> to specify arbitrary default features for a given pool, but I think it
> will come up in the future if we introduce this - it's only logical.
> 
> What we might want to do instead is use this mirroring milestone to add
> support for a proper key-value interface for passing in features and
> other settings for individual rbd images to OpenStack.  I assume it's
> all python dicts with OpenStack, so it shouldn't be hard?  I know that
> getting patches into OpenStack can be frustrating at times and I might
> be underestimating the importance of the use case you have in mind, but
> patching our OpenStack drivers rather than adding what essentially is
> a workaround to librbd makes a lot more sense to me.
> 

It would be less work to skip adding the pool-level defaults which is a plus given everything else required.  However, putting aside how long it would take for the required changes to trickle down from OpenStack, Qemu, etc (since I agree that shouldn't drive design), in some ways your proposal could be seen as blurring the configuration encapsulation between clients and Ceph. 

Is the goal to configure my storage policies in one place or should I have to update all my client configuration settings (not that big of a deal if you are using something like Puppet to push down consistent configs across your servers)? Trying to think like an end-user, I think I would prefer configuring it once within the storage system itself.  I am not familiar with any other storage systems that configure mirroring via OpenStack config files, but I could be wrong since there are a lot of volume drivers now.

I do like the idea of key/value configuration pairs on image create and I had even proposed that a few weeks ago on a separate email thread since we shouldn't keep expanding rbd_create/rbd_clone/rbd_copy for every possible configuration override.  

--

Jason

Re: RBD mirroring CLI proposal ...

[Ilya Dryomov]

On Wed, Sep 23, 2015 at 9:28 PM, Jason Dillaman <dillaman@redhat.com> wrote:
>> So a pool policy is just a set of feature bits?
>
> It would have to store additional details as well.
>
>> I think Cinder at least creates images with rbd_default_features from
>> ceph.conf and adds in layering if it's not set, meaning there is no
>> interface for passing through feature bits (or anything else really,
>> things like striping options, etc).  What pool-level default feature
>> bits infrastructure would do is replace a big (cluster-level) hammer
>> with a smaller (pool-level) hammer.  You'd have to add librbd APIs for
>> it and someone eventually will try to follow suit and add defaults for
>> other settings.  You said you weren't attempting to create a mechanism
>> to specify arbitrary default features for a given pool, but I think it
>> will come up in the future if we introduce this - it's only logical.
>>
>> What we might want to do instead is use this mirroring milestone to add
>> support for a proper key-value interface for passing in features and
>> other settings for individual rbd images to OpenStack.  I assume it's
>> all python dicts with OpenStack, so it shouldn't be hard?  I know that
>> getting patches into OpenStack can be frustrating at times and I might
>> be underestimating the importance of the use case you have in mind, but
>> patching our OpenStack drivers rather than adding what essentially is
>> a workaround to librbd makes a lot more sense to me.
>>
>
> It would be less work to skip adding the pool-level defaults which is a plus given everything else required.  However, putting aside how long it would take for the required changes to trickle down from OpenStack, Qemu, etc (since I agree that shouldn't drive design), in some ways your proposal could be seen as blurring the configuration encapsulation between clients and Ceph.
>
> Is the goal to configure my storage policies in one place or should I have to update all my client configuration settings (not that big of a deal if you are using something like Puppet to push down consistent configs across your servers)? Trying to think like an end-user, I think I would prefer configuring it once within the storage system itself.  I am not familiar with any other storage systems that configure mirroring via OpenStack config files, but I could be wrong since there are a lot of volume drivers now.

I'm not very familiar with OpenStack so I don't know either, I'm just
pointing out that, as far as at least cinder goes, we currently use
a cluster-wide default for something that is inherently a per-image
property, that there is no way to change it, and that there is a way to
configure only a small subset of settings.  I don't see it as blurring
the configuration encapsulation: if a user is creating an image from
OpenStack (or any other client for that matter), they should be able to
specify all the settings they want for a given image and not rely on
cluster-wide or pool-wide defaults.  (Maybe I'm too fixed on this idea
that per-image properties should be per-image and you are trying to
think bigger.  What I'm ranting about here is status quo, mirroring and
the new use cases and configuration challanges it brings along are
somewhat off to the side.)

I'm not against pool-level defaults per se, I just think if we go down
this road it's going to be hard to draw a line in the future, and I want
to make sure we are not adding it just to work around deficiencies in
our OpenStack drivers (and possibly librbd create-like APIs).

Thanks,

                Ilya

Re: RBD mirroring CLI proposal ...

[Jason Dillaman]

> In this case the commands look a little confusing to me, as from their
> names I would rather think they enable/disable mirror for existent
> images too. Also, I don't see a command to check what current
> behaviour is. And, I suppose it would be useful if we could configure
> other default features for a pool (exclusive-lock, object-map, ...)
> Also, I am not sure we should specify <pool-name> this way, as it is
> not consistent with other rbd commands. By default rbd operates on
> 'rbd' pool, which can be changed by --pool option. So what do you
> think if we have something similar to 'rbd feature' commands?
> 
>   rbd [--pool <pool>] default-feature enable <feature>
>   rbd [--pool <pool>] default-feature disable <feature>
>   rbd [--pool <pool>] default-feature show [<feature>]
> 
> (If <feature> is not specified in the last command, all features are
> shown).
> 
> Similarly, it might be useful to have 'rbd feature show' command:
> 
>   rbd feature show <image-spec> [<feature>]
> 
> BTW, where do you think these default feature flags will be stored?
> Storing in pg_pool_t::flags I suppose is the easiest but it looks like
> a layering violation.
> 

I used 'mirror pool enable/disable' to keep all the related commands together.  I wasn't attempting to create a mechanism to specify arbitrary default features for a given pool, only the ability to enable mirroring (by default) on a given pool since that is the use case discussed at CDS. 

Image features can already be seen (along with lots of other image stats) via "rbd info <image-spec>".  Mirror pool settings can be seen via the previously proposed "mirror pool info" command.