experimental features

experimental features

[Sage Weil]

A while back we merged Haomai's experimental OSD backend KeyValueStore.  
We named the config option 'keyvaluestore_dev', hoping to make it clear to 
users that it was still under development, not fully tested, and not yet 
ready for production.  In retrospect, I don't think '_dev' was 
sufficiently scary because many users tried it and ran into 
unexpectd trouble.

There are several other features we've recently added or are considering 
adding that fall into this category.  Having them in the tree is great 
because it streamlines QA and testing, but I want to make sure that 
users are not able to enable the features without being aware of the 
risks.

A few possible suggestions:

- scarier option names, like

  osd objectstore = keyvaluestore_experimental_danger_danger
  ms type = async_experimental_danger_danger
  ms type = xio_experimental_danger_danger

  Once the feature becomes stable, they'll have to adjust their 
config, or we'll need to support both names going forward.

- a separate config option that allows any experimental option

  allow experimental features danger danger = true
  osd objectstore = keyvaluestore
  ms type = xio

  This runs the risk that the user will enable experimental features to 
get X, and later start using Y without realizing Y is also 
experiemental.

- enumerate experiemntal options we want to enable

  allow experimental features danger danger = keyvaluestore, xio
  ms type = xio
  osd objectstore = keyvaluestore

  This has the property that no config change is necessary when the 
feature drops its experimental status.

In all of these cases, we can also make a point of sending something to 
the log on daemon startup.  I don't think too many people will notice 
this, but it is better than nothing.

Other ideas?
sage

Re: experimental features

[Gregory Farnum]

On Fri, Dec 5, 2014 at 9:36 AM, Sage Weil <sweil@redhat.com> wrote:
> A while back we merged Haomai's experimental OSD backend KeyValueStore.
> We named the config option 'keyvaluestore_dev', hoping to make it clear to
> users that it was still under development, not fully tested, and not yet
> ready for production.  In retrospect, I don't think '_dev' was
> sufficiently scary because many users tried it and ran into
> unexpectd trouble.
>
> There are several other features we've recently added or are considering
> adding that fall into this category.  Having them in the tree is great
> because it streamlines QA and testing, but I want to make sure that
> users are not able to enable the features without being aware of the
> risks.
>
> A few possible suggestions:
>
> - scarier option names, like
>
>   osd objectstore = keyvaluestore_experimental_danger_danger
>   ms type = async_experimental_danger_danger
>   ms type = xio_experimental_danger_danger
>
>   Once the feature becomes stable, they'll have to adjust their
> config, or we'll need to support both names going forward.
>
> - a separate config option that allows any experimental option
>
>   allow experimental features danger danger = true
>   osd objectstore = keyvaluestore
>   ms type = xio
>
>   This runs the risk that the user will enable experimental features to
> get X, and later start using Y without realizing Y is also
> experiemental.
>
> - enumerate experiemntal options we want to enable
>
>   allow experimental features danger danger = keyvaluestore, xio
>   ms type = xio
>   osd objectstore = keyvaluestore
>
>   This has the property that no config change is necessary when the
> feature drops its experimental status.
>
> In all of these cases, we can also make a point of sending something to
> the log on daemon startup.  I don't think too many people will notice
> this, but it is better than nothing.
>
> Other ideas?

I don't think these should even be going into release packages for
users to work with. We can build them on the dev gitbuilders for QA
and testing without them ever reaching the hands of users grabbing our
production packages. ;)
-Greg

Re: experimental features

[Mark Nelson]

On 12/05/2014 11:39 AM, Gregory Farnum wrote:
> On Fri, Dec 5, 2014 at 9:36 AM, Sage Weil <sweil@redhat.com> wrote:
>> A while back we merged Haomai's experimental OSD backend KeyValueStore.
>> We named the config option 'keyvaluestore_dev', hoping to make it clear to
>> users that it was still under development, not fully tested, and not yet
>> ready for production.  In retrospect, I don't think '_dev' was
>> sufficiently scary because many users tried it and ran into
>> unexpectd trouble.
>>
>> There are several other features we've recently added or are considering
>> adding that fall into this category.  Having them in the tree is great
>> because it streamlines QA and testing, but I want to make sure that
>> users are not able to enable the features without being aware of the
>> risks.
>>
>> A few possible suggestions:
>>
>> - scarier option names, like
>>
>>    osd objectstore = keyvaluestore_experimental_danger_danger
>>    ms type = async_experimental_danger_danger
>>    ms type = xio_experimental_danger_danger
>>
>>    Once the feature becomes stable, they'll have to adjust their
>> config, or we'll need to support both names going forward.
>>
>> - a separate config option that allows any experimental option
>>
>>    allow experimental features danger danger = true
>>    osd objectstore = keyvaluestore
>>    ms type = xio
>>
>>    This runs the risk that the user will enable experimental features to
>> get X, and later start using Y without realizing Y is also
>> experiemental.
>>
>> - enumerate experiemntal options we want to enable
>>
>>    allow experimental features danger danger = keyvaluestore, xio
>>    ms type = xio
>>    osd objectstore = keyvaluestore
>>
>>    This has the property that no config change is necessary when the
>> feature drops its experimental status.
>>
>> In all of these cases, we can also make a point of sending something to
>> the log on daemon startup.  I don't think too many people will notice
>> this, but it is better than nothing.
>>
>> Other ideas?
>
> I don't think these should even be going into release packages for
> users to work with. We can build them on the dev gitbuilders for QA
> and testing without them ever reaching the hands of users grabbing our
> production packages. ;)
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

I'm in favor of the "allow experimental features" but instead call it:

"ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes things a 
little more explicit. With great power comes great responsibility.

Mark

Re: experimental features

[David Champion]

* On 05 Dec 2014, Sage Weil wrote: 
> adding that fall into this category.  Having them in the tree is great 
> because it streamlines QA and testing, but I want to make sure that 
> users are not able to enable the features without being aware of the 
> risks.
> 
> A few possible suggestions:
> 
> - scarier option names, like
> - a separate config option that allows any experimental option
> - enumerate experiemntal options we want to enable
> Other ideas?

A separate config file for experimental options:
/etc/ceph/danger-danger.conf

-- 
       David Champion • dgc@uchicago.edu • University of Chicago
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] experimental features

[Nigel Williams]

On Sat, Dec 6, 2014 at 4:36 AM, Sage Weil <sweil@redhat.com> wrote:
> - enumerate experiemntal options we want to enable
>...
>   This has the property that no config change is necessary when the
> feature drops its experimental status.

It keeps the risky options in one place too so easier to spot.

> In all of these cases, we can also make a point of sending something to
> the log on daemon startup.  I don't think too many people will notice
> this, but it is better than nothing.

Perhaps change the cluster health status to FRAGILE? or AT_RISK?

Re: experimental features

[Justin Erenkrantz]

On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson <mark.nelson@inktank.com> wrote:
> I'm in favor of the "allow experimental features" but instead call it:
>
> "ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes things a little
> more explicit. With great power comes great responsibility.

+1.

For Subversion, we utilize SVN_I_LOVE_CORRUPTED_XXX for a few options
that can cause data corruption.  -- justin

Re: experimental features

[Fred Yang]

[-- Attachment #1.1: Type: text/plain, Size: 1270 bytes --]

You will have to consider in the real world whoever built the cluster might
not document the dangerous option to let support stuff or successor aware.
Thus any experimental feature considered not safe for production should be
included in a warning message in 'ceph health', and logs, either log it
periodically or log the warning msg upon restart.
Feature-wise, 'ceph health detail' should give you a report over all
important features/options of the cluster as well.

-Fred

On Sun, Dec 7, 2014, 11:15 PM Justin Erenkrantz <justin-XWYNqclEDWm2WkX4dyxwcA@public.gmane.org>
wrote:

> On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson <mark.nelson-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org>
> wrote:
> > I'm in favor of the "allow experimental features" but instead call it:
> >
> > "ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes things a
> little
> > more explicit. With great power comes great responsibility.
>
> +1.
>
> For Subversion, we utilize SVN_I_LOVE_CORRUPTED_XXX for a few options
> that can cause data corruption.  -- justin
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

[-- Attachment #1.2: Type: text/html, Size: 1847 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: experimental features

[Mark Nelson]

I've been thinking for a while that we need another more general command 
than Ceph health to more generally inform you about your cluster.  IE I 
personally don't like having min/max PG warnings in Ceph health (they 
can be independently controlled by ceph.conf options but that kind of 
approach won't scale). I'd like another command that I can run that 
tells me about this kind of thing.  Same thing with experimental 
features.  I don't want ceph health warning me if they've been enabled, 
but I do want to know if they've ever been enabled, when, and whether 
they are still in effect.

Mark

On 12/08/2014 06:57 AM, Fred Yang wrote:
> You will have to consider in the real world whoever built the cluster
> might not document the dangerous option to let support stuff or
> successor aware. Thus any experimental feature considered not safe for
> production should be included in a warning message in 'ceph health', and
> logs, either log it periodically or log the warning msg upon restart.
> Feature-wise, 'ceph health detail' should give you a report over all
> important features/options of the cluster as well.
>
> -Fred
>
> On Sun, Dec 7, 2014, 11:15 PM Justin Erenkrantz <justin@erenkrantz.com
> <mailto:justin@erenkrantz.com>> wrote:
>
>     On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson
>     <mark.nelson@inktank.com <mailto:mark.nelson@inktank.com>> wrote:
>      > I'm in favor of the "allow experimental features" but instead
>     call it:
>      >
>      > "ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes things
>     a little
>      > more explicit. With great power comes great responsibility.
>
>     +1.
>
>     For Subversion, we utilize SVN_I_LOVE_CORRUPTED_XXX for a few options
>     that can cause data corruption.  -- justin
>     --
>     To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>     the body of a message to majordomo@vger.kernel.org
>     <mailto:majordomo@vger.kernel.org>
>     More majordomo info at http://vger.kernel.org/__majordomo-info.html
>     <http://vger.kernel.org/majordomo-info.html>
>

Re: [ceph-users] experimental features

[Christian Balzer]

On Mon, 08 Dec 2014 08:33:25 -0600 Mark Nelson wrote:

> I've been thinking for a while that we need another more general command 
> than Ceph health to more generally inform you about your cluster.  IE I 
> personally don't like having min/max PG warnings in Ceph health (they 
> can be independently controlled by ceph.conf options but that kind of 
> approach won't scale). I'd like another command that I can run that 
> tells me about this kind of thing.  Same thing with experimental 
> features.  I don't want ceph health warning me if they've been enabled, 
> but I do want to know if they've ever been enabled, when, and whether 
> they are still in effect.
> 

Very much agreed.

Expanding on this, setting a cluster to no-scrub will result in a warning
(very arguably) and while a slow request after 30 seconds is WRN worthy,
after something like a minute it ought to be ERR level as this is likely
to have massive impact on clients.

Christian

> Mark
> 
> On 12/08/2014 06:57 AM, Fred Yang wrote:
> > You will have to consider in the real world whoever built the cluster
> > might not document the dangerous option to let support stuff or
> > successor aware. Thus any experimental feature considered not safe for
> > production should be included in a warning message in 'ceph health',
> > and logs, either log it periodically or log the warning msg upon
> > restart. Feature-wise, 'ceph health detail' should give you a report
> > over all important features/options of the cluster as well.
> >
> > -Fred
> >
> > On Sun, Dec 7, 2014, 11:15 PM Justin Erenkrantz <justin@erenkrantz.com
> > <mailto:justin@erenkrantz.com>> wrote:
> >
> >     On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson
> >     <mark.nelson@inktank.com <mailto:mark.nelson@inktank.com>> wrote:
> >      > I'm in favor of the "allow experimental features" but instead
> >     call it:
> >      >
> >      > "ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes
> >      > things
> >     a little
> >      > more explicit. With great power comes great responsibility.
> >
> >     +1.
> >
> >     For Subversion, we utilize SVN_I_LOVE_CORRUPTED_XXX for a few
> > options that can cause data corruption.  -- justin
> >     --
> >     To unsubscribe from this list: send the line "unsubscribe
> > ceph-devel" in the body of a message to majordomo@vger.kernel.org
> >     <mailto:majordomo@vger.kernel.org>
> >     More majordomo info at http://vger.kernel.org/__majordomo-info.html
> >     <http://vger.kernel.org/majordomo-info.html>
> >
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 


-- 
Christian Balzer        Network/Systems Engineer                
chibi@gol.com   	Global OnLine Japan/Fusion Communications
http://www.gol.com/

Re: experimental features

[Sage Weil]

On Sun, 7 Dec 2014, Justin Erenkrantz wrote:
> On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson <mark.nelson-4GqslpFJ+cxBDgjK7y7TUQ@public.gmane.org> wrote:
> > I'm in favor of the "allow experimental features" but instead call it:
> >
> > "ALLOW UNRECOVERABLE DATA CORRUPTING FEATURES" which makes things a little
> > more explicit. With great power comes great responsibility.
> 
> +1.
> 
> For Subversion, we utilize SVN_I_LOVE_CORRUPTED_XXX for a few options
> that can cause data corruption.  -- justin

Thanks, I think we should go for this one.

1. a generic option

 enable unrecoverable data corrupting features = foo bar baz

2. rename keyvaluestore-dev to keyvaluestore (and require 
'osd-objectstore-keyvaluestore' it be listed above).

3. include 'ms-type-async', 'filestore-zfs-snap' as experimental.  And 
whatever else we want to flag...

sage