'Immutable bit' on pools to prevent deletion

'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

Hi,

Although the userland tools like 'ceph' and 'rados' have a safeguard
against fat fingers when it comes to removing a pool there is no such
safeguard when using native librados.

The danger still exists that by accident you remove a pool which is then
completely gone, no way to restore it.

This is still something I find quite dangerous, so I was thinking about
a additional 'Immutable bit' which could be set on a pool before
rados_pool_delete() allows this pool to be removed.

Is it a sane thing to look at 'features' which pools could have? Other
features which might be set on a pool:

- Read Only (all write operations return -EPERM)
- Delete Protected

It's just that looking at a 20TB RBD pool and thinking that just one API
call could remove this pool make me a bit scared.

Am I the only one or is this something worth looking in to?

-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: 'Immutable bit' on pools to prevent deletion

[Dan Van Der Ster]

Hi Wido,

+1 for safeguards.

Yeah that is scary: it's one api call to delete a pool, and perhaps even a client with w capability on a pool can delete it?? (I didn’t try...)

I can think of many ways that fat fingers can create crazy loads, deny client access, ...

 1. changing pool size
 2. setting pool quotas
 3. unplanned PG splitting
 4. creating an EC pool on a cluster with dumpling clients
 5. reweight-by-utilization
 6. changing crush rules/tunables

—yes-i-really-really-mean-it is nice when it’s there. But regardless it is probably not a good practice to work daily (or run librados cron jobs) in a shell that has access to the client.admin keyring. I’ve thought of using sudo to restrict our admin shell to subset of ceph admin commands. But even better would be a internal bit which locks out the API beneath “ceph osd pool …” and “ceph osd crush …”, even for client.admin.

Maybe this is already possible by creating a client.admin-readonly account for daily work and crons, and limit access to client.admin except when absolutely necessary ?

Cheers, Dan


> On 15 Jan 2015, at 15:46, Wido den Hollander <wido@42on.com> wrote:
> 
> Hi,
> 
> Although the userland tools like 'ceph' and 'rados' have a safeguard
> against fat fingers when it comes to removing a pool there is no such
> safeguard when using native librados.
> 
> The danger still exists that by accident you remove a pool which is then
> completely gone, no way to restore it.
> 
> This is still something I find quite dangerous, so I was thinking about
> a additional 'Immutable bit' which could be set on a pool before
> rados_pool_delete() allows this pool to be removed.
> 
> Is it a sane thing to look at 'features' which pools could have? Other
> features which might be set on a pool:
> 
> - Read Only (all write operations return -EPERM)
> - Delete Protected
> 
> It's just that looking at a 20TB RBD pool and thinking that just one API
> call could remove this pool make me a bit scared.
> 
> Am I the only one or is this something worth looking in to?
> 
> -- 
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Mike Dawson]

+1

Thanks,
Mike Dawson


On 1/15/2015 9:46 AM, Wido den Hollander wrote:
> Hi,
>
> Although the userland tools like 'ceph' and 'rados' have a safeguard
> against fat fingers when it comes to removing a pool there is no such
> safeguard when using native librados.
>
> The danger still exists that by accident you remove a pool which is then
> completely gone, no way to restore it.
>
> This is still something I find quite dangerous, so I was thinking about
> a additional 'Immutable bit' which could be set on a pool before
> rados_pool_delete() allows this pool to be removed.
>
> Is it a sane thing to look at 'features' which pools could have? Other
> features which might be set on a pool:
>
> - Read Only (all write operations return -EPERM)
> - Delete Protected
>
> It's just that looking at a 20TB RBD pool and thinking that just one API
> call could remove this pool make me a bit scared.
>
> Am I the only one or is this something worth looking in to?
>

Re: 'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

On 01/15/2015 04:39 PM, Dan Van Der Ster wrote:
> Hi Wido,
> 
> +1 for safeguards.
> 
> Yeah that is scary: it's one api call to delete a pool, and perhaps even a client with w capability on a pool can delete it?? (I didn’t try...)
> 

Quick try, yes! Created a pool and user which only has access to that
pool. I was able to remove that pool.

> I can think of many ways that fat fingers can create crazy loads, deny client access, ...
> 

Sure, but none of those actually make you loose your data.

I know that you should create backups, but by accident removing a pool
is something that is very dangerous and it will take a lot of time to
restore from backups.

>  1. changing pool size
>  2. setting pool quotas
>  3. unplanned PG splitting
>  4. creating an EC pool on a cluster with dumpling clients
>  5. reweight-by-utilization
>  6. changing crush rules/tunables
> 
> —yes-i-really-really-mean-it is nice when it’s there. But regardless it is probably not a good practice to work daily (or run librados cron jobs) in a shell that has access to the client.admin keyring. I’ve thought of using sudo to restrict our admin shell to subset of ceph admin commands. But even better would be a internal bit which locks out the API beneath “ceph osd pool …” and “ceph osd crush …”, even for client.admin.
> 
> Maybe this is already possible by creating a client.admin-readonly account for daily work and crons, and limit access to client.admin except when absolutely necessary ?
> 

That would be great indeed. The client.admin key currently has all the
capabilities and I would indeed like a RO account.

But still, another safeguard against deleting pools would be something
I'd like to see.

Wido

> Cheers, Dan
> 
> 
>> On 15 Jan 2015, at 15:46, Wido den Hollander <wido@42on.com> wrote:
>>
>> Hi,
>>
>> Although the userland tools like 'ceph' and 'rados' have a safeguard
>> against fat fingers when it comes to removing a pool there is no such
>> safeguard when using native librados.
>>
>> The danger still exists that by accident you remove a pool which is then
>> completely gone, no way to restore it.
>>
>> This is still something I find quite dangerous, so I was thinking about
>> a additional 'Immutable bit' which could be set on a pool before
>> rados_pool_delete() allows this pool to be removed.
>>
>> Is it a sane thing to look at 'features' which pools could have? Other
>> features which might be set on a pool:
>>
>> - Read Only (all write operations return -EPERM)
>> - Delete Protected
>>
>> It's just that looking at a 20TB RBD pool and thinking that just one API
>> call could remove this pool make me a bit scared.
>>
>> Am I the only one or is this something worth looking in to?
>>
>> -- 
>> Wido den Hollander
>> 42on B.V.
>> Ceph trainer and consultant
>>
>> Phone: +31 (0)20 700 9902
>> Skype: contact42on
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Yehuda Sadeh]

On Thu, Jan 15, 2015 at 6:46 AM, Wido den Hollander <wido@42on.com> wrote:
> Hi,
>
> Although the userland tools like 'ceph' and 'rados' have a safeguard
> against fat fingers when it comes to removing a pool there is no such
> safeguard when using native librados.
>
> The danger still exists that by accident you remove a pool which is then
> completely gone, no way to restore it.
>
> This is still something I find quite dangerous, so I was thinking about
> a additional 'Immutable bit' which could be set on a pool before
> rados_pool_delete() allows this pool to be removed.
>
> Is it a sane thing to look at 'features' which pools could have? Other
> features which might be set on a pool:
>
> - Read Only (all write operations return -EPERM)
> - Delete Protected
>
> It's just that looking at a 20TB RBD pool and thinking that just one API
> call could remove this pool make me a bit scared.
>
> Am I the only one or is this something worth looking in to?

I completely agree. A while back I opened an issue for that, #9792,
with some suggestions:

- require a special key for this command
- pool removal doesn't apply immediately, but rather first switches
pool to 'pending removal'
- pending removal state reflected in the cluster status
- operation can be cancelled when pending

Yehuda

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Thu, 15 Jan 2015, Yehuda Sadeh wrote:
> On Thu, Jan 15, 2015 at 6:46 AM, Wido den Hollander <wido@42on.com> wrote:
> > Hi,
> >
> > Although the userland tools like 'ceph' and 'rados' have a safeguard
> > against fat fingers when it comes to removing a pool there is no such
> > safeguard when using native librados.
> >
> > The danger still exists that by accident you remove a pool which is then
> > completely gone, no way to restore it.
> >
> > This is still something I find quite dangerous, so I was thinking about
> > a additional 'Immutable bit' which could be set on a pool before
> > rados_pool_delete() allows this pool to be removed.
> >
> > Is it a sane thing to look at 'features' which pools could have? Other
> > features which might be set on a pool:
> >
> > - Read Only (all write operations return -EPERM)
> > - Delete Protected
> >
> > It's just that looking at a 20TB RBD pool and thinking that just one API
> > call could remove this pool make me a bit scared.
> >
> > Am I the only one or is this something worth looking in to?
> 
> I completely agree. A while back I opened an issue for that, #9792,
> with some suggestions:
> 
> - require a special key for this command

The original version of the --yes-i-really-really-mean-it patch actually 
did something like this, but it was complicated and a huge pain to script 
all of the tests so we went with the simpler path (specify pool name twice 
and a scary option).

> - pool removal doesn't apply immediately, but rather first switches
> pool to 'pending removal'
> - pending removal state reflected in the cluster status
> - operation can be cancelled when pending

This amounts of a two stage process, which I think makes a lot of sense.  
I see two reasonably simple options:

1) Add a 'nodelete' pool flag, so that you have to do

 ceph osd pool set foo nodelete false
 ceph osd pool delete foo foo --yes-i-swear-i-etc

Then the question is whether we implicitly set this flag on all existing 
pools.. meh.  But we won't break any current users of the API that create 
and destroy pools.

2) Add a 'allow-delete' flag as Yehuda suggests, so you have

 ceph osd pool set foo allow-delete true
 ceph osd pool delete foo ...

Then secondary question is whether the cluster should implicitly clear the 
allow-delete after some time period (maybe 'pending-delete' would make 
more sense in that case), or whether we deny IO during that period.  Seems 
perhaps too complicated.

The downside here is that there will be huge patchset that fixes all the
tests to do this.  Probably worth it though if it avoids a very bad day 
for some poor admin...

sage

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

In addition (or instead of) making the API harder to fat-finger, we could 
also add a mon config option like

 mon allow pool deletion = false

that defaults off.  Then, to delete any pool, you need to update ceph.conf 
and restart mons or inject the config option change (ceph daemon 
mon.`hostname` conig set ... on the leader) or the API will give you 
EPERM. 

This offers some protection even for client.admin key users if we prevent 
injectargs for that option (maybe feasible) and they don't have access to 
the actual mon machine.

It also makes it easy to drop these protections for our QA because we can 
just stick that option in the teuthology and vstart.sh ceph.conf files.

sage

Re: 'Immutable bit' on pools to prevent deletion

[Gregory Farnum]

On Thu, Jan 15, 2015 at 9:44 AM, Sage Weil <sweil@redhat.com> wrote:
> In addition (or instead of) making the API harder to fat-finger, we could
> also add a mon config option like
>
>  mon allow pool deletion = false
>
> that defaults off.  Then, to delete any pool, you need to update ceph.conf
> and restart mons or inject the config option change (ceph daemon
> mon.`hostname` conig set ... on the leader) or the API will give you
> EPERM.
>
> This offers some protection even for client.admin key users if we prevent
> injectargs for that option (maybe feasible) and they don't have access to
> the actual mon machine.

What would that buy us? Preventing injectargs on it would require mon
restarts, which is unfortunate — and makes it sounds more like a
security feature than a safety blanket.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Thu, 15 Jan 2015, Gregory Farnum wrote:
> On Thu, Jan 15, 2015 at 9:44 AM, Sage Weil <sweil@redhat.com> wrote:
> > In addition (or instead of) making the API harder to fat-finger, we could
> > also add a mon config option like
> >
> >  mon allow pool deletion = false
> >
> > that defaults off.  Then, to delete any pool, you need to update ceph.conf
> > and restart mons or inject the config option change (ceph daemon
> > mon.`hostname` conig set ... on the leader) or the API will give you
> > EPERM.
> >
> > This offers some protection even for client.admin key users if we prevent
> > injectargs for that option (maybe feasible) and they don't have access to
> > the actual mon machine.
> 
> What would that buy us? Preventing injectargs on it would require mon
> restarts, which is unfortunate ? and makes it sounds more like a
> security feature than a safety blanket.

I meant 'ceph tell mon.* injectargs ...' as distinct from 'ceph daemon ... 
config set', which requires access to the host.  But yeah, if we went to 
the effort to limit injectargs (maybe a blanket option that disables 
injectargs on mons?), it could double as a security feature.

But whether it may also useful for security doesn't change whether it is a 
good safety blanket.  I like it because it's simple, easy to implement, 
and easy to disable for testing... :)

sage

Re: 'Immutable bit' on pools to prevent deletion

[Gregory Farnum]

Oh, I think it might be fine to require setting a config option before
you delete stuff; I just don't want to prevent the option from being
injectable. :)

On Thu, Jan 15, 2015 at 10:07 AM, Sage Weil <sweil@redhat.com> wrote:
> On Thu, 15 Jan 2015, Gregory Farnum wrote:
>> On Thu, Jan 15, 2015 at 9:44 AM, Sage Weil <sweil@redhat.com> wrote:
>> > In addition (or instead of) making the API harder to fat-finger, we could
>> > also add a mon config option like
>> >
>> >  mon allow pool deletion = false
>> >
>> > that defaults off.  Then, to delete any pool, you need to update ceph.conf
>> > and restart mons or inject the config option change (ceph daemon
>> > mon.`hostname` conig set ... on the leader) or the API will give you
>> > EPERM.
>> >
>> > This offers some protection even for client.admin key users if we prevent
>> > injectargs for that option (maybe feasible) and they don't have access to
>> > the actual mon machine.
>>
>> What would that buy us? Preventing injectargs on it would require mon
>> restarts, which is unfortunate ? and makes it sounds more like a
>> security feature than a safety blanket.
>
> I meant 'ceph tell mon.* injectargs ...' as distinct from 'ceph daemon ...
> config set', which requires access to the host.  But yeah, if we went to
> the effort to limit injectargs (maybe a blanket option that disables
> injectargs on mons?), it could double as a security feature.
>
> But whether it may also useful for security doesn't change whether it is a
> good safety blanket.  I like it because it's simple, easy to implement,
> and easy to disable for testing... :)
>
> sage
>

Re: 'Immutable bit' on pools to prevent deletion

[John Spray]

On Thu, Jan 15, 2015 at 6:07 PM, Sage Weil <sweil@redhat.com> wrote:
>> What would that buy us? Preventing injectargs on it would require mon
>> restarts, which is unfortunate ? and makes it sounds more like a
>> security feature than a safety blanket.
>
> I meant 'ceph tell mon.* injectargs ...' as distinct from 'ceph daemon ...
> config set', which requires access to the host.  But yeah, if we went to
> the effort to limit injectargs (maybe a blanket option that disables
> injectargs on mons?), it could double as a security feature.
>
> But whether it may also useful for security doesn't change whether it is a
> good safety blanket.  I like it because it's simple, easy to implement,
> and easy to disable for testing... :)

The trouble with this is admin socket part is that any tool that
manages Ceph must use the admin socket interface as well as the normal
over-the-network command interface, and by extension must be able to
execute locally on a mon.  We would no longer have a comprehensive
remote management interface for the mon: management tools would have
to run some code locally too.

I think it's sufficient to require two API calls (set the flag or
config option, then do the delete) within the remote API, rather than
requiring that anyone driving the interface knows how to speak two
network protocols (usual mon remote command + SSH-to-asok).

John

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Thu, 15 Jan 2015, John Spray wrote:
> On Thu, Jan 15, 2015 at 6:07 PM, Sage Weil <sweil@redhat.com> wrote:
> >> What would that buy us? Preventing injectargs on it would require mon
> >> restarts, which is unfortunate ? and makes it sounds more like a
> >> security feature than a safety blanket.
> >
> > I meant 'ceph tell mon.* injectargs ...' as distinct from 'ceph daemon ...
> > config set', which requires access to the host.  But yeah, if we went to
> > the effort to limit injectargs (maybe a blanket option that disables
> > injectargs on mons?), it could double as a security feature.
> >
> > But whether it may also useful for security doesn't change whether it is a
> > good safety blanket.  I like it because it's simple, easy to implement,
> > and easy to disable for testing... :)
> 
> The trouble with this is admin socket part is that any tool that
> manages Ceph must use the admin socket interface as well as the normal
> over-the-network command interface, and by extension must be able to
> execute locally on a mon.  We would no longer have a comprehensive
> remote management interface for the mon: management tools would have
> to run some code locally too.

True.. if we make that option enabled by default.  If we it's off by 
default them it's an opt-in layer of protection.  Most clusters don't have 
ephemeral pools so I think lots of people would want this.
 
> I think it's sufficient to require two API calls (set the flag or
> config option, then do the delete) within the remote API, rather than
> requiring that anyone driving the interface knows how to speak two
> network protocols (usual mon remote command + SSH-to-asok).

Yeah...

sage

Re: 'Immutable bit' on pools to prevent deletion

[John Spray]

On Thu, Jan 15, 2015 at 7:07 PM, Sage Weil <sweil@redhat.com> wrote:
>> The trouble with this is admin socket part is that any tool that
>> manages Ceph must use the admin socket interface as well as the normal
>> over-the-network command interface, and by extension must be able to
>> execute locally on a mon.  We would no longer have a comprehensive
>> remote management interface for the mon: management tools would have
>> to run some code locally too.
>
> True.. if we make that option enabled by default.  If we it's off by
> default them it's an opt-in layer of protection.  Most clusters don't have
> ephemeral pools so I think lots of people would want this.

+1, the problem goes away if it's opt-in, should be easy enough for
API consumers to inspect the conf setting and give a nice informative
error if the safety catch is engaged.  I can imagine wanting to engage
this ahead of plugging in a GUI or some config management recipes that
you didn't quite trust yet.

John

Re: 'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

On 01/15/2015 08:07 PM, Sage Weil wrote:
> On Thu, 15 Jan 2015, John Spray wrote:
>> On Thu, Jan 15, 2015 at 6:07 PM, Sage Weil <sweil@redhat.com> wrote:
>>>> What would that buy us? Preventing injectargs on it would require mon
>>>> restarts, which is unfortunate ? and makes it sounds more like a
>>>> security feature than a safety blanket.
>>>
>>> I meant 'ceph tell mon.* injectargs ...' as distinct from 'ceph daemon ...
>>> config set', which requires access to the host.  But yeah, if we went to
>>> the effort to limit injectargs (maybe a blanket option that disables
>>> injectargs on mons?), it could double as a security feature.
>>>
>>> But whether it may also useful for security doesn't change whether it is a
>>> good safety blanket.  I like it because it's simple, easy to implement,
>>> and easy to disable for testing... :)
>>
>> The trouble with this is admin socket part is that any tool that
>> manages Ceph must use the admin socket interface as well as the normal
>> over-the-network command interface, and by extension must be able to
>> execute locally on a mon.  We would no longer have a comprehensive
>> remote management interface for the mon: management tools would have
>> to run some code locally too.
> 
> True.. if we make that option enabled by default.  If we it's off by 
> default them it's an opt-in layer of protection.  Most clusters don't have 
> ephemeral pools so I think lots of people would want this.
>  

If this is the easiest route I'm +1 for that way.

I'd turn it off right away on the clusters I manage. Pools don't change
that often and I simply want another safeguard against deleting them.

>> I think it's sufficient to require two API calls (set the flag or
>> config option, then do the delete) within the remote API, rather than
>> requiring that anyone driving the interface knows how to speak two
>> network protocols (usual mon remote command + SSH-to-asok).
> 
> Yeah...
> 
> sage
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: 'Immutable bit' on pools to prevent deletion

[Sebastien Han]

Hum if I understand correctly you’re all more in favour of a conf setting in the ceph.conf;
The problem for me is that this will apply to all the pools by default and I’ll have to inject an arg to change this.
Injecting the arg will remove this “lock" and then all of the sudden all the pools become deletable through the lib again (who knows what users can do simultaneously)

I’m more in favour of a new flag to set to the pool, something like:

ceph osd pool set foo protect true
ceph osd pool delete foo foo —yes….
ERROR: pool foo is protected against deletion

ceph osd pool delete foo protect false
ceph osd pool delete foo foo —yes….
Pool successfully deleted

The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
We stick with the "ceph osd pool delete foo foo —yes….” command as well, so we don’t change too much things.

Moreover we can also make use of a config option to protect all new created pools by default:

mon protect pool default = true

This automatically set the protected flag to a new pool.

What do you think?

> On 15 Jan 2015, at 18:24, Sage Weil <sage@newdream.net> wrote:
> 
> Then secondary question is whether the cluster should implicitly clear the 
> allow-delete after some time period (maybe 'pending-delete' would make 
> more sense in that case), or whether we deny IO during that period.  Seems 
> perhaps too complicated.


Cheers.
–––– 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien.han@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

On 01/16/2015 10:50 AM, Sebastien Han wrote:
> Hum if I understand correctly you’re all more in favour of a conf setting in the ceph.conf;
> The problem for me is that this will apply to all the pools by default and I’ll have to inject an arg to change this.
> Injecting the arg will remove this “lock" and then all of the sudden all the pools become deletable through the lib again (who knows what users can do simultaneously)
> 

No, from what I understand it's easier to implement, not the better way.

> I’m more in favour of a new flag to set to the pool, something like:
> 
> ceph osd pool set foo protect true
> ceph osd pool delete foo foo —yes….
> ERROR: pool foo is protected against deletion
> 
> ceph osd pool delete foo protect false
> ceph osd pool delete foo foo —yes….
> Pool successfully deleted
>

Something like that per pool seems better to me as well. But I'd then
opt for a 'feature' which can be set on a pool.

ceph osd pool set foo nodelete
ceph osd pool set foo nopgchange
ceph osd pool set foo nosizechange


> The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
> We stick with the "ceph osd pool delete foo foo —yes….” command as well, so we don’t change too much things.
> 
> Moreover we can also make use of a config option to protect all new created pools by default:
> 
> mon protect pool default = true
> 
> This automatically set the protected flag to a new pool.
> 
> What do you think?
> 

Setting a nodelete flag or something like that by default is fine with
me. Like Sage mentioned earlier, almost nobody will have ephemeral pools
in their cluster. You don't want to loose data because you accidentally
removed a pool.

Wido

>> On 15 Jan 2015, at 18:24, Sage Weil <sage@newdream.net> wrote:
>>
>> Then secondary question is whether the cluster should implicitly clear the 
>> allow-delete after some time period (maybe 'pending-delete' would make 
>> more sense in that case), or whether we deny IO during that period.  Seems 
>> perhaps too complicated.
> 
> 
> Cheers.
> –––– 
> Sébastien Han 
> Cloud Architect 
> 
> "Always give 100%. Unless you're giving blood."
> 
> Phone: +33 (0)1 49 70 99 72 
> Mail: sebastien.han@enovance.com 
> Address : 11 bis, rue Roquépine - 75008 Paris
> Web : www.enovance.com - Twitter : @enovance 
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Fri, 16 Jan 2015, Wido den Hollander wrote:
> On 01/16/2015 10:50 AM, Sebastien Han wrote:
> > Hum if I understand correctly you?re all more in favour of a conf setting in the ceph.conf;
> > The problem for me is that this will apply to all the pools by default and I?ll have to inject an arg to change this.
> > Injecting the arg will remove this ?lock" and then all of the sudden all the pools become deletable through the lib again (who knows what users can do simultaneously)
> > 
> 
> No, from what I understand it's easier to implement, not the better way.

I'd like to do both, actually.  :)

> > I?m more in favour of a new flag to set to the pool, something like:
> > 
> > ceph osd pool set foo protect true
> > ceph osd pool delete foo foo ?yes?.
> > ERROR: pool foo is protected against deletion
> > 
> > ceph osd pool delete foo protect false
> > ceph osd pool delete foo foo ?yes?.
> > Pool successfully deleted
> >
> 
> Something like that per pool seems better to me as well. But I'd then
> opt for a 'feature' which can be set on a pool.
> 
> ceph osd pool set foo nodelete
> ceph osd pool set foo nopgchange
> ceph osd pool set foo nosizechange

I like this since it fits into the current flags nicely.  The downside is 
we don't grandfather existing pools on upgrade.  Not sure if people think 
that's a good idea.

> > The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
> > We stick with the "ceph osd pool delete foo foo ?yes?.? command as well, so we don?t change too much things.
> > 
> > Moreover we can also make use of a config option to protect all new created pools by default:
> > 
> > mon protect pool default = true
> > 
> > This automatically set the protected flag to a new pool.
> > 
> > What do you think?
> > 
> 
> Setting a nodelete flag or something like that by default is fine with
> me. Like Sage mentioned earlier, almost nobody will have ephemeral pools
> in their cluster. You don't want to loose data because you accidentally
> removed a pool.

We should mirror this option:

OPTION(osd_pool_default_flag_hashpspool, OPT_BOOL, true)   // use new pg 
hashing to prevent pool/pg overlap

So,

osd_pool_default_flag_nodelete = true
osd_pool_default_flag_nopgchange = true
osd_pool_default_flag_nosizechange = true

The big question for me is should we enable these by default in hammer?

sage



> 
> Wido
> 
> >> On 15 Jan 2015, at 18:24, Sage Weil <sage@newdream.net> wrote:
> >>
> >> Then secondary question is whether the cluster should implicitly clear the 
> >> allow-delete after some time period (maybe 'pending-delete' would make 
> >> more sense in that case), or whether we deny IO during that period.  Seems 
> >> perhaps too complicated.
> > 
> > 
> > Cheers.
> > ???? 
> > S?bastien Han 
> > Cloud Architect 
> > 
> > "Always give 100%. Unless you're giving blood."
> > 
> > Phone: +33 (0)1 49 70 99 72 
> > Mail: sebastien.han@enovance.com 
> > Address : 11 bis, rue Roqu?pine - 75008 Paris
> > Web : www.enovance.com - Twitter : @enovance 
> > 
> 
> 
> -- 
> Wido den Hollander
> 42on B.V.
> Ceph trainer and consultant
> 
> Phone: +31 (0)20 700 9902
> Skype: contact42on
> 
> 

Re: 'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

> Op 16 jan. 2015 om 15:47 heeft Sage Weil <sage@newdream.net> het volgende geschreven:
> 
>> On Fri, 16 Jan 2015, Wido den Hollander wrote:
>>> On 01/16/2015 10:50 AM, Sebastien Han wrote:
>>> Hum if I understand correctly you?re all more in favour of a conf setting in the ceph.conf;
>>> The problem for me is that this will apply to all the pools by default and I?ll have to inject an arg to change this.
>>> Injecting the arg will remove this ?lock" and then all of the sudden all the pools become deletable through the lib again (who knows what users can do simultaneously)
>> 
>> No, from what I understand it's easier to implement, not the better way.
> 
> I'd like to do both, actually.  :)
> 

Sounds good!

>>> I?m more in favour of a new flag to set to the pool, something like:
>>> 
>>> ceph osd pool set foo protect true
>>> ceph osd pool delete foo foo ?yes?.
>>> ERROR: pool foo is protected against deletion
>>> 
>>> ceph osd pool delete foo protect false
>>> ceph osd pool delete foo foo ?yes?.
>>> Pool successfully deleted
>> 
>> Something like that per pool seems better to me as well. But I'd then
>> opt for a 'feature' which can be set on a pool.
>> 
>> ceph osd pool set foo nodelete
>> ceph osd pool set foo nopgchange
>> ceph osd pool set foo nosizechange
> 
> I like this since it fits into the current flags nicely.  The downside is 
> we don't grandfather existing pools on upgrade.  Not sure if people think 
> that's a good idea.
> 
>>> The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
>>> We stick with the "ceph osd pool delete foo foo ?yes?.? command as well, so we don?t change too much things.
>>> 
>>> Moreover we can also make use of a config option to protect all new created pools by default:
>>> 
>>> mon protect pool default = true
>>> 
>>> This automatically set the protected flag to a new pool.
>>> 
>>> What do you think?
>> 
>> Setting a nodelete flag or something like that by default is fine with
>> me. Like Sage mentioned earlier, almost nobody will have ephemeral pools
>> in their cluster. You don't want to loose data because you accidentally
>> removed a pool.
> 
> We should mirror this option:
> 
> OPTION(osd_pool_default_flag_hashpspool, OPT_BOOL, true)   // use new pg 
> hashing to prevent pool/pg overlap
> 
> So,
> 
> osd_pool_default_flag_nodelete = true
> osd_pool_default_flag_nopgchange = true
> osd_pool_default_flag_nosizechange = true
> 
> The big question for me is should we enable these by default in hammer?
> 

I would say yes. We should probably protect users against something stupid which makes them loose data.

Data safety is prio #1

Wido


> sage
> 
> 
> 
>> 
>> Wido
>> 
>>>> On 15 Jan 2015, at 18:24, Sage Weil <sage@newdream.net> wrote:
>>>> 
>>>> Then secondary question is whether the cluster should implicitly clear the 
>>>> allow-delete after some time period (maybe 'pending-delete' would make 
>>>> more sense in that case), or whether we deny IO during that period.  Seems 
>>>> perhaps too complicated.
>>> 
>>> 
>>> Cheers.
>>> ???? 
>>> S?bastien Han 
>>> Cloud Architect 
>>> 
>>> "Always give 100%. Unless you're giving blood."
>>> 
>>> Phone: +33 (0)1 49 70 99 72 
>>> Mail: sebastien.han@enovance.com 
>>> Address : 11 bis, rue Roqu?pine - 75008 Paris
>>> Web : www.enovance.com - Twitter : @enovance
>> 
>> 
>> -- 
>> Wido den Hollander
>> 42on B.V.
>> Ceph trainer and consultant
>> 
>> Phone: +31 (0)20 700 9902
>> Skype: contact42on
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Alex Elsayed]

Wido den Hollander wrote:

<snip>
> Is it a sane thing to look at 'features' which pools could have? Other
> features which might be set on a pool:
> 
> - Read Only (all write operations return -EPERM)
> - Delete Protected

There's another pool feature I'd find very useful: a WORM flag, that permits 
only create & append (at the RADOS level, not the RBD level as was an 
Emperor blueprint).

In particular, I'd _love_ being able to make something that takes Postgres 
WAL logs and puts them in such a pool, providing real guarantees re: 
consistency. Similarly, audit logs and such for compliance.

Re: 'Immutable bit' on pools to prevent deletion

[Wido den Hollander]

On 01/17/2015 03:31 AM, Alex Elsayed wrote:
> Wido den Hollander wrote:
> 
> <snip>
>> Is it a sane thing to look at 'features' which pools could have? Other
>> features which might be set on a pool:
>>
>> - Read Only (all write operations return -EPERM)
>> - Delete Protected
> 
> There's another pool feature I'd find very useful: a WORM flag, that permits 
> only create & append (at the RADOS level, not the RBD level as was an 
> Emperor blueprint).
> 

Yes, that seems like a good addition. If we introduce the system of
'flags' for pools such a flag could be implemented as well.

> In particular, I'd _love_ being able to make something that takes Postgres 
> WAL logs and puts them in such a pool, providing real guarantees re: 
> consistency. Similarly, audit logs and such for compliance.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


-- 
Wido den Hollander
42on B.V.
Ceph trainer and consultant

Phone: +31 (0)20 700 9902
Skype: contact42on

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Fri, 16 Jan 2015, Alex Elsayed wrote:
> Wido den Hollander wrote:
> 
> <snip>
> > Is it a sane thing to look at 'features' which pools could have? Other
> > features which might be set on a pool:
> > 
> > - Read Only (all write operations return -EPERM)
> > - Delete Protected
> 
> There's another pool feature I'd find very useful: a WORM flag, that permits 
> only create & append (at the RADOS level, not the RBD level as was an 
> Emperor blueprint).
> 
> In particular, I'd _love_ being able to make something that takes Postgres 
> WAL logs and puts them in such a pool, providing real guarantees re: 
> consistency. Similarly, audit logs and such for compliance.

How would you want this to work?

- If the bit is set, object creates are allowed, but not deletes?  What 
about append?

- Are you allowed to clear the bit with something like 'ceph osd pool set 
<pool> worm false' ?

sage

Re: 'Immutable bit' on pools to prevent deletion

[Alex Elsayed]

Sage Weil wrote:

> On Fri, 16 Jan 2015, Alex Elsayed wrote:
>> Wido den Hollander wrote:
>> 
>> <snip>
>> > Is it a sane thing to look at 'features' which pools could have? Other
>> > features which might be set on a pool:
>> > 
>> > - Read Only (all write operations return -EPERM)
>> > - Delete Protected
>> 
>> There's another pool feature I'd find very useful: a WORM flag, that
>> permits only create & append (at the RADOS level, not the RBD level as
>> was an Emperor blueprint).
>> 
>> In particular, I'd _love_ being able to make something that takes
>> Postgres WAL logs and puts them in such a pool, providing real guarantees
>> re: consistency. Similarly, audit logs and such for compliance.
> 
> How would you want this to work?
> 
> - If the bit is set, object creates are allowed, but not deletes?  What
> about append?
> 
> - Are you allowed to clear the bit with something like 'ceph osd pool set
> <pool> worm false' ?

I'd say that a WORM pool would allow 'create' and 'append' only - that fits 
well with the classic notions of WORM media, and would allow natural 
implementations of virtualized WORM tape libraries and such for people who 
need compatibility (if only object creation was supported, you get issues 
where either you have absurd tiny objects or risk data loss on failure to 
write a larger buffered chunk). Similarly, audit records (like selinux logs, 
that might be written by log daemons) don't really come in nice object-sized 
chunks. You want to always have the newest in the archive, so you really 
don't want to buffer up to that.

I'd also figure that WORM's main purpose is assurance/compliance - you want 
to _know_ that nobody could have turned the bit off, futzed with the data, 
and then turned it back on. Otherwise, you'd just write your clients to only 
use create/append, and have no need for WORM at the pool level. Because of 
that, if the flag can be cleared via commands, it should be possible for the 
admins to forbid it (by flat denying it in config, via some keying system, 
via the bits being a ratchet, whatever - I'm not especially concerned by how 
the guarantee is provided, so long as it can be).

Setting it should probably also be privileged, since it'd be trivial to 
cause a DOS by setting it on (say) a CephFS pool - although handling that 
concern is likely out-of-scope for now, since there are easier ways to ruin 
someone's day at the RADOS level.

Re: 'Immutable bit' on pools to prevent deletion

[David Zafman]

The most secure way would be one in which you can only create pools with 
WORM set and can't ever change the WORM state of a pool.  I like this 
simple/secure approach as a first cut.

David

On 1/17/15 11:09 AM, Alex Elsayed wrote:
> Sage Weil wrote:
>
>> On Fri, 16 Jan 2015, Alex Elsayed wrote:
>>> Wido den Hollander wrote:
>>>
>>> <snip>
>>>> Is it a sane thing to look at 'features' which pools could have? Other
>>>> features which might be set on a pool:
>>>>
>>>> - Read Only (all write operations return -EPERM)
>>>> - Delete Protected
>>> There's another pool feature I'd find very useful: a WORM flag, that
>>> permits only create & append (at the RADOS level, not the RBD level as
>>> was an Emperor blueprint).
>>>
>>> In particular, I'd _love_ being able to make something that takes
>>> Postgres WAL logs and puts them in such a pool, providing real guarantees
>>> re: consistency. Similarly, audit logs and such for compliance.
>> How would you want this to work?
>>
>> - If the bit is set, object creates are allowed, but not deletes?  What
>> about append?
>>
>> - Are you allowed to clear the bit with something like 'ceph osd pool set
>> <pool> worm false' ?
> I'd say that a WORM pool would allow 'create' and 'append' only - that fits
> well with the classic notions of WORM media, and would allow natural
> implementations of virtualized WORM tape libraries and such for people who
> need compatibility (if only object creation was supported, you get issues
> where either you have absurd tiny objects or risk data loss on failure to
> write a larger buffered chunk). Similarly, audit records (like selinux logs,
> that might be written by log daemons) don't really come in nice object-sized
> chunks. You want to always have the newest in the archive, so you really
> don't want to buffer up to that.
>
> I'd also figure that WORM's main purpose is assurance/compliance - you want
> to _know_ that nobody could have turned the bit off, futzed with the data,
> and then turned it back on. Otherwise, you'd just write your clients to only
> use create/append, and have no need for WORM at the pool level. Because of
> that, if the flag can be cleared via commands, it should be possible for the
> admins to forbid it (by flat denying it in config, via some keying system,
> via the bits being a ratchet, whatever - I'm not especially concerned by how
> the guarantee is provided, so long as it can be).
>
> Setting it should probably also be privileged, since it'd be trivial to
> cause a DOS by setting it on (say) a CephFS pool - although handling that
> concern is likely out-of-scope for now, since there are easier ways to ruin
> someone's day at the RADOS level.
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: 'Immutable bit' on pools to prevent deletion

[Mykola Golub]

On Fri, Jan 16, 2015 at 06:46:31AM -0800, Sage Weil wrote:
> On Fri, 16 Jan 2015, Wido den Hollander wrote:
> >
> > Something like that per pool seems better to me as well. But I'd then
> > opt for a 'feature' which can be set on a pool.
> > 
> > ceph osd pool set foo nodelete
> > ceph osd pool set foo nopgchange
> > ceph osd pool set foo nosizechange
> 
> I like this since it fits into the current flags nicely.  The downside is 
> we don't grandfather existing pools on upgrade.  Not sure if people think 
> that's a good idea.
> 
> > > The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
> > > We stick with the "ceph osd pool delete foo foo ?yes?.? command as well, so we don?t change too much things.
> > > 
> > > Moreover we can also make use of a config option to protect all new created pools by default:
> > > 
> > > mon protect pool default = true
> > > 
> > > This automatically set the protected flag to a new pool.
> > > 
> > > What do you think?
> > > 
> > 
> > Setting a nodelete flag or something like that by default is fine with
> > me. Like Sage mentioned earlier, almost nobody will have ephemeral pools
> > in their cluster. You don't want to loose data because you accidentally
> > removed a pool.
> 
> We should mirror this option:
> 
> OPTION(osd_pool_default_flag_hashpspool, OPT_BOOL, true)   // use new pg 
> hashing to prevent pool/pg overlap
> 
> So,
> 
> osd_pool_default_flag_nodelete = true
> osd_pool_default_flag_nopgchange = true
> osd_pool_default_flag_nosizechange = true
> 
> The big question for me is should we enable these by default in hammer?

Please see https://github.com/ceph/ceph/pull/3409

I have not enabled any by default.

Also, I have not mirrored CEPH_FEATURE_OSDHASHPSPOOL, should I?

-- 
Mykola Golub

Re: 'Immutable bit' on pools to prevent deletion

[Sage Weil]

On Mon, 19 Jan 2015, Mykola Golub wrote:
> On Fri, Jan 16, 2015 at 06:46:31AM -0800, Sage Weil wrote:
> > On Fri, 16 Jan 2015, Wido den Hollander wrote:
> > >
> > > Something like that per pool seems better to me as well. But I'd then
> > > opt for a 'feature' which can be set on a pool.
> > > 
> > > ceph osd pool set foo nodelete
> > > ceph osd pool set foo nopgchange
> > > ceph osd pool set foo nosizechange
> > 
> > I like this since it fits into the current flags nicely.  The downside is 
> > we don't grandfather existing pools on upgrade.  Not sure if people think 
> > that's a good idea.
> > 
> > > > The good thing with that is that owners of the pool (or admin), will be able to set this flag or remove it.
> > > > We stick with the "ceph osd pool delete foo foo ?yes?.? command as well, so we don?t change too much things.
> > > > 
> > > > Moreover we can also make use of a config option to protect all new created pools by default:
> > > > 
> > > > mon protect pool default = true
> > > > 
> > > > This automatically set the protected flag to a new pool.
> > > > 
> > > > What do you think?
> > > > 
> > > 
> > > Setting a nodelete flag or something like that by default is fine with
> > > me. Like Sage mentioned earlier, almost nobody will have ephemeral pools
> > > in their cluster. You don't want to loose data because you accidentally
> > > removed a pool.
> > 
> > We should mirror this option:
> > 
> > OPTION(osd_pool_default_flag_hashpspool, OPT_BOOL, true)   // use new pg 
> > hashing to prevent pool/pg overlap
> > 
> > So,
> > 
> > osd_pool_default_flag_nodelete = true
> > osd_pool_default_flag_nopgchange = true
> > osd_pool_default_flag_nosizechange = true
> > 
> > The big question for me is should we enable these by default in hammer?
> 
> Please see https://github.com/ceph/ceph/pull/3409
> 
> I have not enabled any by default.

Looks great!

> Also, I have not mirrored CEPH_FEATURE_OSDHASHPSPOOL, should I?

No need... this is all in the mon and there are no compatibility issues 
that I can see.

Thanks!
sage