cephx auth question

cephx auth question

[John Axel Eriksson]

I asked a similar question in a previous email but I didn't get any
satisfying answers. What exactly does cephx auth secure?
From the wiki I just get "this makes your cluster more secure", well
from what? If I run on an internal network accessible only
by a few trusted people - what does cephx auth secure in such a scenario?

In that previous email I got the answer that it secures the cluster
from mistakenly connecting to wrong cluster with rados and
accidentally deleting a pool... well, can rados really "accidentally"
connect to the wrong cluster? Only if I have the wrong config
file right? And if I have the wrong config-file won't it be possible
that I also have the "wrong" key in that case?

Another scenario would be if I take down an OSD that just sits in
storage for say 6 months and then someone starts that machine
again - with key-based auth that OSD wouldn't be able to
connect(somehow? but if it has a working key?) but without auth it
could
possibly connect and wreak havoc in the cluster (since it is so much
behind perhaps in both version of software and what's stored on it).
I thought marking and OSD as down or out would do that?

Are those the main reasons for having cephx auth? Is it to secure the
cluster against people (myself included) making mistakes or  from
hacking, or is there some technical reason that I don't know of or understand?

The reason I'm asking is because having cephx enabled makes cluster
setup much more complicated...


Thanks,

John

Re: cephx auth question

[Wido den Hollander]

Hi,

On 06/12/2012 11:31 AM, John Axel Eriksson wrote:
> I asked a similar question in a previous email but I didn't get any
> satisfying answers. What exactly does cephx auth secure?

I wanted to get back on your e-mail from yesterday, but you beat me to 
it! :)

>  From the wiki I just get "this makes your cluster more secure", well
> from what? If I run on an internal network accessible only
> by a few trusted people - what does cephx auth secure in such a scenario?
>

Never ever trust anything, even in a local network. Security has more 
aspects then just uninvited guests accessing or modifying your data.

Like I said in the previous e-mail, it prevents you from having an old 
machine or just a plain configuration error on a client disrupting 
cluster service.

> In that previous email I got the answer that it secures the cluster
> from mistakenly connecting to wrong cluster with rados and
> accidentally deleting a pool... well, can rados really "accidentally"
> connect to the wrong cluster? Only if I have the wrong config
> file right? And if I have the wrong config-file won't it be possible
> that I also have the "wrong" key in that case?
>

The "rados" command doesn't really need a config file, you can also 
specify the monitor address manually.

$ rados -p <mon address> rmpool data

If you'd issue that with the wrong monitor address while playing in your 
local network you could remove the pool on the wrong cluster.

With cephx in place you also have to specify the correct key, it's just 
a second layer in place from preventing something like this happening.

> Another scenario would be if I take down an OSD that just sits in
> storage for say 6 months and then someone starts that machine
> again - with key-based auth that OSD wouldn't be able to
> connect(somehow? but if it has a working key?) but without auth it
> could
> possibly connect and wreak havoc in the cluster (since it is so much
> behind perhaps in both version of software and what's stored on it).
> I thought marking and OSD as down or out would do that?
>

If you mark an OSD out it will get marked as "in" again when you start 
up the OSD, but that also depends on your settings like "noautoin".

When you take an OSD out of production with cephx you would also remove 
the corresponding OSD key from the cluster keyring.

> Are those the main reasons for having cephx auth? Is it to secure the
> cluster against people (myself included) making mistakes or  from
> hacking, or is there some technical reason that I don't know of or understand?
>

It can serve multiple purposes. In a bigger env you should never trust 
any machine. So if you have a client which only has access to the "rbd" 
pool for running Virtual Machines, it shouldn't get more permissions 
than that.

So yes, it's again human error, but if a machine gets hacked, you can 
prevent that machine from ruining your whole cluster.

> The reason I'm asking is because having cephx enabled makes cluster
> setup much more complicated...
>

cephx might be complicated at first glance, but it isn't really that hard.

It is up to you if you enable it, but take the recommendations above 
into account when making your decision.

Wido

>
> Thanks,
>
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cephx auth question

[John Axel Eriksson]

Ok I think I have a better understanding now.

I really don't want to setup a cluster using mkcephfs, it "feels" like
that's something built for test setups. If I(and I do) want to
automate the setup of OSDs, MONs and RGW it definitely makes it more
complicated using key based auth. I've seen the chef recipes but I'm
not sure they're complete in any way.

I don't entirely understand if there is an order to the setup, if I
have to partly setup the MON first, then OSDs, then configure the MON,
then configure the OSDs and lastly configure RGW. If that's the case
it's kind of complicated and adding keys into the mix doesn't help.

I guess giving more time to my ceph-research will help me get a better
understanding of the auth parts and the rest. I'll have to be less
impatient ;-).

Thanks for responding,

John

On Tue, Jun 12, 2012 at 11:52 AM, Wido den Hollander <wido@widodh.nl> wrote:
> Hi,
>
>
> On 06/12/2012 11:31 AM, John Axel Eriksson wrote:
>>
>> I asked a similar question in a previous email but I didn't get any
>> satisfying answers. What exactly does cephx auth secure?
>
>
> I wanted to get back on your e-mail from yesterday, but you beat me to it!
> :)
>
>
>>  From the wiki I just get "this makes your cluster more secure", well
>> from what? If I run on an internal network accessible only
>> by a few trusted people - what does cephx auth secure in such a scenario?
>>
>
> Never ever trust anything, even in a local network. Security has more
> aspects then just uninvited guests accessing or modifying your data.
>
> Like I said in the previous e-mail, it prevents you from having an old
> machine or just a plain configuration error on a client disrupting cluster
> service.
>
>
>> In that previous email I got the answer that it secures the cluster
>> from mistakenly connecting to wrong cluster with rados and
>> accidentally deleting a pool... well, can rados really "accidentally"
>> connect to the wrong cluster? Only if I have the wrong config
>> file right? And if I have the wrong config-file won't it be possible
>> that I also have the "wrong" key in that case?
>>
>
> The "rados" command doesn't really need a config file, you can also specify
> the monitor address manually.
>
> $ rados -p <mon address> rmpool data
>
> If you'd issue that with the wrong monitor address while playing in your
> local network you could remove the pool on the wrong cluster.
>
> With cephx in place you also have to specify the correct key, it's just a
> second layer in place from preventing something like this happening.
>
>
>> Another scenario would be if I take down an OSD that just sits in
>> storage for say 6 months and then someone starts that machine
>> again - with key-based auth that OSD wouldn't be able to
>> connect(somehow? but if it has a working key?) but without auth it
>> could
>> possibly connect and wreak havoc in the cluster (since it is so much
>> behind perhaps in both version of software and what's stored on it).
>> I thought marking and OSD as down or out would do that?
>>
>
> If you mark an OSD out it will get marked as "in" again when you start up
> the OSD, but that also depends on your settings like "noautoin".
>
> When you take an OSD out of production with cephx you would also remove the
> corresponding OSD key from the cluster keyring.
>
>
>> Are those the main reasons for having cephx auth? Is it to secure the
>> cluster against people (myself included) making mistakes or  from
>> hacking, or is there some technical reason that I don't know of or
>> understand?
>>
>
> It can serve multiple purposes. In a bigger env you should never trust any
> machine. So if you have a client which only has access to the "rbd" pool for
> running Virtual Machines, it shouldn't get more permissions than that.
>
> So yes, it's again human error, but if a machine gets hacked, you can
> prevent that machine from ruining your whole cluster.
>
>
>> The reason I'm asking is because having cephx enabled makes cluster
>> setup much more complicated...
>>
>
> cephx might be complicated at first glance, but it isn't really that hard.
>
> It is up to you if you enable it, but take the recommendations above into
> account when making your decision.
>
> Wido
>
>>
>> Thanks,
>>
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cephx auth question

[Sage Weil]

A few things that Wido missed:

On Tue, 12 Jun 2012, John Axel Eriksson wrote:

> I asked a similar question in a previous email but I didn't get any
> satisfying answers. What exactly does cephx auth secure?
> >From the wiki I just get "this makes your cluster more secure", well
> from what? If I run on an internal network accessible only
> by a few trusted people - what does cephx auth secure in such a scenario?

There is some very preliminary docs at 

	http://ceph.com/docs/wip-auth/ops/manage/security/

Basically:

 * mutual authentication between client and server
 * capabilities limiting what clients are allowed to do

But:

 * no encryption of over-the-wire communications
 * no protection from hijacked TCP sessions (although simpler 
   man-in-the-middle attacks aren't possible)

> In that previous email I got the answer that it secures the cluster
> from mistakenly connecting to wrong cluster with rados and
> accidentally deleting a pool... well, can rados really "accidentally"
> connect to the wrong cluster?

The client can accidentally connect to the wrong cluster if fed the 
wrong monitor address or config.

> Only if I have the wrong config
> file right? And if I have the wrong config-file won't it be possible
> that I also have the "wrong" key in that case?

Yes.  

That said, daemons are careful to validate the cluster fsid before 
communicating regardless of whether cephx is enabled or not. 

You *can* put fsid = ... in the config file, but the client side isn't 
validating it currently, and that won't help if you point at the wrong 
config file.

> Another scenario would be if I take down an OSD that just sits in
> storage for say 6 months and then someone starts that machine
> again - with key-based auth that OSD wouldn't be able to
> connect(somehow? but if it has a working key?) but without auth it
> could
> possibly connect and wreak havoc in the cluster (since it is so much
> behind perhaps in both version of software and what's stored on it).
> I thought marking and OSD as down or out would do that?

Marking out manually will prevent the osd from being marked in 
automatically, but it will still start up and 'join' the cluster.  
However, whenever there is an incompatible change, we bump protocol 
numbers or add feature fits, so it shouldn't be able to do any damage.

I think the main motivation is to prevent a non-root user from, say, 
deleting your data pool, or to limit such 'admin' activity to a select 
number of nodes (e.g., the monitor hosts).

sage

> Are those the main reasons for having cephx auth? Is it to secure the
> cluster against people (myself included) making mistakes or  from
> hacking, or is there some technical reason that I don't know of or understand?
> 
> The reason I'm asking is because having cephx enabled makes cluster
> setup much more complicated...
> 
> 
> Thanks,
> 
> John
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: cephx auth question

[John Axel Eriksson]

Thanks alot for your input Sage!

I get a feeling that this is good for a larger cluster, especially
when you have many people logging in - some without root access.
We are currently a VERY small company (two people have logins and root
access - noone else has a login at all). We've been running
another distributed object storage solution for quite some time which
has no authentication whatsoever and it has worked just fine for
us. Now we're looking for a replacement since we're not entirely happy
with our current solution - it hasn't been entirely stable and we've
lost data twice.

I guess we could simply start off without authentication and later on
enable it if we do need it. Is that relatively easy to do in a running
cluster?

Also is there a good place where I could read up on exactly what
happens if an OSD or something else fails? For example with the
default two replicas - will that data be copied to another OSD if one
fails, do I have to manually do anything, can I read the data as long
as there is one copy somewhere?

john

On Tue, Jun 12, 2012 at 6:56 PM, Sage Weil <sage@inktank.com> wrote:
> A few things that Wido missed:
>
> On Tue, 12 Jun 2012, John Axel Eriksson wrote:
>
>> I asked a similar question in a previous email but I didn't get any
>> satisfying answers. What exactly does cephx auth secure?
>> >From the wiki I just get "this makes your cluster more secure", well
>> from what? If I run on an internal network accessible only
>> by a few trusted people - what does cephx auth secure in such a scenario?
>
> There is some very preliminary docs at
>
>        http://ceph.com/docs/wip-auth/ops/manage/security/
>
> Basically:
>
>  * mutual authentication between client and server
>  * capabilities limiting what clients are allowed to do
>
> But:
>
>  * no encryption of over-the-wire communications
>  * no protection from hijacked TCP sessions (although simpler
>   man-in-the-middle attacks aren't possible)
>
>> In that previous email I got the answer that it secures the cluster
>> from mistakenly connecting to wrong cluster with rados and
>> accidentally deleting a pool... well, can rados really "accidentally"
>> connect to the wrong cluster?
>
> The client can accidentally connect to the wrong cluster if fed the
> wrong monitor address or config.
>
>> Only if I have the wrong config
>> file right? And if I have the wrong config-file won't it be possible
>> that I also have the "wrong" key in that case?
>
> Yes.
>
> That said, daemons are careful to validate the cluster fsid before
> communicating regardless of whether cephx is enabled or not.
>
> You *can* put fsid = ... in the config file, but the client side isn't
> validating it currently, and that won't help if you point at the wrong
> config file.
>
>> Another scenario would be if I take down an OSD that just sits in
>> storage for say 6 months and then someone starts that machine
>> again - with key-based auth that OSD wouldn't be able to
>> connect(somehow? but if it has a working key?) but without auth it
>> could
>> possibly connect and wreak havoc in the cluster (since it is so much
>> behind perhaps in both version of software and what's stored on it).
>> I thought marking and OSD as down or out would do that?
>
> Marking out manually will prevent the osd from being marked in
> automatically, but it will still start up and 'join' the cluster.
> However, whenever there is an incompatible change, we bump protocol
> numbers or add feature fits, so it shouldn't be able to do any damage.
>
> I think the main motivation is to prevent a non-root user from, say,
> deleting your data pool, or to limit such 'admin' activity to a select
> number of nodes (e.g., the monitor hosts).
>
> sage
>
>> Are those the main reasons for having cephx auth? Is it to secure the
>> cluster against people (myself included) making mistakes or  from
>> hacking, or is there some technical reason that I don't know of or understand?
>>
>> The reason I'm asking is because having cephx enabled makes cluster
>> setup much more complicated...
>>
>>
>> Thanks,
>>
>> John
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: cephx auth question

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 5121 bytes --]

On Tue, 12 Jun 2012, John Axel Eriksson wrote:
> Thanks alot for your input Sage!
> 
> I get a feeling that this is good for a larger cluster, especially
> when you have many people logging in - some without root access.
> We are currently a VERY small company (two people have logins and root
> access - noone else has a login at all). We've been running
> another distributed object storage solution for quite some time which
> has no authentication whatsoever and it has worked just fine for
> us. Now we're looking for a replacement since we're not entirely happy
> with our current solution - it hasn't been entirely stable and we've
> lost data twice.
> 
> I guess we could simply start off without authentication and later on
> enable it if we do need it. Is that relatively easy to do in a running
> cluster?

Yes.  The doc linked below outlines the steps that will be needed.

> Also is there a good place where I could read up on exactly what
> happens if an OSD or something else fails? For example with the
> default two replicas - will that data be copied to another OSD if one
> fails, do I have to manually do anything, can I read the data as long
> as there is one copy somewhere?

http://ceph.com/docs/master/ops/manage/failures/

has lots of information about managing and diagnosing failures.  If there 
are gaps, please let us know!

sage


> 
> john
> 
> On Tue, Jun 12, 2012 at 6:56 PM, Sage Weil <sage@inktank.com> wrote:
> > A few things that Wido missed:
> >
> > On Tue, 12 Jun 2012, John Axel Eriksson wrote:
> >
> >> I asked a similar question in a previous email but I didn't get any
> >> satisfying answers. What exactly does cephx auth secure?
> >> >From the wiki I just get "this makes your cluster more secure", well
> >> from what? If I run on an internal network accessible only
> >> by a few trusted people - what does cephx auth secure in such a scenario?
> >
> > There is some very preliminary docs at
> >
> >        http://ceph.com/docs/wip-auth/ops/manage/security/
> >
> > Basically:
> >
> >  * mutual authentication between client and server
> >  * capabilities limiting what clients are allowed to do
> >
> > But:
> >
> >  * no encryption of over-the-wire communications
> >  * no protection from hijacked TCP sessions (although simpler
> >   man-in-the-middle attacks aren't possible)
> >
> >> In that previous email I got the answer that it secures the cluster
> >> from mistakenly connecting to wrong cluster with rados and
> >> accidentally deleting a pool... well, can rados really "accidentally"
> >> connect to the wrong cluster?
> >
> > The client can accidentally connect to the wrong cluster if fed the
> > wrong monitor address or config.
> >
> >> Only if I have the wrong config
> >> file right? And if I have the wrong config-file won't it be possible
> >> that I also have the "wrong" key in that case?
> >
> > Yes.
> >
> > That said, daemons are careful to validate the cluster fsid before
> > communicating regardless of whether cephx is enabled or not.
> >
> > You *can* put fsid = ... in the config file, but the client side isn't
> > validating it currently, and that won't help if you point at the wrong
> > config file.
> >
> >> Another scenario would be if I take down an OSD that just sits in
> >> storage for say 6 months and then someone starts that machine
> >> again - with key-based auth that OSD wouldn't be able to
> >> connect(somehow? but if it has a working key?) but without auth it
> >> could
> >> possibly connect and wreak havoc in the cluster (since it is so much
> >> behind perhaps in both version of software and what's stored on it).
> >> I thought marking and OSD as down or out would do that?
> >
> > Marking out manually will prevent the osd from being marked in
> > automatically, but it will still start up and 'join' the cluster.
> > However, whenever there is an incompatible change, we bump protocol
> > numbers or add feature fits, so it shouldn't be able to do any damage.
> >
> > I think the main motivation is to prevent a non-root user from, say,
> > deleting your data pool, or to limit such 'admin' activity to a select
> > number of nodes (e.g., the monitor hosts).
> >
> > sage
> >
> >> Are those the main reasons for having cephx auth? Is it to secure the
> >> cluster against people (myself included) making mistakes or  from
> >> hacking, or is there some technical reason that I don't know of or understand?
> >>
> >> The reason I'm asking is because having cephx enabled makes cluster
> >> setup much more complicated...
> >>
> >>
> >> Thanks,
> >>
> >> John
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>
> >>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
>