Keys & caps

Keys & caps

[Székelyi Szabolcs]

Hello,

this far I accessed my Ceph (0.48) FS with the client.admin key, but I'd like 
to change that since I don't want to allow clients to control the cluster.

I thought I should create a new key, give it some caps (don't exactly know 
which ones), and distribute it to clients. Here are some things I don't 
know/understand:

* What do the r, w, x, and * caps ("permissions"?) mean on a mon, mds, or osd?

* What's the difference between (for example) 'allow rw' and 'rw' caps? `ceph 
auth` seems to understand (and display as specified) both forms.

Not to run into issues with caps, I've created a key with the same caps as the 
current client.admin, but with a different key, called client.access_fs. I 
planned to narrow down the caps after I made the mount work with the new key.

$ sudo ceph auth list
installed auth entries: 
[...]
client.access_fs
        key: AQ...==
        caps: [mds] allow
        caps: [mon] allow *
        caps: [osd] allow *
[...]

I've copied the key to the client as /etc/ceph/keyring:

[client.access_fs]
        key = AQ...==

I have 'keyring = /etc/ceph/keyring' in my ceph.conf [global] section on the 
client. Now trying to mount the filesystem fails with

$ sudo mount /mnt/ceph
ceph-fuse[1784]: starting ceph client
ceph-fuse[1784]: ceph mount failed with (1) Operation not permitted
ceph-fuse[1782]: mount failed: (1) Operation not permitted

Adding '-o name=client.access_fs' to the command line yields the same result. 
If I copy the client.admin key into the keyring file, it works. (However, 
adding the same mount option again (so the name= parameter and the key name in 
the file do not match), I get an error (which seems okay to me, not sure about 
the correctness):

$ sudo mount /cloud/ -o name=client.access_fs
ceph-fuse[1835]: starting ceph client
ceph-fuse[1835]: starting fuse
fuse: unknown option `name=client.access_fs'
2012-07-09 16:03:18.343793 7fb0e7b8b780 -1 fuse_lowlevel_new failed
ceph-fuse[1835]: fuse finished with error 33
ceph-fuse[1833]: mount failed: (33) Numerical argument out of domain

Could you clear up the confusion in my head? :)

Thanks,
-- 
cc

Re: Keys & caps

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3550 bytes --]

Hi,

On Mon, 9 Jul 2012, Székelyi Szabolcs wrote:
> Hello,
> 
> this far I accessed my Ceph (0.48) FS with the client.admin key, but I'd like 
> to change that since I don't want to allow clients to control the cluster.
> 
> I thought I should create a new key, give it some caps (don't exactly know 
> which ones), and distribute it to clients. Here are some things I don't 
> know/understand:
> 
> * What do the r, w, x, and * caps ("permissions"?) mean on a mon, mds, or osd?

They roughly correspond to read, write, execute.  The distinction is 
subtle and poorly specfied for mon caps; just use the documented values 
for now.  For the mds, it's just 'allow', as we haven't properly defined 
those yet either.  The osd ones are the most useful and well defined.  
There is a grammar in osd/OSDCap.h if you can grok the boost::spirit 
syntax.  In addition to rwx, you can grant by pool or object prefix.  
There is some preliminary/incomplete support for fine-grained control over 
the use of rados classes.

> * What's the difference between (for example) 'allow rw' and 'rw' caps? `ceph 
> auth` seems to understand (and display as specified) both forms.

'rw' is nonsense... it needs to be (a list of) 'allow <something>' 
phrase(s).  From the monitor's perspective, the osd and mds caps are 
opaque, so it won't enforce a particular syntax.  At some point we can 
make it verify they parse properly, but that hasn't happened yet.

> Not to run into issues with caps, I've created a key with the same caps as the 
> current client.admin, but with a different key, called client.access_fs. I 
> planned to narrow down the caps after I made the mount work with the new key.
> 
> $ sudo ceph auth list
> installed auth entries: 
> [...]
> client.access_fs
>         key: AQ...==
>         caps: [mds] allow
>         caps: [mon] allow *
>         caps: [osd] allow *
> [...]
> 
> I've copied the key to the client as /etc/ceph/keyring:
> 
> [client.access_fs]
>         key = AQ...==
> 
> I have 'keyring = /etc/ceph/keyring' in my ceph.conf [global] section on the 
> client. Now trying to mount the filesystem fails with
> 
> $ sudo mount /mnt/ceph
> ceph-fuse[1784]: starting ceph client
> ceph-fuse[1784]: ceph mount failed with (1) Operation not permitted
> ceph-fuse[1782]: mount failed: (1) Operation not permitted
> 
> Adding '-o name=client.access_fs' to the command line yields the same result. 
> If I copy the client.admin key into the keyring file, it works. (However, 
> adding the same mount option again (so the name= parameter and the key name in 
> the file do not match), I get an error (which seems okay to me, not sure about 
> the correctness):
> 
> $ sudo mount /cloud/ -o name=client.access_fs
> ceph-fuse[1835]: starting ceph client
> ceph-fuse[1835]: starting fuse
> fuse: unknown option `name=client.access_fs'
> 2012-07-09 16:03:18.343793 7fb0e7b8b780 -1 fuse_lowlevel_new failed
> ceph-fuse[1835]: fuse finished with error 33
> ceph-fuse[1833]: mount failed: (33) Numerical argument out of domain
> 
> Could you clear up the confusion in my head? :)

The problem is that the mount.ceph command doesn't understand keyrings; it 
only understands secret= and secretfile=.  There is a longstanding feature 
bug open for this

	http://tracker.newdream.net/issues/266

but it hasn't been prioritized.  Sorry for the confusion!  It will happen 
soon.  

In the meantime, you need

	 -o secretfile=/path/to/secretfile,name=access_fs

sage

Re: Keys & caps

[Székelyi Szabolcs]

On 2012. July 9. 09:33:22 Sage Weil wrote:
> On Mon, 9 Jul 2012, Székelyi Szabolcs wrote:
> > this far I accessed my Ceph (0.48) FS with the client.admin key, but I'd
> > like to change that since I don't want to allow clients to control the
> > cluster.
> > 
> > I thought I should create a new key, give it some caps (don't exactly know
> > which ones), and distribute it to clients. Here are some things I don't
> > know/understand:
> > 
> > * What do the r, w, x, and * caps ("permissions"?) mean on a mon, mds, or
> > osd?
> 
> They roughly correspond to read, write, execute.  The distinction is
> subtle and poorly specfied for mon caps; just use the documented values
> for now.

Does this mean that what I'm trying to achieve is not possible at the moment? 
I'd like to give access to my clients to the data in the filesystem, but not 
control over the cluster. My thought was that removing some mon caps from the 
clients' keys will get me there. But from what you write, it looks to me like 
if a client can access the data in the filesystem, it can also (for example) 
bring the cluster down...

> The problem is that the mount.ceph command doesn't understand keyrings; it
> only understands secret= and secretfile=.  There is a longstanding feature
> bug open for this
> 
> 	http://tracker.newdream.net/issues/266
> 
> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
> soon.
> 
> In the meantime, you need
> 
> 	 -o secretfile=/path/to/secretfile,name=access_fs

Is this also true for the FUSE client? I have obscure memories about big 
differences between the kernel and the FUSE client, for example the latter 
being able to read ceph.conf, and get the necessary info, including the 
keyring file, from there. Maybe I didn't emphasize it, but that's what I'm 
using.

Thanks,
-- 
cc


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Keys & caps

[Gregory Farnum]

On Mon, Jul 9, 2012 at 10:27 AM, Székelyi Szabolcs <szekelyi@niif.hu> wrote:
> On 2012. July 9. 09:33:22 Sage Weil wrote:
>> On Mon, 9 Jul 2012, Székelyi Szabolcs wrote:
>> > this far I accessed my Ceph (0.48) FS with the client.admin key, but I'd
>> > like to change that since I don't want to allow clients to control the
>> > cluster.
>> >
>> > I thought I should create a new key, give it some caps (don't exactly know
>> > which ones), and distribute it to clients. Here are some things I don't
>> > know/understand:
>> >
>> > * What do the r, w, x, and * caps ("permissions"?) mean on a mon, mds, or
>> > osd?
>>
>> They roughly correspond to read, write, execute.  The distinction is
>> subtle and poorly specfied for mon caps; just use the documented values
>> for now.
>
> Does this mean that what I'm trying to achieve is not possible at the moment?
> I'd like to give access to my clients to the data in the filesystem, but not
> control over the cluster. My thought was that removing some mon caps from the
> clients' keys will get me there. But from what you write, it looks to me like
> if a client can access the data in the filesystem, it can also (for example)
> bring the cluster down...

I believe your filesystem clients only need  'allow r' on the
monitors, and they should be good with 'allow rw' on the OSDs (though
still "allow" on the MDS). This is distinct from the "allow *" cap and
does minimize some ability to cause damage. :)

>> The problem is that the mount.ceph command doesn't understand keyrings; it
>> only understands secret= and secretfile=.  There is a longstanding feature
>> bug open for this
>>
>>       http://tracker.newdream.net/issues/266
>>
>> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
>> soon.
>>
>> In the meantime, you need
>>
>>        -o secretfile=/path/to/secretfile,name=access_fs
>
> Is this also true for the FUSE client? I have obscure memories about big
> differences between the kernel and the FUSE client, for example the latter
> being able to read ceph.conf, and get the necessary info, including the
> keyring file, from there. Maybe I didn't emphasize it, but that's what I'm
> using.

When using ceph-fuse, you want to specify the name with
--name=access_fs — no -o required.
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Keys & caps

[Székelyi Szabolcs]

On 2012. July 10. 13:09:10 Gregory Farnum wrote:
> On Mon, Jul 9, 2012 at 10:27 AM, Székelyi Szabolcs <szekelyi@niif.hu> wrote:
> > On 2012. July 9. 09:33:22 Sage Weil wrote:
> >> On Mon, 9 Jul 2012, Székelyi Szabolcs wrote:
> >> > this far I accessed my Ceph (0.48) FS with the client.admin key, but
> >> > I'd
> >> > like to change that since I don't want to allow clients to control the
> >> > cluster.
> >> > 
> >> > I thought I should create a new key, give it some caps (don't exactly
> >> > know
> >> > which ones), and distribute it to clients. Here are some things I don't
> >> > know/understand:
> >> > 
> >> > * What do the r, w, x, and * caps ("permissions"?) mean on a mon, mds,
> >> > or
> >> > osd?
> >> 
> >> They roughly correspond to read, write, execute.  The distinction is
> >> subtle and poorly specfied for mon caps; just use the documented values
> >> for now.
> > 
> > Does this mean that what I'm trying to achieve is not possible at the
> > moment? I'd like to give access to my clients to the data in the
> > filesystem, but not control over the cluster. My thought was that
> > removing some mon caps from the clients' keys will get me there. But from
> > what you write, it looks to me like if a client can access the data in
> > the filesystem, it can also (for example) bring the cluster down...
> 
> I believe your filesystem clients only need  'allow r' on the
> monitors, and they should be good with 'allow rw' on the OSDs (though
> still "allow" on the MDS). This is distinct from the "allow *" cap and
> does minimize some ability to cause damage. :)

Great, thanks!

> >> The problem is that the mount.ceph command doesn't understand keyrings;
> >> it
> >> only understands secret= and secretfile=.  There is a longstanding
> >> feature
> >> bug open for this
> >> 
> >>       http://tracker.newdream.net/issues/266
> >> 
> >> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
> >> soon.
> >> 
> >> In the meantime, you need
> >> 
> >>        -o secretfile=/path/to/secretfile,name=access_fs
> > 
> > Is this also true for the FUSE client? I have obscure memories about big
> > differences between the kernel and the FUSE client, for example the latter
> > being able to read ceph.conf, and get the necessary info, including the
> > keyring file, from there. Maybe I didn't emphasize it, but that's what I'm
> > using.
> 
> When using ceph-fuse, you want to specify the name with
> --name=access_fs — no -o required.

Okay, so I don't need the "client." prefix for the key name, but how can I 
specify the --name= option in fstab? Or is it possible to specify it in 
ceph.conf, so it can be read by ceph-fuse when called by mount?

Thanks,
-- 
cc


--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Keys & caps

[Sage Weil]

On Wed, 11 Jul 2012, Sz?kelyi Szabolcs wrote:
> > >> The problem is that the mount.ceph command doesn't understand keyrings;
> > >> it
> > >> only understands secret= and secretfile=.  There is a longstanding
> > >> feature
> > >> bug open for this
> > >> 
> > >>       http://tracker.newdream.net/issues/266
> > >> 
> > >> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
> > >> soon.
> > >> 
> > >> In the meantime, you need
> > >> 
> > >>        -o secretfile=/path/to/secretfile,name=access_fs
> > > 
> > > Is this also true for the FUSE client? I have obscure memories about big
> > > differences between the kernel and the FUSE client, for example the latter
> > > being able to read ceph.conf, and get the necessary info, including the
> > > keyring file, from there. Maybe I didn't emphasize it, but that's what I'm
> > > using.
> > 
> > When using ceph-fuse, you want to specify the name with
> > --name=access_fs ? no -o required.
> 
> Okay, so I don't need the "client." prefix for the key name, but how can I 
> specify the --name= option in fstab? Or is it possible to specify it in 
> ceph.conf, so it can be read by ceph-fuse when called by mount?

Er sorry, you actually want either --id foo or --name client.foo (they are 
equivalent) when dealing with ceph-fuse (or other userland daemons/tools).

I'm not sure what the best way of putting fuse-based file systems in fstab 
is, though.  One way or another, though, ceph-fuse needs to be told on the 
command line who to authenticate as (and which sections of ceph.conf to 
pay attention to).

sage

Re: Keys & caps

[Gregory Farnum]

On Tue, Jul 10, 2012 at 4:25 PM, Sage Weil <sage@inktank.com> wrote:
> On Wed, 11 Jul 2012, Sz?kelyi Szabolcs wrote:
>> > >> The problem is that the mount.ceph command doesn't understand keyrings;
>> > >> it
>> > >> only understands secret= and secretfile=.  There is a longstanding
>> > >> feature
>> > >> bug open for this
>> > >>
>> > >>       http://tracker.newdream.net/issues/266
>> > >>
>> > >> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
>> > >> soon.
>> > >>
>> > >> In the meantime, you need
>> > >>
>> > >>        -o secretfile=/path/to/secretfile,name=access_fs
>> > >
>> > > Is this also true for the FUSE client? I have obscure memories about big
>> > > differences between the kernel and the FUSE client, for example the latter
>> > > being able to read ceph.conf, and get the necessary info, including the
>> > > keyring file, from there. Maybe I didn't emphasize it, but that's what I'm
>> > > using.
>> >
>> > When using ceph-fuse, you want to specify the name with
>> > --name=access_fs ? no -o required.
>>
>> Okay, so I don't need the "client." prefix for the key name, but how can I
>> specify the --name= option in fstab? Or is it possible to specify it in
>> ceph.conf, so it can be read by ceph-fuse when called by mount?
>
> Er sorry, you actually want either --id foo or --name client.foo (they are
> equivalent) when dealing with ceph-fuse (or other userland daemons/tools).

Nope — I checked this locally before sending it out. --name foo worked
out fine. Although I certainly can't keep straight when we require the
type prefix versus when it's added for us. *sigh*


> I'm not sure what the best way of putting fuse-based file systems in fstab
> is, though.  One way or another, though, ceph-fuse needs to be told on the
> command line who to authenticate as (and which sections of ceph.conf to
> pay attention to).

This is my concern, too. I don't think you can specify default IDs for
ceph-fuse right now. :/

>
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Keys & caps

[Sage Weil]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2792 bytes --]

On Tue, 10 Jul 2012, Gregory Farnum wrote:
> On Tue, Jul 10, 2012 at 4:25 PM, Sage Weil <sage@inktank.com> wrote:
> > On Wed, 11 Jul 2012, Sz?kelyi Szabolcs wrote:
> >> > >> The problem is that the mount.ceph command doesn't understand keyrings;
> >> > >> it
> >> > >> only understands secret= and secretfile=.  There is a longstanding
> >> > >> feature
> >> > >> bug open for this
> >> > >>
> >> > >>       http://tracker.newdream.net/issues/266
> >> > >>
> >> > >> but it hasn't been prioritized.  Sorry for the confusion!  It will happen
> >> > >> soon.
> >> > >>
> >> > >> In the meantime, you need
> >> > >>
> >> > >>        -o secretfile=/path/to/secretfile,name=access_fs
> >> > >
> >> > > Is this also true for the FUSE client? I have obscure memories about big
> >> > > differences between the kernel and the FUSE client, for example the latter
> >> > > being able to read ceph.conf, and get the necessary info, including the
> >> > > keyring file, from there. Maybe I didn't emphasize it, but that's what I'm
> >> > > using.
> >> >
> >> > When using ceph-fuse, you want to specify the name with
> >> > --name=access_fs ? no -o required.
> >>
> >> Okay, so I don't need the "client." prefix for the key name, but how can I
> >> specify the --name= option in fstab? Or is it possible to specify it in
> >> ceph.conf, so it can be read by ceph-fuse when called by mount?
> >
> > Er sorry, you actually want either --id foo or --name client.foo (they are
> > equivalent) when dealing with ceph-fuse (or other userland daemons/tools).
> 
> Nope ? I checked this locally before sending it out. --name foo worked
> out fine. Although I certainly can't keep straight when we require the
> type prefix versus when it's added for us. *sigh*

Oh, I bet the code tries to be friendly and will behave without the 
'client.' if you leave it off... but we shouldn't advertise that.  
Everyone, forget what you just heard!

:) sage


> 
> 
> > I'm not sure what the best way of putting fuse-based file systems in fstab
> > is, though.  One way or another, though, ceph-fuse needs to be told on the
> > command line who to authenticate as (and which sections of ceph.conf to
> > pay attention to).
> 
> This is my concern, too. I don't think you can specify default IDs for
> ceph-fuse right now. :/
> 
> >
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Keys & caps

[Székelyi Szabolcs]

On 2012. July 10. 16:25:47 Sage Weil wrote:
> On Wed, 11 Jul 2012, Sz?kelyi Szabolcs wrote:
> > > >> The problem is that the mount.ceph command doesn't understand
> > > >> keyrings;
> > > >> it
> > > >> only understands secret= and secretfile=.  There is a longstanding
> > > >> feature
> > > >> bug open for this
> > > >> 
> > > >>       http://tracker.newdream.net/issues/266
> > > >> 
> > > >> but it hasn't been prioritized.  Sorry for the confusion!  It will
> > > >> happen
> > > >> soon.
> > > >> 
> > > >> In the meantime, you need
> > > >> 
> > > >>        -o secretfile=/path/to/secretfile,name=access_fs
> > > > 
> > > > Is this also true for the FUSE client? I have obscure memories about
> > > > big
> > > > differences between the kernel and the FUSE client, for example the
> > > > latter
> > > > being able to read ceph.conf, and get the necessary info, including
> > > > the
> > > > keyring file, from there. Maybe I didn't emphasize it, but that's what
> > > > I'm
> > > > using.
> > > 
> > > When using ceph-fuse, you want to specify the name with
> > > --name=access_fs ? no -o required.
> > 
> > Okay, so I don't need the "client." prefix for the key name, but how can I
> > specify the --name= option in fstab? Or is it possible to specify it in
> > ceph.conf, so it can be read by ceph-fuse when called by mount?
> 
> Er sorry, you actually want either --id foo or --name client.foo (they are
> equivalent) when dealing with ceph-fuse (or other userland daemons/tools).
> 
> I'm not sure what the best way of putting fuse-based file systems in fstab
> is, though.  One way or another, though, ceph-fuse needs to be told on the
> command line who to authenticate as (and which sections of ceph.conf to
> pay attention to).

This seems to be possible to pass a single argument to the userspace mount 
program with the following syntax in fstab:

ceph-fuse#--name=client.access_fs	/mnt/ceph	fuse	defaults	0 0

-- 
cc