how to mount a specific pool in cephs

how to mount a specific pool in cephs

[Paul Pettigrew]

Hi all

Running Ceph version 0.45-1precise on 12.04 x64.

I cannot find the right syntax to mount a specific pool created, that has a customised replica level set, which is different to the default "data" pool (i.e. we have an additional pool called "backup"). Can somebody please advise the right syntax??

I tried the following per: http://ceph.newdream.net/docs/master/man/8/mount.ceph/
    mount -t ceph 10.32.0.10:6789:/backup /mnt/ceph -vv -o name=admin,secret=ThisIsNotTheRealSecret==
And it hard-hung the server, and needed a hard power restart (ouch!).

The following works fine, but it is not connecting to the "backup" pool (with no pool name specified, it is going to pool0 called "data"):
    mount -t ceph 10.32.0.10:6789:/ /mnt/ceph -vv -o name=admin,secret=ThisIsNotTheRealSecret==

Very many thanks

Paul

Re: how to mount a specific pool in cephs

[Tommi Virtanen]

On Sun, Apr 29, 2012 at 23:16, Paul Pettigrew
<Paul.Pettigrew@mach.com.au> wrote:
> I cannot find the right syntax to mount a specific pool created, that has a customised replica level set, which is different to the default "data" pool (i.e. we have an additional pool called "backup"). Can somebody please advise the right syntax??
>
> I tried the following per: http://ceph.newdream.net/docs/master/man/8/mount.ceph/
>    mount -t ceph 10.32.0.10:6789:/backup /mnt/ceph -vv -o name=admin,secret=ThisIsNotTheRealSecret==
> And it hard-hung the server, and needed a hard power restart (ouch!).
>
> The following works fine, but it is not connecting to the "backup" pool (with no pool name specified, it is going to pool0 called "data"):
>    mount -t ceph 10.32.0.10:6789:/ /mnt/ceph -vv -o name=admin,secret=ThisIsNotTheRealSecret==

You don't mount pools directly; there's filesystem metadata (as
managed by metadata servers) that is needed too.

What you probably want is to specify that a subtree of your ceph dfs
stores the file data in a separate pool, using "cephfs
/mnt/ceph/some/subtree set_layout --pool 6". Note that a numerical
pool id is currently required.

http://ceph.newdream.net/docs/master/man/8/cephfs/

You can mount any subtree of the ceph dfs directly, using
10.32.0.10:6789:/some/subtree in your mount command.


The crash you experienced may be completely separate from this, but if
any logs survived, filing a bug would be appreciated.
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to mount a specific pool in cephs

[Grant Ashman]

Tommi Virtanen <tommi.virtanen <at> dreamhost.com> writes:
> 
> You don't mount pools directly; there's filesystem metadata (as
> managed by metadata servers) that is needed too.
> 
> What you probably want is to specify that a subtree of your ceph dfs
> stores the file data in a separate pool, using "cephfs
> /mnt/ceph/some/subtree set_layout --pool 6". Note that a numerical
> pool id is currently required.
> 
> http://ceph.newdream.net/docs/master/man/8/cephfs/
> 
> You can mount any subtree of the ceph dfs directly, using
> 10.32.0.10:6789:/some/subtree in your mount command.
> 

Hi Tommi,

We have tried setting the layout as described below with:
'cephfs /mnt/ceph-backup/ set_layout --pool 3'

However, I only ever receive the following output;
'Error setting layout: Invalid argument'

I can run other view the current layout of the mount 
point, but cannot change the pool layout.

My understanding of the numeric pool is as follows:

root@dsan-test:/mnt# ceph osd dump -o -|grep 'pool'
pool 0 'data' rep size 2 crush_ruleset 0
pool 1 'metadata' rep size 2 crush_ruleset 1
pool 2 'rbd' rep size 2 crush_ruleset 2
pool 3 'backup' rep size 2 crush_ruleset 0 

(omitted detail I thought unnecessary)

Therefore the backup pool which we specifically want to mount is 3?

Are you able to assist with the syntax for cephfs set_layout?

Thanks
Grant

Re: how to mount a specific pool in cephs

[Greg Farnum]

On Tuesday, May 22, 2012 at 5:02 AM, Grant Ashman wrote:
> Tommi Virtanen <tommi.virtanen <at> dreamhost.com (http://dreamhost.com)> writes:
> > 
> > You don't mount pools directly; there's filesystem metadata (as
> > managed by metadata servers) that is needed too.
> > 
> > What you probably want is to specify that a subtree of your ceph dfs
> > stores the file data in a separate pool, using "cephfs
> > /mnt/ceph/some/subtree set_layout --pool 6". Note that a numerical
> > pool id is currently required.
> > 
> > http://ceph.newdream.net/docs/master/man/8/cephfs/
> > 
> > You can mount any subtree of the ceph dfs directly, using
> > 10.32.0.10:6789:/some/subtree in your mount command.
> 
> 
> 
> Hi Tommi,
> 
> We have tried setting the layout as described below with:
> 'cephfs /mnt/ceph-backup/ set_layout --pool 3'
> 
> However, I only ever receive the following output;
> 'Error setting layout: Invalid argument'
> 
> I can run other view the current layout of the mount 
> point, but cannot change the pool layout.
> 
> My understanding of the numeric pool is as follows:
> 
> root@dsan-test:/mnt# ceph osd dump -o -|grep 'pool'
> pool 0 'data' rep size 2 crush_ruleset 0
> pool 1 'metadata' rep size 2 crush_ruleset 1
> pool 2 'rbd' rep size 2 crush_ruleset 2
> pool 3 'backup' rep size 2 crush_ruleset 0 
> 
> (omitted detail I thought unnecessary)
> 
> Therefore the backup pool which we specifically want to mount is 3?
> 
> Are you able to assist with the syntax for cephfs set_layout?
That's the right pool ID; yes. I believe the problem is that the cephfs tool currently requires you to fill in all the fields, not just the one you wish to change. Try that (setting all the other values to match what you see when you view the layout). :)
-Greg

RE: how to mount a specific pool in cephs

[Grant Ashman]

>That's the right pool ID; yes. I believe the problem is that the cephfs tool currently requires you to fill in all the fields, not ?>just the one you wish to change. Try that (setting all the other values to match what you see when you view the layout). :) -> Greg

Hi,

I've tried setting all values the same as the current layout - changing only the pool number and I still get;

root@dsan-test:/mnt/ceph-test# cephfs /mnt/ceph-backup show_layout
layout.data_pool:     0
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1
root@dsan-test:/mnt/ceph-test# 
root@dsan-test:/mnt/ceph-test# cephfs /mnt/ceph-backup set_layout -p 3 -s 4194304 -u 4194304 -c 1
Error setting layout: Invalid argument

If I leave all values exactly the same i.e '-p 0' the command runs without any error output. However, changing the pool from anything but 0 results in 'Error setting layout: Invalid argument'

Any ideas?

Grant

Re: how to mount a specific pool in cephs

[Greg Farnum]

On Tuesday, May 22, 2012 at 2:12 PM, Grant Ashman wrote:
> > That's the right pool ID; yes. I believe the problem is that the cephfs tool currently requires you to fill in all the fields, not ?>just the one you wish to change. Try that (setting all the other values to match what you see when you view the layout). :) -> Greg
>  
>  
>  
> Hi,
>  
> I've tried setting all values the same as the current layout - changing only the pool number and I still get;
>  
> root@dsan-test:/mnt/ceph-test# cephfs /mnt/ceph-backup show_layout
> layout.data_pool: 0
> layout.object_size: 4194304
> layout.stripe_unit: 4194304
> layout.stripe_count: 1
> root@dsan-test:/mnt/ceph-test#  
> root@dsan-test:/mnt/ceph-test# cephfs /mnt/ceph-backup set_layout -p 3 -s 4194304 -u 4194304 -c 1
> Error setting layout: Invalid argument
>  
> If I leave all values exactly the same i.e '-p 0' the command runs without any error output. However, changing the pool from anything but 0 results in 'Error setting layout: Invalid argument'
>  
> Any ideas?
Oh, I got this conversation confused with another one. You also need to specify the pool as a valid pool to store filesystem data in, if you haven't done that already:
"ceph mds add_data_pool <poolname>"


And you may not actually need to specify all options — I'd been assuming since it broke but I don't remember if that's actually the case (I think it's changed over the versions). 
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to mount a specific pool in cephs

[Grant Ashman]

Greg Farnum <greg <at> inktank.com> writes:
> Oh, I got this conversation confused with another one.
> You also need to specify the pool as a valid pool to
> store filesystem data in, if you haven't done that already:
> "ceph mds add_data_pool <poolname>"
> 
> 

Thanks Greg,

However, I still get the same error :(
When I specify the add data pool I get the following:
(with or without the additional values)

root@dsan-test:~# ceph mds add_data_pool backup
added data pool 0 to mdsmap
root@dsan-test:~# cephfs /mnt/ceph-backup set_layout -p 3
Error setting layout: Invalid argument

Seems to be pointing at pool 0 instead of pool 3 like it should?

Sorry if this has all been covered before, I've not found any 
resolution and the ability to mount specific pools I.E "data"
for production data and "backup" for 1N backup data is a huge
priority to begin using Ceph.

Re: how to mount a specific pool in cephs

[Greg Farnum]

On Tuesday, May 22, 2012 at 2:31 PM, Grant Ashman wrote:
> Greg Farnum <greg <at> inktank.com (http://inktank.com)> writes:
> > Oh, I got this conversation confused with another one.
> > You also need to specify the pool as a valid pool to
> > store filesystem data in, if you haven't done that already:
> > "ceph mds add_data_pool <poolname>"
>  
>  
>  
> Thanks Greg,
>  
> However, I still get the same error :(
> When I specify the add data pool I get the following:
> (with or without the additional values)
>  
> root@dsan-test:~# ceph mds add_data_pool backup
> added data pool 0 to mdsmap

Okay, that's not right — it should say pool 3.
According to the docs I found you ran that correctly, but let's try running "ceph mds add_data_pool 3" and see if that resolves correctly.
*goes to look at code*
Argh, yep, it's expecting a pool ID, not a pool name. Gah.
  
> root@dsan-test:~# cephfs /mnt/ceph-backup set_layout -p 3
> Error setting layout: Invalid argument
>  
> Seems to be pointing at pool 0 instead of pool 3 like it should?
>  
> Sorry if this has all been covered before, I've not found any  
> resolution and the ability to mount specific pools I.E "data"
> for production data and "backup" for 1N backup data is a huge
> priority to begin using Ceph.

No problem — we haven't generated docs for this yet and obviously we need to at some point.

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to mount a specific pool in cephs

[Grant Ashman]

Greg Farnum <greg <at> inktank.com> writes:

> > When I specify the add data pool I get the following:
> > (with or without the additional values)
> >  
> > root <at> dsan-test:~# ceph mds add_data_pool backup
> > added data pool 0 to mdsmap
> 
> Okay, that's not right — it should say pool 3.
> According to the docs I found you ran that correctly, but let's try running 
"ceph mds add_data_pool 3" and
> see if that resolves correctly.
> *goes to look at code*
> Argh, yep, it's expecting a pool ID, not a pool name. Gah.
> 
> > root <at> dsan-test:~# cephfs /mnt/ceph-backup set_layout -p 3
> > Error setting layout: Invalid argument
> >  
> > Seems to be pointing at pool 0 instead of pool 3 like it should?
> >  

Awesome, that seemed to work!
However, I feel a bit silly - what I'm after is:

/mnt/ceph-data - mounted to pool 0 (data)
/mnt/ceph-backup - mounted to pool 3 (backup)

but this seemed to change both to mount to pool 3?

Am I simply doing something wrong at my mount stage?

The process to mount my specific pool as I understand it is;

1. mount -t ceph 1.2.3.4:/ /mnt/ceph-backup
2. ceph mds add_data_pool 3
3. cephfs /mnt/ceph-backup/ -p 3 

Which should give me /ceph-backup mounted to the backup pool.

Should I simply then do a 
mount -t ceph 1.2.3.4:/ /mnt/ceph-data/
to get the /ceph-data/ pool mounted to the pool 0 data pool?




--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: how to mount a specific pool in cephs

[Gregory Farnum]

On Tuesday, May 22, 2012 at 2:51 PM, Grant Ashman wrote:
> Awesome, that seemed to work!
> However, I feel a bit silly - what I'm after is:
> 
> /mnt/ceph-data - mounted to pool 0 (data)
> /mnt/ceph-backup - mounted to pool 3 (backup)
> 
> but this seemed to change both to mount to pool 3?
> 
> Am I simply doing something wrong at my mount stage?
> 
> The process to mount my specific pool as I understand it is;
> 
> 1. mount -t ceph 1.2.3.4:/ /mnt/ceph-backup
> 2. ceph mds add_data_pool 3
> 3. cephfs /mnt/ceph-backup/ -p 3 
> 
> Which should give me /ceph-backup mounted to the backup pool.
> 
> Should I simply then do a 
> mount -t ceph 1.2.3.4:/ /mnt/ceph-data/
> to get the /ceph-data/ pool mounted to the pool 0 data pool?

Right now, you can't actually get fully-distinct filesystem trees in a single Ceph cluster. That will probably come at some point in the future, and will require different MDS daemons and separate data & metadata pools for each filesystem.

What you *can* do right now is:
1) Set the pool a subtree writes all its new data to, and
2) mount subtrees of the filesystem, and
3) give specific clients read/write access to only a subset of the pools used by the filesystem hierarchy.

This means that you could have your root directory contain a "live" and a "backup" directory:
mount -t ceph 1.2.3.4:/ /mnt/ceph
mkdir /mnt/ceph/live; mkdir /mnt/ceph/backup

Leave the "live" directory using pool 0 while setting the "backup" directory to use pool 3:
cephfs /mnt/ceph/backup -p 3; umount /mnt/ceph

And then run one of
mount -t ceph 1.2.3.4:/live /mnt/ceph
or
mount -t ceph 1.2.3.4:/backup /mnt/ceph-backup

On your live and backup servers, respectively. (Or mount both and copy files to backup, or whatever.) Your live servers won't see the backup space, and the backup space won't see the live space, and they will live in different pools that you can place on different OSDs. (Although the metadata can't be segregated right now; you'll want a different strategy for that.)

Make sense?
-Greg

Re: how to mount a specific pool in cephs

[Grant Ashman]

Greg Farnum <greg <at> inktank.com> writes:
> That's the right pool ID; yes. I believe the problem is that the cephfs tool 
> currently requires you to fill in
> all the fields, not just the one you wish to change. Try that (setting all the 
> other values to match what you
> see when you view the layout). :)
> -Greg
> 

Hi,

I've tried setting all values the same as the current layout.
Changing only the pool number I still get;

root@dsan-test:# cephfs /mnt/ceph-backup show_layout
layout.data_pool:     0
layout.object_size:   4194304
layout.stripe_unit:   4194304
layout.stripe_count:  1
root@dsan-test:#
root@dsan-test:# cephfs /mnt/ceph-backup set_layout -p 3 -s 4194304 
-u 4194304 -c 1 Error setting layout: Invalid argument

If I leave all values exactly the same i.e '-p 0' the command runs without any 
error output. 
However, changing the pool from anything but 0 results in 'Error 
setting layout: Invalid argument'

Any ideas?

Grant