Clusters and pools

Clusters and pools

[Ryan Nicholson]

I'm running a cluster based on 4 hosts that each have 3 fast, SCSI osd's, and 1 very large SATA osd, meaning, 12 fast osd's and 4 slow osd's total. I wish to segregate these into 2 pools, that operate independently. The goal is to use the faster disks as an area to hold rbd based VM's, and the larger area to host rbd-base large volumes (to start), and possibly have that become just a big cephfs area once, the fs side of things is considered more stable.

Now, I've been thrown a couple options, and am still unsettled. Which is best?:
- Create 2 independent clusters; one with the 12 SCSI osd's and the other with just the 4 large OSD's on the same hosts. This seems to be more complex from a scripting and boot time standpoint, but easier for my head.
- Create a single cluster and use CRUSH rules to separate the two. This one STILL has me lost, as I'm having trouble understanding the crushmap syntax, the Crushmap import/export commands, and the other mkpool or otherwise commands from the docs in order to "make rbd's come from this faster pool", while "cephfs, you come from this slower pool". I really would like to entertain this path, however, as this allows ceph to handle the entire situation, and, it would seem more elegant.

I'm also open to other options as well. 

Thanks!

Ryan Nicholson

Re: Clusters and pools

[Thomas Mueller]

Am Wed, 25 Jul 2012 04:34:45 +0000 schrieb Ryan Nicholson:

> I'm running a cluster based on 4 hosts that each have 3 fast, SCSI
> osd's, and 1 very large SATA osd, meaning, 12 fast osd's and 4 slow
> osd's total. I wish to segregate these into 2 pools, that operate
> independently. The goal is to use the faster disks as an area to hold
> rbd based VM's, and the larger area to host rbd-base large volumes (to
> start), and possibly have that become just a big cephfs area once, the
> fs side of things is considered more stable.
> 

slightly another topic, but maybe you could also give TIER try:

http://www.lessfs.com/wordpress/?p=776

http://sourceforge.net/projects/tier/files/

- Thomas

Re: Clusters and pools

[Wido den Hollander]

On 07/25/2012 06:34 AM, Ryan Nicholson wrote:
> I'm running a cluster based on 4 hosts that each have 3 fast, SCSI osd's, and 1 very large SATA osd, meaning, 12 fast osd's and 4 slow osd's total. I wish to segregate these into 2 pools, that operate independently. The goal is to use the faster disks as an area to hold rbd based VM's, and the larger area to host rbd-base large volumes (to start), and possibly have that become just a big cephfs area once, the fs side of things is considered more stable.
>
> Now, I've been thrown a couple options, and am still unsettled. Which is best?:
> - Create 2 independent clusters; one with the 12 SCSI osd's and the other with just the 4 large OSD's on the same hosts. This seems to be more complex from a scripting and boot time standpoint, but easier for my head.
> - Create a single cluster and use CRUSH rules to separate the two. This one STILL has me lost, as I'm having trouble understanding the crushmap syntax, the Crushmap import/export commands, and the other mkpool or otherwise commands from the docs in order to "make rbd's come from this faster pool", while "cephfs, you come from this slower pool". I really would like to entertain this path, however, as this allows ceph to handle the entire situation, and, it would seem more elegant.
>
> I'm also open to other options as well.

The "easiest" way to approach this:

Set up the cluster with the 12 fast OSD's first en leave the other 4 out 
of the configuration.

Get everything up and running and play with it.

Then, add the 4 remaining OSD's to the cluster:
1. Add them to ceph.conf
2. Increment max_osd
3. Add them to the keyring
4. Format the OSD's
5. Start the OSD's

Now they should show up in your "ceph -s" output, but no data will go to 
them.

The next step is to export your current crushmap:

$ ceph osd getcrushmap -o crushmap
$ crushtool -d crushmap -o crushmap.txt

You should now add 4 new hosts to the crushmap, something like 
"hostA-slow" and add one OSD under each of them.

Now you can add a new rack called "slowrbd" for example, add a new pool 
and a new rule afterwards.

Compile crushmap.txt back again to "crushmap" and load it into the cluster.

You can now create a new pool with a specific crushrule.

All the data in that pool will go onto those 4 slower OSD's.

Wido

>
> Thanks!
>
> Ryan Nicholson
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

RE: Clusters and pools

[Ryan Nicholson]

Thanks, Wido, for your help. Here's another question for the group:

I've created 2 new rados pools;

SCSI and
Large
I've also REMOVED 'data' from the pools. <--not sure if this is an issue.

How do I setup mountpoints? I wish to mount SCSI as CephFS, or, LaCie as CephFS. My goal is, from the client's perspective to treat these two entities as separate. In production, only RBD's will be on SCSI, and CephFS stuff would live on Large but I do want to learn how to configure this before I take the system out of testing.

I wasn't sure if there was an option for mount.ceph that allowed me to select the pool? Say, x.x.x.x:6789:??:/ or otherwise.

Thanks!

Ryan


-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Wido den Hollander
Sent: Wednesday, July 25, 2012 10:02 AM
To: Ryan Nicholson
Cc: ceph-devel@vger.kernel.org
Subject: Re: Clusters and pools



On 07/25/2012 06:34 AM, Ryan Nicholson wrote:
> I'm running a cluster based on 4 hosts that each have 3 fast, SCSI osd's, and 1 very large SATA osd, meaning, 12 fast osd's and 4 slow osd's total. I wish to segregate these into 2 pools, that operate independently. The goal is to use the faster disks as an area to hold rbd based VM's, and the larger area to host rbd-base large volumes (to start), and possibly have that become just a big cephfs area once, the fs side of things is considered more stable.
>
> Now, I've been thrown a couple options, and am still unsettled. Which is best?:
> - Create 2 independent clusters; one with the 12 SCSI osd's and the other with just the 4 large OSD's on the same hosts. This seems to be more complex from a scripting and boot time standpoint, but easier for my head.
> - Create a single cluster and use CRUSH rules to separate the two. This one STILL has me lost, as I'm having trouble understanding the crushmap syntax, the Crushmap import/export commands, and the other mkpool or otherwise commands from the docs in order to "make rbd's come from this faster pool", while "cephfs, you come from this slower pool". I really would like to entertain this path, however, as this allows ceph to handle the entire situation, and, it would seem more elegant.
>
> I'm also open to other options as well.

The "easiest" way to approach this:

Set up the cluster with the 12 fast OSD's first en leave the other 4 out of the configuration.

Get everything up and running and play with it.

Then, add the 4 remaining OSD's to the cluster:
1. Add them to ceph.conf
2. Increment max_osd
3. Add them to the keyring
4. Format the OSD's
5. Start the OSD's

Now they should show up in your "ceph -s" output, but no data will go to them.

The next step is to export your current crushmap:

$ ceph osd getcrushmap -o crushmap
$ crushtool -d crushmap -o crushmap.txt

You should now add 4 new hosts to the crushmap, something like "hostA-slow" and add one OSD under each of them.

Now you can add a new rack called "slowrbd" for example, add a new pool and a new rule afterwards.

Compile crushmap.txt back again to "crushmap" and load it into the cluster.

You can now create a new pool with a specific crushrule.

All the data in that pool will go onto those 4 slower OSD's.

Wido

>
> Thanks!
>
> Ryan Nicholson
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" 
> in the body of a message to majordomo@vger.kernel.org More majordomo 
> info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Clusters and pools

[Wido den Hollander]

On 07/27/2012 03:25 AM, Ryan Nicholson wrote:
> Thanks, Wido, for your help. Here's another question for the group:
>
> I've created 2 new rados pools;
>
> SCSI and
> Large
> I've also REMOVED 'data' from the pools. <--not sure if this is an issue.
>
> How do I setup mountpoints? I wish to mount SCSI as CephFS, or, LaCie as CephFS. My goal is, from the client's perspective to treat these two entities as separate. In production, only RBD's will be on SCSI, and CephFS stuff would live on Large but I do want to learn how to configure this before I take the system out of testing.
>
> I wasn't sure if there was an option for mount.ceph that allowed me to select the pool? Say, x.x.x.x:6789:??:/ or otherwise.
>

I'm not completely sure about this one. But the "data" pool is being 
used by the filesystem.

There is only one tree for the filesystem, but sub-directories can be 
given separate pools.

So, create the "data" pool again and create a directory called "scsi" 
and "large".

With the cephfs tool you should now be able to say that all the data of 
the scsi directory goes into the SCSI pool and all the data of the large 
directory goes into the pool "large".

I haven't done this myself yet, so I'm not completely sure about the 
syntax for the cephfs tool.

Wido

> Thanks!
>
> Ryan
>
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Wido den Hollander
> Sent: Wednesday, July 25, 2012 10:02 AM
> To: Ryan Nicholson
> Cc: ceph-devel@vger.kernel.org
> Subject: Re: Clusters and pools
>
>
>
> On 07/25/2012 06:34 AM, Ryan Nicholson wrote:
>> I'm running a cluster based on 4 hosts that each have 3 fast, SCSI osd's, and 1 very large SATA osd, meaning, 12 fast osd's and 4 slow osd's total. I wish to segregate these into 2 pools, that operate independently. The goal is to use the faster disks as an area to hold rbd based VM's, and the larger area to host rbd-base large volumes (to start), and possibly have that become just a big cephfs area once, the fs side of things is considered more stable.
>>
>> Now, I've been thrown a couple options, and am still unsettled. Which is best?:
>> - Create 2 independent clusters; one with the 12 SCSI osd's and the other with just the 4 large OSD's on the same hosts. This seems to be more complex from a scripting and boot time standpoint, but easier for my head.
>> - Create a single cluster and use CRUSH rules to separate the two. This one STILL has me lost, as I'm having trouble understanding the crushmap syntax, the Crushmap import/export commands, and the other mkpool or otherwise commands from the docs in order to "make rbd's come from this faster pool", while "cephfs, you come from this slower pool". I really would like to entertain this path, however, as this allows ceph to handle the entire situation, and, it would seem more elegant.
>>
>> I'm also open to other options as well.
>
> The "easiest" way to approach this:
>
> Set up the cluster with the 12 fast OSD's first en leave the other 4 out of the configuration.
>
> Get everything up and running and play with it.
>
> Then, add the 4 remaining OSD's to the cluster:
> 1. Add them to ceph.conf
> 2. Increment max_osd
> 3. Add them to the keyring
> 4. Format the OSD's
> 5. Start the OSD's
>
> Now they should show up in your "ceph -s" output, but no data will go to them.
>
> The next step is to export your current crushmap:
>
> $ ceph osd getcrushmap -o crushmap
> $ crushtool -d crushmap -o crushmap.txt
>
> You should now add 4 new hosts to the crushmap, something like "hostA-slow" and add one OSD under each of them.
>
> Now you can add a new rack called "slowrbd" for example, add a new pool and a new rule afterwards.
>
> Compile crushmap.txt back again to "crushmap" and load it into the cluster.
>
> You can now create a new pool with a specific crushrule.
>
> All the data in that pool will go onto those 4 slower OSD's.
>
> Wido
>
>>
>> Thanks!
>>
>> Ryan Nicholson
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel"
>> in the body of a message to majordomo@vger.kernel.org More majordomo
>> info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>