Ceph Production Environment Setup?

Ceph Production Environment Setup?

[femi anjorin]

Please can anyone  an advise  on how exactly a CEPH production
environment should look like? and what the configuration files should
be. My hardwares include the following:

Server A, B, C configuration
CPU - Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz
RAM - 16GB
Hard drive -  500GB
SSD - 120GB

Server D,E,F,G,H,J configuration
CPU - Intel(R) Atom(TM) CPU D525   @ 1.80GHz
RAM - 4 GB
Boot drive -  320gb
SSD - 120 GB
Storage drives - 16 X 2 TB

I am thinking of these configurations but i am not sure.
Server A - MDS and MON
Server B - MON
Server C - MON
Server D, E,F,G,H,J - OSD

Regards.

Re: Ceph Production Environment Setup?

[Joao Eduardo Luis]

On 01/29/2013 02:56 AM, femi anjorin wrote:
> Please can anyone  an advise  on how exactly a CEPH production
> environment should look like? and what the configuration files should
> be. My hardwares include the following:
>
> Server A, B, C configuration
> CPU - Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz
> RAM - 16GB
> Hard drive -  500GB
> SSD - 120GB
>
> Server D,E,F,G,H,J configuration
> CPU - Intel(R) Atom(TM) CPU D525   @ 1.80GHz
> RAM - 4 GB
> Boot drive -  320gb
> SSD - 120 GB
> Storage drives - 16 X 2 TB
>
> I am thinking of these configurations but i am not sure.
> Server A - MDS and MON
> Server B - MON
> Server C - MON
> Server D, E,F,G,H,J - OSD
>

Those 16GB RAM on the monitor nodes vs the 4GB RAM on the osd nodes seem 
to be a bit wrong to me.  The OSDs tend to require much more RAM, for 
instance for recovery, while the monitor is not as heavy on the memory 
-- if a cluster grows significantly large, the in-memory maps may grow a 
lot too, but that reason alone shouldn't be the reason you would give a 
monitor 16GB RAM and 4GB for an osd.

Furthermore, I see you have 16x2TB storage drives.  Is that per OSD 
node?  I'm assuming that's what you're aiming to do, so how many OSDs 
were you thinking of running on the same host?  Usually we go for 1 OSD 
per drive, but you might have something else on your mind.  I am not an 
expert on server configuration, but my point is that, if you are going 
to have more than one OSD on the same host, your RAM sure looks smaller 
than what I would envision.

BTW, not sure if you're placing SSDs on the monitor/mds nodes with the 
same intent as when you place it on the OSD nodes (keeping the osd 
journal, maybe?), but if you indeed intend to keep the daemons journals 
in them I think you should know that the monitor and the mds don't keep 
a journal.  The monitors do keep a store on disk, but the mds don't even 
do that, instead keeping its data directly on the osds and whatever it 
needs in-memory.

   -Joao

Re: Ceph Production Environment Setup?

[Martin B Nielsen]

There is also the hardware recommendation page in the ceph docs (
http://ceph.com/docs/master/install/hardware-recommendations/ )

Basically they recommend something like ~ 1GHz CPU (or 1 core/osd),
500M-1GB RAM pr OSD daemon. Also most run with 1 OSD daemon pr. disk
(so if you put 16x disk pr. node you'll vastly overpower your atom
cpu)

Overall, while the cluster chugs along happily the hw specs are
relatively modest; as soon as it starts to recover you'll see high
cpu/mem usage.

Cheers,
Martin


On Tue, Jan 29, 2013 at 3:56 AM, femi anjorin <femi.anjorin@gmail.com> wrote:
>
> Please can anyone  an advise  on how exactly a CEPH production
> environment should look like? and what the configuration files should
> be. My hardwares include the following:
>
> Server A, B, C configuration
> CPU - Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz
> RAM - 16GB
> Hard drive -  500GB
> SSD - 120GB
>
> Server D,E,F,G,H,J configuration
> CPU - Intel(R) Atom(TM) CPU D525   @ 1.80GHz
> RAM - 4 GB
> Boot drive -  320gb
> SSD - 120 GB
> Storage drives - 16 X 2 TB
>
> I am thinking of these configurations but i am not sure.
> Server A - MDS and MON
> Server B - MON
> Server C - MON
> Server D, E,F,G,H,J - OSD
>
> Regards.
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph Production Environment Setup?

[femi anjorin]

Thanks.  I have upgraded all the systems to quad core processor
machine with 32 GB RAM although i still have 16 hard drives on each of
the storage nodes.

16 hard drives means i should have 16 OSD daemon but i dont know what
the OSD configuration should look like in ceph.conf.

I mounted the disk under the OSD data directory according to
http://ceph.com/docs/master/rados/deployment/mkcephfs/
the mount looks like this:
/dev/sda1 on /var/lib/ceph/osd/ceph-0
/dev/sdb1 on /var/lib/ceph/osd/ceph-1
/dev/sdc1 on /var/lib/ceph/osd/ceph-2
/dev/sdd1 on /var/lib/ceph/osd/ceph-3
/dev/sde1 on /var/lib/ceph/osd/ceph-4
/dev/sdf1 on /var/lib/ceph/osd/ceph-5
/dev/sdg1 on /var/lib/ceph/osd/ceph-6
/dev/sdh1 on /var/lib/ceph/osd/ceph-7
/dev/sdi1 on /var/lib/ceph/osd/ceph-8
/dev/sdj1 on /var/lib/ceph/osd/ceph-9
/dev/sdk1 on /var/lib/ceph/osd/ceph-10
/dev/sdl1 on /var/lib/ceph/osd/ceph-11
/dev/sdm1 on /var/lib/ceph/osd/ceph-12
/dev/sdn1 on /var/lib/ceph/osd/ceph-13
/dev/sdo1 on /var/lib/ceph/osd/ceph-14
/dev/sdp1 on /var/lib/ceph/osd/ceph-15

BUT I dont know how the OSD configuration should look like?  I see the
following in this ceph reference :
http://ceph.com/docs/master/rados/deployment/mkcephfs/

"For each [osd.n] section of your configuration file, specify the
storage device. For example:
[osd.1]
        devs = /dev/sda
[osd.2]
        devs = /dev/sdb "

I guess this is a configuration for one hard drive. What should the
OSD config be with 16 drives in one host?



Regards,
Femi.


On Tue, Jan 29, 2013 at 1:39 PM, Martin B Nielsen <martin@unity3d.com> wrote:
> There is also the hardware recommendation page in the ceph docs (
> http://ceph.com/docs/master/install/hardware-recommendations/ )
>
> Basically they recommend something like ~ 1GHz CPU (or 1 core/osd),
> 500M-1GB RAM pr OSD daemon. Also most run with 1 OSD daemon pr. disk
> (so if you put 16x disk pr. node you'll vastly overpower your atom
> cpu)
>
> Overall, while the cluster chugs along happily the hw specs are
> relatively modest; as soon as it starts to recover you'll see high
> cpu/mem usage.
>
> Cheers,
> Martin
>
>
> On Tue, Jan 29, 2013 at 3:56 AM, femi anjorin <femi.anjorin@gmail.com> wrote:
>>
>> Please can anyone  an advise  on how exactly a CEPH production
>> environment should look like? and what the configuration files should
>> be. My hardwares include the following:
>>
>> Server A, B, C configuration
>> CPU - Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz
>> RAM - 16GB
>> Hard drive -  500GB
>> SSD - 120GB
>>
>> Server D,E,F,G,H,J configuration
>> CPU - Intel(R) Atom(TM) CPU D525   @ 1.80GHz
>> RAM - 4 GB
>> Boot drive -  320gb
>> SSD - 120 GB
>> Storage drives - 16 X 2 TB
>>
>> I am thinking of these configurations but i am not sure.
>> Server A - MDS and MON
>> Server B - MON
>> Server C - MON
>> Server D, E,F,G,H,J - OSD
>>
>> Regards.
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html