What would a good OSD node hardware configuration look like?

What would a good OSD node hardware configuration look like?

[Dennis Jacobfeuerborn]

Hi,
I'm thinking about building a ceph cluster and I'm wondering what a good
configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
16-disk systems.
Would it make sense to make each disk an individual OSD or should I perhaps
create several raid-0 and create OSDs from those?
Also what is the best setup for the journal? If I understand it correctly
then each OSD needs its own journal and that should be a separate disk but
that would be quite wasteful it seems. Would it make sense to put in two
small SSD disks in a raid-1 configuration and create a filesystem for each
OSD journal on it?
How does the number of OSDs/Nodes affect the performance of say a single dd
operation? Will blocks be distributed over the cluster and written/read in
parallel or does the number only improve concurrency rather than benefit
single threaded workloads?

Regards,
  Dennis

Re: What would a good OSD node hardware configuration look like?

[Josh Durgin]

On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
> Hi,
> I'm thinking about building a ceph cluster and I'm wondering what a good
> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
> 16-disk systems.
> Would it make sense to make each disk an individual OSD or should I perhaps
> create several raid-0 and create OSDs from those?

This mainly depends on your ratio of disks to cpu/ram. Generally we
recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
running 1 OSD/disk is pretty common. It makes recovering from a
single disk failure faster.

> Also what is the best setup for the journal? If I understand it correctly
> then each OSD needs its own journal and that should be a separate disk but
> that would be quite wasteful it seems. Would it make sense to put in two
> small SSD disks in a raid-1 configuration and create a filesystem for each
> OSD journal on it?

This is certainly possible. It's a bit less overhead if you give each
osd it's own partition of the ssd(s) instead of going through another
filesystem.

I suspect it would be better to not use raid-1, since these ssds will be
receiving all the data the osds write as well. If they're in raid-1 
instead of being used independently, their lifetimes might be much
shorter.

> How does the number of OSDs/Nodes affect the performance of say a single dd
> operation? Will blocks be distributed over the cluster and written/read in
> parallel or does the number only improve concurrency rather than benefit
> single threaded workloads?

In cephfs and rbd, objects are distributed over the cluster, but the
OSDs/node ratio doesn't really affect the performance. It's more
dependent on the workload and striping policy. For example, with
a small stripe size, small sequential writes will benefit from more
osds, but the number per node isn't particularly important.

Josh

Re: What would a good OSD node hardware configuration look like?

[Dennis Jacobfeuerborn]

On 11/06/2012 01:14 AM, Josh Durgin wrote:
> On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
>> Hi,
>> I'm thinking about building a ceph cluster and I'm wondering what a good
>> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
>> 16-disk systems.
>> Would it make sense to make each disk an individual OSD or should I perhaps
>> create several raid-0 and create OSDs from those?
> 
> This mainly depends on your ratio of disks to cpu/ram. Generally we
> recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
> running 1 OSD/disk is pretty common. It makes recovering from a
> single disk failure faster.

So basically a 2Ghz quad-core CPU and 8GB RAM would be sufficient for 8 OSDs?

>> Also what is the best setup for the journal? If I understand it correctly
>> then each OSD needs its own journal and that should be a separate disk but
>> that would be quite wasteful it seems. Would it make sense to put in two
>> small SSD disks in a raid-1 configuration and create a filesystem for each
>> OSD journal on it?
> 
> This is certainly possible. It's a bit less overhead if you give each
> osd it's own partition of the ssd(s) instead of going through another
> filesystem.
> 
> I suspect it would be better to not use raid-1, since these ssds will be
> receiving all the data the osds write as well. If they're in raid-1 instead
> of being used independently, their lifetimes might be much
> shorter.

My primary concern here is fault tolerance. What happens when the journal
disk dies? Can ceph cope with that and write directly to the OSDs or would
that mean that with a single shared disk for all OSDs a failure would mean
the entire system is effectively offline for ceph?

>> How does the number of OSDs/Nodes affect the performance of say a single dd
>> operation? Will blocks be distributed over the cluster and written/read in
>> parallel or does the number only improve concurrency rather than benefit
>> single threaded workloads?
> 
> In cephfs and rbd, objects are distributed over the cluster, but the
> OSDs/node ratio doesn't really affect the performance. It's more
> dependent on the workload and striping policy. For example, with
> a small stripe size, small sequential writes will benefit from more
> osds, but the number per node isn't particularly important.

By OSDs/Nodes I really meant "OSDs or nodes" and not the ratio. What I'm
trying to understand is if a) the number of nodes plays a significant role
when it comes to performance (e.g. a 4 node cluster with large disks vs. a
16 node cluster with smaller disks) and b) how much of an impact the number
of OSDs has on the cluster e.g. an 8 node cluster with each node being a
single OSD (with all disks as raid-0) vs. an 8 node cluster with say 64
OSDs (each node with 8 disks as individual OSDs).

What I'm trying to find is a good baseline hardware configuration that
works well with the algorithms and assumptions made by cephs design i.e. if
cepth works better with many smaller OSDs rather than a few larger ones
then that would obviously influence the overall design of the box.

Regards,
  Dennis

Re: What would a good OSD node hardware configuration look like?

[Josh Durgin]

On 11/05/2012 06:49 PM, Dennis Jacobfeuerborn wrote:
> On 11/06/2012 01:14 AM, Josh Durgin wrote:
>> On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I'm thinking about building a ceph cluster and I'm wondering what a good
>>> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
>>> 16-disk systems.
>>> Would it make sense to make each disk an individual OSD or should I perhaps
>>> create several raid-0 and create OSDs from those?
>>
>> This mainly depends on your ratio of disks to cpu/ram. Generally we
>> recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
>> running 1 OSD/disk is pretty common. It makes recovering from a
>> single disk failure faster.
>
> So basically a 2Ghz quad-core CPU and 8GB RAM would be sufficient for 8 OSDs?

Yes, although more RAM will be better (providing more page cache).

>>> Also what is the best setup for the journal? If I understand it correctly
>>> then each OSD needs its own journal and that should be a separate disk but
>>> that would be quite wasteful it seems. Would it make sense to put in two
>>> small SSD disks in a raid-1 configuration and create a filesystem for each
>>> OSD journal on it?
>>
>> This is certainly possible. It's a bit less overhead if you give each
>> osd it's own partition of the ssd(s) instead of going through another
>> filesystem.
>>
>> I suspect it would be better to not use raid-1, since these ssds will be
>> receiving all the data the osds write as well. If they're in raid-1 instead
>> of being used independently, their lifetimes might be much
>> shorter.
>
> My primary concern here is fault tolerance. What happens when the journal
> disk dies? Can ceph cope with that and write directly to the OSDs or would
> that mean that with a single shared disk for all OSDs a failure would mean
> the entire system is effectively offline for ceph?

I'm going to point to some messages in the archives to avoid repetition:

http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/6377

>>> How does the number of OSDs/Nodes affect the performance of say a single dd
>>> operation? Will blocks be distributed over the cluster and written/read in
>>> parallel or does the number only improve concurrency rather than benefit
>>> single threaded workloads?
>>
>> In cephfs and rbd, objects are distributed over the cluster, but the
>> OSDs/node ratio doesn't really affect the performance. It's more
>> dependent on the workload and striping policy. For example, with
>> a small stripe size, small sequential writes will benefit from more
>> osds, but the number per node isn't particularly important.
>
> By OSDs/Nodes I really meant "OSDs or nodes" and not the ratio. What I'm
> trying to understand is if a) the number of nodes plays a significant role
> when it comes to performance (e.g. a 4 node cluster with large disks vs. a
> 16 node cluster with smaller disks) and b) how much of an impact the number
> of OSDs has on the cluster e.g. an 8 node cluster with each node being a
> single OSD (with all disks as raid-0) vs. an 8 node cluster with say 64
> OSDs (each node with 8 disks as individual OSDs).

Generally more smaller nodes will recover faster from a node or disk 
failure than a few larger node, since the remaining OSDs recover in
parallel. There are some other advantages of many small nodes. Wido and
Stefan covered this well in this thread:

http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10212

> What I'm trying to find is a good baseline hardware configuration that
> works well with the algorithms and assumptions made by cephs design i.e. if
> cepth works better with many smaller OSDs rather than a few larger ones
> then that would obviously influence the overall design of the box.
>
> Regards,
>    Dennis
>

Re: What would a good OSD node hardware configuration look like?

[Dennis Jacobfeuerborn]

On 11/06/2012 08:30 PM, Josh Durgin wrote:
> On 11/05/2012 06:49 PM, Dennis Jacobfeuerborn wrote:
>> On 11/06/2012 01:14 AM, Josh Durgin wrote:
>>> On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
>>>> Hi,
>>>> I'm thinking about building a ceph cluster and I'm wondering what a good
>>>> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
>>>> 16-disk systems.
>>>> Would it make sense to make each disk an individual OSD or should I
>>>> perhaps
>>>> create several raid-0 and create OSDs from those?
>>>
>>> This mainly depends on your ratio of disks to cpu/ram. Generally we
>>> recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
>>> running 1 OSD/disk is pretty common. It makes recovering from a
>>> single disk failure faster.
>>
>> So basically a 2Ghz quad-core CPU and 8GB RAM would be sufficient for 8
>> OSDs?
> 
> Yes, although more RAM will be better (providing more page cache).
> 
>>>> Also what is the best setup for the journal? If I understand it correctly
>>>> then each OSD needs its own journal and that should be a separate disk but
>>>> that would be quite wasteful it seems. Would it make sense to put in two
>>>> small SSD disks in a raid-1 configuration and create a filesystem for each
>>>> OSD journal on it?
>>>
>>> This is certainly possible. It's a bit less overhead if you give each
>>> osd it's own partition of the ssd(s) instead of going through another
>>> filesystem.
>>>
>>> I suspect it would be better to not use raid-1, since these ssds will be
>>> receiving all the data the osds write as well. If they're in raid-1 instead
>>> of being used independently, their lifetimes might be much
>>> shorter.
>>
>> My primary concern here is fault tolerance. What happens when the journal
>> disk dies? Can ceph cope with that and write directly to the OSDs or would
>> that mean that with a single shared disk for all OSDs a failure would mean
>> the entire system is effectively offline for ceph?
> 
> I'm going to point to some messages in the archives to avoid repetition:
> 
> http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/6377
> 
>>>> How does the number of OSDs/Nodes affect the performance of say a
>>>> single dd
>>>> operation? Will blocks be distributed over the cluster and written/read in
>>>> parallel or does the number only improve concurrency rather than benefit
>>>> single threaded workloads?
>>>
>>> In cephfs and rbd, objects are distributed over the cluster, but the
>>> OSDs/node ratio doesn't really affect the performance. It's more
>>> dependent on the workload and striping policy. For example, with
>>> a small stripe size, small sequential writes will benefit from more
>>> osds, but the number per node isn't particularly important.
>>
>> By OSDs/Nodes I really meant "OSDs or nodes" and not the ratio. What I'm
>> trying to understand is if a) the number of nodes plays a significant role
>> when it comes to performance (e.g. a 4 node cluster with large disks vs. a
>> 16 node cluster with smaller disks) and b) how much of an impact the number
>> of OSDs has on the cluster e.g. an 8 node cluster with each node being a
>> single OSD (with all disks as raid-0) vs. an 8 node cluster with say 64
>> OSDs (each node with 8 disks as individual OSDs).
> 
> Generally more smaller nodes will recover faster from a node or disk
> failure than a few larger node, since the remaining OSDs recover in
> parallel. There are some other advantages of many small nodes. Wido and
> Stefan covered this well in this thread:
> 
> http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10212
> 

So that sound like a raid-1 (or potentially a raid-10) is pretty much a
must when using a shared ssd disk for the journals for more than one OSD.
Without redundancy the failure of a single disk (the journal one) would
take down all OSDs on that node making a multi OSD per node setup pointless.

Regards,
  Dennis

Re: What would a good OSD node hardware configuration look like?

[Wido den Hollander]

On 07-11-12 02:35, Dennis Jacobfeuerborn wrote:
> On 11/06/2012 08:30 PM, Josh Durgin wrote:
>> On 11/05/2012 06:49 PM, Dennis Jacobfeuerborn wrote:
>>> On 11/06/2012 01:14 AM, Josh Durgin wrote:
>>>> On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
>>>>> Hi,
>>>>> I'm thinking about building a ceph cluster and I'm wondering what a good
>>>>> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
>>>>> 16-disk systems.
>>>>> Would it make sense to make each disk an individual OSD or should I
>>>>> perhaps
>>>>> create several raid-0 and create OSDs from those?
>>>>
>>>> This mainly depends on your ratio of disks to cpu/ram. Generally we
>>>> recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
>>>> running 1 OSD/disk is pretty common. It makes recovering from a
>>>> single disk failure faster.
>>>
>>> So basically a 2Ghz quad-core CPU and 8GB RAM would be sufficient for 8
>>> OSDs?
>>
>> Yes, although more RAM will be better (providing more page cache).
>>
>>>>> Also what is the best setup for the journal? If I understand it correctly
>>>>> then each OSD needs its own journal and that should be a separate disk but
>>>>> that would be quite wasteful it seems. Would it make sense to put in two
>>>>> small SSD disks in a raid-1 configuration and create a filesystem for each
>>>>> OSD journal on it?
>>>>
>>>> This is certainly possible. It's a bit less overhead if you give each
>>>> osd it's own partition of the ssd(s) instead of going through another
>>>> filesystem.
>>>>
>>>> I suspect it would be better to not use raid-1, since these ssds will be
>>>> receiving all the data the osds write as well. If they're in raid-1 instead
>>>> of being used independently, their lifetimes might be much
>>>> shorter.
>>>
>>> My primary concern here is fault tolerance. What happens when the journal
>>> disk dies? Can ceph cope with that and write directly to the OSDs or would
>>> that mean that with a single shared disk for all OSDs a failure would mean
>>> the entire system is effectively offline for ceph?
>>
>> I'm going to point to some messages in the archives to avoid repetition:
>>
>> http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/6377
>>
>>>>> How does the number of OSDs/Nodes affect the performance of say a
>>>>> single dd
>>>>> operation? Will blocks be distributed over the cluster and written/read in
>>>>> parallel or does the number only improve concurrency rather than benefit
>>>>> single threaded workloads?
>>>>
>>>> In cephfs and rbd, objects are distributed over the cluster, but the
>>>> OSDs/node ratio doesn't really affect the performance. It's more
>>>> dependent on the workload and striping policy. For example, with
>>>> a small stripe size, small sequential writes will benefit from more
>>>> osds, but the number per node isn't particularly important.
>>>
>>> By OSDs/Nodes I really meant "OSDs or nodes" and not the ratio. What I'm
>>> trying to understand is if a) the number of nodes plays a significant role
>>> when it comes to performance (e.g. a 4 node cluster with large disks vs. a
>>> 16 node cluster with smaller disks) and b) how much of an impact the number
>>> of OSDs has on the cluster e.g. an 8 node cluster with each node being a
>>> single OSD (with all disks as raid-0) vs. an 8 node cluster with say 64
>>> OSDs (each node with 8 disks as individual OSDs).
>>
>> Generally more smaller nodes will recover faster from a node or disk
>> failure than a few larger node, since the remaining OSDs recover in
>> parallel. There are some other advantages of many small nodes. Wido and
>> Stefan covered this well in this thread:
>>
>> http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10212
>>
>
> So that sound like a raid-1 (or potentially a raid-10) is pretty much a
> must when using a shared ssd disk for the journals for more than one OSD.
> Without redundancy the failure of a single disk (the journal one) would
> take down all OSDs on that node making a multi OSD per node setup pointless.
>

Except that SSDs will mainly fail due to the amount of write cycles they 
had to endure.

So in RAID-1 your SSDs will fail at almost the same time.

With for example 8 OSDs in a server you better spread them out 50/50 
over two SSDs.

Wido

> Regards,
>    Dennis
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: What would a good OSD node hardware configuration look like?

[Gandalf Corvotempesta]

2012/11/7 Wido den Hollander <wido@widodh.nl>:
> Except that SSDs will mainly fail due to the amount of write cycles they had
> to endure.
>
> So in RAID-1 your SSDs will fail at almost the same time.
>
> With for example 8 OSDs in a server you better spread them out 50/50 over
> two SSDs.

But the OS should be installed on SSD, to allow the use of the whole
SATA/SAS disk as OSDs.
Without RAID-1 between SSDs, a disk failure will result it the whole node down.

Re: What would a good OSD node hardware configuration look like?

[Wido den Hollander]

On 07-11-12 09:17, Gandalf Corvotempesta wrote:
> 2012/11/7 Wido den Hollander <wido@widodh.nl>:
>> Except that SSDs will mainly fail due to the amount of write cycles they had
>> to endure.
>>
>> So in RAID-1 your SSDs will fail at almost the same time.
>>
>> With for example 8 OSDs in a server you better spread them out 50/50 over
>> two SSDs.
>
> But the OS should be installed on SSD, to allow the use of the whole
> SATA/SAS disk as OSDs.
> Without RAID-1 between SSDs, a disk failure will result it the whole node down.
> --

True. You might want to use a small internal USB stick like those from 
Transcend for installing your OS. They are available in 4GB and 8GB.

Let syslog handle your logging to an external device and you have almost 
no write I/O on that thing.

This way you can use the SSDs completely for journaling and shrink them 
using HPA with hdparm.

Still, I think that in the distributed nature it shouldn't matter if you 
loose a node. If loosing a node is a to big impact to your cluster, go 
with smaller nodes.

Wido

> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: What would a good OSD node hardware configuration look like?

[Gandalf Corvotempesta]

2012/11/7 Wido den Hollander <wido@widodh.nl>:
> True. You might want to use a small internal USB stick like those from
> Transcend for installing your OS. They are available in 4GB and 8GB.
>
> Let syslog handle your logging to an external device and you have almost no
> write I/O on that thing.

Ok, but in this case instead of a RAID between 2 SSD will be better to
use one SSD for journal and one SATA disks for OS.

DELL R515 has 12 front disks and 2 internal 2.5'' disks. I'm planning
to use these internal disks as journal and OS and the external disks
as OSD.

> Still, I think that in the distributed nature it shouldn't matter if you
> loose a node. If loosing a node is a to big impact to your cluster, go with
> smaller nodes.

This is true.

Re: What would a good OSD node hardware configuration look like?

[Stefan Priebe - Profihost AG]

Am 06.11.2012 01:14, schrieb Josh Durgin:
> On 11/05/2012 09:13 AM, Dennis Jacobfeuerborn wrote:
>> Hi,
>> I'm thinking about building a ceph cluster and I'm wondering what a good
>> configuration would look like for 4-8 (and maybe more) 2HU 8-disk or 3HU
>> 16-disk systems.
>> Would it make sense to make each disk an individual OSD or should I
>> perhaps
>> create several raid-0 and create OSDs from those?
>
> This mainly depends on your ratio of disks to cpu/ram. Generally we
> recommend 1GB ram and 1Ghz per OSD. If you've got enough cpu/ram,
> running 1 OSD/disk is pretty common. It makes recovering from a
> single disk failure faster.

Just i node while using SSDs i've seen osd processes using up to two 3.6 
Ghz cores. So under heavy load and disks and network with enough speed 1 
ghz might be even too low. Right now i'm calculating with 1 3.6Ghz Core 
per OSD when using 10GBE and SSDs.

Greets,
Stefan