Optimize Ceph cluster (kernel, osd, rbd)

Optimize Ceph cluster (kernel, osd, rbd)

[Ta Ba Tuan]

[-- Attachment #1.1: Type: text/plain, Size: 3230 bytes --]

Hi everyone,

I have *3 nodes (running MON and MDS)*
and *6 data nodes ( 84 OSDs**)*
Each data nodes has configuraions:
   - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
   - RAM: 32GB
   - Disk: 14*4TB
(14disks *4TB *6 data nodes= 84 OSDs)

To optimize Ceph Cluster, *I adjusted some kernel arguments* (nr_request 
in queue and increated read throughput):

#Adjust nr_request in queue (staying in mem - default is 128)
     echo 1024 > /sys/block/sdb/queue/nr_requests
     echo noop > /sys/block/sda/queue/scheduler   (default= noop 
deadline [cfq])
#Increase read throughput  (default: 128)
     echo "512" > /sys/block/*/queue/read_ahead_kb

And, *tuning Ceph configuraion options below:*

[client]

      rbd cache = true
      rbd cache size = 536870912
      rbd cache max dirty = 134217728
      rbd cache target dirty = 33554432
      rbd cache max dirty age = 5

[osd]
     osd data = /var/lib/ceph/osd/cloud-$id
     osd journal = /var/lib/ceph/osd/cloud-$id/journal
     osd journal size = 10000
     osd mkfs type = xfs
     osd mkfs options xfs = "-f -i size=2048"
     osd mount options xfs = "rw,noatime,inode64,logbsize=250k"

     keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
#increasing the number may increase the request processing rate
     osd op threads = 24
#The number of disk threads, which are used to perform background disk 
intensive OSD operations such as scrubbing and snap trimming
     osd disk threads =24
#The number of active recovery requests per OSD at one time. More 
requests will accelerate recovery, but the requests places an increased 
load on the cluster.
     osd recovery max active =1
#writing direct to the journal.
#Allow use of libaio to do asynchronous writes
     journal dio = true
     journal aio = true
#Synchronization interval:
#The maximum/minimum interval in seconds for synchronizing the filestore.
     filestore max sync interval = 100
     filestore min sync interval = 50
#Defines the maximum number of in progress operations the file store 
accepts before blocking on queuing new operations.
     filestore queue max ops = 2000
#The maximum number of bytes for an operation
     filestore queue max bytes = 536870912
#The maximum number of operations the filestore can commit.
     filestore queue committing max ops = 2000 (default =500)
#The maximum number of bytes the filestore can commit.
     filestore queue committing max bytes = 536870912
#When you add or remove Ceph OSD Daemons to a cluster, the CRUSH 
algorithm will want to rebalance the cluster by moving placement groups 
to or from Ceph OSD Daemons to restore the balance. The process of 
migrating placement groups and the objects they contain can reduce the 
cluster's operational performance considerably. To maintain operational 
performance, Ceph performs this migration with 'backfilling', which 
allows Ceph to set backfill operations to a lower priority than requests 
to read or write data.
     osd max backfills = 1


Tomorrow, I'm going to implement Ceph Cluster,
I have very little experience in managing Ceph. So, I hope someone give 
me advices about above arguments and guide me how to best optimize ceph 
cluster?

Thank you so much!
--tuantaba




[-- Attachment #1.2: Type: text/html, Size: 4430 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Optimize Ceph cluster (kernel, osd, rbd)

[Ta Ba Tuan]

[-- Attachment #1.1: Type: text/plain, Size: 3419 bytes --]

Please help me!


On 07/20/2013 02:11 AM, Ta Ba Tuan wrote:
> Hi everyone,
>
> I have *3 nodes (running MON and MDS)*
> and *6 data nodes ( 84 OSDs**)*
> Each data nodes has configuraions:
>   - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
>   - RAM: 32GB
>   - Disk: 14*4TB
> (14disks *4TB *6 data nodes= 84 OSDs)
>
> To optimize Ceph Cluster, *I adjusted some kernel arguments* 
> (nr_request in queue and increated read throughput):
>
> #Adjust nr_request in queue (staying in mem - default is 128)
>     echo 1024 > /sys/block/sdb/queue/nr_requests
>     echo noop > /sys/block/sda/queue/scheduler   (default= noop 
> deadline [cfq])
> #Increase read throughput  (default: 128)
>     echo "512" > /sys/block/*/queue/read_ahead_kb
>
> And, *tuning Ceph configuraion options below:*
>
> [client]
>
>      rbd cache = true
>      rbd cache size = 536870912
>      rbd cache max dirty = 134217728
>      rbd cache target dirty = 33554432
>      rbd cache max dirty age = 5
>
> [osd]
>     osd data = /var/lib/ceph/osd/cloud-$id
>     osd journal = /var/lib/ceph/osd/cloud-$id/journal
>     osd journal size = 10000
>     osd mkfs type = xfs
>     osd mkfs options xfs = "-f -i size=2048"
>     osd mount options xfs = "rw,noatime,inode64,logbsize=250k"
>
>     keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
> #increasing the number may increase the request processing rate
>     osd op threads = 24
> #The number of disk threads, which are used to perform background disk 
> intensive OSD operations such as scrubbing and snap trimming
>     osd disk threads =24
> #The number of active recovery requests per OSD at one time. More 
> requests will accelerate recovery, but the requests places an 
> increased load on the cluster.
>     osd recovery max active =1
> #writing direct to the journal.
> #Allow use of libaio to do asynchronous writes
>     journal dio = true
>     journal aio = true
> #Synchronization interval:
> #The maximum/minimum interval in seconds for synchronizing the filestore.
>     filestore max sync interval = 100
>     filestore min sync interval = 50
> #Defines the maximum number of in progress operations the file store 
> accepts before blocking on queuing new operations.
>     filestore queue max ops = 2000
> #The maximum number of bytes for an operation
>     filestore queue max bytes = 536870912
> #The maximum number of operations the filestore can commit.
>     filestore queue committing max ops = 2000 (default =500)
> #The maximum number of bytes the filestore can commit.
>     filestore queue committing max bytes = 536870912
> #When you add or remove Ceph OSD Daemons to a cluster, the CRUSH 
> algorithm will want to rebalance the cluster by moving placement 
> groups to or from Ceph OSD Daemons to restore the balance. The process 
> of migrating placement groups and the objects they contain can reduce 
> the cluster's operational performance considerably. To maintain 
> operational performance, Ceph performs this migration with 
> 'backfilling', which allows Ceph to set backfill operations to a lower 
> priority than requests to read or write data.
>     osd max backfills = 1
>
>
> Tomorrow, I'm going to implement Ceph Cluster,
> I have very little experience in managing Ceph. So, I hope someone 
> give me advices about above arguments and guide me how to best 
> optimize ceph cluster?
>
> Thank you so much!
> --tuantaba
>
>
>


[-- Attachment #1.2: Type: text/html, Size: 4958 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: Optimize Ceph cluster (kernel, osd, rbd)

[Dan Mick]

I would get the cluster up and running and do some experiments before I 
spent any time on optimization, much less all this.

On 07/20/2013 09:35 AM, Ta Ba Tuan wrote:
> Please help me!
>
>
> On 07/20/2013 02:11 AM, Ta Ba Tuan wrote:
>> Hi everyone,
>>
>> I have *3 nodes (running MON and MDS)*
>> and *6 data nodes ( 84 OSDs**)*
>> Each data nodes has configuraions:
>>   - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
>>   - RAM: 32GB
>>   - Disk: 14*4TB
>> (14disks *4TB *6 data nodes= 84 OSDs)
>>
>> To optimize Ceph Cluster, *I adjusted some kernel arguments*
>> (nr_request in queue and increated read throughput):
>>
>> #Adjust nr_request in queue (staying in mem - default is 128)
>>     echo 1024 > /sys/block/sdb/queue/nr_requests
>>     echo noop > /sys/block/sda/queue/scheduler   (default= noop
>> deadline [cfq])
>> #Increase read throughput  (default: 128)
>>     echo "512" > /sys/block/*/queue/read_ahead_kb
>>
>> And, *tuning Ceph configuraion options below:*
>>
>> [client]
>>
>>      rbd cache = true
>>      rbd cache size = 536870912
>>      rbd cache max dirty = 134217728
>>      rbd cache target dirty = 33554432
>>      rbd cache max dirty age = 5
>>
>> [osd]
>>     osd data = /var/lib/ceph/osd/cloud-$id
>>     osd journal = /var/lib/ceph/osd/cloud-$id/journal
>>     osd journal size = 10000
>>     osd mkfs type = xfs
>>     osd mkfs options xfs = "-f -i size=2048"
>>     osd mount options xfs = "rw,noatime,inode64,logbsize=250k"
>>
>>     keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
>> #increasing the number may increase the request processing rate
>>     osd op threads = 24
>> #The number of disk threads, which are used to perform background disk
>> intensive OSD operations such as scrubbing and snap trimming
>>     osd disk threads =24
>> #The number of active recovery requests per OSD at one time. More
>> requests will accelerate recovery, but the requests places an
>> increased load on the cluster.
>>     osd recovery max active =1
>> #writing direct to the journal.
>> #Allow use of libaio to do asynchronous writes
>>     journal dio = true
>>     journal aio = true
>> #Synchronization interval:
>> #The maximum/minimum interval in seconds for synchronizing the filestore.
>>     filestore max sync interval = 100
>>     filestore min sync interval = 50
>> #Defines the maximum number of in progress operations the file store
>> accepts before blocking on queuing new operations.
>>     filestore queue max ops = 2000
>> #The maximum number of bytes for an operation
>>     filestore queue max bytes = 536870912
>> #The maximum number of operations the filestore can commit.
>>     filestore queue committing max ops = 2000 (default =500)
>> #The maximum number of bytes the filestore can commit.
>>     filestore queue committing max bytes = 536870912
>> #When you add or remove Ceph OSD Daemons to a cluster, the CRUSH
>> algorithm will want to rebalance the cluster by moving placement
>> groups to or from Ceph OSD Daemons to restore the balance. The process
>> of migrating placement groups and the objects they contain can reduce
>> the cluster’s operational performance considerably. To maintain
>> operational performance, Ceph performs this migration with
>> ‘backfilling’, which allows Ceph to set backfill operations to a lower
>> priority than requests to read or write data.
>>     osd max backfills = 1
>>
>>
>> Tomorrow, I'm going to implement Ceph Cluster,
>> I have very little experience in managing Ceph. So, I hope someone
>> give me advices about above arguments and guide me how to best
>> optimize ceph cluster?
>>
>> Thank you so much!
>> --tuantaba
>>
>>
>>
>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>