Optimize Ceph cluster (kernel, osd, rbd)

[Ta Ba Tuan <tuantb-QlevPasa8l681eZEIcUDRw@public.gmane.org>]

[-- Attachment #1.1: Type: text/plain, Size: 3230 bytes --]

Hi everyone,

I have *3 nodes (running MON and MDS)*
and *6 data nodes ( 84 OSDs**)*
Each data nodes has configuraions:
   - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
   - RAM: 32GB
   - Disk: 14*4TB
(14disks *4TB *6 data nodes= 84 OSDs)

To optimize Ceph Cluster, *I adjusted some kernel arguments* (nr_request 
in queue and increated read throughput):

#Adjust nr_request in queue (staying in mem - default is 128)
     echo 1024 > /sys/block/sdb/queue/nr_requests
     echo noop > /sys/block/sda/queue/scheduler   (default= noop 
deadline [cfq])
#Increase read throughput  (default: 128)
     echo "512" > /sys/block/*/queue/read_ahead_kb

And, *tuning Ceph configuraion options below:*

[client]

      rbd cache = true
      rbd cache size = 536870912
      rbd cache max dirty = 134217728
      rbd cache target dirty = 33554432
      rbd cache max dirty age = 5

[osd]
     osd data = /var/lib/ceph/osd/cloud-$id
     osd journal = /var/lib/ceph/osd/cloud-$id/journal
     osd journal size = 10000
     osd mkfs type = xfs
     osd mkfs options xfs = "-f -i size=2048"
     osd mount options xfs = "rw,noatime,inode64,logbsize=250k"

     keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
#increasing the number may increase the request processing rate
     osd op threads = 24
#The number of disk threads, which are used to perform background disk 
intensive OSD operations such as scrubbing and snap trimming
     osd disk threads =24
#The number of active recovery requests per OSD at one time. More 
requests will accelerate recovery, but the requests places an increased 
load on the cluster.
     osd recovery max active =1
#writing direct to the journal.
#Allow use of libaio to do asynchronous writes
     journal dio = true
     journal aio = true
#Synchronization interval:
#The maximum/minimum interval in seconds for synchronizing the filestore.
     filestore max sync interval = 100
     filestore min sync interval = 50
#Defines the maximum number of in progress operations the file store 
accepts before blocking on queuing new operations.
     filestore queue max ops = 2000
#The maximum number of bytes for an operation
     filestore queue max bytes = 536870912
#The maximum number of operations the filestore can commit.
     filestore queue committing max ops = 2000 (default =500)
#The maximum number of bytes the filestore can commit.
     filestore queue committing max bytes = 536870912
#When you add or remove Ceph OSD Daemons to a cluster, the CRUSH 
algorithm will want to rebalance the cluster by moving placement groups 
to or from Ceph OSD Daemons to restore the balance. The process of 
migrating placement groups and the objects they contain can reduce the 
cluster's operational performance considerably. To maintain operational 
performance, Ceph performs this migration with 'backfilling', which 
allows Ceph to set backfill operations to a lower priority than requests 
to read or write data.
     osd max backfills = 1


Tomorrow, I'm going to implement Ceph Cluster,
I have very little experience in managing Ceph. So, I hope someone give 
me advices about above arguments and guide me how to best optimize ceph 
cluster?

Thank you so much!
--tuantaba




[-- Attachment #1.2: Type: text/html, Size: 4430 bytes --]

[-- Attachment #2: Type: text/plain, Size: 178 bytes --]

_______________________________________________
ceph-users mailing list
ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com