From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ta Ba Tuan Subject: Re: Optimize Ceph cluster (kernel, osd, rbd) Date: Sat, 20 Jul 2013 23:35:12 +0700 Message-ID: <51EABC40.3010702@vccloud.vn> References: <51E98F62.8050800@vccloud.vn> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1242573343==" Return-path: In-Reply-To: <51E98F62.8050800-QlevPasa8l681eZEIcUDRw@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org Sender: ceph-users-bounces-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org To: "ceph-devel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , ceph-users List-Id: ceph-devel.vger.kernel.org This is a multi-part message in MIME format. --===============1242573343== Content-Type: multipart/alternative; boundary="------------010800020801020808020708" This is a multi-part message in MIME format. --------------010800020801020808020708 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Please help me! On 07/20/2013 02:11 AM, Ta Ba Tuan wrote: > Hi everyone, > > I have *3 nodes (running MON and MDS)* > and *6 data nodes ( 84 OSDs**)* > Each data nodes has configuraions: > - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz > - RAM: 32GB > - Disk: 14*4TB > (14disks *4TB *6 data nodes= 84 OSDs) > > To optimize Ceph Cluster, *I adjusted some kernel arguments* > (nr_request in queue and increated read throughput): > > #Adjust nr_request in queue (staying in mem - default is 128) > echo 1024 > /sys/block/sdb/queue/nr_requests > echo noop > /sys/block/sda/queue/scheduler (default= noop > deadline [cfq]) > #Increase read throughput (default: 128) > echo "512" > /sys/block/*/queue/read_ahead_kb > > And, *tuning Ceph configuraion options below:* > > [client] > > rbd cache = true > rbd cache size = 536870912 > rbd cache max dirty = 134217728 > rbd cache target dirty = 33554432 > rbd cache max dirty age = 5 > > [osd] > osd data = /var/lib/ceph/osd/cloud-$id > osd journal = /var/lib/ceph/osd/cloud-$id/journal > osd journal size = 10000 > osd mkfs type = xfs > osd mkfs options xfs = "-f -i size=2048" > osd mount options xfs = "rw,noatime,inode64,logbsize=250k" > > keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id > #increasing the number may increase the request processing rate > osd op threads = 24 > #The number of disk threads, which are used to perform background disk > intensive OSD operations such as scrubbing and snap trimming > osd disk threads =24 > #The number of active recovery requests per OSD at one time. More > requests will accelerate recovery, but the requests places an > increased load on the cluster. > osd recovery max active =1 > #writing direct to the journal. > #Allow use of libaio to do asynchronous writes > journal dio = true > journal aio = true > #Synchronization interval: > #The maximum/minimum interval in seconds for synchronizing the filestore. > filestore max sync interval = 100 > filestore min sync interval = 50 > #Defines the maximum number of in progress operations the file store > accepts before blocking on queuing new operations. > filestore queue max ops = 2000 > #The maximum number of bytes for an operation > filestore queue max bytes = 536870912 > #The maximum number of operations the filestore can commit. > filestore queue committing max ops = 2000 (default =500) > #The maximum number of bytes the filestore can commit. > filestore queue committing max bytes = 536870912 > #When you add or remove Ceph OSD Daemons to a cluster, the CRUSH > algorithm will want to rebalance the cluster by moving placement > groups to or from Ceph OSD Daemons to restore the balance. The process > of migrating placement groups and the objects they contain can reduce > the cluster's operational performance considerably. To maintain > operational performance, Ceph performs this migration with > 'backfilling', which allows Ceph to set backfill operations to a lower > priority than requests to read or write data. > osd max backfills = 1 > > > Tomorrow, I'm going to implement Ceph Cluster, > I have very little experience in managing Ceph. So, I hope someone > give me advices about above arguments and guide me how to best > optimize ceph cluster? > > Thank you so much! > --tuantaba > > > --------------010800020801020808020708 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Please help me!


On 07/20/2013 02:11 AM, Ta Ba Tuan wrote:
Hi everyone,

I have 3 nodes (running MON and MDS)
and  6 data nodes ( 84 OSDs )
Each data nodes has configuraions:
  - CPU: 24 processor * Core Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
  - RAM: 32GB
  - Disk: 14*4TB
(14disks *4TB *6 data nodes= 84 OSDs)

To optimize Ceph Cluster, I adjusted some kernel arguments (nr_request in queue and increated read throughput):

#Adjust nr_request in queue (staying in mem - default is 128)
    echo 1024 > /sys/block/sdb/queue/nr_requests
    echo noop > /sys/block/sda/queue/scheduler   (default= noop deadline [cfq])
#Increase read throughput  (default: 128)
    echo "512" > /sys/block/*/queue/read_ahead_kb

And, tuning Ceph configuraion options below:

[client]
 rbd cache = true
 rbd cache size = 536870912
 rbd cache max dirty = 134217728
 rbd cache target dirty = 33554432
 rbd cache max dirty age = 5
[osd]
    osd data = /var/lib/ceph/osd/cloud-$id
    osd journal = /var/lib/ceph/osd/cloud-$id/journal
    osd journal size = 10000
    osd mkfs type = xfs
    osd mkfs options xfs = "-f -i size=2048"
    osd mount options xfs = "rw,noatime,inode64,logbsize=250k"

    keyring = /var/lib/ceph/osd/cloud-$id/keyring.osd.$id
#increasing the number may increase the request processing rate
    osd op threads = 24
#The number of disk threads, which are used to perform background disk intensive OSD operations such as scrubbing and snap trimming
    osd disk threads =24
#The number of active recovery requests per OSD at one time. More requests will accelerate recovery, but the requests places an increased load on the cluster.
    osd recovery max active =1
#writing direct to the journal.
#Allow use of libaio to do asynchronous writes
    journal dio = true
    journal aio = true
#Synchronization interval:
#The maximum/minimum interval in seconds for synchronizing the filestore.
    filestore max sync interval = 100
    filestore min sync interval = 50
#Defines the maximum number of in progress operations the file store accepts before blocking on queuing new operations.
    filestore queue max ops = 2000
#The maximum number of bytes for an operation
    filestore queue max bytes = 536870912
#The maximum number of operations the filestore can commit.
    filestore queue committing max ops = 2000 (default =500)
#The maximum number of bytes the filestore can commit.
    filestore queue committing max bytes = 536870912
#When you add or remove Ceph OSD Daemons to a cluster, the CRUSH algorithm will want to rebalance the cluster by moving placement groups to or from Ceph OSD Daemons to restore the balance. The process of migrating placement groups and the objects they contain can reduce the cluster’s operational performance considerably. To maintain operational performance, Ceph performs this migration with ‘backfilling’, which allows Ceph to set backfill operations to a lower priority than requests to read or write data.
    osd max backfills = 1


Tomorrow, I'm going to implement Ceph Cluster,
I have very little experience in managing Ceph. So, I hope someone give me advices about above arguments and guide me how to best optimize ceph cluster?

Thank you so much!
--tuantaba




--------------010800020801020808020708-- --===============1242573343== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ ceph-users mailing list ceph-users-idqoXFIVOFJgJs9I8MT0rw@public.gmane.org http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com --===============1242573343==--