From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stefan Priebe - Profihost AG Subject: Re: [Share]Performance tunning on Ceph FileStore with SSD backend Date: Tue, 27 May 2014 08:05:32 +0200 Message-ID: <53842B2C.10807@profihost.ag> References: <5383A410.6010002@profihost.ag> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Return-path: Received: from mail-ph.de-nserver.de ([85.158.179.214]:32895 "EHLO mail-ph.de-nserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751605AbaE0GFh (ORCPT ); Tue, 27 May 2014 02:05:37 -0400 In-Reply-To: Sender: ceph-devel-owner@vger.kernel.org List-ID: To: Haomai Wang Cc: "ceph-devel@vger.kernel.org" Am 27.05.2014 06:42, schrieb Haomai Wang: > On Tue, May 27, 2014 at 4:29 AM, Stefan Priebe wrote: >> Hi Haomai, >> >> regarding the FDCache problems you're seeing. Isn't this branch interesting >> for you? Have you ever tested it? >> >> http://lists.ceph.com/pipermail/ceph-commit-ceph.com/2014-January/007399.html >> > > Yes, I noticed it. But my main job is improving performance on 0.67.5 > version. Before this branch, my improvement on this problem is avoid > lfn_find in omap* methods with FileStore > class.(https://www.mail-archive.com/ceph-devel@vger.kernel.org/msg18505.html) Avoids mean just remove them? Are they not needed? Do you have a branch for this? >> Greets, >> Stefan >> >> Am 09.04.2014 12:05, schrieb Haomai Wang: >> >>> Hi all, >>> >>> I would like to share some ideas about how to improve performance on >>> ceph with SSD. Not much preciseness. >>> >>> Our ssd is 500GB and each OSD own a SSD(journal is on the same SSD). >>> ceph version is 0.67.5(Dumping) >>> >>> At first, we find three bottleneck on filestore: >>> 1. fdcache_lock(changed in Firely release) >>> 2. lfn_find in omap_* methods >>> 3. DBObjectMap header >>> >>> According to my understanding and the docs in >>> >>> ObjectStore.h(https://github.com/ceph/ceph/blob/master/src/os/ObjectStore.h), >>> I simply remove lfn_find in omap_* and fdcache_lock. I'm not fully >>> sure the correctness of this change, but it works well still now. >>> >>> DBObjectMap header patch is on the pull request queue and may be >>> merged in the next feature merge window. >>> >>> With things above done, we get much performance improvement in disk >>> util and benchmark results(3x-4x). >>> >>> Next, we find fdcache size become the main bottleneck. For example, if >>> hot data range is 100GB, we need 25000(100GB/4MB) fd to cache. If hot >>> data range is 1TB, we need 250000(1000GB/4MB) fd to cache. With >>> increase "filestore_fd_cache_size", the cost of lookup(FDCache) and >>> cache miss is expensive and can't be afford. The implementation of >>> FDCache isn't O(1). So we only can get high performance on fdcache hit >>> range(maybe 100GB with 10240 fdcache size) and more data exceed the >>> size of fdcaceh will be disaster. If you want to cache more fd(102400 >>> fdcache size), the implementation of FDCache will bring on extra CPU >>> cost(can't be ignore) for each op. >>> >>> Because of the capacity of SSD(several hundreds GB), we try to >>> increase the size of rbd object(16MB) so less fd cache is needed. As >>> for FDCache implementation, we simply discard SimpleLRU but introduce >>> RandomCache. Now we can set much larger fdcache size(near cache all >>> fd) with little overload. >>> >>> With these, we achieve 3x-4x performance improvements on filestore with >>> SSD. >>> >>> Maybe it exists something I missed or something wrong, hope can >>> correct me. I hope it can help to improve FileStore on SSD and push >>> into master branch. >>> >> > > >