From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zdenek Kabelac Subject: Re: dm-cache performance behaviour Date: Tue, 5 Apr 2016 10:36:12 +0200 Message-ID: <570378FC.3030701@redhat.com> References: <20160405071253.GB27444@suselix.suse.de> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: Received: from mx1.redhat.com ([209.132.183.28]:53610 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751985AbcDEIgP (ORCPT ); Tue, 5 Apr 2016 04:36:15 -0400 In-Reply-To: <20160405071253.GB27444@suselix.suse.de> Sender: linux-bcache-owner@vger.kernel.org List-Id: linux-bcache@vger.kernel.org To: Andreas Herrmann Cc: dm-devel@redhat.com, linux-bcache@vger.kernel.org, linux-block@vger.kernel.org Dne 5.4.2016 v 09:12 Andreas Herrmann napsal(a): > Hi, > > I've recently looked at performance behaviour of dm-cache and bcache. > I've repeatedly observed very low performance with dm-cache in > different tests. (Similar tests with bcache showed no such oddities.) > > To rule out user errors that might have caused this, I shortly describe > what I've done and observed. > > - tested kernel version: 4.5.0 > > - backing device: 1.5 TB spinning drive > > - caching device: 128 GB SSD (used for metadata and cache and size > of metadata part calculated based on > https://www.redhat.com/archives/dm-devel/2012-December/msg00046.html) > > - my test procedure consisted of a sequence of tests performing fio > runs with different data sets, fio randread performance (bandwidth > and IOPS) were compared, fio was invoked using something like > > fio --directory=/cached-device --rw=randread --name=fio-1 \ > --size=50G --group_reporting --ioengine=libaio \ > --direct=1 --iodepth=1 --runtime=40 --numjobs=1 > > I've iterated over 10 runs for each of numjobs=1,2,3 and varied the > name parameter to operate with different data sets. > > This procedure implied that with 3 jobs the underlying data set for > the test consisted of 3 files with 50G each which exceeds the size > of the caching device. > > - Between some tests I've tried to empty the cache. For dm-cache I did > this by unmounting the "compound" cache device, switching to cleaner > target, zeroing metadata part of the caching device, recreating > caching device and finally recreating the compound cache device > (during this procedure I kept the backing device unmodified). > > I used dmsetup status to check for success of this operation > (checking for #used_cache_blocks). > If there is an easier way to do this please let me know -- If it's > documented I've missed it. > > - dm-cache parameters: > * cache_mode: writeback > * block size: 512 sectors > * migration_threshold 2048 (default) > > I've observed two oddities: > > (1) Only fio tests with the first data set created (and thus > initially occupying the cache) showed decent performance > results. Subsequent fio tests with another data set showed poor > performance. I think this indicates that SMQ policy does not > properly promote/demote data to/from caching device in my tests. > > (2) I've seen results where performance was actually below "native" > (w/o caching) performance of the backing device. I think that this > should not happen. If a data access falls back to the backing device > due to a cache miss I would have expected to see almost the > performance of the backing device. Maybe this points to a > performance issue in SMQ -- spending too much time in policy code > before falling back to the backing device. > > I've tried to figure out what actually happened in SMQ code in these > cases - but eventually dismissed this. Next I want to check whether > there might be a flaw in my test setup/dm-cache configuration. Hi The dm-cache SMQ/MQ is a 'slow moving' hot-spot cache. So before the block is 'promoted' to the cache - there needs to be a reason for it - and it's not a plain single read. So if the other cache promotes the block to the cache with a single block access you may observe different performance. dm-cache is not targeted for 'quick' promoting of read blocks into a cache - rather 'slow' moving of often used blocks. Unsure how that fits your testing environment and what you try to actually test? Regards PS: 256K dm-cache blocks size is quite large - it really depends upon workload - min supported size is 32K - lvm2 defaults to 64K... Zdenek