From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx106.postini.com [74.125.245.106]) by kanga.kvack.org (Postfix) with SMTP id 47E1A6B005D for ; Wed, 9 Jan 2013 21:51:09 -0500 (EST) Received: from /spool/local by e28smtp02.in.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 10 Jan 2013 08:19:31 +0530 Received: from d28relay01.in.ibm.com (d28relay01.in.ibm.com [9.184.220.58]) by d28dlp02.in.ibm.com (Postfix) with ESMTP id B7603394004D for ; Thu, 10 Jan 2013 08:21:03 +0530 (IST) Received: from d28av03.in.ibm.com (d28av03.in.ibm.com [9.184.220.65]) by d28relay01.in.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r0A2p1qQ46530746 for ; Thu, 10 Jan 2013 08:21:01 +0530 Received: from d28av03.in.ibm.com (loopback [127.0.0.1]) by d28av03.in.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r0A2p21u032167 for ; Thu, 10 Jan 2013 13:51:03 +1100 Date: Thu, 10 Jan 2013 10:50:51 +0800 From: Wanpeng Li Subject: Re: [PATCH] writeback: fix writeback cache thrashing Message-ID: <20130110025051.GA7844@hacker.(null)> Reply-To: Wanpeng Li References: <1356847190-7986-1-git-send-email-linkinjeon@gmail.com> <20130105031817.GA8650@localhost> <20130109151354.GA17353@quack.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20130109151354.GA17353@quack.suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: Jan Kara Cc: Namjae Jeon , Fengguang Wu , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Namjae Jeon , Vivek Trivedi , Dave Chinner , Simon Jeons On Wed, Jan 09, 2013 at 04:13:54PM +0100, Jan Kara wrote: >On Wed 09-01-13 17:26:36, Namjae Jeon wrote: > >> But in one normal scenario, the changes actually results in >> performance degradation. >> >> Results for a??dda?? thread on two devices: >> Before applying Patch: >> #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & >> #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & >> #> >> #> 2000+0 records in >> 2000+0 records out >> 2097152000 bytes (2.0GB) copied, 77.205276 seconds, 25.9MB/s -> USB >> HDD WRITE Speed >> >> [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 >> #> >> #> >> #> 800+0 records in >> 800+0 records out >> 838860800 bytes (800.0MB) copied, 154.528362 seconds, 5.2MB/s -> USB >> Flash WRITE Speed >> >> After applying patch: >> #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & >> dd if=/ >> #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & >> #> >> #> 2000+0 records in >> 2000+0 records out >> 2097152000 bytes (2.0GB) copied, 123.844770 seconds, 16.1MB/s ->USB >> HDD WRITE Speed >> 800+0 records in >> 800+0 records out >> 838860800 bytes (800.0MB) copied, 141.352945 seconds, 5.7MB/s -> USB >> Flash WRITE Speed >> >> [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 >> [1]+ Done dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 >> >> So, after applying our changes: >> 1) USB HDD Write speed dropped from 25.9 -> 16.1 MB/s >> 2) USB Flash Write speed increased marginally from 5.2 -> 5.7 MB/s >> >> Normally if we have a USB Flash and HDD plugged in system. And if we >> initiate the a??dda?? on both the devices. Once dirty memory is more than >> the background threshold, flushing starts for all BDI (The write-back >> for the devices will be kicked by the condition): >> If (global_page_state(NR_FILE_DIRTY) + >> global_page_state(NR_UNSTABLE_NFS) > background_thresh)) >> return true; >> As the slow device and the fast device always make sure that there is >> enough DIRTY data in memory to kick write-back. >> Since, USB Flash is slow, the DIRTY pages corresponding to this device >> is much higher, resulting in returning a??truea?? everytime from >> over_bground_thresh. So, even though HDD might have only few KB of >> dirty data, it is also flushed immediately. >> This frequent flushing of HDD data results in gradually increasing the >> bdi_dirty_limit() for HDD. > Interesting. Thanks for testing! So is this just a problem with initial >writeout fraction estimation. I.e. if you first let dd to USB HDD run for a >couple of seconds to ramp up its fraction and only then start writeout to >USB flash, is there still a problem with USB HDD throughput with the >changed over_bground_thresh() function? > >> But, when we introduce the change to control per BDI i.e., >> if (global_page_state(NR_FILE_DIRTY) + >> global_page_state(NR_UNSTABLE_NFS) > background_thresh && >> reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh) >> >> Now, in this case, when we consider the same scenario, writeback for >> HDD will only be kicked only if a??reclaimable * 2 + bdi_stat_error(bdi) >> * 2 > bdi_bground_thresha?? >> But this condition is not true a lot many number of times, so >> resulting in false. > I'm surprised it's not true so often... dd(1) should easily fill the But after merge the patch, dd can't easily fill the caches since shared writeback cache of HDD is small. >caches. But maybe we are oscilating between below-background-threshold >and at-dirty-limit situations rather quickly. Do you have recordings of >BDI_RECLAIMABLE and BDI_DIRTY from the problematic run? > >> This continuous failure to start write-back for HDD actually results >> in lowering the bdi_dirty_limit for HDD, in a way PAUSING the writer >> thread for HDD. >> This is actually resulting in less number of WRITE operations per >> second for HDD. As, the a??dda?? on USB HDD will be put to long sleep(MAX >> PAUSE) in balance_dirty_pages. >> >> While for USB Flash, its bdi_dirty_limit is kept on increasing as it >> is getting more chance to flush dirty data in over_bground_thresh. As, >> bdi_reclaimable > bdi_dirty_limit is true. So, resulting more number >> of WRITE operation per second for USB Flash. >> From these observations, we feel that these changes might not be >> needed. Please let us know in case we are missing on any point here, >> we can further check more on this. > Well, at least we know changing the condition has unexpected side >effects. I'd like to understand those before discarding the idea - because >in your setup flusher thread must end up writing rather small amount of >pages in each run when it's running continuously and that's not too good >either... > > Honza >-- >Jan Kara >SUSE Labs, CR > >-- >To unsubscribe, send a message with 'unsubscribe linux-mm' in >the body to majordomo@kvack.org. For more info on Linux MM, >see: http://www.linux-mm.org/ . >Don't email: email@kvack.org -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org