From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jan Kara Subject: Re: [PATCH] writeback: fix writeback cache thrashing Date: Wed, 9 Jan 2013 16:13:54 +0100 Message-ID: <20130109151354.GA17353@quack.suse.cz> References: <1356847190-7986-1-git-send-email-linkinjeon@gmail.com> <20130105031817.GA8650@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Fengguang Wu , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, liwanp@linux.vnet.ibm.com, Namjae Jeon , Vivek Trivedi , Jan Kara , Dave Chinner , Simon Jeons To: Namjae Jeon Return-path: Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: linux-fsdevel.vger.kernel.org On Wed 09-01-13 17:26:36, Namjae Jeon wrote: > But in one normal scenario, the changes actually results in > performance degradation. >=20 > Results for =E2=80=98dd=E2=80=99 thread on two devices: > Before applying Patch: > #> dd if=3D/dev/zero of=3D/mnt/sdb2/file1 bs=3D1048576 count=3D800 & > #> dd if=3D/dev/zero of=3D/mnt/sda6/file2 bs=3D1048576 count=3D2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 77.205276 seconds, 25.9MB/s -> USB > HDD WRITE Speed >=20 > [2]+ Done dd if=3D/dev/zero of=3D/mnt/sda6/file2 bs=3D1048576 count=3D2= 000 > #> > #> > #> 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 154.528362 seconds, 5.2MB/s -> USB > Flash WRITE Speed >=20 > After applying patch: > #> dd if=3D/dev/zero of=3D/mnt/sdb2/file1 bs=3D1048576 count=3D800 & > dd if=3D/ > #> dd if=3D/dev/zero of=3D/mnt/sda6/file2 bs=3D1048576 count=3D2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 123.844770 seconds, 16.1MB/s ->USB > HDD WRITE Speed > 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 141.352945 seconds, 5.7MB/s -> USB > Flash WRITE Speed >=20 > [2]+ Done dd if=3D/dev/zero of=3D/mnt/sda6/file2 bs=3D1048576 count=3D2= 000 > [1]+ Done dd if=3D/dev/zero of=3D/mnt/sdb2/file1 bs=3D1048576 count=3D8= 00 >=20 > So, after applying our changes: > 1) USB HDD Write speed dropped from 25.9 -> 16.1 MB/s > 2) USB Flash Write speed increased marginally from 5.2 -> 5.7 MB/s >=20 > Normally if we have a USB Flash and HDD plugged in system. And if we > initiate the =E2=80=98dd=E2=80=99 on both the devices. Once dirty memor= y is more than > the background threshold, flushing starts for all BDI (The write-back > for the devices will be kicked by the condition): > If (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh)) > return true; > As the slow device and the fast device always make sure that there is > enough DIRTY data in memory to kick write-back. > Since, USB Flash is slow, the DIRTY pages corresponding to this device > is much higher, resulting in returning =E2=80=98true=E2=80=99 everytime= from > over_bground_thresh. So, even though HDD might have only few KB of > dirty data, it is also flushed immediately. > This frequent flushing of HDD data results in gradually increasing the > bdi_dirty_limit() for HDD. Interesting. Thanks for testing! So is this just a problem with initial writeout fraction estimation. I.e. if you first let dd to USB HDD run for= a couple of seconds to ramp up its fraction and only then start writeout to USB flash, is there still a problem with USB HDD throughput with the changed over_bground_thresh() function? > But, when we introduce the change to control per BDI i.e., > if (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh && > reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh= ) >=20 > Now, in this case, when we consider the same scenario, writeback for > HDD will only be kicked only if =E2=80=98reclaimable * 2 + bdi_stat_err= or(bdi) > * 2 > bdi_bground_thresh=E2=80=99 > But this condition is not true a lot many number of times, so > resulting in false. I'm surprised it's not true so often... dd(1) should easily fill the caches. But maybe we are oscilating between below-background-threshold and at-dirty-limit situations rather quickly. Do you have recordings of BDI_RECLAIMABLE and BDI_DIRTY from the problematic run? > This continuous failure to start write-back for HDD actually results > in lowering the bdi_dirty_limit for HDD, in a way PAUSING the writer > thread for HDD. > This is actually resulting in less number of WRITE operations per > second for HDD. As, the =E2=80=98dd=E2=80=99 on USB HDD will be put to = long sleep(MAX > PAUSE) in balance_dirty_pages. >=20 > While for USB Flash, its bdi_dirty_limit is kept on increasing as it > is getting more chance to flush dirty data in over_bground_thresh. As, > bdi_reclaimable > bdi_dirty_limit is true. So, resulting more number > of WRITE operation per second for USB Flash. > From these observations, we feel that these changes might not be > needed. Please let us know in case we are missing on any point here, > we can further check more on this. Well, at least we know changing the condition has unexpected side effects. I'd like to understand those before discarding the idea - becaus= e in your setup flusher thread must end up writing rather small amount of pages in each run when it's running continuously and that's not too good either... Honza --=20 Jan Kara SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx130.postini.com [74.125.245.130]) by kanga.kvack.org (Postfix) with SMTP id 193A66B005A for ; Wed, 9 Jan 2013 10:13:59 -0500 (EST) Date: Wed, 9 Jan 2013 16:13:54 +0100 From: Jan Kara Subject: Re: [PATCH] writeback: fix writeback cache thrashing Message-ID: <20130109151354.GA17353@quack.suse.cz> References: <1356847190-7986-1-git-send-email-linkinjeon@gmail.com> <20130105031817.GA8650@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Namjae Jeon Cc: Fengguang Wu , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, liwanp@linux.vnet.ibm.com, Namjae Jeon , Vivek Trivedi , Jan Kara , Dave Chinner , Simon Jeons On Wed 09-01-13 17:26:36, Namjae Jeon wrote: > But in one normal scenario, the changes actually results in > performance degradation. > > Results for a??dda?? thread on two devices: > Before applying Patch: > #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & > #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 77.205276 seconds, 25.9MB/s -> USB > HDD WRITE Speed > > [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 > #> > #> > #> 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 154.528362 seconds, 5.2MB/s -> USB > Flash WRITE Speed > > After applying patch: > #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & > dd if=/ > #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 123.844770 seconds, 16.1MB/s ->USB > HDD WRITE Speed > 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 141.352945 seconds, 5.7MB/s -> USB > Flash WRITE Speed > > [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 > [1]+ Done dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 > > So, after applying our changes: > 1) USB HDD Write speed dropped from 25.9 -> 16.1 MB/s > 2) USB Flash Write speed increased marginally from 5.2 -> 5.7 MB/s > > Normally if we have a USB Flash and HDD plugged in system. And if we > initiate the a??dda?? on both the devices. Once dirty memory is more than > the background threshold, flushing starts for all BDI (The write-back > for the devices will be kicked by the condition): > If (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh)) > return true; > As the slow device and the fast device always make sure that there is > enough DIRTY data in memory to kick write-back. > Since, USB Flash is slow, the DIRTY pages corresponding to this device > is much higher, resulting in returning a??truea?? everytime from > over_bground_thresh. So, even though HDD might have only few KB of > dirty data, it is also flushed immediately. > This frequent flushing of HDD data results in gradually increasing the > bdi_dirty_limit() for HDD. Interesting. Thanks for testing! So is this just a problem with initial writeout fraction estimation. I.e. if you first let dd to USB HDD run for a couple of seconds to ramp up its fraction and only then start writeout to USB flash, is there still a problem with USB HDD throughput with the changed over_bground_thresh() function? > But, when we introduce the change to control per BDI i.e., > if (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh && > reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh) > > Now, in this case, when we consider the same scenario, writeback for > HDD will only be kicked only if a??reclaimable * 2 + bdi_stat_error(bdi) > * 2 > bdi_bground_thresha?? > But this condition is not true a lot many number of times, so > resulting in false. I'm surprised it's not true so often... dd(1) should easily fill the caches. But maybe we are oscilating between below-background-threshold and at-dirty-limit situations rather quickly. Do you have recordings of BDI_RECLAIMABLE and BDI_DIRTY from the problematic run? > This continuous failure to start write-back for HDD actually results > in lowering the bdi_dirty_limit for HDD, in a way PAUSING the writer > thread for HDD. > This is actually resulting in less number of WRITE operations per > second for HDD. As, the a??dda?? on USB HDD will be put to long sleep(MAX > PAUSE) in balance_dirty_pages. > > While for USB Flash, its bdi_dirty_limit is kept on increasing as it > is getting more chance to flush dirty data in over_bground_thresh. As, > bdi_reclaimable > bdi_dirty_limit is true. So, resulting more number > of WRITE operation per second for USB Flash. > From these observations, we feel that these changes might not be > needed. Please let us know in case we are missing on any point here, > we can further check more on this. Well, at least we know changing the condition has unexpected side effects. I'd like to understand those before discarding the idea - because in your setup flusher thread must end up writing rather small amount of pages in each run when it's running continuously and that's not too good either... Honza -- Jan Kara SUSE Labs, CR -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758013Ab3AIPOA (ORCPT ); Wed, 9 Jan 2013 10:14:00 -0500 Received: from cantor2.suse.de ([195.135.220.15]:51850 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757922Ab3AIPN6 (ORCPT ); Wed, 9 Jan 2013 10:13:58 -0500 Date: Wed, 9 Jan 2013 16:13:54 +0100 From: Jan Kara To: Namjae Jeon Cc: Fengguang Wu , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, liwanp@linux.vnet.ibm.com, Namjae Jeon , Vivek Trivedi , Jan Kara , Dave Chinner , Simon Jeons Subject: Re: [PATCH] writeback: fix writeback cache thrashing Message-ID: <20130109151354.GA17353@quack.suse.cz> References: <1356847190-7986-1-git-send-email-linkinjeon@gmail.com> <20130105031817.GA8650@localhost> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed 09-01-13 17:26:36, Namjae Jeon wrote: > But in one normal scenario, the changes actually results in > performance degradation. > > Results for ‘dd’ thread on two devices: > Before applying Patch: > #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & > #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 77.205276 seconds, 25.9MB/s -> USB > HDD WRITE Speed > > [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 > #> > #> > #> 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 154.528362 seconds, 5.2MB/s -> USB > Flash WRITE Speed > > After applying patch: > #> dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 & > dd if=/ > #> dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 & > #> > #> 2000+0 records in > 2000+0 records out > 2097152000 bytes (2.0GB) copied, 123.844770 seconds, 16.1MB/s ->USB > HDD WRITE Speed > 800+0 records in > 800+0 records out > 838860800 bytes (800.0MB) copied, 141.352945 seconds, 5.7MB/s -> USB > Flash WRITE Speed > > [2]+ Done dd if=/dev/zero of=/mnt/sda6/file2 bs=1048576 count=2000 > [1]+ Done dd if=/dev/zero of=/mnt/sdb2/file1 bs=1048576 count=800 > > So, after applying our changes: > 1) USB HDD Write speed dropped from 25.9 -> 16.1 MB/s > 2) USB Flash Write speed increased marginally from 5.2 -> 5.7 MB/s > > Normally if we have a USB Flash and HDD plugged in system. And if we > initiate the ‘dd’ on both the devices. Once dirty memory is more than > the background threshold, flushing starts for all BDI (The write-back > for the devices will be kicked by the condition): > If (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh)) > return true; > As the slow device and the fast device always make sure that there is > enough DIRTY data in memory to kick write-back. > Since, USB Flash is slow, the DIRTY pages corresponding to this device > is much higher, resulting in returning ‘true’ everytime from > over_bground_thresh. So, even though HDD might have only few KB of > dirty data, it is also flushed immediately. > This frequent flushing of HDD data results in gradually increasing the > bdi_dirty_limit() for HDD. Interesting. Thanks for testing! So is this just a problem with initial writeout fraction estimation. I.e. if you first let dd to USB HDD run for a couple of seconds to ramp up its fraction and only then start writeout to USB flash, is there still a problem with USB HDD throughput with the changed over_bground_thresh() function? > But, when we introduce the change to control per BDI i.e., > if (global_page_state(NR_FILE_DIRTY) + > global_page_state(NR_UNSTABLE_NFS) > background_thresh && > reclaimable * 2 + bdi_stat_error(bdi) * 2 > bdi_bground_thresh) > > Now, in this case, when we consider the same scenario, writeback for > HDD will only be kicked only if ‘reclaimable * 2 + bdi_stat_error(bdi) > * 2 > bdi_bground_thresh’ > But this condition is not true a lot many number of times, so > resulting in false. I'm surprised it's not true so often... dd(1) should easily fill the caches. But maybe we are oscilating between below-background-threshold and at-dirty-limit situations rather quickly. Do you have recordings of BDI_RECLAIMABLE and BDI_DIRTY from the problematic run? > This continuous failure to start write-back for HDD actually results > in lowering the bdi_dirty_limit for HDD, in a way PAUSING the writer > thread for HDD. > This is actually resulting in less number of WRITE operations per > second for HDD. As, the ‘dd’ on USB HDD will be put to long sleep(MAX > PAUSE) in balance_dirty_pages. > > While for USB Flash, its bdi_dirty_limit is kept on increasing as it > is getting more chance to flush dirty data in over_bground_thresh. As, > bdi_reclaimable > bdi_dirty_limit is true. So, resulting more number > of WRITE operation per second for USB Flash. > From these observations, we feel that these changes might not be > needed. Please let us know in case we are missing on any point here, > we can further check more on this. Well, at least we know changing the condition has unexpected side effects. I'd like to understand those before discarding the idea - because in your setup flusher thread must end up writing rather small amount of pages in each run when it's running continuously and that's not too good either... Honza -- Jan Kara SUSE Labs, CR