RFC: dirty_ratio back to 40%

RFC: dirty_ratio back to 40%

[Larry Woodman]

We've seen multiple performance regressions linked to the lower(20%)
dirty_ratio.  When performing enough IO to overwhelm the background  
flush daemons the percent of dirty pagecache memory quickly climbs 
to the new/lower dirty_ratio value of 20%.  At that point all writing 
processes are forced to stop and write dirty pagecache pages back to disk.  
This causes performance regressions in several benchmarks as well as causing
a noticeable overall sluggishness.  We all know that the dirty_ratio is
an integrity vs performance trade-off but the file system journaling
will cover any devastating effects in the event of a system crash.

Increasing the dirty_ratio to 40% will regain the performance loss seen
in several benchmarks.  Whats everyone think about this???





------------------------------------------------------------------------

diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index ef27e73..645a462 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
 /*
  * The generator of dirty data starts writeback at this percentage
  */
-int vm_dirty_ratio = 20;
+int vm_dirty_ratio = 40;
 
 /*
  * vm_dirty_bytes starts at 0 (disabled) so that it is a function of

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Heinz Diehl]

On 20.05.2010, Larry Woodman wrote: 

> Increasing the dirty_ratio to 40% will regain the performance loss seen
> in several benchmarks.  Whats everyone think about this???

These are tuneable via sysctl. What I have in my /etc/sysctl.conf is

 vm.dirty_ratio = 4
 vm.dirty_background_ratio = 2
 
This writes back the data more often and frequently, thus preventing the
system from long stalls. 

Works at least for me. AMD Quadcore, 8 GB RAM.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Richard Kennedy]

On 20/05/10 13:29, Heinz Diehl wrote:
> On 20.05.2010, Larry Woodman wrote: 
> lwoodman@redhat.com
>> Increasing the dirty_ratio to 40% will regain the performance loss seen
>> in several benchmarks.  Whats everyone think about this???
> 
> These are tuneable via sysctl. What I have in my /etc/sysctl.conf is
> 
>  vm.dirty_ratio = 4
>  vm.dirty_background_ratio = 2
>  
> This writes back the data more often and frequently, thus preventing the
> system from long stalls. 
> 
> Works at least for me. AMD Quadcore, 8 GB RAM.
> 
get_dirty_limits uses a minimum vm_dirty_ratio of 5, so you can't set it
lower than that (unless you use vm_dirty_bytes).
But it's interesting that you find lowering the dirty_ratio helpful. Do
you have any benchmark results you can share?
regards
Richard

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[KOSAKI Motohiro]

Hi

CC to Nick and Jan

> We've seen multiple performance regressions linked to the lower(20%)
> dirty_ratio.  When performing enough IO to overwhelm the background  
> flush daemons the percent of dirty pagecache memory quickly climbs 
> to the new/lower dirty_ratio value of 20%.  At that point all writing 
> processes are forced to stop and write dirty pagecache pages back to disk.  
> This causes performance regressions in several benchmarks as well as causing
> a noticeable overall sluggishness.  We all know that the dirty_ratio is
> an integrity vs performance trade-off but the file system journaling
> will cover any devastating effects in the event of a system crash.
> 
> Increasing the dirty_ratio to 40% will regain the performance loss seen
> in several benchmarks.  Whats everyone think about this???

In past, Jan Kara also claim the exactly same thing.

	Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
	Date: Wed, 24 Feb 2010 15:34:42 +0100

	> (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
	> with old kernels because customers running e.g. LDAP (using BerkelyDB
	> heavily) were complaining about performance problems.

So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
same distro specific patch. because we can easily imazine other users will face the same
issue in the future.

	Acked-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>

Nick, Jan, if the above is too old and your distro have been dropped the patch, please
correct me.

My motivation is, distro specific patches should keep minimum as far as possible.
It exactly help I and other MM developers handle MM bug report.

Thanks.


> ------------------------------------------------------------------------
> 
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index ef27e73..645a462 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
>  /*
>   * The generator of dirty data starts writeback at this percentage
>   */
> -int vm_dirty_ratio = 20;
> +int vm_dirty_ratio = 40;
>  
>  /*
>   * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Zan Lynx]

On 5/20/10 5:48 PM, KOSAKI Motohiro wrote:
> Hi
>
> CC to Nick and Jan
>
>> We've seen multiple performance regressions linked to the lower(20%)
>> dirty_ratio.  When performing enough IO to overwhelm the background
>> flush daemons the percent of dirty pagecache memory quickly climbs
>> to the new/lower dirty_ratio value of 20%.  At that point all writing
>> processes are forced to stop and write dirty pagecache pages back to disk.
>> This causes performance regressions in several benchmarks as well as causing
>> a noticeable overall sluggishness.  We all know that the dirty_ratio is
>> an integrity vs performance trade-off but the file system journaling
>> will cover any devastating effects in the event of a system crash.
>>
>> Increasing the dirty_ratio to 40% will regain the performance loss seen
>> in several benchmarks.  Whats everyone think about this???
>
> In past, Jan Kara also claim the exactly same thing.
>
> 	Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
> 	Date: Wed, 24 Feb 2010 15:34:42 +0100
>
> 	>  (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
> 	>  with old kernels because customers running e.g. LDAP (using BerkelyDB
> 	>  heavily) were complaining about performance problems.
>
> So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> same distro specific patch. because we can easily imazine other users will face the same
> issue in the future.

On desktop systems the low dirty limits help maintain interactive feel. 
Users expect applications that are saving data to be slow. They do not 
like it when every application in the system randomly comes to a halt 
because of one program stuffing data up to the dirty limit.

The cause and effect for the system slowdown is clear when the dirty 
limit is low. "I saved data and now the system is slow until it is 
done." When the dirty page ratio is very high, the cause and effect is 
disconnected. "I was just web surfing and the system came to a halt."

I think we should expect server admins to do more tuning than desktop 
users, so the default limits should stay low in my opinion.

-- 
Zan Lynx
zlynx@acm.org

"Knowledge is Power.  Power Corrupts.  Study Hard.  Be Evil."

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[KOSAKI Motohiro]

> > So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> > same distro specific patch. because we can easily imazine other users will face the same
> > issue in the future.
> 
> On desktop systems the low dirty limits help maintain interactive feel. 
> Users expect applications that are saving data to be slow. They do not 
> like it when every application in the system randomly comes to a halt 
> because of one program stuffing data up to the dirty limit.

really?
Do you mean our per-task dirty limit wouldn't works?

If so, I think we need fix it. IOW sane per-task dirty limitation seems independent issue 
from per-system dirty limit.


> The cause and effect for the system slowdown is clear when the dirty 
> limit is low. "I saved data and now the system is slow until it is 
> done." When the dirty page ratio is very high, the cause and effect is 
> disconnected. "I was just web surfing and the system came to a halt."
> 
> I think we should expect server admins to do more tuning than desktop 
> users, so the default limits should stay low in my opinion.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Jan Kara]

On Fri 21-05-10 10:11:59, KOSAKI Motohiro wrote:
> > > So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> > > same distro specific patch. because we can easily imazine other users will face the same
> > > issue in the future.
> > 
> > On desktop systems the low dirty limits help maintain interactive feel. 
> > Users expect applications that are saving data to be slow. They do not 
> > like it when every application in the system randomly comes to a halt 
> > because of one program stuffing data up to the dirty limit.
> 
> really?
> Do you mean our per-task dirty limit wouldn't works?
> 
> If so, I think we need fix it. IOW sane per-task dirty limitation seems
> independent issue from per-system dirty limit.
  Well, I don't know about any per-task dirty limits. What function
implements them? Any application that dirties a single page can be caught
and forced to call balance_dirty_pages() and do writeback.
  But generally what we observe on a desktop with lots of dirty memory is
that application needs to allocate memory (either private one or for page
cache) and that triggers direct reclaim so the allocation takes a long
time to finish and thus the application is sluggish...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Ric Wheeler]

On 05/20/2010 08:48 PM, Zan Lynx wrote:
> On 5/20/10 5:48 PM, KOSAKI Motohiro wrote:
>> Hi
>>
>> CC to Nick and Jan
>>
>>> We've seen multiple performance regressions linked to the lower(20%)
>>> dirty_ratio.  When performing enough IO to overwhelm the background
>>> flush daemons the percent of dirty pagecache memory quickly climbs
>>> to the new/lower dirty_ratio value of 20%.  At that point all writing
>>> processes are forced to stop and write dirty pagecache pages back to 
>>> disk.
>>> This causes performance regressions in several benchmarks as well as 
>>> causing
>>> a noticeable overall sluggishness.  We all know that the dirty_ratio is
>>> an integrity vs performance trade-off but the file system journaling
>>> will cover any devastating effects in the event of a system crash.
>>>
>>> Increasing the dirty_ratio to 40% will regain the performance loss seen
>>> in several benchmarks.  Whats everyone think about this???
>>
>> In past, Jan Kara also claim the exactly same thing.
>>
>>     Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
>>     Date: Wed, 24 Feb 2010 15:34:42 +0100
>>
>> >  (*) We ended up increasing dirty_limit in SLES 11 to 40% as it 
>> used to be
>> >  with old kernels because customers running e.g. LDAP (using BerkelyDB
>> >  heavily) were complaining about performance problems.
>>
>> So, I'd prefer to restore the default rather than both Redhat and 
>> SUSE apply exactly
>> same distro specific patch. because we can easily imazine other users 
>> will face the same
>> issue in the future.
>
> On desktop systems the low dirty limits help maintain interactive 
> feel. Users expect applications that are saving data to be slow. They 
> do not like it when every application in the system randomly comes to 
> a halt because of one program stuffing data up to the dirty limit.
>
> The cause and effect for the system slowdown is clear when the dirty 
> limit is low. "I saved data and now the system is slow until it is 
> done." When the dirty page ratio is very high, the cause and effect is 
> disconnected. "I was just web surfing and the system came to a halt."
>
> I think we should expect server admins to do more tuning than desktop 
> users, so the default limits should stay low in my opinion.
>

Have you done any performance testing that shows this?

A laptop the smaller default would spin up drives more often and greatly 
decrease your battery life.

Note that both SLES and RHEL default away from the upstream default.

Ric

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Jan Kara]

  Hi,

On Fri 21-05-10 08:48:57, KOSAKI Motohiro wrote:
> CC to Nick and Jan
  Thanks.

> > We've seen multiple performance regressions linked to the lower(20%)
> > dirty_ratio.  When performing enough IO to overwhelm the background  
> > flush daemons the percent of dirty pagecache memory quickly climbs 
> > to the new/lower dirty_ratio value of 20%.  At that point all writing 
> > processes are forced to stop and write dirty pagecache pages back to disk.  
> > This causes performance regressions in several benchmarks as well as causing
> > a noticeable overall sluggishness.  We all know that the dirty_ratio is
> > an integrity vs performance trade-off but the file system journaling
> > will cover any devastating effects in the event of a system crash.
> > 
> > Increasing the dirty_ratio to 40% will regain the performance loss seen
> > in several benchmarks.  Whats everyone think about this???
> 
> In past, Jan Kara also claim the exactly same thing.
> 
> 	Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
> 	Date: Wed, 24 Feb 2010 15:34:42 +0100
> 
> 	> (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
> 	> with old kernels because customers running e.g. LDAP (using BerkelyDB
> 	> heavily) were complaining about performance problems.
> 
> So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> same distro specific patch. because we can easily imazine other users will face the same
> issue in the future.
> 
> 	Acked-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
> 
> Nick, Jan, if the above is too old and your distro have been dropped the patch, please
> correct me.
  No, SLE11 SP1 still has a patch that increases dirty_ratio to 40. But on
the other hand I agree with Zan that for desktop, 40% of memory for dirty
data is a lot these days and takes a long time to write out (it could
easily be 30s - 1m). On a desktop the memory is much better used as
a read-only pagecache or for memory hungry apps like Firefox or Acrobat
Reader.  So I believe for a desktop the current setting (20) is a better
choice. So until we find a way how to dynamically size the dirty limit, we
have to decide whether we want to have a default setting for a server or
for a desktop... Personally, I don't care very much and I feel my time
would be better spent thinking about dynamic limit sizing rather than
arguing what is better default ;).

									Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[David Miller]

From: Larry Woodman <lwoodman@redhat.com>
Date: Thu, 20 May 2010 07:20:42 -0400

> Increasing the dirty_ratio to 40% will regain the performance loss
> seen in several benchmarks.  Whats everyone think about this???

I've been making this change via sysctl on every single system I have,
and have been doing so for quite some time.

When doing a lot of GIT operations to a non-SSD disk the kernel simply
can't submit the writes early enough to prevent everything getting
backlogged, and then processes pile up being forced to sleep on I/O
for several seconds at a time.

I therefore totally support making this the default, but I know some
people will be against it :-)

Acked-by: David S. Miller <davem@davemloft.net>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Christoph Hellwig]

Did this patch get merged somewhere?

On Thu, May 20, 2010 at 07:20:42AM -0400, Larry Woodman wrote:
> We've seen multiple performance regressions linked to the lower(20%)
> dirty_ratio.  When performing enough IO to overwhelm the background
> flush daemons the percent of dirty pagecache memory quickly climbs
> to the new/lower dirty_ratio value of 20%.  At that point all
> writing processes are forced to stop and write dirty pagecache pages
> back to disk.  This causes performance regressions in several
> benchmarks as well as causing
> a noticeable overall sluggishness.  We all know that the dirty_ratio is
> an integrity vs performance trade-off but the file system journaling
> will cover any devastating effects in the event of a system crash.
> 
> Increasing the dirty_ratio to 40% will regain the performance loss seen
> in several benchmarks.  Whats everyone think about this???
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index ef27e73..645a462 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
> /*
>  * The generator of dirty data starts writeback at this percentage
>  */
> -int vm_dirty_ratio = 20;
> +int vm_dirty_ratio = 40;
> 
> /*
>  * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
---end quoted text---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: RFC: dirty_ratio back to 40%

[Larry Woodman]

On Tue, 2010-06-08 at 14:49 -0400, Christoph Hellwig wrote:
> Did this patch get merged somewhere?

I dont think it ever did, about 1/2 of responses were for it and the
other 1/2 against it.

Larry

> 
> On Thu, May 20, 2010 at 07:20:42AM -0400, Larry Woodman wrote:
> > We've seen multiple performance regressions linked to the lower(20%)
> > dirty_ratio.  When performing enough IO to overwhelm the background
> > flush daemons the percent of dirty pagecache memory quickly climbs
> > to the new/lower dirty_ratio value of 20%.  At that point all
> > writing processes are forced to stop and write dirty pagecache pages
> > back to disk.  This causes performance regressions in several
> > benchmarks as well as causing
> > a noticeable overall sluggishness.  We all know that the dirty_ratio is
> > an integrity vs performance trade-off but the file system journaling
> > will cover any devastating effects in the event of a system crash.
> > 
> > Increasing the dirty_ratio to 40% will regain the performance loss seen
> > in several benchmarks.  Whats everyone think about this???
> > 
> > 
> > 
> > 
> > 
> > ------------------------------------------------------------------------
> > 
> > diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> > index ef27e73..645a462 100644
> > --- a/mm/page-writeback.c
> > +++ b/mm/page-writeback.c
> > @@ -78,7 +78,7 @@ int vm_highmem_is_dirtyable;
> > /*
> >  * The generator of dirty data starts writeback at this percentage
> >  */
> > -int vm_dirty_ratio = 20;
> > +int vm_dirty_ratio = 40;
> > 
> > /*
> >  * vm_dirty_bytes starts at 0 (disabled) so that it is a function of
> > 
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
> ---end quoted text---
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>