[LSF/VM TOPIC] Dynamic sizing of dirty_limit

[LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Jan Kara]

  Hi,

  one more suggestion for discussion:
Currently, the amount of dirtiable memory is fixed - either to a percentage
of ram (dirty_limit) or to a fix number of megabytes. The problem with this
is that if you have application doing random writes on a file (like some
simple databases do), you'll get a big performance improvement if you
increase the amount of dirtiable memory (because you safe quite some
rewrites and also get larger chunks of sequential IO) (*)
On the other hand for sequential IO increasing dirtiable memory (beyond
certain level) does not really help - you end up doing the same IO.  So for
a machine is doing sequential IO, having 10% of memory dirtiable is just
fine (and you probably don't want much more because the memory is better
used for something else), when a machine does random rewrites, going to 40%
might be well worth it. So I'd like to discuss how we could measure that
increasing amount of dirtiable memory helps so that we could implement
dynamic sizing of it.

(*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
with old kernels because customers running e.g. LDAP (using BerkelyDB
heavily) were complaining about performance problems.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Jan Kara]

  Hi,

  one more suggestion for discussion:
Currently, the amount of dirtiable memory is fixed - either to a percentage
of ram (dirty_limit) or to a fix number of megabytes. The problem with this
is that if you have application doing random writes on a file (like some
simple databases do), you'll get a big performance improvement if you
increase the amount of dirtiable memory (because you safe quite some
rewrites and also get larger chunks of sequential IO) (*)
On the other hand for sequential IO increasing dirtiable memory (beyond
certain level) does not really help - you end up doing the same IO.  So for
a machine is doing sequential IO, having 10% of memory dirtiable is just
fine (and you probably don't want much more because the memory is better
used for something else), when a machine does random rewrites, going to 40%
might be well worth it. So I'd like to discuss how we could measure that
increasing amount of dirtiable memory helps so that we could implement
dynamic sizing of it.

(*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
with old kernels because customers running e.g. LDAP (using BerkelyDB
heavily) were complaining about performance problems.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Christoph Lameter]

On Wed, 24 Feb 2010, Jan Kara wrote:

> fine (and you probably don't want much more because the memory is better
> used for something else), when a machine does random rewrites, going to 40%
> might be well worth it. So I'd like to discuss how we could measure that
> increasing amount of dirtiable memory helps so that we could implement
> dynamic sizing of it.

Another issue around dirty limits is that they are global. If you are
running multiple jobs on the same box (memcg or cpusets or you set
affinities to separate the box) then every job may need different dirty
limits. One idea that I had in the past was to set dirty limits based on
nodes or cpusets. But that will not cover the other cases that I have
listed above.

The best solution would be an algorithm that can accomodate multiple loads
and manage the amount of dirty memory automatically.



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Andreas Dilger]

On 2010-02-24, at 09:10, Christoph Lameter wrote:
> On Wed, 24 Feb 2010, Jan Kara wrote:
> fine (and you probably don't want much more because the memory is  
> better
>> used for something else), when a machine does random rewrites,  
>> going to 40%
>> might be well worth it. So I'd like to discuss how we could measure  
>> that
>> increasing amount of dirtiable memory helps so that we could  
>> implement
>> dynamic sizing of it.
>
> Another issue around dirty limits is that they are global. If you are
> running multiple jobs on the same box (memcg or cpusets or you set
> affinities to separate the box) then every job may need different  
> dirty
> limits. One idea that I had in the past was to set dirty limits  
> based on
> nodes or cpusets. But that will not cover the other cases that I have
> listed above.
>
> The best solution would be an algorithm that can accomodate multiple  
> loads
> and manage the amount of dirty memory automatically.


Why not dirty limits per file and/or a function of the IO randomness  
vs the file size?  Doing streaming on a large file can easily be  
detected and limited appropriately (either the filesystem can keep up  
and the "smaller" limit will not be hit, or it can't keep up and the  
application needs to be throttled nearly regardless of what the limit  
is).  Doing streaming or random IO on small files is almost  
indistinguishable anyway and should pretty much be treated as random  
IO subject to a "larger" global limit.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

Re: [LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Andreas Dilger]

On 2010-02-24, at 09:10, Christoph Lameter wrote:
> On Wed, 24 Feb 2010, Jan Kara wrote:
> fine (and you probably don't want much more because the memory is  
> better
>> used for something else), when a machine does random rewrites,  
>> going to 40%
>> might be well worth it. So I'd like to discuss how we could measure  
>> that
>> increasing amount of dirtiable memory helps so that we could  
>> implement
>> dynamic sizing of it.
>
> Another issue around dirty limits is that they are global. If you are
> running multiple jobs on the same box (memcg or cpusets or you set
> affinities to separate the box) then every job may need different  
> dirty
> limits. One idea that I had in the past was to set dirty limits  
> based on
> nodes or cpusets. But that will not cover the other cases that I have
> listed above.
>
> The best solution would be an algorithm that can accomodate multiple  
> loads
> and manage the amount of dirty memory automatically.


Why not dirty limits per file and/or a function of the IO randomness  
vs the file size?  Doing streaming on a large file can easily be  
detected and limited appropriately (either the filesystem can keep up  
and the "smaller" limit will not be hit, or it can't keep up and the  
application needs to be throttled nearly regardless of what the limit  
is).  Doing streaming or random IO on small files is almost  
indistinguishable anyway and should pretty much be treated as random  
IO subject to a "larger" global limit.

Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Boaz Harrosh]

On 02/24/2010 06:10 PM, Christoph Lameter wrote:
> On Wed, 24 Feb 2010, Jan Kara wrote:
> 
>> fine (and you probably don't want much more because the memory is better
>> used for something else), when a machine does random rewrites, going to 40%
>> might be well worth it. So I'd like to discuss how we could measure that
>> increasing amount of dirtiable memory helps so that we could implement
>> dynamic sizing of it.
> 
> Another issue around dirty limits is that they are global. If you are
> running multiple jobs on the same box (memcg or cpusets or you set
> affinities to separate the box) then every job may need different dirty
> limits. One idea that I had in the past was to set dirty limits based on
> nodes or cpusets. But that will not cover the other cases that I have
> listed above.
> 
> The best solution would be an algorithm that can accomodate multiple loads
> and manage the amount of dirty memory automatically.
> 

One more point to consider if changes are made (and should) in this area is:
	The stacking filesystems problem.
There are many examples, here is just a simple one. A local iscsi-target backed
by a file an a filesystem, is logged into from local host, the created block device
is mounted by a filesystem. Such a setup used to dead-lock before and has very poor
dribbling performance today. This is because the upper-layer filesystem consumes all
cache quota and leaves no available cache headroom for the lower-level FS, causing
the lower-level FS a page by page write-out (at best). For example mounting such a
scenario through a UML or VM will solve this problem and will preform optimally.
(The iscsi-initiator + upper FS is inside the UML).
There are endless examples of stacking filesystem examples, including NFS local mounts,
clustered setup with local access to one of the devices, and so on. All these preform
badly.

A per-FS cache limit, (proportional to performance, cache is optimally measured by
a time constant), should easily solve this problem as well.

Boaz

Re: [LSF/VM TOPIC] Dynamic sizing of dirty_limit

[Boaz Harrosh]

On 02/24/2010 06:10 PM, Christoph Lameter wrote:
> On Wed, 24 Feb 2010, Jan Kara wrote:
> 
>> fine (and you probably don't want much more because the memory is better
>> used for something else), when a machine does random rewrites, going to 40%
>> might be well worth it. So I'd like to discuss how we could measure that
>> increasing amount of dirtiable memory helps so that we could implement
>> dynamic sizing of it.
> 
> Another issue around dirty limits is that they are global. If you are
> running multiple jobs on the same box (memcg or cpusets or you set
> affinities to separate the box) then every job may need different dirty
> limits. One idea that I had in the past was to set dirty limits based on
> nodes or cpusets. But that will not cover the other cases that I have
> listed above.
> 
> The best solution would be an algorithm that can accomodate multiple loads
> and manage the amount of dirty memory automatically.
> 

One more point to consider if changes are made (and should) in this area is:
	The stacking filesystems problem.
There are many examples, here is just a simple one. A local iscsi-target backed
by a file an a filesystem, is logged into from local host, the created block device
is mounted by a filesystem. Such a setup used to dead-lock before and has very poor
dribbling performance today. This is because the upper-layer filesystem consumes all
cache quota and leaves no available cache headroom for the lower-level FS, causing
the lower-level FS a page by page write-out (at best). For example mounting such a
scenario through a UML or VM will solve this problem and will preform optimally.
(The iscsi-initiator + upper FS is inside the UML).
There are endless examples of stacking filesystem examples, including NFS local mounts,
clustered setup with local access to one of the devices, and so on. All these preform
badly.

A per-FS cache limit, (proportional to performance, cache is optimally measured by
a time constant), should easily solve this problem as well.

Boaz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>