cgroup memory, blkio and the lovely swapping

cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

Hi,

currently when one process causes heavy swapping, the responsiveness of
the hole system suffers greatly. With the small memleak [1] test tool I
wrote, the effect can be experienced very easily, depending on the
delay the lag can become quite large. If I ensure that 10% of the RAM
stay free for free memory and cache, the system never swaps to death.
That works very well, but if accesses to the swap are very heavy, the
system still lags on all other processes, not only the swapping one.
Putting the swapping process into a blkio cgroup with little weight does
not affect the io or swap io from other processes with larger weight in
their group.

Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
swapping and control to let the pagein respect the blkio.weight value
or even better add a second weight value for swapping io ?



kind regards
  Daniel


[1] https://github.com/poelzi/ulatencyd/blob/master/tests/memleak.c

cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

Hi,

currently when one process causes heavy swapping, the responsiveness of
the hole system suffers greatly. With the small memleak [1] test tool I
wrote, the effect can be experienced very easily, depending on the
delay the lag can become quite large. If I ensure that 10% of the RAM
stay free for free memory and cache, the system never swaps to death.
That works very well, but if accesses to the swap are very heavy, the
system still lags on all other processes, not only the swapping one.
Putting the swapping process into a blkio cgroup with little weight does
not affect the io or swap io from other processes with larger weight in
their group.

Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
swapping and control to let the pagein respect the blkio.weight value
or even better add a second weight value for swapping io ?



kind regards
  Daniel


[1] https://github.com/poelzi/ulatencyd/blob/master/tests/memleak.c

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 08:39:44 +0100
Daniel Poelzleithner <poelzi@poelzi.org> wrote:

> Hi,
> 
> currently when one process causes heavy swapping, the responsiveness of
> the hole system suffers greatly. With the small memleak [1] test tool I
> wrote, the effect can be experienced very easily, depending on the
> delay the lag can become quite large. If I ensure that 10% of the RAM
> stay free for free memory and cache, the system never swaps to death.
> That works very well, but if accesses to the swap are very heavy, the
> system still lags on all other processes, not only the swapping one.
> Putting the swapping process into a blkio cgroup with little weight does
> not affect the io or swap io from other processes with larger weight in
> their group.
> 
> Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
> swapping and control to let the pagein respect the blkio.weight value
> or even better add a second weight value for swapping io ?
> 

Now, blkio cgroup does work only with synchronous I/O(direct I/O)
and never work with swap I/O. And I don't think swap-i/o limit
is a blkio matter.

Memory cgroup is now developping dirty_ratio for memory cgroup.
By that, you can control the number of pages in writeback, in memory cgroup.
I think it will work for you.

Thanks,
-Kame

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 08:39:44 +0100
Daniel Poelzleithner <poelzi@poelzi.org> wrote:

> Hi,
> 
> currently when one process causes heavy swapping, the responsiveness of
> the hole system suffers greatly. With the small memleak [1] test tool I
> wrote, the effect can be experienced very easily, depending on the
> delay the lag can become quite large. If I ensure that 10% of the RAM
> stay free for free memory and cache, the system never swaps to death.
> That works very well, but if accesses to the swap are very heavy, the
> system still lags on all other processes, not only the swapping one.
> Putting the swapping process into a blkio cgroup with little weight does
> not affect the io or swap io from other processes with larger weight in
> their group.
> 
> Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
> swapping and control to let the pagein respect the blkio.weight value
> or even better add a second weight value for swapping io ?
> 

Now, blkio cgroup does work only with synchronous I/O(direct I/O)
and never work with swap I/O. And I don't think swap-i/o limit
is a blkio matter.

Memory cgroup is now developping dirty_ratio for memory cgroup.
By that, you can control the number of pages in writeback, in memory cgroup.
I think it will work for you.

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

On Fri, 4 Mar 2011 16:54:55 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org> wrote:

> Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> and never work with swap I/O. And I don't think swap-i/o limit
> is a blkio matter.

I'm totally unsure about what subsystem it really belongs to. It is
memory for sure, but disk access, which it actually affects, belongs to
the blkio subsystem. Is there a technical reason why swap I/O is not run
through the blkio system ?


> Memory cgroup is now developping dirty_ratio for memory cgroup.
> By that, you can control the number of pages in writeback, in memory
> cgroup. I think it will work for you.

I'm not sure that fixes the fairness problem on swapio. Just having a
larger buffer before a writeback happens will reduce seeks, but not
give fair share of io in swap in. It's good to control over it on
cgroup level, but i doubt it will fix the problem.



kind regards
 Daniel

Re: cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

On Fri, 4 Mar 2011 16:54:55 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> and never work with swap I/O. And I don't think swap-i/o limit
> is a blkio matter.

I'm totally unsure about what subsystem it really belongs to. It is
memory for sure, but disk access, which it actually affects, belongs to
the blkio subsystem. Is there a technical reason why swap I/O is not run
through the blkio system ?


> Memory cgroup is now developping dirty_ratio for memory cgroup.
> By that, you can control the number of pages in writeback, in memory
> cgroup. I think it will work for you.

I'm not sure that fixes the fairness problem on swapio. Just having a
larger buffer before a writeback happens will reduce seeks, but not
give fair share of io in swap in. It's good to control over it on
cgroup level, but i doubt it will fix the problem.



kind regards
 Daniel

Re: cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

On Fri, 4 Mar 2011 16:54:55 +0900
KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:

> Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> and never work with swap I/O. And I don't think swap-i/o limit
> is a blkio matter.

I'm totally unsure about what subsystem it really belongs to. It is
memory for sure, but disk access, which it actually affects, belongs to
the blkio subsystem. Is there a technical reason why swap I/O is not run
through the blkio system ?


> Memory cgroup is now developping dirty_ratio for memory cgroup.
> By that, you can control the number of pages in writeback, in memory
> cgroup. I think it will work for you.

I'm not sure that fixes the fairness problem on swapio. Just having a
larger buffer before a writeback happens will reduce seeks, but not
give fair share of io in swap in. It's good to control over it on
cgroup level, but i doubt it will fix the problem.



kind regards
 Daniel

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 09:11:32 +0100
Daniel Poelzleithner <poelzi@poelzi.org> wrote:

> On Fri, 4 Mar 2011 16:54:55 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> > and never work with swap I/O. And I don't think swap-i/o limit
> > is a blkio matter.
> 
> I'm totally unsure about what subsystem it really belongs to. It is
> memory for sure, but disk access, which it actually affects, belongs to
> the blkio subsystem. Is there a technical reason why swap I/O is not run
> through the blkio system ?
> 

Now, blkio cgroup has no tags on each page. Then, it works only when
it can detect a thread which starts I/O in block layer.
But there is an activity to fix that.

http://marc.info/?l=linux-mm&m=129888823027871&w=2

I think you can discuss swap io handling in this thread.

> 
> > Memory cgroup is now developping dirty_ratio for memory cgroup.
> > By that, you can control the number of pages in writeback, in memory
> > cgroup. I think it will work for you.
> 
> I'm not sure that fixes the fairness problem on swapio. Just having a
> larger buffer before a writeback happens will reduce seeks, but not
> give fair share of io in swap in. It's good to control over it on
> cgroup level, but i doubt it will fix the problem.
> 

swap-in is out-of-control from memcg's view and have no plans.
IHMO, the number of swap-in will be blkio cgroup matter.

Thanks,
-Kame 

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 09:11:32 +0100
Daniel Poelzleithner <poelzi@poelzi.org> wrote:

> On Fri, 4 Mar 2011 16:54:55 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> > and never work with swap I/O. And I don't think swap-i/o limit
> > is a blkio matter.
> 
> I'm totally unsure about what subsystem it really belongs to. It is
> memory for sure, but disk access, which it actually affects, belongs to
> the blkio subsystem. Is there a technical reason why swap I/O is not run
> through the blkio system ?
> 

Now, blkio cgroup has no tags on each page. Then, it works only when
it can detect a thread which starts I/O in block layer.
But there is an activity to fix that.

http://marc.info/?l=linux-mm&m=129888823027871&w=2

I think you can discuss swap io handling in this thread.

> 
> > Memory cgroup is now developping dirty_ratio for memory cgroup.
> > By that, you can control the number of pages in writeback, in memory
> > cgroup. I think it will work for you.
> 
> I'm not sure that fixes the fairness problem on swapio. Just having a
> larger buffer before a writeback happens will reduce seeks, but not
> give fair share of io in swap in. It's good to control over it on
> cgroup level, but i doubt it will fix the problem.
> 

swap-in is out-of-control from memcg's view and have no plans.
IHMO, the number of swap-in will be blkio cgroup matter.

Thanks,
-Kame 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 09:11:32 +0100
Daniel Poelzleithner <poelzi-j3Gp85GwYS0dnm+yROfE0A@public.gmane.org> wrote:

> On Fri, 4 Mar 2011 16:54:55 +0900
> KAMEZAWA Hiroyuki <kamezawa.hiroyu-+CUm20s59erQFUHtdCDX3A@public.gmane.org> wrote:
> 
> > Now, blkio cgroup does work only with synchronous I/O(direct I/O)
> > and never work with swap I/O. And I don't think swap-i/o limit
> > is a blkio matter.
> 
> I'm totally unsure about what subsystem it really belongs to. It is
> memory for sure, but disk access, which it actually affects, belongs to
> the blkio subsystem. Is there a technical reason why swap I/O is not run
> through the blkio system ?
> 

Now, blkio cgroup has no tags on each page. Then, it works only when
it can detect a thread which starts I/O in block layer.
But there is an activity to fix that.

http://marc.info/?l=linux-mm&m=129888823027871&w=2

I think you can discuss swap io handling in this thread.

> 
> > Memory cgroup is now developping dirty_ratio for memory cgroup.
> > By that, you can control the number of pages in writeback, in memory
> > cgroup. I think it will work for you.
> 
> I'm not sure that fixes the fairness problem on swapio. Just having a
> larger buffer before a writeback happens will reduce seeks, but not
> give fair share of io in swap in. It's good to control over it on
> cgroup level, but i doubt it will fix the problem.
> 

swap-in is out-of-control from memcg's view and have no plans.
IHMO, the number of swap-in will be blkio cgroup matter.

Thanks,
-Kame 

Re: cgroup memory, blkio and the lovely swapping

[KAMEZAWA Hiroyuki]

On Fri, 4 Mar 2011 08:39:44 +0100
Daniel Poelzleithner <poelzi-j3Gp85GwYS0dnm+yROfE0A@public.gmane.org> wrote:

> Hi,
> 
> currently when one process causes heavy swapping, the responsiveness of
> the hole system suffers greatly. With the small memleak [1] test tool I
> wrote, the effect can be experienced very easily, depending on the
> delay the lag can become quite large. If I ensure that 10% of the RAM
> stay free for free memory and cache, the system never swaps to death.
> That works very well, but if accesses to the swap are very heavy, the
> system still lags on all other processes, not only the swapping one.
> Putting the swapping process into a blkio cgroup with little weight does
> not affect the io or swap io from other processes with larger weight in
> their group.
> 
> Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
> swapping and control to let the pagein respect the blkio.weight value
> or even better add a second weight value for swapping io ?
> 

Now, blkio cgroup does work only with synchronous I/O(direct I/O)
and never work with swap I/O. And I don't think swap-i/o limit
is a blkio matter.

Memory cgroup is now developping dirty_ratio for memory cgroup.
By that, you can control the number of pages in writeback, in memory cgroup.
I think it will work for you.

Thanks,
-Kame

cgroup memory, blkio and the lovely swapping

[Daniel Poelzleithner]

Hi,

currently when one process causes heavy swapping, the responsiveness of
the hole system suffers greatly. With the small memleak [1] test tool I
wrote, the effect can be experienced very easily, depending on the
delay the lag can become quite large. If I ensure that 10% of the RAM
stay free for free memory and cache, the system never swaps to death.
That works very well, but if accesses to the swap are very heavy, the
system still lags on all other processes, not only the swapping one.
Putting the swapping process into a blkio cgroup with little weight does
not affect the io or swap io from other processes with larger weight in
their group.

Maybe I'm mistaken, but wouldn't it be the easiest way to get fair
swapping and control to let the pagein respect the blkio.weight value
or even better add a second weight value for swapping io ?



kind regards
  Daniel


[1] https://github.com/poelzi/ulatencyd/blob/master/tests/memleak.c