From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail203.messagelabs.com (mail203.messagelabs.com [216.82.254.243]) by kanga.kvack.org (Postfix) with ESMTP id 59C2C900086 for ; Mon, 18 Apr 2011 18:27:29 -0400 (EDT) Received: from kpbe15.cbf.corp.google.com (kpbe15.cbf.corp.google.com [172.25.105.79]) by smtp-out.google.com with ESMTP id p3IMRL3P002450 for ; Mon, 18 Apr 2011 15:27:25 -0700 Received: from qyl38 (qyl38.prod.google.com [10.241.83.230]) by kpbe15.cbf.corp.google.com with ESMTP id p3IMLTis000688 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NOT) for ; Mon, 18 Apr 2011 15:27:19 -0700 Received: by qyl38 with SMTP id 38so3403298qyl.8 for ; Mon, 18 Apr 2011 15:27:14 -0700 (PDT) MIME-Version: 1.0 In-Reply-To: <20110418184240.GA11653@tiehlicka.suse.cz> References: <1302821669-29862-1-git-send-email-yinghan@google.com> <20110415094040.GC8828@tiehlicka.suse.cz> <20110418091351.GC8925@tiehlicka.suse.cz> <20110418184240.GA11653@tiehlicka.suse.cz> Date: Mon, 18 Apr 2011 15:27:14 -0700 Message-ID: Subject: Re: [PATCH V4 00/10] memcg: per cgroup background reclaim From: Ying Han Content-Type: multipart/alternative; boundary=000e0cdfd082ba138304a138e415 Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: KOSAKI Motohiro , Minchan Kim , Daisuke Nishimura , Balbir Singh , Tejun Heo , Pavel Emelyanov , KAMEZAWA Hiroyuki , Andrew Morton , Li Zefan , Mel Gorman , Christoph Lameter , Johannes Weiner , Rik van Riel , Hugh Dickins , Dave Hansen , Zhu Yanhai , linux-mm@kvack.org --000e0cdfd082ba138304a138e415 Content-Type: text/plain; charset=ISO-8859-1 On Mon, Apr 18, 2011 at 11:42 AM, Michal Hocko wrote: > On Mon 18-04-11 10:01:20, Ying Han wrote: > > On Mon, Apr 18, 2011 at 2:13 AM, Michal Hocko wrote: > [...] > > > I see. I am just concerned whether 3rd level of reclaim is a good idea. > > > We would need to do background reclaim anyway (and to preserve the > > > original semantic it has to be somehow watermark controlled). I am just > > > wondering why we have to implement it separately from kswapd. Cannot we > > > just simply trigger global kswapd which would reclaim all cgroups that > > > are under watermarks? [I am sorry for my ignorance if that is what is > > > implemented in the series - I haven't got to the patches yes] > > > > > > > They are different on per-zone reclaim vs per-memcg reclaim. The first > > one is triggered if the zone is under memory pressure and we need > > to free pages to serve further page allocations. The second one is > > triggered if the memcg is under memory pressure and we need to free > > pages to leave room (limit - usage) for the memcg to grow. > > OK, I see. > > > > > > Both of them are needed and that is how it is implemented on the direct > > reclaim path. The kswapd batches only try to > > smooth out the system and memcg performance by reclaiming pages > proactively. > > It doesn't affecting the functionality. > > I am still wondering, isn't this just a nice to have feature rather > than must to have in order to get rid of the global LRU? The per-memcg kswapd is a must-have, and it is less related to the effort of "get rid of global LRU" than the next patch I am looking at "enhance the soft_limit reclaim". So this is the structure we will end up with background reclaim: 1. per-memcg : this patch 2. global: targeting reclaim by replacing the per-zone to soft_limit reclaim direct reclaim: 1. per-memcg: no change from today 2. global: targeting reclaim by replacing the per-zone to soft_limit reclaim. > Doesn't it make transition more complicated. I have noticed many if-else in > kswapd path to > distinguish per-cgroup from the traditional global background reclaim. > > > [...] > > > > > > > Step1: Create a cgroup with 500M memory_limit. > > > > > > $ mkdir /dev/cgroup/memory/A > > > > > > $ echo 500m >/dev/cgroup/memory/A/memory.limit_in_bytes > > > > > > $ echo $$ >/dev/cgroup/memory/A/tasks > > > > > > > > > > > > Step2: Test and set the wmarks. > > > > > > $ cat /dev/cgroup/memory/A/memory.low_wmark_distance > > > > > > 0 > > > > > > $ cat /dev/cgroup/memory/A/memory.high_wmark_distance > > > > > > 0 > > > > > > > > > > > > > > They are used to tune the high/low_marks based on the hard_limit. We > > > might > > > > need to export that configuration to user admin especially on > machines > > > where > > > > they over-commit by hard_limit. > > > > > > I remember there was some resistance against tuning watermarks > > > separately. > > > > > > > This API is based on KAMEZAWA's request. :) > > This was just as FYI. Watermarks were considered internal thing. So I > wouldn't be surprised if this got somehow controversial. > We went back and forth on how to set the high/low wmarks for different configurations (over-commit or not). So far, by giving the user ability to set the wmarks seems the most feasible way of fullfilling the requriment. > > > > > > > > > > > > $ cat /dev/cgroup/memory/A/memory.reclaim_wmarks > > > > > > low_wmark 524288000 > > > > > > high_wmark 524288000 > > > > > > > > > > > > $ echo 50m >/dev/cgroup/memory/A/memory.high_wmark_distance > > > > > > $ echo 40m >/dev/cgroup/memory/A/memory.low_wmark_distance > > > > > > > > > > > > $ cat /dev/cgroup/memory/A/memory.reclaim_wmarks > > > > > > low_wmark 482344960 > > > > > > high_wmark 471859200 > > > > > > > > > > low_wmark is higher than high_wmark? > > > > > > > > > > > > > hah, it is confusing. I have them documented. Basically, low_wmark > > > > triggers reclaim and high_wmark stop the reclaim. And we have > > > > > > > > high_wmark < usage < low_wmark. > > OK, I see how you calculate those watermarks now but it is really > confusing for those who are used to traditional watermark semantic. > that is true. I adopt the initial comment from Mel where we keep the same logic of triggering and stopping kswapd with low/high_wmarks and also comparing the usage_in_bytes to the wmarks. Either way is confusing and guess we just need to document it well. --Ying -- > Michal Hocko > SUSE Labs > SUSE LINUX s.r.o. > Lihovarska 1060/12 > 190 00 Praha 9 > Czech Republic > --000e0cdfd082ba138304a138e415 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

On Mon, Apr 18, 2011 at 11:42 AM, Michal= Hocko <mhocko@suse.= cz> wrote:
On Mon 18-04-11 10:01:20, Ying Han wrote:
> On Mon, Apr 18, 2011 at 2:13 AM, Michal Hocko <mhocko@suse.cz> wrote:
[...]
> > I see. I am just concerned whether 3rd level of= reclaim is a good idea.
> > We would need to do background reclaim anyway (and to preserve th= e
> > original semantic it has to be somehow watermark controlled). I a= m just
> > wondering why we have to implement it separately from kswapd. Can= not we
> > just simply trigger global kswapd which would reclaim all cgroups= that
> > are under watermarks? [I am sorry for my ignorance if that is wha= t is
> > implemented in the series - I haven't got to the patches yes]=
> >
>
> They are different on per-zone reclaim vs per-memcg reclaim. The first=
> one is triggered if the zone is under memory pressure and we need
> to free pages to serve further page allocations. =A0The second one is<= br> > triggered if the memcg is under memory pressure and we need to free > pages to leave room (limit - usage) for the memcg to grow.

OK, I see.


>
> Both of them are needed and that is how it is implemented on the direc= t
> reclaim path. The kswapd batches only try to
> smooth out the system and memcg performance by reclaiming pages proact= ively.
> It doesn't affecting the functionality.

I am still wondering, isn't this just a nice to have feature rath= er than=A0must to have in order to get rid of the global LRU?
<= div>The per-memcg kswapd is a must-have, and it is less related to the effo= rt of "get rid of global LRU" than the next patch I am looking at= "enhance the soft_limit reclaim". So this is the structure we wi= ll end up with

background reclaim:
1. per-memcg : this patch=
2. global: targeting reclaim by replacing the per-zone to soft_l= imit reclaim

direct reclaim:
1. per-memc= g: no change from today
2. global: targeting reclaim by replacing the per-zone to soft_limit r= eclaim.
=A0
Doesn't it m= ake=A0transition more complicated. I have noticed many if-else in kswapd pa= th to
distinguish per-cgroup from the traditional global background reclaim.
<= /blockquote>

=A0

=A0

[...]

> > > > > Step1: Create a cgroup with 500M memory_limit.
> > > > > $ mkdir /dev/cgroup/memory/A
> > > > > $ echo 500m >/dev/cgroup/memory/A/memory.limit_= in_bytes
> > > > > $ echo $$ >/dev/cgroup/memory/A/tasks
> > > > >
> > > > > Step2: Test and set the wmarks.
> > > > > $ cat /dev/cgroup/memory/A/memory.low_wmark_distan= ce
> > > > > 0
> > > > > $ cat /dev/cgroup/memory/A/memory.high_wmark_dista= nce
> > > > > 0
> > > >
> > > >
> > > They are used to tune the high/low_marks based on the hard_l= imit. We
> > might
> > > need to export that configuration to user admin especially o= n machines
> > where
> > > they over-commit by hard_limit.
> >
> > I remember there was some resistance against tuning watermarks > > separately.
> >
>
> This API is based on KAMEZAWA's request. :)

This was just as FYI. Watermarks were considered internal thing. So I=
wouldn't be surprised if this got somehow controversial.

We went back and forth on how to set the high/low wma= rks for different configurations (over-commit or not). So far, by
giving the user ability to set the wmarks seems the most feasible way = of fullfilling the requriment.=A0

>
> >
> > > > > $ cat /dev/cgroup/memory/A/memory.reclaim_wmarks > > > > > low_wmark 524288000
> > > > > high_wmark 524288000
> > > > >
> > > > > $ echo 50m >/dev/cgroup/memory/A/memory.high_wm= ark_distance
> > > > > $ echo 40m >/dev/cgroup/memory/A/memory.low_wma= rk_distance
> > > > >
> > > > > $ cat /dev/cgroup/memory/A/memory.reclaim_wmarks > > > > > low_wmark =A0482344960
> > > > > high_wmark 471859200
> > > >
> > > > low_wmark is higher than high_wmark?
> > > >
> > >
> > > hah, it is confusing. I have them documented. Basically, low= _wmark
> > > triggers reclaim and high_wmark stop the reclaim. And we hav= e
> > >
> > > high_wmark < usage < low_wmark.

OK, I see how you calculate those watermarks now but it is really
confusing for those who are used to traditional watermark semantic.

that is true. =A0I adopt the initial comment f= rom Mel where we keep the same logic of triggering and stopping kswapd with= low/high_wmarks and also comparing the usage_in_bytes to the wmarks.=A0Eit= her way is confusing and guess we just need to document it well.

--Ying

--
Michal Hocko
SUSE Labs
SUSE LINUX s.r.o.
Lihovarska 1060/12
190 00 Praha 9
Czech Republic

--000e0cdfd082ba138304a138e415-- -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org