Re: [PATCH V6 00/10] memcg: per cgroup background reclaim

[Ying Han <yinghan@google.com>]

[-- Attachment #1: Type: text/plain, Size: 3993 bytes --]

On Wed, Apr 27, 2011 at 12:36 AM, Johannes Weiner <hannes@cmpxchg.org>wrote:

> On Fri, Apr 22, 2011 at 08:33:58PM -0700, Ying Han wrote:
> > On Fri, Apr 22, 2011 at 7:34 PM, Johannes Weiner <hannes@cmpxchg.org>
> wrote:
> >
> > > On Fri, Apr 22, 2011 at 07:10:25PM -0700, Ying Han wrote: >
> > > However, i still think there is a need from the admin to have some
> > > controls > of which memcg to do background reclaim proactively
> > > (before global memory > pressure) and that was the initial logic
> > > behind the API.
> > >
> > > That sounds more interesting.  Do you have a specific use case
> > > that requires this?
> >
> > There might be more interesting use cases there, and here is one I
> > can think of:
> >
> > let's say we three jobs A, B and C, and one host with 32G of RAM. We
> > configure each job's hard_limit as their peak memory usage.
> > A: 16G
> > B: 16G
> > C: 10G
> >
> > 1. we start running A with hard_limit 15G, and start running B with
> > hard_limit 15G.
> > 2. we set A and B's soft_limit based on their "hot" memory. Let's say
> > setting A's soft_limit 10G and B's soft_limit 10G.
> > (The soft_limit will be changing based on their runtime memory usage)
> >
> > If no more jobs running on the system, A and B will easily fill up the
> whole
> > system with pagecache pages. Since we are not over-committing the machine
> > with their hard_limit, there will be no pressure to push their memory
> usage
> > down to soft_limit.
> >
> > Now we would like to launch another job C, since we know there are A(16G
> -
> > 10G) + B(16G - 10G)  = 12G "cold" memory can be reclaimed (w/o impacting
> the
> > A and B's performance). So what will happen
> >
> > 1. start running C on the host, which triggers global memory pressure
> right
> > away. If the reclaim is fast, C start growing with the free pages from A
> and
> > B.
> >
> > However, it might be possible that the reclaim can not catch-up with the
> > job's page allocation. We end up with either OOM condition or performance
> > spike on any of the running jobs.
>
> If background reclaim can not catch up, C will go into direct reclaim,
> which will have exactly the same effect, only that C will have to do
> the work itself.
>
> > One way to improve it is to set a wmark on either A/B to be proactively
> > reclaiming pages before launching C. The global memory pressure won't
> help
> > much here since we won't trigger that.
>
> Ok, so you want to use the watermarks to push back and limit the usage
> of A and B to make room for C.  Isn't this exactly what the hard limit
> is for?
>
> similar, but not exactly the same. there is no need to hard cap the memory
usage for A and B in that case.
what we need is to have some period of time that A and B slowly reclaim
pages and leaves some room to
launch C smoothly.


> I don't understand the second sentence: global memory pressure won't
> kick in with only A and B, but it will once C starts up.
>
> In my example, the hard_limit of A+B is less than the machine capacity. And
after we have
per-memcg bg reclaim, ideally we won't trigger global reclaim much.

But when we launch C, we end up over-committing the machine. So the global
reclaim will
fire up quickly.

Anyway, i agree that less newly invented kernel API is good. If you check
the latest past V8 from
Kame, we reduced the two API to one which we only allow setting the
high_wmark_distance and the
low_wmark_distance is set internally. I think this is good enough.

--Ying



> > > min_free_kbytes more or less indirectly provides the same on a global
> > > level, but I don't think anybody tunes it just for aggressiveness of
> > > background reclaim.
> > >
> >
> > Hmm, we do scale that in google workload. With large machines under lots
> of
> > memory pressure and heavily network traffic workload, we would like to
> > reduce the likelyhood of page alloc failure. But this is kind of
> different
> > from what we are talking about here.
>
> My point indeed ;-)
>
>        Hannes
>

[-- Attachment #2: Type: text/html, Size: 5319 bytes --]