From mboxrd@z Thu Jan 1 00:00:00 1970 From: Waiman Long Subject: Re: [PATCH] Documentation: Clarify usage of memory limits Date: Thu, 1 Jun 2023 20:09:42 -0400 Message-ID: References: <20230601183820.3839891-1-schatzberg.dan@gmail.com> <20230601195345.GB157732@cmpxchg.org> Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685664588; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IQBTL4KWFtPVFbs25Ot9x0LJbTVOUU336wX1+2B8Rjk=; b=SupiPRc9oKFa+Gk1jUVkI+qcMzcrPPHZ8VbEQ3wWUX0z4XxvSZoOzdASX12bLcVyFn6jo3 b0551IFjvfk5LrOtKuTyGshq3A8FX87Vb/KKvSTP1VWivXu6n82r+Q5fZislQ58InGqOdH yVwwTyHQdV0XRE9w+Yc8GTTIMu63lgg= Content-Language: en-US In-Reply-To: <20230601195345.GB157732-druUgvl0LCNAfugRpC6u6w@public.gmane.org> List-ID: Content-Type: text/plain; charset="us-ascii"; format="flowed" To: Johannes Weiner Cc: Dan Schatzberg , Tejun Heo , Chris Down , Zefan Li , Jonathan Corbet , "open list:CONTROL GROUP (CGROUP)" , "open list:DOCUMENTATION" , open list On 6/1/23 15:53, Johannes Weiner wrote: > On Thu, Jun 01, 2023 at 03:15:28PM -0400, Waiman Long wrote: >> On 6/1/23 14:38, Dan Schatzberg wrote: >>> The existing documentation refers to memory.high as the "main mechanism >>> to control memory usage." This seems incorrect to me - memory.high can >>> result in reclaim pressure which simply leads to stalls unless some >>> external component observes and actions on it (e.g. systemd-oomd can be >>> used for this purpose). While this is feasible, users are unaware of >>> this interaction and are led to believe that memory.high alone is an >>> effective mechanism for limiting memory. >>> >>> The documentation should recommend the use of memory.max as the >>> effective way to enforce memory limits - it triggers reclaim and results >>> in OOM kills by itself. >> That is not how my understanding of memory.high works. When memory usage >> goes past memory.high, memory reclaim will be initiated to reclaim the >> memory back. Stall happens when memory.usage keep increasing like by >> consuming memory faster than what memory reclaim can recover. When >> memory.max is reached, OOM killer will then kill off the tasks. > This was the initial plan indeed: Slow down the workload and thus slow > the growth; hope that the workload recovers with voluntary frees; set > memory.max as a safety if it keeps going beyond. > > This never panned out. Once workloads are stuck, they might not back > down on their own. By increasingly slowing growth, it becomes harder > and harder for them to reach the memory.max intervention point. > > It's a very brittle configuration strategy. Unless you very carefully > calibrate memory.high and memory.max together with awareness of the > throttling algorithm, workloads that hit memory.high will just go to > sleep indefinitely. They require outside intervention that either > adjusts limits or implements kill policies based on observed sleeps > (they're reported as pressure via psi). > > So the common usecases today end up being that memory.max is for > enforcing kernel OOM kills, and memory.high is a tool to implement > userspace OOM killing policies. > > Dan is right to point out the additional expectations for userspace > management when memory.high is in used. And memory.max is still the > primary, works-out-of-the-box method of memory containment. Thanks for clarification. I have to reset my false assumption. Cheers, Longman