From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.9 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF353C433DF for ; Wed, 20 May 2020 20:26:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 92960207E8 for ; Wed, 20 May 2020 20:26:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=chrisdown.name header.i=@chrisdown.name header.b="PGibgYQ1" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 92960207E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=chrisdown.name Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 027EB80007; Wed, 20 May 2020 16:26:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F1A02900002; Wed, 20 May 2020 16:26:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2FBB80007; Wed, 20 May 2020 16:26:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0174.hostedemail.com [216.40.44.174]) by kanga.kvack.org (Postfix) with ESMTP id CC21F900002 for ; Wed, 20 May 2020 16:26:52 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 82E0F180AD822 for ; Wed, 20 May 2020 20:26:52 +0000 (UTC) X-FDA: 76838231064.13.use51_3bf9514403a4a X-HE-Tag: use51_3bf9514403a4a X-Filterd-Recvd-Size: 5950 Received: from mail-ej1-f66.google.com (mail-ej1-f66.google.com [209.85.218.66]) by imf04.hostedemail.com (Postfix) with ESMTP for ; Wed, 20 May 2020 20:26:51 +0000 (UTC) Received: by mail-ej1-f66.google.com with SMTP id s21so5860910ejd.2 for ; Wed, 20 May 2020 13:26:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chrisdown.name; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=pyAjRlnWUUT1mywco3FJeiLGJZLmkRz0VQZk3yUWI70=; b=PGibgYQ1AtG2afha5buPGXZKWooA7vXxvQmNmO1Idq+C1zDslsESBmlsaVApchVa1L glQJr13dgRd6TvmwuLCmKqJbXvyxIQU3mUqKj5WsYxSaRPgBreBPMdk+Dj13fIlUFGXk IeF+KqfHBc9dvaXAutyupqPzuLkHNVKW3WeOo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=pyAjRlnWUUT1mywco3FJeiLGJZLmkRz0VQZk3yUWI70=; b=ZHm627OIc0Rn/+C6vWMyMLqpTlK5ai71wbtkVfK60n2qiWceocnkcdqUsB39pPalHD 7tU2PCZqTQ9RwKM26gYRdz7rBNr2SzO3RvTpN3Ce0IqPLr4vimJ3fLP6liXmAxvMM1wp 09JE3OTigPGIvWCCNI3HDnzI2i2x9P3BAQvjmSPv3j1O/9ThZzAJYdv6zbWMig2c6nkW W6KiT1wutHyzEp4zzGR1ynvXrVCmEgKYgkYI0hqQmpPnsltTeSWC4I3g/hfP24fpgEY3 Ia+HNEnZE00MkC7fC/NAHNGsA75tv4JalUE7FfITMP9ymXxaD9FWxIDqCTiiWliqmbPP ppxg== X-Gm-Message-State: AOAM5313atdwyDORhTMpi0PsgCa8Ths0WqFkS2M5LnYDf/EwAeRHYuJk y/5+gYwXvSwfKZH4BGyCKOIMbQ== X-Google-Smtp-Source: ABdhPJwgeeyXXyd9EvAUvOOxDLmw29QD9rJLdjvxlqGZ1Pj+7iwL+qGPnFCbaSDWyDov/XvQHBVs0A== X-Received: by 2002:a17:906:7a1c:: with SMTP id d28mr756450ejo.10.1590006410840; Wed, 20 May 2020 13:26:50 -0700 (PDT) Received: from localhost ([2620:10d:c093:400::5:758d]) by smtp.gmail.com with ESMTPSA id dt12sm2822454ejb.102.2020.05.20.13.26.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2020 13:26:50 -0700 (PDT) Date: Wed, 20 May 2020 21:26:50 +0100 From: Chris Down To: Michal Hocko Cc: Andrew Morton , Johannes Weiner , Tejun Heo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@fb.com Subject: Re: [PATCH] mm, memcg: reclaim more aggressively before high allocator throttling Message-ID: <20200520202650.GB558281@chrisdown.name> References: <20200520143712.GA749486@chrisdown.name> <20200520160756.GE6462@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <20200520160756.GE6462@dhcp22.suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Michal Hocko writes: >Let me try to understand the actual problem. The high memory reclaim has >a target which is proportional to the amount of charged memory. For most >requests that would be SWAP_CLUSTER_MAX though (resp. N times that where >N is the number of memcgs in excess up the hierarchy). I can see to be >insufficient if the memcg is already in a large excess but if the >reclaim can make a forward progress this should just work fine because >each charging context should reclaim at least the contributed amount. > >Do you have any insight on why this doesn't work in your situation? >Especially with such a large inactive file list I would be really >surprised if the reclaim was not able to make a forward progress. Reclaim can fail for any number of reasons, which is why we have retries sprinkled all over for it already. It doesn't seem hard to believe that it might just fail for transient reasons and drive us deeper into the hole as a result. In this case, a.) the application is producing tons of dirty pages, and b.) we have really heavy systemwide I/O contention on the affected machines. This high load is one of the reasons that direct and kswapd reclaim cannot keep up, and thus nr_pages can become a number of orders of magnitude larger than SWAP_CLUSTER_MAX. This is trivially reproducible on these machines, it's not an edge case. Putting a trace_printk("%d\n", __LINE__) at non-successful reclaim in shrink_page_list shows that what's happening is always (and I really mean always) the "dirty page and you're not kswapd" check, as expected: if (PageDirty(page)) { /* * Only kswapd can writeback filesystem pages * to avoid risk of stack overflow. But avoid * injecting inefficient single-page IO into * flusher writeback as much as possible: only * write pages when we've encountered many * dirty pages, and when we've already scanned * the rest of the LRU for clean pages and see * the same dirty pages again (PageReclaim). */ if (page_is_file_lru(page) && (!current_is_kswapd() || !PageReclaim(page) || !test_bit(PGDAT_DIRTY, &pgdat->flags))) { /* * Immediately reclaim when written back. * Similar in principal to deactivate_page() * except we already have the page isolated * and know it's dirty */ inc_node_page_state(page, NR_VMSCAN_IMMEDIATE); SetPageReclaim(page); goto activate_locked; } >Now to your patch. I do not like it much to be honest. >MEM_CGROUP_RECLAIM_RETRIES is quite arbitrary and I neither like it in >memory_high_write because the that is an interruptible context so there >shouldn't be a good reason to give up after $FOO number of failed >attempts. try_charge and memory_max_write are slightly different because >we are invoking OOM killer based on the number of failed attempts. As Johannes mentioned, the very intent of memory.high is to have it managed using a userspace OOM killer, which monitors PSI. As such, I'm not sure this distinction means much.