From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55A49C4332F for ; Mon, 28 Nov 2022 18:21:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232628AbiK1SVW (ORCPT ); Mon, 28 Nov 2022 13:21:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60390 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234265AbiK1SUn (ORCPT ); Mon, 28 Nov 2022 13:20:43 -0500 Received: from mail-qv1-xf2d.google.com (mail-qv1-xf2d.google.com [IPv6:2607:f8b0:4864:20::f2d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0952C3D920 for ; Mon, 28 Nov 2022 10:07:28 -0800 (PST) Received: by mail-qv1-xf2d.google.com with SMTP id u10so2352277qvp.4 for ; Mon, 28 Nov 2022 10:07:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20210112.gappssmtp.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=qOn1PUQxMJCGp3yaVSqPelmstrqLulCIcemPhpNVzdU=; b=4tQKhAmlgqJFeuomcyGn+GyBJTF3TXNZND3NJQrXoNmD8007Bxyc2BZx8oFt4hfIHz DcHStdfL2bgNWjZTK0ib7Ohf9zqFJ+82s5fozbOWW3QWW289UEPRstHix3b0fcK1+fOF ud9ZTqyADxWT2iqEUic/48O1fS33VMxPFrU3i1iO79Mwue4ofOjSA5Yhg7qSYgKtqIYq c0XG+rJLlfayL++XpOTdm2AUjwy3zr8n5NSnhkwYqQPKPXDvpOjCuAvDzEVYTP13jtUi c80FdjP5USS2wz5z/YixX3LluJBQGexKDWu0K9Qraubay+EYw+V5uZ1Iy3idEWE+bCZM Fl8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qOn1PUQxMJCGp3yaVSqPelmstrqLulCIcemPhpNVzdU=; b=0AO01Q9npZeTMli6qasDiyVBaux5ocqHJsz32LewZ/08SUWRJF0Zh0qE59r6mvn6YY 41Ksp4fC937Inwvc+IwLVexL7seYmi5sY30VOWGYaxCngITyWLbZ3ij1xUzRm/mOtGpw eC+mFK5/xoGdjn3EaYZ+VcukA2Bh+0VFcB12/FNRlpXqcLrehDM1Uyt1mxQQWu7arlk6 pUAduTRUWigo3WkRSKPUDy0VFexMFZUYd2m0BquPa4Vjm/EaVjp1kv1DTwxOG0rn8cgx OX6qVQ4e1xYN6KI0fGpEKiUCNd0LnDrR/1iDNieHT3uAozFFv7aagJoH7iw1gHwGJ6NG uAyg== X-Gm-Message-State: ANoB5pmj/a3Z7aRXbpDPJdCpXxhiRLmy5Imwy0YwZ+yN08iAT/wBB17o 8WdKNtV3ELAnYdkLIy6HRaoi9A== X-Google-Smtp-Source: AA0mqf6DaYR4aiOKw2PMYmb+XewvGkHvB6VF27wWN2x2o6DHCI1w5vwLSnOS5O+2XZOdfSZK+r2GVg== X-Received: by 2002:a0c:ff28:0:b0:4b8:6953:aed6 with SMTP id x8-20020a0cff28000000b004b86953aed6mr34287361qvt.47.1669658847054; Mon, 28 Nov 2022 10:07:27 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-9175-2920-760a-79fa.res6.spectrum.com. [2603:7000:c01:2716:9175:2920:760a:79fa]) by smtp.gmail.com with ESMTPSA id u19-20020a37ab13000000b006e99290e83fsm8607554qke.107.2022.11.28.10.07.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:07:26 -0800 (PST) Date: Mon, 28 Nov 2022 13:07:25 -0500 From: Johannes Weiner To: Ivan Babrou Cc: Linux MM , Linux Kernel Network Developers , linux-kernel , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Eric Dumazet , "David S. Miller" , Hideaki YOSHIFUJI , David Ahern , Jakub Kicinski , Paolo Abeni , cgroups@vger.kernel.org, kernel-team Subject: Re: Low TCP throughput due to vmpressure with swap enabled Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Nov 22, 2022 at 05:28:24PM -0800, Ivan Babrou wrote: > On Tue, Nov 22, 2022 at 2:11 PM Ivan Babrou wrote: > > > > On Tue, Nov 22, 2022 at 12:05 PM Johannes Weiner wrote: > > > > > > On Mon, Nov 21, 2022 at 04:53:43PM -0800, Ivan Babrou wrote: > > > > Hello, > > > > > > > > We have observed a negative TCP throughput behavior from the following commit: > > > > > > > > * 8e8ae645249b mm: memcontrol: hook up vmpressure to socket pressure > > > > > > > > It landed back in 2016 in v4.5, so it's not exactly a new issue. > > > > > > > > The crux of the issue is that in some cases with swap present the > > > > workload can be unfairly throttled in terms of TCP throughput. > > > > > > Thanks for the detailed analysis, Ivan. > > > > > > Originally, we pushed back on sockets only when regular page reclaim > > > had completely failed and we were about to OOM. This patch was an > > > attempt to be smarter about it and equalize pressure more smoothly > > > between socket memory, file cache, anonymous pages. > > > > > > After a recent discussion with Shakeel, I'm no longer quite sure the > > > kernel is the right place to attempt this sort of balancing. It kind > > > of depends on the workload which type of memory is more imporant. And > > > your report shows that vmpressure is a flawed mechanism to implement > > > this, anyway. > > > > > > So I'm thinking we should delete the vmpressure thing, and go back to > > > socket throttling only if an OOM is imminent. This is in line with > > > what we do at the system level: sockets get throttled only after > > > reclaim fails and we hit hard limits. It's then up to the users and > > > sysadmin to allocate a reasonable amount of buffers given the overall > > > memory budget. > > > > > > Cgroup accounting, limiting and OOM enforcement is still there for the > > > socket buffers, so misbehaving groups will be contained either way. > > > > > > What do you think? Something like the below patch? > > > > The idea sounds very reasonable to me. I can't really speak for the > > patch contents with any sort of authority, but it looks ok to my > > non-expert eyes. > > > > There were some conflicts when cherry-picking this into v5.15. I think > > the only real one was for the "!sc->proactive" condition not being > > present there. For the rest I just accepted the incoming change. > > > > I'm going to be away from my work computer until December 5th, but > > I'll try to expedite my backported patch to a production machine today > > to confirm that it makes the difference. If I can get some approvals > > on my internal PRs, I should be able to provide the results by EOD > > tomorrow. > > I tried the patch and something isn't right here. Thanks for giving it a sping. > With the patch applied I'm capped at ~120MB/s, which is a symptom of a > clamped window. > > I can't find any sockets with memcg->socket_pressure = 1, but at the > same time I only see the following rcv_ssthresh assigned to sockets: Hm, I don't see how socket accounting would alter the network behavior other than through socket_pressure=1. How do you look for that flag? If you haven't yet done something comparable, can you try with tracing to rule out sampling errors? diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 066166aebbef..134b623bee6a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -7211,6 +7211,7 @@ bool mem_cgroup_charge_skmem(struct mem_cgroup *memcg, unsigned int nr_pages, goto success; } memcg->socket_pressure = 1; + trace_printk("skmem charge failed nr_pages=%u gfp=%pGg\n", nr_pages, &gfp_mask); if (gfp_mask & __GFP_NOFAIL) { try_charge(memcg, gfp_mask, nr_pages); goto success;