From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02DAFC433EF for ; Fri, 10 Sep 2021 04:18:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7AC00610C7 for ; Fri, 10 Sep 2021 04:18:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7AC00610C7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 743556B0071; Fri, 10 Sep 2021 00:18:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CAFC6B0072; Fri, 10 Sep 2021 00:18:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 592A76B0073; Fri, 10 Sep 2021 00:18:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 45B056B0071 for ; Fri, 10 Sep 2021 00:18:08 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DB55C2DD64 for ; Fri, 10 Sep 2021 04:18:07 +0000 (UTC) X-FDA: 78570356214.38.B8C0604 Received: from mail-lj1-f169.google.com (mail-lj1-f169.google.com [209.85.208.169]) by imf21.hostedemail.com (Postfix) with ESMTP id A035ED02ADCC for ; Fri, 10 Sep 2021 04:18:07 +0000 (UTC) Received: by mail-lj1-f169.google.com with SMTP id f2so1095490ljn.1 for ; Thu, 09 Sep 2021 21:18:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ACI+kaI5QB65LpV5tns5Bzp71UPPaBwusSSeY5hjw/M=; b=UN22a8qX+iznaU8h3hrhCbhSAThy2z/4cZQRhJa9HP/9GbpEhHGrq+uTnqDDOxToOl iPANYQj68YQgj2rIiuGYjsuhvIv/6D7QguvaSUnjb01ysyVuaoIr1sOnstp1f693Pgup 6VPPEzYfhTLi0OciVmwli19okmRb7mBTn546HKQ/eXlUZybbN0YCzC8nIsqPH0h7FSLI Zj3EC+vn16Exl4TI0NupZZxwri8Ejz1eCyAGZWSLBJhcmsmHQ+EgTiJfxRY+ZXRvs9m6 18R2j4i0rMwrfy6CqWNOUI2vpEQ0wD5TAVYHP10kxW6fQmEM6JKxDynqS1J17HN2A/Fk ZNBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ACI+kaI5QB65LpV5tns5Bzp71UPPaBwusSSeY5hjw/M=; b=tuO887X4riHO/kQ96+nv7ve4/0+3kr9wgrxvnc5PQxD30tYC3JFGDHybEr3GsHTx1F ea2KNwRORDblB1JJc8PapjytTvqEDrKY+xrLyS6o7ZNUN2G5lK8DXkTFgItIQxFrb5Xb mx09zSeZTEt+7kl5U63OXxLAx6coguupywEPxQN3rVUj3jqPLQ8PUAem3Ilm90ddx1H1 0LuUaizTQ4+ZG/w3PG+eyoXEJr1g0apscz640K2LBN38sOYmeM/ee9Nqqt3nH5eiiqkm o7pCOqGoV3kio31SAwBm4TlL+vq8yAP4M5gOXWDPhPy2vfK9pnlNw4MDVgMNqNjkWzLv ceWg== X-Gm-Message-State: AOAM533W2yMoLtW5+vkv1cb5jTG+DDoKBtgGsGb9wg1zRk4R596ap/xW 0CtgVRieVhw5w+mTpueZuG0J/Sd5f+XVtAsAUtH88w== X-Google-Smtp-Source: ABdhPJwNTYErfMkcSAQp56nNRqGOIapdEJwjDN8c5Eor19eb/cvIQkSiwbCb3MYkU/9o7GWKPEXEoe4XcyxbDtjvZ7I= X-Received: by 2002:a2e:964b:: with SMTP id z11mr2542436ljh.91.1631247485805; Thu, 09 Sep 2021 21:18:05 -0700 (PDT) MIME-Version: 1.0 References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> <20210910010842.GA94434@shbuild999.sh.intel.com> <20210910023415.GB94434@shbuild999.sh.intel.com> In-Reply-To: <20210910023415.GB94434@shbuild999.sh.intel.com> From: Shakeel Butt Date: Thu, 9 Sep 2021 21:17:54 -0700 Message-ID: Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression To: Feng Tang Cc: kernel test robot , Andrew Morton , 0day robot , Marek Szyprowski , Hillf Danton , Huang Ying , Johannes Weiner , Michal Hocko , "Michal Koutn??" , Muchun Song , Roman Gushchin , Tejun Heo , LKML , lkp@lists.01.org, Xing Zhengjun , Linux MM , mm-commits@vger.kernel.org, Linus Torvalds Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A035ED02ADCC Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=UN22a8qX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of shakeelb@google.com designates 209.85.208.169 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Stat-Signature: tpd69mbfjike6jjau5dmbjizwcmb5ims X-HE-Tag: 1631247487-119184 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Sep 9, 2021 at 7:34 PM Feng Tang wrote: > > On Thu, Sep 09, 2021 at 06:19:06PM -0700, Shakeel Butt wrote: > [...] > > > > > > I am looking into this. I was hoping we have resolution for [1] as > > > > > > these patches touch similar data structures. > > > > > > > > > > > > [1] https://lore.kernel.org/all/20210811031734.GA5193@xsang-OptiPlex-9020/T/#u > > > > > > > > > > I tried 2 debug methods for that 36.4% vm-scalability regression: > > > > > > > > > > 1. Disable the HW cache prefetcher, no effect on this case > > > > > 2. relayout and add padding to 'struct cgroup_subsys_state', reduce > > > > > the regression to 3.1% > > > > > > > > > > > > > Thanks Feng but it seems like the issue for this commit is different. > > > > Rearranging the layout didn't help. Actually the cause of slowdown is > > > > the call to queue_work() inside __mod_memcg_lruvec_state(). > > > > > > > > At the moment, queue_work() is called after 32 updates. I changed it > > > > to 128 and the slowdown of will-it-scale:page_fault[1|2|3] halved > > > > (from around 10% to 5%). I am unable to run reaim or > > > > will-it-scale:fallocate2 as I was getting weird errors. > > > > > > > > Feng, is it possible for you to run these benchmarks with the change > > > > (basically changing MEMCG_CHARGE_BATCH to 128 in the if condition > > > > before queue_work() inside __mod_memcg_lruvec_state())? > > > > > > When I checked this, I tried different changes, including this batch > > > number change :), but it didn't recover the regression (the regression > > > is slightly reduced to about 12%) > [...] > > > > Another change we can try is to remove this specific queue_work() > > altogether because this is the only significant change for the > > workload. That will give us the base performance number. If that also > > has regression then there are more issues to debug. Thanks a lot for > > your help. > > I just tested with patch removing the queue_work() in __mod_memcg_lruvec_state(), > and the regression is gone. Thanks again for confirming this. I will follow this lead and see how to improve this.