From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D603C433FE for ; Fri, 10 Sep 2021 00:43:56 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B7F8160C40 for ; Fri, 10 Sep 2021 00:43:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B7F8160C40 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 2D2756B0071; Thu, 9 Sep 2021 20:43:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 281D96B0072; Thu, 9 Sep 2021 20:43:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19B4C6B0073; Thu, 9 Sep 2021 20:43:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0101.hostedemail.com [216.40.44.101]) by kanga.kvack.org (Postfix) with ESMTP id 05A2B6B0071 for ; Thu, 9 Sep 2021 20:43:55 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 522F21802144A for ; Fri, 10 Sep 2021 00:43:54 +0000 (UTC) X-FDA: 78569816388.15.8EFF1B3 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf26.hostedemail.com (Postfix) with ESMTP id 193A020019C6 for ; Fri, 10 Sep 2021 00:43:53 +0000 (UTC) Received: by mail-lj1-f172.google.com with SMTP id q21so417523ljj.6 for ; Thu, 09 Sep 2021 17:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=7uuVd30s2WHbNN7lmdDhnwmn6gKKOjm5ZXOnnVPJOAc=; b=gHtlqt+DwPRv1ejJ4sOoZNXBRm7rh4PT1EOgN6JuPS+IpgiuiwPgrOsidYVLyj3dLp PMG/QeOd1hk+EwhqyFBdc6dT7F/fLM0/2aYgcriLKtO7Mj9jft8z8qN1+rbjMq++qbLp CDaqK3Va1LGh9NGm0a7cgsmKOkz/hd/8/r0RXmYCDbP2/dcQ9uRwdbnk//d8ta8fCxqM +A/PWqa8+YYIpm8uSrdpE4hrzS3pNXXjHK7OknZ8zBNYEWiRpLhho8In6csV3u1++gqU Lvc+4pVAbxzSXQRVWDDBobX9oZPxBo4aPEa1dFn+QIjEbQ2b+g5t3Kg+zYr+eta2QMvl sVCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=7uuVd30s2WHbNN7lmdDhnwmn6gKKOjm5ZXOnnVPJOAc=; b=T8TIOEGnNeVelgmI1/0C2/j+nqMKPMjjV7ABuXPdd14mNK7eeRXz8tJEeq1ZK82e2/ I0OhPGCT7gB+NxIVxeoXcxoI/39JXUlEXvYTWOJ9SPEexb1aU/IeRFoLKWqtVmHJkOBm 6QIb4VG07izc5ZfWK7HajbmV0M0sD8rHohC9GDcmG5ssU4wnDy/G7ckdnhftDGfUJsqW GRc6cSUnwNgcATcraFK4ROF4dmqpPyG7vVMdn2pE2/Xf0s0q7PddHf2Uxdd2WjvVKlG7 S5b8JAwk0YcvP+8hw0KDBX6cqySwqxrngMF4TOTnS7zgeI1qXIHipdH+5BVkL1tZsGIn +BLw== X-Gm-Message-State: AOAM530bB9/E+wzDAKPBNwR0wNZEmHHS4dp7mPlJIaQb50sW0dEJOsRH AMkKbIf+Kzu/BPw2liLztEsN28oAwxFIRUlVRja6Lw== X-Google-Smtp-Source: ABdhPJzeCz6w0q2U+ulvoyoJWdHD+nGGyY4JW/MWaqGjcHA3gGLSDaljkucvPJBwYPGeR5s6gbQ6arp4upt1qhQTYN8= X-Received: by 2002:a2e:a363:: with SMTP id i3mr2168651ljn.86.1631234632179; Thu, 09 Sep 2021 17:43:52 -0700 (PDT) MIME-Version: 1.0 References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> In-Reply-To: <20210907033000.GA88160@shbuild999.sh.intel.com> From: Shakeel Butt Date: Thu, 9 Sep 2021 17:43:40 -0700 Message-ID: Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression To: Feng Tang Cc: kernel test robot , Andrew Morton , 0day robot , Marek Szyprowski , Hillf Danton , Huang Ying , Johannes Weiner , Michal Hocko , "Michal Koutn??" , Muchun Song , Roman Gushchin , Tejun Heo , LKML , lkp@lists.01.org, Xing Zhengjun , Linux MM , mm-commits@vger.kernel.org, Linus Torvalds Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: ftfcxthmx59k1mzcr95wcbsjd4nmkiwb Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=gHtlqt+D; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of shakeelb@google.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=shakeelb@google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 193A020019C6 X-HE-Tag: 1631234633-807030 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Sep 6, 2021 at 8:30 PM Feng Tang wrote: > > Hi Shakeel, > > On Sun, Sep 05, 2021 at 03:15:46PM -0700, Shakeel Butt wrote: > > On Sun, Sep 5, 2021 at 5:27 AM kernel test robot wrote: > [...] > > > ========================================================================================= > > > compiler/cpufreq_governor/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase/ucode: > > > gcc-9/performance/1BRD_48G/xfs/x86_64-rhel-8.3/3000/debian-10.4-x86_64-20200603.cgz/lkp-icl-2sp2/disk_rr/aim7/0xd000280 > > > > > > commit: > > > 3c28c7680e ("memcg: switch lruvec stats to rstat") > > > 45208c9105 ("memcg: infrastructure to flush memcg stats") > > > > I am looking into this. I was hoping we have resolution for [1] as > > these patches touch similar data structures. > > > > [1] https://lore.kernel.org/all/20210811031734.GA5193@xsang-OptiPlex-9020/T/#u > > I tried 2 debug methods for that 36.4% vm-scalability regression: > > 1. Disable the HW cache prefetcher, no effect on this case > 2. relayout and add padding to 'struct cgroup_subsys_state', reduce > the regression to 3.1% > Thanks Feng but it seems like the issue for this commit is different. Rearranging the layout didn't help. Actually the cause of slowdown is the call to queue_work() inside __mod_memcg_lruvec_state(). At the moment, queue_work() is called after 32 updates. I changed it to 128 and the slowdown of will-it-scale:page_fault[1|2|3] halved (from around 10% to 5%). I am unable to run reaim or will-it-scale:fallocate2 as I was getting weird errors. Feng, is it possible for you to run these benchmarks with the change (basically changing MEMCG_CHARGE_BATCH to 128 in the if condition before queue_work() inside __mod_memcg_lruvec_state())? For the formal patch/fix, I will write down a better explanation on what should be the batch size. thanks, Shakeel