From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C31AC433EF for ; Thu, 26 May 2022 00:38:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 401648D0003; Wed, 25 May 2022 20:38:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 389C88D0002; Wed, 25 May 2022 20:38:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 201A78D0003; Wed, 25 May 2022 20:38:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0F5ED8D0002 for ; Wed, 25 May 2022 20:38:59 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DEC5033D84 for ; Thu, 26 May 2022 00:38:58 +0000 (UTC) X-FDA: 79506034356.02.390E9E5 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 3771240027 for ; Thu, 26 May 2022 00:38:51 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id i18so391187pfk.7 for ; Wed, 25 May 2022 17:38:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=nGnImQMrxcKuAMPfn2cnaf9VK34koPZI6BdWd8L/dvJkOmKEczFMWc8Yx7hsQFZeVL 9KLxd0tBqQk1/z15iyaTJQGyj+rpJcqO/oigF6/cI0x8g+X/Xo/VBfVgAonu8k6z7tAw rX3SONGBqur0M8KowUWh/wmc3aYT+yOgmGgvthbF1GbnQhkOPFYSCTfv0l7sg5kj2jls D/lYOdddeRohuY3mRJGYNVA5mDFRmRi5K+xU9HMyUH4O1+TPylZTZEwi6U0ynqv/r7ZQ PEcsEwLY3XMn7/TlPZPqXhp0uzspbOfW5miRUT3ck3ZbCGUmofEeJzc2ijyz7b5LJr0L 9x9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=Mqc8eJnx+Zq34oUFXZ4wThihqRidE+gSRz3v0cCeIx0+L1jyCSFPI+Cr/sOi21DH8/ JO7QayK/c8l4Zpzbe2rby9xHpO9v6H4tXFUqjx/B5SN3FdvJI+VYQ/Wl35YM4zCAdRkn q1iktoIIa1vAfp6PaT1VZnTaRu0p19jChnGAX9Yzc25JpqjrxB8my1QgVYRSWzKvhImo taFzXPWRcc/wcL0RzBj/ADp9JOE2JMfBGHfQFbJnfn7hpYse8qcSiH0KATYjkJMsa+CF 0siebHYtWdCJ+q6Flj2kosJp5aBYQsEbmt7TfqoG6VDEO0TSu8uWK7yGyTQuAZqqT5j+ T09Q== X-Gm-Message-State: AOAM531XNhLLFwx5qdBY/xn6f6Rch6k4VDvFpcBoD/JcuLywL7l1De6p C+ZXqnqdI9RwoFpwGihqXY2vPA== X-Google-Smtp-Source: ABdhPJxQ2PEcWk8DK3E2AzHueuX2Pq6jI8vVF9DbxayvJeVzlRCPUepbJ8f3PDiOx87ktlbbZyxBrw== X-Received: by 2002:a62:1413:0:b0:518:4259:200e with SMTP id 19-20020a621413000000b005184259200emr34153334pfu.41.1653525537153; Wed, 25 May 2022 17:38:57 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id bh8-20020a056a02020800b003f5cc9c31e2sm141353pgb.38.2022.05.25.17.38.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 17:38:56 -0700 (PDT) Date: Thu, 26 May 2022 00:38:52 +0000 From: Sean Christopherson To: Johannes Weiner Cc: Yosry Ahmed , Shakeel Butt , Marc Zyngier , Tejun Heo , Zefan Li , James Morse , Alexandru Elisei , Suzuki K Poulose , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Andrew Morton , Michal Hocko , Roman Gushchin , Oliver Upton , Cgroups , Linux Kernel Mailing List , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, Linux-MM Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3771240027 X-Stat-Signature: 8tzeskyu9dnezbyjbgqqg9g8th8d9pgy X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=nGnImQMr; spf=pass (imf11.hostedemail.com: domain of seanjc@google.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=seanjc@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1653525531-294775 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 25, 2022, Johannes Weiner wrote: > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > I don't have enough context to say whether we should piggyback KVM MMU > > pages to the existing NR_PAGETABLE item, but from a high level it > > seems like it would be more helpful if they are a separate stat. > > Anyway, I am willing to go with whatever Sean thinks is best. > > Somebody should work this out and put it into a changelog. It's > permanent ABI. After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to allocate a large number of page tables even though the guest isn't accessing that much memory. But, someone would need to either understand how KVM works to make that connection, or know (or be told) to go look at KVM's stats if they're running VMs to better decipher the stats. And even in the little bit of time I played with this, I found having nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very informative. E.g. when backing a VM with THP versus HugeTLB, nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an order of a magnitude higher with THP. I'm guessing the THP behavior is due to something triggering DoubleMap, but now I want to find out why that's happening. So while I'm pretty sure a clever user could glean the same info by cross-referencing NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the very least prove to be helpful for understanding tradeoffs between VM backing types, and likely even steer folks towards potential optimizations. Baseline: # grep page_table /proc/vmstat nr_page_table_pages 2830 nr_secondary_page_table_pages 0 THP: # grep page_table /proc/vmstat nr_page_table_pages 7584 nr_secondary_page_table_pages 140 HugeTLB: # grep page_table /proc/vmstat nr_page_table_pages 3153 nr_secondary_page_table_pages 153