From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Date: Thu, 26 May 2022 00:38:52 +0000 Message-ID: References: Mime-Version: 1.0 Return-path: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=nGnImQMrxcKuAMPfn2cnaf9VK34koPZI6BdWd8L/dvJkOmKEczFMWc8Yx7hsQFZeVL 9KLxd0tBqQk1/z15iyaTJQGyj+rpJcqO/oigF6/cI0x8g+X/Xo/VBfVgAonu8k6z7tAw rX3SONGBqur0M8KowUWh/wmc3aYT+yOgmGgvthbF1GbnQhkOPFYSCTfv0l7sg5kj2jls D/lYOdddeRohuY3mRJGYNVA5mDFRmRi5K+xU9HMyUH4O1+TPylZTZEwi6U0ynqv/r7ZQ PEcsEwLY3XMn7/TlPZPqXhp0uzspbOfW5miRUT3ck3ZbCGUmofEeJzc2ijyz7b5LJr0L 9x9Q== Content-Disposition: inline In-Reply-To: List-ID: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Johannes Weiner Cc: Yosry Ahmed , Shakeel Butt , Marc Zyngier , Tejun Heo , Zefan Li , James Morse , Alexandru Elisei , Suzuki K Poulose , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Andrew Morton , Michal Hocko , Roman Gushchin , Oliver Upton , Cgroups , Linux Kernel Mailing List , linux-arm-kernel@lists.infradead.or On Wed, May 25, 2022, Johannes Weiner wrote: > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > I don't have enough context to say whether we should piggyback KVM MMU > > pages to the existing NR_PAGETABLE item, but from a high level it > > seems like it would be more helpful if they are a separate stat. > > Anyway, I am willing to go with whatever Sean thinks is best. > > Somebody should work this out and put it into a changelog. It's > permanent ABI. After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to allocate a large number of page tables even though the guest isn't accessing that much memory. But, someone would need to either understand how KVM works to make that connection, or know (or be told) to go look at KVM's stats if they're running VMs to better decipher the stats. And even in the little bit of time I played with this, I found having nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very informative. E.g. when backing a VM with THP versus HugeTLB, nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an order of a magnitude higher with THP. I'm guessing the THP behavior is due to something triggering DoubleMap, but now I want to find out why that's happening. So while I'm pretty sure a clever user could glean the same info by cross-referencing NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the very least prove to be helpful for understanding tradeoffs between VM backing types, and likely even steer folks towards potential optimizations. Baseline: # grep page_table /proc/vmstat nr_page_table_pages 2830 nr_secondary_page_table_pages 0 THP: # grep page_table /proc/vmstat nr_page_table_pages 7584 nr_secondary_page_table_pages 140 HugeTLB: # grep page_table /proc/vmstat nr_page_table_pages 3153 nr_secondary_page_table_pages 153 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4759C433F5 for ; Thu, 26 May 2022 00:39:02 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4EEAB4B2F3; Wed, 25 May 2022 20:39:02 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wqkpvBgNGcQj; Wed, 25 May 2022 20:39:01 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 323364B2A7; Wed, 25 May 2022 20:39:01 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7D90C4B210 for ; Wed, 25 May 2022 20:38:59 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id wpaN3zbItIu7 for ; Wed, 25 May 2022 20:38:58 -0400 (EDT) Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 5EB354B20F for ; Wed, 25 May 2022 20:38:58 -0400 (EDT) Received: by mail-pf1-f170.google.com with SMTP id y1so395269pfr.6 for ; Wed, 25 May 2022 17:38:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=nGnImQMrxcKuAMPfn2cnaf9VK34koPZI6BdWd8L/dvJkOmKEczFMWc8Yx7hsQFZeVL 9KLxd0tBqQk1/z15iyaTJQGyj+rpJcqO/oigF6/cI0x8g+X/Xo/VBfVgAonu8k6z7tAw rX3SONGBqur0M8KowUWh/wmc3aYT+yOgmGgvthbF1GbnQhkOPFYSCTfv0l7sg5kj2jls D/lYOdddeRohuY3mRJGYNVA5mDFRmRi5K+xU9HMyUH4O1+TPylZTZEwi6U0ynqv/r7ZQ PEcsEwLY3XMn7/TlPZPqXhp0uzspbOfW5miRUT3ck3ZbCGUmofEeJzc2ijyz7b5LJr0L 9x9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=G5OTwozVMV/gH0v9JmmXJtEOyWpitRr5Rdp2sNYUTxevapXntuDtWbVVSU/uA8L1z9 DU1rHWwBvnuzA5SewiH5iwSo4NiojvoLPGmI1eawdSD1DHZdnQOJ+ESXCQLIRjAlXO9/ MIpVKt5swVXLAfsyaoN61cF/WYYLrMLJmyJtSWt4q3UHgoL3TOkbaS3LU5xaFPftScOP cCq2a5NeNF9plE9cuyvIOJjKCiin/MDEozvD5Fet2RJFmvWHSPCfE7J0X4CjOtcX1Jw5 WedGu8x2jv9dsDbQaRfwT2fzxIRbC5euNkrgY003Kwno9zg8ZCL0hrdw7gWZnUHgV6OS R7TA== X-Gm-Message-State: AOAM533UbcYWD7uOfb2yvb9tmjf9gM1pe4ySqr+tl8ahley97ELVcm82 qu2XQcST2IjDKgUKMPKhBWIWMw== X-Google-Smtp-Source: ABdhPJxQ2PEcWk8DK3E2AzHueuX2Pq6jI8vVF9DbxayvJeVzlRCPUepbJ8f3PDiOx87ktlbbZyxBrw== X-Received: by 2002:a62:1413:0:b0:518:4259:200e with SMTP id 19-20020a621413000000b005184259200emr34153334pfu.41.1653525537153; Wed, 25 May 2022 17:38:57 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id bh8-20020a056a02020800b003f5cc9c31e2sm141353pgb.38.2022.05.25.17.38.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 17:38:56 -0700 (PDT) Date: Thu, 26 May 2022 00:38:52 +0000 From: Sean Christopherson To: Johannes Weiner Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: Wanpeng Li , kvm@vger.kernel.org, Roman Gushchin , Michal Hocko , Yosry Ahmed , Linux-MM , Zefan Li , kvmarm@lists.cs.columbia.edu, Marc Zyngier , Joerg Roedel , Shakeel Butt , Cgroups , Andrew Morton , linux-arm-kernel@lists.infradead.org, Jim Mattson , Linux Kernel Mailing List , Tejun Heo , Paolo Bonzini , Vitaly Kuznetsov X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Wed, May 25, 2022, Johannes Weiner wrote: > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > I don't have enough context to say whether we should piggyback KVM MMU > > pages to the existing NR_PAGETABLE item, but from a high level it > > seems like it would be more helpful if they are a separate stat. > > Anyway, I am willing to go with whatever Sean thinks is best. > > Somebody should work this out and put it into a changelog. It's > permanent ABI. After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to allocate a large number of page tables even though the guest isn't accessing that much memory. But, someone would need to either understand how KVM works to make that connection, or know (or be told) to go look at KVM's stats if they're running VMs to better decipher the stats. And even in the little bit of time I played with this, I found having nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very informative. E.g. when backing a VM with THP versus HugeTLB, nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an order of a magnitude higher with THP. I'm guessing the THP behavior is due to something triggering DoubleMap, but now I want to find out why that's happening. So while I'm pretty sure a clever user could glean the same info by cross-referencing NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the very least prove to be helpful for understanding tradeoffs between VM backing types, and likely even steer folks towards potential optimizations. Baseline: # grep page_table /proc/vmstat nr_page_table_pages 2830 nr_secondary_page_table_pages 0 THP: # grep page_table /proc/vmstat nr_page_table_pages 7584 nr_secondary_page_table_pages 140 HugeTLB: # grep page_table /proc/vmstat nr_page_table_pages 3153 nr_secondary_page_table_pages 153 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 314DFC433EF for ; Thu, 26 May 2022 00:40:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jtx4djPqoql6wYif2NCEE62BB2JdH4y3rbTlJytUgcI=; b=sMMO8MeTMqaiex o2XJRIv9fa17U28gXf0ATYGQaVuUuX6r8g1APVGB4DTGL556j+gybqpm7X6/OzldTaq6+qXDeD+cj C67I+xEKmKYrLhGstLxaFIvuQG2S9Y/BORIPA3Llj4t5+8Hc6kYF7jgpPgeNxZns6BA3ZPr2jqPT0 bQH/DBB0cGiCNeRjKq5ZacH/AtrbLRSijej5YxUMqyyMTZQlcE3gypB3Oy9V49peJccsdvBuMpg96 7SpbjzYlj/cQb/DinKS2DsG/sdwp6ylp0L+937xbMhzm8YUorCJMrOZfLyOPX77Hs+U3xpieIj2D6 /O6wxStd09gQI02bQMkg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nu1Wr-00D4pX-Om; Thu, 26 May 2022 00:39:05 +0000 Received: from mail-pf1-x435.google.com ([2607:f8b0:4864:20::435]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nu1Wo-00D4nR-30 for linux-arm-kernel@lists.infradead.org; Thu, 26 May 2022 00:39:03 +0000 Received: by mail-pf1-x435.google.com with SMTP id c14so409482pfn.2 for ; Wed, 25 May 2022 17:38:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=nGnImQMrxcKuAMPfn2cnaf9VK34koPZI6BdWd8L/dvJkOmKEczFMWc8Yx7hsQFZeVL 9KLxd0tBqQk1/z15iyaTJQGyj+rpJcqO/oigF6/cI0x8g+X/Xo/VBfVgAonu8k6z7tAw rX3SONGBqur0M8KowUWh/wmc3aYT+yOgmGgvthbF1GbnQhkOPFYSCTfv0l7sg5kj2jls D/lYOdddeRohuY3mRJGYNVA5mDFRmRi5K+xU9HMyUH4O1+TPylZTZEwi6U0ynqv/r7ZQ PEcsEwLY3XMn7/TlPZPqXhp0uzspbOfW5miRUT3ck3ZbCGUmofEeJzc2ijyz7b5LJr0L 9x9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=mY0hwfJC+XHGjC/tVIgfpcY0lDAZ9Q5cMcLr0q492e36WC9xfa2TLOvpfScKnKkvvr wpmMY9O7bmxVHYwPj0rrkf02J7/uI5ddofk2uMA/5/9KEl68CHp76tS9oDewIARJgosG 5Iejo1f/8T/e4trUUOjtZLrDNXlC2/OQI6jjKtIJ84sejTuPr06GrOu7MCC3C+8q8Ukw EsmSKi5KvEBZFrh/jFqjpF3SGeQlZGhxIprFnVFLuTl/YUlOuD5baqHLW2xTX1AM2/q4 d73OlmJQ6MNyBuukCg10ar2CepeSF2wjvekMa+dWwNCV7QWNKjbI++1t1Ez38OYttlWQ a2xA== X-Gm-Message-State: AOAM531PBaGEbmVS3xOyUssMTbRBmiOFp45FJA1JhzPoX2/Z5ePXVuC8 hlX4K+5s6UJ+CYOZDUqLBHfjcg== X-Google-Smtp-Source: ABdhPJxQ2PEcWk8DK3E2AzHueuX2Pq6jI8vVF9DbxayvJeVzlRCPUepbJ8f3PDiOx87ktlbbZyxBrw== X-Received: by 2002:a62:1413:0:b0:518:4259:200e with SMTP id 19-20020a621413000000b005184259200emr34153334pfu.41.1653525537153; Wed, 25 May 2022 17:38:57 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id bh8-20020a056a02020800b003f5cc9c31e2sm141353pgb.38.2022.05.25.17.38.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 17:38:56 -0700 (PDT) Date: Thu, 26 May 2022 00:38:52 +0000 From: Sean Christopherson To: Johannes Weiner Cc: Yosry Ahmed , Shakeel Butt , Marc Zyngier , Tejun Heo , Zefan Li , James Morse , Alexandru Elisei , Suzuki K Poulose , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Andrew Morton , Michal Hocko , Roman Gushchin , Oliver Upton , Cgroups , Linux Kernel Mailing List , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, Linux-MM Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220525_173902_172775_A78747AC X-CRM114-Status: GOOD ( 20.05 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, May 25, 2022, Johannes Weiner wrote: > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > I don't have enough context to say whether we should piggyback KVM MMU > > pages to the existing NR_PAGETABLE item, but from a high level it > > seems like it would be more helpful if they are a separate stat. > > Anyway, I am willing to go with whatever Sean thinks is best. > > Somebody should work this out and put it into a changelog. It's > permanent ABI. After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to allocate a large number of page tables even though the guest isn't accessing that much memory. But, someone would need to either understand how KVM works to make that connection, or know (or be told) to go look at KVM's stats if they're running VMs to better decipher the stats. And even in the little bit of time I played with this, I found having nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very informative. E.g. when backing a VM with THP versus HugeTLB, nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an order of a magnitude higher with THP. I'm guessing the THP behavior is due to something triggering DoubleMap, but now I want to find out why that's happening. So while I'm pretty sure a clever user could glean the same info by cross-referencing NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the very least prove to be helpful for understanding tradeoffs between VM backing types, and likely even steer folks towards potential optimizations. Baseline: # grep page_table /proc/vmstat nr_page_table_pages 2830 nr_secondary_page_table_pages 0 THP: # grep page_table /proc/vmstat nr_page_table_pages 7584 nr_secondary_page_table_pages 140 HugeTLB: # grep page_table /proc/vmstat nr_page_table_pages 3153 nr_secondary_page_table_pages 153 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE995C433EF for ; Thu, 26 May 2022 00:39:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344576AbiEZAjC (ORCPT ); Wed, 25 May 2022 20:39:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344599AbiEZAi7 (ORCPT ); Wed, 25 May 2022 20:38:59 -0400 Received: from mail-pf1-x435.google.com (mail-pf1-x435.google.com [IPv6:2607:f8b0:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 072B9B41F1 for ; Wed, 25 May 2022 17:38:57 -0700 (PDT) Received: by mail-pf1-x435.google.com with SMTP id b135so370277pfb.12 for ; Wed, 25 May 2022 17:38:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=nGnImQMrxcKuAMPfn2cnaf9VK34koPZI6BdWd8L/dvJkOmKEczFMWc8Yx7hsQFZeVL 9KLxd0tBqQk1/z15iyaTJQGyj+rpJcqO/oigF6/cI0x8g+X/Xo/VBfVgAonu8k6z7tAw rX3SONGBqur0M8KowUWh/wmc3aYT+yOgmGgvthbF1GbnQhkOPFYSCTfv0l7sg5kj2jls D/lYOdddeRohuY3mRJGYNVA5mDFRmRi5K+xU9HMyUH4O1+TPylZTZEwi6U0ynqv/r7ZQ PEcsEwLY3XMn7/TlPZPqXhp0uzspbOfW5miRUT3ck3ZbCGUmofEeJzc2ijyz7b5LJr0L 9x9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=7P8MdhBUHeNYERj02eaiVlo3+1akuXNgpyLjujYYpb0=; b=fLhR/43EskDNA1Up5Q4gqW7+n1Q/oZpzMre4qnPcLhSZi2acgSrYuZHXzcTMYVhDUX wXLmZVpUyuNePRWKUtldr/UjtUOrw2wky3QmY3PAoHwsfU1oFj0pJc2ngdcV8HUGpXXJ WTOu2VpSovh9UEcCqD9yvhJdPY8k80jLaNo33iBVSQD/dmAPiB41W2Senct+mXv6aEj9 1U2XXGfJmzSC9NeR9zAO+1KudGs/CAY9vaPRak783cElZ2oWdtLDL2XXMN5+41swuK7Y kqlgx2P8OZXtu9WbLDdJIsKZaztzPlN5ayLO3niaAnCQdT8a0yLzJPGs4EyhF53aKtRs DD2A== X-Gm-Message-State: AOAM530sDn6kJbPz/mDkxyF93veaoCuMpB1ekIM91n174uL34vAQvAnn qZnQ2RlWs//P2Tfgr9l8IVh8Ag== X-Google-Smtp-Source: ABdhPJxQ2PEcWk8DK3E2AzHueuX2Pq6jI8vVF9DbxayvJeVzlRCPUepbJ8f3PDiOx87ktlbbZyxBrw== X-Received: by 2002:a62:1413:0:b0:518:4259:200e with SMTP id 19-20020a621413000000b005184259200emr34153334pfu.41.1653525537153; Wed, 25 May 2022 17:38:57 -0700 (PDT) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id bh8-20020a056a02020800b003f5cc9c31e2sm141353pgb.38.2022.05.25.17.38.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 17:38:56 -0700 (PDT) Date: Thu, 26 May 2022 00:38:52 +0000 From: Sean Christopherson To: Johannes Weiner Cc: Yosry Ahmed , Shakeel Butt , Marc Zyngier , Tejun Heo , Zefan Li , James Morse , Alexandru Elisei , Suzuki K Poulose , Paolo Bonzini , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , Andrew Morton , Michal Hocko , Roman Gushchin , Oliver Upton , Cgroups , Linux Kernel Mailing List , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, Linux-MM Subject: Re: [PATCH v4 1/4] mm: add NR_SECONDARY_PAGETABLE to count secondary page table uses. Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Wed, May 25, 2022, Johannes Weiner wrote: > On Tue, May 24, 2022 at 03:31:52PM -0700, Yosry Ahmed wrote: > > I don't have enough context to say whether we should piggyback KVM MMU > > pages to the existing NR_PAGETABLE item, but from a high level it > > seems like it would be more helpful if they are a separate stat. > > Anyway, I am willing to go with whatever Sean thinks is best. > > Somebody should work this out and put it into a changelog. It's > permanent ABI. After a lot of waffling, my vote is to add a dedicated NR_SECONDARY_PAGETABLE. It's somewhat redundant from a KVM perspective, as NR_SECONDARY_PAGETABLE will scale with KVM's per-VM pages_{4k,2m,1g} stats unless the guest is doing something bizarre, e.g. accessing only 4kb chunks of 2mb pages so that KVM is forced to allocate a large number of page tables even though the guest isn't accessing that much memory. But, someone would need to either understand how KVM works to make that connection, or know (or be told) to go look at KVM's stats if they're running VMs to better decipher the stats. And even in the little bit of time I played with this, I found having nr_page_table_pages side-by-side with nr_secondary_page_table_pages to be very informative. E.g. when backing a VM with THP versus HugeTLB, nr_secondary_page_table_pages is roughly the same, but nr_page_table_pages is an order of a magnitude higher with THP. I'm guessing the THP behavior is due to something triggering DoubleMap, but now I want to find out why that's happening. So while I'm pretty sure a clever user could glean the same info by cross-referencing NR_PAGETABLE stats with KVM stats, I think having NR_SECONDARY_PAGETABLE will at the very least prove to be helpful for understanding tradeoffs between VM backing types, and likely even steer folks towards potential optimizations. Baseline: # grep page_table /proc/vmstat nr_page_table_pages 2830 nr_secondary_page_table_pages 0 THP: # grep page_table /proc/vmstat nr_page_table_pages 7584 nr_secondary_page_table_pages 140 HugeTLB: # grep page_table /proc/vmstat nr_page_table_pages 3153 nr_secondary_page_table_pages 153