From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82E72C433EF for ; Fri, 8 Apr 2022 19:12:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 089066B0072; Fri, 8 Apr 2022 15:12:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03A0F6B0073; Fri, 8 Apr 2022 15:12:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E19DD6B0074; Fri, 8 Apr 2022 15:11:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id D2BCF6B0072 for ; Fri, 8 Apr 2022 15:11:59 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9E36B25D2E for ; Fri, 8 Apr 2022 19:11:59 +0000 (UTC) X-FDA: 79334656758.03.F857F6C Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf30.hostedemail.com (Postfix) with ESMTP id 1DCE08000B for ; Fri, 8 Apr 2022 19:11:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1649445118; x=1680981118; h=message-id:date:mime-version:to:cc:references:from: subject:in-reply-to:content-transfer-encoding; bh=+rU6SoAJQ46qQf+w0uFH/OvSXa690ikdV/2AaUpsdzc=; b=Nxk3YGxP6294KwAGSZFQDu23LJdFRO5KzLI0uC4rdqclkNdeptuo6IYk nc2ftFHRq+CQn49wyJoy5tQck7Ua6Z9G7kRQLx/DTkV6evtu8CKhS9cvH 2o/Zp0DkOSEWTy0m9XdOFwBC5kLbsBU8ROhYo7z53DfTH6zbkjC5+hGJS 8GQudZsGA7EQS5oSWSpZA9xOEdrd5ubSkbG092b3UGfknsZ/OfFXzkOJ1 jmlTAGImEb8xGoyPKdrxONo7rRWMlGhUllJdYxWtb2ajVlxcgye7mtLsk RWKhacNnF9yws+1tJdTZYAQ4Wq5BBNaug1E5mU6WQY2ePxpudglA1F8u2 g==; X-IronPort-AV: E=McAfee;i="6400,9594,10311"; a="260510725" X-IronPort-AV: E=Sophos;i="5.90,245,1643702400"; d="scan'208";a="260510725" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2022 12:11:56 -0700 X-IronPort-AV: E=Sophos;i="5.90,245,1643702400"; d="scan'208";a="525492958" Received: from tsungtae-mobl.amr.corp.intel.com (HELO [10.134.43.198]) ([10.134.43.198]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Apr 2022 12:11:55 -0700 Message-ID: <93a7cfdf-02e6-6880-c563-76b01c9f41f5@intel.com> Date: Fri, 8 Apr 2022 12:11:58 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Content-Language: en-US To: "Kirill A. Shutemov" , Borislav Petkov , Andy Lutomirski , Sean Christopherson , Andrew Morton , Joerg Roedel , Ard Biesheuvel Cc: Andi Kleen , Kuppuswamy Sathyanarayanan , David Rientjes , Vlastimil Babka , Tom Lendacky , Thomas Gleixner , Peter Zijlstra , Paolo Bonzini , Ingo Molnar , Varad Gautam , Dario Faggioli , Brijesh Singh , Mike Rapoport , David Hildenbrand , x86@kernel.org, linux-mm@kvack.org, linux-coco@lists.linux.dev, linux-efi@vger.kernel.org, linux-kernel@vger.kernel.org, Mike Rapoport References: <20220405234343.74045-1-kirill.shutemov@linux.intel.com> <20220405234343.74045-2-kirill.shutemov@linux.intel.com> From: Dave Hansen Subject: Re: [PATCHv4 1/8] mm: Add support for unaccepted memory In-Reply-To: <20220405234343.74045-2-kirill.shutemov@linux.intel.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Stat-Signature: ehohzareusmhrw8ur1fz7abi7d3kwq9i X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1DCE08000B Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Nxk3YGxP; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf30.hostedemail.com: domain of dave.hansen@intel.com has no SPF policy when checking 192.55.52.120) smtp.mailfrom=dave.hansen@intel.com X-Rspam-User: X-HE-Tag: 1649445117-723473 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/5/22 16:43, Kirill A. Shutemov wrote: > Kernel only needs to accept memory once after boot, so during the boot > and warm up phase there will be a lot of memory acceptance. After things > are settled down the only price of the feature if couple of checks for > PageUnaccepted() in allocate and free paths. The check refers a hot > variable (that also encodes PageBuddy()), so it is cheap and not visible > on profiles. Let's also not sugar-coat this. Page acceptance is hideously slow. It's agonizingly slow. To boot, it's done holding a global spinlock with interrupts disabled (see patch 6/8). At the very, very least, each acceptance operation involves a couple of what are effectively ring transitions, a 2MB memset(), and a bunch of cache flushing. The system is going to be downright unusable during this time, right? Sure, it's *temporary* and only happens once at boot. But, it's going to suck. Am I over-stating this in any way? The ACCEPT_MEMORY vmstat is good to have around. Thanks for adding it. But, I think we should also write down some guidance like: If your TDX system seems as slow as snail after boot, look at the "accept_memory" counter in /proc/vmstat. If it is incrementing, then TDX memory acceptance is likely to blame. Do we need anything more discrete to tell users when acceptance is over? For instance, maybe they run something and it goes really slow, they watch "accept_memory" until it stops. They rejoice at their good fortune! Then, memory allocation starts falling over to a new node and the agony beings anew. I can think of dealing with this in two ways: cat /sys/.../unaccepted_pages_left which just walks the bitmap and counts the amount of pages remaining. or something like: echo 1 > /sys/devices/system/node/node0/make_the_pain_stop Which will, well, make the pain stop on node0.