From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 855C9C41513 for ; Tue, 14 May 2024 14:05:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=gJVVqM3ooQDOWkzsDRH8EXmATDr0dYQAfcAN1LPnUCM=; b=Ac864f6Q8E/e8G SHfTmNIXHzBCPZkwxoo5top84Oa8ZblrMjMKgXk/xyqstQaOxGZ8F0gYDJFpO/xXU7Qzdc7a1N30x dcdNEWbtD65U2aNcF99Y75UPnQ5WR/KSCLRadqaHq90Y2tWVls/E8W05sl3zRmOGdoYEIhFZMvy7p Qam4vaiz6HgTCBp1SwK8ekHmORQAtFXNUsBHIoJl3r9f2DMWvz986gX00umyekOqur/HX9mft+yPb 5EzEBKJxm6tXBqaGMITLi8R3rSzvycuGStlRmf+Z8hDQflSncnsmjuQD3f7mQkAoBQ1MZTp5cw8J8 UAc3tLGG7vsy+YzaZkvg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1s6smG-0000000G7QF-0Xto; Tue, 14 May 2024 14:05:12 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1s6smC-0000000G7Oq-2F8q for linux-riscv@lists.infradead.org; Tue, 14 May 2024 14:05:10 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 84B7F61259; Tue, 14 May 2024 14:05:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EF3EC2BD10; Tue, 14 May 2024 14:05:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715695505; bh=vWD0ovFiRllts29Px0UYaBuDJVqdTXuD1sCxL0AtQiU=; h=From:To:Cc:Subject:Date:From; b=IeU775cxWUq6Ff2jYBrz1fY/qpA3rlsD2LpXwXJVs2CWriqNzxtqWnrwlllqwpehG DS5YAoD+0P67s+cz9lOE0A9i5GfCCgKKdZVlHQ0FnrVg39cQRpzl19XoYmdP9fDlts NdcHl1Effw0mxgV1KNEB5CNPOXCsMDV7SY4JnT3betckAp6Ur4EF5P6A04lJ84Magv 4lfLitICt/+e8+VP3yhrfJW2JOaheJ5R4ryZ6vzJm+d6m/XQ0nZim82j16JgfFs8Db IZdHC5oMeEZ0Ul26L14T0WcsjHupn7VcW6+vKK76+aO12P5vJs/NGgLsnplZ4A1wKF xKTjWndDhbrAA== From: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= To: Alexandre Ghiti , Albert Ou , David Hildenbrand , Palmer Dabbelt , Paul Walmsley , linux-riscv@lists.infradead.org Subject: [PATCH v2 0/8] riscv: Memory Hot(Un)Plug support Date: Tue, 14 May 2024 16:04:38 +0200 Message-Id: <20240514140446.538622-1-bjorn@kernel.org> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240514_070508_716782_85715BEE X-CRM114-Status: GOOD ( 30.62 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Stoakes , Chethan Seshadri , linux-kernel@vger.kernel.org, Andrew Bresticker , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Santosh Mamila , linux-mm@kvack.org, Sivakumar Munnangi , virtualization@lists.linux-foundation.org, Oscar Salvador Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org RnJvbTogQmrDtnJuIFTDtnBlbCA8Ympvcm5Acml2b3NpbmMuY29tPgoKPT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQpNZW1vcnkg SG90KFVuKVBsdWcgc3VwcG9ydCAoYW5kIFpPTkVfREVWSUNFKSBmb3IgdGhlIFJJU0MtViBwb3J0 Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT0KCkludHJvZHVjdGlvbgo9PT09PT09PT09PT0KClRvIHF1b3RlICJEb2N1bWVudGF0 aW9uL2FkbWluLWd1aWRlL21tL21lbW9yeS1ob3RwbHVnLnJzdCI6ICJNZW1vcnkKaG90KHVuKXBs dWcgYWxsb3dzIGZvciBpbmNyZWFzaW5nIGFuZCBkZWNyZWFzaW5nIHRoZSBzaXplIG9mIHBoeXNp Y2FsCm1lbW9yeSBhdmFpbGFibGUgdG8gYSBtYWNoaW5lIGF0IHJ1bnRpbWUuIgoKVGhpcyBzZXJp ZXMgYWRkcyBtZW1vcnkgaG90KHVuKXBsdWdnaW5nLCBhbmQgWk9ORV9ERVZJQ0Ugc3VwcG9ydCBm b3IKdGhlIFJJU0MtViBMaW51eCBwb3J0LgoKSSdtIHNlbmRpbmcgdGhpcyBzZXJpZXMgd2hpbGUg TFNGL01NL0JQRiBpcyBvbi1nb2luZywgYW5kIHdpdGggc29tZQpsdWNrIHNvbWUgTU0gcGVyc29u IGNhbiByZXZpZXcgdGhlIHNlcmllcyB3aGlsZSB6b25pbmcgb3V0IG9uIGEgdGFsay4KOy0pCgpN TSBjb25maWd1cmF0aW9uCj09PT09PT09PT09PT09PT0KClJJU0MtViBNTSBoYXMgdGhlIGZvbGxv d2luZyBjb25maWd1cmF0aW9uOgoKICogTWVtb3J5IGJsb2NrcyBhcmUgMTI4TSwgYW5hbG9nb3Vz IHRvIHg4Ni02NC4gSXQgdXNlcyBQTUQKICAgKCJodWdlcGFnZSIpIHZtZW1tYXBzLiBGcm9tIHRo YXQgZm9sbG93cyB0aGF0IDJNIChQTUQpIHdvcnRoIG9mCiAgIHZtZW1tYXAgc3BhbnMgMzI3Njgg cGFnZXMgw6EgNEsgd2hpY2ggZ2V0cyB1cyAxMjhNLgoKICogVGhlIHBhZ2VibG9jayBzaXplIGlz IHRoZSBtaW5pbXVtIG1pbmltdW0gdmlydGlvX21lbSBzaXplLCBhbmQgb24KICAgUklTQy1WIGl0 J3MgMk0gKDJeOSAqIDRLKS4KCkltcGxlbWVudGF0aW9uCj09PT09PT09PT09PT09CgpUaGUgUEdE IHRhYmxlIG9uIFJJU0MtViBpcyBzaGFyZWQvY29waWVkIGJldHdlZW4gZm9yIGFsbCBwcm9jZXNz ZXMuIFRvCmF2b2lkIGRvaW5nIHBhZ2UgdGFibGUgc3luY2hyb25pemF0aW9uLCB0aGUgZmlyc3Qg cGF0Y2ggKHBhdGNoIDEpCnByZS1hbGxvY2F0ZWQgdGhlIFBHRCBlbnRyaWVzIGZvciB2bWVtbWFw L2RpcmVjdCBtYXAuIEJ5IGRvaW5nIHRoYXQKdGhlIGluaXRfbW0gUEdEIHdpbGwgYmUgZml4ZWQg YXQga2VybmVsIGluaXQsIGFuZCBzeW5jaHJvbml6YXRpb24gY2FuCmJlIGF2b2lkZWQgYWxsIHRv Z2V0aGVyLgoKVGhlIGZvbGxvd2luZyB0d28gcGF0Y2hlcyAocGF0Y2ggMi0zKSBkb2VzIHNvbWUg cHJlcGFyYXRpb25zLCBmb2xsb3dlZApieSB0aGUgYWN0dWFsIE1IUCBpbXBsZW1lbnRhdGlvbiAo cGF0Y2ggNC01KS4gVGhlbiwgTUhQIGFuZCB2aXJ0aW8tbWVtCmFyZSBlbmFibGVkIChwYXRjaCA2 LTcpLCBhbmQgZmluYWxseSBaT05FX0RFVklDRSBzdXBwb3J0IGlzIGFkZGVkCihwYXRjaCA4KS4K Ck1IUCBhbmQgbG9ja2luZwo9PT09PT09PT09PT09PT0KClRMO0RSOiBUaGUgTUhQIGRvZXMgbm90 IHN0ZXAgb24gYW55IHRvZXMsIGV4Y2VwdCBmb3IgcHRkdW1wLgpBZGRpdGlvbmFsIGxvY2tpbmcg aXMgcmVxdWlyZWQgZm9yIHB0ZHVtcC4KCkxvbmcgdmVyc2lvbjogRm9yIHYyIEkgc3BlbnQgc29t ZSB0aW1lIGRpZ2dpbmcgaW50byBpbml0X21tCnN5bmNocm9uaXphdGlvbi91cGRhdGUuIEhlcmUg YXJlIG15IGZpbmRpbmdzLCBhbmQgSSdkIGxvdmUgdGhlbSB0byBiZQpjb3JyZWN0ZWQgaWYgaW5j b3JyZWN0LgoKSXQncyBiZWVuIGEgZ25hcmx5IHBhdGguLi4KClRoZSBgaW5pdF9tbWAgc3RydWN0 dXJlIGlzIGEgc3BlY2lhbCBtbSAocGVyaGFwcyBub3QgYSAicmVhbCIgb25lKS4KSXQncyBhICJs YXp5IGNvbnRleHQiIHRoYXQgdHJhY2tzIGtlcm5lbCBwYWdlIHRhYmxlIHJlc291cmNlcywgZS5n LiwKdGhlIGtlcm5lbCBwYWdlIHRhYmxlIChzd2FwcGVyX3BnX2RpciksIGEga2VybmVsIHBhZ2Vf dGFibGVfbG9jayAobW9yZQphYm91dCB0aGUgdXNhZ2UgYmVsb3cpLCBtbWFwX2xvY2ssIGFuZCBz dWNoLgoKYGluaXRfbW1gIGRvZXMgbm90IHRyYWNrL2NvbnRhaW4gYW55IFZNQXMuIEhhdmluZyB0 aGUgYGluaXRfbW1gIGlzCmNvbnZlbmllbnQsIHNvIHRoYXQgdGhlIHJlZ3VsYXIga2VybmVsIHBh Z2UgdGFibGUgd2Fsay9tb2RpZnkKZnVuY3Rpb25zIGNhbiBiZSB1c2VkLgoKTm93LCBgaW5pdF9t bWAgYmVpbmcgc3BlY2lhbCBtZWFucyB0aGF0IHRoZSBsb2NraW5nIGZvciBrZXJuZWwgcGFnZQp0 YWJsZXMgYXJlIHNwZWNpYWwgYXMgd2VsbC4KCk9uIFJJU0MtViB0aGUgUEdEICh0b3AtbGV2ZWwg cGFnZSB0YWJsZSBzdHJ1Y3R1cmUpLCBzaW1pbGFyIHRvIHg4NiwgaXMKc2hhcmVkIChjb3BpZWQp IHdpdGggdXNlciBwcm9jZXNzZXMuIElmIHRoZSBrZXJuZWwgUEdEIGlzIG1vZGlmaWVkLCBpdApo YXMgdG8gYmUgc3luY2hlZCB0byB1c2VyLW1vZGUgcHJvY2Vzc2VzIFBHRHMuIFRoaXMgaXMgYXZv aWRlZCBieQpwcmUtcG9wdWxhdGluZyB0aGUgUEdELCBzbyBpdCdsbCBiZSBmaXhlZCBmcm9tIGJv b3QuCgpUaGUgaW4ta2VybmVsIHBnZCByZWdpb25zIGFyZSBkb2N1bWVudGVkIGluCmBEb2N1bWVu dGF0aW9uL2FyY2gvcmlzY3Yvdm0tbGF5b3V0LnJzdGAuCgpUaGUgZGlzdGluY3QgcmVnaW9ucyBh cmU6CiAqIHZtZW1tYXAKICogdm1hbGxvYy9pb3JlbWFwIHNwYWNlCiAqIGRpcmVjdCBtYXBwaW5n IG9mIGFsbCBwaHlzaWNhbCBtZW1vcnkKICoga2FzYW4KICogbW9kdWxlcywgQlBGCiAqIGtlcm5l bAoKTWVtb3J5IGhvdHBsdWcgaXMgdGhlIHByb2Nlc3Mgb2YgYWRkaW5nL3JlbW92aW5nIG1lbW9y eSB0by9mcm9tIHRoZQprZXJuZWwuCgpBZGRpbmcgaXMgZG9uZSBpbiB0d28gcGhhc2VzOgogMS4g QWRkIHRoZSBtZW1vcnkgdG8gdGhlIGtlcm5lbAogMi4gT25saW5lIG1lbW9yeSwgbWFraW5nIGl0 IGF2YWlsYWJsZSB0byB0aGUgcGFnZSBhbGxvY2F0b3IuCgpTdGVwIDEgaXMgcGFydGlhbGx5IGFy Y2hpdGVjdHVyZSBkZXBlbmRlbnQsIGFuZCB1cGRhdGVzIHRoZSBpbml0X21tCnBhZ2UgdGFibGU6 CiAqIFVwZGF0ZSB0aGUgZGlyZWN0IG1hcCBwYWdlIHRhYmxlcy4gVGhlIGRpcmVjdCBtYXAgaXMg YSBsaW5lYXIgbWFwLAogICByZXByZXNlbnRpbmcgYWxsIHBoeXNpY2FsIG1lbW9yeTogYHZpcnQg PSBwaHlzICsgUEFHRV9PRkZTRVRgCiAqIEFkZCBhIGBzdHJ1Y3QgcGFnZWAgZm9yIGVhY2ggYWRk ZWQgcGFnZSBvZiBtZW1vcnkuIFVwZGF0ZSB0aGUKICAgdm1lbW1hcCAodmlydHVhbCBtYXBwaW5n IHRvIHRoZSBgc3RydWN0IHBhZ2VgLCBzbyB3ZSBjYW4gZWFzaWx5CiAgIHRyYW5zZm9ybSBhIGtl cm5lbCB2aXJ0dWFsIGFkZHJlc3MgdG8gYSBgc3RydWN0IHBhZ2UgKmAgYWRkcmVzcy4KCkZyb20g YW4gTUhQIHBlcnNwZWN0aXZlLCB0aGVyZSBhcmUgdHdvIHJlZ2lvbnMgb2YgdGhlIFBHRCB0aGF0 IGFyZQp1cGRhdGVkOgogKiB2bWVtbWFwCiAqIGRpcmVjdCBtYXBwaW5nIG9mIGFsbCBwaHlzaWNh bCBtZW1vcnkKClRoZSBgc3RydWN0IG1tX3N0cnVjdGAgaGFzIGEgY291cGxlIG9mIGxvY2tzIGlu IHBsYXk6CiAqIGBzcGlubG9ja190IHBhZ2VfdGFibGVfbG9ja2AgcHJvdGVjdHMgdGhlIHBhZ2Ug dGFibGUsIGFuZCBzb21lCiAgICBjb3VudGVycwogKiBgc3RydWN0IHJ3X3NlbWFwaG9yZSBtbWFw X2xvY2tgIHByb3RlY3QgYW4gbW0ncyBWTUFzCgpOb3RlIGFnYWluIHRoYXQgYGluaXRfbW1gIGRv ZXMgbm90IGNvbnRhaW4gYW55IFZNQXMsIGJ1dCBzdGlsbCB1c2VzCnRoZSBtbWFwX2xvY2sgaW4g c29tZSBwbGFjZXMuCgpUaGUgYHBhZ2VfdGFibGVfbG9ja2Agd2FzIG9yaWdpbmFsbHkgdXNlZCB0 byB0byBwcm90ZWN0IGFsbCBwYWdlcwp0YWJsZXMsIGJ1dCBtb3JlIHJlY2VudGx5IGEgc3BsaXQg cGFnZSB0YWJsZSBsb2NrIGhhcyBiZWVuIGludHJvZHVjZWQuClRoZSBzcGxpdCBsb2NrIGhhcyBh IHBlci10YWJsZSBsb2NrIGZvciB0aGUgUFRFIGFuZCBQTUQgdGFibGVzLiBJZgpzcGxpdCBsb2Nr IGlzIGRpc2FibGVkLCBhbGwgdGFibGVzIGFyZSBndWFyZGVkIGJ5CmBtbS0+cGFnZV90YWJsZV9s b2NrYCAoZm9yIHVzZXIgcHJvY2Vzc2VzKS4gU3BsaXQgcGFnZSB0YWJsZSBsb2NrcyBhcmUKbm90 IHVzZWQgZm9yIGluaXRfbW0uCgpNSFAgb3BlcmF0aW9ucyBpcyB0eXBpY2FsbHkgc3luY2hyb25p emVkIHVzaW5nCmBERUZJTkVfU1RBVElDX1BFUkNQVV9SV1NFTShtZW1faG90cGx1Z19sb2NrKWAu CgpBY3RvcnMKLS0tLS0tCgpUaGUgZm9sbG93aW5nIG5vbi1NSFAgYWN0b3JzIGluIHRoZSBrZXJu ZWwgdHJhdmVyc2VzIChyZWFkKSwgYW5kL29yCm1vZGlmaWVzIHRoZSBrZXJuZWwgUEdELgoKICog YHB0ZHVtcGAKCiAgIFdhbGtzIHRoZSBlbnRpcmUgYGluaXRfbW1gLCB2aWEgYHB0ZHVtcF93YWxr X3BnZCgpYCB3aXRoIHRoZQogICBgbW1hcF93cml0ZV9sb2NrKGluaXRfbW0pYCB0YWtlbi4KCiAg IE9ic2VydmF0aW9uOiBwdGR1bXAgY2FuIHJhY2Ugd2l0aCBNSFAsIGFuZCBuZWVkcyBhZGRpdGlv bmFsIGxvY2tpbmcKICAgdG8gYXZvaWQgY3Jhc2hlcy9yYWNlcy4KCiAqIGBzZXRfZGlyZWN0Xypg IC8gYGFyY2gvcmlzY3YvbW0vcGFnZWF0dHIuY2AKCiAgIFRoZSBgc2V0X2RpcmVjdF8qYCBmdW5j dGlvbmFsaXR5IGlzIHVzZWQgdG8gInN5bmNocm9uaXplIiB0aGUKICAgZGlyZWN0IG1hcCB0byBv dGhlciBrZXJuZWwgbWFwcGluZ3MsIGUuZy4gbW9kdWxlcy9rZXJuZWwgdGV4dC4gVGhlCiAgIGRp cmVjdCBtYXAgaXMgdXNpbmcgImFzIGxhcmdlIGh1Z2UgdGFibGUgbWFwcGluZ3MgYXMgcG9zc2li bGUiLAogICB3aGljaCBtZWFucyB0aGF0IHRoZSBgc2V0X2RpcmVjdF8qYCBtaWdodCBuZWVkIHRv IHNwbGl0IHRoZSBkaXJlY3QKICAgbWFwLgoKICBUaGUgYHNldF9kaXJlY3RfKmAgZnVuY3Rpb25z IG9wZXJhdGVzIHdpdGggdGhlCiAgYG1tYXBfd3JpdGVfbG9jayhpbml0X21tKWAgdGFrZW4uCgog IE9ic2VydmF0aW9uOiBgc2V0X2RpcmVjdF8qYCB1c2VzIHRoZSBkaXJlY3QgbWFwLCBidXQgd2ls bCBuZXZlcgogIG1vZGlmeSB0aGUgc2FtZSBlbnRyeSBhcyBNSFAuIElmIHRoZXJlIGlzIGEgbWFw cGluZywgdGhhdCBlbnRyeSB3aWxsCiAgbmV2ZXIgcmFjZSB3aXRoIE1IUC4gRnVydGhlciwgTUhQ IGFjdHMgd2hlbiBtZW1vcnkgaXMgb2ZmbGluZS4KCiAqIEhWTyAvIGBtbS9odWdldGxiX3ZtZW1t YXBgCgogICBIVk8gb3B0aW1pemVzIHRoZSBiYWNraW5nIGBzdHJ1Y3QgcGFnZWAgZm9yIGh1Z2V0 bGIgcGFnZXMsIHdoaWNoCiAgIG1lYW5zIGNoYW5naW5nIHRoZSAidm1lbW1hcCIgcmVnaW9uLiBI Vk8gY2FuIHNwbGl0IChtZXJnZT8pIGEKICAgdm1lbW1hcCBwbWQuIEhvd2V2ZXIsIGl0IHdpbGwg bmV2ZXIgcmFjZSB3aXRoIE1IUCwgc2luY2UgSFZPIG9ubHkKICAgb3BlcmF0ZXMgYXQgb25saW5l IG1lbW9yeS4gSFZPIGNhbm5vdCB0b3VjaCBtZW1vcnkgYmVpbmcgTUhQIGFkZGVkCiAgIG9yIHJl bW92ZWQuCgogKiBgYXBwbHlfdG9fcGFnZV9yYW5nZWAKCiAgIFdhbGtzIGEgcmFuZ2UsIGNyZWF0 ZXMgcGFnZXMgYW5kIGFwcGxpZXMgYSBjYWxsYmFjayAoc2V0dGluZwogICBwZXJtaXNzaW9ucykg Zm9yIHRoZSBwYWdlLgoKICAgV2hlbiBjcmVhdGluZyBhIHRhYmxlLCBpdCBtaWdodCB1c2UgYGlu dCBfX3B0ZV9hbGxvY19rZXJuZWwocG1kX3QKICAgKnBtZClgIHdoaWNoIHRha2VzIHRoZSBgaW5p dF9tbS5wYWdlX3RhYmxlX2xvY2tgIHRvIHN5bmNocm9uaXplIHBtZAogICBwb3B1bGF0ZS4KCiAg IFVzZWQgYnk6IGBtbS92bWFsbG9jLmNgIGFuZCBgbW0va2FzYW4vc2hhZG93LmNgLiBUaGUgS0FT QU4gY2FsbGJhY2sKICAgdGFrZXMgdGhlIGBpbml0X21tLnBhZ2VfdGFibGVfbG9ja2AgdG8gc3lu Y2hyb25pemUgcHRlIGNyZWF0aW9uLgoKICAgT2JzZXJ2YXRpb25zOiBgYXBwbHlfdG9fcGFnZV9y YW5nZWAgYXBwbGllcyB0byB0aGUgInZtYWxsb2MvaW9yZW1hcAogICBzcGFjZSIgcmVnaW9uLCBh bmQgImthc2FuIiByZWdpb24uICpOb3QqIGFmZmVjdGVkIGJ5IE1IUC4KCiAqIGBhcHBseV90b19l eGlzdGluZ19wYWdlX3JhbmdlYAoKICAgV2Fsa3MgYSByYW5nZSwgYXBwbGllcyBhIGNhbGxiYWNr IChzZXR0aW5nIHBlcm1pc3Npb25zKSBmb3IgdGhlCiAgIHBhZ2UgKG5vIHBhZ2UgY3JlYXRpb24p LgoKICAgVXNlZCBieTogYGtlcm5lbC9icGYvYXJlbmEuY2AgYW5kIGBtbS9rYXNhbi9zaGFkb3cu Y2AuIFRoZSBLQVNBTgogICBjYWxsYmFjayB0YWtlcyB0aGUgYGluaXRfbW0ucGFnZV90YWJsZV9s b2NrYCB0byBzeW5jaHJvbml6ZSBwdGUKICAgY3JlYXRpb24uICpOb3QqIGFmZmVjdGVkIGJ5IE1I UCByZWdpb25zLgoKICogYGFwcGx5X3RvX2V4aXN0aW5nX3BhZ2VfcmFuZ2VgIGFwcGxpZXMgdG8g dGhlICJ2bWFsbG9jL2lvcmVtYXAKICAgc3BhY2UiIHJlZ2lvbiwgYW5kICJrYXNhbiIgcmVnaW9u LiAqTm90KiBhZmZlY3RlZCBieSBNSFAgcmVnaW9ucy4KCiAqICBgaW9yZW1hcF9wYWdlX3Jhbmdl YCBhbmQgYHZtYXBfcGFnZV9yYW5nZWAKCiAgICBVc2VzIHRoZSBzYW1lIGludGVybmFsIGZ1bmN0 aW9uLCBhbmQgbWlnaHQgY3JlYXRlIHRhYmxlIGVudHJpZXMgYXQKICAgIHRoZSAidm1hbGxvYy9p b3JlbWFwIHNwYWNlIiByZWdpb24uIENhbiBjYWxsCiAgICBgX19wdGVfYWxsb2Nfa2VybmVsKClg IHdoaWNoIHRha2VzIHRoZSBgaW5pdF9tbS5wYWdlX3RhYmxlX2xvY2tgCiAgICBzeW5jaHJvbml6 aW5nIHBtZCBwb3B1bGF0ZSBpbiB0aGUgcmVnaW9uLiAqTm90KiBhZmZlY3RlZCBieSBNSFAKICAg IHJlZ2lvbnMuCgpTdW1tYXJ5OgogICogTUhQIGFkZCB3aWxsIG5ldmVyIG1vZGlmeSB0aGUgc2Ft ZSBwYWdlIHRhYmxlIGVudHJpZXMsIGFzIGFueSBvZgogICAgdGhlIG90aGVyIGFjdG9ycy4KICAq IE1IUCByZW1vdmUgaXMgZG9uZSB3aGVuIG1lbW9yeSBpcyBvZmZsaW5lZCwgYW5kIHdpbGwgbm90 IGNsYXNoCiAgICB3aXRoIGFueSBvZiB0aGUgYWN0b3JzLgogICogRnVuY3Rpb25zIHRoYXQgd2Fs ayB0aGUgZW50aXJlIGtlcm5lbCBwYWdlIHRhYmxlIG5lZWQKICAgIHN5bmNocm9uaXphdGlvbgoK ICAqIEl0J3Mgc3VmZmljaWVudCB0byBhZGQgdGhlIE1IUCBsb2NrIHB0ZHVtcC4KClRlc3RpbmcK PT09PT09PQoKVGhpcyBzZXJpZXMgYWRkcyBiYXNpYyBEVCBzdXBwb3J0ZWQgaG90cGx1Z2dpbmcu IFRoZXJlIGlzIGEgUUVNVSBwYXRjaAplbmFibGluZyBNSFAgZm9yIHRoZSBSSVNDLVYgInZpcnQi IG1hY2hpbmUgaGVyZTogWzFdCgpBQ1BJL01TSSBzdXBwb3J0IGlzIHN0aWxsIGluIHRoZSBtYWtp bmcgZm9yIFJJU0MtViwgYW5kIHByaW9yIHByb3BlcgooQUNQSSkgUENJIE1TSSBzdXBwb3J0IGxh bmRzIFsyXSBhbmQgTlVNQSBTUkFUIHN1cHBvcnQgWzNdLCBpdCBoYXJkIHRvCnRyeSBpdCBvdXQu CgpJJ3ZlIHByZXBhcmVkIGEgUUVNVSBicmFuY2ggd2l0aCBwcm9wZXIgQUNQSSBHRUQvUEMtRElN TSBzdXBwb3J0IFs0XSwKYW5kIGEgdGhpcyBzZXJpZXMgd2l0aCB0aGUgcmVxdWlyZWQgcHJlcmVx dWlzaXRlcyBbNV0gKEFJQSwgQUNQSSBBSUEKTUFEVCwgQUNQSSBOVU1BIFNSQVQpLgoKClRvIHRl c3Qgd2l0aCB2aXJ0aW8tbWVtLCBlLmcuOgogIHwgcWVtdS1zeXN0ZW0tcmlzY3Y2NCBcCiAgfCAg ICAgLW1hY2hpbmUgdmlydCxhaWE9YXBsaWMtaW1zaWMgXAogIHwgICAgIC1jcHUgcnY2NCx2PXRy dWUsdmxlbj0yNTYsZWxlbj02NCxoPXRydWUsemJrYj1vbix6YmtjPW9uLHpia3g9b24semtyPW9u LHprdD1vbixzdmludmFsPW9uLHN2bmFwb3Q9b24sc3ZwYm10PW9uIFwKICB8ICAgICAtbm9kZWZh dWx0cyBcCiAgfCAgICAgLW5vZ3JhcGhpYyAtc21wIDggLWtlcm5lbCBydjY0LXUtYm9vdC5iaW4g XAogIHwgICAgIC1kcml2ZSBmaWxlPXJvb3Rmcy5pbWcsZm9ybWF0PXJhdyxpZj12aXJ0aW8gXAog IHwgICAgIC1kZXZpY2UgdmlydGlvLXJuZy1wY2kgXAogIHwgICAgIC1tIDE2RyxzbG90cz0zLG1h eG1lbT0zMkcgXAogIHwgICAgIC1vYmplY3QgbWVtb3J5LWJhY2tlbmQtcmFtLGlkPW1lbTAsc2l6 ZT0xNkcgXAogIHwgICAgIC1udW1hIG5vZGUsbm9kZWlkPTAsbWVtZGV2PW1lbTAgXAogIHwgICAg IC1zZXJpYWwgY2hhcmRldjpjaGFyMCBcCiAgfCAgICAgLW1vbiBjaGFyZGV2PWNoYXIwLG1vZGU9 cmVhZGxpbmUgXAogIHwgICAgIC1jaGFyZGV2IHN0ZGlvLG11eD1vbixpZD1jaGFyMCBcCiAgfCAg ICAgLWRldmljZSBwY2ktc2VyaWFsLGlkPXNlcmlhbDAsY2hhcmRldj1jaGFyMCBcCiAgfCAgICAg LW9iamVjdCBtZW1vcnktYmFja2VuZC1yYW0saWQ9dm1lbTAsc2l6ZT0yRyBcCiAgfCAgICAgLWRl dmljZSB2aXJ0aW8tbWVtLXBjaSxpZD12bTAsbWVtZGV2PXZtZW0wLG5vZGU9MAoKSW4gdGhlIFFF TVUgbW9uaXRvcjoKICB8IChxZW11KSBpbmZvIG1lbW9yeS1kZXZpY2VzCiAgfCAocWVtdSkgcW9t LXNldCB2bTAgcmVxdWVzdGVkLXNpemUgMUcKCi4uLnRvIHRlc3QgREFYL0tNRU0sIHVzZSB0aGUg Zm9sbG93IFFFTVUgcGFyYW1ldGVyczoKICB8ICAtb2JqZWN0IG1lbW9yeS1iYWNrZW5kLWZpbGUs aWQ9bWVtMSxzaGFyZT1vbixtZW0tcGF0aD12aXJ0aW9fcG1lbS5pbWcsc2l6ZT00RyBcCiAgfCAg LWRldmljZSB2aXJ0aW8tcG1lbS1wY2ksbWVtZGV2PW1lbTEsaWQ9bnYxCgphbmQgdGhlIHJlZ3Vs YXIgbmRjdGwvZGF4Y3RsIGRhbmNlLgoKSWYgeW91J3JlIGJyYXZlIHRvIHRyeSB0aGUgQUNQSSBi cmFuY2gsIGFkZCAiYWNwaT1vbiIgdG8gIi1tYWNoaW5lCnZpcnQiLCBhbmQgdGVzdCBQQy1ESU1N IE1IUCAoaW4gYWRkaXRpb24gdG8gdmlydGlvLXtwfSxtZW0pOgoKSW4gdGhlIFFFTVUgbW9uaXRv cjoKICB8IChxZW11KSBvYmplY3RfYWRkIG1lbW9yeS1iYWNrZW5kLXJhbSxpZD1tZW0xLHNpemU9 MUcKICB8IChxZW11KSBkZXZpY2VfYWRkIHBjLWRpbW0saWQ9ZGltbTEsbWVtZGV2PW1lbTEKCndo ZXJlICJydjY0LXUtYm9vdC5iaW4iIGlzIFUtYm9vdCB3aXRoIEVGSS9BQ1BJLXN1cHBvcnQgKHVz ZSBbNl0gaWYKeW91J3JlIGxhenkpLgoKUmVtb3ZlICJhY3BpPW9uIiB0byBydW4gd2l0aCBEVC4K CiAqIEVuYWJsZSB2aXJ0aW8tbWVtIGZvciBSSVNDLVYKICogQWRkIHByb3BlciBob3RwbHVnIHN1 cHBvcnQgZm9yIHZpcnRpby1tZW0KIApUaGFua3MgdG8gQWxleCwgRGF2aWQsIGFuZCBBbmRyZXcg Zm9yIGFsbCBjb21tZW50cy90ZXN0cy9maXh1cHMuCgpDaGFuZ2Vsb2cKPT09PT09PT09Cgp2MS0+ djI6CiAqIFJldmlld2VkIGEgbG90IG9mIE1IUCBsb2NraW5nIHNjZW5hcmlvcwogKiBWYXJpb3Vz IGNvbmZpZyBidWlsZCBpc3N1ZXMgKGUuZy4gYnVpbGQgIUNPTkZJR19NRU1PUllfSE9UUExVRykg KEFuZHJldykKICogQWRkZWQgYXJjaF9nZXRfbWFwcGFibGVfcmFuZ2UoKSBpbXBsZW1lbnRhdGlv bgogKiBBY3F1aXJlIE1IUCBsb2NrIGZvciBwdGR1bXAsIGFuYWxvZ291cyB0byBhcm02NAogKiBB Q1BJIE1IUCB0ZXN0cwogKiBBZGQgWk9ORV9ERVZJQ0UgcGF0Y2gKClJlZmVyZW5jZXMKPT09PT09 PT09PQoKWzFdIGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL3FlbXUtZGV2ZWwvMjAyNDA1MTQxMTA2 MTUuMzk5MDY1LTEtYmpvcm5Aa2VybmVsLm9yZy8KWzJdIGh0dHBzOi8vbG9yZS5rZXJuZWwub3Jn L2xpbnV4LXJpc2N2LzIwMjQwNTAxMTIxNzQyLjEyMTU3OTItMS1zdW5pbHZsQHZlbnRhbmFtaWNy by5jb20vClszXSBodHRwczovL2xvcmUua2VybmVsLm9yZy9saW51eC1yaXNjdi9jb3Zlci4xNzEz Nzc4MjM2LmdpdC5oYWlibzEueHVAaW50ZWwuY29tLwpbNF0gaHR0cHM6Ly9naXRodWIuY29tL2Jq b3RvL3FlbXUvY29tbWl0cy92aXJ0aW8tbWVtLXBjLWRpbW0tbWhwLWFjcGkvCls1XSBodHRwczov L2dpdGh1Yi5jb20vYmpvdG8vbGludXgvY29tbWl0cy9taHAtdjItbmV4dC1hY3BpLwpbNl0gaHR0 cHM6Ly9naXRodWIuY29tL2Jqb3RvL3Jpc2N2LXJvb3Rmcy11dGlscy90cmVlL2FjcGkKCgpUaGFu a3MsCkJqw7ZybgoKCkJqw7ZybiBUw7ZwZWwgKDgpOgogIHJpc2N2OiBtbTogUHJlLWFsbG9jYXRl IHZtZW1tYXAvZGlyZWN0IG1hcCBQR0QgZW50cmllcwogIHJpc2N2OiBtbTogQ2hhbmdlIGF0dHJp YnV0ZSBmcm9tIF9faW5pdCB0byBfX21lbWluaXQgZm9yIHBhZ2UKICAgIGZ1bmN0aW9ucwogIHJp c2N2OiBtbTogUmVmYWN0b3IgY3JlYXRlX2xpbmVhcl9tYXBwaW5nX3JhbmdlKCkgZm9yIG1lbW9y eSBob3QgYWRkCiAgcmlzY3Y6IG1tOiBBZGQgbWVtb3J5IGhvdHBsdWdnaW5nIHN1cHBvcnQKICBy aXNjdjogbW06IFRha2UgbWVtb3J5IGhvdHBsdWcgcmVhZC1sb2NrIGR1cmluZyBrZXJuZWwgcGFn ZSB0YWJsZSBkdW1wCiAgcmlzY3Y6IEVuYWJsZSBtZW1vcnkgaG90cGx1Z2dpbmcgZm9yIFJJU0Mt VgogIHZpcnRpby1tZW06IEVuYWJsZSB2aXJ0aW8tbWVtIGZvciBSSVNDLVYKICByaXNjdjogbW06 IEFkZCBzdXBwb3J0IGZvciBaT05FX0RFVklDRQoKIGFyY2gvcmlzY3YvS2NvbmZpZyAgICAgICAg ICAgICAgICAgICAgfCAgIDMgKwogYXJjaC9yaXNjdi9pbmNsdWRlL2FzbS9rYXNhbi5oICAgICAg ICB8ICAgNCArLQogYXJjaC9yaXNjdi9pbmNsdWRlL2FzbS9tbXUuaCAgICAgICAgICB8ICAgNCAr LQogYXJjaC9yaXNjdi9pbmNsdWRlL2FzbS9wZ3RhYmxlLTY0LmggICB8ICAyMCArKwogYXJjaC9y aXNjdi9pbmNsdWRlL2FzbS9wZ3RhYmxlLWJpdHMuaCB8ICAgMSArCiBhcmNoL3Jpc2N2L2luY2x1 ZGUvYXNtL3BndGFibGUuaCAgICAgIHwgIDE3ICstCiBhcmNoL3Jpc2N2L21tL2luaXQuYyAgICAg ICAgICAgICAgICAgIHwgMzIwICsrKysrKysrKysrKysrKysrKysrKystLS0tCiBhcmNoL3Jpc2N2 L21tL3B0ZHVtcC5jICAgICAgICAgICAgICAgIHwgICAzICsKIGRyaXZlcnMvdmlydGlvL0tjb25m aWcgICAgICAgICAgICAgICAgfCAgIDIgKy0KIDkgZmlsZXMgY2hhbmdlZCwgMzI4IGluc2VydGlv bnMoKyksIDQ2IGRlbGV0aW9ucygtKQoKCmJhc2UtY29tbWl0OiAwYTE2YTE3Mjg3OTAxMmM0MmY1 NWFlOGMyODgzZTE3YzFlNGUzODhmCi0tIAoyLjQwLjEKCgpfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNjdiBtYWlsaW5nIGxpc3QKbGludXgt cmlzY3ZAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWls bWFuL2xpc3RpbmZvL2xpbnV4LXJpc2N2Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7293144D12 for ; Tue, 14 May 2024 14:05:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=140.211.166.133 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715695510; cv=none; b=hkjI9IHubBtLfZs+cKToVhFt7gPJKgHj34+4jO1FCzEPBnS5vOG/lQFxAcI7nCmY9Te3GwLZ5CEe7VMGwGO13XFB3doTUIv/gdndMUk1Yt+bWELwJZQWVDKPiOOiusBNP09tEnf4uPMC+qhilc9TgebemAw519u9qQdTp34jbMk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715695510; c=relaxed/simple; bh=vWD0ovFiRllts29Px0UYaBuDJVqdTXuD1sCxL0AtQiU=; h=From:To:Cc:Subject:Date:Message-Id:MIME-Version:Content-Type; b=QuYkRzBx5qygtN7Vk5E80ogtRfPz36EgU5HBBudfEyFocOMr9vZI5uPf6fgAWMBqeNXurDCGPZ1al4KoYvejvb25/pDUcWERm/hslRMwRkD18KKmiWpTBV+srsPU0V+X1OsKd0ntY2LeRUZU2CQXgg1yfjXtGV+axy78wjC5e2Y= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=IeU775cx; arc=none smtp.client-ip=140.211.166.133 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="IeU775cx" Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 76072403C7 for ; Tue, 14 May 2024 14:05:08 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org X-Spam-Flag: NO X-Spam-Score: -4.401 X-Spam-Level: Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id V8CjaQCHJa3e for ; Tue, 14 May 2024 14:05:07 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2604:1380:4641:c500::1; helo=dfw.source.kernel.org; envelope-from=bjorn@kernel.org; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp2.osuosl.org EE137402E8 Authentication-Results: smtp2.osuosl.org; dmarc=pass (p=none dis=none) header.from=kernel.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org EE137402E8 Authentication-Results: smtp2.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=IeU775cx Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by smtp2.osuosl.org (Postfix) with ESMTPS id EE137402E8 for ; Tue, 14 May 2024 14:05:06 +0000 (UTC) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 84B7F61259; Tue, 14 May 2024 14:05:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EF3EC2BD10; Tue, 14 May 2024 14:05:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1715695505; bh=vWD0ovFiRllts29Px0UYaBuDJVqdTXuD1sCxL0AtQiU=; h=From:To:Cc:Subject:Date:From; b=IeU775cxWUq6Ff2jYBrz1fY/qpA3rlsD2LpXwXJVs2CWriqNzxtqWnrwlllqwpehG DS5YAoD+0P67s+cz9lOE0A9i5GfCCgKKdZVlHQ0FnrVg39cQRpzl19XoYmdP9fDlts NdcHl1Effw0mxgV1KNEB5CNPOXCsMDV7SY4JnT3betckAp6Ur4EF5P6A04lJ84Magv 4lfLitICt/+e8+VP3yhrfJW2JOaheJ5R4ryZ6vzJm+d6m/XQ0nZim82j16JgfFs8Db IZdHC5oMeEZ0Ul26L14T0WcsjHupn7VcW6+vKK76+aO12P5vJs/NGgLsnplZ4A1wKF xKTjWndDhbrAA== From: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= To: Alexandre Ghiti , Albert Ou , David Hildenbrand , Palmer Dabbelt , Paul Walmsley , linux-riscv@lists.infradead.org Cc: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Andrew Bresticker , Chethan Seshadri , Lorenzo Stoakes , Oscar Salvador , Santosh Mamila , Sivakumar Munnangi , Sunil V L , linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org Subject: [PATCH v2 0/8] riscv: Memory Hot(Un)Plug support Date: Tue, 14 May 2024 16:04:38 +0200 Message-Id: <20240514140446.538622-1-bjorn@kernel.org> X-Mailer: git-send-email 2.40.1 Precedence: bulk X-Mailing-List: virtualization@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Björn Töpel ================================================================ Memory Hot(Un)Plug support (and ZONE_DEVICE) for the RISC-V port ================================================================ Introduction ============ To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory hot(un)plug allows for increasing and decreasing the size of physical memory available to a machine at runtime." This series adds memory hot(un)plugging, and ZONE_DEVICE support for the RISC-V Linux port. I'm sending this series while LSF/MM/BPF is on-going, and with some luck some MM person can review the series while zoning out on a talk. ;-) MM configuration ================ RISC-V MM has the following configuration: * Memory blocks are 128M, analogous to x86-64. It uses PMD ("hugepage") vmemmaps. From that follows that 2M (PMD) worth of vmemmap spans 32768 pages á 4K which gets us 128M. * The pageblock size is the minimum minimum virtio_mem size, and on RISC-V it's 2M (2^9 * 4K). Implementation ============== The PGD table on RISC-V is shared/copied between for all processes. To avoid doing page table synchronization, the first patch (patch 1) pre-allocated the PGD entries for vmemmap/direct map. By doing that the init_mm PGD will be fixed at kernel init, and synchronization can be avoided all together. The following two patches (patch 2-3) does some preparations, followed by the actual MHP implementation (patch 4-5). Then, MHP and virtio-mem are enabled (patch 6-7), and finally ZONE_DEVICE support is added (patch 8). MHP and locking =============== TL;DR: The MHP does not step on any toes, except for ptdump. Additional locking is required for ptdump. Long version: For v2 I spent some time digging into init_mm synchronization/update. Here are my findings, and I'd love them to be corrected if incorrect. It's been a gnarly path... The `init_mm` structure is a special mm (perhaps not a "real" one). It's a "lazy context" that tracks kernel page table resources, e.g., the kernel page table (swapper_pg_dir), a kernel page_table_lock (more about the usage below), mmap_lock, and such. `init_mm` does not track/contain any VMAs. Having the `init_mm` is convenient, so that the regular kernel page table walk/modify functions can be used. Now, `init_mm` being special means that the locking for kernel page tables are special as well. On RISC-V the PGD (top-level page table structure), similar to x86, is shared (copied) with user processes. If the kernel PGD is modified, it has to be synched to user-mode processes PGDs. This is avoided by pre-populating the PGD, so it'll be fixed from boot. The in-kernel pgd regions are documented in `Documentation/arch/riscv/vm-layout.rst`. The distinct regions are: * vmemmap * vmalloc/ioremap space * direct mapping of all physical memory * kasan * modules, BPF * kernel Memory hotplug is the process of adding/removing memory to/from the kernel. Adding is done in two phases: 1. Add the memory to the kernel 2. Online memory, making it available to the page allocator. Step 1 is partially architecture dependent, and updates the init_mm page table: * Update the direct map page tables. The direct map is a linear map, representing all physical memory: `virt = phys + PAGE_OFFSET` * Add a `struct page` for each added page of memory. Update the vmemmap (virtual mapping to the `struct page`, so we can easily transform a kernel virtual address to a `struct page *` address. >From an MHP perspective, there are two regions of the PGD that are updated: * vmemmap * direct mapping of all physical memory The `struct mm_struct` has a couple of locks in play: * `spinlock_t page_table_lock` protects the page table, and some counters * `struct rw_semaphore mmap_lock` protect an mm's VMAs Note again that `init_mm` does not contain any VMAs, but still uses the mmap_lock in some places. The `page_table_lock` was originally used to to protect all pages tables, but more recently a split page table lock has been introduced. The split lock has a per-table lock for the PTE and PMD tables. If split lock is disabled, all tables are guarded by `mm->page_table_lock` (for user processes). Split page table locks are not used for init_mm. MHP operations is typically synchronized using `DEFINE_STATIC_PERCPU_RWSEM(mem_hotplug_lock)`. Actors ------ The following non-MHP actors in the kernel traverses (read), and/or modifies the kernel PGD. * `ptdump` Walks the entire `init_mm`, via `ptdump_walk_pgd()` with the `mmap_write_lock(init_mm)` taken. Observation: ptdump can race with MHP, and needs additional locking to avoid crashes/races. * `set_direct_*` / `arch/riscv/mm/pageattr.c` The `set_direct_*` functionality is used to "synchronize" the direct map to other kernel mappings, e.g. modules/kernel text. The direct map is using "as large huge table mappings as possible", which means that the `set_direct_*` might need to split the direct map. The `set_direct_*` functions operates with the `mmap_write_lock(init_mm)` taken. Observation: `set_direct_*` uses the direct map, but will never modify the same entry as MHP. If there is a mapping, that entry will never race with MHP. Further, MHP acts when memory is offline. * HVO / `mm/hugetlb_vmemmap` HVO optimizes the backing `struct page` for hugetlb pages, which means changing the "vmemmap" region. HVO can split (merge?) a vmemmap pmd. However, it will never race with MHP, since HVO only operates at online memory. HVO cannot touch memory being MHP added or removed. * `apply_to_page_range` Walks a range, creates pages and applies a callback (setting permissions) for the page. When creating a table, it might use `int __pte_alloc_kernel(pmd_t *pmd)` which takes the `init_mm.page_table_lock` to synchronize pmd populate. Used by: `mm/vmalloc.c` and `mm/kasan/shadow.c`. The KASAN callback takes the `init_mm.page_table_lock` to synchronize pte creation. Observations: `apply_to_page_range` applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP. * `apply_to_existing_page_range` Walks a range, applies a callback (setting permissions) for the page (no page creation). Used by: `kernel/bpf/arena.c` and `mm/kasan/shadow.c`. The KASAN callback takes the `init_mm.page_table_lock` to synchronize pte creation. *Not* affected by MHP regions. * `apply_to_existing_page_range` applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP regions. * `ioremap_page_range` and `vmap_page_range` Uses the same internal function, and might create table entries at the "vmalloc/ioremap space" region. Can call `__pte_alloc_kernel()` which takes the `init_mm.page_table_lock` synchronizing pmd populate in the region. *Not* affected by MHP regions. Summary: * MHP add will never modify the same page table entries, as any of the other actors. * MHP remove is done when memory is offlined, and will not clash with any of the actors. * Functions that walk the entire kernel page table need synchronization * It's sufficient to add the MHP lock ptdump. Testing ======= This series adds basic DT supported hotplugging. There is a QEMU patch enabling MHP for the RISC-V "virt" machine here: [1] ACPI/MSI support is still in the making for RISC-V, and prior proper (ACPI) PCI MSI support lands [2] and NUMA SRAT support [3], it hard to try it out. I've prepared a QEMU branch with proper ACPI GED/PC-DIMM support [4], and a this series with the required prerequisites [5] (AIA, ACPI AIA MADT, ACPI NUMA SRAT). To test with virtio-mem, e.g.: | qemu-system-riscv64 \ | -machine virt,aia=aplic-imsic \ | -cpu rv64,v=true,vlen=256,elen=64,h=true,zbkb=on,zbkc=on,zbkx=on,zkr=on,zkt=on,svinval=on,svnapot=on,svpbmt=on \ | -nodefaults \ | -nographic -smp 8 -kernel rv64-u-boot.bin \ | -drive file=rootfs.img,format=raw,if=virtio \ | -device virtio-rng-pci \ | -m 16G,slots=3,maxmem=32G \ | -object memory-backend-ram,id=mem0,size=16G \ | -numa node,nodeid=0,memdev=mem0 \ | -serial chardev:char0 \ | -mon chardev=char0,mode=readline \ | -chardev stdio,mux=on,id=char0 \ | -device pci-serial,id=serial0,chardev=char0 \ | -object memory-backend-ram,id=vmem0,size=2G \ | -device virtio-mem-pci,id=vm0,memdev=vmem0,node=0 In the QEMU monitor: | (qemu) info memory-devices | (qemu) qom-set vm0 requested-size 1G ...to test DAX/KMEM, use the follow QEMU parameters: | -object memory-backend-file,id=mem1,share=on,mem-path=virtio_pmem.img,size=4G \ | -device virtio-pmem-pci,memdev=mem1,id=nv1 and the regular ndctl/daxctl dance. If you're brave to try the ACPI branch, add "acpi=on" to "-machine virt", and test PC-DIMM MHP (in addition to virtio-{p},mem): In the QEMU monitor: | (qemu) object_add memory-backend-ram,id=mem1,size=1G | (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 where "rv64-u-boot.bin" is U-boot with EFI/ACPI-support (use [6] if you're lazy). Remove "acpi=on" to run with DT. * Enable virtio-mem for RISC-V * Add proper hotplug support for virtio-mem Thanks to Alex, David, and Andrew for all comments/tests/fixups. Changelog ========= v1->v2: * Reviewed a lot of MHP locking scenarios * Various config build issues (e.g. build !CONFIG_MEMORY_HOTPLUG) (Andrew) * Added arch_get_mappable_range() implementation * Acquire MHP lock for ptdump, analogous to arm64 * ACPI MHP tests * Add ZONE_DEVICE patch References ========== [1] https://lore.kernel.org/qemu-devel/20240514110615.399065-1-bjorn@kernel.org/ [2] https://lore.kernel.org/linux-riscv/20240501121742.1215792-1-sunilvl@ventanamicro.com/ [3] https://lore.kernel.org/linux-riscv/cover.1713778236.git.haibo1.xu@intel.com/ [4] https://github.com/bjoto/qemu/commits/virtio-mem-pc-dimm-mhp-acpi/ [5] https://github.com/bjoto/linux/commits/mhp-v2-next-acpi/ [6] https://github.com/bjoto/riscv-rootfs-utils/tree/acpi Thanks, Björn Björn Töpel (8): riscv: mm: Pre-allocate vmemmap/direct map PGD entries riscv: mm: Change attribute from __init to __meminit for page functions riscv: mm: Refactor create_linear_mapping_range() for memory hot add riscv: mm: Add memory hotplugging support riscv: mm: Take memory hotplug read-lock during kernel page table dump riscv: Enable memory hotplugging for RISC-V virtio-mem: Enable virtio-mem for RISC-V riscv: mm: Add support for ZONE_DEVICE arch/riscv/Kconfig | 3 + arch/riscv/include/asm/kasan.h | 4 +- arch/riscv/include/asm/mmu.h | 4 +- arch/riscv/include/asm/pgtable-64.h | 20 ++ arch/riscv/include/asm/pgtable-bits.h | 1 + arch/riscv/include/asm/pgtable.h | 17 +- arch/riscv/mm/init.c | 320 ++++++++++++++++++++++---- arch/riscv/mm/ptdump.c | 3 + drivers/virtio/Kconfig | 2 +- 9 files changed, 328 insertions(+), 46 deletions(-) base-commit: 0a16a172879012c42f55ae8c2883e17c1e4e388f -- 2.40.1