From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8144DC27C5E for ; Wed, 5 Jun 2024 11:41:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=7VHiGe07S4UBWXj6aXuBJoEgXIRUxAxQBr2vnJATGwY=; b=CXZOESxF5H6Dp0 XpZhnaKpd+0Ipq6bDQszRFTGllxQVYIwBPSqF/HzCJY0yyMN2iEyjlOxh7dsdT5m21QRsblbZ/Xvn RDVqRkwvSZ8lSqJse3JxQqrfBpnuN2qsr6cQ/fTXcQ+u/SJQr4FuuXsrGt8w0Z4dgfC1qGki04PiZ xUGEAza5K6t9mrCVftrtsu+57ycTI8IogxIIeEgU9E+94zOrs4OWScQshndtlmRxTBDRk0f4PBipm pWDhdzc9xBWIfldQdRIQYO+jHQmV35rxdLaW6TaAsB24P6XIs8gUkeAQdaNNi0vlsctwpJY4dwaK4 xNs4wzYFfyCMKfEa6Dzw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEp19-00000005jpM-2NaH; Wed, 05 Jun 2024 11:41:23 +0000 Received: from sin.source.kernel.org ([145.40.73.55]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sEp14-00000005jm7-1MKD for linux-riscv@lists.infradead.org; Wed, 05 Jun 2024 11:41:21 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id DDD41CE1774; Wed, 5 Jun 2024 11:41:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8A7DC3277B; Wed, 5 Jun 2024 11:41:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717587674; bh=S3u0bjQpavDADgmw6jZLWSNIcNZGzJbh3evgRUz4DTk=; h=From:To:Cc:Subject:Date:From; b=LaL7bLdq+6cwrcQHLBpD94M8DbKhYUAqVsushnVvy0cLD86AL6hUnHO3M7GhiEeCh yOli8oZSSzQglUtdfZ9lQdxULAIbKEkNV9nUt8mivwieLLYJthz4StgifM6pNibyp8 XyqvB20qwl3nEV7TZWuL6SUKTpv8NL8vfrL/PGaUY3Bd8sqdoNPunneetDIZKUKIOh ZpRqWoiS5ox4iSnObOMf9bR4taHjlUzZR5El7C38znb4weoT7oDvB8xt+sm302YqBT fHPt/fuN+mI6QageH31bWvLU/ml0sWxrVS6X1C4IDyR8yEkzb+/Z6P1Q4BE82JbCBj KdcdPmXk4qenw== From: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= To: Alexandre Ghiti , Albert Ou , David Hildenbrand , Palmer Dabbelt , Paul Walmsley , linux-riscv@lists.infradead.org, Oscar Salvador Subject: [PATCH v4 00/11] riscv: Memory Hot(Un)Plug support Date: Wed, 5 Jun 2024 13:40:43 +0200 Message-ID: <20240605114100.315918-1-bjorn@kernel.org> X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240605_044118_783385_6E94E81C X-CRM114-Status: GOOD ( 30.40 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Stoakes , Chethan Seshadri , linux-kernel@vger.kernel.org, Andrew Bresticker , =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Santosh Mamila , linux-mm@kvack.org, Sivakumar Munnangi , virtualization@lists.linux-foundation.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org RnJvbTogQmrDtnJuIFTDtnBlbCA8Ympvcm5Acml2b3NpbmMuY29tPgoKPT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQpNZW1vcnkg SG90KFVuKVBsdWcgc3VwcG9ydCAoYW5kIFpPTkVfREVWSUNFKSBmb3IgdGhlIFJJU0MtViBwb3J0 Cj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT0KCihGb3IgdGhlIHJlc3RsZXNzIGZvbGtzOiBjaGFuZ2UgbG9nIGluIHRoZSBib3R0 b20hKQoKSW50cm9kdWN0aW9uCj09PT09PT09PT09PQoKVG8gcXVvdGUgIkRvY3VtZW50YXRpb24v YWRtaW4tZ3VpZGUvbW0vbWVtb3J5LWhvdHBsdWcucnN0IjogIk1lbW9yeQpob3QodW4pcGx1ZyBh bGxvd3MgZm9yIGluY3JlYXNpbmcgYW5kIGRlY3JlYXNpbmcgdGhlIHNpemUgb2YgcGh5c2ljYWwK bWVtb3J5IGF2YWlsYWJsZSB0byBhIG1hY2hpbmUgYXQgcnVudGltZS4iCgpUaGlzIHNlcmllcyBh ZGRzIG1lbW9yeSBob3QodW4pcGx1Z2dpbmcsIGFuZCBaT05FX0RFVklDRSBzdXBwb3J0IGZvcgp0 aGUgUklTQy1WIExpbnV4IHBvcnQuCgpNTSBjb25maWd1cmF0aW9uCj09PT09PT09PT09PT09PT0K ClJJU0MtViBNTSBoYXMgdGhlIGZvbGxvd2luZyBjb25maWd1cmF0aW9uOgoKICogTWVtb3J5IGJs b2NrcyBhcmUgMTI4TSwgYW5hbG9nb3VzIHRvIHg4Ni02NC4gSXQgdXNlcyBQTUQKICAgKCJodWdl cGFnZSIpIHZtZW1tYXBzLiBGcm9tIHRoYXQgZm9sbG93cyB0aGF0IDJNIChQTUQpIHdvcnRoIG9m CiAgIHZtZW1tYXAgc3BhbnMgMzI3NjggcGFnZXMgw6EgNEsgd2hpY2ggZ2V0cyB1cyAxMjhNLgoK ICogVGhlIHBhZ2VibG9jayBzaXplIGlzIHRoZSBtaW5pbXVtIG1pbmltdW0gdmlydGlvX21lbSBz aXplLCBhbmQgb24KICAgUklTQy1WIGl0J3MgMk0gKDJeOSAqIDRLKS4KCkltcGxlbWVudGF0aW9u Cj09PT09PT09PT09PT09CgpUaGUgUEdEIHRhYmxlIG9uIFJJU0MtViBpcyBzaGFyZWQvY29waWVk IGJldHdlZW4gZm9yIGFsbCBwcm9jZXNzZXMuIFRvCmF2b2lkIGRvaW5nIHBhZ2UgdGFibGUgc3lu Y2hyb25pemF0aW9uLCB0aGUgZmlyc3QgcGF0Y2ggKHBhdGNoIDEpCnByZS1hbGxvY2F0ZWQgdGhl IFBHRCBlbnRyaWVzIGZvciB2bWVtbWFwL2RpcmVjdCBtYXAuIEJ5IGRvaW5nIHRoYXQKdGhlIGlu aXRfbW0gUEdEIHdpbGwgYmUgZml4ZWQgYXQga2VybmVsIGluaXQsIGFuZCBzeW5jaHJvbml6YXRp b24gY2FuCmJlIGF2b2lkZWQgYWxsIHRvZ2V0aGVyLgoKVGhlIGZvbGxvd2luZyB0d28gcGF0Y2hl cyAocGF0Y2ggMi0zKSBkb2VzIHNvbWUgcHJlcGFyYXRpb25zLCBmb2xsb3dlZApieSB0aGUgYWN0 dWFsIE1IUCBpbXBsZW1lbnRhdGlvbiAocGF0Y2ggNC01KS4gVGhlbiwgTUhQIGFuZCB2aXJ0aW8t bWVtCmFyZSBlbmFibGVkIChwYXRjaCA2LTcpLCBhbmQgZmluYWxseSBaT05FX0RFVklDRSBzdXBw b3J0IGlzIGFkZGVkCihwYXRjaCA4KS4KCk1IUCBhbmQgbG9ja2luZwo9PT09PT09PT09PT09PT0K ClRMO0RSOiBUaGUgTUhQIGRvZXMgbm90IHN0ZXAgb24gYW55IHRvZXMsIGV4Y2VwdCBmb3IgcHRk dW1wLgpBZGRpdGlvbmFsIGxvY2tpbmcgaXMgcmVxdWlyZWQgZm9yIHB0ZHVtcC4KCkxvbmcgdmVy c2lvbjogRm9yIHYyIEkgc3BlbnQgc29tZSB0aW1lIGRpZ2dpbmcgaW50byBpbml0X21tCnN5bmNo cm9uaXphdGlvbi91cGRhdGUuIEhlcmUgYXJlIG15IGZpbmRpbmdzLCBhbmQgSSdkIGxvdmUgdGhl bSB0byBiZQpjb3JyZWN0ZWQgaWYgaW5jb3JyZWN0LgoKSXQncyBiZWVuIGEgZ25hcmx5IHBhdGgu Li4KClRoZSBgaW5pdF9tbWAgc3RydWN0dXJlIGlzIGEgc3BlY2lhbCBtbSAocGVyaGFwcyBub3Qg YSAicmVhbCIgb25lKS4KSXQncyBhICJsYXp5IGNvbnRleHQiIHRoYXQgdHJhY2tzIGtlcm5lbCBw YWdlIHRhYmxlIHJlc291cmNlcywgZS5nLiwKdGhlIGtlcm5lbCBwYWdlIHRhYmxlIChzd2FwcGVy X3BnX2RpciksIGEga2VybmVsIHBhZ2VfdGFibGVfbG9jayAobW9yZQphYm91dCB0aGUgdXNhZ2Ug YmVsb3cpLCBtbWFwX2xvY2ssIGFuZCBzdWNoLgoKYGluaXRfbW1gIGRvZXMgbm90IHRyYWNrL2Nv bnRhaW4gYW55IFZNQXMuIEhhdmluZyB0aGUgYGluaXRfbW1gIGlzCmNvbnZlbmllbnQsIHNvIHRo YXQgdGhlIHJlZ3VsYXIga2VybmVsIHBhZ2UgdGFibGUgd2Fsay9tb2RpZnkKZnVuY3Rpb25zIGNh biBiZSB1c2VkLgoKTm93LCBgaW5pdF9tbWAgYmVpbmcgc3BlY2lhbCBtZWFucyB0aGF0IHRoZSBs b2NraW5nIGZvciBrZXJuZWwgcGFnZQp0YWJsZXMgYXJlIHNwZWNpYWwgYXMgd2VsbC4KCk9uIFJJ U0MtViB0aGUgUEdEICh0b3AtbGV2ZWwgcGFnZSB0YWJsZSBzdHJ1Y3R1cmUpLCBzaW1pbGFyIHRv IHg4NiwgaXMKc2hhcmVkIChjb3BpZWQpIHdpdGggdXNlciBwcm9jZXNzZXMuIElmIHRoZSBrZXJu ZWwgUEdEIGlzIG1vZGlmaWVkLCBpdApoYXMgdG8gYmUgc3luY2hlZCB0byB1c2VyLW1vZGUgcHJv Y2Vzc2VzIFBHRHMuIFRoaXMgaXMgYXZvaWRlZCBieQpwcmUtcG9wdWxhdGluZyB0aGUgUEdELCBz byBpdCdsbCBiZSBmaXhlZCBmcm9tIGJvb3QuCgpUaGUgaW4ta2VybmVsIHBnZCByZWdpb25zIGFy ZSBkb2N1bWVudGVkIGluCmBEb2N1bWVudGF0aW9uL2FyY2gvcmlzY3Yvdm0tbGF5b3V0LnJzdGAu CgpUaGUgZGlzdGluY3QgcmVnaW9ucyBhcmU6CiAqIHZtZW1tYXAKICogdm1hbGxvYy9pb3JlbWFw IHNwYWNlCiAqIGRpcmVjdCBtYXBwaW5nIG9mIGFsbCBwaHlzaWNhbCBtZW1vcnkKICoga2FzYW4K ICogbW9kdWxlcywgQlBGCiAqIGtlcm5lbAoKTWVtb3J5IGhvdHBsdWcgaXMgdGhlIHByb2Nlc3Mg b2YgYWRkaW5nL3JlbW92aW5nIG1lbW9yeSB0by9mcm9tIHRoZQprZXJuZWwuCgpBZGRpbmcgaXMg ZG9uZSBpbiB0d28gcGhhc2VzOgogMS4gQWRkIHRoZSBtZW1vcnkgdG8gdGhlIGtlcm5lbAogMi4g T25saW5lIG1lbW9yeSwgbWFraW5nIGl0IGF2YWlsYWJsZSB0byB0aGUgcGFnZSBhbGxvY2F0b3Iu CgpTdGVwIDEgaXMgcGFydGlhbGx5IGFyY2hpdGVjdHVyZSBkZXBlbmRlbnQsIGFuZCB1cGRhdGVz IHRoZSBpbml0X21tCnBhZ2UgdGFibGU6CiAqIFVwZGF0ZSB0aGUgZGlyZWN0IG1hcCBwYWdlIHRh Ymxlcy4gVGhlIGRpcmVjdCBtYXAgaXMgYSBsaW5lYXIgbWFwLAogICByZXByZXNlbnRpbmcgYWxs IHBoeXNpY2FsIG1lbW9yeTogYHZpcnQgPSBwaHlzICsgUEFHRV9PRkZTRVRgCiAqIEFkZCBhIGBz dHJ1Y3QgcGFnZWAgZm9yIGVhY2ggYWRkZWQgcGFnZSBvZiBtZW1vcnkuIFVwZGF0ZSB0aGUKICAg dm1lbW1hcCAodmlydHVhbCBtYXBwaW5nIHRvIHRoZSBgc3RydWN0IHBhZ2VgLCBzbyB3ZSBjYW4g ZWFzaWx5CiAgIHRyYW5zZm9ybSBhIGtlcm5lbCB2aXJ0dWFsIGFkZHJlc3MgdG8gYSBgc3RydWN0 IHBhZ2UgKmAgYWRkcmVzcy4KCkZyb20gYW4gTUhQIHBlcnNwZWN0aXZlLCB0aGVyZSBhcmUgdHdv IHJlZ2lvbnMgb2YgdGhlIFBHRCB0aGF0IGFyZQp1cGRhdGVkOgogKiB2bWVtbWFwCiAqIGRpcmVj dCBtYXBwaW5nIG9mIGFsbCBwaHlzaWNhbCBtZW1vcnkKClRoZSBgc3RydWN0IG1tX3N0cnVjdGAg aGFzIGEgY291cGxlIG9mIGxvY2tzIGluIHBsYXk6CiAqIGBzcGlubG9ja190IHBhZ2VfdGFibGVf bG9ja2AgcHJvdGVjdHMgdGhlIHBhZ2UgdGFibGUsIGFuZCBzb21lCiAgICBjb3VudGVycwogKiBg c3RydWN0IHJ3X3NlbWFwaG9yZSBtbWFwX2xvY2tgIHByb3RlY3QgYW4gbW0ncyBWTUFzCgpOb3Rl IGFnYWluIHRoYXQgYGluaXRfbW1gIGRvZXMgbm90IGNvbnRhaW4gYW55IFZNQXMsIGJ1dCBzdGls bCB1c2VzCnRoZSBtbWFwX2xvY2sgaW4gc29tZSBwbGFjZXMuCgpUaGUgYHBhZ2VfdGFibGVfbG9j a2Agd2FzIG9yaWdpbmFsbHkgdXNlZCB0byB0byBwcm90ZWN0IGFsbCBwYWdlcwp0YWJsZXMsIGJ1 dCBtb3JlIHJlY2VudGx5IGEgc3BsaXQgcGFnZSB0YWJsZSBsb2NrIGhhcyBiZWVuIGludHJvZHVj ZWQuClRoZSBzcGxpdCBsb2NrIGhhcyBhIHBlci10YWJsZSBsb2NrIGZvciB0aGUgUFRFIGFuZCBQ TUQgdGFibGVzLiBJZgpzcGxpdCBsb2NrIGlzIGRpc2FibGVkLCBhbGwgdGFibGVzIGFyZSBndWFy ZGVkIGJ5CmBtbS0+cGFnZV90YWJsZV9sb2NrYCAoZm9yIHVzZXIgcHJvY2Vzc2VzKS4gU3BsaXQg cGFnZSB0YWJsZSBsb2NrcyBhcmUKbm90IHVzZWQgZm9yIGluaXRfbW0uCgpNSFAgb3BlcmF0aW9u cyBpcyB0eXBpY2FsbHkgc3luY2hyb25pemVkIHVzaW5nCmBERUZJTkVfU1RBVElDX1BFUkNQVV9S V1NFTShtZW1faG90cGx1Z19sb2NrKWAuCgpBY3RvcnMKLS0tLS0tCgpUaGUgZm9sbG93aW5nIG5v bi1NSFAgYWN0b3JzIGluIHRoZSBrZXJuZWwgdHJhdmVyc2VzIChyZWFkKSwgYW5kL29yCm1vZGlm aWVzIHRoZSBrZXJuZWwgUEdELgoKICogYHB0ZHVtcGAKCiAgIFdhbGtzIHRoZSBlbnRpcmUgYGlu aXRfbW1gLCB2aWEgYHB0ZHVtcF93YWxrX3BnZCgpYCB3aXRoIHRoZQogICBgbW1hcF93cml0ZV9s b2NrKGluaXRfbW0pYCB0YWtlbi4KCiAgIE9ic2VydmF0aW9uOiBwdGR1bXAgY2FuIHJhY2Ugd2l0 aCBNSFAsIGFuZCBuZWVkcyBhZGRpdGlvbmFsIGxvY2tpbmcKICAgdG8gYXZvaWQgY3Jhc2hlcy9y YWNlcy4KCiAqIGBzZXRfZGlyZWN0XypgIC8gYGFyY2gvcmlzY3YvbW0vcGFnZWF0dHIuY2AKCiAg IFRoZSBgc2V0X2RpcmVjdF8qYCBmdW5jdGlvbmFsaXR5IGlzIHVzZWQgdG8gInN5bmNocm9uaXpl IiB0aGUKICAgZGlyZWN0IG1hcCB0byBvdGhlciBrZXJuZWwgbWFwcGluZ3MsIGUuZy4gbW9kdWxl cy9rZXJuZWwgdGV4dC4gVGhlCiAgIGRpcmVjdCBtYXAgaXMgdXNpbmcgImFzIGxhcmdlIGh1Z2Ug dGFibGUgbWFwcGluZ3MgYXMgcG9zc2libGUiLAogICB3aGljaCBtZWFucyB0aGF0IHRoZSBgc2V0 X2RpcmVjdF8qYCBtaWdodCBuZWVkIHRvIHNwbGl0IHRoZSBkaXJlY3QKICAgbWFwLgoKICBUaGUg YHNldF9kaXJlY3RfKmAgZnVuY3Rpb25zIG9wZXJhdGVzIHdpdGggdGhlCiAgYG1tYXBfd3JpdGVf bG9jayhpbml0X21tKWAgdGFrZW4uCgogIE9ic2VydmF0aW9uOiBgc2V0X2RpcmVjdF8qYCB1c2Vz IHRoZSBkaXJlY3QgbWFwLCBidXQgd2lsbCBuZXZlcgogIG1vZGlmeSB0aGUgc2FtZSBlbnRyeSBh cyBNSFAuIElmIHRoZXJlIGlzIGEgbWFwcGluZywgdGhhdCBlbnRyeSB3aWxsCiAgbmV2ZXIgcmFj ZSB3aXRoIE1IUC4gRnVydGhlciwgTUhQIGFjdHMgd2hlbiBtZW1vcnkgaXMgb2ZmbGluZS4KCiAq IEhWTyAvIGBtbS9odWdldGxiX3ZtZW1tYXBgCgogICBIVk8gb3B0aW1pemVzIHRoZSBiYWNraW5n IGBzdHJ1Y3QgcGFnZWAgZm9yIGh1Z2V0bGIgcGFnZXMsIHdoaWNoCiAgIG1lYW5zIGNoYW5naW5n IHRoZSAidm1lbW1hcCIgcmVnaW9uLiBIVk8gY2FuIHNwbGl0IChtZXJnZT8pIGEKICAgdm1lbW1h cCBwbWQuIEhvd2V2ZXIsIGl0IHdpbGwgbmV2ZXIgcmFjZSB3aXRoIE1IUCwgc2luY2UgSFZPIG9u bHkKICAgb3BlcmF0ZXMgYXQgb25saW5lIG1lbW9yeS4gSFZPIGNhbm5vdCB0b3VjaCBtZW1vcnkg YmVpbmcgTUhQIGFkZGVkCiAgIG9yIHJlbW92ZWQuCgogKiBgYXBwbHlfdG9fcGFnZV9yYW5nZWAK CiAgIFdhbGtzIGEgcmFuZ2UsIGNyZWF0ZXMgcGFnZXMgYW5kIGFwcGxpZXMgYSBjYWxsYmFjayAo c2V0dGluZwogICBwZXJtaXNzaW9ucykgZm9yIHRoZSBwYWdlLgoKICAgV2hlbiBjcmVhdGluZyBh IHRhYmxlLCBpdCBtaWdodCB1c2UgYGludCBfX3B0ZV9hbGxvY19rZXJuZWwocG1kX3QKICAgKnBt ZClgIHdoaWNoIHRha2VzIHRoZSBgaW5pdF9tbS5wYWdlX3RhYmxlX2xvY2tgIHRvIHN5bmNocm9u aXplIHBtZAogICBwb3B1bGF0ZS4KCiAgIFVzZWQgYnk6IGBtbS92bWFsbG9jLmNgIGFuZCBgbW0v a2FzYW4vc2hhZG93LmNgLiBUaGUgS0FTQU4gY2FsbGJhY2sKICAgdGFrZXMgdGhlIGBpbml0X21t LnBhZ2VfdGFibGVfbG9ja2AgdG8gc3luY2hyb25pemUgcHRlIGNyZWF0aW9uLgoKICAgT2JzZXJ2 YXRpb25zOiBgYXBwbHlfdG9fcGFnZV9yYW5nZWAgYXBwbGllcyB0byB0aGUgInZtYWxsb2MvaW9y ZW1hcAogICBzcGFjZSIgcmVnaW9uLCBhbmQgImthc2FuIiByZWdpb24uICpOb3QqIGFmZmVjdGVk IGJ5IE1IUC4KCiAqIGBhcHBseV90b19leGlzdGluZ19wYWdlX3JhbmdlYAoKICAgV2Fsa3MgYSBy YW5nZSwgYXBwbGllcyBhIGNhbGxiYWNrIChzZXR0aW5nIHBlcm1pc3Npb25zKSBmb3IgdGhlCiAg IHBhZ2UgKG5vIHBhZ2UgY3JlYXRpb24pLgoKICAgVXNlZCBieTogYGtlcm5lbC9icGYvYXJlbmEu Y2AgYW5kIGBtbS9rYXNhbi9zaGFkb3cuY2AuIFRoZSBLQVNBTgogICBjYWxsYmFjayB0YWtlcyB0 aGUgYGluaXRfbW0ucGFnZV90YWJsZV9sb2NrYCB0byBzeW5jaHJvbml6ZSBwdGUKICAgY3JlYXRp b24uICpOb3QqIGFmZmVjdGVkIGJ5IE1IUCByZWdpb25zLgoKICogYGFwcGx5X3RvX2V4aXN0aW5n X3BhZ2VfcmFuZ2VgIGFwcGxpZXMgdG8gdGhlICJ2bWFsbG9jL2lvcmVtYXAKICAgc3BhY2UiIHJl Z2lvbiwgYW5kICJrYXNhbiIgcmVnaW9uLiAqTm90KiBhZmZlY3RlZCBieSBNSFAgcmVnaW9ucy4K CiAqICBgaW9yZW1hcF9wYWdlX3JhbmdlYCBhbmQgYHZtYXBfcGFnZV9yYW5nZWAKCiAgICBVc2Vz IHRoZSBzYW1lIGludGVybmFsIGZ1bmN0aW9uLCBhbmQgbWlnaHQgY3JlYXRlIHRhYmxlIGVudHJp ZXMgYXQKICAgIHRoZSAidm1hbGxvYy9pb3JlbWFwIHNwYWNlIiByZWdpb24uIENhbiBjYWxsCiAg ICBgX19wdGVfYWxsb2Nfa2VybmVsKClgIHdoaWNoIHRha2VzIHRoZSBgaW5pdF9tbS5wYWdlX3Rh YmxlX2xvY2tgCiAgICBzeW5jaHJvbml6aW5nIHBtZCBwb3B1bGF0ZSBpbiB0aGUgcmVnaW9uLiAq Tm90KiBhZmZlY3RlZCBieSBNSFAKICAgIHJlZ2lvbnMuCgpTdW1tYXJ5OgogICogTUhQIGFkZCB3 aWxsIG5ldmVyIG1vZGlmeSB0aGUgc2FtZSBwYWdlIHRhYmxlIGVudHJpZXMsIGFzIGFueSBvZgog ICAgdGhlIG90aGVyIGFjdG9ycy4KICAqIE1IUCByZW1vdmUgaXMgZG9uZSB3aGVuIG1lbW9yeSBp cyBvZmZsaW5lZCwgYW5kIHdpbGwgbm90IGNsYXNoCiAgICB3aXRoIGFueSBvZiB0aGUgYWN0b3Jz LgogICogRnVuY3Rpb25zIHRoYXQgd2FsayB0aGUgZW50aXJlIGtlcm5lbCBwYWdlIHRhYmxlIG5l ZWQKICAgIHN5bmNocm9uaXphdGlvbgoKICAqIEl0J3Mgc3VmZmljaWVudCB0byBhZGQgdGhlIE1I UCBsb2NrIHB0ZHVtcC4KClRlc3RpbmcKPT09PT09PQoKVGhpcyBzZXJpZXMgYWRkcyBiYXNpYyBE VCBzdXBwb3J0ZWQgaG90cGx1Z2dpbmcuIFRoZXJlIGlzIGEgUUVNVQpzZXJpZXMgZW5hYmxpbmcg TUhQIGZvciB0aGUgUklTQy1WICJ2aXJ0IiBtYWNoaW5lIGhlcmU6IFsxXQoKQUNQSS9NU0kgc3Vw cG9ydCBpcyBzdGlsbCBpbiB0aGUgbWFraW5nIGZvciBSSVNDLVYsIGFuZCBwcmlvciBwcm9wZXIK KEFDUEkpIFBDSSBNU0kgc3VwcG9ydCBsYW5kcyBbMl0gYW5kIE5VTUEgU1JBVCBzdXBwb3J0IFsz XSwgaXQgaGFyZCB0bwp0cnkgaXQgb3V0LgoKSSd2ZSBwcmVwYXJlZCBhIFFFTVUgYnJhbmNoIHdp dGggcHJvcGVyIEFDUEkgR0VEL1BDLURJTU0gc3VwcG9ydCBbNF0sCmFuZCBhIHRoaXMgc2VyaWVz IHdpdGggdGhlIHJlcXVpcmVkIHByZXJlcXVpc2l0ZXMgWzVdIChBSUEsIEFDUEkgQUlBCk1BRFQs IEFDUEkgTlVNQSBTUkFUKS4KCgpUbyB0ZXN0IHdpdGggdmlydGlvLW1lbSwgZS5nLjoKICB8IHFl bXUtc3lzdGVtLXJpc2N2NjQgXAogIHwgICAgIC1tYWNoaW5lIHZpcnQsYWlhPWFwbGljLWltc2lj IFwKICB8ICAgICAtY3B1IHJ2NjQsdj10cnVlLHZsZW49MjU2LGVsZW49NjQsaD10cnVlLHpia2I9 b24semJrYz1vbix6Ymt4PW9uLHprcj1vbix6a3Q9b24sc3ZpbnZhbD1vbixzdm5hcG90PW9uLHN2 cGJtdD1vbiBcCiAgfCAgICAgLW5vZGVmYXVsdHMgXAogIHwgICAgIC1ub2dyYXBoaWMgLXNtcCA4 IC1rZXJuZWwgcnY2NC11LWJvb3QuYmluIFwKICB8ICAgICAtZHJpdmUgZmlsZT1yb290ZnMuaW1n LGZvcm1hdD1yYXcsaWY9dmlydGlvIFwKICB8ICAgICAtZGV2aWNlIHZpcnRpby1ybmctcGNpIFwK ICB8ICAgICAtbSAxNkcsc2xvdHM9MyxtYXhtZW09MzJHIFwKICB8ICAgICAtb2JqZWN0IG1lbW9y eS1iYWNrZW5kLXJhbSxpZD1tZW0wLHNpemU9MTZHIFwKICB8ICAgICAtbnVtYSBub2RlLG5vZGVp ZD0wLG1lbWRldj1tZW0wIFwKICB8ICAgICAtc2VyaWFsIGNoYXJkZXY6Y2hhcjAgXAogIHwgICAg IC1tb24gY2hhcmRldj1jaGFyMCxtb2RlPXJlYWRsaW5lIFwKICB8ICAgICAtY2hhcmRldiBzdGRp byxtdXg9b24saWQ9Y2hhcjAgXAogIHwgICAgIC1kZXZpY2UgcGNpLXNlcmlhbCxpZD1zZXJpYWww LGNoYXJkZXY9Y2hhcjAgXAogIHwgICAgIC1vYmplY3QgbWVtb3J5LWJhY2tlbmQtcmFtLGlkPXZt ZW0wLHNpemU9MkcgXAogIHwgICAgIC1kZXZpY2UgdmlydGlvLW1lbS1wY2ksaWQ9dm0wLG1lbWRl dj12bWVtMCxub2RlPTAKCndoZXJlICJydjY0LXUtYm9vdC5iaW4iIGlzIFUtYm9vdCB3aXRoIEVG SS9BQ1BJLXN1cHBvcnQgKHVzZSBbNl0gaWYKeW91J3JlIGxhenkpLgoKSW4gdGhlIFFFTVUgbW9u aXRvcjoKICB8IChxZW11KSBpbmZvIG1lbW9yeS1kZXZpY2VzCiAgfCAocWVtdSkgcW9tLXNldCB2 bTAgcmVxdWVzdGVkLXNpemUgMUcKCi4uLnRvIHRlc3QgREFYL0tNRU0sIHVzZSB0aGUgZm9sbG93 IFFFTVUgcGFyYW1ldGVyczoKICB8ICAtb2JqZWN0IG1lbW9yeS1iYWNrZW5kLWZpbGUsaWQ9bWVt MSxzaGFyZT1vbixtZW0tcGF0aD12aXJ0aW9fcG1lbS5pbWcsc2l6ZT00RyBcCiAgfCAgLWRldmlj ZSB2aXJ0aW8tcG1lbS1wY2ksbWVtZGV2PW1lbTEsaWQ9bnYxCgphbmQgdGhlIHJlZ3VsYXIgbmRj dGwvZGF4Y3RsIGRhbmNlLgoKSWYgeW91J3JlIGJyYXZlIHRvIHRyeSB0aGUgQUNQSSBicmFuY2gs IGFkZCAiYWNwaT1vbiIgdG8gIi1tYWNoaW5lCnZpcnQiLCBhbmQgdGVzdCBQQy1ESU1NIE1IUCAo aW4gYWRkaXRpb24gdG8gdmlydGlvLXtwfSxtZW0pOgoKSW4gdGhlIFFFTVUgbW9uaXRvcjoKICB8 IChxZW11KSBvYmplY3RfYWRkIG1lbW9yeS1iYWNrZW5kLXJhbSxpZD1tZW0xLHNpemU9MUcKICB8 IChxZW11KSBkZXZpY2VfYWRkIHBjLWRpbW0saWQ9ZGltbTEsbWVtZGV2PW1lbTEKCllvdSBjYW4g YWxzbyB0cnkgaG90LXJlbW92ZSB3aXRoIHNvbWUgUUVNVSBvcHRpb25zLCBzYXk6CiAgfCAtb2Jq ZWN0IG1lbW9yeS1iYWNrZW5kLWZpbGUsaWQ9bWVtLTEsc2l6ZT0yNTZNLG1lbS1wYXRoPS9wYWdl c2l6ZS0yTUIKICB8IC1kZXZpY2UgcGMtZGltbSxpZD1tZW0xLG1lbWRldj1tZW0tMQogIHwgLW9i amVjdCBtZW1vcnktYmFja2VuZC1maWxlLGlkPW1lbS0yLHNpemU9MUcsbWVtLXBhdGg9L3BhZ2Vz aXplLTFHQgogIHwgLWRldmljZSBwYy1kaW1tLGlkPW1lbTIsbWVtZGV2PW1lbS0yCiAgfCAtb2Jq ZWN0IG1lbW9yeS1iYWNrZW5kLWZpbGUsaWQ9bWVtLTMsc2l6ZT0yNTZNLG1lbS1wYXRoPS9wYWdl c2l6ZS0yTUIKICB8IC1kZXZpY2UgcGMtZGltbSxpZD1tZW0zLG1lbWRldj1tZW0tMwoKUmVtb3Zl ICJhY3BpPW9uIiB0byBydW4gd2l0aCBEVC4KClRoYW5rcyB0byBBbGV4LCBBbmRyZXcsIERhdmlk LCBhbmQgT3NjYXIgZm9yIGFsbApjb21tZW50cy90ZXN0cy9maXh1cHMuCgpDaGFuZ2Vsb2cKPT09 PT09PT09Cgp2My0+djQ6CiAgQ29sbGVjdCB0YWdzCiAgMDAwMi1yaXNjdi1tbS1QcmUtYWxsb2Nh dGUtdm1lbW1hcC1kaXJlY3QtbWFwLWthc2FuLVBHCiAgICAqIFByZS1hbGxvY2F0ZSBLQVNBTiBQ R0QgZW50cmllcyBhcyB3ZWxsIChBbGV4KQogIDAwMDUtcmlzY3YtbW0tQWRkLXBmbl90b19rYWRk ci1pbXBsZW1lbnRhdGlvbgogICAgKiBOZXcgcGF0Y2gsIGFkZCBtaXNzaW5nIGZ1bmN0aW9uIGZv ciBLQVNBTgogIDAwMDYtcmlzY3YtbW0tQWRkLW1lbW9yeS1ob3RwbHVnZ2luZy1zdXBwb3J0CiAg ICAqIFVzZSBwWFhYX2dldCgpIGluc3RlYWQgb2YgUkVBRF9PTkNFIChBbGV4KQogICAgKiBTZXBh cmF0ZWQgdm1lbW1hcCBzdG9yYWdlIGFuZCBwZ3RhYmxlIGFsbG9jYXRpb24uIFVzZSBtYXRjaGlu ZwogICAgICBhbGxvYy9kZWFsbG9jIGZ1bmN0aW9ucyAoZS5nLiBwdGRlc2MgYWxsb2MvZGVsbG9j KSB3aXRoIGFsc28KICAgICAgYXZvaWRzIFdBUk5zIDstKQogICAgKiBDbGVhbmVkIHVwIHRoZSBy ZXNlcnZlZCBwYWdlIGZyZWVpbmcgKEFsZXgpCiAgMDAwOC1yaXNjdi1FbmFibGUtbWVtb3J5LWhv dHBsdWdnaW5nLWZvci1SSVNDLVYKICAgICogU2ltcGxpZmllZCB0aGUgZGVwZW5kZW5jaWVzIChB bGV4KQogICAgKiBBZGRlZCBBUkNIX01IUF9NRU1NQVBfT05fTUVNT1JZX0VOQUJMRQogIDAwMTEt cmlzY3YtRW5hYmxlLURBWC1WTUVNTUFQLW9wdGltaXphdGlvbgogICAgKiBOZXcgcGF0Y2gsIGFk ZGVkIEFSQ0hfV0FOVF9PUFRJTUlaRV9EQVhfVk1FTU1BUAoKdjItPnYzOgogIENvbGxlY3QgdGFn cwogIDAwMDEtcmlzY3YtbW0tUHJvcGVybHktZm9yd2FyZC12bWVtbWFwX3BvcHVsYXRlLWFsdG1h cAogICAgKiBOZXcgcGF0Y2gsIHNwbGl0IGZyb20gYXR0cmlidXRlIGNoYW5nZSAoQWxleCwgT3Nj YXIpCiAgMDAwNS1yaXNjdi1tbS1BZGQtbWVtb3J5LWhvdHBsdWdnaW5nLXN1cHBvcnQKICAgICog Rml4IHBhZ2UgdGFibGUgZnJlZSB0byB0YWtlIFBHX3Jlc2VydmVkIGludG8gY29uc2lkZXJhdGlv biAoRGF2aWQsCiAgICAgIE9zY2FyKQogICAgKiBEZXBlbmQgb24gU1BBUlNFX1ZNRU1NQVAgKEFs ZXgsIERhdmlkLCBPc2NhcikKICAgICogUHJvcGVyIFRMQiBmbHVzaGluZyAoQWxleCkKICAwMDA5 LXJpc2N2LW1tLUFkZC1zdXBwb3J0LWZvci1aT05FX0RFVklDRQogICAgKiBGaXhlZCAzMmIgYnVp bGQKCnYxLT52MjoKICBSZXZpZXdlZCBhIGxvdCBvZiBNSFAgbG9ja2luZyBzY2VuYXJpb3MKICBW YXJpb3VzIGNvbmZpZyBidWlsZCBpc3N1ZXMgKGUuZy4gYnVpbGQgIUNPTkZJR19NRU1PUllfSE9U UExVRykgKEFuZHJldykKICBBZGRlZCBhcmNoX2dldF9tYXBwYWJsZV9yYW5nZSgpIGltcGxlbWVu dGF0aW9uCiAgQWNxdWlyZSBNSFAgbG9jayBmb3IgcHRkdW1wLCBhbmFsb2dvdXMgdG8gYXJtNjQK ICBBQ1BJIE1IUCB0ZXN0cwogIEFkZCBaT05FX0RFVklDRSBwYXRjaAoKUmVmZXJlbmNlcwo9PT09 PT09PT09CgpbMV0gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvcWVtdS1kZXZlbC8yMDI0MDUyMTEw NTYzNS43OTUyMTEtMS1iam9ybkBrZXJuZWwub3JnLwpbMl0gaHR0cHM6Ly9sb3JlLmtlcm5lbC5v cmcvbGludXgtcmlzY3YvMjAyNDA1MDExMjE3NDIuMTIxNTc5Mi0xLXN1bmlsdmxAdmVudGFuYW1p Y3JvLmNvbS8KWzNdIGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xpbnV4LXJpc2N2L2NvdmVyLjE3 MTM3NzgyMzYuZ2l0LmhhaWJvMS54dUBpbnRlbC5jb20vCls0XSBodHRwczovL2dpdGh1Yi5jb20v YmpvdG8vcWVtdS9jb21taXRzL3ZpcnRpby1tZW0tcGMtZGltbS1taHAtYWNwaS12Mi8KWzVdIGh0 dHBzOi8vZ2l0aHViLmNvbS9iam90by9saW51eC9jb21taXRzL21ocC12NC1hY3BpCls2XSBodHRw czovL2dpdGh1Yi5jb20vYmpvdG8vcmlzY3Ytcm9vdGZzLXV0aWxzL3RyZWUvYWNwaQoKClRoYW5r cywKQmrDtnJuCgoKQmrDtnJuIFTDtnBlbCAoMTEpOgogIHJpc2N2OiBtbTogUHJvcGVybHkgZm9y d2FyZCB2bWVtbWFwX3BvcHVsYXRlKCkgYWx0bWFwIHBhcmFtZXRlcgogIHJpc2N2OiBtbTogUHJl LWFsbG9jYXRlIHZtZW1tYXAvZGlyZWN0IG1hcC9rYXNhbiBQR0QgZW50cmllcwogIHJpc2N2OiBt bTogQ2hhbmdlIGF0dHJpYnV0ZSBmcm9tIF9faW5pdCB0byBfX21lbWluaXQgZm9yIHBhZ2UKICAg IGZ1bmN0aW9ucwogIHJpc2N2OiBtbTogUmVmYWN0b3IgY3JlYXRlX2xpbmVhcl9tYXBwaW5nX3Jh bmdlKCkgZm9yIG1lbW9yeSBob3QgYWRkCiAgcmlzY3Y6IG1tOiBBZGQgcGZuX3RvX2thZGRyKCkg aW1wbGVtZW50YXRpb24KICByaXNjdjogbW06IEFkZCBtZW1vcnkgaG90cGx1Z2dpbmcgc3VwcG9y dAogIHJpc2N2OiBtbTogVGFrZSBtZW1vcnkgaG90cGx1ZyByZWFkLWxvY2sgZHVyaW5nIGtlcm5l bCBwYWdlIHRhYmxlIGR1bXAKICByaXNjdjogRW5hYmxlIG1lbW9yeSBob3RwbHVnZ2luZyBmb3Ig UklTQy1WCiAgdmlydGlvLW1lbTogRW5hYmxlIHZpcnRpby1tZW0gZm9yIFJJU0MtVgogIHJpc2N2 OiBtbTogQWRkIHN1cHBvcnQgZm9yIFpPTkVfREVWSUNFCiAgcmlzY3Y6IEVuYWJsZSBEQVggVk1F TU1BUCBvcHRpbWl6YXRpb24KCiBhcmNoL3Jpc2N2L0tjb25maWcgICAgICAgICAgICAgICAgICAg IHwgICA1ICsKIGFyY2gvcmlzY3YvaW5jbHVkZS9hc20va2FzYW4uaCAgICAgICAgfCAgIDQgKy0K IGFyY2gvcmlzY3YvaW5jbHVkZS9hc20vbW11LmggICAgICAgICAgfCAgIDQgKy0KIGFyY2gvcmlz Y3YvaW5jbHVkZS9hc20vcGFnZS5oICAgICAgICAgfCAgIDUgKwogYXJjaC9yaXNjdi9pbmNsdWRl L2FzbS9wZ3RhYmxlLTY0LmggICB8ICAyMCArKwogYXJjaC9yaXNjdi9pbmNsdWRlL2FzbS9wZ3Rh YmxlLWJpdHMuaCB8ICAgMSArCiBhcmNoL3Jpc2N2L2luY2x1ZGUvYXNtL3BndGFibGUuaCAgICAg IHwgIDE5ICstCiBhcmNoL3Jpc2N2L21tL2luaXQuYyAgICAgICAgICAgICAgICAgIHwgMzQ3ICsr KysrKysrKysrKysrKysrKysrKysrLS0tCiBhcmNoL3Jpc2N2L21tL3B0ZHVtcC5jICAgICAgICAg ICAgICAgIHwgICAzICsKIGRyaXZlcnMvdmlydGlvL0tjb25maWcgICAgICAgICAgICAgICAgfCAg IDIgKy0KIDEwIGZpbGVzIGNoYW5nZWQsIDM2NCBpbnNlcnRpb25zKCspLCA0NiBkZWxldGlvbnMo LSkKCgpiYXNlLWNvbW1pdDogZTJjNzliNGM1YzRkODM1MjBhYmI1NzBjYTYzM2RlZDA5NjIxYzBh NgotLSAKMi40My4wCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KbGludXgtcmlzY3YgbWFpbGluZyBsaXN0CmxpbnV4LXJpc2N2QGxpc3RzLmluZnJhZGVh ZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9saW51eC1y aXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7A06193097 for ; Wed, 5 Jun 2024 11:41:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=140.211.166.138 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717587683; cv=none; b=bo+d3FPUS0vZLcm6kl0w+qPuYnTDdo5JJOfAI6on0BnCAdVoXzrhNFEL+Lyuv9SAby9zNyv+zlVxNmEonnV78yz5aGWEmI2IFKEUDFHROlvoiJiNdvP8lSJoeZKgH2YA5lzxAvwx5UiiA4+zBn9NC70xCdRgDxf3QvcoeSpRIKk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717587683; c=relaxed/simple; bh=S3u0bjQpavDADgmw6jZLWSNIcNZGzJbh3evgRUz4DTk=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type; b=rPtS8aU01y3f4TdNJ413jye88E3cLudEqHhK6gQbeax6qqqflJjZI1IpxgCVaozjQ6CHdbsU0moP4viWrw+Bfr1XL6mHjesmMarCZ8SXOJXJuWsK9VAhqZfNdh//A68SAgsxtXGDmXawpcnk0gwsh25ahvBZgZYbktmbl1MotB8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LaL7bLdq; arc=none smtp.client-ip=140.211.166.138 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LaL7bLdq" Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 5BF08820B5 for ; Wed, 5 Jun 2024 11:41:21 +0000 (UTC) X-Virus-Scanned: amavis at osuosl.org X-Spam-Flag: NO X-Spam-Score: -4.401 X-Spam-Level: Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with ESMTP id 0RPp5qeBQxYl for ; Wed, 5 Jun 2024 11:41:20 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=145.40.73.55; helo=sin.source.kernel.org; envelope-from=bjorn@kernel.org; receiver= DMARC-Filter: OpenDMARC Filter v1.4.2 smtp1.osuosl.org 8695A820AD Authentication-Results: smtp1.osuosl.org; dmarc=pass (p=none dis=none) header.from=kernel.org DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 8695A820AD Authentication-Results: smtp1.osuosl.org; dkim=pass (2048-bit key, unprotected) header.d=kernel.org header.i=@kernel.org header.a=rsa-sha256 header.s=k20201202 header.b=LaL7bLdq Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by smtp1.osuosl.org (Postfix) with ESMTPS id 8695A820AD for ; Wed, 5 Jun 2024 11:41:19 +0000 (UTC) Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id DDD41CE1774; Wed, 5 Jun 2024 11:41:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8A7DC3277B; Wed, 5 Jun 2024 11:41:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1717587674; bh=S3u0bjQpavDADgmw6jZLWSNIcNZGzJbh3evgRUz4DTk=; h=From:To:Cc:Subject:Date:From; b=LaL7bLdq+6cwrcQHLBpD94M8DbKhYUAqVsushnVvy0cLD86AL6hUnHO3M7GhiEeCh yOli8oZSSzQglUtdfZ9lQdxULAIbKEkNV9nUt8mivwieLLYJthz4StgifM6pNibyp8 XyqvB20qwl3nEV7TZWuL6SUKTpv8NL8vfrL/PGaUY3Bd8sqdoNPunneetDIZKUKIOh ZpRqWoiS5ox4iSnObOMf9bR4taHjlUzZR5El7C38znb4weoT7oDvB8xt+sm302YqBT fHPt/fuN+mI6QageH31bWvLU/ml0sWxrVS6X1C4IDyR8yEkzb+/Z6P1Q4BE82JbCBj KdcdPmXk4qenw== From: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= To: Alexandre Ghiti , Albert Ou , David Hildenbrand , Palmer Dabbelt , Paul Walmsley , linux-riscv@lists.infradead.org, Oscar Salvador Cc: =?UTF-8?q?Bj=C3=B6rn=20T=C3=B6pel?= , Andrew Bresticker , Chethan Seshadri , Lorenzo Stoakes , Santosh Mamila , Sivakumar Munnangi , Sunil V L , linux-kernel@vger.kernel.org, linux-mm@kvack.org, virtualization@lists.linux-foundation.org Subject: [PATCH v4 00/11] riscv: Memory Hot(Un)Plug support Date: Wed, 5 Jun 2024 13:40:43 +0200 Message-ID: <20240605114100.315918-1-bjorn@kernel.org> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: virtualization@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Björn Töpel ================================================================ Memory Hot(Un)Plug support (and ZONE_DEVICE) for the RISC-V port ================================================================ (For the restless folks: change log in the bottom!) Introduction ============ To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory hot(un)plug allows for increasing and decreasing the size of physical memory available to a machine at runtime." This series adds memory hot(un)plugging, and ZONE_DEVICE support for the RISC-V Linux port. MM configuration ================ RISC-V MM has the following configuration: * Memory blocks are 128M, analogous to x86-64. It uses PMD ("hugepage") vmemmaps. From that follows that 2M (PMD) worth of vmemmap spans 32768 pages á 4K which gets us 128M. * The pageblock size is the minimum minimum virtio_mem size, and on RISC-V it's 2M (2^9 * 4K). Implementation ============== The PGD table on RISC-V is shared/copied between for all processes. To avoid doing page table synchronization, the first patch (patch 1) pre-allocated the PGD entries for vmemmap/direct map. By doing that the init_mm PGD will be fixed at kernel init, and synchronization can be avoided all together. The following two patches (patch 2-3) does some preparations, followed by the actual MHP implementation (patch 4-5). Then, MHP and virtio-mem are enabled (patch 6-7), and finally ZONE_DEVICE support is added (patch 8). MHP and locking =============== TL;DR: The MHP does not step on any toes, except for ptdump. Additional locking is required for ptdump. Long version: For v2 I spent some time digging into init_mm synchronization/update. Here are my findings, and I'd love them to be corrected if incorrect. It's been a gnarly path... The `init_mm` structure is a special mm (perhaps not a "real" one). It's a "lazy context" that tracks kernel page table resources, e.g., the kernel page table (swapper_pg_dir), a kernel page_table_lock (more about the usage below), mmap_lock, and such. `init_mm` does not track/contain any VMAs. Having the `init_mm` is convenient, so that the regular kernel page table walk/modify functions can be used. Now, `init_mm` being special means that the locking for kernel page tables are special as well. On RISC-V the PGD (top-level page table structure), similar to x86, is shared (copied) with user processes. If the kernel PGD is modified, it has to be synched to user-mode processes PGDs. This is avoided by pre-populating the PGD, so it'll be fixed from boot. The in-kernel pgd regions are documented in `Documentation/arch/riscv/vm-layout.rst`. The distinct regions are: * vmemmap * vmalloc/ioremap space * direct mapping of all physical memory * kasan * modules, BPF * kernel Memory hotplug is the process of adding/removing memory to/from the kernel. Adding is done in two phases: 1. Add the memory to the kernel 2. Online memory, making it available to the page allocator. Step 1 is partially architecture dependent, and updates the init_mm page table: * Update the direct map page tables. The direct map is a linear map, representing all physical memory: `virt = phys + PAGE_OFFSET` * Add a `struct page` for each added page of memory. Update the vmemmap (virtual mapping to the `struct page`, so we can easily transform a kernel virtual address to a `struct page *` address. >From an MHP perspective, there are two regions of the PGD that are updated: * vmemmap * direct mapping of all physical memory The `struct mm_struct` has a couple of locks in play: * `spinlock_t page_table_lock` protects the page table, and some counters * `struct rw_semaphore mmap_lock` protect an mm's VMAs Note again that `init_mm` does not contain any VMAs, but still uses the mmap_lock in some places. The `page_table_lock` was originally used to to protect all pages tables, but more recently a split page table lock has been introduced. The split lock has a per-table lock for the PTE and PMD tables. If split lock is disabled, all tables are guarded by `mm->page_table_lock` (for user processes). Split page table locks are not used for init_mm. MHP operations is typically synchronized using `DEFINE_STATIC_PERCPU_RWSEM(mem_hotplug_lock)`. Actors ------ The following non-MHP actors in the kernel traverses (read), and/or modifies the kernel PGD. * `ptdump` Walks the entire `init_mm`, via `ptdump_walk_pgd()` with the `mmap_write_lock(init_mm)` taken. Observation: ptdump can race with MHP, and needs additional locking to avoid crashes/races. * `set_direct_*` / `arch/riscv/mm/pageattr.c` The `set_direct_*` functionality is used to "synchronize" the direct map to other kernel mappings, e.g. modules/kernel text. The direct map is using "as large huge table mappings as possible", which means that the `set_direct_*` might need to split the direct map. The `set_direct_*` functions operates with the `mmap_write_lock(init_mm)` taken. Observation: `set_direct_*` uses the direct map, but will never modify the same entry as MHP. If there is a mapping, that entry will never race with MHP. Further, MHP acts when memory is offline. * HVO / `mm/hugetlb_vmemmap` HVO optimizes the backing `struct page` for hugetlb pages, which means changing the "vmemmap" region. HVO can split (merge?) a vmemmap pmd. However, it will never race with MHP, since HVO only operates at online memory. HVO cannot touch memory being MHP added or removed. * `apply_to_page_range` Walks a range, creates pages and applies a callback (setting permissions) for the page. When creating a table, it might use `int __pte_alloc_kernel(pmd_t *pmd)` which takes the `init_mm.page_table_lock` to synchronize pmd populate. Used by: `mm/vmalloc.c` and `mm/kasan/shadow.c`. The KASAN callback takes the `init_mm.page_table_lock` to synchronize pte creation. Observations: `apply_to_page_range` applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP. * `apply_to_existing_page_range` Walks a range, applies a callback (setting permissions) for the page (no page creation). Used by: `kernel/bpf/arena.c` and `mm/kasan/shadow.c`. The KASAN callback takes the `init_mm.page_table_lock` to synchronize pte creation. *Not* affected by MHP regions. * `apply_to_existing_page_range` applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP regions. * `ioremap_page_range` and `vmap_page_range` Uses the same internal function, and might create table entries at the "vmalloc/ioremap space" region. Can call `__pte_alloc_kernel()` which takes the `init_mm.page_table_lock` synchronizing pmd populate in the region. *Not* affected by MHP regions. Summary: * MHP add will never modify the same page table entries, as any of the other actors. * MHP remove is done when memory is offlined, and will not clash with any of the actors. * Functions that walk the entire kernel page table need synchronization * It's sufficient to add the MHP lock ptdump. Testing ======= This series adds basic DT supported hotplugging. There is a QEMU series enabling MHP for the RISC-V "virt" machine here: [1] ACPI/MSI support is still in the making for RISC-V, and prior proper (ACPI) PCI MSI support lands [2] and NUMA SRAT support [3], it hard to try it out. I've prepared a QEMU branch with proper ACPI GED/PC-DIMM support [4], and a this series with the required prerequisites [5] (AIA, ACPI AIA MADT, ACPI NUMA SRAT). To test with virtio-mem, e.g.: | qemu-system-riscv64 \ | -machine virt,aia=aplic-imsic \ | -cpu rv64,v=true,vlen=256,elen=64,h=true,zbkb=on,zbkc=on,zbkx=on,zkr=on,zkt=on,svinval=on,svnapot=on,svpbmt=on \ | -nodefaults \ | -nographic -smp 8 -kernel rv64-u-boot.bin \ | -drive file=rootfs.img,format=raw,if=virtio \ | -device virtio-rng-pci \ | -m 16G,slots=3,maxmem=32G \ | -object memory-backend-ram,id=mem0,size=16G \ | -numa node,nodeid=0,memdev=mem0 \ | -serial chardev:char0 \ | -mon chardev=char0,mode=readline \ | -chardev stdio,mux=on,id=char0 \ | -device pci-serial,id=serial0,chardev=char0 \ | -object memory-backend-ram,id=vmem0,size=2G \ | -device virtio-mem-pci,id=vm0,memdev=vmem0,node=0 where "rv64-u-boot.bin" is U-boot with EFI/ACPI-support (use [6] if you're lazy). In the QEMU monitor: | (qemu) info memory-devices | (qemu) qom-set vm0 requested-size 1G ...to test DAX/KMEM, use the follow QEMU parameters: | -object memory-backend-file,id=mem1,share=on,mem-path=virtio_pmem.img,size=4G \ | -device virtio-pmem-pci,memdev=mem1,id=nv1 and the regular ndctl/daxctl dance. If you're brave to try the ACPI branch, add "acpi=on" to "-machine virt", and test PC-DIMM MHP (in addition to virtio-{p},mem): In the QEMU monitor: | (qemu) object_add memory-backend-ram,id=mem1,size=1G | (qemu) device_add pc-dimm,id=dimm1,memdev=mem1 You can also try hot-remove with some QEMU options, say: | -object memory-backend-file,id=mem-1,size=256M,mem-path=/pagesize-2MB | -device pc-dimm,id=mem1,memdev=mem-1 | -object memory-backend-file,id=mem-2,size=1G,mem-path=/pagesize-1GB | -device pc-dimm,id=mem2,memdev=mem-2 | -object memory-backend-file,id=mem-3,size=256M,mem-path=/pagesize-2MB | -device pc-dimm,id=mem3,memdev=mem-3 Remove "acpi=on" to run with DT. Thanks to Alex, Andrew, David, and Oscar for all comments/tests/fixups. Changelog ========= v3->v4: Collect tags 0002-riscv-mm-Pre-allocate-vmemmap-direct-map-kasan-PG * Pre-allocate KASAN PGD entries as well (Alex) 0005-riscv-mm-Add-pfn_to_kaddr-implementation * New patch, add missing function for KASAN 0006-riscv-mm-Add-memory-hotplugging-support * Use pXXX_get() instead of READ_ONCE (Alex) * Separated vmemmap storage and pgtable allocation. Use matching alloc/dealloc functions (e.g. ptdesc alloc/delloc) with also avoids WARNs ;-) * Cleaned up the reserved page freeing (Alex) 0008-riscv-Enable-memory-hotplugging-for-RISC-V * Simplified the dependencies (Alex) * Added ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE 0011-riscv-Enable-DAX-VMEMMAP-optimization * New patch, added ARCH_WANT_OPTIMIZE_DAX_VMEMMAP v2->v3: Collect tags 0001-riscv-mm-Properly-forward-vmemmap_populate-altmap * New patch, split from attribute change (Alex, Oscar) 0005-riscv-mm-Add-memory-hotplugging-support * Fix page table free to take PG_reserved into consideration (David, Oscar) * Depend on SPARSE_VMEMMAP (Alex, David, Oscar) * Proper TLB flushing (Alex) 0009-riscv-mm-Add-support-for-ZONE_DEVICE * Fixed 32b build v1->v2: Reviewed a lot of MHP locking scenarios Various config build issues (e.g. build !CONFIG_MEMORY_HOTPLUG) (Andrew) Added arch_get_mappable_range() implementation Acquire MHP lock for ptdump, analogous to arm64 ACPI MHP tests Add ZONE_DEVICE patch References ========== [1] https://lore.kernel.org/qemu-devel/20240521105635.795211-1-bjorn@kernel.org/ [2] https://lore.kernel.org/linux-riscv/20240501121742.1215792-1-sunilvl@ventanamicro.com/ [3] https://lore.kernel.org/linux-riscv/cover.1713778236.git.haibo1.xu@intel.com/ [4] https://github.com/bjoto/qemu/commits/virtio-mem-pc-dimm-mhp-acpi-v2/ [5] https://github.com/bjoto/linux/commits/mhp-v4-acpi [6] https://github.com/bjoto/riscv-rootfs-utils/tree/acpi Thanks, Björn Björn Töpel (11): riscv: mm: Properly forward vmemmap_populate() altmap parameter riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries riscv: mm: Change attribute from __init to __meminit for page functions riscv: mm: Refactor create_linear_mapping_range() for memory hot add riscv: mm: Add pfn_to_kaddr() implementation riscv: mm: Add memory hotplugging support riscv: mm: Take memory hotplug read-lock during kernel page table dump riscv: Enable memory hotplugging for RISC-V virtio-mem: Enable virtio-mem for RISC-V riscv: mm: Add support for ZONE_DEVICE riscv: Enable DAX VMEMMAP optimization arch/riscv/Kconfig | 5 + arch/riscv/include/asm/kasan.h | 4 +- arch/riscv/include/asm/mmu.h | 4 +- arch/riscv/include/asm/page.h | 5 + arch/riscv/include/asm/pgtable-64.h | 20 ++ arch/riscv/include/asm/pgtable-bits.h | 1 + arch/riscv/include/asm/pgtable.h | 19 +- arch/riscv/mm/init.c | 347 +++++++++++++++++++++++--- arch/riscv/mm/ptdump.c | 3 + drivers/virtio/Kconfig | 2 +- 10 files changed, 364 insertions(+), 46 deletions(-) base-commit: e2c79b4c5c4d83520abb570ca633ded09621c0a6 -- 2.43.0