From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8CCCFD18138 for ; Mon, 14 Oct 2024 17:34:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=bQQSHSD7pMPEnyh5pSYNZBMzysQwlv1Sj6SfqEv1kug=; b=BjOdZf4gnj9d6PAcxtHntlgjiK MxxAwGJb/FuZ4TX0ZxXfYrdKXul5T2t/aUa2woWgEnoTvKKcfcFm/HVrO5y0Kqk6HEKhk5QAFN1/a TTcNH0za/hDtumZIg3WCGgtX7GDUfjy7aXWmrrSqNtPlNjjaJRNmTu9ag9kLDGm5uQ6n6BoE6+lJH toaIJYwU/B04txNQFq8QXQxTP4agzm7LYuBAWzvxsoN43DceORgAA0O6vdNZl/asrz0M/ExbqlvTo WhqOlPy1WD6laB8yZpLXGgEkSYjTJM9uFmCN7Ad/KgZgOgnR2M+dstDZ7mzRkxJ3IZQj3wuvq544V xfQKStxQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t0Oxe-000000063kT-0sIp; Mon, 14 Oct 2024 17:34:26 +0000 Received: from mail-pl1-x62e.google.com ([2607:f8b0:4864:20::62e]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t0OwD-000000063U6-0PKY for linux-arm-kernel@lists.infradead.org; Mon, 14 Oct 2024 17:32:58 +0000 Received: by mail-pl1-x62e.google.com with SMTP id d9443c01a7336-20ca388d242so21283745ad.2 for ; Mon, 14 Oct 2024 10:32:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=broadcom.com; s=google; t=1728927176; x=1729531976; darn=lists.infradead.org; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:from:to:cc:subject:date:message-id:reply-to; bh=bQQSHSD7pMPEnyh5pSYNZBMzysQwlv1Sj6SfqEv1kug=; b=GUcn7eMKp1uD1OwT8qiV55gZhifVdGEYmWJQxiqexuZ5q4rJwAS1SgTga+eqkVIsvG 65nhjJ8n2811PP86tQ5krVm8NgMPaSibkwUalO/T2Ek756XgPcaK5BNsTeB8RRgNtoUw OUbKGrF0NLZcpafj00gWfBmVyPkGSXyAINo8Q= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728927176; x=1729531976; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bQQSHSD7pMPEnyh5pSYNZBMzysQwlv1Sj6SfqEv1kug=; b=hUqTzvqNDAU73VtuAKLEaYpGrST3C6FxDiZjdchrNFFVqqquFTMwXA4NQNjrmUZo8L beiDUrTV08/94jBWkqfXReBRbHhDnPDSuJLxi/MMWbGz+tx2e1pbNgLU9gRHPgZ8ps4l FL2syfUXMlFASYPErNnxsZ7SahOpcRaqEx1cZLtEx3KR3Z/IqVTU5frzlRDjA35Er4nz 8jbFzMNSW4Dy7WtFnxz7pdT9CCyqlwxkFBL35Y0iLPQYqpaKQM9E9/nqFeKxDlZhmRmG spHtBv0hHUHA66RnG8Y6pqzFJXVT2Wk1DX9bHEGbDGXFBleJQ8p5uTNEaTWey8ysKDxc L59A== X-Gm-Message-State: AOJu0Yx+DZpxlQe2YN8u2u5w1yh7j8IPbGIOsCTCyTe1P+qaQL9vL7Uf FJbTPj9KivZV44ZTfmCnUO2J9GzXq+kq6h6x6t+sSWuP9gWbJqw7sn522zx+4Q== X-Google-Smtp-Source: AGHT+IEns2eqkxyiAg986XLyfsK4RpsMx2bwZL6BIHWTOuBJ8RnZHMXrUX4qvjPsrHC+Vm+yqwVq0A== X-Received: by 2002:a17:903:1c6:b0:20c:e262:2560 with SMTP id d9443c01a7336-20ce26227c9mr76987585ad.50.1728927175840; Mon, 14 Oct 2024 10:32:55 -0700 (PDT) Received: from [10.67.48.245] ([192.19.223.252]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20c8c0eb7bbsm68863045ad.162.2024.10.14.10.32.54 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 14 Oct 2024 10:32:55 -0700 (PDT) Message-ID: <3e742298-2f38-496c-ba63-1e30d16318c6@broadcom.com> Date: Mon, 14 Oct 2024 10:32:53 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC PATCH v1 00/57] Boot-time page size selection for arm64 To: Ryan Roberts , Andrew Morton , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Greg Marsden , Ivan Ivanov , Kalesh Singh , Marc Zyngier , Mark Rutland , Matthias Brugger , Miroslav Benes , Will Deacon Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <20241014105514.3206191-1-ryan.roberts@arm.com> Content-Language: en-US From: Florian Fainelli Autocrypt: addr=florian.fainelli@broadcom.com; keydata= xsBNBFPAG8ABCAC3EO02urEwipgbUNJ1r6oI2Vr/+uE389lSEShN2PmL3MVnzhViSAtrYxeT M0Txqn1tOWoIc4QUl6Ggqf5KP6FoRkCrgMMTnUAINsINYXK+3OLe7HjP10h2jDRX4Ajs4Ghs JrZOBru6rH0YrgAhr6O5gG7NE1jhly+EsOa2MpwOiXO4DE/YKZGuVe6Bh87WqmILs9KvnNrQ PcycQnYKTVpqE95d4M824M5cuRB6D1GrYovCsjA9uxo22kPdOoQRAu5gBBn3AdtALFyQj9DQ KQuc39/i/Kt6XLZ/RsBc6qLs+p+JnEuPJngTSfWvzGjpx0nkwCMi4yBb+xk7Hki4kEslABEB AAHNMEZsb3JpYW4gRmFpbmVsbGkgPGZsb3JpYW4uZmFpbmVsbGlAYnJvYWRjb20uY29tPsLB IQQQAQgAywUCZWl41AUJI+Jo+hcKAAG/SMv+fS3xUQWa0NryPuoRGjsA3SAUAAAAAAAWAAFr ZXktdXNhZ2UtbWFza0BwZ3AuY29tjDAUgAAAAAAgAAdwcmVmZXJyZWQtZW1haWwtZW5jb2Rp bmdAcGdwLmNvbXBncG1pbWUICwkIBwMCAQoFF4AAAAAZGGxkYXA6Ly9rZXlzLmJyb2FkY29t Lm5ldAUbAwAAAAMWAgEFHgEAAAAEFQgJChYhBNXZKpfnkVze1+R8aIExtcQpvGagAAoJEIEx tcQpvGagWPEH/2l0DNr9QkTwJUxOoP9wgHfmVhqc0ZlDsBFv91I3BbhGKI5UATbipKNqG13Z TsBrJHcrnCqnTRS+8n9/myOF0ng2A4YT0EJnayzHugXm+hrkO5O9UEPJ8a+0553VqyoFhHqA zjxj8fUu1px5cbb4R9G4UAySqyeLLeqnYLCKb4+GklGSBGsLMYvLmIDNYlkhMdnnzsSUAS61 WJYW6jjnzMwuKJ0ZHv7xZvSHyhIsFRiYiEs44kiYjbUUMcXor/uLEuTIazGrE3MahuGdjpT2 IOjoMiTsbMc0yfhHp6G/2E769oDXMVxCCbMVpA+LUtVIQEA+8Zr6mX0Yk4nDS7OiBlvOwE0E U8AbwQEIAKxr71oqe+0+MYCc7WafWEcpQHFUwvYLcdBoOnmJPxDwDRpvU5LhqSPvk/yJdh9k 4xUDQu3rm1qIW2I9Puk5n/Jz/lZsqGw8T13DKyu8eMcvaA/irm9lX9El27DPHy/0qsxmxVmU pu9y9S+BmaMb2CM9IuyxMWEl9ruWFS2jAWh/R8CrdnL6+zLk60R7XGzmSJqF09vYNlJ6Bdbs MWDXkYWWP5Ub1ZJGNJQ4qT7g8IN0qXxzLQsmz6tbgLMEHYBGx80bBF8AkdThd6SLhreCN7Uh IR/5NXGqotAZao2xlDpJLuOMQtoH9WVNuuxQQZHVd8if+yp6yRJ5DAmIUt5CCPcAEQEAAcLB gQQYAQIBKwUCU8AbwgUbDAAAAMBdIAQZAQgABgUCU8AbwQAKCRCTYAaomC8PVQ0VCACWk3n+ obFABEp5Rg6Qvspi9kWXcwCcfZV41OIYWhXMoc57ssjCand5noZi8bKg0bxw4qsg+9cNgZ3P N/DFWcNKcAT3Z2/4fTnJqdJS//YcEhlr8uGs+ZWFcqAPbteFCM4dGDRruo69IrHfyyQGx16s CcFlrN8vD066RKevFepb/ml7eYEdN5SRALyEdQMKeCSf3mectdoECEqdF/MWpfWIYQ1hEfdm C2Kztm+h3Nkt9ZQLqc3wsPJZmbD9T0c9Rphfypgw/SfTf2/CHoYVkKqwUIzI59itl5Lze+R5 wDByhWHx2Ud2R7SudmT9XK1e0x7W7a5z11Q6vrzuED5nQvkhAAoJEIExtcQpvGagugcIAJd5 EYe6KM6Y6RvI6TvHp+QgbU5dxvjqSiSvam0Ms3QrLidCtantcGT2Wz/2PlbZqkoJxMQc40rb fXa4xQSvJYj0GWpadrDJUvUu3LEsunDCxdWrmbmwGRKqZraV2oG7YEddmDqOe0Xm/NxeSobc MIlnaE6V0U8f5zNHB7Y46yJjjYT/Ds1TJo3pvwevDWPvv6rdBeV07D9s43frUS6xYd1uFxHC 7dZYWJjZmyUf5evr1W1gCgwLXG0PEi9n3qmz1lelQ8lSocmvxBKtMbX/OKhAfuP/iIwnTsww 95A2SaPiQZA51NywV8OFgsN0ITl2PlZ4Tp9hHERDe6nQCsNI/Us= In-Reply-To: <20241014105514.3206191-1-ryan.roberts@arm.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241014_103257_276401_D566FCB4 X-CRM114-Status: GOOD ( 39.43 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 10/14/24 03:55, Ryan Roberts wrote: > Hi All, > > Patch bomb incoming... This covers many subsystems, so I've included a core set > of people on the full series and additionally included maintainers on relevant > patches. I haven't included those maintainers on this cover letter since the > numbers were far too big for it to work. But I've included a link to this cover > letter on each patch, so they can hopefully find their way here. For follow up > submissions I'll break it up by subsystem, but for now thought it was important > to show the full picture. > > This RFC series implements support for boot-time page size selection within the > arm64 kernel. arm64 supports 3 base page sizes (4K, 16K, 64K), but to date, page > size has been selected at compile-time, meaning the size is baked into a given > kernel image. As use of larger-than-4K page sizes become more prevalent this > starts to present a problem for distributions. Boot-time page size selection > enables the creation of a single kernel image, which can be told which page size > to use on the kernel command line. > > Why is having an image-per-page size problematic? > ================================================= > > Many traditional distros are now supporting both 4K and 64K. And this means > managing 2 kernel packages, along with drivers for each. For some, it means > multiple installer flavours and multiple ISOs. All of this adds up to a > less-than-ideal level of complexity. Additionally, Android now supports 4K and > 16K kernels. I'm told having to explicitly manage their KABI for each kernel is > painful, and the extra flash space required for both kernel images and the > duplicated modules has been problematic. Boot-time page size selection solves > all of this. > > Additionally, in starting to think about the longer term deployment story for > D128 page tables, which Arm architecture now supports, a lot of the same > problems need to be solved, so this work sets us up nicely for that. > > So what's the down side? > ======================== > > Well nothing's free; Various static allocations in the kernel image must be > sized for the worst case (largest supported page size), so image size is in line > with size of 64K compile-time image. So if you're interested in 4K or 16K, there > is a slight increase to the image size. But I expect that problem goes away if > you're compressing the image - its just some extra zeros. At boot-time, I expect > we could free the unused static storage once we know the page size - although > that would be a follow up enhancement. > > And then there is performance. Since PAGE_SIZE and friends are no longer > compile-time constants, we must look up their values and do arithmetic at > runtime instead of compile-time. My early perf testing suggests this is > inperceptible for real-world workloads, and only has small impact on > microbenchmarks - more on this below. > > Approach > ======== > > The basic idea is to rid the source of any assumptions that PAGE_SIZE and > friends are compile-time constant, but in a way that allows the compiler to > perform the same optimizations as was previously being done if they do turn out > to be compile-time constant. Where constants are required, we use limits; > PAGE_SIZE_MIN and PAGE_SIZE_MAX. See commit log in patch 1 for full description > of all the classes of problems to solve. > > By default PAGE_SIZE_MIN=PAGE_SIZE_MAX=PAGE_SIZE. But an arch may opt-in to > boot-time page size selection by defining PAGE_SIZE_MIN & PAGE_SIZE_MAX. arm64 > does this if the user selects the CONFIG_ARM64_BOOT_TIME_PAGE_SIZE Kconfig, > which is an alternative to selecting a compile-time page size. > > When boot-time page size is active, the arch pgtable geometry macro definitions > resolve to something that can be configured at boot. The arm64 implementation in > this series mainly uses global, __ro_after_init variables. I've tried using > alternatives patching, but that performs worse than loading from memory; I think > due to code size bloat. FWIW, this paragraph was not entirely clear to me until I looked at patch 57 to see that the compile time page size selection had been retained, and could continue to be used as-is. It was somewhat implicit, but not IMHO explicit enough, not a big deal though. Great work, thanks for doing that! This makes me wonder if we could leverage any of that to have a single kernel supporting both LPAE and !LPAE on ARM 32-bit, but that still seems like somewhat more difficult, largely due to the difference in the page table descriptor format (long vs. short). -- Florian