From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BBC2FCD4F54 for ; Fri, 29 May 2026 12:25:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mG0TiHBAekQCoiSIto81YmHTibJe256DIjsJ2ZqbJpY=; b=A4B2adZxhRptf+BtBJAyaB0/68 A8YUI3iy4t+ZgZq+qnNrPISw7JXjt5+VB17LW4UWCCzB1z0WGGj7ikO7cRnCYZheBscQ+npnkEcVJ 11XsGivGJ2o3FdesFADubFgW0fGTRZ8j9OORMxzKe+Y62prO8h+hhLgZIeRlBoDloc/57OUhRVVOx K1j9wGnlF0ToPUScmo0gj9Jim5LqcU5QfEBVkNHHqgj+IjXua/Jb3X8MltHyOA6EwwGmxrR0Bao7K pbgsYOrV4aLxHHNlavgbYwayRSRln4wLHmiAQBC79TMjYwQglQ11m3RJMx2uhL5U2cVj5GdCWWW+O kzH2KNRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wSwH5-00000007NPZ-3V3u; Fri, 29 May 2026 12:25:15 +0000 Received: from tor.source.kernel.org ([172.105.4.254]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1wSwH4-00000007NPT-24z1 for linux-arm-kernel@lists.infradead.org; Fri, 29 May 2026 12:25:14 +0000 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) by tor.source.kernel.org (Postfix) with ESMTP id ADD816054F; Fri, 29 May 2026 12:25:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63FCF1F00893; Fri, 29 May 2026 12:25:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel.org; s=k20260515; t=1780057513; bh=mG0TiHBAekQCoiSIto81YmHTibJe256DIjsJ2ZqbJpY=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=OtP56GBthsMtZdogkYuQhgxvcJD9y8TmL1qKYM+KmphhvQhY1y25OJu6fDlwobbYA db+iYUcz7vdby8g+B/fm7t0YYhG3sFzFRIfGa4grvrwhpIbUBdf+aKue+DVcKkGS0k g1sMU2zqK12C5JbaYtcerv9qsXM+YhlP9oj+Tequ93/cb6vfUQf5Fg4PJgDBwVT2Md IcVDZsYnvHwQhMEKLfJQJsGG6+k3ogkj6YyhzP+8jFEVQCXmJ3GzGNcVKvJqO0SttQ QAq4fpUmIYRjQ4rG+2cXjeb5VYYf7pwj6mE3fZ1Ji+dGLF8f4jQcXB6w6RxV640gJx GKvV/DiKW+b9w== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1wSwH0-00000007P17-3Qg6; Fri, 29 May 2026 12:25:10 +0000 Date: Fri, 29 May 2026 13:25:10 +0100 Message-ID: <865x46v1w9.wl-maz@kernel.org> From: Marc Zyngier To: Leonardo Bras Cc: Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Fuad Tabba , Raghavendra Rao Ananta , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 0/2] Optimize S2 page splitting In-Reply-To: References: <20260515195904.2466381-1-leo.bras@arm.com> <87o6ifaf5z.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/30.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: leo.bras@arm.com, oupton@kernel.org, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, tabba@google.com, rananta@google.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 28 May 2026 18:00:48 +0100, Leonardo Bras wrote: > > On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote: > > On Fri, 15 May 2026 20:59:01 +0100, > > Leonardo Bras wrote: > > > > > > While playing with dirty-bit tracking, I decided to take a look on how page > > > splitting works. Found out all entries are walked, even though we can infer, > > > for instance that: > > > - If a level-3 entry is walked, it means the parent level-2 entry is split > > > - If a split just succeeded in an table entry, it means all children nodes > > > are already split > > > > > > So I tried to optimize it in a way that it does not break other users. > > > > > > My main idea is to introduce positive return values that hint to the > > > pagetable walking mechanism that either siblings or children can be > > > skipped. That should be contained to the visitor function, that returns > > > zero if no error was detected. > > > > > > Numbers on above optimization are promising: > > > A 1GB VM, running on the model, splitting all at the beginning > > > (no manual protect): > > > - Memory was already split (4k pages): -97.33% runtime (-172ms) - 20 runs > > > - THP backed memory: -19.82% runtime (-153ms) - 10 runs > > > - 1x1GB hugetlb memory: -20.65% runtime (-150ms) - 10 runs > > > > > > > I haven't looked at the changes in details, but the methodology is > > quite flawed. For a start, measuring anything on a software model > > (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs > > are completely different from a HW implementation, and even the notion > > of time is pretty inconsistent. > > > > Please run this on actual HW. I'm sure your employer can give you > > access to one of these mythical arm64 toys. Measure things from > > userspace, not from the kernel, so that you have all the overheads. > > Don't add console output, because that will make things far worse. > > > > I'm sure you can hack one of the selftests for this purpose. > > Hello Marc, > > I have ran some tests in real hardware (AmpereOne) and calling from > userspace as you suggested. > > The tests create a 64GB VM, using either regular pages (no split needed), > THP backed memory or Hugetlb. > > Those tests measured the times for the whole syscall that happens to do > eager page splitting, in this case, > 1 - Manual protect and init set: On every KVM_CLEAR_DIRTY_LOG, if the page > was requested to be cleaned, it is also split. > 2 - Otherwise, on dirty log enable (aka KVM_SET_USER_MEMORY_REGION2 with > KVM_MEM_LOG_DIRTY_PAGES set in flags), for the whole memslot is split > at once. > > Tests reflect new suggestions I got in this patchset: instead of using > return values, I am using flags to skip whole levels, or skipping children > nodes. I tested those 2 kernels, on top of vanilla: > a - Skip levels > b - Skip levels, and skip children node > > Results averaged across 4k, 16k and 64k pages, a percentage of time > saved, compared to vanilla kernel. More is better. > > For (1), we have > regular pages: 26.7% (a) and 25.3% (b) > THP: 13.2% (a) and 11.9% (b) > Hugetlb: 14.3% (a) and 13.2% (b) > > For (2), we have: > regular pages: 33.1% (a) and 35.0% (b) > THP: 11.9% (a) and 10.7% (b) > Hugetlb: 13.4% (a) and 13.2% (b) > > On above results, I could notice about 1% overhead showing that skipping > child nodes (testing flag) ends up being counter-productive compared to > just skipping levels. The only case this does not happen is (2a), but it's > not clear on why that happens. > > Based on that, I plan to remove the skip_child patch, and send a v2 with a > flag-based mechanism, which results are shown above. > > Please let me know of any thoughts, suggestions or ideas you might have > about it. Right, that seems compelling. I'd suggest you send a new version with: - a pointer to the test and its the exact invocation parameters so that people other than you can reproduce it - these numbers with a bit of analysis explaining where the gains are With that, we can look at it again and compare results on different workloads and HW. Thanks, M. -- Without deviation from the norm, progress is not possible.