From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 873C72D8393 for ; Fri, 29 May 2026 12:56:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=217.140.110.172 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780059371; cv=none; b=ML6tlTUdsI3d/L5vWoeGrlC60RKgCB2b+ydc7saKoWYLfAiIwuDthirniEk1WA3MIfg2pvJLRjFo/jBHPO0Gq5g4fxHA2G+EArIQ2+ONglyR44CU0q/qOKdDJaiE3mKRdWimDQI3ZYh2S2YhG/UxT2fdVzDFNXM+IU7YN7iPQWg= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780059371; c=relaxed/simple; bh=n9Zij2lTC42O0asoob6by1uG5qjwnMEU9gT53E+5GRU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type:Content-Disposition; b=aLwPIwajLcUdjpNmdgoTHXDyWKEaL5WoIIXx/AcaJJx72i+363AeJqlp/5ZDDB9cZc4dNuaXyotNJM7iy5a+lOooQIWF3F9ut63/tbDGjYqryYb7tfTL/iwFDFG1PV7Q9ZE3LCeoUyPxXtCEF4fzlDP/NZJAqtVZbncXci4dPxg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com; spf=pass smtp.mailfrom=arm.com; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b=qd2CTztf; arc=none smtp.client-ip=217.140.110.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=arm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=arm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=arm.com header.i=@arm.com header.b="qd2CTztf" Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1D25B2247; Fri, 29 May 2026 05:56:03 -0700 (PDT) Received: from devkitleo.cambridge.arm.com (devkitleo.cambridge.arm.com [10.1.196.90]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DE8F63F632; Fri, 29 May 2026 05:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=arm.com; s=foss; t=1780059368; bh=n9Zij2lTC42O0asoob6by1uG5qjwnMEU9gT53E+5GRU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qd2CTztfz4vNPG0hjcltiRX7d1CN+N7kVrQbZgWBMumpPVNtiJl0Z9e1m6kpB6cim OFhbZ9KfEgYrH5GzNV6okJXqNqC6E07ZsSCimFczPiJ1PQM9eiGT1cZIweogUM1h9D WV7mS0/LoD9/gK+rxp6oMAU73NmPzbLjqUEXr9gQ= From: Leonardo Bras To: Marc Zyngier Cc: Leonardo Bras , Oliver Upton , Joey Gouly , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Fuad Tabba , Raghavendra Rao Ananta , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 0/2] Optimize S2 page splitting Date: Fri, 29 May 2026 13:56:03 +0100 Message-ID: X-Mailer: git-send-email 2.54.0 In-Reply-To: <865x46v1w9.wl-maz@kernel.org> References: <20260515195904.2466381-1-leo.bras@arm.com> <87o6ifaf5z.wl-maz@kernel.org> <865x46v1w9.wl-maz@kernel.org> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: 8bit On Fri, May 29, 2026 at 01:25:10PM +0100, Marc Zyngier wrote: > On Thu, 28 May 2026 18:00:48 +0100, > Leonardo Bras wrote: > > > > On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote: > > > On Fri, 15 May 2026 20:59:01 +0100, > > > Leonardo Bras wrote: > > > > > > > > While playing with dirty-bit tracking, I decided to take a look on how page > > > > splitting works. Found out all entries are walked, even though we can infer, > > > > for instance that: > > > > - If a level-3 entry is walked, it means the parent level-2 entry is split > > > > - If a split just succeeded in an table entry, it means all children nodes > > > > are already split > > > > > > > > So I tried to optimize it in a way that it does not break other users. > > > > > > > > My main idea is to introduce positive return values that hint to the > > > > pagetable walking mechanism that either siblings or children can be > > > > skipped. That should be contained to the visitor function, that returns > > > > zero if no error was detected. > > > > > > > > Numbers on above optimization are promising: > > > > A 1GB VM, running on the model, splitting all at the beginning > > > > (no manual protect): > > > > - Memory was already split (4k pages): -97.33% runtime (-172ms) - 20 runs > > > > - THP backed memory: -19.82% runtime (-153ms) - 10 runs > > > > - 1x1GB hugetlb memory: -20.65% runtime (-150ms) - 10 runs > > > > > > > > > > I haven't looked at the changes in details, but the methodology is > > > quite flawed. For a start, measuring anything on a software model > > > (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs > > > are completely different from a HW implementation, and even the notion > > > of time is pretty inconsistent. > > > > > > Please run this on actual HW. I'm sure your employer can give you > > > access to one of these mythical arm64 toys. Measure things from > > > userspace, not from the kernel, so that you have all the overheads. > > > Don't add console output, because that will make things far worse. > > > > > > I'm sure you can hack one of the selftests for this purpose. > > > > Hello Marc, > > > > I have ran some tests in real hardware (AmpereOne) and calling from > > userspace as you suggested. > > > > The tests create a 64GB VM, using either regular pages (no split needed), > > THP backed memory or Hugetlb. > > > > Those tests measured the times for the whole syscall that happens to do > > eager page splitting, in this case, > > 1 - Manual protect and init set: On every KVM_CLEAR_DIRTY_LOG, if the page > > was requested to be cleaned, it is also split. > > 2 - Otherwise, on dirty log enable (aka KVM_SET_USER_MEMORY_REGION2 with > > KVM_MEM_LOG_DIRTY_PAGES set in flags), for the whole memslot is split > > at once. > > > > Tests reflect new suggestions I got in this patchset: instead of using > > return values, I am using flags to skip whole levels, or skipping children > > nodes. I tested those 2 kernels, on top of vanilla: > > a - Skip levels > > b - Skip levels, and skip children node > > > > Results averaged across 4k, 16k and 64k pages, a percentage of time > > saved, compared to vanilla kernel. More is better. > > > > For (1), we have > > regular pages: 26.7% (a) and 25.3% (b) > > THP: 13.2% (a) and 11.9% (b) > > Hugetlb: 14.3% (a) and 13.2% (b) > > > > For (2), we have: > > regular pages: 33.1% (a) and 35.0% (b) > > THP: 11.9% (a) and 10.7% (b) > > Hugetlb: 13.4% (a) and 13.2% (b) > > > > On above results, I could notice about 1% overhead showing that skipping > > child nodes (testing flag) ends up being counter-productive compared to > > just skipping levels. The only case this does not happen is (2a), but it's > > not clear on why that happens. > > > > Based on that, I plan to remove the skip_child patch, and send a v2 with a > > flag-based mechanism, which results are shown above. > > > > Please let me know of any thoughts, suggestions or ideas you might have > > about it. > > Right, that seems compelling. Awesome! > I'd suggest you send a new version with: > > - a pointer to the test and its the exact invocation parameters so > that people other than you can reproduce it > > - these numbers with a bit of analysis explaining where the gains are > > With that, we can look at it again and compare results on different > workloads and HW. > Noted! Will attach test parameters and full results, describing where the gains come from. Thanks! Leo