From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54255346A15 for ; Wed, 8 Jul 2026 03:11:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480307; cv=none; b=OatYf2H4ff13RMWxPTSCagDEH07UELWwylneivDe+X2Rx/kEUp74Uys/Wb47m3g0F2GSMbOQI8y3mA4fOv1H2VCjCL5G4Ko9GOAFc4WVZ8r3LcuB5Yr9eYEr3dREL5VUpMLmBZTY/pZy61He+LZk2K2ZKstoXFiB9jhp+9i4B9s= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480307; c=relaxed/simple; bh=iaIljj69RU4Gmtv58DLd196XZ/6044XHMP5PYXbkemE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MNSaKq/R/yruP4yYdXuJ8IoQNFmOQL+amcQkp++7gm3+qhJtnXgtc/KU3F2maRlOkv0SM6VTkSWZdv74nKJM76tx2F2n4eJjVltMMI0DrrobWc9x+Nt612FdmsMHQKuw9Tmw5PIpOFGsfgGDKsvUwg3ZApr+A0cGXSBp7qMWh60= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GExO9Xjx; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GExO9Xjx" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-380c178acfbso216140a91.1 for ; Tue, 07 Jul 2026 20:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1783480301; x=1784085101; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=HIZhGnigxQeniz/pqQO7mXj7CCL2nhO/uVgxmBbLKD8=; b=GExO9Xjx0/3p8pi2Ss9/+poWbzEfMZf93B6iUY4eOQ1UKusYYH2Mt1Vz7E7SHTCtot 99uo3xs8E3ngMF9zJVS1yuuLcSmAkIC6wa7xPBfEeOeWjXV86EZYzLG8JwsgYsitsOab 3EQvDdYsC3zc5acZWOJjeYZXuFW+xs72qJ3Xn4xoI5c7jyrxqCTUoEhKRcUpHPmjUp54 VHMhVzAdWtMfT+K4t8uj2xCNqE0UcYyAafrjsFGNCcegCCReDNPAifXP8g5uXpsAFtF2 uUTm4pmnOpDBayAv/h59ihNck6SUMZoKJqdHht2wTTrPCQJdDVPBFYc6wBWdY03Yj8Sc OFyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783480301; x=1784085101; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=HIZhGnigxQeniz/pqQO7mXj7CCL2nhO/uVgxmBbLKD8=; b=Wi0T7q3M6CaGmccCirTcu6PA7tDNTYAk+Rx8ZHfLIqG5Bd/0I01Q+7dwleqCrZD1sV hSewMTBcHF6ZSgYXQPvaxWqZkwcrl1hq9szBkwmtfF4qJVwJnoqWoxPPu0feYAE+E7Jf RE/vfEtx3oOn61F+5qMmJNt1c5Wz+VnZ6m2QIGLxejZxNhERjWFL1gi1f8V6i2ODjKfI 1yOM9q/C6/NA5liBaEit3eoVs6p05RVTW9LmszhE7hLv4K9bbx+wHfCWpLrhYQmG01UM 1A5Hq4MUJs/PV9PjvfwYMCLRZAogkxXlrUkFCJHUlY+dPlZ8IDV4pD0zCkJ+dEo7DHM4 HxEw== X-Forwarded-Encrypted: i=1; AHgh+RpGYGiC45xZ0kiGVUH8L2A3Qipg9oMIl+mn0ine+oAIsOerhZYsnsvrW7GxpZEFlB8HwSBIpKtrleZgEgU=@vger.kernel.org X-Gm-Message-State: AOJu0YxSjUf3ykCAJIyVxcNyI3NcELSN1yH7IyVKajfrsf8ML+8T+Ad6 FNzCsQN4HWu0JkWHe+Fh7sHRAu2C+WZrjbIPLNtLtz5xo71J1YsuFLvZFipp27H5/SU/5Iq+c1W YXAccG6/nMVx9WAcM4+BTgg== X-Received: from pjzt12.prod.google.com ([2002:a17:90b:58c:b0:381:1463:3959]) (user=jthoughton job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1d04:b0:37f:fd1f:d30f with SMTP id 98e67ed59e1d1-3893fe5d4a2mr640925a91.12.1783480300469; Tue, 07 Jul 2026 20:11:40 -0700 (PDT) Date: Wed, 8 Jul 2026 03:11:18 +0000 In-Reply-To: <20260708031129.3503195-1-jthoughton@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260708031129.3503195-1-jthoughton@google.com> X-Mailer: git-send-email 2.55.0.795.g602f6c329a-goog Message-ID: <20260708031129.3503195-9-jthoughton@google.com> Subject: [PATCH 08/18] arm64: Implement try_update_vmemmap_pte using the AF trick From: James Houghton To: Will Deacon , Catalin Marinas , Muchun Song , Oscar Salvador Cc: Nikos Nikoleris , Linu Cherian , Mark Rutland , David Hildenbrand , Andrew Morton , Ryan Roberts , Nanyong Sun , Yu Zhao , Frank van der Linden , David Rientjes , James Houghton , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" try_update_vmemmap_pte() must modify vmemmap PTEs without introducing a time window where other CPUs on the system might fault. Normally a break-before-make sequence is required to avoid conflicts with cached translations. However, if we can guarantee that the existing translation cannot be cached, a BBM sequence is not needed. Translations with the AF unset may not be cached (see Arm ARM Rule DWZCQ); the implementation of try_update_vmemmap_pte() on arm64 takes advantage of this fact to replace a PTE without BBM and therefore without leaving a window open where PE might fault on this translation. Of course, if some CPUs on the system do not support HW AF management, clearing the AF will introduce potential faults. system_supports_hvo() will return false if any CPUs on the system do not support HW AF. Signed-off-by: James Houghton --- arch/arm64/include/asm/pgtable.h | 53 ++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5f21d3a738ee..7b11aa41d0a0 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1302,8 +1302,7 @@ static inline void __pte_clear(struct mm_struct *mm, __set_pte(ptep, __pte(0)); } -static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_clear_young(pte_t *ptep) { pte_t old_pte, pte; @@ -1315,7 +1314,13 @@ static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma, pte_val(old_pte), pte_val(pte)); } while (pte_val(pte) != pte_val(old_pte)); - return pte_young(pte); + return pte; +} + +static inline bool __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + return pte_young(__ptep_clear_young(ptep)); } static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1793,6 +1798,48 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define __HAVE_ARCH_TRY_UPDATE_VMEMMAP_PTE +static inline int try_update_vmemmap_pte(unsigned long addr, pte_t *ptep, + const pte_t pte) +{ + const int max_attempts = 16; + int attempts = 0; + pte_t old_pte; + + if (!system_supports_hvo()) + return -EOPNOTSUPP; + + /* This routine is only to be used for valid-to-valid transitions. */ + if (WARN_ON_ONCE(!pte_valid(pte))) + return -EINVAL; + + old_pte = __ptep_get(ptep); + + do { + if (WARN_ON_ONCE(!pte_valid(old_pte))) + return -EINVAL; + + /* We should never get a contiguous PTE here. */ + if (WARN_ON_ONCE(pte_valid_cont(old_pte))) + return -EINVAL; + + if (pte_young(old_pte)) { + /* __ptep_clear_young() returns the overwritten PTE */ + old_pte = pte_mkold(__ptep_clear_young(ptep)); + + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); + } + /* + * Translations without AF cannot be cached, so we can replace + * them without BBM. + */ + } while (!try_cmpxchg_relaxed(&pte_val(*ptep), &pte_val(old_pte), + pte_val(pte)) && + ++attempts < max_attempts); + + return attempts == max_attempts ? -EAGAIN : 0; +} + #define clear_full_ptes clear_full_ptes static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full) -- 2.55.0.795.g602f6c329a-goog