From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62E18C83F1A for ; Fri, 18 Jul 2025 09:04:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0556A6B00AC; Fri, 18 Jul 2025 05:04:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 005C06B00AD; Fri, 18 Jul 2025 05:04:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E36E26B00AF; Fri, 18 Jul 2025 05:04:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CE6F96B00AC for ; Fri, 18 Jul 2025 05:04:03 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 60EBB10F950 for ; Fri, 18 Jul 2025 09:04:03 +0000 (UTC) X-FDA: 83676798366.12.5372ACA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 9855640003 for ; Fri, 18 Jul 2025 09:04:01 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752829441; a=rsa-sha256; cv=none; b=mNXFQxM3sxcyojfVHOT24HaMSOaZIvebujpz+CIcRBo0nGSKt20bWBX8NO8TDC9Svgaqru jgv8mWuNYA1feEoqu+gqA+elwYWhowHtYqIpdzax0GkiED9K6kNQKbGrZr3RTAUQY3QfPy wB4du1Li6gG4oU8it9KH9U9hNI5piao= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf11.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752829441; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DoE82x/1J7Ixr5sXY4UOiKnIofx5fPhTTQcAwnDZ1/A=; b=Lt/7sHRdlTQvaZdn8R+AIazmALWQNkVHh9x2XuBDmmnZCOwc9CvZrB+DSp0brcSGm7ZtrY Sya4V6gZ61x1zKKhz9LAq/zNkNzbIspT3gMH/N5HrQYS6odiAFwF8A4vSMzqZozfUXKLDx rAfQ6ix2BJwBtAhXhKO1XFUhsFRQ9kM= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5FAF325DE; Fri, 18 Jul 2025 02:03:53 -0700 (PDT) Received: from MacBook-Pro.blr.arm.com (unknown [10.164.18.51]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id AF5A63F66E; Fri, 18 Jul 2025 02:03:52 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, willy@infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, catalin.marinas@arm.com, will@kernel.org, Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, anshuman.khandual@arm.com, peterx@redhat.com, joey.gouly@arm.com, ioworker0@gmail.com, baohua@kernel.org, kevin.brodsky@arm.com, quic_zhenhuah@quicinc.com, christophe.leroy@csgroup.eu, yangyicong@hisilicon.com, linux-arm-kernel@lists.infradead.org, hughd@google.com, yang@os.amperecomputing.com, ziy@nvidia.com, Dev Jain Subject: [PATCH v5 7/7] arm64: Add batched versions of ptep_modify_prot_start/commit Date: Fri, 18 Jul 2025 14:32:44 +0530 Message-Id: <20250718090244.21092-8-dev.jain@arm.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: <20250718090244.21092-1-dev.jain@arm.com> References: <20250718090244.21092-1-dev.jain@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 9855640003 X-Stat-Signature: 681k5h7am5n5n5xzjd5tnenqrg5tarrr X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1752829441-264815 X-HE-Meta: U2FsdGVkX18OGkkY90mjqYX6nG+2oS/1nbdJXoAS+mOd6u9R2ukWPO6l12hhUseR1rnf9b8AAJAieIdB/JAnb32Q5+nvf7puap3FSSYsV3tQpy8jy3li6PLu+OblEaUp0aEsjRQWlWr+7P6BzHDHCK8ng1R2ToUlo8maHVIJEUMhpMySnn/5tVFaDZ2QxeAXZPXSf8W4OV7GkTbGy0mKLuIK6HAKokE+hMiL7TD+TXPpI57zX7VLFVzKbNwo43+6wTshi8I6OumoPrWAzyNync9/rXDEyvKspBLo6/YoMk68osbLEKKQ2PghgbdCBb6guIUrsaEfCFLYw5b9Tz5FjPqaOelE88g2wqs2RVVPQWn2fxttkPxw1g1LWDB4Bayu5igoM/9LKCYKEAombC+1AqfkyM4ogrB5BlVKL7D/vJGY0gIMwj13R5gZ2Qr2karbrAyLhCFnzM3lcDqCmcxvSMXrmJ/rUvnAdE4CH1VbsdxEmwtVnZ099t6Klw52uNFC8BvxWDoajS3sj/QnyDwJH8wU+XqgNP50DxE17VKHAm+H6gN1bDRbM4iCr4+nFN24ihFkIMbmjZZRoQPfL1hfg5YM+0o9Dny4l45ULVWWEXuQXvOdYrO1SQUsLDkW0Y0S8LccqGlK/4hhvcdLDsIPKIwXdXywq07SrTTrEQzG107w0uqhhGLltzdIrsf8I63emF5nMpn0TbVvJBkevuJI86uJk6V1WbxI1b0e+x2UfRGn2KVHuoAlm19lJmQqtnzZLRiyQsj0HXe/gJTiI7JTLsyrAPr09cJXoNfPZxI87n1dyNvZ4LHV0t3WPPrh16cOA1DdQmR353RSI0NbpMZWNdxTm728vLe6wP0RtNMbzdC+jW20W1/NQIADFYTYiVvRMQvdd1Rc1A01JeulK6/jFuGD5TYZEhlK0Y8UX4UkbpaM/dFzcTejzkxlgunjsMa0ISf+lFyIzB2L56rjmgP vxvS5Gp5 jI9SCSLTDZL4v/8zm/GwSlNvc+W27NutGypWfNpFja0bGne4gc7nESJYPiv1Ejb1R472B9fThML1QZdEW+ThKWTDQVxV0hsrEmOg95ZbV5ra7a8bnVoLVY7FghcIh2xgjjOZw95jcSUW50XJDsmzo29DyxlF7ejhySkOniImdO81r6q94yFgm1fJxgeyBuLyTqjUp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Override the generic definition of modify_prot_start_ptes() to use get_and_clear_full_ptes(). This helper does a TLBI only for the starting and ending contpte block of the range, whereas the current implementation will call ptep_get_and_clear() for every contpte block, thus doing a TLBI on every contpte block. Therefore, we have a performance win. The arm64 definition of pte_accessible() allows us to batch in the errata specific case: #define pte_accessible(mm, pte) \ (mm_tlb_flush_pending(mm) ? pte_present(pte) : pte_valid(pte)) All ptes are obviously present in the folio batch, and they are also valid. Override the generic definition of modify_prot_commit_ptes() to simply use set_ptes() to map the new ptes into the pagetable. Reviewed-by: Ryan Roberts Signed-off-by: Dev Jain --- arch/arm64/include/asm/pgtable.h | 10 ++++++++++ arch/arm64/mm/mmu.c | 28 +++++++++++++++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ba63c8736666..abd2dee416b3 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1643,6 +1643,16 @@ extern void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t new_pte); +#define modify_prot_start_ptes modify_prot_start_ptes +extern pte_t modify_prot_start_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr); + +#define modify_prot_commit_ptes modify_prot_commit_ptes +extern void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr, + pte_t *ptep, pte_t old_pte, pte_t pte, + unsigned int nr); + #ifdef CONFIG_ARM64_CONTPTE /* diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 3d5fb37424ab..abd9725796e9 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include @@ -1524,24 +1525,41 @@ static int __init prevent_bootmem_remove_init(void) early_initcall(prevent_bootmem_remove_init); #endif -pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) +pte_t modify_prot_start_ptes(struct vm_area_struct *vma, unsigned long addr, + pte_t *ptep, unsigned int nr) { + pte_t pte = get_and_clear_full_ptes(vma->vm_mm, addr, ptep, nr, /* full = */ 0); + if (alternative_has_cap_unlikely(ARM64_WORKAROUND_2645198)) { /* * Break-before-make (BBM) is required for all user space mappings * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(ptep_get(ptep))) - return ptep_clear_flush(vma, addr, ptep); + if (pte_accessible(vma->vm_mm, pte) && pte_user_exec(pte)) + __flush_tlb_range(vma, addr, nr * PAGE_SIZE, + PAGE_SIZE, true, 3); } - return ptep_get_and_clear(vma->vm_mm, addr, ptep); + + return pte; +} + +pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) +{ + return modify_prot_start_ptes(vma, addr, ptep, 1); +} + +void modify_prot_commit_ptes(struct vm_area_struct *vma, unsigned long addr, + pte_t *ptep, pte_t old_pte, pte_t pte, + unsigned int nr) +{ + set_ptes(vma->vm_mm, addr, ptep, pte, nr); } void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t old_pte, pte_t pte) { - set_pte_at(vma->vm_mm, addr, ptep, pte); + modify_prot_commit_ptes(vma, addr, ptep, old_pte, pte, 1); } /* -- 2.30.2