From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 061F7C369DC for ; Wed, 7 May 2025 06:03:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0846F6B000A; Wed, 7 May 2025 02:03:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 036066B0083; Wed, 7 May 2025 02:03:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3F9B6B0085; Wed, 7 May 2025 02:03:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C41826B000A for ; Wed, 7 May 2025 02:03:13 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id EE3C6161295 for ; Wed, 7 May 2025 06:03:14 +0000 (UTC) X-FDA: 83415069108.18.60DB8EA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 5D60B80011 for ; Wed, 7 May 2025 06:03:13 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746597793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=BcVnAnq6DWM1Nncsc0Lvcc7TbQQOgnOKHTqRiPNcxHs=; b=040IsR2q7zR5EImfQyzjQ3taJV9j+hWYgnCwrbK7y6gzt29j6+yrmInOBfs+7fhx3+nHdm m4XympCggon/ez/2Nz+NSyp7EoEhE7V9xNb8KBC1/6IqL+LUvLpOQpferZnvUJUZ3Dw5hP 81sRKbD8lonI3iBA3bZfzi2VJiOOoF0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746597793; a=rsa-sha256; cv=none; b=IaKocFv7Z91CrYXYCmcmXE+DfEf6mJX6F6UsxjuMBOYIue8tTEcTEkDOK3cze0H/1cDIW2 ZnrzX9yTilN+mG5WwvqVE0SbeSAlyb8N3ozllPvEy1t5BAEn6vXCrDJJc9ugIhg+baej9t POJFVFcWxJihG+brkUwqGDqrZ+iED1Y= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 3825F2F; Tue, 6 May 2025 23:03:02 -0700 (PDT) Received: from K4MQJ0H1H2.emea.arm.com (K4MQJ0H1H2.blr.arm.com [10.162.43.22]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 9893D3F5A1; Tue, 6 May 2025 23:03:04 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org Cc: Liam.Howlett@oracle.com, lorenzo.stoakes@oracle.com, vbabka@suse.cz, jannh@google.com, pfalcato@suse.de, linux-mm@kvack.org, linux-kernel@vger.kernel.org, david@redhat.com, peterx@redhat.com, ryan.roberts@arm.com, mingo@kernel.org, libang.li@antgroup.com, maobibo@loongson.cn, zhengqi.arch@bytedance.com, baohua@kernel.org, anshuman.khandual@arm.com, willy@infradead.org, ioworker0@gmail.com, yang@os.amperecomputing.com, baolin.wang@linux.alibaba.com, ziy@nvidia.com, hughd@google.com, Dev Jain Subject: [PATCH v2 0/2] Optimize mremap() for large folios Date: Wed, 7 May 2025 11:32:54 +0530 Message-Id: <20250507060256.78278-1-dev.jain@arm.com> X-Mailer: git-send-email 2.39.3 (Apple Git-146) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: msmaodqutuoqtddkhaqonx7wbbzcipx1 X-Rspamd-Queue-Id: 5D60B80011 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1746597793-67491 X-HE-Meta: U2FsdGVkX1+A3HpZEPjOOTAdYYmdaKOVdNxlLF2LwMUB0KwdYZNC1xAaJQN1xrNi6U5NKkDpEWgwhbBOASBgDlPtTR1iVu7ve1birkEi5JpnvlO4X/cGkZ7rXlsQ1SIS5iNVSmWpXtdKs/XX+9tt43MAOmizSICxuok02GPOUjMdEFQVdHP20aH9Viiol/bT9Zo0BBrejF84HHj+oKyDzGWZnsWXxyHwlNj+NtnWQapIzznGy4yNg2f7NNIUyuaXw/oCrB3UHFXLJnJGrWE3NTfw4l0/DDm05QV4mY49iwjYueMe8xMz+CS3FzHUBeJYwjQlpD59hZAzlrFRVDFKWsiV0rqoxa7MqycPPmhDylfV6m+bS7OzhL9N0tXNJlhsNximrtr2vXDBJNOhc6GmYk5C9uKB7Fm2/SiN+lRb2zSzgUuvDYoYRZ330ijvv7jrFOSn+4CyTE5EK2ErltdATXmk/S28kbMOUJ3J5/dyWrOcKay8UJMfsmlg612dB0CoFzzT5qa8yUv3lScdjMhGJ2zLzC2Zt0wgLeqzQ7ebw+m5voLI2Q0V98FAB78qWd1KIZxxXWgQYLAKqyHSDADBxa2CGqVmDpEoJ81jd0vZVG6HYcuGX0NjCm6M/A+tqRan1WU8ne2fO2Bn6CL7AGxvQSAubjWJRcbZ9HpEocUnklN82frJbDcMDlTnvFrx5TLFD9pyoFLbfYEEZk1cLlD1fKLAUgbZ6qFFLTBhtE7Z5rgRdhZOwzwyLKTn9Pt/pCjkoENg3UF+1rcsSDdvZeWvEWQoLAT/eELPOqpx6Bhe55cEcaRIUVFad4Ifz8IITlQOyR/Jwl6pYFb5ZAQRaEpH16o2Z0ZMZBG+qloWmKdzlry4h7I7ixWcNVrChanPJ+UQzLTVxhne0lzZbDd2zr++iFlc2K3nxiq9OzapYgNycOo9WHJsDsBPobpDZdwPHszGOzsDGxAC6XnH3wG02/d s2cM0VNQ BRIw6+gWOGy+2tIHmxhaD7m5LtYJJgEqq8/IyrwZ/ItsyQq2S+NC4U6zwyDU2BUQuUQVkrLYluvjcqR40IQ16mob5Qrg7A1K9Clul X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently move_ptes() iterates through ptes one by one. If the underlying folio mapped by the ptes is large, we can process those ptes in a batch using folio_pte_batch(), thus clearing and setting the PTEs in one go. For arm64 specifically, this results in a 16x reduction in the number of ptep_get() calls (since on a contig block, ptep_get() on arm64 will iterate through all 16 entries to collect a/d bits), and we also elide extra TLBIs through get_and_clear_full_ptes, replacing ptep_get_and_clear. Mapping 512K of memory, memsetting it, remapping it to src + 512K, and munmapping it 10,000 times, the average execution time reduces from 1.9 to 1.2 seconds, giving a 37% performance optimization, on Apple M3 (arm64). Test program for reference: #define _GNU_SOURCE #include #include #include #include #include #include #define SIZE (1UL << 20) // 512 KB int main(void) { void *new_addr, *addr; for (int i = 0; i < 10000; ++i) { addr = mmap((void *)(1UL << 30), SIZE, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); if (addr == MAP_FAILED) { perror("mmap"); return 1; } memset(addr, 0xAA, SIZE); new_addr = mremap(addr, SIZE, SIZE, MREMAP_MAYMOVE | MREMAP_FIXED, addr + SIZE); if (new_addr != (addr + SIZE)) { perror("mremap"); return 1; } munmap(new_addr, SIZE); } } v1->v2: - Expand patch descriptions, move pte declarations to a new line, reduce indentation in patch 2 by introducing mremap_folio_pte_batch(), fix loop iteration (Lorenzo) - Merge patch 2 and 3 (Anshuman, Lorenzo) - Fix maybe_contiguous_pte_pfns (Willy) Dev Jain (2): mm: Call pointers to ptes as ptep mm: Optimize mremap() by PTE batching include/linux/pgtable.h | 29 ++++++++++++++++++++++ mm/mremap.c | 54 +++++++++++++++++++++++++++++------------ 2 files changed, 68 insertions(+), 15 deletions(-) -- 2.30.2