From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A09AC30566F for ; Wed, 7 Oct 2026 00:51:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791334300; cv=none; b=mHlHdXsu5VWjBFCgiXUV4cndX1dkYpjcm63KzGz2TRygSNa5Z6To0rPt1NnD4HV6f30xI94xYHXI9EgTuW8SxN8Usl6b8TVrbpA0YVwHQ78p8usnfqLBLdnPhATa1qUPqpo6IYQUk1/5+rrhKg82QTrTuZsBYPTG5bB3NS9y+Fk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791334300; c=relaxed/simple; bh=yymAVUk/MP0f8c2jMsG6ikeN/KjriEHdLyRJRUAuWp0=; h=Date:To:From:Subject:Message-Id; b=seXeW6VW2RXyxNgt7A0CZdX9fAamFdEoRAkw/swNFk/Fo/2Td1IBa4CeuMLbCncw5MvzV2E+uSrmAEJAUSvrhtZN8wzOabqsx3XLrN1M9/5BylO6WGcRUTZNdrJygBE2vCqsj/gRU9k+eoGGxgh9pukCar0/H6TMt1lfbERvZBg= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=UYj+bXBY; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="UYj+bXBY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 74FCA1F0089C; Wed, 7 Oct 2026 00:51:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1791334299; bh=+p0swSBb4eC9OhSonNFOk5orogZG4V6VKGpupXo4M4E=; h=Date:To:From:Subject; b=UYj+bXBY8f9Ivfa9GcL1OhEiSA08aVEgLUkcuzS2kNVz229N85kRTRaMt9SS3zzL9 OzJDS66wtdhVYsbP8F+6tMD+dVmDUYFJPp6SNmW2Rb8/RUtzjer9TLWBjN2V+JNzQv 0Xh4Kl60CkKVMKRS3JfTfBH36y+7OIM4E8vmD8yw= Date: Tue, 06 Oct 2026 17:51:39 -0700 To: mm-commits@vger.kernel.org,rppt@kernel.org,muchun.song@linux.dev,mingo@redhat.com,kees@kernel.org,david@kernel.org,dave.hansen@linux.intel.com,bp@alien8.de,balbirs@nvidia.com,arnd@arndb.de,apopple@nvidia.com,lizhe.67@bytedance.com,akpm@linux-foundation.org From: Andrew Morton Subject: [merged mm-stable] x86-string-extend-memcpy_flushcache-fixed-size-fastpaths.patch removed from -mm tree Message-Id: <20261007005139.74FCA1F0089C@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The quilt patch titled Subject: x86/string: extend memcpy_flushcache() fixed-size fastpaths has been removed from the -mm tree. Its filename was x86-string-extend-memcpy_flushcache-fixed-size-fastpaths.patch This patch was dropped because it was merged into the mm-stable branch of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm ------------------------------------------------------ From: "Li Zhe" Subject: x86/string: extend memcpy_flushcache() fixed-size fastpaths Date: Mon, 31 Aug 2026 19:16:38 +0800 The x86 memcpy_nontemporal() helper maps to memcpy_flushcache(), and the ZONE_DEVICE template-copy path uses it to copy one struct page at a time. The relevant copy size is sizeof(struct page). On x86_64, the base struct page layout is 64 bytes. Adding either the KMSAN metadata pointers or an out-of-flags last_cpupid field can make it 80 bytes after alignment, and enabling both can make it 96 bytes. memcpy_flushcache() currently only has inline fixed-size cases for 4, 8, and 16 bytes. As a result, these constant-sized struct page copies fall through to __memcpy_flushcache() even though the compiler knows the copy size at the call site. Add fixed-size MOVNTI cases up to 96 bytes so the ZONE_DEVICE template-copy path can keep these struct page copies in the inline memcpy_flushcache() path. This matters for ZONE_DEVICE memmap initialization because the copy happens once per initialized struct page. For a 100 GB fsdax namespace with map=dev, this is about 25 million struct page copies during nd_pmem binding or rebinding. Tested in a VM with a 100 GB fsdax namespace device configured with map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake server. Test procedure: Rebind the nd_pmem and dax_pmem drivers 30 times and collect the memmap initialization time from the pr_debug() output of memmap_init_zone_device(). With memcpy_nontemporal() used by the ZONE_DEVICE template-copy path: Average of rebinds for nd_pmem driver: 150.40 ms Average of rebinds for dax_pmem driver: 161.83 ms With this x86 fixed-size fastpath patch applied: Average of rebinds for nd_pmem driver: 71.93 ms Average of rebinds for dax_pmem driver: 87.37 ms This further reduces the average memmap initialization time measured during rebind by about 52.2% for nd_pmem and 46.0% for dax_pmem. Link: https://lore.kernel.org/20260831111638.76012-8-lizhe.67@bytedance.com Signed-off-by: Li Zhe Signed-off-by: Andrew Morton Suggested-by: Borislav Petkov Acked-by: Borislav Petkov (AMD) Acked-by: Dave Hansen Cc: Alistair Popple Cc: Arnd Bergmann Cc: Balbir Singh Cc: David Hildenbrand (Arm) Cc: Ingo Molnar Cc: Kees Cook Cc: Mike Rapoport (Microsoft) Cc: Muchun Song --- arch/x86/include/asm/string_64.h | 71 ++++++++++++++++++++++------- 1 file changed, 56 insertions(+), 15 deletions(-) --- a/arch/x86/include/asm/string_64.h~x86-string-extend-memcpy_flushcache-fixed-size-fastpaths +++ a/arch/x86/include/asm/string_64.h @@ -82,23 +82,64 @@ int strcmp(const char *cs, const char *c #ifdef CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE #define __HAVE_ARCH_MEMCPY_FLUSHCACHE 1 void __memcpy_flushcache(void *dst, const void *src, size_t cnt); -static __always_inline void memcpy_flushcache(void *dst, const void *src, size_t cnt) + +static __always_inline void movnti_4(void *dst, const void *src) { - if (__builtin_constant_p(cnt)) { - switch (cnt) { - case 4: - asm ("movntil %1, %0" : "=m"(*(u32 *)dst) : "r"(*(u32 *)src)); - return; - case 8: - asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src)); - return; - case 16: - asm ("movntiq %1, %0" : "=m"(*(u64 *)dst) : "r"(*(u64 *)src)); - asm ("movntiq %1, %0" : "=m"(*(u64 *)(dst + 8)) : "r"(*(u64 *)(src + 8))); - return; - } + asm volatile("movntil %1, %0" + : "=m"(*(u32 *)dst) + : "r"(*(const u32 *)src) + : "memory"); +} + +static __always_inline void movnti_8(void *dst, const void *src) +{ + asm volatile("movntiq %1, %0" + : "=m"(*(u64 *)dst) + : "r"(*(const u64 *)src) + : "memory"); +} + +static __always_inline void movnti_16(void *dst, const void *src) +{ + movnti_8(dst, src); + movnti_8(dst + 8, src + 8); +} + +static __always_inline void movnti_32(void *dst, const void *src) +{ + movnti_16(dst, src); + movnti_16(dst + 16, src + 16); +} + +static __always_inline void movnti_64(void *dst, const void *src) +{ + movnti_32(dst, src); + movnti_32(dst + 32, src + 32); +} + +static __always_inline void memcpy_flushcache(void *dst, const void *src, + size_t cnt) +{ + if (!__builtin_constant_p(cnt)) + return __memcpy_flushcache(dst, src, cnt); + + /* + * The relevant fixed-size copies here are the x86_64 struct page sizes: + * 64, 80, and 96 bytes. Keep 32-byte and 48-byte copies inline as well + * instead of sending those nearby fixed-size cases back to + * __memcpy_flushcache(). + */ + switch (cnt) { + case 4: movnti_4(dst, src); break; + case 8: movnti_8(dst, src); break; + case 16: movnti_16(dst, src); break; + case 32: movnti_32(dst, src); break; + case 48: movnti_32(dst, src); movnti_16(dst + 32, src + 32); break; + case 64: movnti_64(dst, src); break; + case 80: movnti_64(dst, src); movnti_16(dst + 64, src + 64); break; + case 96: movnti_64(dst, src); movnti_32(dst + 64, src + 64); break; + default: __memcpy_flushcache(dst, src, cnt); break; } - __memcpy_flushcache(dst, src, cnt); } #define memcpy_nontemporal memcpy_nontemporal _ Patches currently in -mm which might be from lizhe.67@bytedance.com are mm-hugetlb-fix-overbroad-mmu-notifiers-for-unshared-pmds.patch