From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E80A93F44CD for ; Wed, 1 Jul 2026 23:28:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782948525; cv=none; b=IAt9W66YTxnp8LUXLuAPqIOwyKXogE4Q1IdsQ5C4O+OMGhp+V2b+9mR7O8xF71twcean6pV0hpmlgt8EsaTt74P+MTM5RcM/jwIlqRpHzcWcLXx3pTwb1aFnJg3WaOuRj4Nqi9AoxgEirVUnyz4WorjmPeGm9R97vT8RESufkEI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782948525; c=relaxed/simple; bh=zZidrhRSPce/kDc+r+WLJkhF8OiALTX9A0LpiPRqwQE=; h=Date:To:From:Subject:Message-Id; b=da/5rt1z6CagXfSdd2CETQY7HkwOkch7PRwCUPohujCVg551ktDEGwOE9GOEh6QDG76oOOvU0VqckoHsU7KhJSBEQNnQgrkIvDbi/AEZ124+HZBQS3BX33v2pjQ5uiM0zfylyihanYYuQNNAisbllt62GrtxpuS4lLW7ubE0cNc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=XriHgsZX; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="XriHgsZX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 730D41F000E9; Wed, 1 Jul 2026 23:28:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1782948523; bh=/8bx/7cgv6yJOQrPspasD5YIt+c9Kteeny39rUimB0g=; h=Date:To:From:Subject; b=XriHgsZXDooc3R9FGRF0ghUG+OR+xVgorhmQXlPcEWVYzv+Z9DJuK517KR/xOZmwS BXfzukyzvJcpNtsxHZmPNoq1v9B/nP2NuzYQ1tF0iGrcFeMkzYJN3rEE/+stgH6jCC zP1n8GHmB3XG8x42Fe2mnsyiNf41KKrNoHCQ6cqc= Date: Wed, 01 Jul 2026 16:28:43 -0700 To: mm-commits@vger.kernel.org,lizhe.67@bytedance.com,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-use-memcpy_nt-in-zone-device-template-copies.patch added to mm-new branch Message-Id: <20260701232843.730D41F000E9@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm: use memcpy_nt() in zone-device template copies has been added to the -mm mm-new branch. Its filename is mm-use-memcpy_nt-in-zone-device-template-copies.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-use-memcpy_nt-in-zone-device-template-copies.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: "Li Zhe" Subject: mm: use memcpy_nt() in zone-device template copies Date: Wed, 1 Jul 2026 17:05:53 +0800 The template fast path currently uses memcpy() for the actual struct page copy. Switch zone_device_page_init_from_template() to memcpy_nt() and add memcpy_nt_drain() before memmap_init_compound(), before prep_compound_head() updates overlapping tail metadata, and before returning from memmap_init_zone_device(). ZONE_DEVICE memmap initialization is largely write-once: each struct page is populated once, and most destination cachelines are not expected to be reused immediately afterwards. On x86, a regular cached memcpy() can therefore incur write-allocate traffic by pulling destination cachelines into the cache before writeback, and can populate the cache with data that has little near-term reuse. Using memcpy_nt() lets this path request non-temporal stores for that copy pattern, which can reduce cache pollution and avoid part of the associated write-allocate overhead, while architectures without a specialized backend still fall back to memcpy(). When memcpy_nt() maps to non-temporal stores, order those stores before memmap_init_compound(), before prep_compound_head() updates overlapping compound metadata, and before returning from memmap_init_zone_device(). Keep sanitized builds on the slow path so KASAN/KMSAN retain their instrumented stores. Tested in a VM with a 100 GB fsdax namespace device configured with map=dev and a 100 GB devdax namespace (align=2097152) on Intel Ice Lake server. Test procedure: Rebind the nd_pmem and dax_pmem driver 30 times and collect the memmap initialization time from the pr_debug() output of memmap_init_zone_device(). Base(v7.2-rc1): First binding for nd_pmem driver: 1456 ms Average of subsequent rebinds: 244.28 ms First binding for dax_pmem driver: 1462 ms Average of subsequent rebinds: 273.31 ms With this series: First binding for nd_pmem driver: 1272 ms Average of subsequent rebinds: 96.79 ms First binding for dax_pmem driver: 1354 ms Average of subsequent rebinds: 119.04 ms This reduces the average rebind time by about 60.4% for nd_pmem and 56.4% for dax_pmem. Link: https://lore.kernel.org/20260701090553.62691-9-lizhe.67@bytedance.com Signed-off-by: Li Zhe Cc: Alistair Popple Cc: Arnd Bergmann Cc: Balbir Singh Cc: "Borislav Petkov (AMD)" Cc: David Hildenbrand Cc: Ingo Molnar Cc: Kees Cook Cc: Mike Rapoport (Microsoft) Signed-off-by: Andrew Morton --- mm/mm_init.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) --- a/mm/mm_init.c~mm-use-memcpy_nt-in-zone-device-template-copies +++ a/mm/mm_init.c @@ -1059,10 +1059,20 @@ static void __ref zone_device_page_init_ static inline bool zone_device_page_init_optimization_enabled(void) { /* + * Keep sanitized builds on the slow path so their stores stay + * instrumented. + */ + if (IS_ENABLED(CONFIG_KASAN) || IS_ENABLED(CONFIG_KMSAN)) + return false; + + /* * The template fast path copies a preinitialized struct page image. * Skip it when the page_ref_set tracepoint is enabled. */ - return !page_ref_tracepoint_active(page_ref_set); + if (page_ref_tracepoint_active(page_ref_set)) + return false; + + return true; } static inline void zone_device_template_page_init(struct page *template, @@ -1107,7 +1117,7 @@ static void zone_device_page_init_from_t * to the destination page. */ zone_device_page_update_template(template, pfn); - memcpy(page, template, sizeof(*page)); + memcpy_nt(page, template, sizeof(*page)); } /* @@ -1178,6 +1188,15 @@ static void __ref memmap_init_compound(s zone_device_tail_page_init(page, pfn, zone_idx, nid, pgmap, head, order); } + + /* + * When the template path is enabled, order the preceding tail-page copies + * before prep_compound_head() updates the overlapping compound metadata + * in the first tail-page descriptors. If memcpy_nt() fell back to + * regular cached stores, memcpy_nt_drain() may be a no-op. + */ + if (use_template) + memcpy_nt_drain(); prep_compound_head(head, order); } @@ -1247,10 +1266,26 @@ void __ref memmap_init_zone_device(struc if (pfns_per_compound == 1) continue; + /* + * When the template path is enabled, order the preceding head-page copy + * before memmap_init_compound(), which immediately updates compound-head + * metadata. If memcpy_nt() fell back to regular cached stores, + * memcpy_nt_drain() may be a no-op. + */ + if (use_template) + memcpy_nt_drain(); + memmap_init_compound(page, pfn, zone_idx, nid, pgmap, compound_nr_pages(pfn, altmap, pgmap), use_template); } + /* + * Ensure any prior template copies are ordered before returning. + * On architectures where memcpy_nt() used regular cached stores, + * memcpy_nt_drain() may be a no-op. + */ + if (use_template) + memcpy_nt_drain(); pageblock_migratetype_init_range(start_pfn, nr_pages, MIGRATE_MOVABLE, false); _ Patches currently in -mm which might be from lizhe.67@bytedance.com are mm-fix-stale-zone_device-refcount-comment.patch mm-factor-zone-device-page-init-helpers-out-of-__init_zone_device_page.patch mm-add-a-set_page_section_from_pfn-helper.patch mm-add-a-template-based-fast-path-for-zone-device-page-init.patch mm-extend-the-template-fast-path-to-zone-device-compound-tails.patch string-introduce-memcpy_nt-helpers.patch x86-string-extend-memcpy_flushcache-fixed-size-fastpaths.patch mm-use-memcpy_nt-in-zone-device-template-copies.patch