Re: [PATCH v6 6/8] string: introduce memcpy_nt() helpers

[Muchun Song <muchun.song@linux.dev>]

> On Jul 15, 2026, at 14:18, Li Zhe <lizhe.67@bytedance.com> wrote:
> 
> On 7/14/26 5:24 PM, Muchun Song wrote:
>> 
>> 
>> On 2026/7/9 19:25, Li Zhe wrote:
>>> Introduce memcpy_nt() and memcpy_nt_drain() for write-once copy sites
>>> that want a named non-temporal copy primitive plus an explicit drain
>>> step.
>>> 
>>> On x86_64, override both helpers in arch/x86/include/asm/string_64.h
>>> using the usual self-macro pattern, next to the existing
>>> memcpy_flushcache() backend that memcpy_nt() wraps. The x86_64
>>> implementation maps memcpy_nt() to memcpy_flushcache() and uses wmb()
>>> for memcpy_nt_drain(), because that backend issues MOVNTI stores and
>>> callers need an ordering point before later normal stores that depend
>>> on those writes becoming visible.
>>> 
>>> include/linux/string.h provides the generic fallback under
>>> memcpy_nt() as plain memcpy() and leaves memcpy_nt_drain() empty, so
>>> architectures that do not override memcpy_nt() do not pay an
>>> unconditional barrier. Architectures that later grow a specialized
>>> memcpy_nt() backend can override memcpy_nt_drain() with whatever
>>> drain primitive their memory-ordering rules require.
>>> 
>>> The immediate user is the ZONE_DEVICE template-copy path. It populates
>>> struct page descriptors in a write-once pattern, so a regular cached
>>> memcpy() can incur avoidable write-allocate traffic and cache
>>> pollution for data with little near-term reuse.
>>> 
>>> Signed-off-by: Li Zhe <lizhe.67@bytedance.com>
>>> ---
>>>   arch/x86/include/asm/string_64.h | 22 ++++++++++++++++++++++
>>>   include/linux/string.h           | 23 +++++++++++++++++++++++
>>>   2 files changed, 45 insertions(+)
>>> 
>>> diff --git a/arch/x86/include/asm/string_64.h 
>>> b/arch/x86/include/asm/string_64.h
>>> index 4635616863f5..6cb9e0ac7fa0 100644
>>> --- a/arch/x86/include/asm/string_64.h
>>> +++ b/arch/x86/include/asm/string_64.h
>>> @@ -4,6 +4,7 @@
>>> 
>>>   #ifdef __KERNEL__
>>>   #include <linux/jump_label.h>
>>> +#include <asm/barrier.h>
>>> 
>>>   /* Written 2002 by Andi Kleen */
>>> 
>>> @@ -100,6 +101,27 @@ static __always_inline void 
>>> memcpy_flushcache(void *dst, const void *src, size_t
>>>       }
>>>       __memcpy_flushcache(dst, src, cnt);
>>>   }
>>> +
>>> +#define memcpy_nt memcpy_nt
>>> +/*
>>> + * Reuse the existing x86 flushcache backend as the nt copy primitive.
>>> + * Callers pair it with memcpy_nt_drain() when later stores must be
>>> + * ordered after the copy.
>>> + */
>>> +static __always_inline void memcpy_nt(void *dst, const void *src, 
>>> size_t cnt)
>>> +{
>>> +    memcpy_flushcache(dst, src, cnt);
>> 
>> Why not use memcpy_flushcache() directly in device dax path? I don't
>> understand the necessity of introducing memcpy_nt here.
>> 
> 
> The reason for introducing memcpy_nt() is to give generic MM code a
> named non-temporal copy primitive, instead of hardwiring the x86
> memcpy_flushcache() backend into a generic caller.
> 
> On x86, memcpy_nt() maps to memcpy_flushcache() today. On other
> architectures, memcpy_flushcache() may have different semantics, and we
> also do not know whether its implementation would provide the same
> optimization opportunity as on x86. Using memcpy_nt() lets the generic
> caller express the intent while leaving the backend choice to each
> architecture.

Got it. But the 'nt' suffix is overly abbreviated and not direct enough.
Therefore, I suggest avoiding uncommon abbreviations and instead keeping
things as explicit as the 'flushcache' suffix in memcpy_flushcache.

> 
>>> +}
>>> +
>>> +#define memcpy_nt_drain memcpy_nt_drain
>>> +static __always_inline void memcpy_nt_drain(void)
>>> +{
>>> +    /*
>>> +     * Order the prior MOVNTI stores issued by memcpy_flushcache()
>>> +     * before later normal stores.
>>> +     */
>> 
>> I also have a question here: why are we using wmb to guarantee visibility
>> at this stage?
>> 
>> Since we are still in the very early phases of memory initialization 
>> (specifically,
>> struct page initialization), since we are still in an intermediate 
>> initialization
>> state, this shouldn't be visible to other CPUs anyway.
>> 
>> Thanks.
> 
> The drain is not about exposing the intermediate initialization state to
> other CPUs.
> 
> It is there to order the earlier non-temporal stores before the later
> normal stores on the same control path, for example before
> memmap_init_compound() / prep_compound_head() update overlapping
> compound metadata.

For a single core, out-of-order execution is invisible to developers.
Therefore, I'm curious about your reasons for preserving the order? What
would be the consequences of not maintaining it?

Thanks.

> 
> On x86, memcpy_nt() maps to MOVNTI-based memcpy_flushcache(), so
> memcpy_nt_drain() uses wmb(), which maps to the required sfence there.
> 
> Thanks,
> Zhe
> 
>> 
>>> +    wmb();
>>> +}
>>>   #endif
>>> 
>>>   #endif /* __KERNEL__ */
>>> diff --git a/include/linux/string.h b/include/linux/string.h
>>> index 5702daca4326..a109b2f86ca6 100644
>>> --- a/include/linux/string.h
>>> +++ b/include/linux/string.h
>>> @@ -278,6 +278,29 @@ static inline void memcpy_flushcache(void *dst, 
>>> const void *src, size_t cnt)
>>>   }
>>>   #endif
>>> 
>>> +#ifndef memcpy_nt
>>> +/*
>>> + * memcpy_nt() requests a non-temporal copy when the architecture has a
>>> + * suitable backend. Architectures that do not override it fall back to
>>> + * memcpy().
>>> + */
>>> +static inline void memcpy_nt(void *dst, const void *src, size_t cnt)
>>> +{
>>> +    memcpy(dst, src, cnt);
>>> +}
>>> +#endif
>>> +
>>> +#ifndef memcpy_nt_drain
>>> +/*
>>> + * Callers use memcpy_nt_drain() before later normal stores that 
>>> need to
>>> + * be ordered after memcpy_nt(). Architectures without a specialized
>>> + * backend can leave it empty.
>>> + */
>>> +static inline void memcpy_nt_drain(void)
>>> +{
>>> +}
>>> +#endif
>>> +
>>>   void *memchr_inv(const void *s, int c, size_t n);
>>>   char *strreplace(char *str, char old, char new);
>>> 
>>> -- 
>>> 2.20.1