Re: [RFC PATCH 1/1] mm: batch page copies in folio_copy() and folio_mc_copy()

Re: [RFC PATCH 1/1] mm: batch page copies in folio_copy() and folio_mc_copy()

[David Hildenbrand (Arm)]

On 4/27/26 16:20, Shivank Garg wrote:
> Rewrite folio_copy() and folio_mc_copy() as thin wrappers around new
> batched helpers copy_highpages() and copy_mc_highpages().
> 
> The current implementations iterate copy_highpage() (or its #MC-aware
> variant) per 4 KB page. For a single 2 MB folio that loop runs 512
> times and pays, per page:
> 
>   - kmap_local_page() / kunmap_local()
>   - cond_resched()
>   - one invocation of the architecture copy_page()/memcpy() primitive
> 
> The new helpers issue a single copy_mc_to_kernel()/memcpy() over
> the whole contiguous range when CONFIG_HIGHMEM is off and no
> architecture overrides (__HAVE_ARCH_COPY_HIGHPAGE) copy_highpage().
> HIGHMEM and arch overrides keep the existing per-page path.
> 
> Tested on dual-socket AMD EPYC 9655 (Zen 5) with a CXL.mem node.
> In-kernel folio_mc_copy() microbenchmark on 2 MB folios, source
> evicted from cache before each iteration and measured throughput:
> 
>   direction         baseline GB/s   optimized GB/s   speedup
>   DRAM0 -> DRAM1     18.65 ± 1.37    38.03 ± 3.21     2.04x
>   DRAM0 -> CXL       25.46 ± 2.89    39.29 ± 1.17     1.54x
>   CXL   -> DRAM0     20.61 ± 3.95    35.07 ± 0.62     1.70x
> 
> End-to-end move_pages(2) throughput on anonymous 2 MB mTHP folios,
> 1 GB migrated per run:
> 
>   direction         baseline GB/s   optimized GB/s   speedup
>   DRAM0 -> DRAM1      7.20 ± 0.03     8.01 ± 0.02     1.11x
>   DRAM0 -> CXL       11.12 ± 0.15    13.07 ± 0.03     1.18x
>   DRAM1 -> DRAM0      7.21 ± 0.02     7.95 ± 0.02     1.10x
>   CXL   -> DRAM0      9.10 ± 0.05     9.49 ± 0.01     1.04x
> 
> On AMD EPYC 7713 (Zen 3 / Milan, REP_GOOD without FSRM/ERMS) the
> folio_copy() bulk path regresses because memcpy() falls through to
> memcpy_orig (an unrolled movq loop), which is slower than the
> per-page copy_page() (microcoded rep movsq) it replaces. 

Do you know what the reason for that fallback is? Could it be fixed (e.g., when
we detect page alignment or sth like that?)

-- 
Cheers,

David