[PATCH] mm/vmalloc: avoid false sharing with drain_vmap_work

[PATCH] mm/vmalloc: avoid false sharing with drain_vmap_work

[JonasZhou-oc]

free_vmap_area_noflush() queues drain_vmap_work after the number of
lazily freed pages exceeds lazy_max_pages(). Until the worker purges
those pages, concurrent frees keep calling schedule_work(). Even if
the work is already pending, queue_work_on() performs a locked
test_and_set_bit() on the pending bit in the work item.

On the tested x86-64 build, drain_vmap_work and vmap_nodes occupy the
same 64-byte cache line. The work item starts at offset 0 and the
vmap_nodes pointer at offset 32. The latter is read by vmap allocation
and free paths, so updates to the work item invalidate a cache line
read by all CPUs.

Put drain_vmap_work in the cacheline-aligned data section. Tests were
run on Linux 7.2.

On a two-socket Intel Xeon Silver 4208 system using 16 workers, the
runtimes of vmalloc.fix_align, vmalloc.fix_size, and
vmalloc.no_block_alloc decreased by 12.61%, 5.78%, and 6.87%,
respectively. HITM samples for the affected cache line and total HITM
samples decreased by 96.55% and 13.36%, respectively.

Signed-off-by: JonasZhou <jonaszhou-oc@zhaoxin.com>
---
 mm/vmalloc.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index f4fa227a8d7f..6b4b287e91f6 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1088,7 +1088,12 @@ RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_augment_cb,
 static void reclaim_and_purge_vmap_areas(void);
 static BLOCKING_NOTIFIER_HEAD(vmap_notify_list);
 static void drain_vmap_area_work(struct work_struct *work);
-static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
+/*
+ * Keep the work item, whose pending bit is updated by freeing CPUs,
+ * away from vmap metadata read by allocation and free paths.
+ */
+static __cacheline_aligned_in_smp
+DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
 
 static __cacheline_aligned_in_smp atomic_long_t vmap_lazy_nr;
 
-- 
2.43.0

Re: [PATCH] mm/vmalloc: avoid false sharing with drain_vmap_work

[Uladzislau Rezki]

On Tue, Aug 25, 2026 at 06:46:59PM +0800, JonasZhou-oc wrote:
> free_vmap_area_noflush() queues drain_vmap_work after the number of
> lazily freed pages exceeds lazy_max_pages(). Until the worker purges
> those pages, concurrent frees keep calling schedule_work(). Even if
> the work is already pending, queue_work_on() performs a locked
> test_and_set_bit() on the pending bit in the work item.
> 
> On the tested x86-64 build, drain_vmap_work and vmap_nodes occupy the
> same 64-byte cache line. The work item starts at offset 0 and the
> vmap_nodes pointer at offset 32. The latter is read by vmap allocation
> and free paths, so updates to the work item invalidate a cache line
> read by all CPUs.
> 
> Put drain_vmap_work in the cacheline-aligned data section. Tests were
> run on Linux 7.2.
> 
> On a two-socket Intel Xeon Silver 4208 system using 16 workers, the
> runtimes of vmalloc.fix_align, vmalloc.fix_size, and
> vmalloc.no_block_alloc decreased by 12.61%, 5.78%, and 6.87%,
> respectively. HITM samples for the affected cache line and total HITM
> samples decreased by 96.55% and 13.36%, respectively.
> 
> Signed-off-by: JonasZhou <jonaszhou-oc@zhaoxin.com>
> ---
>  mm/vmalloc.c | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index f4fa227a8d7f..6b4b287e91f6 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -1088,7 +1088,12 @@ RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_augment_cb,
>  static void reclaim_and_purge_vmap_areas(void);
>  static BLOCKING_NOTIFIER_HEAD(vmap_notify_list);
>  static void drain_vmap_area_work(struct work_struct *work);
> -static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
> +/*
> + * Keep the work item, whose pending bit is updated by freeing CPUs,
> + * away from vmap metadata read by allocation and free paths.
> + */
> +static __cacheline_aligned_in_smp
> +DECLARE_WORK(drain_vmap_work, drain_vmap_area_work);
>  
>  static __cacheline_aligned_in_smp atomic_long_t vmap_lazy_nr;
>  
> -- 
> 2.43.0
> 
> 
Makes sens to me.

Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>

--
Uladzislau Rezki