[PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Ye Liu]

From: Ye Liu <liuye@kylinos.cn>

The vmap_purge_lock mutex can be held for an extended period by
__purge_vmap_area_lazy() which calls flush_work() to wait for
purge_vmap_node workers while holding the lock.  Under memory
pressure, those workers may themselves be blocked in direct
reclaim trying to acquire the same lock via the
vmap_node_shrink_scan() shrinker callback, creating a circular
dependency that deadlocks the entire system.

Two places acquire vmap_purge_lock from paths that can be reached
during direct reclaim:

1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
   mutex_trylock().  This is a shrinker that only decays the vmap
   pool and returns SHRINK_STOP without freeing memory; skipping a
   decay cycle when the lock is contended is harmless and prevents
   tasks from piling up on the mutex in the direct reclaim path.

2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
   mutex_trylock().  This is called from the vmalloc allocation
   overflow path; if trylock fails, another thread is already
   purging and the allocator's retry will find freed space.  The
   notifier chain provides a fallback if the retry still fails.

Both trylock failures break the circular dependency: the lock
holder's flush_work() can complete because workers are no longer
blocked on vmap_purge_lock in the direct reclaim path.

Fixes: 7679ba6b36db ("mm: vmalloc: add a shrinker to drain vmap pools")
Suggested-by: Uladzislau Rezki <urezki@gmail.com>
Suggested-by: Dev Jain <dev.jain@arm.com>
Signed-off-by: Ye Liu <liuye@kylinos.cn>
---
v2:
 - Use mutex_trylock instead of mutex_lock to acquire vmap_purge_lock,
   as suggested by Uladzislau Rezki and Dev Jain.
 - Link: https://lore.kernel.org/all/20260824095020.1225189-1-ye.liu@linux.dev/
 mm/vmalloc.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index bea9f76ed7e7..e5c68b795a3e 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2440,7 +2440,8 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
 static void reclaim_and_purge_vmap_areas(void)
 
 {
-	mutex_lock(&vmap_purge_lock);
+	if (!mutex_trylock(&vmap_purge_lock))
+		return;
 	purge_fragmented_blocks_allcpus();
 	__purge_vmap_area_lazy(ULONG_MAX, 0, true);
 	mutex_unlock(&vmap_purge_lock);
@@ -5519,10 +5520,20 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
 	struct vmap_node *vn;
 
-	guard(mutex)(&vmap_purge_lock);
+	/*
+	 * This shrinker is invoked from direct reclaim where memory
+	 * pressure is already high.  Blocking on vmap_purge_lock here
+	 * can deadlock the system: the lock holder may be blocked in
+	 * flush_work() waiting for a worker that is stuck in this same
+	 * reclaim path trying to acquire the same lock.  Use trylock
+	 * to avoid this; skipping a pool decay cycle is harmless.
+	 */
+	if (!mutex_trylock(&vmap_purge_lock))
+		return SHRINK_STOP;
 	for_each_vmap_node(vn)
 		decay_va_pool_node(vn, true);
 
+	mutex_unlock(&vmap_purge_lock);
 	return SHRINK_STOP;
 }
 
-- 
2.25.1

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Fri, Aug 28, 2026 at 05:17:53PM +0800, Ye Liu wrote:
> From: Ye Liu <liuye@kylinos.cn>
> 
> The vmap_purge_lock mutex can be held for an extended period by
> __purge_vmap_area_lazy() which calls flush_work() to wait for
> purge_vmap_node workers while holding the lock.  Under memory
> pressure, those workers may themselves be blocked in direct
> reclaim trying to acquire the same lock via the
> vmap_node_shrink_scan() shrinker callback, creating a circular
> dependency that deadlocks the entire system.
> 
> Two places acquire vmap_purge_lock from paths that can be reached
> during direct reclaim:
> 
> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>    mutex_trylock().  This is a shrinker that only decays the vmap
>    pool and returns SHRINK_STOP without freeing memory; skipping a
>    decay cycle when the lock is contended is harmless and prevents
>    tasks from piling up on the mutex in the direct reclaim path.
> 
> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>    mutex_trylock().  This is called from the vmalloc allocation
>    overflow path; if trylock fails, another thread is already
>    purging and the allocator's retry will find freed space.  The
>    notifier chain provides a fallback if the retry still fails.
> 
> Both trylock failures break the circular dependency: the lock
> holder's flush_work() can complete because workers are no longer
> blocked on vmap_purge_lock in the direct reclaim path.
> 
> Fixes: 7679ba6b36db ("mm: vmalloc: add a shrinker to drain vmap pools")
> Suggested-by: Uladzislau Rezki <urezki@gmail.com>
> Suggested-by: Dev Jain <dev.jain@arm.com>
> Signed-off-by: Ye Liu <liuye@kylinos.cn>
> ---
> v2:
>  - Use mutex_trylock instead of mutex_lock to acquire vmap_purge_lock,
>    as suggested by Uladzislau Rezki and Dev Jain.
>  - Link: https://lore.kernel.org/all/20260824095020.1225189-1-ye.liu@linux.dev/
>  mm/vmalloc.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index bea9f76ed7e7..e5c68b795a3e 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -2440,7 +2440,8 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
>  static void reclaim_and_purge_vmap_areas(void)
>  
>  {
> -	mutex_lock(&vmap_purge_lock);
> +	if (!mutex_trylock(&vmap_purge_lock))
> +		return;
>  	purge_fragmented_blocks_allcpus();
>  	__purge_vmap_area_lazy(ULONG_MAX, 0, true);
>  	mutex_unlock(&vmap_purge_lock);
> @@ -5519,10 +5520,20 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
>  {
>  	struct vmap_node *vn;
>  
> -	guard(mutex)(&vmap_purge_lock);
> +	/*
> +	 * This shrinker is invoked from direct reclaim where memory
> +	 * pressure is already high.  Blocking on vmap_purge_lock here
> +	 * can deadlock the system: the lock holder may be blocked in
> +	 * flush_work() waiting for a worker that is stuck in this same
> +	 * reclaim path trying to acquire the same lock.  Use trylock
> +	 * to avoid this; skipping a pool decay cycle is harmless.
> +	 */
> +	if (!mutex_trylock(&vmap_purge_lock))
> +		return SHRINK_STOP;
>  	for_each_vmap_node(vn)
>  		decay_va_pool_node(vn, true);
>  
> +	mutex_unlock(&vmap_purge_lock);
>  	return SHRINK_STOP;
>  }
>  
> -- 
> 2.25.1
> 
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Andrew Morton]

On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:

> From: Ye Liu <liuye@kylinos.cn>
> 
> The vmap_purge_lock mutex can be held for an extended period by
> __purge_vmap_area_lazy() which calls flush_work() to wait for
> purge_vmap_node workers while holding the lock.  Under memory
> pressure, those workers may themselves be blocked in direct
> reclaim trying to acquire the same lock via the
> vmap_node_shrink_scan() shrinker callback, creating a circular
> dependency that deadlocks the entire system.
> 
> Two places acquire vmap_purge_lock from paths that can be reached
> during direct reclaim:
> 
> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>    mutex_trylock().  This is a shrinker that only decays the vmap
>    pool and returns SHRINK_STOP without freeing memory; skipping a
>    decay cycle when the lock is contended is harmless and prevents
>    tasks from piling up on the mutex in the direct reclaim path.
> 
> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>    mutex_trylock().  This is called from the vmalloc allocation
>    overflow path; if trylock fails, another thread is already
>    purging and the allocator's retry will find freed space.  The
>    notifier chain provides a fallback if the retry still fails.
> 
> Both trylock failures break the circular dependency: the lock
> holder's flush_work() can complete because workers are no longer
> blocked on vmap_purge_lock in the direct reclaim path.

Thanks.  AI review expressed a couple of concerns:
	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Dev Jain]

On 28/08/26 11:35 pm, Andrew Morton wrote:
> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> 
>> From: Ye Liu <liuye@kylinos.cn>
>>
>> The vmap_purge_lock mutex can be held for an extended period by
>> __purge_vmap_area_lazy() which calls flush_work() to wait for
>> purge_vmap_node workers while holding the lock.  Under memory
>> pressure, those workers may themselves be blocked in direct
>> reclaim trying to acquire the same lock via the
>> vmap_node_shrink_scan() shrinker callback, creating a circular
>> dependency that deadlocks the entire system.
>>
>> Two places acquire vmap_purge_lock from paths that can be reached
>> during direct reclaim:
>>
>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>>    mutex_trylock().  This is a shrinker that only decays the vmap
>>    pool and returns SHRINK_STOP without freeing memory; skipping a
>>    decay cycle when the lock is contended is harmless and prevents
>>    tasks from piling up on the mutex in the direct reclaim path.
>>
>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>>    mutex_trylock().  This is called from the vmalloc allocation
>>    overflow path; if trylock fails, another thread is already
>>    purging and the allocator's retry will find freed space.  The
>>    notifier chain provides a fallback if the retry still fails.
>>
>> Both trylock failures break the circular dependency: the lock
>> holder's flush_work() can complete because workers are no longer
>> blocked on vmap_purge_lock in the direct reclaim path.
> 
> Thanks.  AI review expressed a couple of concerns:
> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev


Sounds legit to me. Now there is no guarantee of purge being successful, and we
can get a spurious failure.

How about using a WQ_RECLAIM workqueue:

vmap_purge_wq = alloc_workqueue("vmap_purge",
				WQ_MEM_RECLAIM | WQ_PERCPU, 0);

I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
> 
> 
> On 28/08/26 11:35 pm, Andrew Morton wrote:
> > On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> > 
> >> From: Ye Liu <liuye@kylinos.cn>
> >>
> >> The vmap_purge_lock mutex can be held for an extended period by
> >> __purge_vmap_area_lazy() which calls flush_work() to wait for
> >> purge_vmap_node workers while holding the lock.  Under memory
> >> pressure, those workers may themselves be blocked in direct
> >> reclaim trying to acquire the same lock via the
> >> vmap_node_shrink_scan() shrinker callback, creating a circular
> >> dependency that deadlocks the entire system.
> >>
> >> Two places acquire vmap_purge_lock from paths that can be reached
> >> during direct reclaim:
> >>
> >> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
> >>    mutex_trylock().  This is a shrinker that only decays the vmap
> >>    pool and returns SHRINK_STOP without freeing memory; skipping a
> >>    decay cycle when the lock is contended is harmless and prevents
> >>    tasks from piling up on the mutex in the direct reclaim path.
> >>
> >> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
> >>    mutex_trylock().  This is called from the vmalloc allocation
> >>    overflow path; if trylock fails, another thread is already
> >>    purging and the allocator's retry will find freed space.  The
> >>    notifier chain provides a fallback if the retry still fails.
> >>
> >> Both trylock failures break the circular dependency: the lock
> >> holder's flush_work() can complete because workers are no longer
> >> blocked on vmap_purge_lock in the direct reclaim path.
> > 
> > Thanks.  AI review expressed a couple of concerns:
> > 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
> 
> 
> Sounds legit to me. Now there is no guarantee of purge being successful, and we
> can get a spurious failure.
> 
> How about using a WQ_RECLAIM workqueue:
> 
> vmap_purge_wq = alloc_workqueue("vmap_purge",
> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> 
> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
> 
WQ_MEM_RECLAIM makes sense but this is another patch.

I copied here AI comment:
>
> Does replacing this blocking lock with a trylock break synchronization for
> the callers?
>
No it does not. If someone is doing reclaim we do not wait and do not try
to do it again thus fail allocation.

> When vmalloc space is exhausted, alloc_vmap_area() calls
> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
> that free space has actually been reclaimed before looping back to retry:
> mm/vmalloc.c:alloc_vmap_area() {
>    ...
> overflow:
>	if (!purged) {
>		reclaim_and_purge_vmap_areas();
>		purged = 1;
>		goto retry;
>	}
>    ...
>}
> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
> fails and the function returns immediately.
>
If reclaim is in progress and trylock fails a caller repeats only one
time to retry an allocation. There is no any infinite loop.

>
> The allocator then retries
> instantly without waiting for the concurrent purge to complete.
> Because the retry fails and purged is already 1, could this cause the
> allocation to abort and return a spurious vmalloc allocation failure
> (-EBUSY or -ENOMEM)?
>
vmap space can be fragmented and not avail for 32-bit systems. For
64-bit system it is likely impossible.

But, i think we can overt mutex_lock() into mutex_trylock() just only
in the:

static unsigned long
vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
{
	struct vmap_node *vn;

	guard(mutex)(&vmap_purge_lock);
	for_each_vmap_node(vn)
		decay_va_pool_node(vn, true);

	return SHRINK_STOP;
}

so the reclaim path is not blocked. It should also address an issue
reported by the Ye Liu <ye.liu@linux.dev>.

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Andrew Morton]

On Mon, 31 Aug 2026 11:54:22 +0200 Uladzislau Rezki <urezki@gmail.com> wrote:

> But, i think we can overt mutex_lock() into mutex_trylock() just only
> in the:
> 
> static unsigned long
> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> {
> 	struct vmap_node *vn;
> 
> 	guard(mutex)(&vmap_purge_lock);
> 	for_each_vmap_node(vn)
> 		decay_va_pool_node(vn, true);
> 
> 	return SHRINK_STOP;
> }

I *think* that means we can expect a v3.

Ho hum.  I guess I'll queue v2, under the assumption that my
assumptions are usually wrong.

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Dev Jain]

On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
>>
>>
>> On 28/08/26 11:35 pm, Andrew Morton wrote:
>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
>>>
>>>> From: Ye Liu <liuye@kylinos.cn>
>>>>
>>>> The vmap_purge_lock mutex can be held for an extended period by
>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
>>>> purge_vmap_node workers while holding the lock.  Under memory
>>>> pressure, those workers may themselves be blocked in direct
>>>> reclaim trying to acquire the same lock via the
>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
>>>> dependency that deadlocks the entire system.
>>>>
>>>> Two places acquire vmap_purge_lock from paths that can be reached
>>>> during direct reclaim:
>>>>
>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
>>>>    decay cycle when the lock is contended is harmless and prevents
>>>>    tasks from piling up on the mutex in the direct reclaim path.
>>>>
>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>>>>    mutex_trylock().  This is called from the vmalloc allocation
>>>>    overflow path; if trylock fails, another thread is already
>>>>    purging and the allocator's retry will find freed space.  The
>>>>    notifier chain provides a fallback if the retry still fails.
>>>>
>>>> Both trylock failures break the circular dependency: the lock
>>>> holder's flush_work() can complete because workers are no longer
>>>> blocked on vmap_purge_lock in the direct reclaim path.
>>>
>>> Thanks.  AI review expressed a couple of concerns:
>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
>>
>>
>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
>> can get a spurious failure.
>>
>> How about using a WQ_RECLAIM workqueue:
>>
>> vmap_purge_wq = alloc_workqueue("vmap_purge",
>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
>>
>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
>>
> WQ_MEM_RECLAIM makes sense but this is another patch.
> 
> I copied here AI comment:
>>
>> Does replacing this blocking lock with a trylock break synchronization for
>> the callers?
>>
> No it does not. If someone is doing reclaim we do not wait and do not try
> to do it again thus fail allocation.
> 
>> When vmalloc space is exhausted, alloc_vmap_area() calls
>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
>> that free space has actually been reclaimed before looping back to retry:
>> mm/vmalloc.c:alloc_vmap_area() {
>>    ...
>> overflow:
>> 	if (!purged) {
>> 		reclaim_and_purge_vmap_areas();
>> 		purged = 1;
>> 		goto retry;
>> 	}
>>    ...
>> }
>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
>> fails and the function returns immediately.
>>
> If reclaim is in progress and trylock fails a caller repeats only one
> time to retry an allocation. There is no any infinite loop.
> 
>>
>> The allocator then retries
>> instantly without waiting for the concurrent purge to complete.
>> Because the retry fails and purged is already 1, could this cause the
>> allocation to abort and return a spurious vmalloc allocation failure
>> (-EBUSY or -ENOMEM)?
>>
> vmap space can be fragmented and not avail for 32-bit systems. For
> 64-bit system it is likely impossible.
> 
> But, i think we can overt mutex_lock() into mutex_trylock() just only
> in the:
> 
> static unsigned long
> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> {
> 	struct vmap_node *vn;
> 
> 	guard(mutex)(&vmap_purge_lock);
> 	for_each_vmap_node(vn)
> 		decay_va_pool_node(vn, true);
> 
> 	return SHRINK_STOP;
> }
> 
> so the reclaim path is not blocked. It should also address an issue
> reported by the Ye Liu <ye.liu@linux.dev>.

IIUC you are suggesting mutex_trylock() only in the shrinker path. But
then, the following is possible no: take purge lock, try to get a
worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
-> reclaim_and_purge_vmap_areas -> take purge lock?


> 
> --
> Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Ye Liu]

在 2026/9/1 14:22, Dev Jain 写道:
> 
> 
> On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
>> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
>>>
>>>
>>> On 28/08/26 11:35 pm, Andrew Morton wrote:
>>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
>>>>
>>>>> From: Ye Liu <liuye@kylinos.cn>
>>>>>
>>>>> The vmap_purge_lock mutex can be held for an extended period by
>>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
>>>>> purge_vmap_node workers while holding the lock.  Under memory
>>>>> pressure, those workers may themselves be blocked in direct
>>>>> reclaim trying to acquire the same lock via the
>>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
>>>>> dependency that deadlocks the entire system.
>>>>>
>>>>> Two places acquire vmap_purge_lock from paths that can be reached
>>>>> during direct reclaim:
>>>>>
>>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
>>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
>>>>>    decay cycle when the lock is contended is harmless and prevents
>>>>>    tasks from piling up on the mutex in the direct reclaim path.
>>>>>
>>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>>>>>    mutex_trylock().  This is called from the vmalloc allocation
>>>>>    overflow path; if trylock fails, another thread is already
>>>>>    purging and the allocator's retry will find freed space.  The
>>>>>    notifier chain provides a fallback if the retry still fails.
>>>>>
>>>>> Both trylock failures break the circular dependency: the lock
>>>>> holder's flush_work() can complete because workers are no longer
>>>>> blocked on vmap_purge_lock in the direct reclaim path.
>>>>
>>>> Thanks.  AI review expressed a couple of concerns:
>>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
>>>
>>>
>>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
>>> can get a spurious failure.
>>>
>>> How about using a WQ_RECLAIM workqueue:
>>>
>>> vmap_purge_wq = alloc_workqueue("vmap_purge",
>>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
>>>
>>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
>>>
>> WQ_MEM_RECLAIM makes sense but this is another patch.
>>
>> I copied here AI comment:
>>>
>>> Does replacing this blocking lock with a trylock break synchronization for
>>> the callers?
>>>
>> No it does not. If someone is doing reclaim we do not wait and do not try
>> to do it again thus fail allocation.
>>
>>> When vmalloc space is exhausted, alloc_vmap_area() calls
>>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
>>> that free space has actually been reclaimed before looping back to retry:
>>> mm/vmalloc.c:alloc_vmap_area() {
>>>    ...
>>> overflow:
>>> 	if (!purged) {
>>> 		reclaim_and_purge_vmap_areas();
>>> 		purged = 1;
>>> 		goto retry;
>>> 	}
>>>    ...
>>> }
>>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
>>> fails and the function returns immediately.
>>>
>> If reclaim is in progress and trylock fails a caller repeats only one
>> time to retry an allocation. There is no any infinite loop.
>>
>>>
>>> The allocator then retries
>>> instantly without waiting for the concurrent purge to complete.
>>> Because the retry fails and purged is already 1, could this cause the
>>> allocation to abort and return a spurious vmalloc allocation failure
>>> (-EBUSY or -ENOMEM)?
>>>
>> vmap space can be fragmented and not avail for 32-bit systems. For
>> 64-bit system it is likely impossible.
>>
>> But, i think we can overt mutex_lock() into mutex_trylock() just only
>> in the:
>>
>> static unsigned long
>> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
>> {
>> 	struct vmap_node *vn;
>>
>> 	guard(mutex)(&vmap_purge_lock);
>> 	for_each_vmap_node(vn)
>> 		decay_va_pool_node(vn, true);
>>
>> 	return SHRINK_STOP;
>> }
>>
>> so the reclaim path is not blocked. It should also address an issue
>> reported by the Ye Liu <ye.liu@linux.dev>.
> 
> IIUC you are suggesting mutex_trylock() only in the shrinker path. But
> then, the following is possible no: take purge lock, try to get a
> worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
> -> reclaim_and_purge_vmap_areas -> take purge lock?
> 
> 

Personally, I lean toward the WQ_MEM_RECLAIM workqueue solution. 
After taking a closer look at the code, I noticed a subtle but 
potentially problematic scenario:

When drain_vmap_area_work acquires vmap_purge_lock and calls into 
__purge_vmap_area_lazy, it may subsequently invoke queue_work/queue_work_on 
on the same CPU's system_wq. If the newly queued work ends up waiting 
for an available worker on that same CPU, while the current worker is
blocked waiting for that very work to complete (via flush_work), 
we could end up with a self-deadlock on a single CPU.

Theoretically, this seems possible. I suspect the reason we don't 
see widespread reports of such deadlocks is that the nr_purge_helpers 
logic limits the number of asynchronous workers; when resources are tight, 
it falls back to synchronous execution (purge_vmap_node directly), 
which avoids queuing additional work. 

Using a dedicated workqueue with WQ_MEM_RECLAIM would provide a clean, 
explicit isolation—ensuring forward progress under memory pressure and 
eliminating the risk of interfering with other subsystems' workqueues. 
I believe this approach is more robust in the long run.

Perhaps like the code below：
the dedicated queue eliminates the self‑deadlock risk, and the trylock 
in the shrinker prevents recursive lock attempts from reclaim contexts.

diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index bea9f76ed7e7..68fc1f5acb2f 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -2218,6 +2218,9 @@ static unsigned long lazy_max_pages(void)
  */
 static DEFINE_MUTEX(vmap_purge_lock);
 
+/* Workqueue for lazy vmap purging; WQ_MEM_RECLAIM guarantees progress. */
+static struct workqueue_struct *vmap_purge_wq;
+
 /* for per-CPU blocks */
 static void purge_fragmented_blocks_allcpus(void);
 
@@ -2408,9 +2411,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
                                INIT_WORK(&vn->purge_work, purge_vmap_node);
 
                                if (cpumask_test_cpu(i, cpu_online_mask))
-                                       schedule_work_on(i, &vn->purge_work);
+                                       queue_work_on(i, vmap_purge_wq, &vn->purge_work);
                                else
-                                       schedule_work(&vn->purge_work);
+                                       queue_work(vmap_purge_wq, &vn->purge_work);
 
                                nr_purge_helpers--;
                        } else {
@@ -5519,10 +5522,14 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
 {
        struct vmap_node *vn;
 
-       guard(mutex)(&vmap_purge_lock);
+       if (!mutex_trylock(&vmap_purge_lock))
+               return SHRINK_STOP;
+
        for_each_vmap_node(vn)
                decay_va_pool_node(vn, true);
 
+       mutex_unlock(&vmap_purge_lock);
+
        return SHRINK_STOP;
 }
 
@@ -5575,6 +5582,17 @@ void __init vmalloc_init(void)
         * Now we can initialize a free vmap space.
         */
        vmap_init_free_space();
+
+       /*
+        * A dedicated workqueue for lazy vmap purging.  WQ_MEM_RECLAIM
+        * reserves a rescue worker so queued purge work items are executed
+        * even under memory pressure, when workers of the system workqueue
+        * may be stuck in direct reclaim.
+        */
+       vmap_purge_wq = alloc_workqueue("vmap_purge",
+                                       WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+       WARN_ON(!vmap_purge_wq);
+
        vmap_initialized = true;


>>
>> --
>> Uladzislau Rezki
> 

-- 
Thanks,
Ye Liu

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Mon, Aug 31, 2026 at 06:42:52PM -0700, Andrew Morton wrote:
> On Mon, 31 Aug 2026 11:54:22 +0200 Uladzislau Rezki <urezki@gmail.com> wrote:
> 
> > But, i think we can overt mutex_lock() into mutex_trylock() just only
> > in the:
> > 
> > static unsigned long
> > vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> > {
> > 	struct vmap_node *vn;
> > 
> > 	guard(mutex)(&vmap_purge_lock);
> > 	for_each_vmap_node(vn)
> > 		decay_va_pool_node(vn, true);
> > 
> > 	return SHRINK_STOP;
> > }
> 
> I *think* that means we can expect a v3.
> 
> Ho hum.  I guess I'll queue v2, under the assumption that my
> assumptions are usually wrong.
>
Yep :) 

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Tue, Sep 01, 2026 at 11:52:11AM +0530, Dev Jain wrote:
> 
> 
> On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
> > On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
> >>
> >>
> >> On 28/08/26 11:35 pm, Andrew Morton wrote:
> >>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> >>>
> >>>> From: Ye Liu <liuye@kylinos.cn>
> >>>>
> >>>> The vmap_purge_lock mutex can be held for an extended period by
> >>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
> >>>> purge_vmap_node workers while holding the lock.  Under memory
> >>>> pressure, those workers may themselves be blocked in direct
> >>>> reclaim trying to acquire the same lock via the
> >>>> vmap_node_shrink_scan() shrinker callback, creating a circular
> >>>> dependency that deadlocks the entire system.
> >>>>
> >>>> Two places acquire vmap_purge_lock from paths that can be reached
> >>>> during direct reclaim:
> >>>>
> >>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
> >>>>    mutex_trylock().  This is a shrinker that only decays the vmap
> >>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
> >>>>    decay cycle when the lock is contended is harmless and prevents
> >>>>    tasks from piling up on the mutex in the direct reclaim path.
> >>>>
> >>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
> >>>>    mutex_trylock().  This is called from the vmalloc allocation
> >>>>    overflow path; if trylock fails, another thread is already
> >>>>    purging and the allocator's retry will find freed space.  The
> >>>>    notifier chain provides a fallback if the retry still fails.
> >>>>
> >>>> Both trylock failures break the circular dependency: the lock
> >>>> holder's flush_work() can complete because workers are no longer
> >>>> blocked on vmap_purge_lock in the direct reclaim path.
> >>>
> >>> Thanks.  AI review expressed a couple of concerns:
> >>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
> >>
> >>
> >> Sounds legit to me. Now there is no guarantee of purge being successful, and we
> >> can get a spurious failure.
> >>
> >> How about using a WQ_RECLAIM workqueue:
> >>
> >> vmap_purge_wq = alloc_workqueue("vmap_purge",
> >> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> >>
> >> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
> >>
> > WQ_MEM_RECLAIM makes sense but this is another patch.
> > 
> > I copied here AI comment:
> >>
> >> Does replacing this blocking lock with a trylock break synchronization for
> >> the callers?
> >>
> > No it does not. If someone is doing reclaim we do not wait and do not try
> > to do it again thus fail allocation.
> > 
> >> When vmalloc space is exhausted, alloc_vmap_area() calls
> >> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
> >> that free space has actually been reclaimed before looping back to retry:
> >> mm/vmalloc.c:alloc_vmap_area() {
> >>    ...
> >> overflow:
> >> 	if (!purged) {
> >> 		reclaim_and_purge_vmap_areas();
> >> 		purged = 1;
> >> 		goto retry;
> >> 	}
> >>    ...
> >> }
> >> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
> >> fails and the function returns immediately.
> >>
> > If reclaim is in progress and trylock fails a caller repeats only one
> > time to retry an allocation. There is no any infinite loop.
> > 
> >>
> >> The allocator then retries
> >> instantly without waiting for the concurrent purge to complete.
> >> Because the retry fails and purged is already 1, could this cause the
> >> allocation to abort and return a spurious vmalloc allocation failure
> >> (-EBUSY or -ENOMEM)?
> >>
> > vmap space can be fragmented and not avail for 32-bit systems. For
> > 64-bit system it is likely impossible.
> > 
> > But, i think we can overt mutex_lock() into mutex_trylock() just only
> > in the:
> > 
> > static unsigned long
> > vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> > {
> > 	struct vmap_node *vn;
> > 
> > 	guard(mutex)(&vmap_purge_lock);
> > 	for_each_vmap_node(vn)
> > 		decay_va_pool_node(vn, true);
> > 
> > 	return SHRINK_STOP;
> > }
> > 
> > so the reclaim path is not blocked. It should also address an issue
> > reported by the Ye Liu <ye.liu@linux.dev>.
> 
> IIUC you are suggesting mutex_trylock() only in the shrinker path. But
> then, the following is possible no: take purge lock, try to get a
> worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
> -> reclaim_and_purge_vmap_areas -> take purge lock?
> 
It is possible what you describe. Usually when no memory, the work item
can be delayed for a while. As it was noted WQ_RECLAIM and separate WQ
which has a rescue worker.

IMO, it should be a separate patch.

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Tue, Sep 01, 2026 at 05:07:30PM +0800, Ye Liu wrote:
> 
> 
> 在 2026/9/1 14:22, Dev Jain 写道:
> > 
> > 
> > On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
> >> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
> >>>
> >>>
> >>> On 28/08/26 11:35 pm, Andrew Morton wrote:
> >>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> >>>>
> >>>>> From: Ye Liu <liuye@kylinos.cn>
> >>>>>
> >>>>> The vmap_purge_lock mutex can be held for an extended period by
> >>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
> >>>>> purge_vmap_node workers while holding the lock.  Under memory
> >>>>> pressure, those workers may themselves be blocked in direct
> >>>>> reclaim trying to acquire the same lock via the
> >>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
> >>>>> dependency that deadlocks the entire system.
> >>>>>
> >>>>> Two places acquire vmap_purge_lock from paths that can be reached
> >>>>> during direct reclaim:
> >>>>>
> >>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
> >>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
> >>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
> >>>>>    decay cycle when the lock is contended is harmless and prevents
> >>>>>    tasks from piling up on the mutex in the direct reclaim path.
> >>>>>
> >>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
> >>>>>    mutex_trylock().  This is called from the vmalloc allocation
> >>>>>    overflow path; if trylock fails, another thread is already
> >>>>>    purging and the allocator's retry will find freed space.  The
> >>>>>    notifier chain provides a fallback if the retry still fails.
> >>>>>
> >>>>> Both trylock failures break the circular dependency: the lock
> >>>>> holder's flush_work() can complete because workers are no longer
> >>>>> blocked on vmap_purge_lock in the direct reclaim path.
> >>>>
> >>>> Thanks.  AI review expressed a couple of concerns:
> >>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
> >>>
> >>>
> >>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
> >>> can get a spurious failure.
> >>>
> >>> How about using a WQ_RECLAIM workqueue:
> >>>
> >>> vmap_purge_wq = alloc_workqueue("vmap_purge",
> >>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> >>>
> >>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
> >>>
> >> WQ_MEM_RECLAIM makes sense but this is another patch.
> >>
> >> I copied here AI comment:
> >>>
> >>> Does replacing this blocking lock with a trylock break synchronization for
> >>> the callers?
> >>>
> >> No it does not. If someone is doing reclaim we do not wait and do not try
> >> to do it again thus fail allocation.
> >>
> >>> When vmalloc space is exhausted, alloc_vmap_area() calls
> >>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
> >>> that free space has actually been reclaimed before looping back to retry:
> >>> mm/vmalloc.c:alloc_vmap_area() {
> >>>    ...
> >>> overflow:
> >>> 	if (!purged) {
> >>> 		reclaim_and_purge_vmap_areas();
> >>> 		purged = 1;
> >>> 		goto retry;
> >>> 	}
> >>>    ...
> >>> }
> >>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
> >>> fails and the function returns immediately.
> >>>
> >> If reclaim is in progress and trylock fails a caller repeats only one
> >> time to retry an allocation. There is no any infinite loop.
> >>
> >>>
> >>> The allocator then retries
> >>> instantly without waiting for the concurrent purge to complete.
> >>> Because the retry fails and purged is already 1, could this cause the
> >>> allocation to abort and return a spurious vmalloc allocation failure
> >>> (-EBUSY or -ENOMEM)?
> >>>
> >> vmap space can be fragmented and not avail for 32-bit systems. For
> >> 64-bit system it is likely impossible.
> >>
> >> But, i think we can overt mutex_lock() into mutex_trylock() just only
> >> in the:
> >>
> >> static unsigned long
> >> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> >> {
> >> 	struct vmap_node *vn;
> >>
> >> 	guard(mutex)(&vmap_purge_lock);
> >> 	for_each_vmap_node(vn)
> >> 		decay_va_pool_node(vn, true);
> >>
> >> 	return SHRINK_STOP;
> >> }
> >>
> >> so the reclaim path is not blocked. It should also address an issue
> >> reported by the Ye Liu <ye.liu@linux.dev>.
> > 
> > IIUC you are suggesting mutex_trylock() only in the shrinker path. But
> > then, the following is possible no: take purge lock, try to get a
> > worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
> > -> reclaim_and_purge_vmap_areas -> take purge lock?
> > 
> > 
> 
> Personally, I lean toward the WQ_MEM_RECLAIM workqueue solution. 
> After taking a closer look at the code, I noticed a subtle but 
> potentially problematic scenario:
> 
> When drain_vmap_area_work acquires vmap_purge_lock and calls into 
> __purge_vmap_area_lazy, it may subsequently invoke queue_work/queue_work_on 
> on the same CPU's system_wq. If the newly queued work ends up waiting 
> for an available worker on that same CPU, while the current worker is
> blocked waiting for that very work to complete (via flush_work), 
> we could end up with a self-deadlock on a single CPU.
> 
> Theoretically, this seems possible. I suspect the reason we don't 
> see widespread reports of such deadlocks is that the nr_purge_helpers 
> logic limits the number of asynchronous workers; when resources are tight, 
> it falls back to synchronous execution (purge_vmap_node directly), 
> which avoids queuing additional work. 
> 
> Using a dedicated workqueue with WQ_MEM_RECLAIM would provide a clean, 
> explicit isolation—ensuring forward progress under memory pressure and 
> eliminating the risk of interfering with other subsystems' workqueues. 
> I believe this approach is more robust in the long run.
> 
> Perhaps like the code below：
> the dedicated queue eliminates the self‑deadlock risk, and the trylock 
> in the shrinker prevents recursive lock attempts from reclaim contexts.
> 
> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> index bea9f76ed7e7..68fc1f5acb2f 100644
> --- a/mm/vmalloc.c
> +++ b/mm/vmalloc.c
> @@ -2218,6 +2218,9 @@ static unsigned long lazy_max_pages(void)
>   */
>  static DEFINE_MUTEX(vmap_purge_lock);
>  
> +/* Workqueue for lazy vmap purging; WQ_MEM_RECLAIM guarantees progress. */
> +static struct workqueue_struct *vmap_purge_wq;
> +
>  /* for per-CPU blocks */
>  static void purge_fragmented_blocks_allcpus(void);
>  
> @@ -2408,9 +2411,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
>                                 INIT_WORK(&vn->purge_work, purge_vmap_node);
>  
>                                 if (cpumask_test_cpu(i, cpu_online_mask))
> -                                       schedule_work_on(i, &vn->purge_work);
> +                                       queue_work_on(i, vmap_purge_wq, &vn->purge_work);
>                                 else
> -                                       schedule_work(&vn->purge_work);
> +                                       queue_work(vmap_purge_wq, &vn->purge_work);
>  
>                                 nr_purge_helpers--;
>                         } else {
> @@ -5519,10 +5522,14 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
>  {
>         struct vmap_node *vn;
>  
> -       guard(mutex)(&vmap_purge_lock);
> +       if (!mutex_trylock(&vmap_purge_lock))
> +               return SHRINK_STOP;
> +
>         for_each_vmap_node(vn)
>                 decay_va_pool_node(vn, true);
>  
> +       mutex_unlock(&vmap_purge_lock);
> +
>         return SHRINK_STOP;
>  }
>  
> @@ -5575,6 +5582,17 @@ void __init vmalloc_init(void)
>          * Now we can initialize a free vmap space.
>          */
>         vmap_init_free_space();
> +
> +       /*
> +        * A dedicated workqueue for lazy vmap purging.  WQ_MEM_RECLAIM
> +        * reserves a rescue worker so queued purge work items are executed
> +        * even under memory pressure, when workers of the system workqueue
> +        * may be stuck in direct reclaim.
> +        */
> +       vmap_purge_wq = alloc_workqueue("vmap_purge",
> +                                       WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> +       WARN_ON(!vmap_purge_wq);
> +
>         vmap_initialized = true;
> 
I agree. We should have it and it should be as separate patch, i.e.
split vmap_node_shrink_scan() and dedicated per-cpu WQs per vmap drain.

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Tue, Sep 01, 2026 at 06:43:43PM +0200, Uladzislau Rezki wrote:
> On Tue, Sep 01, 2026 at 05:07:30PM +0800, Ye Liu wrote:
> > 
> > 
> > 在 2026/9/1 14:22, Dev Jain 写道:
> > > 
> > > 
> > > On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
> > >> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
> > >>>
> > >>>
> > >>> On 28/08/26 11:35 pm, Andrew Morton wrote:
> > >>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> > >>>>
> > >>>>> From: Ye Liu <liuye@kylinos.cn>
> > >>>>>
> > >>>>> The vmap_purge_lock mutex can be held for an extended period by
> > >>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
> > >>>>> purge_vmap_node workers while holding the lock.  Under memory
> > >>>>> pressure, those workers may themselves be blocked in direct
> > >>>>> reclaim trying to acquire the same lock via the
> > >>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
> > >>>>> dependency that deadlocks the entire system.
> > >>>>>
> > >>>>> Two places acquire vmap_purge_lock from paths that can be reached
> > >>>>> during direct reclaim:
> > >>>>>
> > >>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
> > >>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
> > >>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
> > >>>>>    decay cycle when the lock is contended is harmless and prevents
> > >>>>>    tasks from piling up on the mutex in the direct reclaim path.
> > >>>>>
> > >>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
> > >>>>>    mutex_trylock().  This is called from the vmalloc allocation
> > >>>>>    overflow path; if trylock fails, another thread is already
> > >>>>>    purging and the allocator's retry will find freed space.  The
> > >>>>>    notifier chain provides a fallback if the retry still fails.
> > >>>>>
> > >>>>> Both trylock failures break the circular dependency: the lock
> > >>>>> holder's flush_work() can complete because workers are no longer
> > >>>>> blocked on vmap_purge_lock in the direct reclaim path.
> > >>>>
> > >>>> Thanks.  AI review expressed a couple of concerns:
> > >>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
> > >>>
> > >>>
> > >>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
> > >>> can get a spurious failure.
> > >>>
> > >>> How about using a WQ_RECLAIM workqueue:
> > >>>
> > >>> vmap_purge_wq = alloc_workqueue("vmap_purge",
> > >>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> > >>>
> > >>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
> > >>>
> > >> WQ_MEM_RECLAIM makes sense but this is another patch.
> > >>
> > >> I copied here AI comment:
> > >>>
> > >>> Does replacing this blocking lock with a trylock break synchronization for
> > >>> the callers?
> > >>>
> > >> No it does not. If someone is doing reclaim we do not wait and do not try
> > >> to do it again thus fail allocation.
> > >>
> > >>> When vmalloc space is exhausted, alloc_vmap_area() calls
> > >>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
> > >>> that free space has actually been reclaimed before looping back to retry:
> > >>> mm/vmalloc.c:alloc_vmap_area() {
> > >>>    ...
> > >>> overflow:
> > >>> 	if (!purged) {
> > >>> 		reclaim_and_purge_vmap_areas();
> > >>> 		purged = 1;
> > >>> 		goto retry;
> > >>> 	}
> > >>>    ...
> > >>> }
> > >>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
> > >>> fails and the function returns immediately.
> > >>>
> > >> If reclaim is in progress and trylock fails a caller repeats only one
> > >> time to retry an allocation. There is no any infinite loop.
> > >>
> > >>>
> > >>> The allocator then retries
> > >>> instantly without waiting for the concurrent purge to complete.
> > >>> Because the retry fails and purged is already 1, could this cause the
> > >>> allocation to abort and return a spurious vmalloc allocation failure
> > >>> (-EBUSY or -ENOMEM)?
> > >>>
> > >> vmap space can be fragmented and not avail for 32-bit systems. For
> > >> 64-bit system it is likely impossible.
> > >>
> > >> But, i think we can overt mutex_lock() into mutex_trylock() just only
> > >> in the:
> > >>
> > >> static unsigned long
> > >> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> > >> {
> > >> 	struct vmap_node *vn;
> > >>
> > >> 	guard(mutex)(&vmap_purge_lock);
> > >> 	for_each_vmap_node(vn)
> > >> 		decay_va_pool_node(vn, true);
> > >>
> > >> 	return SHRINK_STOP;
> > >> }
> > >>
> > >> so the reclaim path is not blocked. It should also address an issue
> > >> reported by the Ye Liu <ye.liu@linux.dev>.
> > > 
> > > IIUC you are suggesting mutex_trylock() only in the shrinker path. But
> > > then, the following is possible no: take purge lock, try to get a
> > > worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
> > > -> reclaim_and_purge_vmap_areas -> take purge lock?
> > > 
> > > 
> > 
> > Personally, I lean toward the WQ_MEM_RECLAIM workqueue solution. 
> > After taking a closer look at the code, I noticed a subtle but 
> > potentially problematic scenario:
> > 
> > When drain_vmap_area_work acquires vmap_purge_lock and calls into 
> > __purge_vmap_area_lazy, it may subsequently invoke queue_work/queue_work_on 
> > on the same CPU's system_wq. If the newly queued work ends up waiting 
> > for an available worker on that same CPU, while the current worker is
> > blocked waiting for that very work to complete (via flush_work), 
> > we could end up with a self-deadlock on a single CPU.
> > 
> > Theoretically, this seems possible. I suspect the reason we don't 
> > see widespread reports of such deadlocks is that the nr_purge_helpers 
> > logic limits the number of asynchronous workers; when resources are tight, 
> > it falls back to synchronous execution (purge_vmap_node directly), 
> > which avoids queuing additional work. 
> > 
> > Using a dedicated workqueue with WQ_MEM_RECLAIM would provide a clean, 
> > explicit isolation—ensuring forward progress under memory pressure and 
> > eliminating the risk of interfering with other subsystems' workqueues. 
> > I believe this approach is more robust in the long run.
> > 
> > Perhaps like the code below：
> > the dedicated queue eliminates the self‑deadlock risk, and the trylock 
> > in the shrinker prevents recursive lock attempts from reclaim contexts.
> > 
> > diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> > index bea9f76ed7e7..68fc1f5acb2f 100644
> > --- a/mm/vmalloc.c
> > +++ b/mm/vmalloc.c
> > @@ -2218,6 +2218,9 @@ static unsigned long lazy_max_pages(void)
> >   */
> >  static DEFINE_MUTEX(vmap_purge_lock);
> >  
> > +/* Workqueue for lazy vmap purging; WQ_MEM_RECLAIM guarantees progress. */
> > +static struct workqueue_struct *vmap_purge_wq;
> > +
> >  /* for per-CPU blocks */
> >  static void purge_fragmented_blocks_allcpus(void);
> >  
> > @@ -2408,9 +2411,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
> >                                 INIT_WORK(&vn->purge_work, purge_vmap_node);
> >  
> >                                 if (cpumask_test_cpu(i, cpu_online_mask))
> > -                                       schedule_work_on(i, &vn->purge_work);
> > +                                       queue_work_on(i, vmap_purge_wq, &vn->purge_work);
> >                                 else
> > -                                       schedule_work(&vn->purge_work);
> > +                                       queue_work(vmap_purge_wq, &vn->purge_work);
> >  
> >                                 nr_purge_helpers--;
> >                         } else {
> > @@ -5519,10 +5522,14 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> >  {
> >         struct vmap_node *vn;
> >  
> > -       guard(mutex)(&vmap_purge_lock);
> > +       if (!mutex_trylock(&vmap_purge_lock))
> > +               return SHRINK_STOP;
> > +
> >         for_each_vmap_node(vn)
> >                 decay_va_pool_node(vn, true);
> >  
> > +       mutex_unlock(&vmap_purge_lock);
> > +
> >         return SHRINK_STOP;
> >  }
> >  
> > @@ -5575,6 +5582,17 @@ void __init vmalloc_init(void)
> >          * Now we can initialize a free vmap space.
> >          */
> >         vmap_init_free_space();
> > +
> > +       /*
> > +        * A dedicated workqueue for lazy vmap purging.  WQ_MEM_RECLAIM
> > +        * reserves a rescue worker so queued purge work items are executed
> > +        * even under memory pressure, when workers of the system workqueue
> > +        * may be stuck in direct reclaim.
> > +        */
> > +       vmap_purge_wq = alloc_workqueue("vmap_purge",
> > +                                       WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> > +       WARN_ON(!vmap_purge_wq);
> > +
> >         vmap_initialized = true;
> > 
> I agree. We should have it and it should be as separate patch, i.e.
> split vmap_node_shrink_scan() and dedicated per-cpu WQs per vmap drain.
> 
And i sent out already the WQ_UNBOUND | WQ_MEM_RECLAIM and separate WQ
for vmap drain logic. It looks like Andrew/me forgot about it:

https://lore.kernel.org/all/20260331202352.879718-1-urezki@gmail.com/

--
Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Dev Jain]

On 01/09/26 10:29 pm, Uladzislau Rezki wrote:
> On Tue, Sep 01, 2026 at 06:43:43PM +0200, Uladzislau Rezki wrote:
>> On Tue, Sep 01, 2026 at 05:07:30PM +0800, Ye Liu wrote:
>>>
>>>
>>> 在 2026/9/1 14:22, Dev Jain 写道:
>>>>
>>>>
>>>> On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
>>>>> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
>>>>>>
>>>>>>
>>>>>> On 28/08/26 11:35 pm, Andrew Morton wrote:
>>>>>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
>>>>>>>
>>>>>>>> From: Ye Liu <liuye@kylinos.cn>
>>>>>>>>
>>>>>>>> The vmap_purge_lock mutex can be held for an extended period by
>>>>>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
>>>>>>>> purge_vmap_node workers while holding the lock.  Under memory
>>>>>>>> pressure, those workers may themselves be blocked in direct
>>>>>>>> reclaim trying to acquire the same lock via the
>>>>>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
>>>>>>>> dependency that deadlocks the entire system.
>>>>>>>>
>>>>>>>> Two places acquire vmap_purge_lock from paths that can be reached
>>>>>>>> during direct reclaim:
>>>>>>>>
>>>>>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
>>>>>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
>>>>>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
>>>>>>>>    decay cycle when the lock is contended is harmless and prevents
>>>>>>>>    tasks from piling up on the mutex in the direct reclaim path.
>>>>>>>>
>>>>>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
>>>>>>>>    mutex_trylock().  This is called from the vmalloc allocation
>>>>>>>>    overflow path; if trylock fails, another thread is already
>>>>>>>>    purging and the allocator's retry will find freed space.  The
>>>>>>>>    notifier chain provides a fallback if the retry still fails.
>>>>>>>>
>>>>>>>> Both trylock failures break the circular dependency: the lock
>>>>>>>> holder's flush_work() can complete because workers are no longer
>>>>>>>> blocked on vmap_purge_lock in the direct reclaim path.
>>>>>>>
>>>>>>> Thanks.  AI review expressed a couple of concerns:
>>>>>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
>>>>>>
>>>>>>
>>>>>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
>>>>>> can get a spurious failure.
>>>>>>
>>>>>> How about using a WQ_RECLAIM workqueue:
>>>>>>
>>>>>> vmap_purge_wq = alloc_workqueue("vmap_purge",
>>>>>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
>>>>>>
>>>>>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
>>>>>>
>>>>> WQ_MEM_RECLAIM makes sense but this is another patch.
>>>>>
>>>>> I copied here AI comment:
>>>>>>
>>>>>> Does replacing this blocking lock with a trylock break synchronization for
>>>>>> the callers?
>>>>>>
>>>>> No it does not. If someone is doing reclaim we do not wait and do not try
>>>>> to do it again thus fail allocation.
>>>>>
>>>>>> When vmalloc space is exhausted, alloc_vmap_area() calls
>>>>>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
>>>>>> that free space has actually been reclaimed before looping back to retry:
>>>>>> mm/vmalloc.c:alloc_vmap_area() {
>>>>>>    ...
>>>>>> overflow:
>>>>>> 	if (!purged) {
>>>>>> 		reclaim_and_purge_vmap_areas();
>>>>>> 		purged = 1;
>>>>>> 		goto retry;
>>>>>> 	}
>>>>>>    ...
>>>>>> }
>>>>>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
>>>>>> fails and the function returns immediately.
>>>>>>
>>>>> If reclaim is in progress and trylock fails a caller repeats only one
>>>>> time to retry an allocation. There is no any infinite loop.
>>>>>
>>>>>>
>>>>>> The allocator then retries
>>>>>> instantly without waiting for the concurrent purge to complete.
>>>>>> Because the retry fails and purged is already 1, could this cause the
>>>>>> allocation to abort and return a spurious vmalloc allocation failure
>>>>>> (-EBUSY or -ENOMEM)?
>>>>>>
>>>>> vmap space can be fragmented and not avail for 32-bit systems. For
>>>>> 64-bit system it is likely impossible.
>>>>>
>>>>> But, i think we can overt mutex_lock() into mutex_trylock() just only
>>>>> in the:
>>>>>
>>>>> static unsigned long
>>>>> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
>>>>> {
>>>>> 	struct vmap_node *vn;
>>>>>
>>>>> 	guard(mutex)(&vmap_purge_lock);
>>>>> 	for_each_vmap_node(vn)
>>>>> 		decay_va_pool_node(vn, true);
>>>>>
>>>>> 	return SHRINK_STOP;
>>>>> }
>>>>>
>>>>> so the reclaim path is not blocked. It should also address an issue
>>>>> reported by the Ye Liu <ye.liu@linux.dev>.
>>>>
>>>> IIUC you are suggesting mutex_trylock() only in the shrinker path. But
>>>> then, the following is possible no: take purge lock, try to get a
>>>> worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
>>>> -> reclaim_and_purge_vmap_areas -> take purge lock?
>>>>
>>>>
>>>
>>> Personally, I lean toward the WQ_MEM_RECLAIM workqueue solution. 
>>> After taking a closer look at the code, I noticed a subtle but 
>>> potentially problematic scenario:
>>>
>>> When drain_vmap_area_work acquires vmap_purge_lock and calls into 
>>> __purge_vmap_area_lazy, it may subsequently invoke queue_work/queue_work_on 
>>> on the same CPU's system_wq. If the newly queued work ends up waiting 
>>> for an available worker on that same CPU, while the current worker is
>>> blocked waiting for that very work to complete (via flush_work), 
>>> we could end up with a self-deadlock on a single CPU.
>>>
>>> Theoretically, this seems possible. I suspect the reason we don't 
>>> see widespread reports of such deadlocks is that the nr_purge_helpers 
>>> logic limits the number of asynchronous workers; when resources are tight, 
>>> it falls back to synchronous execution (purge_vmap_node directly), 
>>> which avoids queuing additional work. 
>>>
>>> Using a dedicated workqueue with WQ_MEM_RECLAIM would provide a clean, 
>>> explicit isolation—ensuring forward progress under memory pressure and 
>>> eliminating the risk of interfering with other subsystems' workqueues. 
>>> I believe this approach is more robust in the long run.
>>>
>>> Perhaps like the code below：
>>> the dedicated queue eliminates the self‑deadlock risk, and the trylock 
>>> in the shrinker prevents recursive lock attempts from reclaim contexts.
>>>
>>> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
>>> index bea9f76ed7e7..68fc1f5acb2f 100644
>>> --- a/mm/vmalloc.c
>>> +++ b/mm/vmalloc.c
>>> @@ -2218,6 +2218,9 @@ static unsigned long lazy_max_pages(void)
>>>   */
>>>  static DEFINE_MUTEX(vmap_purge_lock);
>>>  
>>> +/* Workqueue for lazy vmap purging; WQ_MEM_RECLAIM guarantees progress. */
>>> +static struct workqueue_struct *vmap_purge_wq;
>>> +
>>>  /* for per-CPU blocks */
>>>  static void purge_fragmented_blocks_allcpus(void);
>>>  
>>> @@ -2408,9 +2411,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
>>>                                 INIT_WORK(&vn->purge_work, purge_vmap_node);
>>>  
>>>                                 if (cpumask_test_cpu(i, cpu_online_mask))
>>> -                                       schedule_work_on(i, &vn->purge_work);
>>> +                                       queue_work_on(i, vmap_purge_wq, &vn->purge_work);
>>>                                 else
>>> -                                       schedule_work(&vn->purge_work);
>>> +                                       queue_work(vmap_purge_wq, &vn->purge_work);
>>>  
>>>                                 nr_purge_helpers--;
>>>                         } else {
>>> @@ -5519,10 +5522,14 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
>>>  {
>>>         struct vmap_node *vn;
>>>  
>>> -       guard(mutex)(&vmap_purge_lock);
>>> +       if (!mutex_trylock(&vmap_purge_lock))
>>> +               return SHRINK_STOP;
>>> +
>>>         for_each_vmap_node(vn)
>>>                 decay_va_pool_node(vn, true);
>>>  
>>> +       mutex_unlock(&vmap_purge_lock);
>>> +
>>>         return SHRINK_STOP;
>>>  }
>>>  
>>> @@ -5575,6 +5582,17 @@ void __init vmalloc_init(void)
>>>          * Now we can initialize a free vmap space.
>>>          */
>>>         vmap_init_free_space();
>>> +
>>> +       /*
>>> +        * A dedicated workqueue for lazy vmap purging.  WQ_MEM_RECLAIM
>>> +        * reserves a rescue worker so queued purge work items are executed
>>> +        * even under memory pressure, when workers of the system workqueue
>>> +        * may be stuck in direct reclaim.
>>> +        */
>>> +       vmap_purge_wq = alloc_workqueue("vmap_purge",
>>> +                                       WQ_MEM_RECLAIM | WQ_PERCPU, 0);
>>> +       WARN_ON(!vmap_purge_wq);
>>> +
>>>         vmap_initialized = true;
>>>
>> I agree. We should have it and it should be as separate patch, i.e.
>> split vmap_node_shrink_scan() and dedicated per-cpu WQs per vmap drain.
>>
> And i sent out already the WQ_UNBOUND | WQ_MEM_RECLAIM and separate WQ
> for vmap drain logic. It looks like Andrew/me forgot about it:
> 
> https://lore.kernel.org/all/20260331202352.879718-1-urezki@gmail.com/

Great! Perhaps resend it and we can review it?


> 
> --
> Uladzislau Rezki

Re: [PATCH v2] mm: vmalloc: fix vmap_purge_lock livelock under memory pressure

[Uladzislau Rezki]

On Wed, Sep 02, 2026 at 10:00:57AM +0530, Dev Jain wrote:
> 
> 
> On 01/09/26 10:29 pm, Uladzislau Rezki wrote:
> > On Tue, Sep 01, 2026 at 06:43:43PM +0200, Uladzislau Rezki wrote:
> >> On Tue, Sep 01, 2026 at 05:07:30PM +0800, Ye Liu wrote:
> >>>
> >>>
> >>> 在 2026/9/1 14:22, Dev Jain 写道:
> >>>>
> >>>>
> >>>> On 31/08/26 3:24 pm, Uladzislau Rezki wrote:
> >>>>> On Mon, Aug 31, 2026 at 11:39:14AM +0530, Dev Jain wrote:
> >>>>>>
> >>>>>>
> >>>>>> On 28/08/26 11:35 pm, Andrew Morton wrote:
> >>>>>>> On Fri, 28 Aug 2026 17:17:53 +0800 Ye Liu <ye.liu@linux.dev> wrote:
> >>>>>>>
> >>>>>>>> From: Ye Liu <liuye@kylinos.cn>
> >>>>>>>>
> >>>>>>>> The vmap_purge_lock mutex can be held for an extended period by
> >>>>>>>> __purge_vmap_area_lazy() which calls flush_work() to wait for
> >>>>>>>> purge_vmap_node workers while holding the lock.  Under memory
> >>>>>>>> pressure, those workers may themselves be blocked in direct
> >>>>>>>> reclaim trying to acquire the same lock via the
> >>>>>>>> vmap_node_shrink_scan() shrinker callback, creating a circular
> >>>>>>>> dependency that deadlocks the entire system.
> >>>>>>>>
> >>>>>>>> Two places acquire vmap_purge_lock from paths that can be reached
> >>>>>>>> during direct reclaim:
> >>>>>>>>
> >>>>>>>> 1. vmap_node_shrink_scan(): replace blocking guard(mutex) with
> >>>>>>>>    mutex_trylock().  This is a shrinker that only decays the vmap
> >>>>>>>>    pool and returns SHRINK_STOP without freeing memory; skipping a
> >>>>>>>>    decay cycle when the lock is contended is harmless and prevents
> >>>>>>>>    tasks from piling up on the mutex in the direct reclaim path.
> >>>>>>>>
> >>>>>>>> 2. reclaim_and_purge_vmap_areas(): replace mutex_lock() with
> >>>>>>>>    mutex_trylock().  This is called from the vmalloc allocation
> >>>>>>>>    overflow path; if trylock fails, another thread is already
> >>>>>>>>    purging and the allocator's retry will find freed space.  The
> >>>>>>>>    notifier chain provides a fallback if the retry still fails.
> >>>>>>>>
> >>>>>>>> Both trylock failures break the circular dependency: the lock
> >>>>>>>> holder's flush_work() can complete because workers are no longer
> >>>>>>>> blocked on vmap_purge_lock in the direct reclaim path.
> >>>>>>>
> >>>>>>> Thanks.  AI review expressed a couple of concerns:
> >>>>>>> 	https://sashiko.dev/#/patchset/20260828091753.299295-1-ye.liu@linux.dev
> >>>>>>
> >>>>>>
> >>>>>> Sounds legit to me. Now there is no guarantee of purge being successful, and we
> >>>>>> can get a spurious failure.
> >>>>>>
> >>>>>> How about using a WQ_RECLAIM workqueue:
> >>>>>>
> >>>>>> vmap_purge_wq = alloc_workqueue("vmap_purge",
> >>>>>> 				WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> >>>>>>
> >>>>>> I see the same pattern in __lru_add_drain_all() and kmem_cache_init_late().
> >>>>>>
> >>>>> WQ_MEM_RECLAIM makes sense but this is another patch.
> >>>>>
> >>>>> I copied here AI comment:
> >>>>>>
> >>>>>> Does replacing this blocking lock with a trylock break synchronization for
> >>>>>> the callers?
> >>>>>>
> >>>>> No it does not. If someone is doing reclaim we do not wait and do not try
> >>>>> to do it again thus fail allocation.
> >>>>>
> >>>>>> When vmalloc space is exhausted, alloc_vmap_area() calls
> >>>>>> reclaim_and_purge_vmap_areas() and relies on its blocking behavior to ensure
> >>>>>> that free space has actually been reclaimed before looping back to retry:
> >>>>>> mm/vmalloc.c:alloc_vmap_area() {
> >>>>>>    ...
> >>>>>> overflow:
> >>>>>> 	if (!purged) {
> >>>>>> 		reclaim_and_purge_vmap_areas();
> >>>>>> 		purged = 1;
> >>>>>> 		goto retry;
> >>>>>> 	}
> >>>>>>    ...
> >>>>>> }
> >>>>>> With this patch, if another thread holds vmap_purge_lock, mutex_trylock()
> >>>>>> fails and the function returns immediately.
> >>>>>>
> >>>>> If reclaim is in progress and trylock fails a caller repeats only one
> >>>>> time to retry an allocation. There is no any infinite loop.
> >>>>>
> >>>>>>
> >>>>>> The allocator then retries
> >>>>>> instantly without waiting for the concurrent purge to complete.
> >>>>>> Because the retry fails and purged is already 1, could this cause the
> >>>>>> allocation to abort and return a spurious vmalloc allocation failure
> >>>>>> (-EBUSY or -ENOMEM)?
> >>>>>>
> >>>>> vmap space can be fragmented and not avail for 32-bit systems. For
> >>>>> 64-bit system it is likely impossible.
> >>>>>
> >>>>> But, i think we can overt mutex_lock() into mutex_trylock() just only
> >>>>> in the:
> >>>>>
> >>>>> static unsigned long
> >>>>> vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> >>>>> {
> >>>>> 	struct vmap_node *vn;
> >>>>>
> >>>>> 	guard(mutex)(&vmap_purge_lock);
> >>>>> 	for_each_vmap_node(vn)
> >>>>> 		decay_va_pool_node(vn, true);
> >>>>>
> >>>>> 	return SHRINK_STOP;
> >>>>> }
> >>>>>
> >>>>> so the reclaim path is not blocked. It should also address an issue
> >>>>> reported by the Ye Liu <ye.liu@linux.dev>.
> >>>>
> >>>> IIUC you are suggesting mutex_trylock() only in the shrinker path. But
> >>>> then, the following is possible no: take purge lock, try to get a
> >>>> worker thread, worker thread is stuck in vmalloc -> alloc_vmap_area
> >>>> -> reclaim_and_purge_vmap_areas -> take purge lock?
> >>>>
> >>>>
> >>>
> >>> Personally, I lean toward the WQ_MEM_RECLAIM workqueue solution. 
> >>> After taking a closer look at the code, I noticed a subtle but 
> >>> potentially problematic scenario:
> >>>
> >>> When drain_vmap_area_work acquires vmap_purge_lock and calls into 
> >>> __purge_vmap_area_lazy, it may subsequently invoke queue_work/queue_work_on 
> >>> on the same CPU's system_wq. If the newly queued work ends up waiting 
> >>> for an available worker on that same CPU, while the current worker is
> >>> blocked waiting for that very work to complete (via flush_work), 
> >>> we could end up with a self-deadlock on a single CPU.
> >>>
> >>> Theoretically, this seems possible. I suspect the reason we don't 
> >>> see widespread reports of such deadlocks is that the nr_purge_helpers 
> >>> logic limits the number of asynchronous workers; when resources are tight, 
> >>> it falls back to synchronous execution (purge_vmap_node directly), 
> >>> which avoids queuing additional work. 
> >>>
> >>> Using a dedicated workqueue with WQ_MEM_RECLAIM would provide a clean, 
> >>> explicit isolation—ensuring forward progress under memory pressure and 
> >>> eliminating the risk of interfering with other subsystems' workqueues. 
> >>> I believe this approach is more robust in the long run.
> >>>
> >>> Perhaps like the code below：
> >>> the dedicated queue eliminates the self‑deadlock risk, and the trylock 
> >>> in the shrinker prevents recursive lock attempts from reclaim contexts.
> >>>
> >>> diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> >>> index bea9f76ed7e7..68fc1f5acb2f 100644
> >>> --- a/mm/vmalloc.c
> >>> +++ b/mm/vmalloc.c
> >>> @@ -2218,6 +2218,9 @@ static unsigned long lazy_max_pages(void)
> >>>   */
> >>>  static DEFINE_MUTEX(vmap_purge_lock);
> >>>  
> >>> +/* Workqueue for lazy vmap purging; WQ_MEM_RECLAIM guarantees progress. */
> >>> +static struct workqueue_struct *vmap_purge_wq;
> >>> +
> >>>  /* for per-CPU blocks */
> >>>  static void purge_fragmented_blocks_allcpus(void);
> >>>  
> >>> @@ -2408,9 +2411,9 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end,
> >>>                                 INIT_WORK(&vn->purge_work, purge_vmap_node);
> >>>  
> >>>                                 if (cpumask_test_cpu(i, cpu_online_mask))
> >>> -                                       schedule_work_on(i, &vn->purge_work);
> >>> +                                       queue_work_on(i, vmap_purge_wq, &vn->purge_work);
> >>>                                 else
> >>> -                                       schedule_work(&vn->purge_work);
> >>> +                                       queue_work(vmap_purge_wq, &vn->purge_work);
> >>>  
> >>>                                 nr_purge_helpers--;
> >>>                         } else {
> >>> @@ -5519,10 +5522,14 @@ vmap_node_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
> >>>  {
> >>>         struct vmap_node *vn;
> >>>  
> >>> -       guard(mutex)(&vmap_purge_lock);
> >>> +       if (!mutex_trylock(&vmap_purge_lock))
> >>> +               return SHRINK_STOP;
> >>> +
> >>>         for_each_vmap_node(vn)
> >>>                 decay_va_pool_node(vn, true);
> >>>  
> >>> +       mutex_unlock(&vmap_purge_lock);
> >>> +
> >>>         return SHRINK_STOP;
> >>>  }
> >>>  
> >>> @@ -5575,6 +5582,17 @@ void __init vmalloc_init(void)
> >>>          * Now we can initialize a free vmap space.
> >>>          */
> >>>         vmap_init_free_space();
> >>> +
> >>> +       /*
> >>> +        * A dedicated workqueue for lazy vmap purging.  WQ_MEM_RECLAIM
> >>> +        * reserves a rescue worker so queued purge work items are executed
> >>> +        * even under memory pressure, when workers of the system workqueue
> >>> +        * may be stuck in direct reclaim.
> >>> +        */
> >>> +       vmap_purge_wq = alloc_workqueue("vmap_purge",
> >>> +                                       WQ_MEM_RECLAIM | WQ_PERCPU, 0);
> >>> +       WARN_ON(!vmap_purge_wq);
> >>> +
> >>>         vmap_initialized = true;
> >>>
> >> I agree. We should have it and it should be as separate patch, i.e.
> >> split vmap_node_shrink_scan() and dedicated per-cpu WQs per vmap drain.
> >>
> > And i sent out already the WQ_UNBOUND | WQ_MEM_RECLAIM and separate WQ
> > for vmap drain logic. It looks like Andrew/me forgot about it:
> > 
> > https://lore.kernel.org/all/20260331202352.879718-1-urezki@gmail.com/
> 
> Great! Perhaps resend it and we can review it?
> 
I will and add you to Cc.

--
Uladzislau Rezki