* [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
@ 2026-11-06 1:39 Jinjiang Tu
2026-11-06 8:28 ` Xu Xin (vivo)
` (2 more replies)
0 siblings, 3 replies; 7+ messages in thread
From: Jinjiang Tu @ 2026-11-06 1:39 UTC (permalink / raw)
To: akpm, david, xu.xin, chengming.zhou, liam, ljs, vbabka, jannh,
pfalcato, kees, ziy, gourry, anthony.yznaga, linux-mm
Cc: wangkefeng.wang, sunnanyong, tujinjiang
create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
while holding mmap_write_lock(), and calls ksm_exit() before dropping the
lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
again when the mm is ksmd's current scan cursor, leading to deadlock.
The early registration is no longer needed after commit 590c03ca6a3f
("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
calls __ksm_enter() when the first mergeable VMA is created. Remove
ksm_execve() and its stubs.
Tested with ksm_functional_tests, all tests pass.
Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
---
include/linux/ksm.h | 13 -------------
mm/ksm.c | 16 +++++++++-------
mm/vma_exec.c | 10 ----------
tools/testing/vma/include/stubs.h | 5 -----
4 files changed, 9 insertions(+), 35 deletions(-)
diff --git a/include/linux/ksm.h b/include/linux/ksm.h
index d39d0d5483a2..5e228935cede 100644
--- a/include/linux/ksm.h
+++ b/include/linux/ksm.h
@@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
}
}
-static inline int ksm_execve(struct mm_struct *mm)
-{
- if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
- return __ksm_enter(mm);
-
- return 0;
-}
-
static inline void ksm_exit(struct mm_struct *mm)
{
if (mm_flags_test(MMF_VM_MERGEABLE, mm))
@@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
{
}
-static inline int ksm_execve(struct mm_struct *mm)
-{
- return 0;
-}
-
static inline void ksm_exit(struct mm_struct *mm)
{
}
diff --git a/mm/ksm.c b/mm/ksm.c
index dbfb8f98b189..2e0f894208d8 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
mm_slot_free(mm_slot_cache, mm_slot);
/*
- * Only clear MMF_VM_MERGEABLE. We must not clear
- * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
- * perhaps their mm_struct has just been added to ksm_mm_slot
- * list, and its process has not yet officially started running
- * or has not yet performed mmap/brk to allocate anonymous VMAS.
+ * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
+ * per-process setting inherited across exec, so it must
+ * stay set until userspace disables it with
+ * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
+ * re-registers the mm through ksm_vma_flags().
*/
mm_flags_clear(MMF_VM_MERGEABLE, mm);
mmap_read_unlock(mm);
@@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
/*
* Generally, the flags here always include MMF_VM_MERGEABLE.
- * However, in rare cases, this flag may be cleared by ksmd who
- * scans a cycle without finding any mergeable vma.
+ * However, a newly execed process has not been added to
+ * ksm_mm_slot. The first mergeable VMA registers the mm
+ * with ksmd here. ksmd can also clear it after a full scan
+ * cycle without finding any mergeable VMA.
*/
if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
__ksm_enter(mm);
diff --git a/mm/vma_exec.c b/mm/vma_exec.c
index 586c52155942..2c644e18b55d 100644
--- a/mm/vma_exec.c
+++ b/mm/vma_exec.c
@@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
goto err_free;
}
- /*
- * Need to be called with mmap write lock
- * held, to avoid race with ksmd.
- */
- err = ksm_execve(mm);
- if (err)
- goto err_ksm;
-
vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
vma_set_anonymous(vma);
@@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
return 0;
err:
- ksm_exit(mm);
-err_ksm:
mmap_write_unlock(mm);
err_free:
*vmap = NULL;
diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
index f786a585caf0..9ab4e524f78e 100644
--- a/tools/testing/vma/include/stubs.h
+++ b/tools/testing/vma/include/stubs.h
@@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
{
}
-static inline int ksm_execve(struct mm_struct *mm)
-{
- return 0;
-}
-
static inline void ksm_exit(struct mm_struct *mm)
{
}
--
2.54.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-06 1:39 [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path Jinjiang Tu
@ 2026-11-06 8:28 ` Xu Xin (vivo)
2026-11-07 2:48 ` Jinjiang Tu
2026-11-06 8:46 ` Xu Xin (vivo)
2026-11-07 19:06 ` David Hildenbrand (Arm)
2 siblings, 1 reply; 7+ messages in thread
From: Xu Xin (vivo) @ 2026-11-06 8:28 UTC (permalink / raw)
To: Jinjiang Tu
Cc: akpm, david, chengming.zhou, liam, ljs, vbabka, jannh, pfalcato,
kees, ziy, gourry, anthony.yznaga, linux-mm, wangkefeng.wang,
sunnanyong
On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
>
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
We can go into more detail. like:
"The early registration ksm_execve() during the exec() is no longer required,
since when the process subsequently carries out the actual memory-allocation
operations, such as brk() or mmap(), ksm_vma_flags is invoked to re-check
MMF_VM_MERGE_ANY and register the mm with KSM."
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
>
> Tested with ksm_functional_tests, all tests pass.
If possible, can you provide a reproducible testcase in
tools/testing/selftests/mm/ksm_functional_tests.c in V2 ?
Generally Looks good to me, thanks!
Acked-by: Xu Xin (vivo) <xu.xin@linux.dev>
Thanks!
>
> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
> ---
> include/linux/ksm.h | 13 -------------
> mm/ksm.c | 16 +++++++++-------
> mm/vma_exec.c | 10 ----------
> tools/testing/vma/include/stubs.h | 5 -----
> 4 files changed, 9 insertions(+), 35 deletions(-)
>
> diff --git a/include/linux/ksm.h b/include/linux/ksm.h
> index d39d0d5483a2..5e228935cede 100644
> --- a/include/linux/ksm.h
> +++ b/include/linux/ksm.h
> @@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
> }
> }
>
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> - if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
> - return __ksm_enter(mm);
> -
> - return 0;
> -}
> -
> static inline void ksm_exit(struct mm_struct *mm)
> {
> if (mm_flags_test(MMF_VM_MERGEABLE, mm))
> @@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
> {
> }
>
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> - return 0;
> -}
> -
> static inline void ksm_exit(struct mm_struct *mm)
> {
> }
> diff --git a/mm/ksm.c b/mm/ksm.c
> index dbfb8f98b189..2e0f894208d8 100644
> --- a/mm/ksm.c
> +++ b/mm/ksm.c
> @@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
>
> mm_slot_free(mm_slot_cache, mm_slot);
> /*
> - * Only clear MMF_VM_MERGEABLE. We must not clear
> - * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
> - * perhaps their mm_struct has just been added to ksm_mm_slot
> - * list, and its process has not yet officially started running
> - * or has not yet performed mmap/brk to allocate anonymous VMAS.
> + * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
> + * per-process setting inherited across exec, so it must
> + * stay set until userspace disables it with
> + * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
> + * re-registers the mm through ksm_vma_flags().
> */
> mm_flags_clear(MMF_VM_MERGEABLE, mm);
> mmap_read_unlock(mm);
> @@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
> vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
> /*
> * Generally, the flags here always include MMF_VM_MERGEABLE.
> - * However, in rare cases, this flag may be cleared by ksmd who
> - * scans a cycle without finding any mergeable vma.
> + * However, a newly execed process has not been added to
> + * ksm_mm_slot. The first mergeable VMA registers the mm
> + * with ksmd here. ksmd can also clear it after a full scan
> + * cycle without finding any mergeable VMA.
> */
> if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
> __ksm_enter(mm);
> diff --git a/mm/vma_exec.c b/mm/vma_exec.c
> index 586c52155942..2c644e18b55d 100644
> --- a/mm/vma_exec.c
> +++ b/mm/vma_exec.c
> @@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
> goto err_free;
> }
>
> - /*
> - * Need to be called with mmap write lock
> - * held, to avoid race with ksmd.
> - */
> - err = ksm_execve(mm);
> - if (err)
> - goto err_ksm;
> -
> vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
> vma_set_anonymous(vma);
>
> @@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
> return 0;
>
> err:
> - ksm_exit(mm);
> -err_ksm:
> mmap_write_unlock(mm);
> err_free:
> *vmap = NULL;
> diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
> index f786a585caf0..9ab4e524f78e 100644
> --- a/tools/testing/vma/include/stubs.h
> +++ b/tools/testing/vma/include/stubs.h
> @@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
> {
> }
>
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> - return 0;
> -}
> -
> static inline void ksm_exit(struct mm_struct *mm)
> {
> }
> --
> 2.54.0
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-06 8:28 ` Xu Xin (vivo)
@ 2026-11-07 2:48 ` Jinjiang Tu
0 siblings, 0 replies; 7+ messages in thread
From: Jinjiang Tu @ 2026-11-07 2:48 UTC (permalink / raw)
To: Xu Xin (vivo)
Cc: akpm, david, chengming.zhou, liam, ljs, vbabka, jannh, pfalcato,
kees, ziy, gourry, anthony.yznaga, linux-mm, wangkefeng.wang,
sunnanyong
在 2026/11/6 16:28, Xu Xin (vivo) 写道:
> On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
Thanks for review.
> We can go into more detail. like:
> "The early registration ksm_execve() during the exec() is no longer required,
> since when the process subsequently carries out the actual memory-allocation
> operations, such as brk() or mmap(), ksm_vma_flags is invoked to re-check
> MMF_VM_MERGE_ANY and register the mm with KSM."
Will update it.
>
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
>>
>> Tested with ksm_functional_tests, all tests pass.
> If possible, can you provide a reproducible testcase in
> tools/testing/selftests/mm/ksm_functional_tests.c in V2 ?
ksm_functional_tests.c has tested KSM setting is inherited
after fork/exec.
Or do you mean reproduce the deadlock? It only reproduce
when insert_vm_struct() fails to allocate memory, it's
almost impossible to trigger in practice.
>
> Generally Looks good to me, thanks!
>
> Acked-by: Xu Xin (vivo) <xu.xin@linux.dev>
>
> Thanks!
>
>> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
>> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
>> ---
>> include/linux/ksm.h | 13 -------------
>> mm/ksm.c | 16 +++++++++-------
>> mm/vma_exec.c | 10 ----------
>> tools/testing/vma/include/stubs.h | 5 -----
>> 4 files changed, 9 insertions(+), 35 deletions(-)
>>
>> diff --git a/include/linux/ksm.h b/include/linux/ksm.h
>> index d39d0d5483a2..5e228935cede 100644
>> --- a/include/linux/ksm.h
>> +++ b/include/linux/ksm.h
>> @@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>> }
>> }
>>
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> - if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
>> - return __ksm_enter(mm);
>> -
>> - return 0;
>> -}
>> -
>> static inline void ksm_exit(struct mm_struct *mm)
>> {
>> if (mm_flags_test(MMF_VM_MERGEABLE, mm))
>> @@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>> {
>> }
>>
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> - return 0;
>> -}
>> -
>> static inline void ksm_exit(struct mm_struct *mm)
>> {
>> }
>> diff --git a/mm/ksm.c b/mm/ksm.c
>> index dbfb8f98b189..2e0f894208d8 100644
>> --- a/mm/ksm.c
>> +++ b/mm/ksm.c
>> @@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
>>
>> mm_slot_free(mm_slot_cache, mm_slot);
>> /*
>> - * Only clear MMF_VM_MERGEABLE. We must not clear
>> - * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
>> - * perhaps their mm_struct has just been added to ksm_mm_slot
>> - * list, and its process has not yet officially started running
>> - * or has not yet performed mmap/brk to allocate anonymous VMAS.
>> + * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
>> + * per-process setting inherited across exec, so it must
>> + * stay set until userspace disables it with
>> + * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
>> + * re-registers the mm through ksm_vma_flags().
>> */
>> mm_flags_clear(MMF_VM_MERGEABLE, mm);
>> mmap_read_unlock(mm);
>> @@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
>> vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
>> /*
>> * Generally, the flags here always include MMF_VM_MERGEABLE.
>> - * However, in rare cases, this flag may be cleared by ksmd who
>> - * scans a cycle without finding any mergeable vma.
>> + * However, a newly execed process has not been added to
>> + * ksm_mm_slot. The first mergeable VMA registers the mm
>> + * with ksmd here. ksmd can also clear it after a full scan
>> + * cycle without finding any mergeable VMA.
>> */
>> if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
>> __ksm_enter(mm);
>> diff --git a/mm/vma_exec.c b/mm/vma_exec.c
>> index 586c52155942..2c644e18b55d 100644
>> --- a/mm/vma_exec.c
>> +++ b/mm/vma_exec.c
>> @@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>> goto err_free;
>> }
>>
>> - /*
>> - * Need to be called with mmap write lock
>> - * held, to avoid race with ksmd.
>> - */
>> - err = ksm_execve(mm);
>> - if (err)
>> - goto err_ksm;
>> -
>> vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
>> vma_set_anonymous(vma);
>>
>> @@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>> return 0;
>>
>> err:
>> - ksm_exit(mm);
>> -err_ksm:
>> mmap_write_unlock(mm);
>> err_free:
>> *vmap = NULL;
>> diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
>> index f786a585caf0..9ab4e524f78e 100644
>> --- a/tools/testing/vma/include/stubs.h
>> +++ b/tools/testing/vma/include/stubs.h
>> @@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
>> {
>> }
>>
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> - return 0;
>> -}
>> -
>> static inline void ksm_exit(struct mm_struct *mm)
>> {
>> }
>> --
>> 2.54.0
>>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-06 1:39 [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path Jinjiang Tu
2026-11-06 8:28 ` Xu Xin (vivo)
@ 2026-11-06 8:46 ` Xu Xin (vivo)
2026-11-07 2:56 ` Jinjiang Tu
2026-11-07 19:06 ` David Hildenbrand (Arm)
2 siblings, 1 reply; 7+ messages in thread
From: Xu Xin (vivo) @ 2026-11-06 8:46 UTC (permalink / raw)
To: Jinjiang Tu, akpm, david
Cc: akpm, david, chengming.zhou, liam, ljs, vbabka, jannh, pfalcato,
kees, ziy, gourry, anthony.yznaga, linux-mm, wangkefeng.wang,
sunnanyong
On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
>
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
>
> Tested with ksm_functional_tests, all tests pass.
>
> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
Need to Cc stable@vger.kernel.org ?
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-06 8:46 ` Xu Xin (vivo)
@ 2026-11-07 2:56 ` Jinjiang Tu
0 siblings, 0 replies; 7+ messages in thread
From: Jinjiang Tu @ 2026-11-07 2:56 UTC (permalink / raw)
To: Xu Xin (vivo), akpm, david
Cc: chengming.zhou, liam, ljs, vbabka, jannh, pfalcato, kees, ziy,
gourry, anthony.yznaga, linux-mm, wangkefeng.wang, sunnanyong
在 2026/11/6 16:46, Xu Xin (vivo) 写道:
> On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
>>
>> Tested with ksm_functional_tests, all tests pass.
>>
>> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
>> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
> Need to Cc stable@vger.kernel.org ?
Indeed.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-06 1:39 [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path Jinjiang Tu
2026-11-06 8:28 ` Xu Xin (vivo)
2026-11-06 8:46 ` Xu Xin (vivo)
@ 2026-11-07 19:06 ` David Hildenbrand (Arm)
2026-11-09 1:53 ` Jinjiang Tu
2 siblings, 1 reply; 7+ messages in thread
From: David Hildenbrand (Arm) @ 2026-11-07 19:06 UTC (permalink / raw)
To: Jinjiang Tu, akpm, xu.xin, chengming.zhou, liam, ljs, vbabka,
jannh, pfalcato, kees, ziy, gourry, anthony.yznaga, linux-mm
Cc: wangkefeng.wang, sunnanyong
On 11/6/26 02:39, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
>
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
I assume you mean the
} else if (mm_slot) {
mmap_write_lock(mm);
mmap_write_unlock(mm);
}
Can that actually trigger?
IOW, can we have an slot, and the slot can already have a mm_slot->rmap_list?
--
Cheers,
David
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path
2026-11-07 19:06 ` David Hildenbrand (Arm)
@ 2026-11-09 1:53 ` Jinjiang Tu
0 siblings, 0 replies; 7+ messages in thread
From: Jinjiang Tu @ 2026-11-09 1:53 UTC (permalink / raw)
To: David Hildenbrand (Arm), akpm, xu.xin, chengming.zhou, liam, ljs,
vbabka, jannh, pfalcato, kees, ziy, gourry, anthony.yznaga,
linux-mm
Cc: wangkefeng.wang, sunnanyong
在 2026/11/8 3:06, David Hildenbrand (Arm) 写道:
> On 11/6/26 02:39, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
> I assume you mean the
>
> } else if (mm_slot) {
> mmap_write_lock(mm);
> mmap_write_unlock(mm);
> }
>
> Can that actually trigger?
>
> IOW, can we have an slot, and the slot can already have a mm_slot->rmap_list?
This doesn't need mm_slot->rmap_list != NULL.
In __ksm_exit(), if ksm_scan.mm_slot == mm_slot, will lock and unlock mmap rwsem.
mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
if (ksm_scan.mm_slot == mm_slot)
goto unlock;
...
unlock:
...
} else if (mm_slot) {
mmap_write_lock(mm);
mmap_write_unlock(mm);
}
`ksm_scan.mm_slot == mm_slot` means ksmd is currently scanning current's mm_slot.
After ksm_execve(), current's mm_slot has been added to ksm_mm_head list, and
ksmd could find the mm_slot in scan_get_next_rmap_item().
>
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2026-11-09 1:53 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-06 1:39 [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path Jinjiang Tu
2026-11-06 8:28 ` Xu Xin (vivo)
2026-11-07 2:48 ` Jinjiang Tu
2026-11-06 8:46 ` Xu Xin (vivo)
2026-11-07 2:56 ` Jinjiang Tu
2026-11-07 19:06 ` David Hildenbrand (Arm)
2026-11-09 1:53 ` Jinjiang Tu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox