[PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Jinjiang Tu]

create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
while holding mmap_write_lock(), and calls ksm_exit() before dropping the
lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
again when the mm is ksmd's current scan cursor, leading to deadlock.

The early registration is no longer needed after commit 590c03ca6a3f
("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
calls __ksm_enter() when the first mergeable VMA is created. Remove
ksm_execve() and its stubs.

Tested with ksm_functional_tests, all tests pass.

Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
---
 include/linux/ksm.h               | 13 -------------
 mm/ksm.c                          | 16 +++++++++-------
 mm/vma_exec.c                     | 10 ----------
 tools/testing/vma/include/stubs.h |  5 -----
 4 files changed, 9 insertions(+), 35 deletions(-)

diff --git a/include/linux/ksm.h b/include/linux/ksm.h
index d39d0d5483a2..5e228935cede 100644
--- a/include/linux/ksm.h
+++ b/include/linux/ksm.h
@@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
 	}
 }
 
-static inline int ksm_execve(struct mm_struct *mm)
-{
-	if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
-		return __ksm_enter(mm);
-
-	return 0;
-}
-
 static inline void ksm_exit(struct mm_struct *mm)
 {
 	if (mm_flags_test(MMF_VM_MERGEABLE, mm))
@@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
 {
 }
 
-static inline int ksm_execve(struct mm_struct *mm)
-{
-	return 0;
-}
-
 static inline void ksm_exit(struct mm_struct *mm)
 {
 }
diff --git a/mm/ksm.c b/mm/ksm.c
index dbfb8f98b189..2e0f894208d8 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
 
 		mm_slot_free(mm_slot_cache, mm_slot);
 		/*
-		 * Only clear MMF_VM_MERGEABLE. We must not clear
-		 * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
-		 * perhaps their mm_struct has just been added to ksm_mm_slot
-		 * list, and its process has not yet officially started running
-		 * or has not yet performed mmap/brk to allocate anonymous VMAS.
+		 * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
+		 * per-process setting inherited across exec, so it must
+		 * stay set until userspace disables it with
+		 * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
+		 * re-registers the mm through ksm_vma_flags().
 		 */
 		mm_flags_clear(MMF_VM_MERGEABLE, mm);
 		mmap_read_unlock(mm);
@@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
 		vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
 		/*
 		 * Generally, the flags here always include MMF_VM_MERGEABLE.
-		 * However, in rare cases, this flag may be cleared by ksmd who
-		 * scans a cycle without finding any mergeable vma.
+		 * However, a newly execed process has not been added to
+		 * ksm_mm_slot. The first mergeable VMA registers the mm
+		 * with ksmd here. ksmd can also clear it after a full scan
+		 * cycle without finding any mergeable VMA.
 		 */
 		if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
 			__ksm_enter(mm);
diff --git a/mm/vma_exec.c b/mm/vma_exec.c
index 586c52155942..2c644e18b55d 100644
--- a/mm/vma_exec.c
+++ b/mm/vma_exec.c
@@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
 		goto err_free;
 	}
 
-	/*
-	 * Need to be called with mmap write lock
-	 * held, to avoid race with ksmd.
-	 */
-	err = ksm_execve(mm);
-	if (err)
-		goto err_ksm;
-
 	vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
 	vma_set_anonymous(vma);
 
@@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
 	return 0;
 
 err:
-	ksm_exit(mm);
-err_ksm:
 	mmap_write_unlock(mm);
 err_free:
 	*vmap = NULL;
diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
index f786a585caf0..9ab4e524f78e 100644
--- a/tools/testing/vma/include/stubs.h
+++ b/tools/testing/vma/include/stubs.h
@@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
 {
 }
 
-static inline int ksm_execve(struct mm_struct *mm)
-{
-	return 0;
-}
-
 static inline void ksm_exit(struct mm_struct *mm)
 {
 }
-- 
2.54.0

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Xu Xin (vivo)]

On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
> 
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()

We can go into more detail. like: 
"The early registration ksm_execve() during the exec() is no longer required,
since when the process subsequently carries out the actual memory-allocation
operations, such as brk() or mmap(), ksm_vma_flags is invoked to re-check
MMF_VM_MERGE_ANY and register the mm with KSM."

> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
> 
> Tested with ksm_functional_tests, all tests pass.

If possible, can you provide a reproducible testcase in
tools/testing/selftests/mm/ksm_functional_tests.c in V2 ?

Generally Looks good to me, thanks!

Acked-by: Xu Xin (vivo) <xu.xin@linux.dev>

Thanks!

> 
> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
> ---
>  include/linux/ksm.h               | 13 -------------
>  mm/ksm.c                          | 16 +++++++++-------
>  mm/vma_exec.c                     | 10 ----------
>  tools/testing/vma/include/stubs.h |  5 -----
>  4 files changed, 9 insertions(+), 35 deletions(-)
> 
> diff --git a/include/linux/ksm.h b/include/linux/ksm.h
> index d39d0d5483a2..5e228935cede 100644
> --- a/include/linux/ksm.h
> +++ b/include/linux/ksm.h
> @@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>  	}
>  }
>  
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> -	if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
> -		return __ksm_enter(mm);
> -
> -	return 0;
> -}
> -
>  static inline void ksm_exit(struct mm_struct *mm)
>  {
>  	if (mm_flags_test(MMF_VM_MERGEABLE, mm))
> @@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>  {
>  }
>  
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> -	return 0;
> -}
> -
>  static inline void ksm_exit(struct mm_struct *mm)
>  {
>  }
> diff --git a/mm/ksm.c b/mm/ksm.c
> index dbfb8f98b189..2e0f894208d8 100644
> --- a/mm/ksm.c
> +++ b/mm/ksm.c
> @@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
>  
>  		mm_slot_free(mm_slot_cache, mm_slot);
>  		/*
> -		 * Only clear MMF_VM_MERGEABLE. We must not clear
> -		 * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
> -		 * perhaps their mm_struct has just been added to ksm_mm_slot
> -		 * list, and its process has not yet officially started running
> -		 * or has not yet performed mmap/brk to allocate anonymous VMAS.
> +		 * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
> +		 * per-process setting inherited across exec, so it must
> +		 * stay set until userspace disables it with
> +		 * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
> +		 * re-registers the mm through ksm_vma_flags().
>  		 */
>  		mm_flags_clear(MMF_VM_MERGEABLE, mm);
>  		mmap_read_unlock(mm);
> @@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
>  		vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
>  		/*
>  		 * Generally, the flags here always include MMF_VM_MERGEABLE.
> -		 * However, in rare cases, this flag may be cleared by ksmd who
> -		 * scans a cycle without finding any mergeable vma.
> +		 * However, a newly execed process has not been added to
> +		 * ksm_mm_slot. The first mergeable VMA registers the mm
> +		 * with ksmd here. ksmd can also clear it after a full scan
> +		 * cycle without finding any mergeable VMA.
>  		 */
>  		if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
>  			__ksm_enter(mm);
> diff --git a/mm/vma_exec.c b/mm/vma_exec.c
> index 586c52155942..2c644e18b55d 100644
> --- a/mm/vma_exec.c
> +++ b/mm/vma_exec.c
> @@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>  		goto err_free;
>  	}
>  
> -	/*
> -	 * Need to be called with mmap write lock
> -	 * held, to avoid race with ksmd.
> -	 */
> -	err = ksm_execve(mm);
> -	if (err)
> -		goto err_ksm;
> -
>  	vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
>  	vma_set_anonymous(vma);
>  
> @@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>  	return 0;
>  
>  err:
> -	ksm_exit(mm);
> -err_ksm:
>  	mmap_write_unlock(mm);
>  err_free:
>  	*vmap = NULL;
> diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
> index f786a585caf0..9ab4e524f78e 100644
> --- a/tools/testing/vma/include/stubs.h
> +++ b/tools/testing/vma/include/stubs.h
> @@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
>  {
>  }
>  
> -static inline int ksm_execve(struct mm_struct *mm)
> -{
> -	return 0;
> -}
> -
>  static inline void ksm_exit(struct mm_struct *mm)
>  {
>  }
> -- 
> 2.54.0
> 

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Xu Xin (vivo)]

On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
> 
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
> 
> Tested with ksm_functional_tests, all tests pass.
> 
> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>

Need to Cc stable@vger.kernel.org ？

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Jinjiang Tu]

在 2026/11/6 16:28, Xu Xin (vivo) 写道:
> On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()

Thanks for review.

> We can go into more detail. like:
> "The early registration ksm_execve() during the exec() is no longer required,
> since when the process subsequently carries out the actual memory-allocation
> operations, such as brk() or mmap(), ksm_vma_flags is invoked to re-check
> MMF_VM_MERGE_ANY and register the mm with KSM."

Will update it.

>
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
>>
>> Tested with ksm_functional_tests, all tests pass.
> If possible, can you provide a reproducible testcase in
> tools/testing/selftests/mm/ksm_functional_tests.c in V2 ?

ksm_functional_tests.c has tested KSM setting is inherited
after fork/exec.

Or do you mean reproduce the deadlock? It only reproduce
when insert_vm_struct() fails to allocate memory, it's
almost impossible to trigger in practice.

>
> Generally Looks good to me, thanks!
>
> Acked-by: Xu Xin (vivo) <xu.xin@linux.dev>
>
> Thanks!
>
>> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
>> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
>> ---
>>   include/linux/ksm.h               | 13 -------------
>>   mm/ksm.c                          | 16 +++++++++-------
>>   mm/vma_exec.c                     | 10 ----------
>>   tools/testing/vma/include/stubs.h |  5 -----
>>   4 files changed, 9 insertions(+), 35 deletions(-)
>>
>> diff --git a/include/linux/ksm.h b/include/linux/ksm.h
>> index d39d0d5483a2..5e228935cede 100644
>> --- a/include/linux/ksm.h
>> +++ b/include/linux/ksm.h
>> @@ -66,14 +66,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>>   	}
>>   }
>>   
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> -	if (mm_flags_test(MMF_VM_MERGE_ANY, mm))
>> -		return __ksm_enter(mm);
>> -
>> -	return 0;
>> -}
>> -
>>   static inline void ksm_exit(struct mm_struct *mm)
>>   {
>>   	if (mm_flags_test(MMF_VM_MERGEABLE, mm))
>> @@ -118,11 +110,6 @@ static inline void ksm_fork(struct mm_struct *mm, struct mm_struct *oldmm)
>>   {
>>   }
>>   
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> -	return 0;
>> -}
>> -
>>   static inline void ksm_exit(struct mm_struct *mm)
>>   {
>>   }
>> diff --git a/mm/ksm.c b/mm/ksm.c
>> index dbfb8f98b189..2e0f894208d8 100644
>> --- a/mm/ksm.c
>> +++ b/mm/ksm.c
>> @@ -2777,11 +2777,11 @@ static struct ksm_rmap_item *scan_get_next_rmap_item(struct page **page)
>>   
>>   		mm_slot_free(mm_slot_cache, mm_slot);
>>   		/*
>> -		 * Only clear MMF_VM_MERGEABLE. We must not clear
>> -		 * MMF_VM_MERGE_ANY, because for those MMF_VM_MERGE_ANY process,
>> -		 * perhaps their mm_struct has just been added to ksm_mm_slot
>> -		 * list, and its process has not yet officially started running
>> -		 * or has not yet performed mmap/brk to allocate anonymous VMAS.
>> +		 * Only clear MMF_VM_MERGEABLE. MMF_VM_MERGE_ANY is the
>> +		 * per-process setting inherited across exec, so it must
>> +		 * stay set until userspace disables it with
>> +		 * prctl(PR_SET_MEMORY_MERGE, 0). A later mergeable VMA
>> +		 * re-registers the mm through ksm_vma_flags().
>>   		 */
>>   		mm_flags_clear(MMF_VM_MERGEABLE, mm);
>>   		mmap_read_unlock(mm);
>> @@ -2909,8 +2909,10 @@ vma_flags_t ksm_vma_flags(struct mm_struct *mm, const struct file *file,
>>   		vma_flags_set(&vma_flags, VMA_MERGEABLE_BIT);
>>   		/*
>>   		 * Generally, the flags here always include MMF_VM_MERGEABLE.
>> -		 * However, in rare cases, this flag may be cleared by ksmd who
>> -		 * scans a cycle without finding any mergeable vma.
>> +		 * However, a newly execed process has not been added to
>> +		 * ksm_mm_slot. The first mergeable VMA registers the mm
>> +		 * with ksmd here. ksmd can also clear it after a full scan
>> +		 * cycle without finding any mergeable VMA.
>>   		 */
>>   		if (unlikely(!mm_flags_test(MMF_VM_MERGEABLE, mm)))
>>   			__ksm_enter(mm);
>> diff --git a/mm/vma_exec.c b/mm/vma_exec.c
>> index 586c52155942..2c644e18b55d 100644
>> --- a/mm/vma_exec.c
>> +++ b/mm/vma_exec.c
>> @@ -128,14 +128,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>>   		goto err_free;
>>   	}
>>   
>> -	/*
>> -	 * Need to be called with mmap write lock
>> -	 * held, to avoid race with ksmd.
>> -	 */
>> -	err = ksm_execve(mm);
>> -	if (err)
>> -		goto err_ksm;
>> -
>>   	vma_flags_set_mask(&flags, VMA_STACK_FLAGS);
>>   	vma_set_anonymous(vma);
>>   
>> @@ -163,8 +155,6 @@ int create_init_stack_vma(struct mm_struct *mm, struct vm_area_struct **vmap,
>>   	return 0;
>>   
>>   err:
>> -	ksm_exit(mm);
>> -err_ksm:
>>   	mmap_write_unlock(mm);
>>   err_free:
>>   	*vmap = NULL;
>> diff --git a/tools/testing/vma/include/stubs.h b/tools/testing/vma/include/stubs.h
>> index f786a585caf0..9ab4e524f78e 100644
>> --- a/tools/testing/vma/include/stubs.h
>> +++ b/tools/testing/vma/include/stubs.h
>> @@ -55,11 +55,6 @@ static inline void free_pgd_range(struct mmu_gather *tlb,
>>   {
>>   }
>>   
>> -static inline int ksm_execve(struct mm_struct *mm)
>> -{
>> -	return 0;
>> -}
>> -
>>   static inline void ksm_exit(struct mm_struct *mm)
>>   {
>>   }
>> -- 
>> 2.54.0
>>

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Jinjiang Tu]

在 2026/11/6 16:46, Xu Xin (vivo) 写道:
> On Fri, Nov 06, 2026 at 09:39:06AM +0800, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
>>
>> Tested with ksm_functional_tests, all tests pass.
>>
>> Fixes: 3a9e567ca45f ("mm/ksm: fix ksm exec support for prctl")
>> Signed-off-by: Jinjiang Tu <tujinjiang@huawei.com>
> Need to Cc stable@vger.kernel.org ？

Indeed.

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[David Hildenbrand (Arm)]

On 11/6/26 02:39, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
> 
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.

I assume you mean the

} else if (mm_slot) {
	mmap_write_lock(mm);
	mmap_write_unlock(mm);
}

Can that actually trigger?

IOW, can we have an slot, and the slot can already have a mm_slot->rmap_list?

-- 
Cheers,

David

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[Jinjiang Tu]

在 2026/11/8 3:06, David Hildenbrand (Arm) 写道:
> On 11/6/26 02:39, Jinjiang Tu wrote:
>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>
>> The early registration is no longer needed after commit 590c03ca6a3f
>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>> ksm_execve() and its stubs.
> I assume you mean the
>
> } else if (mm_slot) {
> 	mmap_write_lock(mm);
> 	mmap_write_unlock(mm);
> }
>
> Can that actually trigger?
>
> IOW, can we have an slot, and the slot can already have a mm_slot->rmap_list?

This doesn't need mm_slot->rmap_list != NULL.

In __ksm_exit(), if ksm_scan.mm_slot == mm_slot, will lock and unlock mmap rwsem.


  mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
  if (ksm_scan.mm_slot == mm_slot)
  	goto unlock;

  ...

unlock:
  ...

  } else if (mm_slot) {
	mmap_write_lock(mm);
	mmap_write_unlock(mm);
  }


`ksm_scan.mm_slot == mm_slot` means ksmd is currently scanning current's mm_slot.
After ksm_execve(), current's mm_slot has been added to ksm_mm_head list, and
ksmd could find the mm_slot in scan_get_next_rmap_item().

>

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[David Hildenbrand (Arm)]

On 11/9/26 02:53, Jinjiang Tu wrote:
> 
> 在 2026/11/8 3:06, David Hildenbrand (Arm) 写道:
>> On 11/6/26 02:39, Jinjiang Tu wrote:
>>> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
>>> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
>>> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
>>> again when the mm is ksmd's current scan cursor, leading to deadlock.
>>>
>>> The early registration is no longer needed after commit 590c03ca6a3f
>>> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
>>> calls __ksm_enter() when the first mergeable VMA is created. Remove
>>> ksm_execve() and its stubs.
>> I assume you mean the
>>
>> } else if (mm_slot) {
>>     mmap_write_lock(mm);
>>     mmap_write_unlock(mm);
>> }
>>
>> Can that actually trigger?
>>
>> IOW, can we have an slot, and the slot can already have a mm_slot->rmap_list?
> 
> This doesn't need mm_slot->rmap_list != NULL.
> 
> In __ksm_exit(), if ksm_scan.mm_slot == mm_slot, will lock and unlock mmap rwsem.
> 
> 
>  mm_slot = mm_slot_entry(slot, struct ksm_mm_slot, slot);
>  if (ksm_scan.mm_slot == mm_slot)
>      goto unlock;
> 
>  ...
> 
> unlock:
>  ...
> 
>  } else if (mm_slot) {
>     mmap_write_lock(mm);
>     mmap_write_unlock(mm);
>  }
> 
> 
> `ksm_scan.mm_slot == mm_slot` means ksmd is currently scanning current's mm_slot.
> After ksm_execve(), current's mm_slot has been added to ksm_mm_head list, and
> ksmd could find the mm_slot in scan_get_next_rmap_item().

Right, nothing stops that from happening. It's all rather unlikely to happen in
practice, but I agree with the fix (and it's an obvious improvement).

I'll pick this up an CC stable.

Reviewed-by: David Hildenbrand (Arm) <david@kernel.org>

-- 
Cheers,

David

Re: [PATCH] mm/ksm: fix ksm_exit() deadlock on exec error path

[David Hildenbrand]

On Fri, 06 Nov 2026 09:39:06 +0800, Jinjiang Tu wrote:
> create_init_stack_vma() registers the new mm with ksmd via ksm_execve()
> while holding mmap_write_lock(), and calls ksm_exit() before dropping the
> lock. When insert_vm_struct() fails, __ksm_exit() takes mmap_write_lock()
> again when the mm is ksmd's current scan cursor, leading to deadlock.
> 
> The early registration is no longer needed after commit 590c03ca6a3f
> ("mm/ksm: fix exec/fork inheritance support for prctl"). ksm_vma_flags()
> calls __ksm_enter() when the first mergeable VMA is created. Remove
> ksm_execve() and its stubs.
> 
> [...]

Applied, thanks!

tree: mm/core.git
branch: ksm-7.5.misc

Applied patches will appear soon in mm-next, followed by linux-next.

Please report any outstanding bugs that were missed during review, so we
can either drop the patch series or decide how to fix it up.

It is encouraged to provide Acked-bys, Reviewed-bys, and Tested-bys even
though the patches have already been applied. If possible, patch trailers
will be updated.

Note that commit hashes shown below are subject to change due to rebase,
trailer updates or similar. Further, this is not a guarantee that the
patches will go upstream. If in doubt, please check the listed branch.

[1/1] mm/ksm: fix ksm_exit() deadlock on exec error path
      https://git.kernel.org/mm/core/c/fbcdf9e8

-- 
Cheers,

David