[PATCH 0/1] kernel/module.c: don't allow modprobe to hang forever on a module load

[PATCH 0/1] kernel/module.c: don't allow modprobe to hang forever on a module load

[Konstantin Khorenko]

After commit 6e6de3dee51a ("kernel/module.c: Only return -EEXIST
for modules that have finished loading")
the simple test leads to hanged modprobe process (INTERRUPTIBLE):

1. Make sure nft_ct module is not used by your firewall rules.
2. Run 3 copies of
   # export i=0; while true; if [[ $(($i % 100)) -eq 0 ]] ; then \
     echo "i=$i"; fi; do modprobe nft_ct; i=$(($i + 1)); done

3. Run 2 copies of
   # while true; do rmmod nft_ct; done

Hanged "modprobe" process will appear in ~ 10 seconds.

   # cat /proc/30184/stack
   [<0>] load_module+0x53f/0x2060
   [<0>] __do_sys_finit_module+0xd2/0x100
   [<0>] do_syscall_64+0x5b/0x1c0
   [<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Konstantin Khorenko (1):
  kernel/module.c: wakeup processes in module_wq on module unload

 kernel/module.c | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.15.1

[PATCH 1/1] kernel/module.c: wakeup processes in module_wq on module unload

[Konstantin Khorenko]

Fix the race between load and unload a kernel module.

sys_delete_module()
 try_stop_module()
  mod->state = _GOING
					add_unformed_module()
					 old = find_module_all()
					 (old->state == _GOING =>
					  wait_event_interruptible())

					 During pre-condition
					 finished_loading() rets 0
					 schedule()
					 (never gets waken up later)
 free_module()
  mod->state = _UNFORMED
   list_del_rcu(&mod->list)
   (dels mod from "modules" list)

return

The race above leads to modprobe hanging forever on loading
a module.

Error paths on loading module call wake_up_all(&module_wq) after
freeing module, so let's do the same on straight module unload.

Fixes: 6e6de3dee51a ("kernel/module.c: Only return -EEXIST
for modules that have finished loading")

Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
---
 kernel/module.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/kernel/module.c b/kernel/module.c
index ff2d7359a418..cb09a5f37a5f 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -1033,6 +1033,8 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
 	strlcpy(last_unloaded_module, mod->name, sizeof(last_unloaded_module));
 
 	free_module(mod);
+	/* someone could wait for the module in add_unformed_module() */
+	wake_up_all(&module_wq);
 	return 0;
 out:
 	mutex_unlock(&module_mutex);
-- 
2.15.1

Re: [PATCH 1/1] kernel/module.c: wakeup processes in module_wq on module unload

[Prarit Bhargava]

On 11/13/19 4:29 AM, Konstantin Khorenko wrote:
> Fix the race between load and unload a kernel module.
> 
> sys_delete_module()
>  try_stop_module()
>   mod->state = _GOING
> 					add_unformed_module()
> 					 old = find_module_all()
> 					 (old->state == _GOING =>
> 					  wait_event_interruptible())
> 
> 					 During pre-condition
> 					 finished_loading() rets 0
> 					 schedule()
> 					 (never gets waken up later)
>  free_module()
>   mod->state = _UNFORMED
>    list_del_rcu(&mod->list)
>    (dels mod from "modules" list)
> 
> return
> 
> The race above leads to modprobe hanging forever on loading
> a module.
> 
> Error paths on loading module call wake_up_all(&module_wq) after
> freeing module, so let's do the same on straight module unload.
> 
> Fixes: 6e6de3dee51a ("kernel/module.c: Only return -EEXIST
> for modules that have finished loading")
> 
> Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>
> ---
>  kernel/module.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/kernel/module.c b/kernel/module.c
> index ff2d7359a418..cb09a5f37a5f 100644
> --- a/kernel/module.c
> +++ b/kernel/module.c
> @@ -1033,6 +1033,8 @@ SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
>  	strlcpy(last_unloaded_module, mod->name, sizeof(last_unloaded_module));
>  
>  	free_module(mod);
> +	/* someone could wait for the module in add_unformed_module() */
> +	wake_up_all(&module_wq);

free_module() acts on that *this* module so it makes sense to include the
wake_up_all(&module_wq) call in delete_module.  IIRC I did try a module load &
unload stress test but it looks like my testing wasn't hard enough :(

In any case,

Reviewed-by: Prarit Bhargava <prarit@redhat.com>

P.

>  	return 0;
>  out:
>  	mutex_unlock(&module_mutex);
> 

Re: [PATCH 1/1] kernel/module.c: wakeup processes in module_wq on module unload

[Jessica Yu]

+++ Konstantin Khorenko [13/11/19 12:29 +0300]:
>Fix the race between load and unload a kernel module.
>
>sys_delete_module()
> try_stop_module()
>  mod->state = _GOING
>					add_unformed_module()
>					 old = find_module_all()
>					 (old->state == _GOING =>
>					  wait_event_interruptible())
>
>					 During pre-condition
>					 finished_loading() rets 0
>					 schedule()
>					 (never gets waken up later)
> free_module()
>  mod->state = _UNFORMED
>   list_del_rcu(&mod->list)
>   (dels mod from "modules" list)
>
>return
>
>The race above leads to modprobe hanging forever on loading
>a module.
>
>Error paths on loading module call wake_up_all(&module_wq) after
>freeing module, so let's do the same on straight module unload.
>
>Fixes: 6e6de3dee51a ("kernel/module.c: Only return -EEXIST
>for modules that have finished loading")
>
>Signed-off-by: Konstantin Khorenko <khorenko@virtuozzo.com>

Thanks Konstantin for catching this. I've applied this to
modules-next.

Jessica