[PATCH v2] kernel/fork: Increase minimum number of allowed threads

[PATCH v2] kernel/fork: Increase minimum number of allowed threads

[Hauke Mehrtens]

A modern Linux system creates much more than 20 threads at bootup.
When I booted up OpenWrt in qemu the system sometimes failed to boot up
when it wanted to create the 419th thread. The VM had 128MB RAM and the
calculation in set_max_threads() calculated that max_threads should be
set to 419. When the system booted up it tried to notify the user space
about every device it created because CONFIG_UEVENT_HELPER was set and
used. I counted 1299 calls to call_usermodehelper_setup(), all of
them try to create a new thread and call the userspace hotplug script in
it.

This fixes bootup of Linux on systems with low memory.

I saw the problem with qemu 10.0.2 using these commands:
qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic

Cc: stable@vger.kernel.org
Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
---
 kernel/fork.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/fork.c b/kernel/fork.c
index 7966c9a1c163..388299525f3c 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -115,7 +115,7 @@
 /*
  * Minimum number of threads to boot the kernel
  */
-#define MIN_THREADS 20
+#define MIN_THREADS 600
 
 /*
  * Maximum number of threads
-- 
2.50.1

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[Greg KH]

On Sat, Jul 12, 2025 at 01:08:29AM +0200, Hauke Mehrtens wrote:
> A modern Linux system creates much more than 20 threads at bootup.
> When I booted up OpenWrt in qemu the system sometimes failed to boot up
> when it wanted to create the 419th thread. The VM had 128MB RAM and the
> calculation in set_max_threads() calculated that max_threads should be
> set to 419. When the system booted up it tried to notify the user space
> about every device it created because CONFIG_UEVENT_HELPER was set and
> used. I counted 1299 calls to call_usermodehelper_setup(), all of
> them try to create a new thread and call the userspace hotplug script in
> it.
> 
> This fixes bootup of Linux on systems with low memory.
> 
> I saw the problem with qemu 10.0.2 using these commands:
> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
> ---
>  kernel/fork.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 7966c9a1c163..388299525f3c 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -115,7 +115,7 @@
>  /*
>   * Minimum number of threads to boot the kernel
>   */
> -#define MIN_THREADS 20
> +#define MIN_THREADS 600
>  
>  /*
>   * Maximum number of threads
> -- 
> 2.50.1
> 
> 

Hi,

This is the friendly patch-bot of Greg Kroah-Hartman.  You have sent him
a patch that has triggered this response.  He used to manually respond
to these common problems, but in order to save his sanity (he kept
writing the same thing over and over, yet to different people), I was
created.  Hopefully you will not take offence and will fix the problem
in your patch and resubmit it so that it can be accepted into the Linux
kernel tree.

You are receiving this message because of the following common error(s)
as indicated below:

- This looks like a new version of a previously submitted patch, but you
  did not list below the --- line any changes from the previous version.
  Please read the section entitled "The canonical patch format" in the
  kernel file, Documentation/process/submitting-patches.rst for what
  needs to be done here to properly describe this.

If you wish to discuss this problem further, or you have questions about
how to resolve this issue, please feel free to respond to this email and
Greg will reply once he has dug out from the pending patches received
from other developers.

thanks,

greg k-h's patch email bot

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[Jiri Slaby]

Cc wqueue & umode helper folks

On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
> A modern Linux system creates much more than 20 threads at bootup.
> When I booted up OpenWrt in qemu the system sometimes failed to boot up
> when it wanted to create the 419th thread. The VM had 128MB RAM and the
> calculation in set_max_threads() calculated that max_threads should be
> set to 419. When the system booted up it tried to notify the user space
> about every device it created because CONFIG_UEVENT_HELPER was set and
> used. I counted 1299 calls to call_usermodehelper_setup(), all of
> them try to create a new thread and call the userspace hotplug script in
> it.
> 
> This fixes bootup of Linux on systems with low memory.
> 
> I saw the problem with qemu 10.0.2 using these commands:
> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
> 
> Cc: stable@vger.kernel.org
> Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
> ---
>   kernel/fork.c | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/kernel/fork.c b/kernel/fork.c
> index 7966c9a1c163..388299525f3c 100644
> --- a/kernel/fork.c
> +++ b/kernel/fork.c
> @@ -115,7 +115,7 @@
>   /*
>    * Minimum number of threads to boot the kernel
>    */
> -#define MIN_THREADS 20
> +#define MIN_THREADS 600

As David noted, this is not the proper fix. It appears that usermode 
helper should use limited thread pool. I.e. instead of 
system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with 
max_active set to max_threads divided by some arbitrary constant (3? 4?)?

regards,
-- 
js
suse labs

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[David Laight]

On Thu, 17 Jul 2025 07:26:59 +0200
Jiri Slaby <jirislaby@kernel.org> wrote:

> Cc wqueue & umode helper folks
> 
> On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
> > A modern Linux system creates much more than 20 threads at bootup.
> > When I booted up OpenWrt in qemu the system sometimes failed to boot up
> > when it wanted to create the 419th thread. The VM had 128MB RAM and the
> > calculation in set_max_threads() calculated that max_threads should be
> > set to 419. When the system booted up it tried to notify the user space
> > about every device it created because CONFIG_UEVENT_HELPER was set and
> > used. I counted 1299 calls to call_usermodehelper_setup(), all of
> > them try to create a new thread and call the userspace hotplug script in
> > it.
> > 
> > This fixes bootup of Linux on systems with low memory.
> > 
> > I saw the problem with qemu 10.0.2 using these commands:
> > qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
> > 
> > Cc: stable@vger.kernel.org
> > Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
> > ---
> >   kernel/fork.c | 2 +-
> >   1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/kernel/fork.c b/kernel/fork.c
> > index 7966c9a1c163..388299525f3c 100644
> > --- a/kernel/fork.c
> > +++ b/kernel/fork.c
> > @@ -115,7 +115,7 @@
> >   /*
> >    * Minimum number of threads to boot the kernel
> >    */
> > -#define MIN_THREADS 20
> > +#define MIN_THREADS 600  
> 
> As David noted, this is not the proper fix. It appears that usermode 
> helper should use limited thread pool. I.e. instead of 
> system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with 
> max_active set to max_threads divided by some arbitrary constant (3? 4?)?

Or maybe just 1 ?
I'd guess all the threads either block in the same place or just block
each other??

	David

> 
> regards,

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[Hauke Mehrtens]

On 7/17/25 23:34, David Laight wrote:
> On Thu, 17 Jul 2025 07:26:59 +0200
> Jiri Slaby <jirislaby@kernel.org> wrote:
> 
>> Cc wqueue & umode helper folks
>>
>> On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
>>> A modern Linux system creates much more than 20 threads at bootup.
>>> When I booted up OpenWrt in qemu the system sometimes failed to boot up
>>> when it wanted to create the 419th thread. The VM had 128MB RAM and the
>>> calculation in set_max_threads() calculated that max_threads should be
>>> set to 419. When the system booted up it tried to notify the user space
>>> about every device it created because CONFIG_UEVENT_HELPER was set and
>>> used. I counted 1299 calls to call_usermodehelper_setup(), all of
>>> them try to create a new thread and call the userspace hotplug script in
>>> it.
>>>
>>> This fixes bootup of Linux on systems with low memory.
>>>
>>> I saw the problem with qemu 10.0.2 using these commands:
>>> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
>>>
>>> Cc: stable@vger.kernel.org
>>> Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
>>> ---
>>>    kernel/fork.c | 2 +-
>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/kernel/fork.c b/kernel/fork.c
>>> index 7966c9a1c163..388299525f3c 100644
>>> --- a/kernel/fork.c
>>> +++ b/kernel/fork.c
>>> @@ -115,7 +115,7 @@
>>>    /*
>>>     * Minimum number of threads to boot the kernel
>>>     */
>>> -#define MIN_THREADS 20
>>> +#define MIN_THREADS 600
>>
>> As David noted, this is not the proper fix. It appears that usermode
>> helper should use limited thread pool. I.e. instead of
>> system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with
>> max_active set to max_threads divided by some arbitrary constant (3? 4?)?
> 
> Or maybe just 1 ?
> I'd guess all the threads either block in the same place or just block
> each other??

I will reduce the number of threads. Maybe to max 5 or maybe just one.

I think we should still increase the minimum number of threads, but 
something like 60 to 100 should be fine. It is calculated based RAM size 
128MB RAM resulted already in 419 max threads.

Hauke

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[Hauke Mehrtens]

Hi,

I am not exactly sure how I should limit the number of parallel user 
mode helper calls.
The user mode helper is calling wait_for_initramfs() so it could be that 
some calls are getting queued at the early bootup. This is probably the 
problem I am hitting.

I do not want to block the device creation till the user mode helper 
finished. This could also result in a deadlock and would probably slow 
down bootup.

When I limit the number of user mode helper calls to 1 and let the 
others wait in a system queue, I might block other unrelated tasks in 
the system queue.

I would create an own queue and let the async user mode helper wait in 
this queue to only execute one at a time. When one of them needs a long 
time in user space it would block the others. This workqueue would also 
be active all the time. After the bootup it would probably not do much 
work any more.

I do not like any of these solutions. Do you have better ideas?

Hauke

On 7/18/25 00:52, Hauke Mehrtens wrote:
> On 7/17/25 23:34, David Laight wrote:
>> On Thu, 17 Jul 2025 07:26:59 +0200
>> Jiri Slaby <jirislaby@kernel.org> wrote:
>>
>>> Cc wqueue & umode helper folks
>>>
>>> On 12. 07. 25, 1:08, Hauke Mehrtens wrote:
>>>> A modern Linux system creates much more than 20 threads at bootup.
>>>> When I booted up OpenWrt in qemu the system sometimes failed to boot up
>>>> when it wanted to create the 419th thread. The VM had 128MB RAM and the
>>>> calculation in set_max_threads() calculated that max_threads should be
>>>> set to 419. When the system booted up it tried to notify the user space
>>>> about every device it created because CONFIG_UEVENT_HELPER was set and
>>>> used. I counted 1299 calls to call_usermodehelper_setup(), all of
>>>> them try to create a new thread and call the userspace hotplug 
>>>> script in
>>>> it.
>>>>
>>>> This fixes bootup of Linux on systems with low memory.
>>>>
>>>> I saw the problem with qemu 10.0.2 using these commands:
>>>> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
>>>>
>>>> Cc: stable@vger.kernel.org
>>>> Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
>>>> ---
>>>>    kernel/fork.c | 2 +-
>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/kernel/fork.c b/kernel/fork.c
>>>> index 7966c9a1c163..388299525f3c 100644
>>>> --- a/kernel/fork.c
>>>> +++ b/kernel/fork.c
>>>> @@ -115,7 +115,7 @@
>>>>    /*
>>>>     * Minimum number of threads to boot the kernel
>>>>     */
>>>> -#define MIN_THREADS 20
>>>> +#define MIN_THREADS 600
>>>
>>> As David noted, this is not the proper fix. It appears that usermode
>>> helper should use limited thread pool. I.e. instead of
>>> system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with
>>> max_active set to max_threads divided by some arbitrary constant (3? 
>>> 4?)?
>>
>> Or maybe just 1 ?
>> I'd guess all the threads either block in the same place or just block
>> each other??
> 
> I will reduce the number of threads. Maybe to max 5 or maybe just one.
> 
> I think we should still increase the minimum number of threads, but 
> something like 60 to 100 should be fine. It is calculated based RAM size 
> 128MB RAM resulted already in 419 max threads.
> 
> Hauke

Re: [PATCH v2] kernel/fork: Increase minimum number of allowed threads

[David Laight]

On Sun, 20 Jul 2025 17:28:20 +0200
Hauke Mehrtens <hauke@hauke-m.de> wrote:

> Hi,
> 
> I am not exactly sure how I should limit the number of parallel user 
> mode helper calls.
> The user mode helper is calling wait_for_initramfs() so it could be that 
> some calls are getting queued at the early bootup. This is probably the 
> problem I am hitting.
> 
> I do not want to block the device creation till the user mode helper 
> finished. This could also result in a deadlock and would probably slow 
> down bootup.
> 
> When I limit the number of user mode helper calls to 1 and let the 
> others wait in a system queue, I might block other unrelated tasks in 
> the system queue.
> 
> I would create an own queue and let the async user mode helper wait in 
> this queue to only execute one at a time. When one of them needs a long 
> time in user space it would block the others. This workqueue would also 
> be active all the time. After the bootup it would probably not do much 
> work any more.
> 
> I do not like any of these solutions. Do you have better ideas?

Could you put the requests onto a private queue but use a system work queue
function the clear the queue - only starting the function if there isn't
a copy running?

	David

> 
> Hauke
> 
> On 7/18/25 00:52, Hauke Mehrtens wrote:
> > On 7/17/25 23:34, David Laight wrote:  
> >> On Thu, 17 Jul 2025 07:26:59 +0200
> >> Jiri Slaby <jirislaby@kernel.org> wrote:
> >>  
> >>> Cc wqueue & umode helper folks
> >>>
> >>> On 12. 07. 25, 1:08, Hauke Mehrtens wrote:  
> >>>> A modern Linux system creates much more than 20 threads at bootup.
> >>>> When I booted up OpenWrt in qemu the system sometimes failed to boot up
> >>>> when it wanted to create the 419th thread. The VM had 128MB RAM and the
> >>>> calculation in set_max_threads() calculated that max_threads should be
> >>>> set to 419. When the system booted up it tried to notify the user space
> >>>> about every device it created because CONFIG_UEVENT_HELPER was set and
> >>>> used. I counted 1299 calls to call_usermodehelper_setup(), all of
> >>>> them try to create a new thread and call the userspace hotplug 
> >>>> script in
> >>>> it.
> >>>>
> >>>> This fixes bootup of Linux on systems with low memory.
> >>>>
> >>>> I saw the problem with qemu 10.0.2 using these commands:
> >>>> qemu-system-aarch64 -machine virt -cpu cortex-a57 -nographic
> >>>>
> >>>> Cc: stable@vger.kernel.org
> >>>> Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
> >>>> ---
> >>>>    kernel/fork.c | 2 +-
> >>>>    1 file changed, 1 insertion(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/kernel/fork.c b/kernel/fork.c
> >>>> index 7966c9a1c163..388299525f3c 100644
> >>>> --- a/kernel/fork.c
> >>>> +++ b/kernel/fork.c
> >>>> @@ -115,7 +115,7 @@
> >>>>    /*
> >>>>     * Minimum number of threads to boot the kernel
> >>>>     */
> >>>> -#define MIN_THREADS 20
> >>>> +#define MIN_THREADS 600  
> >>>
> >>> As David noted, this is not the proper fix. It appears that usermode
> >>> helper should use limited thread pool. I.e. instead of
> >>> system_unbound_wq, alloc_workqueue("", WQ_UNBOUND, max_active) with
> >>> max_active set to max_threads divided by some arbitrary constant (3? 
> >>> 4?)?  
> >>
> >> Or maybe just 1 ?
> >> I'd guess all the threads either block in the same place or just block
> >> each other??  
> > 
> > I will reduce the number of threads. Maybe to max 5 or maybe just one.
> > 
> > I think we should still increase the minimum number of threads, but 
> > something like 60 to 100 should be fine. It is calculated based RAM size 
> > 128MB RAM resulted already in 419 max threads.
> > 
> > Hauke  
>