Re: init's children list is long and slows reaping children.

Nick Piggin
2007-04-11T07:28:47Z

Jeff Garzik wrote:
> Linus Torvalds wrote:
>
>>
>> On Fri, 6 Apr 2007, Jeff Garzik wrote:
>>
>>> I would rather change the implementation under the hood to start per-CPU
>>> threads on demand, similar to a thread-pool implementation.
>>>
>>> Boxes with $BigNum CPUs probably won't ever use half of those threads.
>>
>>
>> The counter-argument is that boxes with $BigNum CPU's really don't
>> hurt from it either, and having per-process data structures is often
>> simpler and more efficient than trying to have some thread pool.
>
>
> Two points here:
>
> * A lot of the users in the current kernel tree don't rely on the
> per-CPU qualities. They just need multiple threads running.
>
> * Even with per-CPU data structures and code, you don't necessarily have
> to keep a thread alive and running for each CPU. Reap the ones that
> haven't been used in $TimeFrame, and add thread creation to the slow
> path that already exists in the bowels of schedule_work().
>
> Or if some kernel hacker is really motivated, all workqueue users in the
> kernel would benefit from a "thread audit", looking at working
> conditions to decide if the new kthread APIs are more appropriate.

spawn on demand would require heuristics and complexity though. And
I think there is barely any positive tradeoff to weigh it against.

>> IOW, once we get the processes off the global list, there just isn't
>> any downside from them. Sure, they use some memory, but people who buy
>> 1024-cpu machines won't care about a few kB per CPU..
>>
>> So the *only* downside is literally the process list, and one
>> suggested patch already just removes kernel threads entirely from the
>> parenthood lists.
>>
>> The other potential downside could be "ps is slow", but on the other
>> hand, having the things stick around and have things like CPU-time
>> accumulate is probably worth it - if there are some issues, they'd
>> show up properly accounted for in a way that process pools would have
>> a hard time doing.
>
>
> Regardless of how things are shuffled about internally, there will
> always be annoying overhead /somewhere/ when you have a metric ton of
> kernel threads. I think that people should also be working on ways to
> make the kernel threads a bit more manageable for the average human.

There are a few per CPU, but they should need no human management to
speak of.

Presumably if you have a 1024 CPU system, you'd generally want to be
running at least 1024 of your own processes there, so you already need
some tools to handle that magnitude of processes anyway.

>> So I really don't think this is worth changing things over, apart from
>> literally removing them from process lists, which I think everybody
>> agrees we should just do - it just never even came up before!
>
>
> I think there is a human downside. For an admin you have to wade
> through a ton of processes on your machine, if you are attempting to
> evaluate the overall state of the machine. Just google around for all
> the admins complaining about the explosion of kernel threads on
> production machines :)

User tools should be improved. It shouldn't be too hard to be able to
aggregate kernel thread stats into a single top entry, for example.

I'm not saying the number of threads couldn't be cut down, but there
is still be an order of magnitude problem there...

--
SUSE Labs, Novell Inc.