Re: [PATCH] improve stop_machine performance

Dimitri Sivanich
2010-03-05T14:11:50Z

On Fri, Mar 05, 2010 at 09:22:00AM +0900, Tejun Heo wrote:
> Hello,
>
> On 03/05/2010 06:20 AM, Dimitri Sivanich wrote:
> > On systems with large cpu counts, we've been seeing long bootup times
> > associated with stop_machine operations. I've noticed that by simply
> > removing the creation of the workqueue and associated percpu variables
> > in subsequent stop_machine calls, we can reduce boot times on a
> > 1024 processor SGI UV system from 25-30 (or more) minutes down to 12
> > minutes.
> >
> > The attached patch does this in a simple way by removing the
> > stop_machine_destroy interface, thereby by leaving the workqueues and
> > percpu variables for later use once they are created.
> >
> > If people are against having these areas around after boot, maybe there
> > are some alternatives that will still allow for this optimization:
> >
> > - Set a timer to go off after a configurable number of minutes, at
> > which point the workqueue areas will be deleted.
> >
> > - Keep the stop_machine_destroy function, but somehow run it at the tail
> > end of boot (after modules have loaded), rather than running it at
> > every stop_machine call.
>
> Yeah, I can indeed imagine that creating and destroying all those
> workers on every module load during boot would be very costly if there
> are lots of CPUs. How about sharing the migration thread so that it
> serves as one-per-cpu uninterruptible RT simple thread pool? It's not
> like these things can run taking their turns anyway. I'll go ahead
> and make something up.
>

It seems reasonable as long as setup is fast enough. Will that thread indeed become fully uninterruptible (not affected by anything including scheduler decisions like sched_rt_period_us/sched_rt_runtime_us, etc..)?