Re: [RFC 2/2] Make x86 calibrate_delay run in parallel.

[Robin Holt <holt@sgi.com>]

On Thu, Mar 31, 2011 at 08:58:05AM +0200, Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
> > 
> > * Yinghai Lu <yinghai@kernel.org> wrote:
> > 
> > > On Tue, Dec 14, 2010 at 5:58 PM,  <Robin@sgi.com> wrote:
> > > >
> > > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing
> > > > up the cpus.  By specifying lpj=<value>, we reduced that to 75 seconds.
> > > > Andi Kleen suggested we rework the calibrate_delay calls to run in
> > > > parallel.  With that code in place, a test boot of the same machine took
> > > > 61 seconds to bring the cups up.  I am not sure how we beat the lpj=
> > > > case, but it did outperform.
> > > >
> > > > One thing to note is the total BogoMIPS value is also consistently higher.
> > > > I am wondering if this is an effect with the cores being in performance
> > > > mode.  I did notice that the parallel calibrate_delay calls did cause the
> > > > fans on the machine to ramp up to full speed where the normal sequential
> > > > calls did not cause them to budge at all.
> > > 
> > > please check attached patch, that could calibrate correctly.
> > > 
> > > Thanks
> > > 
> > > Yinghai
> > 
> > > [PATCH -v2] x86: Make calibrate_delay run in parallel.
> > > 
> > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing
> > > up the cpus.  By specifying lpj=<value>, we reduced that to 75 seconds.
> > > Andi Kleen suggested we rework the calibrate_delay calls to run in
> > > parallel.
> > 
> > The risk wit that suggestion is that it will spectacularly miscalibrate on 
> > hyperthreading systems.

I am not trying to be argumentative.  I never got an understanding of
what was going wrong with that earlier patch and am hoping for some
understanding now.

Why does it spectacularly miscalibrate?  Can anything be done to correct
that miscalibration?  Doesn't this patch indicate another problem with
the calibration for hotplug cpus?  Isn't there already a problem if
you boot a cpu normally, then hot-remove a hyperthread of a cpu, run a
userland task which fully loads up all the cores on that socket, then
hot-add that hyperthread back in?  If the lpj value is that volatile,
what value does it really have?

Thank you for your insights,
Robin