Re: [RFC 2/2] Make x86 calibrate_delay run in parallel.

[Ingo Molnar <mingo@elte.hu>]

* Robin Holt <holt@sgi.com> wrote:

> On Thu, Mar 31, 2011 at 08:58:05AM +0200, Ingo Molnar wrote:
> > 
> > * Ingo Molnar <mingo@elte.hu> wrote:
> > 
> > > 
> > > * Yinghai Lu <yinghai@kernel.org> wrote:
> > > 
> > > > On Tue, Dec 14, 2010 at 5:58 PM,  <Robin@sgi.com> wrote:
> > > > >
> > > > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing
> > > > > up the cpus.  By specifying lpj=<value>, we reduced that to 75 seconds.
> > > > > Andi Kleen suggested we rework the calibrate_delay calls to run in
> > > > > parallel.  With that code in place, a test boot of the same machine took
> > > > > 61 seconds to bring the cups up.  I am not sure how we beat the lpj=
> > > > > case, but it did outperform.
> > > > >
> > > > > One thing to note is the total BogoMIPS value is also consistently higher.
> > > > > I am wondering if this is an effect with the cores being in performance
> > > > > mode.  I did notice that the parallel calibrate_delay calls did cause the
> > > > > fans on the machine to ramp up to full speed where the normal sequential
> > > > > calls did not cause them to budge at all.
> > > > 
> > > > please check attached patch, that could calibrate correctly.
> > > > 
> > > > Thanks
> > > > 
> > > > Yinghai
> > > 
> > > > [PATCH -v2] x86: Make calibrate_delay run in parallel.
> > > > 
> > > > On a 4096 cpu machine, we noticed that 318 seconds were taken for bringing
> > > > up the cpus.  By specifying lpj=<value>, we reduced that to 75 seconds.
> > > > Andi Kleen suggested we rework the calibrate_delay calls to run in
> > > > parallel.
> > > 
> > > The risk wit that suggestion is that it will spectacularly miscalibrate on 
> > > hyperthreading systems.
> 
> I am not trying to be argumentative.  I never got an understanding of
> what was going wrong with that earlier patch and am hoping for some
> understanding now.

Well, if calibrate_delay() calls run in parallel then different hyperthreads 
will impact each other.

> Why does it spectacularly miscalibrate?  Can anything be done to correct
> that miscalibration?  Doesn't this patch indicate another problem with
> the calibration for hotplug cpus?  Isn't there already a problem if
> you boot a cpu normally, then hot-remove a hyperthread of a cpu, run a
> userland task which fully loads up all the cores on that socket, then
> hot-add that hyperthread back in?  If the lpj value is that volatile,
> what value does it really have?

The typical CPU hotplug usecase is suspend/resume, where we bring down the CPUs 
in a more or less controlled manner.

Yes, you could achieve something similar by frobbing /sys/*/*/online but that's 
a big difference to *always* running the calibration loops in parallel.

I'd argue for the opposite direction: only calibrate a physical CPU once per 
CPU per bootup - this would also make CPU hotplug faster btw.

( Virtual CPUs (KVM, etc.) need a recalibration per bringup, because the new 
  CPU could be running on different hardware - but that's a detail: 4096 UV
  CPUs are not in this category. )

Really, there's no good reason why every CPU should be calibrated on a system 
running identical CPUs, right? Mixed-frequency systems are rather elusive on 
x86.

Thanks,

	Ingo