Re: [PATCH 4/4] nohz: Set isolcpus when nohz_full is set

[Mike Galbraith <umgwanakikbuti@gmail.com>]

On Thu, 2015-05-21 at 15:06 +0200, Frederic Weisbecker wrote:
> On Thu, May 21, 2015 at 05:57:59AM -0700, Paul E. McKenney wrote:
> > On Thu, May 21, 2015 at 05:42:46PM +0530, Afzal Mohammed wrote:
> > > Hi,
> > > 
> > > On Wed, May 20, 2015 at 02:00:26PM -0700, Paul E. McKenney wrote:
> > > 
> > > > > > Given that kernel initiated association to isolcpus, a user turning
> > > > > > NO_HZ_FULL_ALL on had better not have much generic load to manage.  If
> > > > > 
> > > > > On a quad-core desktop system with NO_HZ_FULL_ALL, hackbench took 3x
> > > > > time as compared to w/o this patch, except boot cpu every one else
> > > > > jobless. Though NO_HZ_FULL_ALL (afaik) is not meant for generic load,
> > > > > it was working fine, but not after this - it is now like a single core
> > > > > system.
> > > > 
> > > > I have to ask...  What is your use case?  What are you wanting NO_HZ_FULL
> > > > to do for you?
> > > 
> > > I was just playing NO_HZ_FULL with tip-[sched,timers]-* changes.
> > > 
> > > Thought that shutting down ticks as much as possible would be
> > > beneficial to normal loads too, though it has been mentioned to be used
> > > for specialized loads. Seems like drawbacks due to it weigh against
> > > normal loads, but haven't so far observed any (on a laptop with normal
> > > activities) before this change.
> > 
> > Indeed, NO_HZ_FULL is special purpose.  You normally would select
> > NO_HZ_FULL_ALL only on a system intended for heavy compute without
> > normal-workload distractions or for some real-time systems.  For mixed
> > workloads, you would build with NO_HZ_FULL (but not NO_HZ_FULL_ALL) and
> > use the boot parameters to select which CPUs are to be running the
> > specialized portion of the workload.
> > 
> > And you would of course need to lead enough CPUs running normally to
> > handle the non-specialized portion of the workload.
> > 
> > This sort of thing has traditionally required specialized kernels,
> > so the cool thing here is that we can make Linux do it.  Though, as
> > you noticed, careful configuration is still required.
> > 
> > Seem reasonable?
> 
> That said if he saw a big performance regression after applying these patches,
> then there is likely a problem in the patchset. Well it could be due to that mode
> which loops on full dynticks before resuming to userspace. Indeed when that is
> enabled, I expect real throughput issues on workloads doing lots of kernel <->
> userspace roundtrips. We just need to make sure this thing only works when requested.
> 
> Anyway, I need to look at the patchset.

I think it's a mistake to make any rash assumption and/or mandate that
the user WILL use nohz_full CPUs immediately, or even at all for that
matter.  nohz_full currently is nothing but a CPU attribute, period,
nothing more, nothing less.  The overhead that comes with the it is the
only thing that suggests that the set really really should be used
exclusively for isolated compute loads, and even then, they had better
be pure userspace due to the hefty border crossing overhead.

Let that overhear be seriously reduced, as per the Ingo/Andy discussion,
and presto, the things become a very interesting to dynamic sets.  You
can really really use them as you see fit, and on the fly, it doesn't
cost you an arm and a leg just to turn it on.  The only restriction
being the static set declaration, that you have to manage until that too
goes away.

I see no reason to turn nohz_full into a rigid construct.

	-Mike