Re: [PATCH] sched, fair: Allow a small degree of load imbalance between SD_NUMA domains v2

Mel Gorman
2020-01-08T18:03:24Z

On Wed, Jan 08, 2020 at 05:46:57PM +0100, Vincent Guittot wrote:
> > Allowing just 1 extra task would work for netperf in some cases except when
> > softirq is involved. It would partially work for IO on ext4 as it's only
> > communicating with one journal thread but a bit more borderline for XFS
> > due to workqueue usage. XFS is not a massive concern in this context as
> > the workqueue is close to the IO issuer and short-lived so I don't think
> > it would crop up much for load balancing unlike ext4 where jbd2 could be
> > very active.
> >
> > If v4 of the patch fails to meet approval then I'll try a patch that
>
> My main concern with v4 was the mismatch between the computed value and the goal to not overload the LLCs
>

Fair enough.

> > allows a hard-coded imbalance of 2 tasks (one communicating task and
>
> If there is no good way to compute the allowed imbalance, a hard coded
> value of 2 is probably simple value to start with

Indeed.

>
> > one kthread) regardless of NUMA domain span up to 50% of utilisation
>
> Are you sure that it's necessary ? This degree of imbalance already applies only if the group has spare capacity
>
> something like
>
> + /* Consider allowing a small imbalance between NUMA groups */
> + if (env->sd->flags & SD_NUMA) {
> +
> + /*
> + * Until we found a good way to compute an acceptable
> + * degree of imbalance linked to the system topology
> + * and that will not impatc mem bandwith and latency,
> + * let start with a fixed small value.
> + */
> + imbalance_adj = 2;
> +
> + /*
> + * Ignore small imbalances when the busiest group has
> + * low utilisation.
> + */
> + env->imbalance -= min(env->imbalance, imbalance_adj);
> + }
>

This is more or less what I had in mind with the exception that the "low
utilisation" part of the comment would go away. The 50% utilisation may
be unnecessary and was based simply on the idea that at that point memory
bandwidth, HT considerations or both would also be dominating factors. I
can leave out the check and add it in as a separate patch if proven to
be necessary.

--
Mel Gorman
SUSE Labs