Fixes for 4.19 stable

Fixes for 4.19 stable

[Vishnu Rangayyan]

Hi,

I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101
(had all 4 backports applied earlier to our tree):

commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list
commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion

perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load:
  58.88%  ksoftirqd/0  [kernel.kallsyms]  [k] update_blocked_averages

Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.

commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream
commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream

The second one doesn’t apply cleanly, there’s a small change for the last diff, where pelt needs renaming to task (cfg_rq_clock_pelt was cfs_rq_clock_task in 4.19.y) in that unchanged part of the diff:

@@ -7700,10 +7720,6 @@ static void update_blocked_averages(int cpu)
 	for_each_leaf_cfs_rq(rq, cfs_rq) {
 		struct sched_entity *se;
 
-		/* throttled entities do not contribute to load */
-		if (throttled_hierarchy(cfs_rq))
-			continue;
-
 		if (update_cfs_rq_load_avg(cfs_rq_clock_pelt(cfs_rq), cfs_rq))
 			update_tg_load_avg(cfs_rq, 0);


I can post that patch with that rename if required.

Thanks 
Vishnu

Re: Fixes for 4.19 stable

[Sasha Levin]

On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
>Hi,
>
>I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after these 2 back ports went in for 4.19.101
>(had all 4 backports applied earlier to our tree):
>
>commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix insertion in rq->leaf_cfs_rq_list
>commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add tmp_alone_branch assertion
>
>perf shows for any given ksoftirqd, with 20k-30k processes on the system with high scheduler load:
>  58.88%  ksoftirqd/0  [kernel.kallsyms]  [k] update_blocked_averages
>
>Can we backport these 2 also, confirmed that it fixes this behavior of ksoftirqd.
>
>commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream
>commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream

Do we also need bef69dd87828 ("sched/cpufreq: Move the
cfs_rq_util_change() call to cpufreq_update_util()") then?

-- 
Thanks,
Sasha

Re: Fixes for 4.19 stable

[Vishnu Rangayyan]

Hi Sasha,

Not sure of this, looks relevant but I'm no expert on this code.
This particular change bef69dd87828 doesn't apply cleanly, need to 
backport it. I'll do that now and retest on the failing setup and report 
back.

Best
Vishnu

On 2/29/20 7:11 PM, Sasha Levin wrote:
> On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
>> Hi,
>>
>> I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after 
>> these 2 back ports went in for 4.19.101
>> (had all 4 backports applied earlier to our tree):
>>
>> commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix 
>> insertion in rq->leaf_cfs_rq_list
>> commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add 
>> tmp_alone_branch assertion
>>
>> perf shows for any given ksoftirqd, with 20k-30k processes on the 
>> system with high scheduler load:
>>  58.88%  ksoftirqd/0  [kernel.kallsyms]  [k] update_blocked_averages
>>
>> Can we backport these 2 also, confirmed that it fixes this behavior of 
>> ksoftirqd.
>>
>> commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream
>> commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
> 
> Do we also need bef69dd87828 ("sched/cpufreq: Move the
> cfs_rq_util_change() call to cpufreq_update_util()") then?
> 

Re: Fixes for 4.19 stable

[Vincent Guittot]

On Tue, 3 Mar 2020 at 00:13, Vishnu Rangayyan
<vishnu.rangayyan@apple.com> wrote:
>
> Hi Sasha,
>
> Not sure of this, looks relevant but I'm no expert on this code.
> This particular change bef69dd87828 doesn't apply cleanly, need to
> backport it. I'll do that now and retest on the failing setup and report
> back.
>
> Best
> Vishnu
>
> On 2/29/20 7:11 PM, Sasha Levin wrote:
> > On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
> >> Hi,
> >>
> >> I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after
> >> these 2 back ports went in for 4.19.101
> >> (had all 4 backports applied earlier to our tree):
> >>
> >> commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix
> >> insertion in rq->leaf_cfs_rq_list
> >> commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add
> >> tmp_alone_branch assertion
> >>
> >> perf shows for any given ksoftirqd, with 20k-30k processes on the
> >> system with high scheduler load:
> >>  58.88%  ksoftirqd/0  [kernel.kallsyms]  [k] update_blocked_averages
> >>
> >> Can we backport these 2 also, confirmed that it fixes this behavior of
> >> ksoftirqd.
> >>
> >> commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream
> >> commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
> >
> > Do we also need bef69dd87828 ("sched/cpufreq: Move the
> > cfs_rq_util_change() call to cpufreq_update_util()") then?

This patch is not related to the 2 patches above. It removes some
spurious call to cfs_rq_util_change() and some wrong ordering but this
is a fixed overhead which is not impacted by the number of cgroups
unlike the 2 patches above. This patch will not help with the high
load of update_blocked_averages

> >

Re: Fixes for 4.19 stable

[gregkh]

On Tue, Mar 03, 2020 at 09:03:15AM +0100, Vincent Guittot wrote:
> On Tue, 3 Mar 2020 at 00:13, Vishnu Rangayyan
> <vishnu.rangayyan@apple.com> wrote:
> >
> > Hi Sasha,
> >
> > Not sure of this, looks relevant but I'm no expert on this code.
> > This particular change bef69dd87828 doesn't apply cleanly, need to
> > backport it. I'll do that now and retest on the failing setup and report
> > back.
> >
> > Best
> > Vishnu
> >
> > On 2/29/20 7:11 PM, Sasha Levin wrote:
> > > On Fri, Feb 28, 2020 at 12:13:54PM -0800, Vishnu Rangayyan wrote:
> > >> Hi,
> > >>
> > >> I still see high (upto 30%) ksoftirqd cpu use with 4.19.101+ after
> > >> these 2 back ports went in for 4.19.101
> > >> (had all 4 backports applied earlier to our tree):
> > >>
> > >> commit f6783319737f28e4436a69611853a5a098cbe974 sched/fair: Fix
> > >> insertion in rq->leaf_cfs_rq_list
> > >> commit 5d299eabea5a251fbf66e8277704b874bbba92dc sched/fair: Add
> > >> tmp_alone_branch assertion
> > >>
> > >> perf shows for any given ksoftirqd, with 20k-30k processes on the
> > >> system with high scheduler load:
> > >>  58.88%  ksoftirqd/0  [kernel.kallsyms]  [k] update_blocked_averages
> > >>
> > >> Can we backport these 2 also, confirmed that it fixes this behavior of
> > >> ksoftirqd.
> > >>
> > >> commit 039ae8bcf7a5f4476f4487e6bf816885fb3fb617 upstream
> > >> commit 31bc6aeaab1d1de8959b67edbed5c7a4b3cdbe7c upstream
> > >
> > > Do we also need bef69dd87828 ("sched/cpufreq: Move the
> > > cfs_rq_util_change() call to cpufreq_update_util()") then?
> 
> This patch is not related to the 2 patches above. It removes some
> spurious call to cfs_rq_util_change() and some wrong ordering but this
> is a fixed overhead which is not impacted by the number of cgroups
> unlike the 2 patches above. This patch will not help with the high
> load of update_blocked_averages

Thanks for letting us know, the other two patches are now queued up.

greg k-h