[6.1] Please apply cc6003916ed46d7a67d91ee32de0f9138047d55f

[6.1] Please apply cc6003916ed46d7a67d91ee32de0f9138047d55f

[Luiz Capitulino]

Hi,

As reported before[1], we found another regression in 6.1 when doing
performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT
being enabled by default by the following upstream commit if you have the
right config dependencies enabled (commit is introduced in v5.16-rc1):

"""
commit c597bfddc9e9e8a63817252b67c3ca0e544ace26
Author: Frederic Weisbecker <frederic@kernel.org>
Date: Tue Sep 14 12:31:34 2021 +0200

sched: Provide Kconfig support for default dynamic preempt mode
"""

We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT
disabled in different perf benchmarks (including UnixBench process
creation and redis). The root cause is explained in the commit log
below which is merged in 6.3 and applies (almost) clealy on 6.1.59.

"""
commit cc6003916ed46d7a67d91ee32de0f9138047d55f
Author: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Date:   Sat Jan 21 12:39:42 2023 +0900

     lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default

     In workloads where this_cpu operations are frequently performed,
     enabling DEBUG_PREEMPT may result in significant increase in
     runtime overhead due to frequent invocation of
     __this_cpu_preempt_check() function.

     This can be demonstrated through benchmarks such as hackbench where this
     configuration results in a 10% reduction in performance, primarily due to
     the added overhead within memcg charging path.
"""

[1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.com/

Re: [6.1] Please apply cc6003916ed46d7a67d91ee32de0f9138047d55f

[Hyeonggon Yoo]

On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino <luizcap@amazon.com> wrote:
>
> Hi,
>
> As reported before[1], we found another regression in 6.1 when doing
> performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT
> being enabled by default by the following upstream commit if you have the
> right config dependencies enabled (commit is introduced in v5.16-rc1):
>
> """
> commit c597bfddc9e9e8a63817252b67c3ca0e544ace26
> Author: Frederic Weisbecker <frederic@kernel.org>
> Date: Tue Sep 14 12:31:34 2021 +0200
>
> sched: Provide Kconfig support for default dynamic preempt mode
> """
>
> We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT
> disabled in different perf benchmarks (including UnixBench process
> creation and redis). The root cause is explained in the commit log
> below which is merged in 6.3 and applies (almost) clealy on 6.1.59.

Oh, I should've sent it to the stable. Thanks for sending it!

Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction
of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels
but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION)
so distros that were using PREEMPT_VOLUNTARY are silently affected by that.

It looks appropriate to be backported to the stable tree (to me).
Hmm but I think it should be backported to 5.15 too?

> """
> commit cc6003916ed46d7a67d91ee32de0f9138047d55f
> Author: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> Date:   Sat Jan 21 12:39:42 2023 +0900
>
>      lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default
>
>      In workloads where this_cpu operations are frequently performed,
>      enabling DEBUG_PREEMPT may result in significant increase in
>      runtime overhead due to frequent invocation of
>      __this_cpu_preempt_check() function.
>
>      This can be demonstrated through benchmarks such as hackbench where this
>      configuration results in a 10% reduction in performance, primarily due to
>      the added overhead within memcg charging path.
> """
>
> [1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.com/

Re: [6.1] Please apply cc6003916ed46d7a67d91ee32de0f9138047d55f

[Greg KH]

On Fri, Oct 20, 2023 at 11:46:36AM +0900, Hyeonggon Yoo wrote:
> On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino <luizcap@amazon.com> wrote:
> >
> > Hi,
> >
> > As reported before[1], we found another regression in 6.1 when doing
> > performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT
> > being enabled by default by the following upstream commit if you have the
> > right config dependencies enabled (commit is introduced in v5.16-rc1):
> >
> > """
> > commit c597bfddc9e9e8a63817252b67c3ca0e544ace26
> > Author: Frederic Weisbecker <frederic@kernel.org>
> > Date: Tue Sep 14 12:31:34 2021 +0200
> >
> > sched: Provide Kconfig support for default dynamic preempt mode
> > """
> >
> > We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT
> > disabled in different perf benchmarks (including UnixBench process
> > creation and redis). The root cause is explained in the commit log
> > below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
> 
> Oh, I should've sent it to the stable. Thanks for sending it!
> 
> Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction
> of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels
> but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION)
> so distros that were using PREEMPT_VOLUNTARY are silently affected by that.
> 
> It looks appropriate to be backported to the stable tree (to me).
> Hmm but I think it should be backported to 5.15 too?

Now queued up, thanks.

greg k-h

Re: [6.1] Please apply cc6003916ed46d7a67d91ee32de0f9138047d55f

[Luiz Capitulino]

On Fri, Oct 20, 2023 at 11:46:36AM +0900, Hyeonggon Yoo wrote:

> On Fri, Oct 20, 2023 at 10:27 AM Luiz Capitulino <luizcap@amazon.com> wrote:
> >
> > Hi,
> >
> > As reported before[1], we found another regression in 6.1 when doing
> > performance comparisons with 5.10. This one is caused by CONFIG_DEBUG_PREEMPT
> > being enabled by default by the following upstream commit if you have the
> > right config dependencies enabled (commit is introduced in v5.16-rc1):
> >
> > """
> > commit c597bfddc9e9e8a63817252b67c3ca0e544ace26
> > Author: Frederic Weisbecker <frederic@kernel.org>
> > Date: Tue Sep 14 12:31:34 2021 +0200
> >
> > sched: Provide Kconfig support for default dynamic preempt mode
> > """
> >
> > We found up to 8% performance improvement with CONFIG_DEBUG_PREEMPT
> > disabled in different perf benchmarks (including UnixBench process
> > creation and redis). The root cause is explained in the commit log
> > below which is merged in 6.3 and applies (almost) clealy on 6.1.59.
> 
> Oh, I should've sent it to the stable. Thanks for sending it!

Thanks for doing the original fix! :)

> Yes, DEBUG_PREEMPT was unintentionally enabled after the introduction
> of PREEMPT_DYNAMIC. It was already enabled by default for PREEMPTION=y kernels
> but PREEMPT_DYNAMIC always enables PREEMPT_BUILD (and hence PREEMPTION)
> so distros that were using PREEMPT_VOLUNTARY are silently affected by that.
> 
> It looks appropriate to be backported to the stable tree (to me).
> Hmm but I think it should be backported to 5.15 too?

Yeah, I see that Greg applied it to 5.15 and 5.10 as well.

I posted it only for 6.1 because the worst case seems to happen after
c597bfddc9 where CONFIG_DEBUG_PREEMPT may be enabled automatically.
But having the fix in earlier kernels is certainly good as well.

- Luiz

> 
> > """
> > commit cc6003916ed46d7a67d91ee32de0f9138047d55f
> > Author: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> > Date:   Sat Jan 21 12:39:42 2023 +0900
> >
> >      lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default
> >
> >      In workloads where this_cpu operations are frequently performed,
> >      enabling DEBUG_PREEMPT may result in significant increase in
> >      runtime overhead due to frequent invocation of
> >      __this_cpu_preempt_check() function.
> >
> >      This can be demonstrated through benchmarks such as hackbench where this
> >      configuration results in a 10% reduction in performance, primarily due to
> >      the added overhead within memcg charging path.
> > """
> >
> > [1] https://lore.kernel.org/stable/010edf5a-453d-4c98-9c07-12e75d3f983c@amazon.com/