[REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Joseph Salisbury]

Hi Rafael,

We are seeing a reproducible MySQL Sysbench threads regression.  A 
bisect indicated the following commit as the first bad commit:
f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")

The regression was found in Oracle kernel performance testing on OCI VM 
shapes:

VM Details:
* VM.Standard2.1:
       x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM

  * VM.Standard.A1.Flex.2:
       Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM


The ResultsDB runs show the regression in the Sysbench threads metric:

   - VM.Standard2.1:       333 -> 236  (-29.1%)
   - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)

A test kernel was built with f4c31b07b136 reverted and the performance 
regression was recovered.

 From the code, it is possible the regression is due to the new 
previous-wakeup heuristic in the special idle cases.  Before the commit:

   - no cpuidle driver:
       tick_nohz_idle_stop_tick()
       default_idle_call()

   - one idle state:
       tick_nohz_idle_retain_tick()
       cpuidle state 0

After the commit, both paths use:

   idle_call_stop_or_retain_tick(got_tick)

Here, got_tick is true if the CPU was woken from the previous idle-loop 
iteration by the scheduler tick, and false otherwise. On this 
workload/platform combination, using that previous wakeup source to 
decide whether to stop or retain the tick appears to change NOHZ 
behavior enough to regress this wakeup-heavy threaded workload.


Do you have any thoughts on whether this is an expected tradeoff of the 
new heuristic, or whether the special cases need a narrower condition?  
For our stable kernels, the immediate candidate fix is to revert the 
backport, but before doing that I wanted to ask whether upstream would 
prefer a targeted adjustment.

I can collect additional data if useful, for example cpuidle 
driver/state information, timer interrupt counts, idle residency, perf 
stat, or scheduler trace data from the affected OCI shapes.

#regzbot introduced: f4c31b07b136839e0fb3026f8a5b6543e3b14d2f


Thanks,

Joe

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Rafael J. Wysocki (Intel)]

Hi,

On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
<joseph.salisbury@oracle.com> wrote:
>
> Hi Rafael,
>
> We are seeing a reproducible MySQL Sysbench threads regression.  A
> bisect indicated the following commit as the first bad commit:
> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>
> The regression was found in Oracle kernel performance testing on OCI VM
> shapes:
>
> VM Details:
> * VM.Standard2.1:
>        x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>
>   * VM.Standard.A1.Flex.2:
>        Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>
>
> The ResultsDB runs show the regression in the Sysbench threads metric:
>
>    - VM.Standard2.1:       333 -> 236  (-29.1%)
>    - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>
> A test kernel was built with f4c31b07b136 reverted and the performance
> regression was recovered.
>
>  From the code, it is possible the regression is due to the new
> previous-wakeup heuristic in the special idle cases.  Before the commit:
>
>    - no cpuidle driver:
>        tick_nohz_idle_stop_tick()
>        default_idle_call()
>
>    - one idle state:
>        tick_nohz_idle_retain_tick()
>        cpuidle state 0

I think that this is your case and the tick stops for you sometimes
now while it had never stopped before.

Can you confirm?

Overall, it would be good to know the idle state lists for both the VM
and the host.

> After the commit, both paths use:
>
>    idle_call_stop_or_retain_tick(got_tick)
>
> Here, got_tick is true if the CPU was woken from the previous idle-loop
> iteration by the scheduler tick, and false otherwise. On this
> workload/platform combination, using that previous wakeup source to
> decide whether to stop or retain the tick appears to change NOHZ
> behavior enough to regress this wakeup-heavy threaded workload.
>
> Do you have any thoughts on whether this is an expected tradeoff of the
> new heuristic,

For the second case, yes.  It is expected that stopping the tick more
often may cause performance to drop.

> or whether the special cases need a narrower condition?

Let's first identify the case this happens in.

> For our stable kernels, the immediate candidate fix is to revert the
> backport, but before doing that I wanted to ask whether upstream would
> prefer a targeted adjustment.
>
> I can collect additional data if useful, for example cpuidle
> driver/state information, timer interrupt counts, idle residency, perf
> stat, or scheduler trace data from the affected OCI shapes.

So instead of reverting the commit, please try the appended change
(modulo gmail-induced whitespace breakage).

---
 kernel/sched/idle.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -250,7 +250,7 @@ static void cpuidle_idle_call(bool stop_
          */
         cpuidle_reflect(dev, entered_state);
     } else {
-        idle_call_stop_or_retain_tick(stop_tick);
+        idle_call_stop_or_retain_tick(false);

         /*
          * If there is only a single idle state (or none), there is

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Christian Loehle]

On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
> Hi,
> 
> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
> <joseph.salisbury@oracle.com> wrote:
>>
>> Hi Rafael,
>>
>> We are seeing a reproducible MySQL Sysbench threads regression.  A
>> bisect indicated the following commit as the first bad commit:
>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>>
>> The regression was found in Oracle kernel performance testing on OCI VM
>> shapes:
>>
>> VM Details:
>> * VM.Standard2.1:
>>        x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>>
>>   * VM.Standard.A1.Flex.2:
>>        Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>>
>>
>> The ResultsDB runs show the regression in the Sysbench threads metric:
>>
>>    - VM.Standard2.1:       333 -> 236  (-29.1%)
>>    - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>>
>> A test kernel was built with f4c31b07b136 reverted and the performance
>> regression was recovered.
>>
>>  From the code, it is possible the regression is due to the new
>> previous-wakeup heuristic in the special idle cases.  Before the commit:
>>
>>    - no cpuidle driver:
>>        tick_nohz_idle_stop_tick()
>>        default_idle_call()
>>
>>    - one idle state:
>>        tick_nohz_idle_retain_tick()
>>        cpuidle state 0
> 
> I think that this is your case and the tick stops for you sometimes
> now while it had never stopped before.
> 
> Can you confirm?
> 
> Overall, it would be good to know the idle state lists for both the VM
> and the host.
> 
+1, but also which HZ are you using?
Both systems reported have 2 logical CPUs then? Were higher core counts also
tested and how does it affect them?

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Joseph Salisbury]

On 7/6/26 10:29 AM, Christian Loehle wrote:
> On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
>> Hi,
>>
>> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
>> <joseph.salisbury@oracle.com> wrote:
>>> Hi Rafael,
>>>
>>> We are seeing a reproducible MySQL Sysbench threads regression.  A
>>> bisect indicated the following commit as the first bad commit:
>>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>>>
>>> The regression was found in Oracle kernel performance testing on OCI VM
>>> shapes:
>>>
>>> VM Details:
>>> * VM.Standard2.1:
>>>         x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>>>
>>>    * VM.Standard.A1.Flex.2:
>>>         Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>>>
>>>
>>> The ResultsDB runs show the regression in the Sysbench threads metric:
>>>
>>>     - VM.Standard2.1:       333 -> 236  (-29.1%)
>>>     - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>>>
>>> A test kernel was built with f4c31b07b136 reverted and the performance
>>> regression was recovered.
>>>
>>>   From the code, it is possible the regression is due to the new
>>> previous-wakeup heuristic in the special idle cases.  Before the commit:
>>>
>>>     - no cpuidle driver:
>>>         tick_nohz_idle_stop_tick()
>>>         default_idle_call()
>>>
>>>     - one idle state:
>>>         tick_nohz_idle_retain_tick()
>>>         cpuidle state 0
>> I think that this is your case and the tick stops for you sometimes
>> now while it had never stopped before.
>>
>> Can you confirm?
>>
>> Overall, it would be good to know the idle state lists for both the VM
>> and the host.
>>
> +1, but also which HZ are you using?
> Both systems reported have 2 logical CPUs then? Were higher core counts also
> tested and how does it affect them?
Hi Rafael, Christian,

Thanks for the feedback!

We will collect the requested runtime data from the affected systems 
before testing the proposed one-line change.

For Christian’s HZ question, I can answer from the UEK kernel configs. 
The relevant configs for the affected kernel are:

- x86_64 / VM.Standard2.1:
     - CONFIG_HZ=1000
     - CONFIG_HZ_1000=y
     - CONFIG_NO_HZ_FULL=y
     - CONFIG_NO_HZ=y
     - CONFIG_CPU_IDLE=y
     - CONFIG_INTEL_IDLE=y
     - CONFIG_ACPI_PROCESSOR_IDLE=y

- arm64 / VM.Standard.A1.Flex.2:
     - CONFIG_HZ=250
     - CONFIG_HZ_250=y
     - CONFIG_NO_HZ_FULL=y
     - CONFIG_NO_HZ=y
     - CONFIG_CPU_IDLE=y
     - CONFIG_ACPI_PROCESSOR_IDLE=y
     - CONFIG_ARM_PSCI_CPUIDLE is not set

The ResultsDB metadata for the two reported systems shows both had 2 
logical CPUs:

  - VM.Standard2.1: 1 OCPU / 2 hardware threads
  - VM.Standard.A1.Flex.2: 2 vCPU threads

I do not yet have higher-core-count results for this same comparison.  I 
will check whether those runs exist, and if not, whether we can schedule 
them.

Thanks,

Joe

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Joseph Salisbury]

Hi Rafael, Christian,

On 7/6/26 10:29 AM, Christian Loehle wrote:
> On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
>> Hi,
>>
>> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
>> <joseph.salisbury@oracle.com> wrote:
>>> Hi Rafael,
>>>
>>> We are seeing a reproducible MySQL Sysbench threads regression.  A
>>> bisect indicated the following commit as the first bad commit:
>>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>>>
>>> The regression was found in Oracle kernel performance testing on OCI VM
>>> shapes:
>>>
>>> VM Details:
>>> * VM.Standard2.1:
>>>         x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>>>
>>>    * VM.Standard.A1.Flex.2:
>>>         Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>>>
>>>
>>> The ResultsDB runs show the regression in the Sysbench threads metric:
>>>
>>>     - VM.Standard2.1:       333 -> 236  (-29.1%)
>>>     - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>>>
>>> A test kernel was built with f4c31b07b136 reverted and the performance
>>> regression was recovered.
>>>
>>>   From the code, it is possible the regression is due to the new
>>> previous-wakeup heuristic in the special idle cases.  Before the commit:
>>>
>>>     - no cpuidle driver:
>>>         tick_nohz_idle_stop_tick()
>>>         default_idle_call()
>>>
>>>     - one idle state:
>>>         tick_nohz_idle_retain_tick()
>>>         cpuidle state 0
>> I think that this is your case and the tick stops for you sometimes
>> now while it had never stopped before.
>>
>> Can you confirm?
The guest-visible data does not show the single-idle-state cpuidle case.
Both affected guests report:

/sys/devices/system/cpu/cpuidle/current_driver = none
/sys/devices/system/cpu/cpuidle/current_governor = menu

There are also no /sys/devices/system/cpu/cpu*/cpuidle entries on either 
guest.  So from the guest data, this looks like the no-cpuidle-driver 
special case rather than the one-idle-state case.

The full revert of f4c31b07b136 recovered the regression. I also tested 
Rafael's suggested one-line change, applied as:

     -        idle_call_stop_or_retain_tick(stop_tick);
    +        idle_call_stop_or_retain_tick(false);

That test kernel still showed regressed performance on VM.Standard2.1.


>>
>> Overall, it would be good to know the idle state lists for both the VM
>> and the host.
For the VMs, there are no guest cpuidle state lists exposed because the 
cpuidle driver is "none".

I do not currently have the host-side idle-state lists from the OCI 
hosts.  I can try to get that data if it would still be useful.
>>
> +1, but also which HZ are you using?
VM.Standard2.1, x86_64:

CONFIG_HZ_1000=y
CONFIG_HZ=1000

VM.Standard.A1.Flex.2, aarch64:

CONFIG_HZ_250=y
CONFIG_HZ=250

> Both systems reported have 2 logical CPUs then?

Yes:

VM.Standard2.1:

CPU(s): 2
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1

VM.Standard.A1.Flex.2:

CPU(s): 2
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1

> Were higher core counts also
> tested and how does it affect them?
Yes. Higher-core-count runs were checked. The regression appears limited 
to the smaller core-count shapes.

The current data shows regressions on:

- VM.Standard2.1
- VM.Standard.A1.Flex.2
- VM.Standard.E4.Flex.1

The larger tested shapes did not show the same regression pattern. The 
test runs use one sysbench thread per online CPU/core count as encoded 
in the metric name.

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Christian Loehle]

On 7/24/26 18:20, Joseph Salisbury wrote:
> Hi Rafael, Christian,
> 
> On 7/6/26 10:29 AM, Christian Loehle wrote:
>> On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
>>> Hi,
>>>
>>> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
>>> <joseph.salisbury@oracle.com> wrote:
>>>> Hi Rafael,
>>>>
>>>> We are seeing a reproducible MySQL Sysbench threads regression.  A
>>>> bisect indicated the following commit as the first bad commit:
>>>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>>>>
>>>> The regression was found in Oracle kernel performance testing on OCI VM
>>>> shapes:
>>>>
>>>> VM Details:
>>>> * VM.Standard2.1:
>>>>         x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>>>>
>>>>    * VM.Standard.A1.Flex.2:
>>>>         Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>>>>
>>>>
>>>> The ResultsDB runs show the regression in the Sysbench threads metric:
>>>>
>>>>     - VM.Standard2.1:       333 -> 236  (-29.1%)
>>>>     - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>>>>
>>>> A test kernel was built with f4c31b07b136 reverted and the performance
>>>> regression was recovered.
>>>>
>>>>   From the code, it is possible the regression is due to the new
>>>> previous-wakeup heuristic in the special idle cases.  Before the commit:
>>>>
>>>>     - no cpuidle driver:
>>>>         tick_nohz_idle_stop_tick()
>>>>         default_idle_call()
>>>>
>>>>     - one idle state:
>>>>         tick_nohz_idle_retain_tick()
>>>>         cpuidle state 0
>>> I think that this is your case and the tick stops for you sometimes
>>> now while it had never stopped before.
>>>
>>> Can you confirm?
> The guest-visible data does not show the single-idle-state cpuidle case.
> Both affected guests report:
> 
> /sys/devices/system/cpu/cpuidle/current_driver = none
> /sys/devices/system/cpu/cpuidle/current_governor = menu
> 
> There are also no /sys/devices/system/cpu/cpu*/cpuidle entries on either guest.  So from the guest data, this looks like the no-cpuidle-driver special case rather than the one-idle-state case.
> 
> The full revert of f4c31b07b136 recovered the regression. I also tested Rafael's suggested one-line change, applied as:
> 
>     -        idle_call_stop_or_retain_tick(stop_tick);
>    +        idle_call_stop_or_retain_tick(false);
> 
> That test kernel still showed regressed performance on VM.Standard2.1.
> 
> 
>>>
>>> Overall, it would be good to know the idle state lists for both the VM
>>> and the host.
> For the VMs, there are no guest cpuidle state lists exposed because the cpuidle driver is "none".
> 
> I do not currently have the host-side idle-state lists from the OCI hosts.  I can try to get that data if it would still be useful.
>>>
>> +1, but also which HZ are you using?
> VM.Standard2.1, x86_64:
> 
> CONFIG_HZ_1000=y
> CONFIG_HZ=1000
> 
> VM.Standard.A1.Flex.2, aarch64:
> 
> CONFIG_HZ_250=y
> CONFIG_HZ=250
> 
>> Both systems reported have 2 logical CPUs then?
> 
> Yes:
> 
> VM.Standard2.1:
> 
> CPU(s): 2
> Thread(s) per core: 2
> Core(s) per socket: 1
> Socket(s): 1
> 
> VM.Standard.A1.Flex.2:
> 
> CPU(s): 2
> Thread(s) per core: 1
> Core(s) per socket: 2
> Socket(s): 1
> 
>> Were higher core counts also
>> tested and how does it affect them?
> Yes. Higher-core-count runs were checked. The regression appears limited to the smaller core-count shapes.
> 
> The current data shows regressions on:
> 
> - VM.Standard2.1
> - VM.Standard.A1.Flex.2
> - VM.Standard.E4.Flex.1
> 
> The larger tested shapes did not show the same regression pattern. The test runs use one sysbench thread per online CPU/core count as encoded in the metric name.
> 
> 

Interesting, so your guests (no cpuidle) need the tick stopped at every
idle entry to not regress, i.e. the below?
Is there anything obvious that shows why that would be? Maybe in the
hypervisor behaviour?

----

diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 052435f4d3e3..d91a102ef028 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -194,8 +194,7 @@ static void cpuidle_idle_call(bool stop_tick)
        }
 
        if (cpuidle_not_available(drv, dev)) {
-               idle_call_stop_or_retain_tick(stop_tick);
-
+               idle_call_stop_or_retain_tick(true);
                default_idle_call();
                goto exit_idle;
        }

[PATCH] sched/idle: Stop the tick when no cpuidle driver is available

[Christian Loehle]

Commit f4c31b07b136 ("sched: idle: Consolidate the handling of two
special cases") changed the no-cpuidle-driver path to use the previous
tick wakeup heuristic.

Oracle reported a substantial sysbench regression on small OCI VM shapes
with no cpuidle driver available. Reverting the commit recovered the lost
performance.

Restore the previous tick handling for the no-driver case while retaining
the heuristic for the single-state cpuidle case.

Fixes: f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
Reported-by: Joseph Salisbury <joseph.salisbury@oracle.com>
Closes: https://lore.kernel.org/all/096b42fa-107f-450d-b3b1-03bcad3f1e04@oracle.com/
Signed-off-by: Christian Loehle <christian.loehle@arm.com>
---
Here's the actual patch, I'd still be curious about why this happens here...

 kernel/sched/idle.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
index 052435f4d3e3..a814f9d5759f 100644
--- a/kernel/sched/idle.c
+++ b/kernel/sched/idle.c
@@ -194,7 +194,7 @@ static void cpuidle_idle_call(bool stop_tick)
 	}
 
 	if (cpuidle_not_available(drv, dev)) {
-		idle_call_stop_or_retain_tick(stop_tick);
+		tick_nohz_idle_stop_tick();
 
 		default_idle_call();
 		goto exit_idle;
-- 
2.34.1

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Zhan Xusheng]

From: Zhan Xusheng <zhanxusheng1024@gmail.com>

On Tue, Jul 28, 2026 at 09:30:39 +0100, Christian Loehle wrote:
> Interesting, so your guests (no cpuidle) need the tick stopped at every
> idle entry to not regress, i.e. the below?
> Is there anything obvious that shows why that would be? Maybe in the
> hypervisor behaviour?

I think it lines up with the got_tick heuristic rather than anything
hypervisor-specific in the guest kernel path.

do_idle() resets got_tick to false at the top of every idle episode and
passes it as stop_tick, so the first idle iteration always calls
idle_call_stop_or_retain_tick(false); tick_nohz_idle_enter() hasn't stopped
the tick at that point, so that takes the retain branch. Only after the
tick fires once (got_tick becomes true) does a later iteration stop it.
So with f4c31b the no-driver path leaves the periodic tick armed at the
start of every idle episode, whereas the old code stopped it
unconditionally.

That also explains the test results: forcing (false) keeps the retain and
still regresses, while (true) / the direct tick_nohz_idle_stop_tick()
restores the old always-stop and recovers.

The hypervisor is then where the consequence shows up: a guest that leaves
its tick running keeps a ~1/HZ timer pending, so the host sees an imminent
timer and keeps waking/scheduling the vCPU instead of letting it idle. That
is at least consistent with the shapes - the x86 shape at HZ=1000 regressed
more (-29%) than the arm shape at HZ=250 (-10%), i.e. more retained ticks,
more interference. For the no-driver bare halt there is no governor/state
selection that a retained tick could help, so stopping unconditionally is
strictly better.

Thanks,
Zhan Xusheng

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Rafael J. Wysocki (Intel)]

On Wed, Jul 29, 2026 at 4:36 AM Zhan Xusheng <zhanxusheng1024@gmail.com> wrote:
>
> From: Zhan Xusheng <zhanxusheng1024@gmail.com>
>
> On Tue, Jul 28, 2026 at 09:30:39 +0100, Christian Loehle wrote:
> > Interesting, so your guests (no cpuidle) need the tick stopped at every
> > idle entry to not regress, i.e. the below?
> > Is there anything obvious that shows why that would be? Maybe in the
> > hypervisor behaviour?
>
> I think it lines up with the got_tick heuristic rather than anything
> hypervisor-specific in the guest kernel path.
>
> do_idle() resets got_tick to false at the top of every idle episode and
> passes it as stop_tick, so the first idle iteration always calls
> idle_call_stop_or_retain_tick(false); tick_nohz_idle_enter() hasn't stopped
> the tick at that point, so that takes the retain branch. Only after the
> tick fires once (got_tick becomes true) does a later iteration stop it.
> So with f4c31b the no-driver path leaves the periodic tick armed at the
> start of every idle episode, whereas the old code stopped it
> unconditionally.

Sure.

That's an intentional modification on the premise that stopping the
tick at the beginning of every idle episode would be more overhead
that letting it run until it actually fires.

> That also explains the test results: forcing (false) keeps the retain and
> still regresses, while (true) / the direct tick_nohz_idle_stop_tick()
> restores the old always-stop and recovers.

No, by itself, it doesn't explain anything.  In particular, it doesn't
explain why stopping the tick upfront in every idle episode leads to
better performance.

> The hypervisor is then where the consequence shows up: a guest that leaves
> its tick running keeps a ~1/HZ timer pending, so the host sees an imminent
> timer and keeps waking/scheduling the vCPU instead of letting it idle.

Right, but then why is performance better when the vCPU is put to
sleep at the beginning of every idle episode?

> That is at least consistent with the shapes - the x86 shape at HZ=1000 regressed
> more (-29%) than the arm shape at HZ=250 (-10%), i.e. more retained ticks,
> more interference.

What interference do you mean in particular?

It looks like putting an idle vCPU to sleep upfront allows the other
vCPUs to do more work: Is this what you mean?

> For the no-driver bare halt there is no governor/state selection that a retained tick could help, so stopping
> unconditionally is strictly better.

Well, I'm not sure I can agree with this statement just yet.

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Rafael J. Wysocki (Intel)]

On Tue, Jul 28, 2026 at 10:30 AM Christian Loehle
<christian.loehle@arm.com> wrote:
>
> On 7/24/26 18:20, Joseph Salisbury wrote:
> > Hi Rafael, Christian,
> >
> > On 7/6/26 10:29 AM, Christian Loehle wrote:
> >> On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
> >>> Hi,
> >>>
> >>> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
> >>> <joseph.salisbury@oracle.com> wrote:
> >>>> Hi Rafael,
> >>>>
> >>>> We are seeing a reproducible MySQL Sysbench threads regression.  A
> >>>> bisect indicated the following commit as the first bad commit:
> >>>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
> >>>>
> >>>> The regression was found in Oracle kernel performance testing on OCI VM
> >>>> shapes:
> >>>>
> >>>> VM Details:
> >>>> * VM.Standard2.1:
> >>>>         x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
> >>>>
> >>>>    * VM.Standard.A1.Flex.2:
> >>>>         Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
> >>>>
> >>>>
> >>>> The ResultsDB runs show the regression in the Sysbench threads metric:
> >>>>
> >>>>     - VM.Standard2.1:       333 -> 236  (-29.1%)
> >>>>     - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
> >>>>
> >>>> A test kernel was built with f4c31b07b136 reverted and the performance
> >>>> regression was recovered.
> >>>>
> >>>>   From the code, it is possible the regression is due to the new
> >>>> previous-wakeup heuristic in the special idle cases.  Before the commit:
> >>>>
> >>>>     - no cpuidle driver:
> >>>>         tick_nohz_idle_stop_tick()
> >>>>         default_idle_call()
> >>>>
> >>>>     - one idle state:
> >>>>         tick_nohz_idle_retain_tick()
> >>>>         cpuidle state 0
> >>> I think that this is your case and the tick stops for you sometimes
> >>> now while it had never stopped before.
> >>>
> >>> Can you confirm?
> > The guest-visible data does not show the single-idle-state cpuidle case.
> > Both affected guests report:
> >
> > /sys/devices/system/cpu/cpuidle/current_driver = none
> > /sys/devices/system/cpu/cpuidle/current_governor = menu
> >
> > There are also no /sys/devices/system/cpu/cpu*/cpuidle entries on either guest.  So from the guest data, this looks like the no-cpuidle-driver special case rather than the one-idle-state case.
> >
> > The full revert of f4c31b07b136 recovered the regression. I also tested Rafael's suggested one-line change, applied as:
> >
> >     -        idle_call_stop_or_retain_tick(stop_tick);
> >    +        idle_call_stop_or_retain_tick(false);
> >
> > That test kernel still showed regressed performance on VM.Standard2.1.
> >
> >
> >>>
> >>> Overall, it would be good to know the idle state lists for both the VM
> >>> and the host.
> > For the VMs, there are no guest cpuidle state lists exposed because the cpuidle driver is "none".
> >
> > I do not currently have the host-side idle-state lists from the OCI hosts.  I can try to get that data if it would still be useful.
> >>>
> >> +1, but also which HZ are you using?
> > VM.Standard2.1, x86_64:
> >
> > CONFIG_HZ_1000=y
> > CONFIG_HZ=1000
> >
> > VM.Standard.A1.Flex.2, aarch64:
> >
> > CONFIG_HZ_250=y
> > CONFIG_HZ=250
> >
> >> Both systems reported have 2 logical CPUs then?
> >
> > Yes:
> >
> > VM.Standard2.1:
> >
> > CPU(s): 2
> > Thread(s) per core: 2
> > Core(s) per socket: 1
> > Socket(s): 1
> >
> > VM.Standard.A1.Flex.2:
> >
> > CPU(s): 2
> > Thread(s) per core: 1
> > Core(s) per socket: 2
> > Socket(s): 1
> >
> >> Were higher core counts also
> >> tested and how does it affect them?
> > Yes. Higher-core-count runs were checked. The regression appears limited to the smaller core-count shapes.
> >
> > The current data shows regressions on:
> >
> > - VM.Standard2.1
> > - VM.Standard.A1.Flex.2
> > - VM.Standard.E4.Flex.1
> >
> > The larger tested shapes did not show the same regression pattern. The test runs use one sysbench thread per online CPU/core count as encoded in the metric name.
> >
> >
>
> Interesting, so your guests (no cpuidle) need the tick stopped at every
> idle entry to not regress, i.e. the below?
> Is there anything obvious that shows why that would be? Maybe in the
> hypervisor behaviour?

Well, this is all unclear to me and it looks like the host is using
the tick stopping in the guest to drive vCPU scheduling decisions or
similar.

I'm guessing (but not sure at all) that arch_cpu_idle() in the x86
version of the guest is just native_safe_halt() which then traps to
the host which does something.  And that may depend on whether or not
the tick has been stopped in the guest.

I'm not going to apply any changes related to this without a clear
understanding of what is really going on and there is too little
information for that ATM.

> ----
>
> diff --git a/kernel/sched/idle.c b/kernel/sched/idle.c
> index 052435f4d3e3..d91a102ef028 100644
> --- a/kernel/sched/idle.c
> +++ b/kernel/sched/idle.c
> @@ -194,8 +194,7 @@ static void cpuidle_idle_call(bool stop_tick)
>         }
>
>         if (cpuidle_not_available(drv, dev)) {
> -               idle_call_stop_or_retain_tick(stop_tick);
> -
> +               idle_call_stop_or_retain_tick(true);
>                 default_idle_call();
>                 goto exit_idle;
>         }
>

Re: [REGRESSION] sched/idle: Sysbench threads regression after f4c31b07b136

[Christian Loehle]

On 7/29/26 19:25, Rafael J. Wysocki (Intel) wrote:
> On Tue, Jul 28, 2026 at 10:30 AM Christian Loehle
> <christian.loehle@arm.com> wrote:
>>
>> On 7/24/26 18:20, Joseph Salisbury wrote:
>>> Hi Rafael, Christian,
>>>
>>> On 7/6/26 10:29 AM, Christian Loehle wrote:
>>>> On 7/2/26 19:47, Rafael J. Wysocki (Intel) wrote:
>>>>> Hi,
>>>>>
>>>>> On Thu, Jul 2, 2026 at 6:30 PM Joseph Salisbury
>>>>> <joseph.salisbury@oracle.com> wrote:
>>>>>> Hi Rafael,
>>>>>>
>>>>>> We are seeing a reproducible MySQL Sysbench threads regression.  A
>>>>>> bisect indicated the following commit as the first bad commit:
>>>>>> f4c31b07b136 ("sched: idle: Consolidate the handling of two special cases")
>>>>>>
>>>>>> The regression was found in Oracle kernel performance testing on OCI VM
>>>>>> shapes:
>>>>>>
>>>>>> VM Details:
>>>>>> * VM.Standard2.1:
>>>>>>         x86 OCI VM shape, 1 OCPU / 2 hardware threads, about 14.5 GB RAM
>>>>>>
>>>>>>    * VM.Standard.A1.Flex.2:
>>>>>>         Arm/Ampere A1 flexible VM shape, 2 vCPU threads, about 10.9 GB RAM
>>>>>>
>>>>>>
>>>>>> The ResultsDB runs show the regression in the Sysbench threads metric:
>>>>>>
>>>>>>     - VM.Standard2.1:       333 -> 236  (-29.1%)
>>>>>>     - VM.Standard.A1.Flex.2: 1286 -> 1152 (-10.4%)
>>>>>>
>>>>>> A test kernel was built with f4c31b07b136 reverted and the performance
>>>>>> regression was recovered.
>>>>>>
>>>>>>   From the code, it is possible the regression is due to the new
>>>>>> previous-wakeup heuristic in the special idle cases.  Before the commit:
>>>>>>
>>>>>>     - no cpuidle driver:
>>>>>>         tick_nohz_idle_stop_tick()
>>>>>>         default_idle_call()
>>>>>>
>>>>>>     - one idle state:
>>>>>>         tick_nohz_idle_retain_tick()
>>>>>>         cpuidle state 0
>>>>> I think that this is your case and the tick stops for you sometimes
>>>>> now while it had never stopped before.
>>>>>
>>>>> Can you confirm?
>>> The guest-visible data does not show the single-idle-state cpuidle case.
>>> Both affected guests report:
>>>
>>> /sys/devices/system/cpu/cpuidle/current_driver = none
>>> /sys/devices/system/cpu/cpuidle/current_governor = menu
>>>
>>> There are also no /sys/devices/system/cpu/cpu*/cpuidle entries on either guest.  So from the guest data, this looks like the no-cpuidle-driver special case rather than the one-idle-state case.
>>>
>>> The full revert of f4c31b07b136 recovered the regression. I also tested Rafael's suggested one-line change, applied as:
>>>
>>>     -        idle_call_stop_or_retain_tick(stop_tick);
>>>    +        idle_call_stop_or_retain_tick(false);
>>>
>>> That test kernel still showed regressed performance on VM.Standard2.1.
>>>
>>>
>>>>>
>>>>> Overall, it would be good to know the idle state lists for both the VM
>>>>> and the host.
>>> For the VMs, there are no guest cpuidle state lists exposed because the cpuidle driver is "none".
>>>
>>> I do not currently have the host-side idle-state lists from the OCI hosts.  I can try to get that data if it would still be useful.
>>>>>
>>>> +1, but also which HZ are you using?
>>> VM.Standard2.1, x86_64:
>>>
>>> CONFIG_HZ_1000=y
>>> CONFIG_HZ=1000
>>>
>>> VM.Standard.A1.Flex.2, aarch64:
>>>
>>> CONFIG_HZ_250=y
>>> CONFIG_HZ=250
>>>
>>>> Both systems reported have 2 logical CPUs then?
>>>
>>> Yes:
>>>
>>> VM.Standard2.1:
>>>
>>> CPU(s): 2
>>> Thread(s) per core: 2
>>> Core(s) per socket: 1
>>> Socket(s): 1
>>>
>>> VM.Standard.A1.Flex.2:
>>>
>>> CPU(s): 2
>>> Thread(s) per core: 1
>>> Core(s) per socket: 2
>>> Socket(s): 1
>>>
>>>> Were higher core counts also
>>>> tested and how does it affect them?
>>> Yes. Higher-core-count runs were checked. The regression appears limited to the smaller core-count shapes.
>>>
>>> The current data shows regressions on:
>>>
>>> - VM.Standard2.1
>>> - VM.Standard.A1.Flex.2
>>> - VM.Standard.E4.Flex.1
>>>
>>> The larger tested shapes did not show the same regression pattern. The test runs use one sysbench thread per online CPU/core count as encoded in the metric name.
>>>
>>>
>>
>> Interesting, so your guests (no cpuidle) need the tick stopped at every
>> idle entry to not regress, i.e. the below?
>> Is there anything obvious that shows why that would be? Maybe in the
>> hypervisor behaviour?
> 
> Well, this is all unclear to me and it looks like the host is using
> the tick stopping in the guest to drive vCPU scheduling decisions or
> similar.
> 
> I'm guessing (but not sure at all) that arch_cpu_idle() in the x86
> version of the guest is just native_safe_halt() which then traps to
> the host which does something.  And that may depend on whether or not
> the tick has been stopped in the guest.
> 

Maybe some of my intuition, I'd be surprised if the slightly delayed
idle entry (because the vCPU stops the tick) explains the big difference,
so it's likely either:
- Hypervisor changes behaviour of a vCPU depending on when it's next
vtimer is armed (leaving the tick on means sooner of course)
- The tick wakeup itself is cause of the different observed behaviour.

To distinguish the two it would be helpful to know how many of the wakeups
are tick wakeups in the "leave tick enabled" case.
Also just to confirm, the reported metrics are just from a single workload
instance right? So we don't have to consider cases of "Leaving tick on doesn't
allow other system vCPUs to use the CPU as effectively", right? Or is this
possible here?

> I'm not going to apply any changes related to this without a clear
> understanding of what is really going on and there is too little
> information for that ATM.

Agreed.

> [snip]