All of lore.kernel.org
 help / color / mirror / Atom feed
* Kernel 6.14.11 dl_server_timer(...) causing IPI/Function Call Interrupts on isolcpu/nohz_full cores, performance regression
@ 2025-07-30 16:51 David Haufe
  2025-07-31  7:01 ` Juri Lelli
  0 siblings, 1 reply; 14+ messages in thread
From: David Haufe @ 2025-07-30 16:51 UTC (permalink / raw)
  To: Juri Lelli; +Cc: linux-kernel

[1.] Kernel 6.14.11 dl_server_timer(...) causing IPI/Function Call
Interrupts on isolcpu/nohz_full cores, performance regression
[2.] The code for dl_server_timer is causing new IPI/Function Call
Interrupts to fire on isolcpu/nohz_full cores which previously had no
interrupts. When there is a single, SCHED_OTHER process running on an
isolcpu/nohz_full core, dl_server_timer executes on a housekeeping
core. This ultimately invokes add_nr_running() and
sched_update_tick_dependency() and finally tick_nohz_dep_set_cpu().
Setting the single process running on an isolcpu/nohz_full core to
FIFO (rt priority) prevents this new interrupt, as it is not seen as a
fair schedule process anymore. Having to use rt priority is
unnecessary and a regression to prior kernels. Kernel function_graph
trace below showing core 0 (housekeeping) sending the IPI to core 19
(nohz_full, isolcpu, rcu_nocb_poll) which is running a single
SCHED_OTHER process. I believe this has been observed by others.
https://community.clearlinux.org/t/sysjitter-worse-in-kernel-6-12-than-6-6/10206

dl_server_timer() {
  0)               |    raw_spin_rq_lock_nested() {
  0)               |      _raw_spin_lock() {
  0)   0.051 us    |        preempt_count_add();
  0)   0.160 us    |      }
  0)   0.357 us    |    }
  0)               |    update_rq_clock() {
  0)   0.115 us    |      arch_scale_cpu_capacity();
  0)   0.498 us    |    }
  0)   0.222 us    |    fair_server_has_tasks();
  0)               |    enqueue_dl_entity() {
  0)   0.056 us    |      replenish_dl_entity();
  0)   0.050 us    |      sched_can_stop_tick();
  0)               |      tick_nohz_dep_set_cpu() {
  0)   0.051 us    |        preempt_count_add();
  0)               |        tick_nohz_full_kick_cpu() {
  0)   0.052 us    |          preempt_count_add();
  0)               |          __smp_call_single_queue() {
  0)               |            /* csd_queue_cpu: cpu=19
callsite=return_to_handler+0x0/0x40 func=nohz_full_kick_func
csd=000000001d10a127 */
  0)   0.178 us    |            call_function_single_prep_ipi();
  0)               |            /* ipi_send_cpu: cpu=19
callsite=return_to_handler+0x0/0x40
callback=generic_smp_call_function_single_interrupt+0x0/0x20 */
  0)               |            native_send_call_func_single_ipi() {
  0)               |              x2apic_send_IPI() {
  0)   0.084 us    |                __x2apic_send_IPI_dest();
  0)   0.203 us    |              }
  0)   0.298 us    |            }
  0)   0.713 us    |          }
  0)   0.053 us    |          preempt_count_sub();
  0)   1.080 us    |        }
  0)   0.052 us    |        preempt_count_sub();
  0)   1.586 us    |      }
  0)   0.237 us    |      cpupri_set();
  0)               |      cpudl_set() {
  0)               |        _raw_spin_lock_irqsave() {
  0)   0.053 us    |          preempt_count_add();
  0)   0.176 us    |        }
  0)   0.141 us    |        cpudl_heapify_up();
  0)               |        _raw_spin_unlock_irqrestore() {
  0)   0.052 us    |          preempt_count_sub();
  0)   0.158 us    |        }
  0)   0.725 us    |      }
  0)   3.014 us    |    }
  0)               |    resched_curr() {
  0)               |      __resched_curr() {
  0)               |        /* ipi_send_cpu: cpu=19
callsite=return_to_handler+0x0/0x40 callback=0x0 */
  0)               |        native_smp_send_reschedule() {
  0)               |          x2apic_send_IPI() {
  0)   0.084 us    |            __x2apic_send_IPI_dest();
  0)   0.200 us    |          }
  0)   0.296 us    |        }
  0)   0.580 us    |      }
  0)   0.678 us    |    }
  0)               |    raw_spin_rq_unlock() {
  0)               |      _raw_spin_unlock() {
  0)   0.052 us    |        preempt_count_sub();
  0)   0.159 us    |      }
  0)   0.260 us    |    }
  0)   5.436 us    |  }

[3.] SCHED_DEADLINE, CFS, nohz, isolcpu, nohz_full_kick_func
[4.] 6.14.11, code appears first in 6.12
[5.] 6.4.16 does not have this issue

-- 
DISCLAIMER: NOTICE REGARDING PRIVACY AND CONFIDENTIALITY 

The information 
contained in and/or accompanying this communication is intended only for 
use by the addressee(s) named herein and may contain legally privileged 
and/or confidential information. If you are not the intended recipient of 
this e-mail, you are hereby notified that any dissemination, distribution 
or copying of this information, and any attachments thereto, is strictly 
prohibited. If you have received this e-mail in error, please immediately 
notify the sender and permanently delete the original and any copy of any 
e-mail and any printout thereof. Electronic transmissions cannot be 
guaranteed to be secure or error-free. The sender therefore does not accept 
liability for any errors or omissions in the contents of this message which 
arise as a result of e-mail transmission. Simplex Trading, LLC and its 
affiliates reserves the right to intercept, monitor, and retain electronic 
communications to and from its system as permitted by law. Simplex Trading, 
LLC is a registered Broker Dealer with CBOE and a Member of SIPC.

^ permalink raw reply	[flat|nested] 14+ messages in thread
end of thread, other threads:[~2026-04-24 12:38 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-30 16:51 Kernel 6.14.11 dl_server_timer(...) causing IPI/Function Call Interrupts on isolcpu/nohz_full cores, performance regression David Haufe
2025-07-31  7:01 ` Juri Lelli
2025-07-31 17:50   ` David Haufe
     [not found]   ` <CAKJHwtOZkrR9kEj+tffq=o0i1fPi3P+8BTHz3RyPDmn=uDOF7g@mail.gmail.com>
2025-08-01  9:06     ` Juri Lelli
2025-08-01 15:28       ` David Haufe
2025-08-04 14:49         ` Juri Lelli
2025-08-04 15:44           ` David Haufe
2025-08-04 16:58             ` Juri Lelli
2025-11-24 20:30               ` David Haufe
2025-11-25 13:58                 ` Juri Lelli
2025-11-25 19:33                   ` David Haufe
2025-12-03 14:29                     ` Juri Lelli
2026-04-10 16:39                     ` Cao Ruichuang
2026-04-24 12:38                       ` Juri Lelli

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.