Linux real-time development
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>,
	Juri Lelli <juri.lelli@redhat.com>,
	Vincent Guittot <vincent.guittot@linaro.org>,
	Dietmar Eggemann <dietmar.eggemann@arm.com>,
	Ben Segall <bsegall@google.com>, Mel Gorman <mgorman@suse.de>,
	Valentin Schneider <vschneid@redhat.com>,
	K Prateek Nayak <kprateek.nayak@amd.com>,
	Kyle McMartin <jkkm@meta.com>,
	linux-kernel@vger.kernel.org, stable@vger.kernel.org,
	Linux RT Development <linux-rt-devel@lists.linux.dev>,
	Clark Williams <williams@redhat.com>,
	Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	John Kacur <jkacur@redhat.com>
Subject: Re: [PATCH sched/core] sched/rt: Fix RT_PUSH_IPI soft lockup loop
Date: Thu, 14 May 2026 11:15:31 -1000	[thread overview]
Message-ID: <agY7c_uARe72fhwa@slm.duckdns.org> (raw)
In-Reply-To: <20260514100300.1d594c7a@gandalf.local.home>

Hello, Steven.

On Thu, May 14, 2026 at 10:03:00AM -0400, Steven Rostedt wrote:
> I was thinking about this more and does disabling the RT_PUSH_IPI cause any
> problems for you?
> 
>   # echo NO_RT_PUSH_IPI > /sys/kernel/debug/sched/features

Not at all. This is actually the mitigation that we deployed across the
affected machines.

...
> -/* RT IPI pull logic requires IRQ_WORK */
> -#if defined(CONFIG_IRQ_WORK) && defined(CONFIG_SMP)
> +/*
> + * RT IPI pull logic requires IRQ_WORK and doesn't make sense for uniprocessors.
> + * If CONFIG_IRQ_FORCED_THREADING isn't set, then softirqs do not run as threads
> + * and can cause latency larger than what RT_PUSH_IPI can save, killing the
> + * effect of it.
> + */
> +#if defined(CONFIG_IRQ_WORK) && defined(CONFIG_SMP) &&	\
> +	defined(CONFIG_IRQ_FORCED_THREADING)
>  # define HAVE_RT_PUSH_IPI
>  #endif

Maybe it should trigger on force_irqthreads so that it's active only when
irq threads are actully enabled.

Whichever way it's done tho, wouldn't this still leave machines in that
config susceptible to IPI storms? It took a combination of factors to
trigger - mpi3mr's threaded irq, psimon activated by systemd, and sustained
network load - but those factors are not that exotic.

Thanks.

-- 
tejun

  reply	other threads:[~2026-05-14 21:15 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20260506235716.2530720-1-tj@kernel.org>
     [not found] ` <20260507141437.GJ3102624@noisy.programming.kicks-ass.net>
2026-05-12 15:37   ` [PATCH sched/core] sched/rt: Fix RT_PUSH_IPI soft lockup loop Steven Rostedt
2026-05-12 18:07     ` Tejun Heo
2026-05-12 21:28       ` Steven Rostedt
2026-05-13 19:39         ` Tejun Heo
2026-05-14  0:24           ` Steven Rostedt
2026-05-14  0:53             ` Tejun Heo
2026-05-14  1:31               ` Steven Rostedt
2026-05-14  1:42                 ` Tejun Heo
2026-05-14  2:01                   ` Steven Rostedt
2026-05-14  4:48                     ` Tejun Heo
2026-05-14 14:03                       ` Steven Rostedt
2026-05-14 21:15                         ` Tejun Heo [this message]
2026-05-14 23:43                           ` Steven Rostedt
2026-05-12 20:10     ` Valentin Schneider

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=agY7c_uARe72fhwa@slm.duckdns.org \
    --to=tj@kernel.org \
    --cc=bigeasy@linutronix.de \
    --cc=bsegall@google.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=jkacur@redhat.com \
    --cc=jkkm@meta.com \
    --cc=juri.lelli@redhat.com \
    --cc=kprateek.nayak@amd.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=stable@vger.kernel.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=williams@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox