The Linux Kernel Mailing List
 help / color / mirror / Atom feed
From: Thomas Gleixner <tglx@kernel.org>
To: Tony Rodriguez <unixpro1970@gmail.com>,
	Thorsten Leemhuis <regressions@leemhuis.info>
Cc: John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de>,
	Greg KH <gregkh@linuxfoundation.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Linux kernel regressions list <regressions@lists.linux.dev>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: the stuttering regression in 7.0: should I have done something different?
Date: Sun, 10 May 2026 23:29:56 +0200	[thread overview]
Message-ID: <871pfj9cmj.ffs@tglx> (raw)
In-Reply-To: <3332123b-9e11-4895-9ab3-1707fba5815c@gmail.com>

On Fri, May 08 2026 at 13:15, Tony Rodriguez wrote:
> Just confirmed on my end today.  This regression also impacts both 
> SPARC64 S7-2 and SPARC64 T7-1 on v7.0.4 and v7.1-rc2 as well. Different 
> systems using the same exact kernels.
>
> ** Please see points (A1) (A2) (B1) (B2)
>
> Once again, I am not experiencing such issues when "my patch" (link 
> below) is added to address this regression.
>
> https://github.com/sparclinux/issues/issues/79#issuecomment-4362173884

Github issues are really not helpful.

> PS - On May 2nd 2026 at 9:42 PM: I also sent an email to Thomas Gleixner 
> regarding this issue.  Will be happy to validate any patches from your 
> end regarding this issue, as time permits me to do so.

Sorry, that mail got lost as it was in reply to a random other archived
thread which has absolutely nothing to do with the problem at hand.

I just looked at your github thing. Despite your changelog claiming
otherwise your "fix" breaks the DoS protection completely. It's a
polished version of a revert.

It also lacks a proper root cause analysis. This list:

    - skipped programming events when delta <= min_delta_ns
    - changed force semantics for overdue events
    - introduced a sticky next_event_forced state
    - returned success even when no event was programmed

does not qualify and is actually wrong.

The code does not unconditionally skip the programming of events when
delta <= min_delta_ns. It only does so conditionally when the previous
force programmed min_delta_ns event has not been delivered to the kernel
yet, i.e. dev->next_event_forced is still set.

That flag is only set when the minimal value has been successfully
programmed and it _is_ cleared on the next timer interrupt, which should
obviously happen due to this minimal delta programming. It is also
cleared when a new event > min_delta_ns is successfully programmed
_before_ the previous one was delivered.

IOW, the core code programmed the hardware with the min_delta_ns
(min_delta_ticks) timeout and the SPARC clockevents driver returned
success (0). Now the core code refuses to do further reprogramming with
the min_delta_ns timeout as that would shift the expiry (interrupt)
further out until the interrupt actually is delivered or some other
event which is not below the min_delta_ns threshold is programmed.

So let's assume that this logic is causing the problem, then the only
explanation for the observed behaviour is that the expected interrupt
due to a forced min_delta_ns programming is never delivered. 

That made me look into the SPARC specific set_next_event() functions. I
don't know which variant your machines are using, but all of them have
the same underlying problem. The interrupt is based on a equal
comparator, so the programming logic for each of the tick variants is:

$variant_add_compare(delta)
{
      cmp = read_timer() + delta;
      write_comparator(cmp);
      now = read_timer();
      return (now - cmp) > 0;
}

and the actual set_next_event() function which is invoked from the core
code does:

   return tick_operations.add_compare(delta) ? -ETIME : 0;

IOW, when the timer read _after_ writing the comparator value is ahead
of the comparator value the operation failed. Looks about right in
theory.

But then there is the reality of hardware which ruins everything. I've
banged my head against the wall many years ago when debugging a similar
issue with the x86 HPET which has the same hardware design failure of
using a compare equal comparator instead of having a compare less than
equal one. See the lengthy comment in hpet_clkevt_set_next_event() for
further information.

Can you apply the debug patch below, which will disable tracing once it
hits the hung task detector and then retrieve the trace?

If that's not possible as the system is unresponsive, then please add
'ftrace_dump_on_oops' on the kernel command line or enable it after boot
in /proc/sys/kernel and let the kernel panic when it hits the hung task
detector fail.

Thanks,

        tglx
---
--- a/arch/sparc/kernel/time_64.c
+++ b/arch/sparc/kernel/time_64.c
@@ -732,8 +732,10 @@ void __irq_entry timer_interrupt(int irq
 	if (unlikely(!evt->event_handler)) {
 		printk(KERN_WARNING
 		       "Spurious SPARC64 timer interrupt on cpu %d\n", cpu);
-	} else
+	} else {
+		trace_printk("Invoking handler %pS\n", evt->event_handler);
 		evt->event_handler(evt);
+	}
 
 	irq_exit();
 
--- a/kernel/hung_task.c
+++ b/kernel/hung_task.c
@@ -248,6 +248,7 @@ static void hung_task_info(struct task_s
 	 * accordingly
 	 */
 	if (sysctl_hung_task_warnings || hung_task_call_panic) {
+		tracing_off();
 		if (sysctl_hung_task_warnings > 0)
 			sysctl_hung_task_warnings--;
 		pr_err("INFO: task %s:%d blocked%s for more than %ld seconds.\n",
--- a/kernel/time/clockevents.c
+++ b/kernel/time/clockevents.c
@@ -370,18 +370,22 @@ int clockevents_program_event(struct clo
 		delta = min(delta, (int64_t) dev->max_delta_ns);
 		cycles = ((u64)delta * dev->mult) >> dev->shift;
 		if (!dev->set_next_event((unsigned long) cycles, dev)) {
+			trace_printk("Successfully programmed %lld %lld\n", expires, delta);
 			dev->next_event_forced = 0;
 			return 0;
 		}
 	}
 
-	if (dev->next_event_forced)
+	if (dev->next_event_forced) {
+		trace_printk("Skipping %lld %lld\n", expires, delta);
 		return 0;
+	}
 
 	if (dev->set_next_event(dev->min_delta_ticks, dev)) {
 		if (!force || clockevents_program_min_delta(dev))
 			return -ETIME;
 	}
+	trace_printk("Force programmed min delta %lld %lld\n", expires, delta);
 	dev->next_event_forced = 1;
 	return 0;
 }

  parent reply	other threads:[~2026-05-10 21:29 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <4dd98a32-d1d6-43de-910c-7e487503177e@leemhuis.info>
2026-05-08  5:51 ` the stuttering regression in 7.0: should I have done something different? John Paul Adrian Glaubitz
2026-05-08  6:33   ` Thorsten Leemhuis
     [not found]     ` <D5D19776-C809-4284-9417-F9A860877B98@gmail.com>
2026-05-08  7:50       ` Thorsten Leemhuis
2026-05-08 20:15         ` Tony Rodriguez
2026-05-08 20:21           ` Tony Rodriguez
2026-05-10 21:29           ` Thomas Gleixner [this message]
2026-05-11  3:13             ` Tony Rodriguez

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=871pfj9cmj.ffs@tglx \
    --to=tglx@kernel.org \
    --cc=glaubitz@physik.fu-berlin.de \
    --cc=gregkh@linuxfoundation.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=regressions@leemhuis.info \
    --cc=regressions@lists.linux.dev \
    --cc=torvalds@linux-foundation.org \
    --cc=unixpro1970@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox