Re: [PATCH v2 2/2] irqchip/riscv-imsic: Complete vector moves without a timer

[Thomas Gleixner <tglx@kernel.org>]

On Thu, Nov 12 2026 at 07:54, Troy Mitchell wrote:
> When a vector moves, the new CPU clears move_prev only after the old CPU
> has released the old vector. If the new CPU syncs first, it arms a timer
> on itself and retries a jiffy later. That CPU is often the one entering
> idle and taking over the broadcast interrupt, with its tick already
> stopped, so the timer is not programmed.

When that timer expires _before_ the earliest programmed timer on the
remote CPU then that remote CPU is kicked out of idle to reevaluate.

So your problem description does not make any sense at all.

If the timer code fails to kick that remote CPU out for re-evaluation
then this needs to be fixed and not worked around somewhere else with
dubious claims.

> The move then completes only on the next interrupt of that CPU, which
> took up to 0.5s on an idle SpacemiT K3. Until then the IRQ is still
> marked as moving and further affinity changes of it return -EBUSY.

What's the problem with that? The notification to cleanup can be
arbitrarily long delayed. See below.

> Let the old CPU queue the sync work on the new CPU once it has released
> the old vector, and remove the retry timer. Wait only while the old
> vector still points to the new vector, since a released old vector may
> already belong to another move.

That's even more bogus. The old CPU does not queue anything.

The issue with these completely misdesigned interrupt controllers is
that the only safe way to migrate an interrupt is to do the following:

   1) Defer the affinity change to the interrupt context of the original
      target CPU.

   2) When the next interrupt arrives on the original target CPU switch
      the affinity over to the new target CPU (carefully...)

   3) If the interrupt is pending again on the original target CPU after
      switching affinity send an IPI to the new target to kick that
      vector.

   4) When the new target CPU receives the first interrupt (it does not
      matter whether it's from the device or from the IPI in #3) it
      tells the original target CPU to cleanup the stale
      associations. That's what the timer is for.

So the requirement is that the new target CPU has to receive an
interrupt to be able to release the associations on the original target
CPU and to re-enable affinity settings. This can obviously cause
arbitrarily long delays if #3 is not the case as the cleanup has to wait
for #4.

That has absolutely nothing to do with the timer and changing that to an
IPI as that only shortens the cleanup delay from _ONE_ jiffies to almost
instant. Who the hell cares?

Why is that requirement there in the first place?

  Because there neither is a guarantee that the affinity change takes
  place immediately when the update happens nor there is a guarantee
  that an interrupt message which was sent to the old CPU is delivered
  before the pending check in #3 happens. There are arbitrary device
  specific delays between the point where the device reads the target
  message and the point where it is sent and fabrics have delays too.

A lot of people spent a massive amount of time and head scratching to
come up with a solution for that especially for non-maskable MSI
interrupts.

Coming back to your change.

Of course this can be done with an IPI from the _new_ target CPU to
the _old_ target CPU, but only under one condition:

    The IPI vector cannot be higher prioritized than the to be cleaned
    up device vector.

Why?

Contrary to level interrupts which stay asserted until the interrupt
handler resets the condition, edge interrupts (e.g. PCI/MSI[-X]) are
from a device POV fire and forget. There is no guarantee that a device
waits for handler response before sending the next one.

So due to this and the arbitrary hardware (device/fabric) delays you can
end up in a situation where there is an interrupt pending either in the
old target or in the new target after #3 above or on both.

For the on both case assume that the new target CPU wins the race to
handle it and it sends the IPI to the old target CPU, which is currently
in an interupt disabled region. Once the old target CPU re-enables
interrupts it handles the highest priority interrupt first, which is the
IPI. Now the IPI handler detects that the lower priority device interrupt is
pending and retriggers the cleanup IPI on itself ---> livelock.

Sure, you can make that so that it ignores that situation and waits for
the new target CPU to re-issue the clenaup IPI on the next interrupt,
but that's still not solving the arbitrary delay problem, i.e. waiting
for that next interrupt which might take forever....

I'm still puzzled that at least _two_ decades after it was publicly
known that the x86 APIC is a design disaster hardware people still
insisted on making the same mistakes over and over. It's not only RISCV
folks, there are more insanities like that in other architectures...

But please read back on the initial IMSIC discussion which led up to
merging this driver. There is a ton of more context there which I'm not
going to repeat.

Thanks,

        tglx




_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv