Re: [Qemu-devel] QEMU ARM SMP: IPI delivery delayed until next main loop event // how to improve IPI latency?

[Peter Maydell <peter.maydell@linaro.org>]

On 12 June 2015 at 17:38, Alex Züpke <alexander.zuepke@hs-rm.de> wrote:
> Hi,
>
> I'm benchmarking some IPI (== inter-processor-interrupt) synchronization stuff of my custom kernel on QEMU ARM (qemu-system-arm -M vexpress-a15 -smp 2) and ran into the following problem: pending IPIs are delayed until the QEMU main loop receives an event (for example the timer interrupt expires or I press a key on the console).
>
> The following timing diagram tries to show this:
>
>   CPU #0                       CPU #1
>   ======                       ======
>   ... other stuff ...          WFI (wait for interrupt, like x86 "HLT")
>   send SGI in MPCore
>   polls for completeness
>                  <time passes ...>
>   polls ...
>                  <... and passes ...>
>   still polls ...
>                  <... and passes ...>
>   still polls ...
>                  <... and passes ...>
>
>
>                  <timer interrupt expires>
>                  <now QEMU switches to CPU #1>
>                                receives IPI
>                                signals completeness
>                                WFI
>                  <QEMU switches to CPU #0>
>   polling done
>   process timer interrupt
>   ...

Right. The problem is that we don't have any way of telling
that CPU 0 is just sat busy waiting for CPU 1.

> It works as expects (I get thousands of IPIs per second now), but
> it does not "feel right", so is there a better way to improve the
> responsiveness of IPI handling in QEMU?

Probably the best approach would be to have something in
arm_cpu_set_irq() which says "if we are CPU X and we've
just caused an interrupt to be set for CPU Y, then we
should ourselves yield back to the main loop".

Something like this, maybe, though I have done no more testing
than checking it doesn't actively break kernel booting :-)

--- a/target-arm/cpu.c
+++ b/target-arm/cpu.c
@@ -325,6 +325,18 @@ static void arm_cpu_set_irq(void *opaque, int
irq, int level)
     default:
         hw_error("arm_cpu_set_irq: Bad interrupt line %d\n", irq);
     }
+
+    /* If we are currently executing code for CPU X, and this
+     * CPU we've just triggered an interrupt on is CPU Y, then
+     * make CPU X yield control back to the main loop at the
+     * end of the TB it's currently executing.
+     * This avoids problems where the interrupt was an IPI
+     * and CPU X would otherwise sit busy looping for the rest
+     * of its timeslice because Y hasn't had a chance to run.
+     */
+    if (current_cpu && current_cpu != cs) {
+        cpu_exit(current_cpu);
+    }
 }

 static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)


-- PMM