From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:37011)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z3Ryy-0006Ow-QC
	for qemu-devel@nongnu.org; Fri, 12 Jun 2015 12:39:05 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z3Ryv-0004Zl-K6
	for qemu-devel@nongnu.org; Fri, 12 Jun 2015 12:39:04 -0400
Received: from mxout-1k.itc.hs-rm.de ([195.72.102.133]:51568)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z3Ryv-0004VB-As
	for qemu-devel@nongnu.org; Fri, 12 Jun 2015 12:39:01 -0400
Received: from EXCHANGE-3K.hds.local (exchange-3k.itc.hs-rm.de [10.10.62.124])
	by mxout-1k.itc.hs-rm.de (8.14.4/8.14.4) with ESMTP id t5CGcvUv002396
	for <qemu-devel@nongnu.org>; Fri, 12 Jun 2015 18:38:57 +0200
	(envelope-from Alexander.Zuepke@hs-rm.de)
Message-ID: <557B0B21.2030009@hs-rm.de>
Date: Fri, 12 Jun 2015 18:38:57 +0200
From: =?UTF-8?B?QWxleCBaw7xwa2U=?= <alexander.zuepke@hs-rm.de>
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
Subject: [Qemu-devel] QEMU ARM SMP: IPI delivery delayed until next main
 loop event // how to improve IPI latency?
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: qemu-devel@nongnu.org

Hi,

I'm benchmarking some IPI (== inter-processor-interrupt) synchronization stuff of my custom kernel on QEMU ARM (qemu-system-arm -M vexpress-a15 -smp 2) and ran into the following problem: pending IPIs are delayed until the QEMU main loop receives an event (for example the timer interrupt expires or I press a key on the console).

The following timing diagram tries to show this:

  CPU #0                       CPU #1
  ======                       ======
  ... other stuff ...          WFI (wait for interrupt, like x86 "HLT")
  send SGI in MPCore
  polls for completeness
                 <time passes ...>
  polls ...
                 <... and passes ...>
  still polls ...
                 <... and passes ...>
  still polls ...
                 <... and passes ...>

  
                 <timer interrupt expires>
                 <now QEMU switches to CPU #1>
                               receives IPI
                               signals completeness
                               WFI
                 <QEMU switches to CPU #0>
  polling done
  process timer interrupt
  ...


My timer is setup to generate an interrupt once a second, so I only get 1 IPI interrupt per second on QEMU. When I run the test on real hardware (i.MX6Q), I get millions of IPIs instead.


I tried to "fix" this by forcing QEMU back into the main loop and added a call to qemu_notify_event() in the IPI-sending path of the ARM interrupt controller:


diff --git a/hw/intc/arm_gic.c b/hw/intc/arm_gic.c
index c1d2e70..20dba75 100644
--- a/hw/intc/arm_gic.c
+++ b/hw/intc/arm_gic.c
@@ -21,6 +21,7 @@
 #include "hw/sysbus.h"
 #include "gic_internal.h"
 #include "qom/cpu.h"
+#include "qemu/main-loop.h"
 
 //#define DEBUG_GIC
 
@@ -898,6 +899,7 @@ static void gic_dist_writel(void *opaque, hwaddr offset,
             target_cpu = ctz32(mask);
         }
         gic_update(s);
+        qemu_notify_event();
         return;
     }
     gic_dist_writew(opaque, offset, value & 0xffff, attrs);



It works as expects (I get thousands of IPIs per second now), but it does not "feel right", so is there a better way to improve the responsiveness of IPI handling in QEMU?

Best regards
Alex