From mboxrd@z Thu Jan  1 00:00:00 1970
Received: from eggs.gnu.org ([2001:4830:134:3::10]:42318)
	by lists.gnu.org with esmtp (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z4Vd1-0006my-Hh
	for qemu-devel@nongnu.org; Mon, 15 Jun 2015 10:44:48 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z4Vct-0005Rk-4o
	for qemu-devel@nongnu.org; Mon, 15 Jun 2015 10:44:44 -0400
Received: from mxout-1k.itc.hs-rm.de ([195.72.102.133]:55305)
	by eggs.gnu.org with esmtp (Exim 4.71)
	(envelope-from <Alexander.Zuepke@hs-rm.de>) id 1Z4Vcs-0005Qa-Ql
	for qemu-devel@nongnu.org; Mon, 15 Jun 2015 10:44:39 -0400
Message-ID: <557EE4D2.5010202@hs-rm.de>
Date: Mon, 15 Jun 2015 16:44:34 +0200
From: =?UTF-8?B?QWxleCBaw7xwa2U=?= <alexander.zuepke@hs-rm.de>
MIME-Version: 1.0
References: <557B0B21.2030009@hs-rm.de>
	<CAFEAcA8iCmgwgmhTkRryTyVBSxS+Cbh6bGSLqt7EPM028F1tUg@mail.gmail.com>
In-Reply-To: <CAFEAcA8iCmgwgmhTkRryTyVBSxS+Cbh6bGSLqt7EPM028F1tUg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Subject: Re: [Qemu-devel] QEMU ARM SMP: IPI delivery delayed until next main
 loop event // how to improve IPI latency?
List-Id: <qemu-devel.nongnu.org>
List-Unsubscribe: <https://lists.nongnu.org/mailman/options/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=unsubscribe>
List-Archive: <http://lists.nongnu.org/archive/html/qemu-devel>
List-Post: <mailto:qemu-devel@nongnu.org>
List-Help: <mailto:qemu-devel-request@nongnu.org?subject=help>
List-Subscribe: <https://lists.nongnu.org/mailman/listinfo/qemu-devel>,
	<mailto:qemu-devel-request@nongnu.org?subject=subscribe>
To: Peter Maydell <peter.maydell@linaro.org>
Cc: QEMU Developers <qemu-devel@nongnu.org>

Hi Peter,

Am 12.06.2015 um 20:03 schrieb Peter Maydell:
> On 12 June 2015 at 17:38, Alex Z=C3=BCpke <alexander.zuepke@hs-rm.de> w=
rote:
>> Hi,
>>
>> I'm benchmarking some IPI (=3D=3D inter-processor-interrupt) synchroni=
zation stuff of my custom kernel on QEMU ARM (qemu-system-arm -M vexpress=
-a15 -smp 2) and ran into the following problem: pending IPIs are delayed=
 until the QEMU main loop receives an event (for example the timer interr=
upt expires or I press a key on the console).
>>
>> The following timing diagram tries to show this:
>>
>>   CPU #0                       CPU #1
>>   =3D=3D=3D=3D=3D=3D                       =3D=3D=3D=3D=3D=3D
>>   ... other stuff ...          WFI (wait for interrupt, like x86 "HLT"=
)
>>   send SGI in MPCore
>>   polls for completeness
>>                  <time passes ...>
>>   polls ...
>>                  <... and passes ...>
>>   still polls ...
>>                  <... and passes ...>
>>   still polls ...
>>                  <... and passes ...>
>>
>>
>>                  <timer interrupt expires>
>>                  <now QEMU switches to CPU #1>
>>                                receives IPI
>>                                signals completeness
>>                                WFI
>>                  <QEMU switches to CPU #0>
>>   polling done
>>   process timer interrupt
>>   ...
>=20
> Right. The problem is that we don't have any way of telling
> that CPU 0 is just sat busy waiting for CPU 1.
>=20
>> It works as expects (I get thousands of IPIs per second now), but
>> it does not "feel right", so is there a better way to improve the
>> responsiveness of IPI handling in QEMU?
>=20
> Probably the best approach would be to have something in
> arm_cpu_set_irq() which says "if we are CPU X and we've
> just caused an interrupt to be set for CPU Y, then we
> should ourselves yield back to the main loop".
>=20
> Something like this, maybe, though I have done no more testing
> than checking it doesn't actively break kernel booting :-)


Thanks! One more check for "level" is needed to get it work:

--- a/target-arm/cpu.c
+++ b/target-arm/cpu.c
@@ -325,6 +325,18 @@ static void arm_cpu_set_irq(void *opaque, int irq, i=
nt level)
     default:
         hw_error("arm_cpu_set_irq: Bad interrupt line %d\n", irq);
     }
+
+    /* If we are currently executing code for CPU X, and this
+     * CPU we've just triggered an interrupt on is CPU Y, then
+     * make CPU X yield control back to the main loop at the
+     * end of the TB it's currently executing.
+     * This avoids problems where the interrupt was an IPI
+     * and CPU X would otherwise sit busy looping for the rest
+     * of its timeslice because Y hasn't had a chance to run.
+     */
+    if (level && current_cpu && current_cpu !=3D cs) {
+        cpu_exit(current_cpu);
+    }
 }
=20
 static void arm_cpu_kvm_set_irq(void *opaque, int irq, int level)


Do you need a testcase for this?



Best regards
Alex