From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0113.outbound.protection.outlook.com [157.56.111.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 6C6E01A06C9 for ; Wed, 22 Apr 2015 22:06:55 +1000 (AEST) Message-ID: <55378EC4.2080302@freescale.com> Date: Wed, 22 Apr 2015 15:06:28 +0300 From: Purcareata Bogdan MIME-Version: 1.0 To: Scott Wood Subject: Re: [PATCH 0/2] powerpc/kvm: Enable running guests on RT Linux References: <1424251955-308-1-git-send-email-bogdan.purcareata@freescale.com> <54E73A6C.9080500@suse.de> <54E740E7.5090806@redhat.com> <54E74A8C.30802@linutronix.de> <1424734051.4698.17.camel@freescale.com> <54EF196E.4090805@redhat.com> <54EF2025.80404@linutronix.de> <1424999159.4698.78.camel@freescale.com> <55158E6D.40304@freescale.com> <1428016310.22867.289.camel@freescale.com> <551E4A41.1080705@freescale.com> <1428096375.22867.369.camel@freescale.com> <55262DD3.2050707@freescale.com> <1428623611.22867.561.camel@freescale.com> <5534DAA4.3050809@freescale.com> <1429577566.4352.68.camel@freescale.com> In-Reply-To: <1429577566.4352.68.camel@freescale.com> Content-Type: text/plain; charset="utf-8"; format=flowed Cc: Laurentiu Tudor , linux-rt-users@vger.kernel.org, Sebastian Andrzej Siewior , Alexander Graf , linux-kernel@vger.kernel.org, Bogdan Purcareata , mihai.caraman@freescale.com, Paolo Bonzini , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 21.04.2015 03:52, Scott Wood wrote: > On Mon, 2015-04-20 at 13:53 +0300, Purcareata Bogdan wrote: >> There was a weird situation for .kvmppc_mpic_set_epr - its corresponding inner >> function is kvmppc_set_epr, which is a static inline. Removing the static inline >> yields a compiler crash (Segmentation fault (core dumped) - >> scripts/Makefile.build:441: recipe for target 'arch/powerpc/kvm/kvm.o' failed), >> but that's a different story, so I just let it be for now. Point is the time may >> include other work after the lock has been released, but before the function >> actually returned. I noticed this was the case for .kvm_set_msi, which could >> work up to 90 ms, not actually under the lock. This made me change what I'm >> looking at. > > kvm_set_msi does pretty much nothing outside the lock -- I suspect > you're measuring an interrupt that happened as soon as the lock was > released. That's exactly right. I've seen things like a timer interrupt occuring right after the spinlock_irqrestore, but before kvm_set_msi actually returned. [...] >> Or perhaps a different stress scenario involving a lot of VCPUs >> and external interrupts? > > You could instrument the MPIC code to find out how many loop iterations > you maxed out on, and compare that to the theoretical maximum. Numbers are pretty low, and I'll try to explain based on my observations. The problematic section in openpic_update_irq is this [1], since it loops through all VCPUs, and IRQ_local_pipe further calls IRQ_check, which loops through all pending interrupts for a VCPU [2]. The guest interfaces are virtio-vhostnet, which are based on MSI (/proc/interrupts in guest shows they are MSI). For external interrupts to the guest, the irq_source destmask is currently 0, and last_cpu is 0 (unitialized), so [1] will go on and deliver the interrupt directly and unicast (no VCPUs loop). I activated the pr_debugs in arch/powerpc/kvm/mpic.c, to see how many interrupts are actually pending for the destination VCPU. At most, there were 3 interrupts - n_IRQ = {224,225,226} - even for 24 flows of ping flood. I understand that guest virtio interrupts are cascaded over 1 or a couple of shared MSI interrupts. So worst case, in this scenario, was checking the priorities for 3 pending interrupts for 1 VCPU. Something like this (some of my prints included): [61010.582033] openpic_update_irq: destmask 1 last_cpu 0 [61010.582034] openpic_update_irq: Only one CPU is allowed to receive this IRQ [61010.582036] IRQ_local_pipe: IRQ 224 active 0 was 1 [61010.582037] IRQ_check: irq 226 set ivpr_pr=8 pr=-1 [61010.582038] IRQ_check: irq 225 set ivpr_pr=8 pr=-1 [61010.582039] IRQ_check: irq 224 set ivpr_pr=8 pr=-1 It would be really helpful to get your comments regarding whether these are realistical number for everyday use, or they are relevant only to this particular scenario. - Can these interrupts be used in directed delivery, so that the destination mask can include multiple VCPUs? The MPIC manual states that timer and IPI interrupts are supported for directed delivery, altough I'm not sure how much of this is used in the emulation. I know that kvmppc uses the decrementer outside of the MPIC. - How are virtio interrupts cascaded over the shared MSI interrupts? /proc/device-tree/soc@e0000000/msi@41600/interrupts in the guest shows 8 values - 224 - 231 - so at most there might be 8 pending interrupts in IRQ_check, is that correct? Looking forward to your feedback. [1] http://lxr.free-electrons.com/source/arch/powerpc/kvm/mpic.c#L454 [2] http://lxr.free-electrons.com/source/arch/powerpc/kvm/mpic.c#L303 [3] https://www-01.ibm.com/chips/techlib/techlib.nsf/techdocs/F27971551C9EED8E8525774A0048770A/$file/mpic_db_05_16_2011.pdf Best regards, Bogdan P. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Purcareata Bogdan Subject: Re: [PATCH 0/2] powerpc/kvm: Enable running guests on RT Linux Date: Wed, 22 Apr 2015 15:06:28 +0300 Message-ID: <55378EC4.2080302@freescale.com> References: <1424251955-308-1-git-send-email-bogdan.purcareata@freescale.com> <54E73A6C.9080500@suse.de> <54E740E7.5090806@redhat.com> <54E74A8C.30802@linutronix.de> <1424734051.4698.17.camel@freescale.com> <54EF196E.4090805@redhat.com> <54EF2025.80404@linutronix.de> <1424999159.4698.78.camel@freescale.com> <55158E6D.40304@freescale.com> <1428016310.22867.289.camel@freescale.com> <551E4A41.1080705@freescale.com> <1428096375.22867.369.camel@freescale.com> <55262DD3.2050707@freescale.com> <1428623611.22867.561.camel@freescale.com> <5534DAA4.3050809@freescale.com> <1429577566.4352.68.camel@freescale.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Cc: Laurentiu Tudor , linux-rt-users@vger.kernel.org, Sebastian Andrzej Siewior , Alexander Graf , linux-kernel@vger.kernel.org, Bogdan Purcareata , mihai.caraman@freescale.com, Paolo Bonzini , Thomas Gleixner , linuxppc-dev@lists.ozlabs.org To: Scott Wood Return-path: In-Reply-To: <1429577566.4352.68.camel@freescale.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" List-Id: linux-rt-users.vger.kernel.org T24gMjEuMDQuMjAxNSAwMzo1MiwgU2NvdHQgV29vZCB3cm90ZToKPiBPbiBNb24sIDIwMTUtMDQt MjAgYXQgMTM6NTMgKzAzMDAsIFB1cmNhcmVhdGEgQm9nZGFuIHdyb3RlOgo+PiBUaGVyZSB3YXMg YSB3ZWlyZCBzaXR1YXRpb24gZm9yIC5rdm1wcGNfbXBpY19zZXRfZXByIC0gaXRzIGNvcnJlc3Bv bmRpbmcgaW5uZXIKPj4gZnVuY3Rpb24gaXMga3ZtcHBjX3NldF9lcHIsIHdoaWNoIGlzIGEgc3Rh dGljIGlubGluZS4gUmVtb3ZpbmcgdGhlIHN0YXRpYyBpbmxpbmUKPj4geWllbGRzIGEgY29tcGls ZXIgY3Jhc2ggKFNlZ21lbnRhdGlvbiBmYXVsdCAoY29yZSBkdW1wZWQpIC0KPj4gc2NyaXB0cy9N YWtlZmlsZS5idWlsZDo0NDE6IHJlY2lwZSBmb3IgdGFyZ2V0ICdhcmNoL3Bvd2VycGMva3ZtL2t2 bS5vJyBmYWlsZWQpLAo+PiBidXQgdGhhdCdzIGEgZGlmZmVyZW50IHN0b3J5LCBzbyBJIGp1c3Qg bGV0IGl0IGJlIGZvciBub3cuIFBvaW50IGlzIHRoZSB0aW1lIG1heQo+PiBpbmNsdWRlIG90aGVy IHdvcmsgYWZ0ZXIgdGhlIGxvY2sgaGFzIGJlZW4gcmVsZWFzZWQsIGJ1dCBiZWZvcmUgdGhlIGZ1 bmN0aW9uCj4+IGFjdHVhbGx5IHJldHVybmVkLiBJIG5vdGljZWQgdGhpcyB3YXMgdGhlIGNhc2Ug Zm9yIC5rdm1fc2V0X21zaSwgd2hpY2ggY291bGQKPj4gd29yayB1cCB0byA5MCBtcywgbm90IGFj dHVhbGx5IHVuZGVyIHRoZSBsb2NrLiBUaGlzIG1hZGUgbWUgY2hhbmdlIHdoYXQgSSdtCj4+IGxv b2tpbmcgYXQuCj4KPiBrdm1fc2V0X21zaSBkb2VzIHByZXR0eSBtdWNoIG5vdGhpbmcgb3V0c2lk ZSB0aGUgbG9jayAtLSBJIHN1c3BlY3QKPiB5b3UncmUgbWVhc3VyaW5nIGFuIGludGVycnVwdCB0 aGF0IGhhcHBlbmVkIGFzIHNvb24gYXMgdGhlIGxvY2sgd2FzCj4gcmVsZWFzZWQuCgpUaGF0J3Mg ZXhhY3RseSByaWdodC4gSSd2ZSBzZWVuIHRoaW5ncyBsaWtlIGEgdGltZXIgaW50ZXJydXB0IG9j Y3VyaW5nIHJpZ2h0IAphZnRlciB0aGUgc3BpbmxvY2tfaXJxcmVzdG9yZSwgYnV0IGJlZm9yZSBr dm1fc2V0X21zaSBhY3R1YWxseSByZXR1cm5lZC4KClsuLi5dCgo+PiAgIE9yIHBlcmhhcHMgYSBk aWZmZXJlbnQgc3RyZXNzIHNjZW5hcmlvIGludm9sdmluZyBhIGxvdCBvZiBWQ1BVcwo+PiBhbmQg ZXh0ZXJuYWwgaW50ZXJydXB0cz8KPgo+IFlvdSBjb3VsZCBpbnN0cnVtZW50IHRoZSBNUElDIGNv ZGUgdG8gZmluZCBvdXQgaG93IG1hbnkgbG9vcCBpdGVyYXRpb25zCj4geW91IG1heGVkIG91dCBv biwgYW5kIGNvbXBhcmUgdGhhdCB0byB0aGUgdGhlb3JldGljYWwgbWF4aW11bS4KCk51bWJlcnMg YXJlIHByZXR0eSBsb3csIGFuZCBJJ2xsIHRyeSB0byBleHBsYWluIGJhc2VkIG9uIG15IG9ic2Vy dmF0aW9ucy4KClRoZSBwcm9ibGVtYXRpYyBzZWN0aW9uIGluIG9wZW5waWNfdXBkYXRlX2lycSBp cyB0aGlzIFsxXSwgc2luY2UgaXQgbG9vcHMgCnRocm91Z2ggYWxsIFZDUFVzLCBhbmQgSVJRX2xv Y2FsX3BpcGUgZnVydGhlciBjYWxscyBJUlFfY2hlY2ssIHdoaWNoIGxvb3BzIAp0aHJvdWdoIGFs bCBwZW5kaW5nIGludGVycnVwdHMgZm9yIGEgVkNQVSBbMl0uCgpUaGUgZ3Vlc3QgaW50ZXJmYWNl cyBhcmUgdmlydGlvLXZob3N0bmV0LCB3aGljaCBhcmUgYmFzZWQgb24gTVNJIAooL3Byb2MvaW50 ZXJydXB0cyBpbiBndWVzdCBzaG93cyB0aGV5IGFyZSBNU0kpLiBGb3IgZXh0ZXJuYWwgaW50ZXJy dXB0cyB0byB0aGUgCmd1ZXN0LCB0aGUgaXJxX3NvdXJjZSBkZXN0bWFzayBpcyBjdXJyZW50bHkg MCwgYW5kIGxhc3RfY3B1IGlzIDAgKHVuaXRpYWxpemVkKSwgCnNvIFsxXSB3aWxsIGdvIG9uIGFu ZCBkZWxpdmVyIHRoZSBpbnRlcnJ1cHQgZGlyZWN0bHkgYW5kIHVuaWNhc3QgKG5vIFZDUFVzIGxv b3ApLgoKSSBhY3RpdmF0ZWQgdGhlIHByX2RlYnVncyBpbiBhcmNoL3Bvd2VycGMva3ZtL21waWMu YywgdG8gc2VlIGhvdyBtYW55IGludGVycnVwdHMgCmFyZSBhY3R1YWxseSBwZW5kaW5nIGZvciB0 aGUgZGVzdGluYXRpb24gVkNQVS4gQXQgbW9zdCwgdGhlcmUgd2VyZSAzIGludGVycnVwdHMgCi0g bl9JUlEgPSB7MjI0LDIyNSwyMjZ9IC0gZXZlbiBmb3IgMjQgZmxvd3Mgb2YgcGluZyBmbG9vZC4g SSB1bmRlcnN0YW5kIHRoYXQgCmd1ZXN0IHZpcnRpbyBpbnRlcnJ1cHRzIGFyZSBjYXNjYWRlZCBv dmVyIDEgb3IgYSBjb3VwbGUgb2Ygc2hhcmVkIE1TSSBpbnRlcnJ1cHRzLgoKU28gd29yc3QgY2Fz ZSwgaW4gdGhpcyBzY2VuYXJpbywgd2FzIGNoZWNraW5nIHRoZSBwcmlvcml0aWVzIGZvciAzIHBl bmRpbmcgCmludGVycnVwdHMgZm9yIDEgVkNQVS4gU29tZXRoaW5nIGxpa2UgdGhpcyAoc29tZSBv ZiBteSBwcmludHMgaW5jbHVkZWQpOgoKWzYxMDEwLjU4MjAzM10gb3BlbnBpY191cGRhdGVfaXJx OiBkZXN0bWFzayAxIGxhc3RfY3B1IDAKWzYxMDEwLjU4MjAzNF0gb3BlbnBpY191cGRhdGVfaXJx OiBPbmx5IG9uZSBDUFUgaXMgYWxsb3dlZCB0byByZWNlaXZlIHRoaXMgSVJRCls2MTAxMC41ODIw MzZdIElSUV9sb2NhbF9waXBlOiBJUlEgMjI0IGFjdGl2ZSAwIHdhcyAxCls2MTAxMC41ODIwMzdd IElSUV9jaGVjazogaXJxIDIyNiBzZXQgaXZwcl9wcj04IHByPS0xCls2MTAxMC41ODIwMzhdIElS UV9jaGVjazogaXJxIDIyNSBzZXQgaXZwcl9wcj04IHByPS0xCls2MTAxMC41ODIwMzldIElSUV9j aGVjazogaXJxIDIyNCBzZXQgaXZwcl9wcj04IHByPS0xCgpJdCB3b3VsZCBiZSByZWFsbHkgaGVs cGZ1bCB0byBnZXQgeW91ciBjb21tZW50cyByZWdhcmRpbmcgd2hldGhlciB0aGVzZSBhcmUgCnJl YWxpc3RpY2FsIG51bWJlciBmb3IgZXZlcnlkYXkgdXNlLCBvciB0aGV5IGFyZSByZWxldmFudCBv bmx5IHRvIHRoaXMgCnBhcnRpY3VsYXIgc2NlbmFyaW8uCgotIENhbiB0aGVzZSBpbnRlcnJ1cHRz IGJlIHVzZWQgaW4gZGlyZWN0ZWQgZGVsaXZlcnksIHNvIHRoYXQgdGhlIGRlc3RpbmF0aW9uIApt YXNrIGNhbiBpbmNsdWRlIG11bHRpcGxlIFZDUFVzPyBUaGUgTVBJQyBtYW51YWwgc3RhdGVzIHRo YXQgdGltZXIgYW5kIElQSSAKaW50ZXJydXB0cyBhcmUgc3VwcG9ydGVkIGZvciBkaXJlY3RlZCBk ZWxpdmVyeSwgYWx0b3VnaCBJJ20gbm90IHN1cmUgaG93IG11Y2ggb2YgCnRoaXMgaXMgdXNlZCBp biB0aGUgZW11bGF0aW9uLiBJIGtub3cgdGhhdCBrdm1wcGMgdXNlcyB0aGUgZGVjcmVtZW50ZXIg b3V0c2lkZSAKb2YgdGhlIE1QSUMuCgotIEhvdyBhcmUgdmlydGlvIGludGVycnVwdHMgY2FzY2Fk ZWQgb3ZlciB0aGUgc2hhcmVkIE1TSSBpbnRlcnJ1cHRzPyAKL3Byb2MvZGV2aWNlLXRyZWUvc29j QGUwMDAwMDAwL21zaUA0MTYwMC9pbnRlcnJ1cHRzIGluIHRoZSBndWVzdCBzaG93cyA4IHZhbHVl cyAKLSAyMjQgLSAyMzEgLSBzbyBhdCBtb3N0IHRoZXJlIG1pZ2h0IGJlIDggcGVuZGluZyBpbnRl cnJ1cHRzIGluIElSUV9jaGVjaywgaXMgCnRoYXQgY29ycmVjdD8KCkxvb2tpbmcgZm9yd2FyZCB0 byB5b3VyIGZlZWRiYWNrLgoKWzFdIGh0dHA6Ly9seHIuZnJlZS1lbGVjdHJvbnMuY29tL3NvdXJj ZS9hcmNoL3Bvd2VycGMva3ZtL21waWMuYyNMNDU0ClsyXSBodHRwOi8vbHhyLmZyZWUtZWxlY3Ry b25zLmNvbS9zb3VyY2UvYXJjaC9wb3dlcnBjL2t2bS9tcGljLmMjTDMwMwpbM10gCmh0dHBzOi8v d3d3LTAxLmlibS5jb20vY2hpcHMvdGVjaGxpYi90ZWNobGliLm5zZi90ZWNoZG9jcy9GMjc5NzE1 NTFDOUVFRDhFODUyNTc3NEEwMDQ4NzcwQS8kZmlsZS9tcGljX2RiXzA1XzE2XzIwMTEucGRmCgpC ZXN0IHJlZ2FyZHMsCkJvZ2RhbiBQLgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwpMaW51eHBwYy1kZXYgbWFpbGluZyBsaXN0CkxpbnV4cHBjLWRldkBsaXN0 cy5vemxhYnMub3JnCmh0dHBzOi8vbGlzdHMub3psYWJzLm9yZy9saXN0aW5mby9saW51eHBwYy1k ZXY= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752481AbbDVMWy (ORCPT ); Wed, 22 Apr 2015 08:22:54 -0400 Received: from mail-bn1on0137.outbound.protection.outlook.com ([157.56.110.137]:35231 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750762AbbDVMWw (ORCPT ); Wed, 22 Apr 2015 08:22:52 -0400 Authentication-Results: freescale.com; dkim=none (message not signed) header.d=none; Message-ID: <55378EC4.2080302@freescale.com> Date: Wed, 22 Apr 2015 15:06:28 +0300 From: Purcareata Bogdan User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.5.0 MIME-Version: 1.0 To: Scott Wood CC: Sebastian Andrzej Siewior , Paolo Bonzini , Alexander Graf , Bogdan Purcareata , , , , , Thomas Gleixner , Laurentiu Tudor Subject: Re: [PATCH 0/2] powerpc/kvm: Enable running guests on RT Linux References: <1424251955-308-1-git-send-email-bogdan.purcareata@freescale.com> <54E73A6C.9080500@suse.de> <54E740E7.5090806@redhat.com> <54E74A8C.30802@linutronix.de> <1424734051.4698.17.camel@freescale.com> <54EF196E.4090805@redhat.com> <54EF2025.80404@linutronix.de> <1424999159.4698.78.camel@freescale.com> <55158E6D.40304@freescale.com> <1428016310.22867.289.camel@freescale.com> <551E4A41.1080705@freescale.com> <1428096375.22867.369.camel@freescale.com> <55262DD3.2050707@freescale.com> <1428623611.22867.561.camel@freescale.com> <5534DAA4.3050809@freescale.com> <1429577566.4352.68.camel@freescale.com> In-Reply-To: <1429577566.4352.68.camel@freescale.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [192.88.166.1] X-ClientProxiedBy: AM3PR03CA035.eurprd03.prod.outlook.com (10.141.191.163) To BN1PR03MB185.namprd03.prod.outlook.com (10.255.200.139) X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:BN1PR03MB185; X-Microsoft-Antispam-PRVS: X-Forefront-Antispam-Report: BMV:1;SFV:NSPM;SFS:(10019020)(6009001)(6049001)(51704005)(377424004)(24454002)(92566002)(122386002)(46102003)(4001350100001)(50466002)(36756003)(77156002)(62966003)(77096005)(15975445007)(2950100001)(40100003)(93886004)(64126003)(23676002)(110136001)(47776003)(83506001)(19580405001)(80316001)(87976001)(66066001)(65956001)(86362001)(87266999)(54356999)(65816999)(50986999)(76176999)(33656002)(19580395003)(42186005)(99136001)(4001430100001)(4001450100001)(42262002);DIR:OUT;SFP:1102;SCL:1;SRVR:BN1PR03MB185;H:[10.171.74.74];FPR:;SPF:None;MLV:sfv;LANG:en; X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(5005006)(5002010);SRVR:BN1PR03MB185;BCL:0;PCL:0;RULEID:;SRVR:BN1PR03MB185; X-Forefront-PRVS: 0554B1F54F X-OriginatorOrg: freescale.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Apr 2015 12:06:44.8504 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN1PR03MB185 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 21.04.2015 03:52, Scott Wood wrote: > On Mon, 2015-04-20 at 13:53 +0300, Purcareata Bogdan wrote: >> There was a weird situation for .kvmppc_mpic_set_epr - its corresponding inner >> function is kvmppc_set_epr, which is a static inline. Removing the static inline >> yields a compiler crash (Segmentation fault (core dumped) - >> scripts/Makefile.build:441: recipe for target 'arch/powerpc/kvm/kvm.o' failed), >> but that's a different story, so I just let it be for now. Point is the time may >> include other work after the lock has been released, but before the function >> actually returned. I noticed this was the case for .kvm_set_msi, which could >> work up to 90 ms, not actually under the lock. This made me change what I'm >> looking at. > > kvm_set_msi does pretty much nothing outside the lock -- I suspect > you're measuring an interrupt that happened as soon as the lock was > released. That's exactly right. I've seen things like a timer interrupt occuring right after the spinlock_irqrestore, but before kvm_set_msi actually returned. [...] >> Or perhaps a different stress scenario involving a lot of VCPUs >> and external interrupts? > > You could instrument the MPIC code to find out how many loop iterations > you maxed out on, and compare that to the theoretical maximum. Numbers are pretty low, and I'll try to explain based on my observations. The problematic section in openpic_update_irq is this [1], since it loops through all VCPUs, and IRQ_local_pipe further calls IRQ_check, which loops through all pending interrupts for a VCPU [2]. The guest interfaces are virtio-vhostnet, which are based on MSI (/proc/interrupts in guest shows they are MSI). For external interrupts to the guest, the irq_source destmask is currently 0, and last_cpu is 0 (unitialized), so [1] will go on and deliver the interrupt directly and unicast (no VCPUs loop). I activated the pr_debugs in arch/powerpc/kvm/mpic.c, to see how many interrupts are actually pending for the destination VCPU. At most, there were 3 interrupts - n_IRQ = {224,225,226} - even for 24 flows of ping flood. I understand that guest virtio interrupts are cascaded over 1 or a couple of shared MSI interrupts. So worst case, in this scenario, was checking the priorities for 3 pending interrupts for 1 VCPU. Something like this (some of my prints included): [61010.582033] openpic_update_irq: destmask 1 last_cpu 0 [61010.582034] openpic_update_irq: Only one CPU is allowed to receive this IRQ [61010.582036] IRQ_local_pipe: IRQ 224 active 0 was 1 [61010.582037] IRQ_check: irq 226 set ivpr_pr=8 pr=-1 [61010.582038] IRQ_check: irq 225 set ivpr_pr=8 pr=-1 [61010.582039] IRQ_check: irq 224 set ivpr_pr=8 pr=-1 It would be really helpful to get your comments regarding whether these are realistical number for everyday use, or they are relevant only to this particular scenario. - Can these interrupts be used in directed delivery, so that the destination mask can include multiple VCPUs? The MPIC manual states that timer and IPI interrupts are supported for directed delivery, altough I'm not sure how much of this is used in the emulation. I know that kvmppc uses the decrementer outside of the MPIC. - How are virtio interrupts cascaded over the shared MSI interrupts? /proc/device-tree/soc@e0000000/msi@41600/interrupts in the guest shows 8 values - 224 - 231 - so at most there might be 8 pending interrupts in IRQ_check, is that correct? Looking forward to your feedback. [1] http://lxr.free-electrons.com/source/arch/powerpc/kvm/mpic.c#L454 [2] http://lxr.free-electrons.com/source/arch/powerpc/kvm/mpic.c#L303 [3] https://www-01.ibm.com/chips/techlib/techlib.nsf/techdocs/F27971551C9EED8E8525774A0048770A/$file/mpic_db_05_16_2011.pdf Best regards, Bogdan P.