From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 829D8C43331 for ; Thu, 7 Nov 2019 16:04:29 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 459EB218AE for ; Thu, 7 Nov 2019 16:04:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 459EB218AE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id CB3034ACA7; Thu, 7 Nov 2019 11:04:28 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id F2Fj8e1eH8G4; Thu, 7 Nov 2019 11:04:27 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id CA8874ACA5; Thu, 7 Nov 2019 11:04:26 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id B93FE4ACAB for ; Thu, 7 Nov 2019 11:04:25 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 8vpleHFXA1p5 for ; Thu, 7 Nov 2019 11:04:24 -0500 (EST) Received: from inca-roads.misterjones.org (inca-roads.misterjones.org [213.251.177.50]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 92B534AC83 for ; Thu, 7 Nov 2019 11:04:24 -0500 (EST) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by cheepnis.misterjones.org with esmtpsa (TLSv1.2:DHE-RSA-AES128-GCM-SHA256:128) (Exim 4.80) (envelope-from ) id 1iSkGk-0008Bm-6O; Thu, 07 Nov 2019 17:04:22 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: [PATCH 0/2] KVM: arm64: Reduce occurence of GICv4 doorbells on non-oversubscribed systems Date: Thu, 7 Nov 2019 16:04:10 +0000 Message-Id: <20191107160412.30301-1-maz@kernel.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on cheepnis.misterjones.org); SAEximRunCond expanded to false X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu As I was cleaning up some of the GICv4 code to make way for GICv4.1 it occured to me that we could drastically reduce the impact of the GICv4 doorbells on systems that are not oversubscribed (each vcpu "owns" a physical CPU). The technique borrows its logic from the way we disable WFE trapping when a vcpu is the only process on the CPU run-queue. If this vcpu is the target of VLPIs, it is then beneficial not to trap blocking WFIs and to leave the vcpu waiting for interrupts in guest state. All we need to do here is to track whether VLPIs are associated to a vcpu (which is easily done by using a counter that we update on MAPI, DISCARD and MOVI). It has been *very lightly* tested on a D05, and behaved pretty well in my limited test cases (I get almost no doorbell at all in the non oversubscribed case, and the usual hailstorm as soon as there is oversubscription). I'd welcome some testing on more current HW. Marc Zyngier (2): KVM: vgic-v4: Track the number of VLPIs per vcpu KVM: arm64: Opportunistically turn off WFI trapping when using direct LPI injection arch/arm/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_emulate.h | 9 +++++++-- include/linux/irqchip/arm-gic-v4.h | 2 ++ virt/kvm/arm/arm.c | 4 ++-- virt/kvm/arm/vgic/vgic-init.c | 1 + virt/kvm/arm/vgic/vgic-its.c | 3 +++ virt/kvm/arm/vgic/vgic-v4.c | 2 ++ 7 files changed, 19 insertions(+), 6 deletions(-) -- 2.20.1 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CAEBDFC6195 for ; Thu, 7 Nov 2019 16:04:30 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A0E37218AE for ; Thu, 7 Nov 2019 16:04:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Blwo1XPp" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A0E37218AE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To :From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=PoAnrEwSAtFpuZIKAdQhcZYyVShC+Hc3xepaIGuUs0o=; b=Blwo1XPpE2JiWN LnlgcBZyELx3Skze9OB+ff84z4PJUR+zV0aOedUKDUSx9XIQFnCYz5mYK4mIcj8OJvVjXwsLDmMBp q+Cd2mnmVikmi8mIZsP4dB0S7yjdoFPqabYnhZYDKGt1VJZC7SfzM8NcD6JvXrYpF5nwCZeypXO32 5a7xPgNB8VrtJf9yXKAKfFq3danCwoVNbGokmO5bhihXU8p0RWvfEO3uN1d+1f39DU9pMtVW/SEph MmfW/TulWoeUT2rj3t7/lZF3+4TvLaTR0FMzJMCKIN70fyse/1TVBGLIWfgTBuwbrsfF3us2ck9oY VcvUOb8J9i+ouUkm74iw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1iSkGr-0005jr-Lz; Thu, 07 Nov 2019 16:04:29 +0000 Received: from inca-roads.misterjones.org ([213.251.177.50]) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1iSkGo-0005iC-BJ for linux-arm-kernel@lists.infradead.org; Thu, 07 Nov 2019 16:04:27 +0000 Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by cheepnis.misterjones.org with esmtpsa (TLSv1.2:DHE-RSA-AES128-GCM-SHA256:128) (Exim 4.80) (envelope-from ) id 1iSkGk-0008Bm-6O; Thu, 07 Nov 2019 17:04:22 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: [PATCH 0/2] KVM: arm64: Reduce occurence of GICv4 doorbells on non-oversubscribed systems Date: Thu, 7 Nov 2019 16:04:10 +0000 Message-Id: <20191107160412.30301-1-maz@kernel.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on cheepnis.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20191107_080426_537984_4B937219 X-CRM114-Status: GOOD ( 11.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: James Morse , Julien Thierry , Suzuki K Poulose Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org As I was cleaning up some of the GICv4 code to make way for GICv4.1 it occured to me that we could drastically reduce the impact of the GICv4 doorbells on systems that are not oversubscribed (each vcpu "owns" a physical CPU). The technique borrows its logic from the way we disable WFE trapping when a vcpu is the only process on the CPU run-queue. If this vcpu is the target of VLPIs, it is then beneficial not to trap blocking WFIs and to leave the vcpu waiting for interrupts in guest state. All we need to do here is to track whether VLPIs are associated to a vcpu (which is easily done by using a counter that we update on MAPI, DISCARD and MOVI). It has been *very lightly* tested on a D05, and behaved pretty well in my limited test cases (I get almost no doorbell at all in the non oversubscribed case, and the usual hailstorm as soon as there is oversubscription). I'd welcome some testing on more current HW. Marc Zyngier (2): KVM: vgic-v4: Track the number of VLPIs per vcpu KVM: arm64: Opportunistically turn off WFI trapping when using direct LPI injection arch/arm/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_emulate.h | 9 +++++++-- include/linux/irqchip/arm-gic-v4.h | 2 ++ virt/kvm/arm/arm.c | 4 ++-- virt/kvm/arm/vgic/vgic-init.c | 1 + virt/kvm/arm/vgic/vgic-its.c | 3 +++ virt/kvm/arm/vgic/vgic-v4.c | 2 ++ 7 files changed, 19 insertions(+), 6 deletions(-) -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.1 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AF4DC43331 for ; Thu, 7 Nov 2019 16:04:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 54DFC218AE for ; Thu, 7 Nov 2019 16:04:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1573142666; bh=H6Ign5lSBgOVZuqj31XgX9GVSaik4GG3m+HjKK4n1eU=; h=From:To:Cc:Subject:Date:List-ID:From; b=rmZ+vhI62SEHjawGC+mTn9guQx2gddpkNoK41EUdqQa2TMSAD7KvNXAxI7kTRL7Yl 08Q1WUwCmAjuxsm/FJtG+nJwZUu7zgOoCqygWdTPphIeVouXgI1bazRI6qQglW5JWA OO4E7eF6f6d6SIqF2i5XiUq7Z/IRBKXJ4sICqd5A= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388564AbfKGQEZ (ORCPT ); Thu, 7 Nov 2019 11:04:25 -0500 Received: from inca-roads.misterjones.org ([213.251.177.50]:56966 "EHLO inca-roads.misterjones.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387459AbfKGQEY (ORCPT ); Thu, 7 Nov 2019 11:04:24 -0500 Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by cheepnis.misterjones.org with esmtpsa (TLSv1.2:DHE-RSA-AES128-GCM-SHA256:128) (Exim 4.80) (envelope-from ) id 1iSkGk-0008Bm-6O; Thu, 07 Nov 2019 17:04:22 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Cc: James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH 0/2] KVM: arm64: Reduce occurence of GICv4 doorbells on non-oversubscribed systems Date: Thu, 7 Nov 2019 16:04:10 +0000 Message-Id: <20191107160412.30301-1-maz@kernel.org> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on cheepnis.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As I was cleaning up some of the GICv4 code to make way for GICv4.1 it occured to me that we could drastically reduce the impact of the GICv4 doorbells on systems that are not oversubscribed (each vcpu "owns" a physical CPU). The technique borrows its logic from the way we disable WFE trapping when a vcpu is the only process on the CPU run-queue. If this vcpu is the target of VLPIs, it is then beneficial not to trap blocking WFIs and to leave the vcpu waiting for interrupts in guest state. All we need to do here is to track whether VLPIs are associated to a vcpu (which is easily done by using a counter that we update on MAPI, DISCARD and MOVI). It has been *very lightly* tested on a D05, and behaved pretty well in my limited test cases (I get almost no doorbell at all in the non oversubscribed case, and the usual hailstorm as soon as there is oversubscription). I'd welcome some testing on more current HW. Marc Zyngier (2): KVM: vgic-v4: Track the number of VLPIs per vcpu KVM: arm64: Opportunistically turn off WFI trapping when using direct LPI injection arch/arm/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_emulate.h | 9 +++++++-- include/linux/irqchip/arm-gic-v4.h | 2 ++ virt/kvm/arm/arm.c | 4 ++-- virt/kvm/arm/vgic/vgic-init.c | 1 + virt/kvm/arm/vgic/vgic-its.c | 3 +++ virt/kvm/arm/vgic/vgic-v4.c | 2 ++ 7 files changed, 19 insertions(+), 6 deletions(-) -- 2.20.1