From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20652C43461 for ; Thu, 3 Sep 2020 15:26:31 +0000 (UTC) Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by mail.kernel.org (Postfix) with ESMTP id 958C1208CA for ; Thu, 3 Sep 2020 15:26:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="eFusFhxU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 958C1208CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvmarm-bounces@lists.cs.columbia.edu Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 48ACC4B25A; Thu, 3 Sep 2020 11:26:30 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6zHLmlcQM767; Thu, 3 Sep 2020 11:26:27 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 866AF4B2BB; Thu, 3 Sep 2020 11:26:27 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id A29F14B280 for ; Thu, 3 Sep 2020 11:26:25 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m2L-toPFpQdX for ; Thu, 3 Sep 2020 11:26:24 -0400 (EDT) Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 0BCF44B2BB for ; Thu, 3 Sep 2020 11:26:24 -0400 (EDT) Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AB6642072A; Thu, 3 Sep 2020 15:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1599146782; bh=8diWhh0vQL5HYykVI49PgHaLPAgX0kRPHzqmaQWLOls=; h=From:To:Cc:Subject:Date:From; b=eFusFhxUiCVyfGJetxIodBv0tSPJaTcDRpy7v6ycgpd2p15loF0UoUChIv4AjVfyl E1EWwVo2fCbfwJ/7OdVnj01XzPbA8i5+AXlAKJCKpC6Ws+d/cbKNTbwo5/WRuCYFjh NINfTBstZUih33OaWbnO04zoCyb3pg4xfJbKKjaM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kDr80-008vT9-Qx; Thu, 03 Sep 2020 16:26:20 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: [PATCH 00/23] KVM: arm64: rVIC/rVID PV interrupt controller Date: Thu, 3 Sep 2020 16:25:47 +0100 Message-Id: <20200903152610.1078827-1-maz@kernel.org> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kernel-team@android.com, Christoffer.Dall@arm.com, lorenzo.pieralisi@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: Lorenzo Pieralisi , kernel-team@android.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Anyone vaguely familiar with the ARM interrupt architecture (also known as GIC) would certainly agree that it isn't the simplest thing to deal with. Its features are ranging from simple, bare metal interrupt delivery to full blown direct injection into a guest. It is also horribly complex, full of backward[-compatibility] features, and it is very hard to reason about what is going on in the system at any given time. For a hypervisor such as KVM, the GIC is an invasive beast that accounts for a large part of the privileged software we run, as it implements the whole of the architecture with bells and whistles as it tries to cater for all possible guests. At the same time, we have an ongoing effort to make KVM/arm64 a more "verifiable" hypervisor, by allowing only a small amount of code to run at EL2. Moving most of the GIC emulation to userspace would involve sacrificing performance (the architecture really doesn't lend itself to a split model, despite the appearances), and proving that it is actually completely safe is almost an impossible task (I have seen people trying!). Another approach is to bite the bullet, and design from the ground up an interrupt controller that: - works well enough for workloads that mostly deal with virtual interrupts (no device assignment), - is as simple as possible for the hypervisor to implement. Since we cannot retrospectively hack the HW, this is a paravirtualized interrupt controller, where every single operation results in a trap. Yes, it looks like it would be terribly expensive. Or not. The result of the above is a specification from ARM [1] that defines the RVIC and RVID components that make the interrupt controller. It is an *Alpha* spec, to it is very much subject to changes (the hypercall numbers have been redacted out to make that explicit). The result of the above result is this patch series, which provides Linux drivers for rVIC and rVID, as well as a KVM implementation that can be exposed to guests. Most of the patches are a big refactor of the KVM/arm64 code to allow a non-GIC irqchip to be exposed to the guest, as the code that actually deals with delivering interrupts is pretty simple. I intend to carry on refactoring this as more structures could be made irqchip agnostic. We have: - Support for the rVIC/rVID PV interrupt controller architecture in a guest - A large rework of the way the vgic integrates with the rest of KVM, mostly punching a vgic-shaped hole in the code, and replacing it with a set of optional callbacks that an interrupt controller implementation can provide, or not. - A rVIC/rVID implementation for KVM/arm64. This is based on my previously posted IPI-as-IRQ series How does it fare? Well, it's not even bad. There is a bit more overhead than with an actual GIC, but you need to squint really hard to see a difference. Turns out that interacting with a HW interrupt controller isn't free either... Of course, YMMV, and I'd happily look at performance figures if someone has the guts to put them together. What is missing: - Patches for userspace to actually start a rVIC-equipped guest. I have pushed a kvmtool branch at [2]. This is just a terrible pile of hacks, don't trust it to do anything right. It works well enough to spawn a guest with virtio-pci and deliver MSIs though. - DT bindings, which I need to write up - ACPI? Why not... Things that are *not* in the spec: - MSIs. I've made them up in the driver and KVM, and I don't think we can do without them. I intend to feed that requirement back to ARM. - RVID level interrupts. We need them to implement PCI INTx. - Priorities. Not clear whether we really need those, and it would certainly complixify the design. This has been lightly tested on a A55-based system running VHE, and equipped with a GICv2, as well as a couple of nVHE systems with both GICv2 and GICv3. VHE+GICv3 is still untested, as I lack the platform (someone please send me an Ampere Altra box ;-). Patches are based on v5.9-rc3 + the IPI patches, and a branch with everything stacked together is at [3]. [1] https://developer.arm.com/architectures/system-architectures/software-standards/rvic [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/kvmtool.git/log/?h=rvic [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/rvic Marc Zyngier (23): irqchip: Add Reduced Virtual Interrupt Controller driver irqchip/rvic: Add support for untrusted interrupt allocation irqchip: Add Reduced Virtual Interrupt Distributor support irqchip/rvid: Add PCI MSI support KVM: arm64: Move GIC model out of the distributor KVM: arm64: vgic-v3: Move early init to kvm_vgic_create() KVM: arm64: Add irqchip callback structure to kvm_arch KVM: arm64: Move kvm_vgic_destroy to kvm_irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_init() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_[un]blocking() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_load/put() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_pending_irq() to irqchip_flow KVM: arm64: Move vgic resource mapping on first run to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_{sync,flush}_hwstate() to irqchip_flow KVM: arm64: nVHE: Only save/restore GICv3 state if modeling a GIC KVM: arm64: Move interrupt injection to irqchip_flow KVM: arm64: Move mapping of HW interrupts into irqchip_flow KVM: arm64: Move set_owner into irqchip_flow KVM: arm64: Turn vgic_initialized into irqchip_finalized KVM: arm64: Move irqfd routing to irqchip_flow KVM: arm64: Tighten msis_require_devid reporting KVM: arm64: Add a rVIC/rVID in-kernel implementation KVM: arm64: Add debugfs files for the rVIC/rVID implementation arch/arm64/include/asm/kvm_host.h | 11 +- arch/arm64/include/asm/kvm_irq.h | 136 +++ arch/arm64/include/uapi/asm/kvm.h | 9 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arch_timer.c | 36 +- arch/arm64/kvm/arm.c | 141 ++- arch/arm64/kvm/hyp/nvhe/switch.c | 12 +- arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pmu-emul.c | 10 +- arch/arm64/kvm/rvic-cpu.c | 1213 +++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-debug.c | 7 +- arch/arm64/kvm/vgic/vgic-init.c | 133 ++- arch/arm64/kvm/vgic/vgic-irqfd.c | 72 +- arch/arm64/kvm/vgic/vgic-its.c | 2 +- arch/arm64/kvm/vgic/vgic-kvm-device.c | 18 +- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic-mmio.c | 10 +- arch/arm64/kvm/vgic/vgic-v2.c | 5 - arch/arm64/kvm/vgic/vgic-v3.c | 26 +- arch/arm64/kvm/vgic/vgic.c | 55 +- arch/arm64/kvm/vgic/vgic.h | 37 + drivers/irqchip/Kconfig | 12 + drivers/irqchip/Makefile | 2 + drivers/irqchip/irq-rvic.c | 595 ++++++++++++ drivers/irqchip/irq-rvid.c | 441 +++++++++ include/kvm/arm_rvic.h | 41 + include/kvm/arm_vgic.h | 33 - include/linux/cpuhotplug.h | 1 + include/linux/irqchip/irq-rvic.h | 100 ++ include/uapi/linux/kvm.h | 2 + 30 files changed, 2907 insertions(+), 264 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_irq.h create mode 100644 arch/arm64/kvm/rvic-cpu.c create mode 100644 drivers/irqchip/irq-rvic.c create mode 100644 drivers/irqchip/irq-rvid.c create mode 100644 include/kvm/arm_rvic.h create mode 100644 include/linux/irqchip/irq-rvic.h -- 2.27.0 _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DC55C433E2 for ; Thu, 3 Sep 2020 15:28:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D9472072A for ; Thu, 3 Sep 2020 15:28:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="QCRvhqX+"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="eFusFhxU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D9472072A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:To:From: Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender :Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=dRfwtQIgu1E9w/KW/OSRP4oz/ix6+yLaPimoRTo4cko=; b=QCRvhqX+fDrqXW9dpZkfkZNm7b +++G3vyt6mfPLD62e8VrkO+CGcSR3evN2pOArTTqnKsfKe5x2lW6GC8tc5LDgmFc/DvqsH2WXH2h1 TWxyv+91p9GNWEGnf5UVxZ70TRQYb2wD1u4Lja0ZY4S7nRyTRChMq8QfhrzBTmAQx5WRrlPIrZk4d AiduEr75TlycmKB9UdUyRiWXs5Um3+nug1GJI/K0R+q0gEGwSTJwHFAPJHK6Jk0ArTcxhqxrBc9DD 44f7WKRyN2uV5++MR9UCZC9lvXIw+10aqDOC9D24LPWkY4KS9GVR5Ue3AGlykroQaHn7hPYYN2I/C wF5sGDcQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDr86-0004rM-Bb; Thu, 03 Sep 2020 15:26:26 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kDr83-0004qL-IR for linux-arm-kernel@lists.infradead.org; Thu, 03 Sep 2020 15:26:24 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AB6642072A; Thu, 3 Sep 2020 15:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1599146782; bh=8diWhh0vQL5HYykVI49PgHaLPAgX0kRPHzqmaQWLOls=; h=From:To:Cc:Subject:Date:From; b=eFusFhxUiCVyfGJetxIodBv0tSPJaTcDRpy7v6ycgpd2p15loF0UoUChIv4AjVfyl E1EWwVo2fCbfwJ/7OdVnj01XzPbA8i5+AXlAKJCKpC6Ws+d/cbKNTbwo5/WRuCYFjh NINfTBstZUih33OaWbnO04zoCyb3pg4xfJbKKjaM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kDr80-008vT9-Qx; Thu, 03 Sep 2020 16:26:20 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Subject: [PATCH 00/23] KVM: arm64: rVIC/rVID PV interrupt controller Date: Thu, 3 Sep 2020 16:25:47 +0100 Message-Id: <20200903152610.1078827-1-maz@kernel.org> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kernel-team@android.com, Christoffer.Dall@arm.com, lorenzo.pieralisi@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200903_112623_754179_F8F79DB5 X-CRM114-Status: GOOD ( 29.97 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lorenzo Pieralisi , Suzuki K Poulose , Christoffer Dall , James Morse , kernel-team@android.com, Julien Thierry Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Anyone vaguely familiar with the ARM interrupt architecture (also known as GIC) would certainly agree that it isn't the simplest thing to deal with. Its features are ranging from simple, bare metal interrupt delivery to full blown direct injection into a guest. It is also horribly complex, full of backward[-compatibility] features, and it is very hard to reason about what is going on in the system at any given time. For a hypervisor such as KVM, the GIC is an invasive beast that accounts for a large part of the privileged software we run, as it implements the whole of the architecture with bells and whistles as it tries to cater for all possible guests. At the same time, we have an ongoing effort to make KVM/arm64 a more "verifiable" hypervisor, by allowing only a small amount of code to run at EL2. Moving most of the GIC emulation to userspace would involve sacrificing performance (the architecture really doesn't lend itself to a split model, despite the appearances), and proving that it is actually completely safe is almost an impossible task (I have seen people trying!). Another approach is to bite the bullet, and design from the ground up an interrupt controller that: - works well enough for workloads that mostly deal with virtual interrupts (no device assignment), - is as simple as possible for the hypervisor to implement. Since we cannot retrospectively hack the HW, this is a paravirtualized interrupt controller, where every single operation results in a trap. Yes, it looks like it would be terribly expensive. Or not. The result of the above is a specification from ARM [1] that defines the RVIC and RVID components that make the interrupt controller. It is an *Alpha* spec, to it is very much subject to changes (the hypercall numbers have been redacted out to make that explicit). The result of the above result is this patch series, which provides Linux drivers for rVIC and rVID, as well as a KVM implementation that can be exposed to guests. Most of the patches are a big refactor of the KVM/arm64 code to allow a non-GIC irqchip to be exposed to the guest, as the code that actually deals with delivering interrupts is pretty simple. I intend to carry on refactoring this as more structures could be made irqchip agnostic. We have: - Support for the rVIC/rVID PV interrupt controller architecture in a guest - A large rework of the way the vgic integrates with the rest of KVM, mostly punching a vgic-shaped hole in the code, and replacing it with a set of optional callbacks that an interrupt controller implementation can provide, or not. - A rVIC/rVID implementation for KVM/arm64. This is based on my previously posted IPI-as-IRQ series How does it fare? Well, it's not even bad. There is a bit more overhead than with an actual GIC, but you need to squint really hard to see a difference. Turns out that interacting with a HW interrupt controller isn't free either... Of course, YMMV, and I'd happily look at performance figures if someone has the guts to put them together. What is missing: - Patches for userspace to actually start a rVIC-equipped guest. I have pushed a kvmtool branch at [2]. This is just a terrible pile of hacks, don't trust it to do anything right. It works well enough to spawn a guest with virtio-pci and deliver MSIs though. - DT bindings, which I need to write up - ACPI? Why not... Things that are *not* in the spec: - MSIs. I've made them up in the driver and KVM, and I don't think we can do without them. I intend to feed that requirement back to ARM. - RVID level interrupts. We need them to implement PCI INTx. - Priorities. Not clear whether we really need those, and it would certainly complixify the design. This has been lightly tested on a A55-based system running VHE, and equipped with a GICv2, as well as a couple of nVHE systems with both GICv2 and GICv3. VHE+GICv3 is still untested, as I lack the platform (someone please send me an Ampere Altra box ;-). Patches are based on v5.9-rc3 + the IPI patches, and a branch with everything stacked together is at [3]. [1] https://developer.arm.com/architectures/system-architectures/software-standards/rvic [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/kvmtool.git/log/?h=rvic [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/rvic Marc Zyngier (23): irqchip: Add Reduced Virtual Interrupt Controller driver irqchip/rvic: Add support for untrusted interrupt allocation irqchip: Add Reduced Virtual Interrupt Distributor support irqchip/rvid: Add PCI MSI support KVM: arm64: Move GIC model out of the distributor KVM: arm64: vgic-v3: Move early init to kvm_vgic_create() KVM: arm64: Add irqchip callback structure to kvm_arch KVM: arm64: Move kvm_vgic_destroy to kvm_irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_init() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_[un]blocking() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_load/put() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_pending_irq() to irqchip_flow KVM: arm64: Move vgic resource mapping on first run to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_{sync,flush}_hwstate() to irqchip_flow KVM: arm64: nVHE: Only save/restore GICv3 state if modeling a GIC KVM: arm64: Move interrupt injection to irqchip_flow KVM: arm64: Move mapping of HW interrupts into irqchip_flow KVM: arm64: Move set_owner into irqchip_flow KVM: arm64: Turn vgic_initialized into irqchip_finalized KVM: arm64: Move irqfd routing to irqchip_flow KVM: arm64: Tighten msis_require_devid reporting KVM: arm64: Add a rVIC/rVID in-kernel implementation KVM: arm64: Add debugfs files for the rVIC/rVID implementation arch/arm64/include/asm/kvm_host.h | 11 +- arch/arm64/include/asm/kvm_irq.h | 136 +++ arch/arm64/include/uapi/asm/kvm.h | 9 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arch_timer.c | 36 +- arch/arm64/kvm/arm.c | 141 ++- arch/arm64/kvm/hyp/nvhe/switch.c | 12 +- arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pmu-emul.c | 10 +- arch/arm64/kvm/rvic-cpu.c | 1213 +++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-debug.c | 7 +- arch/arm64/kvm/vgic/vgic-init.c | 133 ++- arch/arm64/kvm/vgic/vgic-irqfd.c | 72 +- arch/arm64/kvm/vgic/vgic-its.c | 2 +- arch/arm64/kvm/vgic/vgic-kvm-device.c | 18 +- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic-mmio.c | 10 +- arch/arm64/kvm/vgic/vgic-v2.c | 5 - arch/arm64/kvm/vgic/vgic-v3.c | 26 +- arch/arm64/kvm/vgic/vgic.c | 55 +- arch/arm64/kvm/vgic/vgic.h | 37 + drivers/irqchip/Kconfig | 12 + drivers/irqchip/Makefile | 2 + drivers/irqchip/irq-rvic.c | 595 ++++++++++++ drivers/irqchip/irq-rvid.c | 441 +++++++++ include/kvm/arm_rvic.h | 41 + include/kvm/arm_vgic.h | 33 - include/linux/cpuhotplug.h | 1 + include/linux/irqchip/irq-rvic.h | 100 ++ include/uapi/linux/kvm.h | 2 + 30 files changed, 2907 insertions(+), 264 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_irq.h create mode 100644 arch/arm64/kvm/rvic-cpu.c create mode 100644 drivers/irqchip/irq-rvic.c create mode 100644 drivers/irqchip/irq-rvid.c create mode 100644 include/kvm/arm_rvic.h create mode 100644 include/linux/irqchip/irq-rvic.h -- 2.27.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6C7EC43461 for ; Thu, 3 Sep 2020 15:26:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A57DC2072A for ; Thu, 3 Sep 2020 15:26:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1599146793; bh=8diWhh0vQL5HYykVI49PgHaLPAgX0kRPHzqmaQWLOls=; h=From:To:Cc:Subject:Date:List-ID:From; b=VPAvU7uvsSHtb/POcjupcHawAx4z4lhXxf/oShIJRgZ6f3Uxwau3DStGWn/xwOE5R +TLA9flEqggpUsFrJpgE9NUsaCVj/3N6HcUkHjIIiY0t7TaxAOwkcY6WqErwYpnZNO vQiPS+eiG1CxEXmE37pb652ueRAYQBT+oZU73K1o= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728529AbgICP0d (ORCPT ); Thu, 3 Sep 2020 11:26:33 -0400 Received: from mail.kernel.org ([198.145.29.99]:42254 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726327AbgICP0X (ORCPT ); Thu, 3 Sep 2020 11:26:23 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AB6642072A; Thu, 3 Sep 2020 15:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1599146782; bh=8diWhh0vQL5HYykVI49PgHaLPAgX0kRPHzqmaQWLOls=; h=From:To:Cc:Subject:Date:From; b=eFusFhxUiCVyfGJetxIodBv0tSPJaTcDRpy7v6ycgpd2p15loF0UoUChIv4AjVfyl E1EWwVo2fCbfwJ/7OdVnj01XzPbA8i5+AXlAKJCKpC6Ws+d/cbKNTbwo5/WRuCYFjh NINfTBstZUih33OaWbnO04zoCyb3pg4xfJbKKjaM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kDr80-008vT9-Qx; Thu, 03 Sep 2020 16:26:20 +0100 From: Marc Zyngier To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org Cc: kernel-team@android.com, Christoffer Dall , Lorenzo Pieralisi , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH 00/23] KVM: arm64: rVIC/rVID PV interrupt controller Date: Thu, 3 Sep 2020 16:25:47 +0100 Message-Id: <20200903152610.1078827-1-maz@kernel.org> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kernel-team@android.com, Christoffer.Dall@arm.com, lorenzo.pieralisi@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Anyone vaguely familiar with the ARM interrupt architecture (also known as GIC) would certainly agree that it isn't the simplest thing to deal with. Its features are ranging from simple, bare metal interrupt delivery to full blown direct injection into a guest. It is also horribly complex, full of backward[-compatibility] features, and it is very hard to reason about what is going on in the system at any given time. For a hypervisor such as KVM, the GIC is an invasive beast that accounts for a large part of the privileged software we run, as it implements the whole of the architecture with bells and whistles as it tries to cater for all possible guests. At the same time, we have an ongoing effort to make KVM/arm64 a more "verifiable" hypervisor, by allowing only a small amount of code to run at EL2. Moving most of the GIC emulation to userspace would involve sacrificing performance (the architecture really doesn't lend itself to a split model, despite the appearances), and proving that it is actually completely safe is almost an impossible task (I have seen people trying!). Another approach is to bite the bullet, and design from the ground up an interrupt controller that: - works well enough for workloads that mostly deal with virtual interrupts (no device assignment), - is as simple as possible for the hypervisor to implement. Since we cannot retrospectively hack the HW, this is a paravirtualized interrupt controller, where every single operation results in a trap. Yes, it looks like it would be terribly expensive. Or not. The result of the above is a specification from ARM [1] that defines the RVIC and RVID components that make the interrupt controller. It is an *Alpha* spec, to it is very much subject to changes (the hypercall numbers have been redacted out to make that explicit). The result of the above result is this patch series, which provides Linux drivers for rVIC and rVID, as well as a KVM implementation that can be exposed to guests. Most of the patches are a big refactor of the KVM/arm64 code to allow a non-GIC irqchip to be exposed to the guest, as the code that actually deals with delivering interrupts is pretty simple. I intend to carry on refactoring this as more structures could be made irqchip agnostic. We have: - Support for the rVIC/rVID PV interrupt controller architecture in a guest - A large rework of the way the vgic integrates with the rest of KVM, mostly punching a vgic-shaped hole in the code, and replacing it with a set of optional callbacks that an interrupt controller implementation can provide, or not. - A rVIC/rVID implementation for KVM/arm64. This is based on my previously posted IPI-as-IRQ series How does it fare? Well, it's not even bad. There is a bit more overhead than with an actual GIC, but you need to squint really hard to see a difference. Turns out that interacting with a HW interrupt controller isn't free either... Of course, YMMV, and I'd happily look at performance figures if someone has the guts to put them together. What is missing: - Patches for userspace to actually start a rVIC-equipped guest. I have pushed a kvmtool branch at [2]. This is just a terrible pile of hacks, don't trust it to do anything right. It works well enough to spawn a guest with virtio-pci and deliver MSIs though. - DT bindings, which I need to write up - ACPI? Why not... Things that are *not* in the spec: - MSIs. I've made them up in the driver and KVM, and I don't think we can do without them. I intend to feed that requirement back to ARM. - RVID level interrupts. We need them to implement PCI INTx. - Priorities. Not clear whether we really need those, and it would certainly complixify the design. This has been lightly tested on a A55-based system running VHE, and equipped with a GICv2, as well as a couple of nVHE systems with both GICv2 and GICv3. VHE+GICv3 is still untested, as I lack the platform (someone please send me an Ampere Altra box ;-). Patches are based on v5.9-rc3 + the IPI patches, and a branch with everything stacked together is at [3]. [1] https://developer.arm.com/architectures/system-architectures/software-standards/rvic [2] https://git.kernel.org/pub/scm/linux/kernel/git/maz/kvmtool.git/log/?h=rvic [3] https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git/log/?h=irq/rvic Marc Zyngier (23): irqchip: Add Reduced Virtual Interrupt Controller driver irqchip/rvic: Add support for untrusted interrupt allocation irqchip: Add Reduced Virtual Interrupt Distributor support irqchip/rvid: Add PCI MSI support KVM: arm64: Move GIC model out of the distributor KVM: arm64: vgic-v3: Move early init to kvm_vgic_create() KVM: arm64: Add irqchip callback structure to kvm_arch KVM: arm64: Move kvm_vgic_destroy to kvm_irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_init() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_[un]blocking() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_load/put() to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_pending_irq() to irqchip_flow KVM: arm64: Move vgic resource mapping on first run to irqchip_flow KVM: arm64: Move kvm_vgic_vcpu_{sync,flush}_hwstate() to irqchip_flow KVM: arm64: nVHE: Only save/restore GICv3 state if modeling a GIC KVM: arm64: Move interrupt injection to irqchip_flow KVM: arm64: Move mapping of HW interrupts into irqchip_flow KVM: arm64: Move set_owner into irqchip_flow KVM: arm64: Turn vgic_initialized into irqchip_finalized KVM: arm64: Move irqfd routing to irqchip_flow KVM: arm64: Tighten msis_require_devid reporting KVM: arm64: Add a rVIC/rVID in-kernel implementation KVM: arm64: Add debugfs files for the rVIC/rVID implementation arch/arm64/include/asm/kvm_host.h | 11 +- arch/arm64/include/asm/kvm_irq.h | 136 +++ arch/arm64/include/uapi/asm/kvm.h | 9 + arch/arm64/kvm/Makefile | 2 +- arch/arm64/kvm/arch_timer.c | 36 +- arch/arm64/kvm/arm.c | 141 ++- arch/arm64/kvm/hyp/nvhe/switch.c | 12 +- arch/arm64/kvm/hypercalls.c | 7 + arch/arm64/kvm/pmu-emul.c | 10 +- arch/arm64/kvm/rvic-cpu.c | 1213 +++++++++++++++++++++++++ arch/arm64/kvm/vgic/vgic-debug.c | 7 +- arch/arm64/kvm/vgic/vgic-init.c | 133 ++- arch/arm64/kvm/vgic/vgic-irqfd.c | 72 +- arch/arm64/kvm/vgic/vgic-its.c | 2 +- arch/arm64/kvm/vgic/vgic-kvm-device.c | 18 +- arch/arm64/kvm/vgic/vgic-mmio-v3.c | 2 +- arch/arm64/kvm/vgic/vgic-mmio.c | 10 +- arch/arm64/kvm/vgic/vgic-v2.c | 5 - arch/arm64/kvm/vgic/vgic-v3.c | 26 +- arch/arm64/kvm/vgic/vgic.c | 55 +- arch/arm64/kvm/vgic/vgic.h | 37 + drivers/irqchip/Kconfig | 12 + drivers/irqchip/Makefile | 2 + drivers/irqchip/irq-rvic.c | 595 ++++++++++++ drivers/irqchip/irq-rvid.c | 441 +++++++++ include/kvm/arm_rvic.h | 41 + include/kvm/arm_vgic.h | 33 - include/linux/cpuhotplug.h | 1 + include/linux/irqchip/irq-rvic.h | 100 ++ include/uapi/linux/kvm.h | 2 + 30 files changed, 2907 insertions(+), 264 deletions(-) create mode 100644 arch/arm64/include/asm/kvm_irq.h create mode 100644 arch/arm64/kvm/rvic-cpu.c create mode 100644 drivers/irqchip/irq-rvic.c create mode 100644 drivers/irqchip/irq-rvid.c create mode 100644 include/kvm/arm_rvic.h create mode 100644 include/linux/irqchip/irq-rvic.h -- 2.27.0