From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6289C10F07 for ; Sat, 2 Dec 2023 12:37:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232659AbjLBMUb (ORCPT ); Sat, 2 Dec 2023 07:20:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229472AbjLBMU3 (ORCPT ); Sat, 2 Dec 2023 07:20:29 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30654E6 for ; Sat, 2 Dec 2023 04:20:35 -0800 (PST) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9E7EAC433C7; Sat, 2 Dec 2023 12:20:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1701519634; bh=ejaFDXno/ycGNYAzsMsIvPwtjPT2anJN2aFKbFwFV04=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=PvfwcIfWLh/wjDpU29WREENELuCw5zM+gDaYXkHN1dTQAhMOVMLcKZx0qF+XtbDww Cp8cd0bGhix99JP0W3laRsC/uu0mW48n0dssmWtApkTuYAK+QRQqGRTEDM1Tj7nq3r DfFE8WCm4e9My7K4ry5kpd+J4TThkfjNRic1IjDpX9BZv3TYIgAnHPOagQROkylFKY 5CFgXaV5HBHRcNzJxPImcQlZ6BOZyrC8D0rgwpxvA5733vzb24/gDQwiSJc3V3YH5z 1HzD2DElwwq/vVfI1K8HndQt7lDMscl/WRpH88NW+QoGuwn8AExtxvvT1rAjw09rBI sO/nNsSPAusbQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1r9Oz2-000oDc-6w; Sat, 02 Dec 2023 12:20:32 +0000 Date: Sat, 02 Dec 2023 12:20:31 +0000 Message-ID: <87fs0k94og.wl-maz@kernel.org> From: Marc Zyngier To: Kunkun Jiang Cc: , , , , , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Gavin Shan , Jean-Philippe Brucker , "open list:IRQCHIP DRIVERS" , , , Subject: Re: [RFC PATCH] KVM: arm/arm64: GICv4: Support shared VLPI In-Reply-To: <952bd5dc-dd20-acc3-d77e-c9b14e5728d3@huawei.com> References: <20231102143507.840-1-jiangkunkun@huawei.com> <87msvt6cc7.wl-maz@kernel.org> <1fb8353e-e9c4-2570-c2ca-ec537c18ac4d@huawei.com> <86edh228xx.wl-maz@kernel.org> <952bd5dc-dd20-acc3-d77e-c9b14e5728d3@huawei.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: jiangkunkun@huawei.com, dongli.zhang@oracle.com, cohuck@redhat.com, jasowang@redhat.com, stefanha@redhat.com, mst@redhat.com, oliver.upton@linux.dev, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, gshan@redhat.com, jean-philippe@linaro.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, wanghaibin.wang@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Kunkun, On Wed, 08 Nov 2023 09:45:51 +0000, Kunkun Jiang wrote: > > Hi Marc, > > On 2023/11/6 23:33, Marc Zyngier wrote: > > On Mon, 06 Nov 2023 14:59:01 +0000, > > Kunkun Jiang wrote: > >> The virtio-pci driver write entry1-6 > >> massage.data in the msix-table and trap to QEMU for processing. The > >> massage.data is as follow: > >>> entry-0 0 > >>> entry-1 1 > >>> entry-2 1 > >>> entry-3 1 > >>> entry-4 1 > >>> entry-5 1 > >>> entry-6 1 > > Urgh... is vp_modern_queue_vector() used in your configuration? This > > is ... terrible. > I encountered this problem using the 4.19 version kernel, but not the > 5.10 version. This vp_modern_queue_vector() function does not exist > in 4.19, but it uses 'vp_iowrite16(msix_vec, &cfg->queue_msix_vector)', > the same as vp_modern_queue_vector(). > > In the past two days, I learned about the virtio driver and made some > new discoveries. When 'num_queues' is greater than maxcpus, it will > fall back into MSI-X with one shared for queues. The two patches[1], > submitted by Dongli, limits the number of hw queues used by > virtio-blk/virtio-scsi by 'nr_cpu_ids'. The two patches were merged > in 5.1-rc2. And the patch related virtio-blk was merged into the 4.19 > stable branch.The patch related virtio-scsi was not merged. > [1] > https://lore.kernel.org/all/1553682995-5682-1-git-send-email-dongli.zhang@oracle.com/ > > This is the earliest discussion. > https://lore.kernel.org/all/e4afe4c5-0262-4500-aeec-60f30734b4fc@default/ > > I don't know if there are other circumstances that would cause it to > fall back into MSI-X with one shared for queues. At least the hack > method is possible. > > I wonder if PCIe actually allows this sort of thing. > Do you think the virtio driver should be modified? I think the virtio driver should stop messing with the MSI-X configuration behind the kernel's back. For example, what happens if the kernel needs to do a disable_irq() on the "shared" interrupt? It will mask the interrupt in *one* of the vectors, and the interrupt will still be screaming. This is terribly broken, even on x86. > > In any case, this sort of behaviour breaks so many thing in KVM's > > implementation that I'd recommend you disable GICv4 until we have a > > good solution for that. > There seems to be no restriction in the GIC specification that multiple > host irqs cannot be mapped to the same vlpi. Or maybe I didn't notice. > Do you think there are any risks? Please see 5.2.10 ("Restrictions for INTID mapping rules"), which clearly forbids the case we have here: "Maps multiple EventID-DeviceID combinations to the same virtual LPI INTID-vPEID.". > GICv3 does not have this issue, but is this configuration legal? With GICv3, the ITS doesn't see multiple mappings to the same LPI. Each DeviceID/EventID pair has its own LPI, and KVM will just see the injection callback from VFIO. Honestly, the virtio driver is broken (irrespective of the architecture), and incompatible with the GIC architecture. M. -- Without deviation from the norm, progress is not possible.