From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,BITCOIN_OBFU_SUBJ, INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 428E0C4320A for ; Wed, 25 Aug 2021 15:33:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2DA476109E for ; Wed, 25 Aug 2021 15:33:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241783AbhHYPeQ (ORCPT ); Wed, 25 Aug 2021 11:34:16 -0400 Received: from mail.kernel.org ([198.145.29.99]:35478 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241514AbhHYPeK (ORCPT ); Wed, 25 Aug 2021 11:34:10 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 408A46109E; Wed, 25 Aug 2021 15:33:24 +0000 (UTC) Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1mIuu2-007Agb-6J; Wed, 25 Aug 2021 16:33:22 +0100 Date: Wed, 25 Aug 2021 16:33:21 +0100 Message-ID: <87a6l5pmim.wl-maz@kernel.org> From: Marc Zyngier To: Andre Przywara Cc: kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, Alexandru Elisei , Thomas Gleixner , Will Deacon Subject: Re: [PATCH][kvmtool] virtio/pci: Correctly handle MSI-X masking while MSI-X is disabled In-Reply-To: <87tujeq5ey.wl-maz@kernel.org> References: <20210821120742.855712-1-maz@kernel.org> <20210823174833.05adee5d@slackpad.fritz.box> <87tujeq5ey.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: andre.przywara@arm.com, kvm@vger.kernel.org, kvmarm@lists.cs.columbia.edu, kernel-team@android.com, alexandru.elisei@arm.com, tglx@linutronix.de, will@kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, 24 Aug 2021 15:32:53 +0100, Marc Zyngier wrote: > > Hi Andre, > > On Mon, 23 Aug 2021 17:48:33 +0100, > Andre Przywara wrote: > > > > On Sat, 21 Aug 2021 13:07:42 +0100 > > Marc Zyngier wrote: > > > > Hi Marc, > > > > > Since Linux commit 7d5ec3d36123 ("PCI/MSI: Mask all unused MSI-X > > > entries"), kvmtool segfaults when the guest boots and tries to > > > disable all the MSI-X entries of a virtio device while MSI-X itself > > > is disabled. > > > > > > What Linux does is seems perfectly correct. However, kvmtool uses > > > a different decoding depending on whether MSI-X is enabled for > > > this device or not. Which seems pretty wrong. > > > > While I really wish this would be wrong, I think this is > > indeed how this is supposed to work: The Virtio legacy spec makes the > > existence of those two virtio config fields dependent on the > > (dynamic!) enablement status of MSI-X. This is reflected in: > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/virtio_pci.h#n72 > > and explicitly mentioned as a footnote in the virtio 0.9.5 spec[1]: > > "3) ie. once you enable MSI-X on the device, the other fields move. If > > you turn it off again, they move back!" > > Madness! What was Rusty on at the time? I really hope the bitcoin > thing is buying him better stuff... > > > I agree that this looks like a bad idea, but I am afraid we are stuck > > with this. It looks like the Linux driver is at fault here, it should > > not issue the config access when MSIs are disabled. Something like this > > (untested): > > > > --- a/drivers/virtio/virtio_pci_legacy.c > > +++ b/drivers/virtio/virtio_pci_legacy.c > > @@ -103,6 +103,9 @@ static void vp_reset(struct virtio_device *vdev) > > > > static u16 vp_config_vector(struct virtio_pci_device *vp_dev, u16 vector) > > { > > + if (!vp_dev->msix_enabled) > > + return VIRTIO_MSI_NO_VECTOR; > > + > > /* Setup the vector used for configuration events */ > > iowrite16(vector, vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > /* Verify we had enough resources to assign the vector */ > > > > This is just my first idea after looking at this, happy to stand > > corrected or hear about a better solution. > > I don't think this works. It instead completely disables MSI-X, which > is a total bore. I think the only way to deal with it is to quirk it > to prevent the bulk masking to take effect before MSI-X is enabled. Actually, let me correct myself. I tested the wrong configuration (why isn't --force-pci the bloody default in kvmtool?). This patch doesn't fix anything at all, and kvmtool just explodes. Having dug further, it isn't the config space that causes problems, but the programming of the MSI-X vectors. I'm starting to suspect the layout of the MSI-X bar in kvmtool. M. -- Without deviation from the norm, progress is not possible.