From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53286C433E0 for ; Sat, 6 Feb 2021 02:50:28 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CA8D064F91 for ; Sat, 6 Feb 2021 02:50:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA8D064F91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 8B0FB203CE; Sat, 6 Feb 2021 02:50:27 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bags9Lqk2rbd; Sat, 6 Feb 2021 02:50:26 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by silver.osuosl.org (Postfix) with ESMTP id 9429820035; Sat, 6 Feb 2021 02:50:26 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 6A3ECC0174; Sat, 6 Feb 2021 02:50:26 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1F1F9C013A for ; Sat, 6 Feb 2021 02:50:25 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0A18C203CE for ; Sat, 6 Feb 2021 02:50:25 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id eBonE9ZpTzx6 for ; Sat, 6 Feb 2021 02:50:23 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by silver.osuosl.org (Postfix) with ESMTPS id D8C2F20035 for ; Sat, 6 Feb 2021 02:50:23 +0000 (UTC) IronPort-SDR: gn7t8TkaX/d4J4IvvY0okCI43Di8F2RQM+0ttkuoAWhWPp39CDXgEPn921qdu8+LyzeI2WS38t mz2JLuU+GOaw== X-IronPort-AV: E=McAfee;i="6000,8403,9886"; a="178956661" X-IronPort-AV: E=Sophos;i="5.81,156,1610438400"; d="scan'208";a="178956661" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 Feb 2021 18:50:23 -0800 IronPort-SDR: ZYEbH79NCcVuUb33cx3dcwdgg7j3x4nPjn2QVCXDsdC/XVOKH784RPFUHYTu+KoyR77KdjlkA4 qqg5sKQajzxg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,156,1610438400"; d="scan'208";a="373801568" Received: from fmsmsx603.amr.corp.intel.com ([10.18.126.83]) by fmsmga008.fm.intel.com with ESMTP; 05 Feb 2021 18:50:23 -0800 Received: from fmsmsx612.amr.corp.intel.com (10.18.126.92) by fmsmsx603.amr.corp.intel.com (10.18.126.83) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Fri, 5 Feb 2021 18:50:22 -0800 Received: from fmsmsx610.amr.corp.intel.com (10.18.126.90) by fmsmsx612.amr.corp.intel.com (10.18.126.92) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2106.2; Fri, 5 Feb 2021 18:50:22 -0800 Received: from fmsmsx610.amr.corp.intel.com ([10.18.126.90]) by fmsmsx610.amr.corp.intel.com ([10.18.126.90]) with mapi id 15.01.2106.002; Fri, 5 Feb 2021 18:50:22 -0800 From: "Derrick, Jonathan" To: "helgaas@kernel.org" Subject: Re: [PATCH v2 2/2] PCI: vmd: Disable MSI/X remapping when possible Thread-Topic: [PATCH v2 2/2] PCI: vmd: Disable MSI/X remapping when possible Thread-Index: AQHW+ylCpE2jdsG6GkKlPiUMLGcsOqpKo4cAgABQ7wA= Date: Sat, 6 Feb 2021 02:50:22 +0000 Message-ID: <1a91d03da88eb81dff5782686e08eb34ecce7e45.camel@intel.com> References: <20210205215718.GA202474@bjorn-Precision-5520> In-Reply-To: <20210205215718.GA202474@bjorn-Precision-5520> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.132] Content-ID: <3799C4C95BAF7C40BC897FB30EB52955@intel.com> MIME-Version: 1.0 Cc: "Patel, Nirmal" , "will@kernel.org" , "Karkra, Kapil" , "iommu@lists.linux-foundation.org" , "linux-pci@vger.kernel.org" , "dwmw2@infradead.org" X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Fri, 2021-02-05 at 15:57 -0600, Bjorn Helgaas wrote: > On Thu, Feb 04, 2021 at 12:09:06PM -0700, Jon Derrick wrote: > > VMD will retransmit child device MSI/X using its own MSI/X table and > > requester-id. This limits the number of MSI/X available to the whole > > child device domain to the number of VMD MSI/X interrupts. > > > > Some VMD devices have a mode where this remapping can be disabled, > > allowing child device interrupts to bypass processing with the VMD MSI/X > > domain interrupt handler and going straight the child device interrupt > > handler, allowing for better performance and scaling. The requester-id > > still gets changed to the VMD endpoint's requester-id, and the interrupt > > remapping handlers have been updated to properly set IRTE for child > > device interrupts to the VMD endpoint's context. > > > > Some VMD platforms have existing production BIOS which rely on MSI/X > > remapping and won't explicitly program the MSI/X remapping bit. This > > re-enables MSI/X remapping on unload. > > Trivial comments below. Would you mind using "MSI-X" instead of > "MSI/X" so it matches the usage in the PCIe specs? Several mentions > above (including subject) and below. Thanks Bjorn, will do > > > Signed-off-by: Jon Derrick > > --- > > drivers/pci/controller/vmd.c | 60 ++++++++++++++++++++++++++++-------- > > 1 file changed, 48 insertions(+), 12 deletions(-) > > > > diff --git a/drivers/pci/controller/vmd.c b/drivers/pci/controller/vmd.c > > index 5e80f28f0119..a319ce49645b 100644 > > --- a/drivers/pci/controller/vmd.c > > +++ b/drivers/pci/controller/vmd.c > > @@ -59,6 +59,13 @@ enum vmd_features { > > * be used for MSI remapping > > */ > > VMD_FEAT_OFFSET_FIRST_VECTOR = (1 << 3), > > + > > + /* > > + * Device can bypass remapping MSI/X transactions into its MSI/X table, > > + * avoding the requirement of a VMD MSI domain for child device > > s/avoding/avoiding/ > > > + * interrupt handling > > Maybe a period at the end of the sentence. > > > + */ > > + VMD_FEAT_BYPASS_MSI_REMAP = (1 << 4), > > }; > > > > /* > > @@ -306,6 +313,15 @@ static struct msi_domain_info vmd_msi_domain_info = { > > .chip = &vmd_msi_controller, > > }; > > > > +static void vmd_enable_msi_remapping(struct vmd_dev *vmd, bool enable) > > +{ > > + u16 reg; > > + > > + pci_read_config_word(vmd->dev, PCI_REG_VMCONFIG, ®); > > + reg = enable ? (reg & ~0x2) : (reg | 0x2); > > Would be nice to have a #define for 0x2. > > > + pci_write_config_word(vmd->dev, PCI_REG_VMCONFIG, reg); > > +} > > + > > static int vmd_create_irq_domain(struct vmd_dev *vmd) > > { > > struct fwnode_handle *fn; > > @@ -325,6 +341,13 @@ static int vmd_create_irq_domain(struct vmd_dev *vmd) > > > > static void vmd_remove_irq_domain(struct vmd_dev *vmd) > > { > > + /* > > + * Some production BIOS won't enable remapping between soft reboots. > > + * Ensure remapping is restored before unloading the driver. > > + */ > > + if (!vmd->msix_count) > > + vmd_enable_msi_remapping(vmd, true); > > + > > if (vmd->irq_domain) { > > struct fwnode_handle *fn = vmd->irq_domain->fwnode; > > > > @@ -679,15 +702,31 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) > > > > sd->node = pcibus_to_node(vmd->dev->bus); > > > > - ret = vmd_create_irq_domain(vmd); > > - if (ret) > > - return ret; > > - > > /* > > - * Override the irq domain bus token so the domain can be distinguished > > - * from a regular PCI/MSI domain. > > + * Currently MSI remapping must be enabled in guest passthrough mode > > + * due to some missing interrupt remapping plumbing. This is probably > > + * acceptable because the guest is usually CPU-limited and MSI > > + * remapping doesn't become a performance bottleneck. > > */ > > - irq_domain_update_bus_token(vmd->irq_domain, DOMAIN_BUS_VMD_MSI); > > + if (!(features & VMD_FEAT_BYPASS_MSI_REMAP) || offset[0] || offset[1]) { > > + ret = vmd_alloc_irqs(vmd); > > + if (ret) > > + return ret; > > + > > + vmd_enable_msi_remapping(vmd, true); > > + > > + ret = vmd_create_irq_domain(vmd); > > + if (ret) > > + return ret; > > + > > + /* > > + * Override the irq domain bus token so the domain can be > > + * distinguished from a regular PCI/MSI domain. > > + */ > > + irq_domain_update_bus_token(vmd->irq_domain, DOMAIN_BUS_VMD_MSI); > > + } else { > > + vmd_enable_msi_remapping(vmd, false); > > + } > > > > pci_add_resource(&resources, &vmd->resources[0]); > > pci_add_resource_offset(&resources, &vmd->resources[1], offset[0]); > > @@ -753,10 +792,6 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id) > > if (features & VMD_FEAT_OFFSET_FIRST_VECTOR) > > vmd->first_vec = 1; > > > > - err = vmd_alloc_irqs(vmd); > > - if (err) > > - return err; > > - > > spin_lock_init(&vmd->cfg_lock); > > pci_set_drvdata(dev, vmd); > > err = vmd_enable_domain(vmd, features); > > @@ -825,7 +860,8 @@ static const struct pci_device_id vmd_ids[] = { > > .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP,}, > > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_VMD_28C0), > > .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW | > > - VMD_FEAT_HAS_BUS_RESTRICTIONS,}, > > + VMD_FEAT_HAS_BUS_RESTRICTIONS | > > + VMD_FEAT_BYPASS_MSI_REMAP,}, > > {PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x467f), > > .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP | > > VMD_FEAT_HAS_BUS_RESTRICTIONS | > > -- > > 2.27.0 > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu