From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BF8E5C433EF for ; Tue, 14 Jun 2022 02:56:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 827E060758; Tue, 14 Jun 2022 02:56:15 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id n4FnWzGTFiac; Tue, 14 Jun 2022 02:56:14 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 357FC60B00; Tue, 14 Jun 2022 02:56:14 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1146CC0032; Tue, 14 Jun 2022 02:56:14 +0000 (UTC) Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) by lists.linuxfoundation.org (Postfix) with ESMTP id 1275DC002D for ; Tue, 14 Jun 2022 02:56:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id E7637409ED for ; Tue, 14 Jun 2022 02:56:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp4.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=intel.com Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WsnvFRq83N_G for ; Tue, 14 Jun 2022 02:56:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by smtp4.osuosl.org (Postfix) with ESMTPS id C21F141619 for ; Tue, 14 Jun 2022 02:56:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1655175371; x=1686711371; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6v/TvD3cHFaKqMIzUzQPUP7REA0SWO/3xA9jf2Ll40w=; b=PwGh8jGahLnIX1EyLzC0z2Y2wodGG1BDFAZfRotHnhDXSdC/C7n10JT6 qn4MVM+81sCpToZMUiKI+7jbuB8XBYQdIHWG0jiJLc5tuAJsuuuqw03jG jEHE0PMggwsojc4rvDdutYyCe3IgYxykXK5gIUeB8zTMQfYVBDwuO30K1 r10ZVRRrwQYKqvJuPE3/38Di4IKpcfHZBk9WxoAVN3fwWaYAjxVcfkQZu PELCpc9f8RLLHKpMq1qBzjEu0Vm4oYAPQl/ZKbZL0pPF4Z7WFU/WCWQwS 7NEW+ymYfgL7RcPAnDFWuWhUrd9SWCDfNPMG9rJjae//x9EKYYmFRnF7r A==; X-IronPort-AV: E=McAfee;i="6400,9594,10377"; a="303887180" X-IronPort-AV: E=Sophos;i="5.91,298,1647327600"; d="scan'208";a="303887180" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jun 2022 19:56:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.91,298,1647327600"; d="scan'208";a="588166779" Received: from allen-box.sh.intel.com ([10.239.159.48]) by fmsmga007.fm.intel.com with ESMTP; 13 Jun 2022 19:56:08 -0700 From: Lu Baolu To: Joerg Roedel , Kevin Tian , Ashok Raj , Christoph Hellwig , Jason Gunthorpe Subject: [PATCH v2 12/12] iommu/vt-d: Convert global spinlock into per domain ones Date: Tue, 14 Jun 2022 10:51:37 +0800 Message-Id: <20220614025137.1632762-13-baolu.lu@linux.intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220614025137.1632762-1-baolu.lu@linux.intel.com> References: <20220614025137.1632762-1-baolu.lu@linux.intel.com> MIME-Version: 1.0 Cc: Will Deacon , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, Jacob jun Pan , Robin Murphy X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" Using a global device_domain_lock spinlock to protect per-domain device tracking lists is an inefficient way, especially considering this lock is also needed in the hot paths. This optimizes the locking mechanism by converting the global lock to per domain lock. On the other hand, as the device tracking lists are never accessed in any interrupt context, there is no need to disable interrupts while spinning. Replace irqsave variant with spinlock calls. Signed-off-by: Lu Baolu --- drivers/iommu/intel/iommu.h | 1 + drivers/iommu/intel/iommu.c | 45 +++++++++++++++---------------------- 2 files changed, 19 insertions(+), 27 deletions(-) diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h index 6724703d573b..cc304ff09a7b 100644 --- a/drivers/iommu/intel/iommu.h +++ b/drivers/iommu/intel/iommu.h @@ -541,6 +541,7 @@ struct dmar_domain { u8 force_snooping : 1; /* Create IOPTEs with snoop control */ u8 set_pte_snp:1; + spinlock_t lock; /* Protect device tracking lists */ struct list_head devices; /* all devices' list */ struct dma_pte *pgd; /* virtual address */ diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index aa3dea1c9f13..60e70682a190 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -310,7 +310,6 @@ static int iommu_skip_te_disable; #define IDENTMAP_GFX 2 #define IDENTMAP_AZALIA 4 -static DEFINE_SPINLOCK(device_domain_lock); const struct iommu_ops intel_iommu_ops; static bool translation_pre_enabled(struct intel_iommu *iommu) @@ -534,9 +533,8 @@ static int domain_update_device_node(struct dmar_domain *domain) { struct device_domain_info *info; int nid = NUMA_NO_NODE; - unsigned long flags; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_for_each_entry(info, &domain->devices, link) { /* * There could possibly be multiple device numa nodes as devices @@ -548,7 +546,7 @@ static int domain_update_device_node(struct dmar_domain *domain) if (nid != NUMA_NO_NODE) break; } - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); return nid; } @@ -1375,12 +1373,11 @@ iommu_support_dev_iotlb(struct dmar_domain *domain, struct intel_iommu *iommu, u8 bus, u8 devfn) { struct device_domain_info *info = NULL, *tmp; - unsigned long flags; if (!iommu->qi) return NULL; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_for_each_entry(tmp, &domain->devices, link) { if (tmp->iommu == iommu && tmp->bus == bus && tmp->devfn == devfn) { @@ -1389,7 +1386,7 @@ iommu_support_dev_iotlb(struct dmar_domain *domain, struct intel_iommu *iommu, break; } } - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); return info; } @@ -1398,9 +1395,8 @@ static void domain_update_iotlb(struct dmar_domain *domain) { struct device_domain_info *info; bool has_iotlb_device = false; - unsigned long flags; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_for_each_entry(info, &domain->devices, link) { if (info->ats_enabled) { has_iotlb_device = true; @@ -1408,7 +1404,7 @@ static void domain_update_iotlb(struct dmar_domain *domain) } } domain->has_iotlb_device = has_iotlb_device; - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); } static void iommu_enable_dev_iotlb(struct device_domain_info *info) @@ -1499,17 +1495,15 @@ static void __iommu_flush_dev_iotlb(struct device_domain_info *info, static void iommu_flush_dev_iotlb(struct dmar_domain *domain, u64 addr, unsigned mask) { - unsigned long flags; struct device_domain_info *info; if (!domain->has_iotlb_device) return; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_for_each_entry(info, &domain->devices, link) __iommu_flush_dev_iotlb(info, addr, mask); - - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); } static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, @@ -1769,6 +1763,7 @@ static struct dmar_domain *alloc_domain(unsigned int type) domain->flags |= DOMAIN_FLAG_USE_FIRST_LEVEL; domain->has_iotlb_device = false; INIT_LIST_HEAD(&domain->devices); + spin_lock_init(&domain->lock); return domain; } @@ -2442,7 +2437,6 @@ static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev) { struct device_domain_info *info = dev_iommu_priv_get(dev); struct intel_iommu *iommu; - unsigned long flags; u8 bus, devfn; int ret; @@ -2454,9 +2448,9 @@ static int domain_add_dev_info(struct dmar_domain *domain, struct device *dev) if (ret) return ret; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_add(&info->link, &domain->devices); - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); info->domain = domain; /* PASID table is mandatory for a PCI device in scalable mode. */ @@ -4134,7 +4128,6 @@ static void dmar_remove_one_dev_info(struct device *dev) struct device_domain_info *info = dev_iommu_priv_get(dev); struct dmar_domain *domain = info->domain; struct intel_iommu *iommu = info->iommu; - unsigned long flags; if (!dev_is_real_dma_subdevice(info->dev)) { if (dev_is_pci(info->dev) && sm_supported(iommu)) @@ -4146,9 +4139,9 @@ static void dmar_remove_one_dev_info(struct device *dev) intel_pasid_free_table(info->dev); } - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&domain->lock); list_del(&info->link); - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&domain->lock); domain_detach_iommu(domain, iommu); } @@ -4432,7 +4425,7 @@ static bool domain_support_force_snooping(struct dmar_domain *domain) struct device_domain_info *info; bool support = true; - assert_spin_locked(&device_domain_lock); + assert_spin_locked(&domain->lock); list_for_each_entry(info, &domain->devices, link) { if (!ecap_sc_support(info->iommu->ecap)) { support = false; @@ -4447,8 +4440,7 @@ static void domain_set_force_snooping(struct dmar_domain *domain) { struct device_domain_info *info; - assert_spin_locked(&device_domain_lock); - + assert_spin_locked(&domain->lock); /* * Second level page table supports per-PTE snoop control. The * iommu_map() interface will handle this by setting SNP bit. @@ -4466,20 +4458,19 @@ static void domain_set_force_snooping(struct dmar_domain *domain) static bool intel_iommu_enforce_cache_coherency(struct iommu_domain *domain) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); - unsigned long flags; if (dmar_domain->force_snooping) return true; - spin_lock_irqsave(&device_domain_lock, flags); + spin_lock(&dmar_domain->lock); if (!domain_support_force_snooping(dmar_domain)) { - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&dmar_domain->lock); return false; } domain_set_force_snooping(dmar_domain); dmar_domain->force_snooping = true; - spin_unlock_irqrestore(&device_domain_lock, flags); + spin_unlock(&dmar_domain->lock); return true; } -- 2.25.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu