From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBC1E185213; Fri, 29 Mar 2024 12:42:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711716124; cv=none; b=X9VwY9zxPYXLnL4KX5JdoTAmcCzbVjeU806WICZspYy4Gn7M30SfTC3IRb7IQqw4LT8UND35sbxLhBfXlZReG7VVRB1ocmUEITp4KoZ/gw5oLwtPxpN2w9V9yhmj/aNauL0MWgOBgZW1cadN+CUKwi08gmHn5XBwZJ+le1gnUTI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711716124; c=relaxed/simple; bh=Gy/fCpdLTfmFkgTs9RjvmJNa3ujJqwIxDi6ILMKZabo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q4PfxWf3JoujvW/TNjQcMExAS7AGQTvjm/tyIU87xQ41oKt363Qn71SLktCdfpVuUAANQvZ8wHHlI7CLF8bm10DZILTbUi+jAElp5ESLVvqgJ+NpZC3r1BFn0RMmpCyoA6N1dADVDU4AqnJMvEcoX5CgFFYJTTBLRHOl5WvI5Ok= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oHjZ0Kps; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oHjZ0Kps" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2B25C433F1; Fri, 29 Mar 2024 12:42:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711716123; bh=Gy/fCpdLTfmFkgTs9RjvmJNa3ujJqwIxDi6ILMKZabo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oHjZ0Kps3vOgNYxhS+346XgvBVNl6tWev2SgQ/b8CXkk6S+CcOdrOTJJiWPcPzKN7 n+mXFwF3i6vFzpptcRCD1tXnYL+wJoXBc8dCzX0u0hdxgkWG3O/fc4ITLgNjV1yiNU 1penVTT59Q2OMIwIih09pzdndoKXTKNTXRvGLwHn5LDzpIZb34Docwe3tEllU/JBk2 T7KSt9BpLp5rGQfjM8DRRG4dB9iFwR5mgKnwHPhdTNqTBRcoAJTTHeMZ8nMytEZWBI XsFf0QqhxkeyPAPDNBlpE6hU/vRLOxPg6rqq9mAGShDzNS3pZ2MGn+diyoZ+9JI9Cx MuIGtukZ2o6uw== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Jason Gunthorpe , Michael Shavit , Nicolin Chen , Mostafa Saleh , Shameer Kolothum , Moritz Fischer , Will Deacon , Sasha Levin , joro@8bytes.org, jgg@ziepe.ca, robin.murphy@arm.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Subject: [PATCH AUTOSEL 6.8 66/98] iommu/arm-smmu-v3: Hold arm_smmu_asid_lock during all of attach_dev Date: Fri, 29 Mar 2024 08:37:37 -0400 Message-ID: <20240329123919.3087149-66-sashal@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329123919.3087149-1-sashal@kernel.org> References: <20240329123919.3087149-1-sashal@kernel.org> Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore X-stable-base: Linux 6.8.2 Content-Transfer-Encoding: 8bit From: Jason Gunthorpe [ Upstream commit 9f7c68911579bc15c57d227d021ccd253da2b635 ] The BTM support wants to be able to change the ASID of any smmu_domain. When it goes to do this it holds the arm_smmu_asid_lock and iterates over the target domain's devices list. During attach of a S1 domain we must ensure that the devices list and CD are in sync, otherwise we could miss CD updates or a parallel CD update could push an out of date CD. This is pretty complicated, and almost works today because arm_smmu_detach_dev() removes the master from the linked list before working on the CD entries, preventing parallel update of the CD. However, it does have an issue where the CD can remain programed while the domain appears to be unattached. arm_smmu_share_asid() will then not clear any CD entriess and install its own CD entry with the same ASID concurrently. This creates a small race window where the IOMMU can see two ASIDs pointing to different translations. CPU0 CPU1 arm_smmu_attach_dev() arm_smmu_detach_dev() spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_del(&master->domain_head); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_mmu_notifier_get() arm_smmu_alloc_shared_cd() arm_smmu_share_asid(): // Does nothing due to list_del above arm_smmu_update_ctx_desc_devices() arm_smmu_tlb_inv_asid() arm_smmu_write_ctx_desc() ** Now the ASID is in two CDs with different translation arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL); Solve this by wrapping most of the attach flow in the arm_smmu_asid_lock. This locks more than strictly needed to prepare for the next patch which will reorganize the order of the linked list, STE and CD changes. Move arm_smmu_detach_dev() till after we have initialized the domain so the lock can be held for less time. Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Tested-by: Shameer Kolothum Tested-by: Nicolin Chen Tested-by: Moritz Fischer Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/5-v6-96275f25c39d+2d4-smmuv3_newapi_p1_jgg@nvidia.com Signed-off-by: Will Deacon Signed-off-by: Sasha Levin --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 22 ++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0ffb1cf17e0b2..f3f2e47b6d488 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2398,8 +2398,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -EBUSY; } - arm_smmu_detach_dev(master); - mutex_lock(&smmu_domain->init_mutex); if (!smmu_domain->smmu) { @@ -2414,6 +2412,16 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (ret) return ret; + /* + * Prevent arm_smmu_share_asid() from trying to change the ASID + * of either the old or new domain while we are working on it. + * This allows the STE and the smmu_domain->devices list to + * be inconsistent during this routine. + */ + mutex_lock(&arm_smmu_asid_lock); + + arm_smmu_detach_dev(master); + master->domain = smmu_domain; /* @@ -2439,13 +2447,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } } - /* - * Prevent SVA from concurrently modifying the CD or writing to - * the CD entry - */ - mutex_lock(&arm_smmu_asid_lock); ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd); - mutex_unlock(&arm_smmu_asid_lock); if (ret) { master->domain = NULL; goto out_list_del; @@ -2455,13 +2457,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_install_ste_for_dev(master); arm_smmu_enable_ats(master); - return 0; + goto out_unlock; out_list_del: spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_del(&master->domain_head); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +out_unlock: + mutex_unlock(&arm_smmu_asid_lock); return ret; } -- 2.43.0 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 044BCCD11DD for ; Fri, 29 Mar 2024 12:45:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-ID:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yWlmzQnH4ZBYTfyeD3oeRZO/ZtAw2tH2HVgnFAiqiWY=; b=YKto9l2Hg/xmOa 272+mJXmg3mPz6k9WVOPY6ah0VFF9Q8+9mvDltdXRdEQnzSu2p+9BloXWXt5xrlPRsktL2FwF+rHg uFq9HXKaN9pkrRNtviD6xFjElbFlKhUrdFYyUWOBjuCeQ6ODw9oa71Ao/d+5XaqDNRJiNN3BzoBzg eqJvHMY6uVNntNXNjR/l7kcKKnIRcZeDU7SCzhNe9Jt0GJf9t7pCAKl6vQgKzwqBFnvocge4vukk6 D18t5O/XAd3mJYNNyA0ii0XoWRIGO+o74tnzJRP7YxGddSE1hNuiwk7LU/S6wvgxJx5RvWCM3+PpK AFe4brYUgKTOYU9miodQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rqBb8-00000000RS1-1JDW; Fri, 29 Mar 2024 12:44:43 +0000 Received: from desiato.infradead.org ([2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rqBYj-00000000Qj2-0XjC for linux-arm-kernel@bombadil.infradead.org; Fri, 29 Mar 2024 12:42:15 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Transfer-Encoding:MIME-Version :References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=20mSidlvTpoIOGMSsu5DJL6q+TDuIvPua1tu6Dfq5UU=; b=Is8dUjI+1WDDoDrEyaW4Uk65yL vRAO+mE+xah1YilPWBSzIksEUH++MPuc20hE11MkBSoZEllbbmcQvyeBlgEdQJ/SnaQdNgSmd1+6l 45V7xW9wS7Axll2wowc+MeWB6sLVnLTsO6Rt+eTJU70MDLwxdOqOFYY4KShJzGkedHkXSCPLHzJEB /eUPvFhdhHU1OWpWEv4IorIKf2pq/sBpV+A9s8w0rzs+wP4SBTDVpTjtcrZhX8kOaB1FTailWu8Xz QRoy5mJwuM2ELWrCrzy6gxFxO5D6sUGzZiy1ijv4pRaLcrNzEuv2P24KHprUOZCiwi4wOMg/lRqDp 57Sjfsxg==; Received: from sin.source.kernel.org ([2604:1380:40e1:4800::1]) by desiato.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rqBYe-00000001Z34-1hdO for linux-arm-kernel@lists.infradead.org; Fri, 29 Mar 2024 12:42:11 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 30980CE2F9A; Fri, 29 Mar 2024 12:42:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C2B25C433F1; Fri, 29 Mar 2024 12:42:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1711716123; bh=Gy/fCpdLTfmFkgTs9RjvmJNa3ujJqwIxDi6ILMKZabo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oHjZ0Kps3vOgNYxhS+346XgvBVNl6tWev2SgQ/b8CXkk6S+CcOdrOTJJiWPcPzKN7 n+mXFwF3i6vFzpptcRCD1tXnYL+wJoXBc8dCzX0u0hdxgkWG3O/fc4ITLgNjV1yiNU 1penVTT59Q2OMIwIih09pzdndoKXTKNTXRvGLwHn5LDzpIZb34Docwe3tEllU/JBk2 T7KSt9BpLp5rGQfjM8DRRG4dB9iFwR5mgKnwHPhdTNqTBRcoAJTTHeMZ8nMytEZWBI XsFf0QqhxkeyPAPDNBlpE6hU/vRLOxPg6rqq9mAGShDzNS3pZ2MGn+diyoZ+9JI9Cx MuIGtukZ2o6uw== From: Sasha Levin To: linux-kernel@vger.kernel.org, stable@vger.kernel.org Cc: Jason Gunthorpe , Michael Shavit , Nicolin Chen , Mostafa Saleh , Shameer Kolothum , Moritz Fischer , Will Deacon , Sasha Levin , joro@8bytes.org, jgg@ziepe.ca, robin.murphy@arm.com, baolu.lu@linux.intel.com, linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev Subject: [PATCH AUTOSEL 6.8 66/98] iommu/arm-smmu-v3: Hold arm_smmu_asid_lock during all of attach_dev Date: Fri, 29 Mar 2024 08:37:37 -0400 Message-ID: <20240329123919.3087149-66-sashal@kernel.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240329123919.3087149-1-sashal@kernel.org> References: <20240329123919.3087149-1-sashal@kernel.org> MIME-Version: 1.0 X-stable: review X-Patchwork-Hint: Ignore X-stable-base: Linux 6.8.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240329_124209_387894_99B49A4D X-CRM114-Status: GOOD ( 20.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org From: Jason Gunthorpe [ Upstream commit 9f7c68911579bc15c57d227d021ccd253da2b635 ] The BTM support wants to be able to change the ASID of any smmu_domain. When it goes to do this it holds the arm_smmu_asid_lock and iterates over the target domain's devices list. During attach of a S1 domain we must ensure that the devices list and CD are in sync, otherwise we could miss CD updates or a parallel CD update could push an out of date CD. This is pretty complicated, and almost works today because arm_smmu_detach_dev() removes the master from the linked list before working on the CD entries, preventing parallel update of the CD. However, it does have an issue where the CD can remain programed while the domain appears to be unattached. arm_smmu_share_asid() will then not clear any CD entriess and install its own CD entry with the same ASID concurrently. This creates a small race window where the IOMMU can see two ASIDs pointing to different translations. CPU0 CPU1 arm_smmu_attach_dev() arm_smmu_detach_dev() spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_del(&master->domain_head); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); arm_smmu_mmu_notifier_get() arm_smmu_alloc_shared_cd() arm_smmu_share_asid(): // Does nothing due to list_del above arm_smmu_update_ctx_desc_devices() arm_smmu_tlb_inv_asid() arm_smmu_write_ctx_desc() ** Now the ASID is in two CDs with different translation arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, NULL); Solve this by wrapping most of the attach flow in the arm_smmu_asid_lock. This locks more than strictly needed to prepare for the next patch which will reorganize the order of the linked list, STE and CD changes. Move arm_smmu_detach_dev() till after we have initialized the domain so the lock can be held for less time. Reviewed-by: Michael Shavit Reviewed-by: Nicolin Chen Reviewed-by: Mostafa Saleh Tested-by: Shameer Kolothum Tested-by: Nicolin Chen Tested-by: Moritz Fischer Signed-off-by: Jason Gunthorpe Link: https://lore.kernel.org/r/5-v6-96275f25c39d+2d4-smmuv3_newapi_p1_jgg@nvidia.com Signed-off-by: Will Deacon Signed-off-by: Sasha Levin --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 22 ++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 0ffb1cf17e0b2..f3f2e47b6d488 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2398,8 +2398,6 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) return -EBUSY; } - arm_smmu_detach_dev(master); - mutex_lock(&smmu_domain->init_mutex); if (!smmu_domain->smmu) { @@ -2414,6 +2412,16 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) if (ret) return ret; + /* + * Prevent arm_smmu_share_asid() from trying to change the ASID + * of either the old or new domain while we are working on it. + * This allows the STE and the smmu_domain->devices list to + * be inconsistent during this routine. + */ + mutex_lock(&arm_smmu_asid_lock); + + arm_smmu_detach_dev(master); + master->domain = smmu_domain; /* @@ -2439,13 +2447,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } } - /* - * Prevent SVA from concurrently modifying the CD or writing to - * the CD entry - */ - mutex_lock(&arm_smmu_asid_lock); ret = arm_smmu_write_ctx_desc(master, IOMMU_NO_PASID, &smmu_domain->cd); - mutex_unlock(&arm_smmu_asid_lock); if (ret) { master->domain = NULL; goto out_list_del; @@ -2455,13 +2457,15 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) arm_smmu_install_ste_for_dev(master); arm_smmu_enable_ats(master); - return 0; + goto out_unlock; out_list_del: spin_lock_irqsave(&smmu_domain->devices_lock, flags); list_del(&master->domain_head); spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); +out_unlock: + mutex_unlock(&arm_smmu_asid_lock); return ret; } -- 2.43.0 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel