From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D23B13C6600 for ; Thu, 19 Mar 2026 11:14:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.113 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773918862; cv=none; b=PEYpJwwOCUHMSm/7lqBrf4BJVb57o70xE0Xp3sk+O712oFkFR9y/9vbscVjz4gyGZ2aGCKaSTn/SACsCnOaf9RCXc/0q7BTSqC/mPn/fnW2fx12LRrGPz9swGKRz5/0IZRlbL2xO1NPZQ4H636CCPGPkCJVbBl8mWrDeSMvlBzM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773918862; c=relaxed/simple; bh=b1tM+V5dwKzsd7BnMNYgg6No/d9bOTd4ipJd+ZEuaIs=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=GTTA0rnb20uPfi54uLw2zvNaAYnUf6rILgg8gfuDkYYT+CtgkeZDOfU5R41mvXZ4J6g87TPXuzOJ31XjbNDA5nPlrOWjxo5KrHplU0jkGGjJIcnRjxEn5zE+eIz1zCEBLPb1XAeYGmN5iaq6mBTWe7rUFgVwTvAAEqxl/I/pDJs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=AshwTswT; arc=none smtp.client-ip=115.124.30.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="AshwTswT" DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1773918851; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=KziEQjzET2tmGaf79QP1vQNf62aUkhu64wFtz1OLZ7A=; b=AshwTswTh3iyjdLaGw/42Az5k6snL1ZbTegJoLa0JQKj+BRqWv3g235uPNETt3Ni368PzTQsfMs7FgEgugPEu/gmw6/wafdkALS8inufhRCUnv9WfgH4TM4Lohde+z5CkSpphW+eIiuNpgo3OktwELkV1WVpsbuDpMkba71/6DI= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037009110;MF=xueshuai@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0X.IN-cS_1773918850; Received: from 30.246.163.250(mailfrom:xueshuai@linux.alibaba.com fp:SMTPD_---0X.IN-cS_1773918850 cluster:ay36) by smtp.aliyun-inc.com; Thu, 19 Mar 2026 19:14:11 +0800 Message-ID: Date: Thu, 19 Mar 2026 19:14:21 +0800 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH rc v2] iommu: Fix nested pci_dev_reset_iommu_prepare/done() To: Nicolin Chen , joro@8bytes.org, kevin.tian@intel.com Cc: will@kernel.org, robin.murphy@arm.com, baolu.lu@linux.intel.com, jgg@nvidia.com, iommu@lists.linux.dev, linux-kernel@vger.kernel.org References: <20260319043135.1153534-1-nicolinc@nvidia.com> From: Shuai Xue In-Reply-To: <20260319043135.1153534-1-nicolinc@nvidia.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 3/19/26 12:31 PM, Nicolin Chen wrote: > Shuai found that cxl_reset_bus_function() calls pci_reset_bus_function() > internally while both are calling pci_dev_reset_iommu_prepare/done(). > > As pci_dev_reset_iommu_prepare() doesn't support re-entry, the inner call > will trigger a WARN_ON and return -EBUSY, resulting in failing the entire > device reset. > > On the other hand, removing the outer calls in the PCI callers is unsafe. > As pointed out by Kevin, device-specific quirks like reset_hinic_vf_dev() > execute custom firmware waits after their inner pcie_flr() completes. If > the IOMMU protection relies solely on the inner reset, the IOMMU will be > unblocked prematurely while the device is still resetting. > > Instead, fix this by making pci_dev_reset_iommu_prepare/done() reentrant. > > Introduce a 'reset_cnt' in struct iommu_group. Safely increment/decrement > the reference counter in the nested calls, ensuring the IOMMU domains are > only restored when the outermost reset finally completes. > > Fixes: c279e83953d9 ("iommu: Introduce pci_dev_reset_iommu_prepare/done()") > Cc: stable@vger.kernel.org > Reported-by: Shuai Xue > Closes: https://lore.kernel.org/all/absKsk7qQOwzhpzv@Asurada-Nvidia/ > Suggested-by: Kevin Tian > Signed-off-by: Nicolin Chen > --- > Changelog > v2: > * Fix in the helpers by allowing re-entry > v1: > https://lore.kernel.org/all/20260318220028.1146905-1-nicolinc@nvidia.com/ > > drivers/iommu/iommu.c | 15 ++++++++++++--- > 1 file changed, 12 insertions(+), 3 deletions(-) > > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 35db51780954..16155097b27c 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -68,6 +68,7 @@ struct iommu_group { > struct iommu_domain *resetting_domain; > struct iommu_domain *domain; > struct list_head entry; > + unsigned int reset_cnt; Nit: consider renaming reset_cnt to reset_depth or reset_nesting to better convey that this tracks nesting level, not a count of total resets. > unsigned int owner_cnt; > void *owner; > }; > @@ -3961,9 +3962,10 @@ int pci_dev_reset_iommu_prepare(struct pci_dev *pdev) > > guard(mutex)(&group->mutex); > > - /* Re-entry is not allowed */ > - if (WARN_ON(group->resetting_domain)) > - return -EBUSY; > + if (group->resetting_domain) { > + group->reset_cnt++; > + return 0; > + } > > ret = __iommu_group_alloc_blocking_domain(group); pci_dev_reset_iommu_prepare/done() have NO singleton group check. They operate on the specific pdev passed in, but use group-wide state (resetting_domain, reset_cnt) to track the reset lifecycle. Interestingly, the broken_worker in patch 3 of the ATC timeout series DOES have an explicit singleton check: if (list_is_singular(&group->devices)) { /* Note: only support group with a single device */ This reveals an implicit assumption: the entire prepare/done mechanism works correctly only for singleton groups. For multi-device groups: - prepare() only detaches the specific pdev, leaving other devices in the group still attached to the original domain - The group-wide resetting_domain/reset_cnt state can be corrupted by concurrent resets on different devices (as discussed above) If prepare/done is truly meant only for singleton groups, it should enforce this explicitly: if (!list_is_singular(&group->devices)) return -EOPNOTSUPP; If it's meant to support multi-device groups, then the per-device vs group-wide state mismatch needs to be resolved — either by making the state per-device, or by detaching/restoring all devices in the group together. Thanks. Shuai