From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 25A28CE7B19 for ; Fri, 6 Sep 2024 14:10:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ujb0GqJ3xRZSHqC4dECaWBTeqQNVyJrVkjDvMYA7ws4=; b=FVHMvI8SH8ct3NQeqtT3TC0JJe t2uoxhu9nvkKTvwywp95GyYr5c3EW5+pGi/kOBnYkWk1IyRGH4wSakHvcyFUgWPHKc5ApYrglXoWm qIG679+S37jyRUSHXUiGAVqxJcRJY29fFwQkuVC2/9f/vENNGmdnb9eoi86r+Hes3tgkXrDUXBsnV rO9XhiNBhxPGE6x/fOdYyKwbZiwfDfqZmG8vWIVWsWv8XyATTVz5QZvwGOnrZqTmwRZHQFdovhM7x atwKehuiuSLUh+/cO+7mqdRne0RL6s/3elBU4ZtzsWRnIkx7g8kj1oNCuRqifqa0Ya9TBPOqLJ3t/ 8js2tkXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1smZf1-0000000CTbl-2Lja; Fri, 06 Sep 2024 14:10:03 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1smZe1-0000000CTNn-0y7d for linux-arm-kernel@lists.infradead.org; Fri, 06 Sep 2024 14:09:03 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 1BA2CA44FB8; Fri, 6 Sep 2024 14:08:53 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C00E7C4CEC4; Fri, 6 Sep 2024 14:08:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1725631740; bh=tv+Skcv7zYTt33ybSpFMuvSve7Xp3+8oAkSEA7XyXOQ=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=gKVuvlD2kt7HjyuLIhHM8J4yDo+MukLXpcI2OHoJG/V79ZQb8Pu41Dp2hBE+QSoUG QWEtNq/0PzPN5ZQaFHScZ2yCNeM6uMi94kMg4NPMHu7HgRYncurOVPPc2S14vr3Kaz hPRTO6Ig9kPmwvh9oCNKFwcGZ8r9hl48tBTBvbGp74lCbrZXIsuK3jli7RFzBuwxAw MtCT9Pz1ixeI2JXi8gI/h+Wrqll6Dt3culk9yJeRI0hocIs/W8u4iNmAQRbtgvM548 RxRd5NgKyVa7suFCL8bKJ1MYCbvdhT8rn836bHJDSDY+c6SttZ/R4c6+6Ml2SfXpvW YC5CmdZ3FbTtw== Date: Fri, 6 Sep 2024 15:08:54 +0100 From: Will Deacon To: Jason Gunthorpe Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Eric Auger , Jean-Philippe Brucker , Jerry Snitselaar , Moritz Fischer , Michael Shavit , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v9 02/14] iommu/arm-smmu-v3: Start building a generic PASID layer Message-ID: <20240906140853.GA17187@willie-the-truck> References: <0-v9-5cd718286059+79186-smmuv3_newapi_p2b_jgg@nvidia.com> <2-v9-5cd718286059+79186-smmuv3_newapi_p2b_jgg@nvidia.com> <20240702145705.GA4135@willie-the-truck> <20240709193905.GL107163@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240709193905.GL107163@nvidia.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240906_070901_421382_56BDFD5D X-CRM114-Status: GOOD ( 35.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Jason, Sorry, it's taken me ages to get back to this after applying the series. On Tue, Jul 09, 2024 at 04:39:05PM -0300, Jason Gunthorpe wrote: > On Tue, Jul 02, 2024 at 03:57:05PM +0100, Will Deacon wrote: > > > > @@ -611,10 +599,9 @@ void arm_smmu_sva_remove_dev_pasid(struct iommu_domain *domain, > > > struct arm_smmu_bond *bond = NULL, *t; > > > struct arm_smmu_master *master = dev_iommu_priv_get(dev); > > > > > > + arm_smmu_remove_pasid(master, to_smmu_domain(domain), id); > > > + > > > mutex_lock(&sva_lock); > > > - > > > - arm_smmu_clear_cd(master, id); > > > > This looks a bit alarming, as you're effectively moving the CD > > modification outside of the critical section. I assume we're relying on > > the iommu group mutex to serialise this in the caller? I can't see any > > consistent locking in the driver for arm_smmu_clear_cd(). > > I see Nicolin got this - but yes, sva_lock has nothing to do with > CD. CD/STE has always been protected by the group_mutex. > > > As an additional patch, perhaps we should consider documenting what each > > lock in the driver protects and the lock ordering requirements they > > have? > > There is still a bunch of rework to do here, it may be better to > complete the rework than to try to document it, but let me know which > ones you are interested in and I'll write some thing. I think listing the locks we have in the driver and describing both what they protect and the ordering between them would be really helpful. > sva_lock is almost gone, it just locks the IOPF flow on the master > because we are doing the IOPF flow in the wrong place. The IOPF > enable/disbale should be done in attach under the group mutex. > > init_mutex will be deleted once the iommu_domain_alloc_paging() > conversion is done. > > asid_lock.. Is a place holder for the nascent BTM support. It locks > domain->asid only. The BTM patches make this per-instance instead of > global. Unfortunately that locking scheme doesn't work 100%, but I > have a notion how to fix it.. > > asid_lock is also going to need some reconsidering when we make the > domain able to attach to multiple instances which is something iommufd > wants. > > devices_lock protects the device list only, excluding the special > nr_ats_masters thing. > > Then the hidden group mutex makes all the ops touching master single > threaded, and the driver has always quietly relied on this. It > protects the STE/CD and parts of the master. > > So I think we end up with only the group mutex, asid_lock and > devices_lock it is OK. > > The order is group -> asid -> devices, which is layed out clearly in > the attach functions. > > > We've got a few global locks with generic names and after a few > > rounds of refactoring it's really hard to know who's responsible for > > what, especially now that we have stale comments referring to > > arm_smmu_share_asid(). > > > We've also grown a number of places where we > > drop a lock in the callee and immediately re-take it in the caller, > > which tends to be a source of bugs. > > Do we? Can you point to what you noticed? As I recall, I just noticed that: - We have a bunch of comments around 'arm_smmu_asid_lock' that refer to arm_smmu_share_asid(), which no longer exists. - arm_smmu_remove_dev_pasid() drops the asid_lock only to have it retaken in the callee via ->attach_dev(). - arm_smmu_attach_dev() takes/drops/re-takes the devices_lock indirectly when it calls arm_smmu_attach_prepare() and arm_smmu_attach_commit(). - arm_smmu_attach_dev() takes/drops 'arm_smmu_asid_lock' via arm_smmu_domain_finalise()) and then re-takes it before the attach. Please note, I'm not saying that there's a bug here, just that it would be easier to work with if we had some documentation and lock ordering assertions. Will