From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fout-b6-smtp.messagingengine.com (fout-b6-smtp.messagingengine.com [202.12.124.149]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6768A280CD2; Wed, 4 Feb 2026 14:48:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.149 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770216488; cv=none; b=Td3iNlPcmuQeFThy7cXwmYMhj5iFjGd87dEjQR4TRqYfwY+6EI3E7XeyKs2YYIjRoUaheqt9afS//+RLeTMQpomZS8myI5f8nmogtDJpGA5A0SgNG4cEi/qxmlWU5lCvtS8pisWe/vdmIRC+h/cqNkLOXD2Ge84Tyez0V1orq7Y= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1770216488; c=relaxed/simple; bh=ukFfnHNJ5St3Rj6hRUImngtiQiwt7CXxdu0IEyYKDrw=; h=Date:From:To:Cc:Subject:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LfxRs5/2G0/9qAJrt8Y4E24AVmWY3M+OC5Jq8N8mUc3Fp3pdo1gyPduh51yuodalFyW3udkNPsF9qdYE4zZ38WG/bOznRw58oVSOCafEhMCXqa32NCz+n1N7QkqATe8/vvuUunzh4MIXZ/LrF8xlGOqFDNB+fODhpEgqaC6NByo= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=shazbot.org; spf=pass smtp.mailfrom=shazbot.org; dkim=pass (2048-bit key) header.d=shazbot.org header.i=@shazbot.org header.b=0o8WaWC+; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=jzmMz8cj; arc=none smtp.client-ip=202.12.124.149 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=shazbot.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shazbot.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shazbot.org header.i=@shazbot.org header.b="0o8WaWC+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jzmMz8cj" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfout.stl.internal (Postfix) with ESMTP id 459111D00099; Wed, 4 Feb 2026 09:48:05 -0500 (EST) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Wed, 04 Feb 2026 09:48:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shazbot.org; h= cc:cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm2; t=1770216485; x=1770302885; bh=OAqECMTNWylYMxvQZWo48cfQCG+FzsGTUnm7TRzruJc=; b= 0o8WaWC+JS0r9CyNO/nTgXfRQDJ56/yoy70axLrKphXjIXJxrk75jlyRIc0mFozg Lk9yZLXvit+j0LAW6FO4NKvl1M/1uw0AqI/Kv0zLHGMB4xaXYj7dVhwRDcsAdFbe XldCsL0y3BWhA1RU+xIRj0DlrCxgJ+y9G+eS5tdLTz/xq9KeKueV28uAkNWAkYki JFt10+lcANDhqdHiJ+aWcx0rZPmqY4AcPIAS78Sywq78D8ETU6OsQVbv4v/NDVLx tcRW2AWOeJfeGEo9hYpBcCvKQi7VmCM0uEGGCIeEXOrD7uBLnfVA8mCfURXdb+Zo 0O72ph1Y+QfDvfTonvb22Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1770216485; x= 1770302885; bh=OAqECMTNWylYMxvQZWo48cfQCG+FzsGTUnm7TRzruJc=; b=j zmMz8cjFVoG/rsm+0syFV2GjH1G9Jl5OSmRwqy86uRGwWgXestcZk51Jl2OtfOhi 5dT2To2mQwZKbEcO3KiJ6YVzN/OWqj+r5xAY4f4Pcl47DGV+vjJCwJT82kWxuiYI U3D3g6AIwCcaYR2plHhaB4epoyk0ucC5lxbtg6+tuSzhDaJBG2MwMz5qymt6du14 u5x4iw9+Urw3wJMot+A2nGSTxrNjdniDBsJvL0IV/uztGqUeYAV7FueJ6jd7hMI2 V0Qujs8/84hPQEZQbg81ddFd/smKComYzHat9t/3Z/A9zAv532+oJhyaXykyLIuz /MklR4ShSvjuoAZODm5ow== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefgedrtddtgddukedvjedvucetufdoteggodetrf dotffvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfurfetoffkrfgpnffqhgenuceu rghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmnecujf gurhepfffhvfevuffkjghfofggtgfgsehtjeertdertddvnecuhfhrohhmpeetlhgvgicu hghilhhlihgrmhhsohhnuceorghlvgigsehshhgriigsohhtrdhorhhgqeenucggtffrrg htthgvrhhnpedvkeefjeekvdduhfduhfetkedugfduieettedvueekvdehtedvkefgudeg veeuueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpe grlhgvgiesshhhrgiisghothdrohhrghdpnhgspghrtghpthhtohepfeegpdhmohguvgep shhmthhpohhuthdprhgtphhtthhopehlvghonheskhgvrhhnvghlrdhorhhgpdhrtghpth htohepshhumhhithdrshgvmhifrghlsehlihhnrghrohdrohhrghdprhgtphhtthhopegt hhhrihhsthhirghnrdhkohgvnhhighesrghmugdrtghomhdprhgtphhtthhopegrlhgvgi grnhguvghrrdguvghutghhvghrsegrmhgurdgtohhmpdhrtghpthhtoheprghirhhlihgv ugesghhmrghilhdrtghomhdprhgtphhtthhopehsihhmohhnrgesfhhffihllhdrtghhpd hrtghpthhtohepkhhrrgigvghlsehrvgguhhgrthdrtghomhdprhgtphhtthhopegumhhi thhrhidrohhsihhpvghnkhhosegtohhllhgrsghorhgrrdgtohhmpdhrtghpthhtohepgh hurhgthhgvthgrnhhsihhnghhhsegthhhrohhmihhumhdrohhrgh X-ME-Proxy: Feedback-ID: i03f14258:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 4 Feb 2026 09:48:01 -0500 (EST) Date: Wed, 4 Feb 2026 07:47:59 -0700 From: Alex Williamson To: Leon Romanovsky Cc: Sumit Semwal , Christian =?UTF-8?B?S8O2bmln?= , Alex Deucher , David Airlie , Simona Vetter , Gerd Hoffmann , Dmitry Osipenko , Gurchetan Singh , Chia-I Wu , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , Lucas De Marchi , Thomas =?UTF-8?B?SGVsbHN0csO2bQ==?= , Rodrigo Vivi , Jason Gunthorpe , Kevin Tian , Joerg Roedel , Will Deacon , Robin Murphy , Felix Kuehling , Ankit Agrawal , Vivek Kasireddy , linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, linux-kernel@vger.kernel.org, amd-gfx@lists.freedesktop.org, virtualization@lists.linux.dev, intel-xe@lists.freedesktop.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, kvm@vger.kernel.org Subject: Re: [PATCH v7 4/8] vfio: Wait for dma-buf invalidation to complete Message-ID: <20260204074759.108f579e@shazbot.org> In-Reply-To: <20260131-dmabuf-revoke-v7-4-463d956bd527@nvidia.com> References: <20260131-dmabuf-revoke-v7-0-463d956bd527@nvidia.com> <20260131-dmabuf-revoke-v7-4-463d956bd527@nvidia.com> X-Mailer: Claws Mail 4.3.1 (GTK 3.24.51; x86_64-pc-linux-gnu) Precedence: bulk X-Mailing-List: linux-media@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit On Sat, 31 Jan 2026 07:34:14 +0200 Leon Romanovsky wrote: > From: Leon Romanovsky > > dma-buf invalidation is handled asynchronously by the hardware, so VFIO > must wait until all affected objects have been fully invalidated. > > In addition, the dma-buf exporter is expecting that all importers unmap any > buffers they previously mapped. > > Fixes: 5d74781ebc86 ("vfio/pci: Add dma-buf export support for MMIO regions") > Reviewed-by: Kevin Tian > Signed-off-by: Leon Romanovsky > --- > drivers/vfio/pci/vfio_pci_dmabuf.c | 61 +++++++++++++++++++++++++++++++++++--- > 1 file changed, 57 insertions(+), 4 deletions(-) Reviewed-by: Alex Williamson > > diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c > index d8ceafabef48..78d47e260f34 100644 > --- a/drivers/vfio/pci/vfio_pci_dmabuf.c > +++ b/drivers/vfio/pci/vfio_pci_dmabuf.c > @@ -17,6 +17,8 @@ struct vfio_pci_dma_buf { > struct dma_buf_phys_vec *phys_vec; > struct p2pdma_provider *provider; > u32 nr_ranges; > + struct kref kref; > + struct completion comp; > u8 revoked : 1; > }; > > @@ -44,27 +46,46 @@ static int vfio_pci_dma_buf_attach(struct dma_buf *dmabuf, > return 0; > } > > +static void vfio_pci_dma_buf_done(struct kref *kref) > +{ > + struct vfio_pci_dma_buf *priv = > + container_of(kref, struct vfio_pci_dma_buf, kref); > + > + complete(&priv->comp); > +} > + > static struct sg_table * > vfio_pci_dma_buf_map(struct dma_buf_attachment *attachment, > enum dma_data_direction dir) > { > struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv; > + struct sg_table *ret; > > dma_resv_assert_held(priv->dmabuf->resv); > > if (priv->revoked) > return ERR_PTR(-ENODEV); > > - return dma_buf_phys_vec_to_sgt(attachment, priv->provider, > - priv->phys_vec, priv->nr_ranges, > - priv->size, dir); > + ret = dma_buf_phys_vec_to_sgt(attachment, priv->provider, > + priv->phys_vec, priv->nr_ranges, > + priv->size, dir); > + if (IS_ERR(ret)) > + return ret; > + > + kref_get(&priv->kref); > + return ret; > } > > static void vfio_pci_dma_buf_unmap(struct dma_buf_attachment *attachment, > struct sg_table *sgt, > enum dma_data_direction dir) > { > + struct vfio_pci_dma_buf *priv = attachment->dmabuf->priv; > + > + dma_resv_assert_held(priv->dmabuf->resv); > + > dma_buf_free_sgt(attachment, sgt, dir); > + kref_put(&priv->kref, vfio_pci_dma_buf_done); > } > > static void vfio_pci_dma_buf_release(struct dma_buf *dmabuf) > @@ -287,6 +308,9 @@ int vfio_pci_core_feature_dma_buf(struct vfio_pci_core_device *vdev, u32 flags, > goto err_dev_put; > } > > + kref_init(&priv->kref); > + init_completion(&priv->comp); > + > /* dma_buf_put() now frees priv */ > INIT_LIST_HEAD(&priv->dmabufs_elm); > down_write(&vdev->memory_lock); > @@ -331,9 +355,33 @@ void vfio_pci_dma_buf_move(struct vfio_pci_core_device *vdev, bool revoked) > > if (priv->revoked != revoked) { > dma_resv_lock(priv->dmabuf->resv, NULL); > - priv->revoked = revoked; > + if (revoked) > + priv->revoked = true; > dma_buf_invalidate_mappings(priv->dmabuf); > + dma_resv_wait_timeout(priv->dmabuf->resv, > + DMA_RESV_USAGE_BOOKKEEP, false, > + MAX_SCHEDULE_TIMEOUT); > dma_resv_unlock(priv->dmabuf->resv); > + if (revoked) { > + kref_put(&priv->kref, vfio_pci_dma_buf_done); > + wait_for_completion(&priv->comp); > + } else { > + /* > + * Kref is initialize again, because when revoke > + * was performed the reference counter was decreased > + * to zero to trigger completion. > + */ > + kref_init(&priv->kref); > + /* > + * There is no need to wait as no mapping was > + * performed when the previous status was > + * priv->revoked == true. > + */ > + reinit_completion(&priv->comp); > + dma_resv_lock(priv->dmabuf->resv, NULL); > + priv->revoked = false; > + dma_resv_unlock(priv->dmabuf->resv); > + } > } > fput(priv->dmabuf->file); > } > @@ -354,7 +402,12 @@ void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev) > priv->vdev = NULL; > priv->revoked = true; > dma_buf_invalidate_mappings(priv->dmabuf); > + dma_resv_wait_timeout(priv->dmabuf->resv, > + DMA_RESV_USAGE_BOOKKEEP, false, > + MAX_SCHEDULE_TIMEOUT); > dma_resv_unlock(priv->dmabuf->resv); > + kref_put(&priv->kref, vfio_pci_dma_buf_done); > + wait_for_completion(&priv->comp); > vfio_device_put_registration(&vdev->vdev); > fput(priv->dmabuf->file); > } >