From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0b-00082601.pphosted.com (mx0b-00082601.pphosted.com [67.231.153.30]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2DEC3EFD1F for ; Wed, 6 May 2026 15:55:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=67.231.153.30 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778082936; cv=none; b=RM+Pz/p2FfxBnKnv+pgsXBPU7vJzgF4OwygJ4bfmbN+3KYgnJXzO2Bt5CTtqzh5BtdN46AatjHMGUvXpaAxRRMtTMSPtKRZRgpJ5jjTENzm7Wv50eTmoaruGO3duXg+W8I8mqA5RfPQf4dvA0/PQptu7HvfB4bfi0A8mwcKETZs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778082936; c=relaxed/simple; bh=yAE/Ym57GFy/KISCQT6Y8a/+2nm5BztVchD6EC2QOxQ=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=u6sW5Cash9UTooLxGlgqkKG+7/yyz2fnTQyUYXnsUr5rRAKtJktiCglF6lI5CzBGrjysnQTBqXAOctJheEjXZM8umdhfdZryxP3BY36ln/2f7VwwN6dxBm+neI20/kklTFCDsY+tNIc2b+CWcDAr8iNyPL6qoaXl0yk3MdRL7KI= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com; spf=pass smtp.mailfrom=meta.com; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b=V5KJ2Qsv; arc=none smtp.client-ip=67.231.153.30 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=meta.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=meta.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=meta.com header.i=@meta.com header.b="V5KJ2Qsv" Received: from pps.filterd (m0528005.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6466eEbO3782287 for ; Wed, 6 May 2026 08:55:31 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=cc :content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=s2048-2025-q2; bh=Utn027AUC8UDDUJXMpqeQ33wB0VHRXSGiBE+zdS1lww=; b=V5KJ2QsvDqXI jsaAHALFGR6sjzWQOy2R8fDT1m2z8Vp+BEzdkIo2E0PzKZT3MoRqQ2oxFN/cf+RI zNdxXgZ/yC9TXtV3oASmSQ58occoBlUXUmIfLY3rEyKedGP+G26q9XfNbpy/cyyL ATxSfUlKU+0cbiv5isn7bvGMH98bzLSc6Cor0iZ7DZsrSsJL5Y4HlAHYbDQh47an JgHWaoOdPrg/9kinsjy7I1FZDdWBQ1dYEjz8FgwQ2xDR3PrtkNygxgXG3PaHOc5k UAwldnK7l+6FDmkB2j0Yjq5S7uSzqNd6Utnc2cINNmewbNDHUuno2cDWgkWZsFdm 9itz8d4OIQ== Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 4dx2uhwgu1-1 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NOT) for ; Wed, 06 May 2026 08:55:31 -0700 (PDT) Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-44d9ace59efso2828902f8f.1 for ; Wed, 06 May 2026 08:55:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1778082930; x=1778687730; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Utn027AUC8UDDUJXMpqeQ33wB0VHRXSGiBE+zdS1lww=; b=DxK9p7oL3Ua4J7waqlprV3s8h4pS24y3t+TslTE01LrV9iTRKVVf8pc5IIos6Aeelm UD+SSePjZ6wTD2rrXUxFJ/enks7v8i6FPSgjecOagYB7gJLy49P5Sy3klyfPSAFXkijy aHqmCTohCG4OSXHVN+PJLP9x+ZzzGKeDmFdtktFamHSubqBVOhsBPSnbLrIlHx+MusZe fS3efnbYJwT+yTyX6F6zD+KllE6JEM5UKHmIMx03ThaOBXn0Sp/c8LErl/iHyRjM/BtK 5t4RJAZSd3PZ7CwgoOHDHl9/F4OCwgPAcJrFy2+cx2zyw9W8J6g1NBhvoWZPP65w24WQ G05w== X-Forwarded-Encrypted: i=1; AFNElJ9fbm9bPeRZzHM1d0xD4S10xqCjdwLh7a7e9lI9KSbE1zPTg5aAuxqHShHhGxl5eXysceEryoFTZd2bX8I=@vger.kernel.org X-Gm-Message-State: AOJu0YzzR0e7bEtd2IJ0AnpgLlSf+PDYlWLBPjzvjr1cnIzeRbPCjCHY 8stX9dlQJuAcfLzuJXJNKRpT83zXJH+X4KlntpgPnm5n3KTtpCWbV8N9e+SPhPGHknu/Zp+s+7x UD7wNndoBjg8P2yO8tS0R7y98eNW7Byvy1ZcL+5W7toG6blvbOwqCzJdT07CgTsIL X-Gm-Gg: AeBDietIey4DOPasR+TDVluE2oNpU1X84d1bpyC+choPV0jrIALK4TvVbHePEVie1K4 YAU9sq6y0TqrbUY4DhiKcJDDYucCtk+VesiHpj/+Xt+cQA+JyjaR+OKB/l5iMXiNdqYEhyPZPPg 2KesaVhPdY1ZBerjwVuSB2Bk5WbZ6R4z2/CDuPxPTghxzQBKMeZC0/2o0VQM3ZPelAguwQZOy5t e6UVjAFOpJBOMnU6fqqK4JQrsAn2BaMTOH9qvj3B4JFcYFr7QTgmSTdMUiSVh2Re6QssRtaNVsx 0Rb5OlT2+UYwxvGhTeuKkY0h3G29O1wD3N9OyKn9kMY69iSl8w9acvSekiitiOOsbx44MNoCA+N X1r0J6xo8/d86C1vyty/EtNPNjzx3EJNj58F86TxCLwfSlBEKbcpy5wCvjLil0B774IkAYHjlyB bgmyLomEjGEMXBguM1mZbEwLzI+AjN5r/prw/j4PZ3bp52qgREE8F9CYtmDFG07VkL9zDIZoAJz tO3rBh8115nSvLwxBRnlf6kSDN69vVLAA== X-Received: by 2002:a05:6000:2082:b0:441:1c18:f779 with SMTP id ffacd0b85a97d-4515da967c3mr6651399f8f.37.1778082930017; Wed, 06 May 2026 08:55:30 -0700 (PDT) X-Received: by 2002:a05:6000:2082:b0:441:1c18:f779 with SMTP id ffacd0b85a97d-4515da967c3mr6651329f8f.37.1778082929442; Wed, 06 May 2026 08:55:29 -0700 (PDT) Received: from ?IPV6:2001:8b0:8b6:13d4:102e:f2af:e074:5cde? (e.d.c.5.4.7.0.e.f.a.2.f.e.2.0.1.4.d.3.1.6.b.8.0.0.b.8.0.1.0.0.2.ip6.arpa. [2001:8b0:8b6:13d4:102e:f2af:e074:5cde]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-45052483166sm13251331f8f.7.2026.05.06.08.55.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 06 May 2026 08:55:28 -0700 (PDT) Message-ID: Date: Wed, 6 May 2026 16:55:27 +0100 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/9] vfio/pci: Fix vfio_pci_dma_buf_cleanup() double-put Content-Language: en-GB To: Leon Romanovsky Cc: Alex Williamson , Jason Gunthorpe , Alex Mastro , =?UTF-8?Q?Christian_K=C3=B6nig?= , Mahmoud Adam , David Matlack , =?UTF-8?B?QmrDtnJuIFTDtnBlbA==?= , Sumit Semwal , Kevin Tian , Ankit Agrawal , Pranjal Shrivastava , Alistair Popple , Vivek Kasireddy , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, linaro-mm-sig@lists.linaro.org, kvm@vger.kernel.org, =?UTF-8?Q?Carlos_L=C3=B3pez?= References: <20260416131815.2729131-1-mattev@meta.com> <20260416131815.2729131-2-mattev@meta.com> <20260501131236.278ac431@shazbot.org> <9304aada-ee84-4cf2-a1d7-82313eda07aa@meta.com> <20260506152937.GJ11063@unreal> From: Matt Evans In-Reply-To: <20260506152937.GJ11063@unreal> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Proofpoint-ORIG-GUID: VYMbB4IE-xbaqERLMmWWIXOP9_QF6pTp X-Proofpoint-GUID: VYMbB4IE-xbaqERLMmWWIXOP9_QF6pTp X-Authority-Analysis: v=2.4 cv=DtFmPm/+ c=1 sm=1 tr=0 ts=69fb6473 cx=c_pps a=CsXZvLRfiTx/ye2xXAwb9g==:117 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=7x6HtfJdh03M6CCDgxCd:22 a=jCddH8ec0KUNCymVuxII:22 a=VwQbUJbxAAAA:8 a=UqCG9HQmAAAA:8 a=Ikd4Dj_1AAAA:8 a=VabnemYjAAAA:8 a=bm6tUXqWTanbjDFH1akA:9 a=QEXdDO2ut3YA:10 a=F7q00xkr9EfWfQvbdVXI:22 a=gKebqoRLp9LExxC7YDUY:22 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTA2MDE1NiBTYWx0ZWRfX4OOq5Y+AQdHj 3Xv1FbqHcZ9UHKYSPlQtGPtgZWAsDWoZ9zX1o8T7SMY3OGlb9KLXtTN9iDL9ylxW7Xp3wQni+i/ dksRj1oHAp4cl4HsNGEClIzNCcOz/+20EvrlWbQCbbD0NOwIfa1LoHxDrLnHO90N5fmd5MxtLB9 C32AuOu9qTtopgRJfcd4vLwqYCQq3GMpq6xRIpt7AXAIC+oYvysU4nHVxktBW+9MWr70ruzL1rQ ivtEqeJUQ65qnzcCn3RgGPFkYIMrZFpk5v3hkpcWMEIpKVY0CdBjhPHThQPVrEteiLFtjWwnPNl Tc8Xjq4ndp5JJh8s9c7I0nmgNg5EYBqatW7QLpKNkCosV7qX8SfxnLDXPGQ1PUWrnD58/V9OA6P X952nb3czWNOa8tG89+11FUBdWQ//uy8ay2Im5kAjdKJMmQqJYrA4opL5XoLiJHBDFq6dRglLBN tY/nw4PMfRoI+1AcZpg== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-06_01,2026-05-06_01,2025-10-01_01 Hi Leon, On 06/05/2026 16:29, Leon Romanovsky wrote: > > On Wed, May 06, 2026 at 02:53:31PM +0100, Matt Evans wrote: >> Hi Alex, >> >> On 01/05/2026 20:12, Alex Williamson wrote: >>> >>> On Thu, 16 Apr 2026 06:17:44 -0700 >>> Matt Evans wrote: >>> >>>> vfio_pci_dma_buf_cleanup() assumed all VFIO device DMABUFs need to be >>>> revoked. However, if vfio_pci_dma_buf_move() revokes DMABUFs before >>>> the fd/device closes, then vfio_pci_dma_buf_cleanup() would do a >>>> second/underflowing kref_put() then wait_for_completion() on a >>>> completion that never fires. Fixed by predicating on revocation >>>> status. >>>> >>>> This could happen if PCI_COMMAND_MEMORY is cleared before closing the >>>> device fd (but the scenario is more likely to hit when future commits >>>> add more methods to revoke DMABUFs). >>>> >>>> Fixes: 1a8a5227f2299 ("vfio: Wait for dma-buf invalidation to complete") >>>> Signed-off-by: Matt Evans >>>> --- >>>> >>>> (Just a fix, but later "vfio/pci: Convert BAR mmap() to use a DMABUF" >>>> and "vfio/pci: Permanently revoke a DMABUF on request" depend on this >>>> context, so including in this series.) >>> >>> We really need a fix for this split out from this series, It's already >>> been shown[1] that this is trivially reachable. Carlos proposed[2] a >>> similar solution to the one below. I was concurrently working on the >>> issued and suggested an alternative[3]. Let's pick a solution for >>> 7.1-rc. Thanks, >> >> It looks like [3] is progressing, so I'll drop this one when I can rebase >> onto it. >> >> I noticed [3] removes the dma_resv_lock(priv->dmabuf->resv) around the >> priv->vdev = NULL, and this series' vfio_pci_mmap_huge_fault() relies on >> vdev only changing whilst resv is held to resolve a race between a fault and >> cleanup (see patch 7 of this series). The handler takes resv so that it can >> stably test vdev in order to take memory_lock. > > I think that you should rely on priv->revoked and not on priv->vdev. Needs both unfortunately, as the fault handler ultimately needs to take vdev->memory_lock. Matt > > Thanks > >> >> Must your fix change vdev outside of holding resv? I'm still sketching >> alternatives; at first glance perhaps the fault handler could rely on vdev >> being valid if !revoked, which can be tested holding [only] resv. >> >> >> Thanks, >> >> Matt >> >>> >>> Alex >>> >>> [1]https://lore.kernel.org/all/GVXPR02MB12019AA6014F27EF5D773E89BFB372@GVXPR02MB12019.eurprd02.prod.outlook.com/ >>> [2]https://lore.kernel.org/all/20260429182736.409323-2-clopez@suse.de/ >>> [3]https://lore.kernel.org/all/20260429142242.70f746b4@nvidia.com/ >>> >>>> drivers/vfio/pci/vfio_pci_dmabuf.c | 9 +++++++-- >>>> 1 file changed, 7 insertions(+), 2 deletions(-) >>>> >>>> diff --git a/drivers/vfio/pci/vfio_pci_dmabuf.c b/drivers/vfio/pci/vfio_pci_dmabuf.c >>>> index 281ba7d69567..04478b7415a0 100644 >>>> --- a/drivers/vfio/pci/vfio_pci_dmabuf.c >>>> +++ b/drivers/vfio/pci/vfio_pci_dmabuf.c >>>> @@ -395,20 +395,25 @@ void vfio_pci_dma_buf_cleanup(struct vfio_pci_core_device *vdev) >>>> down_write(&vdev->memory_lock); >>>> list_for_each_entry_safe(priv, tmp, &vdev->dmabufs, dmabufs_elm) { >>>> + bool was_revoked; >>>> + >>>> if (!get_file_active(&priv->dmabuf->file)) >>>> continue; >>>> dma_resv_lock(priv->dmabuf->resv, NULL); >>>> list_del_init(&priv->dmabufs_elm); >>>> priv->vdev = NULL; >>>> + was_revoked = priv->revoked; >>>> priv->revoked = true; >>>> dma_buf_invalidate_mappings(priv->dmabuf); >>>> dma_resv_wait_timeout(priv->dmabuf->resv, >>>> DMA_RESV_USAGE_BOOKKEEP, false, >>>> MAX_SCHEDULE_TIMEOUT); >>>> dma_resv_unlock(priv->dmabuf->resv); >>>> - kref_put(&priv->kref, vfio_pci_dma_buf_done); >>>> - wait_for_completion(&priv->comp); >>>> + if (!was_revoked) { >>>> + kref_put(&priv->kref, vfio_pci_dma_buf_done); >>>> + wait_for_completion(&priv->comp); >>>> + } >>>> vfio_device_put_registration(&vdev->vdev); >>>> fput(priv->dmabuf->file); >>>> } >>> >>