From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A3F26C52D7F for ; Thu, 15 Aug 2024 23:42:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Type:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aDQ6KGtRA6VfahmfS/FtakSo54eExWxil8Kq0kZby6I=; b=k8zg+mgJ0uHRAypsqDmZ5qKO20 6z+FEtFBXma2b+YQkGOXW+hCPJqhJcdSFDYTvVgaslbZ6Ox3Zt3CnFbZBirfKfGB0uJAkEuniql9N a2tOKfjALhTBZmviE+mFbIegTJ4tmKdbsaPrPI6Xj/X8/BQjIxZo2L4P27e5dlYwRwJ2Fa2U9PSH3 zBFQMxF+vGWthNUG1LzNLMLANKz123Xv8tYg0Qu70nqdP4rL9CFikRTT9SDOjXdMOOiZh6eWxb9DT 9UzTlrjN5kJ4ND4duj4iRNH95i7zrbb7cnxjad2bqhbkdMWB8i1HlxIFiI8ImLCpmfOMltXAzQkAF eHw8AsOQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sek6Z-0000000BJj0-46Rd; Thu, 15 Aug 2024 23:42:07 +0000 Received: from mail-dm6nam10on20609.outbound.protection.outlook.com ([2a01:111:f400:7e88::609] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sek5w-0000000BJdi-1216 for linux-arm-kernel@lists.infradead.org; Thu, 15 Aug 2024 23:41:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=suPcTF7XO7GYBzmg25nYaGaVtzzNNwUMDv6JYrNPzQT1aY8ldiqnmvcDTyRVVFUBHvRKhvElyqj/XmPF5FJTMqe0flte4nB46ZIWSZfc5+lsON5i69CAD8NRLnRWsMQQTBmDX4Q+kYALZBTaCzniEn0iVppUxTRDu8PcXhmnzO4A5BBqzT3SN79fZaCm+NQooOf0IsincJg+U4QCFh4Ghj+GLtw41mYSsc1rtr2Ly27AjrxxqFAm039aAUhExiByxFtXbmz53keS7SY3yc2bymhYtkCDqilJLc5+WTMVBBh6n+KZAALC34Yh2cTwYt9xjz+Pw4CeTdpppnF0Pjdh1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=aDQ6KGtRA6VfahmfS/FtakSo54eExWxil8Kq0kZby6I=; b=uAZARJzrGiyXlsI+k30W7wENa0+o9QveHXtrOVw+jPCl17IeAMbbpcJkWozm1p8jSgr/rB+URX5knuvO1Xix5BW1Xktlvg7DlnyO8jNv/gIHu+XTDmtFRNIAWepyhHF0lLdujpKZOJnbs7cu3tuidKg4F0E2P2C8V1cW6hoG7kdwIpKxd/GS1J+Tdb2QxSGFJIW9LG3/5OprVeuCwhznbC59RePaTRX7Xl2OtD44HLmziWs3kwR9gSfnj2xLidXVyT8tBprabw7ieSVeY2FJMkDNpTbL+YqP2scWYtRJQ331jIoA1as+hzYKuzad4BE/qH7FQek7sWFaHKoRyNFTUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=aDQ6KGtRA6VfahmfS/FtakSo54eExWxil8Kq0kZby6I=; b=TQvOqm4HuK4cEeQTN/BFvRqEKJV6FbpzJXzdoFhD/fS26naHskkjVrBe3WQV/6hKfeMWeXodaQi2b6LfpW5Nv3Tff/TF69fNlM54w5kRF6BTItElerw2NgOvWtRCoEc/6g2NUm+8kSYs7kCzzBOBSck78ufBaJZOFSO1mqAskLcyDdHGQxgqHHnfhTnvm8RCexLnh8xc4Ut6iBpADXvUG8Dj44n7UPuAwZZygqCx1VYQID5qoh0DcQCFr0bbwZRUQWJ30n6qFo2NHU/JrQ5mA6eH6EJmyyRp7rlVtwus5stWW9NrhoIjF415im0MHRaoYgtyvGVKwQQQPFSgAmmRNQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by CH3PR12MB8850.namprd12.prod.outlook.com (2603:10b6:610:167::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7875.18; Thu, 15 Aug 2024 23:41:22 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%3]) with mapi id 15.20.7875.016; Thu, 15 Aug 2024 23:41:21 +0000 Date: Thu, 15 Aug 2024 20:41:19 -0300 From: Jason Gunthorpe To: Nicolin Chen Cc: kevin.tian@intel.com, will@kernel.org, joro@8bytes.org, suravee.suthikulpanit@amd.com, robin.murphy@arm.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, shuah@kernel.org, linux-kernel@vger.kernel.org, iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kselftest@vger.kernel.org Subject: Re: [PATCH v1 05/16] iommufd/viommu: Add IOMMU_VIOMMU_SET/UNSET_VDEV_ID ioctl Message-ID: <20240815234119.GX2032816@nvidia.com> References: <20240815190848.GP2032816@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: BL1PR13CA0381.namprd13.prod.outlook.com (2603:10b6:208:2c0::26) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|CH3PR12MB8850:EE_ X-MS-Office365-Filtering-Correlation-Id: bdda73a4-9c0e-4508-7754-08dcbd83c465 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?pRL8t2syPYV76sNNS1QmF+Oxq6T3qzSEoX17rRFXdLD5flOSp8OoNrUmT+Tg?= =?us-ascii?Q?gOK9XNDO+bIzYXXIsaTriJjNRfiNQs8Z8fRZUxlJBnlGfmkDmdDwNYctrqwe?= =?us-ascii?Q?duglyHEawYSfisXnwt/9HBFoNMK5D0i73gT15wpMzer8Pt/EoSWfI4ovmJIa?= =?us-ascii?Q?pbfVQrrFetcp1QmHwMJVbtAFeuaA4OcpO9QREDCWpFtovXPADasKP4LWx7TL?= =?us-ascii?Q?onV+IF24TJdrAk2AlXe09En/Nv4baoI+tl/AQxdYyB2doHdf3cNjN1+JhAqJ?= =?us-ascii?Q?cw2YNZmdC06eviYVXMSifNzZT0vWgKtrnpzUZTSxPE/pknZb9oh4SVMN6Ixf?= =?us-ascii?Q?W51VWUlwoclhvZQheTqq4XBMN7y2R0EJUZLLJOZ1VAGrcVo4r5jKRHov43ps?= =?us-ascii?Q?KpD6hnAR+xfg3iPp4d6yVBRturoGsV4gYsRahzIzHMWvB0/u2q7RFQK50MNC?= =?us-ascii?Q?8STCYd0sK/Fn0yeK0o7ewkUe/jWA+E2t8skIjMAfFR5KSG6quV0PzLd6WM/d?= =?us-ascii?Q?3qZGVsL4NT9jh4VYnwo/WVjAz1T64JkkwLZiqEk4/TRVUtYhM0hVhWjDqwy1?= =?us-ascii?Q?FTo5WnB5Tt0LwQDLt/xYwkhFHqEgG7qBW6Uz3kcuf4/2hqqQmbT0n+jZMc6M?= =?us-ascii?Q?bg4vWPqMMxrMc3Bilr0o2hLrnfS9nklyITJpHSm6QiiZbxsJ65LDCkfg3a2Z?= =?us-ascii?Q?hs0xV/m7Wgk4phd/2Ay4I8obSWFNZTvw3RCMEzJ0iy3aKgwuzQpTCVPhipEd?= =?us-ascii?Q?A3bscxcnVELPXhlk9xsLg6xEG/tl3jOJqu56hD8gGYhpQZEKb29utLZS3P4F?= =?us-ascii?Q?LaTaptdRsywUDfkorQqe03K7Aj9w3gs3Lw2DSDAdpKmu7OQ/XJ7j68/Vu062?= =?us-ascii?Q?eBS7sWLAgNJ+jyTBtRdCs+VfQ+B1KRzKNoXGm1IO2tmHKT73OVXrrUgDgxrs?= =?us-ascii?Q?pQVQKIYwG408zp3o/YxGhwig0Lt3CM+Sg+cqAUAN2u+GKGPgMXucScYtScCH?= =?us-ascii?Q?QPfJEfMNlmhFj12ADAAsKxGt5zMgjhrScuVyLOiLXADIWbxpDRETfqIr5n2i?= =?us-ascii?Q?4PUNkY6qejDus9wTvFKUZbmuKD+mcCU7OkBp6Q8iCvmEHwmMB5gh2PI3dS3e?= =?us-ascii?Q?ahJ/tw37LIVGqmg+3hLMZXL3Q7qtHX588ojTaouhJSBYptauN0H+E/OTbcGm?= =?us-ascii?Q?/x1Mkvs/cbiPx2unU2hCQiVhRHlaGlkC8kIRjWinxqw9Nfsbt+G/naR7+sQ4?= =?us-ascii?Q?yGjoVbdlIxHAErqqlaulwCw4/i39CgujoL+c/iXe86DJcqJSa1yIQ2hdoiKV?= =?us-ascii?Q?wOUI5yt6e7ovfND0IQbBw4LM/Je36+qQHiXF46sxFlj5pw=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?ccroInATYRB8KdlJsd4CAsizvTGqpAG/cAw3Yy7gLdsD/2SAWzh2y5iQewRf?= =?us-ascii?Q?HGoBX8wYcFlPW8W4PCZLui43nccYc8AODkzcxrpM2ZXGDVlGsE6SPPzdglG8?= =?us-ascii?Q?WTUq1vn7fRcTNDkoaPO5+m7JWcENFajCY1j9jNgyzzNomStzM2Oo4F/6KWIS?= =?us-ascii?Q?zyBlwhHKTn1b029adGdm7AQbIzFfmOONCiSdAqkts03+ta3g1lg6gCTN7+LU?= =?us-ascii?Q?l0PbTECTmONJr3JGNElH+ihW8bEXxMi4XXCSn+HR1hO1ElReSO1YVuNeheEw?= =?us-ascii?Q?mKTUpprh2e+IcK1f8yNUDRDa6mdExUqt0vFjzg9Mrgb4EkwogmZTQdp8uie+?= =?us-ascii?Q?O8rNG/RB8KO6TDkLTR+yBH+g749682D8kSkrgg1qpF7LqZCUYqe9PLKFMCNX?= =?us-ascii?Q?pn4m/ucfFhBQWy8OytMkm6K2opEulnWsEUeqaw3qZBRPycHQ20uQCRKVrfCk?= =?us-ascii?Q?bW+uipPE/g5d6mG2P2Pb7Nnt2sV6MELm4hGotEacVnnun/a9YIl5iSxG2sXs?= =?us-ascii?Q?YK+HftX0RNveBDr36z2m/LkRVEzdIA2/ROVjpVmTFanLZ0FxeAHDfHIwQhgg?= =?us-ascii?Q?nQkBo0eE2GkfVhlVBioEFJCn8oRFZNG+mkohO3htXmU/jER6Ol30WKqvm4eU?= =?us-ascii?Q?5/HZ9ZOidnKcb6MtzYYCFvjRIa9cgX5s4lOBume16l+YF4h0fK5ZnL2oC8tG?= =?us-ascii?Q?o3q7P6/GmcZ4HWqtWV/D9expqheUNNSS7jNtrUX320RHBnokWWg2AX18gIUv?= =?us-ascii?Q?9qyksrV/01VcWmtgRLWTUoa1TIMQu46srzL+82frnLu+nkkbohgTi0u6RgWp?= =?us-ascii?Q?o1Cy+xmrCcg8gOvm2PenFCaPZdxyof1a+5xaR0KAagJWJXB5B6kb9JzUQrU7?= =?us-ascii?Q?vP4cOHeH3A3Nlyp4JlNqNXf9jy7Hvu2tZ+Mx1IJTeV4GlSL8Svu9DGK2XlhS?= =?us-ascii?Q?ydywZwSNLwY2YIPPa/CztNjgz6MOnK+6x3vCsqmXOTrV+jDYH4CeOeqn5W+g?= =?us-ascii?Q?M7arU1HulrCIERvQN4whQiGA+KhJiQ76Qxtwiwf0lOGr1NTLO1+Pk1wezKNj?= =?us-ascii?Q?KYIH3vIlWsYHTMABnZfF1fLCvHKH2qM4ktH+5GLtwsjilwkDuucRz/qZhdAU?= =?us-ascii?Q?P0ojDE/Q32GkZKU6RsvTFrQsGj+ucGClv9YFTyw2Qv6EZnI2sFi2ef8gNeGU?= =?us-ascii?Q?Vv9xSmLZNQe7vqIimWbxgaAo1wfMyqMdCb8CyzpBK0VoJW8Fy1WcYWTzw/Ey?= =?us-ascii?Q?BMYbQSly05AC2WVKfP3fdG0JEGaj9yJ7eaETO2EYTz7Nc0xUVU+4HrWvftg0?= =?us-ascii?Q?a5tfrwJlMRLhYp0Wstpf2SIjoN6yTKp9Z8To1I3eQVYvXbQtEg/r8fEa8Nkl?= =?us-ascii?Q?lK7AYCwaGDgftfx1xEyZk8KOQjV4lN7Zz7n/YgmYVZOlim9eBvu7D5VsnR/V?= =?us-ascii?Q?g8UgeT+exya5lDag5zziIDVyog163NpgKnnDrNCkxgca9SJXbtk5INc86MSD?= =?us-ascii?Q?8/yWQ4cPtemOLRlPv0bAsT0elXKXAMYp1cvzP/hBJE8ljsWE0m4bAh8pwiQy?= =?us-ascii?Q?1Cj5Zxu4pIxZUiHwU0I=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: bdda73a4-9c0e-4508-7754-08dcbd83c465 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Aug 2024 23:41:21.4956 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FAYYTV+pVEV4nmQVohQdxaSWiZC8JXwXMc2JqNdqqRJncVyUUGLKge0sNNIRPgQm X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8850 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240815_164128_310468_5BD6AD51 X-CRM114-Status: GOOD ( 27.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Aug 15, 2024 at 12:46:24PM -0700, Nicolin Chen wrote: > On Thu, Aug 15, 2024 at 04:08:48PM -0300, Jason Gunthorpe wrote: > > On Wed, Aug 07, 2024 at 01:10:46PM -0700, Nicolin Chen wrote: > > > > > +int iommufd_viommu_set_vdev_id(struct iommufd_ucmd *ucmd) > > > +{ > > > + struct iommu_viommu_set_vdev_id *cmd = ucmd->cmd; > > > + struct iommufd_hwpt_nested *hwpt_nested; > > > + struct iommufd_vdev_id *vdev_id, *curr; > > > + struct iommufd_hw_pagetable *hwpt; > > > + struct iommufd_viommu *viommu; > > > + struct iommufd_device *idev; > > > + int rc = 0; > > > + > > > + if (cmd->vdev_id > ULONG_MAX) > > > + return -EINVAL; > > > + > > > + idev = iommufd_get_device(ucmd, cmd->dev_id); > > > + if (IS_ERR(idev)) > > > + return PTR_ERR(idev); > > > + hwpt = idev->igroup->hwpt; > > > + > > > + if (hwpt == NULL || hwpt->obj.type != IOMMUFD_OBJ_HWPT_NESTED) { > > > + rc = -EINVAL; > > > + goto out_put_idev; > > > + } > > > + hwpt_nested = container_of(hwpt, struct iommufd_hwpt_nested, common); > > > > This doesn't seem like a necessary check, the attached hwpt can change > > after this is established, so this can't be an invariant we enforce. > > > > If you want to do 1:1 then somehow directly check if the idev is > > already linked to a viommu. > > But idev can't link to a viommu without a proxy hwpt_nested? Why not? The idev becomes linked to the viommu when the dev id is set Unless we are also going to enforce the idev is always attached to a nested then I don't think we need to check it here. Things will definately not entirely work as expected if the vdev is directly attached to the s2 or a blocking, but it won't harm anything. > the stage-2 only configuration should have an identity hwpt_nested > right? Yes, that is the right way to use the API > > It has to work by having the iommu driver directly access the xarray > > and the entirely under the spinlock the iommu driver can translate the > > vSID to the pSID and the let go and push the invalidation to HW. No > > races. > > Maybe the iommufd_viommu_invalidate ioctl handler should hold that > xa_lock around the viommu->ops->cache_invalidate, and then add lock > assert in iommufd_viommu_find_device? That doesn't seem like a great idea, you can't do copy_from_user under a spinlock. > > xa_lock(&viommu->vdev_ids); > > vdev_id = xa_load(&viommu->vdev_ids, cmd->vdev_id); > > if (!vdev_id || vdev_id->vdev_id != cmd->vdev_id (????) || vdev_id->dev != idev->dev) > > err > > __xa_erase(&viommu->vdev_ids, cmd->vdev_id); > > xa_unlock((&viommu->vdev_ids); > > I've changed to xa_cmpxchg() in my local tree. Would it be simpler? No, that is still not right, you can't take the vdev_id outside the lock at all. Even for cmpxchng because the vdev_id could have been freed and reallocated by another thread. You must combine the validation of the vdev_id with the erase under a single critical region. Jason