From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3DF22CD4848 for ; Thu, 5 Sep 2024 03:09:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:CC:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=16iVbfaLUsSAsQkr8m/sNAnYjbCJ5otSK/xmLQggjJs=; b=UXONLKjf8ENWg9LLom8g/vGqEx /TKIa+v9oEe93NvgICSGAT3h7tKWyRb+1+0cfMHaaUPiGY4bU7FnHWi0D5xa2vQqqoPiztqFSbqhm Rbjy+3zzd88dYrTQbL5VvW/z5eSMkgHO474At63/1Ku8JgBS0c37Kljlpg2EgGzfbG8iNiav18slh fyJT82V0Giap2932GbN5YUmpRkpEDdb+i+je3U75RNNa5FL+9AvWw2TI/jssY/J6rzQErUTkImx8v inKiPIoyGTvH0pAGBn5SIXUSWhyMWzEZm0ry2k0U639uVXtgCQM7Cg7QSNJRrfGKGzC6fIfMNc9+l FhyLutEg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sm2s0-00000006m6K-11az; Thu, 05 Sep 2024 03:09:16 +0000 Received: from mail-mw2nam04on20618.outbound.protection.outlook.com ([2a01:111:f403:240a::618] helo=NAM04-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sm2r1-00000006lly-0LXg for linux-arm-kernel@lists.infradead.org; Thu, 05 Sep 2024 03:08:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=sobc/1sWp14CoUItC7tHiHHcqbTAgtpNwxl4Ef0fPILZR217SDQkxjVVG3sjLkq4uIh7xAAueQ2zr2R+uwRqb1JAVlo36ExtFulXd3t0MYBhFGIpXkBJdj21uAnBzQPJb0eE8ZPWeLpEcTkTLe4cZqQ6/s0B2yXKdtxmjNEMUO+1UFA2ell0Iu3+UdT636fneZQtrwQux2DMYuPFuU4C1OUTurFHTJuHIwn2tGlHQW3hY7fVMshMJXxpcbHs36EdoG4H6cE2/qr+Ew6w8ZGA5DnrWR+5QHPk0Fvgvh+KNdJlOZ1vWTIcDG9nTDer7BVSgFMNmV2f8FVXtA8U5FzjVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=16iVbfaLUsSAsQkr8m/sNAnYjbCJ5otSK/xmLQggjJs=; b=T6qJ3i70zIjuQYJSzlhLPo2feKFBLj7omyTIxhbIY7mK73waJ/osBp19O2CbT9mxT3MEl/SGONKLqplN3V0e+XJHYZ8aY+L7w/XN6Ue01MQZxNF7aYjWzpm698I7BdZ9gvIOieGzXn8W7y3UucOSp3GPmlcmr799FUDSQo4f13vDX4AoAHKR+HrQlyAw6vyaQEfHVuHZLUAPy2Vr/FaLMJny5Q1ZxlkCQblXpYAi6CKLHCmZp9vlpYuI16bIi6s7jZuiVvdzOfGP0jB1/bYCQpIoNUaDGXZA6OIONQbwaNzrg/kTcOJxRVZeGyijaAzWmr1JkKhB91m+UbCezhAxBA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=huawei.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=16iVbfaLUsSAsQkr8m/sNAnYjbCJ5otSK/xmLQggjJs=; b=ZNxQEaSIMlqnz3+4CG+0Un082FDpeBeWLE4LDH3BcWyy2nxXrvBSZlQtKAlhhA4wGUm3+CuNZ+rwFFI22O3G65SpTuGBxA5iEokxQPHWs57Cmwf9j7prMp0/H5EgxXboEaGx6aUSx5pbYJGdipTn0NEdHiXlk2jEBZ/DhRxYEOnSEcaEu72EWNXvMcnvn1W2CJwNtX6WUKvlp4vzed2PvIo0BoWFQl3C7oIPCCGfaLdpt4KYe7ydyW8rPIHcj4NCCsTQAhtQISbNsY5xQul6c7qEKIQXDJ9KopfepwHB4x3LGDBX8Ftcrw0ghtEd/Nme4T+ttUXRGo/GydRPJ8riYA== Received: from BN0PR02CA0039.namprd02.prod.outlook.com (2603:10b6:408:e5::14) by PH8PR12MB6915.namprd12.prod.outlook.com (2603:10b6:510:1bc::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.27; Thu, 5 Sep 2024 03:08:09 +0000 Received: from BN2PEPF0000449F.namprd02.prod.outlook.com (2603:10b6:408:e5:cafe::15) by BN0PR02CA0039.outlook.office365.com (2603:10b6:408:e5::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.27 via Frontend Transport; Thu, 5 Sep 2024 03:08:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BN2PEPF0000449F.mail.protection.outlook.com (10.167.243.150) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7918.13 via Frontend Transport; Thu, 5 Sep 2024 03:08:07 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 4 Sep 2024 20:07:55 -0700 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 4 Sep 2024 20:07:55 -0700 Received: from Asurada-Nvidia (10.127.8.9) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4 via Frontend Transport; Wed, 4 Sep 2024 20:07:54 -0700 Date: Wed, 4 Sep 2024 20:07:52 -0700 From: Nicolin Chen To: Jason Gunthorpe CC: Baolu Lu , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v2 02/19] iommufd/viommu: Add IOMMUFD_OBJ_VIOMMU and IOMMU_VIOMMU_ALLOC ioctl Message-ID: References: <55918c41-65c4-435c-860b-b2a177b0d364@linux.intel.com> <20240904162621.GN3915968@nvidia.com> <20240904233707.GA1358970@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20240904233707.GA1358970@nvidia.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF0000449F:EE_|PH8PR12MB6915:EE_ X-MS-Office365-Filtering-Correlation-Id: 8e924a2e-2f03-457c-fe27-08dccd57f728 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|82310400026|36860700013; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?tagqVyc45esUSFxa7QxdtCV6Yrj/pqEIxjO8+ynx6rfhoOIzTvHDbguJrApT?= =?us-ascii?Q?JqwnAX24Oc5jNXOdJbgnzQzrFd6CKBRb4uo2qbRZqx5sRN7mdSOuL3kIiVEy?= =?us-ascii?Q?lKPuSVhFCNDdOFogK3n7LRFdPj4B3p+2YOnm5lf6SnUV/IZ6zlU99pCyFSJQ?= =?us-ascii?Q?krUJyxnboWkSahE0WTV503e/DVlvKk1x8iPrNeGw2eExIxQd/wrhKSjQWI8U?= =?us-ascii?Q?A8oq1HqE4gXDSSWnV2Ae8tLRv1L/oF49YvnGM/iDS0i7Gpq7WDZSB6X2OuOH?= =?us-ascii?Q?cxqr9CTwXWNOA+0TbErE05PTGimrRMAXcjhzELZ1f9Hy6T3bs/cJ8ZKXPGVp?= =?us-ascii?Q?0g/scyKI3CPe8m4RKBQKmvv77siR9Og9HQlIx8YzW+RB2eLDaSMVDYtTEKV5?= =?us-ascii?Q?OmNj6gLgss48ncQE6sJBcy/znEfFOgSDeL97XYH8CHC/wREv3Y+sVp4VS5OW?= =?us-ascii?Q?O3OQvnivKtSI/vNPXa7CQmFKwquzML/GcqVeUClfk0M5pEl1uQi6RjoE42CO?= =?us-ascii?Q?EpZABiBLnBN6vTJeXubVUqS5G1/au8pFexeORjuAy3dKPph8Yac4PlSonXWN?= =?us-ascii?Q?Iz+4oEl92K9ZFE62Gz6j22nlIRh/COYh1RE0F/GTc1VOO7FF06z4GoHP7ekY?= =?us-ascii?Q?DZ9gSGEOYbr0hESe6Ur5zE5aIm49pcSs6OqGtsXl03XY/hZE/AdxkgbUBGVo?= =?us-ascii?Q?EysWAlVmAh3XrW4GNCxmuXlETLLmiWbufZNIof3CnX9LqRWeg3/A4b27GpxX?= =?us-ascii?Q?es2C+P0beJ3M0OB47oO8zdLuSukU5M8kJ3HVdgcTc2mAu2HdDlfci5Y/sK+6?= =?us-ascii?Q?3rXjL8qA9BbgpBBCWzlOuynG4+sqcs2XZ7+rBE1bbBNggHsswgt35Pb9K98U?= =?us-ascii?Q?K/MgptGzP+FLV+ui4wwnXP6pI/DyQd3x85xD0GBYSshL6bwLG7xaisi9eJYS?= =?us-ascii?Q?aPCfn7oNKyHIoJvjmWt5Xhts/544sXItFbBZnW+Hl2joHafByiSkjBqno+Qb?= =?us-ascii?Q?6BcW3chuDAIH5Km4V4HB0B7vKGcGQKG+Z0qKw8Ha0VlzyWMT9d/AGXQZRrdU?= =?us-ascii?Q?A8I1hGSAHspBSqVAuI1oHAMJ025n5kDjjwFRv+O1E1Ky58yoQZuuUqZGlneP?= =?us-ascii?Q?QAr08Nv8H9z6VcQr3k1JmhrQbENkiEoWXPlDiRLenw3g+FeRrDUKSQmi7LgN?= =?us-ascii?Q?x+b6qEaBO6tq7uxF4/ibMK0BnAhJbDsXiJBFAdT9iltEon2LpF/AmktkGNyo?= =?us-ascii?Q?YE9B5kiJexP6iJPkSJlVsYNG4EqlJXmagRSTB/WCsBXcHpsyhAC42EMnqbp8?= =?us-ascii?Q?DgCt+8syzebAaM5lmVE7Hm2ZFRnhnKanSGUDGzz96qBdNWsRDQm02jE3CmYZ?= =?us-ascii?Q?xanwbuUeNQB5CDla+XVOhjGfFBXerTRyCgmy6nPV5KkPb/335vtZi9lMdj61?= =?us-ascii?Q?m6fa9MzvjxS1hr9XE2qIDcku8kDeg85S?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(82310400026)(36860700013);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Sep 2024 03:08:07.0244 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 8e924a2e-2f03-457c-fe27-08dccd57f728 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF0000449F.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH8PR12MB6915 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240904_200815_152176_6D6A5B4B X-CRM114-Status: GOOD ( 30.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Sep 04, 2024 at 08:37:07PM -0300, Jason Gunthorpe wrote: > On Wed, Sep 04, 2024 at 10:29:26AM -0700, Nicolin Chen wrote: > > On Wed, Sep 04, 2024 at 01:26:21PM -0300, Jason Gunthorpe wrote: > > > On Sun, Sep 01, 2024 at 10:27:09PM -0700, Nicolin Chen wrote: > > > > On Sun, Sep 01, 2024 at 10:39:17AM +0800, Baolu Lu wrote: > > > > > On 2024/8/28 0:59, Nicolin Chen wrote: > > > > > > +int iommufd_viommu_alloc_ioctl(struct iommufd_ucmd *ucmd) > > > > > > +{ > > > > > > + struct iommu_viommu_alloc *cmd = ucmd->cmd; > > > > > > + struct iommufd_hwpt_paging *hwpt_paging; > > > > > > + struct iommufd_viommu *viommu; > > > > > > + struct iommufd_device *idev; > > > > > > + int rc; > > > > > > + > > > > > > + if (cmd->flags) > > > > > > + return -EOPNOTSUPP; > > > > > > + > > > > > > + idev = iommufd_get_device(ucmd, cmd->dev_id); > > > > > > > > > > Why does a device reference count is needed here? When is this reference > > > > > count released after the VIOMMU is allocated? > > > > > > > > Hmm, it was used to get dev->iommu->iommu_dev to pin the VIOMMU to > > > > a physical IOMMU instance (in v1). Jason suggested to remove that, > > > > yet I didn't realize that this idev is now completely useless. > > > > > > > > With that being said, a parent HWPT could be shared across VIOMUs > > > > allocated for the same VM. So, I think we do need a dev pointer to > > > > know which physical instance the VIOMMU allocates for, especially > > > > for a driver-managed VIOMMU. > > > > > > Eventually you need a way to pin the physical iommu, without pinning > > > any idevs. Not sure how best to do that > > > > Just trying to clarify "without pinning any idevs", does it mean > > we shouldn't pass in an idev_id to get dev->iommu->iommu_dev? > > From userspace we have no choice but to use an idev_id to locate the > physical iommu > > But since we want to support hotplug it is rather problematic if that > idev is permanently locked down. Agreed. Thanks for clarification. > > Otherwise, iommu_probe_device_lock and iommu_device_lock in the > > iommu.c are good enough to lock dev->iommu and iommu->list. And > > I think we just need an iommu helper refcounting the dev_iommu > > (or iommu_device) as we previously discussed. > > If you have a ref on an idev then the iommu_dev has to be stable, so > you can just incr some refcount and then drop the idev stuff. Yes. The small routine would be like: (1) Lock/get idev, dev->iommu, and dev->iommu->list (2) Increase the refcount at dev_iommu (3) Save the dev_iommu to viommu (4) Unlock/put those in (1). Thank you Nicolin