From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B50E8D2E01E for ; Wed, 23 Oct 2024 07:41:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:CC:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Cc6h/YzzQqgj6jEa/pB+5U8cxgBjKfQcfixAXu5zEpQ=; b=IS6RSoYo8+FRr3hzwudqZqGxNb Ql3a/3gzneVxoqIkGZ4w6lG5frZ9mCqbe2NNSqDoq74gSRGKA5iRUrlVOAR215lO9Pl8Eq2J3jTnQ PSvnW51Tm98qEn+gQ9S9BFyVy4b74EGekza9pJnWDFf9wiFEYEPdAs7ExAgk3DfSKVoGItGQ8eFt4 Pava1e5y77k8ylvAM2BTm4rePst9FplfCM9hXKsG2fCHno3crRTOylwjWO7H8HI6j4hILi8U3Oi83 4qe/MFlAOig+vVyhbwoZEf0tzvUBrlWZRN0+CgT1EJS3qnklJJvhKhZMoKt+MnFy3ZQ7HKYV4/2UI AOGI8roQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1t3Vzv-0000000DQZ6-34d3; Wed, 23 Oct 2024 07:41:39 +0000 Received: from mail-dm6nam10on2061d.outbound.protection.outlook.com ([2a01:111:f403:2413::61d] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1t3VhZ-0000000DMxR-353a for linux-arm-kernel@lists.infradead.org; Wed, 23 Oct 2024 07:22:43 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KxynW0yd9uKlnA2G7347GgKGze66LusGwFQneColv+bb58LtfnybiNTadl0zIQiG7mnhBTLtBTJrplNnXJy9QfUeWBViyPnTf1nkQSLsxE3kjASteaKYi05wdMWAK8lW+ym8yz9AuBRZ2W+cq9an0NYF/7hEXwwb99Ojh+6APFacDj6UfL4g12p0xHa4T7jMDCsUV0KvFRIQv0HQEgkQf1Vno05ZbB/LeGLOzgS1DY0D8mB7tZLhZfaeWpY8ZQEpdHtncnYY8d9nOZ0UNWzfY8vFt8q5EgCXqr1Bnv+3xsibxiWsjbgHP6gg3QOMGt4fp82De7oMz03kBJTW7zh/rQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Cc6h/YzzQqgj6jEa/pB+5U8cxgBjKfQcfixAXu5zEpQ=; b=W43Gt9xy1DH3KQL03o/OyAYtt6GXGA/46WKyLmOIzFeUeADU1zkA9Wph42mCerj2gSvNsmGT0+1KxSFuwdA1geEY5mTvYwj6edGAdpyWLtCvFANmse+2IzActpVQPWHFBYMz+UqGibW0bEeIUlrVIopCo1vR+9KlG/wuGSM0Z4z7dqwiuFP6vo+eSmpFowfm7ChACyzjzD8AHo9IQ2/a9DsUFeMkBm25D5pWKUiS8EAoXhXcHCoegb62ySSpSv6rNND2IQVmuqE+Zakz0kRZDQQyMk3SPcxGeDXoOcOcqfysF/xd4N0S58svDjil//ambIL4DXPSBiZS/lWhyJcvTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=huawei.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Cc6h/YzzQqgj6jEa/pB+5U8cxgBjKfQcfixAXu5zEpQ=; b=G6krWYzK1Fe1f7j1ZCi3otqxPyICOLsBzt/H5/5IUiq8nhK+wy8hCY0jgd6GnfRKZLPkGkNqXcHzVHEak8eymNXL7USDIvX1DAWA8LyDsamylHa3xQqehO5uUeacz4NoKkC7Xc04eZAU/Pzu82s+SRq5fYP0E26VTeR8Jcp1Ms2MRZ8cKReTS3VnOEYH+Dgzqj0iDPPajfSuqA4f0y5URb/Mse7oef4HAgA6h8Zmd39Ug8QBpHNJJCae6zikVaZPo5HC/BmAxoqNooASMdzGFoxff3uHbVFBZXZPu84y9Nx7wrVdp/HxQ1bq/zJOoHtIo5YxLV3xBKzOqZEiwvvqGA== Received: from CH2PR08CA0004.namprd08.prod.outlook.com (2603:10b6:610:5a::14) by DS7PR12MB8249.namprd12.prod.outlook.com (2603:10b6:8:ea::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.29; Wed, 23 Oct 2024 07:22:33 +0000 Received: from CH2PEPF0000009D.namprd02.prod.outlook.com (2603:10b6:610:5a:cafe::e7) by CH2PR08CA0004.outlook.office365.com (2603:10b6:610:5a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.30 via Frontend Transport; Wed, 23 Oct 2024 07:22:33 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by CH2PEPF0000009D.mail.protection.outlook.com (10.167.244.25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8093.14 via Frontend Transport; Wed, 23 Oct 2024 07:22:32 +0000 Received: from rnnvmail202.nvidia.com (10.129.68.7) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 23 Oct 2024 00:22:18 -0700 Received: from rnnvmail201.nvidia.com (10.129.68.8) by rnnvmail202.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Wed, 23 Oct 2024 00:22:18 -0700 Received: from Asurada-Nvidia (10.127.8.9) by mail.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4 via Frontend Transport; Wed, 23 Oct 2024 00:22:17 -0700 Date: Wed, 23 Oct 2024 00:22:15 -0700 From: Nicolin Chen To: Jason Gunthorpe CC: , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v1 04/10] iommufd/viommu: Allow drivers to control vdev_id lifecycle Message-ID: References: <3ab48d4978318938911d91833654b296947668ad.1724777091.git.nicolinc@nvidia.com> <20240905180119.GY1358970@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20240905180119.GY1358970@nvidia.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PEPF0000009D:EE_|DS7PR12MB8249:EE_ X-MS-Office365-Filtering-Correlation-Id: a445fd23-dabe-4ef4-9d62-08dcf33375e2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|376014|36860700013|7416014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?Iy38ArqFaE8WWs88SZGglWrn89mY84Ebr8yT5dsHiUHWQ5SFPxZzdNgCvrhg?= =?us-ascii?Q?rUpmmy35uvRLS6uNCICVQRzZubWP2U1OEKwiSiKNIvgHn+ADS56/sP3kMkDS?= =?us-ascii?Q?GfKfbK124D3/Nwt4RsBFpmQMzygRdZLKn6NF80dxF1ibrVDHjvDOmWlCXei+?= =?us-ascii?Q?1p70HglNJPYLCHDyBlyoYPNioIbvjG3XXus6tn0hIdeVR3rvoHoNTyH69XBn?= =?us-ascii?Q?ekNUHSrgh5FcSan7N3HzSbalG5tySc9be62uv3OTLGdVdRcF9Qwh3th5X0Tz?= =?us-ascii?Q?Ne2Yf2Bx1CS3ysqM6a0kQdqbB4Y0V6V60Fuh6+v94FOjaBHYsS+maop2N1Us?= =?us-ascii?Q?T6PrvcnSqAyKWEog9ykc4YNruMcY3HFvK5hSVY5ND8Oy+BKllshuOhFryxbg?= =?us-ascii?Q?KoqngCyvOxTeadTrE9nGbov3m8wq+M4gwfhVQePHT5WzAicjEX123W6bPnZr?= =?us-ascii?Q?tIK8npiAaj/KKXG3xGzW2Pox9yHDpvKS9s8CHjfL6k54OJZ6ipR36ad+mPbF?= =?us-ascii?Q?7NMDak3Hd1lMlb54mzXLb/WwV4kaQSJl6kTYR/EAJ+VsKhHlgmOt/JUB/7HR?= =?us-ascii?Q?/e0Sxie4/yOUrE2wkJLD0oQk1Y37OSHC6+5vmAVIOHcqiALQoel5bkbl6Btc?= =?us-ascii?Q?ntZYX2y/qWBDekM/VDA3uRnUejNJ0+2lP2uHspmvmdCx5cvuKb2xWtpUesvF?= =?us-ascii?Q?GH7x6HYpaE5nsIuMBfvY5Ixw4OcX3hollBF1lpWGo/7gZGh17txhqMqj5c4K?= =?us-ascii?Q?gN5M08sA0PHnHvUZ9XoRIsLVsFUdty9v9lF8cIKLPv42LM33rszHfLmLz6uN?= =?us-ascii?Q?7rU9zOc/RitQfHgNHzo/mG/pJ8c4S+tGocgQnRXv0rn0kURc5CENtTlxi+2G?= =?us-ascii?Q?9QFl99nbYO2Fk4OzHtmI9ikDNOzx2BXgh5Vc6Wb68oxzvP3frOW0Va5Opkv0?= =?us-ascii?Q?4LTIbTdeI11A6BNiFoy9D4OnzMLbloK4nIw2LHauo3atVJDES/JSqb2NcTYm?= =?us-ascii?Q?dK7ObKGMrMQ4aq247tEPnVr1ybt/Tj5lDTmCMJISaDeyOaIxgCoEw5vFZKTe?= =?us-ascii?Q?3OXq/vdONqagV0iGdPUO8HJj7jconC932ELfyhaTRphjnt/rZWWDVorcjc+Q?= =?us-ascii?Q?AgrpUasFjMkeWetNE1b0IMZpq6QFQDpO7r1PDQWFr/oosNJ15vAYbp/EDiDy?= =?us-ascii?Q?8i6TvqInIuY2CuJj+ZbanfIgvoupMzFjnwcdqC06oRzOHTFi00aCo/D0n5dZ?= =?us-ascii?Q?P3RH/l2Mt9DKTZBAUFRPh5NQ1J4ZIhYjiNUgWumYreZztZj0IKJRwrD8NKZW?= =?us-ascii?Q?1YbhY6e0s79MzI5CxhKPUORt3ZIhXq0BzSLNoDNviGshhDLBxNvDWZ6KTyEe?= =?us-ascii?Q?XMLBHhIL0yy3BrjDNlNiXUKytEobD4viKUehHW8gQCWTDQGHTg=3D=3D?= X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(376014)(36860700013)(7416014)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Oct 2024 07:22:32.4872 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: a445fd23-dabe-4ef4-9d62-08dcf33375e2 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CH2PEPF0000009D.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB8249 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241023_002241_819122_9B409784 X-CRM114-Status: GOOD ( 29.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Sep 05, 2024 at 03:01:19PM -0300, Jason Gunthorpe wrote: > On Tue, Aug 27, 2024 at 10:02:06AM -0700, Nicolin Chen wrote: > > The iommufd core provides a lookup helper for an IOMMU driver to find a > > device pointer by device's per-viommu virtual ID. Yet a driver may need > > an inverted lookup to find a device's per-viommu virtual ID by a device > > pointer, e.g. when reporting virtual IRQs/events back to the user space. > > In this case, it'd be unsafe for iommufd core to do an inverted lookup, > > as the driver can't track the lifecycle of a viommu object or a vdev_id > > object. > > > > Meanwhile, some HW can even support virtual device ID lookup by its HW- > > accelerated virtualization feature. E.g. Tegra241 CMDQV HW supports to > > execute vanilla guest-issued SMMU commands containing virtual Stream ID > > but requires software to configure a link between virtual Stream ID and > > physical Stream ID via HW registers. So not only the iommufd core needs > > a vdev_id lookup table, drivers will want one too. > > > > Given the two justifications above, it's the best practice to provide a > > a pair of set_vdev_id/unset_vdev_id ops in the viommu ops, so a driver > > can implement them to control a vdev_id's lifecycle, and configure the > > HW properly if required. > > I think the lifecycle rules should be much simpler. > > If a nested domain is attached to a STE/RID/device then the vIOMMU > affiliated with that nested domain is pinned while the STE is in place > > So the driver only need to provide locking around attach changing the > STE's vIOMMU vs async operations translating from a STE to a > vIOMMU. This can be a simple driver lock of some kind, ie a rwlock > across the STE table. > > Generally that is how all the async events should work, go from the > STE to the VIOMMU to a iommufd callback to the iommufd event > queue. iommufd will translate the struct device from the driver to an > idev_id (or maybe even a vdevid) the same basic way the PRI code works I am trying to draft something following this, and here is what it would look like: ------------------------draft--------------------------- struct arm_smmu_master { .... + struct rw_semaphore lock; + struct arm_vsmmu *vsmmu; .... }; ->attach_dev() { down_write(&master->lock); if (domain->type == IOMMU_DOMAIN_NESTED) master->vsmmu = to_smmu_domain(domain)->vsmmu; else master->vsmmu = NULL; up_write(&master->lock); } isr() { down_read(&master->lock); if (master->vsmmu) { xa_lock(&master->vsmmu->core.vdevs); vdev = iommufd_viommu_dev_to_vdev(&master->vsmmu->core, master->dev); if (vdev) { struct iommu_virq_arm_smmuv3 virq_data = evt; virq_data.evt[0] &= ~EVTQ_0_SID; virq_data.evt[0] |= FIELD_PREP(EVTQ_0_SID, vdev->id); return iommufd_viommu_report_irq( vdev->viommu, IOMMU_VIRQ_TYPE_ARM_SMMUV3, &virq_data, sizeof(virq_data)); } else { rc = -ENOENT; } xa_unlock(&master->vsmmu->core.vdevs); } up_read(&master->lock); } -------------------------------------------------------- [Comparison] | This v1 | Draft 1. Adds to master | A lock and vdev ptr | A lock and viommu ptr 2. Set/unset ptr | In ->vdevice_alloc/free | In all ->attach_dev 3. Do dev_to_vdev | master->vdev->id | attach_handle? Both solutions needs a driver-level lock and an extra pointer in the master structure. And both ISR routines require that driver- level lock to avoid race against attach_dev v.s. vdev alloc/free. Overall, taking step.3 into consideration, it seems that letting master lock&hold the vdev pointer (i.e. this v1) is simpler? As for the implementation of iommufd_viommu_dev_to_vdev(), I read the attach_handle part in the PRI code, yet I don't see the lock that protects the handle returned by iommu_attach_handle_get() in iommu_report_device_fault/find_fault_handler(). And I see the kdoc of iommu_attach_handle_get() mentioning: "Callers are required to synchronize the call of iommu_attach_handle_get() with domain attachment and detachment. The attach handle can only be used during its life cycle." But the caller iommu_report_device_fault() is an async event that cannot guarantee the lifecycle. Would you please shed some light? Thanks Nicolin