From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2059.outbound.protection.outlook.com [40.107.244.59]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BA5C211A2E; Wed, 22 Jan 2025 12:55:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.59 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737550520; cv=fail; b=ctaMMP295ElSYdKuuDAZZ+XHxIYJJAxvUSm5Cld+poqn16lRT8nMXL0iTp/h0qNtFiC3bmhks1SftdvccouOjoudwBJttfDehV0EKJRyDwX+d6LlgfG7CA8179jbJcsI0w6+9QkqiMdGgpTwNFMAgJv0FO0urwprXjsOE88QwjM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737550520; c=relaxed/simple; bh=/uvKcNeVU82eCdCNPuJzlToZ/Bt3bnpXbHZaeBPHMAM=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=dNL/XZViOUts2c7r3bSb8P5XzqRSkFAIAuTHWRrTEpJqunFYwx5mZYFKyouba8wZgsjb/RidY7EDfqFv72lXb1JICi2HGfcv6UHZxUWRqVHG391+N1Q2MA0qOb86K2M8ERbua9D25nArtYsPFflva4UCMpuU10svtgXo93ZuZ/s= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=l6h3bVAX; arc=fail smtp.client-ip=40.107.244.59 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="l6h3bVAX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=tSz+P4OggB+irpgBy3gvyBjL2MWzxh7gzT7CJflX6Ovbbwgv5m5D7p+gpR7mV5acd5Ug3S8I3XvGNmhqc5pV4A4R404HN3cp1ieiL4lRWSz2KMZ+3PWE6nvPc+M+XmMHgNtge60lzsp1/CRvnaAre3PrgzsR2KilxIJZC69JJO+LAcl75v7cIn5Y6n9QppcBL/bD+Tn0eqhnatqmAywKOJuGohSa+LEIcm74Fvv6KX7YHivm4lyXoJRmK4+zgSKLZTpklcraRJGKTvyWRQC6SvIv3NuCcFB5ntrEH5JKjkF2ODCCoJJA5XuDlM5aD0mgNTAzOTbDSi/iRz9RLgybsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=e9Rv93f99GexKSvfPx6sLDdktLNxaLWeG6NvVvAzyYg=; b=aWv5Vq/H4pu9CEtd9Q9EWcUiy91mllz+CqWk8XvdjiV3TFACYyB5axlIRN8CL1fItGf/NyYvXro9swz+bC1ya0Z5Dko7QMlP01DUPCvcAVBQpXWtRVRmUZ+D6IMF5hvh+M48VH5y5wjfcy0zq8j0/BR67OO/ZBRse2yYMCMb4Wfkk2jgBnejER3cvXfBev2FXfXV57VU5C0K4c8b7IGJeFZPlXTHbSgPASctJ6eESbpRphgSSng67ZAMCVKq6vZkarOShRnmsKmRwzVVN2SxRYoSMud/hZqusHVBMCs9tqdFYdTPmMUxRbIbduSjABwT0LJ54Z8x0Xzacni048b/Hw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=e9Rv93f99GexKSvfPx6sLDdktLNxaLWeG6NvVvAzyYg=; b=l6h3bVAXyqDq8qGMuzi/5prH+9wF2gIAKbaIsO65p6pdVZ6XTOhkrFHyWRjRvSM7n8P+bJ1yLrV4gGK/CeHG1YHxTcZNtpXbGnqTl6f7+UwfWmnsDOHE9LUj28Xfid1VR+xWWa5zxePOQjjPDqkH4niJ2tq9kGQFbqxKksBOItR0EaWIlL4mou6aBGr1aQnRYnsxoOkKY4Y/jLAitHIkzrjW4q/lXY+TJNowQJQJBqo3dV8TQmkA6RCbrMH2YelZWKO6WwkRAT0KEZS9HbGWn6DjrcgB7ug9cP8rHWGFNT0hAskOawIGLEd5aNEBTLQvN7FKfUHV7OsnMMpPcYNBFA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) by DS7PR12MB6214.namprd12.prod.outlook.com (2603:10b6:8:96::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8356.21; Wed, 22 Jan 2025 12:55:14 +0000 Received: from CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732]) by CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732%5]) with mapi id 15.20.8356.010; Wed, 22 Jan 2025 12:55:13 +0000 Date: Wed, 22 Jan 2025 08:55:12 -0400 From: Jason Gunthorpe To: Xu Yilun Cc: Baolu Lu , Alexey Kardashevskiy , kvm@vger.kernel.org, dri-devel@lists.freedesktop.org, linux-media@vger.kernel.org, linaro-mm-sig@lists.linaro.org, sumit.semwal@linaro.org, christian.koenig@amd.com, pbonzini@redhat.com, seanjc@google.com, alex.williamson@redhat.com, vivek.kasireddy@intel.com, dan.j.williams@intel.com, yilun.xu@intel.com, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, lukas@wunner.de, yan.y.zhao@intel.com, daniel.vetter@ffwll.ch, leon@kernel.org, zhenzhong.duan@intel.com, tao1.su@intel.com Subject: Re: [RFC PATCH 08/12] vfio/pci: Create host unaccessible dma-buf for private device Message-ID: <20250122125512.GD5556@nvidia.com> References: <20250114133553.GB5556@nvidia.com> <17cd9b77-4620-4883-9a6a-8d1cab822c88@amd.com> <20250115130102.GM5556@nvidia.com> <20250117132523.GA5556@nvidia.com> <20250120132525.GH5556@nvidia.com> <20250121174303.GV5556@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: BN9PR03CA0861.namprd03.prod.outlook.com (2603:10b6:408:13d::26) To CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8659:EE_|DS7PR12MB6214:EE_ X-MS-Office365-Filtering-Correlation-Id: 1cef4729-a565-4b50-7985-08dd3ae40317 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|7416014|376014|366016; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?ypWP5+EccDxdzVgMbsUrmY4cawKBKRg+d6JnwQ/Gl052BlG0UPTBbqhUavTD?= =?us-ascii?Q?zOuJjmX79J/6llkxWZ5d+RSrAhjWi4pV+W1t+2xVtCw4qlWkTjfeIYbNWen5?= =?us-ascii?Q?VJtfjWAL91BYUhQmo7z40xFfsE25lbMEZncHYfR9KSJCpqLgI+qE2QTCQu34?= =?us-ascii?Q?IYz+z4/01/PzRmlwOWm1cHQmJ//CYz0wW5KYphGVkyunS6ueUyTBvKIWEgd5?= =?us-ascii?Q?Q0NFSbw09J7HJv0Hd3M/gO2YYWtX1SqdyClw7dadXAZnvho7vlNwD47D2n0S?= =?us-ascii?Q?iXYLcALXvDQMSsZs26s0x7QWCH1ntoVmLMWDla6SJhcPMM9AHTl3ZL+IOcuQ?= =?us-ascii?Q?1msXIbiYRZLDry+IuHboQ2CtdmvEQJpl6/ah9Af3kwU60m4o0Jug1mUnwZHk?= =?us-ascii?Q?6UtXRThZKJEMGw06q4QGeVOLqbD4p7c7MAAZguWnUlK+gBuxxQYjHUt5HbAQ?= =?us-ascii?Q?PbC9Nbp+9HcFIDmUJpjQytR0ZM2fwxWxnsJOBKlscMLBQzgSTl0s2hVolp9A?= =?us-ascii?Q?7XTNlYJZznOiU5BH9wwyP+TFVysJJjXvNs/mGBoPaBCILPSPgVr4oBDfWny5?= =?us-ascii?Q?RxfYD7d5cEiS/PgxV0emRx2BjcRAkG26dCcxGxBOKyQzcPfAGEBQQeZHIcw9?= =?us-ascii?Q?fIBDmymMSvPVVrlAgiEAbn7ef4486H/iETO+IR43QPwCKUN8muqsWESUK5u9?= =?us-ascii?Q?nYnuCDtZibQnTZ5r0ssYcfSHgTA6KckIxWMVfo2oEPLCNTwjFXF+QlJx6+Vo?= =?us-ascii?Q?FCdzMNpON0e8xjcKS9gPbHrWiwhC9/u63gPNoHJrk7yWFg5nThvo92GtEMAv?= =?us-ascii?Q?o5Fb9xxxNYZY9l4Pqlep7abX6vJiCzgyEwTMJO3lCJ5RMtS1naSM2Beeu0OL?= =?us-ascii?Q?LkZ0yjRJKvnG8d5JRj3WSBZQ5tbFEazduxBIRzDnhtAySSZJy/VOj3EN8ITn?= =?us-ascii?Q?SZjOr8VEYj9IE8xQVVxF52BS6shnXd8MrUzPwDu15ST1Dl1LMm5zq9qrZrgS?= =?us-ascii?Q?Y8Z72BeDTnPMsklX5XysDpqtFF+7xXreYVDabdERI1noBgiVH07azekkGbZk?= =?us-ascii?Q?i0mB0mzZzRxZTT/OS6j76NvM4OUdeoBqS7e9AavuWxyMug+pBvfk7I+V4lEr?= =?us-ascii?Q?N8h+KX2EKt6IIzi19iT5hjKDXNdXFOrirLzW9EJLpb2jjtghs3+CHEj5slYh?= =?us-ascii?Q?N+dS3PhJks5x0Kxq6pyEsjCf6ms1b0RbrA5akTFffVkiZ8NaB54m6NTgiGsa?= =?us-ascii?Q?8/zFqf4UDRNWs22+UbRgRZkCb6/lXmxl/aLVNncimB/xTiREApUywCZLtoBz?= =?us-ascii?Q?G+yLDh2FZoX37tcyuETGh5TSlmj07Ls86eMp2ugWZD+Jvoz9Q2Sq/hjedO69?= =?us-ascii?Q?c+lBF+TaMGfUcESunh+ULcP4JisJ?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB8659.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(7416014)(376014)(366016);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MgTgMVQvEEhxijo+a3IYTWsXV56IBmU4HZuyRxoiPN33rzVVL9gkcKa9uLDV?= =?us-ascii?Q?KUXppY3aXl6G7SpUcSmRQPUYjnsnzx42/Baa5Mqg3FRcvGHmerKqbuvmtTtM?= =?us-ascii?Q?olI0/knLt6O3GJrJTxysvMTY9vOdoyE+zLl7eVMSXe2lc2uBU4ooN3Y1Xrk2?= =?us-ascii?Q?fsEgUBeKZCKy/RwZ2h4dYlEV9Kekyminftmuy8GeL9f7/0LqdGh6ltDZ6vni?= =?us-ascii?Q?Dz3nnNeZ4FM1Hxky7Ptfcz3EnEHXg+uuH2IpYCQXYPE3wH+OHQwCFJlFkgh8?= =?us-ascii?Q?jYFxop3jG6BO6+HnSlnZ2OPoTseDU/NW4M+tLiOGCvA5L5HRHfPbkSzsDeX+?= =?us-ascii?Q?qnGD9edsHISe7zqeBCI4cCeN47eOHThgcnKVMT9NcDWjmS6J2PKPYaDhsi6S?= =?us-ascii?Q?zRodaOr0+YrGSBN/2zB4VwPpOW3efF9p5thOMLRMy7NSt9i0ORTLU88uFWDF?= =?us-ascii?Q?SPNNe+H5+4QSyjrUxMsUtfznHBzB+8NeKvVPiFkNcZZRrvkJ74AlUJTLhHe9?= =?us-ascii?Q?qDqLEUaW9Et3rjC/5i5qMv1JnY1WncaygKJ4mFhFYmAdBxTtZB9DgOn7jk3L?= =?us-ascii?Q?LcAzuxGraXo3V50YFnx3TmOZFGzwNd5C95W5Y6e0osxsQIOur7h3O3wYuY2o?= =?us-ascii?Q?GuoRvlnp8OmzYUVkYoYZKsfjK2udkT85FgKVjZDO0RV4eG4HopibIu0owRuA?= =?us-ascii?Q?d21TEBg78bOIFQfQ4eM3VOY1Snu7kLmKYoWrhqKYBcY0IChBkPeKZRdzarWd?= =?us-ascii?Q?TogpEEGCcaviSfjL1UJ6fdEJbZ43G6yG+gAROTXeoKoEAWJBmH+4ydVNo6oY?= =?us-ascii?Q?Z9PJJrRdHT2ECpJnyJVkpoxUR05GlLz37Oe0nIfU10n3vQC0hyHsy8zQKR7E?= =?us-ascii?Q?NJ8wdTExLRrEACNvDE0a3FeUuHIT2CIvbX6PxWbyFHPKdYVk1fknCODrfLsA?= =?us-ascii?Q?GlGPTi3FZN5sVz5XvEsW3oUqitJjUK9inY36JSbcghuzA6Msnlj749oniZgy?= =?us-ascii?Q?jBDwLXvwyB55rGZN/QGdP3cCdCk5SYPF/zFgjb53jdk/2oRHhR8imcH0oNf7?= =?us-ascii?Q?623URapheZBiaArWvfluHaVMtSu/G/5zH19OUI9qBeV1LczvqjxKCXTBB0za?= =?us-ascii?Q?hE4oZ+n9rwWkmxgqp7myDtLDX4xme18LJXeOd3GfRi5iT3Ff4csfAdDf7wM3?= =?us-ascii?Q?angBMw9dE23YVL/Gb3/3S2HIopT7mNRvDQ02ftm3z1ATVZaVL10tlbcaJvt7?= =?us-ascii?Q?SHTUK02cktVpSFSccTPK1pjavNBfB6uR52WWk50tAw491VdDnYwx78L7W3Vz?= =?us-ascii?Q?mNsAZDOZ1A5X2CSzvw2hEjswcR2bWpG9LLU5KalIIGEsqvjqUq19Lz3zFyPw?= =?us-ascii?Q?yGWWRTIrpfZb6Gd148x2Z4YJaeRfwU6HWjVYG9AbBAyaPZt5A1d9nQcavCBd?= =?us-ascii?Q?kuH1uXCK3RuTwC5RT+iNhYWo6KQ482YdV3aYjsSGiwPsl0z1JKFw2rG6c+bd?= =?us-ascii?Q?HvSXQLupJoTPwmy8wxsn7BS1qTKiSFC4NcqkSpCunuQtlJAnJtcT67LbKt1y?= =?us-ascii?Q?C4rC1D+hu2dpccBDaQY=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1cef4729-a565-4b50-7985-08dd3ae40317 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8659.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Jan 2025 12:55:13.7515 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: I5v7XMHBFL+4AML48zTwXP9DvSQvnfINNuZcP474hf/Qtp6ODlDDBcDb7IBNG2+e X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR12MB6214 On Wed, Jan 22, 2025 at 12:32:56PM +0800, Xu Yilun wrote: > On Tue, Jan 21, 2025 at 01:43:03PM -0400, Jason Gunthorpe wrote: > > On Tue, Jun 25, 2024 at 05:12:10AM +0800, Xu Yilun wrote: > > > > > When VFIO works as a TEE user in VM, it means an attester (e.g. PCI > > > subsystem) has already moved the device to RUN state. So VFIO & DPDK > > > are all TEE users, no need to manipulate TDISP state between them. > > > AFAICS, this is the most preferred TIO usage in CoCo-VM. > > > > No, unfortunately. Part of the motivation to have the devices be > > unlocked when the VM starts is because there is an expectation that a > > driver in the VM will need to do untrusted operations to boot up the > > I assume these operations are device specific. Yes > > device before it can be switched to the run state. > > > > So any vfio use case needs to imagine that VFIO starts with an > > untrusted device, does stuff to it, then pushes everything through to > > I have concern that VFIO has to do device specific stuff. Our current > expectation is a specific device driver deals with the untrusted > operations, then user writes a 'bind' device sysfs node which detaches > the driver for untrusted, do the attestation and accept, and try match > the driver for trusted (e.g. VFIO). I don't see this as working, VFIO will FLR the device which will destroy anything that was done prior. VFIO itself has to do the sequence and the VFIO userspace has to contain the device specific stuff. The bind/unbind dance for untrusted->trusted would need to be internalized in VFIO without unbinding. The main motivation for the bind/unbind flow was to manage the DMA API, which VFIO does not use. Jason