From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 69FE7EE57D7 for ; Wed, 11 Sep 2024 23:02:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Type:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=vr6OTcZE+Ko9pe9FcK0UYGwWv0Jj/QukkvBLDu1H6D0=; b=PSGmTtXwasdpuiedZRJHPzQ5Eh 9QF2iQijdSCSc8EbXaz0YZI8u9ySr5W0eg+2QViQERkRVTUBiDxABZiGW+6MqNhDMI2HaCVFTtXBo N27op7/eq729WZmPC6VNCI31rmiWA0BvgAJTcQHRMv5BpbNaBBPjOKk5KCKnzSUE34XKrb4v7iH3l JEwAtTTHGGUDp0AxXl00bwJfx65TziTq2M9XpzP2VwXSY8a39j86p1XPKlZLG0efBZ5/3dZPiWnsR VlaNYFME6kRamzj6cVgtKQ3o9/ZWlSDqIRW+aD9ghxx/LsrIjarqpT37tdMaUuA7rLXSiJzq0D3r3 zDomYLpg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1soWLl-0000000BJ3Q-3uno; Wed, 11 Sep 2024 23:02:13 +0000 Received: from mail-bn8nam12on20608.outbound.protection.outlook.com ([2a01:111:f403:2418::608] helo=NAM12-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1soWKg-0000000BIn4-3TDL for linux-arm-kernel@lists.infradead.org; Wed, 11 Sep 2024 23:01:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=ES9307ANzjbErQD56/okaIoB1zdI8BasqpyKn2tkzWI0vw9hy90ik69p+O7HNHz2i6LS7ph9jw7W2b14Uqno+Bh5gm4OALK+QtH3+zvFpflMhcYzOgr2fT8dmtC/plhYqBpbEyOoNnwN9FNUDYt23sXpHja2OioyhN+1hiZlH550TpWJBcWrNc7+EZ0MGFmzTwyL2JbU1YUTzAMi67UYv03iRHe4aNuOY0ee3M64z2EdvnHtr+LOpk3aXGPtgOakr2XADt/8ufd0o+7m7hd9l6IJRuVjTVnBqxlIvz7v9MI468DODbBtmy3eg92m1t0ySIIXlPr6cpMfK3Qt0p/UkA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vr6OTcZE+Ko9pe9FcK0UYGwWv0Jj/QukkvBLDu1H6D0=; b=RNWXEm9E1PWRvisAgT54aHMqjIodhWINXlW/eqOlNJp9GPp125hxzJzYjWq34RpOLFBSj5OdsHq6UxOAJoxMJ/2IkDIiMGBGgKd4SE+hgQbKxE9bsvIYSaJxSS2VF2NO10ryRpSzyEhhdeiAEzbg/dbSA+tL4nLfZpq/QElb9KE7PdBdyLSvJ8ICpfmeMXn/680CYMFZGKoqhCD0w+MMxngHOMqLcSM8usvE4DLCZOplvyDdx4qyonxCkTdksHamCZR4M+v1GTyCH3xGy+3Z38kgctulbBm4g/VbABrLhBxtwZWfjDRhzlntNIwRhsyW9AqbPmDJ2/w4C8R7ghvTeg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vr6OTcZE+Ko9pe9FcK0UYGwWv0Jj/QukkvBLDu1H6D0=; b=ug9M3aBG+c0oiwfAVSsqk2oXpCzHvw7XXSi2rtg422OjCEKvhWUheDyaz1y2hp2CM/fgEBfnKjP84D7Lz9G7xD+MK/weznItM/59XYNXcNq7ZZEqszOSX9x/imGQCoulkVyUTGeexXPbXa9Pjsbsi1KBe3YjdExFhioAyuUhUJHYGpI9Chp2f9ECIlHC/HWTmNSaWQSrleURlnakaKjWh3wgYPhOEU7/Wq901yERGgwxmSJtk5k1bC0pnxkhHWS6/uD8i1E6kevmscHyrgwRjOoWLD/5k1BTvHaqSp4STu259NmUrwmdGgN28BgLo9HJTJ2IK/T4s3HOesy5FxbrAw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) by MW6PR12MB8950.namprd12.prod.outlook.com (2603:10b6:303:24a::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.17; Wed, 11 Sep 2024 23:00:57 +0000 Received: from CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8]) by CH3PR12MB7763.namprd12.prod.outlook.com ([fe80::8b63:dd80:c182:4ce8%5]) with mapi id 15.20.7939.022; Wed, 11 Sep 2024 23:00:57 +0000 Date: Wed, 11 Sep 2024 20:00:54 -0300 From: Jason Gunthorpe To: "Tian, Kevin" Cc: Nicolin Chen , "will@kernel.org" , "joro@8bytes.org" , "suravee.suthikulpanit@amd.com" , "robin.murphy@arm.com" , "dwmw2@infradead.org" , "baolu.lu@linux.intel.com" , "shuah@kernel.org" , "linux-kernel@vger.kernel.org" , "iommu@lists.linux.dev" , "linux-arm-kernel@lists.infradead.org" , "linux-kselftest@vger.kernel.org" , "eric.auger@redhat.com" , "jean-philippe@linaro.org" , "mdf@kernel.org" , "mshavit@google.com" , "shameerali.kolothum.thodi@huawei.com" , "smostafa@google.com" , "Liu, Yi L" Subject: Re: [PATCH v2 17/19] iommu/arm-smmu-v3: Add arm_smmu_viommu_cache_invalidate Message-ID: <20240911230054.GP58321@nvidia.com> References: <4b61aba3bc6c1cce628d9db44d5b18ea567a8be1.1724776335.git.nicolinc@nvidia.com> <20240905162039.GT1358970@nvidia.com> <20240905182148.GA1358970@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: YQBP288CA0038.CANP288.PROD.OUTLOOK.COM (2603:10b6:c01:9d::11) To CH3PR12MB7763.namprd12.prod.outlook.com (2603:10b6:610:145::10) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB7763:EE_|MW6PR12MB8950:EE_ X-MS-Office365-Filtering-Correlation-Id: 2ecfc301-0cf3-44a6-3b9b-08dcd2b59877 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|366016|376014|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?b68xM506sf57Iv4ZZJH1ViWJMCv7rQPN3WEdRzjWq3xPHvE9WeSu0BRfHm2r?= =?us-ascii?Q?2nfdYhY8ukd1DM3IbuFgGENqoCbJwmym1Oxzf7CzEXTGs86vDbaLgJ2kN09W?= =?us-ascii?Q?UDUtXr78lIBGYkQaPzXqLddJKBpbHNg3NB4iZ8VfHVN/PFX93nG2quPrSdLx?= =?us-ascii?Q?JGdDw/g2cYYR5OemkEQTKuJuSX3wAfGfKPJdiu7AgLD4J529LmPUFmn1cLj+?= =?us-ascii?Q?sOpkVnF5F4JAQE7wWXeJAIJmS07MFgwBCjVLGs74FArp5xXzWIWq6E6GrGBS?= =?us-ascii?Q?pV1e+KEMDhp8u6yyAVzjwxlo3mWtUvrK8yuyv0J16/ee3o3oGnAUr89rpSg+?= =?us-ascii?Q?1sj3WS9ho8Rel+x3VLjAXshtZTAfuq58jXtbA58ZnhekikZ01EH+/N2x8ObM?= =?us-ascii?Q?yHmr1pMJe2oNSij0eGmc/nGzihatm4hPkj5CcCcp2uEpSqmmzk5E6JLxhoOb?= =?us-ascii?Q?voCpmnxThTIjNrsYrc4tzoSbgzqK2r4UgjCAcT9sEbBSRB9xLqmbDmCzKixe?= =?us-ascii?Q?Qv/JEdCX6WT1Y13LG3Ly3q2q+PW6GoqBfTesxm9TYTNeksC9JzpbQi6rylth?= =?us-ascii?Q?JGnl/tV0sOIuYkBFi7MUsLP4Z73wgEWrRHpqowTkhV0gTmo7C+EsKP4+u7VJ?= =?us-ascii?Q?Y1R2w+7t/Zp1XUm68IUFOIyjejb42OTpFz1YosP+aLIhngdCyxj4sv0iOubt?= =?us-ascii?Q?KZKHWchDEsjpDNQ3NSru0/CVZrdRwBcbBKjLN6TOdbKulXB5COG9P6z7KsZD?= =?us-ascii?Q?xoONoA7mRagh5rp/uO/Ki7yLg5/jy1/gmYZYPxcLjpAJco7sEtZt7ORhW5TJ?= =?us-ascii?Q?ryNlCdV7xserlWc1eGEz6E6Fuk3SAXhUS+QK+uZN8znKTxQJXOSGeGfB6oX4?= =?us-ascii?Q?Mw8SeZ6cTWfOoUXSn6tV+SveLLRaYbr7e2YyVsOrWAv11hNUBXGn12LabLUO?= =?us-ascii?Q?s4gxb53wrR/nHOq2xIp1pawcOSiqiJPjihLWVENwnkLbz/woRmY/bJS5k4FD?= =?us-ascii?Q?OGhPm8LNa0Nj7e0E9bDq5mOQ7f72EGJhPv9ox+EH3hrNLyLxGcSkQB8L8kTi?= =?us-ascii?Q?HiqomNVmjBK/WNa8OuJ1oxnKTeiod99R9d/O7uN0N0R09K6B/C858Y2LroUa?= =?us-ascii?Q?fyJWwOl9UGMPx2pv3sMqw/sSobAEAZsYM15nkPQcgVNB2ETYgFZJB4pfZkJp?= =?us-ascii?Q?sn+oVQGXYY655c1WVxYOa7mxRK3ALK0KfcezRGTF23VXdcZT65mHitfWxG3J?= =?us-ascii?Q?mkI84wj76AqmBCcbmwtg2Hq36bKmAhoXHtDNBPkSlBjbIQ1tvaCHg+LfM6ja?= =?us-ascii?Q?VH9dVGM2J+rgwTWtUtNRgOJgXUEDUNpLDgYJeoMP9cwUhQ=3D=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB7763.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(366016)(376014)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?EN+5bBAuErBlMKBH4FblflMqX0dcGg3NDl6JesnIEzXTjpg6nOsGmR7k09Gv?= =?us-ascii?Q?ClOSq89fivM3i9QwcVT7fIUCTcvlflc6ulSoIr6q8TPfe2munhx2oER/6wZt?= =?us-ascii?Q?DqKNdRs66DsV1FbKx6vWMQ4frQD1ka6IIMYIX0UVlhN4Ov6W3F6XIyJbJls3?= =?us-ascii?Q?b+noy4gT0ajXYNEA4maRDypWjHjSp1jAnrHcY3o3a2b0cf8sNL+x+kAIae7J?= =?us-ascii?Q?fFt32Rh1EieRX2WQVrp7c5feLztBAyIPcne8ZZxXnmFMuW0qEG91DBcXv7P/?= =?us-ascii?Q?ocbj7gvUkr3Tp4IWBLBYk6RRARv3225j9LJ+KXL4rUcQhumvGdtI/EGpeZqk?= =?us-ascii?Q?BuUQcUzV60dBU4MRATcVpQG3W+FdnWuBGsqxVTZox2sqrprcPEAxVM6/tFJ3?= =?us-ascii?Q?q/twjRXHcZhFhR+F6j7TmiLkBTQDg312qIfvpIMOAmrC2DzRaI5y+1cjrJhy?= =?us-ascii?Q?Z3phDPK2e2wjVfnveD0KA8B3xJsdorfGdm01qQrCvWVTaaDHgIlWSbhaqSTO?= =?us-ascii?Q?bXjYFhsNQms8A+Oki5aKlvF4DdY1WCrnLKJTOonhL8bRe1T1UZB2pcYIVFAY?= =?us-ascii?Q?dcnQM2aqAvyMIy8WbRnBRazXIcapVgTzEgf0TTeNXiRtcUgFj6HW9TP2VbkQ?= =?us-ascii?Q?WZt14AIwZr7Ch+0Uxv7oYaLRhxGxsxTzzIqGQ9+uhaI9wEXabgTyq+Vb1ww0?= =?us-ascii?Q?fgWuzbdaLMvQlD9yem1KJxg59c3qS5G5e59uo9tH30MnHVRv0z1JkSCuHB8I?= =?us-ascii?Q?+vI+3fqCxychT1/YnGp732DbfJwdyV07ldRS96p+/AcaM6mfQ5rh5n7PfQU+?= =?us-ascii?Q?S93mvvYNcf4aeu7+9VD+5ANeCYofwKwtEIq7bHjvNHCv1iuicG9JE2RM+37x?= =?us-ascii?Q?qfDU0wA/8/QkFrEMw0sZuqezNHSRaSbbMldLB+bGzRWCxMlC0aeSX53g9cOe?= =?us-ascii?Q?t2sf1BG4Ed/WT3TsGhWAV0TSsOzFR5Qw3LovnNnYCRfzjoVf6bd5I32ZzW1K?= =?us-ascii?Q?Yo81ZsWAJQUTcK3QJnCrbtYeKxU9Gt9T0KOJ04DQaBfZYnExgVIJmGNmrG0d?= =?us-ascii?Q?ja1CqbR5cZl8Qy8a3+8g2kqNkkNoiUfGoO5qauY+n764XCCtGdTK60jeufaV?= =?us-ascii?Q?gxnRowbACJS6ABhN2YhVoVNKBXhRKG2iCFT/75PKvsu7RMerOqhe7YqP8sed?= =?us-ascii?Q?b5jIkbJUzJv9zmgUuzBFN08ZLQ9xcty3Vc90D/coUfxOrzPjUEKb4vHrM+XM?= =?us-ascii?Q?w/Bnu15V9MG75DAPH0cd5ARlBIT67HXhUYtSZDGv33Tc5mdrIMcEqA96yrn5?= =?us-ascii?Q?8dxLOQ9M96RUK0zBlONHKm+s2AzESqBix84jNGy+PKMmCoVSyylYwXs14dcw?= =?us-ascii?Q?izBDYXsPDr0OPrd9W0V0kUrgY1KQLHS7z2C7umTvShVXYtODc43bba+jcMt2?= =?us-ascii?Q?bF3UxYRBlE+Qf8YtlvkzYTofDmX3uxt3EsqErcAh3/gu3GEdj6acaxSHrlJj?= =?us-ascii?Q?fgNETXPIH/0p7AQyhCyYn+V8qFuQau7zdbIzVyqnt5jSaUYe7F+nfG4FSxQf?= =?us-ascii?Q?//qY4yWpcqEUEs6bH5UlVcdv7b95+Ho9Zpp+AUSR?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2ecfc301-0cf3-44a6-3b9b-08dcd2b59877 X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB7763.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2024 23:00:57.0129 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1v7cIdn0V5eY+yC78OsVckpTahFojRzOdbr/qCvEGl6OCySIfnR2Ai+f+wtRanVE X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW6PR12MB8950 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240911_160106_894994_81DF756F X-CRM114-Status: GOOD ( 29.35 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Sep 11, 2024 at 06:25:16AM +0000, Tian, Kevin wrote: > > From: Jason Gunthorpe > > Sent: Friday, September 6, 2024 2:22 AM > > > > On Thu, Sep 05, 2024 at 11:00:49AM -0700, Nicolin Chen wrote: > > > On Thu, Sep 05, 2024 at 01:20:39PM -0300, Jason Gunthorpe wrote: > > > > On Tue, Aug 27, 2024 at 09:59:54AM -0700, Nicolin Chen wrote: > > > > > > > > > +static int arm_smmu_viommu_cache_invalidate(struct > > iommufd_viommu *viommu, > > > > > + struct iommu_user_data_array > > *array) > > > > > +{ > > > > > + struct iommu_domain *domain = > > iommufd_viommu_to_parent_domain(viommu); > > > > > + > > > > > + return __arm_smmu_cache_invalidate_user( > > > > > + to_smmu_domain(domain), viommu, array); > > > > > > > > I'd like to have the viommu struct directly hold the VMID. The nested > > > > parent should be sharable between multiple viommus, it doesn't make > > > > any sense that it would hold the vmid. > > > > > > > > This is struggling because it is trying too hard to not have the > > > > driver allocate the viommu, and I think we should just go ahead and do > > > > that. Store the vmid, today copied from the nesting parent in the vmid > > > > private struct. No need for iommufd_viommu_to_parent_domain(), just > > > > rework the APIs to pass the vmid down not a domain. > > > > > > OK. When I designed all this stuff, we still haven't made mind > > > about sharing the s2 domain, i.e. moving the VMID, which might > > > need a couple of more patches to achieve. > > > > Yes, many more patches, and don't try to do it now.. But we can copy > > the vmid from the s2 and place it in the viommu struct during > > allocation time. > > does it assume that a viommu object cannot span multiple physical > IOMMUs so there is only one vmid per viommu? Yes, the viommu is not intended to cross physical iommus, it is intended to contain objects, like invalidation queues, that are tied to single piommus only. If someone does want to make vIOMMU that unifies multiple pIOMMUS, and they have the feature set that would make that possible, then they will need multiple IOMMUFD VFIOMMU objects and will have to divide up the nested domains appropriately. Jason