From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2070.outbound.protection.outlook.com [40.107.244.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 703909468; Thu, 21 Mar 2024 17:32:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.70 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711042358; cv=fail; b=Uurp50oBr9qyIBV/1oawgGycpgU7tDHU/F08Uhj3zDGtlwOOIGq8WLzR7E+UI7x3Vt8e5e3dW0YdJHENheNLs4+tY+Rxu/1qxrBkg1AQY7p62mRIpxPP2wBt6zEGkhSPlsqVvfxMlSvQHD4MGW05zElunc3Ibipubmnniv2XO0A= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1711042358; c=relaxed/simple; bh=/pDluwAE5YzAm1cxkhM0MpFw+bz/bSC2q4vu7KgpMHU=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=XPnMwlq0/qRK990pzC/T9S8fi32kOTjOxZWIU8le1XAFoHt6Ow6pZqFV+YRu8tHWTpoQAancK7+fWGiq3PBqJFZ5jcljRlEd/IeoMpNDfgAC967aLvo+jIxlZ/1QkTc/rjoFOaHahcWIF8plOr2BL5pIPICq0ljam46Y203VCM8= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=EgNzmWNM; arc=fail smtp.client-ip=40.107.244.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="EgNzmWNM" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hHcGny/IjM8HSkgamFWacPO7YnKiShQXcsGDxxEZDVlbZ6b4/R2/z395fkENeUN29WpcVyWKyHIY83spYJes+dKLlqDWXAShhe2ZRbnPtyphmx0JU9+dv9soIq4/LPZ9m4WNNKF68iK66kaLyZldz/XGX4RA1+vmoC3UxPXKVZEcXVGy7STIjEeI5f1ZzVIaO6470sfhDBc/Uy4gAYe/Iipze80DYS0G6XF+SLSccZm8v7ZupcDAmHKQcMNMzYPzsKZDFd9UX11W0WbP9ws67tkMX56j1vPl70UJGUx29Rwq6bXIIsmsGTC+rAN/OeLNvUU1pB98D6rL07PTdEeQFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lx34rZOUTYnKnxDeHAKD6ihqpn0FmS4OQPjKRgX5ttg=; b=aGWSjVr5efScBJPAed5wn8rxka/xLqM59Uk9GHlQ+0fmUvmIrMJtoQSMvA4cH7OgZMxSDBpnHF5OrY9Qe7KnwsdSUw8Wu+fZD3cm3rmfVOCKGCTiTwM/jRXQlnjZF8qlmCXrgO9DvkYyYVFQT8Uc0OVx9mWD9SLJPrXTBRulnD3iMWZ5f9HqHD5evTYKaUzokelVr5+s8reORHNtG6k8sgYk1i8UpvsMQW/zJpYW3JnZet5SbC1WZTc4ot9+fdgpTNINLFDudC2si5oRTwc+zdBQlsE50oq0GZkwhXx4HxlSlL9m+ovCsQrnUw26cQF/DziIC77um0y93qr8JeUqUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lx34rZOUTYnKnxDeHAKD6ihqpn0FmS4OQPjKRgX5ttg=; b=EgNzmWNMndTQgZZ9TaFmv9qgflSig8+6RiQIp/mrTrWzK0a8q2spyz9+5S5xAY7J/4+fmnkmE5O+t0Y5Qu55Z4VLBcw0K2LUjIOyHHP0MuAPl3KC703c5g7yHt6MwOmPGcKCl79LOT91bX2Br583HNo05bwzlHUGq4AgQQKZsEDNnfFdxY5BKla6mKnl18gb40gBZGWGy3TZkC4Taarg4fcxqda6VxWSnpeNjweBSHIA2IVJU4bqQcYC0z7whM2QTUMMhzfwDmtUI30B3E7pcQzvxYIcmNb9GwcY0UXITVe9XINp87CcvcdzOEpIXZ7nitKsEjT1c5OcOVLKQGfcUQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CY8PR12MB8297.namprd12.prod.outlook.com (2603:10b6:930:79::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.34; Thu, 21 Mar 2024 17:32:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7386.030; Thu, 21 Mar 2024 17:32:33 +0000 Date: Thu, 21 Mar 2024 14:32:32 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Message-ID: <20240321173232.GM1044235@nvidia.com> References: <0-v5-9a37e0c884ce+31e3-smmuv3_newapi_p2_jgg@nvidia.com> <14-v5-9a37e0c884ce+31e3-smmuv3_newapi_p2_jgg@nvidia.com> <20240321132839.GK1044235@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: MN2PR01CA0038.prod.exchangelabs.com (2603:10b6:208:23f::7) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CY8PR12MB8297:EE_ X-MS-Office365-Filtering-Correlation-Id: 58cb20bc-7a82-4525-f042-08dc49cce443 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 229HL6R7LbCT/0QGSQCnCLoc2VlXdxiRsDUK9M6L8nXua4MmDZYPQjusZfxUvJhZfvLN1Fzgd9B6TAIVvvI96qO3c5m/vFPGZW80h7AS3F022hPeUQ558vgXH9HDn9208nS6Bc5616T8A/vq5b9HdGRozSjkGyLTBOwSLi2Nz34Nhv4kv+7Noo//xuPN4Y/hqvN4V1HvvF13JYpX0ev4/k2/7EamY9UyN1k5gYiSwD8OfFbic9QnB8H09VE5xVC5cDfhYZ1+EhDTPKAbWjREnfoEjf8Mkr+r2LWoiXsi+1mAiUNGDG7AIpmKe6nfg40oUJuHH1D50Bvyzh0knO6U5IrkvOUiWpr6wfeGamhj1cw5ZcMcEojPwtRRQCaZ8rFgBzj9F2szk8NUJcfBUaaCcXAWQHPJSJY2qT0VnZKnbm+PdkR/xbzcYSPyGp6QzdeMHwiS/3IoQ1rMXl0oz5SV3IGKf47iYv2uck3cjLjEsKLmyAUZEOs9xDr5vJgVX1gyQrd63sNGbmdzyOk6zLdyiZfvcEQ2F+hVD5NN1xWbTcSUQ0rngStjNShVvJJGKk0mR+GfCSYOrgWfaBPLMGHmVG8RYqnqqEtO5sL4BcT/sVIk3PmeFAKBJIu3Vsa/7Jp9slzZTSUqJMFPDcEVfkq8OIGbr722IP99kJ7c6daCnIQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(1800799015)(7416005)(376005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?amR5a1dPUTNRUzhsd0xBYXhjQndGeWlySERLL2ZXdllsczQxSWRTRHhaOUlC?= =?utf-8?B?TjBYMTBpcDZFVGpDQVRTdU84cGFVU1pUNDFmamNRNGZ1SXU0b0hUVFpzeUQy?= =?utf-8?B?cUpvYytXQXB5Y0J3eWY0NEw1bGRjTDhyaVM4ZGl3cENQOUF2dGl1SnViNFVC?= =?utf-8?B?dzk4Z29iWWJLU01ERWJicFlLREZjNVBBWStTNzQyTHlVM0p0VWh0U3M2Zjly?= =?utf-8?B?YmdqWTc3WGhiLzUvdjZwZ3duaHZOYzNkUXZLK2lGM3Jjb2g1U1grdW5vN29P?= =?utf-8?B?NmZTT2V3VFNBbkJwRTV2Q0JvZnBlY2p3RXFNeVVLRHVaaldBZFdXVW1YUXJO?= =?utf-8?B?WVRYWUNRMXljcENzVHBGOGhXQmJiOE5YZCtXTGZQdFo1Ulc2aElQWGF0REtn?= =?utf-8?B?b0N6cVlvWUtzc2RuZ0dla09RRG1ERnFLNXVicDF5K0JMTmNWbEEwLzhicGlP?= =?utf-8?B?bFc5QzVydy9UMkx6OEVUMUdQdnBCeFd5N2JZcTc2WmZSVFdCdHlVdXFkcm5Y?= =?utf-8?B?NVQ0VkJIWmhmQVFjUExub00xWFYralRoTmlyNDVzVGY3ejNZaGd6RzBhS3Ju?= =?utf-8?B?Wno5cFczUUhlOE1WV0tZS1pzbjFuTkt3aUo5LzZwSVNGSkNmN2t4Ukg3c3J1?= =?utf-8?B?cVZsaXBQRm1JWXdjQzNPMzJOZGozRGs3dXZaSWRmUldYaEtxNExTejZLNWl2?= =?utf-8?B?NzVUS0lvSXZPYzNNTUhWc1R4b2UycFVWdjEzNXRoeVJCYlRlOFNkTW5Lcjk2?= =?utf-8?B?N0UzaStTTjFabFVCTkpwT1dJcC9vUkF3eUhhMTNWVnNpMk1FaEttaEc0a1FC?= =?utf-8?B?TlhacGt6R0xTdUxrS29xeWZBWGplVXhpYVY0WnlkODY1b3FKZmhuY0oyQmJI?= =?utf-8?B?WlNiVWJRbHNNSXJwS3V5Vk9uRUpDZDNyeGx6bWRxaGF0KzhhS0FrNUJSeDRv?= =?utf-8?B?WVVscmZaQlJCdnF3TlVoZ2ptVVAvdHZPektOSCtXL0wyVmhmOFM0Z3VpQllG?= =?utf-8?B?VTVSaFRDZlljZHNqQ1hYU0kyU1NpYnp3MEgvOVpzZk82L2NiazUzdmdIdjNs?= =?utf-8?B?SXlZVXVja1FWVG1jTGdUMWpFWXEySDR0OUM4VmtFdzlGNVRrb3ZveHc4bkM1?= =?utf-8?B?ekJTMGczY29TVG9rSVd5RHloOHZkQnMvcXZLS1NoSDVNSmtCUVFLazdlRGtu?= =?utf-8?B?Q0pUTVFYdGJITWl3amJjNGNiTDBvd1JHREkzaWlpbkFPbUc1Tk1maDNkb3hs?= =?utf-8?B?aFNib0J4ZlhZbmFkWm5NQ0lacDNHTHBkSzRqM3NURUQwdHF5eU5nRHFkTHhu?= =?utf-8?B?czllZXQ2a254MG9VcHJZVUUvNHZUNXhJS09QTjFwbFFybWpVM2NDMmp5NFZE?= =?utf-8?B?dFVqeTVPSldIWkhBZWhBbHlZRjh2STFxalRnd2ltdTNmSEVoazhqMWRSUEph?= =?utf-8?B?YnNtODQ1b0VGUlEyT1pWbFJTVXF6cEtRNzliak9FYjBuSWJCc3pUakpuNE5j?= =?utf-8?B?ZjJXd3BybnJxYXZ4VXRPTXhwSmhCV1R6NlJNMW5VOGpCbWNHT3E5MFo2Y1dj?= =?utf-8?B?eDBHRFdMa3NrS0FpUHFnUldkaTFFYnJWK1NRbWt1Y1labHAvYVpVc0VwRnQr?= =?utf-8?B?VjJZdzVkQ2ZrV3c2Yy9QQjhVOHFjWVVseXF4WnJrM0F0S1dFZTArOGpFYWZq?= =?utf-8?B?Kzl6dTZicEFSbXJWbHJ4a1Q2NnAzZXdSNVV4ZGM2WkRZK3hQU0dnVXdKazlR?= =?utf-8?B?UjFkY2xtdUxNam51WG44RWNRK2M5cEF0NFN4K0hkSU1XYm91Z3pBNnNSU2tu?= =?utf-8?B?ejR2bHZ5K1lFajZuZ1RkeU5SWVppZWFiTC8vNFcwRkJNSWxLcHlEbjFOQzZ1?= =?utf-8?B?SmEyWSt1SDhKd0d6SnQ3ZWNLN1YyaEl0TnE4MU14dzNDdVNLL2IwcWtpTTdV?= =?utf-8?B?bDFmWjFBNnV6OHBkSmxDbUZEcEdpZUNlSnpsQ0I4ZUY3dnZGNkhrZ3FsTFo2?= =?utf-8?B?a3pQMlRqRWpRNjJWcDBLcVNTbllPaER4SmNtNjNoNHNhUFkxSmdRdmVETjFs?= =?utf-8?B?Yno2WnNDSi9meERISUpGRE80cjE1d3EweURyM25veEFjdVZNd3cyR3JZVzJz?= =?utf-8?Q?rQTFS/G6zP+nuzqG51xjL/AvV?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58cb20bc-7a82-4525-f042-08dc49cce443 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2024 17:32:33.2779 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7xAeaTLRrZxSi7nn+Cvo/yZxt+uxipmxRocEUApe0h2Bwfb+HS3cNRZ0MJnNp4M1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8297 On Thu, Mar 21, 2024 at 10:53:20PM +0800, Michael Shavit wrote: > On Thu, Mar 21, 2024 at 9:28 PM Jason Gunthorpe wrote: > > > > On Thu, Mar 21, 2024 at 08:26:43PM +0800, Michael Shavit wrote: > > > Overall I think the patch works, but it took me a while to really > > > digest the big picture. Just to make sure I fully understand: > > > > > > We're trying to satisfy the following invariants for correctness: > > > 1. Devices cannot get translations from a domain that was detached > > > after arm_smmu_attach_dev() returns. > > > > Yes, regardless of ATS this is the API requirement. > > > > > 2. Devices cannot get cached translations from a domain after > > > arm_smmu_atc_inv_domain() returns, regardless of whether the domain is > > > simultaneously being attached/detached from a device. > > > > There is no simultaneously here since the group lock is held by the > > core code. > > I meant a call to arm_smmu_attach_dev concurrent with > arm_smmu_atc_inv_domain (through tlb_flush_all). Oh, yes that is the concurrency that drives alot of this organization > > Yes, we can't just disable ATS due to the PASIDs. A SVA PASID might be > > present and disabling ATS to change the RID domain would completely > > wreck it. Today this scenario is prevented by sva_enable, which is > > removed in following patches. > > What do you mean by wrecking? We might slow it down to a crawl but we > wouldn't be corrupting or destroying anything by disabling ATS while > SVA is running would we? > Oh does disabling ATS abort all transactions rather than making it transparent? As you deduced, without ATS a non-present page will return an abort back to the device instead of a non-present ATS reply which should then trigger PRI. IOW the SVA domain can no longer handle page faults and is effectively non-functioning if ATS is disabled. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94827C6FD1F for ; Thu, 21 Mar 2024 17:33:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yjhOb4WGJtNVbS+LVGg/D8yWrDbZaSg/Eqdky4MhN78=; b=1IBOrAvHcGpeoo ukFs9SHL0XlF/Jrz9sx7qNhPU1JR47yieqfrxa0qdzNVcDFyuM0NWV3dmk5tpgz2NQ+xkFDphiJAe a1woeQpbKuOKdD91fKBHsGJnLLyUOUkySAZcHiSccms+ZtcdTMI/QxMYCRurlVpTpi4mbnpfrW/Qu SEPr7g+Z0nSxeYTICmj1ql2LNjyFcRfHGdXDQ/LzIlHZz1WsilC8lUADOlis4tchhp3rCv/Bsdfwt 2wWxR2E3ivttzKDQxee7ml6ADPSs1YfwXzhyj5FJUeS/v8cGAa5zyhqqTLYeu0M3mb7Lt7+EokNMD asLIXVryenTnCZBeDxnw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnMHW-000000045zf-34Rf; Thu, 21 Mar 2024 17:32:46 +0000 Received: from mail-bn1nam02on20600.outbound.protection.outlook.com ([2a01:111:f403:2407::600] helo=NAM02-BN1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rnMHU-000000045xv-2Ce9 for linux-arm-kernel@lists.infradead.org; Thu, 21 Mar 2024 17:32:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hHcGny/IjM8HSkgamFWacPO7YnKiShQXcsGDxxEZDVlbZ6b4/R2/z395fkENeUN29WpcVyWKyHIY83spYJes+dKLlqDWXAShhe2ZRbnPtyphmx0JU9+dv9soIq4/LPZ9m4WNNKF68iK66kaLyZldz/XGX4RA1+vmoC3UxPXKVZEcXVGy7STIjEeI5f1ZzVIaO6470sfhDBc/Uy4gAYe/Iipze80DYS0G6XF+SLSccZm8v7ZupcDAmHKQcMNMzYPzsKZDFd9UX11W0WbP9ws67tkMX56j1vPl70UJGUx29Rwq6bXIIsmsGTC+rAN/OeLNvUU1pB98D6rL07PTdEeQFw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=lx34rZOUTYnKnxDeHAKD6ihqpn0FmS4OQPjKRgX5ttg=; b=aGWSjVr5efScBJPAed5wn8rxka/xLqM59Uk9GHlQ+0fmUvmIrMJtoQSMvA4cH7OgZMxSDBpnHF5OrY9Qe7KnwsdSUw8Wu+fZD3cm3rmfVOCKGCTiTwM/jRXQlnjZF8qlmCXrgO9DvkYyYVFQT8Uc0OVx9mWD9SLJPrXTBRulnD3iMWZ5f9HqHD5evTYKaUzokelVr5+s8reORHNtG6k8sgYk1i8UpvsMQW/zJpYW3JnZet5SbC1WZTc4ot9+fdgpTNINLFDudC2si5oRTwc+zdBQlsE50oq0GZkwhXx4HxlSlL9m+ovCsQrnUw26cQF/DziIC77um0y93qr8JeUqUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=lx34rZOUTYnKnxDeHAKD6ihqpn0FmS4OQPjKRgX5ttg=; b=EgNzmWNMndTQgZZ9TaFmv9qgflSig8+6RiQIp/mrTrWzK0a8q2spyz9+5S5xAY7J/4+fmnkmE5O+t0Y5Qu55Z4VLBcw0K2LUjIOyHHP0MuAPl3KC703c5g7yHt6MwOmPGcKCl79LOT91bX2Br583HNo05bwzlHUGq4AgQQKZsEDNnfFdxY5BKla6mKnl18gb40gBZGWGy3TZkC4Taarg4fcxqda6VxWSnpeNjweBSHIA2IVJU4bqQcYC0z7whM2QTUMMhzfwDmtUI30B3E7pcQzvxYIcmNb9GwcY0UXITVe9XINp87CcvcdzOEpIXZ7nitKsEjT1c5OcOVLKQGfcUQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) by CY8PR12MB8297.namprd12.prod.outlook.com (2603:10b6:930:79::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7386.34; Thu, 21 Mar 2024 17:32:33 +0000 Received: from DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222]) by DM6PR12MB3849.namprd12.prod.outlook.com ([fe80::6aec:dbca:a593:a222%5]) with mapi id 15.20.7386.030; Thu, 21 Mar 2024 17:32:33 +0000 Date: Thu, 21 Mar 2024 14:32:32 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Nicolin Chen , patches@lists.linux.dev, Shameerali Kolothum Thodi Subject: Re: [PATCH v5 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Message-ID: <20240321173232.GM1044235@nvidia.com> References: <0-v5-9a37e0c884ce+31e3-smmuv3_newapi_p2_jgg@nvidia.com> <14-v5-9a37e0c884ce+31e3-smmuv3_newapi_p2_jgg@nvidia.com> <20240321132839.GK1044235@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MN2PR01CA0038.prod.exchangelabs.com (2603:10b6:208:23f::7) To DM6PR12MB3849.namprd12.prod.outlook.com (2603:10b6:5:1c7::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB3849:EE_|CY8PR12MB8297:EE_ X-MS-Office365-Filtering-Correlation-Id: 58cb20bc-7a82-4525-f042-08dc49cce443 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 229HL6R7LbCT/0QGSQCnCLoc2VlXdxiRsDUK9M6L8nXua4MmDZYPQjusZfxUvJhZfvLN1Fzgd9B6TAIVvvI96qO3c5m/vFPGZW80h7AS3F022hPeUQ558vgXH9HDn9208nS6Bc5616T8A/vq5b9HdGRozSjkGyLTBOwSLi2Nz34Nhv4kv+7Noo//xuPN4Y/hqvN4V1HvvF13JYpX0ev4/k2/7EamY9UyN1k5gYiSwD8OfFbic9QnB8H09VE5xVC5cDfhYZ1+EhDTPKAbWjREnfoEjf8Mkr+r2LWoiXsi+1mAiUNGDG7AIpmKe6nfg40oUJuHH1D50Bvyzh0knO6U5IrkvOUiWpr6wfeGamhj1cw5ZcMcEojPwtRRQCaZ8rFgBzj9F2szk8NUJcfBUaaCcXAWQHPJSJY2qT0VnZKnbm+PdkR/xbzcYSPyGp6QzdeMHwiS/3IoQ1rMXl0oz5SV3IGKf47iYv2uck3cjLjEsKLmyAUZEOs9xDr5vJgVX1gyQrd63sNGbmdzyOk6zLdyiZfvcEQ2F+hVD5NN1xWbTcSUQ0rngStjNShVvJJGKk0mR+GfCSYOrgWfaBPLMGHmVG8RYqnqqEtO5sL4BcT/sVIk3PmeFAKBJIu3Vsa/7Jp9slzZTSUqJMFPDcEVfkq8OIGbr722IP99kJ7c6daCnIQ= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB3849.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366007)(1800799015)(7416005)(376005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?amR5a1dPUTNRUzhsd0xBYXhjQndGeWlySERLL2ZXdllsczQxSWRTRHhaOUlC?= =?utf-8?B?TjBYMTBpcDZFVGpDQVRTdU84cGFVU1pUNDFmamNRNGZ1SXU0b0hUVFpzeUQy?= =?utf-8?B?cUpvYytXQXB5Y0J3eWY0NEw1bGRjTDhyaVM4ZGl3cENQOUF2dGl1SnViNFVC?= =?utf-8?B?dzk4Z29iWWJLU01ERWJicFlLREZjNVBBWStTNzQyTHlVM0p0VWh0U3M2Zjly?= =?utf-8?B?YmdqWTc3WGhiLzUvdjZwZ3duaHZOYzNkUXZLK2lGM3Jjb2g1U1grdW5vN29P?= =?utf-8?B?NmZTT2V3VFNBbkJwRTV2Q0JvZnBlY2p3RXFNeVVLRHVaaldBZFdXVW1YUXJO?= =?utf-8?B?WVRYWUNRMXljcENzVHBGOGhXQmJiOE5YZCtXTGZQdFo1Ulc2aElQWGF0REtn?= =?utf-8?B?b0N6cVlvWUtzc2RuZ0dla09RRG1ERnFLNXVicDF5K0JMTmNWbEEwLzhicGlP?= =?utf-8?B?bFc5QzVydy9UMkx6OEVUMUdQdnBCeFd5N2JZcTc2WmZSVFdCdHlVdXFkcm5Y?= =?utf-8?B?NVQ0VkJIWmhmQVFjUExub00xWFYralRoTmlyNDVzVGY3ejNZaGd6RzBhS3Ju?= =?utf-8?B?Wno5cFczUUhlOE1WV0tZS1pzbjFuTkt3aUo5LzZwSVNGSkNmN2t4Ukg3c3J1?= =?utf-8?B?cVZsaXBQRm1JWXdjQzNPMzJOZGozRGs3dXZaSWRmUldYaEtxNExTejZLNWl2?= =?utf-8?B?NzVUS0lvSXZPYzNNTUhWc1R4b2UycFVWdjEzNXRoeVJCYlRlOFNkTW5Lcjk2?= =?utf-8?B?N0UzaStTTjFabFVCTkpwT1dJcC9vUkF3eUhhMTNWVnNpMk1FaEttaEc0a1FC?= =?utf-8?B?TlhacGt6R0xTdUxrS29xeWZBWGplVXhpYVY0WnlkODY1b3FKZmhuY0oyQmJI?= =?utf-8?B?WlNiVWJRbHNNSXJwS3V5Vk9uRUpDZDNyeGx6bWRxaGF0KzhhS0FrNUJSeDRv?= =?utf-8?B?WVVscmZaQlJCdnF3TlVoZ2ptVVAvdHZPektOSCtXL0wyVmhmOFM0Z3VpQllG?= =?utf-8?B?VTVSaFRDZlljZHNqQ1hYU0kyU1NpYnp3MEgvOVpzZk82L2NiazUzdmdIdjNs?= =?utf-8?B?SXlZVXVja1FWVG1jTGdUMWpFWXEySDR0OUM4VmtFdzlGNVRrb3ZveHc4bkM1?= =?utf-8?B?ekJTMGczY29TVG9rSVd5RHloOHZkQnMvcXZLS1NoSDVNSmtCUVFLazdlRGtu?= =?utf-8?B?Q0pUTVFYdGJITWl3amJjNGNiTDBvd1JHREkzaWlpbkFPbUc1Tk1maDNkb3hs?= =?utf-8?B?aFNib0J4ZlhZbmFkWm5NQ0lacDNHTHBkSzRqM3NURUQwdHF5eU5nRHFkTHhu?= =?utf-8?B?czllZXQ2a254MG9VcHJZVUUvNHZUNXhJS09QTjFwbFFybWpVM2NDMmp5NFZE?= =?utf-8?B?dFVqeTVPSldIWkhBZWhBbHlZRjh2STFxalRnd2ltdTNmSEVoazhqMWRSUEph?= =?utf-8?B?YnNtODQ1b0VGUlEyT1pWbFJTVXF6cEtRNzliak9FYjBuSWJCc3pUakpuNE5j?= =?utf-8?B?ZjJXd3BybnJxYXZ4VXRPTXhwSmhCV1R6NlJNMW5VOGpCbWNHT3E5MFo2Y1dj?= =?utf-8?B?eDBHRFdMa3NrS0FpUHFnUldkaTFFYnJWK1NRbWt1Y1labHAvYVpVc0VwRnQr?= =?utf-8?B?VjJZdzVkQ2ZrV3c2Yy9QQjhVOHFjWVVseXF4WnJrM0F0S1dFZTArOGpFYWZq?= =?utf-8?B?Kzl6dTZicEFSbXJWbHJ4a1Q2NnAzZXdSNVV4ZGM2WkRZK3hQU0dnVXdKazlR?= =?utf-8?B?UjFkY2xtdUxNam51WG44RWNRK2M5cEF0NFN4K0hkSU1XYm91Z3pBNnNSU2tu?= =?utf-8?B?ejR2bHZ5K1lFajZuZ1RkeU5SWVppZWFiTC8vNFcwRkJNSWxLcHlEbjFOQzZ1?= =?utf-8?B?SmEyWSt1SDhKd0d6SnQ3ZWNLN1YyaEl0TnE4MU14dzNDdVNLL2IwcWtpTTdV?= =?utf-8?B?bDFmWjFBNnV6OHBkSmxDbUZEcEdpZUNlSnpsQ0I4ZUY3dnZGNkhrZ3FsTFo2?= =?utf-8?B?a3pQMlRqRWpRNjJWcDBLcVNTbllPaER4SmNtNjNoNHNhUFkxSmdRdmVETjFs?= =?utf-8?B?Yno2WnNDSi9meERISUpGRE80cjE1d3EweURyM25veEFjdVZNd3cyR3JZVzJz?= =?utf-8?Q?rQTFS/G6zP+nuzqG51xjL/AvV?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 58cb20bc-7a82-4525-f042-08dc49cce443 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB3849.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Mar 2024 17:32:33.2779 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7xAeaTLRrZxSi7nn+Cvo/yZxt+uxipmxRocEUApe0h2Bwfb+HS3cNRZ0MJnNp4M1 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8297 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240321_103244_589840_57266DA4 X-CRM114-Status: GOOD ( 22.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1LCBNYXIgMjEsIDIwMjQgYXQgMTA6NTM6MjBQTSArMDgwMCwgTWljaGFlbCBTaGF2aXQg d3JvdGU6Cj4gT24gVGh1LCBNYXIgMjEsIDIwMjQgYXQgOToyOOKAr1BNIEphc29uIEd1bnRob3Jw ZSA8amdnQG52aWRpYS5jb20+IHdyb3RlOgo+ID4KPiA+IE9uIFRodSwgTWFyIDIxLCAyMDI0IGF0 IDA4OjI2OjQzUE0gKzA4MDAsIE1pY2hhZWwgU2hhdml0IHdyb3RlOgo+ID4gPiBPdmVyYWxsIEkg dGhpbmsgdGhlIHBhdGNoIHdvcmtzLCBidXQgaXQgdG9vayBtZSBhIHdoaWxlIHRvIHJlYWxseQo+ ID4gPiBkaWdlc3QgdGhlIGJpZyBwaWN0dXJlLiBKdXN0IHRvIG1ha2Ugc3VyZSBJIGZ1bGx5IHVu ZGVyc3RhbmQ6Cj4gPiA+Cj4gPiA+IFdlJ3JlIHRyeWluZyB0byBzYXRpc2Z5IHRoZSBmb2xsb3dp bmcgaW52YXJpYW50cyBmb3IgY29ycmVjdG5lc3M6Cj4gPiA+IDEuIERldmljZXMgY2Fubm90IGdl dCB0cmFuc2xhdGlvbnMgZnJvbSBhIGRvbWFpbiB0aGF0IHdhcyBkZXRhY2hlZAo+ID4gPiBhZnRl ciBhcm1fc21tdV9hdHRhY2hfZGV2KCkgcmV0dXJucy4KPiA+Cj4gPiBZZXMsIHJlZ2FyZGxlc3Mg b2YgQVRTIHRoaXMgaXMgdGhlIEFQSSByZXF1aXJlbWVudC4KPiA+Cj4gPiA+IDIuIERldmljZXMg Y2Fubm90IGdldCBjYWNoZWQgdHJhbnNsYXRpb25zIGZyb20gYSBkb21haW4gYWZ0ZXIKPiA+ID4g YXJtX3NtbXVfYXRjX2ludl9kb21haW4oKSByZXR1cm5zLCByZWdhcmRsZXNzIG9mIHdoZXRoZXIg dGhlIGRvbWFpbiBpcwo+ID4gPiBzaW11bHRhbmVvdXNseSBiZWluZyBhdHRhY2hlZC9kZXRhY2hl ZCBmcm9tIGEgZGV2aWNlLgo+ID4KPiA+IFRoZXJlIGlzIG5vIHNpbXVsdGFuZW91c2x5IGhlcmUg c2luY2UgdGhlIGdyb3VwIGxvY2sgaXMgaGVsZCBieSB0aGUKPiA+IGNvcmUgY29kZS4KPiAKPiBJ IG1lYW50IGEgY2FsbCB0byBhcm1fc21tdV9hdHRhY2hfZGV2IGNvbmN1cnJlbnQgd2l0aAo+IGFy bV9zbW11X2F0Y19pbnZfZG9tYWluICh0aHJvdWdoIHRsYl9mbHVzaF9hbGwpLgoKT2gsIHllcyB0 aGF0IGlzIHRoZSBjb25jdXJyZW5jeSB0aGF0IGRyaXZlcyBhbG90IG9mIHRoaXMgb3JnYW5pemF0 aW9uCgo+ID4gWWVzLCB3ZSBjYW4ndCBqdXN0IGRpc2FibGUgQVRTIGR1ZSB0byB0aGUgUEFTSURz LiBBIFNWQSBQQVNJRCBtaWdodCBiZQo+ID4gcHJlc2VudCBhbmQgZGlzYWJsaW5nIEFUUyB0byBj aGFuZ2UgdGhlIFJJRCBkb21haW4gd291bGQgY29tcGxldGVseQo+ID4gd3JlY2sgaXQuIFRvZGF5 IHRoaXMgc2NlbmFyaW8gaXMgcHJldmVudGVkIGJ5IHN2YV9lbmFibGUsIHdoaWNoIGlzCj4gPiBy ZW1vdmVkIGluIGZvbGxvd2luZyBwYXRjaGVzLgo+IAo+IFdoYXQgZG8geW91IG1lYW4gYnkgd3Jl Y2tpbmc/IFdlIG1pZ2h0IHNsb3cgaXQgZG93biB0byBhIGNyYXdsIGJ1dCB3ZQo+IHdvdWxkbid0 IGJlIGNvcnJ1cHRpbmcgb3IgZGVzdHJveWluZyBhbnl0aGluZyBieSBkaXNhYmxpbmcgQVRTIHdo aWxlCj4gU1ZBIGlzIHJ1bm5pbmcgd291bGQgd2U/Cj4gT2ggZG9lcyBkaXNhYmxpbmcgQVRTIGFi b3J0IGFsbCB0cmFuc2FjdGlvbnMgcmF0aGVyIHRoYW4gbWFraW5nIGl0IHRyYW5zcGFyZW50PwoK QXMgeW91IGRlZHVjZWQsIHdpdGhvdXQgQVRTIGEgbm9uLXByZXNlbnQgcGFnZSB3aWxsIHJldHVy biBhbiBhYm9ydApiYWNrIHRvIHRoZSBkZXZpY2UgaW5zdGVhZCBvZiBhIG5vbi1wcmVzZW50IEFU UyByZXBseSB3aGljaCBzaG91bGQKdGhlbiB0cmlnZ2VyIFBSSS4KCklPVyB0aGUgU1ZBIGRvbWFp biBjYW4gbm8gbG9uZ2VyIGhhbmRsZSBwYWdlIGZhdWx0cyBhbmQgaXMgZWZmZWN0aXZlbHkKbm9u LWZ1bmN0aW9uaW5nIGlmIEFUUyBpcyBkaXNhYmxlZC4KIApKYXNvbgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5n IGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5p bmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=