From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABDF93212 for ; Tue, 7 Nov 2023 14:00:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="GPDnW4Qv" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TZ4fxp6I06kZH4p9FjJ7OM+d2Md0zUG3DIqrOZcRmP947hXWUBxw1IP+U4n9pktRFo5enjpB8cJlhQkWgD5LqPp9TxsdQmZFBuXzai8M2Y3cMpmSRqSBZRRq5q8WA4amWV3hoaN8wYl0J+24hLnEpL3IYV60GOsx+R5GQoC3izZPZsZ77d90OibTRmrxT2RyleBXJQSu60Wwc5hCXvJWfyJ98DqgGgZ7Hp/2ETmJibfMfht+G9gF/sxAhE9cY9a0ByJ/dSFDAxniFDNlSwlYZYjKgGJjg6N4T2Rb909+AclF19cZLnWhZbNeGIt4fUh0l9h8ptNvGaVtBkyxl4slIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+WPaVyTBY+XVTy+mS9z1DTHPPesrFag92G2WKOnml5k=; b=VWR6LqReDjrUaSn2+xUaQ/lejDCLTA0oih6heoi+m4MpL6lgV4ywQht4kYVPyCOEgn9iq+KLa3cRB2FtOBpZXtz6ilKeZ0kkQU34ibKEGe8qZP4pLryvHJk3zJEHEtbOeXmnJVUpGWG42WpMd6D91C7uhZJR+1lSGwTSX/6fJRiS/aCq7X4FrI1xLfJ6e7afud7e6HMnHEXaQ2lf/c98b1F1xgh8AnfXf2j7KO609AEMFMhIKNJgwUFcYSLImkec0cXwrQGI4/PQzaS7HiIEvd2PLTMOqrljZPBXT7S/c0T9lQ4Zu7bUOKK2y7Em/IKQJTwyicni59EnxIj952u5oQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WPaVyTBY+XVTy+mS9z1DTHPPesrFag92G2WKOnml5k=; b=GPDnW4QvMrGAjiXonMYVrr76Rrr2tY1lbI9wr9Kky0Hm+0JzTawhHOqLkxBaXBMcz0eZN/IbHWjtQIjZfr5n03J1hncy5Skgpe7CgCJoCN7y+CXpbD+7af/23sI2ldSM/Mr6J666WJqsDgzhtAVpfSmZlZHw6M6DpDmBBwnU208z9/fphjuve8IXoT70qEBOlslH8NkV8yXU/SB2qGb3UPi1ZAqKaOiKccW5ZapNAW14LyvuIeMfjdoU+BfKCKsUNtb0GTILdJAVFcpiz4/0diPQq/7u69fC66i/xHo/HTQgF32QWDq/BzOSA/6ubZUBDtgAVf1Tb8R9iRPjPaHozg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by LV2PR12MB6016.namprd12.prod.outlook.com (2603:10b6:408:14e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.27; Tue, 7 Nov 2023 14:00:31 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::b53a:1092:9be2:cfb9]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::b53a:1092:9be2:cfb9%5]) with mapi id 15.20.6954.027; Tue, 7 Nov 2023 14:00:31 +0000 Date: Tue, 7 Nov 2023 10:00:30 -0400 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Jean-Philippe Brucker , Nicolin Chen Subject: Re: [PATCH v2 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain Message-ID: <20231107140030.GN4488@nvidia.com> References: <0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com> <21-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: BLAPR03CA0058.namprd03.prod.outlook.com (2603:10b6:208:32d::33) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|LV2PR12MB6016:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d32a2d8-3e12-4f5d-2071-08dbdf99e768 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: clZxHBz5XkxLTQJCHpTHuQ4uliA58fZ+UBmel9FWsjJL6asyR0M0wyleOYkzFqx+ZPq/INimVhfLtuZDZjoRPN0AH26G2yxnHltLhhiz3dg0znAOzOhIoRByeluPKb5ljQ3AI0vlBGMJ01yKuqX8+MPCdZ/qHyzVQN2P7lq/gjkDKf0AFMoOr/QwfzHR7RDHh4Fo7ZNeKpzJvrp3viy0nJUwqpRsrODiGcRVPn3VoDH1FWq837a9+wF/Fbaj8NPIh/Smph8TSe6U2somFzpFCVZTa7Ucso4hQ/OBHBjoHKUpZwrDXvMXN+tFsBACeXfG6dTcAsMaGqONRGRw9Asmuygo6RthYU9peHuAgBRDWEDUqi/QXPgGzi52hcmfoazBPDycj0QE5z8ZLBmrdggp+lIPXVfB6gklNU0XwpppqxJ82a8SZeBoFrZkJYLMNgQMi1HJcJ3PGJRM5vPEFq1x99ucRwp53tZyZnl4BvH73i1x+MNCTZm4ACRvXGT9oy/FRVcc8upQKvOkh00G0Qpp601vN40ghs+7855GWykPoqkKbPG3PXzzrZuOPxq6Kemv X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(376002)(346002)(396003)(366004)(39860400002)(230922051799003)(1800799009)(64100799003)(451199024)(186009)(107886003)(1076003)(2616005)(66946007)(83380400001)(26005)(6506007)(478600001)(66556008)(6486002)(53546011)(6512007)(2906002)(4326008)(8676002)(6916009)(8936002)(316002)(41300700001)(36756003)(5660300002)(38100700002)(86362001)(54906003)(66476007)(33656002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dW43clRSOW5RMUN1aEw5SFFsajhhMGpqaFZEM05wNytZNmIzZFgxWUt4T1Zh?= =?utf-8?B?QTBHQVJIVjhka3NzR0tIbXFsLzhXT1I5N1QvRlJvMklqRzdMQUU1dGc3WlVm?= =?utf-8?B?c1pkL21mWDJnVEhwWUQzY3B3TFRlblFxeTZ2MlBkZzFjcm9VaS81ZnE5LzJ0?= =?utf-8?B?STdVMDFTa1QyaDhGVnhjdVpXUGNONkljaEtoUnFtdEpVYmU3RmpZS1pBbnpl?= =?utf-8?B?WXcwY0Rnc2h0aG5relZLYVVhdzcvWEpyS09XRWFCU1VVY2dHZEtVUEdMdjhu?= =?utf-8?B?UEtQNVNWZE55SktvTjRDdEdydnJhNVRJWlVCOHRXUmxOUTRrVXJyZjJONits?= =?utf-8?B?czBES0dVQUlUdy9sOTNCc25QTys0UnBwUjRkZURxbnRaQjAvTWxmL3BaM2Np?= =?utf-8?B?NmwwWmxOWWIvaTVpSnBXdEh4SzBHeEpoV1piY25MZlJ5WDBxVjc0YmlSZXJO?= =?utf-8?B?VXB3QkIvZ0QydGt1MmhTbW5DNkxzU3FRNHZVRFpBci9XdlNzNTkrWEkxRUor?= =?utf-8?B?Tm1zcm5jY3lSNFZ1VTl2Wm5TN09sVktHWFd0YlNoS2h6SVVCWmw5VXpVQkU4?= =?utf-8?B?WUhMNlV4ejMzajE4Q243SFplbjhidGhuY3lvSFZ2cWhmVHhBb1FrVlZUdkhV?= =?utf-8?B?SG9ybEE1QjZMYUpYWGtiRVFKVDhMRk40L25ZRndZdlZPM3B3SGFic3hvUkJX?= =?utf-8?B?WnZtUnZSQzdGd0ZPSHdXZWNxN243T0czcTVBWVRUdTk5YnlVc0wvVVJHeStj?= =?utf-8?B?dTdLZm5qakZ5MnlWRlFTVVN0RFp2dWM1cDR3b1A4NjVUcG1BQXRNVUJDMmt6?= =?utf-8?B?bHNQaUw4akIreDc0cmdCTUxYSDcwLzB1cVdkTFlXMzZMMkJRL0tSSlAyODNz?= =?utf-8?B?TlFrYTRIMXpzM2NvZ1d4OVBCRjdFc2hWbktMZnB5bVVteEwrMkYwYVExMmZu?= =?utf-8?B?RGFtSzNHNDQzdnprU3U0UFd5TFpyOEU4V2xLQzlwWWY4L3duUE9nc0dRS0Z5?= =?utf-8?B?a2txbmFLYnlZV3BobDIvellER1FGbnNmY1QyQk5vdC9yczErQXdMOGptdmk2?= =?utf-8?B?RGVFZGtPOWo4VDdlSGQxOTFWN2FkZWRZeDdOZldzb1VUKzBXT3FkOGhCS2pC?= =?utf-8?B?NkZMdWVETEhwSzJ5ejdXWTJ0cVI1K251M2lmbHdoUEdhMkNuMUxDblFyT0tJ?= =?utf-8?B?aHUxMVYyM1ZremlPaElSZmFyOWxOelJjQlY3c0ljNk0vK2FFZ1BOQ3VtczZS?= =?utf-8?B?MnBpOFcwUHVlaExDd0g4WGt4Q2M1WTNZR1VvV3YyY3ExTnhKRmJhNm9LblBY?= =?utf-8?B?TlVYWmZwRjdtZ2JtT3JXdTZHbnJzLzJERS9hNWZEaGtHUlVJdEdVcjBOUnlB?= =?utf-8?B?TUxzTVJlK0xPTHIrT2N2UmVVVjRQdGpoNmsrMXZZVmdHVEp0K05mS2lYZjIr?= =?utf-8?B?WUhyQXlnZlNlVEhDL2JDcThaY1FvV1BtcVZTOG9qWXUvUzM0Mjl1TC9xQjVi?= =?utf-8?B?WWdGdm1YR2g0MXRDdVlNNkh5V2laYmRaYU8reVBvUzdYaC9xMVJPMU5vbk4z?= =?utf-8?B?cWtDdFJKOTFJeUtCWGJRSWRoS1NWS3NwVjNrcFRicHppejBUbUF3bTh6ZWli?= =?utf-8?B?b1JoY2dTY1NEN0FUU1h1SUQrMW42M21yWDBWdkZ3dDFXN3RSK3huTnhuQWc4?= =?utf-8?B?NmpGYTgyVWRRNnVqRndtTnBIaXE4bFJoSmxoR0dlT1FjbHFMNFdkZTRnZEYx?= =?utf-8?B?b3M4dERWZXNLNUNmVTNBdUt0NFB1N0RTUHJoSmY4VGZEdVFDM284VndJUmx3?= =?utf-8?B?eXVJQVpLOHE2WUlZUHNYSVdlM09Kd2FTcTFINFBOYTlnaWYyQ2p2bWJONCtL?= =?utf-8?B?YVRnTVl5TFJaMHgyVThPejlYUmJCb0FtNXU4VnJPSXFLNkt0U2xlcnFpRGFR?= =?utf-8?B?MVoyWDZtNUIyUDdGcmVqVHdqN05WcjdVRXhZYVVUZTVKSVlBM0gvZlJLeDZ2?= =?utf-8?B?UThtamsyWEwreU16MTJHNnI0cjV2VEtPeHRlNFk4YmdYWDhZRmkyVTc0OGpo?= =?utf-8?B?MGZlN2hnUWZPc0V5aGNRVFR0R0JHOUtxdy82bS9zTHJpbUF2Mnc5YWxLT1JO?= =?utf-8?Q?MB4UnJ+uNBJ3aQpqriOn9BIOJ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d32a2d8-3e12-4f5d-2071-08dbdf99e768 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2023 14:00:30.9835 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ISCKcCzFTaUJYj7LFrkeMHUYvaaHaHJ35MvVCdFE9pL3OycL8Tkh1AU+zngJ03se X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6016 On Tue, Nov 07, 2023 at 09:28:08PM +0800, Michael Shavit wrote: > On Thu, Nov 2, 2023 at 7:37 AM Jason Gunthorpe wrote: > > [...] > > @@ -309,24 +169,26 @@ static void arm_smmu_mm_release(struct mmu_notifier *mn, struct mm_struct *mm) > > struct arm_smmu_cd target; > > struct arm_smmu_cd *cdptr; > > > > - cdptr = arm_smmu_get_cd_ptr(master, mm->pasid); > > + cdptr = arm_smmu_get_cd_ptr(master, master_domain->ssid); > > if (WARN_ON(!cdptr)) > > continue; > > - arm_smmu_make_sva_cd(&target, master, NULL, smmu_mn->cd->asid); > > - arm_smmu_write_cd_entry(master, mm->pasid, cdptr, &target); > > + arm_smmu_make_sva_cd(&target, master, NULL, > > + smmu_domain->cd.asid, > > + smmu_domain->btm_invalidation); > > + arm_smmu_write_cd_entry(master, master_domain->ssid, cdptr, > > + &target); > > } > > spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); > > > > - arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_mn->cd->asid); > > - arm_smmu_atc_inv_domain_sva(smmu_domain, mm->pasid, 0, 0); > > - > > - smmu_mn->cleared = true; > > - mutex_unlock(&sva_lock); > > + arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->cd.asid); > > Similar questions to patch 11 from the v1, but why is it ok to remove > the ATC invalidation here? It isn't, it is a mistake as well! > Did you perhaps mean to remove the TLB invalidation instead (for which > it's IIUC ok to delay the invalidation to when the domain/asid is > freed, since those cache entries won't give a hit while the CD is > cleared)? Hmm. I found this: * When EPDx == 1, a translation table walk through TTBx causes F_TRANSLATION. - Note: The Armv8-A VMSA allows a TLB hit to occur for an input address associated with an EPD bit set to 1, but the translation table walk is disabled upon miss. So we need to flush the ASID too when using EPD to disable it. Like this: arm_smmu_tlb_inv_asid(smmu_domain->smmu, smmu_domain->asid); + arm_smmu_atc_inv_domain(smmu_domain, 0, 0); } Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 96FC9C4332F for ; Tue, 7 Nov 2023 14:01:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=94mAQQwXQebawPxboDSaRMoHG2QOprd4JdD84y1P3+w=; b=QKrjbP7Hp471TG lYt2S6nCzl4hI8luJDjaEuu69l3fCiNTYa2F7ylxt32ldDSBh7k4Jy0bFCv1d2CFc/Opb/iygSdKW tAr8qml/wNU6Dx6s8kkayChhspH0j9mch75punu5Wm9u54uqmQmIL7oQyKAwNMJas8ab2rjl6XOB4 jmHnzGNsHap8dQT3SeGKHljA1rn6QwwUclTqnRnXnX/yPePTyKeHpxP35pgtRkrz+dd/7dLlRTYay Cml28KFGFoGiOHfezninJnd5EiB4m3UNTgb9QFfvuw0ppHd2yvwGjSV6xqBt0ZXz1GRSMGAHVjOJb soED2rXxsC7wrf0/4Gsg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r0MdI-001ggg-17; Tue, 07 Nov 2023 14:00:44 +0000 Received: from mail-dm6nam10on20607.outbound.protection.outlook.com ([2a01:111:f400:7e88::607] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1r0MdF-001gfi-0d for linux-arm-kernel@lists.infradead.org; Tue, 07 Nov 2023 14:00:42 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TZ4fxp6I06kZH4p9FjJ7OM+d2Md0zUG3DIqrOZcRmP947hXWUBxw1IP+U4n9pktRFo5enjpB8cJlhQkWgD5LqPp9TxsdQmZFBuXzai8M2Y3cMpmSRqSBZRRq5q8WA4amWV3hoaN8wYl0J+24hLnEpL3IYV60GOsx+R5GQoC3izZPZsZ77d90OibTRmrxT2RyleBXJQSu60Wwc5hCXvJWfyJ98DqgGgZ7Hp/2ETmJibfMfht+G9gF/sxAhE9cY9a0ByJ/dSFDAxniFDNlSwlYZYjKgGJjg6N4T2Rb909+AclF19cZLnWhZbNeGIt4fUh0l9h8ptNvGaVtBkyxl4slIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=+WPaVyTBY+XVTy+mS9z1DTHPPesrFag92G2WKOnml5k=; b=VWR6LqReDjrUaSn2+xUaQ/lejDCLTA0oih6heoi+m4MpL6lgV4ywQht4kYVPyCOEgn9iq+KLa3cRB2FtOBpZXtz6ilKeZ0kkQU34ibKEGe8qZP4pLryvHJk3zJEHEtbOeXmnJVUpGWG42WpMd6D91C7uhZJR+1lSGwTSX/6fJRiS/aCq7X4FrI1xLfJ6e7afud7e6HMnHEXaQ2lf/c98b1F1xgh8AnfXf2j7KO609AEMFMhIKNJgwUFcYSLImkec0cXwrQGI4/PQzaS7HiIEvd2PLTMOqrljZPBXT7S/c0T9lQ4Zu7bUOKK2y7Em/IKQJTwyicni59EnxIj952u5oQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+WPaVyTBY+XVTy+mS9z1DTHPPesrFag92G2WKOnml5k=; b=GPDnW4QvMrGAjiXonMYVrr76Rrr2tY1lbI9wr9Kky0Hm+0JzTawhHOqLkxBaXBMcz0eZN/IbHWjtQIjZfr5n03J1hncy5Skgpe7CgCJoCN7y+CXpbD+7af/23sI2ldSM/Mr6J666WJqsDgzhtAVpfSmZlZHw6M6DpDmBBwnU208z9/fphjuve8IXoT70qEBOlslH8NkV8yXU/SB2qGb3UPi1ZAqKaOiKccW5ZapNAW14LyvuIeMfjdoU+BfKCKsUNtb0GTILdJAVFcpiz4/0diPQq/7u69fC66i/xHo/HTQgF32QWDq/BzOSA/6ubZUBDtgAVf1Tb8R9iRPjPaHozg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by LV2PR12MB6016.namprd12.prod.outlook.com (2603:10b6:408:14e::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6954.27; Tue, 7 Nov 2023 14:00:31 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::b53a:1092:9be2:cfb9]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::b53a:1092:9be2:cfb9%5]) with mapi id 15.20.6954.027; Tue, 7 Nov 2023 14:00:31 +0000 Date: Tue, 7 Nov 2023 10:00:30 -0400 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Jean-Philippe Brucker , Nicolin Chen Subject: Re: [PATCH v2 21/27] iommu/arm-smmu-v3: Put the SVA mmu notifier in the smmu_domain Message-ID: <20231107140030.GN4488@nvidia.com> References: <0-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com> <21-v2-16665a652079+5947-smmuv3_newapi_p2_jgg@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: BLAPR03CA0058.namprd03.prod.outlook.com (2603:10b6:208:32d::33) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|LV2PR12MB6016:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d32a2d8-3e12-4f5d-2071-08dbdf99e768 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: clZxHBz5XkxLTQJCHpTHuQ4uliA58fZ+UBmel9FWsjJL6asyR0M0wyleOYkzFqx+ZPq/INimVhfLtuZDZjoRPN0AH26G2yxnHltLhhiz3dg0znAOzOhIoRByeluPKb5ljQ3AI0vlBGMJ01yKuqX8+MPCdZ/qHyzVQN2P7lq/gjkDKf0AFMoOr/QwfzHR7RDHh4Fo7ZNeKpzJvrp3viy0nJUwqpRsrODiGcRVPn3VoDH1FWq837a9+wF/Fbaj8NPIh/Smph8TSe6U2somFzpFCVZTa7Ucso4hQ/OBHBjoHKUpZwrDXvMXN+tFsBACeXfG6dTcAsMaGqONRGRw9Asmuygo6RthYU9peHuAgBRDWEDUqi/QXPgGzi52hcmfoazBPDycj0QE5z8ZLBmrdggp+lIPXVfB6gklNU0XwpppqxJ82a8SZeBoFrZkJYLMNgQMi1HJcJ3PGJRM5vPEFq1x99ucRwp53tZyZnl4BvH73i1x+MNCTZm4ACRvXGT9oy/FRVcc8upQKvOkh00G0Qpp601vN40ghs+7855GWykPoqkKbPG3PXzzrZuOPxq6Kemv X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(376002)(346002)(396003)(366004)(39860400002)(230922051799003)(1800799009)(64100799003)(451199024)(186009)(107886003)(1076003)(2616005)(66946007)(83380400001)(26005)(6506007)(478600001)(66556008)(6486002)(53546011)(6512007)(2906002)(4326008)(8676002)(6916009)(8936002)(316002)(41300700001)(36756003)(5660300002)(38100700002)(86362001)(54906003)(66476007)(33656002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?dW43clRSOW5RMUN1aEw5SFFsajhhMGpqaFZEM05wNytZNmIzZFgxWUt4T1Zh?= =?utf-8?B?QTBHQVJIVjhka3NzR0tIbXFsLzhXT1I5N1QvRlJvMklqRzdMQUU1dGc3WlVm?= =?utf-8?B?c1pkL21mWDJnVEhwWUQzY3B3TFRlblFxeTZ2MlBkZzFjcm9VaS81ZnE5LzJ0?= =?utf-8?B?STdVMDFTa1QyaDhGVnhjdVpXUGNONkljaEtoUnFtdEpVYmU3RmpZS1pBbnpl?= =?utf-8?B?WXcwY0Rnc2h0aG5relZLYVVhdzcvWEpyS09XRWFCU1VVY2dHZEtVUEdMdjhu?= =?utf-8?B?UEtQNVNWZE55SktvTjRDdEdydnJhNVRJWlVCOHRXUmxOUTRrVXJyZjJONits?= =?utf-8?B?czBES0dVQUlUdy9sOTNCc25QTys0UnBwUjRkZURxbnRaQjAvTWxmL3BaM2Np?= =?utf-8?B?NmwwWmxOWWIvaTVpSnBXdEh4SzBHeEpoV1piY25MZlJ5WDBxVjc0YmlSZXJO?= =?utf-8?B?VXB3QkIvZ0QydGt1MmhTbW5DNkxzU3FRNHZVRFpBci9XdlNzNTkrWEkxRUor?= =?utf-8?B?Tm1zcm5jY3lSNFZ1VTl2Wm5TN09sVktHWFd0YlNoS2h6SVVCWmw5VXpVQkU4?= =?utf-8?B?WUhMNlV4ejMzajE4Q243SFplbjhidGhuY3lvSFZ2cWhmVHhBb1FrVlZUdkhV?= =?utf-8?B?SG9ybEE1QjZMYUpYWGtiRVFKVDhMRk40L25ZRndZdlZPM3B3SGFic3hvUkJX?= =?utf-8?B?WnZtUnZSQzdGd0ZPSHdXZWNxN243T0czcTVBWVRUdTk5YnlVc0wvVVJHeStj?= =?utf-8?B?dTdLZm5qakZ5MnlWRlFTVVN0RFp2dWM1cDR3b1A4NjVUcG1BQXRNVUJDMmt6?= =?utf-8?B?bHNQaUw4akIreDc0cmdCTUxYSDcwLzB1cVdkTFlXMzZMMkJRL0tSSlAyODNz?= =?utf-8?B?TlFrYTRIMXpzM2NvZ1d4OVBCRjdFc2hWbktMZnB5bVVteEwrMkYwYVExMmZu?= =?utf-8?B?RGFtSzNHNDQzdnprU3U0UFd5TFpyOEU4V2xLQzlwWWY4L3duUE9nc0dRS0Z5?= =?utf-8?B?a2txbmFLYnlZV3BobDIvellER1FGbnNmY1QyQk5vdC9yczErQXdMOGptdmk2?= =?utf-8?B?RGVFZGtPOWo4VDdlSGQxOTFWN2FkZWRZeDdOZldzb1VUKzBXT3FkOGhCS2pC?= =?utf-8?B?NkZMdWVETEhwSzJ5ejdXWTJ0cVI1K251M2lmbHdoUEdhMkNuMUxDblFyT0tJ?= =?utf-8?B?aHUxMVYyM1ZremlPaElSZmFyOWxOelJjQlY3c0ljNk0vK2FFZ1BOQ3VtczZS?= =?utf-8?B?MnBpOFcwUHVlaExDd0g4WGt4Q2M1WTNZR1VvV3YyY3ExTnhKRmJhNm9LblBY?= =?utf-8?B?TlVYWmZwRjdtZ2JtT3JXdTZHbnJzLzJERS9hNWZEaGtHUlVJdEdVcjBOUnlB?= =?utf-8?B?TUxzTVJlK0xPTHIrT2N2UmVVVjRQdGpoNmsrMXZZVmdHVEp0K05mS2lYZjIr?= =?utf-8?B?WUhyQXlnZlNlVEhDL2JDcThaY1FvV1BtcVZTOG9qWXUvUzM0Mjl1TC9xQjVi?= =?utf-8?B?WWdGdm1YR2g0MXRDdVlNNkh5V2laYmRaYU8reVBvUzdYaC9xMVJPMU5vbk4z?= =?utf-8?B?cWtDdFJKOTFJeUtCWGJRSWRoS1NWS3NwVjNrcFRicHppejBUbUF3bTh6ZWli?= =?utf-8?B?b1JoY2dTY1NEN0FUU1h1SUQrMW42M21yWDBWdkZ3dDFXN3RSK3huTnhuQWc4?= =?utf-8?B?NmpGYTgyVWRRNnVqRndtTnBIaXE4bFJoSmxoR0dlT1FjbHFMNFdkZTRnZEYx?= =?utf-8?B?b3M4dERWZXNLNUNmVTNBdUt0NFB1N0RTUHJoSmY4VGZEdVFDM284VndJUmx3?= =?utf-8?B?eXVJQVpLOHE2WUlZUHNYSVdlM09Kd2FTcTFINFBOYTlnaWYyQ2p2bWJONCtL?= =?utf-8?B?YVRnTVl5TFJaMHgyVThPejlYUmJCb0FtNXU4VnJPSXFLNkt0U2xlcnFpRGFR?= =?utf-8?B?MVoyWDZtNUIyUDdGcmVqVHdqN05WcjdVRXhZYVVUZTVKSVlBM0gvZlJLeDZ2?= =?utf-8?B?UThtamsyWEwreU16MTJHNnI0cjV2VEtPeHRlNFk4YmdYWDhZRmkyVTc0OGpo?= =?utf-8?B?MGZlN2hnUWZPc0V5aGNRVFR0R0JHOUtxdy82bS9zTHJpbUF2Mnc5YWxLT1JO?= =?utf-8?Q?MB4UnJ+uNBJ3aQpqriOn9BIOJ?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d32a2d8-3e12-4f5d-2071-08dbdf99e768 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2023 14:00:30.9835 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ISCKcCzFTaUJYj7LFrkeMHUYvaaHaHJ35MvVCdFE9pL3OycL8Tkh1AU+zngJ03se X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6016 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231107_060041_256475_1AEB1ECB X-CRM114-Status: GOOD ( 14.53 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCBOb3YgMDcsIDIwMjMgYXQgMDk6Mjg6MDhQTSArMDgwMCwgTWljaGFlbCBTaGF2aXQg d3JvdGU6Cj4gT24gVGh1LCBOb3YgMiwgMjAyMyBhdCA3OjM34oCvQU0gSmFzb24gR3VudGhvcnBl IDxqZ2dAbnZpZGlhLmNvbT4gd3JvdGU6Cj4gPiBbLi4uXQo+ID4gQEAgLTMwOSwyNCArMTY5LDI2 IEBAIHN0YXRpYyB2b2lkIGFybV9zbW11X21tX3JlbGVhc2Uoc3RydWN0IG1tdV9ub3RpZmllciAq bW4sIHN0cnVjdCBtbV9zdHJ1Y3QgKm1tKQo+ID4gICAgICAgICAgICAgICAgIHN0cnVjdCBhcm1f c21tdV9jZCB0YXJnZXQ7Cj4gPiAgICAgICAgICAgICAgICAgc3RydWN0IGFybV9zbW11X2NkICpj ZHB0cjsKPiA+Cj4gPiAtICAgICAgICAgICAgICAgY2RwdHIgPSBhcm1fc21tdV9nZXRfY2RfcHRy KG1hc3RlciwgbW0tPnBhc2lkKTsKPiA+ICsgICAgICAgICAgICAgICBjZHB0ciA9IGFybV9zbW11 X2dldF9jZF9wdHIobWFzdGVyLCBtYXN0ZXJfZG9tYWluLT5zc2lkKTsKPiA+ICAgICAgICAgICAg ICAgICBpZiAoV0FSTl9PTighY2RwdHIpKQo+ID4gICAgICAgICAgICAgICAgICAgICAgICAgY29u dGludWU7Cj4gPiAtICAgICAgICAgICAgICAgYXJtX3NtbXVfbWFrZV9zdmFfY2QoJnRhcmdldCwg bWFzdGVyLCBOVUxMLCBzbW11X21uLT5jZC0+YXNpZCk7Cj4gPiAtICAgICAgICAgICAgICAgYXJt X3NtbXVfd3JpdGVfY2RfZW50cnkobWFzdGVyLCBtbS0+cGFzaWQsIGNkcHRyLCAmdGFyZ2V0KTsK PiA+ICsgICAgICAgICAgICAgICBhcm1fc21tdV9tYWtlX3N2YV9jZCgmdGFyZ2V0LCBtYXN0ZXIs IE5VTEwsCj4gPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc21tdV9kb21h aW4tPmNkLmFzaWQsCj4gPiArICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgc21t dV9kb21haW4tPmJ0bV9pbnZhbGlkYXRpb24pOwo+ID4gKyAgICAgICAgICAgICAgIGFybV9zbW11 X3dyaXRlX2NkX2VudHJ5KG1hc3RlciwgbWFzdGVyX2RvbWFpbi0+c3NpZCwgY2RwdHIsCj4gPiAr ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgJnRhcmdldCk7Cj4gPiAgICAg ICAgIH0KPiA+ICAgICAgICAgc3Bpbl91bmxvY2tfaXJxcmVzdG9yZSgmc21tdV9kb21haW4tPmRl dmljZXNfbG9jaywgZmxhZ3MpOwo+ID4KPiA+IC0gICAgICAgYXJtX3NtbXVfdGxiX2ludl9hc2lk KHNtbXVfZG9tYWluLT5zbW11LCBzbW11X21uLT5jZC0+YXNpZCk7Cj4gPiAtICAgICAgIGFybV9z bW11X2F0Y19pbnZfZG9tYWluX3N2YShzbW11X2RvbWFpbiwgbW0tPnBhc2lkLCAwLCAwKTsKPiA+ IC0KPiA+IC0gICAgICAgc21tdV9tbi0+Y2xlYXJlZCA9IHRydWU7Cj4gPiAtICAgICAgIG11dGV4 X3VubG9jaygmc3ZhX2xvY2spOwo+ID4gKyAgICAgICBhcm1fc21tdV90bGJfaW52X2FzaWQoc21t dV9kb21haW4tPnNtbXUsIHNtbXVfZG9tYWluLT5jZC5hc2lkKTsKPiAKPiBTaW1pbGFyIHF1ZXN0 aW9ucyB0byBwYXRjaCAxMSBmcm9tIHRoZSB2MSwgYnV0IHdoeSBpcyBpdCBvayB0byByZW1vdmUK PiB0aGUgQVRDIGludmFsaWRhdGlvbiBoZXJlPyAKCkl0IGlzbid0LCBpdCBpcyBhIG1pc3Rha2Ug YXMgd2VsbCEKCj4gRGlkIHlvdSBwZXJoYXBzIG1lYW4gdG8gcmVtb3ZlIHRoZSBUTEIgaW52YWxp ZGF0aW9uIGluc3RlYWQgKGZvciB3aGljaAo+IGl0J3MgSUlVQyBvayB0byBkZWxheSB0aGUgaW52 YWxpZGF0aW9uIHRvIHdoZW4gdGhlIGRvbWFpbi9hc2lkIGlzCj4gZnJlZWQsIHNpbmNlIHRob3Nl IGNhY2hlIGVudHJpZXMgd29uJ3QgZ2l2ZSBhIGhpdCB3aGlsZSB0aGUgQ0QgaXMKPiBjbGVhcmVk KT8KCkhtbS4gSSBmb3VuZCB0aGlzOgoKKiBXaGVuIEVQRHggPT0gMSwgYSB0cmFuc2xhdGlvbiB0 YWJsZSB3YWxrIHRocm91Z2ggVFRCeCBjYXVzZXMgRl9UUkFOU0xBVElPTi4KCi0gTm90ZTogVGhl IEFybXY4LUEgVk1TQSBhbGxvd3MgYSBUTEIgaGl0IHRvIG9jY3VyIGZvciBhbiBpbnB1dAogIGFk ZHJlc3MgYXNzb2NpYXRlZCB3aXRoIGFuIEVQRCBiaXQgc2V0IHRvIDEsIGJ1dCB0aGUgdHJhbnNs YXRpb24KICB0YWJsZSB3YWxrIGlzIGRpc2FibGVkIHVwb24gbWlzcy4KClNvIHdlIG5lZWQgdG8g Zmx1c2ggdGhlIEFTSUQgdG9vIHdoZW4gdXNpbmcgRVBEIHRvIGRpc2FibGUgaXQuCgpMaWtlIHRo aXM6CgogICAgICAgIGFybV9zbW11X3RsYl9pbnZfYXNpZChzbW11X2RvbWFpbi0+c21tdSwgc21t dV9kb21haW4tPmFzaWQpOworICAgICAgIGFybV9zbW11X2F0Y19pbnZfZG9tYWluKHNtbXVfZG9t YWluLCAwLCAwKTsKIH0KCkphc29uCgpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdApsaW51eC1hcm0ta2Vy bmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1h bi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg==