From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2048.outbound.protection.outlook.com [40.107.92.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 37EDF2E417 for ; Thu, 26 Oct 2023 14:11:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="P5JFE/oz" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NA7sX5sU/Qebr76QBYToOc9rMbgCdVh7b0O8reIajwRt+R8rrHFEszlpRzinxlPIPqb4S0ybePp+DwjyL3Kn4/y8vBOvznymZwGd5AWvMq6J2ti/B+rUYJARi6fQlGtOg/46DZWxBhfb0eNj5AqtgqL7XMnyTjkM/wduPbmHUmKJU/kci/ACdcmUgvATiCWbhb8qRz6GkLdVkuEnV+DjUr4kHxieM0kKWHDLqwsvXTeSxNHo4GkyhW7yisJNlfX31x/eCBUjBHgctPvEF+MJjPh2L7pa1mFyrA+bx2j7SN6PtRzz/6DjRd1ePgyvqVS7lG1tu3lSisXwbSr8eULgcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ihWbLqG/tz8dOo8xB5kn0PVmFNtLG3vbNmHxB6OaZ4c=; b=TJTmRW70kLbDOvISOC8Q7pkkchISuAn01O1Nj8VmkWFbXoFIiJh/tOk9reaUIUcChQf+vVtBAqKNCYQCivn4o1pqV2oKFw0XlUjOr9ZymjWr+M2lh/MvHX4N6UMozUqDIDvwL5M2qCdAqugP0qtHH4EkTTPAGFFUG88EtijRzW/WRET5ORcDDwNwku0KnPAGClJdLgx5d6Rw/vzo4KKn0jZWmXQBRKkrFdsz5Eelv+258qJyswwFTX3alABVDWEzfjEM898clYAQZhG6MB7fzNqCG8P+O8nfQU7pFLAIPpLzyp9lWLbetUkZ37vIsc9Fnkj+vftUpC6fmue/oO5hCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ihWbLqG/tz8dOo8xB5kn0PVmFNtLG3vbNmHxB6OaZ4c=; b=P5JFE/ozIXvfy3HLOQk3iJ6fxDOm9LDC0euLu5bRgd40aM0Q5AYppt1XtVuB0WcOXZ/TC20q5s9CCgbH8Sy05mMxKhzjHCgVa+DF45RiJLAejz7JU/I7qpgLCFngqwpvxn6TXj43BAuFz0/JBSHqAAaQ31gjVlfGhnY/M91b5+GLy5tNUooy9tA9lc3QWFcZUsCaXJFc8vXYmhg2QVGKk1QiO/L/gSmvdDDYg0JHWEGQF/QvNofPp8KwQ/ed6rviDCJe9P/G8Uk5XNHh2kPKAdUgLne2XrQoOpzximd87ngnLSoWX966/ZmuKVowzRZOVr+AiWIy0rlzSAk+GxmbYg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by DM4PR12MB6638.namprd12.prod.outlook.com (2603:10b6:8:b5::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Thu, 26 Oct 2023 14:11:03 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::11a1:b0c7:7c88:9480]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::11a1:b0c7:7c88:9480%4]) with mapi id 15.20.6933.022; Thu, 26 Oct 2023 14:11:03 +0000 Date: Thu, 26 Oct 2023 11:11:01 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Jean-Philippe Brucker , Nicolin Chen Subject: Re: [PATCH 11/27] iommu/arm-smmu-v3: Lift CD programming out of the SVA notifier code Message-ID: <20231026141101.GO3952@nvidia.com> References: <0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com> <11-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com> <20231024234608.GD911568@nvidia.com> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: SA9PR13CA0078.namprd13.prod.outlook.com (2603:10b6:806:23::23) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|DM4PR12MB6638:EE_ X-MS-Office365-Filtering-Correlation-Id: a031b77a-73b8-457a-e55d-08dbd62d6342 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IDpS4higXByi1FS0CU7+ouIyzvyKkV5eulaD6000DPChBleYd+qutV96eJg4WVcwb8JL36xWwNwfHxAGv6EhX5i5DWILppZjZT1v6/HR3BhzlkAWijkZIcBqBsnKU1jTiI3qHxLYhEOg1l5SuCYDb9Sz8rQJp+7nYOWVOh60IDTED2t72u4bc+hHqBD/hWFSBclniS1gbtCWruzjuu6sJL23PveQ/Ti2LnQ85h8tc+w6zIyPsOhpRCOWj1+UdO/XRZRQ9JJERCBjxWfR9HLp1qk54e7wJYJ3zRRE5baErGHwJk7TCngckXEha4ABD1I19s8MQwLYNiyBn6MFQziVqdLUHcDN2vj6NUmWpJhvvKhA57SHu5PiQot6y9YNLtm6r5BhWkGLccJmM14mSr/T2krp/VmKPUOqG2SY5Z+DvliP07OGPlkFQF9vNtD62YVMAGtkEi1jHrLHPGId80jO5KjSK5VRCswmSNdH4IggLnSQ3dvMegbF1qaxQRIVKVzA7rrIlZYOpVjJH0IQlazqPj3OpRUj7fMee1x0yKdB5HssUeRpBXy+oSsA8rFRPYtW X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(376002)(136003)(346002)(39860400002)(366004)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(66946007)(5660300002)(2906002)(41300700001)(316002)(6916009)(4326008)(54906003)(66476007)(66556008)(53546011)(478600001)(6486002)(8936002)(8676002)(6506007)(107886003)(66899024)(2616005)(1076003)(26005)(6512007)(83380400001)(86362001)(38100700002)(33656002)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M2VuSmg1ejFaVm5aMmRlSVFIRVpxcUpFLzdKUTU2UnhtditCVmk4eHdWWkNt?= =?utf-8?B?L0p6WXEvUy9yek5NZzlJSlk0bHA2VmRrRDNGRkV6cDcrNHRKejFDcURCTWJx?= =?utf-8?B?NU1zaEJ1ZnpyNldadmQ0RjNlcEtBUWhwY2w4TC9zQWZ4czYzdWhVUWRRTUdj?= =?utf-8?B?d0NZbFFmN2pmYVhKSyswdFgyVVd1SldaQ01va3B5cnA2Mko3WEIxRldPa0JD?= =?utf-8?B?VlByUHFvY1hxM1I5c0dHbjRQTnk4NkgxOFNCN1JRcEJ6cStuUEZOZkxYNTJR?= =?utf-8?B?emx3MTVYeFEwTkRGYnRKd0xEZTBERDAwUlhiMGJzUXpmdlBEZE5VNXJSM0Zp?= =?utf-8?B?NndvdmtZWEZaSEFLQ0wvQkd1N2Q4cG9FTzZmTFk1blFqK3c1NU5oRmZ0Q1li?= =?utf-8?B?WC9EcG5HR3NWS1NQNlhubm9GNmxaU3lIK2NLVWhxTmQ5QVNvTWlqU0FXTWlV?= =?utf-8?B?eVd1Z3p4WkhicUtxeVNLNHZlQUNOdUpUTU94Y05rYzQvbDlMZkhFWWd5bkZL?= =?utf-8?B?c29rblc2a29DcEVaQTF6YXluT2tBQ3IzU0Qya0VldndpbXlxL1Y3b09CSktL?= =?utf-8?B?U0FSK2YrdU9ZM3dra1hBRmVkaVBvL2NJMTNBWmh1cDF0Z09UbHpHemZidzQy?= =?utf-8?B?dGlYbzhDeEZBbWcyT2pISlM3bmUrTC9MNkNKaXU0NmlGaURXTytJU2piaXpY?= =?utf-8?B?YzhvOFNxdVB3UlNyMVpPQjVLZVNEUVJWWkFEOXFxVnpZcEtyL3pEaGlwZjVz?= =?utf-8?B?ZjBVNWh1RlNhMmFYQWl5TmtnTkRlNFRCNnpMQzJvYWt1OElrZzh1S0xPQW5k?= =?utf-8?B?ODhPUlNtWEVuU1VCMUxqTEM5S0kwTkFPV0M4RjlZOUFxVWdSZzdocmZMVUdL?= =?utf-8?B?bjl3UXVxL2UxWDFoU2NtTUwrWEFjMFZpWkJNaGcxVmE5UHBFQVQ1d3pkdzFX?= =?utf-8?B?RWNjejMxUDFRN21SUmI1bHoxcjFKUUNPM3llQzU3aTc0eFYrNk9wTEFuMFlY?= =?utf-8?B?UnhmYVdqVUxyWUtCc3NBMVlsU0lTZlZQVXpXZFVQMCtZRnZOblNFcWk3RmZM?= =?utf-8?B?RnhjUEdOOFZ5ZU92T3dFOXdlQUk3VFV3UnJKWXg0cGdTTlNzSWNLbTJxV3d1?= =?utf-8?B?YWtpdHdiVE1xbUIxZUhvQjNqSE14NWkweGhKZDhnZmR0bUtXakhka3Y2aVRh?= =?utf-8?B?YVZqL1o1N2xFZmFBUEdtNUdPamd2c09OOUlQUmZxQm9VWENMaUQva0VHeGZE?= =?utf-8?B?Z3l3YTZBY1FwUU54cmY3LzNWOHZMZDBxY0RDRmF4THNQWlVXK05kNDNBNk5h?= =?utf-8?B?QmNhU0JsS1U4dHhuZGlxU0IvZ2s5Rm5xcC9PMkZaK2VuMGxLcWkvZHZXTm5H?= =?utf-8?B?bUtSUHl3NEZwSnp0RWN1UjduckYrWVo2cGhLYWJaQ2R3RVcxRXNNRVpnQlQr?= =?utf-8?B?YWdTSjdiaVllMzlJWG1LcVVMSkRrS1BZRGNLOGdTUDFVYSs3Wks1STBTb1RP?= =?utf-8?B?dU8vekVKNFk0MkREK1BCalVCdVN2VTRJak0yQU9nb1lXQXlWOEhmVnJvYTJn?= =?utf-8?B?K1NJRGEvUTBHRWpnZjhYNVVBTGV1Nm5PcDkxWVJTTEFrVEZBbkMxMzAwdy9N?= =?utf-8?B?dEk5eER3OGtSN29wTERoYUZzaW5KNEZIYkFzNHB3dVRNSis2ZnMvWEI2RXI2?= =?utf-8?B?SlBHQlB2MmxkVjZOQmJJRXY0NHpMZUJkbVdiNlkvQ3M3cTRmL3F6b3NmVlI1?= =?utf-8?B?Yy85b21LNjJyNmc4OHUvUmxBMkRKMDAwREtSWU9uQkZRSXhac0VuK2lZTXlR?= =?utf-8?B?b1FOMlBlakRYa1RGaVV2VTNxcUR3N0NpWWltWE1nT0UyRkpzYjFOMHFDUWxx?= =?utf-8?B?YlRFUXBVTmJCQ2pZazIwbVNjL0tZVUVUQ2RlUXRlWFRJMmN1L3BnQnhkc2Ry?= =?utf-8?B?YWt5M0J5NWpIcEYrNnJ5YU10VlM1K1o5bjVFS2xJTnR0WDhHZHRubVlwWWtw?= =?utf-8?B?K0Nmd2V5OHlMSUNNZjZuOFdneGNhQXZXejNPOVVaVGZ5TUlSYUxxOEcrWm5K?= =?utf-8?B?TjY1TFhOZmlxS0Y0VnhrN2R1emIwSjJlSVVjNGxubXZKQ1dNREdPMXdrM2ZN?= =?utf-8?Q?SLhE=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a031b77a-73b8-457a-e55d-08dbd62d6342 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 14:11:03.1625 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ydF8ehT3eC6RBbCwIso60wwcJFxkq22jDziCYwvSYybSFUSBSwxZHIYUUxTQTy9g X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6638 On Thu, Oct 26, 2023 at 03:31:41PM +0800, Michael Shavit wrote: > On Wed, Oct 25, 2023 at 7:46 AM Jason Gunthorpe wrote: > > > > On Tue, Oct 24, 2023 at 02:34:28PM +0800, Michael Shavit wrote: > > > On Thu, Oct 12, 2023 at 7:26 AM Jason Gunthorpe wrote: > > > > [...] > > > > -static void arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) > > > > +static struct arm_smmu_ctx_desc * > > > > +arm_smmu_mmu_notifier_put(struct arm_smmu_mmu_notifier *smmu_mn) > > > > { > > > > struct mm_struct *mm = smmu_mn->mn.mm; > > > > struct arm_smmu_ctx_desc *cd = smmu_mn->cd; > > > > struct arm_smmu_domain *smmu_domain = smmu_mn->domain; > > > > - struct arm_smmu_master *master; > > > > - unsigned long flags; > > > > > > > > if (!refcount_dec_and_test(&smmu_mn->refs)) > > > > - return; > > > > + return cd; > > > > > > > > list_del(&smmu_mn->list); > > > > > > > > - spin_lock_irqsave(&smmu_domain->devices_lock, flags); > > > > - list_for_each_entry(master, &smmu_domain->devices, domain_head) > > > > - arm_smmu_clear_cd(master, mm->pasid); > > > > - spin_unlock_irqrestore(&smmu_domain->devices_lock, flags); > > > > - > > > > /* > > > > * If we went through clear(), we've already invalidated, and no > > > > * new TLB entry can have been formed. > > > > > > This re-orders the TLB invalidation before the CD entry is cleared. > > > Couldn't a misbehaving device form TLB entries in this time interval > > > that we'd want to avoid? > > > > Hum.. No for the 'inv_asid', but yes for the 'atc_inv_domain' > > Just to confirm, why "No for the 'inv_asid'"? My best guess: > 1. Transactions don't hit the TLB entries unless there's a valid CD > configured with that ASID > 2. You're relying on those TLB entries being cleared elsewhere in the > code, when freeing/reclaiming the ASID from the domain. Ahhh I was too thoughtless to say that! Yes, we need to have the CD removed before doing the IOTLB invalidation too because we are trying to clear the IOTLB of that ASID so the ASID is clean before going back to the allocator. I got a bit confused because later on in the series that specific invalidation is moved into arm_smmu_domain_free_id() and moved over to domain deallocation, but at this point we don't have that yet so it is still doing something important. Regardless it is fixed! > But this also makes me curious why we bother with an ASID invalidation > in the first place if it's not required for correctness. ASIDs put back into the xarray for allocation must be clean in the IOTLB as we don't do an invalidation when we allocate an unused ASID from the xarray. An ASID that is not clean could have stale translations that are not valid. See patch 22 Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49352C25B48 for ; Thu, 26 Oct 2023 14:11:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=JCVIhIyvAi4LnRjkjhkeSvK7DUHBDjbMiCgbxUyawSw=; b=quHkuJ0WSKtVKG z94eSnQkxZuBeo+ba6pfdNUoCml8VJ7VbdMSJUU8phRbz5UKAPbsnxc7+2b40KZbqex+kWvXHWXTg 8DmFr5if89lv0UP835X09yzPoZgcJTErbabl0rJV1F4woj8224ncd8u145+Lwpg/6/UhskGPJwYx8 pbtTJ2H//p6YxUK1vIPglShSn5sAAa89tJIaPJE6um9ET9U0KbPh1KdPBcL5iKDwteCgCgD6GAy3n TgZlDLJfJiNwI131UcDxt5dOSyxClUBt/dTOvQUkF2my8GzfdHxryOggN46Ph7UDlvZoxNVbvC/qK z35KwSwFph/VXS8tPY/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qw14p-00EdPT-2x; Thu, 26 Oct 2023 14:11:11 +0000 Received: from mail-bn7nam10on2060a.outbound.protection.outlook.com ([2a01:111:f400:7e8a::60a] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qw14n-00EdOj-0d for linux-arm-kernel@lists.infradead.org; Thu, 26 Oct 2023 14:11:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=NA7sX5sU/Qebr76QBYToOc9rMbgCdVh7b0O8reIajwRt+R8rrHFEszlpRzinxlPIPqb4S0ybePp+DwjyL3Kn4/y8vBOvznymZwGd5AWvMq6J2ti/B+rUYJARi6fQlGtOg/46DZWxBhfb0eNj5AqtgqL7XMnyTjkM/wduPbmHUmKJU/kci/ACdcmUgvATiCWbhb8qRz6GkLdVkuEnV+DjUr4kHxieM0kKWHDLqwsvXTeSxNHo4GkyhW7yisJNlfX31x/eCBUjBHgctPvEF+MJjPh2L7pa1mFyrA+bx2j7SN6PtRzz/6DjRd1ePgyvqVS7lG1tu3lSisXwbSr8eULgcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ihWbLqG/tz8dOo8xB5kn0PVmFNtLG3vbNmHxB6OaZ4c=; b=TJTmRW70kLbDOvISOC8Q7pkkchISuAn01O1Nj8VmkWFbXoFIiJh/tOk9reaUIUcChQf+vVtBAqKNCYQCivn4o1pqV2oKFw0XlUjOr9ZymjWr+M2lh/MvHX4N6UMozUqDIDvwL5M2qCdAqugP0qtHH4EkTTPAGFFUG88EtijRzW/WRET5ORcDDwNwku0KnPAGClJdLgx5d6Rw/vzo4KKn0jZWmXQBRKkrFdsz5Eelv+258qJyswwFTX3alABVDWEzfjEM898clYAQZhG6MB7fzNqCG8P+O8nfQU7pFLAIPpLzyp9lWLbetUkZ37vIsc9Fnkj+vftUpC6fmue/oO5hCA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ihWbLqG/tz8dOo8xB5kn0PVmFNtLG3vbNmHxB6OaZ4c=; b=P5JFE/ozIXvfy3HLOQk3iJ6fxDOm9LDC0euLu5bRgd40aM0Q5AYppt1XtVuB0WcOXZ/TC20q5s9CCgbH8Sy05mMxKhzjHCgVa+DF45RiJLAejz7JU/I7qpgLCFngqwpvxn6TXj43BAuFz0/JBSHqAAaQ31gjVlfGhnY/M91b5+GLy5tNUooy9tA9lc3QWFcZUsCaXJFc8vXYmhg2QVGKk1QiO/L/gSmvdDDYg0JHWEGQF/QvNofPp8KwQ/ed6rviDCJe9P/G8Uk5XNHh2kPKAdUgLne2XrQoOpzximd87ngnLSoWX966/ZmuKVowzRZOVr+AiWIy0rlzSAk+GxmbYg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by DM4PR12MB6638.namprd12.prod.outlook.com (2603:10b6:8:b5::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6907.33; Thu, 26 Oct 2023 14:11:03 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::11a1:b0c7:7c88:9480]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::11a1:b0c7:7c88:9480%4]) with mapi id 15.20.6933.022; Thu, 26 Oct 2023 14:11:03 +0000 Date: Thu, 26 Oct 2023 11:11:01 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, Joerg Roedel , linux-arm-kernel@lists.infradead.org, Robin Murphy , Will Deacon , Jean-Philippe Brucker , Nicolin Chen Subject: Re: [PATCH 11/27] iommu/arm-smmu-v3: Lift CD programming out of the SVA notifier code Message-ID: <20231026141101.GO3952@nvidia.com> References: <0-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com> <11-v1-afbb86647bbd+5-smmuv3_newapi_p2_jgg@nvidia.com> <20231024234608.GD911568@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: SA9PR13CA0078.namprd13.prod.outlook.com (2603:10b6:806:23::23) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|DM4PR12MB6638:EE_ X-MS-Office365-Filtering-Correlation-Id: a031b77a-73b8-457a-e55d-08dbd62d6342 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IDpS4higXByi1FS0CU7+ouIyzvyKkV5eulaD6000DPChBleYd+qutV96eJg4WVcwb8JL36xWwNwfHxAGv6EhX5i5DWILppZjZT1v6/HR3BhzlkAWijkZIcBqBsnKU1jTiI3qHxLYhEOg1l5SuCYDb9Sz8rQJp+7nYOWVOh60IDTED2t72u4bc+hHqBD/hWFSBclniS1gbtCWruzjuu6sJL23PveQ/Ti2LnQ85h8tc+w6zIyPsOhpRCOWj1+UdO/XRZRQ9JJERCBjxWfR9HLp1qk54e7wJYJ3zRRE5baErGHwJk7TCngckXEha4ABD1I19s8MQwLYNiyBn6MFQziVqdLUHcDN2vj6NUmWpJhvvKhA57SHu5PiQot6y9YNLtm6r5BhWkGLccJmM14mSr/T2krp/VmKPUOqG2SY5Z+DvliP07OGPlkFQF9vNtD62YVMAGtkEi1jHrLHPGId80jO5KjSK5VRCswmSNdH4IggLnSQ3dvMegbF1qaxQRIVKVzA7rrIlZYOpVjJH0IQlazqPj3OpRUj7fMee1x0yKdB5HssUeRpBXy+oSsA8rFRPYtW X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(376002)(136003)(346002)(39860400002)(366004)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(66946007)(5660300002)(2906002)(41300700001)(316002)(6916009)(4326008)(54906003)(66476007)(66556008)(53546011)(478600001)(6486002)(8936002)(8676002)(6506007)(107886003)(66899024)(2616005)(1076003)(26005)(6512007)(83380400001)(86362001)(38100700002)(33656002)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?M2VuSmg1ejFaVm5aMmRlSVFIRVpxcUpFLzdKUTU2UnhtditCVmk4eHdWWkNt?= =?utf-8?B?L0p6WXEvUy9yek5NZzlJSlk0bHA2VmRrRDNGRkV6cDcrNHRKejFDcURCTWJx?= =?utf-8?B?NU1zaEJ1ZnpyNldadmQ0RjNlcEtBUWhwY2w4TC9zQWZ4czYzdWhVUWRRTUdj?= =?utf-8?B?d0NZbFFmN2pmYVhKSyswdFgyVVd1SldaQ01va3B5cnA2Mko3WEIxRldPa0JD?= =?utf-8?B?VlByUHFvY1hxM1I5c0dHbjRQTnk4NkgxOFNCN1JRcEJ6cStuUEZOZkxYNTJR?= =?utf-8?B?emx3MTVYeFEwTkRGYnRKd0xEZTBERDAwUlhiMGJzUXpmdlBEZE5VNXJSM0Zp?= =?utf-8?B?NndvdmtZWEZaSEFLQ0wvQkd1N2Q4cG9FTzZmTFk1blFqK3c1NU5oRmZ0Q1li?= =?utf-8?B?WC9EcG5HR3NWS1NQNlhubm9GNmxaU3lIK2NLVWhxTmQ5QVNvTWlqU0FXTWlV?= =?utf-8?B?eVd1Z3p4WkhicUtxeVNLNHZlQUNOdUpUTU94Y05rYzQvbDlMZkhFWWd5bkZL?= =?utf-8?B?c29rblc2a29DcEVaQTF6YXluT2tBQ3IzU0Qya0VldndpbXlxL1Y3b09CSktL?= =?utf-8?B?U0FSK2YrdU9ZM3dra1hBRmVkaVBvL2NJMTNBWmh1cDF0Z09UbHpHemZidzQy?= =?utf-8?B?dGlYbzhDeEZBbWcyT2pISlM3bmUrTC9MNkNKaXU0NmlGaURXTytJU2piaXpY?= =?utf-8?B?YzhvOFNxdVB3UlNyMVpPQjVLZVNEUVJWWkFEOXFxVnpZcEtyL3pEaGlwZjVz?= =?utf-8?B?ZjBVNWh1RlNhMmFYQWl5TmtnTkRlNFRCNnpMQzJvYWt1OElrZzh1S0xPQW5k?= =?utf-8?B?ODhPUlNtWEVuU1VCMUxqTEM5S0kwTkFPV0M4RjlZOUFxVWdSZzdocmZMVUdL?= =?utf-8?B?bjl3UXVxL2UxWDFoU2NtTUwrWEFjMFZpWkJNaGcxVmE5UHBFQVQ1d3pkdzFX?= =?utf-8?B?RWNjejMxUDFRN21SUmI1bHoxcjFKUUNPM3llQzU3aTc0eFYrNk9wTEFuMFlY?= =?utf-8?B?UnhmYVdqVUxyWUtCc3NBMVlsU0lTZlZQVXpXZFVQMCtZRnZOblNFcWk3RmZM?= =?utf-8?B?RnhjUEdOOFZ5ZU92T3dFOXdlQUk3VFV3UnJKWXg0cGdTTlNzSWNLbTJxV3d1?= =?utf-8?B?YWtpdHdiVE1xbUIxZUhvQjNqSE14NWkweGhKZDhnZmR0bUtXakhka3Y2aVRh?= =?utf-8?B?YVZqL1o1N2xFZmFBUEdtNUdPamd2c09OOUlQUmZxQm9VWENMaUQva0VHeGZE?= =?utf-8?B?Z3l3YTZBY1FwUU54cmY3LzNWOHZMZDBxY0RDRmF4THNQWlVXK05kNDNBNk5h?= =?utf-8?B?QmNhU0JsS1U4dHhuZGlxU0IvZ2s5Rm5xcC9PMkZaK2VuMGxLcWkvZHZXTm5H?= =?utf-8?B?bUtSUHl3NEZwSnp0RWN1UjduckYrWVo2cGhLYWJaQ2R3RVcxRXNNRVpnQlQr?= =?utf-8?B?YWdTSjdiaVllMzlJWG1LcVVMSkRrS1BZRGNLOGdTUDFVYSs3Wks1STBTb1RP?= =?utf-8?B?dU8vekVKNFk0MkREK1BCalVCdVN2VTRJak0yQU9nb1lXQXlWOEhmVnJvYTJn?= =?utf-8?B?K1NJRGEvUTBHRWpnZjhYNVVBTGV1Nm5PcDkxWVJTTEFrVEZBbkMxMzAwdy9N?= =?utf-8?B?dEk5eER3OGtSN29wTERoYUZzaW5KNEZIYkFzNHB3dVRNSis2ZnMvWEI2RXI2?= =?utf-8?B?SlBHQlB2MmxkVjZOQmJJRXY0NHpMZUJkbVdiNlkvQ3M3cTRmL3F6b3NmVlI1?= =?utf-8?B?Yy85b21LNjJyNmc4OHUvUmxBMkRKMDAwREtSWU9uQkZRSXhac0VuK2lZTXlR?= =?utf-8?B?b1FOMlBlakRYa1RGaVV2VTNxcUR3N0NpWWltWE1nT0UyRkpzYjFOMHFDUWxx?= =?utf-8?B?YlRFUXBVTmJCQ2pZazIwbVNjL0tZVUVUQ2RlUXRlWFRJMmN1L3BnQnhkc2Ry?= =?utf-8?B?YWt5M0J5NWpIcEYrNnJ5YU10VlM1K1o5bjVFS2xJTnR0WDhHZHRubVlwWWtw?= =?utf-8?B?K0Nmd2V5OHlMSUNNZjZuOFdneGNhQXZXejNPOVVaVGZ5TUlSYUxxOEcrWm5K?= =?utf-8?B?TjY1TFhOZmlxS0Y0VnhrN2R1emIwSjJlSVVjNGxubXZKQ1dNREdPMXdrM2ZN?= =?utf-8?Q?SLhE=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a031b77a-73b8-457a-e55d-08dbd62d6342 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 26 Oct 2023 14:11:03.1625 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ydF8ehT3eC6RBbCwIso60wwcJFxkq22jDziCYwvSYybSFUSBSwxZHIYUUxTQTy9g X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB6638 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231026_071109_256757_4FEDBFAD X-CRM114-Status: GOOD ( 26.82 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVGh1LCBPY3QgMjYsIDIwMjMgYXQgMDM6MzE6NDFQTSArMDgwMCwgTWljaGFlbCBTaGF2aXQg d3JvdGU6Cj4gT24gV2VkLCBPY3QgMjUsIDIwMjMgYXQgNzo0NuKAr0FNIEphc29uIEd1bnRob3Jw ZSA8amdnQG52aWRpYS5jb20+IHdyb3RlOgo+ID4KPiA+IE9uIFR1ZSwgT2N0IDI0LCAyMDIzIGF0 IDAyOjM0OjI4UE0gKzA4MDAsIE1pY2hhZWwgU2hhdml0IHdyb3RlOgo+ID4gPiBPbiBUaHUsIE9j dCAxMiwgMjAyMyBhdCA3OjI24oCvQU0gSmFzb24gR3VudGhvcnBlIDxqZ2dAbnZpZGlhLmNvbT4g d3JvdGU6Cj4gPiA+ID4gWy4uLl0KPiA+ID4gPiAtc3RhdGljIHZvaWQgYXJtX3NtbXVfbW11X25v dGlmaWVyX3B1dChzdHJ1Y3QgYXJtX3NtbXVfbW11X25vdGlmaWVyICpzbW11X21uKQo+ID4gPiA+ ICtzdGF0aWMgc3RydWN0IGFybV9zbW11X2N0eF9kZXNjICoKPiA+ID4gPiArYXJtX3NtbXVfbW11 X25vdGlmaWVyX3B1dChzdHJ1Y3QgYXJtX3NtbXVfbW11X25vdGlmaWVyICpzbW11X21uKQo+ID4g PiA+ICB7Cj4gPiA+ID4gICAgICAgICBzdHJ1Y3QgbW1fc3RydWN0ICptbSA9IHNtbXVfbW4tPm1u Lm1tOwo+ID4gPiA+ICAgICAgICAgc3RydWN0IGFybV9zbW11X2N0eF9kZXNjICpjZCA9IHNtbXVf bW4tPmNkOwo+ID4gPiA+ICAgICAgICAgc3RydWN0IGFybV9zbW11X2RvbWFpbiAqc21tdV9kb21h aW4gPSBzbW11X21uLT5kb21haW47Cj4gPiA+ID4gLSAgICAgICBzdHJ1Y3QgYXJtX3NtbXVfbWFz dGVyICptYXN0ZXI7Cj4gPiA+ID4gLSAgICAgICB1bnNpZ25lZCBsb25nIGZsYWdzOwo+ID4gPiA+ Cj4gPiA+ID4gICAgICAgICBpZiAoIXJlZmNvdW50X2RlY19hbmRfdGVzdCgmc21tdV9tbi0+cmVm cykpCj4gPiA+ID4gLSAgICAgICAgICAgICAgIHJldHVybjsKPiA+ID4gPiArICAgICAgICAgICAg ICAgcmV0dXJuIGNkOwo+ID4gPiA+Cj4gPiA+ID4gICAgICAgICBsaXN0X2RlbCgmc21tdV9tbi0+ bGlzdCk7Cj4gPiA+ID4KPiA+ID4gPiAtICAgICAgIHNwaW5fbG9ja19pcnFzYXZlKCZzbW11X2Rv bWFpbi0+ZGV2aWNlc19sb2NrLCBmbGFncyk7Cj4gPiA+ID4gLSAgICAgICBsaXN0X2Zvcl9lYWNo X2VudHJ5KG1hc3RlciwgJnNtbXVfZG9tYWluLT5kZXZpY2VzLCBkb21haW5faGVhZCkKPiA+ID4g PiAtICAgICAgICAgICAgICAgYXJtX3NtbXVfY2xlYXJfY2QobWFzdGVyLCBtbS0+cGFzaWQpOwo+ ID4gPiA+IC0gICAgICAgc3Bpbl91bmxvY2tfaXJxcmVzdG9yZSgmc21tdV9kb21haW4tPmRldmlj ZXNfbG9jaywgZmxhZ3MpOwo+ID4gPiA+IC0KPiA+ID4gPiAgICAgICAgIC8qCj4gPiA+ID4gICAg ICAgICAgKiBJZiB3ZSB3ZW50IHRocm91Z2ggY2xlYXIoKSwgd2UndmUgYWxyZWFkeSBpbnZhbGlk YXRlZCwgYW5kIG5vCj4gPiA+ID4gICAgICAgICAgKiBuZXcgVExCIGVudHJ5IGNhbiBoYXZlIGJl ZW4gZm9ybWVkLgo+ID4gPgo+ID4gPiBUaGlzIHJlLW9yZGVycyB0aGUgVExCIGludmFsaWRhdGlv biBiZWZvcmUgdGhlIENEIGVudHJ5IGlzIGNsZWFyZWQuCj4gPiA+IENvdWxkbid0IGEgbWlzYmVo YXZpbmcgZGV2aWNlIGZvcm0gVExCIGVudHJpZXMgaW4gdGhpcyB0aW1lIGludGVydmFsCj4gPiA+ IHRoYXQgd2UnZCB3YW50IHRvIGF2b2lkPwo+ID4KPiA+IEh1bS4uIE5vIGZvciB0aGUgJ2ludl9h c2lkJywgYnV0IHllcyBmb3IgdGhlICdhdGNfaW52X2RvbWFpbicKPiAKPiBKdXN0IHRvIGNvbmZp cm0sIHdoeSAiTm8gZm9yIHRoZSAnaW52X2FzaWQnIj8gTXkgYmVzdCBndWVzczoKPiAxLiBUcmFu c2FjdGlvbnMgZG9uJ3QgaGl0IHRoZSBUTEIgZW50cmllcyB1bmxlc3MgdGhlcmUncyBhIHZhbGlk IENECj4gY29uZmlndXJlZCB3aXRoIHRoYXQgQVNJRAo+IDIuIFlvdSdyZSByZWx5aW5nIG9uIHRo b3NlIFRMQiBlbnRyaWVzIGJlaW5nIGNsZWFyZWQgZWxzZXdoZXJlIGluIHRoZQo+IGNvZGUsIHdo ZW4gZnJlZWluZy9yZWNsYWltaW5nIHRoZSBBU0lEIGZyb20gdGhlIGRvbWFpbi4KCkFoaGggSSB3 YXMgdG9vIHRob3VnaHRsZXNzIHRvIHNheSB0aGF0IQoKWWVzLCB3ZSBuZWVkIHRvIGhhdmUgdGhl IENEIHJlbW92ZWQgYmVmb3JlIGRvaW5nIHRoZSBJT1RMQgppbnZhbGlkYXRpb24gdG9vIGJlY2F1 c2Ugd2UgYXJlIHRyeWluZyB0byBjbGVhciB0aGUgSU9UTEIgb2YgdGhhdCBBU0lECnNvIHRoZSBB U0lEIGlzIGNsZWFuIGJlZm9yZSBnb2luZyBiYWNrIHRvIHRoZSBhbGxvY2F0b3IuCgpJIGdvdCBh IGJpdCBjb25mdXNlZCBiZWNhdXNlIGxhdGVyIG9uIGluIHRoZSBzZXJpZXMgdGhhdCBzcGVjaWZp YwppbnZhbGlkYXRpb24gaXMgbW92ZWQgaW50byBhcm1fc21tdV9kb21haW5fZnJlZV9pZCgpIGFu ZCBtb3ZlZCBvdmVyIHRvCmRvbWFpbiBkZWFsbG9jYXRpb24sIGJ1dCBhdCB0aGlzIHBvaW50IHdl IGRvbid0IGhhdmUgdGhhdCB5ZXQgc28gaXQgaXMKc3RpbGwgZG9pbmcgc29tZXRoaW5nIGltcG9y dGFudC4gUmVnYXJkbGVzcyBpdCBpcyBmaXhlZCEKCj4gQnV0IHRoaXMgYWxzbyBtYWtlcyBtZSBj dXJpb3VzIHdoeSB3ZSBib3RoZXIgd2l0aCBhbiBBU0lEIGludmFsaWRhdGlvbgo+IGluIHRoZSBm aXJzdCBwbGFjZSBpZiBpdCdzIG5vdCByZXF1aXJlZCBmb3IgY29ycmVjdG5lc3MuCgpBU0lEcyBw dXQgYmFjayBpbnRvIHRoZSB4YXJyYXkgZm9yIGFsbG9jYXRpb24gbXVzdCBiZSBjbGVhbiBpbiB0 aGUKSU9UTEIgYXMgd2UgZG9uJ3QgZG8gYW4gaW52YWxpZGF0aW9uIHdoZW4gd2UgYWxsb2NhdGUg YW4gdW51c2VkIEFTSUQKZnJvbSB0aGUgeGFycmF5LiBBbiBBU0lEIHRoYXQgaXMgbm90IGNsZWFu IGNvdWxkIGhhdmUgc3RhbGUKdHJhbnNsYXRpb25zIHRoYXQgYXJlIG5vdCB2YWxpZC4gU2VlIHBh dGNoIDIyCgpKYXNvbgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0 cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGlu Zm8vbGludXgtYXJtLWtlcm5lbAo=