From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2071.outbound.protection.outlook.com [40.107.93.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 153CB6FBE for ; Mon, 21 Aug 2023 13:50:13 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bbVIDBRF1fq/l/VlxYpBHAC1wRNCreRBFNATBC1+l2IOTnZlMvLJCFE4hj3RSpp+cEXlPy3LB5PYRnXylrzNkuC35UKshBlljyUL/Nj9tcR99s5LmSkwgLiU/9meuSMSe+38gw57yHaXh5Xj8WRI0OoR1xo1dOaejH55UJflIFTyL3AlTBTeBjl/bsHXT+VEwRsWrl618KvXzgEP6o8hIli+sGAQhN5FwTZzymVzf2D45Mf2u3AQjhpJC3B3OA/klU8+jaXz34KZpFI2NJ5LhIw71NWnBlNQtajMzbla66Kr3OfwcAvWCNxAf6N/Nnv/cadGqXneu3Siet2R2E+kPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PiTFDJUb/Oiur5ylRf7zJNhdYPXmsI7JvGU8NB3RvPw=; b=JwyPKXvqCmoGsrfjIsiYneH9BEi6lsvF1uw742gXsUv8iB+NMlgP31uWsrNo9ywn+7URiBuuTCf6rDWGIKAShqgZxbK14qcP+Xh1UtyCJgZHFOKmO+600Op7OCaE0j01cjFwS9mw7Ma6QAM8m/gBSGN5xSkswG4z62D7NAUTBcMKkakTBr2UGXmpwSiiXECNtJwdqqCeW+J0ciCOzyiFT1kID8ENzmPPuU7/PopRVROwXgywatf5XpzQ1th6N6HnkAafQbU5d3sJH3yKcDIARhmj4Ngh1QvMJRKlTMFEy5cQtu73xVenMFLdbThWU/2OXWmSDso0d+IfQ2TXdpZ+Aw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PiTFDJUb/Oiur5ylRf7zJNhdYPXmsI7JvGU8NB3RvPw=; b=OD6nQmneL2BRSUepF8kS7dwTrb+1Q4ercnAgO2I0y0sj/T383EZbKAEv9vxAPz19cJ0Z0czGMg87Le296SExeQq1lBel9gbXPF6dFApaAa5Kv6vYrZJ5I0Trn0YyfTZzGVxm/QOxSMPWBLBm7mf6jNLor/lezj3QvFrIx6KJ+Y6Ym6+lW3M1nJf7IrU7iW2kqlq9TtHGGM9J/zVqSwBe9+tQkUScGhcZ9yqihdFOiBsOlyyFczW4QAYKiQICZh3NpQ4uuJ89Nrd2TzqLhZAviNnVDC80NMaaR5pr6gfz3U+MEvb5nrZAyNeZeGK+zPGohyVZ+vDehC6PDqlaL8/2SQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by LV3PR12MB9353.namprd12.prod.outlook.com (2603:10b6:408:21b::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.31; Mon, 21 Aug 2023 13:50:11 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::5111:16e8:5afe:1da1]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::5111:16e8:5afe:1da1%6]) with mapi id 15.20.6699.020; Mon, 21 Aug 2023 13:50:11 +0000 Date: Mon, 21 Aug 2023 10:50:09 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, nicolinc@nvidia.com, tina.zhang@intel.com, jean-philippe@linaro.org, robin.murphy@arm.com Subject: Re: [RFC PATCH v1 3/8] iommu/arm-smmu-v3-sva: Allocate new ASID from installed_smmus Message-ID: References: <20230817182055.1770180-1-mshavit@google.com> <20230818021629.RFC.v1.3.I326c62dc062aed8d901d319aa665dbe983c7904c@changeid> Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-ClientProxiedBy: MN2PR20CA0054.namprd20.prod.outlook.com (2603:10b6:208:235::23) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|LV3PR12MB9353:EE_ X-MS-Office365-Filtering-Correlation-Id: fb39326f-ce08-4361-351a-08dba24d896e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sTXuFwIRq4PxCBTJbmzsbOPyv87ABUD6yVVbcVg7QJCQYlbK0w2ymzljL8nnbOA1IqddF8JodIcVNGpi7kbFX5WYZiRLjEoZeGVxNQ9xQh4ootdFMSeDENntTUQUGj1/4stas/MU8MsyeTUk7DxMpJabSScF8yTCMS+kSrm0NyzMvInXmNLHdYy6YP7c/x4ypxfAjZI1X0Q9Oa/wqKvgnJdm969NGLucP59zDdIQEicVUWMzfTanRDSEdczCRSlRc/LjqbEEIEash20TEoPXUBxyeYmzXLuBYGbxJb0GBmSAn9Uhq7jlSs6S8PkWPhFCrHTSKyM0DlptpM8rfcnWNiGzKxJNGyR1rFQUy3h/t1rnW0xGf5+tX+sQryh8F/lRLVEMxY87DCMOjR4rmx7GCF9319f8NqitYm+m6qM6Cu1dG4AhOqjc9Z08qnPVaNnhmzFAn7In/RjYElB8+Ew9XjtgVqYfIwhXIA+zIO6WxPylnMHCHHIQnsvNAjw1ls0eHjTRhnJF0YY6zxJT8gdcfWv1OKZhCVJ5+7egNbt42Oe6PqFot0cX7+of15yMCIwj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(39860400002)(346002)(376002)(366004)(1800799009)(451199024)(186009)(5660300002)(83380400001)(38100700002)(316002)(66476007)(66556008)(6916009)(478600001)(2906002)(41300700001)(8936002)(8676002)(4326008)(53546011)(6512007)(2616005)(66946007)(6506007)(26005)(6486002)(86362001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YVVsODRqcVV4MjZqei9pWFFxdU5Ocks1UWJraUV0WDYxV2daS2R3T21hTGxw?= =?utf-8?B?Q3N1YVkzUXdLaDArYmFlNjZJWDRrazYzWGxucFlEOSt4RzBnYVhHRUpyTjRy?= =?utf-8?B?YnNQTDVwRVFwNEdkaDl3ODVFSUR5NmFWbTRTUGxRVG9IakVCdWJubjZFdWov?= =?utf-8?B?cUR4MHlDbWM1VzhsWlM3eWEwVUtPMTdWREZWTzZuSXQ1aXFvVlpBSFF0cEJJ?= =?utf-8?B?enhrTExYalp0YUtwT3o2MVZGRVRqc09ibnRNVGttTlRzejdaMG03SHh6dW9N?= =?utf-8?B?L2pQcHZNbk5DcVdOV3RYblpPdlZONHA0RkRxL0pjMEo1czcyVzdYYVE3bEsr?= =?utf-8?B?N05SeG1IdmtEU0NhRkNacFBCTDRCRFNMMnpiQTRwajNEL2ovWlRaTnhTRi81?= =?utf-8?B?Qzl4TS9xZUMyV1hpczZBS2VqcHV0T0UvZjBTdEhKWmt2SFVhL25ldmtSajFN?= =?utf-8?B?RkpLakhMMHFHdU5jUnkwd1NrSFVRc1kzS0dEQjFWZnNPL3JJWWdlRzA0d0ZL?= =?utf-8?B?d1BXRE1VTHIvNHVmQ3RZaVhRWkZFbXVGRnhVcy8yVXZvbGZCeWdvUzQ0d1E4?= =?utf-8?B?Wmx1bVh4UU5ETVcxT09VUDhlUzFKVUt5T2dpdUFoM1hROVQrNjZXeUNUZ1JE?= =?utf-8?B?SDdqc1lDWkhnNWFBcmU0Zm9EY3BtVUFXc09vMkxNdjgzTHB4Z1Vsd3VoSlhC?= =?utf-8?B?a2I5M0ZMUHhwaUhTVldXTUhDYkluZlFXOHFCc0RYMmFYejh5Q3BJdFpCY25t?= =?utf-8?B?SzB3SkZHM2FBaGlBYVhSMDlZMXBPVytOeFh0bU9WM0pMczlIZmxHMVliRFgv?= =?utf-8?B?WEtaampMVm1ybEZRYzNGc2JGRzR5VHJHTnJxQnlWT1Zhei91dTlQYTJOT1BF?= =?utf-8?B?QXIvSm5ZRXM2K2VpMk1YZDRxRkNyMWtsQXZ0TlNjS0J2QWdtOTdHMVA0R3lw?= =?utf-8?B?T2dyb0p2ZTVpVG82OXVGTmRrZ0o2QW5qcEcweVlCNVFGa0tUUEhVOFNRSmkx?= =?utf-8?B?RDNVRFJCRExVdmRtNGVzcXE2MnNhWmg3N0pzTzNwZGtlK3hudXptWFNXK2s4?= =?utf-8?B?aUR1ekhhWGV2K01PdmJyNGhMNjMzdnVHdXRSSzlPRHFjRW05bHVoVFU4dFFU?= =?utf-8?B?dTNqSUtzOFhkb2JRY3VZeHZ3ZVJFemhKYmY5NWVIL1FNYVdnME9kY0N2TElT?= =?utf-8?B?SlRlZzM5ZzJ1VHdPOWVGSzFMWkRNNDhXTzg4M3JiMDA3ME8vT1Z2amtrMTUr?= =?utf-8?B?SVVuSHNnNlZsMWdWUnFvbVFqdmhmYkIraWlZYVdGRjBOQk5HOEpZWExncXI5?= =?utf-8?B?bHN0ZFpTaXdPVTVnVlB6aU1yV0tOcXlFVXUrOWlaQ3d3dmRqd1FyZmR3dE84?= =?utf-8?B?em1RNThUZUc2dVNQYUNyek9pQmk2UzZkZ0I0VUZmMUE3TUd2L0V5VjFZcWNL?= =?utf-8?B?Ti9Tbm1OdmIrdGw5cHlGOE1yNzREK1hsMUpjbTFHeVNoV09ERklyUWV4VWI5?= =?utf-8?B?dUlvbXc5QzZVV3l6eWFmLzdPSmJFSFFIeUxQZlNkd2pXYmNhd21KbjNwWnhj?= =?utf-8?B?d2tIcGZDYmJOekNzWlNiSVVxeGo5VG9mZjFVMXZTajdIcDFWYW85RkEyQ1o0?= =?utf-8?B?VU4xU0NjYmt5UVRTWEdUZFhmVTRjK3ljOTdYUmpXK3JNaTlvSEpCa2N6cVBt?= =?utf-8?B?MEUzQUJiY092amFsVk96L1ROWEJXNkwxSW1NRVRNTERJVUN5bkNiTXpnVG1h?= =?utf-8?B?bXJnUlRkS0FremgweW5pVGNsTWsyRjZWUmlHRDNoTi8zd0N6Q2NOeVJnMTZB?= =?utf-8?B?eVRNWDFLMGdaeVdHYTF2czBHcGljSk5HOUpwWHhBQks1am10ZVdyTCtFWno2?= =?utf-8?B?ZjZhckt1bEZOOG1XdGxjd3dZanVQOXFxemI5dCtHRDBicTBFRE16ZEpwVmVI?= =?utf-8?B?ZmNUYTJnQU5NeThHMFFzcDFYTmFNUmN5STA1UHlMbzV2cGxmeW5oRHQvR3F5?= =?utf-8?B?amljZUN5WGtpRXd4MWF2RE04bTdJaWVuU3N0UU8yczJGRk40VFdzVjBRQ1Vw?= =?utf-8?B?TzhqVW1QZ2oyVHJaamVWOUprY1FVd0FRazNPbUJHVFlmTVppVzVUc2hxbmVr?= =?utf-8?Q?ius32ufL/VM/9PdU3o29hgeJE?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb39326f-ce08-4361-351a-08dba24d896e X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2023 13:50:11.1243 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KOu8BfkKPeKLVTSfF90vK8lDrRUovbFN27VvK2sf53PLLyGcTRAJUoF8vLn/2zoH X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9353 On Mon, Aug 21, 2023 at 09:38:40PM +0800, Michael Shavit wrote: > On Mon, Aug 21, 2023 at 7:54 PM Jason Gunthorpe wrote: > > > > On Mon, Aug 21, 2023 at 05:31:23PM +0800, Michael Shavit wrote: > > > On Fri, Aug 18, 2023 at 2:38 AM Jason Gunthorpe wrote: > > > > > > > > On Fri, Aug 18, 2023 at 02:16:25AM +0800, Michael Shavit wrote: > > > > > Pick an ASID that is within the supported range of all SMMUs that the > > > > > domain is installed to. > > > > > > > > > > Signed-off-by: Michael Shavit > > > > > --- > > > > > > > > This seems like a pretty niche scenario, maybe we should just keep a > > > > global for the max ASID? > > > > > > > > Otherwise we need a code to change the ASID, even for non-SVA domains, > > > > when the domain is installed in different devices if the current ASID > > > > is over the instance max.. > > > > > > This RFC took the other easy way out for this problem by rejecting > > > attaching a domain if its currently assigned ASID/VMID > > > is out of range when attaching to a new SMMU. But I'm not sure > > > which of the two options is the right trade-off. > > > Especially if we move VMID to a global allocator (which I plan to add > > > for v2), setting a global maximum for VMID of 256 sounds small. > > > > IMHO the simplest and best thing is to make both vmid and asid as > > local allocators. Then alot of these problems disappear > > Well that does sound like the most flexible, but IMO quite a lot more > complicated. > > I'll post a v2 RFC that removes the `iommu/arm-smmu-v3: Add list of > installed_smmus` patch and uses a flat master list in smmu_domain as > suggested by Robin, for comparison with the v1. But at a glance using a > local allocator would require: > 1. Keeping that patch so we can track the asid/vmid for a domain on a > per smmu instance You'd have to store the cache tag in the per-master struct on that list and take it out of the domain struct. Ie the list of attached masters contains the per-master cache tag instead of a global cache tag. The only place you need the cache tag is when iterating over the list of masters, so it is OK. If the list of masters is sorted by smmu then the first master of each smmu can be used to perform the cache tag invalidation, then the rest of the list is the ATC invalidation. The looping code will be a bit ugly. > 2. Keeping a map in the smmu struct so that arm_smmu_share_asid can > find any arm_smmu_installed_smmu that need to have their asid > updated Yes, the global xarray moves into the smmu > (on a loop over every smmu the domain in arm_smmu_mmu_notifier_get is > attached to, which just at a glance looks headache inducing because of > sva's piggybacking on the rid domain.) Not every smmu, just the one you are *currently* attaching to. We don't care if the *other* smmu's have different ASIDs, maybe they are not using BTM, or won't use SVA. We care that *our* smmu has the right ASID when we go to attach the domain. So the logic is not really any different, you already know the smmu because it is attaching, and you do the same locking xarray stuff as was done globally, just against this local smmu. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3AB1EE4996 for ; Mon, 21 Aug 2023 13:50:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=hok3R5Xg5cmO48+q5BmAq5lUB0kAMpktdjzb+pi0AMU=; b=fRjX65+9ZkbrU3 xUAdpctSXLbxjQVKyPR8IDpDRmh+SOLGc7IDBfT89TN9TEsjhIjU857JOGLE/w138KVw+K1m3WQOG 9o74IWJtSYYn9pr8qU+VgFlyLqRJUQ1+YP7SQC016lsk7VX7Vh2RuJwiHGXUGREUdWzmuoEGJ3zpw oZtPqEG5/HpzrpD82IMBNdy4L0oe+6Bq+p0VFsl1a2hCkj5VM5/028L46rcEcNojqy2/RQ0LL+Khy gTUWpwmv6dItJXvGeoYec9R183fiDf9v00EpZN7WlS5s5XLRg3h59ANi/N7r0ezk0u4dFWphTW/p+ PDEw5i4uRefp4aBAebSg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qY5IW-00E74y-01; Mon, 21 Aug 2023 13:50:24 +0000 Received: from mail-dm6nam10on20626.outbound.protection.outlook.com ([2a01:111:f400:7e88::626] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qY5IQ-00E73v-0u for linux-arm-kernel@lists.infradead.org; Mon, 21 Aug 2023 13:50:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bbVIDBRF1fq/l/VlxYpBHAC1wRNCreRBFNATBC1+l2IOTnZlMvLJCFE4hj3RSpp+cEXlPy3LB5PYRnXylrzNkuC35UKshBlljyUL/Nj9tcR99s5LmSkwgLiU/9meuSMSe+38gw57yHaXh5Xj8WRI0OoR1xo1dOaejH55UJflIFTyL3AlTBTeBjl/bsHXT+VEwRsWrl618KvXzgEP6o8hIli+sGAQhN5FwTZzymVzf2D45Mf2u3AQjhpJC3B3OA/klU8+jaXz34KZpFI2NJ5LhIw71NWnBlNQtajMzbla66Kr3OfwcAvWCNxAf6N/Nnv/cadGqXneu3Siet2R2E+kPw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PiTFDJUb/Oiur5ylRf7zJNhdYPXmsI7JvGU8NB3RvPw=; b=JwyPKXvqCmoGsrfjIsiYneH9BEi6lsvF1uw742gXsUv8iB+NMlgP31uWsrNo9ywn+7URiBuuTCf6rDWGIKAShqgZxbK14qcP+Xh1UtyCJgZHFOKmO+600Op7OCaE0j01cjFwS9mw7Ma6QAM8m/gBSGN5xSkswG4z62D7NAUTBcMKkakTBr2UGXmpwSiiXECNtJwdqqCeW+J0ciCOzyiFT1kID8ENzmPPuU7/PopRVROwXgywatf5XpzQ1th6N6HnkAafQbU5d3sJH3yKcDIARhmj4Ngh1QvMJRKlTMFEy5cQtu73xVenMFLdbThWU/2OXWmSDso0d+IfQ2TXdpZ+Aw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PiTFDJUb/Oiur5ylRf7zJNhdYPXmsI7JvGU8NB3RvPw=; b=OD6nQmneL2BRSUepF8kS7dwTrb+1Q4ercnAgO2I0y0sj/T383EZbKAEv9vxAPz19cJ0Z0czGMg87Le296SExeQq1lBel9gbXPF6dFApaAa5Kv6vYrZJ5I0Trn0YyfTZzGVxm/QOxSMPWBLBm7mf6jNLor/lezj3QvFrIx6KJ+Y6Ym6+lW3M1nJf7IrU7iW2kqlq9TtHGGM9J/zVqSwBe9+tQkUScGhcZ9yqihdFOiBsOlyyFczW4QAYKiQICZh3NpQ4uuJ89Nrd2TzqLhZAviNnVDC80NMaaR5pr6gfz3U+MEvb5nrZAyNeZeGK+zPGohyVZ+vDehC6PDqlaL8/2SQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by LV3PR12MB9353.namprd12.prod.outlook.com (2603:10b6:408:21b::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6678.31; Mon, 21 Aug 2023 13:50:11 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::5111:16e8:5afe:1da1]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::5111:16e8:5afe:1da1%6]) with mapi id 15.20.6699.020; Mon, 21 Aug 2023 13:50:11 +0000 Date: Mon, 21 Aug 2023 10:50:09 -0300 From: Jason Gunthorpe To: Michael Shavit Cc: iommu@lists.linux.dev, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, will@kernel.org, nicolinc@nvidia.com, tina.zhang@intel.com, jean-philippe@linaro.org, robin.murphy@arm.com Subject: Re: [RFC PATCH v1 3/8] iommu/arm-smmu-v3-sva: Allocate new ASID from installed_smmus Message-ID: References: <20230817182055.1770180-1-mshavit@google.com> <20230818021629.RFC.v1.3.I326c62dc062aed8d901d319aa665dbe983c7904c@changeid> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MN2PR20CA0054.namprd20.prod.outlook.com (2603:10b6:208:235::23) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|LV3PR12MB9353:EE_ X-MS-Office365-Filtering-Correlation-Id: fb39326f-ce08-4361-351a-08dba24d896e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: sTXuFwIRq4PxCBTJbmzsbOPyv87ABUD6yVVbcVg7QJCQYlbK0w2ymzljL8nnbOA1IqddF8JodIcVNGpi7kbFX5WYZiRLjEoZeGVxNQ9xQh4ootdFMSeDENntTUQUGj1/4stas/MU8MsyeTUk7DxMpJabSScF8yTCMS+kSrm0NyzMvInXmNLHdYy6YP7c/x4ypxfAjZI1X0Q9Oa/wqKvgnJdm969NGLucP59zDdIQEicVUWMzfTanRDSEdczCRSlRc/LjqbEEIEash20TEoPXUBxyeYmzXLuBYGbxJb0GBmSAn9Uhq7jlSs6S8PkWPhFCrHTSKyM0DlptpM8rfcnWNiGzKxJNGyR1rFQUy3h/t1rnW0xGf5+tX+sQryh8F/lRLVEMxY87DCMOjR4rmx7GCF9319f8NqitYm+m6qM6Cu1dG4AhOqjc9Z08qnPVaNnhmzFAn7In/RjYElB8+Ew9XjtgVqYfIwhXIA+zIO6WxPylnMHCHHIQnsvNAjw1ls0eHjTRhnJF0YY6zxJT8gdcfWv1OKZhCVJ5+7egNbt42Oe6PqFot0cX7+of15yMCIwj X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(396003)(39860400002)(346002)(376002)(366004)(1800799009)(451199024)(186009)(5660300002)(83380400001)(38100700002)(316002)(66476007)(66556008)(6916009)(478600001)(2906002)(41300700001)(8936002)(8676002)(4326008)(53546011)(6512007)(2616005)(66946007)(6506007)(26005)(6486002)(86362001)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?YVVsODRqcVV4MjZqei9pWFFxdU5Ocks1UWJraUV0WDYxV2daS2R3T21hTGxw?= =?utf-8?B?Q3N1YVkzUXdLaDArYmFlNjZJWDRrazYzWGxucFlEOSt4RzBnYVhHRUpyTjRy?= =?utf-8?B?YnNQTDVwRVFwNEdkaDl3ODVFSUR5NmFWbTRTUGxRVG9IakVCdWJubjZFdWov?= =?utf-8?B?cUR4MHlDbWM1VzhsWlM3eWEwVUtPMTdWREZWTzZuSXQ1aXFvVlpBSFF0cEJJ?= =?utf-8?B?enhrTExYalp0YUtwT3o2MVZGRVRqc09ibnRNVGttTlRzejdaMG03SHh6dW9N?= =?utf-8?B?L2pQcHZNbk5DcVdOV3RYblpPdlZONHA0RkRxL0pjMEo1czcyVzdYYVE3bEsr?= =?utf-8?B?N05SeG1IdmtEU0NhRkNacFBCTDRCRFNMMnpiQTRwajNEL2ovWlRaTnhTRi81?= =?utf-8?B?Qzl4TS9xZUMyV1hpczZBS2VqcHV0T0UvZjBTdEhKWmt2SFVhL25ldmtSajFN?= =?utf-8?B?RkpLakhMMHFHdU5jUnkwd1NrSFVRc1kzS0dEQjFWZnNPL3JJWWdlRzA0d0ZL?= =?utf-8?B?d1BXRE1VTHIvNHVmQ3RZaVhRWkZFbXVGRnhVcy8yVXZvbGZCeWdvUzQ0d1E4?= =?utf-8?B?Wmx1bVh4UU5ETVcxT09VUDhlUzFKVUt5T2dpdUFoM1hROVQrNjZXeUNUZ1JE?= =?utf-8?B?SDdqc1lDWkhnNWFBcmU0Zm9EY3BtVUFXc09vMkxNdjgzTHB4Z1Vsd3VoSlhC?= =?utf-8?B?a2I5M0ZMUHhwaUhTVldXTUhDYkluZlFXOHFCc0RYMmFYejh5Q3BJdFpCY25t?= =?utf-8?B?SzB3SkZHM2FBaGlBYVhSMDlZMXBPVytOeFh0bU9WM0pMczlIZmxHMVliRFgv?= =?utf-8?B?WEtaampMVm1ybEZRYzNGc2JGRzR5VHJHTnJxQnlWT1Zhei91dTlQYTJOT1BF?= =?utf-8?B?QXIvSm5ZRXM2K2VpMk1YZDRxRkNyMWtsQXZ0TlNjS0J2QWdtOTdHMVA0R3lw?= =?utf-8?B?T2dyb0p2ZTVpVG82OXVGTmRrZ0o2QW5qcEcweVlCNVFGa0tUUEhVOFNRSmkx?= =?utf-8?B?RDNVRFJCRExVdmRtNGVzcXE2MnNhWmg3N0pzTzNwZGtlK3hudXptWFNXK2s4?= =?utf-8?B?aUR1ekhhWGV2K01PdmJyNGhMNjMzdnVHdXRSSzlPRHFjRW05bHVoVFU4dFFU?= =?utf-8?B?dTNqSUtzOFhkb2JRY3VZeHZ3ZVJFemhKYmY5NWVIL1FNYVdnME9kY0N2TElT?= =?utf-8?B?SlRlZzM5ZzJ1VHdPOWVGSzFMWkRNNDhXTzg4M3JiMDA3ME8vT1Z2amtrMTUr?= =?utf-8?B?SVVuSHNnNlZsMWdWUnFvbVFqdmhmYkIraWlZYVdGRjBOQk5HOEpZWExncXI5?= =?utf-8?B?bHN0ZFpTaXdPVTVnVlB6aU1yV0tOcXlFVXUrOWlaQ3d3dmRqd1FyZmR3dE84?= =?utf-8?B?em1RNThUZUc2dVNQYUNyek9pQmk2UzZkZ0I0VUZmMUE3TUd2L0V5VjFZcWNL?= =?utf-8?B?Ti9Tbm1OdmIrdGw5cHlGOE1yNzREK1hsMUpjbTFHeVNoV09ERklyUWV4VWI5?= =?utf-8?B?dUlvbXc5QzZVV3l6eWFmLzdPSmJFSFFIeUxQZlNkd2pXYmNhd21KbjNwWnhj?= =?utf-8?B?d2tIcGZDYmJOekNzWlNiSVVxeGo5VG9mZjFVMXZTajdIcDFWYW85RkEyQ1o0?= =?utf-8?B?VU4xU0NjYmt5UVRTWEdUZFhmVTRjK3ljOTdYUmpXK3JNaTlvSEpCa2N6cVBt?= =?utf-8?B?MEUzQUJiY092amFsVk96L1ROWEJXNkwxSW1NRVRNTERJVUN5bkNiTXpnVG1h?= =?utf-8?B?bXJnUlRkS0FremgweW5pVGNsTWsyRjZWUmlHRDNoTi8zd0N6Q2NOeVJnMTZB?= =?utf-8?B?eVRNWDFLMGdaeVdHYTF2czBHcGljSk5HOUpwWHhBQks1am10ZVdyTCtFWno2?= =?utf-8?B?ZjZhckt1bEZOOG1XdGxjd3dZanVQOXFxemI5dCtHRDBicTBFRE16ZEpwVmVI?= =?utf-8?B?ZmNUYTJnQU5NeThHMFFzcDFYTmFNUmN5STA1UHlMbzV2cGxmeW5oRHQvR3F5?= =?utf-8?B?amljZUN5WGtpRXd4MWF2RE04bTdJaWVuU3N0UU8yczJGRk40VFdzVjBRQ1Vw?= =?utf-8?B?TzhqVW1QZ2oyVHJaamVWOUprY1FVd0FRazNPbUJHVFlmTVppVzVUc2hxbmVr?= =?utf-8?Q?ius32ufL/VM/9PdU3o29hgeJE?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: fb39326f-ce08-4361-351a-08dba24d896e X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Aug 2023 13:50:11.1243 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KOu8BfkKPeKLVTSfF90vK8lDrRUovbFN27VvK2sf53PLLyGcTRAJUoF8vLn/2zoH X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV3PR12MB9353 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230821_065018_324487_1926FF93 X-CRM114-Status: GOOD ( 33.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCBBdWcgMjEsIDIwMjMgYXQgMDk6Mzg6NDBQTSArMDgwMCwgTWljaGFlbCBTaGF2aXQg d3JvdGU6Cj4gT24gTW9uLCBBdWcgMjEsIDIwMjMgYXQgNzo1NOKAr1BNIEphc29uIEd1bnRob3Jw ZSA8amdnQG52aWRpYS5jb20+IHdyb3RlOgo+ID4KPiA+IE9uIE1vbiwgQXVnIDIxLCAyMDIzIGF0 IDA1OjMxOjIzUE0gKzA4MDAsIE1pY2hhZWwgU2hhdml0IHdyb3RlOgo+ID4gPiBPbiBGcmksIEF1 ZyAxOCwgMjAyMyBhdCAyOjM44oCvQU0gSmFzb24gR3VudGhvcnBlIDxqZ2dAbnZpZGlhLmNvbT4g d3JvdGU6Cj4gPiA+ID4KPiA+ID4gPiBPbiBGcmksIEF1ZyAxOCwgMjAyMyBhdCAwMjoxNjoyNUFN ICswODAwLCBNaWNoYWVsIFNoYXZpdCB3cm90ZToKPiA+ID4gPiA+IFBpY2sgYW4gQVNJRCB0aGF0 IGlzIHdpdGhpbiB0aGUgc3VwcG9ydGVkIHJhbmdlIG9mIGFsbCBTTU1VcyB0aGF0IHRoZQo+ID4g PiA+ID4gZG9tYWluIGlzIGluc3RhbGxlZCB0by4KPiA+ID4gPiA+Cj4gPiA+ID4gPiBTaWduZWQt b2ZmLWJ5OiBNaWNoYWVsIFNoYXZpdCA8bXNoYXZpdEBnb29nbGUuY29tPgo+ID4gPiA+ID4gLS0t Cj4gPiA+ID4KPiA+ID4gPiBUaGlzIHNlZW1zIGxpa2UgYSBwcmV0dHkgbmljaGUgc2NlbmFyaW8s IG1heWJlIHdlIHNob3VsZCBqdXN0IGtlZXAgYQo+ID4gPiA+IGdsb2JhbCBmb3IgdGhlIG1heCBB U0lEPwo+ID4gPiA+Cj4gPiA+ID4gT3RoZXJ3aXNlIHdlIG5lZWQgYSBjb2RlIHRvIGNoYW5nZSB0 aGUgQVNJRCwgZXZlbiBmb3Igbm9uLVNWQSBkb21haW5zLAo+ID4gPiA+IHdoZW4gdGhlIGRvbWFp biBpcyBpbnN0YWxsZWQgaW4gZGlmZmVyZW50IGRldmljZXMgaWYgdGhlIGN1cnJlbnQgQVNJRAo+ ID4gPiA+IGlzIG92ZXIgdGhlIGluc3RhbmNlIG1heC4uCj4gPiA+Cj4gPiA+IFRoaXMgUkZDIHRv b2sgdGhlIG90aGVyIGVhc3kgd2F5IG91dCBmb3IgdGhpcyBwcm9ibGVtIGJ5IHJlamVjdGluZwo+ ID4gPiBhdHRhY2hpbmcgYSBkb21haW4gaWYgaXRzIGN1cnJlbnRseSBhc3NpZ25lZCBBU0lEL1ZN SUQKPiA+ID4gaXMgb3V0IG9mIHJhbmdlIHdoZW4gYXR0YWNoaW5nIHRvIGEgbmV3IFNNTVUuIEJ1 dCBJJ20gbm90IHN1cmUKPiA+ID4gd2hpY2ggb2YgdGhlIHR3byBvcHRpb25zIGlzIHRoZSByaWdo dCB0cmFkZS1vZmYuCj4gPiA+IEVzcGVjaWFsbHkgaWYgd2UgbW92ZSBWTUlEIHRvIGEgZ2xvYmFs IGFsbG9jYXRvciAod2hpY2ggSSBwbGFuIHRvIGFkZAo+ID4gPiBmb3IgdjIpLCBzZXR0aW5nIGEg Z2xvYmFsIG1heGltdW0gZm9yIFZNSUQgb2YgMjU2IHNvdW5kcyBzbWFsbC4KPiA+Cj4gPiBJTUhP IHRoZSBzaW1wbGVzdCBhbmQgYmVzdCB0aGluZyBpcyB0byBtYWtlIGJvdGggdm1pZCBhbmQgYXNp ZCBhcwo+ID4gbG9jYWwgYWxsb2NhdG9ycy4gVGhlbiBhbG90IG9mIHRoZXNlIHByb2JsZW1zIGRp c2FwcGVhcgo+IAo+IFdlbGwgdGhhdCBkb2VzIHNvdW5kIGxpa2UgdGhlIG1vc3QgZmxleGlibGUs IGJ1dCBJTU8gcXVpdGUgYSBsb3QgbW9yZQo+IGNvbXBsaWNhdGVkLgo+IAo+IEknbGwgcG9zdCBh IHYyIFJGQyB0aGF0IHJlbW92ZXMgdGhlIGBpb21tdS9hcm0tc21tdS12MzogQWRkIGxpc3Qgb2YK PiBpbnN0YWxsZWRfc21tdXNgIHBhdGNoIGFuZCB1c2VzIGEgZmxhdCBtYXN0ZXIgbGlzdCBpbiBz bW11X2RvbWFpbiBhcwo+IHN1Z2dlc3RlZCBieSBSb2JpbiwgZm9yIGNvbXBhcmlzb24gd2l0aCB0 aGUgdjEuIEJ1dCBhdCBhIGdsYW5jZSB1c2luZyBhCj4gbG9jYWwgYWxsb2NhdG9yIHdvdWxkIHJl cXVpcmU6Cgo+IDEuIEtlZXBpbmcgdGhhdCBwYXRjaCBzbyB3ZSBjYW4gdHJhY2sgdGhlIGFzaWQv dm1pZCBmb3IgYSBkb21haW4gb24gYQo+IHBlciBzbW11IGluc3RhbmNlCgpZb3UnZCBoYXZlIHRv IHN0b3JlIHRoZSBjYWNoZSB0YWcgaW4gdGhlIHBlci1tYXN0ZXIgc3RydWN0IG9uIHRoYXQKbGlz dCBhbmQgdGFrZSBpdCBvdXQgb2YgdGhlIGRvbWFpbiBzdHJ1Y3QuCgpJZSB0aGUgbGlzdCBvZiBh dHRhY2hlZCBtYXN0ZXJzIGNvbnRhaW5zIHRoZSBwZXItbWFzdGVyIGNhY2hlIHRhZwppbnN0ZWFk IG9mIGEgZ2xvYmFsIGNhY2hlIHRhZy4KClRoZSBvbmx5IHBsYWNlIHlvdSBuZWVkIHRoZSBjYWNo ZSB0YWcgaXMgd2hlbiBpdGVyYXRpbmcgb3ZlciB0aGUgbGlzdApvZiBtYXN0ZXJzLCBzbyBpdCBp cyBPSy4KCklmIHRoZSBsaXN0IG9mIG1hc3RlcnMgaXMgc29ydGVkIGJ5IHNtbXUgdGhlbiB0aGUg Zmlyc3QgbWFzdGVyIG9mIGVhY2gKc21tdSBjYW4gYmUgdXNlZCB0byBwZXJmb3JtIHRoZSBjYWNo ZSB0YWcgaW52YWxpZGF0aW9uLCB0aGVuIHRoZSByZXN0Cm9mIHRoZSBsaXN0IGlzIHRoZSBBVEMg aW52YWxpZGF0aW9uLgoKVGhlIGxvb3BpbmcgY29kZSB3aWxsIGJlIGEgYml0IHVnbHkuCgo+IDIu IEtlZXBpbmcgYSBtYXAgaW4gdGhlIHNtbXUgc3RydWN0IHNvIHRoYXQgYXJtX3NtbXVfc2hhcmVf YXNpZCBjYW4KPiBmaW5kIGFueSBhcm1fc21tdV9pbnN0YWxsZWRfc21tdSB0aGF0IG5lZWQgdG8g aGF2ZSB0aGVpciBhc2lkCj4gdXBkYXRlZAoKWWVzLCB0aGUgZ2xvYmFsIHhhcnJheSBtb3ZlcyBp bnRvIHRoZSBzbW11Cgo+IChvbiBhIGxvb3Agb3ZlciBldmVyeSBzbW11IHRoZSBkb21haW4gaW4g YXJtX3NtbXVfbW11X25vdGlmaWVyX2dldCBpcwo+IGF0dGFjaGVkIHRvLCB3aGljaCBqdXN0IGF0 IGEgZ2xhbmNlIGxvb2tzIGhlYWRhY2hlIGluZHVjaW5nIGJlY2F1c2Ugb2YKPiBzdmEncyBwaWdn eWJhY2tpbmcgb24gdGhlIHJpZCBkb21haW4uKQoKTm90IGV2ZXJ5IHNtbXUsIGp1c3QgdGhlIG9u ZSB5b3UgYXJlICpjdXJyZW50bHkqIGF0dGFjaGluZyB0by4gV2UKZG9uJ3QgY2FyZSBpZiB0aGUg Km90aGVyKiBzbW11J3MgaGF2ZSBkaWZmZXJlbnQgQVNJRHMsIG1heWJlIHRoZXkgYXJlCm5vdCB1 c2luZyBCVE0sIG9yIHdvbid0IHVzZSBTVkEuCgpXZSBjYXJlIHRoYXQgKm91ciogc21tdSBoYXMg dGhlIHJpZ2h0IEFTSUQgd2hlbiB3ZSBnbyB0byBhdHRhY2ggdGhlCmRvbWFpbi4KClNvIHRoZSBs b2dpYyBpcyBub3QgcmVhbGx5IGFueSBkaWZmZXJlbnQsIHlvdSBhbHJlYWR5IGtub3cgdGhlIHNt bXUKYmVjYXVzZSBpdCBpcyBhdHRhY2hpbmcsIGFuZCB5b3UgZG8gdGhlIHNhbWUgbG9ja2luZyB4 YXJyYXkgc3R1ZmYgYXMKd2FzIGRvbmUgZ2xvYmFsbHksIGp1c3QgYWdhaW5zdCB0aGlzIGxvY2Fs IHNtbXUuCgpKYXNvbgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0 cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGlu Zm8vbGludXgtYXJtLWtlcm5lbAo=