From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA55DCDD1DC for ; Fri, 27 Sep 2024 20:45:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:CC:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=LWhOd+csSt57Wle1XgkQ8Vr2H7MNBgXyHyv1XbbzMC8=; b=uOU3cg78Wavfz7Xrck4N2wzpUp X4nfP8ajYGLxDAb+7nFJV7Sgi7jisQONKMLIGE1DwwpEWk4OnsR07mGlS9WLAFuKgq0Yg7VYwDQjt mVdw4x/hMl8PTvgPXl3xlxcHdpbPrNPb1zCTfIIiEQ9vTLOvB7LoLSDNdvMA4yjj9txQnJWbgum+e 0KzbHd2+rNJ3tCOOL8rG9TNPILd36XWu39QaTn3l100l6owzmzamQGw0Orz2LYxNWOJpV4gwJ2R1i wQcT5MVsRw8bz4A1pRf3X1kDgppBRuZSE/FisNa9YjM6lBPnimV6bIit12GdykA/+g2t+McGJiekz uAm6Qr8Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1suHqT-0000000C4rZ-0tnp; Fri, 27 Sep 2024 20:45:45 +0000 Received: from mail-bn8nam11on2060c.outbound.protection.outlook.com ([2a01:111:f403:2414::60c] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1suHpH-0000000C4jo-144I for linux-arm-kernel@lists.infradead.org; Fri, 27 Sep 2024 20:44:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kblS7hrdxouyN0tL20XcxqXhlUy0IYH779w/pPT/lR06f/PAQUqpg3mZEnsoBAG6B5FWZa84pNoKwIjpLxBlenb0/q/w9qSKm17sCJFYjB1pt+xDjI81iBoXUm1WL4ihFGG59Sx/sjtapFvhRqDeIoGVDqcn/nusDLgJrFrKFBw+bxwbnDVAN1Pr49YFBTF2jNG0bhkf5F3n51cJ9QN/20/3ghqLP6NirsPCv8+39MXTw5zwyMvqvAG7paDLOOlvTrLWCyf6OoeTPCitzxmFwuTeScGkT5eXi/zAT81qLwk1wcRWQ9Xj88Pc575P3ITPO1n5UsUhSAnj6Zk4vUOnNg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LWhOd+csSt57Wle1XgkQ8Vr2H7MNBgXyHyv1XbbzMC8=; b=pdaL+JeFbo9gpUJjF0FtIEdT8eioFt+R57qj6BP22xBm2UG8M3u9WHzIS/4qqgtAMjjUQZNMZgM3duanpUBVfm+3mGVelCz1PHKTLyxcYjN4YAyAo3FPjjj2u5sXRjSOOcwlCnFcF1MbT67MwsCKzfogN/1py/VobjereWIyMw6DSRnKKUOnT+pOCeI2D6CT1Ph6tAvzLgTe1UH9ZDUw5mtW75cJYBRGEJIawoqPtkk+Hp4djmtm1xS5SfGQNBE/tLWe1o7FuRD1LztcMYiWSaekPQEMKu2rHOGX6z1XtVOr6MRwfBZoeMFRu0YtcJOrVVR2lyvRahy6eo+faIqKTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.161) smtp.rcpttodomain=intel.com smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LWhOd+csSt57Wle1XgkQ8Vr2H7MNBgXyHyv1XbbzMC8=; b=f2lm1paTflzLRzkFGcJT2fXD5vsBmtWnC85+5yICXI8622Vr8vUPB8Rd05B3POdnNev2i3LZ+jqJ/CFtvYF8bWsr43esGc53N6jLzmWhxoAnWu9ZXrdAyels+6w2jG6oD4uF1/RvcSN7cgUjy+xjnqRC+DhcxhLs4k/O84Mf1sYhiVkpXbXg8PF0AappQ2fW22PqwWQc9VMfmrr0fM4TnmaMy2cCbVjk5i2qL8orP97DeDphf2WRwRLGB06j3yY1OiSCluE41M+VRRlnyV6kHNpJpkJt1ooV+3WQS6gVaj1u74vSx5dpaA1Rm8WBdba7w3qe28DXXU0aJ3+g11E05Q== Received: from SN7PR04CA0153.namprd04.prod.outlook.com (2603:10b6:806:125::8) by MN6PR12MB8543.namprd12.prod.outlook.com (2603:10b6:208:47b::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8005.21; Fri, 27 Sep 2024 20:44:20 +0000 Received: from SA2PEPF000015CA.namprd03.prod.outlook.com (2603:10b6:806:125:cafe::9) by SN7PR04CA0153.outlook.office365.com (2603:10b6:806:125::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8005.22 via Frontend Transport; Fri, 27 Sep 2024 20:44:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.161) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.161 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.161; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.161) by SA2PEPF000015CA.mail.protection.outlook.com (10.167.241.200) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8005.15 via Frontend Transport; Fri, 27 Sep 2024 20:44:19 +0000 Received: from rnnvmail204.nvidia.com (10.129.68.6) by mail.nvidia.com (10.129.200.67) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 27 Sep 2024 13:44:07 -0700 Received: from rnnvmail202.nvidia.com (10.129.68.7) by rnnvmail204.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Fri, 27 Sep 2024 13:44:07 -0700 Received: from Asurada-Nvidia (10.127.8.14) by mail.nvidia.com (10.129.68.7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4 via Frontend Transport; Fri, 27 Sep 2024 13:44:06 -0700 Date: Fri, 27 Sep 2024 13:44:04 -0700 From: Nicolin Chen To: Yi Liu CC: , , , , , , , , , , , , , , , , , , Subject: Re: [PATCH v2 00/19] iommufd: Add VIOMMU infrastructure (Part-1) Message-ID: References: <82632802-c55a-4199-b685-8b594a8e7104@intel.com> <8440cd72-ced6-4887-b724-ce6a1650db13@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <8440cd72-ced6-4887-b724-ce6a1650db13@intel.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SA2PEPF000015CA:EE_|MN6PR12MB8543:EE_ X-MS-Office365-Filtering-Correlation-Id: 578282f2-b0b3-4a59-48ff-08dcdf35295f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|7416014|376014|36860700013|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?aEreRviyPRQsrM7sP+xgpNclKylODlS6v9dB0pxXugGcDq3sZ0fPXOHa4Oax?= =?us-ascii?Q?9/ZS5Jyg4ksyL1M2yTxYU9/LUzqzyg0rQZwYfhoSAxgufgqdfARHyeRjpi9t?= =?us-ascii?Q?RIv6+vRPgVY31QyPWj+RbbYwYvBEdPD7/Va2/Rr8aXZw+kayzMZBkNQOVcTq?= =?us-ascii?Q?U1fwlHcf/vgbsvlPS7JWi7VXoNQ4l/XKIUWlDmtl4mnXcR+NDPptOK8JWqYD?= =?us-ascii?Q?gX6BbIJ26iM24WrT4qbOkpcLjfO2Us/3TSvkfAB+zDCPj5LF4HajuFgvwHY1?= =?us-ascii?Q?2r2iHIW0/XbCioGhAtbzP3BKrCOJHH5LztZ+saGypt8dg+aJgPBziJwQokSr?= =?us-ascii?Q?o6At/SQOR7pKJherdmRZ3q5fQQKCFPvALVcWo4Gs/PyX0TkJ+RlNkEENEAPW?= =?us-ascii?Q?UNqUpU/6sTtrSWy33mtcK8UeRieDWz4plgPjZKDDUVkWGiL3KyNvwZOyAcTo?= =?us-ascii?Q?dVK6Hdp/KjLS+odQFTVr+TCyY/qoh+oyeyP/o/yuYHdKY/GOYIVEUE4gXd5S?= =?us-ascii?Q?kZexgu7jbpWKMpPyeMyfTnh75jrxXPauk7/gS/c1njG4AfMc//5ZVPibjX1K?= =?us-ascii?Q?uZp2wMRrCbUe+l42a3pvUIX3jdKdH5HA9bMrUAnL3xT45sl7yKArURVoky4y?= =?us-ascii?Q?PWZP9yTXIrLcTujYIEgScVD+D4aH51gJ4xgrGZ86oHWiT9ewC7lB0n5+14As?= =?us-ascii?Q?sG5IYIa8jvlEE5XHH0DUeOLPY68ykRmj2OVekb5Rfq1EzJAB833flK5Ser+2?= =?us-ascii?Q?cDQiLv7o53KJnmhQ/SzLUsW33ZjQc79mpJl8yElnVcuwOV6wJ1+E3bApRXyV?= =?us-ascii?Q?+TpOVhXm5g3ZcXUUIpWGsI1zGnuNrCvwp9wqJychCssKdrFr/XcUhDoqL3en?= =?us-ascii?Q?bHRJSMSVtcvdX7WT2oevJXNolIbdHKOGKZSr+yHE9XCSvbRotJOrbJAAgUNK?= =?us-ascii?Q?opDAnapnRFVT1NhAwQ47ZM1YIUsspiPeitP9mENO1oNvn+N0/q2sZdQgMWi6?= =?us-ascii?Q?lsGPT8paY+wV0MbDUAQt13Qy8WqvtyfSfpsTmqdE2MSX+zrdQkQo6RiUWZBo?= =?us-ascii?Q?2DYGnZT27cL+n8m6S5RFB5JHeIxXejdQ1EsprxfoMxnXHaXt8vFsJ7S9lZO/?= =?us-ascii?Q?Tn57C7uSxmq/k3AkgpbLMmmA28o44ZyokzRTD3pzZ3pR/snYPCg8mHAze1Qx?= =?us-ascii?Q?4iE/n9rjjzQsJyutnGN83cK6n+FunRvdfcocIEXaMPVL3laKvFuw0luZnGeD?= =?us-ascii?Q?GCu+54pVewBKXoq8JhZIh+/Gih6zMAF79zvy0aWn8VbBdc4fZ8s8K++kXobh?= =?us-ascii?Q?L+JsOd9k98S+D5kbFYRwIK29OisOKu3tc6uCrAXshZppkt1wRsMDRyl9IpW7?= =?us-ascii?Q?KEoATXsVx+XVxQ8by3B2yOtSntNTHAAW8mIE2Pv/QuSPYPxtjDhHJv76eZpE?= =?us-ascii?Q?KO8IzMTzXqcyz/qU15y1kBoQ1pGeBrbc?= X-Forefront-Antispam-Report: CIP:216.228.117.161;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge2.nvidia.com;CAT:NONE;SFS:(13230040)(82310400026)(7416014)(376014)(36860700013)(1800799024);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Sep 2024 20:44:19.8732 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 578282f2-b0b3-4a59-48ff-08dcdf35295f X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.161];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SA2PEPF000015CA.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN6PR12MB8543 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240927_134431_306795_E372784B X-CRM114-Status: GOOD ( 27.55 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, Sep 27, 2024 at 08:12:20PM +0800, Yi Liu wrote: > On 2024/9/27 14:32, Nicolin Chen wrote: > > On Fri, Sep 27, 2024 at 01:54:45PM +0800, Yi Liu wrote: > > > > > > Baolu told me that Intel may have the same: different domain IDs > > > > > > on different IOMMUs; multiple IOMMU instances on one chip: > > > > > > https://lore.kernel.org/linux-iommu/cf4fe15c-8bcb-4132-a1fd-b2c8ddf2731b@linux.intel.com/ > > > > > > So, I think we are having the same situation here. > > > > > > > > > > yes, it's called iommu unit or dmar. A typical Intel server can have > > > > > multiple iommu units. But like Baolu mentioned in that thread, the intel > > > > > iommu driver maintains separate domain ID spaces for iommu units, which > > > > > means a given iommu domain has different DIDs when associated with > > > > > different iommu units. So intel side is not suffering from this so far. > > > > > > > > An ARM SMMU has its own VMID pool as well. The suffering comes > > > > from associating VMIDs to one shared parent S2 domain. > > > > > > Is this because of the VMID is tied with a S2 domain? > > > > On ARM, yes. VMID is a part of S2 domain stuff. > > > > > > Does a DID per S1 nested domain or parent S2? If it is per S2, > > > > I think the same suffering applies when we share the S2 across > > > > IOMMU instances? > > > > > > per S1 I think. The iotlb efficiency is low as S2 caches would be > > > tagged with different DIDs even the page table is the same. :) > > > > On ARM, the stage-1 is tagged with an ASID (Address Space ID) > > while the stage-2 is tagged with a VMID. Then an invalidation > > for a nested S1 domain must require the VMID from the S2. The > > ASID may be also required if the invalidation is specific to > > that address space (otherwise, broadcast per VMID.) > Looks like the nested s1 caches are tagged with both ASID and VMID. Yea, my understanding is similar. If both stages are enabled for a nested translation, VMID is tagged for S1 cache too. > > I feel these two might act somehow similarly to the two DIDs > > during nested translations? > > not quite the same. Is it possible that the ASID is the same for stage-1? > Intel VT-d side can have the pasid to be the same. Like the gIOVA, all > devices use the same ridpasid. Like the scenario I replied to Baolu[1], > do er choose to use different DIDs to differentiate the caches for the > two devices. On ARM, each S1 domain (either a normal stage-1 PASID=0 domain or an SVA PASID>0 domain) has a unique ASID. So it unlikely has the situation of two identical ASIDs if they are on the same vIOMMU, because the ASID pool is per IOMMU instance (whether p or v). With two vIOMMU instances, there might be the same ASIDs but they will be tagged with different VMIDs. > [1] > https://lore.kernel.org/linux-iommu/4bc9bd20-5aae-440d-84fd-f530d0747c23@intel.com/ Is "gIOVA" a type of invalidation that only uses "address" out of "PASID, DID and address"? I.e. PASID and DID are not provided via the invalidation request, so it's going to broadcast all viommus? Thanks Nicolin