From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B4198C47DB3 for ; Wed, 31 Jan 2024 14:12:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=GrDATisZ1Q5cZrY25aAPM45vxA1nh/yEHQxjRG151fE=; b=T9kko42ho+hr4U Xf2p4f7VilEFsb0VR5R7LMuq2cgK34osl5SgJmaqYemzzU0r/qBDcuTUgUF43egpU+zNB816QmiID fRLp1nApR1L0VhIF/79srO1GIZcOlMlZ20g1ubOJmwRjSEMVC+H+RPFY3LJHtKFvYMV998f4lsGtV z9GrFHFb6YQu4FkOrJbd7UbvVK4n5PsscCo89adUcO23cypPQ5oOymq3rKfrO/YtpglMPQcH+4jVI ncPogw77uNG4+1fRrUk7rvwx3kURkr4CwyzlLSgEHNSzWoiz5uWrzIMWeVU8gZBrL/LvypAByf+BM dSnETi0L8rKjm68xWUow==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVBKH-00000003oFY-3Qo4; Wed, 31 Jan 2024 14:12:29 +0000 Received: from mail-bn7nam10on20600.outbound.protection.outlook.com ([2a01:111:f403:2009::600] helo=NAM10-BN7-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rVBKF-00000003oE7-28ZU for linux-arm-kernel@lists.infradead.org; Wed, 31 Jan 2024 14:12:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fQY82E560LJvEUz3ToRI7vqeBuIy//Ut6mZE7m4hRtlUsqU1PCL/3z6NJnRV5NwLS44QUrKoNue4I6PjsnM85OD0scaUqwK+ZOVi9Ik+X8fNKjmAMkWD9qPtyhagP69+3iOmT2OB6C40xsL2embnyUSbUTmM1TQ7Glp7UTAfAFGiydt3Qz/95b+dnJfPluwQ4imPrXuyOBLgF/I1/5TMRhKgnwwEvUzL2NPYXqTtAQkG6LSZTl5/b9im0rYWP67RHrTgxi1OeWs3U5ZiLtPvT0pLK5BsXBXW4nkhSbVGY3ajTMPcNhns3WFAXGUAub3bfBC7tr6v2c6+SjgT/OjZYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5Qy3pK3LRXnxFJT8/zwRp+sto7WjN0MxXBV2XCLc2+0=; b=YxM+ZxInBayick1APm2doQzOWxUnd8Ejkzm+0PDjOCixwXOU5NmTfql1BJP6NwhQDW3y0+0DCUBXcjt//SetpZc2GJggok5ze7inRSHqYVsr/kn7EFAJjTbPSxxfRlBa43eqxbGTj5DlCMAbGgRyNHIA5JNQTuTew5G9IrkM66GvXQs8r5qOSlnmGYaRLEyXe8kf1xiY519KbTIPalEiUCbA0mitiBAJ8d8+IjngirCR2u3YZAaY3FgbI2Gih/F0y17kijyuGhKHRFmoaqMraDmJ/O38ePFgoDRW7MaYIUJRKMEwBNB3V0ONq1k9ULthb0ve8dZYXVik4/sNwgjG6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5Qy3pK3LRXnxFJT8/zwRp+sto7WjN0MxXBV2XCLc2+0=; b=aEMWJ/oIDyq3G+pwjYlG6gnx5TCwT+8LE+tSEN+jVreEu2CNh9UYzKQMDFNr66DJASSdD+/NH5BiFAoN+7hUKEOBIvLtveDGt845c4M3ttwMlfgMlq0LbZlzlYIRzlZ2VqQu+Y91ZWs83kI5wP2UeE95NdFVQJwu1ulicgcZbrlsCHbnajHwZhnX1GaJJPVgaMxDKqAHyPCO1HDWP3wDH1KQ5pF442oZ9l0HN3FTUloF6WiyYmE3luTxLZ+DY/+Lx/DljMHRTU81nJsFmM2yWIoLKoNHBTKrmVJMhnICN1NJ468ea4pjJtnFeL/iM+nLyJOF8o2CevQvJAunEOVrjA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by CY5PR12MB6204.namprd12.prod.outlook.com (2603:10b6:930:23::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.24; Wed, 31 Jan 2024 14:12:18 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873%6]) with mapi id 15.20.7228.029; Wed, 31 Jan 2024 14:12:18 +0000 Date: Wed, 31 Jan 2024 10:12:17 -0400 From: Jason Gunthorpe To: Shameerali Kolothum Thodi Cc: "iommu@lists.linux.dev" , Joerg Roedel , "linux-arm-kernel@lists.infradead.org" , Robin Murphy , Will Deacon , Eric Auger , Jean-Philippe Brucker , Moritz Fischer , Michael Shavit , Nicolin Chen , "patches@lists.linux.dev" Subject: Re: [PATCH v4 14/27] iommu/arm-smmu-v3: Make changing domains be hitless for ATS Message-ID: <20240131141217.GG1455070@nvidia.com> References: <0-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com> <14-v4-e7091cdd9e8d+43b1-smmuv3_newapi_p2_jgg@nvidia.com> <78401f5d541042f5a71f13bf8af7ed3f@huawei.com> Content-Disposition: inline In-Reply-To: <78401f5d541042f5a71f13bf8af7ed3f@huawei.com> X-ClientProxiedBy: BL1P221CA0026.NAMP221.PROD.OUTLOOK.COM (2603:10b6:208:2c5::27) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|CY5PR12MB6204:EE_ X-MS-Office365-Filtering-Correlation-Id: 8ea9631f-0c19-43ac-00a9-08dc2266a1e7 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: UBnn3U6kai+OYPcPV16VHVBw5PsQB0w2PUbN9AyQeDIJT6zbycrRxf8ANzddu9wbvi7Iij2nkZoMPDnjgfG+LHKaMO+1oQghtkGIyfi1COjgsErYhGYN/0ob5m/GPSaux8ruPU8h8wUtEURsJHVavG+Ld1XwKndIzGWBiCyL3pRDKLZF4eRdyjsFE60zqsJcwsXFtVmXC4lJ0JytLE33Hlwa8H4r/EIMiUbRmLISd9FNvwEURRUoBZccfkH8p9/GFcNF27THBH5kCkIj1Vx7CatfVlTNlBU05JaGOkqdcru2o2pDasWnDVBEC9WfSTBZcelb3OGaBHbrWMdkNrxWpQ03p1Wm+hQQkzPGktzSefetGdJ3pHQ+aR105fj8GL9DjprWtc7jHCwyAROYC3s8YoTe9iiPK2MkayUm+7n+ZI6TePgQqKBupjBkZDi+UgjQlPkZAgy5bFh57gA8HMOKtoULzqvYwuSBdhFGgbdynDETmqyqFC3jOC3KTrVTdshh5qYLuujc9kLGgNLOt5xddrReRFF+13GFaDHAG43BDqlYcpWHLkU6lSVehX46k2+B X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(366004)(376002)(39860400002)(346002)(136003)(396003)(230922051799003)(451199024)(186009)(1800799012)(64100799003)(26005)(2616005)(1076003)(83380400001)(478600001)(6486002)(41300700001)(6506007)(6512007)(36756003)(8936002)(4326008)(8676002)(316002)(66476007)(66556008)(6916009)(66946007)(54906003)(2906002)(5660300002)(33656002)(38100700002)(7416002)(86362001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?w+Hw1anDfTjJPY8MixYlsYJe6GAtiDOqGu3Qke/kLcme8KzMeWQEwAlgzUSq?= =?us-ascii?Q?nXJNT0mAbqslMqXExB8SPponsMK/hqKeXoPm4KeEHcWDkp0BbB22LbXhNSNe?= =?us-ascii?Q?wB7rW+5QiWkG+1Eu9HHlxsaG6w0yOdTCrACeHa9p1xCOHk8x6H2hssp4mpju?= =?us-ascii?Q?h08SJZbEz7oRTJQ1GrHIPETW5CMgFRfl/m36cAjWjJ8zSa6fCo4VXevhsnOx?= =?us-ascii?Q?jXmQJ5YFZEbxC9EWIjINGtrPWQ4xFLLLyyPnSu5+jFC6dctGCsgdaC8j2+Nb?= =?us-ascii?Q?iy7j0yBcSSe+nE9SlC0tuj4gle0MIMycFimaQpq3Q58GCEQCq/4QeJWzY1NP?= =?us-ascii?Q?IvUQtha+9h4ESSEQT+BqrM4HQrEOmAjxjHcJGC4mahTryBr4QiXf8guNRsSJ?= =?us-ascii?Q?u42TCN9wFxIy9ThtaBZ/RKR3jWyMtnvN0wNIlNgzVvocv8glgUn2jBXRgRtK?= =?us-ascii?Q?etcyzvr6Ejneuky3+UJk6Db5tUKbL8nlT/byJvi+4Bq+atT7K364+BctEoRU?= =?us-ascii?Q?EOKkrWq8o57fOeU21nxgI81qVT9YEiyrUKsATEbohIh41UHgr9r4ZpjYH4CR?= =?us-ascii?Q?jHr+4Q8TgCquC4bQ7s3BAhR+WTfliGZ4p0f75OGkaAZdPTOQU7FahVaWtEPz?= =?us-ascii?Q?9qHFVJ8md6Dn0SWtC6CXjfaEAPyPc0Z3PYDmSIPzb8DY6xnXc7J0h92wFn6g?= =?us-ascii?Q?LVHdq6y5VD4WgI5ekbUEdG02BcCXY2XoUvLuQrIB4velaVVdJwN5u5yCWMUP?= =?us-ascii?Q?fobwrOtFobmT4Z7OE21y0J9wUOhk8upDYvFOvWDIngmSRu0TZJn04k3rZSyG?= =?us-ascii?Q?mLPoCvLWugrhFBiqZX5gGUjHyszCktDxCDca3VXz0TvYEj1287x+xIS2m1KJ?= =?us-ascii?Q?CewOIOKBM95PRtpZWTSeKjMisu5etehWBtZYlHyUSbqxQYx/s6RR90TJhPti?= =?us-ascii?Q?rrC0lEDXmVgO4Z+lTPOVkERcRf5USKvw3mz32F4gRIrc8o3F5sPx01Cpm1YD?= =?us-ascii?Q?zlHPSeR6iQJ8q0/sOn2WHdMJ5ekJAJlYYjHcCkEQxQ8U+uOpfqMnuRsgMOvf?= =?us-ascii?Q?P29WRvPSktK+J6UQdlYwpKWYylzs5XqNvj6+ZDdlaDS2XdgyORalrpELAGGh?= =?us-ascii?Q?mCFRqdoPBmxSb05557w4AEUVXDkKXU6uRbHp5A/A5PsPc/Z1vTFXyOAOn4ki?= =?us-ascii?Q?x49NlUM1sSjvW0zYMLzQkBhIR/sFD0/uHHlTKz2XQHZ4Zd0fICRkfF4akpCz?= =?us-ascii?Q?FgDPKxcilccpCKeA3o+KcSB/58i3gQS1pIDOlQk3EV2PV48GRRLWFh5CIcft?= =?us-ascii?Q?xXs2zZ5qjvkm/0ncQvw/vaoijZNAzhgridX19jikdwCYJZTMaagRlUw/Vr+U?= =?us-ascii?Q?RQo9X7QlYOOe5Z6Rcd6sVwvGVj0V+c0haJpWQpHKRATEYxbUWmeEfA4P+hc6?= =?us-ascii?Q?4r+BEDBZkkQDCjBKlqRqNuTHxwyI6nfovekZdLlxtYNS+DrZi/iXBm17EIrB?= =?us-ascii?Q?MGX5Qg6lyQ8NoGCZxk9sPbxRN4DN77XZoXl6cEkWsZsBAXJ7G0nCZLQefRGZ?= =?us-ascii?Q?mQgf9rEX7a96HRPCGZC0uwO0KZVl5gDdvY0y/KR1?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ea9631f-0c19-43ac-00a9-08dc2266a1e7 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Jan 2024 14:12:17.9312 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1q9IXQ/WX6Qcz5RYXZ60uZNO5qZJtjG3WBJM4APLif2VRqZFp9LhjIHS4RJ3soKp X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY5PR12MB6204 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240131_061227_581605_0690EE23 X-CRM114-Status: GOOD ( 18.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Jan 31, 2024 at 11:20:48AM +0000, Shameerali Kolothum Thodi wrote: > > +static void arm_smmu_attach_commit(struct arm_smmu_master *master, > > + struct attach_state *state) > > +{ > > + lockdep_assert_held(&arm_smmu_asid_lock); > > + > > + if (state->want_ats && !master->ats_enabled) { > > + arm_smmu_enable_ats(master); > > + } else if (master->ats_enabled) { > > + /* > > + * The translation has changed, flush the ATC. At this point the > > + * SMMU is translating for the new domain and both the > > old&new > > + * domain will issue invalidations. > > + */ > > + arm_smmu_atc_inv_master(master); > > + } > > + master->ats_enabled = state->want_ats; > > + > > + arm_smmu_remove_master_domain(master, > > + iommu_get_domain_for_dev(master- > > >dev)); > > I think the core has not yet set the group->domain since we are still in attach(). Any chance you are not running exactly v4? There was an earlier version on github that had exactly what you said, Nicolin found it and it was fixed before v4 was posted: static struct arm_smmu_domain * to_smmu_domain_devices(struct iommu_domain *domain) { /* The domain can be NULL only when processing the first attach */ if (!domain) return NULL; ^^^^^^^^^^^^^^^^^^^^^ This was added if (domain->type & __IOMMU_DOMAIN_PAGING) return to_smmu_domain(domain); return NULL; } static void arm_smmu_remove_master_domain(struct arm_smmu_master *master, struct iommu_domain *domain) { struct arm_smmu_domain *smmu_domain = to_smmu_domain_devices(domain); struct arm_smmu_master_domain *master_domain; unsigned long flags; if (!smmu_domain) <---- On first attach smmu_domain will be NULL return; (this is what it looks like in my version of v4 at commit 595a8ea0f0ad60e80412dcfbc36b74af7116ba4a) Otherwise I don't have a guess.. Can you narrow it a bit more? Thanks, Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel