From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2057.outbound.protection.outlook.com [40.107.244.57]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCBE7131E4F for ; Thu, 25 Jan 2024 17:47:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.107.244.57 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706204855; cv=fail; b=sfv2CuuUS6PLhksV6P0meaymy+THoeCWeOLjlzvixMyMzhTNuJe3bOW6M2ylZvoG4VlXlAav8WBD6UlU8BtVv40c0ByLJaf1vlrf8VeEpu+TWWuNWEIhDhyK8bG0acGHv55ElSmrxq5d8uecheQIv0fr7/IK/NXzDA2q3ZtzyFw= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706204855; c=relaxed/simple; bh=WhiS8nkA/13kBCqUhE7R4xSHYn8wFboELu3JNnY464U=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=VoiarGTg0N3IROtmWSgpY5Z+Rj7ny0mFyWPl6e5y8hljP6v5JGhT4nuzOxKnzbghgp0IO9a/IN4ha0E5WxNeWz1GGqyd8ilsoZsTaAS9INqkeryJCGYP7MhQ9Q0QSnIVIqX/11VQHklsKDS09k7YHamzTOcfRuwh6xwk+V+QzwI= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=JGtW8UAh; arc=fail smtp.client-ip=40.107.244.57 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="JGtW8UAh" ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M6EbbaRXXAr2aRmIhqgYbbNY03Y6J9QVeSYYwz/L+LaK0ZA40RBJbpFm4UsGobIhCidfLh0Fct8Yx7p67woKh8YDhCKWPWxSMiEmiWtX6eYUhEayoGhxgg/MRBVi+o+tumOFZlo0rKH+CVjvcN9AS7OPJwL6M8rvwCbtF3oiKb4clsL/xzwys8uk3LkqxEJj94Ehmd+/oPdqVyVCOIdKVTn71NuSq+HHj6FhQa5D4vtbTxR1Kv3MliWeWacGXuqTqNJRU0CPNRyyjlqd/IMp71qliwJw9qpiVMKf4MugU+EsRBk9bfYmt+YJo2LmDKAyh7s1la7GzJyEBxYT0RpPZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PdAGPjO2Ukc9uHhaC6W7AP6OBCk0cg0VxKoUH4ln5Bs=; b=ku0d2bdQz5BaJiYm8lxkT7gIPRlQF5Z5TBdqhVJ/HMx2wDqeCuo5ksSZBx9ee21dEoIAldbMhSs4OBNPrA8vy/r/z0W+96W8tcto96fNVo95dncq9R60gOlEeYFGb7uhlgdqAoFX074hWkCqW3ECDxPJdS2SFP8ZAvuD/a0WNTI3xRKFVv2eVgToUb2JmWSwRWokQJ0qP+yGZ47b34NDkazT10yHrJSH3RWSvfSfFEJyDFTqZtVHy/uaAnRmwl3scgTqb4kfMYZeCpUwSJhajE2h44t8PxahyKJhkY6HfxroWDaxZhQaHBr1vvw33YGzjefpdgBub0QS+q+L+CYrog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PdAGPjO2Ukc9uHhaC6W7AP6OBCk0cg0VxKoUH4ln5Bs=; b=JGtW8UAhXzgo9WDCtKUjSSxaBV9aXJ5bi07f02dJVrbnot4kelE5c8kD9tJP+KpClXVrqypMkQYOyeCvpMhjBuQI1haQWRRZK8zdcYUwOvenh9B9REorsFd25+KvDxSeMc0B16owqAm18a4R1a3rE7Qh19GzlcKHbXHvDhI3bp5gZzJwwzjBSLLSs512oPUFUTQTXuLc+lwD0s3As/fJjoMWbeP9988qQTX5njbZ6ZwpNmq7xbxryQ9eYD42cIG0tXQfVmGRKbjz/KXxaMyP7aT7ZUGh1Sm3HfX+HFhEMu+oEgALbKg2oygXiqV29WC0WyOtXNtlCMc1lGali5AypA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SN7PR12MB8132.namprd12.prod.outlook.com (2603:10b6:806:321::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.32; Thu, 25 Jan 2024 17:47:30 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873%6]) with mapi id 15.20.7228.022; Thu, 25 Jan 2024 17:47:30 +0000 Date: Thu, 25 Jan 2024 13:47:28 -0400 From: Jason Gunthorpe To: Nicolin Chen Cc: "will@kernel.org" , Robin Murphy , "joro@8bytes.org" , "jean-philippe@linaro.org" , Alistair Popple , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "iommu@lists.linux.dev" Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in struct io_pgtable_cfg Message-ID: <20240125174728.GR1455070@nvidia.com> References: <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com> <20240122130152.GP734935@nvidia.com> <20240122175700.GB1455070@nvidia.com> <20240125135537.GP1455070@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DS7PR07CA0019.namprd07.prod.outlook.com (2603:10b6:5:3af::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SN7PR12MB8132:EE_ X-MS-Office365-Filtering-Correlation-Id: 735c78b4-4914-4631-d6a2-08dc1dcdb402 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KOiLm1H/oKcpFYVLSFBeQSBAkpD9zqw3M9IOsPT59wppvSfx1YzcuBv94RN08BM+KZW2NTGZDsBD97vSPdm7xrchBHpoSEV9smlBYiz3Sq2ZpC1ncB/FwwK2UpMwDis8m/4fnJ85QRDYo/HpiPCfz1aSP4px7EJb8+5LRZ5U6Q4je93Yn7IM4G4vgdWZaWnFY0H86a+uGzl2fOWk/N8jm95rM1DGg9mIYKxf1Zb8QMA4TLzBmiUTEn79mVqjWXmrKwvqyrdORlE7kqqAyjNp4XJkSda2gcT5pSNlTQnCDl+ZLUxifuHoTQfkLXycnyGkix4Wgwd/vnXIACnP7qDhE/UP7lW0QG0RjnEhNP7ETafv99qa8iGOh4FNKyw65K8YmMTKodewBEfjNhBXQIRhvu3yno2wWogWoFA+7m7Pd/2uMYD5aQ94pGMeP+S/cnmT0to6gM0HijId9WXhjLP02+znuHB8R4dfIwMpwwmvN7qd9oWF9zjYVbyzEOmbYDUFCuY5jNXnMXw2WJn1DFZEOdHGaePSftcLfZ7HvPCCgB8hTdssd8oP+v4D6YX4XtHU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(136003)(396003)(366004)(376002)(230922051799003)(186009)(451199024)(64100799003)(1800799012)(26005)(1076003)(478600001)(2616005)(83380400001)(6512007)(86362001)(38100700002)(66556008)(5660300002)(316002)(33656002)(8676002)(41300700001)(6486002)(66946007)(37006003)(8936002)(2906002)(6506007)(54906003)(36756003)(66476007)(6636002)(4326008)(6862004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7MMPUIiA7mjh2rPTRFBqx+5qjVHHF/bn3ei8n+ZTaAum9y221s1NnCSK+ueQ?= =?us-ascii?Q?u7RBxBjGnJiA8FHZoCQAZ5BRQ99s9lZWInezn1x5mxGycps6slJtA/yH4aFo?= =?us-ascii?Q?2Q1Q3TKnN2uIafZbN16bTUNqWhFnriY857v/NNZ0GGhjEDb84kO253twSuky?= =?us-ascii?Q?/W+0v5Bo16gkUhdswAy9xAp433KWCMbVZ1itaU4MyRn8A5HshameXskwmdAe?= =?us-ascii?Q?m1QKTNj7lsW0w99xSGvwY+vTWzS+av7PgrhenMZphfztxiI8zo/tvBuVcX0g?= =?us-ascii?Q?870xN/2/snP1sjo6kW4q2XS4+0+YXkk1cHLEyiIIQDKa5BySQydkeHj3obem?= =?us-ascii?Q?alJCByljrtim5Yd+Y+lB+RvzqlBGi8erNyaWYEs7udrrl+TuiffanLayaVWG?= =?us-ascii?Q?HrqEmXD7nHI4MuTJkeVQysRkxukpuvQcT7hWSz63CVKFbFgEG3PUsJWTWlpW?= =?us-ascii?Q?5s/xZnmqLhD5NiX+ox+9kf8omiqiXi/qgUdpWMvnB9veCX5PTlDwtRwfsdSG?= =?us-ascii?Q?N5C+ttg4SMR/YPbe3+e+lJdAy4YJb+83+q+ZLbrv1mspQLN631nPEz19M2lo?= =?us-ascii?Q?Nus6bh/O/WIgqtMIU9DCSE3DfJS8Ge6XJ5QaR7zsEFvWqgwydSS26EP9czwV?= =?us-ascii?Q?W3DauLL1f1C19JSCgLivvNbwga2oWfMYAjQvfXqitiHDuKzkAuyTwE46Pmw/?= =?us-ascii?Q?P+qnbvlX6l60LCksG72VPagBWP1sg5+fYjnpRYm0O2K+tuwH/P5q0eVh+YzJ?= =?us-ascii?Q?1qePUJdZjYFSRW8Pd1Q6GJVWeDPfpMwGRt+ESxnuRlMsSi8qVAtkhxNfDHVi?= =?us-ascii?Q?AIK4cMvEkKL54XkTBSXDsgtndtPLtKjgxU1t9ffzTSHJA26ubT8TNzUH8Fur?= =?us-ascii?Q?TAjYa7PLl48l93IVrG4ywyc/zQQfW8RVcHD+C05i0pPHFe3Rk5Lk+QbPZyX1?= =?us-ascii?Q?TeMBG3z/f0014emFCoLb/BX1HsRmEC0I4v9gtB5BigVSH+aLwsPzqtuBegVn?= =?us-ascii?Q?vdrRBLlfKDjUXfZ3NBabX4YlQpL8SZz6KCwPBza0ZEzHPF//ZEWqqDafABkn?= =?us-ascii?Q?k17QydQytug78QqyA2cgKG+ppEK2WB2Od7NPVmaOBjK5lrjSSSP36t6rHPnS?= =?us-ascii?Q?ZmAtB10W+W2Ps1/IfpUA9yRsaL3oMPeRjB6gNTuKRKGM/On/Jk0QmIAnEe5m?= =?us-ascii?Q?1oXzzFSMWXuaG/1Q698969X+EtW2rU9rBb7n98X5y7agodjs7HYuVObZy4Hr?= =?us-ascii?Q?Ot7rxXHDuOMRjYtST/diciUAQGffqzDnj1f27O8TBNYtYUBSgJRAOWVfxCV3?= =?us-ascii?Q?ptmtIdtVKdBK4CFEunxc74hArM1lE2FvpM8ySeOMxsoZ/UPGbFosXsjDSIlC?= =?us-ascii?Q?Sc+sTS0gzmG2Iaxr5dnYIz9gfo6uVvpEqgVoDetyzS7Q8HMKdG7lD34hxDCe?= =?us-ascii?Q?On59xOvb0DnIQz/Gcqlqho3PnhF1OgD+3vyI/XSSy44ZW7DJ3bUkl4Ehh6qU?= =?us-ascii?Q?OswmtsMmdJjHYIt/QKoQ06nXXhTbJnY7LLLjzpvzTa3s9U9wnigAcWWbftSv?= =?us-ascii?Q?fau3sj07xNEH0MfJm6vD9nCY9HNSNCdQN0SDk+Zd?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 735c78b4-4914-4631-d6a2-08dc1dcdb402 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 17:47:30.6601 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a4MEB4jMfa04Hs+4+BFaWFG7/gPyFxsHuHJNGQUqucgZgGJKNkF6WzHzQGHUZmGa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8132 On Thu, Jan 25, 2024 at 09:23:00AM -0800, Nicolin Chen wrote: > > When the soft lockup issue is solved you can consider if a tunable is > > still interesting.. > > Yea, it would be on top of the soft lockup fix. I assume we are > still going with your change: arm_smmu_inv_range_too_big, though > I wonder if we should apply before your rework series to make it > a bug fix.. It depends what change you settle on.. > > > > Maybe it is really just a simple thing - compute how many invalidation > > > > commands are needed, if they don't all fit in the current queue space, > > > > then do an invalidate all instead? > > > > > > The queue could actually have a large space. But one large-size > > > invalidation would be divided into batches that have to execute > > > back-to-back. And the batch size is 64 commands in 64-bit case, > > > which might be too small as a cap. > > > > Yes, some notable code reorganizing would be needed to implement > > something like this > > > > Broadly I'd sketch sort of: > > > > - Figure out how fast the HW can execute a lot of commands > > - The above should drive some XX maximum number of commands, maybe we > > need to measure at boot, IDK > > - Strongly time bound SVA invalidation: > > * No more than XX commands, if more needed then push invalidate > > all > > * All commands must fit in the available queue space, if more > > needed then push invalidate all > > - The total queue depth must not be larger than YY based on the > > retire rate so that even a full queue will complete invalidation > > below the target time. > > > > A tunable indicating what the SVA time bound target should be might be > > appropriate.. > > Thanks for listing it out. I will draft something with that, and > should we just confine it to SVA or non DMA callers in general? Also, how much of this SVA issue is multithreaded? Will multiple command queues improve anything? Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2175C47258 for ; Thu, 25 Jan 2024 17:47:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=E7N/LuM6eY5IeERiCxb6cezww/UUQu78xSRhUtRix9U=; b=Jhpg5vdHeRlR3p PhvrY7qnZFfeUg9cXibGEPNL+4/6JzL0FP03QW2IBSfCj9RA+Jti0wYDWIIIumsZIzYbrhYfS9uCa Myjj6CCwoOW+75d0gLU3KRsxWVypo38U9tgnyF/uPeY448EUnH44XGwPYM8OhoDklnukuDBaxogiG MdmKcKPmDxuRnrLCw+l36gUOCIZGJUPtQVGjluJrZEjYlAA0lFyXoRPMzI/TsyIPCVm4OUGH1Rq8I Lc/JPqT23KP1pSSSQtvUefa+mAWLL1jU23jjG20U0/yKEg8uMXMxVE0e6h3rQxAbdPRsJ5nq3BANw 9nmv5OM51XPlxjb4WmGg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rT3pH-00000001CrU-0Stn; Thu, 25 Jan 2024 17:47:43 +0000 Received: from mail-mw2nam12on20601.outbound.protection.outlook.com ([2a01:111:f403:200a::601] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rT3pE-00000001Cps-0y8a for linux-arm-kernel@lists.infradead.org; Thu, 25 Jan 2024 17:47:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=M6EbbaRXXAr2aRmIhqgYbbNY03Y6J9QVeSYYwz/L+LaK0ZA40RBJbpFm4UsGobIhCidfLh0Fct8Yx7p67woKh8YDhCKWPWxSMiEmiWtX6eYUhEayoGhxgg/MRBVi+o+tumOFZlo0rKH+CVjvcN9AS7OPJwL6M8rvwCbtF3oiKb4clsL/xzwys8uk3LkqxEJj94Ehmd+/oPdqVyVCOIdKVTn71NuSq+HHj6FhQa5D4vtbTxR1Kv3MliWeWacGXuqTqNJRU0CPNRyyjlqd/IMp71qliwJw9qpiVMKf4MugU+EsRBk9bfYmt+YJo2LmDKAyh7s1la7GzJyEBxYT0RpPZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=PdAGPjO2Ukc9uHhaC6W7AP6OBCk0cg0VxKoUH4ln5Bs=; b=ku0d2bdQz5BaJiYm8lxkT7gIPRlQF5Z5TBdqhVJ/HMx2wDqeCuo5ksSZBx9ee21dEoIAldbMhSs4OBNPrA8vy/r/z0W+96W8tcto96fNVo95dncq9R60gOlEeYFGb7uhlgdqAoFX074hWkCqW3ECDxPJdS2SFP8ZAvuD/a0WNTI3xRKFVv2eVgToUb2JmWSwRWokQJ0qP+yGZ47b34NDkazT10yHrJSH3RWSvfSfFEJyDFTqZtVHy/uaAnRmwl3scgTqb4kfMYZeCpUwSJhajE2h44t8PxahyKJhkY6HfxroWDaxZhQaHBr1vvw33YGzjefpdgBub0QS+q+L+CYrog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PdAGPjO2Ukc9uHhaC6W7AP6OBCk0cg0VxKoUH4ln5Bs=; b=JGtW8UAhXzgo9WDCtKUjSSxaBV9aXJ5bi07f02dJVrbnot4kelE5c8kD9tJP+KpClXVrqypMkQYOyeCvpMhjBuQI1haQWRRZK8zdcYUwOvenh9B9REorsFd25+KvDxSeMc0B16owqAm18a4R1a3rE7Qh19GzlcKHbXHvDhI3bp5gZzJwwzjBSLLSs512oPUFUTQTXuLc+lwD0s3As/fJjoMWbeP9988qQTX5njbZ6ZwpNmq7xbxryQ9eYD42cIG0tXQfVmGRKbjz/KXxaMyP7aT7ZUGh1Sm3HfX+HFhEMu+oEgALbKg2oygXiqV29WC0WyOtXNtlCMc1lGali5AypA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SN7PR12MB8132.namprd12.prod.outlook.com (2603:10b6:806:321::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7202.32; Thu, 25 Jan 2024 17:47:30 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::96dd:1160:6472:9873%6]) with mapi id 15.20.7228.022; Thu, 25 Jan 2024 17:47:30 +0000 Date: Thu, 25 Jan 2024 13:47:28 -0400 From: Jason Gunthorpe To: Nicolin Chen Cc: "will@kernel.org" , Robin Murphy , "joro@8bytes.org" , "jean-philippe@linaro.org" , Alistair Popple , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "iommu@lists.linux.dev" Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in struct io_pgtable_cfg Message-ID: <20240125174728.GR1455070@nvidia.com> References: <61f9b371-7c45-26b1-ec0f-600765280c89@arm.com> <20240122130152.GP734935@nvidia.com> <20240122175700.GB1455070@nvidia.com> <20240125135537.GP1455070@nvidia.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: DS7PR07CA0019.namprd07.prod.outlook.com (2603:10b6:5:3af::7) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SN7PR12MB8132:EE_ X-MS-Office365-Filtering-Correlation-Id: 735c78b4-4914-4631-d6a2-08dc1dcdb402 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KOiLm1H/oKcpFYVLSFBeQSBAkpD9zqw3M9IOsPT59wppvSfx1YzcuBv94RN08BM+KZW2NTGZDsBD97vSPdm7xrchBHpoSEV9smlBYiz3Sq2ZpC1ncB/FwwK2UpMwDis8m/4fnJ85QRDYo/HpiPCfz1aSP4px7EJb8+5LRZ5U6Q4je93Yn7IM4G4vgdWZaWnFY0H86a+uGzl2fOWk/N8jm95rM1DGg9mIYKxf1Zb8QMA4TLzBmiUTEn79mVqjWXmrKwvqyrdORlE7kqqAyjNp4XJkSda2gcT5pSNlTQnCDl+ZLUxifuHoTQfkLXycnyGkix4Wgwd/vnXIACnP7qDhE/UP7lW0QG0RjnEhNP7ETafv99qa8iGOh4FNKyw65K8YmMTKodewBEfjNhBXQIRhvu3yno2wWogWoFA+7m7Pd/2uMYD5aQ94pGMeP+S/cnmT0to6gM0HijId9WXhjLP02+znuHB8R4dfIwMpwwmvN7qd9oWF9zjYVbyzEOmbYDUFCuY5jNXnMXw2WJn1DFZEOdHGaePSftcLfZ7HvPCCgB8hTdssd8oP+v4D6YX4XtHU X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(346002)(39860400002)(136003)(396003)(366004)(376002)(230922051799003)(186009)(451199024)(64100799003)(1800799012)(26005)(1076003)(478600001)(2616005)(83380400001)(6512007)(86362001)(38100700002)(66556008)(5660300002)(316002)(33656002)(8676002)(41300700001)(6486002)(66946007)(37006003)(8936002)(2906002)(6506007)(54906003)(36756003)(66476007)(6636002)(4326008)(6862004);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?7MMPUIiA7mjh2rPTRFBqx+5qjVHHF/bn3ei8n+ZTaAum9y221s1NnCSK+ueQ?= =?us-ascii?Q?u7RBxBjGnJiA8FHZoCQAZ5BRQ99s9lZWInezn1x5mxGycps6slJtA/yH4aFo?= =?us-ascii?Q?2Q1Q3TKnN2uIafZbN16bTUNqWhFnriY857v/NNZ0GGhjEDb84kO253twSuky?= =?us-ascii?Q?/W+0v5Bo16gkUhdswAy9xAp433KWCMbVZ1itaU4MyRn8A5HshameXskwmdAe?= =?us-ascii?Q?m1QKTNj7lsW0w99xSGvwY+vTWzS+av7PgrhenMZphfztxiI8zo/tvBuVcX0g?= =?us-ascii?Q?870xN/2/snP1sjo6kW4q2XS4+0+YXkk1cHLEyiIIQDKa5BySQydkeHj3obem?= =?us-ascii?Q?alJCByljrtim5Yd+Y+lB+RvzqlBGi8erNyaWYEs7udrrl+TuiffanLayaVWG?= =?us-ascii?Q?HrqEmXD7nHI4MuTJkeVQysRkxukpuvQcT7hWSz63CVKFbFgEG3PUsJWTWlpW?= =?us-ascii?Q?5s/xZnmqLhD5NiX+ox+9kf8omiqiXi/qgUdpWMvnB9veCX5PTlDwtRwfsdSG?= =?us-ascii?Q?N5C+ttg4SMR/YPbe3+e+lJdAy4YJb+83+q+ZLbrv1mspQLN631nPEz19M2lo?= =?us-ascii?Q?Nus6bh/O/WIgqtMIU9DCSE3DfJS8Ge6XJ5QaR7zsEFvWqgwydSS26EP9czwV?= =?us-ascii?Q?W3DauLL1f1C19JSCgLivvNbwga2oWfMYAjQvfXqitiHDuKzkAuyTwE46Pmw/?= =?us-ascii?Q?P+qnbvlX6l60LCksG72VPagBWP1sg5+fYjnpRYm0O2K+tuwH/P5q0eVh+YzJ?= =?us-ascii?Q?1qePUJdZjYFSRW8Pd1Q6GJVWeDPfpMwGRt+ESxnuRlMsSi8qVAtkhxNfDHVi?= =?us-ascii?Q?AIK4cMvEkKL54XkTBSXDsgtndtPLtKjgxU1t9ffzTSHJA26ubT8TNzUH8Fur?= =?us-ascii?Q?TAjYa7PLl48l93IVrG4ywyc/zQQfW8RVcHD+C05i0pPHFe3Rk5Lk+QbPZyX1?= =?us-ascii?Q?TeMBG3z/f0014emFCoLb/BX1HsRmEC0I4v9gtB5BigVSH+aLwsPzqtuBegVn?= =?us-ascii?Q?vdrRBLlfKDjUXfZ3NBabX4YlQpL8SZz6KCwPBza0ZEzHPF//ZEWqqDafABkn?= =?us-ascii?Q?k17QydQytug78QqyA2cgKG+ppEK2WB2Od7NPVmaOBjK5lrjSSSP36t6rHPnS?= =?us-ascii?Q?ZmAtB10W+W2Ps1/IfpUA9yRsaL3oMPeRjB6gNTuKRKGM/On/Jk0QmIAnEe5m?= =?us-ascii?Q?1oXzzFSMWXuaG/1Q698969X+EtW2rU9rBb7n98X5y7agodjs7HYuVObZy4Hr?= =?us-ascii?Q?Ot7rxXHDuOMRjYtST/diciUAQGffqzDnj1f27O8TBNYtYUBSgJRAOWVfxCV3?= =?us-ascii?Q?ptmtIdtVKdBK4CFEunxc74hArM1lE2FvpM8ySeOMxsoZ/UPGbFosXsjDSIlC?= =?us-ascii?Q?Sc+sTS0gzmG2Iaxr5dnYIz9gfo6uVvpEqgVoDetyzS7Q8HMKdG7lD34hxDCe?= =?us-ascii?Q?On59xOvb0DnIQz/Gcqlqho3PnhF1OgD+3vyI/XSSy44ZW7DJ3bUkl4Ehh6qU?= =?us-ascii?Q?OswmtsMmdJjHYIt/QKoQ06nXXhTbJnY7LLLjzpvzTa3s9U9wnigAcWWbftSv?= =?us-ascii?Q?fau3sj07xNEH0MfJm6vD9nCY9HNSNCdQN0SDk+Zd?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 735c78b4-4914-4631-d6a2-08dc1dcdb402 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 17:47:30.6601 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: a4MEB4jMfa04Hs+4+BFaWFG7/gPyFxsHuHJNGQUqucgZgGJKNkF6WzHzQGHUZmGa X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN7PR12MB8132 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240125_094740_351956_2BB2F27E X-CRM114-Status: GOOD ( 28.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Jan 25, 2024 at 09:23:00AM -0800, Nicolin Chen wrote: > > When the soft lockup issue is solved you can consider if a tunable is > > still interesting.. > > Yea, it would be on top of the soft lockup fix. I assume we are > still going with your change: arm_smmu_inv_range_too_big, though > I wonder if we should apply before your rework series to make it > a bug fix.. It depends what change you settle on.. > > > > Maybe it is really just a simple thing - compute how many invalidation > > > > commands are needed, if they don't all fit in the current queue space, > > > > then do an invalidate all instead? > > > > > > The queue could actually have a large space. But one large-size > > > invalidation would be divided into batches that have to execute > > > back-to-back. And the batch size is 64 commands in 64-bit case, > > > which might be too small as a cap. > > > > Yes, some notable code reorganizing would be needed to implement > > something like this > > > > Broadly I'd sketch sort of: > > > > - Figure out how fast the HW can execute a lot of commands > > - The above should drive some XX maximum number of commands, maybe we > > need to measure at boot, IDK > > - Strongly time bound SVA invalidation: > > * No more than XX commands, if more needed then push invalidate > > all > > * All commands must fit in the available queue space, if more > > needed then push invalidate all > > - The total queue depth must not be larger than YY based on the > > retire rate so that even a full queue will complete invalidation > > below the target time. > > > > A tunable indicating what the SVA time bound target should be might be > > appropriate.. > > Thanks for listing it out. I will draft something with that, and > should we just confine it to SVA or non DMA callers in general? Also, how much of this SVA issue is multithreaded? Will multiple command queues improve anything? Jason _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel