From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4DB95C47258 for ; Thu, 25 Jan 2024 19:56:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=bHKzWT0q29UBJBVkWoqH7S60XxhJK4pDfcMUAJWTMCw=; b=mdoNtt/YztI/Pc fK6TMvAS2DRB4yYwlqpCW1ke1g0u/4fkJGOLrFY2gHoF8KmWaqLpWf9vzyzucK1teeJ9ifDuSwWor AbZzzDAp517S6T1r9MvZ+zvCTIFgjx8AcAnQA2u2yfMNdbiX3eAGrMESMTapWIdLix5M16zSc3hEE 6eytVJsuG7X+tnucJGLfBmO2p7pm91yNwThhZ8ibnUkZfjMuDaufYlIP0piI8OmGNXLaYiyLM05Lu pJuJqI7JB03n4lunHyUgbYhiVD/UEN3GBCioEaHAXm/7Uh8730KB53BBopridqfx1nbg9kT+lp5OL vOEMt79ee6HW/H4winJQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rT5pl-00000001fMf-3QwR; Thu, 25 Jan 2024 19:56:21 +0000 Received: from mail-dm6nam12on20600.outbound.protection.outlook.com ([2a01:111:f403:2417::600] helo=NAM12-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rT5pi-00000001fM6-2tWg for linux-arm-kernel@lists.infradead.org; Thu, 25 Jan 2024 19:56:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bIYPZR7FQhqz/aW2wpNXnQPoXENQNXbTFz6UbuH7EJ0XMdVdKcthzL8ko3Sfc93ROiG6B7eVANVgvgScK/KsZWYTKSBlLvt4kWPbiJAL19HMn7flZ1wIcoQYqR8JbmxP68QYSuwlp0M5U2zmkk8snJP4f5J9MqR8TBfLOj+wfFOEEdJ7fX/tIQeQcVvHEJF1TEZFvXED5Uan8vwGG+a5Z5ONHDqdmUVPuWfVN9VzZP5LfFbSWulACpAQC0WV1/167tlZO+woj1Ufd2n3X3AfkuuN8ENejnJFgBL5CfpHuJs0mNFi2wAfJurzKol8VKGZ3+3XTu2fbr14cZuPEdHN2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=KhZqUncc1hO7qGO0tinPjv4U9XLEdukstuowcIhOXho=; b=YYQBJiWBXkLzlzA2e2Lq9uuo21eYy5MEhCQQgjTc/ln4NU5CfKQ5QygU4+EUPAkb0MkHE7K0ZQZ517Cf9uzV1dVhMpPv9H4w6+0FfmBSUnf2BgMQxMDiyTGiXjzuFQiTQt6j3sFCgAsvaSQsqbWueMdckKkB5B4kDOZMM6lKEhhI7SuU9pyv4LsioIDwnxiPPDIp/vvpudNGcou4hy/pax01lhLofWGQcOYkReZ3hVP1bzUj/Gq/La03WpBNboalq+tC35n0ObDkZolCOKb9UOWPUTCDEGvlZmEDWH7FFTnAwcRy3QZHkM3r3zZ1pUjMXoPgx4+QMbekzWwWE7Q5tg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.117.160) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KhZqUncc1hO7qGO0tinPjv4U9XLEdukstuowcIhOXho=; b=uaFbojnD+JxfGL5svBY538vwvw6KOKbJRgQwx4GFCZnUk5Jzx4+NZx8pBrX3OqtehniuZFWSz1wzdeoV3erkRjvXbpGQh8Hc68TpfY9V1gvliPg3ti9fW0IMFZ8OueSlExTYzeukIYDlT4LaQRzZgEY7Zhs+h8LVmi7r6DrXapiRpS0obshwQdtNQ8crRpW9szm4Ua1MyU3UTB/ogX8DxF6kklCtk/JMR/bY9PBe8NibBIqKcwVB2y6DkCOWqA9Yv4xSfHIXBAedVvgE3hmMA4IaQOm6WTsSUXv2INIPJpBchzrsQ1sjW/hYFsxhON95DEgLoY7GpFSwRWfGcWPU2w== Received: from MN2PR16CA0064.namprd16.prod.outlook.com (2603:10b6:208:234::33) by IA0PR12MB8837.namprd12.prod.outlook.com (2603:10b6:208:491::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.27; Thu, 25 Jan 2024 19:56:11 +0000 Received: from BL6PEPF0001AB52.namprd02.prod.outlook.com (2603:10b6:208:234:cafe::d6) by MN2PR16CA0064.outlook.office365.com (2603:10b6:208:234::33) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.27 via Frontend Transport; Thu, 25 Jan 2024 19:56:11 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.117.160) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.117.160 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.117.160; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.117.160) by BL6PEPF0001AB52.mail.protection.outlook.com (10.167.241.4) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7181.14 via Frontend Transport; Thu, 25 Jan 2024 19:56:11 +0000 Received: from rnnvmail205.nvidia.com (10.129.68.10) by mail.nvidia.com (10.129.200.66) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 25 Jan 2024 11:55:54 -0800 Received: from rnnvmail204.nvidia.com (10.129.68.6) by rnnvmail205.nvidia.com (10.129.68.10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41; Thu, 25 Jan 2024 11:55:53 -0800 Received: from Asurada-Nvidia (10.127.8.9) by mail.nvidia.com (10.129.68.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.41 via Frontend Transport; Thu, 25 Jan 2024 11:55:53 -0800 Date: Thu, 25 Jan 2024 11:55:51 -0800 From: Nicolin Chen To: Jason Gunthorpe CC: "will@kernel.org" , Robin Murphy , "joro@8bytes.org" , "jean-philippe@linaro.org" , Alistair Popple , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "iommu@lists.linux.dev" Subject: Re: [PATCH 1/3] iommu/io-pgtable-arm: Add nents_per_pgtable in struct io_pgtable_cfg Message-ID: References: <20240122130152.GP734935@nvidia.com> <20240122175700.GB1455070@nvidia.com> <20240125135537.GP1455070@nvidia.com> <20240125174728.GR1455070@nvidia.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20240125174728.GR1455070@nvidia.com> X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL6PEPF0001AB52:EE_|IA0PR12MB8837:EE_ X-MS-Office365-Filtering-Correlation-Id: ff578901-742e-4b87-fcb6-08dc1ddfae1a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: w07oHQ/T6ovrx05gMbFdQqCwrIgV8PfmBmK6rQxhtBbbASrtD2ZWG2NWXK2jriUeK9sgBXIDHO1qbDYtDF7qux6Ww3IuzI5gWbgt0QnRXwTni2QuOwo5oBb8+4s0LABCxwhXXqfEOTLT5fMtJR4NOuMsM1p+DIV/cNk6nNo1Ys/1I762YUBGD2qFG/wT29AxbnPhMw/FW09XNPflnd0Ui7Z+GWLTAClCDjlQA5ZJxcQMIf4NfhYQw+kc+eZcPAkNscgl5kKXtj2wTOoPeSHNJLAsoUhOp8ixJsMrx+eQQR7yWYRbK2QeS/PfkTi5OZgMxZBA1+un2bvSAUQ2hqAAmuza+aTt2ukWXLT0wOzqTjRBoGpB1Rie+YRUdCvwuGXBZ+YNbi2LDKVObX7Kh7lCQAYmmeVoAIcn5xeYkxYv586a8GN+gvZWGLrKm23dpEKZiib/T/OvZOFc2SsqNdE5q0M9qj2h+GW0sFgpkiy7nRy6HgKUXso235BmeN9HyD6gNDRvYL7gOy8NaBxthKgd77f8Bi+lqsiyVSEPwHtjJEy3ieZSgFbaGhbt0YvW9mI56Z9D/BfQeuA+Tcsswt3cL8MMEAhLWJzYb2J7fWOBexLurBKAy1l5eCDo/HXx017Kto+129RBgY2+xDzjyNbqUxmeT1JtsIXeiGWV2ASYa4ETJa1iljka+NtJy0EF6amL+eqdxcuJGyfun9dKoLVRYKUE04JVzg0wMq0k7Cl8t723SitMFPuBicw59CqDrw0j X-Forefront-Antispam-Report: CIP:216.228.117.160;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc6edge1.nvidia.com;CAT:NONE;SFS:(13230031)(4636009)(396003)(136003)(376002)(39860400002)(346002)(230922051799003)(186009)(64100799003)(82310400011)(451199024)(1800799012)(36840700001)(46966006)(40470700004)(8676002)(6862004)(4326008)(8936002)(316002)(70586007)(2906002)(5660300002)(86362001)(70206006)(6636002)(36860700001)(47076005)(7636003)(356005)(54906003)(82740400003)(9686003)(478600001)(83380400001)(26005)(426003)(336012)(41300700001)(33716001)(40480700001)(40460700003)(55016003);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 19:56:11.3653 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ff578901-742e-4b87-fcb6-08dc1ddfae1a X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.117.160];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BL6PEPF0001AB52.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8837 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240125_115618_795658_0CAF12D5 X-CRM114-Status: GOOD ( 34.38 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Jan 25, 2024 at 01:47:28PM -0400, Jason Gunthorpe wrote: > On Thu, Jan 25, 2024 at 09:23:00AM -0800, Nicolin Chen wrote: > > > When the soft lockup issue is solved you can consider if a tunable is > > > still interesting.. > > > > Yea, it would be on top of the soft lockup fix. I assume we are > > still going with your change: arm_smmu_inv_range_too_big, though > > I wonder if we should apply before your rework series to make it > > a bug fix.. > > It depends what change you settle on.. I mean your arm_smmu_inv_range_too_big patch. Should it be a bug fix CCing the stable tree? My previous SVA fix was, by the way. > > > > > Maybe it is really just a simple thing - compute how many invalidation > > > > > commands are needed, if they don't all fit in the current queue space, > > > > > then do an invalidate all instead? > > > > > > > > The queue could actually have a large space. But one large-size > > > > invalidation would be divided into batches that have to execute > > > > back-to-back. And the batch size is 64 commands in 64-bit case, > > > > which might be too small as a cap. > > > > > > Yes, some notable code reorganizing would be needed to implement > > > something like this > > > > > > Broadly I'd sketch sort of: > > > > > > - Figure out how fast the HW can execute a lot of commands > > > - The above should drive some XX maximum number of commands, maybe we > > > need to measure at boot, IDK > > > - Strongly time bound SVA invalidation: > > > * No more than XX commands, if more needed then push invalidate > > > all > > > * All commands must fit in the available queue space, if more > > > needed then push invalidate all > > > - The total queue depth must not be larger than YY based on the > > > retire rate so that even a full queue will complete invalidation > > > below the target time. > > > > > > A tunable indicating what the SVA time bound target should be might be > > > appropriate.. > > > > Thanks for listing it out. I will draft something with that, and > > should we just confine it to SVA or non DMA callers in general? > > Also, how much of this SVA issue is multithreaded? Will multiple > command queues improve anything? The bottleneck from measurement is mostly at SMMU consuming the commands with a single CMDQ HW, so multithreading unlikely helps. And VCMDQ only provides a multi-queue interface/wrapper for VM isolations. Thanks Nic _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel