From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58325C61DF4 for ; Fri, 24 Nov 2023 12:01:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230392AbjKXMBs (ORCPT ); Fri, 24 Nov 2023 07:01:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229580AbjKXMBq (ORCPT ); Fri, 24 Nov 2023 07:01:46 -0500 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C79410CB; Fri, 24 Nov 2023 04:01:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1700827312; x=1732363312; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=vnwBM3tljMEcetDptjzZqGvpgTMc7X/b1g4fgPwK904=; b=Uj0oenYzTYiL9qG9CwElg20KuwOHnXMOXWFtWVjVrBoIdMDJafJXrwMI 4pOaP4o4A2ET155mlCNltecOPRsUb7gO2rvscRWGrefCvh0lsiFG8A9T2 OMON5UNZ7sJMSZajn37Vu9QuvVqBvHnnVcumvSsghrsZRGjENWIbKbCcg KoWWIqI6WQzQOElvGj9Psi9gR4UOD6z0Y8x5C84z4aaRpugtIfmzandOv IV8aZhqSpjskZSDvXmFYAO9V0mfz9SRntziEQnH4c3LOh+u9yyoiTTjC4 wxCUsWP0NwWcr/6IB5dhvffb7dS50X9yURf35Rt2PPMczQg+Rzn4L3MNR A==; X-IronPort-AV: E=McAfee;i="6600,9927,10902"; a="13967944" X-IronPort-AV: E=Sophos;i="6.04,224,1695711600"; d="scan'208";a="13967944" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Nov 2023 04:01:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10902"; a="838068621" X-IronPort-AV: E=Sophos;i="6.04,224,1695711600"; d="scan'208";a="838068621" Received: from moqiongz-mobl.ccr.corp.intel.com (HELO [10.255.28.60]) ([10.255.28.60]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Nov 2023 04:01:47 -0800 Message-ID: <7a683525-07ca-4ff1-97bd-0193d07dc857@linux.intel.com> Date: Fri, 24 Nov 2023 20:01:45 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: baolu.lu@linux.intel.com, Yi Liu , Jacob Pan , Yan Zhao , iommu@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v7 00/12] iommu: Prepare to deliver page faults to user space To: liulongfang , Joerg Roedel , Will Deacon , Robin Murphy , Jason Gunthorpe , Kevin Tian , Jean-Philippe Brucker , Nicolin Chen References: <20231115030226.16700-1-baolu.lu@linux.intel.com> <1a029033-3c9e-aeab-06bf-1e7020c2bc7d@huawei.com> Content-Language: en-US From: Baolu Lu In-Reply-To: <1a029033-3c9e-aeab-06bf-1e7020c2bc7d@huawei.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2023/11/24 14:30, liulongfang wrote: > On 2023/11/15 11:02, Lu Baolu Wrote: >> When a user-managed page table is attached to an IOMMU, it is necessary >> to deliver IO page faults to user space so that they can be handled >> appropriately. One use case for this is nested translation, which is >> currently being discussed in the mailing list. >> >> I have posted a RFC series [1] that describes the implementation of >> delivering page faults to user space through IOMMUFD. This series has >> received several comments on the IOMMU refactoring, which I am trying to >> address in this series. >> >> The major refactoring includes: >> >> - [PATCH 01 ~ 04] Move include/uapi/linux/iommu.h to >> include/linux/iommu.h. Remove the unrecoverable fault data definition. >> - [PATCH 05 ~ 06] Remove iommu_[un]register_device_fault_handler(). >> - [PATCH 07 ~ 10] Separate SVA and IOPF. Make IOPF a generic page fault >> handling framework. >> - [PATCH 11 ~ 12] Improve iopf framework for iommufd use. >> >> This is also available at github [2]. >> >> [1] https://lore.kernel.org/linux-iommu/20230530053724.232765-1-baolu.lu@linux.intel.com/ >> [2] https://github.com/LuBaolu/intel-iommu/commits/preparatory-io-pgfault-delivery-v7 >> >> Change log: >> v7: >> - Rebase to v6.7-rc1. >> - Export iopf_group_response() for global use. >> - Release lock when calling iopf handler. >> - The whole series has been verified to work for SVA case on Intel >> platforms by Zhao Yan. Add her Tested-by to affected patches. >> >> v6: https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ >> - [PATCH 09/12] Check IS_ERR() against the iommu domain. [Jingqi/Jason] >> - [PATCH 12/12] Rename the comments and name of iopf_queue_flush_dev(), >> no functionality changes. [Kevin] >> - All patches rebased on the latest iommu/core branch. >> >> v5: https://lore.kernel.org/linux-iommu/20230914085638.17307-1-baolu.lu@linux.intel.com/ >> - Consolidate per-device fault data management. (New patch 11) >> - Improve iopf_queue_flush_dev(). (New patch 12) >> >> v4: https://lore.kernel.org/linux-iommu/20230825023026.132919-1-baolu.lu@linux.intel.com/ >> - Merge iommu_fault_event and iopf_fault. They are duplicate. >> - Move iommu_report_device_fault() and iommu_page_response() to >> io-pgfault.c. >> - Move iommu_sva_domain_alloc() to iommu-sva.c. >> - Add group->domain and use it directly in sva fault handler. >> - Misc code refactoring and refining. >> >> v3: https://lore.kernel.org/linux-iommu/20230817234047.195194-1-baolu.lu@linux.intel.com/ >> - Convert the fault data structures from uAPI to kAPI. >> - Merge iopf_device_param into iommu_fault_param. >> - Add debugging on domain lifetime for iopf. >> - Remove patch "iommu: Change the return value of dev_iommu_get()". >> - Remove patch "iommu: Add helper to set iopf handler for domain". >> - Misc code refactoring and refining. >> >> v2: https://lore.kernel.org/linux-iommu/20230727054837.147050-1-baolu.lu@linux.intel.com/ >> - Remove unrecoverable fault data definition as suggested by Kevin. >> - Drop the per-device fault cookie code considering that doesn't make >> much sense for SVA. >> - Make the IOMMU page fault handling framework generic. So that it can >> available for use cases other than SVA. >> >> v1: https://lore.kernel.org/linux-iommu/20230711010642.19707-1-baolu.lu@linux.intel.com/ >> >> Lu Baolu (12): >> iommu: Move iommu fault data to linux/iommu.h >> iommu/arm-smmu-v3: Remove unrecoverable faults reporting >> iommu: Remove unrecoverable fault data >> iommu: Cleanup iopf data structure definitions >> iommu: Merge iopf_device_param into iommu_fault_param >> iommu: Remove iommu_[un]register_device_fault_handler() >> iommu: Merge iommu_fault_event and iopf_fault >> iommu: Prepare for separating SVA and IOPF >> iommu: Make iommu_queue_iopf() more generic >> iommu: Separate SVA and IOPF >> iommu: Consolidate per-device fault data management >> iommu: Improve iopf_queue_flush_dev() >> >> include/linux/iommu.h | 266 +++++++--- >> drivers/iommu/intel/iommu.h | 2 +- >> drivers/iommu/iommu-sva.h | 71 --- >> include/uapi/linux/iommu.h | 161 ------ >> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 14 +- >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 51 +- >> drivers/iommu/intel/iommu.c | 25 +- >> drivers/iommu/intel/svm.c | 8 +- >> drivers/iommu/io-pgfault.c | 469 ++++++++++++------ >> drivers/iommu/iommu-sva.c | 66 ++- >> drivers/iommu/iommu.c | 232 --------- >> MAINTAINERS | 1 - >> drivers/iommu/Kconfig | 4 + >> drivers/iommu/Makefile | 3 +- >> drivers/iommu/intel/Kconfig | 1 + >> 15 files changed, 601 insertions(+), 773 deletions(-) >> delete mode 100644 drivers/iommu/iommu-sva.h >> delete mode 100644 include/uapi/linux/iommu.h >> > > Tested-By: Longfang Liu Thank you for the testing. > > The Arm SVA mode based on HiSilicon crypto accelerator completed the functional test > and performance test of page fault scenarios. > 1. The IOMMU page fault processing function is normal. > 2. Performance test on 128 core ARM platform. performance is reduced: > > Threads Performance > 8 -0.77% > 16 -1.1% > 32 -0.31% > 64 -0.49% > 128 -0.72% > 256 -1.7% > 384 -4.94% > 512 NA（iopf timeout） > > Finally, continuing to increase the number of threads will cause iommu's page fault > processing to time out(more than 4.2 seconds). > This problem occurs both in the before version(kernel6.7-rc1) and > in the after modification's version. Probably you can check whether commit 6bbd42e2df8f ("mmu_notifiers: call invalidate_range() when invalidating TLBs") matters. It was discussed in this thread. https://lore.kernel.org/linux-iommu/20231117090933.75267-1-baolu.lu@linux.intel.com/ Best regards, baolu