From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.7]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2330B200B2 for ; Fri, 24 Nov 2023 12:01:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=linux.intel.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=intel.com header.i=@intel.com header.b="EGp9mrNJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1700827313; x=1732363313; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=vnwBM3tljMEcetDptjzZqGvpgTMc7X/b1g4fgPwK904=; b=EGp9mrNJRFFZ0N98qfBCtLuInT5UwMTSdRp4Qjq5eai2JVnTKDaZ0d7b mahw7JtTTyGNELLLzaHnq6JqKBHJEnioqJeOC34iGHfAU7JURj80JMuzl mv+p/XVsLhqIjAW7PcdVU3eMQKxQc1BNzyXarkM2aZ2GH6OO+ZFBhSJVu RaPB+DDUKlbHks1Ru5ZBTiIl7m0FRbmg+8uYLicBAoP2AarF2kbtwoCGz KPDHFC8808ONB1dGxbA0Dg5pBLB047ESxxuyXeUuGEWkopEyN1+i9U2Z4 Swj3dyJF8zenZ6vSxSo8O6RKPud1uJTngoZ2ku7ai2ueG2o5C7+4Dc13K g==; X-IronPort-AV: E=McAfee;i="6600,9927,10902"; a="13967943" X-IronPort-AV: E=Sophos;i="6.04,224,1695711600"; d="scan'208";a="13967943" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmvoesa101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Nov 2023 04:01:51 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10902"; a="838068621" X-IronPort-AV: E=Sophos;i="6.04,224,1695711600"; d="scan'208";a="838068621" Received: from moqiongz-mobl.ccr.corp.intel.com (HELO [10.255.28.60]) ([10.255.28.60]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Nov 2023 04:01:47 -0800 Message-ID: <7a683525-07ca-4ff1-97bd-0193d07dc857@linux.intel.com> Date: Fri, 24 Nov 2023 20:01:45 +0800 Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Cc: baolu.lu@linux.intel.com, Yi Liu , Jacob Pan , Yan Zhao , iommu@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v7 00/12] iommu: Prepare to deliver page faults to user space To: liulongfang , Joerg Roedel , Will Deacon , Robin Murphy , Jason Gunthorpe , Kevin Tian , Jean-Philippe Brucker , Nicolin Chen References: <20231115030226.16700-1-baolu.lu@linux.intel.com> <1a029033-3c9e-aeab-06bf-1e7020c2bc7d@huawei.com> Content-Language: en-US From: Baolu Lu In-Reply-To: <1a029033-3c9e-aeab-06bf-1e7020c2bc7d@huawei.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2023/11/24 14:30, liulongfang wrote: > On 2023/11/15 11:02, Lu Baolu Wrote: >> When a user-managed page table is attached to an IOMMU, it is necessary >> to deliver IO page faults to user space so that they can be handled >> appropriately. One use case for this is nested translation, which is >> currently being discussed in the mailing list. >> >> I have posted a RFC series [1] that describes the implementation of >> delivering page faults to user space through IOMMUFD. This series has >> received several comments on the IOMMU refactoring, which I am trying to >> address in this series. >> >> The major refactoring includes: >> >> - [PATCH 01 ~ 04] Move include/uapi/linux/iommu.h to >> include/linux/iommu.h. Remove the unrecoverable fault data definition. >> - [PATCH 05 ~ 06] Remove iommu_[un]register_device_fault_handler(). >> - [PATCH 07 ~ 10] Separate SVA and IOPF. Make IOPF a generic page fault >> handling framework. >> - [PATCH 11 ~ 12] Improve iopf framework for iommufd use. >> >> This is also available at github [2]. >> >> [1] https://lore.kernel.org/linux-iommu/20230530053724.232765-1-baolu.lu@linux.intel.com/ >> [2] https://github.com/LuBaolu/intel-iommu/commits/preparatory-io-pgfault-delivery-v7 >> >> Change log: >> v7: >> - Rebase to v6.7-rc1. >> - Export iopf_group_response() for global use. >> - Release lock when calling iopf handler. >> - The whole series has been verified to work for SVA case on Intel >> platforms by Zhao Yan. Add her Tested-by to affected patches. >> >> v6: https://lore.kernel.org/linux-iommu/20230928042734.16134-1-baolu.lu@linux.intel.com/ >> - [PATCH 09/12] Check IS_ERR() against the iommu domain. [Jingqi/Jason] >> - [PATCH 12/12] Rename the comments and name of iopf_queue_flush_dev(), >> no functionality changes. [Kevin] >> - All patches rebased on the latest iommu/core branch. >> >> v5: https://lore.kernel.org/linux-iommu/20230914085638.17307-1-baolu.lu@linux.intel.com/ >> - Consolidate per-device fault data management. (New patch 11) >> - Improve iopf_queue_flush_dev(). (New patch 12) >> >> v4: https://lore.kernel.org/linux-iommu/20230825023026.132919-1-baolu.lu@linux.intel.com/ >> - Merge iommu_fault_event and iopf_fault. They are duplicate. >> - Move iommu_report_device_fault() and iommu_page_response() to >> io-pgfault.c. >> - Move iommu_sva_domain_alloc() to iommu-sva.c. >> - Add group->domain and use it directly in sva fault handler. >> - Misc code refactoring and refining. >> >> v3: https://lore.kernel.org/linux-iommu/20230817234047.195194-1-baolu.lu@linux.intel.com/ >> - Convert the fault data structures from uAPI to kAPI. >> - Merge iopf_device_param into iommu_fault_param. >> - Add debugging on domain lifetime for iopf. >> - Remove patch "iommu: Change the return value of dev_iommu_get()". >> - Remove patch "iommu: Add helper to set iopf handler for domain". >> - Misc code refactoring and refining. >> >> v2: https://lore.kernel.org/linux-iommu/20230727054837.147050-1-baolu.lu@linux.intel.com/ >> - Remove unrecoverable fault data definition as suggested by Kevin. >> - Drop the per-device fault cookie code considering that doesn't make >> much sense for SVA. >> - Make the IOMMU page fault handling framework generic. So that it can >> available for use cases other than SVA. >> >> v1: https://lore.kernel.org/linux-iommu/20230711010642.19707-1-baolu.lu@linux.intel.com/ >> >> Lu Baolu (12): >> iommu: Move iommu fault data to linux/iommu.h >> iommu/arm-smmu-v3: Remove unrecoverable faults reporting >> iommu: Remove unrecoverable fault data >> iommu: Cleanup iopf data structure definitions >> iommu: Merge iopf_device_param into iommu_fault_param >> iommu: Remove iommu_[un]register_device_fault_handler() >> iommu: Merge iommu_fault_event and iopf_fault >> iommu: Prepare for separating SVA and IOPF >> iommu: Make iommu_queue_iopf() more generic >> iommu: Separate SVA and IOPF >> iommu: Consolidate per-device fault data management >> iommu: Improve iopf_queue_flush_dev() >> >> include/linux/iommu.h | 266 +++++++--- >> drivers/iommu/intel/iommu.h | 2 +- >> drivers/iommu/iommu-sva.h | 71 --- >> include/uapi/linux/iommu.h | 161 ------ >> .../iommu/arm/arm-smmu-v3/arm-smmu-v3-sva.c | 14 +- >> drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 51 +- >> drivers/iommu/intel/iommu.c | 25 +- >> drivers/iommu/intel/svm.c | 8 +- >> drivers/iommu/io-pgfault.c | 469 ++++++++++++------ >> drivers/iommu/iommu-sva.c | 66 ++- >> drivers/iommu/iommu.c | 232 --------- >> MAINTAINERS | 1 - >> drivers/iommu/Kconfig | 4 + >> drivers/iommu/Makefile | 3 +- >> drivers/iommu/intel/Kconfig | 1 + >> 15 files changed, 601 insertions(+), 773 deletions(-) >> delete mode 100644 drivers/iommu/iommu-sva.h >> delete mode 100644 include/uapi/linux/iommu.h >> > > Tested-By: Longfang Liu Thank you for the testing. > > The Arm SVA mode based on HiSilicon crypto accelerator completed the functional test > and performance test of page fault scenarios. > 1. The IOMMU page fault processing function is normal. > 2. Performance test on 128 core ARM platform. performance is reduced: > > Threads Performance > 8 -0.77% > 16 -1.1% > 32 -0.31% > 64 -0.49% > 128 -0.72% > 256 -1.7% > 384 -4.94% > 512 NA（iopf timeout） > > Finally, continuing to increase the number of threads will cause iommu's page fault > processing to time out(more than 4.2 seconds). > This problem occurs both in the before version(kernel6.7-rc1) and > in the after modification's version. Probably you can check whether commit 6bbd42e2df8f ("mmu_notifiers: call invalidate_range() when invalidating TLBs") matters. It was discussed in this thread. https://lore.kernel.org/linux-iommu/20231117090933.75267-1-baolu.lu@linux.intel.com/ Best regards, baolu