From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68B6B7C for ; Wed, 17 Aug 2022 07:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1660720861; x=1692256861; h=message-id:date:mime-version:cc:subject:to:references: from:in-reply-to:content-transfer-encoding; bh=rzGWskO+7SbZcBTwGa/L5Pm0RSFGECEGooXJW/tWynE=; b=QckWkhXnpLsw6jDwS8aGRlozv3TeY5m1hEIFciDq9zlJmRNtO5AeIUk9 XDYpV53DpT+g29r/XEEIi0V1gEEk2bM9tPjChfbA4uq2Lu1cKgVSCgoU+ smvGilfe9UIZp0BHt55A89EoAipGR0LBa/tH7dDNMw0SGU+jYOOglXO+G VvPTg0gJTUEKUkFMwftp/j3rHNncMSeMW1+236GLAH69CDj0yU59rQmfl OawX87KuQq11vmtNJR83XslcN/d4eMRWqv+Bd40HqhT29kfmn4TaLImgt 2noZWwWInclaYx1F1YSWqnN4xfDgOdQ7RSeS9nwFvyWvA0JJ3UgPf+UJA A==; X-IronPort-AV: E=McAfee;i="6400,9594,10441"; a="293700414" X-IronPort-AV: E=Sophos;i="5.93,242,1654585200"; d="scan'208";a="293700414" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Aug 2022 00:20:52 -0700 X-IronPort-AV: E=Sophos;i="5.93,242,1654585200"; d="scan'208";a="667487039" Received: from jzhan60-mobl1.ccr.corp.intel.com (HELO [10.254.209.228]) ([10.254.209.228]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Aug 2022 00:20:48 -0700 Message-ID: <37ba5f53-079f-c04b-5c3e-678c87bbe0ff@linux.intel.com> Date: Wed, 17 Aug 2022 15:20:46 +0800 Precedence: bulk X-Mailing-List: iommu@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.11.0 Cc: baolu.lu@linux.intel.com, Bart Van Assche , Sasha Levin , David Woodhouse , Joerg Roedel , iommu@lists.linux.dev, Will Deacon , Robin Murphy , Kevin Tian , Ashok Raj , Christoph Hellwig , Jason Gunthorpe , Liu Yi L , Jacob jun Pan , linux-kernel@vger.kernel.org, Scarlett Gourley , James Sewart , Jack O'Sullivan Subject: Re: lockdep splat due to klist iteration from atomic context in Intel IOMMU driver Content-Language: en-US To: Lennert Buytenhek References: <94921c7a-778c-911d-4d72-d7815d95c46b@linux.intel.com> From: Baolu Lu In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 2022/8/17 14:09, Lennert Buytenhek wrote: > On Wed, Aug 17, 2022 at 12:04:10PM +0800, Baolu Lu wrote: > >>>>> On a build of 7ebfc85e2cd7 ("Merge tag 'net-6.0-rc1' of >>>>> git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net"), with >>>>> CONFIG_INTEL_IOMMU_DEBUGFS enabled, I am seeing the lockdep splat >>>>> below when an I/O page fault occurs on a machine with an Intel >>>>> IOMMU in it. >>>>> >>>>> The issue seems to be the klist iterator functions using >>>>> spin_*lock_irq*() but the klist insertion functions using >>>>> spin_*lock(), combined with the Intel DMAR IOMMU driver iterating >>>>> over klists from atomic (hardirq) context as of commit 8ac0b64b9735 >>>>> ("iommu/vt-d: Use pci_get_domain_bus_and_slot() in pgtable_walk()") >>>>> when CONFIG_INTEL_IOMMU_DEBUGFS is enabled, where >>>>> pci_get_domain_bus_and_slot() calls into bus_find_device() which >>>>> iterates over klists. >>>>> >>>>> I found this commit from 2018: >>>>> >>>>> commit 624fa7790f80575a4ec28fbdb2034097dc18d051 >>>>> Author: Bart Van Assche >>>>> Date: Fri Jun 22 14:54:49 2018 -0700 >>>>> >>>>> scsi: klist: Make it safe to use klists in atomic context >>>>> >>>>> This commit switched lib/klist.c:klist_{prev,next} from >>>>> spin_{,un}lock() to spin_{lock_irqsave,unlock_irqrestore}(), but left >>>>> the spin_{,un}lock() calls in add_{head,tail}() untouched. >>>>> >>>>> The simplest fix for this would be to switch lib/klist.c:add_{head,tail}() >>>>> over to use the IRQ-safe spinlock variants as well? >>>> Another possibility would be to evaluate whether it is safe to revert commit >>>> 624fa7790f80 ("scsi: klist: Make it safe to use klists in atomic context"). >>>> That commit is no longer needed by the SRP transport driver since the legacy >>>> block layer has been removed from the kernel. >>> And then to fix the 6.0-rc1 iommu/vt-d lockdep splat with >>> CONFIG_INTEL_IOMMU_DEBUGFS enabled, we could convert the Intel DMAR >>> IRQ handler to a threaded IRQ handler. We (Arista) carry the patch >>> below in our kernel tree, and the last two hunks of the patch do >>> exactly that, for the same reason (having to call >>> pci_get_domain_bus_and_slot() from the IRQ handler) but this is >>> probably too big of a change for 6.0-rc. >>> >>> >>> >>> commit 90a8e7da0facf198692a641fcfe6f89c478608e0 >>> Author: Lennert Buytenhek >>> Date: Wed Jul 13 15:34:30 2022 +0300 >>> >>> iommu/vt-d: Use report_iommu_fault() >>> This patch makes iommu/vt-d call report_iommu_fault() when an I/O >>> page fault occurs, which has two effects: >>> 1) It allows device drivers to register a callback to be notified >>> of I/O page faults, via the iommu_set_fault_handler() API. >>> 2) It triggers the io_page_fault tracepoint in report_iommu_fault() >>> when an I/O page fault occurs. >>> The latter point is the main aim of this patch, as it allows >>> rasdaemon-like daemons to be notified of I/O page faults, and to >>> possibly initiate corrective action in response. >> >> The IOMMU subsystem already has a framework to handle I/O page faults: >> >> commit fc36479db74e9 "iommu: Add a page fault handler" >> >> And below series, >> >> https://lore.kernel.org/linux-iommu/20220817012024.3251276-1-baolu.lu@linux.intel.com/ >> >> is trying to make it more generic. It seems to be more suitable for your >> case. >> >> The report_iommu_fault() probably will be replaced by >> iommu_register_device_fault_handler() eventually. So I don't encourage >> its usage in the VT-d driver. > > We use the iommu/io_page_fault tracepoint from userspace to be notified > of (non-ATS) I/O page faults so that we can detect malfunctioning PCIe > devices, which in our systems are typically switch/router line cards, > and take corrective action, such as restarting the offending line card. Yes. Make sense. > > Calling report_iommu_fault() causes the iommu/io_page_fault tracepoint > to be invoked, which is why we made the AMD and Intel IOMMU drivers use > report_iommu_fault() in our kernel tree. Can iommu_register_device_fault_handler() also serve your case? report_iommu_fault() is domain based, while the former is device based. > > It seems that iommu_queue_iopf() is specific to the SVA use case, while > we are not using SVA, in which case it would not address our use case. > (We don't care about knowing about ATS translation faults, we just want > to know when a non-ATS PCI device is malfunctioning.) The iommu_queue_iopf() is for recoverable I/O page fault. Your case only cares about unrecoverable DMA faults. So it's not suitable for you. Sorry for the misunderstanding. Best regards, baolu