From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.0 required=3.0 tests=BAYES_00,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E88CC433E0 for ; Mon, 1 Feb 2021 18:51:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4782464DA8 for ; Mon, 1 Feb 2021 18:51:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231296AbhBASvR (ORCPT ); Mon, 1 Feb 2021 13:51:17 -0500 Received: from mail.kernel.org ([198.145.29.99]:35728 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229763AbhBASvP (ORCPT ); Mon, 1 Feb 2021 13:51:15 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A5BD864DA8; Mon, 1 Feb 2021 18:50:34 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1l6eHQ-00BLAx-Lg; Mon, 01 Feb 2021 18:50:32 +0000 Date: Mon, 01 Feb 2021 18:50:32 +0000 Message-ID: <87o8h3lj0n.wl-maz@kernel.org> From: Marc Zyngier To: John Garry Cc: Thomas Gleixner , Zhou Wang , "linux-kernel@vger.kernel.org" Subject: Re: PCI MSI issue with reinserting a driver In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: john.garry@huawei.com, tglx@linutronix.de, wangzhou1@hisilicon.com, linux-kernel@vger.kernel.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi John, On Mon, 01 Feb 2021 18:34:59 +0000, John Garry wrote: > > Just a heads-up, by chance I noticed that I can't re-insert a specific > driver on v5.11-rc6: > > [ 64.356023] hisi_dma 0000:7b:00.0: Adding to iommu group 31 > [ 64.368627] hisi_dma 0000:7b:00.0: enabling device (0000 -> 0002) > [ 64.384156] hisi_dma 0000:7b:00.0: Failed to allocate MSI vectors! > [ 64.397180] hisi_dma: probe of 0000:7b:00.0 failed with error -28 > > That's with CONFIG_DEBUG_TEST_DRIVER_REMOVE=y > > Bisect tells me that this is the first bad commit: > 4615fbc3788d genirq/irqdomain: Don't try to free an interrupt that has > no mapping > > The relevant driver code is > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/dma/hisi_dma.c#n547 > > That driver only allocates 30 MSI, so maybe there's a problem with not > allocating (and freeing) all 32 MSI. Are they Multi-MSI (and not MSI-X)? > I'll have a bit more of a look tomorrow. Here's my suspicion: two of the interrupts are mapped in the low-level domain (the ITS, I'd expect in your case), but they have never been mapped at the higher level. On teardown, we only get rid of the 30 that were actually mapped, and leave the last two dangling in the ITS domain, and thus the ITS device resources are never freed. On reload, we request another 32 interrupts, which can't be satisfied for this device. Assuming I got it right, the question is: why weren't these interrupts mapped in the PCI domain the first place. And if I got it wrong, I'm even more curious! Thanks, M. -- Without deviation from the norm, progress is not possible.