From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C8581C433EF for ; Thu, 21 Apr 2022 09:52:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QR6SuaHz3xW66VZ+cSck/z8k/aTByAmGRX3Ff5aE1gQ=; b=ENdEY/0CNN+uB2 OH4RuW1i8Lr32qH+lasWc7KylLQnma7h7xNOTcCnSN/qYM3n348ebXa5/ju+YDv8NxyHAUtRosdbJ B5owdE/FOaMS0+P3jELqM+SwqEgQ6ODCN7iv/cQEZRhmQxooe4HfLpOiaLzcWYdLGJFj46IbGhigw q8OrALaa44ulqeiPuGlFewI8YLSVg8MHSDcfVs7mpRuClUlkSGGdVSd4ky/l5Axg6EuUWZplATh+s bIqoyaXDL5r7ulp6LMHd/1PEFZiq3HoFZJUoa5DgkVkC5S44IvDK73WfzOWuLksmuvVX0mOWi7grT Q+0WqDc2TIMfySwvCedA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nhTTN-00CtWX-OF; Thu, 21 Apr 2022 09:51:37 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nhTTK-00CtVO-KD for linux-arm-kernel@lists.infradead.org; Thu, 21 Apr 2022 09:51:36 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 12D1C61881; Thu, 21 Apr 2022 09:51:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CE881C385BC; Thu, 21 Apr 2022 09:51:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650534692; bh=jMwQwLbuWnaM2uCtDSMfJpp4AWMuWZkYRi9r7x7RXXk=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=k76LKDqgJTADOaMELaBdUWOzYi7WdS3SBu4mgf7ZgMN1M+dJldRYbMEkTa4ZzFc8M +m5rB4hSWvQsBvI1UvtUYj7fo4jtolIIRxjCJZ3GVufWn+soiyXOpBP9mQehyKMJqR mZWmqUJN4Na17Xm3F5ftZ25jCBDTyXBW1QcMVv0ESnLOC++RnKfYYbPdqHvqO/eqWN enq9+fb8DsJEDbi2dzt1vELy64nsqYuIlAbOn7SWcEgq4jxykjiPiZxToiXLc485FI 2ExNGy52RdqWinqzIgCRFXPCBQXs8WTnXhvMtziXY4RLsZh/SuT4rDRaesZadST4Z5 GVQfq6bV6zc9A== Received: from sofa.misterjones.org ([185.219.108.64] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nhTTG-005qk7-8G; Thu, 21 Apr 2022 10:51:30 +0100 Date: Thu, 21 Apr 2022 10:51:30 +0100 Message-ID: <87fsm6ahnh.wl-maz@kernel.org> From: Marc Zyngier To: Nathan Rossi Cc: linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Nathan Rossi , Andrew Lunn , Gregory Clement , Sebastian Hesselbarth , Thomas Gleixner Subject: Re: [PATCH] irqchip/armada-370-xp: Enable MSI affinity configuration In-Reply-To: References: <20220421015728.86912-1-nathan@nathanrossi.com> <87mtgfgx7d.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: nathan@nathanrossi.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, nathan.rossi@digi.com, andrew@lunn.ch, gregory.clement@bootlin.com, sebastian.hesselbarth@gmail.com, tglx@linutronix.de X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220421_025134_783861_23DF20D1 X-CRM114-Status: GOOD ( 52.90 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, 21 Apr 2022 09:32:23 +0100, Nathan Rossi wrote: > > On Thu, 21 Apr 2022 at 16:54, Marc Zyngier wrote: > > > > Hi Nathan, > > > > On Thu, 21 Apr 2022 02:57:28 +0100, > > Nathan Rossi wrote: > > > > > > From: Nathan Rossi > > > > > > With multiple devices attached via PCIe to an Armada 385 it is possible > > > to overwhelm a single CPU with MSI interrupts. Under certain scenarios > > > configuring the interrupts to be handled by more than one CPU would > > > prevent the system from being overwhelmed. However the > > > irqchip-aramada-370-xp driver is configured to only handle MSIs on the > > > boot CPU, and provides no affinity configuration. > > > > > > This change adds support to the armada-370-xp driver to allow for > > > configuring the affinity of specific MSI irqs and to generate the > > > interrupts on secondary CPUs. This is done by enabling the private > > > doorbell for all online CPUs and configures all CPUs to unmask MSI > > > specific private doorbell bits. The CPU affinity selection of the > > > interrupt is handled by the target list of the software triggered > > > interrupt value, which is provided as the MSI message. The message has > > > the associated CPU bit set for the target CPU. For private doorbell > > > interrupts only one bit can be set otherwise all CPUs will receive the > > > interrupt, so the lowest CPU in the affinity mask is used. This means > > > that by default the first CPU will handle all the interrupts as was the > > > case before. > > > > > > Signed-off-by: Nathan Rossi > > > --- > > > drivers/irqchip/irq-armada-370-xp.c | 34 ++++++++++++++++++++++++++++++++-- > > > 1 file changed, 32 insertions(+), 2 deletions(-) > > > > > > diff --git a/drivers/irqchip/irq-armada-370-xp.c b/drivers/irqchip/irq-armada-370-xp.c > > > index 5b8d571c04..42c257f576 100644 > > > --- a/drivers/irqchip/irq-armada-370-xp.c > > > +++ b/drivers/irqchip/irq-armada-370-xp.c > > > @@ -209,15 +209,37 @@ static struct msi_domain_info armada_370_xp_msi_domain_info = { > > > > > > static void armada_370_xp_compose_msi_msg(struct irq_data *data, struct msi_msg *msg) > > > { > > > +#ifdef CONFIG_SMP > > > + unsigned int cpu = cpumask_first(irq_data_get_effective_affinity_mask(data)); > > > + > > > + msg->data = (1 << (cpu + 8)) | (data->hwirq + PCI_MSI_DOORBELL_START); > > > > BIT(cpu + 8) | ... > > > > > +#else > > > + msg->data = 0xf00 | (data->hwirq + PCI_MSI_DOORBELL_START); > > > > This paints the existing code a bit differently. This seems to target > > all 4 CPUs. Why is that? I'd expect only bit 8 to be set, and the > > whole #ifdefery to go away. > > I am not sure why this is targeting 4 CPUs, it will be masked by the > percpu doorbell mask register and is effectively BIT(8). At least > based on the documentation I have (only for armada 370/38x), which is > why I left it as an #ifdef. I was also not able to find any specifics > as to why it is targeting all 4 CPUs in git history. However this > value was added with the initial driver implementation when only > armada 370 was available in the kernel, so it is perhaps an > inconsistent value that was never an issue due to the bits being > reserved. I will remove the #ifdef in a v2 patch that addresses your > other comments. I guess we can get at least some testing from the platform maintainers to check that this doesn't regress the UP systems. > > > > > > +#endif > > > msg->address_lo = lower_32_bits(msi_doorbell_addr); > > > msg->address_hi = upper_32_bits(msi_doorbell_addr); > > > - msg->data = 0xf00 | (data->hwirq + PCI_MSI_DOORBELL_START); > > > } > > > > > > static int armada_370_xp_msi_set_affinity(struct irq_data *irq_data, > > > const struct cpumask *mask, bool force) > > > { > > > - return -EINVAL; > > > +#ifdef CONFIG_SMP > > > + unsigned int cpu; > > > + > > > + if (!force) > > > + cpu = cpumask_any_and(mask, cpu_online_mask); > > > + else > > > + cpu = cpumask_first(mask); > > > + > > > + if (cpu >= nr_cpu_ids) > > > + return -EINVAL; > > > + > > > + irq_data_update_effective_affinity(irq_data, cpumask_of(cpu)); > > > + > > > + return IRQ_SET_MASK_OK; > > > +#else > > > + return -EINVAL; > > > +#endif > > > } > > > > > > static struct irq_chip armada_370_xp_msi_bottom_irq_chip = { > > > @@ -482,6 +504,7 @@ static void armada_xp_mpic_smp_cpu_init(void) > > > static void armada_xp_mpic_reenable_percpu(void) > > > { > > > unsigned int irq; > > > + u32 reg; > > > > > > /* Re-enable per-CPU interrupts that were enabled before suspend */ > > > for (irq = 0; irq < ARMADA_370_XP_MAX_PER_CPU_IRQS; irq++) { > > > @@ -501,6 +524,13 @@ static void armada_xp_mpic_reenable_percpu(void) > > > } > > > > > > ipi_resume(); > > > + > > > + /* Enable MSI doorbell mask and combined cpu local interrupt */ > > > + reg = readl(per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_MSK_OFFS) > > > + | PCI_MSI_DOORBELL_MASK; > > > + writel(reg, per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_MSK_OFFS); > > > + /* Unmask local doorbell interrupt */ > > > + writel(1, per_cpu_int_base + ARMADA_370_XP_INT_CLEAR_MASK_OFFS); > > > > This is a duplicate of what is already in armada_370_xp_msi_init(). > > Please refactor it so that this doesn't happen twice on the first CPU. > > It is duplicated, however armada_xp_mpic_reenable_percpu is not called > on the boot cpu as the setup is called with cpuhp_setup_state_nocalls. Ah, right. Make sure we can get rid of the code duplication then. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel