From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A84DDC54E67 for ; Tue, 26 Mar 2024 20:11:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sGq8ro38RPwVsqG9FKd862QaVfAaNdKT4FdPnIV0xpg=; b=WKx+SlTsTbmnst I1l9tHbAnEbqsbuGX3y24jNLt7SfhgrlFvXw5PaAzUHRqedFUH8tMEg2FcaDpgSJbb2eLucUBfSkZ yvYv8DcdnWXIeFBSLlp2fjOyuKEh54QOn9LdRZvIsAcvvP6HzGc2gl+jEqQJ96xTNrqanj2FC8zV1 Nc6/kHeWvl0Xmg+97md8c22XxS29+bDnx4/K6AhkkqsSnbC60n1231R5hwBQNGVjXCVtoq2lqZpVg Hw6rfWlLffpmzwN+ASNJAxSDcGXPkKu89g350TDKu4h1jdSsTErRYOQ2h+n6a/hxAsf8QAZdSph/0 sK41LzP+3mgNHyFn2Ing==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpD8e-00000006HBt-1O9k; Tue, 26 Mar 2024 20:11:16 +0000 Received: from galois.linutronix.de ([2a0a:51c0:0:12e:550::1]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1rpD8W-00000006H9S-47bG for linux-arm-kernel@lists.infradead.org; Tue, 26 Mar 2024 20:11:14 +0000 From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1711483864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bLYsrPiq5J0wZCfj/q/s3RyXPXdnzXn/EJnAMhRjnNM=; b=XhiNBpO7K8eF9NaGaXZqaLBez6KTX8VZWWfXmIAeqJ38bZ2sg52o8z1mYLrpCVMcVVTXkx 2aXCo08qv0112q6lnpGPfibcRvvL04rKEvKBnI/nZsqObQPXdQTWpAovN6DdqiKd44BCoQ 8X+e0brBjmhiXsWKJ0AXH0hlaMfBVQ2O9Syu3KfynU5MRZ5chzdzAh1P/O8ruiz0ed0x5Z PgyoYN2F+dnco5oFw2dw9f92M1JW81eK2LzZqDPw5OnsOTa8vvD5xXWYhZ1C0hdTfVSJCb jBTfhdX3d3XuTBemBJif5eoit1Pj9evYeQu9Q2r+3wr+p+IvSoUbG1g0B88P3Q== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1711483864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=bLYsrPiq5J0wZCfj/q/s3RyXPXdnzXn/EJnAMhRjnNM=; b=so1j1EtXxDx/CBeJgbfuTx9V9+Jc4ZFzK7AJYhWwQ99HWJlykwLf84Y2x4cgUdipnALRYC fXEJ1PLg9n/sjzBw== To: Jason Gunthorpe , "Peng Fan (OSS)" Cc: Will Deacon , Robin Murphy , Joerg Roedel , Marc Zyngier , Bixuan Cui , linux-arm-kernel@lists.infradead.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, Peng Fan Subject: Re: [PATCH 3/3] iommu/arm-smmu-v3: support suspend/resume In-Reply-To: <87bk72fcf4.ffs@tglx> References: <20240324-smmu-v3-v1-0-11bc96e156a5@nxp.com> <20240324-smmu-v3-v1-3-11bc96e156a5@nxp.com> <20240325134848.GA8419@ziepe.ca> <87bk72fcf4.ffs@tglx> Date: Tue, 26 Mar 2024 21:11:03 +0100 Message-ID: <87msqkeotk.ffs@tglx> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240326_131109_405776_8D7A6823 X-CRM114-Status: GOOD ( 22.70 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Mar 25 2024 at 18:29, Thomas Gleixner wrote: > On Mon, Mar 25 2024 at 10:48, Jason Gunthorpe wrote: >> On Sun, Mar 24, 2024 at 08:29:00PM +0800, Peng Fan (OSS) wrote: >>> + if (smmu->features & ARM_SMMU_FEAT_PRI) { >>> + desc = irq_get_msi_desc(smmu->priq.q.irq); >>> + if (desc) { >>> + get_cached_msi_msg(smmu->priq.q.irq, &msg); >>> + arm_smmu_write_msi_msg(desc, &msg); >>> + } >>> + } >>> +} >> >> I wonder if this should be done instead by converting the driver away >> from platform MSI to the new MSI mechanism? > > There is work in progress for that. Should come around in the next > weeks. But that won't solve it. The above is a horrible hack and I think this should be solved completely differently. On suspend the core interrupt code disables all interrupts which are not marked as wakeup interrupts. On resume it reenables them including restoring the affinity setting. So this just should work, but it does not because the MSI message is only written once, when the interrupt is activated. Further affinity changes affect only the ITS table and do not result in a message write. The most trivial way w/o changing any core code for it, would be to free the interrupts on suspend and request them on resume again, because that would deactivate and reactivate it. Though that does not work if the IOMMU is resumed in the early resume path. But we can enforce writing the message in the resume path. See the untested below. Thanks, tglx --- --- a/include/linux/irq.h +++ b/include/linux/irq.h @@ -223,6 +223,7 @@ struct irq_data { * irqchip have flag IRQCHIP_ENABLE_WAKEUP_ON_SUSPEND set. * IRQD_RESEND_WHEN_IN_PROGRESS - Interrupt may fire when already in progress in which * case it must be resent at the next available opportunity. + * IRQD_RESUMING - Interrupt is resumed after suspend */ enum { IRQD_TRIGGER_MASK = 0xf, @@ -249,6 +250,7 @@ enum { IRQD_AFFINITY_ON_ACTIVATE = BIT(28), IRQD_IRQ_ENABLED_ON_SUSPEND = BIT(29), IRQD_RESEND_WHEN_IN_PROGRESS = BIT(30), + IRQD_RESUMING = BIT(31), }; #define __irqd_to_state(d) ACCESS_PRIVATE((d)->common, state_use_accessors) @@ -443,6 +445,11 @@ static inline bool irqd_needs_resend_whe return __irqd_to_state(d) & IRQD_RESEND_WHEN_IN_PROGRESS; } +static inline bool irqd_resuming(struct irq_data *d) +{ + return __irqd_to_state(d) & IRQD_RESUMING; +} + #undef __irqd_to_state static inline irq_hw_number_t irqd_to_hwirq(struct irq_data *d) --- a/kernel/irq/msi.c +++ b/kernel/irq/msi.c @@ -617,6 +617,7 @@ static unsigned int msi_domain_get_hwsiz static inline void irq_chip_write_msi_msg(struct irq_data *data, struct msi_msg *msg) { + irq_data_get_msi_desc(data)->msg = *msg; data->chip->irq_write_msi_msg(data, msg); } @@ -652,7 +653,10 @@ int msi_domain_set_affinity(struct irq_d int ret; ret = parent->chip->irq_set_affinity(parent, mask, force); - if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) { + if (ret < 0) + return ret; + + if (ret != IRQ_SET_MASK_OK_DONE || irqd_resuming(irq_data)) { BUG_ON(irq_chip_compose_msi_msg(irq_data, msg)); msi_check_level(irq_data->domain, msg); irq_chip_write_msi_msg(irq_data, msg); --- a/kernel/irq/pm.c +++ b/kernel/irq/pm.c @@ -177,7 +177,13 @@ static void resume_irq(struct irq_desc * irq_state_set_masked(desc); resume: desc->istate &= ~IRQS_SUSPENDED; + /* + * Ensure that MSI messages get rewritten on resume. The device + * might have lost it due to power disabling. + */ + irqd_set(&desc->irq_data, IRQD_RESUMING); __enable_irq(desc); + irqd_clear(&desc->irq_data, IRQD_RESUMING); } static void resume_irqs(bool want_early) _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel