Linux-ARM-Kernel Archive on lore.kernel.org
 help / color / mirror / Atom feed
From: Peng Fan <peng.fan@linux.dev>
To: Francesco Dolcini <francesco@dolcini.it>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>,
	Peng Fan <peng.fan@oss.nxp.com>,
	Francesco Dolcini <francesco.dolcini@toradex.com>,
	Peng Fan <peng.fan@nxp.com>,
	linux-remoteproc@vger.kernel.org, imx@lists.linux.dev,
	linux-arm-kernel@lists.infradead.org,
	linux-rt-devel@lists.linux.dev,
	Bjorn Andersson <andersson@kernel.org>,
	Clark Williams <clrkwllms@kernel.org>,
	Fabio Estevam <festevam@gmail.com>, Frank Li <Frank.Li@nxp.com>,
	Jassi Brar <jassisinghbrar@gmail.com>,
	Mathieu Poirier <mathieu.poirier@linaro.org>,
	Pengutronix Kernel Team <kernel@pengutronix.de>,
	Sascha Hauer <s.hauer@pengutronix.de>,
	Steven Rostedt <rostedt@goodmis.org>
Subject: Re: [PATCH v3 00/10] mailbox: imx: Use threaded handler to avoid kworker in imx's remoteproc
Date: Thu, 19 Nov 2026 22:30:06 +0800	[thread overview]
Message-ID: <av8H7l6C8fmiHdbV@fedora> (raw)
In-Reply-To: <20261119114041.GA166708@francesco-nb>

On Thu, Nov 19, 2026 at 12:40:41PM +0100, Francesco Dolcini wrote:
>Hello Peng, Sebastian
>
>On Mon, Nov 16, 2026 at 11:23:31AM +0100, Sebastian Andrzej Siewior wrote:
>> On 2026-11-16 10:23:41 [+0100], Francesco Dolcini wrote:
>> > Our CI shows a ~10-15% eMMC sequential read throughput drop on i.MX8QXP
>> > and i.MX8DX (Colibri iMX8X, Cortex-A35), starting with v7.2-rc1 and
>> > still present in v7.4-rc3 (hdparm --direct -t: ~129 MB/s -> 105-120
>> > MB/s on i.MX8DX). i.MX8QM, i.MX8M and i.MX9 are not affected.
>> > 
>> > 5d1c0000.mailbox interrupt count before the test went from ~8k on v6.18
>> > to 60k+ on v7.3/v7.4.
>> > 
>> > Peng: any idea/suggestion? Does it makes any sense to you?
>> 
>> Can you pin point it to a patch in this series? My guess is that
>> something with the masking/ unmasking went wrong. The increase might be
>> result of the interrupt keeping firing while the threaded interrupt did
>> not have a chance to run.
>
>Reverting 36cac4b5101f, 3225a745f517, e55bea377d49 and 692e1bc96a5d
>restores 149 MB/s with schedutil.
>
>
>On Thu, Nov 19, 2026 at 02:25:55PM +0800, Peng Fan wrote:
>> On Mon, Nov 16, 2026 at 10:23:41AM +0100, Francesco Dolcini wrote:
>> >Peng: any idea/suggestion? Does it makes any sense to you?
>> 
>> Please help test whether this helps:
>> 
>> diff --git a/drivers/mailbox/imx-mailbox.c b/drivers/mailbox/imx-mailbox.c
>> index 683ccea6f2ba..e29044cf4b92 100644
>> --- a/drivers/mailbox/imx-mailbox.c
>> +++ b/drivers/mailbox/imx-mailbox.c
>> @@ -635,7 +635,7 @@ static int imx_mu_startup(struct mbox_chan *chan)
>>  {
>>         struct imx_mu_priv *priv = to_imx_mu_priv(chan->mbox);
>>         struct imx_mu_con_priv *cp = chan->con_priv;
>> -       unsigned long irq_flag = IRQF_NO_THREAD;
>> +       unsigned long irq_flag = IRQF_NO_THREAD | IRQF_ONESHOT;
>>         int ret;
>> 
>>         pm_runtime_get_sync(priv->dev);
>
>It does not help.
>	
>The analysis below was done with Claude help, on measurements I took on a
>Colibri iMX8QXP. I hope you can pick it up from there, this is not my
>area of expertise, and the best I can do is to send a revert for those 4
>commits.
>
>Every SCU RPC now wakes the SCHED_FIFO irq thread (TX done and RX), and
>while an RT task is runnable schedutil requests the maximum CPU frequency.
>The A35 clock is CLK_GET_RATE_NOCACHE, so each frequency change is
>followed by an SCU RPC reading the rate back, which wakes the irq thread
>again. The loop never settles, even when idle, and the transitions (ATF +
>SCU) are what costs the eMMC throughput (and I assume there is more
>impact on performance than just this).

Not sure whether this is root cause, per my understanding, cpu freq update
is not that high frequency call.

Let me setup a i.MX8QXP board and see what's going on there. BTW, i.MX8QM should
also have same issue, but seems not impact in your report, a bit werid,
might be because i.MX8QM has more cores :)

Have you tried pin your emmc test thread to core3, or core2, just not core0.

Regards
Peng

>
>v7.4-rc3, 4 CPUs, idle irq/cpufreq transitions per 10s, hdparm --direct -t:
>
>                               irq    trans  schedutil  performance
>  v6.18                          -      -    149.7 MB/s     -
>  v7.4-rc3                   16203  10800    113 MB/s   150 MB/s
>  v7.4-rc3 + IRQF_ONESHOT     7832   7813    113 MB/s   150 MB/s
>  v7.4-rc3 + revert 4 commits   60     36    149 MB/s   150 MB/s
>
>The 4 commits are 36cac4b5101f, 3225a745f517, e55bea377d49 and
>692e1bc96a5d; reverting only the RX part (e55bea377d49) is not enough.
>
>
>#regzbot introduced: 692e1bc96a5d^..36cac4b5101f
>#regzbot title: mailbox: imx: threaded IRQ handling causes a cpufreq transition storm on SCU-based i.MX8
>
>Thanks,
>Francesco
>
>


  reply	other threads:[~2026-11-19 14:30 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-17  6:55 [PATCH v3 00/10] mailbox: imx: Use threaded handler to avoid kworker in imx's remoteproc Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 01/10] mailbox: imx: Forward the timeout/ error in imx_mu_generic_tx() Sebastian Andrzej Siewior
2026-06-18  1:50   ` Peng Fan
2026-06-18  1:57     ` Peng Fan
2026-06-22 11:24   ` Peng Fan
2026-06-24  7:44     ` Sebastian Andrzej Siewior
2026-06-26  8:23       ` Peng Fan
2026-06-26  8:34         ` Sebastian Andrzej Siewior
2026-06-26  9:00           ` Peng Fan (OSS)
2026-06-29 15:38             ` Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 02/10] mailbox: imx: Add a channel shutdown field Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 03/10] mailbox: imx: Use devm_pm_runtime_enable() Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 04/10] mailbox: imx: use devm_of_platform_populate() Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 05/10] mailbox: imx: Use channel index instead of zero in imx_mu_specific_rx() Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 06/10] mailbox: imx: Start splitting the IRQ handler in primary and threaded handler Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 07/10] mailbox: imx: Move the RX part of the mailbox into the " Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 08/10] mailbox: imx: Move the RXDB " Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 09/10] mailbox: imx: Don't force-thread the primary handler Sebastian Andrzej Siewior
2026-06-17  6:55 ` [PATCH v3 10/10] remoteproc: imx_rproc: Invoke the callback directly Sebastian Andrzej Siewior
2026-11-16  9:23 ` [PATCH v3 00/10] mailbox: imx: Use threaded handler to avoid kworker in imx's remoteproc Francesco Dolcini
2026-11-16 10:23   ` Sebastian Andrzej Siewior
2026-11-19 11:40     ` Francesco Dolcini
2026-11-19 14:30       ` Peng Fan [this message]
2026-11-19 14:44         ` Francesco Dolcini
2026-11-19  6:25   ` Peng Fan

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av8H7l6C8fmiHdbV@fedora \
    --to=peng.fan@linux.dev \
    --cc=Frank.Li@nxp.com \
    --cc=andersson@kernel.org \
    --cc=bigeasy@linutronix.de \
    --cc=clrkwllms@kernel.org \
    --cc=festevam@gmail.com \
    --cc=francesco.dolcini@toradex.com \
    --cc=francesco@dolcini.it \
    --cc=imx@lists.linux.dev \
    --cc=jassisinghbrar@gmail.com \
    --cc=kernel@pengutronix.de \
    --cc=linux-arm-kernel@lists.infradead.org \
    --cc=linux-remoteproc@vger.kernel.org \
    --cc=linux-rt-devel@lists.linux.dev \
    --cc=mathieu.poirier@linaro.org \
    --cc=peng.fan@nxp.com \
    --cc=peng.fan@oss.nxp.com \
    --cc=rostedt@goodmis.org \
    --cc=s.hauer@pengutronix.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox