* [PATCH v2 0/2] remoteproc: k3: Recover a crashed remote core @ 2026-11-16 18:41 Chrispine Tinega 2026-11-16 18:41 ` [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core Chrispine Tinega 2026-11-16 18:41 ` [PATCH v2 2/2] remoteproc: k3-m4: Enable crash recovery for a core Linux booted Chrispine Tinega 0 siblings, 2 replies; 6+ messages in thread From: Chrispine Tinega @ 2026-11-16 18:41 UTC (permalink / raw) To: andersson, mathieu.poirier Cc: linux-remoteproc, linux-kernel, francesco, Chrispine Tinega A K3 remote core can tell Linux it has crashed by sending RP_MBOX_CRASH, but the K3 drivers only log it. Nothing reaches the remoteproc core, so the crash is not counted and recovery never runs. The core stays halted until the next boot, with its rpmsg channels dead. Patch 1 reports the crash with rproc_report_crash(). This applies to every K3 remote that uses the common mailbox callback. Patch 2 enables recovery on the M4F when Linux booted it. Its stop is a plain reset, so recovery does not need the crashed core to answer. In IPC-only mode recovery stays disabled, because it would boot the firmware Linux has, which need not be the one started before Linux. The R5F and DSP drivers keep recovery disabled. A crashed core there moves to RPROC_CRASHED, and can be stopped and started again through sysfs. On the R5F, recovery itself would not restart the core yet: the reset it needs is applied in unprepare(), which recovery does not call. Tested on an AM625 board (BLIIoT BL350) running TI's 6.12 kernel with the equivalent change. The M4F runs Zephyr, whose fatal-error handler sends RP_MBOX_CRASH. After a forced fault, recovery reset and rebooted the M4F in 71 ms, and Linux kept running. Mainline's M4F stop is the same plain reset that this test used for a crashed core. The Zephyr side is https://github.com/zephyrproject-rtos/zephyr/pull/125770. Changes in v2: - Patch 1: drop the duplicate crash print, and note the R5F limit in the commit message. (Francesco) - Patch 2: keep recovery disabled in IPC-only mode. (Francesco) - Rebased onto rproc-next. v1: https://lore.kernel.org/all/20261109054830.42715-1-dev@chrispinetinega.com/ Chrispine Tinega (2): remoteproc: k3: Report a remote crash to the remoteproc core remoteproc: k3-m4: Enable crash recovery for a core Linux booted drivers/remoteproc/ti_k3_common.c | 6 +----- drivers/remoteproc/ti_k3_m4_remoteproc.c | 6 +++++- 2 files changed, 6 insertions(+), 6 deletions(-) base-commit: 3035b6708f61259c93a368238e206dcec313e584 -- 2.53.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core 2026-11-16 18:41 [PATCH v2 0/2] remoteproc: k3: Recover a crashed remote core Chrispine Tinega @ 2026-11-16 18:41 ` Chrispine Tinega 2026-11-18 16:57 ` Francesco Valla 2026-11-16 18:41 ` [PATCH v2 2/2] remoteproc: k3-m4: Enable crash recovery for a core Linux booted Chrispine Tinega 1 sibling, 1 reply; 6+ messages in thread From: Chrispine Tinega @ 2026-11-16 18:41 UTC (permalink / raw) To: andersson, mathieu.poirier Cc: linux-remoteproc, linux-kernel, francesco, Chrispine Tinega A K3 remote core that faults can send RP_MBOX_CRASH over the mailbox, but k3_rproc_mbox_callback() only logs it. The remoteproc core never learns of the crash, so its recovery never runs, the crash is not counted, and the core stays halted with every rpmsg channel to it dead. Call rproc_report_crash(), which may be called from atomic context. The core then moves to RPROC_CRASHED and the crash is counted. Where the driver leaves recovery enabled, recovery stops the core and boots it again. Where recovery is disabled, the core can still be stopped and started again through sysfs. Recovery on its own cannot yet restart an R5F: the reset the R5F needs before it starts again is applied in its unprepare() op, which sysfs stop calls and recovery does not. The R5F driver keeps recovery disabled, so this does not change what happens there. Signed-off-by: Chrispine Tinega <dev@chrispinetinega.com> --- Changes in v2: - Drop the dev_err(). rproc_report_crash() already logs the crash and names the core. (Francesco) - Say in the commit message that recovery cannot yet restart an R5F. (Francesco) drivers/remoteproc/ti_k3_common.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/drivers/remoteproc/ti_k3_common.c b/drivers/remoteproc/ti_k3_common.c index 4b6da3363..43a7a21fd 100644 --- a/drivers/remoteproc/ti_k3_common.c +++ b/drivers/remoteproc/ti_k3_common.c @@ -60,11 +60,7 @@ void k3_rproc_mbox_callback(struct mbox_client *client, void *data) switch (msg) { case RP_MBOX_CRASH: - /* - * remoteproc detected an exception, but error recovery is not - * supported. So, just log this for now - */ - dev_err(dev, "K3 rproc %s crashed\n", rproc->name); + rproc_report_crash(rproc, RPROC_FATAL_ERROR); break; case RP_MBOX_ECHO_REPLY: dev_info(dev, "received echo reply from %s\n", rproc->name); -- 2.53.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core 2026-11-16 18:41 ` [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core Chrispine Tinega @ 2026-11-18 16:57 ` Francesco Valla 2026-11-18 19:44 ` Chrispine Tinega 0 siblings, 1 reply; 6+ messages in thread From: Francesco Valla @ 2026-11-18 16:57 UTC (permalink / raw) To: Chrispine Tinega Cc: andersson, mathieu.poirier, linux-remoteproc, linux-kernel Hi Chrispine, On Mon, Nov 16, 2026 at 09:41:44PM +0300, Chrispine Tinega wrote: > A K3 remote core that faults can send RP_MBOX_CRASH over the mailbox, > but k3_rproc_mbox_callback() only logs it. The remoteproc core never > learns of the crash, so its recovery never runs, the crash is not > counted, and the core stays halted with every rpmsg channel to it dead. > > Call rproc_report_crash(), which may be called from atomic context. The > core then moves to RPROC_CRASHED and the crash is counted. Where the > driver leaves recovery enabled, recovery stops the core and boots it > again. Where recovery is disabled, the core can still be stopped and > started again through sysfs. > > Recovery on its own cannot yet restart an R5F: the reset the R5F needs > before it starts again is applied in its unprepare() op, which sysfs > stop calls and recovery does not. The R5F driver keeps recovery > disabled, so this does not change what happens there. > Did you have the chance to take a look at the review sashiko did [1]? I think the points about the leaks in IPC mode make sense and are worth an investigation, both for the M4 and R5F cases. > Signed-off-by: Chrispine Tinega <dev@chrispinetinega.com> > --- > Changes in v2: > - Drop the dev_err(). rproc_report_crash() already logs the crash > and names the core. (Francesco) > - Say in the commit message that recovery cannot yet restart an R5F. > (Francesco) > > drivers/remoteproc/ti_k3_common.c | 6 +----- > 1 file changed, 1 insertion(+), 5 deletions(-) > > diff --git a/drivers/remoteproc/ti_k3_common.c b/drivers/remoteproc/ti_k3_common.c > index 4b6da3363..43a7a21fd 100644 > --- a/drivers/remoteproc/ti_k3_common.c > +++ b/drivers/remoteproc/ti_k3_common.c > @@ -60,11 +60,7 @@ void k3_rproc_mbox_callback(struct mbox_client *client, void *data) > > switch (msg) { > case RP_MBOX_CRASH: > - /* > - * remoteproc detected an exception, but error recovery is not > - * supported. So, just log this for now > - */ > - dev_err(dev, "K3 rproc %s crashed\n", rproc->name); > + rproc_report_crash(rproc, RPROC_FATAL_ERROR); > break; > case RP_MBOX_ECHO_REPLY: > dev_info(dev, "received echo reply from %s\n", rproc->name); > -- > 2.53.0 > > Regards, Francesco [1] https://sashiko.dev/#/patchset/20261116184146.1897281-1-dev%40chrispinetinega.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core 2026-11-18 16:57 ` Francesco Valla @ 2026-11-18 19:44 ` Chrispine Tinega 2026-11-22 13:57 ` Francesco Valla 0 siblings, 1 reply; 6+ messages in thread From: Chrispine Tinega @ 2026-11-18 19:44 UTC (permalink / raw) To: Francesco Valla Cc: andersson, mathieu.poirier, linux-remoteproc, linux-kernel Hi Francesco, On Wed, Nov 18, 2026 at 05:57:10PM +0100, Francesco Valla wrote: > Did you have the chance to take a look at the review sashiko did? > I think the points about the leaks in IPC mode make sense and are worth > an investigation, both for the M4 and R5F cases. I had not, thanks. Both of its points on patch 1 hold, and they have one cause: v2 reports a crash for a core that Linux did not boot. 1. IPC-only mode. Once a DETACHED or ATTACHED core is RPROC_CRASHED, rproc_detach() refuses it with -EINVAL. The DSP and R5F remove paths only detach an RPROC_ATTACHED core, so they skip it and leak its carveouts and subdevices. The R5F also has no start or stop ops in this mode. On the M4F and DSP, a shutdown would reset a core that Linux never started. 2. Probe. All three drivers request the mailbox before devm_rproc_add(), so a crash queued during probe is never cancelled if probe then fails. In IPC-only mode the crash work can itself cause the failure, by setting RPROC_CRASHED before rproc_validate() runs. A crash the core sent before Linux probed is enough, since it waits in the mailbox until the channel is opened. For v3 I report the crash only when the core is RPROC_RUNNING, and keep the log line otherwise: case RP_MBOX_CRASH: if (rproc->state == RPROC_RUNNING) rproc_report_crash(rproc, RPROC_FATAL_ERROR); else dev_err(dev, "K3 rproc %s crashed\n", rproc->name); break; An IPC-only core is never RPROC_RUNNING, and no core is before devm_rproc_add(). A core that Linux booted takes the same path as in v2, so your stress test still covers it. Patch 2 is unchanged. Recovering an IPC-only core needs the core to handle a crashed attached core, for instance as RPROC_FEAT_ATTACH_ON_RECOVERY does. I would leave that for a separate series. The review also points at an unbalanced put_device() in k3_rproc_unprepare(). I think that one is older than this series: stopping an attached M4F or DSP through sysfs reaches it with no crash at all. I will check it on a board and send it separately. v3 builds with W=1 and checkpatch is clean. I will post it once your stress test is done, unless you would rather have it now. Regards, Chrispine ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core 2026-11-18 19:44 ` Chrispine Tinega @ 2026-11-22 13:57 ` Francesco Valla 0 siblings, 0 replies; 6+ messages in thread From: Francesco Valla @ 2026-11-22 13:57 UTC (permalink / raw) To: Chrispine Tinega Cc: andersson, mathieu.poirier, linux-remoteproc, linux-kernel Hi Chrispine, On Wed, Nov 18, 2026 at 10:44:03PM +0300, Chrispine Tinega wrote: > Hi Francesco, > > On Wed, Nov 18, 2026 at 05:57:10PM +0100, Francesco Valla wrote: > > Did you have the chance to take a look at the review sashiko did? > > I think the points about the leaks in IPC mode make sense and are worth > > an investigation, both for the M4 and R5F cases. > > I had not, thanks. Both of its points on patch 1 hold, and they have > one cause: v2 reports a crash for a core that Linux did not boot. > > 1. IPC-only mode. Once a DETACHED or ATTACHED core is RPROC_CRASHED, > rproc_detach() refuses it with -EINVAL. The DSP and R5F remove paths > only detach an RPROC_ATTACHED core, so they skip it and leak its > carveouts and subdevices. The R5F also has no start or stop ops in > this mode. On the M4F and DSP, a shutdown would reset a core that > Linux never started. > > 2. Probe. All three drivers request the mailbox before > devm_rproc_add(), so a crash queued during probe is never cancelled > if probe then fails. In IPC-only mode the crash work can itself > cause the failure, by setting RPROC_CRASHED before rproc_validate() > runs. A crash the core sent before Linux probed is enough, since it > waits in the mailbox until the channel is opened. > > For v3 I report the crash only when the core is RPROC_RUNNING, and keep > the log line otherwise: > > case RP_MBOX_CRASH: > if (rproc->state == RPROC_RUNNING) > rproc_report_crash(rproc, RPROC_FATAL_ERROR); > else > dev_err(dev, "K3 rproc %s crashed\n", rproc->name); > break; > > An IPC-only core is never RPROC_RUNNING, and no core is before > devm_rproc_add(). A core that Linux booted takes the same path as in > v2, so your stress test still covers it. Patch 2 is unchanged. > > Recovering an IPC-only core needs the core to handle a crashed attached > core, for instance as RPROC_FEAT_ATTACH_ON_RECOVERY does. I would leave > that for a separate series. > > The review also points at an unbalanced put_device() in > k3_rproc_unprepare(). I think that one is older than this series: > stopping an attached M4F or DSP through sysfs reaches it with no crash > at all. I will check it on a board and send it separately. > > v3 builds with W=1 and checkpatch is clean. I will post it once your > stress test is done, unless you would rather have it now. > I'd prefer not to declare a Tested-by against a known buggy patch. Please send a V3, and I'll reply there. > Regards, > Chrispine > Reagrds, Francesco ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH v2 2/2] remoteproc: k3-m4: Enable crash recovery for a core Linux booted 2026-11-16 18:41 [PATCH v2 0/2] remoteproc: k3: Recover a crashed remote core Chrispine Tinega 2026-11-16 18:41 ` [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core Chrispine Tinega @ 2026-11-16 18:41 ` Chrispine Tinega 1 sibling, 0 replies; 6+ messages in thread From: Chrispine Tinega @ 2026-11-16 18:41 UTC (permalink / raw) To: andersson, mathieu.poirier Cc: linux-remoteproc, linux-kernel, francesco, Chrispine Tinega The M4 driver disables recovery, but nothing reported a crash to the remoteproc core, so there was nothing to recover from. Now that RP_MBOX_CRASH reaches rproc_report_crash(), recovery resets the M4F and boots its firmware again, while Linux keeps running. The stop is a plain reset, so it does not need the crashed core's cooperation. Enable recovery by default when Linux booted the M4F. Keep it disabled in IPC-only mode, where Linux only attached: recovery boots rproc->firmware from Linux, which need not be the firmware started before Linux, and nothing would tell the user it was replaced. Userspace can still change it through /sys/class/remoteproc/remoteprocN/recovery. Signed-off-by: Chrispine Tinega <dev@chrispinetinega.com> --- Changes in v2: - Keep recovery disabled in IPC-only mode, where Linux only attached and recovery could boot a different firmware unnoticed. (Francesco) - Retitled from "remoteproc: k3-m4: Leave crash recovery enabled". drivers/remoteproc/ti_k3_m4_remoteproc.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/drivers/remoteproc/ti_k3_m4_remoteproc.c b/drivers/remoteproc/ti_k3_m4_remoteproc.c index 4c46f4a25..b2a187ea4 100644 --- a/drivers/remoteproc/ti_k3_m4_remoteproc.c +++ b/drivers/remoteproc/ti_k3_m4_remoteproc.c @@ -56,7 +56,6 @@ static int k3_m4_rproc_probe(struct platform_device *pdev) return -ENOMEM; rproc->has_iommu = false; - rproc->recovery_disabled = true; kproc = rproc->priv; kproc->dev = dev; kproc->rproc = rproc; @@ -105,6 +104,11 @@ static int k3_m4_rproc_probe(struct platform_device *pdev) /* configure devices for either remoteproc or IPC-only mode */ if (p_state) { rproc->state = RPROC_DETACHED; + /* + * Recovery boots rproc->firmware, which is not necessarily what + * was started before Linux attached. + */ + rproc->recovery_disabled = true; dev_info(dev, "configured M4F for IPC-only mode\n"); } else { dev_info(dev, "configured M4F for remoteproc mode\n"); -- 2.53.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-11-22 13:57 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-16 18:41 [PATCH v2 0/2] remoteproc: k3: Recover a crashed remote core Chrispine Tinega 2026-11-16 18:41 ` [PATCH v2 1/2] remoteproc: k3: Report a remote crash to the remoteproc core Chrispine Tinega 2026-11-18 16:57 ` Francesco Valla 2026-11-18 19:44 ` Chrispine Tinega 2026-11-22 13:57 ` Francesco Valla 2026-11-16 18:41 ` [PATCH v2 2/2] remoteproc: k3-m4: Enable crash recovery for a core Linux booted Chrispine Tinega
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox