[PATCH 0/9] ASoC: SOF: ipc4/Intel: Support for firmware exception handling

[PATCH 0/9] ASoC: SOF: ipc4/Intel: Support for firmware exception handling

[Peter Ujfalusi]

Hi,

When a firmware crashes it creats a panic information into a telemetry slot.
The panic format is defined by Zephyr, includes stack and additional information
to help to identify the reason for the crash.
Part of the firmware exception handling the firmware also sends an
EXCEPTION_CAUGHT notification.

This series implements the kernel side handling of the exception:
print information into the kernel log
export the whole telemetry slot to user space for tools extract additional
information from the panic dump.

Regards,
Peter
---
Rander Wang (9):
  ASoC: SOF: Xtensa: dump ar registers to restore call stack
  ASoC: SOF: ipc4-mtrace: move debug slot related definitions to
    header.h
  ASoC: SOF: ipc4: add a helper function to search debug slot
  ASoC: SOF: ipc4: add definition of telemetry slot for exception
    handling
  ASoC: SOF: ipc4: add exception node in sof debugfs directory
  ASoC: SOF: Intel: add telemetry retrieval support on Intel platforms
  ASoC: SOF: Intel: mtl: dump dsp stack
  ASoC: SOF: Intel: hda: add ipc4 FW panic support on CAVS 2.5+
    platforms
  ASoC: SOF: ipc4: handle EXCEPTION_CAUGHT notification from firmware

 include/sound/sof/ipc4/header.h | 17 ++++++
 sound/soc/sof/Makefile          |  2 +-
 sound/soc/sof/intel/Makefile    |  3 +-
 sound/soc/sof/intel/hda.c       | 14 +++++
 sound/soc/sof/intel/hda.h       |  1 +
 sound/soc/sof/intel/mtl.c       |  3 ++
 sound/soc/sof/intel/telemetry.c | 95 +++++++++++++++++++++++++++++++++
 sound/soc/sof/intel/telemetry.h | 35 ++++++++++++
 sound/soc/sof/intel/tgl.c       |  1 +
 sound/soc/sof/ipc4-mtrace.c     | 34 ++++--------
 sound/soc/sof/ipc4-priv.h       |  3 ++
 sound/soc/sof/ipc4-telemetry.c  | 95 +++++++++++++++++++++++++++++++++
 sound/soc/sof/ipc4-telemetry.h  | 73 +++++++++++++++++++++++++
 sound/soc/sof/ipc4.c            | 29 ++++++++++
 sound/soc/sof/xtensa/core.c     | 11 ++++
 15 files changed, 391 insertions(+), 25 deletions(-)
 create mode 100644 sound/soc/sof/intel/telemetry.c
 create mode 100644 sound/soc/sof/intel/telemetry.h
 create mode 100644 sound/soc/sof/ipc4-telemetry.c
 create mode 100644 sound/soc/sof/ipc4-telemetry.h

-- 
2.42.0

[PATCH 1/9] ASoC: SOF: Xtensa: dump ar registers to restore call stack

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

On Xtensa platform ar0 is for caller address and ar1 is for stack
address. The ar register dump can be used to rebuild call stack with
FW elf file by debug tools.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/xtensa/core.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/sound/soc/sof/xtensa/core.c b/sound/soc/sof/xtensa/core.c
index bebbe3a2865c..7c91a919eadc 100644
--- a/sound/soc/sof/xtensa/core.c
+++ b/sound/soc/sof/xtensa/core.c
@@ -132,6 +132,17 @@ static void xtensa_stack(struct snd_sof_dev *sdev, const char *level, void *oops
 				   buf, sizeof(buf), false);
 		dev_printk(level, sdev->dev, "0x%08x: %s\n", stack_ptr + i * 4, buf);
 	}
+
+	if (!xoops->plat_hdr.numaregs)
+		return;
+
+	dev_printk(level, sdev->dev, "AR registers:\n");
+	/* the number of ar registers is a multiple of 4 */
+	for (i = 0; i < xoops->plat_hdr.numaregs; i += 4) {
+		hex_dump_to_buffer(xoops->ar + i, 16, 16, 4,
+				   buf, sizeof(buf), false);
+		dev_printk(level, sdev->dev, "%#x: %s\n", i * 4, buf);
+	}
 }
 
 const struct dsp_arch_ops sof_xtensa_arch_ops = {
-- 
2.42.0

[PATCH 2/9] ASoC: SOF: ipc4-mtrace: move debug slot related definitions to header.h

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

The macro definitions of debug slot can be used by gdb, telemetry
and mtrace log, so move these definitions to header.h from mtrace.
Then these macro definitions can be shared

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 include/sound/sof/ipc4/header.h | 17 +++++++++++++++++
 sound/soc/sof/ipc4-mtrace.c     | 34 +++++++++++----------------------
 2 files changed, 28 insertions(+), 23 deletions(-)

diff --git a/include/sound/sof/ipc4/header.h b/include/sound/sof/ipc4/header.h
index c58f00ef054a..574a9d581f88 100644
--- a/include/sound/sof/ipc4/header.h
+++ b/include/sound/sof/ipc4/header.h
@@ -515,6 +515,23 @@ struct sof_ipc4_notify_resource_data {
 	uint32_t data[6];
 } __packed __aligned(4);
 
+#define SOF_IPC4_DEBUG_DESCRIPTOR_SIZE		12 /* 3 x u32 */
+
+/*
+ * The debug memory window is divided into 16 slots, and the
+ * first slot is used as a recorder for the other 15 slots.
+ */
+#define SOF_IPC4_MAX_DEBUG_SLOTS		15
+#define SOF_IPC4_DEBUG_SLOT_SIZE		0x1000
+
+/* debug log slot types */
+#define SOF_IPC4_DEBUG_SLOT_UNUSED		0x00000000
+#define SOF_IPC4_DEBUG_SLOT_CRITICAL_LOG	0x54524300 /* byte 0: core ID */
+#define SOF_IPC4_DEBUG_SLOT_DEBUG_LOG		0x474f4c00 /* byte 0: core ID */
+#define SOF_IPC4_DEBUG_SLOT_GDB_STUB		0x42444700
+#define SOF_IPC4_DEBUG_SLOT_TELEMETRY		0x4c455400
+#define SOF_IPC4_DEBUG_SLOT_BROKEN		0x44414544
+
 /** @}*/
 
 #endif
diff --git a/sound/soc/sof/ipc4-mtrace.c b/sound/soc/sof/ipc4-mtrace.c
index 2b4659a1768e..9f1e33ee8826 100644
--- a/sound/soc/sof/ipc4-mtrace.c
+++ b/sound/soc/sof/ipc4-mtrace.c
@@ -41,24 +41,12 @@
  * The two pointers are offsets within the buffer.
  */
 
-#define SOF_MTRACE_DESCRIPTOR_SIZE		12 /* 3 x u32 */
-
 #define FW_EPOCH_DELTA				11644473600LL
 
-#define INVALID_SLOT_OFFSET			0xffffffff
 #define MAX_ALLOWED_LIBRARIES			16
-#define MAX_MTRACE_SLOTS			15
-
-#define SOF_MTRACE_PAGE_SIZE			0x1000
-#define SOF_MTRACE_SLOT_SIZE			SOF_MTRACE_PAGE_SIZE
-
-/* debug log slot types */
-#define SOF_MTRACE_SLOT_UNUSED			0x00000000
-#define SOF_MTRACE_SLOT_CRITICAL_LOG		0x54524300 /* byte 0: core ID */
-#define SOF_MTRACE_SLOT_DEBUG_LOG		0x474f4c00 /* byte 0: core ID */
-#define SOF_MTRACE_SLOT_GDB_STUB		0x42444700
-#define SOF_MTRACE_SLOT_TELEMETRY		0x4c455400
-#define SOF_MTRACE_SLOT_BROKEN			0x44414544
+
+#define SOF_IPC4_INVALID_SLOT_OFFSET		0xffffffff
+
  /* for debug and critical types */
 #define SOF_MTRACE_SLOT_CORE_MASK		GENMASK(7, 0)
 #define SOF_MTRACE_SLOT_TYPE_MASK		GENMASK(31, 8)
@@ -140,7 +128,7 @@ static int sof_ipc4_mtrace_dfs_open(struct inode *inode, struct file *file)
 	if (unlikely(ret))
 		goto out;
 
-	core_data->log_buffer = kmalloc(SOF_MTRACE_SLOT_SIZE, GFP_KERNEL);
+	core_data->log_buffer = kmalloc(SOF_IPC4_DEBUG_SLOT_SIZE, GFP_KERNEL);
 	if (!core_data->log_buffer) {
 		debugfs_file_put(file->f_path.dentry);
 		ret = -ENOMEM;
@@ -212,13 +200,13 @@ static ssize_t sof_ipc4_mtrace_dfs_read(struct file *file, char __user *buffer,
 		return 0;
 	}
 
-	if (core_data->slot_offset == INVALID_SLOT_OFFSET)
+	if (core_data->slot_offset == SOF_IPC4_INVALID_SLOT_OFFSET)
 		return 0;
 
 	/* The log data buffer starts after the two pointer in the slot */
 	log_buffer_offset =  core_data->slot_offset + (sizeof(u32) * 2);
 	/* The log data size excludes the pointers */
-	log_buffer_size = SOF_MTRACE_SLOT_SIZE - (sizeof(u32) * 2);
+	log_buffer_size = SOF_IPC4_DEBUG_SLOT_SIZE - (sizeof(u32) * 2);
 
 	read_ptr = core_data->host_read_ptr;
 	write_ptr = core_data->dsp_write_ptr;
@@ -510,13 +498,13 @@ static void sof_mtrace_find_core_slots(struct snd_sof_dev *sdev)
 	u32 slot_desc_type_offset, type, core;
 	int i;
 
-	for (i = 0; i < MAX_MTRACE_SLOTS; i++) {
+	for (i = 0; i < SOF_IPC4_MAX_DEBUG_SLOTS; i++) {
 		/* The type is the second u32 in the slot descriptor */
 		slot_desc_type_offset = sdev->debug_box.offset;
-		slot_desc_type_offset += SOF_MTRACE_DESCRIPTOR_SIZE * i + sizeof(u32);
+		slot_desc_type_offset += SOF_IPC4_DEBUG_DESCRIPTOR_SIZE * i + sizeof(u32);
 		sof_mailbox_read(sdev, slot_desc_type_offset, &type, sizeof(type));
 
-		if ((type & SOF_MTRACE_SLOT_TYPE_MASK) == SOF_MTRACE_SLOT_DEBUG_LOG) {
+		if ((type & SOF_MTRACE_SLOT_TYPE_MASK) == SOF_IPC4_DEBUG_SLOT_DEBUG_LOG) {
 			core = type & SOF_MTRACE_SLOT_CORE_MASK;
 
 			if (core >= sdev->num_cores) {
@@ -533,7 +521,7 @@ static void sof_mtrace_find_core_slots(struct snd_sof_dev *sdev)
 			 * debug_box + SOF_MTRACE_SLOT_SIZE offset
 			 */
 			core_data->slot_offset = sdev->debug_box.offset;
-			core_data->slot_offset += SOF_MTRACE_SLOT_SIZE * (i + 1);
+			core_data->slot_offset += SOF_IPC4_DEBUG_SLOT_SIZE * (i + 1);
 			dev_dbg(sdev->dev, "slot%d is used for core%u\n", i, core);
 			if (core_data->delayed_pos_update) {
 				sof_ipc4_mtrace_update_pos(sdev, core);
@@ -633,7 +621,7 @@ int sof_ipc4_mtrace_update_pos(struct snd_sof_dev *sdev, int core)
 
 	core_data = &priv->cores[core];
 
-	if (core_data->slot_offset == INVALID_SLOT_OFFSET) {
+	if (core_data->slot_offset == SOF_IPC4_INVALID_SLOT_OFFSET) {
 		core_data->delayed_pos_update = true;
 		return 0;
 	}
-- 
2.42.0

[PATCH 3/9] ASoC: SOF: ipc4: add a helper function to search debug slot

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Currently IPC4 supports GDB slot, telemetry slot and
debug slot. This helper function will be used to get
the slot offset in debug windows for further processing.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/ipc4-priv.h |  3 +++
 sound/soc/sof/ipc4.c      | 23 +++++++++++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/sound/soc/sof/ipc4-priv.h b/sound/soc/sof/ipc4-priv.h
index a5d0b2eae464..9e69b7c29117 100644
--- a/sound/soc/sof/ipc4-priv.h
+++ b/sound/soc/sof/ipc4-priv.h
@@ -120,4 +120,7 @@ void sof_ipc4_update_cpc_from_manifest(struct snd_sof_dev *sdev,
 				       struct sof_ipc4_fw_module *fw_module,
 				       struct sof_ipc4_base_module_cfg *basecfg);
 
+size_t sof_ipc4_find_debug_slot_offset_by_type(struct snd_sof_dev *sdev,
+					       u32 slot_type);
+
 #endif
diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c
index 6c33ee5af32b..4673c53f6c03 100644
--- a/sound/soc/sof/ipc4.c
+++ b/sound/soc/sof/ipc4.c
@@ -547,6 +547,29 @@ static int sof_ipc4_init_msg_memory(struct snd_sof_dev *sdev)
 	return 0;
 }
 
+size_t sof_ipc4_find_debug_slot_offset_by_type(struct snd_sof_dev *sdev,
+					       u32 slot_type)
+{
+	size_t slot_desc_type_offset;
+	u32 type;
+	int i;
+
+	/* The type is the second u32 in the slot descriptor */
+	slot_desc_type_offset = sdev->debug_box.offset + sizeof(u32);
+	for (i = 0; i < SOF_IPC4_MAX_DEBUG_SLOTS; i++) {
+		sof_mailbox_read(sdev, slot_desc_type_offset, &type, sizeof(type));
+
+		if (type == slot_type)
+			return sdev->debug_box.offset + (i + 1) * SOF_IPC4_DEBUG_SLOT_SIZE;
+
+		slot_desc_type_offset += SOF_IPC4_DEBUG_DESCRIPTOR_SIZE;
+	}
+
+	dev_dbg(sdev->dev, "Slot type %#x is not available in debug window\n", slot_type);
+	return 0;
+}
+EXPORT_SYMBOL(sof_ipc4_find_debug_slot_offset_by_type);
+
 static int ipc4_fw_ready(struct snd_sof_dev *sdev, struct sof_ipc4_msg *ipc4_msg)
 {
 	int inbox_offset, inbox_size, outbox_offset, outbox_size;
-- 
2.42.0

[PATCH 4/9] ASoC: SOF: ipc4: add definition of telemetry slot for exception handling

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Core dump includes hardware platform information, cpu registers and
exception call stack. FW saves core dump to telemetry slot in shared
memory window for host in the event of FW exception. This patch creates
exception node in debugfs for user to dump telemetry data.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/Makefile         |  2 +-
 sound/soc/sof/ipc4-telemetry.c | 95 ++++++++++++++++++++++++++++++++++
 sound/soc/sof/ipc4-telemetry.h | 73 ++++++++++++++++++++++++++
 3 files changed, 169 insertions(+), 1 deletion(-)
 create mode 100644 sound/soc/sof/ipc4-telemetry.c
 create mode 100644 sound/soc/sof/ipc4-telemetry.h

diff --git a/sound/soc/sof/Makefile b/sound/soc/sof/Makefile
index 744d40bd8c8b..42dc48e53964 100644
--- a/sound/soc/sof/Makefile
+++ b/sound/soc/sof/Makefile
@@ -10,7 +10,7 @@ snd-sof-objs +=	ipc3.o ipc3-loader.o ipc3-topology.o ipc3-control.o ipc3-pcm.o\
 endif
 ifneq ($(CONFIG_SND_SOC_SOF_INTEL_IPC4),)
 snd-sof-objs += ipc4.o ipc4-loader.o ipc4-topology.o ipc4-control.o ipc4-pcm.o\
-		ipc4-mtrace.o
+		ipc4-mtrace.o ipc4-telemetry.o
 endif
 
 # SOF client support
diff --git a/sound/soc/sof/ipc4-telemetry.c b/sound/soc/sof/ipc4-telemetry.c
new file mode 100644
index 000000000000..ec4ae9674364
--- /dev/null
+++ b/sound/soc/sof/ipc4-telemetry.c
@@ -0,0 +1,95 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license.  When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2018-2023 Intel Corporation. All rights reserved.
+//
+
+#include <linux/debugfs.h>
+#include <linux/io.h>
+#include <linux/pm_runtime.h>
+#include <sound/sof/debug.h>
+#include <sound/sof/ipc4/header.h>
+#include "sof-priv.h"
+#include "ops.h"
+#include "ipc4-telemetry.h"
+#include "ipc4-priv.h"
+
+static void __iomem *sof_ipc4_query_exception_address(struct snd_sof_dev *sdev)
+{
+	u32 type = SOF_IPC4_DEBUG_SLOT_TELEMETRY;
+	size_t telemetry_slot_offset;
+	u32 offset;
+
+	telemetry_slot_offset = sof_ipc4_find_debug_slot_offset_by_type(sdev, type);
+	if (!telemetry_slot_offset)
+		return NULL;
+
+	/* skip the first separator magic number */
+	offset = telemetry_slot_offset + sizeof(u32);
+
+	return sdev->bar[sdev->mailbox_bar] + offset;
+}
+
+static ssize_t sof_telemetry_entry_read(struct file *file, char __user *buffer,
+					size_t count, loff_t *ppos)
+{
+	struct snd_sof_dfsentry *dfse = file->private_data;
+	struct snd_sof_dev *sdev = dfse->sdev;
+	void __iomem *io_addr;
+	loff_t pos = *ppos;
+	size_t size_ret;
+	u8 *buf;
+
+	if (pos < 0)
+		return -EINVAL;
+	/* skip the first separator magic number */
+	if (pos >= SOF_IPC4_DEBUG_SLOT_SIZE - 4 || !count)
+		return 0;
+	if (count > SOF_IPC4_DEBUG_SLOT_SIZE - 4 - pos)
+		count = SOF_IPC4_DEBUG_SLOT_SIZE - 4 - pos;
+
+	io_addr = sof_ipc4_query_exception_address(sdev);
+	if (!io_addr)
+		return -EFAULT;
+
+	buf = kzalloc(SOF_IPC4_DEBUG_SLOT_SIZE - 4, GFP_KERNEL);
+	if (!buf)
+		return -ENOMEM;
+
+	memcpy_fromio(buf, io_addr, SOF_IPC4_DEBUG_SLOT_SIZE - 4);
+	size_ret = copy_to_user(buffer, buf + pos, count);
+	if (size_ret) {
+		kfree(buf);
+		return -EFAULT;
+	}
+
+	*ppos = pos + count;
+	kfree(buf);
+
+	return count;
+}
+
+static const struct file_operations sof_telemetry_fops = {
+	.open = simple_open,
+	.read = sof_telemetry_entry_read,
+};
+
+void sof_ipc4_create_exception_debugfs_node(struct snd_sof_dev *sdev)
+{
+	struct snd_sof_dfsentry *dfse;
+
+	dfse = devm_kzalloc(sdev->dev, sizeof(*dfse), GFP_KERNEL);
+	if (!dfse)
+		return;
+
+	dfse->type = SOF_DFSENTRY_TYPE_IOMEM;
+	dfse->size = SOF_IPC4_DEBUG_SLOT_SIZE - 4;
+	dfse->access_type = SOF_DEBUGFS_ACCESS_ALWAYS;
+	dfse->sdev = sdev;
+
+	list_add(&dfse->list, &sdev->dfsentry_list);
+
+	debugfs_create_file("exception", 0444, sdev->debugfs_root, dfse, &sof_telemetry_fops);
+}
diff --git a/sound/soc/sof/ipc4-telemetry.h b/sound/soc/sof/ipc4-telemetry.h
new file mode 100644
index 000000000000..ab3599e3d87d
--- /dev/null
+++ b/sound/soc/sof/ipc4-telemetry.h
@@ -0,0 +1,73 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2023 Intel Corporation. All rights reserved.
+ */
+
+#ifndef __SOUND_SOC_SOF_IPC4_TELEMETRY_H
+#define __SOUND_SOC_SOF_IPC4_TELEMETRY_H
+
+/* Target code */
+enum sof_ipc4_coredump_tgt_code {
+	COREDUMP_TGT_UNKNOWN = 0,
+	COREDUMP_TGT_X86,
+	COREDUMP_TGT_X86_64,
+	COREDUMP_TGT_ARM_CORTEX_M,
+	COREDUMP_TGT_RISC_V,
+	COREDUMP_TGT_XTENSA,
+};
+
+#define COREDUMP_ARCH_HDR_ID 'A'
+#define COREDUMP_HDR_ID0 'Z'
+#define COREDUMP_HDR_ID1 'E'
+
+#define XTENSA_BLOCK_HDR_VER		2
+#define XTENSA_CORE_DUMP_SEPARATOR	0x0DEC0DEB
+#define XTENSA_CORE_AR_REGS_COUNT	16
+#define XTENSA_SOC_INTEL_ADSP		3
+#define XTENSA_TOOL_CHAIN_ZEPHYR	1
+#define XTENSA_TOOL_CHAIN_XCC		2
+
+/* Coredump header */
+struct sof_ipc4_coredump_hdr {
+	/* 'Z', 'E' as identifier of file */
+	char id[2];
+
+	/* Identify the version of the header */
+	u16 hdr_version;
+
+	/* Indicate which target (e.g. architecture or SoC) */
+	u16 tgt_code;
+
+	/* Size of uintptr_t in power of 2. (e.g. 5 for 32-bit, 6 for 64-bit) */
+	u8 ptr_size_bits;
+
+	u8 flag;
+
+	/* Reason for the fatal error */
+	u32 reason;
+} __packed;
+
+/* Architecture-specific block header */
+struct sof_ipc4_coredump_arch_hdr {
+	/* COREDUMP_ARCH_HDR_ID to indicate this is a architecture-specific block */
+	char id;
+
+	/* Identify the version of this block */
+	u16 hdr_version;
+
+	/* Number of bytes following the header */
+	u16 num_bytes;
+} __packed;
+
+struct sof_ipc4_telemetry_slot_data {
+	u32 separator;
+	struct sof_ipc4_coredump_hdr hdr;
+	struct sof_ipc4_coredump_arch_hdr arch_hdr;
+	u32 arch_data[];
+} __packed;
+
+void sof_ipc4_create_exception_debugfs_node(struct snd_sof_dev *sdev);
+#endif
-- 
2.42.0

[PATCH 5/9] ASoC: SOF: ipc4: add exception node in sof debugfs directory

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

The exception node is created when FW is ready and clear to
zero when FW post boot.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/ipc4.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c
index 4673c53f6c03..82f2f196c9c2 100644
--- a/sound/soc/sof/ipc4.c
+++ b/sound/soc/sof/ipc4.c
@@ -15,6 +15,7 @@
 #include "sof-audio.h"
 #include "ipc4-fw-reg.h"
 #include "ipc4-priv.h"
+#include "ipc4-telemetry.h"
 #include "ops.h"
 
 static const struct sof_ipc4_fw_status {
@@ -598,6 +599,8 @@ static int ipc4_fw_ready(struct snd_sof_dev *sdev, struct sof_ipc4_msg *ipc4_msg
 	sdev->debug_box.offset = snd_sof_dsp_get_window_offset(sdev,
 							SOF_IPC4_DEBUG_WINDOW_IDX);
 
+	sof_ipc4_create_exception_debugfs_node(sdev);
+
 	dev_dbg(sdev->dev, "mailbox upstream 0x%x - size 0x%x\n",
 		inbox_offset, inbox_size);
 	dev_dbg(sdev->dev, "mailbox downstream 0x%x - size 0x%x\n",
-- 
2.42.0

[PATCH 6/9] ASoC: SOF: Intel: add telemetry retrieval support on Intel platforms

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Telemetry data is decoded based on intel xtensa design and printed in
kernel log by sof debug framework.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/intel/Makefile    |  3 +-
 sound/soc/sof/intel/telemetry.c | 95 +++++++++++++++++++++++++++++++++
 sound/soc/sof/intel/telemetry.h | 35 ++++++++++++
 3 files changed, 132 insertions(+), 1 deletion(-)
 create mode 100644 sound/soc/sof/intel/telemetry.c
 create mode 100644 sound/soc/sof/intel/telemetry.h

diff --git a/sound/soc/sof/intel/Makefile b/sound/soc/sof/intel/Makefile
index 030574dbc998..6489d0660d58 100644
--- a/sound/soc/sof/intel/Makefile
+++ b/sound/soc/sof/intel/Makefile
@@ -7,7 +7,8 @@ snd-sof-intel-hda-common-objs := hda.o hda-loader.o hda-stream.o hda-trace.o \
 				 hda-dsp.o hda-ipc.o hda-ctrl.o hda-pcm.o \
 				 hda-dai.o hda-dai-ops.o hda-bus.o \
 				 skl.o hda-loader-skl.o \
-				 apl.o cnl.o tgl.o icl.o mtl.o lnl.o hda-common-ops.o
+				 apl.o cnl.o tgl.o icl.o mtl.o lnl.o hda-common-ops.o \
+				 telemetry.o
 
 snd-sof-intel-hda-mlink-objs := hda-mlink.o
 
diff --git a/sound/soc/sof/intel/telemetry.c b/sound/soc/sof/intel/telemetry.c
new file mode 100644
index 000000000000..1a3b5c28a6f0
--- /dev/null
+++ b/sound/soc/sof/intel/telemetry.c
@@ -0,0 +1,95 @@
+// SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause)
+//
+// This file is provided under a dual BSD/GPLv2 license.  When using or
+// redistributing this file, you may do so under either license.
+//
+// Copyright(c) 2023 Intel Corporation. All rights reserved.
+
+/* telemetry data queried from debug window */
+
+#include <sound/sof/ipc4/header.h>
+#include <sound/sof/xtensa.h>
+#include "../ipc4-priv.h"
+#include "../sof-priv.h"
+#include "hda.h"
+#include "telemetry.h"
+
+void sof_ipc4_intel_dump_telemetry_state(struct snd_sof_dev *sdev, u32 flags)
+{
+	static const char invalid_slot_msg[] = "Core dump is not available due to";
+	struct sof_ipc4_telemetry_slot_data *telemetry_data;
+	struct sof_ipc_dsp_oops_xtensa *xoops;
+	struct xtensa_arch_block *block;
+	u32 slot_offset;
+	char *level;
+
+	level = (flags & SOF_DBG_DUMP_OPTIONAL) ? KERN_DEBUG : KERN_ERR;
+
+	slot_offset = sof_ipc4_find_debug_slot_offset_by_type(sdev, SOF_IPC4_DEBUG_SLOT_TELEMETRY);
+	if (!slot_offset)
+		return;
+
+	telemetry_data = kmalloc(sizeof(*telemetry_data), GFP_KERNEL);
+	if (!telemetry_data)
+		return;
+	sof_mailbox_read(sdev, slot_offset, telemetry_data, sizeof(*telemetry_data));
+	if (telemetry_data->separator != XTENSA_CORE_DUMP_SEPARATOR) {
+		dev_err(sdev->dev, "%s invalid separator %#x\n", invalid_slot_msg,
+			telemetry_data->separator);
+		goto free_telemetry_data;
+	}
+
+	block = kmalloc(sizeof(*block), GFP_KERNEL);
+	if (!block)
+		goto free_telemetry_data;
+
+	sof_mailbox_read(sdev, slot_offset + sizeof(*telemetry_data), block, sizeof(*block));
+	if (block->soc != XTENSA_SOC_INTEL_ADSP) {
+		dev_err(sdev->dev, "%s invalid SOC %d\n", invalid_slot_msg, block->soc);
+		goto free_block;
+	}
+
+	if (telemetry_data->hdr.id[0] != COREDUMP_HDR_ID0 ||
+	    telemetry_data->hdr.id[1] != COREDUMP_HDR_ID1 ||
+	    telemetry_data->arch_hdr.id != COREDUMP_ARCH_HDR_ID) {
+		dev_err(sdev->dev, "%s invalid coredump header %c%c, arch hdr %c\n",
+			invalid_slot_msg, telemetry_data->hdr.id[0],
+			telemetry_data->hdr.id[1],
+			telemetry_data->arch_hdr.id);
+		goto free_block;
+	}
+
+	switch (block->toolchain) {
+	case XTENSA_TOOL_CHAIN_ZEPHYR:
+		dev_printk(level, sdev->dev, "FW is built with Zephyr toolchain\n");
+		break;
+	case XTENSA_TOOL_CHAIN_XCC:
+		dev_printk(level, sdev->dev, "FW is built with XCC toolchain\n");
+		break;
+	default:
+		dev_printk(level, sdev->dev, "Unknown toolchain is used\n");
+		break;
+	}
+
+	xoops = kzalloc(struct_size(xoops, ar, XTENSA_CORE_AR_REGS_COUNT), GFP_KERNEL);
+	if (!xoops)
+		goto free_block;
+
+	xoops->exccause = block->exccause;
+	xoops->excvaddr = block->excvaddr;
+	xoops->epc1 = block->pc;
+	xoops->ps = block->ps;
+	xoops->sar = block->sar;
+
+	xoops->plat_hdr.numaregs = XTENSA_CORE_AR_REGS_COUNT;
+	memcpy((void *)xoops->ar, block->ar, XTENSA_CORE_AR_REGS_COUNT * sizeof(u32));
+
+	sof_oops(sdev, level, xoops);
+	sof_stack(sdev, level, xoops, NULL, 0);
+
+	kfree(xoops);
+free_block:
+	kfree(block);
+free_telemetry_data:
+	kfree(telemetry_data);
+}
diff --git a/sound/soc/sof/intel/telemetry.h b/sound/soc/sof/intel/telemetry.h
new file mode 100644
index 000000000000..3c2b23c75f5d
--- /dev/null
+++ b/sound/soc/sof/intel/telemetry.h
@@ -0,0 +1,35 @@
+/* SPDX-License-Identifier: (GPL-2.0-only OR BSD-3-Clause) */
+/*
+ * This file is provided under a dual BSD/GPLv2 license.  When using or
+ * redistributing this file, you may do so under either license.
+ *
+ * Copyright(c) 2023 Intel Corporation. All rights reserved.
+ *
+ * telemetry data in debug windows
+ */
+
+#ifndef _SOF_INTEL_TELEMETRY_H
+#define _SOF_INTEL_TELEMETRY_H
+
+#include "../ipc4-telemetry.h"
+
+struct xtensa_arch_block {
+	u8	soc; /* should be equal to XTENSA_SOC_INTEL_ADSP */
+	u16	version;
+	u8	toolchain; /* ZEPHYR or XCC */
+
+	u32	pc;
+	u32	exccause;
+	u32	excvaddr;
+	u32	sar;
+	u32	ps;
+	u32	scompare1;
+	u32	ar[XTENSA_CORE_AR_REGS_COUNT];
+	u32	lbeg;
+	u32	lend;
+	u32	lcount;
+} __packed;
+
+void sof_ipc4_intel_dump_telemetry_state(struct snd_sof_dev *sdev, u32 flags);
+
+#endif /* _SOF_INTEL_TELEMETRY_H */
-- 
2.42.0

[PATCH 7/9] ASoC: SOF: Intel: mtl: dump dsp stack

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Dump dsp stack with sof_ipc4_intel_dump_telemetry_state since dsp stack
information is included by telemetry data. This also supports lnl since
the mtl code is reused.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/intel/mtl.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/sound/soc/sof/intel/mtl.c b/sound/soc/sof/intel/mtl.c
index f9412517eaf2..254dbbeac1d0 100644
--- a/sound/soc/sof/intel/mtl.c
+++ b/sound/soc/sof/intel/mtl.c
@@ -18,6 +18,7 @@
 #include "hda-ipc.h"
 #include "../sof-audio.h"
 #include "mtl.h"
+#include "telemetry.h"
 
 static const struct snd_sof_debugfs_map mtl_dsp_debugfs[] = {
 	{"hda", HDA_DSP_HDA_BAR, 0, 0x4000, SOF_DEBUGFS_ACCESS_ALWAYS},
@@ -320,6 +321,8 @@ void mtl_dsp_dump(struct snd_sof_dev *sdev, u32 flags)
 	romdbgsts = snd_sof_dsp_read(sdev, HDA_DSP_BAR, MTL_DSP_REG_HFFLGPXQWY + 0x8 * 3);
 	dev_printk(level, sdev->dev, "ROM feature bit%s enabled\n",
 		   romdbgsts & BIT(24) ? "" : " not");
+
+	sof_ipc4_intel_dump_telemetry_state(sdev, flags);
 }
 
 static bool mtl_dsp_primary_core_is_enabled(struct snd_sof_dev *sdev)
-- 
2.42.0

[PATCH 8/9] ASoC: SOF: Intel: hda: add ipc4 FW panic support on CAVS 2.5+ platforms

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Get the FW panic information from telemetry data in memory window and
dump it to kernel log. The old platforms before CAVS 2.5+ don't support
it since there is no support in FW for them.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/intel/hda.c | 14 ++++++++++++++
 sound/soc/sof/intel/hda.h |  1 +
 sound/soc/sof/intel/tgl.c |  1 +
 3 files changed, 16 insertions(+)

diff --git a/sound/soc/sof/intel/hda.c b/sound/soc/sof/intel/hda.c
index 15e6779efaa3..02c82ccb9f66 100644
--- a/sound/soc/sof/intel/hda.c
+++ b/sound/soc/sof/intel/hda.c
@@ -31,6 +31,7 @@
 #include "../sof-pci-dev.h"
 #include "../ops.h"
 #include "hda.h"
+#include "telemetry.h"
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/sof_intel.h>
@@ -731,6 +732,19 @@ void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags)
 	}
 }
 
+void hda_ipc4_dsp_dump(struct snd_sof_dev *sdev, u32 flags)
+{
+	char *level = (flags & SOF_DBG_DUMP_OPTIONAL) ? KERN_DEBUG : KERN_ERR;
+
+	/* print ROM/FW status */
+	hda_dsp_get_state(sdev, level);
+
+	if (flags & SOF_DBG_DUMP_REGS)
+		sof_ipc4_intel_dump_telemetry_state(sdev, flags);
+	else
+		hda_dsp_dump_ext_rom_status(sdev, level, flags);
+}
+
 static bool hda_check_ipc_irq(struct snd_sof_dev *sdev)
 {
 	const struct sof_intel_dsp_desc *chip;
diff --git a/sound/soc/sof/intel/hda.h b/sound/soc/sof/intel/hda.h
index 2b228c63905b..7c575ba9462c 100644
--- a/sound/soc/sof/intel/hda.h
+++ b/sound/soc/sof/intel/hda.h
@@ -603,6 +603,7 @@ int hda_dsp_shutdown_dma_flush(struct snd_sof_dev *sdev);
 int hda_dsp_shutdown(struct snd_sof_dev *sdev);
 int hda_dsp_set_hw_params_upon_resume(struct snd_sof_dev *sdev);
 void hda_dsp_dump(struct snd_sof_dev *sdev, u32 flags);
+void hda_ipc4_dsp_dump(struct snd_sof_dev *sdev, u32 flags);
 void hda_ipc_dump(struct snd_sof_dev *sdev);
 void hda_ipc_irq_dump(struct snd_sof_dev *sdev);
 void hda_dsp_d0i3_work(struct work_struct *work);
diff --git a/sound/soc/sof/intel/tgl.c b/sound/soc/sof/intel/tgl.c
index bb9f20253c99..4a61f6d28ae5 100644
--- a/sound/soc/sof/intel/tgl.c
+++ b/sound/soc/sof/intel/tgl.c
@@ -102,6 +102,7 @@ int sof_tgl_ops_init(struct snd_sof_dev *sdev)
 
 		/* debug */
 		sof_tgl_ops.ipc_dump	= cnl_ipc4_dump;
+		sof_tgl_ops.dbg_dump	= hda_ipc4_dsp_dump;
 
 		sof_tgl_ops.set_power_state = hda_dsp_set_power_state_ipc4;
 	}
-- 
2.42.0

[PATCH 9/9] ASoC: SOF: ipc4: handle EXCEPTION_CAUGHT notification from firmware

[Peter Ujfalusi]

From: Rander Wang <rander.wang@intel.com>

Driver will receive exception IPC message and process it by
snd_sof_dsp_panic.

Signed-off-by: Rander Wang <rander.wang@intel.com>
Reviewed-by: Péter Ujfalusi <peter.ujfalusi@linux.intel.com>
Reviewed-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Reviewed-by: Pierre-Louis Bossart <pierre-louis.bossart@linux.intel.com>
Reviewed-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@linux.intel.com>
---
 sound/soc/sof/ipc4.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/sound/soc/sof/ipc4.c b/sound/soc/sof/ipc4.c
index 82f2f196c9c2..3f4d57dba972 100644
--- a/sound/soc/sof/ipc4.c
+++ b/sound/soc/sof/ipc4.c
@@ -645,6 +645,9 @@ static void sof_ipc4_rx_msg(struct snd_sof_dev *sdev)
 	case SOF_IPC4_NOTIFY_LOG_BUFFER_STATUS:
 		sof_ipc4_mtrace_update_pos(sdev, SOF_IPC4_LOG_CORE_GET(ipc4_msg->primary));
 		break;
+	case SOF_IPC4_NOTIFY_EXCEPTION_CAUGHT:
+		snd_sof_dsp_panic(sdev, 0, true);
+		break;
 	default:
 		dev_dbg(sdev->dev, "Unhandled DSP message: %#x|%#x\n",
 			ipc4_msg->primary, ipc4_msg->extension);
-- 
2.42.0

Re: [PATCH 0/9] ASoC: SOF: ipc4/Intel: Support for firmware exception handling

[Mark Brown]

On Tue, 19 Sep 2023 12:24:07 +0300, Peter Ujfalusi wrote:
> When a firmware crashes it creats a panic information into a telemetry slot.
> The panic format is defined by Zephyr, includes stack and additional information
> to help to identify the reason for the crash.
> Part of the firmware exception handling the firmware also sends an
> EXCEPTION_CAUGHT notification.
> 
> This series implements the kernel side handling of the exception:
> print information into the kernel log
> export the whole telemetry slot to user space for tools extract additional
> information from the panic dump.
> 
> [...]

Applied to

   https://git.kernel.org/pub/scm/linux/kernel/git/broonie/sound.git for-next

Thanks!

[1/9] ASoC: SOF: Xtensa: dump ar registers to restore call stack
      commit: 58bb5081cba130f12c26d8e4d5e9416a0272f07e
[2/9] ASoC: SOF: ipc4-mtrace: move debug slot related definitions to header.h
      commit: 4287205065f244f4d40ae6aa7875b3ebffedcf8d
[3/9] ASoC: SOF: ipc4: add a helper function to search debug slot
      commit: a397899f81d52202265d4977a99085f53e426826
[4/9] ASoC: SOF: ipc4: add definition of telemetry slot for exception handling
      commit: ab05061d25806515358d184eb4d305f7f12befdc
[5/9] ASoC: SOF: ipc4: add exception node in sof debugfs directory
      commit: 80b567f8995757d36008f835853cea8d2f7c34c0
[6/9] ASoC: SOF: Intel: add telemetry retrieval support on Intel platforms
      commit: c8b54a2f7af41740b5faad2f6846d927b14369ca
[7/9] ASoC: SOF: Intel: mtl: dump dsp stack
      commit: e449b18ff03c2f90430d00486fd713854b28c077
[8/9] ASoC: SOF: Intel: hda: add ipc4 FW panic support on CAVS 2.5+ platforms
      commit: eb6e5dab11401c64f5d5576c71e5fc0a4c7b321a
[9/9] ASoC: SOF: ipc4: handle EXCEPTION_CAUGHT notification from firmware
      commit: c1c48fd6bbe788458e3685fea74bdb3cb148ff93

All being well this means that it will be integrated into the linux-next
tree (usually sometime in the next 24 hours) and sent to Linus during
the next merge window (or sooner if it is a bug fix), however if
problems are discovered then the patch may be dropped or reverted.

You may get further e-mails resulting from automated or manual testing
and review of the tree, please engage with people reporting problems and
send followup patches addressing any issues that are reported if needed.

If any updates are required or you are submitting further changes they
should be sent as incremental updates against current git, existing
patches will not be replaced.

Please add any relevant lists and maintainers to the CCs when replying
to this mail.

Thanks,
Mark