RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Ruipeng Qi]

<+How a kernel client driver can register region with minidump
<+------------------------------------------------------------
<+
<+Client driver can use ``qcom_minidump_region_register`` API's to register
<+and ``qcom_minidump_region_unregister`` to unregister their region from
<+minidump driver.
<+
<+Client needs to fill their region by filling ``qcom_minidump_region``
<+structure object which consists of the region name, region's virtual
<+and physical address and its size.

Hi, Mukesh, wish you a good holiday :)

I have the following idea, please help me to assess whether this can be
implemented or not. As we all know, most of the kernel objects are
allocated by the slab sub-system.I wonder if we can dump all memory
keeped by the slab sub-system? If so,  we got most of the kernel objects
which will be helpful to fix problems when we run with system issues.

How can we do this? From the description above, I think we should
register one region for each slab,  for each slab will have some pages,
and the memory between each slab is non-continuous. As we all
know, there are millions of slabs in the system, so if we dump slabs
in this way, it will introduce a heavy overhead.

I am not very familiar with qualcomm minidump, maybe my thought
is wrong. Looking forward to your reply!

Best Regards
Ruipeng

[Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Mukesh Ojha]

Add the qualcomm minidump guide for the users which tries to cover
the dependency, API use and the way to test and collect minidump
on Qualcomm supported SoCs.

Signed-off-by: Mukesh Ojha <quic_mojha@quicinc.com>
---
 Documentation/admin-guide/index.rst         |   1 +
 Documentation/admin-guide/qcom_minidump.rst | 272 ++++++++++++++++++++++++++++
 2 files changed, 273 insertions(+)
 create mode 100644 Documentation/admin-guide/qcom_minidump.rst

diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 43ea35613dfc..251d070486c2 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -120,6 +120,7 @@ configure specific aspects of kernel behavior to your liking.
    perf-security
    pm/index
    pnp
+   qcom_minidump
    rapidio
    ras
    rtc
diff --git a/Documentation/admin-guide/qcom_minidump.rst b/Documentation/admin-guide/qcom_minidump.rst
new file mode 100644
index 000000000000..b492f2b79639
--- /dev/null
+++ b/Documentation/admin-guide/qcom_minidump.rst
@@ -0,0 +1,272 @@
+Qualcomm minidump feature
+=========================
+
+Introduction
+------------
+
+Minidump is a best effort mechanism to collect useful and predefined
+data for first level of debugging on end user devices running on
+Qualcomm SoCs. It is built on the premise that System on Chip (SoC)
+or subsystem part of SoC crashes, due to a range of hardware and
+software bugs. Hence, the ability to collect accurate data is only
+a best-effort. The data collected could be invalid or corrupted, data
+collection itself could fail, and so on.
+
+Qualcomm devices in engineering mode provides a mechanism for generating
+full system RAM dumps for post-mortem debugging. But in some cases it's
+however not feasible to capture the entire content of RAM. The minidump
+mechanism provides the means for selected region should be included in
+the ramdump.
+
+
+::
+
+   +-----------------------------------------------+
+   |   DDR                       +-------------+   |
+   |                             |      SS0-ToC|   |
+   | +----------------+     +----------------+ |   |
+   | |Shared memory   |     |         SS1-ToC| |   |
+   | |(SMEM)          |     |                | |   |
+   | |                | +-->|--------+       | |   |
+   | |G-ToC           | |   | SS-ToC  \      | |   |
+   | |+-------------+ | |   | +-----------+  | |   |
+   | ||-------------| | |   | |-----------|  | |   |
+   | || SS0-ToC     | | | +-|<|SS1 region1|  | |   |
+   | ||-------------| | | | | |-----------|  | |   |
+   | || SS1-ToC     |-|>+ | | |SS1 region2|  | |   |
+   | ||-------------| |   | | |-----------|  | |   |
+   | || SS2-ToC     | |   | | |  ...      |  | |   |
+   | ||-------------| |   | | |-----------|  | |   |
+   | ||  ...        | |   |-|<|SS1 regionN|  | |   |
+   | ||-------------| |   | | |-----------|  | |   |
+   | || SSn-ToC     | |   | | +-----------+  | |   |
+   | |+-------------+ |   | |                | |   |
+   | |                |   | |----------------| |   |
+   | |                |   +>|  regionN       | |   |
+   | |                |   | |----------------| |   |
+   | +----------------+   | |                | |   |
+   |                      | |----------------| |   |
+   |                      +>|  region1       | |   |
+   |                        |----------------| |   |
+   |                        |                | |   |
+   |                        |----------------|-+   |
+   |                        |  region5       |     |
+   |                        |----------------|     |
+   |                        |                |     |
+   |  Region information    +----------------+     |
+   | +---------------+                             |
+   | |region name    |                             |
+   | |---------------|                             |
+   | |region address |                             |
+   | |---------------|                             |
+   | |region size    |                             |
+   | +---------------+                             |
+   +-----------------------------------------------+
+       G-ToC: Global table of contents
+       SS-ToC: Subsystem table of contents
+       SS0-SSn: Subsystem numbered from 0 to n
+
+It depends on SoC where the underlying firmware is keeping the
+minidump global table taking care of subsystem ToC part for
+minidump like for above diagram, it is for shared memory sitting
+in DDR and it is shared among various master however it is possible
+that this could be implemented via memory mapped regions but the
+general idea should remain same. Here, various subsystem could be
+DSP's like ADSP/CDSP/MODEM etc, along with Application processor
+(APSS) where Linux runs. DSP minidump gets collected when DSP's goes
+for recovery followed by a crash. The minidump part of code for
+that resides in ``qcom_rproc_minidump.c``.
+
+
+SMEM as backend
+----------------
+
+In this document, SMEM will be used as the backend implementation
+of minidump.
+
+The core of minidump feature is part of Qualcomm's boot firmware code.
+It initializes shared memory (SMEM), which is a part of DDR and
+allocates a small section of it to minidump table, i.e. also called
+global table of contents (G-ToC). Each subsystem (APSS, ADSP, ...) has
+its own table of segments to be included in the minidump, all
+references from a descriptor in SMEM (G-ToC). Each segment/region has
+some details like name, physical address and its size etc. and it
+could be anywhere scattered in the DDR.
+
+Qualcomm APSS Minidump kernel driver concept
+--------------------------------------------
+
+Qualcomm APSS minidump kernel driver adds the capability to add Linux
+region to be dumped as part of RAM dump collection. At the moment,
+shared memory driver creates platform device for minidump driver and
+give a means to APSS minidump to initialize itself on probe.
+
+This driver provides ``qcom_minidump_region_register`` and
+``qcom_minidump_region_unregister`` API's to register and unregister
+APSS minidump region. It also supports registration for the clients
+who came before minidump driver was initialized. It maintains pending
+list of clients who came before minidump and once minidump is initialized
+it registers them in one go.
+
+To simplify post-mortem debugging, driver creates and maintain an ELF
+header as first region that gets updated each time a new region gets
+registered.
+
+The solution supports extracting the RAM dump/minidump produced either
+over USB or stored to an attached storage device.
+
+Dependency of minidump kernel driver
+------------------------------------
+
+It is to note that whole of minidump depends on Qualcomm boot firmware
+whether it supports minidump or not. So, if the minidump SMEM ID is
+present in shared memory, it indicates that minidump is supported from
+boot firmware and it is possible to dump Linux (APSS) region as part
+of minidump collection.
+
+How a kernel client driver can register region with minidump
+------------------------------------------------------------
+
+Client driver can use ``qcom_minidump_region_register`` API's to register
+and ``qcom_minidump_region_unregister`` to unregister their region from
+minidump driver.
+
+Client needs to fill their region by filling ``qcom_minidump_region``
+structure object which consists of the region name, region's virtual
+and physical address and its size.
+
+Below, is one sample client driver snippet which tries to allocate a
+region from kernel heap of certain size and it writes a certain known
+pattern (that can help in verification after collection that we got
+the exact pattern, what we wrote) and registers it with minidump.
+
+ .. code-block:: c
+
+  #include <soc/qcom/qcom_minidump.h>
+  [...]
+
+
+  [... inside a function ...]
+  struct qcom_minidump_region region;
+
+  [...]
+
+  client_mem_region = kzalloc(region_size, GFP_KERNEL);
+  if (!client_mem_region)
+	return -ENOMEM;
+
+  [... Just write a pattern ...]
+  memset(client_mem_region, 0xAB, region_size);
+
+  [... Fill up the region object ...]
+  strlcpy(region.name, "REGION_A", sizeof(region.name));
+  region.virt_addr = client_mem_region;
+  region.phys_addr = virt_to_phys(client_mem_region);
+  region.size = region_size;
+
+  ret = qcom_minidump_region_register(&region);
+  if (ret < 0) {
+	pr_err("failed to add region in minidump: err: %d\n", ret);
+	return ret;
+  }
+
+  [...]
+
+
+Test
+----
+
+Existing Qualcomm devices already supports entire RAM dump (also called
+full dump) by writing appropriate value to Qualcomm's top control and
+status register (tcsr) in ``driver/firmware/qcom_scm.c`` .
+
+SCM device Tree bindings required to support download mode
+For example (sm8450) ::
+
+	/ {
+
+	[...]
+
+		firmware {
+			scm: scm {
+				compatible = "qcom,scm-sm8450", "qcom,scm";
+				[... tcsr register ... ]
+				qcom,dload-mode = <&tcsr 0x13000>;
+
+				[...]
+			};
+		};
+
+	[...]
+
+		soc: soc@0 {
+
+			[...]
+
+			tcsr: syscon@1fc0000 {
+				compatible = "qcom,sm8450-tcsr", "syscon";
+				reg = <0x0 0x1fc0000 0x0 0x30000>;
+			};
+
+			[...]
+		};
+	[...]
+
+	};
+
+User of minidump can pass ``qcom_scm.download_mode="mini"`` to kernel
+commandline to set the current download mode to minidump.
+Similarly, ``"full"`` is passed to set the download mode to full dump
+where entire RAM dump will be collected while setting it ``"full,mini"``
+will collect minidump along with fulldump.
+
+Writing to sysfs node can also be used to set the mode to minidump::
+
+	echo "mini" > /sys/module/qcom_scm/parameter/download_mode
+
+Once the download mode is set, any kind of crash will make the device collect
+respective dump as per set download mode.
+
+Dump collection
+---------------
+::
+
+	+-----------+
+	|           |
+	|           |         +------+
+	|           |         |      |
+	|           |         +--+---+ Product(Qualcomm SoC)
+	+-----------+             |
+	|+++++++++++|<------------+
+	|+++++++++++|    usb cable
+	+-----------+
+            x86_64 PC
+
+The solution supports a product running with Qualcomm SoC (where minidump)
+is supported from the firmware) connected to x86_64 host PC running PCAT
+tool. It supports downloading the minidump produced from product to the
+host PC over USB or to save the minidump to the product attached storage
+device(UFS/eMMC/SD Card) into minidump dedicated partition.
+
+By default, dumps are downloaded via USB to the attached x86_64 PC running
+PCAT (Qualcomm tool) software. Upon download, we will see a set of binary
+blobs starting with name ``md_*`` in PCAT configured directory in x86_64
+machine, so for above example from the client it will be ``md_REGION_A.BIN``.
+This binary blob depends on region content to determine whether it needs
+external parser support to get the content of the region, so for simple
+plain ASCII text we don't need any parsing and the content can be seen
+just opening the binary file.
+
+To collect the dump to attached storage type, one needs to write appropriate
+value to IMEM register, in that case dumps are collected in rawdump
+partition on the product device itself.
+
+One needs to read the entire rawdump partition and pull out content to
+save it onto the attached x86_64 machine over USB. Later, this rawdump
+can be passed to another tool (``dexter.exe`` [Qualcomm tool]) which
+converts this into the similar binary blobs which we have got it when
+download type was set to USB, i.e. a set of registered regions as blobs
+and their name starts with ``md_*``.
+
+Replacing the ``dexter.exe`` with some open source tool can be added as future
+scope of this document.
-- 
2.7.4

RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Ruipeng Qi]

<+How a kernel client driver can register region with minidump
<+------------------------------------------------------------
<+
<+Client driver can use ``qcom_minidump_region_register`` API's to register
<+and ``qcom_minidump_region_unregister`` to unregister their region from
<+minidump driver.
<+
<+Client needs to fill their region by filling ``qcom_minidump_region``
<+structure object which consists of the region name, region's virtual
<+and physical address and its size.

Hi, Mukesh, wish you a good holiday :)

I have the following idea, please help me to assess whether this can be
implemented or not. As we all know, most of the kernel objects are
allocated by the slab sub-system.I wonder if we can dump all memory
keeped by the slab sub-system? If so,  we got most of the kernel objects
which will be helpful to fix problems when we run with system issues.

How can we do this? From the description above, I think we should
register one region for each slab,  for each slab will have some pages,
and the memory between each slab is non-continuous. As we all
know, there are millions of slabs in the system, so if we dump slabs
in this way, it will introduce a heavy overhead.

I am not very familiar with qualcomm minidump, maybe my thought
is wrong. Looking forward to your reply!

Best Regards
Ruipeng

Re: RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Mukesh Ojha]

On 12/25/2023 7:25 PM, Ruipeng Qi wrote:
> <+How a kernel client driver can register region with minidump
> <+------------------------------------------------------------
> <+
> <+Client driver can use ``qcom_minidump_region_register`` API's to register
> <+and ``qcom_minidump_region_unregister`` to unregister their region from
> <+minidump driver.
> <+
> <+Client needs to fill their region by filling ``qcom_minidump_region``
> <+structure object which consists of the region name, region's virtual
> <+and physical address and its size.
> 
> Hi, Mukesh, wish you a good holiday :)

Hope you had the same..:-)

> 
> I have the following idea, please help me to assess whether this can be
> implemented or not. As we all know, most of the kernel objects are
> allocated by the slab sub-system.I wonder if we can dump all memory
> keeped by the slab sub-system? If so,  we got most of the kernel objects
> which will be helpful to fix problems when we run with system issues.
> 
> How can we do this? From the description above, I think we should
> register one region for each slab,  for each slab will have some pages,
> and the memory between each slab is non-continuous. As we all
> know, there are millions of slabs in the system, so if we dump slabs
> in this way, it will introduce a heavy overhead.
> 
> I am not very familiar with qualcomm minidump, maybe my thought
> is wrong. Looking forward to your reply!

In the current state and in simple terms, Qualcomm Minidump can not do
this, Minidump is more of a consumer driver so, what ever gets
registered with it, it can dump. Qualcomm Minidump serves bigger purpose
to dump content in any kind of crash whether it is kernel or non-kernel
like NOC errors/XPUs etc and both kernel/non-kernel entity can register 
to it, so we gets dump in any kind of system crash.

One more thing, kernel part of minidump, we are calling it APSS Minidump
has limitation of no of entries so it will be difficult to dump 
non-continuous regions after a certain number of registration ~200. However,
we do have a solution in downstream kernel for it like to create a big 
CMA buffer and register this buffer with Minidump so that whatever gets 
dumped in that buffer gets captured during crash and fill up this buffer 
and create elf during panic. I think, similar thing you are also doing 
with your OS-minidump.

I have just glanced into your implementation of OS-minidump, it
more of relying on basic concept of RAM content preserved
across boot and later reading it through procfs but this basic
stuff is common to pstore(ram) as well and pstore has file system 
support why don't you make your driver as one of pstore record and that 
way Qualcomm minidump also gets benefited where entire OS-minidump 
record gets registered with Qualcomm minidump and we get this on panic 
and you get this via pstorefs.

-Mukesh

> 
> Best Regards
> Ruipeng

Re: RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Ruipeng Qi]

On Wed, Jan 3, 2024 at 11:27 PM Mukesh Ojha <quic_mojha@quicinc.com> wrote:
>
>
> One more thing, kernel part of minidump, we are calling it APSS Minidump
> has limitation of no of entries so it will be difficult to dump
> non-continuous regions after a certain number of registration ~200. However,
> we do have a solution in downstream kernel for it like to create a big
> CMA buffer and register this buffer with Minidump so that whatever gets
> dumped in that buffer gets captured during crash and fill up this buffer
> and create elf during panic. I think, similar thing you are also doing
> with your OS-minidump.
>
> I have just glanced into your implementation of OS-minidump, it
> more of relying on basic concept of RAM content preserved
> across boot and later reading it through procfs but this basic
> stuff is common to pstore(ram) as well and pstore has file system
> support why don't you make your driver as one of pstore record and that
> way Qualcomm minidump also gets benefited where entire OS-minidump
> record gets registered with Qualcomm minidump and we get this on panic
> and you get this via pstorefs.
>
Thanks Mukesh！It is a good suggestion to move OS-minidump forward!
By the way, I have some questions here for which I need your assistance.

Firstly,I can reimplement OS-minidump as one of the pstore records to
dump data. The resulting dump file would contain thousands of
non-contiguous memory regions, each with only the virtual address and
size recorded. As far as I know, Qualcomm's minidump can handle
several memory regions, each with a physical address and size.
This seems to be a difference, and I'm curious as to how you deal with
data dumped by OS-minidump. I would really appreciate it if you could
provide more details on your approach.

Secondly, what tools do you use to analyze the dump data, and does it
support crash tool?

Lastly, is Qualcomm minidump compatible with non-Qualcomm SoCs,
and if so, how can one use it?

Best Regards
Ruipeng Qi

Re: RESEND: Re: [Patch v6 03/12] docs: qcom: Add qualcomm minidump guide

[Mukesh Ojha]

On 1/8/2024 9:04 PM, Ruipeng Qi wrote:
> On Wed, Jan 3, 2024 at 11:27 PM Mukesh Ojha <quic_mojha@quicinc.com> wrote:
>>
>>
>> One more thing, kernel part of minidump, we are calling it APSS Minidump
>> has limitation of no of entries so it will be difficult to dump
>> non-continuous regions after a certain number of registration ~200. However,
>> we do have a solution in downstream kernel for it like to create a big
>> CMA buffer and register this buffer with Minidump so that whatever gets
>> dumped in that buffer gets captured during crash and fill up this buffer
>> and create elf during panic. I think, similar thing you are also doing
>> with your OS-minidump.
>>
>> I have just glanced into your implementation of OS-minidump, it
>> more of relying on basic concept of RAM content preserved
>> across boot and later reading it through procfs but this basic
>> stuff is common to pstore(ram) as well and pstore has file system
>> support why don't you make your driver as one of pstore record and that
>> way Qualcomm minidump also gets benefited where entire OS-minidump
>> record gets registered with Qualcomm minidump and we get this on panic
>> and you get this via pstorefs.
>>
> Thanks Mukesh！It is a good suggestion to move OS-minidump forward!
> By the way, I have some questions here for which I need your assistance.
> 
> Firstly,I can reimplement OS-minidump as one of the pstore records to
> dump data. The resulting dump file would contain thousands of
> non-contiguous memory regions, each with only the virtual address and
> size recorded. As far as I know, Qualcomm's minidump can handle
> several memory regions, each with a physical address and size.
> This seems to be a difference, and I'm curious as to how you deal with
> data dumped by OS-minidump. I would really appreciate it if you could
> provide more details on your approach.

What my thought was to think your OS-minidump to be one of pstore record 
similar to existing records like console, ftrace, pmsg, dmesg etc.,
If you follow this series patch 11/12 and 12/12 is trying to get the
pstore(ram) record information and registering with minidump and here
the physical address are of the ramoops record addresses.

So, once you are capturing everything inside in a record, all minidump
has to do is get your Os-minidump record physical address and size
and register with minidump.
> 
> Secondly, what tools do you use to analyze the dump data, and does it
> support crash tool?

Currently, we are trying to capture only pstore ramoops region in text
format and not been using any tool.

Since, Qualcomm minidump is controlled from boot firmware and it can
not be used on non-Qualcomm SoCs so here minidump driver and its usecase
is limited to capture only pstore (ram)records for targets where RAM
content is not guaranteed to be preserved across boots.

So, you can think minidump as one of ramoops backend which will be
dumping all the ramoops regions/records/zones.

+---------+     +---------+   +--------+      +---------+
  | console |     | pmsg    |   | ftrace |     | dmesg   | ...Os-minidump
  +---------+     +---------+   +--------+     +---------+
        |             |             |              |
        |             |             |              |
        +------------------------------------------+
                           |
                          \ /
                   +----------------+
             (1)   |pstore frontends|
                   +----------------+
                           |
                          \ /
                  +------------------- +
             (2)  | pstore backend(ram)|
                  +--------------------+
                           |
                          \ /

                    +---------------+
             (3)    | qcom_minidump |
                    +---------------+


> 
> Lastly, is Qualcomm minidump compatible with non-Qualcomm SoCs,
> and if so, how can one use it?

I already replied it above.

-Mukesh

> 
> Best Regards
> Ruipeng Qi