public inbox for linux-mm@kvack.org
 help / color / mirror / Atom feed
* [PATCH v2 00/25] Introduce meminspect
@ 2026-03-10 20:15 Mukesh Ojha
  2026-03-10 20:15 ` [PATCH v2 01/25] kernel: " Mukesh Ojha
                   ` (25 more replies)
  0 siblings, 26 replies; 42+ messages in thread
From: Mukesh Ojha @ 2026-03-10 20:15 UTC (permalink / raw)
  To: Jonathan Corbet, Shuah Khan, Eugen Hristev, Arnd Bergmann,
	Dennis Zhou, Tejun Heo, Christoph Lameter, Andrew Morton,
	Thomas Gleixner, Peter Zijlstra, Anna-Maria Behnsen,
	Frederic Weisbecker, Ingo Molnar, Juri Lelli, Vincent Guittot,
	Dietmar Eggemann, Steven Rostedt, Ben Segall, Mel Gorman,
	Valentin Schneider, David Hildenbrand, Lorenzo Stoakes,
	Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Kees Cook, Brendan Jackman,
	Johannes Weiner, Zi Yan, Chris Li, Kairui Song, Kemeng Shi,
	Nhat Pham, Baoquan He, Barry Song, Youngjun Park, Petr Mladek,
	John Ogness, Sergey Senozhatsky, Bjorn Andersson, Mathieu Poirier,
	Konrad Dybcio, Mukesh Ojha, Rob Herring, Krzysztof Kozlowski,
	Conor Dooley, Saravana Kannan
  Cc: workflows, linux-doc, linux-kernel, linux-arch, linux-mm,
	linux-arm-msm, linux-remoteproc, devicetree

First of all, I want to thank Eugene for his excellent work on this
series. What began as the Qualcomm Minidump driver from me has now
evolved into meminspect. He also presented meminspect a few months ago
at Linux Plumbers 2025.

Video of the recording is available here for anyone interested:
https://www.youtube.com/watch?v=aDZv4-kOLSc

Introduction:

meminspect is a mechanism which allows the kernel to mark specific
memory areas for memory dumping or specific inspection, statistics,
usage.  Once regions are marked, meminspect keeps an internal list with
the regions in a dedicated table.  Further, these regions can be
accessed using specific API by any interested driver.  Regions being
marked beforehand, when the system is up and running, there is no need
nor dependency on a panic handler, or a working kernel that can dump the
debug information.  meminspect can be primarily used for debugging. The
approach is feasible to work when pstore, kdump, or another mechanism do
not.  Pstore relies on persistent storage, a dedicated RAM area or
flash, which has the disadvantage of having the memory reserved all the
time, or another specific non volatile memory. Some devices cannot keep
the RAM contents on reboot so ramoops does not work. Some devices do not
allow kexec to run another kernel to debug the crashed one.  For such
devices, that have another mechanism to help debugging, like firmware,
kmemdump is a viable solution.

meminspect can create a core image, similar with /proc/vmcore, with only
the registered regions included. This can be loaded into crash tool/gdb
and analyzed. This happens if CRASH_DUMP=y.  To have this working,
specific information from the kernel is registered, and this is done at
meminspect init time, no need for the meminspect users to do anything.

This version of the meminspect patch series includes two drivers that
make use of it: one is the Qualcomm Minidump, and the other one is the
Debug Kinfo backend for Android devices, reworked from this source here:
https://android.googlesource.com/kernel/common/+/refs/heads/android-mainline/drivers/android/debug_kinfo.c
written originally by Jone Chou <jonechou@google.com>

*** History, motivation and available online resources ***

The patch series is based on both minidump and kmemdump previous implementations.

After the three RFC kmemdump versions, considering the ML discussions, it was decided to
move this into kernel/ directory and rework it into naming it meminspect, as Thomas Gleixner
suggested.

Initial version of kmemdump and discussion is available here:
https://lore.kernel.org/lkml/20250422113156.575971-1-eugen.hristev@linaro.org/

Kmemdump has been presented and discussed at Linaro Connect 2025,
including motivation, scope, usability and feasability.
Video of the recording is available here for anyone interested:
https://www.youtube.com/watch?v=r4gII7MX9zQ&list=PLKZSArYQptsODycGiE0XZdVovzAwYNwtK&index=14

Linaro blog on kmemdump can be found here:
https://www.linaro.org/blog/introduction-to-kmemdump/

Linaro blog on kmemdump step by stem using minidump backend is available here:
https://www.linaro.org/blog/kmemdump-step-by-step-on-qualcomm-automotive-platform/

The implementation is based on the initial Pstore/directly mapped zones
published as an RFC here:
https://lore.kernel.org/all/20250217101706.2104498-1-eugen.hristev@linaro.org/

The back-end implementation for qcom_minidump is based on the minidump
patch series and driver written by Mukesh Ojha, thanks:
https://lore.kernel.org/lkml/20240131110837.14218-1-quic_mojha@quicinc.com/

The RFC v2 version with .section creation and macro annotation kmemdump
is available here:
https://lore.kernel.org/all/20250724135512.518487-1-eugen.hristev@linaro.org/

The RFC v3 version with making everything static, which was pretty much rejected due to
all reasons discussed on the public ML:
https://lore.kernel.org/all/20250912150855.2901211-1-eugen.hristev@linaro.org/

*** How to use meminspect with minidump backend on Qualcomm platform guide ***

Prerequisites:
Crash tool compiled with target=ARM64 and minor changes required for
usual crash mode (minimal mode works without the patch) **A patch can be
applied from here https://p.calebs.dev/1687bc ** This patch will be
eventually sent in a reworked way to crash tool.

Target kernel must be built with : CONFIG_DEBUG_INFO_REDUCED=n ; this
will have vmlinux include all the debugging information needed for crash
tool.

Also, the kernel requires these as well: CONFIG_MEMINSPECT,
CONFIG_CRASH_DUMP and the driver CONFIG_QCOM_MINIDUMP

Kernel arguments: Kernel firmware must be set to mode 'mini' by kernel
module parameter like this : qcom_scm.download_mode=mini

After the kernel boots, and minidump module is loaded, everything is
ready for a possible crash.

Once the crash happens, the firmware will kick in and you will see on
the console the message saying Sahara init, etc, that the firmware is
waiting in download mode. (this is subject to firmware supporting this
mode, I am using sa8775p-ride board)

Example of log on the console:
"
[...]
B -   1096414 - usb: init start
B -   1100287 - usb: qusb_dci_platform , 0x19
B -   1105686 - usb: usb3phy: PRIM success: lane_A , 0x60
B -   1107455 - usb: usb2phy: PRIM success , 0x4
B -   1112670 - usb: dci, chgr_type_det_err
B -   1117154 - usb: ID:0x260, value: 0x4
B -   1121942 - usb: ID:0x108, value: 0x1d90
B -   1124992 - usb: timer_start , 0x4c4b40
B -   1129140 - usb: vbus_det_pm_unavail
B -   1133136 - usb: ID:0x252, value: 0x4
B -   1148874 - usb: SUPER , 0x900e
B -   1275510 - usb: SUPER , 0x900e
B -   1388970 - usb: ID:0x20d, value: 0x0
B -   1411113 - usb: ENUM success
B -   1411113 - Sahara Init
B -   1414285 - Sahara Open
"

Once the board is in download mode, you can use the qdl tool (I
personally use edl , have not tried qdl yet), to get all the regions as
separate files.  The tool from the host computer will list the regions
in the order they were downloaded.

Once you have all the files simply use `cat` to put them all together,
in the order of the indexes.  For my kernel config and setup, here is my
cat command : (you can use a script or something, I haven't done that so
far):

`cat md_KELF1.BIN md_Kvmcorein2.BIN md_Kconfig3.BIN \
md_Ktotalram4.BIN md_Kcpu_poss5.BIN md_Kcpu_pres6.BIN \
md_Kcpu_onli7.BIN md_Kcpu_acti8.BIN md_Kmem_sect9.BIN \
md_Kjiffies10.BIN md_Klinux_ba11.BIN md_Knr_threa12.BIN \
md_Knr_irqs13.BIN md_Ktainted_14.BIN md_Ktaint_fl15.BIN \
md_Knode_sta16.BIN md_K__per_cp17.BIN md_Knr_swapf18.BIN \
md_Kinit_uts19.BIN md_Kprintk_r20.BIN md_Kprintk_r21.BIN \
md_Kprb22.BIN md_Kprb_desc23.BIN md_Kprb_info24.BIN \
md_Kprb_data25.BIN  md_Khigh_mem26.BIN md_Kinit_mm27.BIN \
md_Kunknown29.BIN md_Kunknown30.BIN md_Kunknown31.BIN \
md_Kunknown32.BIN md_Kunknown33.BIN md_Kunknown34.BIN \
md_Kunknown35.BIN md_Kunknown37.BIN \
md_Kunknown38.BIN md_Kunknown39.BIN md_Kunknown40.BIN \
md_Kunknown41.BIN md_Kunknown42.BIN md_Kunknown43.BIN \
md_Kunknown44.BIN md_Kunknown45.BIN  md_Kunknown46.BIN \
md_Kunknown47.BIN md_Kunknown48.BIN md_Kunknown49.BIN \
md_Kunknown50.BIN md_Kunknown51.BIN md_Kunknown52.BIN \
md_Kunknown53.BIN md_Kunknown54.BIN   > ./minidump_image`

Once you have the resulted file, use `crash` tool to load it, like this:
`./crash --no_modules --no_panic --no_kmem_cache --zero_excluded vmlinux minidump_image`

There is also a --minimal mode for ./crash that would work without any patch applied
to crash tool, but you can't inspect symbols, etc.

Once you load crash you will see something like this :
      KERNEL: minidump/20260310-235110/vmlinux  [TAINTED]
    DUMPFILE: ./minidump/20260310-235110/minidump_image
        CPUS: 8 [OFFLINE: 7]
        DATE: Thu Jan  1 05:30:00 +0530 1970
      UPTIME: 00:00:27
       TASKS: 0
    NODENAME: qemuarm64
     RELEASE: 7.0.0-rc3-next-20260309-00028-g528b3c656121
     VERSION: #5 SMP PREEMPT Tue Mar 10 18:18:41 UTC 2026
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 0
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"

crash> log
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x514f0014]
[    0.000000] Linux version 7.0.0-rc3-next-20260309-00028-g528b3c656121 (@21e3bca4168f) (aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #5 SMP PREEMPT Tue Mar 10 18:18:41 UTC 2026

*** Debug Kinfo backend driver ***
I need help with the testing of this driver, Anyone who actually wants
to test this, feel free to reply to the patch. we have also written a
simple DT binding for the driver.

Thanks in advance for the review, and apologies if I missed addressing any comment.

-Mukesh 

Changes in v2: https://lore.kernel.org/lkml/20251119154427.1033475-1-eugen.hristev@linaro.org/
 - Fixed doc warnings
 - Fixed kernel-test robot warnings.
 - Took Mike suggestion to remove mark inspect flag for dynamic memory.
 - Added R-b for printk patch.
 - Modified some commit messages for clarity.
 - corrected binding change for debug-kinfo as per Rob suggestion.

Changelog for meminspect v1:
- rename to meminspect
- start on top of v2 actually, with the section and all.
- remove the backend thing, change the API to access the table
- move everything to kernel/
- add dependency to CRASH_DUMP instead of a separate knob
- move the minidump driver to soc/qcom
- integrate the meminspect better into memblock by using a new memblock flag
- minor fixes : use dev_err_probe everywhere, rearrange variable declarations,
remove some useless code, etc.

Changelog for RFC v3:
- V2 available here : https://lore.kernel.org/all/20250724135512.518487-1-eugen.hristev@linaro.org/
- Removed the .section as requested by David Hildenbrand.
- Moved all kmemdump registration(when possible) to vmcoreinfo.
- Because of this, some of the variables that I was registering had to be non-static
so I had to modify this as per David Hildenbrand suggestion.
- Fixed minor things in the Kinfo driver: one field was broken, fixed some
compiler warnings, fixed the copyright and remove some useless includes.
- Moved the whole kmemdump from drivers/debug into mm/ and Kconfigs into mm/Kconfig.debug
and it's now available in kernel hacking, as per Randy Dunlap review
- Reworked some of the Documentation as per review from Jon Corbet

Changelog for RFC v2:
- V1 available here: https://lore.kernel.org/lkml/20250422113156.575971-1-eugen.hristev@linaro.org/
- Reworked the whole minidump implementation based on suggestions from Thomas Gleixner.
This means new API, macros, new way to store the regions inside kmemdump
(ditched the IDR, moved to static allocation, have a static default backend, etc)
- Reworked qcom_minidump driver based on review from Bjorn Andersson
- Reworked printk log buffer registration based on review from Petr Mladek

I appologize if I missed any review comments.
Patches are sent on top on next-20260309 tag

---
Eugen Hristev (21):
      kernel: Introduce meminspect
      init/version: Annotate static information into meminspect
      mm/percpu: Annotate static information into meminspect
      cpu: Annotate static information into meminspect
      genirq/irqdesc: Annotate static information into meminspect
      timers: Annotate static information into meminspect
      kernel/fork: Annotate static information into meminspect
      mm/page_alloc: Annotate static information into meminspect
      mm/show_mem: Annotate static information into meminspect
      mm/swapfile: Annotate static information into meminspect
      kernel/vmcore_info: Register dynamic information into meminspect
      kernel/configs: Register dynamic information into meminspect
      mm/init-mm: Annotate static information into meminspect
      panic: Annotate static information into meminspect
      kallsyms: Annotate static information into meminspect
      mm/mm_init: Annotate static information into meminspect
      sched/core: Annotate runqueues into meminspect
      remoteproc: qcom: Move minidump data structures into its own header
      soc: qcom: Add minidump backend driver
      soc: qcom: smem: Add minidump platform device
      meminspect: Add debug kinfo compatible driver

Mukesh Ojha (4):
      mm/numa: Register node data information into meminspect
      mm/sparse: Register information into meminspect
      printk: Register information into meminspect
      dt-bindings: reserved-memory: Add Google Kinfo Pixel reserved memory

 Documentation/dev-tools/index.rst                  |   1 +
 Documentation/dev-tools/meminspect.rst             | 144 +++++++
 .../bindings/reserved-memory/google,kinfo.yaml     |  46 ++
 MAINTAINERS                                        |  14 +
 drivers/of/platform.c                              |   1 +
 drivers/remoteproc/qcom_common.c                   |  56 +--
 drivers/soc/qcom/Kconfig                           |  13 +
 drivers/soc/qcom/Makefile                          |   1 +
 drivers/soc/qcom/minidump.c                        | 272 ++++++++++++
 drivers/soc/qcom/smem.c                            |  10 +
 include/asm-generic/vmlinux.lds.h                  |  13 +
 include/linux/meminspect.h                         | 263 ++++++++++++
 include/linux/soc/qcom/minidump.h                  |  72 ++++
 init/Kconfig                                       |   1 +
 init/version-timestamp.c                           |   3 +
 init/version.c                                     |   3 +
 kernel/Makefile                                    |   1 +
 kernel/configs.c                                   |   6 +
 kernel/cpu.c                                       |   5 +
 kernel/fork.c                                      |   3 +
 kernel/irq/irqdesc.c                               |   2 +
 kernel/kallsyms.c                                  |   9 +
 kernel/meminspect/Kconfig                          |  30 ++
 kernel/meminspect/Makefile                         |   4 +
 kernel/meminspect/kinfo.c                          | 284 +++++++++++++
 kernel/meminspect/meminspect.c                     | 471 +++++++++++++++++++++
 kernel/panic.c                                     |   4 +
 kernel/printk/printk.c                             |  11 +
 kernel/sched/core.c                                |   2 +
 kernel/time/timer.c                                |   2 +
 kernel/vmcore_info.c                               |   4 +
 mm/init-mm.c                                       |  11 +
 mm/mm_init.c                                       |   2 +
 mm/numa.c                                          |   2 +
 mm/page_alloc.c                                    |   2 +
 mm/percpu.c                                        |   2 +
 mm/show_mem.c                                      |   2 +
 mm/sparse.c                                        |   6 +
 mm/swapfile.c                                      |   2 +
 39 files changed, 1725 insertions(+), 55 deletions(-)
---
base-commit: 343f51842f4ed7143872f3aa116a214a5619a4b9
change-id: 20260311-minidump-v2-eed8da647ce5

Best regards,
-- 
-Mukesh Ojha



^ permalink raw reply	[flat|nested] 42+ messages in thread

end of thread, other threads:[~2026-03-19  7:34 UTC | newest]

Thread overview: 42+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-10 20:15 [PATCH v2 00/25] Introduce meminspect Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 01/25] kernel: " Mukesh Ojha
2026-03-12  4:33   ` Randy Dunlap
2026-03-12  4:46   ` Randy Dunlap
2026-03-16  8:31     ` Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 02/25] init/version: Annotate static information into meminspect Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 03/25] mm/percpu: " Mukesh Ojha
2026-03-12 20:18   ` kernel test robot
2026-03-13  8:06   ` kernel test robot
2026-03-10 20:15 ` [PATCH v2 04/25] cpu: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 05/25] genirq/irqdesc: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 06/25] timers: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 07/25] kernel/fork: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 08/25] mm/page_alloc: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 09/25] mm/show_mem: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 10/25] mm/swapfile: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 11/25] kernel/vmcore_info: Register dynamic " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 12/25] kernel/configs: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 13/25] mm/init-mm: Annotate static " Mukesh Ojha
2026-03-12 14:28   ` kernel test robot
2026-03-10 20:15 ` [PATCH v2 14/25] panic: " Mukesh Ojha
2026-03-10 20:15 ` [PATCH v2 15/25] kallsyms: " Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 16/25] mm/mm_init: " Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 17/25] sched/core: Annotate runqueues " Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 18/25] mm/numa: Register node data information " Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 19/25] mm/sparse: Register " Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 20/25] printk: " Mukesh Ojha
2026-03-16  9:39   ` John Ogness
2026-03-16 10:24     ` Eugen Hristev
2026-03-10 20:16 ` [PATCH v2 21/25] remoteproc: qcom: Move minidump data structures into its own header Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 22/25] soc: qcom: Add minidump backend driver Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 23/25] soc: qcom: smem: Add minidump platform device Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 24/25] dt-bindings: reserved-memory: Add Google Kinfo Pixel reserved memory Mukesh Ojha
2026-03-11  9:05   ` Krzysztof Kozlowski
2026-03-16 11:12     ` Mukesh Ojha
2026-03-10 20:16 ` [PATCH v2 25/25] meminspect: Add debug kinfo compatible driver Mukesh Ojha
2026-03-11  9:09   ` Krzysztof Kozlowski
2026-03-11 22:07   ` Randy Dunlap
2026-03-16  2:24 ` [PATCH v2 00/25] Introduce meminspect Bjorn Andersson
2026-03-16 18:16   ` Mukesh Ojha
2026-03-19  2:55     ` Bjorn Andersson
2026-03-19  7:33       ` Eugen Hristev

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox