Re: [mainline] KDUMP/FADUMP is broken on the mainline kernel on P11 System

[Sourabh Jain <sourabhjain@linux.ibm.com>]

Hello Venkat,

Thanks for sharing the bug report.

Can you please provide additional data about the environment.

1. What was the top commit in your git branch?
2. Kernel command-line of first/production kernel.
3. Is this issue specific to P11?
4. Since you used kdump service to load the capture kernel may I know 
what distro was used?


On 12/01/26 08:51, Venkat Rao Bagalkote wrote:
> Greetings!!
>
>
> KDUMP and FADUMP is broken on P11 system. After configuring 
> KDUMP/FADUMP on the system, upon reboot, system dosent reboot to 
> collect the vmcore. Its just stuck. Need to trigger reboot from HMC.

What do you mean upon reboot? Did you on system crash?

>
>
>
> # makedumpfile -v
> makedumpfile: version 1.7.7 (released on 21 Apr 2025)
> lzo    enabled
> snappy    enabled
> zstd    disabled
>
> cd /root/
> # makedumpfile -v
> makedumpfile: version 1.7.7 (released on 21 Apr 2025)
> lzo    enabled
> snappy    enabled
> zstd    disabled
>
> # systemctl stop kdump.service
> # systemctl start kdump.service
>  systemctl status kdump.service --no-pager
> ● kdump.service - Crash recovery kernel arming
>      Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; 
> preset: enabled)
>      Active: active (exited) since Fri 2026-01-09 09:12:01 CST; 7s ago
>     Process: 2071 ExecStart=/usr/bin/kdumpctl start (code=exited, 
> status=0/SUCCESS)
>    Main PID: 2071 (code=exited, status=0/SUCCESS)
>         CPU: 41.857s
>
> Jan 09 09:11:55 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** Install 
> squash…**
> Jan 09 09:11:56 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Stripping file…**
> Jan 09 09:11:56 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Stripping file…**
> Jan 09 09:11:56 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Squashing the …**
> Jan 09 09:11:58 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Squashing the …**
> Jan 09 09:11:58 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Creating image…**
> Jan 09 09:11:58 ltcden8-lp5.ltc.tadn.ibm.com dracut[2839]: *** 
> Creating initr…**
> Jan 09 09:12:01 ltcden8-lp5.ltc.tadn.ibm.com kdumpctl[2074]: kdump: 
> kexec: lo…el
> Jan 09 09:12:01 ltcden8-lp5.ltc.tadn.ibm.com kdumpctl[2074]: kdump: 
> Starting …K]
> Jan 09 09:12:01 ltcden8-lp5.ltc.tadn.ibm.com systemd[1]: Finished 
> Crash recov…g.
> Hint: Some lines were ellipsized, use -l to show in full.
> # echo 10 > /proc/sys/kernel/panic
> # echo 1 > /proc/sys/kernel/sysrq
> # echo c > /proc/sysrq-trigger
> [  595.605706] sysrq: Trigger a crash
> [  595.605721] Kernel panic - not syncing: sysrq triggered crash
> [  595.605729] CPU: 0 UID: 0 PID: 1840 Comm: bash Kdump: loaded Not 
> tainted 6.17.0 #6 VOLUNTARY
> [  595.605739] Hardware name: IBM,9080-HEX Power11 (architected) 
> 0x820200 0xf000007 of:IBM,FW1110.01 (NH1110_069) hv:phyp pSeries
> [  595.605747] Call Trace:
> [  595.605752] [c0000000a0e17920] [c00000000184c0e8] 
> dump_stack_lvl+0xd0/0xe8 (unreliable)
> [  595.605769] [c0000000a0e17950] [c0000000001b2434] vpanic+0x498/0x518
> [  595.605781] [c0000000a0e17a00] [c0000000001b2504] nmi_panic+0x0/0x13c
> [  595.605791] [c0000000a0e17a20] [c000000000e95d8c] 
> sysrq_handle_crash+0x30/0x34
> [  595.605804] [c0000000a0e17a80] [c000000000e96c98] 
> __handle_sysrq+0x124/0x384
> [  595.605814] [c0000000a0e17b40] [c000000000e97a9c] 
> write_sysrq_trigger+0x184/0x228
> [  595.605826] [c0000000a0e17bc0] [c0000000009f1c54] 
> proc_reg_write+0x18c/0x240
> [  595.605838] [c0000000a0e17c10] [c000000000896588] 
> vfs_write+0x148/0x7d8
> [  595.605848] [c0000000a0e17cf0] [c000000000896eb4] 
> ksys_write+0xa0/0x184
> [  595.605858] [c0000000a0e17d50] [c0000000000393d0] 
> system_call_exception+0x1e0/0x460
> [  595.605869] [c0000000a0e17e50] [c00000000000d05c] 
> system_call_vectored_common+0x15c/0x2ec
> [  595.605882] ---- interrupt: 3000 at 0x7fff9ab33e74
> [  595.605890] NIP:  00007fff9ab33e74 LR: 00007fff9ab33e74 CTR: 
> 0000000000000000
> [  595.605897] REGS: c0000000a0e17e80 TRAP: 3000   Not tainted (6.17.0)
> [  595.605904] MSR:  800000000280f033 
> <SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44422408  XER: 00000000
> [  595.605936] IRQMASK: 0
> [  595.605936] GPR00: 0000000000000004 00007fffc3c7fab0 
> 0000000135727e00 0000000000000001
> [  595.605936] GPR04: 000000014b93c040 0000000000000002 
> 0000000000000010 0000000000000001
> [  595.605936] GPR08: 0000000000000001 0000000000000000 
> 0000000000000000 0000000000000000
> [  595.605936] GPR12: 0000000000000000 00007fff9adfab60 
> 000000014b9fc1d0 00000001357287b8
> [  595.605936] GPR16: 00000001357294d8 0000000020000000 
> 0000000000000000 0000000135639070
> [  595.605936] GPR20: 00000001356cbeb8 00007fffc3c7fc54 
> 00000001356cf8a0 00000001357289bc
> [  595.605936] GPR24: 0000000135728a50 0000000000000000 
> 000000014b93c040 0000000000000002
> [  595.605936] GPR28: 0000000000000002 00007fff9ac318e0 
> 000000014b93c040 0000000000000002
> [  595.606034] NIP [00007fff9ab33e74] 0x7fff9ab33e74
> [  595.606040] LR [00007fff9ab33e74] 0x7fff9ab33e74
> [  595.606046] ---- interrupt: 3000
> 8:41
> # cat /etc/kdump.conf
> # This file contains a series of commands to perform (in order) in the 
> kdump
> # kernel after a kernel crash in the crash kernel(1st kernel) has 
> happened.
> #
> # Directives in this file are only applicable to the kdump initramfs, 
> and have
> # no effect once the root filesystem is mounted and the normal init 
> scripts are
> # processed.
> #
> # Currently, only one dump target and path can be specified.  If the 
> dumping to
> # the configured target fails, the failure action which can be 
> configured via
> # the "failure_action" directive will be performed.
> #
> # Supported options:
> #
> # auto_reset_crashkernel <yes|no>
> #           - whether to reset kernel crashkernel to new default value
> #             or not when kexec-tools updates the default crashkernel 
> value and
> #             existing kernels using the old default kernel 
> crashkernel value.
> #             The default value is yes.
> #
> # raw <partition>
> #           - Will dd /proc/vmcore into <partition>.
> #             Use persistent device names for partition devices,
> #             such as /dev/vg/<devname>.
> #
> # nfs <nfs mount>
> #           - Will mount nfs to <mnt>, and copy /proc/vmcore to
> #             <mnt>/<path>/%HOST-%DATE/, supports DNS.
> #
> # ssh <user@server>
> #           - Will save /proc/vmcore to 
> <user@server>:<path>/%HOST-%DATE/,
> #             supports DNS.
> #             NOTE: make sure the user has write permissions on the 
> server.
> #
> # sshkey <path>
> #           - Will use the sshkey to do ssh dump.
> #             Specify the path of the ssh key to use when dumping
> #             via ssh. The default value is /root/.ssh/kdump_id_rsa.
> #
> # <fs type> <partition>
> #           - Will mount -t <fs type> <partition> <mnt>, and copy
> #             /proc/vmcore to <mnt>/<path>/%HOST_IP-%DATE/.
> #             NOTE: <partition> can be a device node, label or uuid.
> #             It's recommended to use persistent device names
> #             such as /dev/vg/<devname>.
> #             Otherwise it's suggested to use label or uuid.
> #             Supported fs types: ext[234], xfs, btrfs, minix, virtiofs
> #
> # path <path>
> #           - "path" represents the file system path in which vmcore
> #             will be saved.  If a dump target is specified in
> #             kdump.conf, then "path" is relative to the specified
> #             dump target.
> #
> #             Interpretation of "path" changes a bit if the user didn't
> #             specify any dump target explicitly in kdump.conf. In this
> #             case, "path" represents the absolute path from root. The
> #             dump target and adjusted path are arrived at automatically
> #             depending on what's mounted in the current system.
> #
> #             Ignored for raw device dumps.  If unset, will use the 
> default
> #             "/var/crash".
> #
> # core_collector <command> <options>
> #           - This allows you to specify the command to copy
> #             the vmcore.  The default is makedumpfile, which on
> #             some architectures can drastically reduce vmcore size.
> #             See /sbin/makedumpfile --help for a list of options.
> #             Note that the -i and -g options are not needed here,
> #             as the initrd will automatically be populated with a
> #             config file appropriate for the running kernel.
> #             The default core_collector for raw/ssh dump is:
> #             "makedumpfile -F -l --message-level 7 -d 31".
> #             The default core_collector for other targets is:
> #             "makedumpfile -l --message-level 7 -d 31".
> #
> #             "makedumpfile -F" will create a flattened vmcore.
> #             You need to use "makedumpfile -R" to rearrange the dump 
> data to
> #             a normal dumpfile readable with analysis tools.  For 
> example:
> #             "makedumpfile -R vmcore < vmcore.flat".
> #
> #             For core_collector format details, you can refer to
> #             kexec-kdump-howto.txt or kdump.conf manpage.
> #
> # kdump_post <binary | script>
> #           - This directive allows you to run a executable binary
> #             or script after the vmcore dump process terminates.
> #             The exit status of the current dump process is fed to
> #             the executable binary or script as its first argument.
> #             All files under /etc/kdump/post.d are collectively sorted
> #             and executed in lexical order, before binary or script
> #             specified kdump_post parameter is executed.
> #
> # kdump_pre <binary | script>
> #           - Works like the "kdump_post" directive, but instead of 
> running
> #             after the dump process, runs immediately before it.
> #             Exit status of this binary is interpreted as follows:
> #               0 - continue with dump process as usual
> #               non 0 - run the final action (reboot/poweroff/halt)
> #             All files under /etc/kdump/pre.d are collectively sorted 
> and
> #             executed in lexical order, after binary or script specified
> #             kdump_pre parameter is executed.
> #             Even if the binary or script in /etc/kdump/pre.d directory
> #             returns non 0 exit status, the processing is continued.
> #
> # extra_bins <binaries | shell scripts>
> #           - This directive allows you to specify additional binaries or
> #             shell scripts to be included in the kdump initrd.
> #             Generally they are useful in conjunction with a kdump_post
> #             or kdump_pre binary or script which depends on these 
> extra_bins.
> #
> # extra_modules <module(s)>
> #           - This directive allows you to specify extra kernel modules
> #             that you want to be loaded in the kdump initrd.
> #             Multiple modules can be listed, separated by spaces, and 
> any
> #             dependent modules will automatically be included.
> #
> # failure_action <reboot | halt | poweroff | shell | dump_to_rootfs>
> #           - Action to perform in case dumping fails.
> #             reboot:   Reboot the system.
> #             halt:     Halt the system.
> #             poweroff: Power down the system.
> #             shell:    Drop to a bash shell.
> #                       Exiting the shell reboots the system by default,
> #                       or perform "final_action".
> #             dump_to_rootfs:  Dump vmcore to rootfs from initramfs 
> context and
> #                       reboot by default or perform "final_action".
> #                       Useful when non-root dump target is specified.
> #             The default option is "reboot".
> #
> # default <reboot | halt | poweroff | shell | dump_to_rootfs>
> #           - Same as the "failure_action" directive above, but this 
> directive
> #             is obsolete and will be removed in the future.
> #
> # final_action <reboot | halt | poweroff>
> #           - Action to perform in case dumping succeeds. Also performed
> #             when "shell" or "dump_to_rootfs" failure action finishes.
> #             Each action is same as the "failure_action" directive 
> above.
> #             The default is "reboot".
> #
> # force_rebuild <0 | 1>
> #           - By default, kdump initrd will only be rebuilt when 
> necessary.
> #             Specify 1 to force rebuilding kdump initrd every time 
> when kdump
> #             service starts.
> #
> # force_no_rebuild <0 | 1>
> #           - By default, kdump initrd will be rebuilt when necessary.
> #             Specify 1 to bypass rebuilding of kdump initrd.
> #
> #             force_no_rebuild and force_rebuild options are mutually
> #             exclusive and they should not be set to 1 simultaneously.
> #
> # override_resettable <0 | 1>
> #           - Usually an unresettable block device can't be a dump 
> target.
> #             Specifying 1 when you want to dump even though the block
> #             target is unresettable
> #             By default, it is 0, which will not try dumping destined 
> to fail.
> #
> # dracut_args <arg(s)>
> #           - Pass extra dracut options when rebuilding kdump initrd.
> #
> # fence_kdump_args <arg(s)>
> #           - Command line arguments for fence_kdump_send (it can contain
> #             all valid arguments except hosts to send notification to).
> #
> # fence_kdump_nodes <node(s)>
> #           - List of cluster node(s) except localhost, separated by 
> spaces,
> #             to send fence_kdump notifications to.
> #             (this option is mandatory to enable fence_kdump).
> #
>
> #raw /dev/vg/lv_kdump
> #ext4 /dev/vg/lv_kdump
> #ext4 LABEL=/boot
> #ext4 UUID=03138356-5e61-4ab3-b58e-27507ac41937
> #virtiofs myfs
> #nfs my.server.com:/export/tmp
> #nfs [2001:db8::1:2:3:4]:/export/tmp
> #ssh user@my.server.com
> #ssh user@2001:db8::1:2:3:4
> #sshkey /root/.ssh/kdump_id_rsa
> auto_reset_crashkernel yes
> path /var/crash
> core_collector makedumpfile -l --message-level 7 -d 31
> #core_collector scp
> #kdump_post /var/crash/scripts/kdump-post.sh
> #kdump_pre /var/crash/scripts/kdump-pre.sh
> #extra_bins /usr/bin/lftp
> #extra_modules gfs2
> #failure_action shell
> #force_rebuild 1
> #force_no_rebuild 1
> #dracut_args --omit-drivers "cfg80211 snd" --add-drivers "ext2 ext3"
> #fence_kdump_args -p 7410 -f auto -c 0 -i 10
> #fence_kdump_nodes node1 node2
>
>
>
> Regards,
>
> Venkat.
>