Re: Help needed in getting kernel dump in QEMU VM

Re: Help needed in getting kernel dump in QEMU VM

[Baoquan He]

Hi,

On 11/13/21 at 10:40am, Dongliang Mu wrote:
> Hi all KDUMP maintainers,
> 
> I would like to generate a kernel dump within QEMU VM.
> 
> 1. I reproduced the kernel crash [1] in QEMU VM. The QEMU startup
> script is as follows:
> 
> qemu-system-x86_64 \
>   -kernel $KERNEL/arch/x86/boot/bzImage \
>   -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
>   -hda $IMAGE/stretch.img \
>   -net user,hostfwd=tcp::10021-:22 -net nic \
>   -enable-kvm \
>   -nographic \
>   -m 2G \
>   -smp 2 \
>   -pidfile vm.pid \
>   2>&1 | tee vm.log
> 
> The stretch.img is generated by Syzkaller script [1]. -kernel option
> is convenient for loading any other kernels.
> 
> 2. As the .config already has the essential
> configuration(CONFIG_KEXEC, CONFIG_CRASH_DUMP, CONFIG_DEBUG_INFO), I
> did not change this configuration file.
> 
> 3. I installed kdump-tools crash kexec-tools makedumpfile
> linux-image-4.9.0-13-amd64 in the stretch.img. Here I installed
> linux-image-4.9.0-13-amd64 because there is no default kernel in /boot
> directory. And to make kdump-tools working, I modify
> /etc/default/kdump-tools in the following:
> 
> KDUMP_INITRD=/boot/initrd.img-4.9.0-13-amd64
> KDUMP_KERNEL=/boot/vmlinuz-4.9.0-13-amd64

What distros are you using? Asking this because I am sure you are not
using Fedora/RHEL OS. The implementation of kdump tools is different in
each distros, even though the mechanims in kdump code is the same.

When we try to get help from upstream, considering and asking good question
is very important for getting quick response and effective help.

Thanks
Baoquan

> 
> 4. I append "crashkernel=384M-:128M" to the command line in the
> startup script of QEMU.
> 
> 5. After rebooting, kdump service can start successfully, and the
> kdump-config shows:
> 
> root@syzkaller:~# kdump-config show
> DUMP_MODE:        kdump
> USE_KDUMP:        1
> KDUMP_SYSCTL:     kernel.panic_on_oops=1
> KDUMP_COREDIR:    /var/crash
> crashkernel addr: 0x77000000
>    /boot/vmlinuz-4.9.0-13-amd64
> kdump initrd:
>    /boot/initrd.img-4.9.0-13-amd64
> current state:    ready to kdump
> 
> kexec command:
>   /sbin/kexec -p --command-line="earlyprintk=serial oops=panic
> panic_on_warn=1 nmi_watchdog=panic panic=86400 net.ifnames=0
> sysctl.kernel.hung_task_all_cpu_backtrace=1 ima_policy=tcb
> kvm-intel.nested=1 nf-conntrack-ftp.ports=20000
> nf-conntrack-tftp.ports=20000 nf-conntrack-sip.ports=20000
> nf-conntrack-irc.ports=20000 nf-conntrack-sane.ports=20000
> vivid.n_devs=16 vivid.multiplanar=1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2
> netrom.nr_ndevs=16 rose.rose_ndevs=16 spec_store_bypass_disable=prctl
> numa=fake=2 nopcid dummy_hcd.num=8 binder.debug_mask=0
> rcupdate.rcu_expedited=1 root=/dev/sda console=ttyS0 vsyscall=native
> watchdog_thresh=55 workqueue.watchdog_thresh=140 console=ttyS0
> root=/dev/sda debug earlyprintk=serial slub_debug=QUZ irqpoll
> nr_cpus=1 nousb systemd.unit=kdump-tools.service
> ata_piix.prefer_ms_hyperv=0" --initrd=/boot/initrd.img-4.9.0-13-amd64
> /boot/vmlinuz-4.9.0-13-amd64
> 
> 6. When I execute the PoC, the current kernel crashes and then reboots
> into the dump-capture kernel. However, the kernel log shows, it is in
> emergency mode,
> 
> You are in emergency mode. After logging in, type "journalctl -xb" to view
> system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
> try again to boot into default mode.
> 
> Finally, I would like to ask several questions:
> 1) is the emergency mode due to the incorrect command line?
> 2) is this the right way to generate kernel dump from QEMU VM?
> 3) Any comments on the above procedures?
> 
> Thanks very much in advance.
> 
> [1] general protection fault in reiserfs_security_init
> (https://syzkaller.appspot.com/bug?id=8abaedbdeb32c861dc5340544284167dd0e46cde)
> 
> [2] https://github.com/google/syzkaller/blob/master/tools/create-image.sh
> 
> --
> My best regards to you.
> 
>      No System Is Safe!
>      Dongliang Mu



_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Help needed in getting kernel dump in QEMU VM

[Dongliang Mu]

On Mon, Nov 15, 2021 at 3:27 PM Baoquan He <bhe@redhat.com> wrote:
>
> Hi,
>
> On 11/13/21 at 10:40am, Dongliang Mu wrote:
> > Hi all KDUMP maintainers,
> >
> > I would like to generate a kernel dump within QEMU VM.
> >
> > 1. I reproduced the kernel crash [1] in QEMU VM. The QEMU startup
> > script is as follows:
> >
> > qemu-system-x86_64 \
> >   -kernel $KERNEL/arch/x86/boot/bzImage \
> >   -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
> >   -hda $IMAGE/stretch.img \
> >   -net user,hostfwd=tcp::10021-:22 -net nic \
> >   -enable-kvm \
> >   -nographic \
> >   -m 2G \
> >   -smp 2 \
> >   -pidfile vm.pid \
> >   2>&1 | tee vm.log
> >
> > The stretch.img is generated by Syzkaller script [1]. -kernel option
> > is convenient for loading any other kernels.
> >
> > 2. As the .config already has the essential
> > configuration(CONFIG_KEXEC, CONFIG_CRASH_DUMP, CONFIG_DEBUG_INFO), I
> > did not change this configuration file.
> >
> > 3. I installed kdump-tools crash kexec-tools makedumpfile
> > linux-image-4.9.0-13-amd64 in the stretch.img. Here I installed
> > linux-image-4.9.0-13-amd64 because there is no default kernel in /boot
> > directory. And to make kdump-tools working, I modify
> > /etc/default/kdump-tools in the following:
> >
> > KDUMP_INITRD=/boot/initrd.img-4.9.0-13-amd64
> > KDUMP_KERNEL=/boot/vmlinuz-4.9.0-13-amd64
>
> What distros are you using? Asking this because I am sure you are not
> using Fedora/RHEL OS. The implementation of kdump tools is different in
> each distros, even though the mechanims in kdump code is the same.
>

I am using Debian stretch as the guest OS. So kdump-tools kexec
makedumpfile is all from Debian.

> When we try to get help from upstream, considering and asking good question
> is very important for getting quick response and effective help.
>
> Thanks
> Baoquan
>
> >
> > 4. I append "crashkernel=384M-:128M" to the command line in the
> > startup script of QEMU.
> >
> > 5. After rebooting, kdump service can start successfully, and the
> > kdump-config shows:
> >
> > root@syzkaller:~# kdump-config show
> > DUMP_MODE:        kdump
> > USE_KDUMP:        1
> > KDUMP_SYSCTL:     kernel.panic_on_oops=1
> > KDUMP_COREDIR:    /var/crash
> > crashkernel addr: 0x77000000
> >    /boot/vmlinuz-4.9.0-13-amd64
> > kdump initrd:
> >    /boot/initrd.img-4.9.0-13-amd64
> > current state:    ready to kdump
> >
> > kexec command:
> >   /sbin/kexec -p --command-line="earlyprintk=serial oops=panic
> > panic_on_warn=1 nmi_watchdog=panic panic=86400 net.ifnames=0
> > sysctl.kernel.hung_task_all_cpu_backtrace=1 ima_policy=tcb
> > kvm-intel.nested=1 nf-conntrack-ftp.ports=20000
> > nf-conntrack-tftp.ports=20000 nf-conntrack-sip.ports=20000
> > nf-conntrack-irc.ports=20000 nf-conntrack-sane.ports=20000
> > vivid.n_devs=16 vivid.multiplanar=1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2
> > netrom.nr_ndevs=16 rose.rose_ndevs=16 spec_store_bypass_disable=prctl
> > numa=fake=2 nopcid dummy_hcd.num=8 binder.debug_mask=0
> > rcupdate.rcu_expedited=1 root=/dev/sda console=ttyS0 vsyscall=native
> > watchdog_thresh=55 workqueue.watchdog_thresh=140 console=ttyS0
> > root=/dev/sda debug earlyprintk=serial slub_debug=QUZ irqpoll
> > nr_cpus=1 nousb systemd.unit=kdump-tools.service
> > ata_piix.prefer_ms_hyperv=0" --initrd=/boot/initrd.img-4.9.0-13-amd64
> > /boot/vmlinuz-4.9.0-13-amd64
> >
> > 6. When I execute the PoC, the current kernel crashes and then reboots
> > into the dump-capture kernel. However, the kernel log shows, it is in
> > emergency mode,
> >
> > You are in emergency mode. After logging in, type "journalctl -xb" to view
> > system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
> > try again to boot into default mode.
> >
> > Finally, I would like to ask several questions:
> > 1) is the emergency mode due to the incorrect command line?
> > 2) is this the right way to generate kernel dump from QEMU VM?
> > 3) Any comments on the above procedures?
> >
> > Thanks very much in advance.
> >
> > [1] general protection fault in reiserfs_security_init
> > (https://syzkaller.appspot.com/bug?id=8abaedbdeb32c861dc5340544284167dd0e46cde)
> >
> > [2] https://github.com/google/syzkaller/blob/master/tools/create-image.sh
> >
> > --
> > My best regards to you.
> >
> >      No System Is Safe!
> >      Dongliang Mu
>
>

_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec

Re: Help needed in getting kernel dump in QEMU VM

[Baoquan He]

On 11/15/21 at 03:49pm, Dongliang Mu wrote:
> On Mon, Nov 15, 2021 at 3:27 PM Baoquan He <bhe@redhat.com> wrote:
> >
> > Hi,
> >
> > On 11/13/21 at 10:40am, Dongliang Mu wrote:
> > > Hi all KDUMP maintainers,
> > >
> > > I would like to generate a kernel dump within QEMU VM.
> > >
> > > 1. I reproduced the kernel crash [1] in QEMU VM. The QEMU startup
> > > script is as follows:
> > >
> > > qemu-system-x86_64 \
> > >   -kernel $KERNEL/arch/x86/boot/bzImage \
> > >   -append "console=ttyS0 root=/dev/sda debug earlyprintk=serial slub_debug=QUZ"\
> > >   -hda $IMAGE/stretch.img \
> > >   -net user,hostfwd=tcp::10021-:22 -net nic \
> > >   -enable-kvm \
> > >   -nographic \
> > >   -m 2G \
> > >   -smp 2 \
> > >   -pidfile vm.pid \
> > >   2>&1 | tee vm.log
> > >
> > > The stretch.img is generated by Syzkaller script [1]. -kernel option
> > > is convenient for loading any other kernels.
> > >
> > > 2. As the .config already has the essential
> > > configuration(CONFIG_KEXEC, CONFIG_CRASH_DUMP, CONFIG_DEBUG_INFO), I
> > > did not change this configuration file.
> > >
> > > 3. I installed kdump-tools crash kexec-tools makedumpfile
> > > linux-image-4.9.0-13-amd64 in the stretch.img. Here I installed
> > > linux-image-4.9.0-13-amd64 because there is no default kernel in /boot
> > > directory. And to make kdump-tools working, I modify
> > > /etc/default/kdump-tools in the following:
> > >
> > > KDUMP_INITRD=/boot/initrd.img-4.9.0-13-amd64
> > > KDUMP_KERNEL=/boot/vmlinuz-4.9.0-13-amd64
> >
> > What distros are you using? Asking this because I am sure you are not
> > using Fedora/RHEL OS. The implementation of kdump tools is different in
> > each distros, even though the mechanims in kdump code is the same.
> >
> 
> I am using Debian stretch as the guest OS. So kdump-tools kexec
> makedumpfile is all from Debian.

Then I would sugest asking in a Debian/ubuntu forum or mailing list, figure
out if the configuration or setting is correct. 

I never try Debian OS, can't help, sorry.

> 
> > When we try to get help from upstream, considering and asking good question
> > is very important for getting quick response and effective help.
> >
> > Thanks
> > Baoquan
> >
> > >
> > > 4. I append "crashkernel=384M-:128M" to the command line in the
> > > startup script of QEMU.
> > >
> > > 5. After rebooting, kdump service can start successfully, and the
> > > kdump-config shows:
> > >
> > > root@syzkaller:~# kdump-config show
> > > DUMP_MODE:        kdump
> > > USE_KDUMP:        1
> > > KDUMP_SYSCTL:     kernel.panic_on_oops=1
> > > KDUMP_COREDIR:    /var/crash
> > > crashkernel addr: 0x77000000
> > >    /boot/vmlinuz-4.9.0-13-amd64
> > > kdump initrd:
> > >    /boot/initrd.img-4.9.0-13-amd64
> > > current state:    ready to kdump
> > >
> > > kexec command:
> > >   /sbin/kexec -p --command-line="earlyprintk=serial oops=panic
> > > panic_on_warn=1 nmi_watchdog=panic panic=86400 net.ifnames=0
> > > sysctl.kernel.hung_task_all_cpu_backtrace=1 ima_policy=tcb
> > > kvm-intel.nested=1 nf-conntrack-ftp.ports=20000
> > > nf-conntrack-tftp.ports=20000 nf-conntrack-sip.ports=20000
> > > nf-conntrack-irc.ports=20000 nf-conntrack-sane.ports=20000
> > > vivid.n_devs=16 vivid.multiplanar=1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2
> > > netrom.nr_ndevs=16 rose.rose_ndevs=16 spec_store_bypass_disable=prctl
> > > numa=fake=2 nopcid dummy_hcd.num=8 binder.debug_mask=0
> > > rcupdate.rcu_expedited=1 root=/dev/sda console=ttyS0 vsyscall=native
> > > watchdog_thresh=55 workqueue.watchdog_thresh=140 console=ttyS0
> > > root=/dev/sda debug earlyprintk=serial slub_debug=QUZ irqpoll
> > > nr_cpus=1 nousb systemd.unit=kdump-tools.service
> > > ata_piix.prefer_ms_hyperv=0" --initrd=/boot/initrd.img-4.9.0-13-amd64
> > > /boot/vmlinuz-4.9.0-13-amd64
> > >
> > > 6. When I execute the PoC, the current kernel crashes and then reboots
> > > into the dump-capture kernel. However, the kernel log shows, it is in
> > > emergency mode,
> > >
> > > You are in emergency mode. After logging in, type "journalctl -xb" to view
> > > system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to
> > > try again to boot into default mode.
> > >
> > > Finally, I would like to ask several questions:
> > > 1) is the emergency mode due to the incorrect command line?
> > > 2) is this the right way to generate kernel dump from QEMU VM?
> > > 3) Any comments on the above procedures?
> > >
> > > Thanks very much in advance.
> > >
> > > [1] general protection fault in reiserfs_security_init
> > > (https://syzkaller.appspot.com/bug?id=8abaedbdeb32c861dc5340544284167dd0e46cde)
> > >
> > > [2] https://github.com/google/syzkaller/blob/master/tools/create-image.sh
> > >
> > > --
> > > My best regards to you.
> > >
> > >      No System Is Safe!
> > >      Dongliang Mu
> >
> >
> 


_______________________________________________
kexec mailing list
kexec@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec