[Qemu-devel] Failed to set a breakpoint on start_kernel

[Qemu-devel] Failed to set a breakpoint on start_kernel

[Wei Yang]

All

I like qemu very much and know it could debug the kernel.

I tried what I searched on web but couldn't stop at the break point.
Below is what I did.

1. Both host and guest installed the same OS, Fedora16 x86_64.

2. Compile the qemu with
    ./configure --target-list=x86_64-softmmu --enable-kvm
--enable-debug-tcg --enable-debug --enable-trace-backend=simple

3. With this command I can boot up my guest.
    ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
1024  -boot dc fedora16.img -monitor stdio

4. I git clone the kernel source in the guest and make a new kernel and initrd.
    I start the guest with this new kernel successfully

5. I copy out the initrd.img and the .config of kernel to host.
compile the kernel on host.
    the kernel source code is identical on host and gueset,

6. I start the guest with the kernel and initrd on host
    ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
1024  -boot dc fedora16.img -monitor stdio -kernel
~/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
~/git/debug/initramfs-3.0.0.img -append
"root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
rd.md=0 rd.lvm.lv=vg_wizard/lv_swap"

    This works fine.

7. Then I start the guest with gdbstub option
    ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
1024  -boot dc fedora16.img -monitor stdio -kernel
/home/ywywyang/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
/home/ywywyang/git/debug/initramfs-3.0.0.img -append
"root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
rd.md=0 rd.lvm.lv=vg_wizard/lv_swap" -S -gdb tcp::4321

    Then the guest stop at the beginning.

8. Attach the gdb in the kernel source directory
    gdb
    file vmlinux
    target remote localhost:4321
    b start_kernel
    c

   Then the guest will run very happily....

    Also use the "info b " could show the break point is set.

Which step I made a mistake?


-- 
Wei Yang
Help You, Help Me

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Mulyadi Santosa]

Hi...

On Fri, Mar 16, 2012 at 09:43, Wei Yang <weiyang.kernel@gmail.com> wrote:
> 8. Attach the gdb in the kernel source directory
>    gdb
>    file vmlinux

that vmlinux, is that the symbol file of the guest running kernel? And
have you compiled the guest kernel with the option to include debug
info and possibly frame pointer info ?

-- 
regards,

Mulyadi Santosa
Freelance Linux trainer and consultant

blog: the-hydra.blogspot.com
training: mulyaditraining.blogspot.com

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2755 bytes --]

On 2012-03-16 03:43, Wei Yang wrote:
> All
> 
> I like qemu very much and know it could debug the kernel.
> 
> I tried what I searched on web but couldn't stop at the break point.
> Below is what I did.
> 
> 1. Both host and guest installed the same OS, Fedora16 x86_64.
> 
> 2. Compile the qemu with
>     ./configure --target-list=x86_64-softmmu --enable-kvm
> --enable-debug-tcg --enable-debug --enable-trace-backend=simple
> 
> 3. With this command I can boot up my guest.
>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> 1024  -boot dc fedora16.img -monitor stdio
> 
> 4. I git clone the kernel source in the guest and make a new kernel and initrd.
>     I start the guest with this new kernel successfully
> 
> 5. I copy out the initrd.img and the .config of kernel to host.
> compile the kernel on host.
>     the kernel source code is identical on host and gueset,
> 
> 6. I start the guest with the kernel and initrd on host
>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> 1024  -boot dc fedora16.img -monitor stdio -kernel
> ~/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
> ~/git/debug/initramfs-3.0.0.img -append
> "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
> rd.md=0 rd.lvm.lv=vg_wizard/lv_swap"
> 
>     This works fine.
> 
> 7. Then I start the guest with gdbstub option
>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> 1024  -boot dc fedora16.img -monitor stdio -kernel
> /home/ywywyang/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
> /home/ywywyang/git/debug/initramfs-3.0.0.img -append
> "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
> rd.md=0 rd.lvm.lv=vg_wizard/lv_swap" -S -gdb tcp::4321
> 
>     Then the guest stop at the beginning.
> 
> 8. Attach the gdb in the kernel source directory
>     gdb
>     file vmlinux
>     target remote localhost:4321
>     b start_kernel
>     c
> 
>    Then the guest will run very happily....
> 
>     Also use the "info b " could show the break point is set.
> 
> Which step I made a mistake?

Two major issues with this procedure:

1. When using kvm, a soft breakpoint (as set by 'b') will inject a trap
instruction into the guest image - which is not yet loaded after the
bios ran. You need to use a hardware breakpoint in this case.

2. Due to gdb limitations, you cannot switch between 16/32-bit mode (the
CPU starts in 16 bit) and the 64-bit mode of kernel within the same gdb
session. Therefore:
 - let the target run into Linux is active
 - attach gdb
 - issue "hw start_kernel"
 - reboot (e.g. "monitor system_reset")
 - you will hit the breakpoint, and gdb will be usable

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Laurent Vivier]

Le samedi 17 mars 2012 à 09:53 +0100, Jan Kiszka a écrit :
> On 2012-03-16 03:43, Wei Yang wrote:
> > All
> > 
> > I like qemu very much and know it could debug the kernel.
> > 
> > I tried what I searched on web but couldn't stop at the break point.
> > Below is what I did.
> > 
> > 1. Both host and guest installed the same OS, Fedora16 x86_64.
> > 
> > 2. Compile the qemu with
> >     ./configure --target-list=x86_64-softmmu --enable-kvm
> > --enable-debug-tcg --enable-debug --enable-trace-backend=simple
> > 
> > 3. With this command I can boot up my guest.
> >     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> > 1024  -boot dc fedora16.img -monitor stdio
> > 
> > 4. I git clone the kernel source in the guest and make a new kernel and initrd.
> >     I start the guest with this new kernel successfully
> > 
> > 5. I copy out the initrd.img and the .config of kernel to host.
> > compile the kernel on host.
> >     the kernel source code is identical on host and gueset,
> > 
> > 6. I start the guest with the kernel and initrd on host
> >     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> > 1024  -boot dc fedora16.img -monitor stdio -kernel
> > ~/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
> > ~/git/debug/initramfs-3.0.0.img -append
> > "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
> > rd.md=0 rd.lvm.lv=vg_wizard/lv_swap"
> > 
> >     This works fine.
> > 
> > 7. Then I start the guest with gdbstub option
> >     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
> > 1024  -boot dc fedora16.img -monitor stdio -kernel
> > /home/ywywyang/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
> > /home/ywywyang/git/debug/initramfs-3.0.0.img -append
> > "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
> > rd.md=0 rd.lvm.lv=vg_wizard/lv_swap" -S -gdb tcp::4321
> > 
> >     Then the guest stop at the beginning.
> > 
> > 8. Attach the gdb in the kernel source directory
> >     gdb
> >     file vmlinux
> >     target remote localhost:4321
> >     b start_kernel
> >     c
> > 
> >    Then the guest will run very happily....
> > 
> >     Also use the "info b " could show the break point is set.
> > 
> > Which step I made a mistake?
> 
> Two major issues with this procedure:
> 
> 1. When using kvm, a soft breakpoint (as set by 'b') will inject a trap
> instruction into the guest image - which is not yet loaded after the
> bios ran. You need to use a hardware breakpoint in this case.
> 
> 2. Due to gdb limitations, you cannot switch between 16/32-bit mode (the
> CPU starts in 16 bit) and the 64-bit mode of kernel within the same gdb
> session. Therefore:
>  - let the target run into Linux is active
>  - attach gdb
>  - issue "hw start_kernel"
>  - reboot (e.g. "monitor system_reset")
>  - you will hit the breakpoint, and gdb will be usable

You can also try my patch :

http://patchwork.ozlabs.org/patch/137543/

Laurent

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

[ re-added qemu-devel to CC ]

On 2012-03-17 13:10, Wei Yang wrote:
>> Two major issues with this procedure:
>>
>> 1. When using kvm, a soft breakpoint (as set by 'b') will inject a trap
>> instruction into the guest image - which is not yet loaded after the
>> bios ran. You need to use a hardware breakpoint in this case.
>>
>> 2. Due to gdb limitations, you cannot switch between 16/32-bit mode (the
>> CPU starts in 16 bit) and the 64-bit mode of kernel within the same gdb
>> session. Therefore:
>>  - let the target run into Linux is active
>>  - attach gdb
>>  - issue "hw start_kernel"
>>  - reboot (e.g. "monitor system_reset")
>>  - you will hit the breakpoint, and gdb will be usable
>>
>> Jan
>>
>>
> oh, so when qemu run with kvm enabled, I couldn't debug the kernel right?

That's not what I said. You need to be aware of how it works. And, in
contrast to pure emulation, kwm uses a non-transparent mechanism for
injecting software breakpoints. Consider it the price for the gained speed.

> 
> I tried to run qemu with out -enable-kvm, kernel could stop at the break point.
> 
> BTW, I tried "hw start_kernel", but it failed.
> (gdb) hw start_kernel
> Undefined command: "hw".  Try "help".

Sorry, typo. Must be "hb".

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 3270 bytes --]

On 2012-03-17 12:25, Laurent Vivier wrote:
> Le samedi 17 mars 2012 à 09:53 +0100, Jan Kiszka a écrit :
>> On 2012-03-16 03:43, Wei Yang wrote:
>>> All
>>>
>>> I like qemu very much and know it could debug the kernel.
>>>
>>> I tried what I searched on web but couldn't stop at the break point.
>>> Below is what I did.
>>>
>>> 1. Both host and guest installed the same OS, Fedora16 x86_64.
>>>
>>> 2. Compile the qemu with
>>>     ./configure --target-list=x86_64-softmmu --enable-kvm
>>> --enable-debug-tcg --enable-debug --enable-trace-backend=simple
>>>
>>> 3. With this command I can boot up my guest.
>>>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
>>> 1024  -boot dc fedora16.img -monitor stdio
>>>
>>> 4. I git clone the kernel source in the guest and make a new kernel and initrd.
>>>     I start the guest with this new kernel successfully
>>>
>>> 5. I copy out the initrd.img and the .config of kernel to host.
>>> compile the kernel on host.
>>>     the kernel source code is identical on host and gueset,
>>>
>>> 6. I start the guest with the kernel and initrd on host
>>>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
>>> 1024  -boot dc fedora16.img -monitor stdio -kernel
>>> ~/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
>>> ~/git/debug/initramfs-3.0.0.img -append
>>> "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
>>> rd.md=0 rd.lvm.lv=vg_wizard/lv_swap"
>>>
>>>     This works fine.
>>>
>>> 7. Then I start the guest with gdbstub option
>>>     ./../qemu/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -smp 4 -m
>>> 1024  -boot dc fedora16.img -monitor stdio -kernel
>>> /home/ywywyang/git/linux-yinghai/arch/x86_64/boot/bzImage -initrd
>>> /home/ywywyang/git/debug/initramfs-3.0.0.img -append
>>> "root=/dev/mapper/vg_wizard-lv_root ro rd.lvm.lv=vg_wizard/lv_root
>>> rd.md=0 rd.lvm.lv=vg_wizard/lv_swap" -S -gdb tcp::4321
>>>
>>>     Then the guest stop at the beginning.
>>>
>>> 8. Attach the gdb in the kernel source directory
>>>     gdb
>>>     file vmlinux
>>>     target remote localhost:4321
>>>     b start_kernel
>>>     c
>>>
>>>    Then the guest will run very happily....
>>>
>>>     Also use the "info b " could show the break point is set.
>>>
>>> Which step I made a mistake?
>>
>> Two major issues with this procedure:
>>
>> 1. When using kvm, a soft breakpoint (as set by 'b') will inject a trap
>> instruction into the guest image - which is not yet loaded after the
>> bios ran. You need to use a hardware breakpoint in this case.
>>
>> 2. Due to gdb limitations, you cannot switch between 16/32-bit mode (the
>> CPU starts in 16 bit) and the 64-bit mode of kernel within the same gdb
>> session. Therefore:
>>  - let the target run into Linux is active
>>  - attach gdb
>>  - issue "hw start_kernel"
>>  - reboot (e.g. "monitor system_reset")
>>  - you will hit the breakpoint, and gdb will be usable
> 
> You can also try my patch :
> 
> http://patchwork.ozlabs.org/patch/137543/

Unless there is a use case beyond this x86 band-aid, lets focus on
getting gdb right. Reminds me that gdb folks asked me to file a bug
about this - which I still need to do. :-/

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Wei Yang]

2012/3/17 Jan Kiszka <jan.kiszka@web.de>:
> [ re-added qemu-devel to CC ]
>
> On 2012-03-17 13:10, Wei Yang wrote:
>>> Two major issues with this procedure:
>>>
>>> 1. When using kvm, a soft breakpoint (as set by 'b') will inject a trap
>>> instruction into the guest image - which is not yet loaded after the
>>> bios ran. You need to use a hardware breakpoint in this case.
>>>
>>> 2. Due to gdb limitations, you cannot switch between 16/32-bit mode (the
>>> CPU starts in 16 bit) and the 64-bit mode of kernel within the same gdb
>>> session. Therefore:
>>>  - let the target run into Linux is active
>>>  - attach gdb
>>>  - issue "hw start_kernel"
>>>  - reboot (e.g. "monitor system_reset")
>>>  - you will hit the breakpoint, and gdb will be usable
>>>
>>> Jan
>>>
>>>
>> oh, so when qemu run with kvm enabled, I couldn't debug the kernel right?
>
> That's not what I said. You need to be aware of how it works. And, in
> contrast to pure emulation, kwm uses a non-transparent mechanism for
> injecting software breakpoints. Consider it the price for the gained speed.
>

Thanks :)
It works.  Though I don't understand it totally, I get the rough idea of it. :)

>>
>> I tried to run qemu with out -enable-kvm, kernel could stop at the break point.
>>
>> BTW, I tried "hw start_kernel", but it failed.
>> (gdb) hw start_kernel
>> Undefined command: "hw".  Try "help".
>
> Sorry, typo. Must be "hb".
>
> Jan
>



-- 
Richard Yang
Help You, Help Me

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Wei Yang]

>> You can also try my patch :
>>
>> http://patchwork.ozlabs.org/patch/137543/
>
> Unless there is a use case beyond this x86 band-aid, lets focus on
> getting gdb right. Reminds me that gdb folks asked me to file a bug
> about this - which I still need to do. :-/
>
> Jan
>

Jan, I didn't try your patch yet.

You mean  this hardware assist break point is just support on x86 now?

-- 
Richard Yang
Help You, Help Me

Re: [Qemu-devel] Failed to set a breakpoint on start_kernel

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 684 bytes --]

On 2012-03-17 15:16, Wei Yang wrote:
>>> You can also try my patch :
>>>
>>> http://patchwork.ozlabs.org/patch/137543/
>>
>> Unless there is a use case beyond this x86 band-aid, lets focus on
>> getting gdb right. Reminds me that gdb folks asked me to file a bug
>> about this - which I still need to do. :-/
>>
>> Jan
>>
> 
> Jan, I didn't try your patch yet.

Not mine, it's Laurent's patch.

> 
> You mean  this hardware assist break point is just support on x86 now?
> 

I mean that the workaround is only required for x86 and will no longer
be needed once gdb can follow target mode switches between 16, 32, and
64 bit without stumbling like now.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]