[Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

Hello,

I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
without KVM support). The host is also 64 bits. Version of gdb I'm
using is 7.1.

To do that I simply start qemu like the following:

 $ qemu-system-x86_64 -kernel bzImage -append "root=/dev/sda1
console=ttyS0 3" -serial stdio -S -s

and in another term, I start gdb like this:

 $ gdb vmlinux
 $ target remote locahost:1234
 $ b prepare_namespace
 $ continue
 Continuing.
 Remote 'g' packet reply is too long: af75919effffffff0.............

So it fails when gdb stops on the breakpoint with the above message.

If I try to dump the backtrace I got:

 $ bt
 Target is executing.
 $ info thread
 * 1 Thread 1 (CPU#0 [running])  (running)

But the VM seems to be stopped because if I'm asking the status to qemu:

 $ info status
 VM status: paused

I also tried qemu with KVM support but I get one more problem: gdb is
ignoring my breakpoint.

Could anybody help me to make gdb work ?

Thanks.
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Michael Tokarev]

On 22.01.2012 18:20, Francis Moreau wrote:
> Hello,
> 
> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
> without KVM support). The host is also 64 bits. Version of gdb I'm
> using is 7.1.

I'm not sure how it is relevant but at least last (1.0) qemu is
compiled as PIE executable by default, and gdb-7.1 is unable to
debug such executables, PIE support has been intruduced in
gdb-7.2.  You may try recompiling qemu with --disable-pie if
that is the case.

/mjt

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

Hello,

On Sun, Jan 22, 2012 at 6:11 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> On 22.01.2012 18:20, Francis Moreau wrote:
>> Hello,
>>
>> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
>> without KVM support). The host is also 64 bits. Version of gdb I'm
>> using is 7.1.
>
> I'm not sure how it is relevant but at least last (1.0) qemu is
> compiled as PIE executable by default, and gdb-7.1 is unable to
> debug such executables, PIE support has been intruduced in
> gdb-7.2.  You may try recompiling qemu with --disable-pie if
> that is the case.
>

I'm not sure to understand: it seems that you think that I want to
debug qemu itself, but I don't.

I want to debug a kernel running inside a VM.

Am I mis understanding your comment ?

Thanks
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Michael Tokarev]

On 22.01.2012 23:11, Francis Moreau wrote:
[]
> I'm not sure to understand: it seems that you think that I want to
> debug qemu itself, but I don't.
> 
> I want to debug a kernel running inside a VM.
> 
> Am I mis understanding your comment ?

It is the other way around: it was me who misunderstood your question :)

Sorry for the noize.

Thanks,

/mjt

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

On 2012-01-22 15:20, Francis Moreau wrote:
> Hello,
> 
> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
> without KVM support). The host is also 64 bits. Version of gdb I'm
> using is 7.1.
> 
> To do that I simply start qemu like the following:
> 
>  $ qemu-system-x86_64 -kernel bzImage -append "root=/dev/sda1
> console=ttyS0 3" -serial stdio -S -s
> 
> and in another term, I start gdb like this:
> 
>  $ gdb vmlinux
>  $ target remote locahost:1234
>  $ b prepare_namespace
>  $ continue
>  Continuing.
>  Remote 'g' packet reply is too long: af75919effffffff0.............
> 
> So it fails when gdb stops on the breakpoint with the above message.
> 
> If I try to dump the backtrace I got:
> 
>  $ bt
>  Target is executing.
>  $ info thread
>  * 1 Thread 1 (CPU#0 [running])  (running)
> 
> But the VM seems to be stopped because if I'm asking the status to qemu:
> 
>  $ info status
>  VM status: paused
> 
> I also tried qemu with KVM support but I get one more problem: gdb is
> ignoring my breakpoint.
> 
> Could anybody help me to make gdb work ?

When stopping the guest with -S before it booted, gdb will interrupt it
while it is still in 16-bit real mode. Later on, when Linux runs, the
guest is in 64-bit protected mode. gdb is not prepared for such a
switch. All you can do:

- let the guest run until it surely reached 64-bit mode
- interrupt it and set a breakpoint at the desired early-boot location,
  important: if using KVM, set a hardware breakpoint!
- continue and reboot the guest without detaching gdb
- on next boot, the breakpoint will trigger

HTH,
Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Max Filippov]

> When stopping the guest with -S before it booted, gdb will interrupt it
> while it is still in 16-bit real mode. Later on, when Linux runs, the
> guest is in 64-bit protected mode. gdb is not prepared for such a
> switch. All you can do:

Try

set arch i386:x86-64

in the GDB prompt.

-- 
Thanks.
-- Max

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 428 bytes --]

On 2012-01-22 20:16, Max Filippov wrote:
>> When stopping the guest with -S before it booted, gdb will interrupt it
>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>> guest is in 64-bit protected mode. gdb is not prepared for such a
>> switch. All you can do:
> 
> Try
> 
> set arch i386:x86-64
> 
> in the GDB prompt.
> 

Won't help if gdb already connected in 16/32-bit before.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

On Sun, Jan 22, 2012 at 8:21 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2012-01-22 20:16, Max Filippov wrote:
>>> When stopping the guest with -S before it booted, gdb will interrupt it
>>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>>> guest is in 64-bit protected mode. gdb is not prepared for such a
>>> switch. All you can do:
>>
>> Try
>>
>> set arch i386:x86-64
>>
>> in the GDB prompt.
>>
>
> Won't help if gdb already connected in 16/32-bit before.

Why not ? is it a gdb bug ?

Thanks
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 743 bytes --]

On 2012-01-22 20:58, Francis Moreau wrote:
> On Sun, Jan 22, 2012 at 8:21 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2012-01-22 20:16, Max Filippov wrote:
>>>> When stopping the guest with -S before it booted, gdb will interrupt it
>>>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>>>> guest is in 64-bit protected mode. gdb is not prepared for such a
>>>> switch. All you can do:
>>>
>>> Try
>>>
>>> set arch i386:x86-64
>>>
>>> in the GDB prompt.
>>>
>>
>> Won't help if gdb already connected in 16/32-bit before.
> 
> Why not ? is it a gdb bug ?

More than a bug, a deficit in the x86 target management of gdb. Requires
some work, but gdb people are at least aware of the issue.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

On Sun, Jan 22, 2012 at 10:15 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2012-01-22 20:58, Francis Moreau wrote:
>> On Sun, Jan 22, 2012 at 8:21 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>> On 2012-01-22 20:16, Max Filippov wrote:
>>>>> When stopping the guest with -S before it booted, gdb will interrupt it
>>>>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>>>>> guest is in 64-bit protected mode. gdb is not prepared for such a
>>>>> switch. All you can do:
>>>>
>>>> Try
>>>>
>>>> set arch i386:x86-64
>>>>
>>>> in the GDB prompt.
>>>>
>>>
>>> Won't help if gdb already connected in 16/32-bit before.
>>
>> Why not ? is it a gdb bug ?
>
> More than a bug, a deficit in the x86 target management of gdb. Requires
> some work, but gdb people are at least aware of the issue.
>

Thanks again for your help.
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

hello,

On Sun, Jan 22, 2012 at 8:16 PM, Max Filippov <jcmvbkbc@gmail.com> wrote:
>> When stopping the guest with -S before it booted, gdb will interrupt it
>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>> guest is in 64-bit protected mode. gdb is not prepared for such a
>> switch. All you can do:
>
> Try
>
> set arch i386:x86-64
>
> in the GDB prompt.
>

So I started qemu with '-s -S' and connect  gdb to qemu then put the
breakpoint and 'continue'.

gdb stops to the breakpoint and I get: "Remote 'g' packet reply is too
long: feffffff........."

A this point I do "set arch i386:x86-64" but after :

 $ bt
 Target is executing.

I've no idea why gdb think that the target is running. In the qemu's
monitor, doing "info status" reports that the VM is paused.

Thanks
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

Hello,

Thanks for you hints !

On Sun, Jan 22, 2012 at 7:49 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2012-01-22 15:20, Francis Moreau wrote:
>> Hello,
>>
>> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
>> without KVM support). The host is also 64 bits. Version of gdb I'm
>> using is 7.1.
>>
>> To do that I simply start qemu like the following:
>>
>>  $ qemu-system-x86_64 -kernel bzImage -append "root=/dev/sda1
>> console=ttyS0 3" -serial stdio -S -s
>>
>> and in another term, I start gdb like this:
>>
>>  $ gdb vmlinux
>>  $ target remote locahost:1234
>>  $ b prepare_namespace
>>  $ continue
>>  Continuing.
>>  Remote 'g' packet reply is too long: af75919effffffff0.............
>>
>> So it fails when gdb stops on the breakpoint with the above message.
>>
>> If I try to dump the backtrace I got:
>>
>>  $ bt
>>  Target is executing.
>>  $ info thread
>>  * 1 Thread 1 (CPU#0 [running])  (running)
>>
>> But the VM seems to be stopped because if I'm asking the status to qemu:
>>
>>  $ info status
>>  VM status: paused
>>
>> I also tried qemu with KVM support but I get one more problem: gdb is
>> ignoring my breakpoint.
>>
>> Could anybody help me to make gdb work ?
>
> When stopping the guest with -S before it booted, gdb will interrupt it
> while it is still in 16-bit real mode. Later on, when Linux runs, the
> guest is in 64-bit protected mode. gdb is not prepared for such a
> switch. All you can do:
>
> - let the guest run until it surely reached 64-bit mode
> - interrupt it and set a breakpoint at the desired early-boot location,

So I let the kernel boot, and then I'm trying to start and connect gdb
to qemu but unfortunately gdb is segfaulting when trying to connect :(

>  important: if using KVM, set a hardware breakpoint!

ah ok good to know, I'll try to use hw breakpoints.

Thanks
-- 
Francis

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2321 bytes --]

On 2012-01-22 20:25, Francis Moreau wrote:
> Hello,
> 
> Thanks for you hints !
> 
> On Sun, Jan 22, 2012 at 7:49 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>> On 2012-01-22 15:20, Francis Moreau wrote:
>>> Hello,
>>>
>>> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
>>> without KVM support). The host is also 64 bits. Version of gdb I'm
>>> using is 7.1.
>>>
>>> To do that I simply start qemu like the following:
>>>
>>>  $ qemu-system-x86_64 -kernel bzImage -append "root=/dev/sda1
>>> console=ttyS0 3" -serial stdio -S -s
>>>
>>> and in another term, I start gdb like this:
>>>
>>>  $ gdb vmlinux
>>>  $ target remote locahost:1234
>>>  $ b prepare_namespace
>>>  $ continue
>>>  Continuing.
>>>  Remote 'g' packet reply is too long: af75919effffffff0.............
>>>
>>> So it fails when gdb stops on the breakpoint with the above message.
>>>
>>> If I try to dump the backtrace I got:
>>>
>>>  $ bt
>>>  Target is executing.
>>>  $ info thread
>>>  * 1 Thread 1 (CPU#0 [running])  (running)
>>>
>>> But the VM seems to be stopped because if I'm asking the status to qemu:
>>>
>>>  $ info status
>>>  VM status: paused
>>>
>>> I also tried qemu with KVM support but I get one more problem: gdb is
>>> ignoring my breakpoint.
>>>
>>> Could anybody help me to make gdb work ?
>>
>> When stopping the guest with -S before it booted, gdb will interrupt it
>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>> guest is in 64-bit protected mode. gdb is not prepared for such a
>> switch. All you can do:
>>
>> - let the guest run until it surely reached 64-bit mode
>> - interrupt it and set a breakpoint at the desired early-boot location,
> 
> So I let the kernel boot, and then I'm trying to start and connect gdb
> to qemu but unfortunately gdb is segfaulting when trying to connect :(

Try gdb 7.3 or even latest development version (the latter is required
for module debugging - just in case).

> 
>>  important: if using KVM, set a hardware breakpoint!
> 
> ah ok good to know, I'll try to use hw breakpoints.

The reason is that software breakpoints are implemented under kvm by
patching breakpoint instructions into the guest - and those get
overwritten when reloading the kernel after reboot.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [Qemu-devel] Failed to use gdb with qemu 15.1 (with and without kvm support)

[Francis Moreau]

On Sun, Jan 22, 2012 at 8:28 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
> On 2012-01-22 20:25, Francis Moreau wrote:
>> Hello,
>>
>> Thanks for you hints !
>>
>> On Sun, Jan 22, 2012 at 7:49 PM, Jan Kiszka <jan.kiszka@web.de> wrote:
>>> On 2012-01-22 15:20, Francis Moreau wrote:
>>>> Hello,
>>>>
>>>> I'm trying to debug a 64 bits kernel (3.2) using qemu (with and
>>>> without KVM support). The host is also 64 bits. Version of gdb I'm
>>>> using is 7.1.
>>>>
>>>> To do that I simply start qemu like the following:
>>>>
>>>>  $ qemu-system-x86_64 -kernel bzImage -append "root=/dev/sda1
>>>> console=ttyS0 3" -serial stdio -S -s
>>>>
>>>> and in another term, I start gdb like this:
>>>>
>>>>  $ gdb vmlinux
>>>>  $ target remote locahost:1234
>>>>  $ b prepare_namespace
>>>>  $ continue
>>>>  Continuing.
>>>>  Remote 'g' packet reply is too long: af75919effffffff0.............
>>>>
>>>> So it fails when gdb stops on the breakpoint with the above message.
>>>>
>>>> If I try to dump the backtrace I got:
>>>>
>>>>  $ bt
>>>>  Target is executing.
>>>>  $ info thread
>>>>  * 1 Thread 1 (CPU#0 [running])  (running)
>>>>
>>>> But the VM seems to be stopped because if I'm asking the status to qemu:
>>>>
>>>>  $ info status
>>>>  VM status: paused
>>>>
>>>> I also tried qemu with KVM support but I get one more problem: gdb is
>>>> ignoring my breakpoint.
>>>>
>>>> Could anybody help me to make gdb work ?
>>>
>>> When stopping the guest with -S before it booted, gdb will interrupt it
>>> while it is still in 16-bit real mode. Later on, when Linux runs, the
>>> guest is in 64-bit protected mode. gdb is not prepared for such a
>>> switch. All you can do:
>>>
>>> - let the guest run until it surely reached 64-bit mode
>>> - interrupt it and set a breakpoint at the desired early-boot location,
>>
>> So I let the kernel boot, and then I'm trying to start and connect gdb
>> to qemu but unfortunately gdb is segfaulting when trying to connect :(
>
> Try gdb 7.3 or even latest development version (the latter is required
> for module debugging - just in case).

OMG it's working !

One weird thing though: if I put "target remote localhost:1234" in
.gdbinit then I'm getting this in gdb:

   /home/fmoreau/.gdbinit:1: Error in sourced command file:
   Remote 'g' packet reply is too long: c0f6ba3d0088fff....

I've no problem if I don't use .gdbinit.

That's sad that setting arch doesn't work.

>
>>
>>>  important: if using KVM, set a hardware breakpoint!
>>
>> ah ok good to know, I'll try to use hw breakpoints.
>
> The reason is that software breakpoints are implemented under kvm by
> patching breakpoint instructions into the guest - and those get
> overwritten when reloading the kernel after reboot.

Thanks for the information, using hw breakpoint with kvm works !

Thanks a lot !
-- 
Francis